IBM Research
Network Progress Calls
NAMD makes progress engine calls from the compute loops
– Typical frequency is10000 cycles, dynamically tunable
for ( i = 0; i < (i_upper SELF(- 1)); ++i ){
CmiNetworkProgress();
const CompAtom &p_i = p_0[i]; //……………………………
//Compute Pairlists
for (k=0; k<npairi; ++k) { //Compute forces
}
}
void CmiNetworkProgress() { new_time = rts_get_timebase();
if(new_time < lastProgress + PERIOD) { lastProgress = new_time;
return;
}
lastProgress = new_time; AdvanceCommunication();
}
Corporation
IBM Research
MPI Scalability
Charm++ MPI Driver
–Iprobe based implementation
–Higher progress overhead of MPI_Test
–Statically pinned FIFOs for point to point communication
32 |
© 2005 IBM Corporation |
|
IBM Research
Charm++ Native Driver
BGX Message Layer (developed by George Almasi)
–Lower progress overhead
–Active messages
• Easily design complex communication protocols
–Dynamic FIFO mapping
–Low overhead remote memory access
–Interrupts
–Charm++ BGX driver was developed by Chao Huang over this summer
33 |
© 2005 IBM Corporation |
|
IBM Research
BG/L Msglayer
|
|
|
|
Messages |
|
|
|
|
Msg Queues |
||||||||
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|||
SpadMessage |
|
TreeMessage |
TorusMessage |
|
Collective |
|
|||||||||||
|
|
|
|
|
|
|
|
|
|
|
|
|
|
p |
Msq queue |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
o |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
s |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
t |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Torus |
|
|
|
|
Packets |
|
|
|
|
|
|
|
|
Msq queue |
|
|||||
|
|
|
|
|
|
|
|
|
|
|
|
|
|
0 |
|
|
|
|
TreePacket |
|
|
TorusPacket |
|
|
|
|
|
|
|
||||||
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
FIFO |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
||
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2 |
|
|
|
|
|
Dynamically |
|
|
Deterministically |
|
|
|
pinning |
|||||||
|
|
|
routed packet |
|
|
routed packet |
|
|
|
… |
|
|
|||||
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
n-1 |
|
|
|
|
|
Templates |
|
|
|
|
|
|
|
|||||||
|
|
|
|
|
|
|
|
|
|
||||||||
|
|
|
|
|
|
|
|
Scratchpad |
|
||||||||
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Msq queue |
|
|
|
|
|
|
|
|
|
TorusDirectMessage<> |
|
|
|
|
|
|||||
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Advance loop
|
|
|
|
ts |
|
|
|
e |
|
|
|
k |
|
|
|
c |
|
|
|
a |
|
|
|
|
p |
|
|
|
|
( This slide is taken from G. Almási’s talk on the “new” msglayer. )
Network
Coll. network FIFO
Torus FIFOs
I0 0 
1 
2 
H
I1 0 
1 
2 
H
R0 x+
x- y+
y- z+
z-
H
R1 x+
x- y+
y- z+
z-
H
Dispatching
Torus pkt. registry
0
1
2
…
p
Coll. pkt. disp.
34 |
© 2005 IBM Corporation |
|
IBM Research
Optimized Multicast
pinFifo Algorithms
–Decide which of the 6 FIFOs to use when send msg to {x,y,z,t}
–Cones, Chessboard
Dynamic FIFO mapping
–A special send queue that msg can go from whichever FIFO that is not full
35 |
© 2005 IBM Corporation |
|
IBM Research
Communication Pattern in PME
108 
procs
108 procs
36 |
|
© 2005 IBM Corporation |
|
||
|
|
IBM Research
PME
Plane decomposition for 3D-FFT
PME objects placed close to patch objects on the torus
PME optimized through an asynchronous all-to-all with dynamic FIFO mapping
37 |
© 2005 IBM Corporation |
|
IBM Research
Performance Results
© 2005 IBM Corporation
IBM Research
BGX Message layer vs MPI
Fully non-blocking version performed below par on MPI
– Polling overhead high for a list of posted receives
BGX message layer works well with asynchronous communication
# Nodes |
Cutoff |
|
with PME |
||
Msglayer |
MPI* |
Msglayer |
MPI* |
||
|
|||||
APoA1 Benchmark
4 |
2250 |
2250 |
|
|
32 |
314 |
316 |
356 |
371 |
128 |
85 |
91.6 |
103 |
|
512 |
22.7 |
23.8 |
26.7 |
27.8 |
1024 |
13.2 |
13.9 |
14.4 |
17.3 |
2048 |
7.9 |
8.1 |
9.7 |
10.2 |
4096 |
4.8 |
4.9 |
6.8 |
7.3 |
NAMD Co-Processor Mode Performance (ms/step)
Message layer has sender side blocking communication here
39 |
© 2005 IBM Corporation |
|
IBM Research
Blocking vs Overlap
|
Cutoff |
|
with PME |
|
# Nodes |
Blocking Sender |
Non-Blocking |
Blocking Sender |
Non-Blocking |
|
||||
32 |
314 |
313 |
356 |
347 |
128 |
85 |
82 |
103 |
97.2 |
512 |
22.7 |
21.7 |
26.7 |
23.7 |
1024 |
13.2 |
11.9 |
14.4 |
13.8 |
2048 |
7.9 |
7.3 |
9.7 |
8.6 |
4096 |
4.8 |
4.3 |
6.8 |
6.2 |
8192 |
- |
3.7 |
- |
- |
APoA1 Benchmark in Co-Processor Mode
40 |
© 2005 IBM Corporation |
|