Blue Waters
By early December, we expect a single-cabinet test system ("Phase 0") to be delivered to NCSA, and plan to allow participating science team members access to it during the Fall PRAC Workshop (Dec. 13). This system will have 52 XE6 compute nodes (832 AMD Interlagos "Bulldozer" cores) and 32 XK6 compute nodes (256 Interlagos cores plus 32 Fermi accelerators), all in the same interconnect fabric.
Early next year, a substantial (>1.3 PFLOP peak) system ("Phase 1") will be delivered to NCSA for early science runs and possibly some porting and tuning. This system will be available for a few months, after which it will be integrated with the rest of the cabinets comprising the full Blue Waters system. By early January, we will send out detailed information on the process for getting access to the early science system.
The table below summarizes the key parameters of the various Phases of the Blue Waters system. Phase 2 is expected to go into production in Summer 2012, and in Fall 2012, the Phase 3 system will have Kepler accelerators installed in the XK nodes. For more details, see the newly revised Blue Waters PRAC wiki page:
https://wiki.ncsa.illinois.edu/display/BWpublic/User+Information
System Components UIUC/NCSA Blue Waters Phase I System UIUC/NCSA Blue Waters Phase II System UIUC/NCSA Blue Waters Phase III System System XE6 XE6/XK6 XE6/XK7 Cabinets – XE6 47 237 237 Cabinets – XK6 0 32 32 Cabinets – XIO 1 7 7 x86 Peak Performance (PF) 1.41 7.62 7.62 GPU Peak Performance (PF) 0 0 > 4 Peak Performance (PF) 1.41 7.62 >11.5 Compute Nodes – XE6 4,512 22,752 22,752 Compute Nodes – XK6 0 3,072 3,072 x86 Compute Cores 72,192 388,608 388,608 Service and I/O Nodes (total) 96 672 672 - LNET Routers 50 582 582 - Network 4 8 8 - Boot 1 2 2 - SDB 1 2 2 - Internal Only XIO nodes 2 4 4 - Unassigned 38 74 74 XE6 Compute Node Perf (GF) 313.6 313.6 313.6 XK6 Compute Node (GF) NA 156.8 (CPU) 156.8(CPU) + >1,000(GPU) Compute Node Memory (XE6/XK6) 64/32 64/32 64/32 XE6 Compute Node Memory Bandwidth (GB/s) 102.4 102.4 102.4 XK6 Compute Node Memory Bandwidth (GB/s) 51.2 51.2 51.2+>140 Total System Memory (TB) 282 1,518 1,518 Interconnect Architecture 3D Torus 12x8x24 3D Torus 23x24x24 3D Torus 23x24x24 Peak Node Injection Bandwidth (GB/s) 9.6 9.6 9.6 Average Bisection Bandwidth (TB/s) 8.65 17.55 17.55 Minimum Bisection Bandwidth (TB/s) 7.50 10.35 10.35 Peak Interconnect Injection Bandwidth (GB/s) 44.24 254.36 254.36 Storage Cabinets >2 >30 >30 Total Raw Storage (PB) >2.5 >30 >30 Total Usable Storage (PB) >2 >25 >25 Storage SSUs Disks >17000 >17000 External Login Nodes 2 4 4 External Data Movers (esDM) 40 100 100 Peak Power (MW) 2.7 15.7 15.7 Power (Linpack) (MW) 2.4 < 13.6 13.6 Power (Applications) (MW) 2.2 < 12.6 12.75
