NETL’s Joule Supercomputer is up and running at the Office of Fossil Energy’s National Energy Technology Laboratory (NETL) in Morgantown, West Virginia. One of the world’s fastest, most energy-efficient supercomputers, it is intended to help energy researchers discover new materials, optimize designs, and better predict operational characteristics. Joule is not only on the TOP500 list as one of the top 100 supercomputers in the world, but it is also one of the most energy-efficient.


The system was dedicated by Secretary of Energy Dr. Ernest Moniz in July, 2013. “This new capacity will give us the computational muscle to accelerate the development and design of large scale chemical looping reactors and carbon capture technologies that will allow us to use fossil fuels more cleanly,” said Secretary Moniz. “It will ensure that NETL remains on the forefront of this research, which is critical not only to our economic future but to the environment as well.”

A brief overview:

  • 1512 nodes. Each node has two 8-core 2.6 GHz Intel Sandy Bridge CPUs (24,192 total).
  • Each node is equipped with a 40-Gbps QDR Infiniband network interface and connected to a full-bisection bandwidth Fat-Tree topology Infiniband network. This provides full 40Gbps links among all 1512 nodes and allowed the system to achieve 82% efficiency running High-Performance LINPACK.
  • Total of 9 petabytes of disk storage. 1 petabyte of primary disk storage attached to the compute nodes by Infiniband. Storage is mirrored by an identical 1 petabyte array in B39 at NETL Morgantown.
  • OpenSUSE 11.4 is the current base distribution with specially compiled kernels to support parallel processing.
  • Currently, MFiX, ANSYS/FLUENT, Barracuda, VASP, OpenFOAM and other simulation packages are being used on the system. Post-processing/Visualization packages used include Ensight, Paraview, and VisIt.
  • Primary mechanism for interfacing with the HPCEE is remote login from user workstations. Access via VirtualGL/TurboVNC software that provides a full 3D accelerated graphics interface remotely and makes user logins persistent, providing the same active login in moving from one workstation to another, to a laptop, to a personal system, or even to one of large scale SBEUC special purpose displays.
  • SBEUC Visualization Centers are located at all three NETL sites (Morgantown, Pittsburgh Morgantown. These provide access to high-resolution tiled displays, large screen LCDs, and collaboration workstations. Also, these systems are closed tied to the large-memory visualization nodes to speed post-processing.

Housed at NETL’s Simulation-Based Engineering User Center, the Joule Supercomputer is a 503 TFlops (trillion floating-point operations per second) computer that enables the simulation of phenomena that are difficult or impossible to measure,
sbeuc-timessuch as coal jet penetration into a gasifier. With capabilities for running modelling tools at various scales ranging from molecules, to devices, to entire power plants and natural fuel reservoirs, the HPCEE is the backbone providing enhanced visualization, data analysis, and data storage capabilities for NETL’s energy research efforts.

All of the computational, visualization, network hardware, and primary storage servers are installed in a Silicon Graphics ICE Cube Air modular data center. This data centre provides the HPCEE one of the lowest power utilization efficiency (PUE) infrastructures available, with a range of 1.03 to 1.07 PUEs. To date, the HPCEE has one of the lowest recorded PUEs achieved in the industry, using only one percent of total electrical consumption to cool the equipment – far surpassing the DOE Office of the Chief Information Officer’s standard of 40 percent. The increase in efficiency translates to electrical energy cost savings of approximately $450,000 annually.