Experimental HPC systems at IT4Innovations

Experimental HPC and AI systems at IT4Innovations

10.11.2022

In addition to its large supercomputers (list), IT4Innovations also operates smaller complementary systems. These systems represent emerging, non-traditional or highly specialized hardware architectures not found in conventional supercomputing data centers. In the last days of October, Complementary Systems I was put into operation.

Compute partition 1 – Arm A64FX processors

The compute nodes of the first part of Complementary Systems I are built on Arm A64FX processors with integrated fast HBM2 memory. It is essentially a fragment of the world’s most powerful supercomputer in recent years, Fukagu, at Japan’s RIKEN Center of Computational Science (currently the second most powerful supercomputer). The configuration consists of eight HPE Apollo 80 compute nodes interconnected by a 100Gb/s Infiniband network. Compute node configuration:

1× Arm A64FX, 48 cores, 2 GHz, 32GB HBM2 memory
400 GB SSD
HDR Infiniband 100 Gb/s

New programming models, libraries and application development tools are also deployed in complementary systems to extract maximum performance from the hardware. Complementary systems will thus provide scientific teams with the opportunity to test and compare experimental architectures with traditional ones (x86 + Nvidia GPUs) and the opportunity to optimize and accelerate computation in new research areas. The Complementary Systems I project, which went into operation in October, consists of several parts. They are based on servers from Hewlett Packard Enterprise and were delivered and implemented by M Computers from Brno.

Complementary systems at IT4I

Information about complementary systems projects on IT4Innovations.

More information

Compute partition 2 – Intel processors, Intel PMEM, Intel FPGA (Altera)

The compute nodes in this part of the Complementary Systems are based on Intel technologies. The servers are equipped with third-generation Intel Xeon processors, persistent (non-volatile) Intel Optane memory with a total capacity of 2TB and 8TB per server, and Intel Stratix 10 MX2100 FPGA cards. This part consists of two HPE ProLiant DL380 Gen 10 Plus nodes in the configuration:

2× Intel Xeon Gold 6338, 32 cores, 2 GHz
256 GB RAM
8 TB and 2 TB Intel Optane Persistent Memory (NVDIMM)
2 TB NVMe SSD
2× FPGA Bittware 520N-MX (Intel Stratix 10)
HDR Infiniband 100 Gb/s

Compute partition 3 – AMD processors, AMD accelerators, AMD FPGAs (Xilinx)

The third part of Complementary Systems is a figure on AMD technologies. The servers are equipped with third-generation AMD EPYC processors, four AMD Instinct MI100 GPU cards interconnected by a fast bus (AMD Infinity Fabric) and two Xilinx Alveo FPGA cards with different performance. Xilinx is one of AMD’s latest major acquisitions. This part consists of two HPE Apollo 6500 Gen 10+ nodes in the configuration:

2× AMD EPYC 7513, 32 cores, 2.6 GHz
256 GB RAM
2 TB NVMe SSD
4× AMD Instinct MI100 (AMD Infinity Fabric Link)
FPGA Xilinx Alveo U250
FPGA Xilinx Alveo U280
HDR Infiniband 100 Gb/s

Software equipment

An important part of Complementary Systems I is software, which includes environments, compilers, numerical libraries or tools for algorithm development and debugging.

HPE Cray Programming Environment

HPE Cray Programming Environment is a comprehensive tool for developing HPC applications in heterogeneous environments. It supports all Complementary Systems architectures. It includes optimized libraries, support for the most widely used programming languages, and several tools for analyzing, debugging, and optimizing parallel algorithms. HPE support for the tools included in the Cray PE package is included.

Intel oneAPI

OneAPI is Intel’s tool for developing applications deployed on heterogeneous platforms – e.g. CPUs, GPUs, FPGAs and more. In Complementary Systems, it is planned to be used mainly for FPGA cards.

AMD ROCm

ROCm is an AMD software package that includes programming models, development tools, libraries, or integration tools for the most widely used AI frameworks that run on top of AMD GP GPU accelerators.