HPC cluster at IOCB Prague
The Institute of Organic Chemistry and Biochemistry of the CAS (IOCB) uses for its basic research supercomputers to perform calculations in the field of biophysics, focusing primarily on molecular dynamics and quantum chemistry calculations. In 2019, the Institute expanded its infrastructure with a computing cluster with a total of 8,532 processor cores.
Computing cluster public procurement
The subject of the public procurement was to supply, install, configure and deploy sets of computing and control servers, storage and other components including additional services for a period of five years. The computing cluster was supposed to be a single functional unit, and its components were to be interconnected with a low-latency data network used to transmit user data and communicate multiprocessor parallel computations. The computing cluster had to allow the installation of the software needed to successfully complete the OPVVV project. The software included at least the following:
- Free software Gromacs, NAMD, Siesta, Anaconda (Python 3.6+), Octave, PyMol.
- Proprietary software Amber14 +, Gaussian2016 +, Turbomole, VMD and Matlab2018a +, for which the client has already purchased licenses linked to Linux operating system; The cluster had to allow this software to run.
Another condition for the cluster was to enable fast massively parallel (MPP) computations, i.e. calculations spread across multiple cluster nodes and communicating mainly (but not exclusively) via MPI protocols on a low-latency network using Infiniband or Omni-Path communications.
Intel Server Chassis 4 servers / 2U
Because many calculations of different users using shared disk space will run simultaneously in a cluster, configuring common storage also had to allow multiple users to simultaneously write and read data without significantly delaying their data transfer. The cluster configuration also had to allow the operation of the above software using native instructions of the installed processors.
Process of procurement
The Institute of Organic Chemistry and Biochemistry of the Academy of Sciences of the Czech Republic requested to deliver the maximum performance of the computing cluster in the given price budget, which amounted to 48 093 507 CZK (2,1 M USD) including VAT.
The tender was won by M Computers company, which was able to deliver the entire cluster at the required price with the highest computation power and within the deadline. M Computers specializes in similar deliveries of computing clusters and has extensive experience with them. For example, in 2019, M Computers also delivered the most powerful supercomputer cluster with Intel Xeon processors for SMP, MPP, HPC and AI (artificial intelligence) tasks at Czech Technical University in Prague.
HPC cluster implementation
The solution was based primarily on the Intel platform — processors, servers and high-speed interconnection network. The computing cluster servers are built on a dual-processor Intel Xeon platform in 1U, 2U rack servers (Intel Wolf Pass platform) and four servers in 2U chassis with shared fans and power supplies (Intel Buchanan Pass platform). The processors used in the solution are the Intel Xeon Gold 6100 series, which offered the best price / performance ratio. Presence of AVX-512 units is very important for application performance in these processors. The graphics accelerators used are the Nvidia Quadro RTX5000.
Data storage – a disk array, is based on the ZFS file system.
Intel Omni-Path technology provides high-speed inter-node communication, namely the Intel Omni-Path Director Class Switch Series 6 Slot Base with 240× 100 Gbps ports.
The hardware part of the custler was implemented in four days by six technicians. The computing cluster had to be integrated into the existing data room. For this reason, servers are divided into six rack cabinets to optimize their power and cooling. It was necessary to create a tailored cluster connections. Part of the handover was also meeting the performance requirements. Gromacs software for molecular dynamics was launched on all servers to verify performance.
Configuration of HPC cluster
- 1× control server
- 8× CPU compute node (768 GB RAM)
- 20× CPU compute node (192 GB RAM)
- 192× CPU compute node(96 GB RAM)
- 5× GPU compute node (2× GPU akcelerátor)
- 2× storage / disc array
- Intel Omni-Path Architecture network
- communication and data network
- software
Specification of the whole solution:
- Number of all compute nodes: 237
- Number of all compute cores: 8 532
- Total RAM size: 30 TB
- Number of GPU accelerators: 10
- 1 176 meters of optical cables
- 1 122 connectors connected
- price cca 2,1 M USD inc. VAT
Customer – Institute of Organic Chemistry and Biochemistry AS CR, v. I.
The mission of the Institute is independent basic research (with the purpose of gathering new knowledge) in the fields of organic chemistry and biochemistry, and molecular oriented disciplines, which in organic chemistry and biochemistry oriented, with a strong emphasis on the application of the results in practice. Research is focused mainly on the following areas: nucleic acid components, proteins, peptides, natural products, functional synthetic molecules and molecular modeling. The focus of the Institute is to achieve excellence in international competition and to maintain this position in the long term. More at: https://www.uochb.cz/en
Supplier – M Computers s.r.o.
M Computers s.r.o. is a purely Czech private technology company that has been successfully operating on the IT market since 2002 when it transformed from a group of individuals operating since 1996. In 2003, it became a founding member of the nationwide Orange & Green network and co-owner of this brand.
The main products include the supply of computing clusters, mass storage, servers and data storage. It also provides information technology outsourcing and operates a specialized e-shop LenoShop.cz. M Computers is a partner of powerful companies such as Lenovo, Intel, Nvidia, IBM, HPE, Supermicro and Infortrend. He is currently one of the leaders in HPC, supercomputers and artificial intelligence. More at: https://mcomputers.cz/en/
