{"id":11775,"date":"2019-12-23T11:53:31","date_gmt":"2019-12-23T10:53:31","guid":{"rendered":"https:\/\/www.mcomputers.cz\/2019\/12\/23\/vypocetni-cluster-na-uochb-av-cr\/"},"modified":"2020-01-02T19:51:14","modified_gmt":"2020-01-02T18:51:14","slug":"hpc-cluster-at-iocb-prague","status":"publish","type":"post","link":"https:\/\/mcomputers.cz\/en\/2019\/12\/23\/hpc-cluster-at-iocb-prague\/","title":{"rendered":"HPC cluster at IOCB Prague"},"content":{"rendered":"

The Institute of Organic Chemistry and Biochemistry of the CAS (IOCB) uses for its basic research supercomputers to perform calculations in the field of biophysics, focusing primarily on molecular dynamics and quantum chemistry calculations.<\/span> In 2019, the Institute expanded its infrastructure with a computing cluster with a total of 8,532 processor cores.<\/span><\/span><\/strong><\/p>\n

\n

\"UOCHBComputing cluster public procurement
\n<\/span><\/span><\/strong><\/h3>\n

The subject of the public procurement was to supply, install, configure and deploy sets of computing and control servers, storage and other components including additional services for a period of five years. The computing cluster was supposed to be a single functional unit, and its components were to be interconnected with a low-latency data network used to transmit user data and communicate multiprocessor parallel computations. The computing cluster had to allow the installation of the software needed to successfully complete the OPVVV project. The software included at least the following:<\/span><\/p>\n

    \n
  • Free software Gromacs, NAMD, Siesta, Anaconda (Python 3.6+), Octave, PyMol.<\/span><\/li>\n
  • Proprietary software Amber14 +, Gaussian2016 +, Turbomole, VMD and Matlab2018a +, for which the client has already purchased licenses linked to Linux operating system; The cluster had to allow this software to run.<\/span><\/li>\n<\/ul>\n

    \"UOCHBAnother condition for the cluster was to enable fast massively parallel (MPP) computations, i.e. calculations spread across multiple cluster nodes and communicating mainly (but not exclusively) via MPI protocols on a low-latency network using Infiniband or Omni-Path communications.<\/span><\/p>\n

    \n
    Cluster computing nodes had to maximize data throughput because quantum-mechanical and Matlab calculations benefit dramatically from fast access to data in memory. At the same time, computational node processors had to be equipped with a large buffer because molecular dynamics calculations and quantum-mechanical simulations carry out a large number of vector operations, whereby larger caches lead to minimization of “cache misses” and thus dramatically increasing computational speed. At the same time, some of the listed programs depend on vectorization, so for optimal performance it is necessary to use advanced internal processor instructions, such as the AVX512. For Gromacs, NAMD, Amber, and other software relevant to this project, a cluster can significantly improve its performance if CUDA-compatible graphics cards are used for part of the calculation, especially in all-to-all communication(e.g. Gromacs + Plumed or Gaussian).<\/span><\/div>\n
    \n
    \"Intel

    Intel Server Chassis 4 servers \/ 2U<\/p><\/div>\n

    Because many calculations of different users using shared disk space will run simultaneously in a cluster, configuring common storage also had to allow multiple users to simultaneously write and read data without significantly delaying their data transfer. The cluster configuration also had to allow the operation of the above software using native instructions of the installed processors.<\/span><\/div>\n<\/div>\n

    Process of procurement
    \n<\/strong><\/h3>\n
    \n
    \n

    The Institute of Organic Chemistry and Biochemistry of the Academy of Sciences of the Czech Republic requested to deliver the maximum performance of the computing cluster in the given price budget, which amounted to 48 093 507 CZK (2,1 M USD) including VAT.<\/span><\/span><\/p>\n

    The tender was won by M Computers company, which was able to deliver the entire cluster at the required price with the highest computation power and within the deadline.<\/span> M Computers specializes in similar deliveries of computing clusters and has extensive experience with them.<\/span> For example, in 2019, M Computers also delivered the most powerful supercomputer cluster with Intel Xeon processors for SMP, MPP, HPC and AI (artificial intelligence) tasks at Czech Technical University in Prague.<\/span><\/p>\n<\/div>\n<\/div>\n

    \"INTELHPC cluster implementation
    \n<\/strong><\/h3>\n
    \n
    \n

    The solution was based primarily on the Intel platform — processors, servers and high-speed interconnection network. The computing cluster servers are built on a dual-processor Intel Xeon platform in 1U, 2U rack servers (Intel Wolf Pass platform) and four servers in 2U chassis with shared fans and power supplies (Intel Buchanan Pass platform). The processors used in the solution are the Intel Xeon Gold 6100 series, which offered the best price \/ performance ratio. Presence of AVX-512 units is very important for application performance in these processors. The graphics accelerators used are the Nvidia Quadro RTX5000.<\/span><\/p>\n

    Data storage – a disk array, is based on the ZFS file system.<\/p>\n

    Intel Omni-Path technology provides high-speed inter-node communication, namely the Intel Omni-Path Director Class Switch Series 6 Slot Base with 240\u00d7 100 Gbps ports.<\/p>\n

    The hardware part of the custler was implemented in four days by six technicians. The computing cluster had to be integrated into the existing data room. For this reason, servers are divided into six rack cabinets to optimize their power and cooling. It was necessary to create a tailored cluster connections. Part of the handover was also meeting the performance requirements. Gromacs software for molecular dynamics was launched on all servers to verify performance.<\/p>\n<\/div>\n<\/div>\n\n