High performance computing

Learning Results

The course aims at the presentation of modern parallel systems and architectures and their main techniques and programming tools. It also has the purpose to teach integrated parallelization techniques for shared memory, distributed memory, GPU-based and hybrid massively parallel environments, giving emphasis to the analysis, design and implementation of corresponding algorithms and programs for advanced computational problems in several scientific areas, including artificial intelligence and data science. An additional important goal is to provide students with the necessary knowledge and skills for developing efficient parallel applications in modern parallel systems, and the application of all the above in areas of real problems and further research.

Upon successful completion of the course, the student will be able to:

• Identify the most important of the modern categories / architectures of parallel systems and describe their main characteristics.
• Describe the different models of parallel programming nowadays, and realize their differences and their main advantages.
• Understand and further apply efficient parallelization techniques in shared memory, distributed memory and hybrid environments.
• Design efficient parallel algorithms for advanced scientific topics over shared memory, distributed memory, GPU-based, and hybrid platforms.
• Utilize modern parallel programming tools to develop efficient parallel applications.
• Program in modern parallel platforms, such as multicore systems, multi-computers, GPUs / co-processors, etc.
• To measure the performance of parallel programs and applications with modern metrics, techniques and tools.

Skills acquired

• Retrieve, analyse and synthesise data and information by utilising necessary technologies
• Decision-Making
• Work independently / Teamwork
• Advance of free, creative and inductive thinking
• Adapt to new situations
• Work in an interdisciplinary environment

Generate new research ideas

• Modern parallel systems and architectures, supercomputers, multi-core systems, clusters, hybrid systems and architectures.
• Many-core architectures and state-of-the-art GPU-accelerators and co-processors.
• Parallel programming techniques for shared memory, distributed memory, GPU-based, and hybrid environments.
• Parallel algorithms for advanced computational problems (including the areas of artificial intelligence and data science) in massively parallel environments.
• Multi-threaded programming in shared memory systems (using OpenMP).
• Programming of GPUs and co-processors (using the CUDA programming model, OpenCL standard, SIMD programming etc.).
• Measuring the Performance of Parallel Programs.
• Application Development using modern parallel programming tools in hybrid parallel environments (i.e. OpenMP, CUDA, OpenCL, AVX extensions, hybrid programming including MPI, etc.)

• Peter S. Pacheco and Matthew Malensek, An Introduction to Parallel Programming, 3rd edition, Morgan Kaufmann, 2021.
• Kirk D.B., Hwu W.M., Programming Massively Parallel Processors: A Hands-on Approach, 3rd edition, Morgan Kaufmann, 2016.
• Quinn M.J., Parallel Programming in C with MPI and OpenMP, McGraw-Hill, 2003.
• Wilkinson B., Allen M., Parallel Programming – Techniques and Applications Using Networked Work stations and Parallel Computers, Pearson, Prentice Hall, 2006.
• James Jeffers, James Reinders, and Avinash Sodani, Ιntel Xeon Phi Processor High Performance Programming, Morgan Kaufmann, 2nd Edition, 2016. 
• A. Grama, G. Karypis, V. Kumar, A. Gupta, Introduction to Parallel Computing, 2nd Edition, Addison-Wesley, 2003.
• F.T. Leighton, “Introduction to Parallel Algorithms and Architectures: Arrays, Trees, Hypercubes”, Morgan Kaufmann, San Mateo, CA, 1992.
• LLNL OpenMP Tutorial, https://hpc-tutorials.llnl.gov/openmp/
• OpenCL Specification, https://www.khronos.org/registry/OpenCL/specs/opencl-2.1.pdf
• NVIDIA CUDA Tutorial: http://www.nvidia.com/docs/IO/116711/sc11-cuda-c-basics.pdf

Related scientific journals:

1. Transactions on Parallel and Distributed Systems, IEEE
2. International Journal of Parallel Programming, Springer
3. Journal of Parallel and Distributed Computing, Elsevier
4. Transactions on Parallel Computing, ACM