Parallel Processing

Parallel Processing §

• Parallel computing is a computing where the jobs are broken into discrete parts that can be executed concurrently.

  • Parallel systems deal with the simultaneous use of multiple computer resources that can include:
    • A single computer with multiple processors
    • Several computers connected by a network to form a parallel processing cluster (distributed processing) or
    • A combination of both

• There are two basic elements to each architecture

• A control unit (CU) is responsible for the fetching and decoding of instructions, passing to a PU/PE for processing

  • Two instructions streams requires two CUs, one to handle each stream

• A processing unit (PU) (aka processing element PE) is something that works on streams (data and instructions) (e.g. ALU)

  • If you have two data streams, or two instructions streams you need two processing units

• Four categories:

  • SISD
    • S ingle I nstruction S ingle D ata
      • One instruction stream and one data stream
    • A single CU fetches a single instruction stream from memory
    • The CU then directs control signals to the single PU to operate the data stream
      • One operation at a time
    • This is a conventional computer (Von Neumann architecture)
  • SIMD
    • Single Instruction stream, Multiple Data stream
      • Single CU and multiple Pus
    • CU fetches an instruction from memory and after decoding broadcasts control signals to all PUs
      • A global clock drives all PUs
      • That is, at any given time, all PUs are synchronously executing the same instruction but on different sets of data
    • Typically found in graphics card or vector processing e.g. shading pixels
  • MISD
    • Multiple Instruction stream, Single Data stream
    • Multiple processors each executing its own instruction stream to process the same data
  • MIMD
    • Multiple Instructions streams, Multiple Data streams
      • Multiple processors each executing its own instruction stream to process the data stream allocated to it
      • MIMD exploits asynchronous parallelism
        • Instructions and data processed independently
      • A MIMD is a true multiprocessor, in contrast to SIMD
      • Typical, modern day processors use this architecture as well as most parallel computer systems

• Overview

  • A massively parallel system is defined as the use of many processors ( NOT CORES ) or separate computers.

• Communication

  • The major difference in architecture is that massively parallel computers systems use a network infrastructure to support multiple processing units, rather than a bus structure
    • If processors share the same bus, it cannot be considered massively parallel, neither can it if there are small number of processors
    • Processor cores share a bus, so are automatically ruled out
  • Programs running on this architecture communicate by passing messages using the network

• Summary

  • Hardware
    • The network architecture linking each processing core needs to be resilient and capable of shifting large amounts of data
  • Software
    • Software must be written that can make use of the vast number of core. Not all tasks are suitable for massively parallel systems
      • 1 woman can have a baby in 9 months, 9 women cannot have a baby in 1 month
    • Examples of suitable tasks are those that can be broken apart and executed independently. That is asynchronous operations.