In computing, single program, multiple data (SPMD) is a term that has been used to refer to computational models for exploiting parallelism whereby multiple processors cooperate in the execution of a program in order to obtain results faster.
The term SPMD was introduced in 1983 and was used to denote two different computational models:
- by Michel Auguin (University of Nice Sophia-Antipolis) and François Larbey (Thomson/Sintra), as a "fork-and-join" and data-parallel approach where the parallel tasks ("single program") are split-up and run simultaneously in lockstep on multiple SIMD processors with different inputs, and
- by Frederica Darema (IBM), where "all (processors) processes begin executing the same program... but through synchronization directives ... self-schedule themselves to execute different instructions and act on different data" and enabling MIMD parallelization of a given program, and is a more general approach than data-parallel and more efficient than the fork-and-join for parallel execution on general purpose multiprocessors.
The (IBM) SPMD is the most common style of parallel programming and can be considered a subcategory of MIMD in that it refers to MIMD execution of a given ("single") program. It is also a prerequisite for research concepts such as active messages and distributed shared memory.
| Flynn's taxonomy |
|---|
|
Single data stream |
|---|
|
|
|
Multiple data streams |
|---|
|
|
|
SIMD subcategories |
|---|
|
|
|
See also |
|---|
|
|
M. Auguin; F. Labrey (1985). "A Multi-processor SIMD Machine: OPSILA". In K. Waldschmidt; B. Myhrhaug (eds.).
Microcomputers, usage and design. Amsterdam: North-Holland.
ISBN 0-444-87835-1.
1 2 F. Darema-Rogers; D. A. George; V. A. Norton; G. F. Pfister (23 January 1985). "A VM Parallel Environment". IBM Research Report RC 11225 (Report). IBM T. J. Watson Research Center.
1 2 3 4 5 Flynn, Michael J. (September 1972). "Some Computer Organizations and Their Effectiveness" (PDF). IEEE Transactions on Computers. C-21 (9): 948–960. doi:10.1109/TC.1972.5009071.
↑ "Intel® ISPC User's Guide § The ISPC Parallel Execution Model". ispc.github.io.
1 2 "PyTorch/XLA SPMD User Guide — PyTorch/XLA master documentation". docs.pytorch.org.
↑ "OpenMP Support — Clang 14.0.0 documentation". releases.llvm.org.
↑ Kandiah, Vijay; Lustig, Daniel; Villa, Oreste; Nellans, David; Hardavellas, Nikos (17 February 2023). Parsimony: Enabling SIMD/Vector Programming in Standard Compiler Flows. CGO 2023: Proceedings of the 21st ACM/IEEE International Symposium on Code Generation and Optimizati. pp. 186–198. doi:10.1145/3579990.3580019.
↑ Darema, Frederica (1986). "Parallel Applications Development for Shared Memory Systems". IBM Research Report RC12229 (Report). Yorktown Heights, NY: IBM T. J. Watson Research Center.
↑