List of resources on testing distributed systems curated by Andrey Satarin (@asatarin).
Contents
Overview of testing approaches
- Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed Data-Intensive Systems — Great overview of how even simple testing can help a lot, you just need right focus
- What Bugs Live in the Cloud? A Study of 3000+ Issues in Cloud Systems — study of actual bugs in different popular distributed systems (Hadoop MapReduce, HDFS, HBase, Cassandra, ZooKeeper
and Flume)
- TaxDC: A Taxonomy of Non-Deterministic Concurrency Bugs in Datacenter Distributed Systems — comprehensive taxonomy of bugs in distributed systems (Cassandra, Hadoop MapReduce, HBase, ZooKeeper)
- An Empirical Study on Crash Recovery Bugs in Large-Scale Distributed Systems — based on bug database from "What Bugs Live in the Cloud?" paper reseachers focus specifically on crash recovery bugs in Hadoop MapReduce, HBase, Cassandra, ZooKeeper. There is review of this paper by Murat Demirbas in his blog.
- Redundancy does not imply fault tolerance: analysis of distributed storage reactions to single errors and corruptions — study of several distributed systems (Redis, ZooKeeper, MongoDB, Cassandra, Kafka, RethinkDB) on how fault tolerant they are to data corruption and read/write errors
- An empirical study on the correctness of formally verified distributed systems — study of bugs in formally verified distributed systems
- The Case for Limping-Hardware Tolerant Clouds — research on effect of limping hardware on performance of a distributed systems (aka limplock), see also great blog post by Dan Luu on a similiar topic Distributed systems: when limping hardware is worse than dead hardware
- Early detection of configuration errors to reduce failure damage — why and how to test configuration files of your system
- Why Is Random Testing Effective for Partition Tolerance Bugs? — just what it says in a title, authors try to explain why random testing (Jepsen) is effective and introduce notions of test coverage relating to network partition, see also "The Morning Paper" review
- FlyMC: Highly Scalable Testing of Complex Interleavings in Distributed Systems — novel approach of systematically exploring interleavings in distributed systems augmented with static analysis and prioritization. This approach is faster than previous techniques and found old and new bugs in several systems (Cassandra, Ethereum Blockchain, Hadoop, Kudu, Raft LogCabin, Spark, ZooKeeper).
- What bugs cause cloud production incidents? — research focused on bugs (and their resolution strategies) that actually cause production incidents in large-scale distributed services at Microsoft Azure.
- Torturing Databases for Fun and Profit — checking ACID guarantees of open source and commercial databases under power loss, additional material
Technologies for Testing Distributed Systems by Colin Scott
Colin Scott shares his viewpoint from academia on testing distributed systems,
specifically regression testing for correctness and performance bugs.
Testing in a Distributed World by Ines Sombra (RICON 2014)
Great overview of techniques for testing distributed systems. Video is available on Archive
Additional materials could be found in this Github repo
Resilience In Complex Adaptive Systems
These materials are not directly related to testing distributed systems, but they greatly contribute to general understanding of such systems.
State of the art approach to testing stateful distributed systems.
Some notable Jepsen analyses:
Jepsen is used by CockroachDB, VoltDB, Cassandra, ScyllaDB and others.
Companies using TLA+ to verify correctness of algorithms:
Lineage-driven Fault Injection
Netflix adopted lineage-driven fault injection techniques for testing microservices.
Netflix pioneered chaos engineering discipline.
There are two flavors of fuzzing. First, randomized concurrency testing, where the ordering of messages is fuzzed:
And input fuzzing, where message contents or user inputs are fuzzed:
Performance and Benchmarking
See also benchmarking tools.
Specific approaches in different distributed systems
See also formal methods section.
Automated failure injection (see also Lineage-driven Fault Injection):
Random/manual failure injection testing:
See also Chaos Engineering.
They published series of blog posts on testing ScyllaDB:
Series of post on testing at VoltDB:
Additional resources:
CockroachLabs (CockroachDB)
See also formal methods section.
See also formal methods section.
There is also talk from Sean T. Allen on testing stream processing system at Wallaroo Labs (ex. Sendence)
See also formal methods section.
SQLite is not a distributed system by any stretch of the imagination, but provides good example of comprehensive testing of database implementation.
See also formal methods section.