· 5 min read

Understanding the CAP theorem

I've been organizing my thoughts on system design, specifically the CAP theorem, which comes up constantly in engineering discussions. I wanted to turn my handwritten notes into a reference on the actual trade-offs we face when building distributed systems.

This is the breakdown, straight from my notebook.

My notes on CAP theorem

The CAP theorem states that in a distributed system, you can have only 2 out of 3 of the following properties:

  1. Consistency: all nodes see the same data at the same time. When a write is made to one node, all subsequent reads from any node will return that updated value.
  2. Availability: every request to a non-failing node receives a response, without a guarantee that it contains the most recent data.
  3. Partition tolerance: the system continues to operate despite arbitrary message loss or failure of part of the system (i.e., network partitions between nodes). Even if the network is down, the system keeps working.

In any distributed system, partition tolerance is a must. Network failures will happen, and your system needs to handle them. Since partition tolerance is non-negotiable, the CAP theorem really comes down to one choice: when a network partition happens, do you prioritize consistency or availability?

Note: Sometimes showing stale data is better than showing no data at all.

Some systems can never show stale data

Some systems absolutely require consistency, even at the cost of availability.

  • Ticket booking (e.g., BookMyShow): Imagine User A is booking a seat. Due to a network issue, User B selects the same seat. If the system is available but inconsistent, you now have 2 people showing up for the same seat.
  • E-commerce inventory: You can't show the wrong stock numbers.
  • Financial systems: Stock trading platforms need to show up-to-date order book entries.

Most systems can live with a little staleness

In most systems, we can tolerate some inconsistency and should prioritize availability.

  • Social media: If User A updates their profile picture, it's perfectly fine if User B sees the old picture for some time.
  • Netflix: If a description is updated, it doesn't have to be updated instantly everywhere.
  • Review sites: Again, slight delays in data propagation are acceptable.

The test I use is whether it would be catastrophic if users briefly saw inconsistent data. If it would, choose consistency, and if it wouldn't, choose availability.

You can decide feature by feature

As systems grow you can also mix the two and decide feature by feature.

  • BookMyShow: Seat selection needs strong consistency, but viewing event descriptions or general details can prioritize availability (showing outdated data is acceptable).
  • Tinder: Matching needs consistency (if both users swipe right at the same time, they need to match). A slightly outdated profile is acceptable, though (e.g., a user just uploaded a new image).

If you prioritize consistency

  1. Distributed transactions: making multiple data nodes (cache & DB) store data and stay in sync adds complexity. It guarantees consistency across all nodes, but users will likely see higher latency while the system makes sure the data is consistent everywhere.

  2. Single node solutions: using a single DB instance avoids propagation issues. It limits scalability, but having a single source of truth eliminates consistency challenges (e.g., PostgreSQL, SQL, Dynamo, Google Spanner).

If you prioritize availability

An availability-first design can include:

  1. Multiple replicas: adding read replicas with asynchronous replication lets reads be served from any replica, even if it's slightly behind. This improves read performance and availability at the cost of potential staleness.

    • Tech: Redis Clusters, DynamoDB, Cassandra.
  2. CDC (Change Data Capture): it tracks changes in the primary DB and syncs the data asynchronously, streaming changes to a target system like a data warehouse or another application.

Consistency isn't all or nothing

Consistency comes in levels:

  • Strong consistency: all reads reflect the most recent writes. This is the most expensive consistency model in terms of performance. (Used in banks, shopping inventory.)
  • Causal consistency: related events appear in the same order to all users. If Process A communicates a value to Process B, any subsequent actions by Process B will happen after Process A's action. (Used in social media comment threads and collaborative applications.)
  • Read-your-own-writes consistency: a user always sees their own updates first, though other users might see older versions. (e.g., profile updates, where a user might click update again and again if they don't see their change.)
  • Eventual consistency: used for faster, "optimistic" writes. Updates propagate eventually, so the system becomes consistent over time but may be temporarily inconsistent. (Used in social media posts, Google reviews, etc.)

The short version

If every read needs to see the most recent write, prioritize consistency. If it doesn't, prioritize availability.


Used LLMs to correct grammar and typos.

Comments

Loading comments
Leave a commentno sign in
0 / 1000