#012 Demystify "Distributed Systems" with sample code
Here’s a fast, example-driven tour of the five concepts on Distributed Systems , each with a compact code sample you can adapt. I’ll keep the explanations tight with no fluff
1) CAP Theorem (consistency vs. availability)
The article frames CAP as choosing 2 of Consistency, Availability, Partition tolerance (with examples like CP PostgreSQL vs AP Cassandra/DynamoDB). swequiz.com
Example: MongoDB leaning CP with majority write concern
# pip install pymongo
from pymongo import MongoClient, WriteConcern
from bson.objectid import ObjectId
# CP-leaning collection: require majority ack + journaled write
client = MongoClient("mongodb://rs0/") # replica set URI
db = client.get_database("shop")
orders = db.get_collection(
"orders",
write_concern=WriteConcern(w="majority", j=True) # favor consistency
)
order_id = orders.insert_one({"sku": "ABC-123", "qty": 1}).inserted_id
# Stronger read: read from primary only (consistent read)
doc = db.get_collection("orders", read_preference="primary").find_one({"_id": order_id})
print(doc)
If you instead use lower write concern and allow secondaries for reads, you tilt toward availability but may see stale reads during partitions—an AP flavor (per the article’s discussion). swequiz.com
2) Scalability (vertical vs. horizontal)
The post distinguishes scaling up vs. out, notes statelessness, service discovery, and load balancing. swequiz.com
Example: Stateless HTTP service ready for horizontal scaling
# pip install fastapi uvicorn redis
# Run multiple replicas (e.g., 3) behind a load balancer; use Redis for session/state.
from fastapi import FastAPI, Request
import redis
import os
app = FastAPI()
r = redis.Redis(host=os.getenv("REDIS_HOST", "localhost"), port=6379)
@app.get("/health")
def health():
return {"ok": True}
@app.post("/cart/add")
async def add_to_cart(req: Request):
body = await req.json()
user = body["user_id"]
sku = body["sku"]
r.hincrby(f"cart:{user}", sku, 1) # shared state in Redis, not in-process
return {"status": "added"}
# Start N replicas:
# uvicorn app:app --port 8000
# uvicorn app:app --port 8001
# uvicorn app:app --port 8002
# Put them behind NGINX/ALB/Ingress (round-robin / least-connections).
3) Fault Tolerance (retries, failover, graceful degradation)
Highlights replication, failover, retries, circuit breakers (Hystrix/Resilience4j), and graceful degradation. swequiz.com
Example: Retry with exponential backoff + a tiny circuit breaker
# pip install httpx
import time, httpx
class CircuitBreaker:
def __init__(self, fail_threshold=5, reset_after=10):
self.fail_threshold = fail_threshold
self.reset_after = reset_after
self.fail_count = 0
self.state = "CLOSED"
self.opened_at = None
def allow(self):
if self.state == "OPEN":
if time.time() - self.opened_at > self.reset_after:
self.state = "HALF_OPEN"
return True
return False
return True
def record_success(self):
self.fail_count = 0
self.state = "CLOSED"
def record_failure(self):
self.fail_count += 1
if self.fail_count >= self.fail_threshold:
self.state = "OPEN"
self.opened_at = time.time()
cb = CircuitBreaker()
def fetch_with_resilience(url, max_retries=3, base=0.2):
if not cb.allow():
return {"error": "circuit_open"} # fast-fail (graceful degrade path)
for i in range(max_retries):
try:
r = httpx.get(url, timeout=2.0)
r.raise_for_status()
cb.record_success()
return r.json()
except Exception:
time.sleep(base * (2 ** i)) # backoff
cb.record_failure()
return {"error": "failed_after_retries"}
This embodies the “retry/backoff + circuit breaker to prevent cascades,” as described. swequiz.com
4) Latency & Network Communication
Example A: Async I/O to hide network latency during fan-out
# pip install httpx anyio
import anyio, httpx
async def fetch(client, url):
r = await client.get(url, timeout=2.0)
r.raise_for_status()
return r.text
async def aggregate(urls):
async with httpx.AsyncClient() as client:
results = await anyio.gather(*[fetch(client, u) for u in urls])
return results
# anyio.run(aggregate, ["https://api1", "https://api2", "https://api3"])
Example B: Simple cache to cut tail latency on hot keys
# pip install cachetools httpx
import httpx
from cachetools import TTLCache
cache = TTLCache(maxsize=10000, ttl=30) # 30s CDN-like edge cache
def cached_get(url):
if url in cache:
return cache[url]
r = httpx.get(url, timeout=1.5)
r.raise_for_status()
cache[url] = r.text
return r.text
These mirror the article’s emphasis on async comms and caching to reduce perceived latency. swequiz.com
5) Distributed Consensus (Paxos/Raft/Zab)
The post covers Paxos, Raft (leader, heartbeats, log replication), and Zab, with etcd/Kubernetes as a real-world Raft example. swequiz.com
Example: Poor-man’s leader election with etcd (Raft under the hood)
# pip install etcd3
# Run: etcd locally or point ETCD_HOST to your cluster.
import etcd3, os, time
from uuid import uuid4
etcd = etcd3.client(host=os.getenv("ETCD_HOST","localhost"), port=2379)
node_id = str(uuid4())
lease = etcd.lease(5) # 5s TTL
def try_become_leader():
# Create the leader key if absent; attaches TTL so it expires if we die
# Success => this node is leader.
return etcd.transaction(
compare=[etcd.transactions.version("/leaders/serviceX") == 0],
success=[etcd.transactions.put("/leaders/serviceX", node_id, lease)],
failure=[]
)
def am_i_leader():
val, _ = etcd.get("/leaders/serviceX")
return val and val.decode() == node_id
while True:
try_become_leader()
if am_i_leader():
# Send periodic heartbeat by refreshing lease
lease.refresh()
# Handle leader-only work (e.g., scheduling)
print("I am leader:", node_id)
else:
print("Follower; waiting…")
time.sleep(2)
What you get
docker-compose with:
redis,etcd,nginx(LB), and 3 FastAPI replicas.Python scripts for: consensus (etcd leader election), fault tolerance (retries + circuit breaker), latency (async fan-out + TTL cache), and CAP (Mongo majority write + primary reads; code only).
Quick start
unzip distributed-concepts-examples.zip
cd distributed-concepts-examples
# Start the stack (redis, etcd, 3x app replicas, nginx)
docker compose up --build
# Try round-robin across app replicas
curl http://localhost:8080/health
curl -X POST http://localhost:8080/cart/add -H 'Content-Type: application/json' \
-d '{"user_id":"u1","sku":"ABC-123"}'
# Leader election demo (runs locally; etcd must be up)
pip install -r requirements.txt
python app_consensus/leader.py
# Fault-tolerance demo
python app_fault_tolerance/resilience.py
# Latency demos
python app_latency/async_agg.py
python app_latency/cache_example.py
# CAP demo (needs a MongoDB replica set; adjust URI in the file)
python app_cap/mongo_example.py