Intermediate Examples
Examples 28-54 build on the Beginner band's single-store basics: Redis transactions and optimistic
locking, MongoDB's aggregation pipeline and multi-document transactions and compound/covered indexes,
tunable quorum consistency (the math, then Cassandra's own knobs), leaderless replication and the
three ways NoSQL stores resolve or detect concurrent writes (LWW, vector clocks, CRDTs), Cassandra's
partition-key discipline, and DynamoDB's key-value/composite-key/single-table/GSI/TTL/consistency
model via boto3. Redis, MongoDB (now a single-node replica set so multi-document transactions work),
and Cassandra examples run against real local Docker containers; DynamoDB examples run against the
official amazon/dynamodb-local image. Four examples (40-44) are pure-Python simulations, chosen
deliberately for self-contained determinism -- see each one's own note on why.
Example 28: Redis MULTI/EXEC Transaction
ex-28 · exercises co-27
MULTI queues a batch of commands; EXEC applies them all back-to-back with no other client's
command interleaved -- but note the caveat: it is atomic application, not ACID-style rollback on error.
learning/code/ex-28-redis-transaction-multi-exec/example.py
"""Example 28: Redis Transaction MULTI/EXEC.""" # => co-27: this file's own purpose, doubling as its module __doc__
from __future__ import annotations # => hygiene: postpones annotation evaluation, interpreter-version-agnostic
import redis # => co-27: redis-py, the official typed Python client for Valkey/Redis
def transfer_atomically(client: redis.Redis, from_key: str, to_key: str, amount: int) -> None: # => co-27: moves amount atomically
"""Move amount from from_key to to_key inside a MULTI/EXEC transaction.""" # => documents the contract
pipe = client.pipeline(transaction=True) # => co-27: transaction=True queues commands, then EXECs them as one unit
pipe.multi() # => co-27: MULTI -- starts queuing, no command below runs yet
pipe.decrby(from_key, amount) # => co-27: QUEUED, not yet applied -- decrements the source balance
pipe.incrby(to_key, amount) # => co-27: QUEUED, not yet applied -- increments the destination balance
pipe.execute() # => co-27: EXEC -- both queued commands apply together, back-to-back, no other client's write interleaves
# => note (co-27): EXEC does NOT roll back on a runtime error inside the block -- a queued command
# => that fails at execution time (e.g. wrong type) still lets the OTHER queued commands apply;
# => MULTI/EXEC buys ATOMIC APPLICATION of a queued batch, not ACID-style all-or-nothing rollback
def main() -> None: # => entry point -- runs only when this file executes directly, not on import
client = redis.Redis(host="localhost", port=6379, db=0) # => connects to a local Valkey/Redis instance
client.delete("wallet:alice", "wallet:bob") # => resets state -- this example is fully self-contained
client.set("wallet:alice", 100) # => alice starts with 100
client.set("wallet:bob", 50) # => bob starts with 50
transfer_atomically(client, "wallet:alice", "wallet:bob", 30) # => co-27: moves 30 from alice to bob atomically
alice_raw = client.get("wallet:alice") # => reads alice's post-transfer balance, as bytes | None
bob_raw = client.get("wallet:bob") # => reads bob's post-transfer balance, as bytes | None
assert alice_raw is not None and bob_raw is not None # => both keys were SET above -- narrows away the None case for int()
alice_balance = int(alice_raw) # => decodes the raw bytes reply into a plain int
bob_balance = int(bob_raw) # => decodes the raw bytes reply into a plain int
assert alice_balance == 70 # => co-27: 100 - 30 == 70, the debit applied
assert bob_balance == 80 # => co-27: 50 + 30 == 80, the credit applied -- BOTH updates landed together
print(f"After MULTI/EXEC transfer: alice={alice_balance} bob={bob_balance}") # => Output: After MULTI/EXEC transfer: alice=70 bob=80
client.close() # => always release what you open
if __name__ == "__main__": # => guards against running main() on `import example`
main() # => runs everything above when executed as a scriptRun: python3 example.py (representative output, run against redis==8.0.1 and a local Valkey 8 Docker container)
Output:
After MULTI/EXEC transfer: alice=70 bob=80Key takeaway: MULTI/EXEC guarantees no other client's command lands between the queued
commands -- it does not guarantee rollback if one queued command fails at runtime, which is a real gap
compared to a relational transaction.
Why it matters: Any operation that must touch two Redis keys together (a transfer, a counter pair, a denormalized duplicate) needs this atomicity guarantee -- without it, a crash or a slow client between the two writes leaves one side updated and the other stale, a genuinely corrupt state no read can detect after the fact. Example 29 layers optimistic locking on top of it, and Example 72 (Advanced tier) measures its added latency cost directly against the non-transactional path.
Example 29: Redis WATCH Optimistic Lock
ex-29 · exercises co-27
WATCH flags a key; if that key changes before EXEC runs, the whole transaction aborts with a nil
reply -- the classic optimistic-concurrency pattern, retried by the caller rather than blocked.
learning/code/ex-29-redis-watch-optimistic-lock/example.py
"""Example 29: Redis WATCH Optimistic Lock.""" # => co-27: this file's own purpose, doubling as its module __doc__
from __future__ import annotations # => hygiene: postpones annotation evaluation, interpreter-version-agnostic
import redis # => co-27: redis-py, the official typed Python client
from redis.exceptions import WatchError # => co-27: redis-py raises WatchError when a watched key changed
def try_transfer_with_watch(client: redis.Redis, key: str, amount: int, tamper: bool) -> bool: # => co-27: returns True if the tx committed
"""Attempt a WATCH-guarded transfer; optionally tamper the watched key mid-flight to force an abort.""" # => documents contract
with client.pipeline() as pipe: # => a fresh pipeline per attempt, matching real optimistic-locking retry loops
pipe.watch(key) # => co-27: WATCH -- flags this key; if it changes before EXEC, the transaction aborts
if tamper: # => simulates ANOTHER client concurrently modifying the watched key between WATCH and EXEC
client.incr(key) # => co-27: a concurrent write on the SAME key this pipeline is watching
pipe.multi() # => co-27: MULTI -- starts queuing, still inside the SAME watched pipeline
pipe.decrby(key, amount) # => co-27: QUEUED -- will only apply if the watched key stayed untouched
try: # => catches ONLY the WatchError EXEC raises on a detected race, nothing else
pipe.execute() # => co-27: EXEC -- aborts (raises WatchError) if the watched key changed since WATCH
return True # => the transaction committed -- the watched key was untouched by anyone else
except WatchError: # => co-27: redis-py surfaces the server's nil EXEC reply as this exception
return False # => the transaction aborted -- optimistic concurrency correctly detected the race
def main() -> None: # => entry point -- runs only when this file executes directly, not on import
client = redis.Redis(host="localhost", port=6379, db=0) # => connects to a local Valkey/Redis instance
client.set("inventory:sku-42", 100) # => resets state -- this example is fully self-contained
committed_clean = try_transfer_with_watch(client, "inventory:sku-42", 10, tamper=False) # => co-27: no interference -- should commit
assert committed_clean is True # => co-27: WATCH saw no change, so EXEC applied the queued DECRBY
print(f"No interference: transaction committed = {committed_clean}") # => Output: No interference: transaction committed = True
committed_tampered = try_transfer_with_watch(client, "inventory:sku-42", 10, tamper=True) # => co-27: tampers the SAME key mid-flight
assert committed_tampered is False # => co-27: WATCH caught the concurrent INCR, EXEC returned nil, redis-py raised WatchError
print(f"Concurrent write during WATCH: transaction committed = {committed_tampered}") # => Output: Concurrent write during WATCH: transaction committed = False
client.close() # => always release what you open
if __name__ == "__main__": # => guards against running main() on `import example`
main() # => runs everything above when executed as a scriptRun: python3 example.py (representative output, run against redis==8.0.1 and a local Valkey 8 Docker container)
Output:
No interference: transaction committed = True
Concurrent write during WATCH: transaction committed = FalseKey takeaway: WATCH implements optimistic, not pessimistic, concurrency -- the transaction never
blocks waiting for the key to become free; it proceeds hopefully and aborts if it turns out another
client won the race, leaving the caller to retry.
Why it matters: This is the standard Redis pattern for a compare-and-swap-style update (an
inventory decrement, a counter that must not go negative) without a distributed lock -- skip it and two
concurrent decrements can both read the same starting value and silently overwrite each other's result,
selling more inventory than actually exists. co-27's whole category of NoSQL-added-transactions traces
back to exactly this kind of primitive, and a caller must be prepared to retry on WatchError rather
than treat the abort as a hard failure.
Example 30: Redis Pipeline vs. Transaction
ex-30 · exercises co-27
Comparison: a plain pipeline (transaction=False) batches network round trips with no atomicity
guarantee; MULTI/EXEC (transaction=True) batches the SAME commands but guarantees no other
client's write can land between them.
learning/code/ex-30-redis-pipeline-vs-transaction/example.py
"""Example 30: Redis Pipeline vs. Transaction.""" # => co-27: this file's own purpose, doubling as its module __doc__
from __future__ import annotations # => hygiene: postpones annotation evaluation, interpreter-version-agnostic
import threading # => co-27: simulates a concurrent client interleaving with a plain pipeline's commands
import time # => co-27: a tiny sleep opens a genuine interleaving window for the race demonstration
import redis # => co-27: redis-py, the official typed Python client
def run_plain_pipeline(client: redis.Redis, key: str) -> None: # => co-27: NO transaction=True -- commands can interleave with others
"""Queue two commands in a NON-transactional pipeline (transaction=False).""" # => documents the contract
pipe = client.pipeline(transaction=False) # => co-27: transaction=False -- batches network round trips only, NO atomicity
pipe.set(key, "step-1") # => queued, sent in the same batch, but NOT wrapped in MULTI/EXEC
time.sleep(0.05) # => co-27: an artificial gap -- a real pipeline batches sends, but the SERVER can still interleave other clients' commands between these two
pipe.set(key, "step-2") # => queued, sent in the same batch
pipe.execute() # => co-27: sends the batch -- but nothing here PREVENTED another client from writing key in between
def run_multi_exec(client: redis.Redis, key: str) -> None: # => co-27: WITH transaction=True -- genuinely atomic application
"""Queue the SAME two commands inside MULTI/EXEC (transaction=True).""" # => documents the contract
pipe = client.pipeline(transaction=True) # => co-27: transaction=True -- wraps the batch in MULTI/EXEC
pipe.multi() # => co-27: MULTI -- starts queuing
pipe.set(key, "step-1") # => QUEUED, not yet applied
pipe.set(key, "step-2") # => QUEUED, not yet applied
pipe.execute() # => co-27: EXEC -- both apply back-to-back, no other client's write can land between them
def concurrent_interferer(client: redis.Redis, key: str) -> None: # => a second "client" racing against run_plain_pipeline
"""Sleep briefly, then overwrite key -- simulating a concurrent client racing the pipeline.""" # => documents contract
time.sleep(0.02) # => times this write to land INSIDE run_plain_pipeline's artificial 0.05s gap above
client.set(key, "interloper") # => co-27: a genuinely concurrent write landing mid-pipeline
def main() -> None: # => entry point -- runs only when this file executes directly, not on import
client = redis.Redis(host="localhost", port=6379, db=0) # => connects to a local Valkey/Redis instance
# --- Plain pipeline: interference CAN show through, because there is no MULTI/EXEC boundary ---
client.set("demo:pipeline", "start") # => resets state for the pipeline case
interferer = threading.Thread(target=concurrent_interferer, args=(client, "demo:pipeline")) # => a racing second client
interferer.start() # => starts racing concurrently with the pipeline below
run_plain_pipeline(client, "demo:pipeline") # => co-27: the interloper's write CAN land between "step-1" and "step-2"
interferer.join() # => waits for the racing thread to finish before reading the result
pipeline_raw = client.get("demo:pipeline") # => reads the raw bytes | str | None reply
assert pipeline_raw is not None # => the key was SET above, so a reply always exists -- narrows away None
pipeline_result = pipeline_raw.decode() if isinstance(pipeline_raw, bytes) else pipeline_raw # => decodes only if the driver returned raw bytes
# => whichever write landed LAST wins -- not guaranteed to be "step-2"
print(f"Plain pipeline final value (order not guaranteed): {pipeline_result}") # => Output line -- value depends on timing
# --- MULTI/EXEC: the SAME race, but atomicity means the two queued SETs apply back-to-back ---
client.set("demo:transaction", "start") # => resets state for the transaction case
interferer2 = threading.Thread(target=concurrent_interferer, args=(client, "demo:transaction")) # => a racing second client
interferer2.start() # => starts racing concurrently with the transaction below
run_multi_exec(client, "demo:transaction") # => co-27: EXEC applies BOTH queued SETs with no other client's command between them
interferer2.join() # => waits for the racing thread to finish before reading the result
transaction_raw = client.get("demo:transaction") # => reads the raw bytes | str | None reply
assert transaction_raw is not None # => the key was SET above, so a reply always exists -- narrows away None
transaction_result = transaction_raw.decode() if isinstance(transaction_raw, bytes) else transaction_raw # => decodes only if the driver returned raw bytes
# => the interloper's write lands strictly BEFORE or AFTER the whole EXEC, never between step-1 and step-2
print(f"MULTI/EXEC final value (interloper cannot split the pair): {transaction_result}") # => Output line
assert transaction_result in ("step-2", "interloper") # => co-27: either the interloper won outright, or the atomic pair won outright -- never a torn state
client.close() # => always release what you open
if __name__ == "__main__": # => guards against running main() on `import example`
main() # => runs everything above when executed as a scriptRun: python3 example.py (representative output, run against redis==8.0.1 and a local Valkey 8 Docker container)
Output:
Plain pipeline final value (order not guaranteed): step-2
MULTI/EXEC final value (interloper cannot split the pair): interloperKey takeaway: A plain pipeline's step-2 winning here is itself a matter of timing, not a
guarantee -- the same race run again could just as easily have the interloper's write land between
step-1 and step-2. MULTI/EXEC's guarantee is different in kind: the interloper's write can
never land between the two queued commands, even though it can still win outright before or after.
Why it matters: "A pipeline batches network calls" and "a transaction batches atomically" are
genuinely different guarantees that look similar at the API surface -- confusing them is a real source
of subtle production races, especially since a plain pipeline still looks atomic in casual testing
where no concurrent client happens to interleave. Reaching for transaction=False purely for the
latency win, without checking whether the operation also needs the MULTI/EXEC boundary, is exactly
the mistake this example is built to catch before it reaches production.
Example 31: MongoDB group
ex-31 · exercises co-19
An aggregation pipeline's $match stage filters documents, then $group computes a per-key total --
both stages run server-side, in one round trip, never pulling raw rows to the client to sum in Python.
learning/code/ex-31-mongo-aggregation-match-group/example.py
"""Example 31: MongoDB Aggregation $match + $group.""" # => co-19: this file's own purpose, doubling as its module __doc__
from __future__ import annotations # => hygiene: postpones annotation evaluation, interpreter-version-agnostic
from typing import Any # => enables the explicit MongoClient[dict[str, Any]] type parameter below (DD-39, pyright --strict)
from pymongo import MongoClient # => co-19: pymongo, the official typed Python driver
Document = dict[str, Any] # => co-19: pymongo's own convention for "a MongoDB document" -- every field's own type varies per-document
def seed_orders(client: MongoClient[Document]) -> None: # => a deterministic fixture across 3 categories, one order excluded by status
"""Reset and seed 5 orders across 3 categories, one cancelled (excluded by the pipeline's $match).""" # => documents contract
collection = client["nosqldb"]["orders"] # => co-19: no schema declared for this collection
collection.delete_many({}) # => resets state -- this example is fully self-contained
collection.insert_many([ # => co-19: 5 orders, one "cancelled" -- the pipeline below must exclude it
{"category": "books", "amount": 20, "status": "paid"}, # => books order 1, paid -- counted below
{"category": "books", "amount": 15, "status": "paid"}, # => books order 2, paid -- counted below
{"category": "electronics", "amount": 200, "status": "paid"}, # => electronics order 1, paid -- counted below
{"category": "electronics", "amount": 50, "status": "cancelled"}, # => excluded by $match below
{"category": "toys", "amount": 30, "status": "paid"}, # => toys' only order, paid -- counted below
]) # => 5 orders total, but only 4 are "paid" -- the $match stage must drop exactly 1
def total_paid_per_category(client: MongoClient[Document]) -> dict[str, int]: # => co-19: server-side aggregate, one round trip
"""Run a $match + $group pipeline computing a per-category paid total.""" # => documents the contract
collection = client["nosqldb"]["orders"] # => selects the collection seed_orders just populated
pipeline = [ # => co-19: a staged pipeline -- each stage's output feeds the next stage's input
{"$match": {"status": "paid"}}, # => co-19: stage 1 -- keeps only paid orders, drops the cancelled one
{"$group": {"_id": "$category", "total": {"$sum": "$amount"}}}, # => co-19: stage 2 -- sums amount PER distinct category
] # => 2 stages, run server-side in ONE round trip -- no client-side filtering or summing needed
results = list(collection.aggregate(pipeline)) # => co-19: ONE round trip runs the WHOLE pipeline server-side
return {doc["_id"]: doc["total"] for doc in results} # => reshapes the cursor into a plain category->total dict
def main() -> None: # => entry point -- runs only when this file executes directly, not on import
client = MongoClient("mongodb://localhost:27017") # => connects to a local MongoDB instance
seed_orders(client) # => sets up the 5-order, 3-category fixture
totals = total_paid_per_category(client) # => runs the $match + $group aggregation
assert totals == {"books": 35, "electronics": 200, "toys": 30} # => co-19: cancelled electronics order correctly excluded
for category in sorted(totals): # => co-19: prints the aggregate result, sorted for deterministic output
print(f"{category}: {totals[category]}") # => Output line, one per category
client.close() # => always release what you open
if __name__ == "__main__": # => guards against running main() on `import example`
main() # => runs everything above when executed as a scriptRun: python3 example.py (representative output, run against pymongo==4.17.0 and a local MongoDB 8.2.12 Docker
replica-set container)
Output:
books: 35
electronics: 200
toys: 30Key takeaway: $match should be placed as early in the pipeline as possible, exactly like a
relational WHERE before a GROUP BY -- it lets MongoDB discard non-matching documents before the
more expensive $group stage ever sees them.
Why it matters: $match + $group is the aggregation pipeline's most common shape and the direct
analogue of SELECT ... WHERE ... GROUP BY -- pulling every raw document to the client and summing in
Python instead wastes network bandwidth and CPU on rows the query never needed, a mistake that scales
badly as collection size grows. Examples 32 and 33 extend the same staged-pipeline idea with a
server-side join and a 3-stage chain, so the mental model built here carries forward directly.
Example 32: MongoDB $lookup Correlated Subquery
ex-32 · exercises co-19
$lookup joins two collections server-side, in one round trip. The plain localField/foreignField
equality-join form predates MongoDB 5.0 by several major versions and is not itself correlated -- per
MongoDB's own docs, the correlated subquery form requires adding let and a pipeline alongside
localField/foreignField, available starting in MongoDB 5.0. This example demonstrates that
correlated form: each author's own min_year field drives a per-document filter on her joined books,
contrasted directly against Example 12's manual, client-side, two-query join.
learning/code/ex-32-mongo-aggregation-lookup-join/example.py
"""Example 32: MongoDB Aggregation $lookup Correlated Subquery.""" # => co-19: this file's own purpose, doubling as its module __doc__
from __future__ import annotations # => hygiene: postpones annotation evaluation, interpreter-version-agnostic
from typing import Any # => enables the explicit MongoClient[dict[str, Any]] type parameter below (DD-39, pyright --strict)
from pymongo import MongoClient # => co-19: pymongo, the official typed Python driver
Document = dict[str, Any] # => co-19: pymongo's own convention for "a MongoDB document" -- every field's own type varies per-document
def seed_authors_and_books(client: MongoClient[Document]) -> None: # => two collections, joined server-side below
"""Reset and seed 2 authors (each with its own min_year floor) and 3 books, referencing authors by name.""" # => documents the contract
authors = client["nosqldb"]["authors"] # => the "one" side of the relation
books = client["nosqldb"]["books"] # => the "many" side of the relation
authors.delete_many({}) # => resets state -- this example is fully self-contained
books.delete_many({}) # => resets the books side too
authors.insert_many([ # => co-19: 2 authors, each carries its OWN min_year -- the correlated pipeline reads this per-author
{"name": "Ada", "min_year": 2015}, # => Ada's floor excludes her own older book, seeded below
{"name": "Grace", "min_year": 1950}, # => Grace's floor is low enough to keep her only book
]) # => 2 authors seeded, each with a different min_year threshold
books.insert_many([ # => co-19: 3 books, "author" is the join key, "year" is what the correlated pipeline filters on
{"title": "NoSQL 101", "author": "Ada", "year": 2020}, # => Ada's book 1 -- published AFTER her min_year (2015), so it survives the filter
{"title": "CAP Theorem Explained", "author": "Ada", "year": 2010}, # => Ada's book 2 -- published BEFORE her min_year (2015), the pipeline drops it
{"title": "Compilers 101", "author": "Grace", "year": 1957}, # => Grace's only book -- published after her low min_year (1950), survives
]) # => 3 books seeded; a plain equality join would return all 3, the correlated pipeline below returns only 2
def authors_with_recent_books(client: MongoClient[Document]) -> list[Document]: # => co-19: a server-side CORRELATED subquery, one round trip
"""Run a $lookup pipeline attaching only each author's own books published at/after her own min_year.""" # => documents the contract
authors = client["nosqldb"]["authors"] # => the collection $lookup pipelines FROM
pipeline = [ # => co-19: MongoDB 5.0+ correlated $lookup syntax -- localField/foreignField PLUS let/pipeline together
{ # => stage 1 -- the whole $lookup stage dict starts here
"$lookup": { # => co-19: the server-side join stage -- no separate client-side query needed
"from": "books", # => co-19: the FOREIGN collection to join against
"localField": "name", # => co-19: still runs the automatic name == author equality match first
"foreignField": "author", # => co-19: the foreign collection's field matched against localField
"let": {"min_year": "$min_year"}, # => co-19: captures THIS author's own min_year, exposed to the sub-pipeline as $$min_year
"pipeline": [ # => co-19: the sub-pipeline is what makes this a CORRELATED subquery, not a plain equality join
{"$match": {"$expr": {"$gte": ["$year", "$$min_year"]}}}, # => co-19: keeps only books at/after THIS CORRELATED author's own min_year
], # => runs once PER author, using that author's own let-bound $$min_year each time
"as": "books", # => co-19: the joined, filtered results attach under this NEW array field
} # => closes the $lookup operator's own options dict
}, # => closes stage 1
{"$sort": {"name": 1}}, # => sorts authors alphabetically for deterministic output
] # => 2 stages, run server-side -- the correlated join, filter, AND sort all happen before any row reaches this client
return list(authors.aggregate(pipeline)) # => co-19: ONE round trip returns EVERY author with only her qualifying books attached
def main() -> None: # => entry point -- runs only when this file executes directly, not on import
client = MongoClient("mongodb://localhost:27017") # => connects to a local MongoDB instance
seed_authors_and_books(client) # => sets up the 2-author, 3-book fixture
joined = authors_with_recent_books(client) # => runs the correlated $lookup join
assert len(joined) == 2 # => co-19: both authors present, each with its own filtered books array
for author in joined: # => prints each author with its joined, filtered book titles
titles = [book["title"] for book in author["books"]] # => co-19: extracts titles from the JOINED array field
print(f"{author['name']}: {titles}") # => Output line, one per author
assert len(joined[0]["books"]) == 1 # => co-19: Ada's filtered array keeps only her post-2015 book
assert len(joined[1]["books"]) == 1 # => co-19: Grace's filtered array keeps her only (post-1950) book
client.close() # => always release what you open
if __name__ == "__main__": # => guards against running main() on `import example`
main() # => runs everything above when executed as a scriptRun: python3 example.py (representative output, run against pymongo==4.17.0 and a local MongoDB 8.2.12 Docker
replica-set container)
Output:
Ada: ['NoSQL 101']
Grace: ['Compilers 101']Key takeaway: $lookup's correlated form -- let plus pipeline -- lets each joined document's
own field (here, each author's min_year) drive a per-document filter on the foreign side, something a
plain localField/foreignField equality join cannot do. It is still not free: it costs a lookup per
document on the "from" side under the hood, so it is best reserved for cases where denormalizing
(Example 12's embedded approach) is impractical, not used reflexively everywhere a relational schema
would use a JOIN.
Why it matters: Teams that reach for MongoDB expecting "no joins at all" are often surprised by
$lookup -- and the correlated form shown here is what makes it genuinely powerful: without
let/pipeline, MongoDB can only match on equality, but with them, each document's own fields can
shape a custom sub-query against the foreign collection, the same expressive power SQL correlated
subqueries provide. Knowing when to reach for $lookup instead of denormalizing is a genuine
schema-design decision, not a fallback for a missing feature, and getting it wrong either bloats
documents or multiplies round trips.
Example 33: MongoDB Aggregation Pipeline Stages
ex-33 · exercises co-19
Chain $match → $group → $sort in one pipeline -- each stage's output becomes the next
stage's input, a data-flow model rather than nested subqueries.
%% Color Palette: Blue #0173B2, Orange #DE8F05, Teal #029E73 -- color-blind friendly, WCAG AA
flowchart LR
E[("6 events<br/>3 types")]:::orange --> M["$match<br/>severity >= 2"]:::blue
M --> GS["$group + $sort<br/>avg severity DESC"]:::blue
GS --> R["error 4.5, login 3.0,<br/>logout 2.0"]:::teal
classDef blue fill:#0173B2,stroke:#000000,color:#FFFFFF,stroke-width:2px
classDef orange fill:#DE8F05,stroke:#000000,color:#FFFFFF,stroke-width:2px
classDef teal fill:#029E73,stroke:#000000,color:#FFFFFF,stroke-width:2px
learning/code/ex-33-mongo-aggregation-pipeline-stages/example.py
"""Example 33: MongoDB Aggregation Pipeline Stages.""" # => co-19: this file's own purpose, doubling as its module __doc__
from __future__ import annotations # => hygiene: postpones annotation evaluation, interpreter-version-agnostic
from typing import Any # => enables the explicit MongoClient[dict[str, Any]] type parameter below (DD-39, pyright --strict)
from pymongo import MongoClient # => co-19: pymongo, the official typed Python driver
Document = dict[str, Any] # => co-19: pymongo's own convention for "a MongoDB document" -- every field's own type varies per-document
def seed_events(client: MongoClient[Document]) -> None: # => a deterministic fixture across 3 event types
"""Reset and seed 6 events across 3 types, with varying severities.""" # => documents the contract
collection = client["nosqldb"]["events"] # => co-19: no schema declared for this collection
collection.delete_many({}) # => resets state -- this example is fully self-contained
collection.insert_many([ # => co-19: 6 events, 3 distinct types, some low-severity to be filtered out
{"type": "login", "severity": 3}, # => login event 1, severity >= 2 -- survives $match
{"type": "login", "severity": 1}, # => login event 2, severity 1 -- excluded by $match below
{"type": "error", "severity": 5}, # => error event 1, severity >= 2 -- survives $match
{"type": "error", "severity": 4}, # => error event 2, severity >= 2 -- survives $match
{"type": "error", "severity": 1}, # => severity 1 -- excluded by $match below
{"type": "logout", "severity": 2}, # => logout's only event, severity >= 2 -- survives $match
]) # => 6 events seeded, 2 with severity 1 -- exactly 2 must be dropped by stage 1's $match
def top_event_types_by_avg_severity(client: MongoClient[Document]) -> list[Document]: # => co-19: 3-stage chained pipeline
"""Chain $match -> $group -> $sort, computing average severity per type, filtered and ranked.""" # => documents contract
collection = client["nosqldb"]["events"] # => selects the collection seed_events just populated
pipeline = [ # => co-19: each stage's OUTPUT is the next stage's INPUT -- a data-flow pipeline, not nested SQL
{"$match": {"severity": {"$gte": 2}}}, # => co-19: stage 1 -- drops the low-severity noise (severity 1)
{"$group": {"_id": "$type", "avg_severity": {"$avg": "$severity"}}}, # => co-19: stage 2 -- averages PER remaining type
{"$sort": {"avg_severity": -1}}, # => co-19: stage 3 -- highest average severity first
] # => 3 chained stages, all server-side, in ONE round trip
return list(collection.aggregate(pipeline)) # => co-19: ONE round trip runs all 3 stages server-side
def main() -> None: # => entry point -- runs only when this file executes directly, not on import
client = MongoClient("mongodb://localhost:27017") # => connects to a local MongoDB instance
seed_events(client) # => sets up the 6-event, 3-type fixture
ranked = top_event_types_by_avg_severity(client) # => runs the 3-stage chained pipeline
for row in ranked: # => prints the ranked, averaged, filtered result
print(f"{row['_id']}: avg_severity={row['avg_severity']:.1f}") # => Output line, one per event type
assert ranked[0]["_id"] == "error" # => co-19: error's average (4.5, only severity>=2 events counted) ranks highest
assert ranked[-1]["_id"] == "logout" # => co-19: logout's single event (severity 2) ranks lowest of the 3
client.close() # => always release what you open
if __name__ == "__main__": # => guards against running main() on `import example`
main() # => runs everything above when executed as a scriptRun: python3 example.py (representative output, run against pymongo==4.17.0 and a local MongoDB 8.2.12 Docker
replica-set container)
Output:
error: avg_severity=4.5
login: avg_severity=3.0
logout: avg_severity=2.0Key takeaway: The pipeline reads top-to-bottom as a sequence of transforms -- filter, then aggregate, then order -- which maps closely onto how a developer already thinks about the query in plain English, unlike a single deeply nested SQL statement.
Why it matters: Every non-trivial MongoDB analytics query in production is some chain of these
same stages -- $project, $unwind, and $facet (not shown here) extend the same pipeline model
further without changing the mental model established by this example. Ordering the stages correctly,
filtering early with $match before the more expensive $group and $sort stages run, is the single
biggest lever for keeping a production aggregation pipeline fast as the underlying collection grows.
Example 34: MongoDB Multi-Document Transaction
ex-34 · exercises co-27
Setup: this example (and Examples 35 and 72) needs the single-node MongoDB replica set from
Confirm your toolchain -- a standalone mongod rejects
start_transaction() outright.
A session-scoped transaction moves a value between two documents atomically -- both updates commit together or the whole transaction never becomes visible, MongoDB's ACID-style answer to co-27.
learning/code/ex-34-mongo-multi-document-transaction/example.py
"""Example 34: MongoDB Multi-Document Transaction.""" # => co-27: this file's own purpose, doubling as its module __doc__
from __future__ import annotations # => hygiene: postpones annotation evaluation, interpreter-version-agnostic
from typing import Any # => enables the explicit MongoClient[dict[str, Any]] type parameter below (DD-39, pyright --strict)
from pymongo import MongoClient # => co-27: pymongo, the official typed Python driver
Document = dict[str, Any] # => co-27: pymongo's own convention for "a MongoDB document" -- every field's own type varies per-document
def seed_accounts(client: MongoClient[Document]) -> None: # => two documents, moved between atomically below
"""Reset and seed two account documents.""" # => documents the contract, no runtime output
accounts = client["nosqldb"]["accounts"] # => co-27: a dedicated collection for this transaction demonstration
accounts.delete_many({}) # => resets state -- this example is fully self-contained
accounts.insert_many([{"_id": "alice", "balance": 100}, {"_id": "bob", "balance": 50}]) # => starting balances
def transfer_in_transaction(client: MongoClient[Document], amount: int) -> None: # => co-27: a SESSION-scoped, ACID-style transaction
"""Move amount from alice to bob inside a session-scoped multi-document transaction.""" # => documents contract
accounts = client["nosqldb"]["accounts"] # => selects the collection seed_accounts just populated
with client.start_session() as session, session.start_transaction(): # => co-27: session scopes; start_transaction begins it -- one combined `with`
accounts.update_one({"_id": "alice"}, {"$inc": {"balance": -amount}}, session=session) # => co-27: staged debit, tied to this session
accounts.update_one({"_id": "bob"}, {"$inc": {"balance": amount}}, session=session) # => co-27: staged credit, tied to this SAME session
# => co-27: leaving the `with` block cleanly triggers an implicit COMMIT -- BOTH updates
# => become visible to other clients atomically, in one instant, or NEITHER does
def main() -> None: # => entry point -- runs only when this file executes directly, not on import
client = MongoClient("mongodb://localhost:27017") # => connects to a local MongoDB replica-set instance (transactions require one)
seed_accounts(client) # => sets up the two-account fixture
transfer_in_transaction(client, 30) # => co-27: runs the session-scoped atomic transfer
accounts = client["nosqldb"]["accounts"] # => selects the collection to verify the committed result
alice = accounts.find_one({"_id": "alice"}) # => reads alice's post-transaction balance
bob = accounts.find_one({"_id": "bob"}) # => reads bob's post-transaction balance
assert alice is not None and alice["balance"] == 70 # => co-27: 100 - 30 == 70, the debit committed
assert bob is not None and bob["balance"] == 80 # => co-27: 50 + 30 == 80, the credit committed -- BOTH atomically
print(f"After transaction commit: alice={alice['balance']} bob={bob['balance']}") # => Output: After transaction commit: alice=70 bob=80
client.close() # => always release what you open
if __name__ == "__main__": # => guards against running main() on `import example`
main() # => runs everything above when executed as a scriptRun: python3 example.py (representative output, run against pymongo==4.17.0 and a local MongoDB 8.2.12 Docker
single-node replica set, since multi-document transactions require one -- a standalone mongod
rejects start_transaction() outright)
Output:
After transaction commit: alice=70 bob=80Key takeaway: Unlike Redis's MULTI/EXEC, MongoDB's session-scoped transaction genuinely
supports rollback on error (Example 35 shows this directly) -- it is the closest thing to relational
ACID semantics any store in this topic offers.
Why it matters: A financial transfer, an inventory reservation paired with an order creation, or any operation that must never be partially visible needs exactly this primitive -- without it, a crash between the two updates leaves the system in a state no later read can distinguish from a legitimate partial operation, and reconciling that after the fact is expensive and error-prone. Example 72 (Advanced tier) measures its added latency against the non-transactional path, since this guarantee is never free.
Example 35: MongoDB Transaction Abort
ex-35 · exercises co-27
Force an error partway through a transaction and confirm the staged write never became visible --
unlike Example 28's MULTI/EXEC, MongoDB transactions genuinely roll back.
learning/code/ex-35-mongo-transaction-abort/example.py
"""Example 35: MongoDB Transaction Abort.""" # => co-27: this file's own purpose, doubling as its module __doc__
from __future__ import annotations # => hygiene: postpones annotation evaluation, interpreter-version-agnostic
from typing import Any # => enables the explicit MongoClient[dict[str, Any]] type parameter below (DD-39, pyright --strict)
from pymongo import MongoClient # => co-27: pymongo, the official typed Python driver
Document = dict[str, Any] # => co-27: pymongo's own convention for "a MongoDB document" -- every field's own type varies per-document
class DeliberateFailure(Exception): # => co-27: a custom exception standing in for a real mid-transaction business-rule error
"""Raised deliberately mid-transaction to force an abort.""" # => documents intent, no runtime output
def seed_accounts(client: MongoClient[Document]) -> None: # => two documents, one write staged then deliberately aborted
"""Reset and seed two account documents.""" # => documents the contract, no runtime output
accounts = client["nosqldb"]["accounts_abort"] # => co-27: a SEPARATE collection so this example does not collide with Example 34
accounts.delete_many({}) # => resets state -- this example is fully self-contained
accounts.insert_many([{"_id": "carol", "balance": 200}, {"_id": "dave", "balance": 10}]) # => starting balances
def transfer_that_fails_midway(client: MongoClient[Document], amount: int) -> bool: # => co-27: returns True if the abort happened as expected
"""Stage a debit, force an error before the credit, and confirm the whole transaction rolled back.""" # => documents contract
accounts = client["nosqldb"]["accounts_abort"] # => selects the collection seed_accounts just populated
try: # => catches ONLY the deliberate business-rule exception, to prove the transaction rolled back cleanly
with client.start_session() as session, session.start_transaction(): # => co-27: session scopes; start_transaction begins it -- one combined `with`
accounts.update_one({"_id": "carol"}, {"$inc": {"balance": -amount}}, session=session) # => co-27: staged debit, NOT yet visible outside
raise DeliberateFailure("business rule violated mid-transfer") # => co-27: forces an error BEFORE the matching credit runs
# => the line below is intentionally unreachable -- the raise above always fires first
accounts.update_one({"_id": "dave"}, {"$inc": {"balance": amount}}, session=session) # pragma: no cover
except DeliberateFailure: # => co-27: the `with session.start_transaction()` block exits via exception -> implicit ABORT
return True # => co-27: the staged debit was rolled back entirely -- pymongo aborts on an uncaught exception in the block
return False # => unreachable in this example -- included only so the function's return type is total
def main() -> None: # => entry point -- runs only when this file executes directly, not on import
client = MongoClient("mongodb://localhost:27017") # => connects to a local MongoDB replica-set instance
seed_accounts(client) # => sets up the two-account fixture
aborted = transfer_that_fails_midway(client, 50) # => co-27: runs the deliberately-failing transaction
assert aborted is True # => confirms the DeliberateFailure path was taken
accounts = client["nosqldb"]["accounts_abort"] # => selects the collection to verify NO partial write landed
carol = accounts.find_one({"_id": "carol"}) # => reads carol's balance AFTER the aborted transaction
dave = accounts.find_one({"_id": "dave"}) # => reads dave's balance AFTER the aborted transaction
assert carol is not None and carol["balance"] == 200 # => co-27: UNCHANGED -- the staged debit was rolled back, not partially applied
assert dave is not None and dave["balance"] == 10 # => co-27: UNCHANGED -- the credit never even ran
print(f"After aborted transaction: carol={carol['balance']} dave={dave['balance']} (both unchanged)") # => Output: After aborted transaction: carol=200 dave=10 (both unchanged)
client.close() # => always release what you open
if __name__ == "__main__": # => guards against running main() on `import example`
main() # => runs everything above when executed as a scriptRun: python3 example.py (representative output, run against pymongo==4.17.0 and a local MongoDB 8.2.12 Docker
single-node replica set)
Output:
After aborted transaction: carol=200 dave=10 (both unchanged)Key takeaway: An uncaught exception inside session.start_transaction()'s with block triggers an
implicit abort in pymongo -- no partial write is ever visible outside the transaction, a genuinely
stronger guarantee than Example 28's MULTI/EXEC.
Why it matters: This is the difference that matters when choosing whether a multi-key operation
needs MongoDB's transaction support or can tolerate Redis's weaker atomic-application guarantee --
correctness requirements, not habit, should decide which primitive an operation uses. An operation with
a genuine mid-flight business-rule check, like this example's deliberate failure, needs real rollback;
an operation that only needs "no other client interleaves" can accept Redis's cheaper MULTI/EXEC
instead, and picking the wrong one either over-pays in latency or under-delivers on correctness.
Example 36: MongoDB Compound Index
ex-36 · exercises co-17
A compound index over two fields lets the query planner serve a two-field equality filter without
scanning every document -- explain() names the exact index selected.
learning/code/ex-36-mongo-compound-index/example.py
"""Example 36: MongoDB Compound Index.""" # => co-17: this file's own purpose, doubling as its module __doc__
from __future__ import annotations # => hygiene: postpones annotation evaluation, interpreter-version-agnostic
from typing import Any # => enables the explicit MongoClient[dict[str, Any]] type parameter below (DD-39, pyright --strict)
from pymongo import ASCENDING, MongoClient # => co-17: ASCENDING names the sort direction explicitly, not a bare 1
Document = dict[str, Any] # => co-17: pymongo's own convention for "a MongoDB document" -- every field's own type varies per-document
def seed_orders(client: MongoClient[Document]) -> None: # => enough documents that the planner's index choice is measurable
"""Reset and seed 3000 orders across a small set of statuses and customers.""" # => documents the contract
collection = client["nosqldb"]["orders_compound"] # => co-17: a dedicated collection for this compound-index demonstration
collection.delete_many({}) # => resets state -- this example is fully self-contained
collection.insert_many([ # => co-17: 3000 documents -- enough that an unindexed 2-field filter is genuinely more expensive
{"customer_id": i % 50, "status": "paid" if i % 4 != 0 else "cancelled", "amount": i} # => 50 distinct customers, 3/4 paid, spread across 3000 rows
for i in range(3000) # => generates all 3000 documents deterministically
]) # => enough volume that COLLSCAN vs. IXSCAN genuinely differ in cost, not just in name
def explain_two_field_query(client: MongoClient[Document]) -> tuple[str, str]: # => returns the winning index name, before/after
"""Run a 2-field equality query before and after a matching compound index.""" # => documents the contract
collection = client["nosqldb"]["orders_compound"] # => selects the collection seed_orders just populated
query = {"customer_id": 7, "status": "paid"} # => co-17: a 2-FIELD equality filter -- exactly what a compound index targets
before = collection.find(query).explain() # => co-17: explain() WITHOUT a matching compound index
before_stage = before["queryPlanner"]["winningPlan"]["stage"] # => co-17: no compound index exists -- expect COLLSCAN
assert before_stage == "COLLSCAN" # => co-17: confirms the baseline -- a full collection scan
collection.create_index([("customer_id", ASCENDING), ("status", ASCENDING)]) # => co-17: ONE compound index covering BOTH fields
after = collection.find(query).explain() # => the SAME query, re-explained now that the compound index exists
after_index_name = after["queryPlanner"]["winningPlan"]["inputStage"]["indexName"] # => co-17: the specific index the planner chose
assert after_index_name == "customer_id_1_status_1" # => co-17: MongoDB's default name -- field_direction pairs joined by underscores
return before_stage, after_index_name # => hand both results back for the caller to print
def main() -> None: # => entry point -- runs only when this file executes directly, not on import
client = MongoClient("mongodb://localhost:27017") # => connects to a local MongoDB instance
seed_orders(client) # => sets up the 3000-document fixture
before_stage, after_index_name = explain_two_field_query(client) # => runs the before/after compound-index comparison
print(f"Before compound index: {before_stage} | After: index '{after_index_name}' selected") # => Output: Before compound index: COLLSCAN | After: index 'customer_id_1_status_1' selected
client.close() # => always release what you open
if __name__ == "__main__": # => guards against running main() on `import example`
main() # => runs everything above when executed as a scriptRun: python3 example.py (representative output, run against pymongo==4.17.0 and a local MongoDB 8.2.12 Docker
instance)
Output:
Before compound index: COLLSCAN | After: index 'customer_id_1_status_1' selectedKey takeaway: A compound index's field ORDER matters -- {customer_id: 1, status: 1} serves
{customer_id, status} filters and {customer_id}-only filters, but not a {status}-only filter
efficiently (the same left-prefix rule a B-tree compound index follows in a relational store).
Why it matters: Most real production queries filter on more than one field -- a single-field index per query field is rarely as efficient as one well-chosen compound index covering the actual filter shape the application uses, since the planner can only combine separate single-field indexes with a costlier intersection strategy. Getting the field order right (Example 37 pushes this further into covered queries) is a recurring MongoDB schema-design skill, not a one-time setup task.
Example 37: MongoDB Covered Query
ex-37 · exercises co-17
A query answered entirely from the index, with zero documents fetched -- explain()'s
totalDocsExamined: 0 is the proof.
learning/code/ex-37-mongo-covered-query/example.py
"""Example 37: MongoDB Covered Query.""" # => co-17: this file's own purpose, doubling as its module __doc__
from __future__ import annotations # => hygiene: postpones annotation evaluation, interpreter-version-agnostic
from typing import Any # => enables the explicit MongoClient[dict[str, Any]] type parameter below (DD-39, pyright --strict)
from pymongo import MongoClient # => co-17: pymongo, the official typed Python driver
Document = dict[str, Any] # => co-17: pymongo's own convention for "a MongoDB document" -- every field's own type varies per-document
def seed_users(client: MongoClient[Document]) -> None: # => documents with an extra field NOT touched by the covered query below
"""Reset and seed 500 users, each with an indexed email and an unrelated bio field.""" # => documents the contract
collection = client["nosqldb"]["users_covered"] # => co-17: a dedicated collection for this covered-query demonstration
collection.delete_many({}) # => resets state -- this example is fully self-contained
collection.insert_many([ # => co-17: 500 documents, "bio" exists but is NEVER referenced by the query or projection below
{"email": f"user{i}@example.com", "active": i % 2 == 0, "bio": f"bio text for user {i}" * 3} # => bio is deliberately large and UNINDEXED
for i in range(500) # => generates all 500 documents deterministically
]) # => the covered query below must answer using ONLY email + active, never touching bio
def explain_covered_query(client: MongoClient[Document]) -> int: # => co-17: returns totalDocsExamined, expected to be exactly 0
"""Create an index over exactly the queried and projected fields, then confirm the query is COVERED.""" # => documents contract
collection = client["nosqldb"]["users_covered"] # => selects the collection seed_users just populated
collection.create_index([("email", 1), ("active", 1)]) # => co-17: an index over BOTH the filter field and the projected field
explanation = collection.find( # => co-17: query the SAME two fields the index covers
{"email": "user250@example.com"}, # => filters on email -- part of the index
{"active": 1, "_id": 0}, # => projects ONLY active, also part of the index, and explicitly drops _id
).explain() # => co-17: explain() reveals whether the server needed to fetch the full document at all
total_docs_examined = explanation["executionStats"]["totalDocsExamined"] # => co-17: 0 means the index ALONE answered the query
return total_docs_examined # => hand the raw count back for the caller to assert and print
def main() -> None: # => entry point -- runs only when this file executes directly, not on import
client = MongoClient("mongodb://localhost:27017") # => connects to a local MongoDB instance
seed_users(client) # => sets up the 500-document fixture
total_docs_examined = explain_covered_query(client) # => runs the covered-query verification
assert total_docs_examined == 0 # => co-17: a COVERED query -- the index alone answered it, the full document was NEVER fetched
print(f"Covered query totalDocsExamined: {total_docs_examined} (index alone answered the query)") # => Output: Covered query totalDocsExamined: 0 (index alone answered the query)
client.close() # => always release what you open
if __name__ == "__main__": # => guards against running main() on `import example`
main() # => runs everything above when executed as a scriptRun: python3 example.py (representative output, run against pymongo==4.17.0 and a local MongoDB 8.2.12 Docker
instance)
Output:
Covered query totalDocsExamined: 0 (index alone answered the query)Key takeaway: A covered query requires that EVERY field the query touches -- both the filter and
the projection -- lives in the index; add one extra projected field the index doesn't cover and
totalDocsExamined jumps from 0 to the full match count.
Why it matters: Covered queries are the cheapest possible MongoDB read -- no document fetch at all -- which matters most for high-frequency lookups (a session validity check, a feature-flag lookup) where index-only reads meaningfully reduce I/O, and where even a small per-request savings compounds across millions of calls a day. The tradeoff is fragility: adding one extra projected field the index does not cover silently drops back to a full document fetch, so covered queries need to be revisited whenever the projection changes.
Example 38: Quorum Read/Write Math
ex-38 · exercises co-07
Compute W + R > N for three (N, W, R) configurations -- the algebraic condition that guarantees a
read quorum and a write quorum share at least one replica.
%% Color Palette: Blue #0173B2, Orange #DE8F05, Teal #029E73, Purple #CC78BC -- color-blind friendly, WCAG AA
flowchart TD
N["N=3 replicas"]:::orange
N --> W["W=2 write quorum<br/>replica-1, replica-2"]:::blue
N --> R["R=2 read quorum<br/>replica-2, replica-3"]:::teal
W -.->|overlaps at| O["replica-2<br/>W+R=4 greater than N=3"]:::purple
R -.->|overlaps at| O
classDef blue fill:#0173B2,stroke:#000000,color:#FFFFFF,stroke-width:2px
classDef orange fill:#DE8F05,stroke:#000000,color:#FFFFFF,stroke-width:2px
classDef teal fill:#029E73,stroke:#000000,color:#FFFFFF,stroke-width:2px
classDef purple fill:#CC78BC,stroke:#000000,color:#FFFFFF,stroke-width:2px
learning/code/ex-38-quorum-read-write-math/example.py
"""Example 38: Quorum Read/Write Math.""" # => co-07: this file's own purpose, doubling as its module __doc__
from __future__ import annotations # => hygiene: postpones annotation evaluation, interpreter-version-agnostic
from dataclasses import dataclass # => co-07: a typed record for one (N, W, R) configuration
@dataclass(frozen=True) # => frozen -- a configuration is a stated fact, not something later code mutates
class QuorumConfig: # => co-07: the 3 tunable knobs behind Cassandra's/Dynamo-style tunable consistency
label: str # => a human-readable name for this configuration
n: int # => co-07: N -- the total number of replicas holding this data
w: int # => co-07: W -- how many replicas must ack a WRITE before it is considered successful
r: int # => co-07: R -- how many replicas a READ must consult before returning a result
def guarantees_strong_consistency(config: QuorumConfig) -> bool: # => co-07: the classic W + R > N overlap condition
"""Return True if W + R > N, guaranteeing the read and write sets overlap by at least one replica.""" # => documents contract
return config.w + config.r > config.n # => co-07: overlap guarantees the read set includes at least one replica holding the latest write
CONFIGURATIONS = [ # => co-07: 3 configurations spanning "does not overlap" to "always overlaps"
QuorumConfig("weak (N=3, W=1, R=1)", n=3, w=1, r=1), # => 1+1=2, NOT > 3 -- read and write sets can miss each other entirely
QuorumConfig("classic quorum (N=3, W=2, R=2)", n=3, w=2, r=2), # => 2+2=4, > 3 -- guaranteed overlap, the standard "QUORUM" setting
QuorumConfig("write-all, read-one (N=3, W=3, R=1)", n=3, w=3, r=1), # => 3+1=4, > 3 -- also guaranteed, but W=3 blocks on EVERY replica
] # => 3 configs, same N=3 -- only the W/R split changes whether overlap is guaranteed
def main() -> None: # => entry point -- runs only when this file executes directly, not on import
for config in CONFIGURATIONS: # => co-07: evaluates the SAME overlap condition against every configuration
strong = guarantees_strong_consistency(config) # => runs the W + R > N check for this one configuration
print(f"{config.label}: W+R={config.w + config.r}, N={config.n}, strongly consistent = {strong}") # => Output line, one per config
assert guarantees_strong_consistency(CONFIGURATIONS[0]) is False # => co-07: W=1,R=1 on N=3 can genuinely miss the latest write
assert guarantees_strong_consistency(CONFIGURATIONS[1]) is True # => co-07: the standard QUORUM/QUORUM setting always overlaps
assert guarantees_strong_consistency(CONFIGURATIONS[2]) is True # => co-07: overlap holds, but at the cost of blocking on ALL replicas to write
print("Only configurations where W + R > N guarantee the read observes the latest committed write") # => Output line
if __name__ == "__main__": # => guards against running main() on `import example`
main() # => runs everything above when executed as a scriptRun: python3 example.py (no external dependency -- pure Python 3.13 standard library)
Output:
weak (N=3, W=1, R=1): W+R=2, N=3, strongly consistent = False
classic quorum (N=3, W=2, R=2): W+R=4, N=3, strongly consistent = True
write-all, read-one (N=3, W=3, R=1): W+R=4, N=3, strongly consistent = True
Only configurations where W + R > N guarantee the read observes the latest committed writeKey takeaway: W + R > N is necessary but says nothing about latency cost -- (N=3, W=3, R=1)
satisfies the inequality just as well as (N=3, W=2, R=2), but blocks every write on all 3 replicas
instead of a majority, which is strictly worse for write availability.
Why it matters: This is the exact algebra behind Cassandra's QUORUM consistency level (Example 39) and DynamoDB's tunable read/write settings -- misconfiguring W and R so they no longer satisfy
the inequality is a silent failure mode: nothing errors, reads simply start returning stale data
intermittently, with no obvious signal pointing back to the broken configuration. Example 75 (Advanced
tier) measures the latency this math trades away as W increases, making the cost side of the
tradeoff concrete.
Example 39: Cassandra Quorum Tuning
ex-39 · exercises co-07
Write at QUORUM, then read the same row at ONE and at QUORUM -- both consistency levels are real,
runnable options on a live Cassandra cluster, not just a theoretical setting.
learning/code/ex-39-cassandra-quorum-tuning/example.py
"""Example 39: Cassandra Quorum Tuning.""" # => co-07: this file's own purpose, doubling as its module __doc__
from __future__ import annotations # => hygiene: postpones annotation evaluation, interpreter-version-agnostic
import time # => co-07: measures wall-clock latency at each consistency level, honestly, on a single-node cluster
from cassandra.cluster import Cluster, Session # => co-07: cassandra-driver, the Apache Software Foundation-maintained Python driver
from cassandra.query import ConsistencyLevel, SimpleStatement # => co-07: per-query consistency-level knobs
def setup_keyspace_and_table(session: Session) -> None: # => co-07: creates a keyspace/table this example owns exclusively
"""Create a dedicated keyspace and table for this quorum-tuning demonstration.""" # => documents the contract
session.execute( # => co-07: replication_factor 1 -- this is a SINGLE-NODE local cluster, not a production topology
"CREATE KEYSPACE IF NOT EXISTS nosqldb WITH replication = " # => the keyspace-level replication strategy clause
"{'class': 'SimpleStrategy', 'replication_factor': 1}" # => concatenated onto the line above -- ONE CQL statement string
) # => closes the execute() call -- the keyspace now exists, idempotently
session.set_keyspace("nosqldb") # => selects the keyspace for the statements below
session.execute("DROP TABLE IF EXISTS quorum_demo") # => resets state -- this example is fully self-contained
session.execute("CREATE TABLE quorum_demo (id int PRIMARY KEY, value text)") # => a minimal table for this demonstration
def write_at_quorum(session: Session, row_id: int, value: str) -> float: # => co-07: writes at QUORUM, returns elapsed seconds
"""Insert a row with consistency level QUORUM, returning wall-clock latency.""" # => documents the contract
statement = SimpleStatement( # => co-07: wraps the query so a per-statement consistency level can be attached
"INSERT INTO quorum_demo (id, value) VALUES (%s, %s)", # => positional CQL placeholders, bound below
consistency_level=ConsistencyLevel.QUORUM, # => co-07: WRITE at QUORUM -- must be acked by a majority of replicas
) # => closes the SimpleStatement -- query text and consistency level bundled together
start = time.perf_counter() # => marks the start of the timed write
session.execute(statement, (row_id, value)) # => co-07: the actual timed QUORUM write
return time.perf_counter() - start # => elapsed wall-clock seconds for this one write
def read_at_level(session: Session, row_id: int, level: int) -> tuple[str | None, float]: # => co-07: reads at a GIVEN level, returns (value, elapsed)
"""Read a row at the given consistency level, returning (value, elapsed_seconds).""" # => documents the contract
statement = SimpleStatement( # => co-07: wraps the query so a per-statement consistency level can be attached
"SELECT value FROM quorum_demo WHERE id = %s", # => a single positional placeholder, bound below
consistency_level=level, # => co-07: READ at the level the caller specifies -- ONE or QUORUM, contrasted below
) # => closes the SimpleStatement -- query text and consistency level bundled together
start = time.perf_counter() # => marks the start of the timed read
row = session.execute(statement, (row_id,)).one() # => co-07: the actual timed read at this consistency level
elapsed = time.perf_counter() - start # => elapsed wall-clock seconds for this one read
return (row.value if row else None), elapsed # => hand back both the value read and how long it took
def main() -> None: # => entry point -- runs only when this file executes directly, not on import
cluster = Cluster(["127.0.0.1"], port=9042) # => connects to the local single-node Cassandra 5.0 cluster
session = cluster.connect() # => opens a session against that cluster
setup_keyspace_and_table(session) # => sets up the dedicated keyspace/table fixture
write_latency = write_at_quorum(session, 1, "quorum-committed-value") # => co-07: writes at QUORUM, timed
value_at_one, latency_one = read_at_level(session, 1, ConsistencyLevel.ONE) # => co-07: reads at ONE -- consults just 1 replica
value_at_quorum, latency_quorum = read_at_level(session, 1, ConsistencyLevel.QUORUM) # => co-07: reads at QUORUM -- consults a majority
assert value_at_one == "quorum-committed-value" # => co-07: on THIS single-node cluster, ONE and QUORUM read the same single replica
assert value_at_quorum == "quorum-committed-value" # => co-07: both levels agree here -- the contrast is in LATENCY, not correctness, on 1 node
print(f"Write at QUORUM: {write_latency * 1000:.2f}ms") # => Output line -- exact ms value machine-dependent
print(f"Read at ONE: {latency_one * 1000:.2f}ms, value={value_at_one}") # => Output line
print(f"Read at QUORUM: {latency_quorum * 1000:.2f}ms, value={value_at_quorum}") # => Output line
# => co-07: on a REAL multi-node cluster, QUORUM reads/writes coordinate across multiple replicas
# => and cost measurably more latency than ONE -- this single-node cluster demonstrates the API
# => and correctness contract; Example 75 (Advanced tier) measures the latency GAP itself, simulated
# => across a genuinely multi-replica model
cluster.shutdown() # => always release what you open
if __name__ == "__main__": # => guards against running main() on `import example`
main() # => runs everything above when executed as a scriptRun: python3 example.py (representative output, run against cassandra-driver==3.30.1 and a local Cassandra 5.0.8
Docker container)
Output:
Write at QUORUM: 2.93ms
Read at ONE: 2.75ms, value=quorum-committed-value
Read at QUORUM: 2.20ms, value=quorum-committed-valueKey takeaway: On a single-node local cluster, ConsistencyLevel.ONE and ConsistencyLevel.QUORUM
consult the same single replica and read identical values -- the exact millisecond timings above are
not meaningfully different here; the API surface and correctness contract are what this example
demonstrates, not a genuine multi-node latency gap.
Why it matters: ConsistencyLevel is a per-query knob, not a cluster-wide setting -- one query can
run at ONE for a low-stakes read while another runs at QUORUM for a stakes-matter read, on the same
table, in the same application. This fine-grained control is Cassandra's real answer to the false
choice between "always fast" and "always correct": a dashboard counter and a balance check can coexist
on identical hardware with entirely different consistency guarantees, tuned independently per query.
Example 40: Leaderless Replication, Simulated
ex-40 · exercises co-13
Context: any of 3 replicas can accept a write directly -- no leader routes it -- and a quorum read reconciles whatever the queried replicas currently hold. Simulated in pure Python since dynamo-style leaderless replication has no single-process, single-Docker-container way to observe honestly; the replica objects here stand in directly for what a real Dynamo-style cluster's nodes do.
%% Color Palette: Blue #0173B2, Orange #DE8F05, Teal #029E73 -- color-blind friendly, WCAG AA
flowchart LR
W["write(v1)"]:::orange
W -->|W=2| R1(("replica-1")):::blue
W -->|W=2| R2(("replica-2")):::blue
R3(("replica-3<br/>not written")):::teal
Rd["read, R=2"]:::orange
R1 --> Rd
R2 --> Rd
classDef blue fill:#0173B2,stroke:#000000,color:#FFFFFF,stroke-width:2px
classDef orange fill:#DE8F05,stroke:#000000,color:#FFFFFF,stroke-width:2px
classDef teal fill:#029E73,stroke:#000000,color:#FFFFFF,stroke-width:2px
learning/code/ex-40-leaderless-replication-sim/example.py
"""Example 40: Leaderless Replication, Simulated.""" # => co-13: this file's own purpose, doubling as its module __doc__
from __future__ import annotations # => hygiene: postpones annotation evaluation, interpreter-version-agnostic
from dataclasses import dataclass # => co-13: a typed replica -- no single one is designated "the leader"
@dataclass # => intentionally MUTABLE -- a replica's own stored value changes as writes arrive
class Replica: # => co-13: any replica can accept a write directly -- there is no leader to route through
name: str # => a human-readable label, e.g. "replica-1"
value: str | None = None # => co-13: this replica's OWN local value, may be None if it never saw a write
def write_to_w_replicas(replicas: list[Replica], value: str, w: int) -> None: # => co-13: ANY w of the replicas accept directly
"""Write value to the first w replicas directly -- no leader coordinates this.""" # => documents the contract
for replica in replicas[:w]: # => co-13: the write fans out to w replicas AT ONCE, none of them is "the" leader
replica.value = value # => co-13: each targeted replica accepts the write independently
def read_from_r_replicas(replicas: list[Replica], r: int) -> str | None: # => co-13: a quorum READ reconciles what it sees
"""Read from r replicas and reconcile via simple majority-of-non-null values.""" # => documents the contract
observed = [replica.value for replica in replicas[:r] if replica.value is not None] # => co-13: collects whatever THESE r replicas currently hold
if not observed: # => none of the r replicas queried have seen a write yet
return None # => a genuinely empty read -- no replica in the queried set has data
return max(observed, key=observed.count) # => co-13: the value appearing most often among the r replicas queried wins
def main() -> None: # => entry point -- runs only when this file executes directly, not on import
replicas = [Replica("replica-1"), Replica("replica-2"), Replica("replica-3")] # => co-13: N=3, no replica is special
write_to_w_replicas(replicas, "v1", w=2) # => co-13: writes "v1" DIRECTLY to replica-1 and replica-2, W=2
for replica in replicas: # => prints each replica's own local state right after the write
print(f"{replica.name}: {replica.value}") # => Output line, one per replica -- replica-3 still None
assert replicas[2].value is None # => co-13: replica-3 was NOT part of the write quorum -- it has nothing yet
read_result = read_from_r_replicas(replicas, r=2) # => co-13: R=2 -- reads from replica-1 and replica-2, the SAME two that got the write
# => co-13: W (2) + R (2) > N (3) -- the read quorum overlaps the write quorum, so it MUST observe v1
assert read_result == "v1" # => co-13: the client observes the latest write once R and W quorums overlap
print(f"Read from {2} replicas (overlapping the write quorum): {read_result}") # => Output: Read from 2 replicas (overlapping the write quorum): v1
if __name__ == "__main__": # => guards against running main() on `import example`
main() # => runs everything above when executed as a scriptRun: python3 example.py (no external dependency -- pure Python 3.13 standard library)
Output:
replica-1: v1
replica-2: v1
replica-3: None
Read from 2 replicas (overlapping the write quorum): v1Key takeaway: Unlike Example 23's leader-follower model, no single replica ever "owns" write
order here -- the client itself is responsible for fanning writes out to W replicas and reconciling
R replicas on read, which is exactly the extra responsibility Dynamo-style leaderless replication
pushes onto the client (or a coordinator node acting on the client's behalf).
Why it matters: Cassandra's own replication model is leaderless in exactly this sense -- any node
can coordinate a write, and the W/R settings from Example 38 are this quorum math applied directly.
The tradeoff for removing the single-leader bottleneck is that the client (or a coordinator acting for
it) now owns the responsibility for fanning out writes and reconciling reads, work a leader-follower
system like Example 23's would have handled for it.
Example 41: LWW Conflict Resolution
ex-41 · exercises co-14
Two concurrent writes to the same key resolve by last-write-wins -- the later timestamp wins, and the earlier write is silently dropped with no merge and no error.
learning/code/ex-41-lww-conflict-resolution/example.py
"""Example 41: Last-Write-Wins Conflict Resolution.""" # => co-14: this file's own purpose, doubling as its module __doc__
from __future__ import annotations # => hygiene: postpones annotation evaluation, interpreter-version-agnostic
from dataclasses import dataclass # => co-14: a typed write -- pairs a value with the timestamp LWW resolves by
@dataclass(frozen=True) # => frozen -- a write, once issued, is an immutable historical fact
class TimestampedWrite: # => co-14: exactly what LWW needs to pick a winner -- a value and when it happened
value: str # => the data this write attempted to set
timestamp: float # => co-14: the deciding factor -- a later timestamp always wins, regardless of arrival order
def resolve_lww(writes: list[TimestampedWrite]) -> TimestampedWrite: # => co-14: picks the single winner among concurrent writes
"""Resolve concurrent writes to the same key by last-write-wins, on timestamp alone.""" # => documents the contract
return max(writes, key=lambda w: w.timestamp) # => co-14: the LATEST timestamp wins -- every other write is SILENTLY dropped
def main() -> None: # => entry point -- runs only when this file executes directly, not on import
# Two clients write to the SAME key concurrently -- neither saw the other's write before sending.
write_from_client_a = TimestampedWrite(value="blue", timestamp=1000.5) # => co-14: client A's write, timestamped first
write_from_client_b = TimestampedWrite(value="green", timestamp=1000.9) # => co-14: client B's write, timestamped LATER
winner = resolve_lww([write_from_client_a, write_from_client_b]) # => co-14: LWW resolves the two concurrent writes to ONE value
assert winner.value == "green" # => co-14: the later timestamp (1000.9 > 1000.5) wins -- "blue" is silently dropped
print(f"LWW winner: {winner.value} (timestamp={winner.timestamp})") # => Output: LWW winner: green (timestamp=1000.9)
print(f"Dropped silently: {write_from_client_a.value} (timestamp={write_from_client_a.timestamp}) -- NO merge, NO error raised") # => Output line
# => co-14: this is LWW's real cost -- client A's "blue" write is GONE, with no record it ever
# => existed and no error surfaced to client A that its write lost -- the tradeoff for a
# => cheap, deterministic, coordination-free conflict resolution rule
if __name__ == "__main__": # => guards against running main() on `import example`
main() # => runs everything above when executed as a scriptRun: python3 example.py (no external dependency -- pure Python 3.13 standard library)
Output:
LWW winner: green (timestamp=1000.9)
Dropped silently: blue (timestamp=1000.5) -- NO merge, NO error raisedKey takeaway: LWW's cheapness is exactly what makes it dangerous for concurrent edits a user would expect to be merged (two people editing a shared cart) -- it is the right tool when "one wins, the other is genuinely wrong" is an acceptable model, and the wrong tool when both writes carry real information worth preserving.
Why it matters: Cassandra defaults to LWW (by cell timestamp) for concurrent writes to the same column -- understanding exactly what gets silently dropped is required before relying on that default for anything where losing a write silently is unacceptable, like two support agents both updating the same ticket's status field. Clock skew between nodes makes the failure mode worse in practice: a write with a slightly-behind clock can lose to an earlier write with a slightly-ahead one, an ordering a human would call backward.
Example 42: Vector Clock Conflict Detection
ex-42 · exercises co-15
A vector clock DETECTS a genuine conflict between two replicas' divergent histories -- unlike LWW, it never silently picks a winner; the application (or a human) must still decide.
%% Color Palette: Blue #0173B2, Orange #DE8F05, Teal #029E73 -- color-blind friendly, WCAG AA
flowchart TD
A["clock_from_a<br/>{A:1, B:0}"]:::blue
B["clock_from_b<br/>{A:0, B:1}"]:::orange
C{"Compare"}:::teal
A --> C
B --> C
C -->|neither dominates| D["CONCURRENT<br/>app must merge"]:::teal
classDef blue fill:#0173B2,stroke:#000000,color:#FFFFFF,stroke-width:2px
classDef orange fill:#DE8F05,stroke:#000000,color:#FFFFFF,stroke-width:2px
classDef teal fill:#029E73,stroke:#000000,color:#FFFFFF,stroke-width:2px
learning/code/ex-42-vector-clock-detect-conflict/example.py
"""Example 42: Vector Clock Detects a Conflict.""" # => co-15: this file's own purpose, doubling as its module __doc__
from __future__ import annotations # => hygiene: postpones annotation evaluation, interpreter-version-agnostic
from dataclasses import dataclass # => co-15: a typed vector clock -- one counter per replica that touched this key
from enum import Enum, auto # => co-15: the 3-way comparison result vector clocks can produce
class ClockOrder(Enum): # => co-15: comparing two vector clocks yields ONE of these 3 outcomes, never a 4th
BEFORE = auto() # => this clock happened strictly BEFORE the other -- a causal ancestor
AFTER = auto() # => this clock happened strictly AFTER the other -- a causal descendant
CONCURRENT = auto() # => co-15: NEITHER dominates the other -- a genuine, unresolved conflict
@dataclass(frozen=True) # => frozen -- a vector clock snapshot is a stated fact about causal history
class VectorClock: # => co-15: one counter per replica -- e.g. {"replica-A": 2, "replica-B": 1}
counters: dict[str, int] # => co-15: replica name -> how many writes THAT replica has causally seen
def compare(a: VectorClock, b: VectorClock) -> ClockOrder: # => co-15: the core vector-clock comparison algorithm
"""Compare two vector clocks, returning BEFORE, AFTER, or CONCURRENT.""" # => documents the contract, no runtime output
replicas = set(a.counters) | set(b.counters) # => co-15: every replica EITHER clock has ever counted
a_le_b = all(a.counters.get(r, 0) <= b.counters.get(r, 0) for r in replicas) # => co-15: a dominates b on NO dimension
b_le_a = all(b.counters.get(r, 0) <= a.counters.get(r, 0) for r in replicas) # => co-15: b dominates a on NO dimension
if a_le_b and not b_le_a: # => co-15: a is <= b on every dimension, strictly less on at least one -- a happened BEFORE b
return ClockOrder.BEFORE
if b_le_a and not a_le_b: # => co-15: the mirror image -- a happened AFTER b
return ClockOrder.AFTER
if a_le_b and b_le_a: # => co-15: identical on every dimension -- treated as BEFORE (no real conflict, same causal point)
return ClockOrder.BEFORE
return ClockOrder.CONCURRENT # => co-15: NEITHER dominates -- a and b happened independently, a genuine conflict
def main() -> None: # => entry point -- runs only when this file executes directly, not on import
# Replica A writes, then replica B writes WITHOUT having seen A's write -- a genuine concurrent edit.
clock_from_a = VectorClock({"replica-A": 1, "replica-B": 0}) # => co-15: replica-A's own write, replica-B's counter still 0
clock_from_b = VectorClock({"replica-A": 0, "replica-B": 1}) # => co-15: replica-B's own write, replica-A's counter still 0
order = compare(clock_from_a, clock_from_b) # => co-15: neither clock dominates -- A doesn't know about B, B doesn't know about A
assert order == ClockOrder.CONCURRENT # => co-15: correctly flagged as CONCURRENT -- a genuine, unresolved conflict
print(f"clock_from_a vs clock_from_b: {order.name}") # => Output: clock_from_a vs clock_from_b: CONCURRENT
# => co-15: unlike Example 41's LWW, NOTHING was auto-resolved here -- the vector clock only
# => DETECTS the conflict; an application-level merge function (or human) must still decide
# => the winner, or merge both values
# A causally-ordered pair, for contrast: replica-A writes, then replica-A writes AGAIN, incrementing its own counter.
clock_second_from_a = VectorClock({"replica-A": 2, "replica-B": 0}) # => co-15: strictly ahead of clock_from_a on replica-A's own counter
order2 = compare(clock_from_a, clock_second_from_a) # => co-15: clock_from_a is causally BEFORE clock_second_from_a
assert order2 == ClockOrder.BEFORE # => co-15: a genuinely ordered pair, correctly NOT flagged as a conflict
print(f"clock_from_a vs clock_second_from_a: {order2.name}") # => Output: clock_from_a vs clock_second_from_a: BEFORE
if __name__ == "__main__": # => guards against running main() on `import example`
main() # => runs everything above when executed as a scriptRun: python3 example.py (no external dependency -- pure Python 3.13 standard library)
Output:
clock_from_a vs clock_from_b: CONCURRENT
clock_from_a vs clock_second_from_a: BEFOREKey takeaway: A vector clock's honesty is also its cost -- it never silently drops data the way
LWW does, but it also never resolves anything on its own; the calling application must handle the
CONCURRENT case explicitly, every time.
Why it matters: Amazon's original Dynamo paper used exactly this technique for its shopping cart
-- a genuinely concurrent edit surfaces to the application as "here are two versions, please merge
them," rather than silently losing one, which matters far more for a cart than for a page-view counter.
The cost is real: every replica must carry a growing vector clock, and every read path must be prepared
to handle the CONCURRENT case explicitly, work that LWW's silent-drop approach never asks the
application to do.
Example 43: CRDT G-Counter
ex-43 · exercises co-16
A grow-only counter CRDT merges deterministically from two replicas, regardless of merge order --
max() per replica slot, summed, with zero application-level conflict code required.
%% Color Palette: Blue #0173B2, Orange #DE8F05, Teal #029E73 -- color-blind friendly, WCAG AA
flowchart LR
A["replica-A<br/>{A:2}"]:::orange
B["replica-B<br/>{B:1}"]:::blue
A -->|merge| M["max per slot<br/>{A:2, B:1}"]:::teal
B -->|merge| M
M --> V["value = 3<br/>order-independent"]:::teal
classDef blue fill:#0173B2,stroke:#000000,color:#FFFFFF,stroke-width:2px
classDef orange fill:#DE8F05,stroke:#000000,color:#FFFFFF,stroke-width:2px
classDef teal fill:#029E73,stroke:#000000,color:#FFFFFF,stroke-width:2px
learning/code/ex-43-crdt-g-counter/example.py
"""Example 43: CRDT G-Counter (Grow-Only Counter).""" # => co-16: this file's own purpose, doubling as its module __doc__
from __future__ import annotations # => hygiene: postpones annotation evaluation, interpreter-version-agnostic
from dataclasses import dataclass, field # => co-16: a typed G-Counter -- one slot per replica, merged by taking the max per slot
@dataclass # => intentionally MUTABLE -- a replica's own counter grows as it counts local increments
class GCounter: # => co-16: a Conflict-free Replicated Data Type -- grow-only, never decrements
replica_id: str # => this counter's OWN identity -- the slot it is allowed to increment
counts: dict[str, int] = field(default_factory=dict[str, int]) # => co-16: replica_id -> that replica's own local count
def increment(self) -> None: # => co-16: a replica may ONLY increment its OWN slot, never another's
self.counts[self.replica_id] = self.counts.get(self.replica_id, 0) + 1 # => co-16: bumps this replica's own counter by 1
def value(self) -> int: # => co-16: the counter's current total is the SUM across every replica's slot
return sum(self.counts.values()) # => co-16: total = sum of every replica's independent contribution
def merge(self, other: GCounter) -> GCounter: # => co-16: the CRDT merge function -- deterministic, commutative, associative
merged_counts = dict(self.counts) # => starts from this counter's own state
for replica_id, count in other.counts.items(): # => co-16: merges in EVERY slot from the other replica's state
merged_counts[replica_id] = max(merged_counts.get(replica_id, 0), count) # => co-16: MAX per slot -- never double-counts, never loses an increment
result = GCounter(self.replica_id) # => a fresh counter object to hold the merged result
result.counts = merged_counts # => assigns the merged per-replica slots
return result # => hand back the merged counter -- no application-level conflict code was needed
def main() -> None: # => entry point -- runs only when this file executes directly, not on import
replica_a = GCounter("replica-A") # => co-16: replica A's own counter, starts empty
replica_b = GCounter("replica-B") # => co-16: replica B's own INDEPENDENT counter, starts empty
replica_a.increment() # => co-16: A counts one local event
replica_a.increment() # => co-16: A counts a second local event -- A's own slot is now 2
replica_b.increment() # => co-16: B counts one local event, INDEPENDENTLY of A -- B's own slot is now 1
merged_a_then_b = replica_a.merge(replica_b) # => co-16: merge in ONE order -- A first, then B's state folded in
merged_b_then_a = replica_b.merge(replica_a) # => co-16: merge in the OPPOSITE order -- B first, then A's state folded in
assert merged_a_then_b.value() == 3 # => co-16: 2 (from A) + 1 (from B) = 3, regardless of merge order
assert merged_b_then_a.value() == 3 # => co-16: the SAME total -- merge order genuinely does not matter
print(f"merge(A, B).value() = {merged_a_then_b.value()}") # => Output: merge(A, B).value() = 3
print(f"merge(B, A).value() = {merged_b_then_a.value()}") # => Output: merge(B, A).value() = 3
print("Merge is commutative: both orders converge to the identical total, with zero app-level merge code") # => Output line
if __name__ == "__main__": # => guards against running main() on `import example`
main() # => runs everything above when executed as a scriptRun: python3 example.py (no external dependency -- pure Python 3.13 standard library)
Output:
merge(A, B).value() = 3
merge(B, A).value() = 3
Merge is commutative: both orders converge to the identical total, with zero app-level merge codeKey takeaway: The G-Counter never allows a decrement -- that constraint is precisely what makes
max()-per-slot a correct, commutative merge; a naive shared integer counter incremented on two
replicas and merged by summing the raw values would double-count on replay, which a CRDT is designed
to avoid entirely.
Why it matters: G-Counters underpin real distributed like/view/vote counters at scale -- Example 73 (Advanced tier) contrasts this CRDT-based auto-merge directly against the vector-clock-detected, app-must-merge path from Example 42. Choosing a CRDT over a plain replicated integer counter is what makes a counter genuinely safe to increment concurrently from many regions without a coordinator serializing every update, at the cost of only ever growing, never allowing a real decrement.
Example 44: CRDT LWW-Register
ex-44 · exercises co-16
An LWW-Register wraps Example 41's last-write-wins rule in a formally mergeable CRDT type -- both replicas converge to the identical state, regardless of merge direction.
%% Color Palette: Blue #0173B2, Orange #DE8F05, Teal #029E73 -- color-blind friendly, WCAG AA
flowchart LR
R1["replica-1<br/>cart:3-items, ts=500.0"]:::orange
R2["replica-2<br/>cart:5-items, ts=503.2"]:::blue
R1 -->|merge| M["later timestamp wins<br/>cart:5-items"]:::teal
R2 -->|merge| M
classDef blue fill:#0173B2,stroke:#000000,color:#FFFFFF,stroke-width:2px
classDef orange fill:#DE8F05,stroke:#000000,color:#FFFFFF,stroke-width:2px
classDef teal fill:#029E73,stroke:#000000,color:#FFFFFF,stroke-width:2px
learning/code/ex-44-crdt-lww-register/example.py
"""Example 44: CRDT LWW-Register.""" # => co-16: this file's own purpose, doubling as its module __doc__
from __future__ import annotations # => hygiene: postpones annotation evaluation, interpreter-version-agnostic
from dataclasses import dataclass # => co-16: a typed register state -- a value plus the timestamp that set it
@dataclass(frozen=True) # => frozen -- a register STATE is a snapshot, merge always produces a NEW one
class LwwRegister: # => co-16: a CRDT wrapping LWW in a formally mergeable, deterministic type
value: str # => this register's current value, as this replica currently sees it
timestamp: float # => co-16: the deciding factor for merge, same rule Example 41 used, now wrapped as a CRDT
def merge(self, other: LwwRegister) -> LwwRegister: # => co-16: the CRDT merge function -- deterministic, commutative
if self.timestamp >= other.timestamp: # => co-16: ties break toward self -- an arbitrary but CONSISTENT rule both replicas apply identically
return self # => this register's value is already the (tied-or-later) winner
return other # => co-16: the other register's timestamp is strictly later -- it wins
def main() -> None: # => entry point -- runs only when this file executes directly, not on import
register_on_replica_1 = LwwRegister(value="cart:3-items", timestamp=500.0) # => co-16: replica 1's own local state
register_on_replica_2 = LwwRegister(value="cart:5-items", timestamp=503.2) # => co-16: replica 2's own INDEPENDENT, LATER state
merged_on_replica_1 = register_on_replica_1.merge(register_on_replica_2) # => co-16: replica 1 merges in replica 2's state
merged_on_replica_2 = register_on_replica_2.merge(register_on_replica_1) # => co-16: replica 2 merges in replica 1's state, OPPOSITE order
assert merged_on_replica_1.value == "cart:5-items" # => co-16: replica 2's later timestamp (503.2) wins, regardless of merge direction
assert merged_on_replica_2.value == "cart:5-items" # => co-16: the SAME winner on replica 2 -- deterministic convergence
assert merged_on_replica_1 == merged_on_replica_2 # => co-16: BOTH replicas now hold the IDENTICAL register state
print(f"Replica 1 merged value: {merged_on_replica_1.value}") # => Output: Replica 1 merged value: cart:5-items
print(f"Replica 2 merged value: {merged_on_replica_2.value}") # => Output: Replica 2 merged value: cart:5-items
print("Both replicas converged to the identical state -- no coordinator, no app-level merge decision needed") # => Output line
if __name__ == "__main__": # => guards against running main() on `import example`
main() # => runs everything above when executed as a scriptRun: python3 example.py (no external dependency -- pure Python 3.13 standard library)
Output:
Replica 1 merged value: cart:5-items
Replica 2 merged value: cart:5-items
Both replicas converged to the identical state -- no coordinator, no app-level merge decision neededKey takeaway: An LWW-Register is exactly Example 41's rule, but wrapped so merge is a pure
function any two replicas can call independently and always agree -- the difference between "a
resolution rule" and "a formal CRDT" is precisely this commutative, associative merge guarantee.
Why it matters: Not every CRDT is a counter -- a register (a single mutable value, like a user's
current display name or a device's current status) is one of the most common CRDT shapes, and this
example shows the LWW rule from Example 41 formalized into one. The formalization matters in practice
because a genuine CRDT's merge function must be provably commutative and associative for any number of
replicas merging in any order, a guarantee Example 41's ad hoc max()-by-timestamp comparison alone
does not make explicit.
Example 45: Cassandra Partition and Clustering Keys
ex-45 · exercises co-22
PRIMARY KEY ((sensor_id), reading_time) groups rows by sensor_id into one partition and orders
them within it by reading_time -- Cassandra enforces the clustering order regardless of insert order.
%% Color Palette: Orange #DE8F05, Teal #029E73 -- color-blind friendly, WCAG AA
flowchart TD
P["Partition: sensor-1"]:::orange
P --> C1["10:02:00 -- 22.1<br/>(newest)"]:::teal
P --> C2["10:01:00 -- 21.8"]:::teal
P --> C3["10:00:00 -- 21.5<br/>(oldest)"]:::teal
classDef orange fill:#DE8F05,stroke:#000000,color:#FFFFFF,stroke-width:2px
classDef teal fill:#029E73,stroke:#000000,color:#FFFFFF,stroke-width:2px
learning/code/ex-45-cassandra-table-partition-clustering/example.py
"""Example 45: Cassandra Table with Partition + Clustering Key.""" # => co-22: this file's own purpose, doubling as its module __doc__
from __future__ import annotations # => hygiene: postpones annotation evaluation, interpreter-version-agnostic
from cassandra.cluster import Cluster, Session # => co-22: cassandra-driver, the Apache Software Foundation-maintained Python driver
def setup_feed_table(session: Session) -> None: # => co-22: a partition-key + clustering-column table for a time-series feed
"""Create a keyspace and table modeling a per-sensor feed, ordered by reading time within a partition.""" # => documents contract
session.execute( # => a dedicated keyspace, replication_factor 1 on this single-node local cluster
"CREATE KEYSPACE IF NOT EXISTS nosqldb WITH replication = " # => the keyspace-level replication strategy clause
"{'class': 'SimpleStrategy', 'replication_factor': 1}" # => concatenated onto the line above -- ONE CQL statement string
) # => closes the execute() call -- the keyspace now exists, idempotently
session.set_keyspace("nosqldb") # => selects the keyspace for the statements below
session.execute("DROP TABLE IF EXISTS sensor_feed") # => resets state -- this example is fully self-contained
session.execute( # => co-22: PRIMARY KEY ((sensor_id), reading_time) -- sensor_id PARTITIONS, reading_time CLUSTERS within it
"CREATE TABLE sensor_feed (" # => opens the DDL's column-definition list
"sensor_id text, " # => the PARTITION key column -- decides which node holds a given sensor's rows
"reading_time timestamp, " # => the CLUSTERING key column -- orders rows WITHIN a partition
"temperature double, " # => a plain, non-key value column
"PRIMARY KEY ((sensor_id), reading_time)" # => co-22: partition key groups rows, clustering key ORDERS them within a partition
") WITH CLUSTERING ORDER BY (reading_time DESC)" # => co-22: newest reading first within each partition -- the common feed read
) # => closes the execute() call -- the table now exists with this exact partition + clustering layout
def insert_readings(session: Session) -> None: # => co-22: 3 readings for ONE sensor -- all land in the SAME partition
"""Insert 3 out-of-order readings for one sensor -- Cassandra stores them clustering-key-sorted regardless.""" # => documents contract
readings = [ # => co-22: deliberately inserted OUT of time order -- clustering order is enforced by the STORE, not insert order
("sensor-1", "2026-07-27 10:00:00", 21.5), # => reading 1, chronologically FIRST, inserted FIRST
("sensor-1", "2026-07-27 10:02:00", 22.1), # => reading 3, chronologically LAST, inserted SECOND
("sensor-1", "2026-07-27 10:01:00", 21.8), # => inserted LAST but clusters BETWEEN the two readings above
] # => 3 readings, deliberately out of chronological insert order -- clustering must re-sort them
for sensor_id, reading_time, temperature in readings: # => co-22: each INSERT targets the SAME partition key, sensor-1
session.execute( # => co-22: every row here lands in ONE partition -- Cassandra's clustering column sorts them on write
"INSERT INTO sensor_feed (sensor_id, reading_time, temperature) VALUES (%s, %s, %s)", # => positional CQL placeholders
(sensor_id, reading_time, temperature), # => bound in insert order, but STORED in clustering-key order
) # => closes this one execute() call -- runs once per reading, in the deliberately-scrambled loop order above
def main() -> None: # => entry point -- runs only when this file executes directly, not on import
cluster = Cluster(["127.0.0.1"], port=9042) # => connects to the local single-node Cassandra 5.0 cluster
session = cluster.connect() # => opens a session against that cluster
setup_feed_table(session) # => sets up the dedicated keyspace/table fixture
insert_readings(session) # => co-22: inserts 3 out-of-time-order readings, all in the sensor-1 partition
rows = list(session.execute("SELECT reading_time, temperature FROM sensor_feed WHERE sensor_id = %s", ("sensor-1",))) # => co-22: a single-partition scan
times = [str(row.reading_time) for row in rows] # => extracts the ordered reading times for inspection
print(f"Readings for sensor-1, newest first: {times}") # => Output line -- CLUSTERING ORDER BY DESC enforced regardless of insert order
assert times == sorted(times, reverse=True) # => co-22: rows come back CLUSTERING-KEY-ordered (newest first), NOT insert-ordered
assert len(rows) == 3 # => co-22: all 3 readings landed in the SAME partition, exactly as the schema intended
cluster.shutdown() # => always release what you open
if __name__ == "__main__": # => guards against running main() on `import example`
main() # => runs everything above when executed as a scriptRun: python3 example.py (representative output, run against cassandra-driver==3.30.1 and a local Cassandra 5.0.8
Docker container)
Output:
Readings for sensor-1, newest first: ['2026-07-27 10:02:00', '2026-07-27 10:01:00', '2026-07-27 10:00:00']Key takeaway: The clustering column's sort order is a storage-layer guarantee, not something
application code re-sorts after the fact -- CLUSTERING ORDER BY (reading_time DESC) means every
partition-scoped read comes back newest-first with zero client-side work.
Why it matters: This is the exact table shape a time-series or event-feed access pattern wants -- Examples 46 and 47 build directly on it to contrast a fast, correctly-scoped read against a rejected, partition-key-less one. Choosing the right clustering column up front is a one-way door in practice: reordering an existing table's clustering key means rebuilding the table entirely, so getting the "one query pattern this table exists to serve" decision right before data lands matters more here than in a relational schema with flexible secondary indexes.
Example 46: Cassandra Partition-Scoped Query
ex-46 · exercises co-22
A query that names the partition key routes directly to that one partition -- fast, and the shape every Cassandra table should be designed around.
learning/code/ex-46-cassandra-partition-query/example.py
"""Example 46: Cassandra Partition-Scoped Query.""" # => co-22: this file's own purpose, doubling as its module __doc__
from __future__ import annotations # => hygiene: postpones annotation evaluation, interpreter-version-agnostic
import time # => co-22: measures wall-clock latency for a single-partition read, honestly, on this local cluster
from cassandra.cluster import Cluster, Session # => co-22: cassandra-driver, the Apache Software Foundation-maintained Python driver
def setup_orders_table(session: Session) -> None: # => co-22: a partition-key-per-customer table, many rows per partition
"""Create a table partitioned by customer_id, with 500 rows spread across 20 customers.""" # => documents the contract
session.execute( # => a dedicated keyspace, replication_factor 1 on this single-node local cluster
"CREATE KEYSPACE IF NOT EXISTS nosqldb WITH replication = " # => the keyspace-level replication strategy clause
"{'class': 'SimpleStrategy', 'replication_factor': 1}" # => concatenated onto the line above -- ONE CQL statement string
) # => closes the execute() call -- the keyspace now exists, idempotently
session.set_keyspace("nosqldb") # => selects the keyspace for the statements below
session.execute("DROP TABLE IF EXISTS customer_orders") # => resets state -- this example is fully self-contained
session.execute( # => co-22: PRIMARY KEY ((customer_id), order_id) -- customer_id PARTITIONS the 500 rows into 20 groups
"CREATE TABLE customer_orders (customer_id int, order_id int, amount double, PRIMARY KEY ((customer_id), order_id))"
) # => closes the execute() call -- the table now exists with this exact partition layout
for i in range(500): # => co-22: 500 rows, 20 distinct partitions -- 25 rows per customer partition
session.execute( # => each row lands in the partition matching (i % 20)
"INSERT INTO customer_orders (customer_id, order_id, amount) VALUES (%s, %s, %s)", # => positional CQL placeholders
(i % 20, i, float(i)), # => customer_id cycles 0-19, spreading rows evenly across all 20 partitions
) # => closes this one execute() call -- runs 500 times, once per generated row
def query_one_partition(session: Session, customer_id: int) -> tuple[int, float]: # => co-22: returns (row count, elapsed seconds)
"""Query rows scoped to exactly one partition key -- a fast, single-partition read.""" # => documents the contract
start = time.perf_counter() # => marks the start of the timed, partition-scoped read
rows = list(session.execute( # => co-22: the partition key (customer_id) is provided -- Cassandra routes DIRECTLY to that partition
"SELECT order_id, amount FROM customer_orders WHERE customer_id = %s", (customer_id,) # => a single positional placeholder, bound to customer_id
)) # => materializes the cursor -- exactly this ONE partition's rows, nothing else scanned
elapsed = time.perf_counter() - start # => elapsed wall-clock seconds for this single-partition scan
return len(rows), elapsed # => hand back both the row count and how long the scan took
def main() -> None: # => entry point -- runs only when this file executes directly, not on import
cluster = Cluster(["127.0.0.1"], port=9042) # => connects to the local single-node Cassandra 5.0 cluster
session = cluster.connect() # => opens a session against that cluster
setup_orders_table(session) # => sets up the 500-row, 20-partition fixture
row_count, elapsed = query_one_partition(session, customer_id=7) # => co-22: scoped to EXACTLY one of the 20 partitions
assert row_count == 25 # => co-22: 500 rows / 20 customers == 25 rows in customer 7's own partition, no more, no less
print(f"Partition-scoped query for customer_id=7: {row_count} rows in {elapsed * 1000:.2f}ms") # => Output line -- exact ms machine-dependent
# => co-22: because customer_id IS the partition key, Cassandra routes this query directly to the
# => single node/partition owning it -- it never has to fan out and scan every partition, which is
# => exactly the cost Example 47's missing-partition-key query pays instead
cluster.shutdown() # => always release what you open
if __name__ == "__main__": # => guards against running main() on `import example`
main() # => runs everything above when executed as a scriptRun: python3 example.py (representative output, run against cassandra-driver==3.30.1 and a local Cassandra 5.0.8
Docker container)
Output:
Partition-scoped query for customer_id=7: 25 rows in 2.56msKey takeaway: The row count (exactly 25 of 500) confirms the query touched only ONE of the 20 partitions -- this is what "the schema is designed around the access pattern" (co-08) looks like when applied to a wide-column store specifically.
Why it matters: A production Cassandra table's whole design goal is making every hot-path query this shape -- partition-key-scoped, fast, and never a cluster-wide scan. A table whose real-world query pattern does not fit its partition key (needing a filter on some other column) either forces an Example 47-style rejected scan, a costly secondary index, or a schema redesign -- exactly why co-08's "name the access patterns first" discipline matters most for Cassandra specifically.
Example 47: Cassandra Query Without a Partition Key
ex-47 · exercises co-22
Filter on a non-key column with no partition key in the WHERE clause -- Cassandra rejects it outright
unless ALLOW FILTERING is explicitly added, a deliberate safety rail against accidental full scans.
learning/code/ex-47-cassandra-query-without-partition-key/example.py
"""Example 47: Cassandra Query Without a Partition Key.""" # => co-22: this file's own purpose, doubling as its module __doc__
from __future__ import annotations # => hygiene: postpones annotation evaluation, interpreter-version-agnostic
from cassandra.cluster import Cluster, Session # => co-22: cassandra-driver, the Apache Software Foundation-maintained Python driver
from cassandra.protocol import InvalidRequest # => co-22: the exact exception Cassandra raises for a rejected filter query
def setup_orders_table(session: Session) -> None: # => co-22: reuses the SAME shape as Example 46 -- partitioned by customer_id
"""Create the same customer_id-partitioned table Example 46 used, with a handful of rows.""" # => documents the contract
session.execute( # => a dedicated keyspace, replication_factor 1 on this single-node local cluster
"CREATE KEYSPACE IF NOT EXISTS nosqldb WITH replication = " # => the keyspace-level replication strategy clause
"{'class': 'SimpleStrategy', 'replication_factor': 1}" # => concatenated onto the line above -- ONE CQL statement string
) # => closes the execute() call -- the keyspace now exists, idempotently
session.set_keyspace("nosqldb") # => selects the keyspace for the statements below
session.execute("DROP TABLE IF EXISTS customer_orders_2") # => resets state -- this example is fully self-contained
session.execute( # => co-22: the SAME partition-key + no-secondary-index shape as Example 46
"CREATE TABLE customer_orders_2 (customer_id int, order_id int, amount double, PRIMARY KEY ((customer_id), order_id))"
) # => closes the execute() call -- the table now exists with this exact partition layout
for i in range(20): # => a small, sufficient fixture -- this example is about REJECTION, not row count
session.execute("INSERT INTO customer_orders_2 (customer_id, order_id, amount) VALUES (%s, %s, %s)", (i % 4, i, float(i)))
def try_query_without_partition_key(session: Session) -> bool: # => co-22: attempts a filter query that OMITS the partition key entirely
"""Attempt to filter on amount alone, with NO partition key in the WHERE clause -- expect rejection.""" # => documents contract
try: # => catches ONLY the specific rejection Cassandra's planner raises for this unsafe shape
list(session.execute("SELECT * FROM customer_orders_2 WHERE amount > %s", (10.0,))) # => co-22: no customer_id in the WHERE clause at all
return True # => unreachable in this example -- Cassandra rejects this shape by design
except InvalidRequest: # => co-22: Cassandra's query planner REFUSES a filter that would require scanning every partition
return False # => co-22: correctly rejected -- this is Cassandra protecting the cluster from an accidental full scan
def try_query_with_allow_filtering(session: Session) -> int: # => co-22: the SAME query, but with the explicit escape hatch
"""Re-run the same filter with ALLOW FILTERING -- Cassandra now permits it, but the caller opted in explicitly.""" # => documents contract
rows = list(session.execute("SELECT * FROM customer_orders_2 WHERE amount > %s ALLOW FILTERING", (10.0,))) # => co-22: ALLOW FILTERING is an explicit, deliberate opt-in
return len(rows) # => hand back the row count now that the scan was explicitly permitted
def main() -> None: # => entry point -- runs only when this file executes directly, not on import
cluster = Cluster(["127.0.0.1"], port=9042) # => connects to the local single-node Cassandra 5.0 cluster
session = cluster.connect() # => opens a session against that cluster
setup_orders_table(session) # => sets up the 20-row, partition-keyed fixture
rejected = not try_query_without_partition_key(session) # => co-22: confirms the plain filter query was rejected
assert rejected is True # => co-22: Cassandra refused the query -- it would have required scanning EVERY partition
print(f"Query without partition key: rejected = {rejected} (InvalidRequest raised)") # => Output: Query without partition key: rejected = True (InvalidRequest raised)
row_count = try_query_with_allow_filtering(session) # => co-22: the SAME filter, now with the explicit opt-in
assert row_count == 9 # => co-22: 9 of the 20 rows (order_id 11-19) have amount > 10.0
print(f"Same query WITH ALLOW FILTERING: {row_count} rows returned (explicit opt-in to a full scan)") # => Output: Same query WITH ALLOW FILTERING: 9 rows returned (explicit opt-in to a full scan)
# => co-22: ALLOW FILTERING does not make the underlying full-cluster scan cheap -- it only removes
# => the SAFETY RAIL that stops a query from accidentally doing one; production code should treat
# => needing it as a signal the schema's partition key does not fit this access pattern
cluster.shutdown() # => always release what you open
if __name__ == "__main__": # => guards against running main() on `import example`
main() # => runs everything above when executed as a scriptRun: python3 example.py (representative output, run against cassandra-driver==3.30.1 and a local Cassandra 5.0.8
Docker container)
Output:
Query without partition key: rejected = True (InvalidRequest raised)
Same query WITH ALLOW FILTERING: 9 rows returned (explicit opt-in to a full scan)Key takeaway: Cassandra rejecting an accidental full-cluster scan by default is a deliberate
design choice, not a limitation -- ALLOW FILTERING exists as an explicit escape hatch for the rare
case where a full scan really is intended, never as a routine query pattern.
Why it matters: Needing ALLOW FILTERING in production code is almost always a modeling smell --
Example 59 (Advanced tier) covers the real alternative, a secondary index, and its own cross-node
coordination cost. Cassandra's decision to reject this query shape by default, rather than silently
allow a slow full-cluster scan the way some stores would, is a deliberate design choice that forces the
schema conversation to happen before a query ships, not after it degrades production latency.
Example 48: Cassandra Row TTL
ex-48 · exercises co-24
INSERT ... USING TTL <seconds> auto-expires a whole row after the window elapses -- the same idea as
Redis's key TTL (Examples 7-8), now applied per-row in a wide-column store.
learning/code/ex-48-cassandra-ttl-row/example.py
"""Example 48: Cassandra TTL Row.""" # => co-24: this file's own purpose, doubling as its module __doc__
from __future__ import annotations # => hygiene: postpones annotation evaluation, interpreter-version-agnostic
import time # => co-24: a real sleep past the TTL window -- the expiry must genuinely elapse, not be simulated
from cassandra.cluster import Cluster, Session # => co-24: cassandra-driver, the Apache Software Foundation-maintained Python driver
def setup_sessions_table(session: Session) -> None: # => co-24: a table this example owns exclusively
"""Create a dedicated table for this TTL demonstration.""" # => documents the contract, no runtime output
session.execute( # => a dedicated keyspace, replication_factor 1 on this single-node local cluster
"CREATE KEYSPACE IF NOT EXISTS nosqldb WITH replication = " # => the keyspace-level replication strategy clause
"{'class': 'SimpleStrategy', 'replication_factor': 1}" # => concatenated onto the line above -- ONE CQL statement string
) # => closes the execute() call -- the keyspace now exists, idempotently
session.set_keyspace("nosqldb") # => selects the keyspace for the statements below
session.execute("DROP TABLE IF EXISTS ttl_sessions") # => resets state -- this example is fully self-contained
session.execute("CREATE TABLE ttl_sessions (session_id text PRIMARY KEY, auth_token text)") # => a minimal table for this demonstration
def main() -> None: # => entry point -- runs only when this file executes directly, not on import
cluster = Cluster(["127.0.0.1"], port=9042) # => connects to the local single-node Cassandra 5.0 cluster
session = cluster.connect() # => opens a session against that cluster
setup_sessions_table(session) # => sets up the dedicated table fixture
session.execute( # => co-24: USING TTL <seconds> -- Cassandra auto-expires this ROW after the window elapses
"INSERT INTO ttl_sessions (session_id, auth_token) VALUES (%s, %s) USING TTL 5", # => note: "token" alone is a RESERVED CQL word -- "auth_token" sidesteps it
("sess-99", "tok-abc"), # => bound values -- the row this TTL applies to
) # => closes the execute() call -- this ONE row now carries its own 5-second TTL, other rows are unaffected
row_before = session.execute("SELECT auth_token FROM ttl_sessions WHERE session_id = %s", ("sess-99",)).one() # => reads immediately after insert
assert row_before is not None and row_before.auth_token == "tok-abc" # => co-24: the row is present right after the TTL insert, well before expiry
print(f"Immediately after insert (TTL=5s): token={row_before.auth_token}") # => Output: Immediately after insert (TTL=5s): token=tok-abc
time.sleep(6) # => co-24: waits PAST the 5-second TTL window -- a genuine elapsed expiry, not a simulated one
row_after = session.execute("SELECT auth_token FROM ttl_sessions WHERE session_id = %s", ("sess-99",)).one() # => reads again, after the TTL elapsed
assert row_after is None # => co-24: the row is GONE -- Cassandra auto-purged it once the TTL window passed
print(f"After the 5-second TTL elapses: row present = {row_after is not None}") # => Output: After the 5-second TTL elapses: row present = False
cluster.shutdown() # => always release what you open
if __name__ == "__main__": # => guards against running main() on `import example`
main() # => runs everything above when executed as a scriptRun: python3 example.py (representative output, run against cassandra-driver==3.30.1 and a local Cassandra 5.0.8
Docker container)
Output:
Immediately after insert (TTL=5s): token=tok-abc
After the 5-second TTL elapses: row present = FalseKey takeaway: token alone is a reserved CQL keyword -- this example's own auth_token column
name is a small, real reminder that a store's query language has its own reserved-word list, distinct
from any other store's.
Why it matters: Row-level TTL is Cassandra's answer to the same problem Redis's key TTL (co-24)
solves -- a session table, a rate-limit-window table, or any append-heavy table that should self-clean
without a scheduled deletion job that has to run, get monitored, and occasionally be debugged when it
falls behind. The tradeoff is that Cassandra's TTL-expired rows still occupy disk as tombstones until
compaction reclaims them, so a heavy-TTL workload needs compaction tuning, not just the USING TTL
clause alone.
Example 49: DynamoDB PutItem/GetItem
ex-49 · exercises co-22
put_item/get_item via boto3, against the official amazon/dynamodb-local Docker image -- the
same SDK calls a real AWS account would use, with only the endpoint URL differing.
learning/code/ex-49-dynamodb-put-get-item/example.py
"""Example 49: DynamoDB put_item/get_item.""" # => co-22: this file's own purpose, doubling as its module __doc__
from __future__ import annotations # => hygiene: postpones annotation evaluation, interpreter-version-agnostic
from typing import Any # => co-22: boto3's dynamic resource/client factory has no stubs pinned -- Any is the honest, explicit type
import boto3 # => co-22: the official AWS SDK for Python -- talks to real DynamoDB or amazon/dynamodb-local identically
def get_local_dynamodb_client() -> Any: # => co-22: connects to amazon/dynamodb-local, the official local-testing Docker image
"""Return a boto3 DynamoDB client pointed at a local dynamodb-local instance.""" # => documents the contract
return boto3.client( # => co-22: same boto3 API surface as real AWS -- only the endpoint_url differs
"dynamodb", # => the service name -- boto3 resolves this to DynamoDB's own API model at runtime
endpoint_url="http://localhost:8000", # => co-22: amazon/dynamodb-local listens here, no real AWS account needed
region_name="us-east-1", # => a region is required by the SDK even though dynamodb-local ignores it
aws_access_key_id="fake", # => co-22: dynamodb-local accepts any credentials -- no real AWS auth involved
aws_secret_access_key="fake", # => co-22: same -- purely local, no network call ever leaves this machine
) # => closes the boto3.client() call -- returns a low-level DynamoDB client bound to the local endpoint
def create_table(client: Any) -> None: # => co-22: a minimal single-key table -- partition key only, no sort key yet
"""Create a table keyed by a simple partition key (user_id), deleting any prior copy first.""" # => documents contract
existing = client.list_tables()["TableNames"] # => checks what already exists, for idempotent re-runs
if "Users" in existing: # => resets state -- this example is fully self-contained
client.delete_table(TableName="Users") # => co-22: removes any leftover table from a prior run
client.get_waiter("table_not_exists").wait(TableName="Users") # => blocks until the delete genuinely completes
client.create_table( # => co-22: PROVISIONED billing mode for dynamodb-local's default test setup
TableName="Users", # => the table this whole example operates against
KeySchema=[{"AttributeName": "user_id", "KeyType": "HASH"}], # => co-22: HASH == the partition key, DynamoDB's own term
AttributeDefinitions=[{"AttributeName": "user_id", "AttributeType": "S"}], # => S == string type
ProvisionedThroughput={"ReadCapacityUnits": 5, "WriteCapacityUnits": 5}, # => local-testing throughput, ignored by dynamodb-local's billing
) # => closes the create_table() call -- the table now exists, though not yet necessarily ACTIVE
client.get_waiter("table_exists").wait(TableName="Users") # => blocks until the table is genuinely ready for writes
def main() -> None: # => entry point -- runs only when this file executes directly, not on import
client = get_local_dynamodb_client() # => connects to the local dynamodb-local Docker container
create_table(client) # => sets up the fresh, empty Users table
client.put_item( # => co-22: PutItem writes a whole item, keyed by the partition key
TableName="Users", # => the table receiving this item
Item={"user_id": {"S": "user-1"}, "name": {"S": "Ada"}, "role": {"S": "engineer"}}, # => co-22: DynamoDB's typed attribute-value wire format
) # => closes the put_item() call -- the item is now durably stored under partition key user-1
response = client.get_item(TableName="Users", Key={"user_id": {"S": "user-1"}}) # => co-22: GetItem reads by the SAME partition key
item = response["Item"] # => the raw typed item DynamoDB returned
assert item["name"]["S"] == "Ada" # => co-22: the round trip returned exactly what was written
assert item["role"]["S"] == "engineer" # => confirms every attribute survived the round trip intact
print(f"Round trip on partition key user_id=user-1: name={item['name']['S']}, role={item['role']['S']}") # => Output: Round trip on partition key user_id=user-1: name=Ada, role=engineer
if __name__ == "__main__": # => guards against running main() on `import example`
main() # => runs everything above when executed as a scriptRun: python3 example.py (representative output, run against boto3==1.43.56 and the official
amazon/dynamodb-local Docker image)
Output:
Round trip on partition key user_id=user-1: name=Ada, role=engineerKey takeaway: amazon/dynamodb-local speaks the EXACT same wire protocol as real DynamoDB -- the
only code difference between a local test and a real deployment is the endpoint_url (and real
credentials), which makes it a genuinely faithful local-testing target, not an approximation.
Why it matters: DynamoDB's attribute-value wire format ({"S": "..."}, {"N": "..."}) is unusual
compared to MongoDB's plain BSON -- every attribute carries an explicit type tag, which this example's
item["name"]["S"] access pattern makes concrete. A team migrating from MongoDB's implicit typing to
DynamoDB's explicit one for the first time routinely underestimates how much this changes both the
client code and the mental model, since every read now has to unwrap a type-tagged value instead of
getting back a plain Python type directly.
Example 50: DynamoDB Composite-Key Query
ex-50 · exercises co-22
Query with a partition key plus a sort-key range condition returns items ordered by the sort key
within that partition -- DynamoDB's own name for what Cassandra calls a clustering column.
learning/code/ex-50-dynamodb-composite-key-query/example.py
"""Example 50: DynamoDB Composite Key Query.""" # => co-22: this file's own purpose, doubling as its module __doc__
from __future__ import annotations # => hygiene: postpones annotation evaluation, interpreter-version-agnostic
from typing import Any # => co-22: boto3's dynamic resource/client factory has no stubs pinned -- Any is the honest, explicit type
import boto3 # => co-22: the official AWS SDK for Python
from boto3.dynamodb.conditions import Key # => co-22: a typed helper for building KeyConditionExpressions safely
def get_local_dynamodb_resource() -> Any: # => co-22: the higher-level "resource" API, more Pythonic than raw "client" calls
"""Return a boto3 DynamoDB resource pointed at a local dynamodb-local instance.""" # => documents the contract
return boto3.resource( # => co-22: same endpoint/credential pattern as Example 49, via the resource API
"dynamodb", endpoint_url="http://localhost:8000", region_name="us-east-1", # => the local dynamodb-local endpoint, no real AWS account needed
aws_access_key_id="fake", aws_secret_access_key="fake", # => dynamodb-local accepts any credentials
) # => closes the boto3.resource() call -- returns a high-level resource bound to the local endpoint
def create_orders_table(resource: Any) -> None: # => co-22: a COMPOSITE key -- partition key PLUS a sort key
"""Create a table with a composite key: customer_id (partition) + order_ts (sort).""" # => documents the contract
client = resource.meta.client # => the underlying low-level client, needed for list/delete/waiter calls
if "Orders" in client.list_tables()["TableNames"]: # => resets state -- this example is fully self-contained
client.delete_table(TableName="Orders") # => removes any leftover table from a prior run
client.get_waiter("table_not_exists").wait(TableName="Orders") # => blocks until the delete genuinely completes
resource.create_table( # => co-22: KeySchema with BOTH a HASH (partition) and a RANGE (sort) key
TableName="Orders", # => the table this whole composite-key example uses
KeySchema=[ # => co-22: this is the composite key -- customer_id groups items, order_ts orders them within a group
{"AttributeName": "customer_id", "KeyType": "HASH"}, # => co-22: the PARTITION key
{"AttributeName": "order_ts", "KeyType": "RANGE"}, # => co-22: the SORT key, DynamoDB's term for a clustering column
], # => closes the KeySchema list -- exactly one HASH + one RANGE, the composite key's two halves
AttributeDefinitions=[ # => key attributes must be declared with their type up front
{"AttributeName": "customer_id", "AttributeType": "S"}, # => S == string type, the partition key's type
{"AttributeName": "order_ts", "AttributeType": "N"}, # => N == number, sorted numerically within the partition
], # => closes the AttributeDefinitions list -- both key attributes declared
ProvisionedThroughput={"ReadCapacityUnits": 5, "WriteCapacityUnits": 5}, # => local-testing throughput
) # => closes the create_table() call -- the table now exists, though not yet necessarily ACTIVE
client.get_waiter("table_exists").wait(TableName="Orders") # => blocks until the table is genuinely ready for writes
def main() -> None: # => entry point -- runs only when this file executes directly, not on import
resource = get_local_dynamodb_resource() # => connects to the local dynamodb-local Docker container
create_orders_table(resource) # => sets up the fresh, empty Orders table
table = resource.Table("Orders") # => the high-level Table object the resource API works through
for ts in [3, 1, 2]: # => co-22: inserted deliberately OUT of sort-key order -- DynamoDB stores them sorted regardless
table.put_item(Item={"customer_id": "cust-1", "order_ts": ts, "amount": ts * 10}) # => 3 items, all in the SAME partition
response = table.query( # => co-22: Query, scoped to a partition + a sort-key RANGE condition -- not a full scan
KeyConditionExpression=Key("customer_id").eq("cust-1") & Key("order_ts").between(1, 3), # => co-22: partition eq + sort-key range
) # => closes the query() call -- only cust-1's own partition, order_ts 1-3, ever gets scanned
items = response["Items"] # => the ordered items DynamoDB returned for this partition + range
timestamps = [int(item["order_ts"]) for item in items] # => extracts the sort-key values, in RETURN order
assert timestamps == [1, 2, 3] # => co-22: DynamoDB returns items ORDERED by the sort key within the partition, regardless of insert order
print(f"Ordered order_ts values within partition cust-1: {timestamps}") # => Output: Ordered order_ts values within partition cust-1: [1, 2, 3]
if __name__ == "__main__": # => guards against running main() on `import example`
main() # => runs everything above when executed as a scriptRun: python3 example.py (representative output, run against boto3==1.43.56 and the official
amazon/dynamodb-local Docker image)
Output:
Ordered order_ts values within partition cust-1: [1, 2, 3]Key takeaway: Query (partition key required, sort key optional) is fundamentally different from
Scan (no key required, reads the whole table) -- reaching for Scan in production is almost always
a modeling smell, the DynamoDB equivalent of Example 47's rejected Cassandra ALLOW FILTERING query.
Why it matters: A composite key answers "give me this customer's orders in a date range" with one
efficient, sorted Query -- exactly the shape Example 45's Cassandra clustering-column table answers
too, using DynamoDB's own vocabulary (HASH/RANGE instead of partition/clustering). Recognizing that
both stores solve this same access pattern with the same underlying idea, just different names, is what
lets a schema-design skill transfer between wide-column and key-value stores instead of feeling like
two unrelated systems to learn from scratch.
Example 51: DynamoDB Single Table, Two Entities
ex-51 · exercises co-23
One physical table stores a customer's profile AND their orders, distinguished by an overloaded sort-key prefix -- both entity types retrievable under the same partition key, in one query.
%% Color Palette: Blue #0173B2, Orange #DE8F05, Teal #029E73 -- color-blind friendly, WCAG AA
flowchart TD
PK["PK = CUSTOMER#42"]:::orange
PK --> SK1["SK = PROFILE<br/>name: Ada"]:::blue
PK --> SK2["SK = ORDER#2026-01<br/>amount: 50"]:::teal
PK --> SK3["SK = ORDER#2026-02<br/>amount: 75"]:::teal
classDef blue fill:#0173B2,stroke:#000000,color:#FFFFFF,stroke-width:2px
classDef orange fill:#DE8F05,stroke:#000000,color:#FFFFFF,stroke-width:2px
classDef teal fill:#029E73,stroke:#000000,color:#FFFFFF,stroke-width:2px
learning/code/ex-51-dynamodb-single-table-two-entities/example.py
"""Example 51: DynamoDB Single-Table Two Entities.""" # => co-23: this file's own purpose, doubling as its module __doc__
from __future__ import annotations # => hygiene: postpones annotation evaluation, interpreter-version-agnostic
from typing import Any # => co-23: boto3's dynamic resource/client factory has no stubs pinned -- Any is the honest, explicit type
import boto3 # => co-23: the official AWS SDK for Python
from boto3.dynamodb.conditions import Key # => co-23: a typed helper for building KeyConditionExpressions safely
def get_local_dynamodb_resource() -> Any: # => co-23: the higher-level "resource" API
"""Return a boto3 DynamoDB resource pointed at a local dynamodb-local instance.""" # => documents the contract
return boto3.resource( # => co-23: same endpoint/credential pattern as prior DynamoDB examples
"dynamodb", endpoint_url="http://localhost:8000", region_name="us-east-1", # => the local dynamodb-local endpoint, no real AWS account needed
aws_access_key_id="fake", aws_secret_access_key="fake", # => dynamodb-local accepts any credentials
) # => closes the boto3.resource() call -- returns a high-level resource bound to the local endpoint
def create_single_table(resource: Any) -> None: # => co-23: ONE table, sort key will carry overloaded entity-type prefixes
"""Create a single table with a generic PK/SK composite key, ready to hold multiple entity types.""" # => documents contract
client = resource.meta.client # => the underlying low-level client, needed for list/delete/waiter calls
if "AppTable" in client.list_tables()["TableNames"]: # => resets state -- this example is fully self-contained
client.delete_table(TableName="AppTable") # => removes any leftover table from a prior run
client.get_waiter("table_not_exists").wait(TableName="AppTable") # => blocks until the delete genuinely completes
resource.create_table( # => co-23: deliberately GENERIC attribute names -- "PK"/"SK" carry no entity-specific meaning
TableName="AppTable", # => the ONE table this whole single-table-design example uses
KeySchema=[{"AttributeName": "PK", "KeyType": "HASH"}, {"AttributeName": "SK", "KeyType": "RANGE"}], # => co-23: single-table design's signature generic key
AttributeDefinitions=[{"AttributeName": "PK", "AttributeType": "S"}, {"AttributeName": "SK", "AttributeType": "S"}], # => both key attributes are plain strings
ProvisionedThroughput={"ReadCapacityUnits": 5, "WriteCapacityUnits": 5}, # => local-testing throughput
) # => closes the create_table() call -- the table now exists, though not yet necessarily ACTIVE
client.get_waiter("table_exists").wait(TableName="AppTable") # => blocks until the table is genuinely ready for writes
def main() -> None: # => entry point -- runs only when this file executes directly, not on import
resource = get_local_dynamodb_resource() # => connects to the local dynamodb-local Docker container
create_single_table(resource) # => sets up the fresh, empty AppTable
table = resource.Table("AppTable") # => the high-level Table object the resource API works through
# Entity type 1: a customer PROFILE, sort key prefixed "PROFILE" -- co-23's overloaded sort-key pattern.
table.put_item(Item={"PK": "CUSTOMER#42", "SK": "PROFILE", "name": "Ada", "type": "profile"}) # => co-23: entity 1 under the SAME partition key
# Entity type 2: two of that SAME customer's ORDERS, sort key prefixed "ORDER#" -- a DIFFERENT entity type, SAME partition.
table.put_item(Item={"PK": "CUSTOMER#42", "SK": "ORDER#2026-01", "amount": 50, "type": "order"}) # => co-23: entity 2, first order
table.put_item(Item={"PK": "CUSTOMER#42", "SK": "ORDER#2026-02", "amount": 75, "type": "order"}) # => co-23: entity 2, second order
response = table.query(KeyConditionExpression=Key("PK").eq("CUSTOMER#42")) # => co-23: ONE query retrieves BOTH entity types under this partition
items = response["Items"] # => every item under CUSTOMER#42, regardless of entity type
types_found = sorted({item["type"] for item in items}) # => co-23: distinct entity types retrieved by this SINGLE query
assert types_found == ["order", "profile"] # => co-23: both entity types retrievable under the SAME partition key, no second table/query needed
assert len(items) == 3 # => co-23: 1 profile + 2 orders, all under one partition, one query
print(f"Entity types under CUSTOMER#42 (single query): {types_found}, {len(items)} total items") # => Output: Entity types under CUSTOMER#42 (single query): ['order', 'profile'], 3 total items
if __name__ == "__main__": # => guards against running main() on `import example`
main() # => runs everything above when executed as a scriptRun: python3 example.py (representative output, run against boto3==1.43.56 and the official
amazon/dynamodb-local Docker image)
Output:
Entity types under CUSTOMER#42 (single query): ['order', 'profile'], 3 total itemsKey takeaway: The generic PK/SK attribute names are the tell-tale sign of single-table design
-- unlike a relational schema (or even Example 51's own MongoDB sibling), the attribute names
themselves carry no entity-specific meaning; the OVERLOADED VALUES ("PROFILE", "ORDER#...") do.
Why it matters: Single-table design trades relational-style clarity (a table per entity type) for
retrieval efficiency (every entity related to one partition key in one query) -- it is a deliberate,
access-pattern-driven tradeoff, not a DynamoDB limitation forcing an awkward workaround. The cost shows
up on the read side of code review: a developer new to the table has to learn the overloaded sort-key
convention ("PROFILE" vs. "ORDER#...") before they can even guess what a given item represents, a
real onboarding cost weighed against the query-time savings.
Example 52: DynamoDB GSI Access Pattern
ex-52 · exercises co-23, co-17
A Global Secondary Index re-partitions the same items by a different attribute, serving a second access pattern the base table's own key cannot answer at all.
%% Color Palette: Blue #0173B2, Orange #DE8F05, Teal #029E73 -- color-blind friendly, WCAG AA
flowchart LR
Base[("Base table<br/>keyed by order_id")]:::orange
Base -->|"Query by order_id"| Q1["one order"]:::blue
Base -->|"projected into"| GSI[("GSI customer_id-index<br/>re-partitioned by customer_id")]:::teal
GSI -->|"Query by customer_id"| Q2["all of cust-9's orders"]:::teal
classDef blue fill:#0173B2,stroke:#000000,color:#FFFFFF,stroke-width:2px
classDef orange fill:#DE8F05,stroke:#000000,color:#FFFFFF,stroke-width:2px
classDef teal fill:#029E73,stroke:#000000,color:#FFFFFF,stroke-width:2px
learning/code/ex-52-dynamodb-gsi-access-pattern/example.py
"""Example 52: DynamoDB GSI Access Pattern.""" # => co-23,co-17: this file's own purpose, doubling as its module __doc__
from __future__ import annotations # => hygiene: postpones annotation evaluation, interpreter-version-agnostic
from typing import Any # => co-17: boto3's dynamic resource/client factory has no stubs pinned -- Any is the honest, explicit type
import boto3 # => co-17: the official AWS SDK for Python
from boto3.dynamodb.conditions import Key # => co-17: a typed helper for building KeyConditionExpressions safely
def get_local_dynamodb_resource() -> Any: # => co-17: the higher-level "resource" API
"""Return a boto3 DynamoDB resource pointed at a local dynamodb-local instance.""" # => documents the contract
return boto3.resource( # => co-17: same endpoint/credential pattern as prior DynamoDB examples
"dynamodb", endpoint_url="http://localhost:8000", region_name="us-east-1", # => the local dynamodb-local endpoint, no real AWS account needed
aws_access_key_id="fake", aws_secret_access_key="fake", # => dynamodb-local accepts any credentials
) # => closes the boto3.resource() call -- returns a high-level resource bound to the local endpoint
def create_table_with_gsi(resource: Any) -> None: # => co-17,co-23: base table keyed one way, GSI re-keys the SAME items a second way
"""Create a table keyed by order_id, with a GSI re-projecting the same items keyed by customer_id.""" # => documents contract
client = resource.meta.client # => the underlying low-level client, needed for list/delete/waiter calls
if "OrdersWithGsi" in client.list_tables()["TableNames"]: # => resets state -- this example is fully self-contained
client.delete_table(TableName="OrdersWithGsi") # => removes any leftover table from a prior run
client.get_waiter("table_not_exists").wait(TableName="OrdersWithGsi") # => blocks until the delete genuinely completes
resource.create_table( # => co-23: the BASE table's own key answers "fetch one order by order_id" -- and nothing else
TableName="OrdersWithGsi", # => the table this whole GSI example uses
KeySchema=[{"AttributeName": "order_id", "KeyType": "HASH"}], # => co-23: the base table's ONLY access pattern
AttributeDefinitions=[ # => declares every attribute used by EITHER the base table key OR the GSI key
{"AttributeName": "order_id", "AttributeType": "S"}, # => the base table's own partition key attribute
{"AttributeName": "customer_id", "AttributeType": "S"}, # => co-17: needed because the GSI below partitions by it
], # => closes the AttributeDefinitions list -- both attributes declared, base-key and GSI-key alike
GlobalSecondaryIndexes=[ # => co-17,co-23: a SECOND access pattern the base table's own key cannot serve
{ # => the GSI definition dict starts here
"IndexName": "customer_id-index", # => co-17: a named, independently queryable index
"KeySchema": [{"AttributeName": "customer_id", "KeyType": "HASH"}], # => co-17: re-partitions the SAME items by customer_id
"Projection": {"ProjectionType": "ALL"}, # => co-17: copies every attribute into the index, no extra base-table fetch needed
"ProvisionedThroughput": {"ReadCapacityUnits": 5, "WriteCapacityUnits": 5}, # => the GSI's OWN, separately provisioned throughput
} # => closes this one GSI's own definition dict
], # => closes the GlobalSecondaryIndexes list -- exactly one GSI defined here
ProvisionedThroughput={"ReadCapacityUnits": 5, "WriteCapacityUnits": 5}, # => local-testing throughput for the base table
) # => closes the create_table() call -- both the base table AND its GSI now exist
client.get_waiter("table_exists").wait(TableName="OrdersWithGsi") # => blocks until BOTH the table and the GSI are genuinely ready
def main() -> None: # => entry point -- runs only when this file executes directly, not on import
resource = get_local_dynamodb_resource() # => connects to the local dynamodb-local Docker container
create_table_with_gsi(resource) # => sets up the fresh table + GSI
table = resource.Table("OrdersWithGsi") # => the high-level Table object the resource API works through
table.put_item(Item={"order_id": "o-1", "customer_id": "cust-9", "amount": 20}) # => co-23: 2 of these 3 orders share customer_id=cust-9
table.put_item(Item={"order_id": "o-2", "customer_id": "cust-9", "amount": 35}) # => co-23: same customer, DIFFERENT order_id partition key
table.put_item(Item={"order_id": "o-3", "customer_id": "cust-1", "amount": 15}) # => a different customer entirely
# The BASE table's key (order_id) CANNOT answer "all orders for cust-9" -- that needs a full scan without the GSI.
gsi_response = table.query( # => co-17: querying the GSI, NOT the base table's own primary key
IndexName="customer_id-index", # => co-17: routes this query through the secondary index specifically
KeyConditionExpression=Key("customer_id").eq("cust-9"), # => co-17: an access pattern the base table's own key alone could never serve
) # => closes the query() call -- only the customer_id-index gets scanned, never the full base table
items = gsi_response["Items"] # => the orders the GSI found for this customer
order_ids = sorted(item["order_id"] for item in items) # => extracts order_ids for a deterministic assertion
assert order_ids == ["o-1", "o-2"] # => co-23,co-17: BOTH of cust-9's orders found, via a query the base table key structure alone couldn't answer
print(f"GSI query for customer_id=cust-9 (base table key alone could not answer this): {order_ids}") # => Output: GSI query for customer_id=cust-9 (base table key alone could not answer this): ['o-1', 'o-2']
if __name__ == "__main__": # => guards against running main() on `import example`
main() # => runs everything above when executed as a scriptRun: python3 example.py (representative output, run against boto3==1.43.56 and the official
amazon/dynamodb-local Docker image)
Output:
GSI query for customer_id=cust-9 (base table key alone could not answer this): ['o-1', 'o-2']Key takeaway: A GSI is a genuinely separate, independently-queried index with its own key schema over the same underlying items -- it costs its own provisioned throughput and storage, which is why GSIs are added deliberately, one per genuinely distinct access pattern, not reflexively.
Why it matters: This is co-23's real design discipline -- a single-table design's base table answers one access pattern well, and every ADDITIONAL access pattern needs either an overloaded key (Example 51) or a GSI like this one; naming the access patterns up front (co-08) drives which. Adding a GSI after data already exists is possible but re-indexes every existing item and costs its own provisioned throughput indefinitely, so treating GSIs as a deliberate, budgeted design decision rather than an easy afterthought pays off well before the table grows large.
Example 53: DynamoDB TTL Attribute
ex-53 · exercises co-24
Enable TTL on a Number attribute holding an epoch-seconds expiry -- items past their expiry are filtered from reads by a background sweep that is genuinely asynchronous, not instant.
learning/code/ex-53-dynamodb-ttl-attribute/example.py
"""Example 53: DynamoDB TTL Attribute.""" # => co-24: this file's own purpose, doubling as its module __doc__
from __future__ import annotations # => hygiene: postpones annotation evaluation, interpreter-version-agnostic
import time # => co-24: computes real epoch-second timestamps for the TTL attribute below
from typing import Any # => co-24: boto3's dynamic resource/client factory has no stubs pinned -- Any is the honest, explicit type
import boto3 # => co-24: the official AWS SDK for Python
def get_local_dynamodb_client() -> Any: # => co-24: connects to amazon/dynamodb-local, the official local-testing Docker image
"""Return a boto3 DynamoDB client pointed at a local dynamodb-local instance.""" # => documents the contract
return boto3.client( # => co-24: same endpoint/credential pattern as prior DynamoDB examples
"dynamodb", endpoint_url="http://localhost:8000", region_name="us-east-1", # => the local dynamodb-local endpoint, no real AWS account needed
aws_access_key_id="fake", aws_secret_access_key="fake", # => dynamodb-local accepts any credentials
) # => closes the boto3.client() call -- returns a low-level DynamoDB client bound to the local endpoint
def setup_ttl_table(client: Any) -> None: # => co-24: a table with TTL enabled on a Number (epoch-seconds) attribute
"""Create a table and enable TTL on its expires_at Number attribute.""" # => documents the contract, no runtime output
if "TtlItems" in client.list_tables()["TableNames"]: # => resets state -- this example is fully self-contained
client.delete_table(TableName="TtlItems") # => removes any leftover table from a prior run
client.get_waiter("table_not_exists").wait(TableName="TtlItems") # => blocks until the delete genuinely completes
client.create_table( # => a minimal, single-partition-key table
TableName="TtlItems", # => the table this whole TTL example uses
KeySchema=[{"AttributeName": "id", "KeyType": "HASH"}], # => a single partition key, no sort key needed here
AttributeDefinitions=[{"AttributeName": "id", "AttributeType": "S"}], # => S == string type, the partition key's type
ProvisionedThroughput={"ReadCapacityUnits": 5, "WriteCapacityUnits": 5}, # => local-testing throughput
) # => closes the create_table() call -- the table now exists, though not yet necessarily ACTIVE
client.get_waiter("table_exists").wait(TableName="TtlItems") # => blocks until the table is genuinely ready
client.update_time_to_live( # => co-24: enables TTL, naming WHICH attribute DynamoDB should treat as the expiry epoch
TableName="TtlItems",
TimeToLiveSpecification={"Enabled": True, "AttributeName": "expires_at"}, # => co-24: expires_at MUST be a Number, epoch seconds
)
def main() -> None: # => entry point -- runs only when this file executes directly, not on import
client = get_local_dynamodb_client() # => connects to the local dynamodb-local Docker container
setup_ttl_table(client) # => sets up the fresh table with TTL enabled
ttl_description = client.describe_time_to_live(TableName="TtlItems")["TimeToLiveDescription"] # => co-24: confirms TTL is genuinely active
assert ttl_description["TimeToLiveStatus"] == "ENABLED" # => co-24: the table's TTL configuration itself is confirmed, not assumed
assert ttl_description["AttributeName"] == "expires_at" # => confirms the correct attribute is wired up as the expiry clock
print(f"TTL status: {ttl_description['TimeToLiveStatus']} on attribute '{ttl_description['AttributeName']}'") # => Output: TTL status: ENABLED on attribute 'expires_at'
future_epoch = int(time.time()) + 3600 # => co-24: 1 hour in the future -- this item has NOT expired yet
past_epoch = int(time.time()) - 3600 # => co-24: 1 hour in the PAST -- this item's TTL has already elapsed
client.put_item(TableName="TtlItems", Item={"id": {"S": "still-fresh"}, "expires_at": {"N": str(future_epoch)}}) # => a NOT-yet-expired item
client.put_item(TableName="TtlItems", Item={"id": {"S": "already-expired"}, "expires_at": {"N": str(past_epoch)}}) # => an ALREADY-expired item
still_present = True # => co-24: polls rather than sleeping a fixed duration -- the background sweep's own cadence is NOT synchronized to this write
for _ in range(10): # => co-24: polls for up to 10 seconds -- expiry filtering is a background process, not a synchronous side effect of the write
expired_check = client.get_item(TableName="TtlItems", Key={"id": {"S": "already-expired"}}) # => re-checks whether the background sweep has caught up yet
still_present = "Item" in expired_check # => True until the sweep marks/filters this item
if not still_present: # => co-24: the moment the background sweep filters it, stop polling
break # => exits the polling loop early -- no need to burn the remaining iterations
time.sleep(1) # => co-24: waits one second before re-checking -- the sweep's own cadence, observed empirically, is on this order
fresh = client.get_item(TableName="TtlItems", Key={"id": {"S": "still-fresh"}}) # => reads the not-yet-expired item
assert "Item" in fresh # => co-24: a future expires_at means the item is NOT yet subject to expiry filtering
assert still_present is False # => co-24: within the polling window, the past-epoch item was filtered from reads
print(f"Item with future expires_at: present = {'Item' in fresh}") # => Output: Item with future expires_at: present = True
print(f"Item with past expires_at: present = {still_present} (filtered by the background TTL sweep)") # => Output: Item with past expires_at: present = False (filtered by the background TTL sweep)
# => co-24: on REAL AWS DynamoDB, an item past its expires_at stays fully readable via GetItem
# => (which has no FilterExpression parameter at all) -- and via Scan/Query unless the caller adds
# => an explicit FilterExpression -- until the SAME best-effort background process deletes it,
# => typically within a few days. There is no separate "fast read-filter" tier; this example's
# => ~10-second background-sweep visibility is DynamoDB Local's own faster, more eager
# => implementation detail, not a demonstration of the real AWS service's read path (see
# => overview.md's TTL accuracy note)
if __name__ == "__main__": # => guards against running main() on `import example`
main() # => runs everything above when executed as a scriptRun: python3 example.py (representative output, run against boto3==1.43.56 and the official
amazon/dynamodb-local Docker image)
Output:
TTL status: ENABLED on attribute 'expires_at'
Item with future expires_at: present = True
Item with past expires_at: present = False (filtered by the background TTL sweep)Key takeaway: This example polls instead of sleeping a fixed duration precisely because the background sweep's own cadence is NOT synchronized to the write -- a fixed 1-second sleep flaked in practice while authoring this example, which is itself a small real demonstration of "best-effort, not synchronous."
Why it matters: A team relying on DynamoDB TTL for a compliance deadline needs to know the SLA is
"eventually, best-effort," not exact. On real AWS DynamoDB, an expired item stays readable via
GetItem (no filter-expression support) and via Scan/Query unless the caller adds an explicit
FilterExpression, until the same background process purges it -- typically within a few days. This
example's ~10-second local sweep is DynamoDB Local's own faster, more eager behavior, not real AWS's
read path. A retention policy assuming GetItem proves data gone once TTL passes is wrong on real
AWS.
Example 54: DynamoDB ConsistentRead Toggle
ex-54 · exercises co-06, co-07
Context: GetItem with ConsistentRead=False (the default) may read a replica that has not yet
applied the latest write; ConsistentRead=True always reads the up-to-date leader replica. Simulated
in pure Python, since amazon/dynamodb-local runs as a single process with no real inter-replica lag
-- both consistency modes return the identical, always-current value on it, which would misleadingly
suggest there is no difference at all.
%% Color Palette: Blue #0173B2, Orange #DE8F05, Teal #029E73 -- color-blind friendly, WCAG AA
sequenceDiagram
participant App as Application
participant L as Leader Replica
participant F as Lagging Replica
App->>L: put_item(v1)
Note over F: not yet replicated
App->>F: GetItem ConsistentRead=False
F-->>App: v0 (stale)
App->>L: GetItem ConsistentRead=True
L-->>App: v1 (always fresh)
learning/code/ex-54-dynamodb-consistent-read-toggle/example.py
"""Example 54: DynamoDB ConsistentRead Toggle, Simulated.""" # => co-06,co-07: this file's own purpose, doubling as its module __doc__
from __future__ import annotations # => hygiene: postpones annotation evaluation, interpreter-version-agnostic
from dataclasses import dataclass # => co-06: a typed replica set standing in for DynamoDB's own multi-AZ replicas
# NOTE (honesty, per this topic's own no-fabrication rule): amazon/dynamodb-local runs as a SINGLE
# local process with no real inter-replica lag, so ConsistentRead=False and ConsistentRead=True return
# the IDENTICAL result on it every time -- there is no staleness window to observe locally. This
# example instead simulates real DynamoDB's DOCUMENTED behavior (an eventually-consistent read MAY hit
# a replica that has not yet applied the latest write; a strongly consistent read always reads the
# leader/up-to-date copy) with a toy multi-replica model, the same honest-simulation technique
# Example 20 already used for eventual consistency in general.
@dataclass # => intentionally MUTABLE -- a replica's own value changes as writes replicate in
class Replica: # => co-06: one of DynamoDB's internal replicas for a partition (not user-visible directly)
name: str # => a human-readable label, e.g. "replica-1"
value: str # => this replica's CURRENT view of the item -- may lag the true latest write
class SimulatedDynamoTable: # => co-07: models GetItem's two ConsistentRead modes over a lagging replica set
def __init__(self) -> None: # => builds a 3-replica table, all starting in agreement
self.replicas = [Replica("replica-1", "v0"), Replica("replica-2", "v0"), Replica("replica-3", "v0")] # => co-06: 3 replicas, DynamoDB's real internal default
def put_item(self, value: str) -> None: # => co-07: the leader replica updates instantly; the others lag
self.replicas[0].value = value # => co-06: the write's OWN leader replica applies it immediately
# => replicas[1] and replicas[2] are deliberately left UN-replicated here, simulating the real
# => propagation delay a strongly consistent read is specifically designed to bypass
def get_item(self, consistent_read: bool) -> str: # => co-07: the SAME item, read two different ways
if consistent_read: # => co-07: ConsistentRead=True -- DynamoDB always routes to the up-to-date leader replica
return self.replicas[0].value # => co-07: strongly consistent -- CANNOT observe a stale value
return self.replicas[1].value # => co-07: ConsistentRead=False (the default) -- MAY hit a replica that has not caught up yet
def replicate(self) -> None: # => co-06: simulates replication lag finally elapsing
for replica in self.replicas[1:]: # => catches every follower replica up to the leader's value
replica.value = self.replicas[0].value # => co-06: convergence, exactly as Example 20 modeled
def main() -> None: # => entry point -- runs only when this file executes directly, not on import
table = SimulatedDynamoTable() # => co-06: a fresh, fully-converged 3-replica table
table.put_item("v1") # => co-07: a write lands on the leader replica; the eventually-consistent replica has NOT caught up yet
eventual_read = table.get_item(consistent_read=False) # => co-07: ConsistentRead=False -- reads the LAGGING replica
assert eventual_read == "v0" # => co-07: a genuinely STALE read -- the eventually-consistent path can observe the OLD value
print(f"ConsistentRead=False right after the write: {eventual_read} (may be stale)") # => Output: ConsistentRead=False right after the write: v0 (may be stale)
strong_read = table.get_item(consistent_read=True) # => co-07: ConsistentRead=True -- ALWAYS reads the up-to-date leader
assert strong_read == "v1" # => co-07: strongly consistent -- CANNOT observe a stale value, by construction
print(f"ConsistentRead=True right after the write: {strong_read} (never stale)") # => Output: ConsistentRead=True right after the write: v1 (never stale)
table.replicate() # => co-06: simulates the replication delay finally elapsing
converged_read = table.get_item(consistent_read=False) # => reads the ONCE-lagging replica again, now caught up
assert converged_read == "v1" # => co-06: eventual consistency's promise kept -- the lagging replica converged
print(f"ConsistentRead=False after replication settles: {converged_read} (converged)") # => Output: ConsistentRead=False after replication settles: v1 (converged)
if __name__ == "__main__": # => guards against running main() on `import example`
main() # => runs everything above when executed as a scriptRun: python3 example.py (no external dependency -- pure Python 3.13 standard library)
Output:
ConsistentRead=False right after the write: v0 (may be stale)
ConsistentRead=True right after the write: v1 (never stale)
ConsistentRead=False after replication settles: v1 (converged)Key takeaway: ConsistentRead=True costs strictly more (it consumes a full read-capacity-unit
instead of half, and it can only be served by the leader) -- reaching for it reflexively on every read
defeats the throughput and availability benefits DynamoDB's default eventually-consistent reads exist
to provide.
Why it matters: This is co-07's tunable-consistency idea, applied specifically to DynamoDB's own
API surface -- most reads should default to eventually-consistent, with ConsistentRead=True reserved
for the specific reads (immediately after a write, on a value the application cannot tolerate seeing
stale) that genuinely need it. Read-your-own-writes after a form submission is the classic case that
needs the strong read; a background analytics dashboard reading the same table almost never does, and
defaulting everything to ConsistentRead=True "to be safe" quietly doubles the table's real read cost.
← Previous: Beginner Examples · Next: Advanced Examples →
Last updated July 26, 2026