Skip to content
AyoKoding

Artifact: RFC — Event Bus for Shipment Notifications

RFC with options and a decisive trade-off, event bus selection -- exercises co-04.

RFC: Choosing an Event Bus for Shipment Notifications

Problem: the current in-process queue between Shipment API and Notification Worker has no replay capability -- a Notification Worker bug that drops messages cannot be recovered from without re-triggering the original events, and there is no ordering guarantee per shipment.

Option A: Kafka. Pros: message replay from any offset, per-partition ordering (partitioned by shipment_id), horizontal scale proven at far larger volume than Harborlight needs today. Cons: the team has no existing Kafka operational experience; adds a new piece of infrastructure to run and monitor.

Option B: Amazon SQS + SNS fanout. Pros: fully managed, zero new infrastructure to operate, the team already runs other SQS queues. Cons: no message replay once a message is deleted from the queue; no native per-key ordering guarantee (SQS FIFO queues offer ordering but cap throughput well below Harborlight's peak volume).

Decisive trade-off: replay capability is the deciding factor -- the Shipment Platform team has now shipped two Notification Worker bugs in the past year that would have been trivially recoverable with replay, and both required a manual data-reconciliation effort without it. Kafka's operational cost is real but bounded (one well-documented technology to learn); SQS's missing replay is a recurring cost paid every time a future bug ships.

Decision: Kafka.

Open Questions: who owns Kafka cluster operations day to day (undecided -- SRE and Shipment Platform are still discussing); retention period for replay (undecided -- proposal is 7 days, pending a cost estimate).

Last updated July 15, 2026

Command Palette

Search for a command to run...