Skip to main content

​
Drop log in time-bucketed Redis streams

Date: 2026-09-02 Status: approved for implementation

​
Problem

The WebSocket gateway’s drop log is one Redis stream, ws:droplog, holding 24 hours of drop events: about 2,000,000 entries a day at roughly 4 KB each, so an 8 GB value in a single key. Dragonfly splits --maxmemory (13 GiB) across its 32 shards and rejects new-key inserts in any shard whose budget (about 416 MiB) is exhausted, even when global usage is far below the cap. One 8 GB key therefore turns one shard permanently over budget. On 1 to 2 Sep 2026 this rejected 246,777 writes across every service that uses Dragonfly, wedged the WebSocket replay writer, and OOM-killed v3 twice. The stopgap is DROPLOG_MAX_LEN=90000, which caps the log at about one hour of history instead of 24. This spec restores 24 hours without any single key outgrowing a shard.

​
Goals

  • No drop log key larger than a small fraction of a shard budget at any volume the gateway can plausibly reach (target under 200 MB at twice today’s peak).
  • 24 hour retention with no trim job doing the work.
  • No change to the cursor contract: /internal/drops and the hub keep passing <ms>-<seq> stream ids as since, min, max, cursor and head.
  • No change to entry encoding, dedupe semantics, decode-skip handling, the in-memory hour cache’s behaviour, or the board rebuild.
  • Zero-downtime cutover: entries written before the deploy stay readable until they age out.
  • Everything Redis-touching unit-tested against a real stream implementation.

​
Non-goals

  • Shrinking entries, changing what is persisted, or moving the log out of Redis.
  • Touching the limit log (ws:limitlog, 37 MB) or the replay store.
  • Multi-writer support. The drop pump is the single writer today and stays so.

​
Design

​
Keys

Each entry is written to the bucket stream for its id’s millisecond:
At today’s volume a bucket holds 15k to 30k entries, 60 to 120 MB. Keys hash across all 32 shards. Every bucket carries a 25 hour TTL. The TTL is set with EXPIRE the first time a process writes to a bucket (tracked in a small in-memory set of bucket starts this process has already stamped, so a restart re-stamps at most one bucket). Retention therefore needs no trim job. A per-bucket MAXLEN ~ of DROPLOG_BUCKET_MAX_LEN (default 60,000, about 240 MB) remains as a runaway backstop only; the near-cap warning moves to the bucket level.

​
Ids

The writer assigns complete ids explicitly as <ms>-<seq>. ms is the pump’s wall clock at append time, clamped to be non-decreasing within the process; seq is 0 for a new millisecond and increments while consecutive appends share one. (Dragonfly 1.40 also accepts the <ms>-* shorthand, verified 2 Sep 2026, but miniredis does not, and a complete id costs nothing.) The bucket is derived from that same ms, so the invariant every reader relies on holds by construction:
An entry with id <ms>-<seq> lives in bucket bucket(ms), and all entries in bucket b have b <= ms < b + dropLogBucketMs.
Because ids are globally ordered by (ms, seq) and every bucket covers a disjoint time range, reading buckets in time order yields the same global order the single stream had. Before its first append a process seeds the allocator from the newest live bucket (it probes bucket(now)+1, bucket(now), bucket(now)-1 for a top id), so a restart on a clock a few seconds behind its predecessor continues after the existing ids instead of writing behind them into an older bucket that cursor readers have already passed. If Redis answers “ID specified in XADD is equal or smaller than the target stream top item” (a process restarted on a clock behind its predecessor, or a second writer), the writer reads the bucket’s top id, retries once with ms = top.ms and seq = top.seq + 1, and clamps its monotonic clock to that value. top.ms is inside the bucket by the invariant, so the retried id is too. If the retry also fails the batch returns the error and the pump’s existing hold-and-retry handles it.

​
Reading

One iterator module owns bucket enumeration and both directions:
  • forwardScan(start exclusiveId|"", end inclusiveId|"", page, visit) Buckets from bucket(start.ms) (or the retention floor) up to bucket(end.ms) (or bucket(now + bucketMs)). XRANGE with (start on the first bucket and - on later ones; end bounds the last bucket, + elsewhere.
  • backwardScan(end exclusiveOrInclusive|"", min inclusiveId|"", page, visit) Buckets from bucket(end.ms) (or bucket(now + bucketMs)) down to bucket(min.ms) (or the retention floor). XREVRANGE with the caller’s end bound on the first bucket and + on later ones; min bounds the last bucket, - elsewhere.
visit receives one page at a time and returns whether to continue, and the page size is a callback so limit-bounded readers ask each bucket only for what they still need. Callers keep their existing paging, limit, dedupe, decode-skip and scan-cap logic unchanged. Empty buckets cost one round trip each. To keep the steady state at one or two round trips: the bucket window’s floor is clamped to bucket(cutover_ms) once the cutover is known (no bucket can exist before it), and the bucket after bucket(now) is probed only within one minute of a boundary. A full 24 hour sweep over a sparse log is therefore at most ~145 extra round trips, within the existing 60k scan cap’s cost profile. The retention floor is now - 24h - dropLogBucketMs. Readers never enumerate buckets older than the floor, so expired keys are never requested. Consumers and how they map:

​
Cutover

The legacy key ws:droplog is read as one extra pseudo-bucket that sits below every real bucket, but only while it can still hold data:
  • On its first bucketed append the new code does SET NX ws:droplog:cutover_ms <ms>.
  • Readers cache that value (re-read every minute until found). A scan whose range reaches below cutover_ms also scans the legacy key with the same bounds, in the correct position (after all real buckets in a backward scan, before them in a forward scan). A cursor whose ms < cutover_ms maps to the legacy key.
  • Once now > cutover_ms + 25h the legacy key is never consulted again.
  • Railway keeps the previous deployment serving until the new one is healthy, so for up to about a minute the old process still appends to the legacy key while the new one writes buckets. Those few legacy entries carry ids newer than cutover_ms; readers still return them, but in the legacy slot rather than interleaved by id. The hub dedupes by drop id, so the only effect is a one-time ordering wobble inside one page during the cutover minute.
runDropLogTrim keeps running every 10 minutes. It keeps trimming the legacy key to 24 hours and deletes it once empty, and as a belt-and-braces sweep it UNLINKs any bucket key between floor - 6h and floor (a bounded list of at most 36 computed keys, no SCAN). The near-cap warning is emitted per bucket. DROPLOG_MAX_LEN is retired; the Railway variable is removed at deploy.

​
Volume at the memory cache

The hour cache (memCacheRetainMs, 65 minutes) spans 7 buckets. Its warm and tail sync change only in which iterator they call.

​
Failure handling

  • Append errors surface exactly as today; the pump holds the unpersisted tail and retries with backoff.
  • A failed EXPIRE is retried on the next append to that bucket (the bucket is only recorded as stamped after EXPIRE succeeds). Worst case a bucket without a TTL is caught by the trim job’s UNLINK sweep.
  • Decode failures per entry are counted and skipped exactly as today.
  • A cursor pointing into a bucket that has expired simply starts at the oldest live bucket; no error, matching how XRANGE past a trimmed id behaved.

​
Testing

Add github.com/alicebob/miniredis/v2 as a test dependency and run the real go-redis client against it. Streams with explicit ids, XRANGE, XREVRANGE, EXPIRE and TTL are all supported. A dropLogNow func() time.Time seam provides a controllable clock. Coverage required before merge:
  1. Bucket key math: boundaries, floor, id parsing, invariant on written ids.
  2. Append: entries land in the right bucket across a boundary; EXPIRE set once per bucket per process; monotonic clamp; retry after “equal or smaller”.
  3. ReadDropEventsSince: newest window across buckets; cursor paging across a boundary; cursor exactly on a boundary; empty bucket in the middle; dedupe across a bucket boundary.
  4. ReadDropEventsBack: max/cursor and min in different buckets; head; hasMore.
  5. Matched sweep: matches across buckets, dedupe across boundary, scan cap.
  6. Memory cache warm and tail sync across buckets.
  7. Legacy fallback: entries written to ws:droplog before cutover_ms are returned in order behind bucketed ones; ignored after the 25 hour window.
  8. Trim job: legacy trimmed then deleted; stale bucket keys unlinked.
  9. Existing droplog, cache, board and sweep tests keep passing unchanged.

​
Rollout

  1. Merge and deploy v3. First append creates cutover_ms; history stays complete because reads span both stores.
  2. Remove DROPLOG_MAX_LEN from the v3 service. The legacy stream trims itself away within 24 hours.
  3. Verify with INFO stats (oom_rejections flat), MEMORY USAGE on a few bucket keys (under 200 MB), TTL on a bucket (under 25h), and the monitor’s drops layer staying green.
  4. After 25 hours confirm ws:droplog is gone and /internal/drops with min 24 hours back returns a full day.

​
Alternatives considered

  • Postgres partitioned table: cleanest query model, but v3 production has no Postgres write path, so it needs credentials, DigitalOcean firewall changes for Railway egress, and a new id scheme mapped onto the cursor contract.
  • Dragonfly --num_shards=2 on a 32 GB container: config only, but doubles the Dragonfly bill, cuts parallelism, and leaves the 8 GB key to grow again.
  • Hash-sharded streams (ws:droplog:<hash % N>): even key sizes, but every range read becomes an N-way merge and cursors stop mapping to one stream.