> ## Documentation Index
> Fetch the complete documentation index at: https://docs.odds-api.io/llms.txt
> Use this file to discover all available pages before exploring further.

# 2026 09 02 droplog time buckets design

# Drop log in time-bucketed Redis streams

Date: 2026-09-02
Status: approved for implementation

## Problem

The WebSocket gateway's drop log is one Redis stream, `ws:droplog`, holding 24
hours of drop events: about 2,000,000 entries a day at roughly 4 KB each, so an
8 GB value in a single key. Dragonfly splits `--maxmemory` (13 GiB) across its
32 shards and rejects new-key inserts in any shard whose budget (about 416 MiB)
is exhausted, even when global usage is far below the cap. One 8 GB key
therefore turns one shard permanently over budget. On 1 to 2 Sep 2026 this
rejected 246,777 writes across every service that uses Dragonfly, wedged the
WebSocket replay writer, and OOM-killed v3 twice.

The stopgap is `DROPLOG_MAX_LEN=90000`, which caps the log at about one hour of
history instead of 24. This spec restores 24 hours without any single key
outgrowing a shard.

## Goals

* No drop log key larger than a small fraction of a shard budget at any volume
  the gateway can plausibly reach (target under 200 MB at twice today's peak).
* 24 hour retention with no trim job doing the work.
* No change to the cursor contract: `/internal/drops` and the hub keep passing
  `<ms>-<seq>` stream ids as `since`, `min`, `max`, `cursor` and `head`.
* No change to entry encoding, dedupe semantics, decode-skip handling, the
  in-memory hour cache's behaviour, or the board rebuild.
* Zero-downtime cutover: entries written before the deploy stay readable until
  they age out.
* Everything Redis-touching unit-tested against a real stream implementation.

## Non-goals

* Shrinking entries, changing what is persisted, or moving the log out of Redis.
* Touching the limit log (`ws:limitlog`, 37 MB) or the replay store.
* Multi-writer support. The drop pump is the single writer today and stays so.

## Design

### Keys

Each entry is written to the bucket stream for its id's millisecond:

```
ws:droplog:b:<bucketStartMs>      bucketStartMs = ms - ms % dropLogBucketMs
dropLogBucketMs = 10 * 60 * 1000  (10 minutes, 144 buckets per day)
```

At today's volume a bucket holds 15k to 30k entries, 60 to 120 MB. Keys hash
across all 32 shards.

Every bucket carries a 25 hour TTL. The TTL is set with `EXPIRE` the first time
a process writes to a bucket (tracked in a small in-memory set of bucket starts
this process has already stamped, so a restart re-stamps at most one bucket).
Retention therefore needs no trim job. A per-bucket `MAXLEN ~` of
`DROPLOG_BUCKET_MAX_LEN` (default 60,000, about 240 MB) remains as a runaway
backstop only; the near-cap warning moves to the bucket level.

### Ids

The writer assigns complete ids explicitly as `<ms>-<seq>`. `ms` is the pump's
wall clock at append time, clamped to be non-decreasing within the process;
`seq` is 0 for a new millisecond and increments while consecutive appends share
one. (Dragonfly 1.40 also accepts the `<ms>-*` shorthand, verified 2 Sep 2026,
but miniredis does not, and a complete id costs nothing.) The bucket is derived
from that same `ms`, so the invariant every reader relies on holds by
construction:

> An entry with id `<ms>-<seq>` lives in bucket `bucket(ms)`, and all entries
> in bucket `b` have `b <= ms < b + dropLogBucketMs`.

Because ids are globally ordered by `(ms, seq)` and every bucket covers a
disjoint time range, reading buckets in time order yields the same global
order the single stream had.

Before its first append a process seeds the allocator from the newest live
bucket (it probes `bucket(now)+1`, `bucket(now)`, `bucket(now)-1` for a top id),
so a restart on a clock a few seconds behind its predecessor continues after
the existing ids instead of writing behind them into an older bucket that
cursor readers have already passed.

If Redis answers "ID specified in XADD is equal or smaller than the target
stream top item" (a process restarted on a clock behind its predecessor, or a
second writer), the writer reads the bucket's top id, retries once with
`ms = top.ms` and `seq = top.seq + 1`, and clamps its monotonic clock to that
value. `top.ms` is inside the bucket by the invariant, so the retried id is
too. If the retry also fails the batch returns the error and the pump's
existing hold-and-retry handles it.

### Reading

One iterator module owns bucket enumeration and both directions:

* `forwardScan(start exclusiveId|"", end inclusiveId|"", page, visit)`
  Buckets from `bucket(start.ms)` (or the retention floor) up to
  `bucket(end.ms)` (or `bucket(now + bucketMs)`). `XRANGE` with `(start` on the
  first bucket and `-` on later ones; `end` bounds the last bucket, `+`
  elsewhere.
* `backwardScan(end exclusiveOrInclusive|"", min inclusiveId|"", page, visit)`
  Buckets from `bucket(end.ms)` (or `bucket(now + bucketMs)`) down to
  `bucket(min.ms)` (or the retention floor). `XREVRANGE` with the caller's end
  bound on the first bucket and `+` on later ones; `min` bounds the last bucket,
  `-` elsewhere.

`visit` receives one page at a time and returns whether to continue, and the
page size is a callback so limit-bounded readers ask each bucket only for what
they still need. Callers keep their existing paging, limit, dedupe, decode-skip
and scan-cap logic unchanged.

Empty buckets cost one round trip each. To keep the steady state at one or two
round trips: the bucket window's floor is clamped to `bucket(cutover_ms)` once
the cutover is known (no bucket can exist before it), and the bucket after
`bucket(now)` is probed only within one minute of a boundary. A full 24 hour
sweep over a sparse log is therefore at most \~145 extra round trips, within the
existing 60k scan cap's cost profile.

The retention floor is `now - 24h - dropLogBucketMs`. Readers never enumerate
buckets older than the floor, so expired keys are never requested.

Consumers and how they map:

| Consumer                      | Today                             | After                                                                                |
| ----------------------------- | --------------------------------- | ------------------------------------------------------------------------------------ |
| `ReadDropEventsSince("")`     | `XREVRANGE + - limit`, reversed   | `backwardScan` until `limit`, reversed                                               |
| `ReadDropEventsSince(cursor)` | `XRANGE (cursor + limit`          | `forwardScan(cursor, "", ...)` until `limit`                                         |
| `ReadDropEventsBack`          | `XREVRANGE end min limit`         | `backwardScan(cursor, min, ...)` until `limit`; `head` is the first id visited       |
| `sweepMatchedSlice`           | `XREVRANGE` page chain            | `backwardScan(max/cursor, min, page, ...)` with the same match, dedupe and `scanCap` |
| memory cache `warm`           | `XREVRANGE` to the 65 min floor   | `backwardScan("", floorId, ...)`                                                     |
| memory cache `syncTail`       | `XRANGE (last +`                  | `forwardScan(last, "", ...)`                                                         |
| board rebuild                 | `ReadDropEventsSince(24h-ago id)` | unchanged; the cursor id maps to a bucket                                            |
| `/internal/drops`             | passes ids through                | unchanged                                                                            |

### Cutover

The legacy key `ws:droplog` is read as one extra pseudo-bucket that sits below
every real bucket, but only while it can still hold data:

* On its first bucketed append the new code does `SET NX ws:droplog:cutover_ms <ms>`.
* Readers cache that value (re-read every minute until found). A scan whose
  range reaches below `cutover_ms` also scans the legacy key with the same
  bounds, in the correct position (after all real buckets in a backward scan,
  before them in a forward scan). A cursor whose `ms < cutover_ms` maps to the
  legacy key.
* Once `now > cutover_ms + 25h` the legacy key is never consulted again.
* Railway keeps the previous deployment serving until the new one is healthy,
  so for up to about a minute the old process still appends to the legacy key
  while the new one writes buckets. Those few legacy entries carry ids newer
  than `cutover_ms`; readers still return them, but in the legacy slot rather
  than interleaved by id. The hub dedupes by drop id, so the only effect is a
  one-time ordering wobble inside one page during the cutover minute.

`runDropLogTrim` keeps running every 10 minutes. It keeps trimming the legacy
key to 24 hours and deletes it once empty, and as a belt-and-braces sweep it
`UNLINK`s any bucket key between `floor - 6h` and `floor` (a bounded list of at
most 36 computed keys, no `SCAN`). The near-cap warning is emitted per bucket.
`DROPLOG_MAX_LEN` is retired; the Railway variable is removed at deploy.

### Volume at the memory cache

The hour cache (`memCacheRetainMs`, 65 minutes) spans 7 buckets. Its warm and
tail sync change only in which iterator they call.

### Failure handling

* Append errors surface exactly as today; the pump holds the unpersisted tail
  and retries with backoff.
* A failed `EXPIRE` is retried on the next append to that bucket (the bucket is
  only recorded as stamped after `EXPIRE` succeeds). Worst case a bucket
  without a TTL is caught by the trim job's `UNLINK` sweep.
* Decode failures per entry are counted and skipped exactly as today.
* A cursor pointing into a bucket that has expired simply starts at the oldest
  live bucket; no error, matching how `XRANGE` past a trimmed id behaved.

### Testing

Add `github.com/alicebob/miniredis/v2` as a test dependency and run the real
`go-redis` client against it. Streams with explicit ids, `XRANGE`, `XREVRANGE`,
`EXPIRE` and `TTL` are all supported. A `dropLogNow func() time.Time` seam
provides a controllable clock.

Coverage required before merge:

1. Bucket key math: boundaries, floor, id parsing, invariant on written ids.
2. Append: entries land in the right bucket across a boundary; `EXPIRE` set once
   per bucket per process; monotonic clamp; retry after "equal or smaller".
3. `ReadDropEventsSince`: newest window across buckets; cursor paging across a
   boundary; cursor exactly on a boundary; empty bucket in the middle; dedupe
   across a bucket boundary.
4. `ReadDropEventsBack`: `max`/`cursor` and `min` in different buckets; `head`;
   `hasMore`.
5. Matched sweep: matches across buckets, dedupe across boundary, scan cap.
6. Memory cache warm and tail sync across buckets.
7. Legacy fallback: entries written to `ws:droplog` before `cutover_ms` are
   returned in order behind bucketed ones; ignored after the 25 hour window.
8. Trim job: legacy trimmed then deleted; stale bucket keys unlinked.
9. Existing droplog, cache, board and sweep tests keep passing unchanged.

### Rollout

1. Merge and deploy v3. First append creates `cutover_ms`; history stays
   complete because reads span both stores.
2. Remove `DROPLOG_MAX_LEN` from the v3 service. The legacy stream trims itself
   away within 24 hours.
3. Verify with `INFO stats` (`oom_rejections` flat), `MEMORY USAGE` on a few
   bucket keys (under 200 MB), `TTL` on a bucket (under 25h), and the monitor's
   `drops` layer staying green.
4. After 25 hours confirm `ws:droplog` is gone and `/internal/drops` with
   `min` 24 hours back returns a full day.

## Alternatives considered

* Postgres partitioned table: cleanest query model, but v3 production has no
  Postgres write path, so it needs credentials, DigitalOcean firewall changes
  for Railway egress, and a new id scheme mapped onto the cursor contract.
* Dragonfly `--num_shards=2` on a 32 GB container: config only, but doubles the
  Dragonfly bill, cuts parallelism, and leaves the 8 GB key to grow again.
* Hash-sharded streams (`ws:droplog:<hash % N>`): even key sizes, but every
  range read becomes an N-way merge and cursors stop mapping to one stream.