This is an AI assisted bug report as I was seeing increasing latency in querying logs when parseable pods restarted because of node updates/OOM errors or config updates.
/about output:
{
"version": "v2.9.3",
"uiVersion": "v2.9.3",
"commit": "c54b762",
"deploymentId": "01KBNFVYPZ46Y5EKD93MQ9GFVN",
"updateAvailable": false,
"latestVersion": null,
"llmActive": false,
"llmProvider": null,
"oidcActive": false,
"license": {
"name": "AGPL-3.0-only",
"version": "v1",
"plan": "OSS",
"deploymentInfo": "Managed"
},
"mode": "Standalone",
"staging": "/parseable/staging",
"grpcPort": 8001,
"store": {
"type": "S3 bucket",
"path": "---"
},
"analytics": {
"clarityTag": null
}
Description:
Expected: when a node writes to a date partition that a different node has
already written to, it should append its own ManifestItem to
snapshot.manifest_list once, then update that entry in place on subsequent sync
cycles. The list should hold one entry per (date partition, writer).
Observed: it appends a new entry on every sync cycle — one per minute — until
the UTC date rolls over. On our deployment manifest_list reached 3110 entries
across 66 unique manifest paths, making .stream.json 795 KB (99.9% of it
manifest_list) and producing 5664 manifest GETs in a five minute window, with a
single query fetching the same manifest object 33 times.
Root cause
In src/catalog/mod.rs, process_single_partition finds the entry to update by time
range alone:
let pos = meta.snapshot.manifest_list.iter().position(|item| {
item.time_lower_bound <= partition_lower && partition_lower < item.time_upper_bound
});
handle_existing_partition then checks whether that entry belongs to the current
node before allowing an in-place update:
let manifest_file_name = manifest_path("").to_string();
let should_update = manifests[pos].manifest_path.contains(&manifest_file_name);
The two ask different questions. position() returns the first time-overlapping
entry, which need not be the current writer's. Once a foreign entry occupies that
position, should_update is false, the else branch appends, and the next sync
repeats the same lookup with the same result. The current writer never finds its own
entry, so it never updates in place.
While one node owns a date for a full day the two checks coincide by accident —
there is a single entry and it belongs to that node — which is why stable
single-writer deployments never hit this.
Reproduction
- Run Parseable against an S3 backend and ingest into a stream.
- Restart the pod mid-day (any change of node identity will do).
- Poll the snapshot:
aws s3 cp "s3://$BUCKET/$STREAM/.stream/.stream.json" - \
| jq '.snapshot.manifest_list | {total: length, unique: (group_by(.manifest_path) | length)}'
total climbs by one per minute; unique stays constant.
Evidence
Duplicate count equals the minutes between the writer change and end of that UTC day:
| Date |
Manifest owner |
Duplicates |
Equivalent |
| 2026-07-15 |
parseable-698ddb94fc-zrq8f |
1417 |
23.6 h |
| 2026-07-16 |
parseable-7576497bcc-z6774 |
939 |
15.6 h |
| 2026-07-07 |
parseable-698ddb94fc-xdk8n |
459 |
7.7 h |
| 2026-07-28 |
parseable-67d9c77976-hqv2f |
196 |
3.3 h |
| 2026-07-28 |
parseable-6db557fd9f-qrfwf |
27 |
27 m |
Only 6 of 66 paths were affected — exactly the dates where writer identity changed.
Growth is visible directly in .stream.json PUT sizes, one 254-byte entry per sync:
00:37:30 bytes=19047
00:38:30 bytes=19301
00:39:30 bytes=19555
Impact
Query latency. Every duplicate is resolved and fetched during planning. After
collapsing to the 66 unique entries, manifest GETs fell from 5664 per five minutes to
51 per three minutes and .stream.json from 795,499 to 19,047 bytes.
Write amplification. The whole snapshot is re-uploaded every sync. At 795 KB that
is roughly 1.1 GB/day of PUT traffic.
Corrupted statistics. Per-date rollups appear to sum duplicated entries, so
reported figures scale with the duplication factor. 2026-07-15 (1417 duplicates)
reported 31,287,815,348 events against 30,810,252 for the adjacent day, a ratio of
about 1015, while the stream's lifetime total was 6,159,104. Gauges also go negative
once deletions are subtracted from the inflated totals:
parseable_events_ingested{stream="functions"} -103800687
parseable_storage_size{stream="functions",type="data"} -16325422799
A per-date value exceeding the lifetime total is a cheap invariant that would have
surfaced this much sooner.
Suggested fix
Match writer identity in the lookup rather than testing it afterwards:
let manifest_file_name = manifest_path("").to_string();
let pos = meta.snapshot.manifest_list.iter().position(|item| {
item.time_lower_bound <= partition_lower
&& partition_lower < item.time_upper_bound
&& item.manifest_path.contains(&manifest_file_name)
});
A node inheriting a partition then finds no match, appends once, and updates in place
thereafter — which appears to be what the existing should_update check already
intends to support.
/aboutoutput:Description:
Expected: when a node writes to a date partition that a different node has already written to, it should append its own
ManifestItemtosnapshot.manifest_listonce, then update that entry in place on subsequent sync cycles. The list should hold one entry per (date partition, writer).Observed: it appends a new entry on every sync cycle — one per minute — until the UTC date rolls over. On our deployment
manifest_listreached 3110 entries across 66 unique manifest paths, making.stream.json795 KB (99.9% of itmanifest_list) and producing 5664 manifest GETs in a five minute window, with a single query fetching the same manifest object 33 times.Root cause
In
src/catalog/mod.rs,process_single_partitionfinds the entry to update by time range alone:handle_existing_partitionthen checks whether that entry belongs to the current node before allowing an in-place update:The two ask different questions.
position()returns the first time-overlapping entry, which need not be the current writer's. Once a foreign entry occupies that position,should_updateisfalse, the else branch appends, and the next sync repeats the same lookup with the same result. The current writer never finds its own entry, so it never updates in place.While one node owns a date for a full day the two checks coincide by accident — there is a single entry and it belongs to that node — which is why stable single-writer deployments never hit this.
Reproduction
totalclimbs by one per minute;uniquestays constant.Evidence
Duplicate count equals the minutes between the writer change and end of that UTC day:
Only 6 of 66 paths were affected — exactly the dates where writer identity changed. Growth is visible directly in
.stream.jsonPUT sizes, one 254-byte entry per sync:Impact
Query latency. Every duplicate is resolved and fetched during planning. After collapsing to the 66 unique entries, manifest GETs fell from 5664 per five minutes to 51 per three minutes and
.stream.jsonfrom 795,499 to 19,047 bytes.Write amplification. The whole snapshot is re-uploaded every sync. At 795 KB that is roughly 1.1 GB/day of PUT traffic.
Corrupted statistics. Per-date rollups appear to sum duplicated entries, so reported figures scale with the duplication factor.
2026-07-15(1417 duplicates) reported 31,287,815,348 events against 30,810,252 for the adjacent day, a ratio of about 1015, while the stream's lifetime total was 6,159,104. Gauges also go negative once deletions are subtracted from the inflated totals:A per-date value exceeding the lifetime total is a cheap invariant that would have surfaced this much sooner.
Suggested fix
Match writer identity in the lookup rather than testing it afterwards:
A node inheriting a partition then finds no match, appends once, and updates in place thereafter — which appears to be what the existing
should_updatecheck already intends to support.