Skip to content

[Question] ResumableParser: recommended way to detect a truncated stream at EOF? #1047

Description

@Watson1978

In Fluentd we are migrating our JSON stream parsing from yajl-ruby to JSON::ResumableParser (fluent/fluentd#5383). One behavior we want to preserve: when an input stream ends in the middle of a JSON document, the consumer should be able to notice it (Yajl raised a ParseError in that situation; we now emit a warning instead of silently dropping the incomplete tail).

Problem

There doesn't seem to be a single API that answers "did the stream end mid-document?". Our first attempt used rest.empty?, but rest only reflects unconsumed tokenizer bytes, so a stream cut on a complete token boundary (right after a : or ,) leaves rest empty and eos? returning true, even though a document is clearly in progress:

parser = JSON::ResumableParser.new
parser << '{"a":1,'
parser.value while parser.parse
parser.rest          # => ""
parser.eos?          # => true    (but the document is incomplete and gets silently dropped)
parser.partial_value # => {"a"=>1}

Reproduction

require 'bundler/inline'
gemfile do
  source 'https://rubygems.org'
  gem 'json', '= 2.20.0'
end

CASES = [
  ['{"a":1}',           false],  # [input, truncated?]
  ['{"a":1}{"b":2}',    false],
  ['{"a":1} ',          false],
  ['{"a":1}{"b":2',     true],   # cut inside a number token
  ['{"a":1}{"b":',      true],   # cut right after a colon (token boundary)
  ['{"a":1}{',          true],   # cut right after object open
  ['{"a":1,',           true],   # cut right after a comma (token boundary)
]

CASES.each do |input, truncated|
  parser = JSON::ResumableParser.new
  parser << input
  parser.value while parser.parse
  detected = !parser.rest.empty? || !parser.partial_value.nil?
  puts "%-18s rest=%-6s eos?=%-5s partial_value=%-12s detected=%s%s" % [
    input.inspect, parser.rest.inspect, parser.eos?,
    parser.partial_value.inspect, detected,
    detected == truncated ? "" : "  <-- MISMATCH"
  ]
end

Output on json 2.20.0 (on Ruby 4.0.5):

"{\"a\":1}"          rest=""     eos?=true  partial_value=nil          detected=false
"{\"a\":1}{\"b\":2}" rest=""     eos?=true  partial_value=nil          detected=false
"{\"a\":1} "         rest=""     eos?=true  partial_value=nil          detected=false
"{\"a\":1}{\"b\":2"  rest="2"    eos?=false partial_value={"b"=>nil}   detected=true
"{\"a\":1}{\"b\":"   rest=""     eos?=true  partial_value={"b"=>nil}   detected=true
"{\"a\":1}{"         rest="{"    eos?=false partial_value=nil          detected=true
"{\"a\":1,"          rest=""     eos?=true  partial_value={"a"=>1}     detected=true

What we ended up with

rest and partial_value turn out to be complementary (rest catches the mid-token and just-after-{ cuts, partial_value catches the token-boundary cuts), so we currently use:

truncated = !parser.rest.empty? || !parser.partial_value.nil?

Questions

  1. Is this combination the intended way to detect truncation, and is it something we can rely on across versions? Both behaviors were established empirically, and building a partial_value object just for a nil check feels like a workaround.
  2. Is eos? returning true while a document is structurally in progress ('{"a":1,') the intended semantics? If eos? is meant to be tokenizer-level only, a doc note would help; if it's meant to answer "is the parser at a clean end of stream?", this looks like it should return false.
  3. Would you be open to a dedicated predicate (e.g. incomplete? / in_document?) for this?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions