Can `ResumableParser` return whether the last `partial_value` container is completed or not?

[kou]

This is not a critical feature request.

I'm trying to migrate to json gem from json-stream gem in https://github.com/groonga/groonga-command-parser/ by JSON::ResumableParser.

It parses records to be loaded to Groonga (a full text search engine with column store). The records use the following format:

[
{"column_name1": "record1's column_value1", "column_name2": 2, ...},
{"column_name1": "record2's column_value1", "column_name2": 2, ...},
...
]

groonga-command-parser wants to process a record as soon as it's completed. For example, it wants to process the first record when the following chunk is processed:

[
{"column_name1": "record1's column_value1", "column_name2": 2, ...},

We can do it by the following script:

require "json"

parser = JSON::ResumableParser.new
parser << <<-CHUNK
[
{"column1": "value1"},
CHUNK
parser.parse
p parser.partial_value[0] # => {"column1" => "value1"}

But it doesn't work with the following chunk:

[
{"column_name1": "record1's column_value1",

We can't know whether the record is completed or not:

require "json"

parser = JSON::ResumableParser.new
parser << <<-CHUNK
[
{"column1": "value1",
CHUNK
parser.parse
p parser.partial_value[0] # => {"column1" => "value1"} # This is not completed yet

groonga-command-parser has a workaround for it. It ignores the last record:

require "json"

parser = JSON::ResumableParser.new
parser << <<-CHUNK
[
{"column1": "value1"},
CHUNK
parser.parse
p parser.partial_value[0..-2] # => []

parser << <<-CHUNK
{"column1": "value2"},
CHUNK
parser.parse
p parser.partial_value[0..-2] # => [{"column1" => "value1"}] # 1st record

If we can detect whether a record in this example is completed or not, it's useful. But this is not a critical feature request because there is a workaround for this case.

[byroot]

I see. Extracting complete elements of a top level array, is indeed a use case that makes a lot of sense and that I would like to support.

Not quite certain of the best way yet, but I will think about it.

The question though is whether only supporting it for a top-level array is enough, or if we'd need something more flexible.

Perhaps:

parser = JSON::ResumableParser.new
parser << <<~CHUNK
  [
  {"column1": "value1"},
CHUNK
parser.skip_array
while parser.parse
  p parser.value # => {"column1" => "value2"}
end

But what if you want to do the same for an array that is nested inside an object?, e.g.

{
  "events": [
    {...},
    {...}
  ]
}

But either way, I feel like having a way to tell the parser to ignore some levels of the structure is probably the way to go.

NB: I'm at a conference, unclear when I'll have time to experiment with this.

[kou]

Ah, good point.

In the groonga-command-parser use case, the [...] pattern is only needed.

But there is another use case that use the {"XXX": [...]} pattern. It's Groonga's logical_range_filter response: https://github.com/groonga/groonga/blob/05a7fc1ec296ca31c7e4b2f47348032103f13b9d/test/command/suite/sharding/logical_range_filter/command_version/3/json.expected#L28-L102

It has the following structure:

{
  "body": {
    "columns": [
      {...column1...},
      {...column2...},
      ...
    ],
    "records": [
      [...record1...],
      [...record2...],
      ...
    ]
  },
  "header": {...}
}

It's helpful that the records part can be processed as soon as it's parsed. But this is a minor use case...

I also checked an OpenAI API: https://developers.openai.com/api/docs/guides/migrate-to-responses#messages-vs-items

It has the following structure:

{
  ...,
  "output": [
    {...output1...},
    {...output2...},
    ...
  ],
  ...
}

But it's not returned as a stream.

OpenAI API uses server-sent events for streamed response: https://developers.openai.com/api/docs/guides/streaming-responses

It returns small complete JSONs as events. So we don't need to care about OpenAI API case.

[byroot]

On mobile so can’t give a lot of details, but I wonder if the on_load callback already solves the need, as it yields all values as they are completed.

Of course in some case it may not be easy to identify the values you care about

[kou]

It's a good idea.

Can we add ResumableParser#current_nesting? If we have it, we can implement it by the following script:

require "json"

parser = nil
on_load = lambda do |value|
  p value if value.is_a?(Hash) and parser.current_nesting == 1
  value
end
parser = JSON::ResumableParser.new(on_load: on_load)
parser << <<-CHUNK
[
{"column1": "value1", "column2": "value2"},
CHUNK
parser.parse

parser << <<-CHUNK
{"column1": "value2"},
CHUNK
parser.parse

[byroot]

Can we add ResumableParser#current_nesting?

It wouldn't be hard yes.

we can implement it by the following script:

Yes that would work, but probably wouldn't be very efficient, as the lambda would be invoked a lot. So it would be OK as a quick hack, but I would feel awkward documenting this as a supported use case.

The more I think about it, the more I think the clean solution is to go toward StringScanner inspired methods, such as skip_*.

Something like:

parser.skip_array_start # => true
until parser.skip_array_end
  if  parser.parse
    p parser.value
  end
  parser.skip_comma
end

[kou]

Interesting.

It may be useful that skip_* return skipped string or nil instead of boolean like StringScanner does.

[byroot]

Indeed. e.g. to allow seeking into an object, where you can't know the names order, you may need to do something like:

parser.skip_to_any_key(["some_key", "some_other_key"]) # => "some_other_key"

That's just a vague idea though, I'll have to see what's really doable when I'll try to implement it (not sure when).