Slow local file reads when using GetResult.into_stream

[ariel-miculas]

Describe the bug
See apache/datafusion#21450
Root cause: there's a spawn_blocking call per each 8KiBs read from the file, adding significant context switch overhead

To Reproduce
See apache/datafusion#21446
For the tests I've used a c7a.16xlarge ec2 instance, with a trimmed down version of hits.json to 51G (original has 217 GiB), with a warm cache (by running cat hits_50.json > /dev/null)

Expected behavior

A more efficient implementation (e.g. tokio uses a buffer size of 2MiBs when reading files)

Additional context
apache/datafusion#21478 (comment)

[tustvold]

I think the trade-off here depends on which is more precious, memory or CPU/latency. 2MB might be peanuts in an analytics context, but significant in contexts with higher concurrency.

Perhaps we could make this configurable, e.g. as a parameter?

[ariel-miculas]

Even without increasing the buffer size, we could reduce the context switch overhead, e.g. by using a single spawn_blocking thread and an mpsc channel, as I mentioned in apache/datafusion#21478 (comment) when considering alternative implementations.

[tustvold]

Does that not just move the problem into how big to make the mpsc buffer?

[ariel-miculas]

Not really, cause you eliminate the overhead of creating one spawn_blocking task per chunk.

[ariel-miculas]

using an mpsc channel with one slot and sending 8KiB at a time, I get this result:

┏━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query    ┃ json-test-on-main ┃       HEAD ┃       Change ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 0 │        2438.00 ms │ 3268.99 ms │ 1.34x slower │
└──────────┴───────────────────┴────────────┴──────────────┘

whereas with the existing into_stream implementation I get:

┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃ json-test-on-main ┃ test-json-improvement ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │        2938.54 ms │           36468.92 ms │ 12.41x slower │

[tustvold]

In which case SGTM 😄

[ariel-miculas]

The only problem is that a long-living spawn_blocking thread runs into this sort of thing: apache/datafusion#15654