GH-50566: [C++] Introduce simdjson and migrate ObjectParser

[Reranko05]

Rationale for this change

This PR is the first in a series of PRs for GH-35460 to migrate Arrow's JSON implementation from RapidJSON to simdjson.

As the foundation for the migration, this PR introduces simdjson into Arrow's build system, updates the bundled simdjson version to v4.6.4, and migrates the self-contained arrow::json::internal::ObjectParser.

What changes are included in this PR?

This PR:

• Introduces simdjson as a third-party dependency.
• Adds CMake support for discovering and building simdjson.
• Updates the bundled simdjson dependency to v4.6.4.
• Adds system package and vcpkg integration required for simdjson across supported build environments.
• Links simdjson into the Arrow JSON library.
• Migrates arrow::json::internal::ObjectParser from RapidJSON to the simdjson DOM API.
• Adds dedicated unit tests covering:
  • String retrieval
  • Boolean retrieval
  • String map extraction
  • Invalid JSON
  • Missing keys
  • Incorrect types
  • Non-object root documents
  • Empty objects

This establishes the build infrastructure needed for the remaining JSON migration, which will be completed incrementally in follow-up PRs.

Performance

On the ObjectParser benchmark, the simdjson implementation achieved approximately 6.3× higher throughput than the previous RapidJSON implementation (about 84% lower execution time).

Are these changes tested?

Yes.

• Added dedicated unit tests for ObjectParser.
• Verified the ObjectParser JSON test suite locally.
• Updated the build system and validated the dependency integration across supported CI configurations (currently 41 CI checks passing; remaining failures are limited to two Windows-specific workflows under investigation).

• GitHub Issue: [C++] Introduce simdjson and migrate ObjectParser #50566

[github-actions]

⚠️ GitHub issue #35460 has been automatically assigned in GitHub to PR creator.

[Reranko05]

@kou As we discussed, I've started working on the RapidJSON → simdjson migration and will try to split it into small, reviewable PRs. This PR is the first step, introducing the dependency and migrating ObjectParser.

I'm currently investigating the remaining CI failures. The implementation builds locally, the JSON test suite passes, and several CI workflows also pass. The remaining failures all seem to fail during CMake configure because simdjson::simdjson isn't available in some configurations, while other workflows successfully fall back to building simdjson from source.

I've narrowed it down to the dependency integration (FindsimdjsonAlt.cmake / CMake target creation). If there's an existing Arrow pattern you'd recommend for handling this kind of third-party dependency integration, I'd really appreciate your guidance.

[Reranko05]

Hi @kou, I've updated the bundled simdjson to v4.6.4, added the required system packages for the SYSTEM dependency path, and added simdjson to the vcpkg manifest. At this point, 41 CI checks are passing and only two Windows-specific workflows are still failing:

• AMD64 Windows MSVC GLib: link failure with
  unresolved external symbol simdjson::internal::error_codes
• AMD64 Windows C++ RTools 40 ucrt64: failure while building the bundled simdjson dependency with
  fatal error: had to relocate PCH

I investigated the MSVC failure by looking into the vcpkg integration (cpp/vcpkg.json, FindsimdjsonAlt.cmake, ThirdpartyToolchain.cmake) and also tried a couple of Windows-specific compile definition experiments, but I haven't been able to resolve it yet.

Do you have any suggestions on where I should look next for these remaining Windows-specific issues?

[github-actions]

⚠️ GitHub issue #35460 has been automatically assigned in GitHub to PR creator.

[kou]

• AMD64 Windows MSVC GLib: link failure with
  unresolved external symbol simdjson::internal::error_codes

FetchContent will fix it.

• AMD64 Windows C++ RTools 40 ucrt64: failure while building the bundled simdjson dependency with
  fatal error: had to relocate PCH

We may need to disable PCH (pre-compiled headers) with old MinGW.

Could you try this?

diff --git a/ci/scripts/PKGBUILD b/ci/scripts/PKGBUILD
index 6a66d86867..4737a7b1d8 100644
--- a/ci/scripts/PKGBUILD
+++ b/ci/scripts/PKGBUILD
@@ -137,6 +137,7 @@ build() {
     -DARROW_CXXFLAGS="${CPPFLAGS}" \
     -DAWSSDK_SOURCE=BUNDLED \
     -DCMAKE_BUILD_TYPE="release" \
+    -DCMAKE_DISABLE_PRECOMPILE_HEADERS=ON \
     -DCMAKE_INSTALL_PREFIX=${MINGW_PREFIX} \
     -DCMAKE_UNITY_BUILD=OFF \
     -DCMAKE_VERBOSE_MAKEFILE=ON \

See also: https://cmake.org/cmake/help/latest/variable/CMAKE_DISABLE_PRECOMPILE_HEADERS.html

[Reranko05]

Hi @kou, I ran a quick microbenchmark comparing the current RapidJSON implementation and the simdjson On-Demand implementation using the same ObjectParser API (Parse, GetString, GetBool) over 10 million iterations in a Release build.

RapidJSON: 18.33 s average (1.83 µs/iteration)
simdjson On-Demand: 2.91 s average (0.29 µs/iteration)

That is approximately a 6.3× speedup (about 84% lower execution time) for this workload.

[Reranko05]

I updated the simdjson integration to use the FetchContent target as well, but the AMD64 Windows MSVC GLib job still fails with the same unresolved external symbol simdjson::internal::error_codes during linking.

I'm not yet sure why the MSVC configuration is the only one exhibiting this linker issue. Do you have any suggestions?

[Reranko05]

I investigated the MSVC failure further.

The failing job is using the vcpkg-provided simdjson (4.6.4), and simdjson::simdjson is resolved as an imported static library:

TYPE = STATIC_LIBRARY
IMPORTED = TRUE
include path = .../vcpkg_installed/x64-windows/include
library = .../vcpkg_installed/x64-windows/debug/lib/simdjson.lib

However, the build still fails with unresolved references to:

simdjson::internal::error_codes
simdjson::internal::structural_or_whitespace_negated

I tried investigating whether this is related to how the vcpkg package is built or linked, but I haven't been able to isolate the cause.

Do you have any suggestions on where I should look next, or is there something specific about the MSVC/vcpkg configuration that I'm missing?

[kou]

That is approximately a 6.3× speedup (about 84% lower execution time) for this workload.

Great!

The failing job is using the vcpkg-provided simdjson (4.6.4), and simdjson::simdjson is resolved as an imported static library:

Ah, it doesn't use bundled simdjson.

Could you download https://github.com/Reranko05/arrow/pkgs/nuget/simdjson_x64-windows and share it? I don't know how to download it...

I want to check the built simdjson by vcpkg.

[Reranko05]

@kou, I tried FORCE_ANY_NEWER_VERSION TRUE. Most of the previous configuration failures are gone, but some are jobs still aren't finding a suitable simdjson. Do you have another suggestion for this?

[kou]

Could you use the following instead of installing libsimdjson-dev in the Ubuntu 22.04 image? simdjson-dev on Ubuntu 22.04 is old: https://packages.ubuntu.com/jammy/libsimdjson-dev

diff --git a/ci/docker/ubuntu-22.04-cpp.dockerfile b/ci/docker/ubuntu-22.04-cpp.dockerfile
index d098eb346e..71a21852b2 100644
--- a/ci/docker/ubuntu-22.04-cpp.dockerfile
+++ b/ci/docker/ubuntu-22.04-cpp.dockerfile
@@ -226,4 +226,5 @@ ENV absl_SOURCE=BUNDLED \
     PARQUET_BUILD_EXECUTABLES=ON \
     PATH=/usr/lib/ccache/:$PATH \
     PYTHON=python3 \
+    simdjson_SOURCE=BUNDLED \
     xsimd_SOURCE=BUNDLED

[Reranko05]

@kou There are only two CI failures remaining:

ARM64 macOS 14 C++: arrow-s3fs-test times out.
debian-trixie-amd64: the Verify Reproducibility job ends with Error: The operation was canceled.

[Reranko05]

@kou All CI checks have passed now, so I'm going to mark this PR as ready for review.

Should I remove -DCMAKE_DISABLE_PRECOMPILE_HEADERS=ON, or keep it?

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[kou]

Should I remove -DCMAKE_DISABLE_PRECOMPILE_HEADERS=ON, or keep it?

We need to keep it because it needs to build simdjson with old GCC.

[github-actions]

⚠️ GitHub issue #35460 has been automatically assigned in GitHub to PR creator.

[github-actions]

⚠️ GitHub issue #35460 has been automatically assigned in GitHub to PR creator.

[kou]

+1

Could you open a new sub-issue instead of closing GH-35460?

Could you add a performance note to the PR description?

[github-actions]

⚠️ GitHub issue #35460 has been automatically assigned in GitHub to PR creator.

[Reranko05]

@kou opened the sub-issue for this PR, and added performance note to PR desc.

[github-actions]

⚠️ GitHub issue #35460 has been automatically assigned in GitHub to PR creator.

[github-actions]

⚠️ GitHub issue #50566 has been automatically assigned in GitHub to PR creator.

[kou]

Thanks.