Encoding::InvalidByteSequenceError with json 2.8.1

[jterapin]

Hi again!

When using json 2.8.1 with Ruby 3.2.0 (for additional context, we use CodeBuild instances) - been running into the following error:

Encoding::InvalidByteSequenceError: "\xE2" on US-ASCII (Encoding::InvalidByteSequenceError)
/codebuild/output/src3972890485/src/.gems/ruby/3.2.0/gems/json-2.8.1/lib/json/common.rb:204:in `encode'
/codebuild/output/src3972890485/src/.gems/ruby/3.2.0/gems/json-2.8.1/lib/json/common.rb:204:in `parse'
/codebuild/output/src3972890485/src/.gems/ruby/3.2.0/gems/json-2.8.1/lib/json/common.rb:204:in `parse'
/codebuild/output/src3972890485/src/.gems/ruby/3.2.0/gems/json-2.8.1/lib/json/common.rb:710:in `load'
/codebuild/output/src3972890485/src/aws-sdk-ruby/build_tools/services.rb:93:in `load_docs'

The load_docs referred above is here: https://github.com/aws/aws-sdk-ruby/blob/version-3/build_tools/services.rb#L93
Could any of the recent changes have effected this? Reverting back to json 2.7.6 has fixed this for us.

InvalidByteSequenceError: "\xE2" on US-ASCII feels like its trying to read it as ASCII instead of UTF-8.

Any guidance will be appreciated. Thanks much!!!

[byroot]

Yes, I'm making the library progressively stricter about encoding, this one I think I should just note in the CHANGELOG.

InvalidByteSequenceError: "\xE2" on US-ASCII

This means you are passing an UTF-8 strings, but it's encoded as ASCII, looking at your code, it comes directly from File.read, so somehow you must have Encoding.default_external == Encoding::ASCII.

ruby -e 'Encoding.default_external = Encoding::ASCII; p File.read("/etc/passwd").encoding'
#<Encoding:US-ASCII>

I think you should check your $LANG environment variable. But yes, either way you are passing an invalid string to JSON so it's expected it raises. We only have a small exception for binary strings but that's it.

[byroot]

A quick workaround can also be:

JSON.load(File.read(model_path('docs-2.json', models_dir), encoding: Encoding::UTF_8))

[jterapin]

Hi! Just wondering on the ETA on the gem update with this fix? TIA

[byroot]

I'm just waiting a few days to see if some other issues are reported.

[byroot]

2.8.2 was released.

[jterapin]

Hello @byroot - it's me again. We used the latest version of the json gem and still encountered the same error in our CodeBuild project (see below). I may have to make some configuration on the CodeBuild project settings (based on reading this post but before I look into doing that - I wanted to double check with you. Thanks again.

/codebuild/output/src2789127840/src/aws-sdk-ruby/build_tools/services.rb:188: warning: previous definition of Services was here
rake aborted!
Encoding::InvalidByteSequenceError: "\xE2" on US-ASCII (Encoding::InvalidByteSequenceError)
/codebuild/output/src2789127840/src/.gems/ruby/3.2.0/gems/json-2.8.2/lib/json/common.rb:205:in `encode'
/codebuild/output/src2789127840/src/.gems/ruby/3.2.0/gems/json-2.8.2/lib/json/common.rb:205:in `parse'
/codebuild/output/src2789127840/src/.gems/ruby/3.2.0/gems/json-2.8.2/lib/json/common.rb:205:in `parse'
/codebuild/output/src2789127840/src/.gems/ruby/3.2.0/gems/json-2.8.2/lib/json/common.rb:711:in `load'
/codebuild/output/src2789127840/src/aws-sdk-ruby/build_tools/services.rb:93:in `load_docs'
/codebuild/output/src2789127840/src/aws-sdk-ruby/build_tools/services.rb:70:in `build_service'
/codebuild/output/src2789127840/src/aws-sdk-ruby/build_tools/services.rb:51:in `block in services'
/codebuild/output/src2789127840/src/aws-sdk-ruby/build_tools/services.rb:50:in `each'
/codebuild/output/src2789127840/src/aws-sdk-ruby/build_tools/services.rb:50:in `inject'
/codebuild/output/src2789127840/src/aws-sdk-ruby/build_tools/services.rb:50:in `services'
/codebuild/output/src2789127840/src/aws-sdk-ruby/build_tools/services.rb:42:in `each'
/codebuild/output/src2789127840/src/aws-sdk-ruby/tasks/build.rake:7:in `block in <top (required)>'
tasks/sdk.rake:60:in `block in <top (required)>'
/codebuild/output/src2789127840/src/releaseTools/lib/rake_code_build_output.rb:8:in `top_level'
/codebuild/output/src2789127840/src/.gems/ruby/3.2.0/gems/rake-13.2.1/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => build
(See full trace by running task with --trace)

[byroot]

The backtrace clearly shows you are not using .load_file.

[jterapin]

Aaaah, thanks for catching that.