EventBridge Schema Registry uses custom formats not handled by new Validation code

[n2N8Z]

What were you trying to accomplish?
Validate events against EventBridge Schema Registry published schema.

Expected Behavior

AWS published schema can be imported, and events successfully validated against it.

Current Behavior

Importing schema results in fastjsonschema error:
fastjsonschema.exceptions.JsonSchemaDefinitionException: Unknown format: int64

Possible Solution

1. Pass through custom formats Dict.

2. AWS schema not use custom formats on non-string types.

3. fastjsonschema not fail on custom formats.
   Custom formats for non-string types horejsek/python-fastjsonschema#108

Steps to Reproduce (for bugs)

import json
from aws_lambda_powertools.utilities.validation import validator, validate

json_schema_dict = json.loads(json.loads(open('aws.s3@AWSAPICallViaCloudTrail-v2.JSONSchemaDraft4.txt').read())['Content'])
validate(event={}, schema=json_schema_dict)

aws.s3@AWSAPICallViaCloudTrail-v2.JSONSchemaDraft4.txt

Environment

• Powertools version used: 1.9.0
• Packaging format (Layers, PyPi): PyPi
• AWS Lambda function runtime:: N/A
• Debugging logs: N/A

How to enable debug mode**

# paste logs here
fastjsonschema.exceptions.JsonSchemaDefinitionException: Unknown format: int64

[heitorlessa]

Hi @n2N8Z - Thank you for raising this, and for creating a PR on that.

We're all currently in PTO - We'll look into it in early January. In the meantime, wouldn't Parser utility be a better fit in this case?

I wasn't aware you could have custom formats in JSON Schemas, and if fastjsonschema supports ignoring it then I'm for it as long as we're explicit in the docs.

[n2N8Z]

In the meantime, wouldn't Parser utility be a better fit in this case?

I'm trying to validate an event against its schema definition, which is what fastjsonschema does, and so since that is the library you're using for the Validation feature, and because the feature is called "Validation", I figured that was the appropriate feature. If you think that the Parser feature is the feature to be used for event validation, then when would the Validation feature be used ?

I wasn't aware you could have custom formats in JSON Schemas, and if fastjsonschema supports ignoring it then I'm for it as long as we're explicit in the docs.

It's not so much that fastjsonschema supports ignoring the custom formats, fjs defaults to validating formats, including custom formats, and so you need to provide a regex or function to do that. There's a specific issue (horejsek/python-fastjsonschema#108) with the EventBridge schema that I was working on, that results in the custom format validation being skipped even after a validation function is provided, but it must still be provided for the schema to be accepted.

[heitorlessa]

I've just merged that PR with some modifications and I'll circle back here once 1.10 is released with it - Hopefully by end of Friday if we manage to ship other minor improvements.

As for your question - validation vs parser utility - it wouldn't indeed work now that I got my laptop and read through all issues, since this is a schema you don't have control over (EventBridge CloudTrail AWS API Call). Parser uses Pydantic so it can do both parsing as well as deep data validation, and it supports more complex use cases.

Once again, we appreciate your contribution very much and we'll ping here once it's released in PyPi.

Thank you <3

[heitorlessa]

hey @n2N8Z - This is now available on 1.10 on PyPi and on SAR :)

Closing this now, and thank you for the contribution, much appreciated!