SnsSqsEnvelope - [ERROR] KeyError: 'SigningCertURL'

[micaellasry]

I'm using v 1.26.0 and the SnsSqsEnvelope for a lambda function

The raw event I get is:

{
    'Records': [{
        'messageId': '1f96888d-207d-4d28-981e-a30809c488ff',
        'receiptHandle': 'AQEB5VS+EqhAAOfhQ2r2yJ3girxAXlyT81NiXniRUiAu2xoMBlhVDoTKfnT76jhp1XDo7p/XNn8+cCniuwYxDKTIIIO9q58WwjkthhoLx7cNpLncNcx4IcjTjwZtTSfIpEnWMMzuyxnEFG64/mJxpRdMbpu+j4nlYJmJ+TTAZlytCTo8cBYgbIqvN0n2QynHs3XIRnfTni70t05N2Stv+kzayC4b2d1UB9LivrMfr1CcEFdqggkB0jRxEI2Tc7X/Njx007HNsGtUfgg6BzJJ9Gi7fQ==',
        'body': '{\n
            "Type" : "Notification",\n 
            "MessageId" : "d6c32d3c-68be-5725-ba9b-7636d02f8d5f",\n 
            "SequenceNumber" : "10000000000000008000",\n 
            "TopicArn" : "arn:aws:sns:eu-west-1:476991913874:test.fifo",\n 
            "Message" : "****",\n
            "Timestamp" : "2022-05-27T11:30:25.776Z",\n
            "UnsubscribeURL" : "https://sns.eu-west-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:eu-west-1:476991913874:test.fifo:264d6e5d-8aaf-4259-aefa-0f93d53a6025"\n
            }',
        'attributes': {
            'ApproximateReceiveCount': '4080',
            'SentTimestamp': '1653651025798',
            'SequenceNumber': '18870078736313839616',
            'MessageGroupId': '****',
            'SenderId': 'AIDAWYJAWPFU7SUQGUJC6',
            'MessageDeduplicationId': '****',
            'ApproximateFirstReceiveTimestamp': '1653651025798'
        },
        'messageAttributes': {},
        'md5OfBody': '4182f034b0e35d4e6cf845484e3bf647',
        'eventSource': 'aws:sqs',
        'eventSourceARN': 'arn:****',
        'awsRegion': 'eu-west-1'
    }]
}

The Sns body contains UnsubscribeURL but does not contain SigningCertURL

I see in the source code: parser/models/sns.py:31 the following code

def check_sqs_protocol(cls, values):
        sqs_rewritten_keys = ("UnsubscribeURL", "SigningCertURL")
        if any(key in sqs_rewritten_keys for key in values):
            values["UnsubscribeUrl"] = values.pop("UnsubscribeURL")
            values["SigningCertUrl"] = values.pop("SigningCertURL")
        return values

This causes the parsing to fail.
Why are these two values considered together?

[michaelbrewer]

Seems like a bug in the parser model code. I would log it as a bug, as the discussions is not looked at too frequently by the project maintainers:

For which the fix would be as follow:

    @root_validator(pre=True, allow_reuse=True)
    def check_sqs_protocol(cls, values):
        sqs_rewritten_keys = ("UnsubscribeURL", "SigningCertURL")
        if any(key in sqs_rewritten_keys for key in values):
            values["UnsubscribeUrl"] = values.pop("UnsubscribeURL", None)
            values["SigningCertUrl"] = values.pop("SigningCertURL", None)
        return values

Otherwise you can use the SNSEvent data class in aws_lambda_powertools.utilities.data_classes which does not rely on Pydantic

[michaelbrewer]

But from the docs for SNS, it really does seem like SigningCertURL should always be included 🤷🏼 ?

[ran-isenberg]

hey @micaellasry , can you open an issue so this gets visibility?
If it gets visibility we might be able to fix this fast.

[ran-isenberg]

i remeber @heitorlessa added this code at one point, dont remember the bug it solved. do you remember?

[michaelbrewer]

@ran-isenberg it was a hack for backwards compatibility to fix the names for SigningCertURL and UnsubscribeURL (#272)

[michaelbrewer]

@micaellasry how was this event created? An a fully setup SNS queue?

I am not sure really if SigningCertURL is also included for both SNS and SNS over SQS use case. (It does not help that the casing between the two is not consistent :(

[heitorlessa]

Hi @micaellasry, could you create a bug report and provide as much information as possible on the steps so we can try reproduce it? Parser Model is behaving as expected[1].

I suspect you might not want the SNS metadata information within your Lambda but simply the payload - if that's the case, please enable raw message delivery when subscribing your SQS queue to SNS; this will ensure your Lambda only receives the SQS envelope + payload instead of SNS+SQS. Suffice to say you'll then switch to SnsEnvelope too.

That being said, please allow me to explain the history of why that branch logic is there.

When SNS team integrated with Lambda, it corrected the Url to URL case in both SigningCertUrl and UnsubscribeUrl - these were previously crafted prior to 2015 to integrate with SQS, HTTP, etc.

When Parser SNS Model was created it took into account SQS protocol (SNS->SQS) and not Lambda protocol (SNS->Lambda). This meant that customers using SNS->SQS->Lambda had a KeyError when validating their payloads - hence why that branch code to remap in case SQS protocol is used while keeping all other keys intact.

---

[1] I've just re-generated SNS Lambda protocol and SNS SQS protocol events and Parser worked as expected.

SNS->SQS subscription (SNS->SQS->Lambda)

{
    "Records": [
        {
            "messageId": "732a2fe0-ae74-459e-91c1-b9000a32af81",
            "receiptHandle": "AQEB2Fhh2Sujp9h8CW3BUEXLRsuRHwWCqGV4jinH6xQ/x0qcCY0XQ8t4J7M/+qH5oDM/9mcsi7hpFOS+PRYwxU6yLIDt1agwy4vyRbus3qC/juCptnH+tqTaS3WTSEOXLLS3JPGAlga5hX6LYPatg+Vn0PVxy36lf0wyocnnh9iOSsOd4YUIQHNMXWcMfMobTITjUZItJX+6hqGw43ciZ7TmYBbflCNttOiYc6BLBQlCNh6S6HEHErJtBWr7MhgrVFVh15OE3l/WfFsd4NJwutl40NlHxra6V1JMYgtr/D0QVzIVGJc3YQNBv71MU8WIDEAAnvqet1w1wcG3IcvC8s3qtKxzOF43I6maLz8xtLaQ2hCTNMCYhTHasRNpyF34Ou1E81MZzkOQCS1K/am6xDovDQ==",
            "body": "{\n  \"Type\" : \"Notification\",\n  \"MessageId\" : \"992beb60-b6fe-507e-8b53-b0798a6e534c\",\n  \"TopicArn\" : \"arn:aws:sns:eu-west-1:123456789012:powertools265\",\n  \"Message\" : \"'{\\\"hello\\\": \\\"world\\\"}'\",\n  \"Timestamp\" : \"2022-06-07T11:45:20.450Z\",\n  \"SignatureVersion\" : \"1\",\n  \"Signature\" : \"bF03RCWw+vPDNTun1U2kn/Ng3z1k74FSx2Kp2v14HjEUVRFjp6oq+C568TcBUOx3eZLfxwgtW0JSNr+idV+MWIUtXfZxglthj9vOLAKwgZeB8Zu4e4Gi+pEN669LLn5ylO0oGL0SMaFVDfWmDXs5E7x8CEgt5lNmqc5GI88pC/RjIwwS3cpLk+3DGLpgLR/T7wt3s3A2Eq8hjhb7kny4J5WmFdDocbRi21VkmO2CYS+i518kHjYZgqPcUiLOZOJFsYdJU3FWu23AR5EaeCwWLir7mFovjkdLR6xPSui8lB9WFsNvBKkOuWZ6S1xenZJgCCaeX6El9G/61+sMdULTGQ==\",\n  \"SigningCertURL\" : \"https://sns.eu-west-1.amazonaws.com/SimpleNotificationService-7ff5318490ec183fbaddaa2a969abfda.pem\",\n  \"UnsubscribeURL\" : \"https://sns.eu-west-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:eu-west-1:123456789012:powertools265:15189ad7-870e-40e5-a7dd-a48898cd9f86\"\n}",
            "attributes": {
                "ApproximateReceiveCount": "1",
                "SentTimestamp": "1654602320475",
                "SenderId": "AIDAISMY7JYY5F7RTT6AO",
                "ApproximateFirstReceiveTimestamp": "1654602320482"
            },
            "messageAttributes": {},
            "md5OfBody": "fd181a34ede44c6cb573dd6b10c844aa",
            "eventSource": "aws:sqs",
            "eventSourceARN": "arn:aws:sqs:eu-west-1:123456789012:powertools265",
            "awsRegion": "eu-west-1"
        }
    ]
}

SNS->Lambda subscription

{
    "Records": [
        {
            "EventSource": "aws:sns",
            "EventVersion": "1.0",
            "EventSubscriptionArn": "arn:aws:sns:eu-west-1:123456789012:powertools265:3f5bb551-d95e-4088-ac0d-f7dd674a76b6",
            "Sns": {
                "Type": "Notification",
                "MessageId": "fdff66f1-8802-5c34-a142-3de8b330c72f",
                "TopicArn": "arn:aws:sns:eu-west-1:123456789012:powertools265",
                "Subject": null,
                "Message": "'{\"hello\": \"world\"}'",
                "Timestamp": "2022-06-07T11:39:30.302Z",
                "SignatureVersion": "1",
                "Signature": "fF2wK5FJdjFiijJoBaJrgEcVdhHsV1dij8WE+HvQG9TJ6ZrorDzM8503ELfghP2Widj3tG7IrUQSCgqh3EAx2Y2pexzISgEI7V0Q2koCZaknel9x6U/VA2eEOlEAsg/tpdysTRB+TUGyEqLyG9sX7qKS9EFqqCtP6l51PP371Lbw9xnA635drwZ2GwjiExxTiO4Ytwuuqnw58wu2pJFjFW87JFQjQW1WO1e9+ZXOXaqhBtjqQFExhfXZhkbDU3675BPnhjXh2xz3BTh/POL02Gnui1uaE79ZZMduNLehzrzRpFSIc3ZfYIVgMD0Ut/XJFW4MxRt3rJ7BZrIz47NgeA==",
                "SigningCertUrl": "https://sns.eu-west-1.amazonaws.com/SimpleNotificationService-7ff5318490ec183fbaddaa2a969abfda.pem",
                "UnsubscribeUrl": "https://sns.eu-west-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:eu-west-1:123456789012:powertools265:3f5bb551-d95e-4088-ac0d-f7dd674a76b6",
                "MessageAttributes": {}
            }
        }
    ]
}

[michaelbrewer]

I don't think raising a KeyError would be what you would expect ( @ran-isenberg what do you think? ).

I would expect a ValidationError would be raised.

Here is a patch, it this is ever excepted as a bug by the maintainers

diff --git a/aws_lambda_powertools/utilities/parser/models/sns.py b/aws_lambda_powertools/utilities/parser/models/sns.py
index e329162e..9192f3e2 100644
--- a/aws_lambda_powertools/utilities/parser/models/sns.py
+++ b/aws_lambda_powertools/utilities/parser/models/sns.py
@@ -32,7 +32,7 @@ class SnsNotificationModel(BaseModel):
         sqs_rewritten_keys = ("UnsubscribeURL", "SigningCertURL")
         if any(key in sqs_rewritten_keys for key in values):
             values["UnsubscribeUrl"] = values.pop("UnsubscribeURL")
-            values["SigningCertUrl"] = values.pop("SigningCertURL")
+            values["SigningCertUrl"] = values.pop("SigningCertURL", None)
         return values
 
 
diff --git a/tests/functional/parser/test_sns.py b/tests/functional/parser/test_sns.py
index 81158a44..ef9127ae 100644
--- a/tests/functional/parser/test_sns.py
+++ b/tests/functional/parser/test_sns.py
@@ -1,3 +1,4 @@
+import json
 from typing import Any, List
 
 import pytest
@@ -103,3 +104,16 @@ def handle_sns_sqs_json_body(event: List[MySnsBusiness], _: LambdaContext):
 def test_handle_sns_sqs_trigger_event_json_body():  # noqa: F811
     event_dict = load_event("snsSqsEvent.json")
     handle_sns_sqs_json_body(event_dict, LambdaContext())
+
+
+def test_handle_sns_sqs_trigger_event_json_body_missing_field():
+    # GIVEN an event with a missing SigningCertURL
+    event_dict = load_event("snsSqsEvent.json")
+    payload = json.loads(event_dict["Records"][0]["body"])
+    assert payload.pop("SigningCertURL") is not None
+    event_dict["Records"][0]["body"] = json.dumps(payload)
+
+    # WHEN parsing the payload
+    # THEN raise a ValidationError error
+    with pytest.raises(ValidationError):
+        handle_sns_sqs_json_body(event_dict, LambdaContext())

[ran-isenberg]

"validators should either return the parsed value or raise a ValueError, TypeError, or AssertionError (assert statements may be used)."
https://pydantic-docs.helpmanual.io/usage/validators/

if it's the envelope code, i agree, validation error is the way to go

[heitorlessa]

Thank you for your input. I'm gonna lock this discussion as we're deviating from solving the problem of why SigningCertURL wasn't present in the event (bigger issue). The semantics of the exception type will be addressed elsewhere.

[heitorlessa]

@micaellasry sending a friendly reminder to recall our ask to create a bug report to investigate how
you ended up without those keys.

PS: We can't reproduce it using SNS->Lambda, or SNS->SQS->Lambda. What we can reproduce is the KeyError by intentionally removing those keys from the original event.

Thank you!

[heitorlessa]

@micaellasry @ran-isenberg - this internal fix it'll be available in today's release thanks to @rubenfonseca

#1299