Support custom log attributes via extra logger parameter

[eoinsha]

Hi!

I am using the aws-lambda-powertools-python logger for a number of projects but also have existing deployments using structured JSON logging implemented with a custom logger. Our custom logger supports the Python logging extra parameter to supply custom attributes. It would be great if the aws-lambda-powertools-python formatter supported custom attributes so we can take advantage of structured logging for additional attributes.

Example

from aws_lambda_powertools.logging import Logger
logger = Logger()
logger.info('Put aggregation result', extra={'key': 'results/sum.csv', 'size': 130425})

Current output:

{
  "level": "INFO",
  "location": "<module>:5",
  "message": "Put aggregation result",
  "timestamp": "2020-12-23 11:00:47,695",
  "service": "service_undefined",
  "sampling_rate": 0
}

Desired output:

{
  "level": "INFO",
  "location": "<module>:5",
  "message": "Put aggregation result",
  "timestamp": "2020-12-23 11:00:47,695",
  "service": "service_undefined",
  "sampling_rate": 0,
  "key": "results/sum.csv",
  "size": 130425
}

This enables us to do effective log queries, filtering and stats using CloudWatch Logs Insights.

For reference:

• python-json-logger support custom attributes using extra - https://github.com/madzak/python-json-logger
• Usage of the extra keyword argument to add custom attributes to the LogRecord is covered here: https://docs.python.org/3/library/logging.html

Is this something that you would consider supporting?

[heitorlessa]

Hey @eoinsha - Great to hear from you, and thanks for opening this feature request.

We're in PTO so I'll revisit in January - for now, I have one quick question for you:

Logger allows an arbitrary object instead of string today:

logger.info({
'key': 'results/sum.csv',
'size': 130425
})

This will be encapsulated within the message key instead of being top-level keys:

{
...
"message": {
'key': 'results/sum.csv',
'size': 130425}
}

Does this help or you prefer having a string to message plus extra keys as top level keys?

If that helps, we should update the docs to make this more explicit. If not, I also think is fine having an extra key for this purpose for cases where you want to preserve the message

[eoinsha]

Thanks @heitorlessa! I do have a preference for the extra method as it's consistent with the references, what we do already, makes it easier to switch over. But what you show here achieves the same thing, so it looks like so I can use that on other projects. Thanks again and have a great break! :)

[ran-isenberg]

+1 for this feature. Would make it easier to convice people at my org to use it :)

[ran-isenberg]

it's be great if there would be no need for extra= but just send a kwargs

[heitorlessa]

Hey @eoinsha - PR created! Could you double check if that helps you moving forward? #257

@risenberg-cyberark we wouldn't be able to do it without extra as it'd break the logging protocol and make it slightly difficult for those moving in or outside Logger.

cc Other folks who opened issues in the past, as this feature might help them out too @papi-tokei @ivamluz @patrickwerz @alvaropc

[eoinsha]

This looks perfect, thanks @heitorlessa !

[ivamluz]

Thanks, @heitorlessa ! That covers that scenario I raised in the other PR I opened the other day.

[heitorlessa]

Merging it now, and it should be released by Friday along with other minor improvements ;)

will circle back when 1.10.0 is available on PyPi this week