Application logging in AWS Lambda with Serverless plugin and Papertrail

We use Papertrail service to gather logs from all our apps across different platforms and programming languages. So, naturally, when we dived into a serverless world of AWS, one of our first efforts was to find out how to make it work there too.

AWS Lambda sends every console output to CloudWatch logs, so the idea was to grab the logs from there and forward them to Papertrail. The other option would be to send them to Papertrail directly from the app but we didn’t want to increase processing time and wanted the logs of application failures to be completely reliable.

Cloudwatch provides a way to stream log events to Elasticsearch or Lambda. And the second option is interesting here. So we created an independent lambda function which subscribes to the logs of the other functions and forwards them to Papertrail. We took first inspiration from an Apiary repository which utilizes Winston logger for Node.js and its transport to Papertrail.

We didn't want to subscribe the Lambda trigger manually for each new function so we created a simple Serverless plugin which did the work. You can see its first version in an older release of one of our Github repositories. Simply said, it created a hook on deploy:compileEvents event and added an AWS::Logs::SubscriptionFilter and a AWS::Lambda::Permission resources for each function to the resulting CloudFormation template.

However, the logger function and the plugin remained in the source code of each Serverless application until we found a very nice plugin by A Cloud Guru for sending logs to Sumologic which creates the function on the fly. So we drew inspiration there and were able to do the same thing and finally created a standalone plugin which can be easily shared and maintained among more applications.

The plugin ignores implicit Lambda logs (starting with START, END, and REPORT) and adds Lambda request id to each event. Notice that it expects the logs to be in JSON format (and converts them to JSON if they are not).

The installation is simple, add the plugin to your dev dependencies like yarn add @keboola/serverless-papertrail-logging --dev and update your serverless.yml file:

custom:
  papertrail:
    port: 1234
    
plugins:
- '@keboola/serverless-papertrail-logging'

If you want to have the logs enhanced with information about the event and the errors with stack traces, you should check our npm module @keboola/serverless-request-handler. You could also like our Chrome extension to format JSON logs in Papertrail.