Help! My Lambda Stopped Working After a Deploy

Whether you are deploying to a staging or production environment, a broken app that works fine on your local environment can be a tricky thing to fix. On AWS Lambda this is made worse by per-execution log files that can be rather time-consuming to pore through.

Automatic Runtime Management

To keep this post simple, I'll assume that you are using Node on AWS Lambda. But the principle applies to all runtimes.

By default AWS Lambda uses automatic runtime management. You specify the major version e.g. Node 18 and AWS handles setting up the environment for your Lambda.

Minor Releases

What you may not be aware of, is that if your runtime management is set to Auto then AWS will silently update the minor version being used for any given runtime. This new minor version will be applied on your next deploy of the Lambda.

This is obviously convenient, as it means your Lambda runtime stays up to date with any fixes or security patches etc that come with the minor release.

The Problem With That…

Minor releases should not introduce breaking changes. In theory. But in a sea of different dependencies, sometimes that happens, and that's what happened to us a couple of days ago.

We deployed a Lambda which was previously working fine, it broke, and it took a long time to figure out that the reason is because our runtime minor version had been updated in the background.

There was some incompatibility with the runtime minor release and one of our dependencies. If you've exhausted all other obvious reasons as to why your Lambda is broken, this might be a possible explanation.

Coming to this conclusion and then figuring out how to roll back all took quite a while, so hopefully if you're reading this it will save you some time!

Turning off Automatic Runtime Management

In AWS find your Lambda and under the Code tab look at Runtime settings.

Click Edit runtime management configuration.

Here you can turn off Auto mode and switch to Manual mode.

When using Manual mode you will need to specify an ARN address of the exact runtime that you want to use.

Finding the ARN of a Working Deploy

So now all you need is to find the ARN of a runtime used on a working deploy.

The easiest way to do that is to use the Versions feature of Lambda, if you have that turned on (which you should!).

Although Versions doesn't allow us to conveniently roll back, it does display the runtime used in the Code tab in the Runtime settings section.

You can grab the ARN of a previously working deploy, and paste it into the Manual mode settings, redeploy and everything should be working again!

Alternatively you can dig through your log files to a previously working execution and the ARN of the runtime used should be displayed there.

Conclusion

If you're running a production app you probably want to be in full control over your environment. In the same way you most likely freeze your dependency versions, you may also want to freeze your runtime version. Switching to Manual mode helps you do this.