Featured image of post Running Azure Functions in Docker container – crashes on production

Running Azure Functions in Docker container – crashes on production

One day I have deployed my Azure Function app into container to production server (.NET Core on Linux). And app started to fail after few minutes, without any reason, with exit code 137. This post describes how I debugged the problem – to my suprise, it was caused by .NET Garbage Collector.

After searching on web for exit code 137, I found it’s SIGKILL signal – so host environment sent this signal to process in container. Because of that, I tried to start my container and watch stats by command:

1
2
3
$ docker stats
CONTAINER ID        NAME                                      CPU %               MEM USAGE / LIMIT
4feddd624a5e        my-azurefunction-app                      10.39%               398.4MiB / 1.795GiB  

So it’s in memory…

By this stats I was able to recognize, that my container drains memory so quickly, that all free memory is assigned to it after running 30 seconds. So the problem is in memory management.

I searched for memory leakage in function source code and made some memory optimalization, but with no effect. It was strange to me, that these memory leaks are not present in dev environment – app runs stable around 65MB. I started to think, that there is no GC Collect performed by runtime.

.NET Core environment

This was reason, why I started search more detail about .NET Core environment. And I found the problem – .NET Core uses for app different Server Garbage Collector on server, than on developer’s Workstation GC. Server GC is optimized for higher throughput and scalability – and counts with larger resources. Servers GC has different limits for ephemeral memory segment (for objects from generation 0+1). Very different.

GC32-bit64-bit
Workstation GC16 MB256 MB
Server GC64 MB4 GB
Server GC with > 4 logical CPUs32 MB2 GB
Server GC with > 8 logical CPUs16 MB1 GB

So from workstation env my application went to server with 2GB RAM. Because of func SDK is compiled 32-bit, .NET recieved this env:

1
ProcessorCount:4,Is64BitOperatingSystem:True,Is64BitProcess:False

But inside Docker container on production:

1
ProcessorCount:2,Is64BitOperatingSystem:True,Is64BitProcess:True

Switching to Workstation GC

So as solution, I switched GC for my Function app to Workstation GC. It’s not so much optimized, but it works also on so small resources.

You can switch Server GC to Workstation GC by adding value ServerGarbageCollection to .csproj file:

1
2
3
4
5
6
7
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netstandard2.0</TargetFramework>
    <AzureFunctionsVersion>v2</AzureFunctionsVersion>
    <ServerGarbageCollection>false</ServerGarbageCollection>
  </PropertyGroup>
...

If you are interested, you can find more informations about GC in .NET on Microsoft documentation.

comments powered by Disqus
Built with Hugo
Theme Stack designed by Jimmy