One day I have deployed my Azure Function app into container to production server (.NET Core on Linux). And app started to fail after few minutes, without any reason, with exit code 137. This post describes how I debugged the problem – to my suprise, it was caused by .NET Garbage Collector.
After searching on web for exit code 137, I found it’s SIGKILL signal – so host environment sent this signal to process in container. Because of that, I tried to start my container and watch stats by command:
$ docker stats
CONTAINER ID NAME CPU % MEM USAGE / LIMIT 4feddd624a5e my-azurefunction-app 10.39% 398.4MiB / 1.795GiB
So it’s in memory…
By this stats I was able to recognize, that my container drains memory so quickly, that all free memory is assigned to it after running 30 seconds. So the problem is in memory management.
I searched for memory leakage in function source code and made some memory optimalization, but with no effect. It was strange to me, that these memory leaks are not present in dev environment – app runs stable around 65MB. I started to think, that there is no GC Collect performed by runtime.
.NET Core environment
This was reason, why I started search more detail about .NET Core environment. And I found the problem – .NET Core uses for app different Server Garbage Collector on server, than on developer’s Workstation GC. Server GC is optimized for higher throughput and scalability – and counts with larger resources.
Servers GC has different limits for ephemeral memory segment (for objects from generation 0+1).
|Workstation GC||16 MB||256 MB|
|Server GC||64 MB||4 GB|
|Server GC with > 4 logical CPUs||32 MB||2 GB|
|Server GC with > 8 logical CPUs||16 MB||1 GB|
So from workstation env my application went to server with 2GB RAM.
Because of func SDK is compiled 32-bit, .NET recieved this env:
But inside Docker container on production?
Switching to Workstation GC
So as solution, I switched GC for my Function app to Workstation GC. It’s not so much optimized, but it works also on so small resources.
You can switch Server GC to Workstation GC by adding value ServerGarbageCollection to .csproj file:
If you are interested, you can find more informations about GC in .NET on Microsoft documentation.
<Project Sdk="Microsoft.NET.Sdk"> <PropertyGroup> <TargetFramework>netstandard2.0</TargetFramework> <AzureFunctionsVersion>v2</AzureFunctionsVersion> <ServerGarbageCollection>false</ServerGarbageCollection> </PropertyGroup> ...