Introduction
I created the tool I’m discussing in this blog post to simplify and expedite my troubleshooting of FunctionApp deployments. With this post, I am trying to explain how to use the AzureFuncDebugger tool to resolve various FunctionApp deployment issues within complex enterprise environments - perfect for when your code’s all good, but the deployment’s hitting a wall because of some config mix-ups.
Note
AzFuncDebugger tool is available for download on GitHub.
The tool provides a one-page report detailing the current configuration, the behavior of the FunctionApp, and tips for resolving any issues found. “There’s no need to run any temporary VM for troubleshooting or to install SDKs and compile code; simply switch the runtime to .NET and load the package directly from URL on GitHub.
Selected Content Storage not working for FunctionApp
In certain configurations, depending on the chosen plan and operating system, Azure Functions may utilize a File Share (a feature within Azure Storage Accounts) to store the source code for Functions (this storage is referred to as Content Storage throughout this article). Given that Azure Functions is built on the Azure AppService, it also supports package deployment, which can be done through a local ZIP file or a remote URL - and with package deployment, Content Storage configuration gets ignored.
If you’ve implemented the security enhancements from my article on securing internal Azure Functions storage with PrivateLink, you might find the setup quite intricate. Regrettably, any blockage to the Content Storage access can cause your Function App to become unresponsive, resulting in 50x server errors and timeouts after a certain duration. Debugging can be particularly taxing because each change requires a waiting period for the new configuration to be saved, the Function App to restart, and the new configuration to load.
The package deployment method allows for the straightforward deployment of the open-source AzFuncDebugger tool directly from GitHub into a FunctionApp that’s encountering issues with:
| |
This ensures the tool loads successfully without timeouts, and it activates the simulator feature for accessing the Content Storage within the given FunctionApp. By using the TEST_CCONNECTIONSTRING and TEST_CSHARE configuration parameters, you can verify the accessibility of the storage, even in “broken” FunctionApp.
FunctionApp has no network visibility to Content Storage
When a Content Storage (the Storage Account for the File Share) is set to block public access (which is recommended) and is made available only within a vNET via PrivateLink, the following issues commonly occur:
File Share storage is not resolved with internal IP
The Private Endpoint linked to the PrivateLink service is assigned an internal IP from within the designated vNET—for instance, mystorageaccount.blob.core.windows.net gets resolved to 10.0.1.5. Private Endpoint automatically generates a record in the Azure DNS Private Zone, superseding the original public DNS record for the service. This modified record is only applied within the confines of the vNET.
If the FunctionApp fails to recognize the internal IP record, you should verify the DNS server configuration by examining the WEBSITE_DNS_SERVER parameter to ensure it’s correctly set. Additionally, the boolean parameter WEBSITE_CONTENTOVERVNET must be enabled, as it dictates to connect to the Content Storage via the vNET instead of through a standard public internet outbound connection.
Additionally, the Simulator feature of the tool is pretty handy. It lets you see the IP address that’s being resolved and gives you a glimpse into the effective networking configuration. This kind of debug info is a lifesaver for figuring out whether the trouble is with the DNS setup or something in the routing - like maybe a network security group or a firewall playing gatekeeper a bit too strictly.
Other common issues with Content Storage
When you’ve double-checked and confirmed that the DNS and network visibility are on point, the issue might be with credentials. That’s where the Simulator feature comes in really handy again. It’ll show you if the stored credentials are doing their job and even give you a list of the detected Function source codes sitting in the content storage.
The Function doesn’t get loaded
The package deployment has higher priority. If you notice that a new Function isn’t loading or updating, it’s a good idea to check if the WEBSITE_RUN_FROM_PACKAGE configuration parameter is active. This parameter might be preventing the loading of Functions directly from the Content Storage. The tool provides insights into the active configuration of the code location for your FunctionApp.
TLS inspection (untrusted internal CA)
When resource accessed from FunctionApp is routed throught enterprise network, there could be some network appliance (e.g. Azure Firewall or Palo Alto), to inspect the encrypted traffic and gets re-signed by internal CA. With variable TEST_HTTPCLIENT_GET_URL you can perform a HTTP(s) call to any resource and see details about HTTP request and response, including certs details.
Enterprise custom DNS server
Many corporate Azure landing zones in enterprise settings use their own on-premises custom DNS servers to resolve addresses, even within Azure vNETs. For accessing resources over PrivateLink, these DNS zones need to be delegated to the Azure DNS Private Zone. This can be done, for example, through the Azure DNS Private Resolver service. Alternatively, a DNS record for these resources needs to be manually created in the on-premises DNS servers. The debugger tool shows you which DNS servers are actually being used and with TEST_DNS_RESOLVE_DOMAIN variable enables to test DNS record of a specific resource that’s published over PrivateLink.
Whitelisted IP traffic
In scenarios where a FunctionApp is connected to a vNET and needs to access certain resources through this vNET (like when a resource only allows traffic from a specific vNET), it’s crucial to set up the outbound traffic routing correctly. By default, FunctionApp routes traffic via the public internet. To redirect egress traffic through the vNET, you need to set the WEBSITE_VNET_ROUTE_ALL configuration parameter or the AppService setting vnetRouteAllEnabled to true.
The tool can aid in troubleshooting by showing the vNET assigned to the FunctionApp, the IP address assigned to the FunctionApp within that vNET, and the public IP address from which the outbound request was actually sent by the FunctionApp to an external service Ipify.org.
Wrong time zone (including CRON execution)
If your code depends on local time, like running daily tasks at a specific time or performing time-based operations, setting the timezone for your FunctionApp runtime is essential. To specify a timezone for datetime operations, you need to configure WEBSITE_TIME_ZONE parameter. It’s important to note that the format for specifying timezones differs between Windows and Linux runtimes. This same principle applies to other services based on AppService, such as Web Apps or API Apps.
The tool will show you the current time with the currently selected timezone, and it will issue a warning if the timezone format doesn’t match what’s expected for the chosen operating system.
Conclusion
I hope you find the tool helpful. With a few tweaks, I believe it could, with a few adjustments, be adapted to support general AppService deployments if necessary.
In case of any issue, please report on GitHub.