Posts on Blog | Vaclav Jirovsky

Running Zone-Redundant ExpressRoute Circuits with External Storage for Azure VMware Solution

Tue, 23 Sep 2025 00:00:00 +0000

You might have landed on this blog post because of the email Microsoft recently sent to affected customers:

What Changed in Azure

For Azure Public IP resources, Microsoft has more options how to deploy from resiliency perspective. In August 2024, Azure announced that any newly created Public IP will now be zone-redundant by default.

How It Relates to Azure VMware Solution (AVS)

Azure VMware Solution offers two models of integration with a customer’s private Azure virtual network:

Direct virtual network integration – an AVS SDDC is directly available from virtual network. Available only with the latest AVS generation SKU, AV64 Gen 2
ExpressRoute Circuit integration – required for all other SKUs. This establishes a private connection (similar to a “VPN”) between the virtual network and the AVS SDDC via internal ExpressRoute circuit. This circuit itself is provided at no additional cost (included in the AVS pricing) and is not metered.

How an ExpressRoute Circuit Is Deployed

This post won’t go into the full details of deploying an ExpressRoute (you can find that in the official documentation). Instead, let’s look at the high-level steps:

To connect an AVS SDDC to your virtual network, the first step is to prepare a virtual network and create a dedicated subnet called GatewaySubnet.
This subnet is then used by another Azure resource: Azure Virtual Network Gateway. This resource enables secure connectivity between different networks - such as other Azure virtual networks, on-premises environments, or even AWS VPC.

If you’ve worked with the Azure VPN Gateway, that’s essentially a specific type of Virtual Network Gateway.
With the Virtual Network Gateway in place, you can connect multiple networks to a single Azure virtual network.
The gateway requires an associated Azure Public IP address resource. This assignment gives the gateway an external presence, enabling connections to be established between the source network and the target Azure virtual network - it’s used for control/management plane. This Public IP is used for control/management plane, but it’s not used for traffic itself - but if that IP fails, the gateway resource can fail due to not available control/management plane.
Once the connection is established, both networks become routable with each other via theirs private IP addresses.

graph LR
%% Styles
classDef azure fill:#eef6ff,stroke:#4f7dbd,stroke-width:1px;
classDef comp fill:#ffffff,stroke:#7e8aa2,stroke-width:1px;
classDef mgmt fill:#fef9e7,stroke:#d4ac0d,stroke-width:1px,color:#000000;
%% Azure VNet + Gateway
subgraph AZURE_VNET["🟦 Azure Virtual Network (vNET)"]
direction TB
subgraph GWSUB["🧩 GatewaySubnet"]
direction TB
VNG["🔗 Virtual Network Gateway"]
end
end
class AZURE_VNET,GWSUB azure
class VNG comp
%% External connections (left side)
AVS["☁️ Azure VMware Solution (SDDC)"]
ONPREM["🏢 On-Premises Network"]
PIP["🌐 Public IP (control-plane)"]
class AVS,ONPREM comp
class PIP mgmt
%% Data plane
AVS --- VNG
ONPREM --- VNG
%% Control plane dependency
PIP --- VNG

AVS Networking with External Storage

A typical deployment that uses external storage - such as Pure Cloud Block Store or Pure Storage Cloud Azure Native - looks like this:

graph LR
%% Styles
classDef azure fill:#eef6ff,stroke:#4f7dbd,stroke-width:1px;
classDef comp fill:#ffffff,stroke:#7e8aa2,stroke-width:1px;
classDef pure fill:#F04E23,stroke:#C43E19,stroke-width:1px,color:#ffffff;
classDef mgmt fill:#fef9e7,stroke:#d4ac0d,stroke-width:1px,color:#000000;
%% Azure VNet + Gateway
subgraph AZURE_VNET["🟦 Azure Virtual Network"]
direction TB
PURE["🛢️ Pure Storage Cloud (external storage)"]
subgraph GWSUB["🧩 GatewaySubnet"]
direction TB
VNG["🔗 Virtual Network Gateway"]
end
end
class AZURE_VNET,GWSUB azure
class VNG comp
class PURE pure
%% External connections (left side)
AVS["☁️ Azure VMware Solution (SDDC)"]
ONPREM["🏢 On-Premises Network"]
PIP["🌐 Public IP (control-plane)"]
class AVS,ONPREM comp
class PIP mgmt
%% Data plane
AVS --- VNG
ONPREM --- VNG
VNG --- PURE
%% Control plane dependency
PIP --- VNG

Impacts on Resiliency

Now we get to the real topic. We are in the cloud environment and everything still comes back to the fundamental building blocks - availability zones (AZs).

When you deploy an AVS resource, you must choose the availability zone where the AVS SDDC will reside. AVS also supports deploying nodes across multiple zones (using the AVS Stretched Cluster feature). However, in this configuration any external storage solution is not supported.

Because of that limitation, let’s design a Zonal deployment (and repeat in another AZ to spread across AZs) and the best practice is to place your storage in the same AZ as your AVS deployment. Otherwise, you increase the risk of downtime:

If you deploy storage in a different AZ and the AVS’s zone goes down → your workloads are down.

If the storage’s zone goes down → your workloads are also down.

By keeping AVS and storage in the same AZ, you reduce the probability of a cross-AZ dependency failure. That’s why I strongly recommend explicitly specifying the AZ for all related resources and deploying them consistently in the same zone.

Networking

So far, we haven’t looked at the deployment architecture from a resiliency perspective.

Virtual network gateway (ExpressRoute)

Azure offers multiple ExpressRoute gateway SKUs - Standard, High Performance, Ultra Performance, and the AZ-enabled SKUs (ErGw1AZ / ErGw2AZ / ErGw3AZ).
The AZ SKUs are zone-redundant; they deploy gateway instances across availability zones and are designed to survive a single-AZ failure. Non-AZ SKUs are regional and don’t provide zone-level redundancy.

At Pure Storage, we recommend ErGw3AZ SKU for AVS environments that rely on external storage.
It delivers the highest capacity in the AZ family and supports ExpressRoute FastPath, which lets data bypass the gateway for lower latency and higher throughput.

The last puzzle piece: the Public IP

The Public IP attached to the ExpressRoute gateway is what is going on here.

By default, any Public IP was defaultly deployed as a multi-zonal - in all AZs. But this deployment creates distinct per-zone IP instances and the gateway binds to one of them at startup; it does not failover to the sibling IPs in other zones during an outage and might make your connection to an external storage unavailable.

To address this, Microsoft introduced the Zone-redundant Public IP type. This is a single logical IP that automatically spans all zones. With it, the ExpressRoute gateway remains reachable even if one AZ fails - fully aligning with the zone-redundant design of the AZ-enabled (such as ErGw3AZ) SKUs.

Impact to New AVS Deployments

As announced by Microsoft, Azure has changed the default behavior for new deployments.
When you deploy a new Virtual Network Gateway, the associated Public IP will now default to zone-redundant mode.

In this configuration, the Public IP is not shown as a separate resource in your subscription—it is created as a hidden, Microsoft-managed resource. This ensures resiliency by default.

No additional action is required for new AVS deployments.

Impact to Existing AVS Deployments

For existing environments, it is important to validate the resiliency of your AVS external storage setup to avoid potential outages.

Also verify, that the gateway’s Public IP is not a Basic SKU.

Inherited Basic IPs are a legacy artifact from older deployments.
The Basic SKU has been discontinued, and migration to a Standard SKU is required by the end of September 2025.

Microsoft has prepared a guide on how migrate the associated Virtual Network Gateway and its associated Public IP. More details can be found in this KB. Please be sure you make this change during a maintenance window, since it might introduce couple minutes of downtime.

Enterprise-Grade Block Storage with Portworx CSI and Cloud Block Store

Sun, 15 Jun 2025 00:00:00 +0000

Background

Portworx by Pure Storage has introduced a new version of its Kubernetes CSI driver (PX-CSI), designed to simplify how Kubernetes clusters consume persistent volumes. Version 25.4.0 brings official support for Pure Cloud Block Store, enabling enterprise-grade block storage for a new frontier in the cloud - Kubernetes clusters such as Azure Kubernetes Service (AKS).

In this post, we’ll take a closer look at how to install PX-CSI on an Azure Kubernetes Service (AKS) cluster and mount a persistent volume into a container, backed by a Pure Cloud Block Store (CBS) array.

Architecture Overview

Before diving into the installation, let’s briefly review the architecture of our solution. PX-CSI is deployed as a set of managed pods on your AKS cluster and integrates with the Pure Cloud Block Store array to provision and attach volumes on demand. In FlashArray Direct Access mode, each Kubernetes PV directly maps to a volume on the array, and the application’s data is written straight to the array. This provides the performance and data services of Pure’s enterprise storage to your containerized apps, without using any intermediate storage pools on the cluster.

In essence, when a PVC is created with the storage class backed by CBS, PX-CSI communicates with the CBS’s management plane (via its REST API) to create a new volume. That volume is then attached over iSCSI to the appropriate AKS worker node and made available to the requesting pod. This architecture allows you to leverage Pure’s deduplication, compression, snapshots, and high availability features for your cloud-native workloads. What you are going to need is an instance of Pure Cloud Block Store, deployed into the same vNET as AKS will be deployed into.

Step 1 - Create an AKS Cluster

PX-CSI requires a Kubernetes cluster with a minimum of 3 worker nodes to maintain internal KVDB quorum, which is used to manage storage metadata and runtime state . It’s also recommended to use sufficiently sized nodes (e.g., 4 vCPU and 16 GB RAM or larger) to accommodate the storage services alongside your application workloads.

When deploying the AKS cluster, make sure to select the Bring your own virtual network option under the Networking tab. This step is crucial because the AKS nodes must be able to communicate with the Pure Cloud Block Store instance’s endpoints — including the management interface and iSCSI target IPs.

Be sure that no Network Security Groups (NSGs) are in place that could block this communication. For optimal performance and reliability, is recommended deploying the AKS cluster into the same virtual network (vNET) as the Pure Cloud Block Store instance. This ensures low-latency connectivity and helps avoid unexpected network-related costs.

Step 2 - Connect to the AKS Cluster

With the AKS cluster running and the Portworx spec in hand, the next step is to apply the configuration to the cluster. It’s convenient to use Azure Cloud Shell for this, as it comes pre-loaded with Azure CLI and kubectl. Open the Azure Cloud Shell in your browser (from the Azure Portal or via https://shell.azure.com). Make sure your shell is set to Bash for familiarity.

Authenticate and set Kubeconfig - ensure you’re connected to correct AKS:

1

az aks get-credentials --resource-group <<resource group name>> --name <<AKUS cluster name>> --overwrite-existing

Now you are operating in the context of your AKS cluster.

Step 3 - Create the Configuration File (pure.json)

Portworx PX-CSI requires a JSON configuration file with information about your Pure Cloud Block Store array. This file (named pure.json) will contain the management endpoint and API token for your array, enabling Portworx to authenticate and create volumes via the Pure REST API. Portworx also supports Cloud Block Store multi-tenant setups - however, for the purpose of this post, we’ll use a single-tenant setup.

To proceed, we’ll need to create a configuration file that requires two values. The first is the IP address of the CBS management console, which can be found as the floatingManagementIP output variable in the Managed Application resource, representing your CBS array.

From the array management UI, find the Management Endpoint and generate an API Token for an account with Storage Admin privileges (it can be a new dedicated user for Portworx, e.g. portworx-user).

Generate a new API token under Settings > Users and Policies – create a token that doesn’t expire, or note its expiry if it does.

Now, create a file named pure.json with the following structure (replace the placeholder values with your actual endpoint and token):

1
2
3
4
5
6
7
8


{
 "FlashArrays": [
 {
 "MgmtEndPoint": "<pure-cbs-management-endpoint>",
 "APIToken": "<pure-cbs-api-token>"
 }
 ]
}

With pure.json ready, the next step is to provide it to Portworx by creating a Kubernetes Secret.

Upload the pure.json file to your Cloud Shell environment (you can use the upload button in Cloud Shell or use an SCP command to the Cloud Shell storage). Once the file is present in your Cloud Shell, run the following command to create the secret in your AKS cluster:

1
2
3


kubectl create secret generic px-pure-secret \
 --namespace portworx \
 --from-file=pure.json=./pure.json

This command will create a secret named px-pure-secret in the portworx namespace (we will deploy Portworx in the portworx namespace for clarity).

Info

It is important to use the exact name px-pure-secret, as the Portworx driver looks for a secret by this name during startup.

You should see a confirmation like secret/px-pure-secret created.

The secret contains the management endpoint and token and will be mounted by Portworx pods to authenticate with the array.

Tip

Note: Ensure the secret is created in the same namespace where Portworx will be installed. In our case, we use the portworx namespace (if you plan to deploy Portworx in a different namespace, adjust the --namespace accordingly).

The Portworx Operator will pass this secret to the PX-CSI driver automatically on deployment.

Step 4 - Configure Auto-Enabling of Multipath

Since Azure Kubernetes Service (AKS) does not enable Multipath I/O by default on new nodes, we need to configure it manually.

To automate this process, we’ve prepared a DaemonSet that automatically detects new nodes in the cluster and applies the required configuration changes.

To deploy the DaemonSet, simply apply the following YAML:

1

kubectl apply -f https://raw.githubusercontent.com/PureStorage-OpenConnect/cloudblockstore-scripts/refs/heads/main/CBS-Azure-Solutions/px-csi-driver/enable-aks.yaml

Step 5 - Generate the PX-CSI Installation Spec

Portworx provides a convenient online tool, PX-Central (accessible at https://central.portworx.com), to generate a custom installation manifest for your Kubernetes cluster.

We will use PX-Central to configure our Portworx deployment for our needs:

Log in to PX-Central: Navigate to the PX-Central site and log in with your Pure Storage/Portworx credentials. If you don’t have an account, you can create one on the site.
Select PX-CSI and Version: In the spec generator, choose PX-CSI as the product type. Ensure you select version 25.4.0 or higher for PX-CSI, as version 25.4.0 introduced enhancements and is required for the features we need.
Configure Kubernetes Settings: Set Distribution to “None” (since AKS is a standard Kubernetes environment). Select the Kubernetes version that matches your AKS cluster’s version (e.g., 1.31.8). This ensures the generated manifests are compatible with your cluster’s API version.
Enable iSCSI Support: In the Portworx configuration options, make sure to set Storage Area Network to “iSCSI”.

Generate and Download Commands: After filling in the required settings, PX-Central will generate a set of Kubernetes manifests or commands, providing a Portworx Operator installation manifest and a StorageCluster custom resource (CR) manifest tailored to your config.

You may see two kubectl commands - for example, PX-Central might instruct you to run:

1
2
3


kubectl apply -f 'https://install.portworx.com/……..# Install Portworx Operator

kubectl apply -f https://install.portworx.com/……… # Deploy Portworx StorageCluster (PX-CSI)

These are placeholders – PX-Central will give you actual URLs - please keep them handy, we will need them in the next step. The first command installs the Portworx Operator (which manages the deployment), and the second creates the StorageCluster CR which describes the desired Portworx cluster configuration (including the Pure Cloud Block Store integration).

Step 6 - Install the Portworx Operator and PX-CSI Driver

Now it’s time to deploy Portworx on the cluster. We will apply the manifests obtained from PX-Central to install the Portworx Operator and the Portworx StorageCluster specification. The operator will then pull up all necessary components including the PX-CSI driver.

Apply the Portworx Operator manifest: Use the command or file from the PX-Central output to install the operator.

This will create the Portworx Operator deployment and related roles. You should see resources like serviceaccount/portworx-operator and deployment.apps/portworx-operator get created. The operator is a controller that will watch for a StorageCluster custom resource and deploy Portworx components accordingly.

Apply the Portworx StorageCluster spec: Next, apply the PX-CSI cluster spec YAML (the second command from PX-Central). This command creates a StorageCluster object which describes the desired state of Portworx on your cluster. When you create this object, the Portworx Operator will begin deploying the PX-CSI pods.

You can verify the creation of the StorageCluster CR in the output: e.g., storagecluster.core.libopenstorage.org/px-cluster created.

Once both the operator and the StorageCluster manifests are applied, give the system a few minutes to pull the container images and initialize the pods on all nodes.

Step 7 - Verify the Portworx PX-CSI Installation

After installation, you should verify that Portworx and its CSI components are up and running. Use the following command to check the status of Portworx pods:

1

kubectl get pods -n portworx -o wide

This will list all pods in the portworx namespace (assuming you installed Portworx there). You should see the Portworx Operator pod and the PX-CSI driver pods in a Running state. For example, the output might include entries like:

1
2
3
4
5
6


NAME READY STATUS RESTARTS NODE
portworx-operator-5c6b7bbd4d-abcde 1/1 Running 0 aks-nodepool1-123456-vmss000000
px-csi-ext-controller-0 5/5 Running 0 aks-nodepool1-123456-vmss000000
px-csi-ext-node-57zxp 3/3 Running 0 aks-nodepool1-123456-vmss000001
px-csi-ext-node-nrj7k 3/3 Running 0 aks-nodepool1-123456-vmss000002
px-csi-ext-node-l8tj2 3/3 Running 0 aks-nodepool1-123456-vmss000003

In the above example, the operator is running (1/1 containers ready), a CSI controller stateful set pod is running (with multiple containers for provisioner, attacher, etc.), and a CSI node plugin pod is running on each AKS worker node. The exact pod names may vary, but look for pods prefixed with portworx- or px-csi-. All pods should reach Running status.

If any pod is in CrashLoopBackOff state, use kubectl describe or check logs to troubleshoot (common issues might be missing iSCSI packages on nodes or secret misconfiguration).

At this stage, Portworx PX-CSI has been successfully deployed and is integrated with Pure Cloud Block Store! The driver has likely created some default StorageClasses as well for you to use, including one for Pure FlashArray direct volumes.

Step 8 - Create a Persistent Volume Claim (PVC)

With Portworx running, you can now request persistent storage in your AKS cluster that resides on a CBS. Portworx by Pure provides a set of pre-configured StorageClass objects out-of-the-box for various use cases. The one we will use is px-fa-direct-access, which is the StorageClass for Pure FlashArray Direct Access volumes.

This storage class tells Portworx to provision volumes directly on the Pure Cloud Block Store/FlashArray for each PVC, rather than using Portworx-managed pools. Let’s create a sample Pod with PVC to verify everything end-to-end:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: px-demo-pvc
spec:
 storageClassName: px-fa-direct-access
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 10Ti
---
apiVersion: v1
kind: Pod
metadata:
 name: px-demo-pod
spec:
 volumes:
 - name: px-demo-storage
 persistentVolumeClaim:
 claimName: px-demo-pvc
 containers:
 - name: px-demo-container
 image: nginx
 ports:
 - containerPort: 80
 volumeMounts:
 - mountPath: "/px-storage"
 name: px-demo-storage

This PVC requests a 10 TiB volume using the px-fa-direct-access storage class for the Pod px-demo-pod.

Kubernetes will submit the PVC to the Portworx CSI provisioner. Portworx then communicates with array to create a 10 TiB block volume on the array (thin-provisioned), and attaches that volume via iSCSI to one of the AKS nodes (the node where a pod using this PVC would be scheduled, or immediately if binding is immediate).

At this point, you can consume the PVC in any pod. When we log in into the pod, we can see a mounted volume from CBS into the directory /px-storage:

Congratulations – you have successfully set up Portworx PX-CSI on AKS and created a persistent volume on Pure Cloud Block Store!

Conclusion

Portworx CSI brings together the best of Kubernetes and Pure Storage by allowing AKS administrators to consume Pure Cloud Block Store as a native Kubernetes storage resource. In this tutorial, we created a highly-available AKS environment, configured Portworx’s PX-CSI driver, and demonstrated provisioning a PVC backed by Pure’s enterprise-grade cloud storage.

This integration simplifies cloud deployments by offloading data persistence to CBS, giving you features like data reduction and snapshots out-of-the-box on AKS.

Optimizing Storage: Deploying Cloud SAN in Advanced Azure Landing Zones

Fri, 31 Jan 2025 00:00:00 +0000

This post is follow up to my previous post about pricing changes, introduced on Microsoft Build 2024 and their cost implications for cloud storage solution.

Motivation

The Azure Cloud Adoption Framework (CAF) is Microsoft’s proven methodology for guiding organizations through their cloud adoption journey. It provides a structured approach to designing, implementing, and optimizing Azure environments based on industry best practices. CAF helps businesses align their cloud strategy with operational, security, and compliance requirements, ensuring a scalable and well-governed foundation.

Native cloud storage solutions often struggle to meet the high performance, low latency, and resilience demands of enterprise workloads. SAN solutions in the cloud address these challenges by providing highly available, scalable, and feature-rich block storage, similar to on-premises SAN architectures. SAN solutions like Pure Cloud Block Store enable seamless replication, disaster recovery, and multi-cloud mobility, making them ideal for mission-critical applications, databases, and virtualized environments that require consistent performance and data integrity.

Challenges

With adopting a SAN solution into any advanced Azure landing zone (inspired by CAF) comes few challenges that requires to be addressed.

Hidden Costs for the Data Path – Cloud networking and storage transactions can introduce unforeseen costs, especially when data is transferred across regions or via multiple peered vNETs (see more details). Understanding these costs is critical for FinOps efficiency.

No TLS Inspection or Additional Latency – Security policies enforcing TLS decryption can impact SAN replication traffic, potentially introducing performance overhead and requiring additional network configurations. Wrong design can disable to perform a TLS inspection, or in other cases bring an additional latency and lower storage performance.

Network Segregation – Proper network isolation and segmentation are crucial for security and compliance, but on the other hand a SAN solution might become a single point of failure from this perspective.

Deployment Architecture Pattern for a Shared SAN Solution

In this post, I’ll demonstrate this pattern using Pure Cloud Block Store (CBS), a SAN block storage solution designed for both Azure and AWS.

Info

Please note that the best performance and minimum costs for cloud SAN storage solutions are achieved within a single availability zone in a single virtual network deployments.

This post describes a deployments in cases where it is necessary to share a single SAN instance between multiple workloads deployed across multiple virtual networks, while adhering to CAF’s design principles and leveraging cloud benefits.

Key Features

Optimal SAN performance: The data path is optimized, ensuring no additional latency or performance degradation is introduced
Security isolation: Storage access is strictly limited to host VMs, preventing host VMs from communicating with each other
Support for TLS Inspection of traffic: All standard traffic from host VMs can be a subject to TLS inspection. The only exception is the CBS data path, which is routed directly for performance optimization
FinOps optimization: This architecture pattern is designed to minimize unnecessary vNET peering configurations, minimizing hidden costs associated with inter-vNET data transfer.

Diagram

Components

SAN solution (Pure Cloud Block Store in our example)
Virtual network (vNET)
Network interface (NIC)
Azure Firewall
Network Security Group (NSG)
Azure VM

Data path(s)

iSCSI Data Path

Each Host VM is in dual-NIC setup - primary and secondary NICs. A primary NIC is dedicated for standard traffic, routed to Azure Firewall and a secondary NIC is dedicated for iSCSI communication for block storage access.

Host VMs connect directly to CBS over iSCSI via a vNET Peering between host VM vNET and CBS vNET.

Replication Data Path (Between Multiple CBS Instances)

CBS supports various replication types between storage arrays, ranging from ActiveCluster (synchronous) to multiple asynchronous options such as ActiveDR and CloudSnap. These replication processes utilize dedicated network interfaces to ensure efficient data transfer.

In this setup, CBS instances replicate to each other directly via vNET Peering between the CBS #1 vNET and the CBS #2 vNET.

Details

For following examples, consider following IP ranges (IP ranges are selected to make this example easier to understand, you don’t need to follow them):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


- CBS #1 vNET: 192.168.96.0/21
 |- subnet system: 192.168.100.0/24
 |- subnet iscsi: 192.168.101.0/24
 |- subnet mngmt: 192.168.102.0/24
 |- subnet replication: 192.168.103.0/24

- Spoke #1 vNET: 192.168.1.0/22
 |- subnet default: 192.168.1.0/24
 |- subnet cbs-nics: 192.168.2.0/24

- Hub vNET: 10.0.0.0/16
 |- subnet AzureFirewallSubnet: 10.0.1.0/26

- CBS #2 vNET: 192.168.200.0/21
 |- subnet system: 192.168.200.0/24
 |- subnet iscsi: 192.168.201.0/24
 |- subnet mngmt: 192.168.202.0/24
 |- subnet replication: 192.168.203.0/24

NSG configurations

To accomplish all the needs described above, you need to configure the following Network Security Groups:

CBS #1 vNET - iSCSI subnet

Purpose: This NSG allows iSCSI traffic from NICs in the subnet only and blocks any further communication from other vNET/subnets.

1
2
3
4
5
6
7
8


Inbound:
Allow 3260 from <<host VM IPs>> to Any
Deny all


Outbound:
Allow 3260 from Any to <<host VM IPs>>
Deny all

Example:

Tip

For a port configuration of inbound security rules, configure the Destination port range only and keep Source port ranges as wildcards.

Spoke #1 vNET - NSG for VM NICs within default subnet

This NSG is for testing purposes, to make sure the iSCSI traffic to CBS will not be routed via the incorrect interface.

1
2
3


Outbound - add following rule:
Deny 3260 from Any to <<CBS iSCSI subnet range>>
...

Spoke #1 vNET - NSG for VM NICs within cbs-nics subnet

Thanks to this NSG, the iSCSI traffic can be routed via this secondary NIC and communicate with CBS, but can’t communicate with each other.

1
2
3
4
5
6
7


Inbound Security Rules:
Allow 3260 from <<CBS iSCSI subnet range>> to Any
Deny all

Outbound Security Rules:
Allow 3260 from Any to <<CBS iSCSI subnet range>>
Deny all

Example:

Azure Firewall configuration

All non-datapath traffic from CBS (system or management interface) can be routed through Azure Firewall or other type of virtual appliance.

Because there might be a SSL pinning in place, in the following example, we proceed with an Application rule allowing traffic to FQDN *.purestorage.com and management.azure.com without TLS inspection on Port 443 (HTTPS).

Routing configurations

Forced Tunneling

Routing to Azure Firewall is achieved via routing table, with a User-Defined Route (UDR) 0.0.0.0/0 with next hop to Azure Firewall (in our example 10.0.1.4).

CBS vNET

For achieving the forced tunneling to Azure Firewall is this routing table attached to the system and management subnets. Traffic from these CBS interfaces will be routed via Azure Firewall.

Host VM vNET

For achieving the forced tunneling to Azure Firewall is this routing table attached to the default subnet. Standard traffic from the Host VM (except for iSCSI data path) will be routed via Azure Firewall. iSCSI path will be connected directly to the CBS iSCSI subnet via vNET peering.

Host VM Routing Table

To ensure the operation system on the Host VM will know when to use secondary NIC for iSCSI traffic, you need to configure a route to CBS iSCSI interface.

Windows

Before configuring a route, we need to identify the ID of the network interface within the cbs-nics subnet.

Following commands provide information about host VMs network interfaces:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


PS> ipconfig
...
Ethernet adapter Ethernet 3:

 Connection-specific DNS Suffix . :
 Link-local IPv6 Address . . . . . : fe80::5fe4:1cd8:2983:16ad%11
 IPv4 Address. . . . . . . . . . . : 192.168.2.5
 Subnet Mask . . . . . . . . . . . : 255.255.255.0
 Default Gateway . . . . . . . . . :

...


PS> netsh interface ipv4 show interfaces

Idx Met MTU State Name
--- ---------- ---------- ------------ ---------------------------
 1 75 4294967295 connected Loopback Pseudo-Interface 1
 4 10 1500 connected Ethernet
 **11** 10 1500 connected Ethernet 3

To configure a persistent route, use following command:

1

route -p add <<IP range of subnet iscsi>> MASK 255.255.255.0 <<gateway IP of subnet cbs-nics>> METRIC 5015 IF <<network interface Idx>>

Tip

As IP address of subnet gateway use IP address: x.x.x.1

Example:

1

PS> route -p add 192.168.101.0 MASK 255.255.255.0 192.168.2.1 METRIC 5015 IF 11

Linux

Firstly, we need to identify the ID of the network interface within the cbs-nics subnet.

Following commands provide information about host VMs network interfaces:

1
2
3
4
5
6
7
8


$ ip addr show

3: **eth1**: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
 link/ether 7c:1e:52:5d:6d:9c brd ff:ff:ff:ff:ff:ff
 inet 192.168.2.4/24 metric 200 brd 192.168.2.255 scope global eth1
 valid_lft forever preferred_lft forever
 inet6 fe80::7e1e:52ff:fe5d:6d9c/64 scope link
 valid_lft forever preferred_lft forever

To configure a persistent route, use following command:

1

$ route add -net <<IP range of subnet iscsi>> netmask 255.255.255.0 gw <<gateway IP of subnet cbs-nics>> dev <<ntw interface>> metric 5015

Tip

As IP address of subnet gateway use IP address: x.x.x.1

Example:

1

$route add -net 192.168.101.0 netmask 255.255.255.0 gw 192.168.2.1 dev eth1 metric 5015

iSCSI target configuration

For connecting CBS as an iSCSI target, make sure you select the right network interface in subnet cbs-nics to be used for ISCSI initiator.

In our example it’s 192.168.2.5:

For Windows - Powershell option:

1
2
3
4
5
6
7


PS> New-IscsiTargetPortal -TargetPortalAddress 192.168.101.5 -InitiatorPortalAddress 192.168.2.5 -TargetPortalPortNumber 3260

#ct0
Get-IscsiTarget | Connect-IscsiTarget -InitiatorPortalAddress 192.168.2.5 -IsMultipathEnabled $true -IsPersistent $true -TargetPortalAddress 192.168.101.5

#ct1
Get-IscsiTarget | Connect-IscsiTarget -InitiatorPortalAddress 192.168.2.5 -IsMultipathEnabled $true -IsPersistent $true -TargetPortalAddress 192.168.101.8

Or in Windows iSCSI wizard:

May 2024 Azure Updates: Cost Implications for Cloud Storage

Fri, 31 May 2024 00:00:00 +0000

When discussing cloud storage costs, network expenses are often a significant hidden factor. Data transfer between storage and other services can lead to unexpected charges, especially as usage scales. During the Build 2024 event, Microsoft silently published a small but important update to networking costs. This article aims to summarize how networking costs might impact cloud storage in Azure and what to consider during design from a FinOps perspective.

Any costs in this article roughly estimate costs related to network data transfer only and don’t reflect any individual customer agreements. For precise estimates, I recommend checking the latest official pricing, contacting your Microsoft representative, or evaluating a limited PoC to prevent any incurred costs.

Single-AZ transfers within a single vNET

Any data transfers between VMs within a single vNET in a single availability zone (AZ) have been free, and there are no announced plans to start charging for them.

Cross-AZ transfers within a single vNET

To improve your workload resiliency, you might want to scale out underlying resources across multiple availability zones. This architecture reduces the probability of downtime, so when one AZ experiences an outage, the other remains operational.

A bit of history: In 2021, Microsoft announced plans to start billing for data transfers across availability zones (AZs) in the future. Microsoft noted that they would announce the launch date 3 months in advance. This plan remained in place for the past 3 years, until April 2024, when Microsoft began to implement these charges. Additionally, the Microsoft Learn/Azure pricing documentation has been inconsistent, and only the page about Bandwidth pricing mentioned this cost.

With this cost in place, any data transfer between two AZs, even within a single vNET in the same region, is charged per transferred GB.

At end of May, during the Microsoft Build conference, Microsoft has announced in a small blog post that they are removing the cost, and so far, it appears they do not plan to reintroduce it.

Impact to Cloud Storage

When I talk about Cloud Storage in this context, I am referring to TBs of data that are frequently accessed by an application. Due to missing important storage features (thin-provisioning, data deduplication, snapshots, etc.) on native disks services, this type of storage typically leverages a smarter, SAN solution, such as Pure Cloud Block Store™ (CBS).

SAN solutions are typically AZ-specific, meaning the underlying resources are deployed within a single AZ. This architecture reduces storage latency and improves performance.

For workloads deployed in a single AZ, the best practice is to co-locate a SAN in the same AZ as the workload. Deploying into a different AZ might lead to slower performance and violates the main concept of isolated AZs for resiliency. It increases the probability of downtime because if either the application’s AZ or the storage’s AZ experiences an outage, it leads to a solution downtime. Additionally, such an architecture would also bring additional charges to your invoice due to cross-AZ communication charges.

For more important workloads deployed across multiple AZs, you may deploy 2 CBS arrays (into different AZs within same vNET) and leverage a replication between them.

ActiveDR

One of the replication types available in CBS is an asynchronous near-zero RPO, called ActiveDR.

A big advantage of this replication is that only deduplicated and compressed data are transferred, not the raw data. After Microsoft removed the cross-AZ cost, this has no impact on the bill anymore, but it significantly reduces the time required to transfer the “same amount” of data. This will have a significant impact in the next FinOps tip below.

ActiveCluster

Another type of replication between two CBS instances which can be used here is called ActiveCluster™. This setup provides active-active synchronous replication with zero RPO and RTO, where each CBS instance primary serves applications within its respective AZ.

Please note, that in ActiveCluster replication the benefit for replicating only deduplicated and compressed data is not applicable, since every write needs to be done on both devices synchronously.

Multiple vNETs, Single/Cross-AZ Communication

All the scenarios above discussed having all resources within a single vNET.

When a resource is located in a different vNET and accessed via vNET peering/NVA (regardless of whether it’s in the same or a different AZ), another networking cost applies—for data transferred via vNET peering. The vNET peering service itself is free, but ingress and egress traffic is charged at both ends of the peered networks.

VNET Peering within the same region:

Transfer	Price
Inbound data	$0.01 per GB
Outbound data	$0.01 per GB

vNET peering within the same region is charged at $0.02 per GB ($0.01 for egress from the first vNET and $0.01 for ingress to the second vNET), regardless of whether the data physically stays within the same AZ.

Following best practices for creating an Azure landing zone in an enterprise environment typically results in a more complex network topology. The most common is the Hub and Spoke topology, where multiple Spoke vNETs (containing various workloads) are connected to a single Hub vNET, which provides common connectivity to on-premises, security appliances, and more, via vNET peering.

With this topology, try to limit the number of hops as much as possible—each vNET peering in the path incurs charges for both egress and ingress.

For communication between two Spoke vNETs (e.g. a VM consuming storage from SAN in another vNET), consider co-locating these resources into a single vNET or creating a direct vNET peering between these vNETs.

sequenceDiagram
participant vNET 1
participant Hub vNET
participant vNET 2
vNET 1->> Hub vNET: egress from vNET 1 ($0.01) + ingress to Hub vNET ($0.01) = $0.02
Hub vNET ->> vNET 2: egress from Hub vNET ($0.01) + ingress to vNET 2 ($0.01) = $0.02

In total this architecture costs $0.04 per GB.

Model scenario no.1 - transferring 100 TB data, same-region, cross-AZ:

Method	Total
Within single vNET	~~$2,000~~ $0 after April 2024
Hub&spoke (routed via Hub, 2x vNET peerings)	$4,000
Hub&spoke (direct spoke-spoke vNET peering)	$2,000

Model scenario no.2 - transferring 100 TB data, same-region single-AZ:

Method	Total
Within single vNET	$0
Hub&spoke (routed via Hub, 2x vNET peerings)	$4,000
Hub&spoke (direct spoke-spoke vNET peering)	$2,000

AWS difference

There is a significant difference in networking costs between Azure and AWS. AWS does not charge for data transfer between peered VPCs if the source and destination remain within the same availability zone. All variants in Scenario no.2 would not be charged.

Private Link

At the end of this article, I would like to discuss the option of exposing cloud storage via Private Link. Despite its various benefits, Private Endpoints incur additional costs, priced at $0.01 per hour for each Private Endpoint. Additionally, there are costs associated with processing inbound and outbound data transferred via Private Endpoints, and the data transfer charges mentioned earlier still apply. For large data transfers it might be a significant hidden cost.

Playing with GitHub Copilot: Interactive Documentation by RAG

Thu, 28 Mar 2024 00:00:00 +0000

Intro

Disclaimer: Just a heads up, I’m not an AI Expert, I’m ordinary App Innovation architect, just trying to get AI to lend me a hand with my projects. If I slip up here and there, my bad—I’m sort of navigating uncharted waters for me.

While experimenting with GitHub Copilot, I encountered frustration due to its limited or absent knowledge on my specific domain (or product), despite clearly specifying the topic, GitHub Copilot were unable to load them, and with just only basic or no knowledge about the topic tried to “invent” some answers, which were totally hallucinating.

My conversation with GitHub Copilot:

vjirovsky	GitHub Copilot
“create me a Bicep template for Pure Cloud Block Store deployment”
	Sure, here’s a basic example of a Bicep template for deploying Pure Cloud Block Store on Azure. This template creates a virtual network, a subnet, and a network security group with necessary rules. Please note that this is a simplified example and you may need to adjust it according to your specific requirements. `…non-sense Bicep code (truncated)…` Please note that this template does not actually deploy the Pure Cloud Block Store instance because as of now, there is no Azure Resource Manager (ARM) or Bicep support for Pure Cloud Block Store. You would need to use the Pure Storage console or API to deploy the Cloud Block Store instance.
“create me a Bicep template for Pure Cloud Block Store deployment, use guidance here (link)”
	I’m sorry for any confusion, but as an AI, I don’t have the ability to access external URLs or content outside of the information provided in the conversation. This includes the URL you’ve provided.

I understand that GitHub Copilot cannot encompass all knowledge, but what surprised me was its inability to access any supportive resources available online, even when public URLs to documentation are provided. My assumption is that such online access is restricted due to security/license reasons, preventing Copilot from loading external content, at least directly into my session.

Concept Overview

The idea is pretty straightforward - using Retrieval-Augmented Generation (RAG) prompt engineering: it’s about loading every prompt with comprehensive knowledge of my domain or product, akin to filling a suitcase with all essentials - as on Figure 1, a component called ‘Enterprise Knowledge Base’ in the diagram.

A Knowledge Base, especially for tools like GitHub Copilot, can be any form of text document—ranging from source code files to simple text files. In my experiment, I opted for a Markdown formatted document. I chose Markdown because of its structured yet straightforward syntax, which is easily parseable for large language models (LLMs), you can include there also a source code blocks and it’s user-friendly for authors.

Example of helper file

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16



CBS means Pure Cloud Block Store

## Prerequisites before CBS deployment
A CBS requires for deployment following components to be deployed before CBS (headlines 3 in this section):

### Logged user's permissions
A logged user, who deploys a CBS, needs to have a permission Managed Application Contributor on the selected subscription and on the selected subnet(s) for CBS deployment, at the minimum.

### Region / AZ support
See this list of supported regions and Availability Zones to confirm CBS availability in your region and selected Availability zone.

The up-to-date list is always available on URL: https://support.purestorage.com/Pure_Cloud_Block_Store/Pure_Cloud_Block_Store_on_Azure_-__Regions_Support

If a user asks about deployment into any concrete region or AZ, point them to the URL.
...

Using in GitHub Copilot

I have created a single helper file called cbs-azure-copilot.md within which I compiled essential details about the Pure Cloud Block Store product. This included guidance on deployment, best practices, and a selection of deployment examples.

The GitHub Copilot plugin in IDEs, such as Visual Studio Code, can reference a file in a prompt, but it needs to be done the right way. One key limitation is that the file must either be open or part of the repository you’ve selected.

At this moment, you may reference it with special command #file: <<filename>>:

When using a prompt from this article or similar sources, ensure that the #file: command is recognized correctly. From my experience, you may need to manually type out this portion, as the IDE plugin does not support copying and pasting the entire prompt.

Also, to maintain context from the helper file, I recommend repeating its reference in every prompt to ensure Copilot stays informed.

This #file command offers a notable advantage: it enables the simultaneous referencing of a helper file along with an open file containing source code within a single prompt. This feature can be particularly useful for receiving proposal changes from GitHub Copilot, informed by the contents of the helper file.

See example conversation below:

vjirovsky	GitHub Copilot
“please fix my code (something is wrong around Managed App publisher), use helper #file:cbs-azure-copilot.md”
	used reference main.bicep, lines 1-48 Based on the helper file, the publisher ID for ARM deployments of Managed Applications should be `purestoragemarketplaceadmin`.

Usage in GitHub Copilot Enterprise

In the Enterprise version of GitHub Copilot, you have the ability to set it up so that it can include references to your private data (such as our helper file) in the responses generated for all users within your organization. This functionality offers a straightforward way to inject new knowledge into Copilot or to steer its output in a particular direction.

For example, it could be utilized to embed corporate standards directly into the tool’s suggestions. Imagine ensuring that every Bicep template suggested by Copilot automatically aligns more closely with your company’s security or tagging standards.

Additionally, it could serve to standardize the way frequently used content is structured and organized according to standard within your organization.

Conclusion

In summary, using a helper file like cbs-azure-copilot.md in GitHub Copilot offers an easy and effective way for making Copilot even more useful in your coding projects by aligning it with your specific needs and standards.

For those interested, you can find the CBS helper file on GitHub, providing a practical example of how to leverage this approach.

Resources

Demystifying Azure Private Link service

Sat, 03 Feb 2024 00:00:00 +0000

Personally, I find Azure Private Link Service incredibly valuable and beneficial. However, I recognize that it can be confusing at first glance, especially when it’s not actively used and becomes a bit fuzzy in memory. That’s precisely why I’m writing this article – to serve as a reference for future instances when I, and others like me, encounter this challenge.

Introduction

Long story short, Azure Private Link enables one-directional access to resources via a virtual network (vNET) privately. A resource published via Private Link (and its associated services) into a vNET cannot access other resources within the vNET. However, other resources within the vNET can access the published resource. A resource can be any Azure PaaS service (such as Azure KeyVault or Azure CosmosDB, a list of supported services can be found here) or any your workload represented by Azure Load Balancer. The traffic goes via the Microsoft backbone network.

Private Link resources in detail

Azure Private Link Service resource

This resource is specifically designed for publishing a Load Balancer via Private Link. It’s not getting involved in the setup of Azure native PaaS service, just for Load Balancer. In essence, it’s associated with the Frontend IP of the chosen Load Balancer. Each Private Link Service can only be assigned to a single Load Balancer Frontend IP - a Standard Load Balancer can host up to 8 Private Link Services. Private Link Service is a standard Azure resource that requires deployment within a resource group and location.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


param privatelinkServiceName string
param location string
param loadBalancerFrontEndIpConfigurationResourceId string
param loadbalancerId string


resource privatelinkService 'Microsoft.Network/privateLinkServices@2021-05-01' = {
 name: privatelinkServiceName
 location: location
 properties: {
 enableProxyProtocol: false
 loadBalancerFrontendIpConfigurations: [
 {
 id: loadBalancerFrontEndIpConfigurationResourceId
 }
 ]
 ipConfigurations: [
 {
 name: 'snet-provider-default-1'
 properties: {
 privateIPAllocationMethod: 'Dynamic'
 privateIPAddressVersion: 'IPv4'
 subnet: {
 id: reference(loadbalancerId, '2019-06-01').frontendIPConfigurations[0].properties.subnet.id
 }
 primary: false
 }
 }
 ]
 }
}

output id string = privatelinkService.id

Azure Private Endpoint resource

Private Endpoint is a critical component in both approaches, whether for Azure PaaS services or for Load Balancers, and it must be present at all times. It serves as an instance (interface) of the published resource within a virtual network (vNET).

A Private Endpoint has an associated resource, namely the Azure Network Interface (NIC) resource. This NIC allocates an IP address within a virtual network (vNET).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


param keyvaultPrivateEndpointName string
param location string
param keyVaultResourceId string
param subnetResourceId string


resource keyVaultPrivateEndpoint 'Microsoft.Network/privateEndpoints@2022-01-01' = {
 name: keyvaultPrivateEndpointName
 location: location
 properties: {
 privateLinkServiceConnections: [
 {
 name: keyvaultPrivateEndpointName
 properties: {
 groupIds: [
 'vault'
 ]
 privateLinkServiceId: keyVaultResourceId
 }
 }
 ]
 subnet: {
 id: subnetResourceId
 }
 }
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43


param privateEndpointName string
param location string
param vnetName string
param subnetName string
param privateLinkServiceResourceId string


resource privateEndpoint 'Microsoft.Network/privateEndpoints@2023-06-01' = {
 name: privateEndpointName
 location: location
 properties: {
 subnet: {
 id: resourceId('Microsoft.Network/virtualNetworks/subnets', vnetName, subnetName)
 }
 privateLinkServiceConnections: [
 {
 name: privateEndpointName
 properties: {
 privateLinkServiceId: privateLinkServiceResourceId
 }
 }
 ]
 }
}

module networkInterface '_nic-nested.bicep' = {
 name: 'nested'
 params: {
 nicName: last(split(privateEndpoint.properties.networkInterfaces[0].id, '/'))
 }
}

output ipAddressInVnet string = networkInterface.outputs.ip


// _nic-nested.bicep file:
param nicName string

resource networkInterface 'Microsoft.Network/networkInterfaces@2021-08-01' existing = {
 name: nicName
}

output ip string = networkInterface.properties.ipConfigurations[0].properties.privateIPAddress

Why To Care about DNS

The seamless utilization of an Azure PaaS resource via Private Link, without any additional configuration, relies on correct DNS configuration.

Let’s show it on example - when you deploy a Blob Storage, it gets a hostname like: <<something>>.blob.core.windows.net.

Whenever you access the hostname myblobstorage.blob.core.windows.net, it automatically resolves to the assigned public IP address of the Blob Storage service. However, with Private Endpoint, a Network Interface (NIC) representing the Blob Storage is deployed within your vNET, allocated with a private IP address from your vNET range. Therefore, you need to access the Blob Storage using this private IP.

Consider a scenario where you have a Blob Storage with the hostname myblobstorage.blob.core.windows.net, and public access is blocked. If you perform an HTTPS call for https://myblobstorage.blob.core.windows.net/ from a resource within the vNET:

In a standard setup, the hostname myblobstorage.blob.core.windows.net would be translated by a public DNS server to the associated Azure public IP address (e.g., 20.60.181.xxx) and routed via the vNET’s configured internet routing method. However, since the request appears to arrive from the “public internet” rather than the vNET, the Blob Storage service would refuse it due to disabled public access.

1
2
3
4
5
6
7
8


> nslookup myblobstorage.blob.core.windows.net

Server: 168.63.129.16

Non-authoritative answer:
Name: blob.xxxxxx.store.core.windows.net
Addresses:
 20.60.181.xxx

Alternatively, if you wish to use the internal IP address directly, you could perform an HTTPS call for https://10.0.1.2/ (the private IP address within the vNET). In this case, TLS certificate validation comes in the game. The request would be routed to the Blob Storage server correctly, and the server would respond with a TLS certificate issued for myblobstorage.blob.core.windows.net hostname. However, the web client initiating the request will validate the certificate and encounter a TLS error regarding the mismatching certificate (as the certificate is not issued for 10.0.1.2). While turning off certificate validation could be an option, it introduces vulnerability to Man-in-the-Middle (MiTM) attacks, which is not advisable.

To tackle this challenge, the Private Link concept relies on “overwriting DNS records” within the vNET. For this purpose, Azure provides a built-in service called Azure Private DNS Zone. This service allows hosting specific DNS zones, while forwarding the rest of the requests for resolutions to standard DNS servers.

When your vNET utilizes default Azure DNS resolvers, you simply need to create the Azure Private DNS Zone for the selected PaaS service. Corresponding records for the PaaS resource should be configured within this private DNS zone. As a result, any DNS resolution requests originating from the selected vNET will automatically be overwritten to use Private Link’s IP records.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


> nslookup myblobstorage.blob.core.windows.net

Server: 168.63.129.16

Non-authoritative answer:
Name: blob.xxxxxx.store.core.windows.net
Addresses:
 10.0.1.2
Aliases:
 myblobstorage.privatelink.store.core.windows.net

In case your vNET utilizes a custom DNS resolvers, you may set up a forwarder within a Azure vNET, integrated with Azure Private DNS Zone and you may configure your environment to even resolve resources via Private Link from on-premises. More details about Azure DNS Private Resolver can be found here.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


param privateDnsZoneName string
param keyVaultResourceId string
param virtualNetworkResourceId string

resource keyVaultPrivateDnsZone 'Microsoft.Network/privateDnsZones@2020-06-01' = {
 name: privateDnsZoneName
 location: 'global'
 properties: {}
}

resource privateEndpointDnsGroup 'Microsoft.Network/privateEndpoints/privateDnsZoneGroups@2022-01-01' = {
 name: 'vault-PrivateDnsZoneGroup'
 parent: keyVaultPrivateEndpoint
 properties: {
 privateDnsZoneConfigs: [
 {
 name: privateDnsZoneName
 properties: {
 privateDnsZoneId: keyVaultPrivateDnsZone.id
 }
 }
 ]
 }
}

resource keyVaultPrivateDnsZoneVnetLink 'Microsoft.Network/privateDnsZones/virtualNetworkLinks@2020-06-01' = {
 name: uniqueString(keyVaultResourceId)
 parent: keyVaultPrivateDnsZone
 location: 'global'
 properties: {
 registrationEnabled: false
 virtualNetwork: {
 id: virtualNetworkResourceId
 }
 }
}

Key Takeaways

The Vital Role of vNET

Without a virtual network (vNET), consuming a resource via Private Link isn’t possible. The vNET serves as the essential conduit facilitating communication between the published resource and the consuming resource, such as a virtual machine (VM) or Azure Function.

DNS Private Zones

As mentioned earlier, configuring the corresponding Azure DNS Private Zone is crucial for ensuring the smooth operation of Private Endpoint. List of DNS zones for Azure PaaS services, such as privatelink.vaultcore.azure.net for KeyVault, can be found in Microsoft Docs here.

For debugging a effective resolution of DNS records within a vNET you may utilize my open-source tool AzFuncDebugger.

Ingress vs. Egress traffic

Azure Private Link communication is incoming-only from a resource perspective. This means that the resource accessed via Private Link cannot initiate communication with any other resource within the virtual network. Therefore, Private Link should not be considered as an option for routing bi-directional communication between two resources within the virtual network.

For instance, if you access a resource like KeyVault from an AppService, you could utilize publishing KeyVault via Private Endpoint (incoming traffic from KeyVault perspective). However, for outbound traffic from the AppService, you would need to employ standard vNET outbound traffic integration.

Sub-resource Targets

Some Azure PaaS services consist of multiple sub-resources (subdomains), as seen with Azure Storage, which has privatelink.blob.core.windows.net for Blob, privatelink.file.core.windows.net for File, and so on.

This implies that to consume all sub-resources of an Azure Storage resource, you would need to create multiple Private Endpoints, each with its corresponding DNS records, to ensure full functionality.

The full list of sub-resources for Azure PaaS resources can be found here.

Cross-region / Cross-tenant Private Links

One significant benefit of Azure Private Link worth emphasizing is its capability to publish a service from one region to another by creating a Private Endpoint in a different region. Because the traffic travels via the Microsoft backbone network, the additional latency incurred is minimal. However, it’s important to keep in mind that splitting resources of your workload across regions may decrease the resiliency of the workload, as it becomes dependent on two regions.

Another remarkable use case is the ability for Private Link connections to be cross-tenant. As an Azure ISV provider, you can construct a highly intricate SaaS solution and expose only a load balancer to the customer. This load balancer allows the customer to privately consume the SaaS service within their vNETs via Private Link.

Multiple Endpoints

Multiple Private Endpoints can be associated with a single Private Link Service, with a maximum of up to 1000 instances. For instance, this capability allows you to deploy a single Load Balancer for your SaaS solution and share it among multiple customers.

Infrastructure as Code Support

All resources related to Azure Private Link have full support to be deployed via IaC tooling like Azure ARM, Bicep or Terraform. See example within this article to deploy resources via Azure Bicep.

Approvals

Since Azure Private Link supports the creation of cross-tenant connections, establishing a secure connection requires initiating a new Private Link connection from the consumer’s tenant and obtaining approval from the tenant that owns the Private Link Service with resources.

If the consumer has RBAC permissions on the published resources via the Private Link Service, the connection will be automatically approved. However, if the consumer does not have such permissions, manual approval from the customer is required. In this scenario, the Azure Private Link Center interface in the Azure Portal allows customers to view all pending connections and approve or reject Private Link connections as needed.

Costs

Please keep in mind that Private Endpoints incur additional costs, priced at $0.01 per hour for each Private Endpoint. Additionally, there are additional costs associated with the processing of inbound and outbound data transferred via Private Endpoints, which are charged per GB. It’s important to note that these costs could potentially accumulate to significant amounts in certain scenarios. On the other hand, utilizing a Private Link Service itself is free of charge.

Troubleshooting Azure Functions deployments with AzFuncDebugger tool

Wed, 10 Jan 2024 00:00:00 +0000

Introduction

I created the tool I’m discussing in this blog post to simplify and expedite my troubleshooting of FunctionApp deployments. With this post, I am trying to explain how to use the AzureFuncDebugger tool to resolve various FunctionApp deployment issues within complex enterprise environments - perfect for when your code’s all good, but the deployment’s hitting a wall because of some config mix-ups.

Note

AzFuncDebugger tool is available for download on GitHub.

The tool provides a one-page report detailing the current configuration, the behavior of the FunctionApp, and tips for resolving any issues found. “There’s no need to run any temporary VM for troubleshooting or to install SDKs and compile code; simply switch the runtime to .NET and load the package directly from URL on GitHub.

Selected Content Storage not working for FunctionApp

In certain configurations, depending on the chosen plan and operating system, Azure Functions may utilize a File Share (a feature within Azure Storage Accounts) to store the source code for Functions (this storage is referred to as Content Storage throughout this article). Given that Azure Functions is built on the Azure AppService, it also supports package deployment, which can be done through a local ZIP file or a remote URL - and with package deployment, Content Storage configuration gets ignored.

If you’ve implemented the security enhancements from my article on securing internal Azure Functions storage with PrivateLink, you might find the setup quite intricate. Regrettably, any blockage to the Content Storage access can cause your Function App to become unresponsive, resulting in 50x server errors and timeouts after a certain duration. Debugging can be particularly taxing because each change requires a waiting period for the new configuration to be saved, the Function App to restart, and the new configuration to load.

The package deployment method allows for the straightforward deployment of the open-source AzFuncDebugger tool directly from GitHub into a FunctionApp that’s encountering issues with:

1

WEBSITE_RUN_FROM_PACKAGE = https://github.com/vjirovsky/azFuncDebugger/releases/download/v0.7/pre-release.zip

This ensures the tool loads successfully without timeouts, and it activates the simulator feature for accessing the Content Storage within the given FunctionApp. By using the TEST_CCONNECTIONSTRING and TEST_CSHARE configuration parameters, you can verify the accessibility of the storage, even in “broken” FunctionApp.

FunctionApp has no network visibility to Content Storage

When a Content Storage (the Storage Account for the File Share) is set to block public access (which is recommended) and is made available only within a vNET via PrivateLink, the following issues commonly occur:

The Private Endpoint linked to the PrivateLink service is assigned an internal IP from within the designated vNET—for instance, mystorageaccount.blob.core.windows.net gets resolved to 10.0.1.5. Private Endpoint automatically generates a record in the Azure DNS Private Zone, superseding the original public DNS record for the service. This modified record is only applied within the confines of the vNET.

If the FunctionApp fails to recognize the internal IP record, you should verify the DNS server configuration by examining the WEBSITE_DNS_SERVER parameter to ensure it’s correctly set. Additionally, the boolean parameter WEBSITE_CONTENTOVERVNET must be enabled, as it dictates to connect to the Content Storage via the vNET instead of through a standard public internet outbound connection.

Additionally, the Simulator feature of the tool is pretty handy. It lets you see the IP address that’s being resolved and gives you a glimpse into the effective networking configuration. This kind of debug info is a lifesaver for figuring out whether the trouble is with the DNS setup or something in the routing - like maybe a network security group or a firewall playing gatekeeper a bit too strictly.

The Function doesn’t get loaded

The package deployment has higher priority. If you notice that a new Function isn’t loading or updating, it’s a good idea to check if the WEBSITE_RUN_FROM_PACKAGE configuration parameter is active. This parameter might be preventing the loading of Functions directly from the Content Storage. The tool provides insights into the active configuration of the code location for your FunctionApp.

TLS inspection (untrusted internal CA)

When resource accessed from FunctionApp is routed throught enterprise network, there could be some network appliance (e.g. Azure Firewall or Palo Alto), to inspect the encrypted traffic and gets re-signed by internal CA. With variable TEST_HTTPCLIENT_GET_URL you can perform a HTTP(s) call to any resource and see details about HTTP request and response, including certs details.

Enterprise custom DNS server

Many corporate Azure landing zones in enterprise settings use their own on-premises custom DNS servers to resolve addresses, even within Azure vNETs. For accessing resources over PrivateLink, these DNS zones need to be delegated to the Azure DNS Private Zone. This can be done, for example, through the Azure DNS Private Resolver service. Alternatively, a DNS record for these resources needs to be manually created in the on-premises DNS servers. The debugger tool shows you which DNS servers are actually being used and with TEST_DNS_RESOLVE_DOMAIN variable enables to test DNS record of a specific resource that’s published over PrivateLink.

Whitelisted IP traffic

In scenarios where a FunctionApp is connected to a vNET and needs to access certain resources through this vNET (like when a resource only allows traffic from a specific vNET), it’s crucial to set up the outbound traffic routing correctly. By default, FunctionApp routes traffic via the public internet. To redirect egress traffic through the vNET, you need to set the WEBSITE_VNET_ROUTE_ALL configuration parameter or the AppService setting vnetRouteAllEnabled to true.

The tool can aid in troubleshooting by showing the vNET assigned to the FunctionApp, the IP address assigned to the FunctionApp within that vNET, and the public IP address from which the outbound request was actually sent by the FunctionApp to an external service Ipify.org.

Wrong time zone (including CRON execution)

If your code depends on local time, like running daily tasks at a specific time or performing time-based operations, setting the timezone for your FunctionApp runtime is essential. To specify a timezone for datetime operations, you need to configure WEBSITE_TIME_ZONE parameter. It’s important to note that the format for specifying timezones differs between Windows and Linux runtimes. This same principle applies to other services based on AppService, such as Web Apps or API Apps.

The tool will show you the current time with the currently selected timezone, and it will issue a warning if the timezone format doesn’t match what’s expected for the chosen operating system.

Conclusion

I hope you find the tool helpful. With a few tweaks, I believe it could, with a few adjustments, be adapted to support general AppService deployments if necessary.

In case of any issue, please report on GitHub.

Securing Azure Managed Applications Resources

Sat, 14 Oct 2023 00:00:00 +0000

Introduction

Deploying an Azure Marketplace offer is a powerful way to implement a cloud solution provided by Microsoft’s partners into your Azure subscription, while outsourcing the development to the publisher of the given offer. An offer can be anything from simple solution built on top of Virtual Machines or really complex solutions, utilizing AKS containers and other Azure native services - all these services will be deployed with a purchased offer.

There are two models of a solution in Azure Marketplace:

Solution template - this plan type enables to deploy a non-transactable solution (Microsoft doesn’t facilitate the payment), where the customer will manage the solution after deployment themselves
Managed Application - solutions built as Managed Applications enables to deploy a transactable solution, where either the offer’s publisher or the customer manages the solution after deployment

For publisher is there a benefit, that their solution will be displayed in Azure Marketplace, what can help them to accquire new customers.

Managed Application - Introduction

When you deploy a solution based on the Managed Application type, it appears as another resource within your Azure resource group.

With this resource, a new managed resource group is automatically created within your subscription, where all the necessary resources of solution are deployed.

And when you delete the Managed Application resource, Azure Resource Manager will automatically delete the related managed resource group along with all the resources contained within it.

Permissions

A valuable benefit of the Managed Application deployment model is that it allows the publisher to securely access the solution’s resources (within your subscription) using the standard Azure RBAC model. This enables the publisher to monitor, upgrade, or perform maintenance on the solution as needed. The publisher will automatically be granted a Contributor or Owner role for the managed resource group (although this can be opted-out if desired).

It’s a double-edged tool; while it allows customers to out source solution management to the publisher and simply consume it as a service, some customers opt to limit access to reduce their infrastructure’s attack surface. And opting out of management access completely revokes all privileges for the publisher, including read access for monitoring.

Just-in-time Access (JIT)

In response to this challenge, Microsoft has introduced a new feature known as Just-in-time (JIT) access. This feature provides read-only access to the customer’s resources. When necessary, the publisher can request elevated permissions for these resources for a specific time period. The request requires approval from a responsible person on the customer’s side before access is granted.

While this feature effectively addresses the challenge, it has one drawback: it necessitates the use of a Microsoft Entra ID (former Azure Active Directory) P2 license on the customer’s side.

Read-only Access

For customers who lack a P2 License but still need to grant publishers elevated access while maintaining read-only access, there is one available option.

This concept hinges on utilizing the JIT feature, even in scenarios where the customer does not have a P2 License. License detection for JIT occurs during the creation of a JIT access request. If the required license is not present, the request will fail, but the deployment configuration remains unchanged.

As a result, you have the ability to restrict the publisher’s access to solutions that are built on Managed Applications, such as Pure Cloud Block Store.

In this concept, the customer deploys a Managed Application with JIT access enabled and set JIT Approval mode to Manual.

The Approvers list cannot be empty, so you need to add at least one responsible security group or user.

Thanks to the absence of an Microsoft Entra ID license, any attempt by the publisher to gain elevated access will fail, but the publisher still retains desired read-only access, and as a customer, you have more control over your infrastructure.

For customers who have the P2 License, achieving read-only access for publishers is straightforward. Simply set the JIT Approval mode to Manual and refrain from approving any requests.

8 tricks for parameters in Azure Bicep deployments

Tue, 25 Jul 2023 00:00:00 +0000

Use .bicepparam File Format

The Azure Bicep has introduced support for a new file format called .bicepparam starting from version v0.18.4. This format is used to pass parameters into Bicep templates.

The syntax of this new file format is simpler and is more readable than the previous JSON schema type used for ARM deployments.

Example:

1
2
3
4
5
6
7


//enables IntelliCode in VSCode to suggest and validate parameters against bicep template
using './file.bicep'

//parameters
param location = 'westeurope'
param param1 = 'value1'
param param2 = 'value2'

Use sys. Functions In Parameters Files

The .bicepparam file also allows the use of functions from the sys namespace, such as concat(), first(), and uniqueString(). These functions can be used to manipulate and process the parameters passed into Bicep templates.

Example:

1
2
3


param rgName = 'my-awesome-rg'
param uniqueSuffix = '${sys.uniqueString(rgName)}'
param storageName = 'storage${sys.uniqueString(uniqueSuffix)}'

This can help you to simplify your Bicep templates and remove additional parameter transformations inside the template.

Consolidate Parameters In One Place

One property I find frustrating when deploying via Bicep/ARM is that certain parameters, such as rgName and location, must be in some cases specified as extra arguments for the az command during execution, while the rest of the parameters are contained in a .bicepparam or .json parameters file.

Instead of that, I do prefer to have all customization options for the current deployment in the parameter file in one place and make no change to script executing az command.

In a ARM JSON Parameters Files

My trick to consolidate ARM JSON parameters into one place is based on reading the JSON file before the deployment command, and storing the value into variable in script.

For Scripts Based On PowerShell:

1

(Get-Content mytemplate.parameters.json -Raw | ConvertFrom-Json).parameters.resourceGroupName

For Scripts Based On `az-cli` (with `bash`):

I do recommend to use jq tool for reading a JSON file from command line. You can install the jq via multiple ways, one of them is via pip, on Ubuntu as official package:

1

sudo apt-get update && sudo apt-get install jq

on MacOS via Homebrew:

1

brew install jq

When you have jq command ready, you can access the parameter easily:

1

jq .parameters.resourceGroupName.value mytemplate.parameters.json

In .bicepparam Files

The .bicepparam is a new file format, and as such, it can be more challenging to parse the file using other tools. One of the option could be to implement custom parser, but that’s too much complicated for such small task.

As the simpliest way I do prefer to install a full Bicep cli.

Note

The Bicep cli, included in az-cli with commands az bicep, doesn’t contain all Bicep’s commands as in full Bicep cli.

With this full Bicep cli installed, you can compile input .bicepparam into ARM JSON format, which can be easily parsed with the jq tool.

1
2


paramsInJson=`bicep build-params $parametersFilename --stdout | jq -r ".parametersJson"`
resourceGroupName=`echo $paramsInJson | jq -r ".parameters.resourceGroupName.value"`

Pass A Parameter With Dynamic Value

Occasionally, it may be necessary to include a dynamic variable as a parameter during Bicep deployment, which can be sourced from an environment variable or retrieved as output from another REST API. But how to pass into the template?

ARM JSON Parameters

When you use a Bicep deployment with the original ARM JSON parameters file format, there is a option to override the parameter in the parameters file, with passing addditional --parameters argument in key=value format.

1
2
3
4
5
6


az deployment group create \
 --name myDeployment \
 --resource-group rg-bicep \
 --template-file main.bicep \
 --parameters @main.parameters.json \
 --parameters myParameter='$bashVariable'

The myParameter value in JSON parameters files will be overwritten with one in argument.

Note

You can notice the parameters filename starts with @, in .bicepparam is without this character.

Bicep .bicepparam Files

At the time this article was written, the current version of az-cli does not support the use of a .bicepparam file with this trick. The way the .bicepparam file is parsed is different, and attempting to use the --parameters argument results in an error:

1
2
3
4
5
6
7
8


$ az deployment group create \
 --name myDeployment \
 --resource-group rg-bicep \
 --template-file main.bicep \
 --parameters main.bicepparam \
 --parameters myParameter='$bashVariable'

ERROR: Can not use --parameters argument more than once when using a .bicepparam file

According to some comments found in the Azure Bicep project on GitHub, there are plans to change the behavior and enable overwriting parameters using the syntax --parameters main.bicepparam param1=localvalue, but there is no timeline for this change.

My current workaround around this limitation is based again on the bicep build-params command:

compile the .bicepparam file into temp JSON file
modify the temp JSON file (replace)
provide the temp JSON file as a parameter file to az-cli
delete the temp JSON

Pass Current IP Address

A specific scenario I want to demonstrate is how to restrict access to my current public IP address. Although I am not a fan of using IP-based restrictions, I use them often in my temporary PoCs.

To achieve this, I use the above trick in combination with retrieving my IP address from an external service such as http://ifconfig.me:

1

myIpAddress=`curl ifconfig.me 2> /dev/null`

In the .bicepparam file, I create a unique placeholder value (e.g., $myIpAddress), compile into into ARM parameters JSON file and replace it there using the traditional sed command, before it gets processed by deployment command:

1

(...) | sed "s/\$myIpAddress/$myIpAddress/" > temp.json

Use secure() Attribute

Do not forget to annotate all sensitive Bicep parameters with attribute secure(). This decorator prevents to include the value of the parameter into logs and deployment history.

1
2


@secure()
param vmAdminPassword string

Also with this attribute applied, you can integrate Azure Key Vault and store these secrets within Key Vault, and access them during the deployment from Bicep template directly.

Example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


resource keyVault 'Microsoft.KeyVault/vaults@2023-02-01' existing = {
 name: kvName
 scope: resourceGroup(subscriptionId, kvResourceGroup )
}

module sql './sql.bicep' = {
 name: 'deploySQL'
 params: {
 sqlServerName: sqlServerName
 adminLogin: adminLogin
 adminPassword: keyVault.getSecret('vmAdminPassword')
 }
}


// sql.bicep:

@secure()
param adminPassword string

resource sqlServer 'Microsoft.Sql/servers@2022-08-01-preview' = {
 ...
}

Markdown Description

When working with a complex Bicep template, it can be useful to include more complex instructions on how to use specific parameters. This can be done by adding a user guide within the @description() decorator in Markdown language, which will then be displayed in IntelliSense completion. This can help users understand how to use the parameter and make it easier for them to work with the template.

1
2
3
4
5
6
7



@description('''
Here can be a *markdown-enabled* text, displayed in **IntelliSense** in VSCode
1. something A
1. something B
''')
param subnetName string

Using a Single Parameter File for Multiple Bicep Templates

If you have a complex project that consists of multiple Bicep templates, you may need to use the same parameters across several templates. For example, you may need to use the same common Resource Group name in multiple templates.

However, in Bicep, the parameter file must only contain parameters that are relevant to the specific template it is associated with. This means that if a parameter is only used in templateA.bicep, it cannot be included in the parameter file for templateB.bicep.

1

Error BCP259: The parameter "templateBParameter" is assigned in the params file without being declared in the Bicep file.

One workaround for this issue is to include the extra parameters in the other template, but assign them default values and provide an explanation in the description() decorator, that these parameters are not used in that particular template.

Example:

1
2
3
4
5


@description('common parameter used in both templates')
param commonParameterA string

@description('NOT USED in this template')
param templateBParameter string = ''

This allows you to include the same parameters in multiple templates without causing any issues and minimizes confusion of another user of given template.

Cloud Cold Starts - How Can I Secure My Azure Infrastructure Part #1

Wed, 30 Mar 2022 00:00:00 +0000

Tip

In this series of Cloud Cold Starts articles, I’d like to address questions (at least an introduction to the topic) that may come to mind when starting out with the cloud.

Encryption At Rest

When we talk about encryption at rest, we are talking about encrypting data that is only stored and not currently being transferred or processed in memory – if the data is physically stored on the storage device (disk). We want to prevent anyone from connecting to the disk and starting to read the data without our permission – so we have the data encrypted with an encryption key as soon as it is written to the storage and decrypted as soon as it is read from the storage. The encryption key is stored elsewhere than the data. Same purpose as the BitLocker you have on your laptop. Without the encryption key, an attacker would not be able to read the data (within a reasonable time, unless there is a flaw in the encryption mechanism).

Azure recognizes two modes of encryption – client-side encryption (encryption is done outside of Azure and Azure works with encrypted data it doesn’t understand) and server-side encryption, where the encryption is managed within Azure.

Server-side Encryption

Server-side encryption encrypts data automatically during storage on disk. But what encryption key is used for this? Microsoft offers two options, its platform-managed keys and customer-managed keys (CMK, sometimes referred to as BYOK – Bring Your Own Key).

Platform keys are managed end-to-end by Azure and you don’t have to worry about anything – all the necessary key-related activities – generation, rotation, etc. are delegated to Microsoft. This solution is suitable for most workloads, but some workloads handling sensitive data must implement CMK by regulation. This increased security brings all key management tasks to the customer.

What’s really good is that Azure has security in its DNA and encrypts with platform-managed keys the data for almost all services offered – from Azure Storage (including disks for virtual machines), Azure SQL Database (search for TDE feature), or perhaps Azure Database for MySQL and more. There is no additional charge for encryption and encryption cannot be turned off, it is enforced by default.

More info about Encryption at the rest can be found here.

Encryption In Transit

Usually we need to exchange data between some applications or send it to the user – this is another place where we need to be careful about data protection. Sometimes this area is called “data in motion”. When communicating over an unencrypted channel, any attacker can listen and/or modify the content of the data along the way. This is why, in 2022, we use the secured HTTPS protocol (via TLS protocol) to access any website (and our browser verifies the website’s certificate, otherwise it will drop the connection).

To manage any resource in Azure (via the REST API or portal), Azure enforces a connection via an encrypted connection.

Almost every Azure service provides the same level of security (if the protocol of given service supports it), so you can consume a service over a secure connection.

Some services allow you to use also unsecured versions of protocols, so make sure you disable them. A great example of this is the Azure Web App, where HTTP out of the box is enabled. To disable unencrypted connections, just enable the “HTTPS Only” option in the “TLS/SSL settings” section of the Web App.

I also recommend setting the minimum TLS version to 1.2 (as of March 2022, Azure does not yet support the higher TLS version 1.3).

More details about Azure encryption in transit can be found here.

Least Privilege Strategy

Another important level of security for Azure resources is restricting permissions for users, administrators, service accounts – for everybody. Give them only the permissions they need to do their jobs, nothing more.

Tip

Do you have a service account, that configures the DNS zone only?

That service account certainly doesn’t need to have a subscription owner permissions.

Azure has a lot of built-in roles that you can assign a user to a given resource, see here.

In next article I would like to cover networking isolation and key management.

Do you like this format of Cloud Cold Starts articles? Do you have any topic you would like to cover here? Please share your idea in the comments!

Protect Azure Function’s internal Storage via PrivateLink

Wed, 29 Dec 2021 00:00:00 +0000

What data are stored inside Storage Account

Azure Function App uses two subservices blob and files in the Storage Account. It always creates a two blob containers, called azure-webjobs-hosts and azure-webjobs-secrets.

In the azure-webjobs-hosts container – there are metadata files used for logging purposes, locks etc.

In the container azure-webjobs-secrets you can find really interesting data:

every function has JSON file
there is host.json file, containing all function’s keys, including master FunctionApp key, in encrypted form

Function Apps, not using deployment type app package, also stores even functions source code in File shares container, in the site/wwwroot directory.

Out-of-the-box setup

When you create a new Function App via standard Azure Portal wizard, it creates automatically a Storage Account. This Storage Account has opened public endpoint and the Function App interacts with storage via SAS (Shared Access Signature) via public endpoint.

This is not so dangerous setup. as you could read it for first time – all file containers in this Storage Account has for Public access level value Private, so you still need to have storage access key for accessing these files.

On the other hand, in this setup, with a combination of some vulnerability, you could get the content on storage easily compromised and for some workloads you need to reduce the risk with adding extra layer of protection.

PrivateLink approach

This setup is based on limiting access to the Storage Account and requires having a vNET. Public endpoint of Storage Account is disabled and there is created a PrivateLink endpoint of the Storage Account into Function App’s vNET.

Prerequisites

The Function App need to have a plan supporting vNET integration for egress traffic (higher App Service tier, or for consumption-based resources you need to have a Premium plan).

Storage Account needs to be provisioned as General purpose V2 account kind (V1 is not supported by PrivateLink).

Note

I am reminding this, because it happened to me, that Azure Portal user wizard created a V1 Storage Account for Function App. When this happens, just create a new V2 Storage Account and point the Function App to the new storage resource.

Steps to protect Azure Function (via Azure Portal)

Turn off public endpoint on Storage Account

Go to Networking, select Allow access from selected networks and click Save

Set up a PrivateLink endpoint for Storage Account

Go to Networking, select tab Private Endpoint connections and click on +Private Endpoint
Follow wizard, as resource select the Storage Account resource
As target sub-resource select blob value
Repeat steps, only as a target sub-resource select file value

Set up required feature flag for Function App runtime

Go to the Function App resource, select Configuration and select tab Application settings
Add a new Application settings called WEBSITE_CONTENTOVERVNET, with value 1
Save new settings
Test the Function App

Now the Storage Account is not accessible from public internet, but reachable from the Function App.

AWS SSO with Azure AD – Enterprise Adoption Tips

Mon, 19 Jul 2021 00:00:00 +0000

AWS IAM can federate with external IdP for a long time (called Single-Account Access), but this solution requires creating and managing AAD Enterprise app for every federated AWS account. In case you have hundreds of AWS accounts, it brings overhead costs to manage all these applications.

AWS SSO federation

I found AWS SSO as a much better approach because all you have to do is set up and maintain a single AAD Enterprise app – long story short, metadata about users and groups (username, email address,… – NO passwords) are provisioned into AWS SSO service, deployed in AWS master account, together with role definitions (permission set). Inside AWS SSO user interface are available all connected AWS child accounts, you just select a role, destination AWS child account (where the role will be deployed), who should be granted the role in given AWS account and AWS SSO will provision and generate roles with trust relationship into AWS IAMs of selected child accounts.

When user logs into AWS console, after successful SSO login (via SAML protocol) is redirected to AWS SSO Portal, where user see all available AWS accounts and roles. The provisioning is one-way sync made via SCIM protocol and it works that AAD periodically pushes updates about users into AWS SSO service.

This setup brings some limitations or maybe “strange” behavior you need to count with.

Empty first name/surname results in SCIM failure

If you follow the official tutorial provided by Microsoft, you installed the AAD Enterprise application AWS Single Sign-On by AWS from the AAD application gallery.

The application has all the required attribute mapping out of the box. In my AD directory, after the first run of provisioning, some of the users have not been provisioned in AWS SSO with strange SystemForCrossDomainIdentityManagementServiceIncompatible error:

1
2


StatusCode: BadRequest, {"schema":["urn:ietf:params:scim:api:messages:2.0:Error"],
"detail":"Request is unparsable, syntactically incorrect, or violates schema.","status":"400"

After some debugging, I found out that the application doesn’t cover cases when a user doesn’t have filled in first name/surname. In attribute mapping, you can add some default value to these attributes or you can switch these two attributes to expression like Coalesce([surname], [mail]) – in case surname is empty, email value is used.

Scoping users and groups for provisioning

AAD Enterprise app in AWS SSO is used for SSO access (you log in via SAML into AWS SSO portal) and for provisioning (to sync users into AWS SSO via SCIM protocol).

AWS SSO service has few limits you need to have on your mind. Currently, there is a limit of 50 000 users to be provisioned into AWS SSO. This could be an issue in a scenario, you have a large organization of over 50 000 users and just a small subset of users need access to AWS console (developers, security,…) and you will need to limit number of users to be provisioned with scope filter.

There are two modes, how to filter groups/users, who will be provisioned.

Mode #1 - Sync all users and groups scope

In AAD Enterprise app select Sync all users and groups scope and in the section Mapping create a scoping filter (one filter can be for users and one for groups – but there are two isolated filters). This settings will provision all users and groups matching given filter, but it doesn’t mean they will be able to login (on AAD side). AAD checks during login workflow, if user is assigned to AAD Enterprise application. You can turn off this by switching off toggle User assignment required? in Properties.

As result, any user (including users with no assigned roles in AWS) will be able to login into AAD Enterprise app, but they will not see any AWS account after login.

Warning

Important limitation: AAD provisioning scope filter currently (as state in 2022) doesn’t support operator memberOf, so you can’t filter users by membership in a security group

Mode #2 - Sync only assigned users and groups scope

In AAD Enteprise app select Sync only assigned users and groups scope. In this mode it will provision only users and groups, who have been directly assigned to the AAD Enterprise app. If you assign the app to UserA, it will provision UserA into AWS SSO, but with no other membership. If you have assigned the AAD Enterprise app to SecurityGroupB (where UserA is member of), it will provision the security group, UserA and membership information into AWS SSO.

Cons of this approach is that you have to manage list of assigned users in two places – first is the AAD Enterprise app assignment, wait for users to be updated in AWS SSO, and assign requested role inside AWS SSO.

Nested security groups

SCIM doesn’t support nested security groups. If you have a more complicated structure when a user is a member of a security group and the security group is a member of another security group, this membership information will not be transferred into AWS SSO. Only first-level security groups are supported.

Personal Microsoft accounts with aliases

A long time away, I had an idea to rename my personal Microsoft account (former Windows LiveID) hotmail account to use new alias @outlook.com. This idea brought me much trouble – many apps (even in the Microsoft world) can’t handle the situation, somewhere is the account referred as @hotmail.com and somewhere as @outlook.com.

AWS SSO is, unfortunately, the same story – I was not able to get the account working, during login I receive an error message Invalid MFA credentials. I do not recommend using this kind of Microsoft accounts.

Redshift federation support

In case you use federated login into AWS Redshift, I have bad news for you. Federation via AWS SSO is not supported for Redshift services. You will need to onboard an AWS account with deployed Redshift cluster with a traditional Single-Account access approach into AAD in parallel.

Return HTML Response Directly from Azure Functions (.NET)

Sun, 11 Apr 2021 00:00:00 +0000

.NET Core 3.1 and lower

Following code is designed for Azure Functions running on .NET Core 3.1 (or lower). Just create a file HtmlObjectResult.cs and return new class instance as result of your Azure Function.

What about .NET 5

In .NET 5 a guide above will not work. In fact, in .NET 5 (Azure Functions in isolated process) you don’t need any custom code, a Function has during runtime easy access to HttpRequestData object, so you can easily create a response like this:

Make Azure Front Door Origin Secured with Private Link

Sat, 20 Feb 2021 00:00:00 +0000

When you deploy a Azure Front Door (or CDN) service usually you need to protect (with WAF) or you need to offload traffic from your workload – you point users to WAF/CDN url instead of origin (your workload) url. Thanks that you can protect any origin from App Service, Static website (deployed in Storage Account) or another custom origin (e.g. Kubernetes container or some on-prem workload).

Risks for Origin in Public Endpoint Model

Main issue what comes up with this model is restricting the origin to be called by WAF/CDN only – an attacker is able to call origin directly and bypass any WAF protection and perform a SQL injection or DDoS attack on your workload. You could protect your workload by filtering traffic on your backend only, based on HTTP header X-Azure-FDID value (you can find more details here) and whitelisting Azure’s infrastructure IP range (be careful about this option – IP range could change in future – you need to monitor Azure Service Tag AzureFrontDoor.Backend for any changes).

Your workload would stay exposed to public internet and could be vulnerable against attacks with network vectors, but at least there are some security controls in place.

Securing Origin with Private Link

In February 2021 Microsoft introduced a game changer feature – Azure Private Link integration for Front Door.

What is Private Link?

Azure Private Link enables you to access Azure PaaS Services (for example, Azure Storage and SQL Database) and Azure hosted customer-owned/partner services over a private endpoint in your virtual network.

Traffic between your workload and service is not routed via public internet, but via Microsoft private network (backbone) only. Thanks to that your workload can access required service via much more secure way and doesn’t have to be exposed to internet (but they can be if you need).

Private Link is also good option versus IP whitelist filtering, because with private endpoint you associate (whitelist) your workload with a instance, not whole PaaS service.

Additional advantage of Private Link is, that your workload can be even in another subscription, even in another AAD tenant.

How to Secure Static Website Origin on Azure Storage Account

Info

Private Link feature is available for Premium SKU of Azure Front Door only.

Turn On Static Website Feature

For deployment of SPA websites (or static export from WordPress, see my previous article PR Crisis Microsite Deployed in Cloud in Few Hours). I prefer to deploy these files into Azure Blob Storage with turned on Static Website feature.

When you turn on the feature, your will receive an auto-generated URL for your storage account – we will need this URL later.

Anything, what you upload into blob storage folder $web, will be accessible over this auto-generated URL.

Restrict Public Access for Azure Storage

Go to Networking and restrict access to selected networks only. You can also uncheck Allow trusted Microsoft services to access to storage account option.

Set Up Origin in Azure Front Door

Info

As of February 2021 Private Link for Front Door is Preview feature and is available in East US, West 2 US, and South Central US regions only.

As next you need to set up the Storage Account as origin for Front Door. Fill in auto-generated URL of Static website into fields Host name and Origin host header.

Select Enable private link service and select your Storage Account resource. Select a sub resource web. Be sure, that values of Host name and Origin host headers didn’t changed to different one, overwrite them if that happened.

Configuration above is derived from Private Link architecture – it will register in internal DNS zone different record (something close to private IP) for your resource – when Front Door try to access this host name, private IP record is used and it’s not routed via public internet.

Approve a Private Link request

When you submit form from previous section, a request for new Private Link connection has been created.

Go to Private Link Center (you can find it by typing “Private Link” into Search resources field) and approve the request in Pending connections.

The Result

Thanks to settings above, I am able to access the website via Front Door only:

When I try to bypass Front Door and access Storage Account directly, I receive an error:

Caching whole API response in ASP.NET Core

Thu, 16 Jul 2020 00:00:00 +0000

Following method implements REST GET method in Web API controller Order. This method provides information about order from database:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


// GET: order/{orderId}
/// 
/// Get manifest file
/// 
/// Order ID
[HttpGet("order/{orderId}"]
public IActionResult Get(int orderId)
{
 var entry = _repository.GetById(orderId);

 if (entry == null)
 return NotFound("Order not found.");

 return Ok(entry);
}

I needed to add response cache, but I didn’t wanted to care about status codes – if i would only cache the output text, I would miss in cached responses alternative codes like 404 – or i would have to save also some another metadata – which is not so developer-friendly solution.

I got idea – what about cache whole response? Response is object implementing IActionResult, so what about serializate whole response, save to cache – and when is accessed already cached, only load from cache and deserializate and send to client. And how to do it really developer-friendly? I have idea to wrap it into lambda, so original content of method is invoked in lambda and result of the lambda is saved to cache:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34



// GET: order/{orderId}
/// <summary>
/// Get manifest file
/// </summary>
/// <param name="orderId">Order ID</param>
[HttpGet("order/{orderId}"]
public IActionResult Get(int orderId)
{
 var cacheKey = "Get-Order-" + orderId;

 var cachedResponse = _distributedCache.GetString(cacheKey);
 if (!string.IsNullOrEmpty(cachedResponse))
 {
 return JsonConvert.DeserializeObject(cachedResponse);
 }
 }

 Func createResponse = () =>
 {
 var entry = _repository.GetById(orderId);

 if (entry == null)
 return NotFound("Order not found.");

 return Ok(entry);
 };

 var actionResponse = createResponse();
 var serializatedResponse = JsonConvert.SerializeObject(actionResponse);
 _distributedCache.SetString(cacheKey, serializatedResponse);

 return actionResponse;
}

Run .exe Application in Azure Functions in Powerful Way

Sun, 12 Jul 2020 00:00:00 +0000

If your workload fits into Azure Functions limits, you can save costs&time and run your executable totally serverless – you will not need to patch your OS and easily scale workloads dynamically as you need. You can pay for every invocation in consumption plan (with some free grant offered every month), or alternatively you can have dedicated App Service plan, which can be shared between multiple Function Apps (ideal for multiple small workloads to optimize your costs).

In Azure Functions you have multiple options, how to run your .exe – some of workloads can be solved even without writing a line of code, but you really need to ensure the workflow fits limitations of solution.

PowerShell wrapper

Easiest option – you don’t need to write any code (Microsoft did it instead of you) – they have created a wrapper and published on GitHub. This wrapper (written in .csx) enables you to convert executable command to HTTP endpoint.

Example of config:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


{
 "name":"consoleAppToFunctions",
 "input":{
 "command":"ffmpeg.exe -i {inputFile} {output1}",
 "arguments":{
 "inputFile":{
 "url":"https://1drv.ms/v/"
 },
 "output1":{
 "localfile":"out.mp3"
 }
 }
 },
 "output":{
 "folder":"outputFolder",
 "binaryFile":{
 "returnFile":"*.mp3",
 "returnFileName":"yourFile.mp3"
 }
 }

This example would take URL of your input file on OneDrive (as HTTP parameter inputFile), process it and answers with output file as HTTP response.

It will help you when you need to serve a console app as web service, for anything else (for example when you need to call service by alternative trigger like Timer or ServiceBus) you need to customize the wrapper.

Azure Functions custom handlers

Recently introduced feature custom handler enables to implement any programming language that supports HTTP (e.g. Go, Rust). Your executable needs to understand HTTP request and Azure Function acts as proxy (e.g. transform Blob storage trigger to HTTP request) a translate trigger/input payload into HTTP request and optionally process response.

I mention this feature because some kind of workloads can be solved by this easy no code method – when you have some legacy executable working as HTTP server.

More details about custom handlers can be found on Microsoft Docs, samples on GitHub.

Writing custom wrapper

You can write custom wrapper in any of these language – C#, JavaScript (+TypeScript), F#, Java, PowerShell, Python – selection is up to you. Usually I write them in C#, but that’s my personal preferences only.

1
2
3
4
5
6
7
8


Process p = new Process();
p.StartInfo.UseShellExecute = false;
p.StartInfo.FileName = Path.Combine(readonlyCmdDir, "myapp.exe");
p.StartInfo.Arguments = "myinput.txt";
p.Start();
p.WaitForExit();

//p.ExitCode should be equal to 0 - standard exit

Azure Functions Properties And Limitations

When you write your own wrapper, you need to know following properties and limitations about Azure Functions.

Lifecycle in Azure Functions Host

One of the most important thing you need to know about Azure Functions host is that your Function can be invocated in a new fresh host or inside re-used host. You can imagine host as an application, which performs supportive tasks (authentication, HTTP router, listening on triggers, timer planning) and when request matches route (or other trigger fires), host will trigger method Run() of static class, representing your Function.

Azure Function host is an App Service

Every Function App (I mean whole your Function App, not every Function in the app) runs in own sandbox – isolated from other Function Apps running on same machine and providing an additional degree of security. You are not permitted to perform some operations like writing to registry, open raw sockets, creating symlinks etc.

More details about sandbox can be found on GitHub.

Filesystem in App Service

All files in %HOME% (or /home on Linux) directory are persistent. When you scale out your Function App into multiple instances, this directory is shared between all of them.

Resulting effects

All above lead to some behavior you need to count with.

Constructor is called in fresh host only

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13



 public static class MyFunction
 {
 static MyFunction()
 {
 myInitTask();
 }
 [FunctionName("MyFunction")]
 public static async Task Run(...)
 {
 //my task
 }
 }

You could think that myInitTask() will be called during startup of host application – that’s not true, because host app uses lazy-loading, constructor will be called only during first time, when your function is triggered.

When you run the function again (in short period, usually 5-10 minutes), Azure will re-use existing host and call Run() method of your Function only. If your Function will not be called for some time, host will release it from memory and given host will be a fresh host for your Function.

If you would perform any singleton operation in constructor (for example. connection to database), check during every invocation (means in Run() method) you have valid singleton objects (timeout).

Also ensure, you clean all resources in right way in your Run() method, otherwise you can reach out limits, like socket exception, very easily.

Possible conflicts between invocations/nodes

Functions are not processed sequentially, Function host will try to save your costs/time and perform as many Functions as possible in parallel.

When you have a workload, which require an exclusive access for a shared resource you need to redesign solution or implement some locking method.

I have created a test application to illustrate this kind of conflicts – this test app was writing log in common file when invocation of Function started, wait 15 seconds and write into log file when invocation finished.

1
2
3
4
5
6


[11:30:15 PM] [START] from invocationId "0e2e1b8c-2338-4ed3-9d35-77ade55491a2"
[11:30:39 PM] [START] from invocationId "e37c05f2-b55e-40fa-ad4d-cb5a257d79e6"
[11:30:45 PM] [FINISH] from invocationId "0e2e1b8c-2338-4ed3-9d35-77ade55491a2"
[11:30:47 PM] [START] from invocationId "3a77ddbe-1547-4dcf-ba75-26e458e00b50"
[11:31:09 PM] [FINISH] from invocationId "e37c05f2-b55e-40fa-ad4d-cb5a257d79e6"
...

As you can see, interactions with log file were overlapping – if I would have a executable, which requires to create some temp file with constant filename, it could case problems and damage outputs.

Following workaround will work at most cases – if a binary needs to create some temp files in same directory as it runs, most of them will respect current working directory and will create temp files there.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41


[FunctionName("ConvertFileFunction")]
public static async Task Run(
 [HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = null)] HttpRequest req,
 ILogger log,
 ExecutionContext context)
{
 // we need to inject ExecutionContext, which will provide us ID of invocation (our unique ID for creating a directory)
 try{

 #if DEBUG
 var tempRootDir = Path.GetTempPath();
 #else
 // temp location we will have in real Azure Function host 
 var tempRootDir = "D:\\home\\data";
 #endif

 // create a unique temp folder for given invocation of Function
 var workingDir = Path.Combine(tempRootDir, context.InvocationId.ToString());
 Directory.CreateDirectory(workingDir);

 //
 var readonlyCmdDir = Path.Combine(context.FunctionAppDirectory, "cmd");
 // set working directory where to create files
 Directory.SetCurrentDirectory(workingDir);

 try
 {
 .... CREATE A PROCESS...
 }
 catch (Exception e)
 {
 }
 finally
 {
 Directory.SetCurrentDirectory(readonlyCmdDir);

 // we have to clean after ourselves and delete our temp files
 Directory.Delete(workingDir, true);
 }
 }
}

In case a binary relies on TEMP variable, you can “fake” environment variable for given process:

1
2
3



Process p = new Process();
p.startInfo.EnvironmentVariables["TEMP"] = "---MY-VALUE---";

Have you experienced any other properties or limitations of running executables in Azure Functions? please share your thought into comments.

Azure Functions and forbidden sockets exception

Fri, 22 May 2020 00:00:00 +0000

1
2
3


System.Net.Sockets.SocketException
Exception while executing function Function1: An exception occurred while opening a connection to the server.
An attempt was made to access a socket in a way forbidden by its access permissions.

The reason for this error is simple – you reached your’s sandbox (Project Kudu) currently opened outcoming sockets limit and sandbox disabled access for new sockets.

Azure connection limits

Limits per App Service Plans:

App Service Plan	Connection Limit
Free F1	250
Shared D1	250
Basic B1 1 Instance	1920
Basic B2 1 Instance	3968
Basic B3 1 Instance	8064
Standard S1 1 Instance	1920
Standard S1 2 Instances	1920 per instance
Standard S2 1 Instance	3968
Standard S3 1 Instance	8064
Premium P1 1 Instance (Preview)	1920

Source: azurefieldnotes.com

How to solve this issue

Reason 1 – You really need so much connections

In case you really need so much connections opened at once, you have to upgrade you App Plan, according to limits table above.

Reason 2 – You have bug in you app

Check, if your application closes all the opened connections. For example in my application, source of this issue were not-released sockets by HttpClient.

How to use HttpClient (also) in Azure Functions/Web apps correctly

DON’T DO:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


public static class HowToNotUseHttpClientInFunction
{
 [FunctionName("HowToNotUseHttpClientInFunction")]
 public static async Task Run(List urls, ILogger log)
 {
 log.LogInformation("C# HTTP trigger function processed a request.");

 foreach(var url in urls)
 {
 using (var client = new HttpClient())
 {
 var result = await client.GetAsync(url);
 Console.WriteLine(result.Content);
 }
 }

 }
}

Wrapping up HTTPClient into using() section will not work as you probably think – even the application exits, sockets stay to be opened and they will wait for OS’s timeout, after it can be reused. So in this case, function is going to allocate 10 sockets and will release them after timeout.

You can read more details on aspnetmonsters.com.

DO:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


public static class HowToUseHttpClientInFunction
{
 public static HttpClient client = new HttpClient();

 [FunctionName("HowToUseHttpClientInFunction")]
 public static async Task Run(List urls, ILogger log)
 {
 log.LogInformation("C# HTTP trigger function processed a request.");

 foreach (var url in urls)
 {
 var result = await client.GetAsync(url);
 Console.WriteLine(result.Content);
 }

 }
}

This way will open only one connection and reuse it for all HTTP requests needed by application, so you will not probably reach Kudu’s limits.

PR Crisis Microsite Deployed in Cloud in Few Hours

Tue, 24 Mar 2020 00:00:00 +0000

During PR crises of corporations (e.g. faulty service or product), main company websites often become overloaded. This situation typically leads to an increased amount of inbound telephone calls and overloading the call centers. Changing the main company website usually takes some time (they are built on enterprise CMS) and scaling the customer telephone lines is not so easy (and really expensive). All of the above are reasons why most of companies choose to create a crisis-dedicated microsite or move communication to external platform (e.g. Medium).

A microsite solution (looks more professional) needs a decision where to run it. I don’t see any better scenario for a website to be deployed into the cloud – we need to have website scale-able and quickly deployed. I am sure there are even better tools you could use to build the crisis website, but for our solution we had to make a compromise – sometimes because of usability but most importantly because of the time constraint – what is easiest to set up, with what we have the most experience to be able to launch MVP (Minimum Viable Product) as soon as possible.

We wanted to have a solution which would be stable and we didn’t want to make the ministry’s PR department go through a Markdown language training, and so we decided to use WordPress. This CMS brings a friendly user interface but it’s a bigger challenge to configure it to be secure and ready for high performance.

The solution described below is designed to be deployed in Microsoft Azure. It can be also easily ported to AWS (CloudFront, S3, etc.). We selected Azure because most enterprise companies have Azure implemented in their organizations (usually because Azure Active Directory or Office 365).

Solution Architecture

Solution is built on idea, standard users don’t interact with CMS, only but with static HTML files generated by CMS. Thanks to this approach, we can easily achieve the desired performance and security.

WordPress (WP) is installed on a virtual server, which is isolated and accessible only via VPN (or IP filter at least). WP installation is configured for a different URL, than the destination URL of the website. Without this WP installation doesn’t work well. When the author generates static files (output is located on the server’s filesystem), our custom built script is triggered – this script syncs these generated files to Azure Blob Storage via Shared Access Signature (this could be simplified by mounting Blob storage as directory on filesystem).

Azure CDN is placed In front of Blob Storage. This service provides caching of content to the closest physical nodes (and in fact solves all our performance needs). When the author is satisfied with the exported version of articles, the author performs CDN cache invalidation and all readers will see the updated version in a few minutes. Thanks to saving SSL private key in Key Vault, the Azure CDN can behave as our website.

The diagram also shows a „registered app in AAD“. This application is granted to perform the CDN cache invalidation and provides access for our custom WP plugin – this plugin enables to trigger the invalidation from WP user interface via the Azure REST API.

Limitation of this solution is a lack of interactive content – e.g. search, poll etc. However, these features are usually not needed for a crisis website.

Solution Details and Usage

We used plugin WP2Static for static files export. The plugin works as a crawler. It crawls all pages and articles, generates static HTML files from them and downloads all other necessary files (e.g. images, CSS files, etc.). The plugin replaces URL of WP installation with production website URL.

Publishing the new content version was initially performed manually by running a bash script (in early stages of MVP). Few hours later we developed with Vladimir Smitka custom WP plugin (for cache invalidation) and script, which detects new export and uploads it on Blob Storage. This combination enables the authors to make a deployment by themselves from WP directly.

I also recommend setting up HTTPS-only redirection on Azure CDN (+redirection from a www. subdomain) by custom rules in the Rules engine feature.

Result

The website was pretty fast even with 11K concurrent active users. I believe Azure CDN capacity could handle a bigger amount but we haven’t needed it yet.

Despite this fact, the website was stable, according to Hlídač webů service (a monitoring service of the government websites). The graph below shows the moments when content was deployed. The website slowed down at those points.

Checklist Before You Start

Ensure you have an Azure access with sufficient permissions – including all mentioned Azure services above are enabled (Azure Policy/de-registered service type in subscription)
Ensure you have an access to your Azure AD – usually other department have to create application in AAD and assign role for given CDN instance
Set up DNS records
If you want to use a custom SSL certificate, ensure you have private key ready (Azure CDN is able to generate SSL certificate by itself, only needs to have set up DNS records to CDN)
In case you have a custom SSL certificate, ensure you have an access to AAD to set up the Azure CDN so that it has permissions to access the private key in Key Vault.

General Tips for a Crisis Website

Reduce TTL of DNS records to 5 minutes, in case you will need to change those quickly.
Do the same change also for HSTS settings. In case you will misconfigured HTTPS, so you don’t want to cut off all your readers
If your microsite will be subdomain crisis.acmecompany.com, be sure also to establish a redirection from www.crisis.acmecompany.com. The URLs are often preceded with www. by the press regardless of the canon URL form Set up Google Analytics. Even if you don’t want to evaluate results (what is searched by visitors), it’s good to know what the traffic was and for what traffic size you need to prepare your main website

High-availability Version

To get this solution ready also for the possibility of an Azure region outage (where your website is running), I also describe this solution in HA version. However, for the basic crisis websites the solution described above is sufficient.

The entire solution is duplicated also to another failover Azure region. An Azure Traffic Manager is placed in front of the Azure CDN. This service performs a health check and in case of failure of the primary region it automatically switches to the failover region. The Azure Site Recovery service is used for replication of VM with WP administration.

The script and the plugin requires a small modification to perform all changes on both Azure Storage and CDN.

Source Code and Settings

You can find the source code of WP plugin, the script and the necessary settings on my GitHub.

Thanks to Adam J. Sporka for text correction of the article.

Optimize Azure CosmosDB costs

Mon, 02 Sep 2019 00:00:00 +0000

Azure CosmosDB pricing explained

Pricing of this service is not so complicated as it looks on the first try – you pay for two parameters together – for saved document size ($0.25 per GB) + for reserved database performance (throughput) – in Request Units (RU) / second unit.

Minimum performance you have to reserve is 400 RU/s ($0.008 per 1 hour), scaleable by 100 RU/s.

Pricing #1 per document collection

In this pricing mode, you pay for reserved performance per every document collection.

So at the minimum, you will pay for every document collection in your database $23.61 per month (400 RU/s, 1GB storage, Single Region Write).

You can scale performance for every collection separately. But if you don’t fill up reserved performance, you are wasting your money.

Pricing #2 per database

Azure also provides alternative pricing – you can set up reserved performance for whole database. This reserved performance is shared across all collections in database. Minimum is again 400 RU/s

Every collection needs to have at least 100 RU/s – so first 4 collections are included, extra collection will cost you +100 RU/s per collection.

And small note, this pricing mode has one extra requirement – you need to have shard key in all your collections.

Pricing-problematic use case

Imagine you want to migrate f.ex. application, which has 10x MongoDB entity types (collections).

By pricing mode #1 you get amount $233.6 per month (400 RU/s per collection = 4000 RU/s in total) By pricing mode #2 you get amount $58.4 per month (1000 RU/s in total) – both without storage cost

That could be ok for some kind of applications. But what if I have application, which has some primary collections with frequent access and some supportive collections with infrequent access?

So you can join the group of users, who pay $23.61 per month per every infrequent collection. Or you able to not use CosmosDB for infrequent collections, but I would have to manage another connection to another database service, care about SLAs etc.

First reasonable solution is to use the pricing mode #2, but you have to pay at least 100 RU/s per every collection, so if you have a lot of infrequent collections, you still going to pay too much.

Solution based on MongoDB discriminators

When I read the pricing description in detail, I got an idea:

What about put all entities in one collection and mark into BSON serializated entity their type and use it during deserialization?

One of my friend reminded me, that MongoDB C# SDK (MongoDB SDKs for almost every language) already have built-in feature called discriminator. This feature allows you to use polymorphism on entites saved to MongoDB database (more info).

1
2
3
4


[BsonDiscriminator("User")]
public User {
 // fields and properties
}

The idea is simple – I am going to create only one collection in pricing mode #1 (with 400RU/s), where are located entities across all infrequent collections.

Example implementation in MongoDB .NET SDK

ACommonStoredEntityModel.cs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


/// 
/// Abstract class of entity saved in MongoDB
/// 
[BsonIgnoreExtraElements]
[BsonDiscriminator(Required = true)]
public abstract class ACommonStoredEntityModel
{
 const string COMMON_COLLECTION_NAME = "common_storage";

 [BsonId]
 public ObjectId id;

 public ACommonStoredEntityModel()
 {
 }


 /// 
 /// Get collection name in database
 /// 
 /// 
 public static string GetCommonCollectionName()
 {
 return COMMON_COLLECTION_NAME;
 }
}

Category.cs (entity #1)

1
2
3
4


public class Category : ACommonStoredEntityModel
{
 public string title;
}

User.cs (entity #2)

1
2
3
4
5
6



public class User : ACommonStoredEntityModel
{
 public string username;
 public string email;
}

AMongoCommonRepository.cs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61


/// 
/// Repository interface for common collection saved entities
/// Do not forget to register your new Repository in IoC in Startup.cs (ConfigureServices())
/// 
public abstract class AMongoCommonRepository where T : ACommonStoredEntityModel
{
 /// 
 /// collection
 /// 
 protected IMongoCollection _collection;

 /// 
 /// mongodb connection
 /// 
 protected IMongoDatabase _mongoDatabase = null;

 public AMongoCommonRepository(IMongoDatabase mongoDatabase)
 {
 _mongoDatabase = mongoDatabase;


 if (!IsCommonCollectionExists())
 {
 _mongoDatabase.CreateCollection(ACommonStoredEntityModel.GetCommonCollectionName());
 }
 _collection = _mongoDatabase.GetCollection(ACommonStoredEntityModel.GetCommonCollectionName()).OfType();

 if (!BsonClassMap.IsClassMapRegistered(typeof(T)))
 {
 BsonClassMap.RegisterClassMap(cm =>
 {
 cm.SetDiscriminator(typeof(T).Name);
 });
 }

 }

 /// 
 /// checks if given collectionName exists in database
 /// 
 /// 
 public bool IsCommonCollectionExists()
 {
 var filter = new BsonDocument("name", ACommonStoredEntityModel.GetCommonCollectionName());
 //filter by collection name
 var collections = _mongoDatabase.ListCollections(new ListCollectionsOptions { Filter = filter });
 //check for existence
 return collections.Any();
 }


 /// 
 /// Get all entities
 /// 
 /// all entities
 public async Task> GetAllAsync()
 {
 var result = await _collection.FindAsync(u => 1 == 1);
 return result.ToList();
 }
}

CategoryRepository.cs

1
2
3
4
5
6
7


/// 
/// Repository interface for Category entity
/// Do not forget to register Repository in IoC in Startup.cs (ConfigureServices())
/// 
public class CategoryRepository : AMongoCommonRepository
{
}

UserRepository.cs

1
2
3
4
5
6
7


///
/// Repository interface for User entity
/// Do not forget to register Repository in IoC in Startup.cs (ConfigureServices())
/// 
public class UserRepository : AMongoCommonRepository
{
}

By this repository approach you can access these entities like before the change:

Usage

1
2


var users = await _userRepository.GetAllAsync();
var category = await _categoryRepository.GetAllAsync();

In this approach there is no sharding key, but it’s easy to add it.

Results

As you see, SDK automatically adds discriminator attribute _t. Thanks to this attribute MongoDB SDK recognizes entity type, how to deserialize document.

Cost savings – case study

We used this approach in one project – Web API application, which has some collections with frequent access and few with infrequent access. We moved these infrequent collections to this common collection.

As you see, we got about -33% cost savings with fully utilized performance, without any performance issues in application (because existing prepaid reserved database performance was not utilized).

Also we still have possibility to extend performance for common collection (by granularity +-100 RU/s).

This solution will not save you every time, but for similar use cases can be really helpful.

Sections in Azure App configuration as in appsettings.json

Thu, 27 Jun 2019 00:00:00 +0000

Do you have some Application settings in appsettings.json for App Service (or Azure Function) structured in sections? And do you know how to set up same structure as Application settings in Azure?

It’s easy – just write Application settings Key in format Section:MyAttribute

for appsettings.json:

1
2
3
4
5
6


{
 "myGreatApi": {
 "baseUrl": "https://localhost:7071/api/",
 "version": 2,
 }
}

will Application settings look like:

Running Azure Functions in Docker container – crashes on production

Tue, 22 Jan 2019 00:00:00 +0000

After searching on web for exit code 137, I found it’s SIGKILL signal – so host environment sent this signal to process in container. Because of that, I tried to start my container and watch stats by command:

1
2
3


$ docker stats
CONTAINER ID NAME CPU % MEM USAGE / LIMIT
4feddd624a5e my-azurefunction-app 10.39% 398.4MiB / 1.795GiB

So it’s in memory…

By this stats I was able to recognize, that my container drains memory so quickly, that all free memory is assigned to it after running 30 seconds. So the problem is in memory management.

I searched for memory leakage in function source code and made some memory optimalization, but with no effect. It was strange to me, that these memory leaks are not present in dev environment – app runs stable around 65MB. I started to think, that there is no GC Collect performed by runtime.

.NET Core environment

This was reason, why I started search more detail about .NET Core environment. And I found the problem – .NET Core uses for app different Server Garbage Collector on server, than on developer’s Workstation GC. Server GC is optimized for higher throughput and scalability – and counts with larger resources. Servers GC has different limits for ephemeral memory segment (for objects from generation 0+1). Very different.

GC	32-bit	64-bit
Workstation GC	16 MB	256 MB
Server GC	64 MB	4 GB
Server GC with > 4 logical CPUs	32 MB	2 GB
Server GC with > 8 logical CPUs	16 MB	1 GB

So from workstation env my application went to server with 2GB RAM. Because of func SDK is compiled 32-bit, .NET recieved this env:

1

ProcessorCount:4,Is64BitOperatingSystem:True,Is64BitProcess:False

But inside Docker container on production:

1

ProcessorCount:2,Is64BitOperatingSystem:True,Is64BitProcess:True

Switching to Workstation GC

So as solution, I switched GC for my Function app to Workstation GC. It’s not so much optimized, but it works also on so small resources.

You can switch Server GC to Workstation GC by adding value ServerGarbageCollection to .csproj file:

1
2
3
4
5
6
7


<Project Sdk="Microsoft.NET.Sdk">
 <PropertyGroup>
 <TargetFramework>netstandard2.0</TargetFramework>
 <AzureFunctionsVersion>v2</AzureFunctionsVersion>
 <ServerGarbageCollection>false</ServerGarbageCollection>
 </PropertyGroup>
...

If you are interested, you can find more informations about GC in .NET on Microsoft documentation.

Running Azure Functions on your infrastructure

Mon, 10 Dec 2018 00:00:00 +0000

Introduction to Azure Functions

Azure Functions provide easy way how to run serverless functions (small pieces of code) in Azure. You can run your C#/Javascript/F#/Java/Python function with HTTP/DB/queue/file trigger integration. Sample Function:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


 [FunctionName("Function2")]
 public static HttpResponseMessage Run(
 [HttpTrigger(AuthorizationLevel.Anonymous, "get", "post", Route = "ping/")]HttpRequestMessage req,
 ILogger log
 )
 {
 log.LogInformation("Function 2 - I am OK!");

 return req.CreateResponse(HttpStatusCode.OK, "pong");
 }

Dev environment

Microsoft created nice cross-platform environment for developers. You are able to develop and debug Azure Functions on Windows/Linux/Mac with Visual Studio (Code). For testing and debugging you can use Azure Functions Core Tools (func.exe/func).

To install AFCT via npm use command:

1

npm i -g azure-functions-core-tools --unsafe-perm true

But this dev environment requires connection strings to Azure Storage for writing metadata about functions (locks, timers or even log outputs from functions) – variables AzureWebJobsStorage and AzureWebJobsDashboard in local.settings.json file.

Options

Azure Storage Emulator

A service, which emulates Azure’s Storage API (usually is installed with Azure SDK) and can be installed on any Windows computer.

In that case use for both variables value: "UseDevelopmentStorage=true"

++ no costs −− dependency on Windows, stability (especially for concurrent clients)

Azure Storage Acccount

straightforward solution: just create another Storage in Azure and use it for development.

++ easy configuration, same behavior as production −− you have to be online with connection to Azure to run and debug your function

Microsoft’s (production) runtime environment

For production Azure Functions apps Microsoft created more runtime environments. When you deploy your Function to Azure, you can choose between Windows and Linux runtime. Both runtime are available as Docker container image on GitHub.

But they have same issue as dev environment – they needs connection strings to Azure Storage for writing metadata about functions (locks, timers or even log outputs from functions) – variables AzureWebJobsStorage and AzureWebJobsDashboard in local.settings.json file.

Q: So how can I run production server on own infrastructure totally without connection to Azure?
A: You can’t, at least for now. Depedency to Azure Storage is too much strong at the moment.

Introducing Vasek’s AF Host Docker image

But you can get close to that. For this purpose I created my custom runtime Docker image for Azure Functions applications – to have better experience with applications deployed on own infrastructure.

This image is based on Microsoft’s official docker runtime image, but with some modifications – e.g. redirected logs to console, exit code for Docker,… So result is, that for runtime you need only Azure Storage Account for metadata (AzureWebJobsStorage).

Installation

Installation is easy – instead of official docker image use for runtime my image vjirovsky/vaseks-af-host:{runtimelanguage}-{version}

For dotnet runtime v2.0: FROM vjirovsky/vaseks-af-host:dotnet-2.0

Sample project based on Vasek’s AF Host image

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


FROM microsoft/dotnet:2.1-sdk AS installer-env

COPY src/ /src/
RUN cd /src/SampleFunctionApp && \
 mkdir -p /home/site/wwwroot && \
 dotnet publish *.csproj --output /home/site/wwwroot

FROM vjirovsky/vaseks-af-host:dotnet-2.0

ENV AzureWebJobsStorage="---YOUR-STORAGE_CONNECTION_STRING---"

COPY --from=installer-env ["/home/site/wwwroot", "/home/site/wwwroot"]

You can find out sample project on GitHub.

Sample run (of app sample-app):

1

docker run -p 8080:80 -it sample-app:latest

Source code

Source code of Vasek’s Azure Functions Host runtime is available on GitHub.

Force Garbage Collector during debugging in Visual Studio

Fri, 16 Nov 2018 00:00:00 +0000

It’s really easy to force Garbage Collector to collect dead objects from heap.

All you need is command:

1

GC.Collect()

From Visual Studio during debugging

As first your application needs to be paused. This can be done when code reaches breakpoint, or you can use Break All function (Ctrl+Alt+Back or blue pause icon in toolbar)
When the code is paused, open the Immediate Window (another tab of Output by default) Write command into Immediate Window and press Enter:

1

GC.Collect();

Garbage collection should be performed now and expected result is:

1
2


GC.Collect();
Expression has been evaluated and has no value

From code during runtime

You should not to care about GC in application – .NET runtime do it instead of you.

But in case you really need it, you can call GC by:

1
2


GC.Collect();
GC.WaitForPendingFinalizers();

Handling Xbox One Media Remote in UWP applications

Thu, 27 Apr 2017 00:00:00 +0000

If you see it for first time, Xbox One Media Remote is remote controller (based on IR), which you can buy from Microsoft Store for ~$25 as an offical Xbox accessory.

Buttons on Xbox One Media Remote

Handling pressed keys

In App.xaml.cs you have to change RequiresPointerMode:

1
2
3
4
5
6
7


 public App()
 {
 this.InitializeComponent();
 this.Suspending += OnSuspending;

 this.RequiresPointerMode = ApplicationRequiresPointerMode.WhenRequested;
 }

There is example, how you can handle buttons on page:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


public MainPage()
 {
 this.InitializeComponent();
 this.RequiresPointer = RequiresPointer.WhenEngaged;
 this.IsFocusEngagementEnabled = true;
 Window.Current.CoreWindow.KeyDown += CoreWindow_KeyDown;
 }

 private void CoreWindow_KeyDown(Windows.UI.Core.CoreWindow sender, Windows.UI.Core.KeyEventArgs args)
 {
 //let's play with args.VirtualKey
 }

Remote’s buttons and VirtualKeys enum mapping

Button	Runtime equivalent
Home button (*)	`(VirtualKey)(7)`
View button	`System.VirtualKey.GamepadView`
Menu button	`System.VirtualKey.GamepadMenu`
Navigation arrows	`System.VirtualKey.GamepadDPadLeft` `System.VirtualKey.GamepadDPadUp` `System.VirtualKey.GamepadDPadRight` `System.VirtualKey.GamepadDPadDown`
Select button	`System.VirtualKey.GamepadA`
Back button	`System.VirtualKey.GamepadB`
OneGuide button	not available during runtime
Volume+mute buttons	not available during runtime
Channel buttons	`System.VirtualKey.PageUp System.VirtualKey.PageDown`
Media control buttons	not available during runtime

(*) Home button is special – when user press this button, Xbox opens system’s menu (you can’t override that), but system will also send you signal (VirtualKey)(7) – so for example, you can pause game or movie in your app.