Lesson: Creating and Managing Custom Virtual Machine Images in Azure

Introduction: The Power of Custom Images

In the early days of cloud computing, administrators often spent hours manually configuring virtual machines (VMs) after deployment. You would spin up a base image from the Azure Marketplace, log in, install security updates, configure web servers, add custom scripts, and tweak operating system settings. If you needed to scale out to ten or twenty instances, repeating this process was not only tedious but also prone to human error, leading to "configuration drift" where your servers eventually diverged in their settings and performance.

Custom Virtual Machine images solve this problem by providing a standardized, pre-configured template. By creating a custom image, you capture the state of a virtual machine—including your specific software stack, security patches, and configuration files—and save it as a reusable blueprint. When you deploy a new VM from this custom image, it arrives pre-configured, reducing deployment times from hours to minutes. This practice is foundational for infrastructure as code, auto-scaling groups, and maintaining a consistent environment across development, testing, and production stages.

This lesson explores how to create, store, and manage custom VM images in Azure. We will move beyond the basic "capture" method and look at professional workflows using Azure Compute Gallery, which is the industry standard for versioning and sharing images across your organization.

The Evolution of Image Management

Before we dive into the "how-to," it is important to understand the shift in how we handle images. Initially, Azure relied on "Managed Images." While functional, Managed Images are difficult to scale; they are tied to a specific region and lack versioning capabilities. If you update your software, you have to overwrite or manually delete the old image, which can break existing deployments.

Azure Compute Gallery (formerly known as Shared Image Gallery) changed this by introducing a structure that supports versioning, regional replication, and role-based access control. With a gallery, you can organize images into logical sets, provide metadata, and automatically push updates to different geographic regions. This is the approach we will focus on, as it represents the current best practice for enterprise cloud management.

Callout: Managed Images vs. Azure Compute Gallery Managed Images are essentially static snapshots of a VM disk. They are simple to create but offer no lifecycle management. Azure Compute Gallery acts as a container for your images, providing a versioning system (e.g., 1.0.0, 1.0.1) and allowing you to replicate those versions to different Azure regions automatically. Always prefer the Compute Gallery for any production workload.

Preparing a Source Virtual Machine

The process of creating a custom image begins with a "Golden Image" or a "Source VM." This is the machine where you perform the installation and configuration of your software. Before you capture the image, you must ensure the OS is in a generalized state.

Step 1: Configure the Source VM

Start by deploying a standard VM from the Azure Marketplace. Connect to the machine and perform your necessary tasks:

• Install Applications: Install the software required for your specific workload (e.g., Nginx, Python, SQL Server).
• Apply Security Patches: Run the latest OS updates to ensure the image is secure at the moment of creation.
• Remove Sensitive Data: Clear out temporary files, logs, and user-specific credentials. Never bake hardcoded service account passwords into an image.
• Clean Up Logs: Delete logs in /var/log or C:\Windows\Temp to keep the image size small and prevent accidental data leakage.

Step 2: Generalization (The "Sysprep" Process)

Before Azure can turn a VM into an image, it must be "generalized." This process removes machine-specific information, such as the computer name, security identifiers (SIDs), and network interface configurations. If you do not generalize the VM, the instances created from the resulting image will conflict with one another because they will all think they are the original machine.

• For Windows: Run the Sysprep tool located in C:\Windows\System32\Sysprep\sysprep.exe. Select the "Generalize" checkbox and set the shutdown option to "Shutdown."
• For Linux: Use the Azure Linux Agent. Run the command sudo waagent -deprovision user to remove user-specific data and then sudo waagent -deprovision+user to clean up the system. Finally, shut down the machine.

Warning: Irreversible Action Generalizing a virtual machine is an irreversible process. Once you run the Sysprep tool or the Linux deprovisioning command, you will not be able to log into that VM again. Ensure you have taken a final backup or snapshot if you think you might need to revert to the pre-generalized state.

Creating the Azure Compute Gallery

Once your source VM is shut down and generalized, you are ready to move it into the Azure Compute Gallery. Think of the gallery as a library shelf where you store different versions of your custom OS templates.

Steps to create the gallery and image definition:

1. Create the Gallery: In the Azure Portal, search for "Azure Compute Gallery" and select "Create." Provide a name, a region, and a description.
2. Create an Image Definition: Inside the gallery, you need to define the type of image (e.g., "Web-Server-Image"). You will specify the OS type (Linux or Windows), the publishing details, and the recommended minimum/maximum hardware requirements.
3. Create an Image Version: This is where you point the gallery to your generalized source VM. You specify the version number (using semantic versioning, e.g., 1.0.0) and choose the target regions where this image should be replicated.

Command-Line Implementation (Azure CLI)

Using the Azure CLI is often faster and less error-prone than the portal. Here is how you create the gallery and definition using CLI commands:

# Create the Gallery
az sig create --resource-group MyResourceGroup --gallery-name MyGallery

# Create the Image Definition
az sig image-definition create \
   --resource-group MyResourceGroup \
   --gallery-name MyGallery \
   --gallery-image-definition MyImageDef \
   --publisher MyPublisher \
   --offer MyOffer \
   --sku MySku \
   --os-type Linux

# Create the Image Version from the generalized VM
az sig image-version create \
   --resource-group MyResourceGroup \
   --gallery-name MyGallery \
   --gallery-image-definition MyImageDef \
   --gallery-image-version 1.0.0 \
   --managed-image /subscriptions/.../resourceGroups/.../providers/Microsoft.Compute/virtualMachines/MySourceVM

Advanced Configuration: Image Builder

While the manual "capture" process works well for occasional updates, it is not ideal for CI/CD pipelines. If you update your application code weekly, manually spinning up a VM, configuring it, and capturing it becomes a bottleneck. This is where Azure Image Builder (AIB) comes in.

Azure Image Builder is a service that automates the entire process. You provide a JSON configuration file—a template—that tells Azure what to do. AIB then creates a temporary VM, runs your scripts, installs your software, captures the image, and cleans up the temporary resources automatically.

How an Image Builder Template Works

The template is a JSON file that defines three main components:

1. Source: The base image from the Marketplace (e.g., Ubuntu 20.04).
2. Customizer: A list of steps to execute. You can use shell scripts, PowerShell scripts, or Ansible playbooks.
3. Distributor: Where the final image should go (e.g., your Azure Compute Gallery).

Callout: Infrastructure as Code (IaC) Azure Image Builder is the bridge between manual administration and true Infrastructure as Code. By defining your image configuration in a JSON file, you can version control the template in Git, trigger builds through a pipeline, and ensure that every image is built exactly the same way every time.

Best Practices for Image Management

Creating an image is only the first step. Managing the lifecycle of these images is where many organizations struggle. Follow these industry-standard practices to maintain a healthy environment.

1. Implement Semantic Versioning

Always use a clear versioning scheme, such as Major.Minor.Patch (e.g., 2.1.4). This allows your deployment teams to understand the significance of an update. A change in the "Major" version might imply a new OS version, while a "Patch" version might simply be a security update.

2. Automate with Pipelines

Stop creating images manually. Integrate your image creation process into a tool like Azure DevOps or GitHub Actions. When a developer updates the configuration script, the pipeline should automatically trigger an Azure Image Builder task, resulting in a new, tested image version.

3. Use "Golden Image" Hygiene

Keep your source VM as lean as possible. Avoid installing unnecessary software or tools that are not required for the specific function of the VM. A smaller image is faster to deploy, easier to secure, and reduces your storage costs in the Compute Gallery.

4. Regional Replication

One of the primary benefits of the Azure Compute Gallery is its ability to replicate images across multiple regions. If you have a global application, ensure your gallery is configured to push image versions to all regions where you host VMs. This ensures that when you scale out, the images are available locally, reducing latency.

5. Security Scanning

Always run a security scan on your images before they are marked as "production ready." Use tools like Microsoft Defender for Cloud to scan your images for vulnerabilities or misconfigurations. If a vulnerability is found, you should patch the source, create a new version, and deprecate the old, insecure version.

Common Pitfalls and How to Avoid Them

Even with a good process, there are common mistakes that can lead to headaches. Being aware of these will save you significant troubleshooting time.

The "Stale Image" Problem

The Mistake: Teams continue to deploy older image versions that are months or years old. These images lack the latest OS security patches and might be incompatible with newer Azure features. The Fix: Implement a "Deprecation Policy." In the Azure Compute Gallery, you can mark older versions as "deprecated." This prevents users from accidentally selecting them while keeping the images available for legacy systems that might still require them.

Oversized OS Disks

The Mistake: Users create a massive OS disk (e.g., 500GB) for the source VM, and then capture that image. Every single VM created from that image will now have a 500GB disk, even if the actual data occupies only 10GB. The Fix: Use the smallest OS disk possible that meets your requirements. If you need more storage, attach an additional Data Disk to the VM after deployment.

Hardcoded Configurations

The Mistake: Baking environment-specific configurations into the image. For example, hardcoding a database connection string or a development API key into a config file inside the image. The Fix: Use environment variables, Azure Key Vault, or configuration management tools (like Puppet, Chef, or Ansible) to inject environment-specific settings at runtime. The image should be generic; the configuration should be provided by the environment.

Ignoring the Linux/Windows Agent

The Mistake: Forgetting to install or update the Azure VM Agent (waagent for Linux or Azure VM Guest Agent for Windows) before capturing. The Fix: The agent is the communication bridge between the Azure platform and your VM. If it is broken or missing, you won't be able to run custom scripts or perform guest-level management. Always verify the agent is running and up-to-date before generalization.

Quick Reference: Image Creation Workflow

Frequently Asked Questions

Q: Can I convert a standard VM into a custom image without deleting the original?

A: Yes. When you capture an image, you have the option to "Capture" the VM, which effectively turns it into an image. However, be aware that this process usually shuts down and deallocates the VM. If you need to keep the original VM running, create a snapshot of the OS disk instead and create an image from that snapshot.

Q: How many versions should I keep in my gallery?

A: There is no hard limit, but you should adopt a retention policy. Keeping the last 3-5 stable versions is usually sufficient for most organizations. You can use Azure Policy to automatically clean up older versions after a certain period.

Q: Can I share my images across different Azure subscriptions?

A: Yes, the Azure Compute Gallery supports cross-subscription sharing. You can share your gallery with other subscriptions within your Azure Active Directory tenant, which is ideal for large organizations with separate dev, test, and production subscriptions.

Q: Does creating an image cost money?

A: Yes, you pay for the storage used by the image versions in the gallery. You are charged based on the amount of data stored and the number of regions the image is replicated to.

Summary and Key Takeaways

Creating and managing custom VM images is a critical skill for any cloud administrator. By moving away from manual configuration and adopting a structured approach using the Azure Compute Gallery, you ensure that your infrastructure is reliable, secure, and scalable.

Key Takeaways to Remember:

1. Consistency is King: Custom images are the most effective way to eliminate configuration drift across your server fleet.
2. Generalization is Mandatory: Always run Sysprep (Windows) or the deprovisioning agent (Linux) to ensure your images are clean and ready for deployment.
3. Use the Compute Gallery: Abandon legacy Managed Images in favor of the Azure Compute Gallery to benefit from versioning and regional replication.
4. Automate Everything: Transition from manual image creation to automated builds using Azure Image Builder to integrate with your CI/CD pipelines.
5. Practice Lifecycle Management: Use image versioning and deprecation to ensure that your users are always deploying the most secure and up-to-date templates.
6. Security First: Treat your images as production assets. Scan them for vulnerabilities and ensure that no sensitive data or credentials are baked into the disk.
7. Keep it Simple: Build lean images. Offload configuration and environment-specific data to external services like Key Vault or configuration management tools.

By following these principles, you will transform your Azure environment from a collection of manually maintained servers into a robust, automated ecosystem that can scale to meet any business demand. Start small by building a single image manually, then iterate until you have a fully automated pipeline that handles your images with the same rigor as your application code.