Advanced Computer Vision: Custom Vision Training on Azure

Introduction to Custom Vision

In the realm of artificial intelligence, pre-trained models—those built to recognize generic objects like "a car," "a dog," or "a tree"—are incredibly useful for general-purpose tasks. However, real-world business requirements often demand higher specificity. You might need to identify specific industrial parts on an assembly line, detect unique plant diseases in an agricultural setting, or classify distinct types of retail products that look remarkably similar to the untrained eye. This is where Custom Vision comes into play.

Custom Vision is a service within the Azure AI Vision suite that allows developers to build, deploy, and improve their own image classifiers and object detectors. Instead of relying on a one-size-fits-all model, you provide the training data, and Azure handles the underlying machine learning complexity. This capability is vital because it democratizes access to state-of-the-art computer vision technology, enabling businesses to solve niche problems without needing a team of PhD-level data scientists to architect neural networks from scratch.

Understanding how to effectively train these models is a critical skill. It involves more than just uploading images; it requires thoughtful data curation, strategic labeling, and iterative performance evaluation. By the end of this lesson, you will understand the end-to-end lifecycle of Custom Vision projects, from initial data preparation to model deployment and optimization.

The Core Concepts of Custom Vision

Before diving into the mechanics, we must distinguish between the two primary types of projects you can build with Custom Vision: Image Classification and Object Detection.

Image Classification

Image classification involves assigning a single label (or sometimes multiple tags) to an entire image. For example, if you are building a system to sort recycling, the model looks at an image and decides if it is "plastic," "glass," or "paper." The model does not necessarily care where in the image the object is, only that the image as a whole belongs to a specific category.

Object Detection

Object detection is more granular. It involves identifying the location of objects within an image by drawing bounding boxes around them and assigning a label to each box. In an industrial inspection scenario, you might need to detect a "missing screw" and a "misaligned gasket" on a single circuit board. Because the model must identify both the class and the spatial coordinates, object detection is generally more complex to train and requires more precise data labeling.

Callout: Classification vs. Detection While classification answers the question "What is this image?", object detection answers "What is in this image and where is it located?". Choose classification when you have a single primary subject per image and detection when you need to count, locate, or identify multiple distinct objects within the same frame.

Step-by-Step: Preparing Your Training Data

The quality of your model is directly proportional to the quality of your training data. This is the "garbage in, garbage out" principle in action. If your training images are blurry, poorly lit, or mislabeled, your model will struggle to perform in production.

Data Collection Best Practices

You should aim for a representative sample of the data your model will encounter in the real world. If your system will operate in a low-light warehouse, your training images should include low-light conditions. If the camera angle will be top-down, ensure your training data reflects that perspective.

• Diversity: Include images from different angles, lighting conditions, and backgrounds.
• Quantity: While you can start with as few as 15–50 images per tag, more is usually better. Aim for at least 200–500 images per category for production-grade accuracy.
• Consistency: Ensure that the images are clear and that the subject is easily identifiable by a human observer.
• Negative Samples: Include images that do not contain any of your target objects. This helps the model learn what the object is not, reducing false positives.

Labeling Strategy

Labeling is a manual process that requires consistency. If you decide that a "defective component" includes a scratch, make sure every image with a scratch is labeled consistently. If you label a scratch in one image but ignore it in another, the model will receive conflicting signals, leading to confusion during the training phase.

Note: Use the Azure Custom Vision portal’s built-in tagging interface to manage your labels. For large datasets, consider using Azure Machine Learning Data Labeling projects, which offer advanced tools for team-based annotation and automated labeling assistance.

Building a Custom Vision Project

You can interact with Custom Vision through the web-based portal or the Azure SDK for Python. For most initial experiments, the portal is the most intuitive starting point.

Creating the Resource

1. Navigate to the Azure Portal.
2. Search for "Custom Vision" and create a new resource.
3. Choose between "Training" and "Prediction" resources. You can create both under the same resource group for easier management.
4. Once created, visit the Custom Vision portal to create your project.

Configuring Project Settings

When you create a project, you must select the appropriate "Domain." Domains are pre-optimized model architectures for specific scenarios:

• General: Good for most common objects.
• Retail: Optimized for objects found on store shelves.
• Food: Optimized for food items.
• Landmarks: Optimized for static monuments.
• Retail/Manufacturing: Specialized for high-precision scenarios.

Choosing the right domain is crucial because it sets the starting point for your model's neural network weights. If you are building a system for industrial inspection, the "General (Compact)" or "Retail/Manufacturing" domains are usually the best starting points.

Training and Evaluation

Once your images are uploaded and tagged, the next step is to trigger the training process.

Training Types

• Quick Training: Fast, but uses a simpler model architecture. Best for early iterations.
• Advanced Training: Takes longer, but allows you to specify a budget (time) for the training process. This is necessary for achieving the highest possible accuracy.

Interpreting Evaluation Metrics

After the training process completes, Azure will provide you with three critical metrics:

1. Precision: Out of all the times the model predicted a label, how many were correct? High precision means the model is rarely wrong when it makes a guess.
2. Recall: Out of all the actual instances of a label in your test set, how many did the model find? High recall means the model is good at finding all the objects that exist.
3. mAP (Mean Average Precision): This is a combined score that balances precision and recall, providing a single number to judge the overall effectiveness of your model.

Tip: If your precision is high but your recall is low, your model is "cautious"—it only predicts when it is very sure, but it misses many objects. If your recall is high but precision is low, your model is "trigger-happy"—it finds everything, but makes many mistakes. Adjust your probability threshold in the portal to find the right balance for your specific use case.

Programmatic Interaction with the SDK

While the web portal is excellent for manual tasks, professional workflows require automation. Using the Azure AI Vision SDK for Python allows you to integrate model training and prediction into your CI/CD pipelines.

Installing the SDK

pip install azure-cognitiveservices-vision-customvision

Example: Uploading Images via Python

This script demonstrates how to authenticate and upload images to a project.

from azure.cognitiveservices.vision.customvision.training import CustomVisionTrainingClient
from msrest.authentication import ApiKeyCredentials

# Configuration
ENDPOINT = "YOUR_ENDPOINT"
TRAINING_KEY = "YOUR_TRAINING_KEY"
PROJECT_ID = "YOUR_PROJECT_ID"

# Authenticate
credentials = ApiKeyCredentials(in_headers={"Training-key": TRAINING_KEY})
trainer = CustomVisionTrainingClient(ENDPOINT, credentials)

# Uploading an image
with open("image_01.jpg", "rb") as image_contents:
    trainer.create_images_from_data(
        PROJECT_ID,
        image_contents,
        tag_ids=[tag_id] # Replace with actual tag ID
    )

Explanation of the Code

1. Authentication: We use ApiKeyCredentials to authorize our requests against the Azure service.
2. Client Initialization: The CustomVisionTrainingClient serves as the primary interface for managing projects, tags, and images.
3. Data Upload: The create_images_from_data function is the workhorse. It takes the project ID and the binary data of the image. Note that you must have a tag_id ready; you can fetch these IDs using the get_tags() method.

Best Practices for Model Optimization

Achieving 90%+ accuracy is rarely a "one-shot" process. It requires an iterative cycle of training, testing, and refining.

The Iterative Loop

1. Train: Build the first version of your model.
2. Test: Use a "validation set"—a collection of images the model has never seen before.
3. Analyze: Look at the images the model got wrong. Is it misidentifying objects because of background clutter? Is it failing on specific angles?
4. Curate: Add more images of the types the model is failing to identify.
5. Re-train: Repeat the process.

Handling Class Imbalance

If you have 1,000 images of "screws" and only 50 images of "missing screws," the model will become biased toward predicting "screws" every time. This is known as class imbalance. To mitigate this, ensure you have a relatively equal number of images for every tag you define. If you cannot get more images for a rare class, consider using data augmentation (rotating, flipping, or slightly color-shifting existing images) to artificially expand your dataset.

Callout: Data Augmentation Data augmentation is a technique used to artificially increase the size of a training dataset by creating modified versions of images. Simple transformations like horizontal flips, slight zooms, or brightness adjustments can help the model become more invariant to these changes, leading to better performance on real-world data.

Deployment Options

Once your model is performing well, you need to expose it to your applications. Custom Vision offers two primary deployment paths:

Cloud API (Prediction Resource)

You publish your model to a prediction endpoint. Your application sends an HTTP request with an image, and the model returns the classification or bounding box coordinates. This is ideal for applications with consistent internet connectivity and where latency requirements are not extremely strict.

Exportable Models (Edge Deployment)

For scenarios where latency is critical or internet connectivity is unreliable (e.g., a camera on a moving vehicle or a factory floor machine), you can export your model. Azure supports exporting to:

• TensorFlow: For Android or general mobile apps.
• CoreML: For iOS devices.
• ONNX: A vendor-neutral format that runs on various hardware, including Raspberry Pi or NVIDIA Jetson devices.
• Docker: Run the model as a local container, allowing you to build an API that runs entirely on your local infrastructure.

Common Pitfalls and How to Avoid Them

1. Overfitting

Overfitting happens when a model learns the training data too well, including the noise. If your training accuracy is 99% but your real-world performance is 60%, you are likely overfitting.

• Solution: Increase the diversity of your training data. Add more "negative" samples (background images).

2. Ignoring Lighting and Environment

A model trained on perfectly lit studio photos will fail in a dimly lit office.

• Solution: Collect your training data in the actual environment where the model will be deployed. If the environment changes over time, plan to re-train the model periodically.

3. Inconsistent Labeling

If one person labels "defective" as a small crack and another person ignores small cracks, the model will never converge on a reliable definition of "defective."

• Solution: Create a clear, written guide for what constitutes each tag. If working in a team, have one person perform a quality check on the labels of another.

4. Too Few Images

Starting with 10 images per tag is rarely enough for a robust production system.

• Solution: Start small for prototyping, but commit to gathering hundreds of images before moving to production.

Comparison Table: Training vs. Prediction

Advanced Considerations: Active Learning

Active learning is a strategy to optimize your labeling efforts. Instead of labeling thousands of random images, you start with a small set, train a model, and then have that model "suggest" which images it is most uncertain about. You then focus your manual labeling efforts only on those difficult images.

Many advanced teams use this to achieve high accuracy with significantly less effort. By focusing on the "edge cases"—the images that represent the boundaries of your classes—you provide the model with the most valuable information possible. This is significantly more efficient than labeling images that the model can already classify correctly with high confidence.

Summary and Key Takeaways

Custom Vision on Azure provides a powerful, accessible path to building tailored computer vision solutions. By following the structured approach outlined in this lesson, you can move from a business problem to a deployed, accurate model with confidence.

Key Takeaways:

• Data is Paramount: The quality and diversity of your training data are the most significant factors in model performance. Spend more time on data curation than on model configuration.
• Iterative Process: Do not expect perfection in the first training run. Treat model development as an iterative cycle of testing, analyzing failures, and adding targeted data.
• Choose the Right Domain: Selecting the correct domain (e.g., Retail, General, Manufacturing) provides a strong architectural foundation for your specific task.
• Balance Precision and Recall: Understand the trade-off between being "cautious" and "trigger-happy." Use the probability threshold settings to align the model with your business requirements.
• Consider Deployment Early: Determine whether you need cloud-based inference or edge-based deployment at the start of your project, as this impacts whether you need to select an "exportable" domain.
• Monitor and Maintain: A model is not a "set it and forget it" asset. Real-world conditions drift over time; monitor your production performance and re-train as necessary to account for environment changes.
• Avoid Overfitting: If your model performs exceptionally well in the lab but poorly in the field, prioritize adding more diverse, real-world data and negative samples to your training set.

By mastering these concepts, you are well-equipped to solve complex visual identification challenges, providing your organization with custom AI solutions that are both effective and maintainable.