Consuming Change Feed with SDK

Module: Optimize Azure Cosmos DB Solution

Lesson: Consuming Change Feed with SDK

Introduction: Why the Change Feed Matters

In modern distributed systems, data is rarely static. Applications need to react to changes in real-time—whether that involves updating a search index, triggering a notification, or replicating data to a secondary storage system. Azure Cosmos DB provides a built-in feature called the "Change Feed" to handle these requirements. The Change Feed is essentially an ordered list of modifications to documents within a Cosmos DB container, sorted by the time they were modified.

Why does this matter? Without a change feed, developers often resort to "polling," where an application periodically queries the database for new records. Polling is inefficient; it consumes unnecessary Request Units (RUs), introduces latency, and puts undue load on your database. By using the Change Feed, your application becomes event-driven. You only process data when it actually changes, and the SDK handles the heavy lifting of tracking your progress. Mastering the Change Feed is essential for building scalable, responsive, and cost-effective applications on the Azure platform.

Understanding the Mechanics of the Change Feed

The Change Feed works by listening to a Cosmos DB container for any changes. When a document is inserted or updated, the change is appended to the feed. It is important to note that the Change Feed does not record deletions by default; for that, you would need to implement a "soft delete" pattern where you mark a document as deleted rather than removing it entirely.

The feed is persistent, meaning you can process changes from the beginning of time, or you can start from a specific point in time. Because the feed is strictly ordered by modification time, it ensures that your downstream processes receive data in the sequence it occurred. This is critical for maintaining data consistency across different microservices or storage systems.

Callout: Change Feed vs. Polling When you use polling, your application constantly asks the database "Is there anything new?" even if the answer is "no." This burns RUs and creates latency. The Change Feed, conversely, acts as a stream. The database pushes the changes to your application (or your application pulls them as they become available), meaning you only process data that exists. This is the difference between a hungry bird waiting for a worm and a bird that only flies when it sees the worm moving.

The Change Feed Processor Library

While you can interact with the Change Feed directly using the GetChangeFeedIterator method, the recommended approach for most production scenarios is the Change Feed Processor (CFP) library. The CFP simplifies the process of managing state, handling load balancing, and ensuring fault tolerance.

When you use the CFP, you don't have to worry about where you left off or how to distribute the workload across multiple instances of your application. The library uses a "lease" container to store state information, such as which partition of the database is being processed by which instance of your application. If one instance crashes, the other instances detect the failure and automatically take over the partitions that were being handled by the failed instance.

Setting Up the Lease Container

Before you start coding, you must have a lease container. This is a separate container in your Cosmos DB account (or even a different account) that stores the checkpoints for your processor. The processor needs this to know which documents it has already processed.

Note: The lease container should generally be provisioned with a low amount of throughput, as it only stores small metadata documents. You can often use a shared throughput database to keep costs down.

Step-by-Step Implementation: Building a Processor

To implement the Change Feed Processor in a .NET environment, you need to follow a specific workflow. This workflow involves configuring the processor, defining the handler, and starting the service.

1. Define the Change Handler

The handler is a delegate that receives the list of changes. This is where your business logic lives.

// Example of a Change Handler
async Task HandleChangesAsync(
    IReadOnlyCollection<Item> changes, 
    CancellationToken cancellationToken)
{
    foreach (Item item in changes)
    {
        // Add your business logic here
        // For example: Indexing, sending emails, or data transformation
        Console.WriteLine($"Processing document: {item.Id}");
    }
}

2. Configure the Processor

You use a ChangeFeedProcessorBuilder to define how the processor behaves. This includes specifying the lease container, the source container, and the handler.

Container sourceContainer = client.GetContainer("database", "source");
Container leaseContainer = client.GetContainer("database", "leases");

ChangeFeedProcessor processor = sourceContainer
    .GetChangeFeedProcessorBuilder<Item>("myProcessorName", HandleChangesAsync)
    .WithInstanceName("worker-1")
    .WithLeaseContainer(leaseContainer)
    .Build();

3. Start the Processor

Once built, you must start the processor. This is an asynchronous operation.

await processor.StartAsync();
// Keep the application running...
await processor.StopAsync();

Advanced Configuration and Best Practices

While the basic implementation is straightforward, real-world scenarios often require tuning the processor for performance and reliability.

Tuning Batch Size

By default, the processor tries to grab as many changes as it can in a single batch. You can control this using .WithMaxItems(int). If your processing logic involves expensive operations (like calling an external API), you might want to keep the batch size smaller to avoid timeouts or memory pressure.

Handling Errors

What happens if your code throws an exception while processing a batch? By default, the processor will retry. If you don't handle the error, the processor might get stuck in a loop. It is a best practice to wrap your processing logic in a try-catch block and log errors appropriately.

Warning: The Poison Pill Pattern If a single document causes an exception every time it is processed, it becomes a "poison pill." The processor will keep retrying, and your system will be unable to move forward. Always implement logic to identify these problematic documents—perhaps by logging them to a "dead-letter" queue or moving them to a separate container for manual inspection—so the processor can skip them and continue.

Scaling Out

The beauty of the Change Feed Processor is that it is designed to be distributed. If you have a massive amount of data, you can deploy multiple instances of your application. The library will automatically divide the partitions of the source container among the available instances. If you add more instances, the workload is rebalanced automatically.

Practical Use Cases: When to Use What

To truly optimize your Azure Cosmos DB solution, you need to understand which architectural patterns fit the Change Feed.

1. Materialized Views

Often, you store data in a way that is optimized for writes (e.g., a massive JSON document). However, your UI might need a different view of that data. You can use the Change Feed to project the data into a different container that is optimized for specific read queries. This is effectively creating a materialized view in real-time.

2. Data Integration

If you need to push data from Cosmos DB into an analytics engine (like Azure Synapse or Power BI), the Change Feed is your primary tool. You can build a connector that listens to the feed and streams the data into your warehouse.

3. Triggering Notifications

If your application needs to send a push notification or an email whenever a user status changes, the Change Feed is the perfect trigger. Because the feed is reliable and ordered, you ensure that every status change is processed exactly once (in order).

Avoiding Common Pitfalls

Even experienced developers can run into issues with the Change Feed. Here are the most common mistakes and how to avoid them.

1. Ignoring Throughput Consumption Every operation in the Change Feed consumes RUs. If your processor is reading a massive amount of historical data, you might hit your container's throughput limit.

• Solution: Monitor your RU consumption using Azure Monitor. Use the RequestUnits property in the response headers to see how much each batch costs.

2. Tight Coupling Avoid putting heavy business logic directly inside the change handler. If the handler takes too long, the processor will fall behind, and you will see a lag in data processing.

• Solution: Use a producer-consumer pattern. Your change handler should simply drop the message into a queue (like Azure Service Bus or an internal Channel<T>), and a separate background worker should process the queue.

3. Incorrect Lease Container Configuration Using a lease container that is too small or improperly provisioned can lead to contention.

• Solution: Always ensure the lease container is in the same region as the source container to minimize latency. If you are running multiple microservices, ensure each one uses a unique processor name so they don't fight over the same lease documents.

Callout: Designing for Idempotency In distributed systems, it is possible for the same batch of changes to be processed more than once (e.g., if an instance crashes after processing but before saving the lease). Always write your change handlers to be idempotent—meaning that if the same data is processed twice, the end result is the same as if it were processed once. For example, instead of "Increment counter by 1," use "Set counter to X."

Deep Dive: Managing Concurrency and Throughput

When dealing with large-scale applications, you might face scenarios where the Change Feed lag starts to grow. This happens when the rate of incoming changes exceeds the rate at which your processor can handle them. To address this, you have several levers to pull.

First, consider the throughput of your source container. If the source container is hitting its RU limit, the Change Feed will be throttled. You should consider using Autoscale throughput for the source container to handle spikes in traffic. Second, verify the RU settings of your lease container. While it doesn't need much, if you have a massive number of partitions, the lease container itself could become a bottleneck if it's set to a very low RU limit.

Optimizing the Handler Logic

The most common cause of lag is inefficient code inside the HandleChangesAsync method. If you are doing I/O-bound tasks, ensure you are using async/await properly. Avoid blocking calls (like .Result or .Wait()) which can lead to thread starvation. If you are performing multiple database operations, try to batch them using transactional batches if possible.

Monitoring the Lag

You can monitor the lag by using the ChangeFeedProcessor.GetEstimatedLag method. This returns an estimate of how many documents are currently waiting to be processed. You should expose this metric via an API or a dashboard to alert your team if the lag grows beyond a certain threshold.

// Monitoring lag example
long lag = await processor.GetEstimatedLag();
if (lag > 10000)
{
    // Alert the engineering team
    Logger.LogWarning($"High change feed lag detected: {lag} documents.");
}

Implementing the Change Feed with Azure Functions

While the SDK is powerful, Azure Functions provides a "bindings" approach that makes the Change Feed even easier to implement. If you are using C#, you can use the CosmosDBTrigger.

[FunctionName("ProcessChanges")]
public static void Run(
    [CosmosDBTrigger(
        databaseName: "ToDoItems",
        containerName: "Items",
        Connection = "CosmosDBConnection",
        LeaseContainerName = "leases")] IReadOnlyList<Document> input,
    ILogger log)
{
    if (input != null && input.Count > 0)
    {
        log.LogInformation($"Documents modified: {input.Count}");
        foreach (var doc in input)
        {
            log.LogInformation($"Document Id: {doc.Id}");
        }
    }
}

This approach removes the need to manually build the processor, manage the lifecycle, or worry about the lease container configuration, as the Azure Functions runtime handles all of that for you. This is often the preferred route for event-driven microservices.

Security and Networking Best Practices

Since the Change Feed is a stream of your data, it should be treated with the same security posture as the database itself.

1. Use Managed Identities: Avoid storing connection strings in your application configuration. Use Azure Managed Identities to authenticate your application to Cosmos DB. This removes the risk of leaked credentials.
2. Private Links: Ensure your Cosmos DB account is not exposed to the public internet. Use Private Endpoints to keep all traffic within your Azure Virtual Network.
3. Least Privilege: The application consuming the Change Feed only needs Read access to the source container and Read/Write access to the lease container. Configure your RBAC roles accordingly.

Summary and Key Takeaways

The Change Feed is the backbone of event-driven architecture within Cosmos DB. By allowing your application to react to data changes in real-time rather than polling, you create a more efficient, responsive, and robust system.

Key Takeaways:

• Event-Driven Architecture: The Change Feed is a push-based model that eliminates the need for inefficient polling, saving RUs and reducing system latency.
• The Power of the Processor: Always prefer the Change Feed Processor (CFP) library over manual iteration. It provides built-in load balancing, fault tolerance, and state management that would be incredibly difficult to implement from scratch.
• Idempotency is Non-Negotiable: Because distributed systems can experience transient failures, ensure your change handlers are idempotent so that processing the same data twice does not result in corrupted state.
• Monitor Your Lag: Use the GetEstimatedLag method to keep an eye on how far behind your processor is. This is your primary indicator of system health.
• Handle Poison Pills: Always include error handling in your processing logic to identify and isolate documents that cause exceptions, preventing your entire processing pipeline from stalling.
• Scale by Partitioning: The CFP automatically balances the workload based on the partition key of your source container. Design your source container partition keys with this in mind to ensure even distribution of the processing load.
• Azure Functions as a Shortcut: For simpler projects or serverless architectures, the CosmosDBTrigger in Azure Functions is a highly effective way to consume the Change Feed with minimal boilerplate code.

By mastering these concepts, you transition from simply storing data in Cosmos DB to building an active, reactive data ecosystem. Whether you are building real-time dashboards, microservices, or complex data pipelines, the Change Feed is the tool that makes it all possible. Start small, monitor your metrics closely, and scale your processing logic as your data grows.