Referential Integrity with Change Feed

Mastering Referential Integrity with Azure Cosmos DB Change Feed

Introduction: The Challenge of Distributed Data

When you build applications using relational databases like SQL Server or PostgreSQL, you rely on foreign keys to maintain referential integrity. These constraints ensure that a child record cannot exist without a parent, and that data remains consistent across tables. However, when you move to a NoSQL architecture like Azure Cosmos DB, you trade those rigid constraints for massive horizontal scale and low-latency performance. Because Cosmos DB is designed to be partitioned across many nodes, enforcing traditional cross-partition referential integrity at the database engine level is architecturally prohibitive.

This shift leaves developers with a significant challenge: how do you ensure data consistency in a distributed system where related documents might reside in different partitions or even different collections? This is where the Azure Cosmos DB Change Feed becomes an essential tool. By treating database changes as a stream of events, you can build asynchronous processes that maintain relationships, synchronize data, and enforce integrity without blocking your main application workflows. Understanding how to use the Change Feed for these purposes is a foundational skill for any engineer working with distributed cloud data.

Understanding the Cosmos DB Change Feed

The Change Feed is a persistent log of all document modifications within a Cosmos DB container, ordered by the time they occurred. When you enable the Change Feed, it outputs a stream of documents that have been inserted or updated. Deletes are also captured, provided you have enabled "soft delete" patterns or are using the Change Feed with the "full fidelity" mode. This mechanism acts as the backbone for event-driven architectures within the Azure ecosystem.

Instead of trying to force a synchronous "transaction" that spans multiple partitions—which would significantly degrade performance—you allow the primary write to complete successfully. Then, the Change Feed notifies your secondary services (often implemented as Azure Functions) to perform the necessary updates to maintain your business logic constraints. This is the implementation of the "Saga" pattern or event-driven consistency, which is the industry standard for high-scale distributed systems.

Callout: ACID Transactions vs. Eventual Consistency In a standard relational database, ACID transactions ensure that an update to a parent record and its children happens simultaneously or not at all. In Cosmos DB, you have ACID transactions within a single logical partition. When you need to span partitions, you must move to a model of eventual consistency. The Change Feed is the mechanism that bridges the gap between the initial write and the eventual state of related data, allowing your system to remain highly available and performant while still achieving data integrity.

Implementing Referential Integrity Patterns

To effectively use the Change Feed for referential integrity, you must move away from the mindset of "blocking updates" and toward "asynchronous reconciliation." There are three primary patterns used to maintain integrity: the Denormalization Pattern, the Secondary Indexing Pattern, and the Cascading Update Pattern.

1. The Denormalization Pattern

In many cases, you don't actually need a foreign key; you need the related data to be available locally. By copying the necessary fields from a parent document into the child document, you eliminate the need for joins or cross-collection lookups. When the parent document changes, the Change Feed triggers an update to all child documents.

Example Scenario: Imagine an E-commerce system where you have Users and Orders. If a user changes their display name, you want that change reflected in all their past orders.

Step-by-Step Implementation:

1. Primary Write: Update the User document in the Users container.
2. Change Feed Trigger: An Azure Function listens to the Users container.
3. Propagation: The function queries the Orders container for all documents where the UserId matches the updated user.
4. Update: The function patches the UserName field in each relevant Order document.

Tip: Managing Throughput When propagating changes to thousands of child documents, ensure your Azure Function is configured with appropriate concurrency settings. If you perform too many updates simultaneously, you may exceed your Request Unit (RU) budget and trigger 429 "Too Many Requests" errors. Use a batching approach to update child records in logical groups.

2. The Secondary Indexing/Lookup Pattern

Sometimes, you cannot denormalize data because it changes too frequently or is too voluminous. In this case, you use the Change Feed to maintain a "lookup collection" that acts as a cross-reference index. This is essentially a materialized view of your data that is optimized for queries that your primary container cannot handle efficiently.

3. The Cascading Update/Delete Pattern

If you need to enforce a "Delete Cascade" (e.g., if a Project is deleted, all its Tasks must be deleted), the Change Feed is your primary mechanism. Because Cosmos DB does not support cross-partition cascading deletes, you must implement this logic in your middleware.

Practical Code Implementation: The Azure Function Approach

The most common way to consume the Change Feed is via an Azure Function with a Cosmos DB trigger. Below is a C# example demonstrating how to propagate a name change from a Customer record to all associated Invoice records.

public static class IntegrityFunction
{
    [FunctionName("SyncCustomerName")]
    public static async Task Run(
        [CosmosDBTrigger(
            databaseName: "StoreDB",
            containerName: "Customers",
            Connection = "CosmosDBConnection",
            LeaseContainerName = "leases",
            CreateLeaseContainerIfNotExists = true)] IReadOnlyList<Document> input,
        [CosmosDB(
            databaseName: "StoreDB",
            containerName: "Invoices",
            Connection = "CosmosDBConnection")] IAsyncCollector<dynamic> invoiceCollector,
        ILogger log)
    {
        foreach (var customer in input)
        {
            // Extract updated name
            string newName = customer.GetPropertyValue<string>("Name");
            string customerId = customer.Id;

            // Logic to find and update invoices
            // Note: In a real-world scenario, you would query the Invoices container
            // and perform the update via an SDK client.
            log.LogInformation($"Customer {customerId} changed to {newName}. Updating invoices...");
            
            // Perform the update logic here...
        }
    }
}

Explaining the Code

• Trigger: The CosmosDBTrigger attribute tells the Azure Function to watch the Customers container. It uses a leases container to keep track of where it left off in the feed, ensuring that if the function restarts, it doesn't process the same changes twice.
• The Input: The IReadOnlyList<Document> contains the documents that have been modified since the last check.
• The Workflow: Inside the loop, we extract the updated information and then interact with the Invoices container to apply the change.

Best Practices for Reliable Integration

Working with the Change Feed requires a disciplined approach to error handling and idempotency. Because distributed systems are prone to network blips and transient failures, your code must be resilient.

Make Operations Idempotent

An operation is idempotent if performing it multiple times produces the same result as performing it once. If your Azure Function fails halfway through updating a set of invoices, the Change Feed trigger will retry the batch. If your code isn't idempotent, you might end up with duplicate data or corrupted state. Always check if the update is necessary before applying it.

Monitor the "Lag"

The Change Feed is asynchronous, meaning there is a delay between the primary update and the propagation of that update. In your monitoring dashboard, track the "Change Feed Lag." This metric tells you how far behind your consumers are. If the lag starts growing, it indicates that your processing logic is slower than the rate of incoming writes, and you may need to scale out your processing instances.

Handling Deletes

By default, the Change Feed does not explicitly notify you that a document was deleted. To handle referential integrity for deletions, consider these approaches:

• Soft Delete: Add an IsDeleted boolean flag to your documents. When you "delete" a record, set this flag to true. Your Change Feed processor will see this update and can then perform the necessary cleanup operations on related records.
• Change Feed Full Fidelity: Recent versions of Cosmos DB support "full fidelity" mode, which includes information about operations (insert, replace, or delete) directly in the feed. This is the preferred modern way to handle deletions.

Warning: Avoid Infinite Loops A common mistake is to have a Change Feed trigger on Container A that updates Container B, and a trigger on Container B that updates Container A. This creates a circular dependency that can lead to an infinite loop of updates, resulting in massive RU consumption and potential system crashes. Always design your data flow to be unidirectional.

Comparing Data Integrity Strategies

Common Pitfalls and How to Avoid Them

1. Ignoring Partition Key Design

The most common mistake in Cosmos DB is choosing a partition key that causes "hot partitions." When using the Change Feed, if you have a hot partition, your Change Feed processor will struggle to keep up. Ensure your partition key provides high cardinality so that data is distributed evenly across your physical nodes.

2. Over-reliance on "Real-time" Expectations

Developers often assume that the Change Feed is "real-time." While it is very fast, it is still an asynchronous process. If your business requirements dictate that a user must see their updated name on an invoice within milliseconds of updating their profile, you may need to rethink your data modeling—perhaps by keeping the User and Invoice data in the same logical partition.

3. Failing to Handle Exceptions

What happens when the Azure Function fails to update an invoice? If you don't handle the exception, the function will keep retrying the same faulty batch, potentially blocking the entire pipeline. Always implement a "Dead Letter Queue" (DLQ) pattern. If a record fails to process after a set number of retries, move it to a separate container for manual review.

4. Ignoring Throughput Costs

Every update performed by your Change Feed processor consumes Request Units (RUs). If you have a high-traffic system, a single update to a parent document could trigger thousands of updates to child documents. This can significantly increase your monthly Azure bill. Always estimate the "amplification factor" of your operations before deploying to production.

Step-by-Step: Setting Up a Resilient Processor

If you are setting up a Change Feed processor for the first time, follow these steps to ensure a robust environment:

1. Provision the Lease Container: Before starting your function, create a dedicated container for leases. This container should have a small amount of throughput (400 RUs is usually sufficient).
2. Configure the Function App: Set the MaxItemsPerInvocation in your host.json file. A value of 100-500 is typically a good starting point to balance performance and memory usage.
3. Implement Idempotency Logic: Ensure that your update logic checks the current state of the target document. Only perform a ReplaceItemAsync if the data actually needs to change.
4. Enable Monitoring: Configure Azure Monitor and Application Insights to track the ChangeFeedLag metric. Set up alerts for when the lag exceeds a certain threshold (e.g., 5 minutes).
5. Test with Scale: Use a load testing tool to simulate a high volume of writes to your primary container. Observe how the Change Feed processor handles the load and identify where the bottlenecks occur.

The Role of Architecture in Data Integrity

Ultimately, managing referential integrity in Cosmos DB is less about the database features and more about the architecture of your application. You are moving away from a model where the database enforces constraints to a model where the application enforces business rules through event sourcing. This is a powerful shift that allows for unparalleled scale, but it requires a high degree of maturity in how you handle data lifecycles.

When you design your schema, consider the "read-heavy" versus "write-heavy" nature of your application. If your application is write-heavy, you might prefer to keep data normalized and perform joins at query time (or use the Cosmos DB SQL API's JOIN capabilities for small datasets). If your application is read-heavy, you should lean into denormalization and the Change Feed to pre-calculate the data state.

Callout: The "One-Way" Rule Always aim for a unidirectional data flow. Data should flow from the source of truth (the parent) to the derived data (the children). Never allow a child update to trigger a change in the parent unless it is an explicit, well-documented business requirement. Circular dependencies are the primary cause of instability in distributed event-driven systems.

Advanced Considerations: Handling Schema Evolution

What happens when your schema changes? If you decide to add a new field to your Invoice documents, your existing Change Feed processor might not know how to handle the new structure.

Industry standard practice suggests using versioning in your documents. Include a SchemaVersion property in every document. Your Azure Function should be written to handle multiple versions of the schema. This allows you to deploy new code that understands the new schema while the old schema still exists in the database. When a document is updated, the Change Feed processor can "migrate" the document to the latest schema version during the update process.

Summary: Key Takeaways

As we conclude this lesson, remember that the Change Feed is the most powerful tool in your Cosmos DB arsenal for maintaining data integrity in a distributed world. Here are the core principles to keep in mind:

• Embrace Eventual Consistency: Accept that your system will have a state of "near-consistency" rather than "immediate consistency." This trade-off is the price of massive scalability.
• Prioritize Idempotency: Always write your processing logic so that it can be safely re-run without causing data duplication or corruption.
• Monitor System Health: Use the Change Feed Lag metric as your primary health indicator. A growing lag is the first sign of a performance bottleneck.
• Avoid Circular Dependencies: Keep your data flow unidirectional. Parent updates child, but child should never trigger an update back to the parent in a way that creates a loop.
• Use Soft Deletes or Full Fidelity: Explicitly plan for how you will handle record deletions. Relying on implicit behavior will lead to "orphan" records in your secondary containers.
• Batch Carefully: Balance the size of your batches to optimize for throughput without hitting RU limits or memory constraints in your Azure Functions.
• Design for Schema Evolution: Include versioning in your documents so your processors can handle data as it changes over time, preventing breaking changes during deployments.

By applying these patterns, you can build systems that are not only performant and scalable but also maintain the high level of data integrity required for modern enterprise applications. The Change Feed is not just a feature; it is a design philosophy that shifts the burden of consistency from the database engine to the application logic, empowering you to build truly distributed solutions.