Lesson: Cosmos DB Container and Item Operations

Introduction: Mastering Data Management in the Cloud

Azure Cosmos DB is a globally distributed, multi-model database service that allows you to build highly responsive and scalable applications. Unlike traditional relational databases that rely on fixed schemas and rigid table structures, Cosmos DB is designed to handle massive amounts of data with low latency. At the core of this service are two fundamental concepts: containers and items. Understanding how to interact with these entities programmatically is the most critical skill for any developer working within the Azure ecosystem.

When you build applications for the cloud, you are often faced with the challenge of managing data that grows unpredictably. Whether you are building an e-commerce platform that needs to track millions of user orders or a social media feed that handles constant streams of updates, the way you structure your containers and manipulate your items determines the performance, cost, and maintainability of your application. This lesson will guide you through the intricacies of managing these resources using the Azure Cosmos DB .NET SDK, providing you with the technical foundation to build efficient, production-ready data layers.

Understanding the Cosmos DB Hierarchy

Before diving into code, it is essential to visualize how data is organized within Cosmos DB. The hierarchy follows a strict structure that defines the scope of your operations: Account > Database > Container > Item.

• Account: The top-level resource representing your instance of Cosmos DB. It holds your global distribution settings, security configurations, and primary keys.
• Database: A logical container for one or more containers. It acts as a namespace for your collections.
• Container: The fundamental unit of scalability. This is where your data lives. It is schema-agnostic, meaning you can store documents with different structures in the same container.
• Item: The individual piece of data stored within a container. In the context of the SQL API, these are JSON documents.

Callout: Containers vs. Tables In traditional relational databases, you might think of a table as a rigid structure where every row must have the same columns. In Cosmos DB, a container is more fluid. While you define a partition key for the container, the items themselves do not need to share the same schema. This "schema-less" nature allows you to store different types of objects—such as users, orders, and products—within the same container if your design requirements demand it.

Setting Up the Environment

To interact with Cosmos DB, you must first establish a connection using the CosmosClient class. This client is thread-safe and designed to be a singleton in your application. Creating a new client for every request is a common mistake that can lead to socket exhaustion and significant performance degradation.

Initializing the Client

You will need the endpoint URL and the primary key from your Azure portal. It is best practice to store these in environment variables or a secure vault, rather than hardcoding them into your source code.

using Microsoft.Azure.Cosmos;

string endpoint = Environment.GetEnvironmentVariable("COSMOS_ENDPOINT");
string key = Environment.GetEnvironmentVariable("COSMOS_KEY");

CosmosClient client = new CosmosClient(endpoint, key, new CosmosClientOptions()
{
    SerializerOptions = new CosmosSerializationOptions() { PropertyNamingPolicy = CosmosPropertyNamingPolicy.CamelCase }
});

The CosmosSerializationOptions block is particularly useful. By setting the PropertyNamingPolicy to CamelCase, you ensure that your C# class properties (which typically use PascalCase) are correctly mapped to standard JSON format (which uses camelCase) within the database.

Managing Containers

Containers are where you define your throughput and partitioning strategy. Because the partition key is immutable—meaning you cannot change it after the container is created—it is vital to choose the right property for your data access patterns.

Creating a Container

When creating a container, you must specify the container name, the partition key path, and the throughput (Request Units or RUs).

Database database = await client.CreateDatabaseIfNotExistsAsync("RetailDatabase");
ContainerProperties containerProperties = new ContainerProperties("Orders", "/customerId");

// Create container with 400 RU/s
Container container = await database.CreateContainerIfNotExistsAsync(containerProperties, throughput: 400);

Best Practices for Partitioning

Choosing the right partition key is the single most important decision you will make in Cosmos DB. A good partition key should have a high cardinality, meaning there are many possible values for that key. If you choose a key with low cardinality—like "status" where the values are only "pending," "shipped," or "cancelled"—you will end up with "hot partitions" where one partition receives significantly more traffic than others, leading to throttled requests.

Note: Always prefix your partition key path with a forward slash (e.g., /customerId). This indicates to the SDK that you are referencing a property within the JSON document.

Item Operations: CRUD Basics

Once your container is ready, you can perform Create, Read, Update, and Delete (CRUD) operations. These operations are performed using the Item object within your container instance.

Creating an Item

To create an item, you define a C# object (POCO) and pass it to the CreateItemAsync method. The SDK automatically serializes your object into JSON.

public class Order
{
    public string Id { get; set; }
    public string CustomerId { get; set; }
    public decimal Total { get; set; }
}

Order newOrder = new Order { Id = "order_123", CustomerId = "user_abc", Total = 99.99m };
ItemResponse<Order> response = await container.CreateItemAsync(newOrder, new PartitionKey(newOrder.CustomerId));

Reading an Item

Reading an item is extremely efficient if you know both the id and the partition key. This is known as a "point read." Point reads are the fastest way to retrieve data and cost significantly less in terms of Request Units compared to queries.

ItemResponse<Order> response = await container.ReadItemAsync<Order>("order_123", new PartitionKey("user_abc"));
Order order = response.Resource;

Updating an Item

Cosmos DB uses an "upsert" or "replace" model. The ReplaceItemAsync method replaces the entire document with the new version provided.

order.Total = 105.50m;
await container.ReplaceItemAsync(order, order.Id, new PartitionKey(order.CustomerId));

Warning: Be cautious when using ReplaceItemAsync. If you only want to update a single field, you must first read the full document, modify the property locally, and then send the entire document back. If you have a large document with many fields, this can be inefficient.

Advanced Item Operations: Patching

To avoid the overhead of reading and replacing full documents, you can use the Patch API. The Patch API allows you to send partial updates to an existing document. This is highly efficient for large documents or scenarios where multiple processes might be updating different parts of the same document simultaneously.

List<PatchOperation> operations = new List<PatchOperation>
{
    PatchOperation.Replace("/total", 110.00m),
    PatchOperation.Add("/status", "Processed")
};

await container.PatchItemAsync<Order>("order_123", new PartitionKey("user_abc"), operations);

The Patch API is a powerful tool for reducing throughput consumption. By only sending the changes, you reduce the payload size and the number of RUs required for the operation.

Querying Data with SQL

While point reads are best for retrieving specific items, you will often need to query across multiple items. Cosmos DB supports a SQL-like syntax that allows you to filter, project, and sort your data.

Executing a Query

Queries are executed using GetItemQueryIterator. This method returns an iterator that allows you to page through results efficiently.

QueryDefinition queryDefinition = new QueryDefinition("SELECT * FROM c WHERE c.customerId = @cid")
    .WithParameter("@cid", "user_abc");

using FeedIterator<Order> resultSet = container.GetItemQueryIterator<Order>(queryDefinition);

while (resultSet.HasMoreResults)
{
    FeedResponse<Order> response = await resultSet.ReadNextAsync();
    foreach (Order order in response)
    {
        Console.WriteLine($"Order ID: {order.Id}");
    }
}

Best Practices for Querying

• *Avoid SELECT : Only select the fields you need. Projecting only specific fields reduces the data transferred over the wire and lowers the RU cost.
• Use Parameters: Always use parameterized queries to prevent SQL injection and allow the query engine to cache the execution plan.
• Filter by Partition Key: Whenever possible, include the partition key in your WHERE clause. This allows the query to be scoped to a single partition, making it significantly faster and cheaper.

Callout: Request Units (RUs) In Cosmos DB, you don't pay for CPU or RAM directly. You pay for Request Units (RUs). One RU is the cost of reading a 1KB document by its ID. Every operation—whether it's a read, a write, or a complex query—has an RU cost. Understanding how to minimize these costs through point reads, indexing, and efficient partitioning is the key to optimizing your cloud spend.

Handling Concurrency with ETags

In distributed systems, multiple users or processes might attempt to modify the same document at the same time. This can lead to the "lost update" problem, where the last person to save their changes overwrites everyone else's work without knowing it. Cosmos DB uses ETags to handle optimistic concurrency.

Every item in Cosmos DB has an _etag property. When you read an item, you get its current ETag. When you perform an update, you can include that ETag in the request. If the ETag in the database has changed since you last read the document, the operation will fail with a 412 Precondition Failed error.

ItemResponse<Order> response = await container.ReadItemAsync<Order>("order_123", new PartitionKey("user_abc"));
string eTag = response.ETag;

// Attempt to update with the original ETag
await container.ReplaceItemAsync(order, order.Id, new PartitionKey(order.CustomerId), new ItemRequestOptions { IfMatchEtag = eTag });

If another process updated the document in the meantime, the ETag would no longer match, and the SDK would throw an exception, allowing you to handle the conflict gracefully (perhaps by re-reading the document and asking the user to resolve the conflict).

Common Pitfalls and Troubleshooting

Even experienced developers can run into issues when working with Cosmos DB. Being aware of these pitfalls can save you hours of debugging.

1. The "Hot Partition" Problem

As mentioned earlier, choosing a partition key with low cardinality will cause one partition to become overloaded. If your application starts throwing 429 Too Many Requests errors, it is often because your throughput is concentrated on a single partition key value.

• Solution: Re-evaluate your partition key. If you cannot change it, consider adding a "synthetic key" by combining two properties to increase the number of unique partition key values.

2. Ignoring Indexing Policies

Cosmos DB indexes every property by default. While this makes it easy to query, it also consumes more storage and RUs for writes.

• Solution: If you have a large document and only query on a few fields, customize the indexing policy to include only those specific paths. This will reduce your write costs.

3. Creating the Client inside a Loop

The CosmosClient is designed to be a long-lived object. Creating it inside a loop or a scoped request handler will cause the application to open and close connections constantly, leading to socket exhaustion.

• Solution: Register the CosmosClient as a singleton in your dependency injection container (e.g., in Program.cs or Startup.cs).

4. Over-fetching Data

Fetching entire documents when you only need a specific field is a common source of high RU usage.

• Solution: Use the projection features in your SQL queries or leverage the Patch API to retrieve or update only the necessary data points.

Quick Reference: Comparison Table

Step-by-Step: Implementing a Robust Data Access Layer

To ensure your application is maintainable, follow these steps to structure your data access.

1. Define Models: Create plain C# classes that represent your domain entities. Use attributes like [JsonProperty("id")] to control serialization.
2. Configure Dependency Injection: Register the CosmosClient as a singleton.
3. Create a Repository Pattern: Wrap your container operations in a repository class. This separates your business logic from the database-specific SDK code.
4. Implement Error Handling: Use try-catch blocks around all database calls to handle CosmosException. Check for specific status codes like 429 (throttling) or 404 (not found).
5. Enable Logging: Use the built-in diagnostic logging in the Cosmos SDK to monitor the RU cost of your operations.

Example: Repository Pattern Structure

public class OrderRepository : IOrderRepository
{
    private readonly Container _container;

    public OrderRepository(CosmosClient client)
    {
        _container = client.GetContainer("RetailDatabase", "Orders");
    }

    public async Task<Order> GetOrderAsync(string id, string customerId)
    {
        try
        {
            ItemResponse<Order> response = await _container.ReadItemAsync<Order>(id, new PartitionKey(customerId));
            return response.Resource;
        }
        catch (CosmosException ex) when (ex.StatusCode == System.Net.HttpStatusCode.NotFound)
        {
            return null;
        }
    }
}

Best Practices Checklist

• Always use Async: Cosmos DB is I/O intensive. Blocking calls will lead to poor performance.
• Monitor RU usage: Use the Azure Portal metrics to track your RU consumption and adjust your throughput settings as needed.
• Implement Retry Logic: The .NET SDK has built-in retry logic for transient errors. Ensure your configuration allows for sufficient retries during periods of high load.
• Use Change Feed: If you need to react to changes in your data (e.g., sending an email after an order is created), use the Change Feed feature rather than polling the database.
• Optimize for Writes: If your application is write-heavy, consider using bulk execution mode, which allows the SDK to group multiple operations into a single request.

Advanced: Understanding Throughput (Autoscale vs. Manual)

When you create a container, you must decide how to provision throughput. You have two main choices:

• Manual Throughput: You provision a fixed amount of RUs (e.g., 400 RUs). You pay for this amount regardless of whether you use it. This is suitable for predictable, steady workloads.
• Autoscale Throughput: You set a maximum RU limit. Cosmos DB will automatically scale your throughput up or down based on your application's demand, between 10% of the max and the max value. This is ideal for unpredictable or bursty workloads.

Note: Autoscale is generally more cost-effective for applications with variable traffic patterns, as you aren't paying for idle capacity during quiet periods.

Summary and Key Takeaways

Mastering container and item operations is the foundation of building successful applications on Azure Cosmos DB. By focusing on efficient data modeling, correct partitioning, and leveraging the right SDK methods, you can ensure your application remains fast and cost-effective as it scales.

Here are the key takeaways from this lesson:

1. Hierarchy Matters: Always remember the hierarchy of Account, Database, Container, and Item. Your operations are scoped within these levels.
2. Partitioning is Critical: The partition key is the most important design decision. It determines how your data is distributed and how efficiently your queries will run. Choose a key that provides high cardinality.
3. Prioritize Point Reads: Whenever you have the ID and partition key, use ReadItemAsync instead of a SQL query. It is the cheapest and fastest way to interact with your data.
4. Leverage Partial Updates: Use the Patch API to modify specific fields in a document, which minimizes data transfer and RU consumption compared to replacing the entire document.
5. Handle Concurrency: Use ETags and optimistic concurrency to ensure data integrity in multi-user environments.
6. Manage Throughput Wisely: Choose between Manual and Autoscale throughput based on your workload's predictability and budget.
7. Structure for Longevity: Use the repository pattern to keep your code clean, testable, and separate from the specific implementation details of the Cosmos DB SDK.

By applying these principles, you will be well-equipped to handle the data management requirements of any modern, cloud-native application. Continue to experiment with the SDK, monitor your RU usage, and refine your data models as your application evolves.