Locating Restore Points

Lesson: Locating Restore Points in Azure Cosmos DB

Introduction: Why Data Recovery Strategy Matters

In the world of distributed databases, the question is rarely "if" a data loss event will occur, but "when." Whether through an accidental deletion by an administrator, a faulty application logic bug that overwrites critical production data, or a catastrophic regional service outage, the ability to recover your data is the backbone of any enterprise-grade cloud architecture. Azure Cosmos DB provides sophisticated backup and restore capabilities, but these features are only as effective as your ability to identify the correct point in time to which you need to roll back.

Locating a restore point is more than just picking a timestamp from a calendar. It involves understanding the interplay between your backup mode, the retention window, and the specific state of your database at a granular level. When you are faced with a data recovery scenario, the pressure is often high, and the clock is ticking. Understanding exactly how to query for, verify, and select the right restore point is a critical skill for any database administrator or cloud architect. This lesson will walk you through the mechanisms of identifying these points, ensuring that when disaster strikes, you are prepared to act with precision.

Understanding Backup Modes in Cosmos DB

Before you can locate a restore point, you must understand the two primary backup modes offered by Azure Cosmos DB: Periodic and Continuous. Your choice of mode dictates how you search for and define a "restore point."

Periodic Backup Mode

In Periodic backup mode, backups are taken at regular intervals. The frequency and retention period are configured at the account level. When you need to restore, you are essentially looking for the most recent backup that occurred before the "incident" happened. The recovery point is a discrete snapshot.

Continuous Backup Mode

Continuous backup mode is significantly more granular. It allows for point-in-time restores (PITR) to any second within the retention window (up to 30 days). In this mode, you are not looking for a specific snapshot file; you are identifying a specific timestamp that represents the state of the database immediately preceding the data loss event.

Callout: Periodic vs. Continuous Backup Periodic backup is ideal for workloads where occasional data loss is acceptable or where RPO/RTO requirements are less stringent. Continuous backup is designed for mission-critical applications where data loss must be minimized to the lowest possible threshold, allowing for recovery to a specific second in time.

Identifying Restore Points: The Mechanics

When you initiate a restore operation, you are essentially creating a new Cosmos DB account based on the data present in the source account at a specific time. To locate the correct time, you must correlate application logs, audit logs, and database metrics.

Step 1: Correlating Incident Time to Restore Time

The most common mistake is guessing the time of an incident. If an application bug deletes records at 14:02 UTC, you cannot simply restore to 14:02, as the deletion might have already been processed. You need to identify the exact moment the event started and look for a point in time before that event.

Step 2: Using Azure Resource Graph

Azure Resource Graph is a powerful tool for exploring your resources. You can use it to verify the configuration of your backup settings, which confirms whether you are looking for a discrete snapshot or a continuous timeline.

// KQL query to identify backup mode for all Cosmos DB accounts
resources
| where type == "microsoft.documentdb/databaseaccounts"
| extend backupPolicy = properties.backupPolicy
| project name, backupPolicyType = backupPolicy.type

By running this query, you quickly verify which accounts are on Continuous mode, allowing you to proceed with a point-in-time search, versus those on Periodic mode, which require you to look for specific snapshot timestamps.

Working with Continuous Backup (PITR)

When you are in Continuous Backup mode, the "restore point" is a flexible variable. You have a window of time defined by your retention policy. To find the right point, you should use the Azure CLI or PowerShell to list available restore locations and times.

Listing Restoreable Resources via CLI

To identify the restore point, you need to check the restoreParameters of your resource. The following command helps you list the restoreable resources for a given account:

# Get the restoreable resources for a specific Cosmos DB account
az cosmosdb restorable-database-account list --instance-id <instance-id>

This command returns a list of objects that describe the account's state at various points. You will look for the startTime and endTime fields. These fields define the valid range for your recovery.

Practical Example: Finding the "Pre-Incident" Timestamp

Imagine your application suffered a mass-update error at 10:15 AM today. You need to find the latest valid restore point before that.

1. Check Logs: Review your application's telemetry (e.g., Application Insights) to see when the failed process started.
2. Verify Range: Use the Azure CLI to ensure 10:14 AM falls within the minRestoreTime and maxRestoreTime for your account.
3. Select Timestamp: Choose 10:14:30 AM as your target restore point to ensure you are before the corruption event.

Note: Always aim for a buffer of at least 30-60 seconds before the known incident time. This provides a safety margin in case the system clocks between your application server and the database were slightly out of sync.

Working with Periodic Backup

In Periodic mode, the process is slightly different because you are restricted to specific snapshots. You cannot restore to any second; you must restore to the time of the last successful backup.

Using the Azure Portal to Find Snapshots

1. Navigate to your Cosmos DB account in the Azure Portal.
2. Select the Backup & Restore blade.
3. View the list of available backups.
4. Note the specific timestamp of the backup you wish to restore.

Important Considerations for Periodic Backups

Because snapshots are taken at intervals (e.g., every 4 hours), your data loss window is defined by the interval. If you lose data at 10:00 AM and the last backup was at 8:00 AM, you will lose the changes made during those two hours. This is why it is vital to keep track of these intervals when planning your recovery strategy.

Best Practices for Locating Restore Points

Locating a restore point is a high-stakes activity. Following industry standards will help you avoid common pitfalls.

1. Maintain a "Last Known Good" Registry

Many administrators keep a log of major deployments or bulk updates. If you are performing a massive data migration or a schema update, record the exact UTC timestamp before you begin. This gives you a clear, pre-defined reference point if the operation fails.

2. Use Immutable Audit Logs

Application logs should be stored in an immutable format (like Azure Blob Storage with immutability policies). If your database is compromised, your application logs are your best source of truth for identifying when the incident occurred.

3. Test Your Recovery Regularly

A restore plan that has never been tested is not a plan; it is a hope. Conduct quarterly "fire drills" where you restore a production-like environment to a specific point in time. This forces you to practice locating the correct restore point and verifying the data integrity post-restore.

4. Monitor Restore Availability

Ensure that your retention policy matches your business requirements. If your business requires a 48-hour recovery window, but your Periodic backup is set to a 24-hour retention, you are failing your RPO (Recovery Point Objective).

Warning: Data Overlap When performing a restore, you are creating a new account. You cannot restore data directly back into the original account without first migrating it. Always plan for the storage cost and the networking overhead of moving data from the restored account back into your production environment.

Common Pitfalls and How to Avoid Them

Pitfall 1: Time Zone Confusion

One of the most frequent errors is miscalculating time zones. Cosmos DB uses UTC for all internal timestamps. If you are operating in a local time zone (e.g., EST or PST), always convert your incident time to UTC before searching for a restore point.

Pitfall 2: Neglecting the "Restore Window"

Some users assume they can restore from any point in the past. Remember that backups have a retention limit. If you need to recover a file deleted three weeks ago, but your retention is only 7 days, you will be unable to locate a valid restore point.

Pitfall 3: Not Verifying Data Consistency

After locating a restore point and initiating the restore, many users immediately point their application to the new database. Always perform a validation check on a subset of the data in the restored account to ensure the state is indeed what you expected before switching traffic.

Comparison: Locating Restore Points by Mode

Step-by-Step: Executing a Restore with a Selected Point

Once you have identified your target timestamp, follow these steps to initiate the recovery.

Step 1: Identify the Target

Suppose you identified 2023-10-27T10:14:00Z as your target. Ensure you have the resourceId of the original account.

Step 2: Initiate Restore via CLI

Use the following command to start the restoration process into a new account:

az cosmosdb restore \
    --target-database-account-name "my-restored-account" \
    --resource-group "my-resource-group" \
    --restore-source "my-original-account-resource-id" \
    --restore-timestamp "2023-10-27T10:14:00Z" \
    --location "East US"

Step 3: Verify the Restore

The restore operation is asynchronous. You can monitor the status using:

az cosmosdb show \
    --name "my-restored-account" \
    --resource-group "my-resource-group" \
    --query "provisioningState"

Once the state shows "Succeeded," you can connect to the new account and verify your data.

Deep Dive: The Role of Azure Service Health

Sometimes, a restore is not triggered by user error but by a service-level issue. If you suspect an issue with the underlying Cosmos DB service, check the Azure Service Health dashboard before attempting a restore. If there is a known regional outage, a restore might not be necessary or might even be ineffective until the service is fully restored.

Understanding the health of the infrastructure allows you to make an informed decision about whether a restore is the right path forward. If you restore while the underlying service is having issues, you risk adding latency or failure points to your recovery process.

Automation and Scripting for Large-Scale Environments

In large environments with hundreds of Cosmos DB accounts, manual restoration is unfeasible. You should script the identification of restore points.

Automating the Search

You can write a Python script using the azure-mgmt-cosmosdb library to iterate through all accounts, check their backup mode, and output the latest possible restore timestamp. This creates a "recovery catalog" that you can use in an emergency.

# Conceptual snippet for automating restore point identification
from azure.identity import DefaultAzureCredential
from azure.mgmt.cosmosdb import CosmosDBManagementClient

credential = DefaultAzureCredential()
client = CosmosDBManagementClient(credential, subscription_id="...")

def get_latest_restore_time(account_name, rg):
    # Retrieve account details and check backup policy
    account = client.database_accounts.get(rg, account_name)
    if account.backup_policy.type == "Continuous":
        # Logic to extract maxRestoreTime from account properties
        return account.properties.restore_parameters.max_restore_time
    return None

This proactive approach ensures that you aren't scrambling to find the "how-to" during an actual outage. By having a script ready to query these times, you reduce your RTO significantly.

Security Considerations for Restored Data

When you restore data, the new account is a fresh resource. It will have default security settings. You must ensure that:

1. Access Control (RBAC): The new account has the same Role-Based Access Control settings as the original.
2. Network Security: If the original account was locked down to a VNet or specific IP range, the new account must be configured with the same network rules.
3. Encryption: Ensure the encryption settings (e.g., Customer-Managed Keys) are applied if required by your compliance policies.

Failure to replicate these settings can lead to "security leakage," where your restored data is accidentally exposed to unauthorized users or systems.

Dealing with Multi-Region Accounts

If your Cosmos DB account is replicated across multiple regions, the restore process is more complex. You need to consider which region you are restoring from. Typically, you restore into a single region and then configure the replication settings for the new account to match your production environment.

Always document your production replication topology. If you need to restore, you need to know exactly how many regions you need to re-enable to get back to full production parity.

Frequently Asked Questions (FAQ)

Can I restore just one container?

Yes, in many cases, you can restore individual databases or containers rather than the entire account. When you initiate the restore, you can specify the databasesToRestore or collectionsToRestore parameters to limit the scope of the recovery.

What happens to my original account during a restore?

The original account remains untouched. The restore process creates a new account. This is a key safety feature, as it prevents accidental overwriting of the original data while you are attempting to recover.

Is there a cost to keeping an account for restore purposes?

Yes, you are billed for the storage of the restored account. Once you have verified your data and migrated what you need, ensure you delete the restored account to avoid unnecessary charges.

Can I restore across subscriptions?

Yes, you can restore a Cosmos DB account to a different subscription, provided you have the appropriate permissions in both the source and target subscriptions. This is useful for moving data to a "sandbox" or "recovery" subscription.

Key Takeaways

1. Precision is Paramount: Always correlate your restore timestamp with application logs to ensure you are restoring to the exact moment before data corruption occurred. A 60-second buffer is a recommended industry best practice.
2. Know Your Backup Mode: Understand the difference between Periodic and Continuous modes. Periodic is snapshot-based, while Continuous allows for second-by-second granularity.
3. Automate Discovery: Don't rely on manual processes during an emergency. Build scripts or use Resource Graph to maintain a catalog of valid restore points across your environment.
4. Security Parity: A restored account is a new account. Always verify that your network rules, RBAC, and encryption settings match your production standards before allowing application traffic.
5. Test Your Plan: Perform quarterly restoration drills. The time to find out that your restore process fails is during a test, not during a production outage.
6. Mind the Cost: Restored accounts are separate, billable entities. Always decommission them once the data recovery process is complete to optimize your cloud spend.
7. Document the Topology: For multi-region accounts, keep a clear record of your replication configuration to ensure you can quickly rebuild your production architecture after a restore.

By mastering the art of locating restore points, you transform your disaster recovery from a reactive, stressful event into a controlled, predictable operational task. Keep these principles in mind, and you will be well-positioned to protect your data assets against any unexpected scenario.