Lesson: Mastering Event Grid Domains and Topics

Introduction: The Architecture of Event-Driven Systems

In modern cloud computing, the ability for services to communicate asynchronously is not just a benefit; it is a necessity. As applications grow in complexity, moving away from monolithic, request-response architectures toward event-driven designs becomes essential for scalability, reliability, and loose coupling. At the heart of Azure’s event-driven ecosystem lies Azure Event Grid, a fully managed service that enables you to easily manage the flow of events from many different sources to many different destinations.

When you first start with Event Grid, you might create individual topics for each service or application. While this works for small projects, it quickly becomes an administrative burden as your organization scales. This is where Event Grid Domains come into play. Domains provide a way to manage a large number of topics under a single management umbrella. By using domains, you can centralize authentication, authorization, and configuration, allowing you to treat a collection of related topics as a single unit.

Understanding the difference between custom topics and domains—and knowing when to use each—is a critical skill for any cloud architect or developer. This lesson will guide you through the conceptual framework of Event Grid, the specific mechanics of Domains, and the practical implementation strategies you need to build scalable, enterprise-grade event-driven solutions.

Understanding Azure Event Grid Fundamentals

Before diving into Domains, we must ensure a solid grasp of the core concepts of Event Grid. At its simplest, Event Grid is an event broker. It receives events from publishers (the source) and routes them to subscribers (the handlers).

Core Components

• Events: These are the smallest units of information that describe something that has happened. An event might be "a file was uploaded to blob storage" or "a new user was created in your identity system."
• Event Sources: These are the services or applications that generate the events. This could be Azure services like Blob Storage, IoT Hub, or your own custom applications.
• Topics: A topic is an endpoint where the publisher sends the events. You can think of a topic as a category or a channel.
• Event Subscriptions: These are the rules that define which events go to which destination. A subscription can include filters, such as only sending events for a specific file type or a specific user region.
• Event Handlers: These are the services that receive the events, such as Azure Functions, Logic Apps, Webhooks, or Event Hubs.

Callout: Event Grid vs. Other Messaging Services It is common to confuse Event Grid with Service Bus or Event Hubs. Event Grid is designed for event notification and reactive programming. It is optimized for high-throughput, low-latency delivery of discrete events. Service Bus, by contrast, is for message queuing and complex messaging patterns like sessions and transactions. Event Hubs is for telemetry ingestion and high-scale data streaming. Choosing the right tool depends on whether you are reacting to state changes (Event Grid) or processing a stream of data (Event Hubs).

What are Event Grid Domains?

An Event Grid Domain is a management construct for a large number of Event Grid topics related to the same application or organization. Imagine you are building a multi-tenant application where every tenant has their own set of topics. If you created a separate Event Grid topic for each tenant, you would quickly reach the limit of topics allowed per resource group or subscription, and managing the security for thousands of individual topics would become impossible.

With a Domain, you create one resource—the Domain—and then you can create "Domain Topics" within it. These topics share the same endpoint, but they are logically separated.

Why Use Domains?

1. Simplified Management: You can apply Access Control (IAM) at the domain level rather than the individual topic level. If an entire team is responsible for a set of related topics, you grant them access to the domain once.
2. Resource Efficiency: Domains allow you to scale to thousands of topics without hitting the limits that apply to individual, standalone topics.
3. Logical Grouping: Domains provide a natural way to group events by business unit, application, or tenant, making it easier to monitor and audit event traffic.
4. Uniform Configuration: You can configure settings like private endpoints or firewall rules on the domain, and they automatically apply to all the domain topics contained within.

Implementation: Creating and Using Domains

To work with Event Grid Domains, you typically follow a workflow that involves setting up the domain, creating domain topics, and then publishing events to those specific topics.

Step-by-Step: Creating a Domain

You can create a domain using the Azure Portal, Azure CLI, or PowerShell. For developers, the CLI is often the most efficient route for automation.

1. Create a Resource Group:

az group create --name my-event-domain-rg --location eastus

2. Create the Domain:

az eventgrid domain create \
  --resource-group my-event-domain-rg \
  --name my-business-domain \
  --location eastus

3. Create a Domain Topic: Once the domain is created, you can create a topic within it. Notice that the topic is scoped to the domain.

az eventgrid domain topic create \
  --domain-name my-business-domain \
  --name tenant-one-topic \
  --resource-group my-event-domain-rg

Publishing Events to a Domain Topic

When you publish events to a domain, the topic property in the event schema must be set to the name of the domain topic. This tells Event Grid which internal "bucket" the event belongs to.

Here is a sample JSON event structure you would send to your domain endpoint:

[
  {
    "id": "12345",
    "eventType": "recordCreated",
    "subject": "myapp/records/123",
    "eventTime": "2023-10-27T10:00:00Z",
    "data": {
      "recordId": "123",
      "status": "active"
    },
    "topic": "tenant-one-topic"
  }
]

Note: The topic field in the JSON payload is mandatory when publishing to a domain. If you omit it, the service will not know which domain topic to route the event to, and the request will be rejected.

Advanced Routing and Filtering

One of the most powerful features of Event Grid is the ability to filter events at the subscription level. When using domains, you can create subscriptions that filter based on specific criteria, allowing subscribers to receive only the information they care about.

Filtering Strategies

• Subject Filtering: You can filter based on the subject field, such as "begins with" or "ends with". This is useful if you are using a single topic for multiple object types.
• Advanced Filtering: You can filter based on the actual values inside the data payload of your event. For example, if you have a status field in your event data, you can create a subscription that only triggers when status equals "critical".
• Event Type Filtering: You can subscribe only to specific event types, such as userCreated or userDeleted, ignoring all other events sent to that topic.

Example: Advanced Filtering with Azure CLI

Suppose you want a subscriber to only receive events where the department is "Engineering". You can define this during subscription creation:

az eventgrid event-subscription create \
  --source-resource-id /subscriptions/{sub-id}/resourceGroups/my-event-domain-rg/providers/Microsoft.EventGrid/domains/my-business-domain/topics/tenant-one-topic \
  --name engineering-subscriber \
  --endpoint https://mywebhook.com/api/events \
  --advanced-filter data.department StringIn Engineering

This ensures that your webhook endpoint is not overwhelmed by events that are irrelevant to the engineering team.

Best Practices for Event Grid Architecture

Designing an event-driven system requires careful planning. If you ignore these best practices, you risk creating a system that is brittle, difficult to debug, and expensive to maintain.

1. Idempotency is Mandatory

Events in a distributed system can be delivered more than once. This is known as "at-least-once" delivery. Your consumer services must be designed to be idempotent—meaning that if the same event is processed twice, the outcome remains the same. Use database constraints or unique event IDs to track processed events and prevent duplicate side effects.

2. Use Dead-Lettering

Sometimes, an event cannot be delivered to the handler (e.g., the endpoint is down, or the authentication token is expired). Always configure a dead-letter location (usually an Azure Blob Storage container) so that undelivered events are stored rather than lost. Monitor this container regularly to understand why events are failing.

3. Keep Event Payloads Lightweight

Event Grid is designed for event notification, not data transport. Do not put massive objects in the event data. Instead, include the metadata and a URI (like a link to a blob or a database record) that the consumer can use to fetch the full data if needed. This keeps your event throughput high and costs low.

4. Secure Your Endpoints

Ensure your endpoints are protected. If you are using Webhooks, validate the handshake request that Event Grid sends when you create a subscription. Use Azure Active Directory (RBAC) to control who can publish to your domains and who can subscribe to your topics.

Callout: The "Push" Model Distinction Event Grid uses a "push-push" model. It pushes events to the subscriber, and the subscriber must be ready to receive them. This is different from the "pull" model (like Kafka or Event Hubs) where the consumer dictates the pace of processing. Because of this, you must ensure your subscriber can handle the load. If your subscriber is a slow process, consider putting a queue or an Azure Function with high concurrency in front of it.

Common Pitfalls and How to Avoid Them

Pitfall 1: Over-using Topics

A common mistake is creating too many individual topics when a domain would suffice, or conversely, putting too many unrelated events into a single topic.

• How to avoid: Group topics by "domain context." If two services need to share the same security boundary and administrative lifecycle, they belong in the same domain. If they are completely independent, they should likely be in separate domains or separate topics.

Pitfall 2: Ignoring Retry Policies

Event Grid has a default retry policy, but it might not be suitable for your specific application.

• How to avoid: Configure the retry policy (the number of attempts and the time-to-live for an event) based on the volatility of your downstream services. If your service is prone to intermittent outages, increase the retry duration.

Pitfall 3: Security Misconfiguration

Leaving your Event Grid topic endpoint public without proper authentication is a security risk.

• How to avoid: Use Shared Access Signatures (SAS) for publishers, or better yet, use Managed Identities to allow Azure resources to publish to Event Grid without managing keys. Always use HTTPS for all event delivery endpoints.

Quick Reference Table: Event Grid Choices

Deep Dive: Monitoring and Troubleshooting

Even with the best design, things go wrong. Monitoring is the final pillar of a robust event-driven architecture.

Metrics to Watch

Azure Monitor provides several metrics for Event Grid that you should track:

• Publish Success/Failure: If you see a spike in publish failures, check your authentication or ensure the domain topic name is correct.
• Delivery Success/Failure: This tells you if your subscribers are successfully receiving events. A high failure rate usually points to an issue with the subscriber endpoint (e.g., 500 errors, timeouts).
• Matched Events: This tracks how many events successfully matched your subscription filters.

Using Diagnostic Logs

To troubleshoot specific events, enable diagnostic logs and send them to a Log Analytics workspace. You can then run Kusto Query Language (KQL) queries to track the lifecycle of a specific event:

AzureDiagnostics
| where ResourceProvider == "MICROSOFT.EVENTGRID"
| where Category == "DeliveryAttempts"
| project TimeGenerated, OperationName, ResultType, Topic_s, SubscriptionName_s

This query helps you identify exactly which subscriptions are failing and why, allowing for surgical debugging rather than guessing.

Practical Exercise: A Multi-Tenant Scenario

Imagine you are building a SaaS application for three different companies (Tenants A, B, and C). Each tenant needs to receive their own "orderCreated" events.

1. Domain Setup: Create a domain named SaaSOrderDomain.
2. Topics: Create domain topics TenantA, TenantB, and TenantC.
3. Publishing: Your central order service publishes to SaaSOrderDomain. In the event metadata, it sets the topic property to TenantA.
4. Subscription: Tenant A creates a subscription on the TenantA domain topic that points to their own Webhook.
5. Isolation: Tenant B cannot see or subscribe to TenantA's events because they do not have the required permissions to the TenantA topic, even though both exist within the same SaaSOrderDomain.

This architecture provides perfect isolation while keeping your infrastructure footprint small.

Summary and Key Takeaways

Event Grid Domains represent a significant step forward in managing event-driven infrastructure at scale. By grouping related topics, you reduce management overhead, simplify security, and ensure that your event architecture can grow alongside your business.

Key Takeaways for Your Architecture:

• Domains for Scale: Always prefer Domains over individual topics when managing more than a handful of related topics. They are purpose-built for organizational and multi-tenant scaling.
• Logical Separation: Use domain topics as the logical boundaries for your events. Ensure your publishers are correctly tagging events with the appropriate topic name.
• Prioritize Resilience: Always implement dead-lettering and design your subscribers to be idempotent. In a distributed, asynchronous system, you must assume delivery issues will occur.
• Security First: Use Managed Identities and RBAC to secure your event traffic. Avoid hard-coding keys or secrets in your application code.
• Monitor Proactively: Set up alerts on delivery failures. Do not wait for a user to report a missing event; use the built-in diagnostic logs to catch issues early.
• Filter at the Edge: Use advanced filtering to reduce the processing burden on your subscribers. Only send them the events they are actually interested in.
• Keep it Lightweight: Treat events as signals, not data containers. Keep the payload small and rely on external storage or APIs to fetch detailed information.

By mastering these concepts, you move from simply "sending messages" to building a sophisticated, resilient, and manageable event-driven ecosystem. Whether you are connecting internal microservices or integrating with third-party webhooks, these practices will ensure your infrastructure remains a reliable backbone for your applications.