Azure Functions and Correlation Patterns

Correlation patterns holds an important place when you’re dealing with distributed systems. In every software system, it’s very important to be able to identify problems as quickly as possible. When it comes to distributed systems, since an event chain can span across multiple independent systems, you are very likely to spend some extra effort to have observability over a single event chain. CorrelationId pattern comes into play exactly here. I’m not planning to tell you what it is and why it is important, there are great articles out there does that. In this article, I’m planning to explain how you can incorporate correlation id into Azure Functions runtimes in your distributed systems.

There are two main problems you’re trying to solve when it comes to implementing correlation id properly:

• to be able to incorporate continuity of correlation id between downstream and upstream requests of the same event chain.
• to be able to enrich logs produced across the stack with correlation id, with minimum effort.

In this article, I’m mostly going to focus on second problem. Aim is to make sure that every log produced on the codebase is enriched with the correct correlation id behind the scenes.

One of the architectural improvements that came with .Net Core and continued with .Net 5 on Asp.Net was the middleware logic regarding the app lifecycle. You can read about it here in detail. Usually, middleware is how people choose to integrate correlationId handling and log enrichments. It’s overall a very good architectural improvement of what traditionally has been done with Filters in .Net Framework. Also at Beerwulf, we use middleware to incorporate correlation id on our Asp.Net Core runtimes.

The problem with Azure Functions is that it doesn’t have a middleware pattern (at least not yet, I haven’t checked the roadmap or discussions on this). There are couple of alternatives you can follow and I’m not going to claim that the solution I’m going to showcase is a good pattern on all cases. We realized that other patterns to implement correlation id required a pattern that we’d like to avoid as much as possible: a static DI locator. Thus we looked for a solution that didn’t involve a static DI locator and ended up with this.

Function Invocation Filters

While looking for different mechanism that could potentially replace the middleware pattern on Azure Functions, we ran into function invocation filters. It’s basically couple of events in the function lifecycle that’s provided for custom implementations. You can implement IFunctionInvocationFilter interface that’s coming from Microsoft.Azure.WebJobs.Host namespace.

One thing you’ll notice is that it’s marked as obsolete with following message: “Filters is in preview and there may be breaking changes in this area”. Initially, we didn’t want to go down this route due to the message. After some research of discussions on Azure Functions repository, we’ve concluded that it doesn’t pose a significant problem for our codebase to move on with this preview feature.

Enriching with LogContext

In this particular project, we were using SeriLog to extend logging pipeline provided by runtime. SeriLog provides enricher logic on .Net Framework and also as far as I know on .Net Core/5.

To be able use enrichers with SeriLog, you can change as you logging configuration as following

The problem prompted all this train of thought for us was exactly how CorrelationIdEnricher was expected to be by SeriLog. With<TEnricher> method expects TEnricher type to have a public parameterless constructor. But we wanted to inject our correlation id context and let DI mechanism to handle all that. So, if we decided to chase down on this avenue, it was more or less going to mean that we need to inject correlation id context via a static accessor for DI container. At Beerwulf, that’s a pattern we’d like to avoid at all cost.

At this point, we decided to look what other options SeriLog or native runtime logger of Azure Functions offer us. One of the things that caught our attention was LogContext. LogContext is provided by SeriLog and is scoped to the current logical thread, which uses AsyncLocal to preserve the context across async calls. You can configure SeriLog to be enriched from LogContext as following.

At this point, at least it’s a bit clearer how to enrich the logs with correlation id. But to answer how to integrate correlation id context into runtime, we need to implement function invocation filters.

Integrating correlation id into Azure Functions

This part is pretty straightforward once you know how to play with LogContext. We decided to implement a FunctionBase class which could repeatedly used across all our function classes to manage complexity centrally.

Depending on where does Azure Function stands on your chain of events, you can have couple of different implementations on how to integrate correlation id. For us, use cases were as following

• Consuming a queue message, which carries a correlation id to relay forward
• Starting point of a chain of events, which should create a correlation id and relay forward
• Middle or end point of a chain of events, which should carry correlation id forward

FunctionBase class will implement IFunctionInvocationFilter, which has two members:

Task OnExecutingAsync(
  FunctionExecutingContext executingContext,
  CancellationToken cancellationToken);
Task OnExecutedAsync(
  FunctionExecutedContext executedContext,
  CancellationToken cancellationToken);

You can find a demo implementation that does the trick for the correlation id handling as following

We’ll be revisiting this space quite frequently in coming weeks and months as it’s potentially going to change frequently with new releases of Azure WebJobs SDK. But for now, this has been a fun challenge that we wanted to share.