Blob Storage Policy

Info

This guide teaches applying a Nodinite-specific policy to enable logging from the Azure API Management platform to Nodinite.

When the Blob Storage Policy is in place and active, the code running in the Azure API Management platform creates a Nodinite-specific JSON Log Event and sends it to the Blob Storage Container.

Tip

You can use the Nodinite Azure Monitoring Agent to get alerts if you stockpile Log Events. The Nodinite Pickup Log Events Service Logging Agent should consume the Log Events. You can have a backout container for malformed messages, which should also be included in the Monitoring.

The Nodinite Pickup Log Events Service Logging Agent consumes the Nodinite Log Events from the Blob Storage Container. This pattern is Asynchronous.

Tip

You can use the Nodinite Non-Events Monitoring Agent to get alerts if you have too many or too few events within a period.

Configuration Steps

Follow the steps below to enable Logging to Nodinite in your Azure API Management Service APIs:

• Add and configure a Managed Identity
• Add and configure Named Values
• Add Policy Fragments
  • Inbound Policy Fragment
  • Outbound Policy Fragment
  • On-Error Policy Fragment
• Add a reference to Policy Fragments in All APIs
  • Inbound Processing
    • Inbound Policy Fragment
  • Outbound Processing
    • Outbound Policy Fragment
  • On-Error Processing
• Variables
  • Set variable on All operations
  • Set variables on the API level (inbound)
  • Set variables on the API level (outbound)
  • Set variables on the API level (on-error)

Next, you must configure the Nodinite Pickup Log Events Service Logging Agent.

• Consume Log Events from the Blob Storage Container.
• Malformed Log Events should be sent to a backout container.

Next, configure the Nodinite Azure Monitoring Agent to ensure you are not stockpiling Log Events.

• Destination container
• Backout container

Next, create and use Nodinite Log Views with Search Fields to share insights and access to the logged data with your business and key stakeholders.

Next, you can configure the Nodinite Non-Events Monitoring Agent to validate you get the expected number of Log Events during a period.

Managed Identity

In the Blob Storage Policy, you are using a user-assigned Managed Identity.

1. Create (or reuse) a Managed Identity in the Azure Portal.
2. Note the Client ID; you need it in the Named Values section.
   
3. Add the Storage Blob Contributor role to the Managed Identity. Apply on the Storage Account pointed to by the logging-blob-url named value.
   
4. Add the Managed Identity to the API Management Service where you intend to apply and use the Blob Storage Policy.
   

Named Values

The Blob Storage Policy uses the variables to avoid hard-coding values. You can expand the sample with additional properties as your business demands and policy require.

In the Azure Portal, navigate to the API Management Service and click Named Values in the sidebar.

Create the following, and set the values:

1. logging-blob-mid-client-id - Enter the Client ID of the Managed Identity from the previous step.
2. logging-blob-url - Enter the URL for the Blob Storage Container. Ensure the value ends with a trailing slash (/). You can omit the trailing slash if you also change the code in the sample.
   

Policy Fragments

The Blob Storage Policy has three Policy Fragments.

1. Inbound Policy Fragment - Includes the code to create a Nodinite JSON Log Event for Inbound Processing.
2. Outbound Policy Fragment- Includes the code to create a Nodinite JSON Log Event for Outbound Processing.
3. On Error Policy Fragment - Includes the code to create a Nodinite JSON Log Event when there is an error in Inbound and Outbound Processing.

Tip

All events in the sample make use of the x-ms-client-tracking-id header value. This ID facilitates end-to-end logging and is vital to all Nodinite Log Events. You can easily group all Log Events for an interchange in a Nodinite Log View by the built-in Search Field Application interchange id. If you call an Azure Function using the Nodinite Serilog sinks, add this header to the context to retain the value through the whole chain of events. If you invoke a Logic App, you do not need to do anything, the system has this covered for you.

Inbound Policy Fragment

1. In the Azure Portal, navigate to the API Management Service, then, in the sidebar, click Policy Fragments.
2. Create a new Policy Fragment and name it, for example, InboundLoggingPolicyFragment.
   

The fields are all part of a Nodinite Log Event.

<fragment>
	<!-- Start Logging -->
	<set-header name="x-ms-client-tracking-id" exists-action="skip">
		<value>@(Guid.NewGuid().ToString())</value>
	</set-header>
	<set-variable name="correlationId" value="@{
            return context.Request.Headers.GetValueOrDefault("x-ms-client-tracking-id");
        }" />
	<send-request mode="new" timeout="20" response-variable-name="blobdata" ignore-error="true">
		<set-url>@($"{{logging-blob-url}}{Guid.NewGuid().ToString()}.json")</set-url>
		<set-method>PUT</set-method>
		<set-header name="x-ms-version" exists-action="override">
			<value>2019-07-07</value>
		</set-header>
		<set-header name="x-ms-blob-type" exists-action="override">
			<value>BlockBlob</value>
		</set-header>
		<set-body>@{
            var body = context.Request?.Body?.As<string>(preserveContent: true) ?? "";
            var bodyToLog = System.Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(body));

            var headers = context.Request.Headers.Where(h => !h.Key.Equals("authorization", StringComparison.InvariantCultureIgnoreCase) && !h.Key.Equals("ocp-apim-subscription-key", StringComparison.InvariantCultureIgnoreCase));
            var correlationId = context.Variables.GetValueOrDefault<string>("correlationId");

            Dictionary<string, string> contextProperties = new Dictionary<string, string>();
            contextProperties.Add("CorrelationId", correlationId);
        
            foreach (var h in headers) {
                contextProperties.Add(string.Format("Header#{0}", h.Key), String.Join(", ", h.Value));
            }
            
            var requestLogMessage = new {
                LogAgentValueId = 3,
                EndPointName = context.Api.Name + "/" + context.Operation.Name,
                EndPointUri = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
                EndPointDirection = 10,
                EndPointTypeId = 71,
                EventDirection = 21,
                OriginalMessageTypeName = context.Variables.GetValueOrDefault<string>("loggingSchemaRoot", "UndeclaredSchemaRoot") + "#" + context.Variables.GetValueOrDefault<string>("loggingSchemaName", "UndeclaredSchemaName"),
                LogDateTime = DateTime.UtcNow,
                ApplicationInterchangeId = correlationId,
                Context = contextProperties,
                LogText = context.Variables.GetValueOrDefault<string>("logtext", ""),
                Body = bodyToLog
            };

            return JsonConvert.SerializeObject(requestLogMessage);
            }</set-body>
		<authentication-managed-identity client-id="{{logging-blob-mid-client-id}}" resource="https://storage.azure.com" />
	</send-request>
	<!-- End Logging -->
</fragment>

Outbound Policy Fragment

<fragment>
	<!-- Start Logging -->
	<set-header name="x-ms-client-tracking-id" exists-action="skip">
		<value>@(Guid.NewGuid().ToString())</value>
	</set-header>
	<set-variable name="correlationId" value="@{
            return context.Request.Headers.GetValueOrDefault("x-ms-client-tracking-id");
        }" />
	<send-request mode="new" timeout="20" response-variable-name="blobdata" ignore-error="true">
		<set-url>@($"{{logging-blob-url}}{Guid.NewGuid().ToString()}.json")</set-url>
		<set-method>PUT</set-method>
		<set-header name="x-ms-version" exists-action="override">
			<value>2019-07-07</value>
		</set-header>
		<set-header name="x-ms-blob-type" exists-action="override">
			<value>BlockBlob</value>
		</set-header>
		<set-body>@{
            var body = context.Request?.Body?.As<string>(preserveContent: true) ?? "";
            var bodyToLog = System.Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(body));

            var headers = context.Request.Headers.Where(h => !h.Key.Equals("authorization", StringComparison.InvariantCultureIgnoreCase) && !h.Key.Equals("ocp-apim-subscription-key", StringComparison.InvariantCultureIgnoreCase));
            var correlationId = context.Variables.GetValueOrDefault<string>("correlationId");

            Dictionary<string, string> contextProperties = new Dictionary<string, string>();
            contextProperties.Add("CorrelationId", correlationId);
        
            foreach (var h in headers) {
                contextProperties.Add(string.Format("Header#{0}", h.Key), String.Join(", ", h.Value));
            }
            
            var requestLogMessage = new {
                LogAgentValueId = 3,
                EndPointName = context.Api.Name + "/" + context.Operation.Name,
                EndPointUri = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
                EndPointDirection = 10,
                EndPointTypeId = 71,
                EventDirection = 25,
                OriginalMessageTypeName = context.Variables.GetValueOrDefault<string>("loggingSchemaRoot", "UndeclaredSchemaRoot") + "#" + context.Variables.GetValueOrDefault<string>("loggingSchemaName", "UndeclaredSchemaName"),
                LogDateTime = DateTime.UtcNow,
                ApplicationInterchangeId = correlationId,
                Context = contextProperties,
                LogText = context.Variables.GetValueOrDefault<string>("logtext", ""),
                Body = bodyToLog
            };

            return JsonConvert.SerializeObject(requestLogMessage);
            }</set-body>
		<authentication-managed-identity client-id="{{logging-blob-mid-client-id}}" resource="https://storage.azure.com" />
	</send-request>
	<!-- End Logging -->
</fragment>

On-Error Policy Fragment

Note

The LogStatus is set to -1 to indicate an Error condition.

<fragment>
	<!-- Start Logging -->
	<set-header name="x-ms-client-tracking-id" exists-action="skip">
		<value>@(Guid.NewGuid().ToString())</value>
	</set-header>
	<set-variable name="correlationId" value="@{
            return context.Request.Headers.GetValueOrDefault("x-ms-client-tracking-id");
        }" />
	<send-request mode="new" timeout="20" response-variable-name="blobdata" ignore-error="true">
		<set-url>@($"{{logging-blob-url}}{Guid.NewGuid().ToString()}.json")</set-url>
		<set-method>PUT</set-method>
		<set-header name="x-ms-version" exists-action="override">
			<value>2019-07-07</value>
		</set-header>
		<set-header name="x-ms-blob-type" exists-action="override">
			<value>BlockBlob</value>
		</set-header>
		<set-body>@{
            var body = context.Request?.Body?.As<string>(preserveContent: true) ?? "";
            var bodyToLog = System.Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(body));

            var headers = context.Request.Headers.Where(h => !h.Key.Equals("authorization", StringComparison.InvariantCultureIgnoreCase) && !h.Key.Equals("ocp-apim-subscription-key", StringComparison.InvariantCultureIgnoreCase));
            var correlationId = context.Variables.GetValueOrDefault<string>("correlationId");

            Dictionary<string, string> contextProperties = new Dictionary<string, string>();
            contextProperties.Add("CorrelationId", correlationId);
        
            foreach (var h in headers) {
                contextProperties.Add(string.Format("Header#{0}", h.Key), String.Join(", ", h.Value));
            }
            
            var requestLogMessage = new {
                LogAgentValueId = 3,
                EndPointName = context.Api.Name + "/" + context.Operation.Name,
                EndPointUri = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
                EndPointDirection = 10,
                EndPointTypeId = 71,
                EventDirection = 25,
                OriginalMessageTypeName = context.Variables.GetValueOrDefault<string>("loggingSchemaRoot", "UndeclaredSchemaRoot") + "#" + context.Variables.GetValueOrDefault<string>("loggingSchemaName", "UndeclaredSchemaName"),
                LogDateTime = DateTime.UtcNow,
                ApplicationInterchangeId = correlationId,
                Context = contextProperties,
                LogText = context.Variables.GetValueOrDefault<string>("logtext", ""),
                Body = bodyToLog,
                LogStatus = -1
            };

            return JsonConvert.SerializeObject(requestLogMessage);
            }</set-body>
		<authentication-managed-identity client-id="{{logging-blob-mid-client-id}}" resource="https://storage.azure.com" />
	</send-request>
	<!-- End Logging -->
</fragment>

Add a reference to Policy Fragments in All APIs

In the Azure Portal, navigate to the API Management Service, then, in the sidebar, click APIs, then All APIs.

Inbound Processing

Use the Inbound Policy Fragment to create Nodinite Log Events for the Request.

Inbound Policy Fragment (Inbound Processing)

1. Click on the Add policy button.
   
2. Add a reference to the Inbound Policy Fragment.

 <inbound>
    <include-fragment fragment-id="InboundLoggingPolicyFragment" />
</inbound>

Outbound Processing

Use the Outbound Policy Fragment to create Nodinite Log Events for the Response.

Outbound Policy Fragment (Outbound Processing)

1. Click on the Add policy button.
   
2. Add a reference to the Outbound Policy Fragment.

<outbound>
    <include-fragment fragment-id="outboundLoggingPolicyFragment" />
</outbound>

On Error

Use the On-Error Policy Fragment to create Nodinite Log Events when there is an error.

<on-error>
    <include-fragment fragment-id="onerrorLoggingPolicyFragment" />
</on-error>

Complete example with all Policy Fragments

<policies>
    <inbound>
        <include-fragment fragment-id="InboundLoggingPolicyFragment" />
    </inbound>
    <backend>
        <forward-request />
    </backend>
    <outbound>
        <include-fragment fragment-id="outboundLoggingPolicyFragment" />
    </outbound>
    <on-error>
        <include-fragment fragment-id="onerrorLoggingPolicyFragment" />
    </on-error>
</policies>

Variables

In the final steps with the Blob Storage Policy, you need to set a few values. These are commonly used to attribute the Nodinite JSON Log Event. You can develop additional dynamics based on the provided examples.

• Set variable on All operations
• Set variables on the API level (inbound)
• Set variables on the API level (outbound)
• Set variables on the API level (on-error)

The following variables are part of the default Nodinite Blob Storage Policy:

Info

Using the example values, the name of the Message Type in the resulting Nodinite JSON Log Event is Nodinite.Logging.Schemas#Orders. If possible, please use a versioning scheme in real-world scenarios, i.e. Nodinite.Logging.Schemas/1.0#Orders

In the On-Error, the LogStatus is set to -1 (indicating an Error), this is a candidate you may be interested in setting as a variable. Feel free to modify the default Policy Fragments to fit your use case.

• Log Status Codes

Set variable on All operations

In the Azure Portal, navigate to the API Management Service. In the sidebar, click APIs, and then select one of the APIs, then All operations.

Complete example

Below is an example of setting the value for loggingSchemaRoot for all operations (inbound, outbound, and on-error). You can override the value if you want to on the operation level.

<policies>
    <!-- Throttle, authorize, validate, cache, or transform the requests -->
    <inbound>
        <set-variable name="loggingSchemaRoot" value="Nodinite.Logging.Schemas" />
        <base />
    </inbound>
    <!-- Control if and how the requests are forwarded to services  -->
    <backend>
        <base />
    </backend>
    <!-- Customize the responses -->
    <outbound>
        <set-variable name="loggingSchemaRoot" value="Nodinite.Logging.Schemas" />
        <base />
    </outbound>
    <!-- Handle exceptions and customize error responses  -->
    <on-error>
        <set-variable name="loggingSchemaRoot" value="Nodinite.Logging.Schemas" />
        <base />
    </on-error>
</policies>

Set variables on the API level (inbound)

For a specific operation, for example, a POST, in the inbound processing step, you can set the suffix part of the Message Type name as described in the Variables heading. Also, you can set a Log Text (a description field for the Nodinite Log Event)

 <inbound>
    <set-variable name="loggingSchemaName" value="Orders" />
    <set-variable name="logtext" value="Post OK" />
    <base />
    <set-backend-service id="apim-generated-policy" backend-id="nodinitefunctionapps" />
</inbound>

Set variables on the API level (outbound)

In the outbound processing step, we now make changes to the variables. You should set these values according to your business case and match the data in orbit.

 <outbound>
    <set-variable name="loggingSchemaName" value="OrderResponse" />
    <set-variable name="logtext" value="Response OK" />
    <base />
</outbound>

Set variables on the API level (on-error)

In the on-error processing step, we again make changes to the variables. You should set these values according to your business case and match the data in orbit.

 <on-error>
    <set-variable name="loggingSchemaName" value="Error" />
    <set-variable name="logtext" value="Error in processing" />
    <base />
</on-error>

Next Step

Search Fields
Log Views

Related

• Logging with Azure API Management Services

• JSON Log Event

• Event Hub Policy

• Log API Policy

Interested in logging from other Azure Related Services?

• Logging from Azure Logic Apps
• Logging from Azure functions