Event Hub Policy

Info

This guide teaches how to apply a Nodinite-specific policy to enable logging from the Azure API Management platform to Nodinite.

Warning

The Logging with the Event Hub option currently has a 200 KB limit. If you log larger messages than this, there will be no event logged at all(!). To ensure the proper Logging, make sure to use the option and perform the Logging using a Blob Storage Policy to the specified Azure Storage Account.

Event Hub

Your mission is to create a Nodinite Log Event and then put the result to an intermediate storage, in this case the Azure Event Hub. Next, the Nodinite Pickup Log Events Service Logging Agent transfers the logged data to the configured Nodinite instance of for use in self-service enabled Log Views.

How to Enable Nodinite logging from your APIs in Azure API Mgmt Services:

• 1. Configure one or more Event Hub Loggers
• 2. Configure the Policy within your API
• 3. Install and configure the Nodinite Pickup Log Events Service Logging Agent
  • 3a. Configure fetching the Nodinite JSON Log Event from the target Event Hub. You must provide the connection string (Create a new or re-use an existing from the 'Shared access policies' section in the Azure portal).

When the policy is configured and active, the Azure API Management platform creates a Nodinite-specific JSON Log Event that is posted to the named Event Hub. Then, the properly configured Nodinite Pickup Log Events Service Logging Agent asynchronously moves the JSON Log Event to Nodinite.

Event Hub Logger

To log using an Azure Event Hub, you must have a named Event Hub Logger; In the Policy, your refer to it by its name. You cannot change the name. If you want to change the name, you must delete it first and then re-create it with the new name. You can create it using either Nodinite or with a tool like Postman.

Tip

You can use the Nodinite Azure Monitoring Agent to manage and monitor these Event Hub Loggers.

• Monitoring

• Managing

  • Details
  • Edit
  • Delete

• Learn how to call the Azure Rest API using Postman

  • You should use the ApplicationId/ClientID, you created when you followed the Azure Application Access user guide
  • Download Azure REST API Template for Postman

You can also create the Event Hub Logger by performing a PUT method with a Body; a template is provided below:

• {{subscriptionId}}

The {loggerName} is the unique identifier for this logger in your API Management Service ({{APIMGMTServiceName}}). This name is later used in the API Policy.
Replace {loggerName} with the name of the logger, this name should be unique per Event Hub used as the target for logging. Use at least one Event Hub per Environment (Prod, QA, Test, ...)

{
  "properties": {
    "loggerType": "azureEventHub",
    "description": "Sample Event Hub Logger enabling logging from a Policy in Azure API Services to Nodinite",
    "credentials": {
      "name": "{loggerName}",
      "connectionString": "Endpoint=sb://{namespace}.servicebus.windows.net/;SharedAccessKeyName={sharedAccessKeyName};SharedAccessKey={sharedAccessKey};EntityPath={entityPath}"
    }
  }
}

Use this Body as a template and modify it according to your preferences

API Policy Configuration

Depending on what you want to log (Request and/or response), you must add a logging configuration into your policy either in the inbound, outbound or in both configurations.

The template policy configuration below logs the following items:

• Mandatory properties for a Nodinite Log Event
  • Feel free to modify content as required by your business requirements
• HTTP Headers are retained as context properties

  Info

  Some properties are excluded in the sample code

• Body

Replace the '{loggerName}' in both API management logger and the code snippets below for the Policy configuration

The example below creates a basic Nodinite JSON Log Event with some common properties, required and optional.

• Context is optional
• Body is optional

<log-to-eventhub logger-id='{loggerName}'>
    @{
    var body =  context.Request.Body.As<string>(preserveContent: true);

    var bodyToLog = System.Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(body));

    var headers = context.Request.Headers
                .Where(h => h.Key != "Authorization" && h.Key != "Ocp-Apim-Subscription-Key");

    Dictionary<string, string> contextProperties = new Dictionary<string, string>();
    foreach (var h in headers) {
        contextProperties.Add(string.Format("Header#{0}", h.Key), String.Join(", ", h.Value));
    }
    
    var requestLogMessage = new {
        LogAgentValueId = 15,
        EndPointName = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
        EndPointUri = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
        EndPointDirection = 10,
        EndPointTypeId = 71,
        OriginalMessageTypeName = "Nodinite.Schemas/Nodinite.Demo/5.0#Orders",
        LogDateTime = DateTime.UtcNow,
        Context = contextProperties,
        Body = bodyToLog
    };

    return JsonConvert.SerializeObject(requestLogMessage);
    }
</log-to-eventhub>

Example Azure Event Hub Policy

This complete policy includes a configuration to log both the inbound (Request) and the outbound (Response) information.

In this example, we will add a clientTrackingId (guid) if it was not provided in the x-ms-client-tracking-id HTTP header and some other optional properties for a Nodinite Log Event.

x-ms-client-tracking-id

<policies>
    <inbound>
        <!-- creating a tracking/correlation id -->
        <set-variable name="clientTrackingId" value="@{ 
            return Guid.NewGuid().ToString();
        }" />
        <!-- if x-ms-client-tracking-id exists use it, if not, use clientTrackingId -->
        <set-header name="x-ms-client-tracking-id" exists-action="skip">
            <value>@(context.Variables.GetValueOrDefault<string>("clientTrackingId"))</value>
        </set-header>
        <!-- Put on logger -->
        <log-to-eventhub logger-id="{loggerName}">
            @{
                var body =  context.Request.Body.As<string>(preserveContent: true); // in case we need it later...

                var bodyToLog = System.Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(body));

                var headers = context.Request.Headers
                            .Where(h => h.Key != "Authorization" && h.Key != "Ocp-Apim-Subscription-Key");

                var correlationId = headers.Any(h=>h.Key == "x-ms-client-tracking-id") ? headers.First(h => h.Key == "x-ms-client-tracking-id").Value[0] : context.Variables.GetValueOrDefault<string>("clientTrackingId"); 

                Dictionary<string, string> contextProperties = new Dictionary<string, string>();
                contextProperties.Add("Correlation Id", correlationId);
            
                foreach (var h in headers) {
                    contextProperties.Add(string.Format("Header#{0}", h.Key), String.Join(", ", h.Value));
                }
                
                var requestLogMessage = new {
                    LogAgentValueId = 15,
                    EndPointName = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
                    EndPointUri = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
                    EndPointDirection = 10,
                    EndPointTypeId = 71,
                    EventDirection=21,
                    OriginalMessageTypeName = "Nodinite.Schemas/Nodinite.Demo/5.0#Orders",
                    LogDateTime = DateTime.UtcNow,
                    LogStatus = 0,
                    ApplicationInterchangeId = correlationId,
                    Context = contextProperties,
                    LogText = "Hello World from Nodinite API mgmt logging policy",
                    Body = bodyToLog
                };

                return JsonConvert.SerializeObject(requestLogMessage);
            }
        </log-to-eventhub>
    </inbound>
    <backend>
        <forward-request />
    </backend>
    <outbound>
        <!-- if x-ms-client-tracking-id exists use it, if not, use clientTrackingId -->
        <set-header name="x-ms-client-tracking-id" exists-action="skip">
            <value>@(context.Variables.GetValueOrDefault<string>("Correlation Id"))</value>
        </set-header>
        <log-to-eventhub logger-id="{loggerName}">
            @{
                var body =  context.Response.Body.As<string>(preserveContent: true); // in case we need it later...

                var bodyToLog = System.Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(body));

                var headers = context.Response.Headers
                            .Where(h => h.Key != "Authorization" && h.Key != "Ocp-Apim-Subscription-Key");
                
                var correlationId = headers.Any(h=>h.Key == "x-ms-client-tracking-id") ? headers.First(h => h.Key == "x-ms-client-tracking-id").Value[0] : context.Variables.GetValueOrDefault<string>("clientTrackingId"); 

                Dictionary<string, string> contextProperties = new Dictionary<string, string>();
                contextProperties.Add("Correlation Id", correlationId);
                foreach (var h in headers) {
                    contextProperties.Add(string.Format("Header#{0}", h.Key), String.Join(", ", h.Value));
                }
                string logText = "";

                if (context.Response.StatusCode > 300)
                {
                    logText = "Guru meditation";
                }
                
                var requestLogMessage = new {
                    LogAgentValueId = 15,
                    EndPointName = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
                    EndPointUri = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
                    EndPointDirection = 10,
                    EndPointTypeId = 71,
                    EventDirection=25,
                    OriginalMessageTypeName = "Nodinite.Schemas/Nodinite.Demo/5.0#OrderResponses",
                    LogDateTime = DateTime.UtcNow,
                    ApplicationInterchangeId = correlationId,
                    Context = contextProperties,
                    Body = bodyToLog,
                    LogStatus = context.Response.StatusCode,
                    LogText = logText
                };
                return JsonConvert.SerializeObject(requestLogMessage);
            }
        </log-to-eventhub>
    </outbound>
    <on-error />
</policies>

Troubleshooting

Contact our Support for additional guidance if you fail to implement the Logging.

Next Step

Search Fields
Log Views

Related

Azure Application Access
JSON Log Event
Managing
Monitoring
Non Events Agent
Pickup Log Events Service Logging Agent

• Log API Policy

Interested in logging from other Azure Related Services?

• Logging from Azure Logic Apps
• Logging from Azure functions