A close up of a sign

Description automatically generated

 

Overview

Datadog Mule® Integration is an agent-based integration that collects metrics from MuleSoft products and uploads them onto Datadog.

In the Datadog Mule® Integration bundle you will find:

·       120+ metrics

·       4 out-of-the-box dashboards

·       9 out-of-the-box monitors

·       Datadog Connector for Mule 4

You can use these metrics to take advantage of the out-of-the-box dashboards and monitors or you can create your own visualizations.

Content

Overview. 1

OOTB Assets 3

Pre-requisites 3

Dashboards 4

Monitors 8

Datadog Mule® Integration Installation. 11

Prerequisites 11

Installation process. 11

Datadog Mule® Integration Configuration. 13

init_config. 13

instances 15

Configuration process 15

The Connected Apps permissions 16

Full example. 17

System architecture. 18

Pre-requisites 18

IO Connect Services API 18

On-prem Mule servers 18

Mule Networking pre-requisites 19

CloudHub applications. 19

Ports and Hostnames to whitelist 19

IO Connect Services networking requirements. 19

MuleSoft Anypoint networking requirements 19

Datadog networking requirements. 20

Logging ingestion. 21

Logs JSON Layout 21

Logs forwarding. 22

Log collection through the agent [4] 22

Log forwarding through an HTTP appender 23

JMX [On-Prem] 25

JMX metrics at mule runtimes 26

Setting up the Datadog Agent for JMX fetch. 26

Visualizing the JMX metrics. 28

References 29

 

Datadog Mule® Integration is an agent-based integration that collects metrics from MuleSoft® products and uploads them onto Datadog. The metrics are currently collected from the following MuleSoft products:

1.     Mule® runtime.

2.     Exchange.

3.     Access Management.

4.     Object Store v2.

All metrics can be easily identified with the namespace mulesoft.anypoint in the Datadog Metrics Explorer. You can use these metrics to take advantage of the out-of-the-box dashboards and monitors or create your own visualizations.

OOTB Assets

Datadog Mule[EI1] [AFBL2] [AFBL3] ® Integration includes 4 dashboards and 9 monitors to visualize and identify the usage of your infrastructure resources. The data used in the dashboards and monitors is collected by the metrics.

Once you finalize the Datadog Mule® Integration installation, you will be able to check and use all dashboards and monitors in the Datadog page.

Dashboards:

§  Execs: Cost optimization.

§  Operations: Infrastructure.

§  Operations: Resources allocation and usage.

§  Development: Optimizations.

Monitors:

§  CloudHub and On-Premise.

o   Applications stopped.

o   CPU Load.

o   Memory usage.

§  On-Premise.

o   Servers stopped.

o   Application errors.

§  CloudHub.

o   Queue overload.

Pre-requisites

Before using the dashboards, you must know the identifiers of the organization and environments. This will help to filter out the graphics in the dashboards as well as properly use the tags in the metrics.

MuleSoft has documented this in the following articles:

How to know my Organization ID (Org ID) on the Anypoint Platform
https://help.mulesoft.com/s/article/How-to-know-my-Organization-ID-Org-ID-on-the-Anypoint-Platform

How to get Anypoint platform organization details via Anypoint APIs
https://help.mulesoft.com/s/article/How-to-get-Anypoint-platform-organization-details-via-Anypoint-APIs

It’s recommended to save views in the dashboards where these identifiers are needed for proper visualization. Learn more about views here https://docs.datadoghq.com/dashboards/template_variables/#saved-views

Dashboards

Find the assets in the Datadog web page by clicking at the side bar:

The Dashboard List should be like this one:

 

Let’s review how they work.

Execs: Cost Optimization

This dashboard monitors the resources available and not available through time per organization, helping you to identify how are being used easily.

The first section Application and Server failures shows critical and sensitive information about all the infrastructure:

§  Applications stopped time.

§  Applications stopped TOP 10 list.

§  Applications error On-Premise.

§  Applications errors On-Premise TOP 10 list.

The next section presents the usage of the following [EI4] resources:

§  vCores.

§  VPNs.

§  VPCs.

§  Static IPs.

§  Load Balancers.

§  Premium Connectors.

§  Object Store.

The information is displayed as values with its own timeline graphic, showing the values through time:

Those widgets work by selecting the time range and the right variables located at the top:

Pro-tip: Configure the MuleSoft organization and environment identifiers and save the view. See how you can obtain the identifiers and save the view in the Datadog dashboard in the pre-requisites.

By default, the dashboard uses and shows the information of all the organizations and environments (*) in your MuleSoft organization. To bring the information of a specific organization and environment, change the values with the correct Ids and change the time range as needed:

 

Operations: Infrastructure

The dashboard is divided in three sections:

Section 1. Critical: contains alerts that monitor applications critical behavior:

§  CloudHub applications stopped

§  Servers stopped

§  On-Premise application errors

§  Queue overload

These widgets are connected to the monitors, they show when the monitors are in Alert, OK or No Data status:

              

Section 2. CloudHub: shows resource usage by application:

§  Memory used

§  Memory percentage

§  CPU usage

§  CPU percentage

§  Network in and out

§  Message queue and Inflight

Section 3. On-Premise: shows resource usage by target and machine:

§  Memory used

§  Memory percentage (base 256 MB)

§  CPU used

§  Network in and out

It works similar to the dashboard mentioned before, you must select the variables values and time range as needed:

 

Operations: Resources allocation and usage

Pro-tip: Configure the MuleSoft organization and environment identifiers and save the view. See how you can obtain the identifiers and save the view in the Datadog dashboard in the pre-requisites.

This dashboard presents the resources available and used of your infrastructure per main organization.

The dashboard is divided into six sections:

Section 1. Organization vCores:

It displays the VCores assigned and reassigned per organization, also has a subsection that shows:

§  vCores used by environments

§  vCores reassigned per suborganization

§  VPNs usage

§  VPCs usage

§  Load Balancers usage

§  Static IPs usage

Section 2. Resources usage

The next sections include a table that contains resource allocation by organization name, resources reassigned and assigned.

§  Organization VPNs

§  Organization VPCs

§  Organization Load Balancers

§  Organization Static IPs

Section 3. Resource re-allocation

Resources from Anypoint Platform in an organization can be reassigned to a sub-organization. The next collapsible sections show how these resources are reassigned to sub-organizations.

Development: Optimizations

The dashboard displays basic information about CloudHub applications and On-Premise servers.

It is divided into CloudHub and On-Premise sections that shows:

§  CPU used

§  Memory used

§  Network in and out

For CloudHub section, you can select it by application and environment, for On-Premise section by target and host:

 

Monitors

Monitors help to constantly keep track of specific events, triggering an alert or notification whenever such events fulfill a condition. Datadog Mule® Integration comes with pre built-in monitors for events of failures in the Mule applications or servers.

Go to the monitors list by clicking at the side bar:

It should be a list like this one:

All monitors have the [HMJL5] same structure, but each one is configured with specific metrics. The monitor is triggered based on thresholds already set.

If you open a monitor in the Edit section, you will find the alert threshold value defined by default. This is different for each monitor:

If the alert is triggered, the monitor is going to change to red color, like the image below:

You can indicate in the monitor that the issue is fixed by clicking the Resolve button at the top-right corner, the Alert will change the status to OK and the color from red to green:

Once updated to “Resolve”, the Alert Monitor widget in the dashboards, also will be updated to OK status.

A screenshot of a cell phone

Description automatically generated

All alerts can be configured to trigger alerts. This is done by configuring the email address, Slack channel or other communication medium that is configured in your Datadog instance. Click on the “Edit” button on the monitor and scroll down to the Notify your team section and configure accordingly.

A screenshot of a cell phone

Description automatically generated

 

To learn more about monitors go to https://docs.datadoghq.com/monitors/

Datadog Mule® Integration Installation

The Datadog Mule® Integration is installed following the standard Datadog custom integration. This section is intended to show the complete path to follow from the prerequisites to get the integration working.

Prerequisites

1.     Python 2.7+ (Python 3.8 recommended).

2.     Java 8.

3.     pip package installer.

4.     Git.

Installation process

1.     Agent
Go to this link
https://app.datadoghq.com/account/settings#agent/overview and login with your credentials. Select the OS of the target installation platform and follow the steps in that page to download and install the Datadog Agent.

2.     Developer toolkit
Install the Developer toolkit following the link
https://docs.datadoghq.com/developers/integrations/new_check_howto/?tab=configurationtemplate#developer-toolkit

3.     Datadog Mule® Integration
Clone the
https://github.com/DataDog/marketplace/mulesoft_anypoint repository to the configured Developer Toolkit extras repository (by default $HOME/dd/ in Linux systems and %userprofile%\dd in Windows systems).

The git command to run is described in this link
https://docs.datadoghq.com/developers/integrations/new_check_howto/?tab=configurationtemplate#setup

4.     Integration requirements
Install the required python libraries listed in the requirements.in file at root level of
mulesoft_anypoint folder.

The requirement installation can be found at
https://docs.datadoghq.com/developers/guide/custom-python-package/?tab=linux selecting the target SO tab.

5.     Build the wheel
Run the
ddev release build mulesoft_anypoint command to build the package.

6.     Install the wheel. This step must be completed with administrator privileges.
Install the artifact by executing the wheel command:
https://docs.datadoghq.com/developers/integrations/new_check_howto/?tab=configurationtemplate#installing. Usually, the wheel is generated under mulesoft_anypoint/dist.

Before running the command you must verify that you have the necessary access permissions to the wheel and parent directory.

7.     Configure the installation.
Datadog Mule® Integration requires specific parameters to properly gather metrics. Follow the Integration Configuration section.

8.     Restart (or start) the Agent. This step must be completed with administrator privileges.
Follow the target platform OS instructions to restart (or start) the Agent described in
https://docs.datadoghq.com/agent/basic_agent_usage/?tab=agentv6v7

9.     Verify the installation
Verify the integration by running
status command described at the target platform OS instructions https://docs.datadoghq.com/agent/basic_agent_usage/?tab=agentv6v7 , mulesoft_anypoint should be listed in the Running Checks section.

Datadog Mule® Integration Configuration

The Datadog Mule® Integration is a Datadog custom integration and it is configured through a conf.yaml file located at:

·       C:\ProgramData\Datadog\conf.d\mulesoft_anypoint.d\conf.yaml in Windows

·       /etc/datadog-agent/conf.d/mulesoft_anypoint.d/conf.yaml in Linux systems

The configuration in the conf.yaml is divided into two parts:

·       init_config: A common part used by all the executions

·       instances: A list of instances to be scheduled independently

init_config

This section contains the common configuration used by all the instance executions. It contains the following configurations:

·       hosts: Grouping of all the hosts definitions needed by Mulesoft Anypoint Reader. Some hosts are specific to some APIs, if so, it is specified in the description:

o   anypoint: The Anypoint server host url. It is preconfigured with https://anypoint.mulesoft.com but it could be different for EU or GOV Mule Regions, see https://docs.mulesoft.com/access-management/managing-users#prerequisites

o   object_store_v2: The Specific Region Object Store V2 server host url. See https://docs.mulesoft.com/object-store/osv2-apis for the full list of available hosts. This host definition is used by the Object Store API. Example value: https://object-store-us-east-1.anypoint.mulesoft.com

o   object_store_v2_stats: The Object Store V2 Stats server host url. This host definition is used by the Object Store V2 Stats API. It is preconfigured with https://object-store-stats.anypoint.mulesoft.com

o   mule_server: The url or ip of the Server running a Mule Runtime with the Mule Agent. This host definition is used by the ARM APIs Example value: http://localhost:9999

o   oauth_provider: The Oauth Provider url that allows obtain a Bearer token used to make requests to all the APIs. It is preconfigured with https://anypoint.mulesoft.com/accounts/api/v2/oauth2/token but it could be different for EU or GOV Mule Regions, see https://docs.mulesoft.com/access-management/managing-users#prerequisites

·       client_id: The Client ID for the authentication API

·       client_secret: The Client Secret for the authentication API

·       env_id: The Environment ID for environment specific requests

·       org_id: The Organization ID for the requests that requires to specify it

·       api_key: The API key provided when you purchased the product

·       customer_key: The Customer key provided when you purchased the product

·       connection_wait_time: The number of seconds that authentication method will wait until next retry. If not specified it is defaulted to 2

·       connection_attempts_num: The number of retry attempts that authentication method will perform. If not specified it is defaulted to 3

In the Full Example section there is an example of a configuration file with all the values configured.

instances

This section contains a list of instances defined following the YAML list item notation -. Each instance is scheduled independently to run a set of APIs with a specific threads number configuration. Each instance contains the following configurations:

·       min_collection_interval: The time in seconds between executions. If not specified it is defaulted to 15 seconds

·       threads: The number of allowed parallel threads running the instance

·       api_filter: If not specified, all the APIs are executed, otherwise it must contain a list of APIs to run within the instance following the YAML list item notation -.

In the Full Example section there is an example of a configuration file with a list of instances. The example is taken directly from the conf.yaml file distributed with the integration and contains the optimum numbers we recommend for almost any scenario for min_collection_interval and threads.

The metric collection of any instance can be disabled at any time by commenting out the whole instance. This means, commenting the two attributes mentioned above.

Configuration process

The example file provided in datadog_checks/mulesoft_anypoint/data/conf.yaml.example and copied to the conf.d/mulesoft_anypoint.d/conf.yaml.example folder after the installation process contains a configuration with all the possible values already configured and they should just work for the common usage case.

The instances section contains a list of instances that were set to a periodicity and concurrency level according to each API provided information. Even if all these values can be changed, we recommend just go with the defaults.

The main configuration parameters to pay attention are:

·       object_store_v2: In the hosts section, it depends on the CloudHub Region and can be found at the following link https://docs.mulesoft.com/object-store/osv2-apis

·       mule_server: In the hosts section, it depends on the Mule Agent configuration. Following the defaults of the Mule Agent configuration it would be http://localhost:9999 to retrieve the metrics from a Mule Runtime running on the same machine

·       Connected Apps Authentication. The credentials to get access to the Anypoint platform. More information about Connected Apps: https://docs.mulesoft.com/access-management/connected-apps-overview. The two parameters to configure:

o   client_id: It must be configured with the Connected Apps account client_id

o   client_secret: It must be configured with the Connected Apps account client_secret

·       env_id: The Environment ID for environment specific requests. See https://docs.mulesoft.com/access-management/environments and https://help.mulesoft.com/s/question/0D52T00004mXPvSSAW/how-to-find-cloud-hub-environment-id

·       org_id: The Organization ID for the requests that requires to specify it. See https://docs.mulesoft.com/access-management/organization and https://help.mulesoft.com/s/article/How-to-know-my-Organization-ID-Org-ID-on-the-Anypoint-Platform

·       api_key: The API key provided by IO Connect Services when you purchased the product

·       customer_key: The Customer key provided by IO Connect Services when you purchased the product

·       For EU or GOV Mule Regions, the anypoint and oauth_provider should be changed too. See https://docs.mulesoft.com/access-management/managing-users#prerequisites

The Connected Apps permissions

Datadog Mule® Integration requires specific permissions on the MuleSoft Anypoint Platform to collect metrics. The approach to grant such permissions is using a Connected App. Follow instructions in the official MuleSoft documentation to create a connected app.
https://docs.mulesoft.com/access-management/connected-apps-overview

There are some common permissions that must be granted to be able to execute any API and others, specific, used by one or few APIs.

·       Exchange API:

o   Exchange Administrator
This is only required to read data from Exchange. Datadog Mule® Integration does not modify any asset.

·       CloudHub:

o   CloudHub Organization Admin
This is only required to read data from CloudHub. Datadog Mule® Integration does not modify any asset.

o   Read Alerts

·       ARM Rest Services:

o   Read Alerts

o   Read Applications

o   Read Servers

·       Access Management:

o   Read Applications

o   View Organization

·       ARM Monitoring Query:

o   Read Applications

·       Object Store:

o   Manage Application Data

o   Manage stores data

·       Object Store V2 Stats:

o   An administrator user
This is only required to read data from Object Store V2 statistics. Datadog Mule® Integration does not modify any asset. If these metrics are not desired, make sure you comment out this entry in the configuration file in
instances.

The integration does not modify in any manner the assets in Anypoint Platform, those permissions are for read-only.

Full example

Below is a complete configuration example (with fake credentials and ids):

init_config:
    hosts:
          anypoint: https://anypoint.mulesoft.com
          object_store_v2: https://object-store-us-east-1.anypoint.mulesoft.com
          object_store_v2_stats: https://object-store-stats.anypoint.mulesoft.com
          mule_server: http://localhost:9999
          oauth_provider: https://anypoint.mulesoft.com/accounts/api/v2/oauth2/token
    client_id: 035715123cbc31a123456a43143213f3
    client_secret: bAc2345678C34aFF1aB1A12f5A245678
    env_id: a3cc1234-4a24-125b-1a45-1c1aa1a13cad
    org_id: ac2345aa-cc13-1367-bca1-b1234a2aa4aa
    api_key: 548f-1s52-2d5f4f5b34ed
    customer_key: a6a6-b5568ae854e5
    connection_wait_time: 2
    connection_attempts_num: 3
instances:
  - min_collection_interval: 86400
    threads: 32
    api_filter:
      - access_management
  - min_collection_interval: 10
    threads: 32
    api_filter:
      - arm_monitoring_query
  - min_collection_interval: 10
    threads: 32
    api_filter:
      - arm_mule_agent
  - min_collection_interval: 10
    threads: 32
    api_filter:
      - arm_rest_services
  - min_collection_interval: 10
    threads: 32
    api_filter:
      - cloudhub
  - min_collection_interval: 86400
    threads: 32
    api_filter:
      - exchange_experience
  - min_collection_interval: 60
    threads: 32
    api_filter:
      - insight
  - min_collection_interval: 86400
    threads: 32
    api_filter:
      - object_store
  - min_collection_interval: 86400
    threads: 32
    api_filter:
      - object_store_v2_stats

System architecture

The Datadog Mule® Integration is an agent-based integration. This means that the customer must install the agent in a server on-premises and configure it.
https://docs.datadoghq.com/developers/integrations/

Pre-requisites

IO Connect Services API

The Datadog Mule® Integration does a license check via SSL and hence it requires outbound access to:

·      https://api.ioconnectservices.com/

·      Port: 443

On-prem Mule servers

On-prem Mule servers must be registered in Anypoint Runtime Manager (ARM) to be able to collect data. The RM Agent comes in the /bin folder of the Mule runtime, then you can perform the command in CLI. See instructions in https://docs.mulesoft.com/runtime-manager/servers-create

Any server that’s registered in a group or cluster in ARM must be able to gather metrics from those as well.

Mule Networking pre-requisites

In order for the agent to properly connect to ARM in Anypoint, specific network configuration must be allowed. All DNS names, ports and IPs needed to hook the agent are documented in https://docs.mulesoft.com/runtime-manager/rtm-agent-whitelists.

CloudHub applications

Given the cloud nature of applications deployed to CloudHub, all application and server metadata is intrinsically stored by default on Anypoint Control Plane. No special configuration is required other than the needed permissions in the connected app.

Ports and Hostnames to whitelist

The Datadog Mule® Integration must have internet connection on port 443 for outbound connections at least.

In enterprises, it’s very common that all networks are behind a firewall to protect access. In many other cases, reverse proxies are used to protect outbound communications to restricted websites. Customers must configure rules in the firewall and proxies to ensure the communication to all IO Connect Services, Anypoint and Datadog.

IO Connect Services networking requirements

The Datadog Mule® Integration does a license check via SSL and hence it requires outbound access to:

·      https://api.ioconnectservices.com/

·      Port: 443

This is an outbound communication only and it’s initiated by the Datadog agent running on-premise.

MuleSoft Anypoint networking requirements

Communication from Mule servers, installed on-prem, must allow inbound and outbound connections to the following DNS names via port 443 (HTTPS) and 9999 (configurable websocket).

Here is a full list of the FQDNs that need to be whitelisted. Pick the ones corresponding to the region to which you MuleSoft organization belongs to.

·      anypoint.mulesoft.com

·      eu1.anypoint.mulesoft.com

·      mule-manager.anypoint.mulesoft.com

·      mule-manager.eu1.anypoint.mulesoft.com

·      runtime-manager.anypoint.mulesoft.com

·      runtime-manager.eu1.anypoint.mulesoft.com

·      arm-auth-proxy.prod.cloudhub.io

·      arm-auth-proxy.prod-eu.msap.io

·      data-authenticator.anypoint.mulesoft.com

·      data-authenticator.eu1.anypoint.mulesoft.com

·      analytics-ingest.anypoint.mulesoft.com

·      analytics-ingest.eu1.anypoint.mulesoft.com

·      exchange2-asset-manager-kprod.s3.amazonaws.com

·      exchange2-asset-manager-kprod-eu.s3.eu-central-1.amazonaws.com

Learn more about the MuleSoft Anypoint networking requisites in https://docs.mulesoft.com/runtime-manager/rtm-agent-whitelists

Datadog networking requirements

Communication from Datadog agent, installed on-prem, must allow outbound connections to *.datadoghq.com via port 443 (HTTPS). Other ports might be used for specific use cases.

The FQDNs that need to be whitelisted are:

·       trace.agent.datadoghq.com: APM

·       process.datadoghq.com: Live containers

·       *.agent.datadoghq.com: Log collection

·       api.datadoghq.com: Non-critical functions such as checking API Key validity

Modern firewalls can whitelist request based on OSI's layer 7.

Also, you can find these requisites in Datadog site, they are well documented. https://docs.datadoghq.com/agent/guide/network/?tab=agentv6v7

To know the full list of IP ranges that Datadog uses, see the following sites.

·      https://ip-ranges.datadoghq.com/

·      https://ip-ranges.datadoghq.eu/

In Datadog, all communication is outbound, meaning the agent sends data to Datadog and Datadog never requests to client servers.

All communication is done through these ports:

·      443/TCP: port for most Agent data. (Metrics, APM, Live Processes/Containers)

·      123/UDP: Network time protocol (NTP)

·      10516/TCP: port for the Log collection over TCP for Datadog US region, 443/tcp for the Datadog EU region.

·      10255/TCP: port for the Kubernetes http kubelet

·      10250/TCP: port for the Kubernetes https kubelet

The only inbound communication needed is to send data to agents within the same network, like it's the case of APM configured in a Mule application using the Datadog APM Connector to trace processes.

·       5000/TCP: port for the go_expvar server

·       5001/TCP: port on which the IPC api listens

·       5002/TCP: port for the Agent browser GUI to be served

·       8125/UDP: dogstatsd

·       8126/TCP: port for the APM Receiver

Logging ingestion

Logs are part of the observability pillar. To have full observability you must enable logging collection for your applications. This section will show you how to forward the logs into Datadog and the ways to do so for your Mule apps.

Logs JSON Layout

To avoid custom mapping rules for your logs in the Datadog platform and have a correlation between the logs and traces your application can report, we recommend using a JSONLayout formatter in the log4j2 configuration of your Mule apps.

The JSONLayout is placed into your appender config and must follow the below structure and configuration:

<JSONLayout compact="false" eventEol="true" properties="true" stacktraceAsString="true">
        <KeyValuePair key="host" value="${sys:domain}" />
        <KeyValuePair key="appName" value="${sys:domain}" /> 
        <KeyValuePair key="ddsource" value="mule" />
        <KeyValuePair key="service" value="mule" />
        <KeyValuePair key="correlationId" value="$${ctx:correlationId:-}"/>
</JSONLayout>

The layout parameters definition and documentation are in the official log4j2 documentation https://logging.apache.org/log4j/2.x/manual/layouts.html#JSONLayout

$${ctx:correlationId:-}: The <KeyValuePair key="correlationId" value="$${ctx:correlationId:-}"/> Is taking the correlation Id from Log4J2’s MDC[3]. The correlation Id is coming from the Mule event and is injected by the Log4j2 context.

${sys:domain}: This is an automatically generated system property in the CloudHub domain name. For On-Prem you must set the "host" and "appName" values through environment variables or static values, where the "host" is the server where your application is hosted and the "appName" is the name of your application.

Logs forwarding

There are two useful ways to forward logs. The logs can be collected by the Datadog agent from a directory or you can enable an HTTP appender to send the logs to the Datadog API.

Log collection through the agent [4]

Prerequisites

1.     A Datadog agent running.

2.     Your Mule applications must be customer hosted. The Datadog agent must coexist in the same server.

3.     Have a directory where the logs can be collected.

Steps

1.     In your application, add a new file appender in the log4j2 configuration file, find below a sample:

<File name="MyFile" fileName="path/to/your/mule/log.log" immediateFlush="true">

<JSONLayout …/>

</File>

2.     Reference the appender in the Logger configuration as follows:

<Loggers>

<AsyncRoot level="INFO">

<AppenderRef ref="MyFile" />

</AsyncRoot>

</Loggers>

 

Search the folder of the Datadog agent in your documents. There, find the file datadog.yaml and open it. Search the text: #logs_enabled: false change the value to true and uncomment it. Save the changes. 

o   The default path directory of the Datadog agent is:

§  Windows: C:\ProgramData\Datadog

§  Linux: /etc/datadog-agent

3.     In the Datadog agent folder, open conf.d folder. There, create a folder and name it java.d then add into it a file named java.yaml and add the next content:

#Log section
logs:
  - type: file
      path: "/path/to/your/mule/log.log"
      service: java
      source: java
   sourcecategory: sourcecode

Note: Make sure that the path value is the same as the fileName value in the log4j2 file of the Mule application.

Until here you have finished configuring the Agent and your applications. Please notice that this way of logging collection is only available for On-Prem applications, for your MuleSoft-hosted applications you must use an HTTP appender.

Log forwarding through an HTTP appender

Prerequisites

1.     Have outbound HTTP connection.

2.     Have the Datadog’s API key.

3.     For CloudHub, have visible the Disable CloudHub logs checkbox, this is not a visible option in Runtime Manager. A special request to the Mulesoft support team is required to enable this option. More info: https://docs.mulesoft.com/runtime-manager/custom-log-appender

Steps

1.     Create an appender for Datadog in your application log4j2 configuration:

<Http name="DD-HTTP" url="https://http-intake.logs.datadoghq.com/v1/input" verifyHostname="false" method="POST">
    <Property name="DD-API-KEY" value="${sys:ddApiKey}" />
    <Property name="dd.trace_id" value="${ctx:dd.trace_id}" />
 
    <JSONLayout … />
</Http>

1.1   ${sys:ddApiKey}: The DD-API-KEY header is a custom property that will be set with the Datadog’s API key. For On-Prem the value can be set through an environment variable or static value, for CloudHub use the application properties in Runtime Manager.

1.2   ${ctx:dd.trace_id}: The dd.trace_id is a customer header in the HTTP request that helps to inject the trace identifier when using logging with APM tracing. More details about this in the Datadog APM Connector.

2.     This step is required only if you want to log in CloudHub as well as in Datadog for your MuleSoft-Hosted applications. In the log4j2 create an appender with the special tag Log4J2CloudhubLogAppender inside the <Appenders>:

<Appenders>

  <Log4J2CloudhubLogAppender name="CLOUDHUB"

addressProvider="com.mulesoft.ch.logging.DefaultAggregatorAddressProvider"

applicationContext="com.mulesoft.ch.logging.DefaultApplicationContext"

appendRetryIntervalMs="${sys:logging.appendRetryInterval}"

appendMaxAttempts="${sys:logging.appendMaxAttempts}"

batchSendIntervalMs="${sys:logging.batchSendInterval}"

batchMaxRecords="${sys:logging.batchMaxRecords}"

memBufferMaxSize="${sys:logging.memBufferMaxSize}"

journalMaxWriteBatchSize="${sys:logging.journalMaxBatchSize}"

journalMaxFileSize="${sys:logging.journalMaxFileSize}"

clientMaxPacketSize="${sys:logging.clientMaxPacketSize}"

clientConnectTimeoutMs="${sys:logging.clientConnectTimeout}"

clientSocketTimeoutMs="${sys:logging.clientSocketTimeout}"

serverAddressPollIntervalMs="${sys:logging.serverAddressPollInterval}"      serverHeartbeatSendIntervalMs="${sys:logging.serverHeartbeatSendIntervalMs}"

statisticsPrintIntervalMs="${sys:logging.statisticsPrintIntervalMs}">

     <PatternLayout pattern="[%d{MM-dd HH:mm:ss}] %-5p %c{1} [%t]: %m%n"/>

  </Log4J2CloudhubLogAppender>

</Appenders>

3.     Enable your appender(s) to be used:

3.1 CloudHub

<Loggers>
    <AsyncRoot level="INFO">
          <AppenderRef ref="CLOUDHUB"/>
          <AppenderRef ref="DD-HTTP"/>
   </AsyncRoot>
</Loggers>

3.2 On-prem

       
<Loggers>
      <AsyncRoot level="INFO">
              <AppenderRef ref="DD-HTTP" />
      </AsyncRoot>
</Loggers>

Now you have configured the HTTP approach for sending logs, please notice that this option is available for both, customer-hosted and MuleSoft-hosted applications.

Note: When an application is deployed in CloudHub the Runtime Manager always overwrite the log4j2.xml. You need to disable the CloudHub logs to apply your project’s log4j2.xml configuration file in the runtime manager.

 

JMX [On-Prem]

The Mule Runtime Manager agent JMX service allows you to track specific JMX metrics and publish them to external services. The JMX service sends all the gathered metrics to publishers, which in turn publish the metrics to external monitoring tools [5].

Datadog is a monitoring service which has a lightweight Java plugin named JMXFetch. This plugin is called by the Datadog Agent to connect to the MBean Server and collect your application metrics. This plugin sends metrics to the Datadog Agent using the DogStatsD server running within the Agent[6].

Full monitoring of your mule runtimes is accomplished when you install the Datadog Mule integration into the agent, enables the Datadog APM and gather your mule runtime JVM metrics. These metrics enable a deep dive infrastructure inspection and correlate your instances/servers with the APM services, delivering a holistic view of your customer-hosted runtime planes, everything in a single platform.

JMX metrics at mule runtimes

To open JMX metrics consumption at your mule servers for the Datadog agent, you must run your mule runtime or add the below arguments [7] to your mule instance conf folder in a file named wrapper-additional.conf:

·       -M-Dcom.sun.management.jmxremote

o   To allow the JMX client access to the Mule Java VM.

·       -M-Dcom.sun.management.jmxremote.port=portNum

o   To enable JVM monitoring of your instance. In the property above, portNum is the port number through which you want to enable JMX RMI connections.

·       -M-Dcom.sun.management.jmxremote.local.only=true

o   If set to false, remote system monitoring can be configured but if you are running the Datadog agent in the same instance where your mule runtime is running this is not neccessary.

·       -M-Dcom.sun.management.jmxremote.authenticate=false

o   Password authentication for remote monitoring is enabled by default. To disable it, set the following system property when you start the Mule runtime, moreover all traffic is kept locally and no outbound monitoring is configured.

·       -M-Dcom.sun.management.jmxremote.ssl=false

o   To use client SSL authentication then set it true, N/A for local monitoring.

·       -M-Djava.rmi.server.hostname=host

o   To enable JVM monitoring of your instance. In the property above, host is the local IP address of your server to enable JMX RMI connections.

Note: Above properties, like the host and port, will be used later in the Datadog agent configuration.

After running your Mule runtime with the above configuration, the agent can collect these metrics and report them into the Datadog platform, if your Mule runtime was running and you added the properties as a wrapper-additional.conf a restart of your instance is required, let’s configure the agent to do so.

Setting up the Datadog Agent for JMX fetch

Prerequisites

1.     A Datadog agent running.

2.     Your Mule applications must be customer hosted.

3.     The Datadog agent must be able to communicate with the Mule server. Ensure that the configured host and port for the Mule runtime JMX service are reachable.

Configuration/Steps

·       Configure the depicted file at the following link according to your needs and specs, please follow Datadog documentation:

o   https://docs.datadoghq.com/integrations/java/?tab=host#configuration

·       Restart the datadog agent

o   Linux:

§  https://docs.datadoghq.com/agent/guide/agent-commands/?tab=agentv6v7#restart-the-agent

o   Windows:

§  https://docs.datadoghq.com/agent/basic_agent_usage/windows/?tab=gui

·       Let’s check the JMXFetch is running and gathering our instance(s):

o   Please follow the below document to verify that JMXFetch check is executing correctly:

§  https://docs.datadoghq.com/agent/troubleshooting/agent_check_status/?tab=agentv6v7

§  When configured and started successfully, it should show you something similar like the below image:



Visualizing the JMX metrics

Now, let's confirm you are able to see the metrics in Datadog web page. Login into: https://app.datadoghq.com/account/login?next=%2F

·       Once there, go to the left menu, select Dashboards and there, look for JVM Metrics or JVM - Overview dashboard.

o   If the JVM Metrics dashboard is selected, then go up into the dashboard, below the dashboard name, and check for the $host variable, here select your host or “*” to show hosts gathered metrics.

o   If the JVM - Overview dashboard is selected, then go up into the dashboard, below the dashboard name, and check for the $scope variable, here select your host or “*” to show hosts gathered metrics.

o   Please find below an image of what it is depicted above:



·       And that’s it you can visualize your JMX metrics OOB.

 This link contains the list of the metrics you can obtain from JMX.

References

[1] https://www.datadoghq.com/product/log-management/#full-observability

[2] https://logging.apache.org/log4j/2.x/manual/layouts.html#JSONLayout

[3] https://logging.apache.org/log4j/2.x/manual/thread-context.html

[4] https://docs.datadoghq.com/logs/log_collection/?tab=http

[5] https://docs.mulesoft.com/runtime-manager/jmx-service

[6]
https://docs.datadoghq.com/integrations/java/?tab=host#metric-collection

[7]
https://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html