Overview
Datadog Mule® Integration is an agent-based integration that collects metrics from MuleSoft products and uploads them onto Datadog.
In the Datadog Mule® Integration bundle you will find:
· 120+ metrics
· 4 out-of-the-box dashboards
· 9 out-of-the-box monitors
· Datadog Connector for Mule 4
You can use these metrics to take advantage of the out-of-the-box dashboards and monitors or you can create your own visualizations.
Content
Datadog Mule® Integration Installation
Datadog Mule® Integration Configuration
The Connected Apps permissions
Mule Networking pre-requisites
Ports and Hostnames to whitelist
IO Connect Services networking requirements
MuleSoft Anypoint networking requirements
Datadog networking requirements
Log collection through the agent [4]
Log forwarding through an HTTP appender
Setting up the Datadog Agent for JMX fetch
Datadog Mule® Integration is an agent-based integration that collects metrics from MuleSoft® products and uploads them onto Datadog. The metrics are currently collected from the following MuleSoft products:
1. Mule® runtime.
2. Exchange.
3. Access Management.
4. Object Store v2.
All metrics can be easily identified with the namespace mulesoft.anypoint in the Datadog Metrics Explorer. You can use these metrics to take advantage of the out-of-the-box dashboards and monitors or create your own visualizations.
OOTB Assets
Datadog Mule[EI1] [AFBL2] [AFBL3] ® Integration includes 4 dashboards and 9 monitors to visualize and identify the usage of your infrastructure resources. The data used in the dashboards and monitors is collected by the metrics.
Once you finalize the Datadog Mule® Integration installation, you will be able to check and use all dashboards and monitors in the Datadog page.
Dashboards:
§ Execs: Cost optimization.
§ Operations: Infrastructure.
§ Operations: Resources allocation and usage.
§ Development: Optimizations.
Monitors:
§ CloudHub and On-Premise.
o Applications stopped.
o CPU Load.
o Memory usage.
§ On-Premise.
o Servers stopped.
o Application errors.
§ CloudHub.
o Queue overload.
Pre-requisites
Before using the dashboards, you must know the identifiers of the organization and environments. This will help to filter out the graphics in the dashboards as well as properly use the tags in the metrics.
MuleSoft has documented this in the following articles:
How to know
my Organization ID (Org ID) on the Anypoint Platform
https://help.mulesoft.com/s/article/How-to-know-my-Organization-ID-Org-ID-on-the-Anypoint-Platform
How to get
Anypoint platform organization details via Anypoint APIs
https://help.mulesoft.com/s/article/How-to-get-Anypoint-platform-organization-details-via-Anypoint-APIs
It’s recommended to save views in the dashboards where these identifiers are needed for proper visualization. Learn more about views here https://docs.datadoghq.com/dashboards/template_variables/#saved-views
Dashboards
Find the assets in the Datadog web page by clicking at the side bar:
The Dashboard List should be like this one:
Let’s review how they work.
Execs: Cost Optimization
This dashboard monitors the resources available and not available through time per organization, helping you to identify how are being used easily.
The first section Application and Server failures shows critical and sensitive information about all the infrastructure:
§ Applications stopped time.
§ Applications stopped TOP 10 list.
§ Applications error On-Premise.
§ Applications errors On-Premise TOP 10 list.
The next section presents the usage of the following [EI4] resources:
§ vCores.
§ VPNs.
§ VPCs.
§ Static IPs.
§ Load Balancers.
§ Premium Connectors.
§ Object Store.
The information is displayed as values with its own timeline graphic, showing the values through time:
Those widgets work by selecting the time range and the right variables located at the top:
Pro-tip: Configure the MuleSoft organization and environment identifiers and save the view. See how you can obtain the identifiers and save the view in the Datadog dashboard in the pre-requisites.
By default, the dashboard uses and shows the information of all the organizations and environments (*) in your MuleSoft organization. To bring the information of a specific organization and environment, change the values with the correct Ids and change the time range as needed:
Operations: Infrastructure
The dashboard is divided in three sections:
Section 1. Critical: contains alerts that monitor applications critical behavior:
§ CloudHub applications stopped
§ Servers stopped
§ On-Premise application errors
§ Queue overload
These widgets are connected to the monitors, they show when the monitors are in Alert, OK or No Data status:
Section 2. CloudHub: shows resource usage by application:
§ Memory used
§ Memory percentage
§ CPU usage
§ CPU percentage
§ Network in and out
§ Message queue and Inflight
Section 3. On-Premise: shows resource usage by target and machine:
§ Memory used
§ Memory percentage (base 256 MB)
§ CPU used
§ Network in and out
It works similar to the dashboard mentioned before, you must select the variables values and time range as needed:
Operations: Resources allocation and usage
Pro-tip: Configure the MuleSoft organization and environment identifiers and save the view. See how you can obtain the identifiers and save the view in the Datadog dashboard in the pre-requisites.
This dashboard presents the resources available and used of your infrastructure per main organization.
The dashboard is divided into six sections:
Section 1. Organization vCores:
It displays the VCores assigned and reassigned per organization, also has a subsection that shows:
§ vCores used by environments
§ vCores reassigned per suborganization
§ VPNs usage
§ VPCs usage
§ Load Balancers usage
§ Static IPs usage
Section 2. Resources usage
The next sections include a table that contains resource allocation by organization name, resources reassigned and assigned.
§ Organization VPNs
§ Organization VPCs
§ Organization Load Balancers
§ Organization Static IPs
Section 3. Resource re-allocation
Resources from Anypoint Platform in an organization can be reassigned to a sub-organization. The next collapsible sections show how these resources are reassigned to sub-organizations.
Development: Optimizations
The dashboard displays basic information about CloudHub applications and On-Premise servers.
It is divided into CloudHub and On-Premise sections that shows:
§ CPU used
§ Memory used
§ Network in and out
For CloudHub section, you can select it by application and environment, for On-Premise section by target and host:
Monitors
Monitors help to constantly keep track of specific events, triggering an alert or notification whenever such events fulfill a condition. Datadog Mule® Integration comes with pre built-in monitors for events of failures in the Mule applications or servers.
Go to the monitors list by clicking at the side bar:
It should be a list like this one:
All monitors have the [HMJL5] same structure, but each one is configured with specific metrics. The monitor is triggered based on thresholds already set.
If you open a monitor in the Edit section, you will find the alert threshold value defined by default. This is different for each monitor:
If the alert is triggered, the monitor is going to change to red color, like the image below:
You can indicate in the monitor that the issue is fixed by clicking the Resolve button at the top-right corner, the Alert will change the status to OK and the color from red to green:
Once updated to “Resolve”, the Alert Monitor widget in the dashboards, also will be updated to OK status.
All alerts can be configured to trigger alerts. This is done by configuring the email address, Slack channel or other communication medium that is configured in your Datadog instance. Click on the “Edit” button on the monitor and scroll down to the Notify your team section and configure accordingly.
To learn more about monitors go to https://docs.datadoghq.com/monitors/
Datadog Mule® Integration Installation
The Datadog Mule® Integration is installed following the standard Datadog custom integration. This section is intended to show the complete path to follow from the prerequisites to get the integration working.
Prerequisites
1. Python 2.7+ (Python 3.8 recommended).
2. Java 8.
3. pip package installer.
4. Git.
Installation process
1. Agent
Go to this link https://app.datadoghq.com/account/settings#agent/overview and login with your credentials.
Select the OS of the target installation platform and follow the steps in that
page to download and install the Datadog Agent.
2. Developer toolkit
Install the Developer toolkit following the link https://docs.datadoghq.com/developers/integrations/new_check_howto/?tab=configurationtemplate#developer-toolkit
3. Datadog Mule® Integration
Clone the https://github.com/DataDog/marketplace/mulesoft_anypoint repository to the
configured Developer Toolkit extras repository (by default $HOME/dd/ in Linux systems and %userprofile%\dd in Windows systems).
The git command to run is described in this link https://docs.datadoghq.com/developers/integrations/new_check_howto/?tab=configurationtemplate#setup
4. Integration requirements
Install the required python libraries listed in the requirements.in file
at root level of mulesoft_anypoint folder.
The requirement installation can be found at https://docs.datadoghq.com/developers/guide/custom-python-package/?tab=linux selecting the target SO tab.
5. Build the wheel
Run the ddev
release build mulesoft_anypoint
command to build the package.
6. Install the wheel. This step must be completed with administrator
privileges.
Install the artifact by executing the wheel command: https://docs.datadoghq.com/developers/integrations/new_check_howto/?tab=configurationtemplate#installing. Usually, the wheel is generated
under mulesoft_anypoint/dist.
Before running the command you must verify that you have the necessary access
permissions to the wheel and parent directory.
7.
Configure
the installation.
Datadog Mule®
Integration requires specific parameters to properly gather metrics. Follow the
Integration Configuration section.
8. Restart (or start) the Agent. This step must be completed with administrator
privileges.
Follow the target platform OS instructions to restart (or start) the Agent
described in https://docs.datadoghq.com/agent/basic_agent_usage/?tab=agentv6v7
9. Verify the installation
Verify the integration by running status command described at the target
platform OS instructions https://docs.datadoghq.com/agent/basic_agent_usage/?tab=agentv6v7 , mulesoft_anypoint should be listed in the Running Checks section.
Datadog Mule® Integration Configuration
The Datadog Mule® Integration is a Datadog custom integration and it is configured through a conf.yaml file located at:
·
C:\ProgramData\Datadog\conf.d\mulesoft_anypoint.d\conf.yaml
in Windows
·
/etc/datadog-agent/conf.d/mulesoft_anypoint.d/conf.yaml
in Linux systems
The configuration in the conf.yaml is divided into two parts:
·
init_config: A common part used by all the executions
·
instances: A list of instances to be scheduled
independently
init_config
This section contains the common configuration used by all the instance executions. It contains the following configurations:
·
hosts: Grouping of all the hosts definitions
needed by Mulesoft
Anypoint Reader.
Some hosts are specific to some APIs, if so, it is specified in the
description:
o
anypoint: The Anypoint server host url. It is preconfigured with
https://anypoint.mulesoft.com
but it could be different for EU
or GOV Mule Regions, see https://docs.mulesoft.com/access-management/managing-users#prerequisites
o
object_store_v2: The Specific Region Object Store V2 server host url. See https://docs.mulesoft.com/object-store/osv2-apis for the full list of available hosts.
This host definition is used by the Object Store API. Example value: https://object-store-us-east-1.anypoint.mulesoft.com
o
object_store_v2_stats:
The Object Store V2 Stats server host url. This host definition
is used by the Object
Store V2 Stats API.
It is preconfigured with https://object-store-stats.anypoint.mulesoft.com
o
mule_server: The url or ip of the Server running a Mule Runtime with the Mule Agent. This host definition is used by the
ARM APIs Example value: http://localhost:9999
o
oauth_provider:
The Oauth Provider url that
allows obtain a Bearer token used to make requests to all the APIs. It is
preconfigured with https://anypoint.mulesoft.com/accounts/api/v2/oauth2/token
but it could be different for EU
or GOV Mule Regions, see https://docs.mulesoft.com/access-management/managing-users#prerequisites
·
client_id: The Client ID for the authentication
API
·
client_secret:
The Client Secret for the
authentication API
·
env_id: The Environment ID for environment
specific requests
·
org_id: The Organization ID for the requests
that requires to specify it
·
api_key: The API key provided when you
purchased the product
·
customer_key: The Customer key provided when you
purchased the product
·
connection_wait_time:
The number of seconds that
authentication method will wait until next retry. If not specified it is
defaulted to 2
·
connection_attempts_num:
The number of retry attempts
that authentication method will perform. If not specified it is defaulted to 3
In the Full Example section there is an example of a configuration file with all the values configured.
instances
This
section contains a list of instances defined following the YAML list item
notation -. Each instance is scheduled
independently to run a set of APIs with a specific threads number
configuration. Each instance contains the following configurations:
·
min_collection_interval: The time in seconds between
executions. If not specified it is defaulted to 15 seconds
·
threads: The number of allowed parallel threads
running the instance
·
api_filter: If not specified, all the APIs are
executed, otherwise it must contain a list of APIs to run within the instance
following the YAML list item notation -.
In the Full Example section there is an example of a
configuration file with a list of instances. The example is taken directly from
the conf.yaml file distributed with the integration
and contains the optimum numbers we recommend for almost any scenario for
min_collection_interval and threads.
The metric collection of any instance can be disabled at any time by commenting out the whole instance. This means, commenting the two attributes mentioned above.
Configuration process
The
example file provided in datadog_checks/mulesoft_anypoint/data/conf.yaml.example
and copied to the conf.d/mulesoft_anypoint.d/conf.yaml.example
folder after the installation
process contains a configuration with all the possible values already
configured and they should just work for the common usage case.
The
instances section contains a list of instances
that were set to a periodicity and concurrency level according to each API
provided information. Even if all these values can be changed, we recommend
just go with the defaults.
The main configuration parameters to pay attention are:
·
object_store_v2:
In the hosts section, it depends on the CloudHub Region and can be found at the following
link https://docs.mulesoft.com/object-store/osv2-apis
·
mule_server: In the hosts section,
it depends on the Mule
Agent
configuration. Following the defaults of the Mule Agent configuration it would be http://localhost:9999 to retrieve the metrics from a Mule Runtime running on the same machine
· Connected Apps Authentication. The credentials to get access to the Anypoint platform. More information about Connected Apps: https://docs.mulesoft.com/access-management/connected-apps-overview. The two parameters to configure:
o
client_id: It must be configured with the Connected Apps account client_id
o
client_secret:
It must be configured with the Connected Apps account client_secret
·
env_id: The Environment ID for environment
specific requests. See https://docs.mulesoft.com/access-management/environments and https://help.mulesoft.com/s/question/0D52T00004mXPvSSAW/how-to-find-cloud-hub-environment-id
·
org_id: The Organization ID for the requests
that requires to specify it. See https://docs.mulesoft.com/access-management/organization and https://help.mulesoft.com/s/article/How-to-know-my-Organization-ID-Org-ID-on-the-Anypoint-Platform
·
api_key: The API key provided by IO Connect Services when you
purchased the product
·
customer_key: The Customer key provided by IO Connect Services when you
purchased the product
·
For EU or GOV Mule
Regions, the anypoint
and oauth_provider should be changed too. See https://docs.mulesoft.com/access-management/managing-users#prerequisites
The Connected Apps permissions
Datadog Mule® Integration requires specific permissions on
the MuleSoft Anypoint Platform to collect metrics. The approach to grant such
permissions is using a Connected App. Follow instructions in the official
MuleSoft documentation to create a connected app.
https://docs.mulesoft.com/access-management/connected-apps-overview
There are some common permissions that must be granted to be able to execute any API and others, specific, used by one or few APIs.
· Exchange API:
o
Exchange
Administrator
This
is only required to read data from Exchange. Datadog Mule® Integration does
not modify any asset.
· CloudHub:
o
CloudHub
Organization Admin
This
is only required to read data from CloudHub. Datadog Mule® Integration does
not modify any asset.
o Read Alerts
· ARM Rest Services:
o Read Alerts
o Read Applications
o Read Servers
· Access Management:
o Read Applications
o View Organization
· ARM Monitoring Query:
o Read Applications
· Object Store:
o Manage Application Data
o Manage stores data
· Object Store V2 Stats:
o
An administrator
user
This is
only required to read data from Object Store V2 statistics. Datadog Mule®
Integration does not modify any asset. If these metrics are not desired, make
sure you comment out this entry in the configuration file in instances.
The integration does not modify in any manner the assets in Anypoint Platform, those permissions are for read-only.
Full example
Below is a complete configuration example (with fake credentials and ids):
init_config: hosts: anypoint: https://anypoint.mulesoft.com object_store_v2: https://object-store-us-east-1.anypoint.mulesoft.com object_store_v2_stats: https://object-store-stats.anypoint.mulesoft.com mule_server: http://localhost:9999 oauth_provider: https://anypoint.mulesoft.com/accounts/api/v2/oauth2/token client_id: 035715123cbc31a123456a43143213f3 client_secret: bAc2345678C34aFF1aB1A12f5A245678env_id: a3cc1234-4a24-125b-1a45-1c1aa1a13cad
org_id: ac2345aa-cc13-1367-bca1-b1234a2aa4aa
api_key: 548f-1s52-2d5f4f5b34ed customer_key: a6a6-b5568ae854e5 connection_wait_time: 2 connection_attempts_num: 3instances: - min_collection_interval: 86400 threads: 32 api_filter: - access_management - min_collection_interval: 10 threads: 32 api_filter: - arm_monitoring_query - min_collection_interval: 10 threads: 32 api_filter: - arm_mule_agent - min_collection_interval: 10 threads: 32 api_filter: - arm_rest_services - min_collection_interval: 10 threads: 32 api_filter: - cloudhub - min_collection_interval: 86400 threads: 32 api_filter: - exchange_experience - min_collection_interval: 60 threads: 32 api_filter: - insight - min_collection_interval: 86400 threads: 32 api_filter: - object_store - min_collection_interval: 86400 threads: 32 api_filter: - object_store_v2_stats
System architecture
The
Datadog Mule® Integration is an agent-based integration. This means that the
customer must install the agent in a server on-premises and configure it.
https://docs.datadoghq.com/developers/integrations/
Pre-requisites
IO Connect Services API
The Datadog Mule® Integration does a license check via SSL and hence it requires outbound access to:
· https://api.ioconnectservices.com/
· Port: 443
On-prem Mule servers
On-prem Mule servers must be registered in Anypoint Runtime Manager (ARM) to be able to collect data. The RM Agent comes in the /bin folder of the Mule runtime, then you can perform the command in CLI. See instructions in https://docs.mulesoft.com/runtime-manager/servers-create.
Any server that’s registered in a group or cluster in ARM must be able to gather metrics from those as well.
Mule Networking pre-requisites
In order for the agent to properly connect to ARM in Anypoint, specific network configuration must be allowed. All DNS names, ports and IPs needed to hook the agent are documented in https://docs.mulesoft.com/runtime-manager/rtm-agent-whitelists.
CloudHub applications
Given the cloud nature of applications deployed to CloudHub, all application and server metadata is intrinsically stored by default on Anypoint Control Plane. No special configuration is required other than the needed permissions in the connected app.
Ports and Hostnames to whitelist
The Datadog Mule® Integration must have internet connection on port 443 for outbound connections at least.
In enterprises, it’s very common that all networks are behind a firewall to protect access. In many other cases, reverse proxies are used to protect outbound communications to restricted websites. Customers must configure rules in the firewall and proxies to ensure the communication to all IO Connect Services, Anypoint and Datadog.
IO Connect Services networking requirements
The Datadog Mule® Integration does a license check via SSL and hence it requires outbound access to:
· https://api.ioconnectservices.com/
· Port: 443
This is an outbound communication only and it’s initiated by the Datadog agent running on-premise.
MuleSoft Anypoint networking requirements
Communication from Mule servers, installed on-prem, must allow inbound and outbound connections to the following DNS names via port 443 (HTTPS) and 9999 (configurable websocket).
Here is a full list of the FQDNs that need to be whitelisted. Pick the ones corresponding to the region to which you MuleSoft organization belongs to.
· mule-manager.anypoint.mulesoft.com
· mule-manager.eu1.anypoint.mulesoft.com
· runtime-manager.anypoint.mulesoft.com
· runtime-manager.eu1.anypoint.mulesoft.com
· arm-auth-proxy.prod.cloudhub.io
· arm-auth-proxy.prod-eu.msap.io
· data-authenticator.anypoint.mulesoft.com
· data-authenticator.eu1.anypoint.mulesoft.com
· analytics-ingest.anypoint.mulesoft.com
· analytics-ingest.eu1.anypoint.mulesoft.com
· exchange2-asset-manager-kprod.s3.amazonaws.com
· exchange2-asset-manager-kprod-eu.s3.eu-central-1.amazonaws.com
Learn more about the MuleSoft Anypoint networking requisites in https://docs.mulesoft.com/runtime-manager/rtm-agent-whitelists
Datadog networking requirements
Communication from Datadog agent, installed on-prem, must allow outbound connections to *.datadoghq.com via port 443 (HTTPS). Other ports might be used for specific use cases.
The FQDNs that need to be whitelisted are:
· trace.agent.datadoghq.com: APM
· process.datadoghq.com: Live containers
· *.agent.datadoghq.com: Log collection
· api.datadoghq.com: Non-critical functions such as checking API Key validity
Modern firewalls can whitelist request based on OSI's layer 7.
Also, you can find these requisites in Datadog site, they are well documented. https://docs.datadoghq.com/agent/guide/network/?tab=agentv6v7
To know the full list of IP ranges that Datadog uses, see the following sites.
· https://ip-ranges.datadoghq.com/
· https://ip-ranges.datadoghq.eu/
In Datadog, all communication is outbound, meaning the agent sends data to Datadog and Datadog never requests to client servers.
All communication is done through these ports:
· 443/TCP: port for most Agent data. (Metrics, APM, Live Processes/Containers)
· 123/UDP: Network time protocol (NTP)
· 10516/TCP: port for the Log collection over TCP for Datadog US region, 443/tcp for the Datadog EU region.
· 10255/TCP: port for the Kubernetes http kubelet
· 10250/TCP: port for the Kubernetes https kubelet
The only inbound communication needed is to send data to agents within the same network, like it's the case of APM configured in a Mule application using the Datadog APM Connector to trace processes.
· 5000/TCP: port for the go_expvar server
· 5001/TCP: port on which the IPC api listens
· 5002/TCP: port for the Agent browser GUI to be served
· 8125/UDP: dogstatsd
· 8126/TCP: port for the APM Receiver
Logging ingestion
Logs are part of the observability pillar. To have full observability you must enable logging collection for your applications. This section will show you how to forward the logs into Datadog and the ways to do so for your Mule apps.
Logs JSON Layout
To avoid custom mapping rules for your logs in the Datadog platform and have a correlation between the logs and traces your application can report, we recommend using a JSONLayout formatter in the log4j2 configuration of your Mule apps.
The JSONLayout is placed into your appender config and must follow the below structure and configuration:
<JSONLayout compact="false" eventEol="true" properties="true" stacktraceAsString="true"> <KeyValuePair key="host" value="${sys:domain}" /> <KeyValuePair key="appName" value="${sys:domain}" /> <KeyValuePair key="ddsource" value="mule" /> <KeyValuePair key="service" value="mule" /> <KeyValuePair key="correlationId" value="$${ctx:correlationId:-}"/></JSONLayout>The layout parameters definition and documentation are in the official log4j2 documentation https://logging.apache.org/log4j/2.x/manual/layouts.html#JSONLayout
$${ctx:correlationId:-}: The <KeyValuePair
key="correlationId" value="$${ctx:correlationId:-}"/>
Is taking the correlation Id from Log4J2’s MDC[3]. The correlation Id is
coming from the Mule event and is injected by the Log4j2 context.
${sys:domain}:
This is an automatically
generated system property in the CloudHub
domain name. For On-Prem
you must set the "host"
and "appName" values through environment variables
or static values, where the "host"
is the server where your
application is hosted and the "appName"
is the name of your application.
Logs forwarding
There are two useful ways to forward logs. The logs can be collected by the Datadog agent from a directory or you can enable an HTTP appender to send the logs to the Datadog API.
Log collection through the agent [4]
Prerequisites
1. A Datadog agent running.
2. Your Mule applications must be customer hosted. The Datadog agent must coexist in the same server.
3. Have a directory where the logs can be collected.
Steps
1. In your application, add a new file appender in the log4j2 configuration file, find below a sample:
<File name="MyFile"
fileName="path/to/your/mule/log.log"
immediateFlush="true">
<JSONLayout …/>
</File>
2. Reference the appender in the Logger configuration as follows:
<Loggers>
<AsyncRoot level="INFO">
<AppenderRef ref="MyFile" />
</AsyncRoot>
</Loggers>
Search the folder of the Datadog agent in your documents. There, find the file datadog.yaml and open it. Search the text: #logs_enabled: false change the value to true and uncomment it. Save the changes.
o The default path directory of the Datadog agent is:
§ Windows: C:\ProgramData\Datadog
§ Linux: /etc/datadog-agent
3. In the Datadog agent folder, open conf.d folder. There, create a folder and name it java.d then add into it a file named java.yaml and add the next content:
#Log sectionlogs: - type: file path: "/path/to/your/mule/log.log" service: java source: java sourcecategory: sourcecodeNote: Make sure that the path value is the same as the fileName value in the log4j2 file of the Mule application.
Until here you have finished configuring the Agent and your applications. Please notice that this way of logging collection is only available for On-Prem applications, for your MuleSoft-hosted applications you must use an HTTP appender.
Log forwarding through an HTTP appender
Prerequisites
1. Have outbound HTTP connection.
2. Have the Datadog’s API key.
3.
For CloudHub, have visible
the Disable CloudHub logs
checkbox, this is not a visible
option in Runtime Manager. A special request to the Mulesoft support team is required
to enable this option. More info: https://docs.mulesoft.com/runtime-manager/custom-log-appender
Steps
1. Create an appender for Datadog in your application log4j2 configuration:
<Http name="DD-HTTP" url="https://http-intake.logs.datadoghq.com/v1/input" verifyHostname="false" method="POST"> <Property name="DD-API-KEY" value="${sys:ddApiKey}" /> <Property name="dd.trace_id" value="${ctx:dd.trace_id}" /> <JSONLayout … /></Http>1.1
${sys:ddApiKey}:
The DD-API-KEY header is a custom property that will be set with the
Datadog’s API key. For On-Prem the value can be set through an environment
variable or static value, for CloudHub use the application properties in Runtime
Manager.
1.2
${ctx:dd.trace_id}:
The dd.trace_id
is a customer header in the HTTP request that helps to inject the trace
identifier when using logging with APM tracing. More details about this in the
Datadog APM Connector.
2. This
step is required only if you want to log in CloudHub as well as in Datadog for
your MuleSoft-Hosted applications. In the log4j2 create an appender with the special tag Log4J2CloudhubLogAppender inside
the <Appenders>:
<Appenders>
<Log4J2CloudhubLogAppender
name="CLOUDHUB"
addressProvider="com.mulesoft.ch.logging.DefaultAggregatorAddressProvider"
applicationContext="com.mulesoft.ch.logging.DefaultApplicationContext"
appendRetryIntervalMs="${sys:logging.appendRetryInterval}"
appendMaxAttempts="${sys:logging.appendMaxAttempts}"
batchSendIntervalMs="${sys:logging.batchSendInterval}"
batchMaxRecords="${sys:logging.batchMaxRecords}"
memBufferMaxSize="${sys:logging.memBufferMaxSize}"
journalMaxWriteBatchSize="${sys:logging.journalMaxBatchSize}"
journalMaxFileSize="${sys:logging.journalMaxFileSize}"
clientMaxPacketSize="${sys:logging.clientMaxPacketSize}"
clientConnectTimeoutMs="${sys:logging.clientConnectTimeout}"
clientSocketTimeoutMs="${sys:logging.clientSocketTimeout}"
serverAddressPollIntervalMs="${sys:logging.serverAddressPollInterval}"
serverHeartbeatSendIntervalMs="${sys:logging.serverHeartbeatSendIntervalMs}"
statisticsPrintIntervalMs="${sys:logging.statisticsPrintIntervalMs}">
<PatternLayout pattern="[%d{MM-dd HH:mm:ss}] %-5p %c{1} [%t]:
%m%n"/>
</Log4J2CloudhubLogAppender>
</Appenders>
3. Enable your appender(s) to be used:
3.1 CloudHub
<Loggers> <AsyncRoot level="INFO"> <AppenderRef ref="CLOUDHUB"/> <AppenderRef ref="DD-HTTP"/> </AsyncRoot></Loggers>3.2 On-prem
<Loggers>
<AsyncRoot level="INFO"> <AppenderRef ref="DD-HTTP" /> </AsyncRoot>
</Loggers>Now you have configured the HTTP approach for sending logs, please notice that this option is available for both, customer-hosted and MuleSoft-hosted applications.
Note: When an application is deployed in CloudHub the Runtime Manager always overwrite the log4j2.xml. You need to disable the CloudHub logs to apply your project’s log4j2.xml configuration file in the runtime manager.
JMX [On-Prem]
The Mule Runtime Manager agent JMX service allows you to track specific JMX metrics and publish them to external services. The JMX service sends all the gathered metrics to publishers, which in turn publish the metrics to external monitoring tools [5].
Datadog is a monitoring service which has a lightweight Java plugin named JMXFetch. This plugin is called by the Datadog Agent to connect to the MBean Server and collect your application metrics. This plugin sends metrics to the Datadog Agent using the DogStatsD server running within the Agent[6].
Full monitoring of your mule runtimes is accomplished when you install the Datadog Mule integration into the agent, enables the Datadog APM and gather your mule runtime JVM metrics. These metrics enable a deep dive infrastructure inspection and correlate your instances/servers with the APM services, delivering a holistic view of your customer-hosted runtime planes, everything in a single platform.
JMX metrics at mule runtimes
To open JMX metrics consumption at your mule servers for the Datadog agent, you must run your mule runtime or add the below arguments [7] to your mule instance conf folder in a file named wrapper-additional.conf:
· -M-Dcom.sun.management.jmxremote
o To allow the JMX client access to the Mule Java VM.
· -M-Dcom.sun.management.jmxremote.port=portNum
o To enable JVM monitoring of your instance. In the property above, portNum is the port number through which you want to enable JMX RMI connections.
· -M-Dcom.sun.management.jmxremote.local.only=true
o If set to false, remote system monitoring can be configured but if you are running the Datadog agent in the same instance where your mule runtime is running this is not neccessary.
· -M-Dcom.sun.management.jmxremote.authenticate=false
o Password authentication for remote monitoring is enabled by default. To disable it, set the following system property when you start the Mule runtime, moreover all traffic is kept locally and no outbound monitoring is configured.
· -M-Dcom.sun.management.jmxremote.ssl=false
o To use client SSL authentication then set it true, N/A for local monitoring.
· -M-Djava.rmi.server.hostname=host
o
To enable JVM
monitoring of your instance. In the property above, host is the local IP address of your
server to enable JMX RMI connections.
Note: Above properties, like the host and port, will be used later in the Datadog agent configuration.
After running your Mule runtime with the above configuration, the agent can collect these metrics and report them into the Datadog platform, if your Mule runtime was running and you added the properties as a wrapper-additional.conf a restart of your instance is required, let’s configure the agent to do so.
Setting up the Datadog Agent for JMX fetch
Prerequisites
1. A Datadog agent running.
2. Your Mule applications must be customer hosted.
3. The Datadog agent must be able to communicate with the Mule server. Ensure that the configured host and port for the Mule runtime JMX service are reachable.
Configuration/Steps
· Configure the depicted file at the following link according to your needs and specs, please follow Datadog documentation:
o https://docs.datadoghq.com/integrations/java/?tab=host#configuration
· Restart the datadog agent
o Linux:
§ https://docs.datadoghq.com/agent/guide/agent-commands/?tab=agentv6v7#restart-the-agent
o Windows:
§ https://docs.datadoghq.com/agent/basic_agent_usage/windows/?tab=gui
· Let’s check the JMXFetch is running and gathering our instance(s):
o Please follow the below document to verify that JMXFetch check is executing correctly:
§ https://docs.datadoghq.com/agent/troubleshooting/agent_check_status/?tab=agentv6v7
§ When configured and started successfully, it should show you something similar like the below image:
Visualizing the JMX metrics
Now, let's confirm you are able to see the metrics in Datadog web page. Login into: https://app.datadoghq.com/account/login?next=%2F
· Once there, go to the left menu, select Dashboards and there, look for JVM Metrics or JVM - Overview dashboard.
o If the JVM Metrics dashboard is selected, then go up into the dashboard, below the dashboard name, and check for the $host variable, here select your host or “*” to show hosts gathered metrics.
o If the JVM - Overview dashboard is selected, then go up into the dashboard, below the dashboard name, and check for the $scope variable, here select your host or “*” to show hosts gathered metrics.
o Please find below an image of what it is depicted above:
· And that’s it you can visualize your JMX metrics OOB.
This link contains the list of the metrics you can obtain from JMX.
References
[1] https://www.datadoghq.com/product/log-management/#full-observability
[2] https://logging.apache.org/log4j/2.x/manual/layouts.html#JSONLayout
[3] https://logging.apache.org/log4j/2.x/manual/thread-context.html
[4] https://docs.datadoghq.com/logs/log_collection/?tab=http
[5] https://docs.mulesoft.com/runtime-manager/jmx-service
[6] https://docs.datadoghq.com/integrations/java/?tab=host#metric-collection
[7] https://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html