Export Confluent Cloud Metrics to Datadog

At one of my previous assignments, we had an environment where we were running our microservices on Azure AKS clusters and were connecting to Kafka clusters in Confluent Cloud. For our monitoring and logging, we were using Datadog. Datadog has an integration with Confluent, but that is not for the Confluent Cloud platform. Since we had all of our dashboards in Datadog we really liked to have the Confluent Cloud metrics in Datadog as well. With this blog post, I will show you how to get this done when you have a similar setup.

After a bit of searching, I found a Prometheus exporter for extracting metrics from the Confluent Cloud Metrics API. All I needed to do was set up this exporter in combination with the Datadog agent, so I could send those metrics to Datadog. As we were running everything on k8s I also wanted to run this exporter on our clusters as well. On my github you can find the manifest which creates a deployment with a ReplicaSet of one and has two containers, namely the ccloud exporter and the datadog agent. Once deployed it will start fetching metrics and exporting it to Datadog.

 

In Datadog you go to “Dashboard > New Dashboard > New Timeboard“. On your new dashboard, you add a widget by clicking on “Add graph” and you drag the Timeseries widget to the board. By default system.cpu.user is selected as metric, but when you click on it and fill in ccloud then you will see the available metrics. A list of available metrics can also be found in the Confluent Cloud documentation.

 

Documentation

If you need some help with creating dashboards you can check the Datadog documentation here. Once you have set up the widget with the metrics you like, you will get something like the dashboard below.

  • Production

If you have any questions then you can leave a comment or contact Bram via GithubLinkedin, or Twitter.

Originally published on Bram’s blog.

Bram is a cloud native engineer with a specialization in Azure, Kubernetes & Container Security. He is people oriented, inquisitively, eager to learn new things and he likes to take on a challenge.
May 07, 2024 | BLOG | 6 MINUTES

8 questions you were afraid to ask about Talos answerd

Talos is a minimal Kubernetes OS that's quickly gaining popularity because of its ease of use and strong focus on security by default. It has already been …

April 30, 2024 | BLOG | 9 MINUTES

12 Factor: 13 years later

How can we make applications easy to operate? The 12-factor methodology is about 13 years old. How did it age in the cloud-native era? Do we need a 13th …

April 25, 2024 | BLOG | 5 MINUTES

Build your own Python Kubernetes Operator

Yes, you read it right – build a K8s operator in Python! I often get reactions like, "But doesn't it have to be in Golang?" Fortunately, that's not …