Prometheus and Grafana Explained: Monitoring and Visualizing Kubernetes Metrics Like a Pro
- Nitin Yadav
- Blog
About
Set up Prometheus and Grafana for real-time monitoring and visualization in Kubernetes environments. Learn step-by-step configuration and best practices for DevOps success.
Industries
- DevOps, Grafana, Kubernetes, Prometheus
Share Via
Introduction
In the fast-evolving landscape of DevOps, monitoring is a key factor that sets operations apart. In terms of microservices, cloud infrastructure, or container applications, having real-time observability is one of the important requirements. This will ensure smooth operations and prevent high downtime. Prometheus and Grafana play a vital role to monitor and maintain reliable applications hosted on Kubernetes based services, such as AWS. Both of them together make an excellent combination of monitoring and visualization capabilities.
Why Monitoring Matters
Nowadays, often the infrastructure is complex and spans across cloud providers (AWS, GCP, Azure, etc), container environments, and on-premise data centers. Such complex application environment setups make it difficult to identify performance bottlenecks or sudden unexpected outages and the need for scaling the application. So, without proper monitoring, the development or devops teams need to operate without visibility, which demands a lot of effort while resolving the problem and hindering the ability to build reliable systems.
What is Prometheus
Prometheus is an open-source systems monitoring and alerting toolkit.It is widely used to track metrics, analyze trends, and trigger alerts. This monitoring tool is built for dynamic environments. Prometheus collects and stores its metrics as time series data, i.e. metrics information is stored with the timestamp at which it was recorded, alongside optional key-value pairs called labels. Prometheus is used mostly in the Kubernetes based applications.
Key Features of Prometheus
- Time-Series Data Storage: Prometheus collects real-time metrics and stores them efficiently as time series data.
- Flexible Query Language (PromQL): Prometheus provides a functional query language called PromQL (Prometheus Query Language) that lets the user select and aggregate time series data in real time.
- Pull-Based Metrics Collection: Prometheus actively scrapes application metrics from the configured endpoints. So it doesn’t rely on other services to push data. This ensures a more scalable and reliable monitoring setup. Time series collection happens via a pull model over HTTP.
- Alerting Mechanism: Alerting rules allow us to define alert conditions based on Prometheus expression language expressions and to send notifications about firing alerts to an external service. Whenever the alert expression results in one or more vector elements at a given point in time, the alert counts as active for these elements’ label sets.
- Integration Support: Prometheus supports integration with Kubernetes, Docker, and some other services. There are a number of libraries and servers which help in exporting existing metrics from third-party systems as Prometheus metrics. This is useful for cases where it is not feasible to instrument a given system with Prometheus metrics directly.
Architecture of Prometheus
This diagram illustrates the architecture of Prometheus and some of its ecosystem components:Â
Setting up Prometheus
- Download the latest release of Prometheus for your platform. In my case, I am using Linux, so I’ll run the below commands:
2. Configure Prometheus. We can use the below given configuration (yaml) file as an example:Â
3. To start Prometheus with the newly created example configuration file, change to the directory containing the Prometheus binary and run the cli command:Â
4. After running the above command, Prometheus should start up. We can browse to a status page about itself at localhost:9090. Give it a couple of seconds to collect data about itself from its own HTTP metrics endpoint.
5. We can also verify that Prometheus is serving metrics about itself by navigating to its metrics endpoint: localhost:9090/metrics.
6. To use Prometheus’s built-in expression browser, navigate to http://localhost:9090/graph and choose the “Table” view within the “Graph” tab.
7. To graph expressions, navigate to http://localhost:9090/graph and use the “Graph” tab. For example, enter the following expression to graph the per-second rate of chunks being created in the self-scraped Prometheus:
8. To configure sample targets for Prometheus to scrape, use the below given commands. The Node Exporter is used as an example target.Â
9. Once the above commands are executed, we’ll have example targets listening on http://localhost:8080/metrics, http://localhost:8081/metrics, and http://localhost:8082/metrics.
10. To record the time series resulting from this expression into a new metric called job_instance_mode:node_cpu_seconds:avg_rate5m, create a file with the following recording rule and save it as prometheus.rules.yml:Â
11. To make Prometheus pick up the above given new rule, add a rule_files statement in the prometheus.yml file. The config should look like below one:Â
12. The next step is to restart Prometheus with the new configuration and verify that a new time series with the metric name job_instance_mode:node_cpu_seconds:avg_rate5m is now available by querying it through the expression browser or graphing it.
Â
What is Grafana
Grafana is an open source software that helps us to query, visualize, alert on, and explore our metrics, logs, and traces wherever they are stored. Grafana integrates seamlessly with Prometheus. Grafana pulls data from the time-series database of Prometheus. This helps to create charts, graphs, and alerts. With the help of Grafana’s rich visualization, teams can easily spot trends, analyze system health, and diagnose issues. Grafana features its own Prometheus HTTP endpoint that can be scraped by Prometheus, making it fairly easy to get statistics and dashboards.
Â
Key Features of Grafana
- Interactive Dashboards: Grafana provides good real-time, customizable visualizations. The basic building block for visualization in Grafana is the panel, and that panel can contain a graph, a Singlestat, a table, a heatmap, and freetext. Dashboard is a collection of panels, each of which holds a set of variables (things like sensor name, application, and server). These panels are arranged in a grid on the Grafana dashboard
- Multi-Source Compatibility: Grafana supports pulling data from multiple sources, like – Prometheus, Elasticsearch, InfluxDB, etc.
- Alerting and Notifications: Grafana can send alerts based on the set metric thresholds. Grafana alerts can be sent through several different notifiers, including email, PagerDuty, or Slack or texts.
- User Access Controls: Grafana can allow role-based dashboard access. The organization can create a team of users and then assign permissions on folders, Grafana dashboards, and so on.
- Querying and Exploration: Grafana helps to deep dive into the application logs and the metrics to analyze the application performance. Based on the analysis, we can take action if required.
Setting up Grafana
1. Install Grafana. In my case, I’ll install Grafana open source version on Ubuntu OS, so I need to run the below commands:Â
For more info on Grafana installation, check out the official documentation.
2. To sign in to Grafana for the first time, follow the below steps:
a. Use the link to access Grafana: http://localhost:3000/.Â
b. On the sign-in page, enter admin for both the username and password. These are the default credentials for Grafana admin user.
c. Click Sign in.
d. If successful, you’ll see a prompt to change the password.
e. Click OK on the prompt and change your password.
3. To create a dashboard in Grafana console, follow these steps:
a. Click on the Dashboards menu.Â
b. Click on Create dashboard.
c. click Add visualization.
d. In the dialog box that opens, click — Grafana —
e. Now a query will be configured and the Random Walk dashboard will be generated.
f. Click Refresh to query the data source.
g. When finished editing the panel, click Save dashboard.
How Prometheus and Grafana Work Together
- Prometheus collects and stores time-series metrics from various components.
- Grafana queries Prometheus and visualizes the data in real-time.
- Alerting in Prometheus and Grafana helps to detect anomalies as early as possible.
- Together, they provide a comprehensive monitoring solution. This allows DevOps teams to track the infrastructure performance easily.
Conclusion
Prometheus and Grafana form a powerful duo for monitoring and visualizing Kubernetes environments. Prometheus offers robust time-series data collection and alerting capabilities, while Grafana turns this raw data into insightful visual dashboards. When used together, they empower DevOps teams with real-time observability, faster issue detection, and data-driven decision-making. Whether you’re scaling a microservices architecture or ensuring the uptime of critical cloud-native applications, integrating Prometheus and Grafana into your Kubernetes setup is a smart move towards building resilient, high-performing systems.
Â
Frequently asked questions
 Prometheus is used in Kubernetes to collect, store, and query real-time metrics. It helps monitor application performance, system resource usage, and cluster health, enabling proactive alerting and faster issue detection.
 Grafana connects to Prometheus as a data source, queries its time-series metrics, and displays them using interactive dashboards. This integration provides visual insights into system health and performance trends.
 Monitoring in Kubernetes is essential to detect failures, performance issues, and scaling needs. It offers real-time visibility, helping DevOps teams maintain system reliability and ensure efficient resource usage.
Prometheus offers powerful querying (PromQL), pull-based metrics collection, built-in alerting, and efficient time-series storage. It’s ideal for dynamic, containerized environments like Kubernetes.
 Yes, Prometheus includes an alerting mechanism that uses rules to trigger notifications when specific conditions are met. It integrates with Alertmanager to send alerts via email, Slack, or other channels.
Yes, Grafana offers a free and open-source version that includes core features like dashboard creation, visualization, and alerting. Paid plans offer advanced features and support for enterprise use.
Download the Prometheus binary, extract the archive, configure the prometheus.yml file, and run Prometheus using ./prometheus –config.file=prometheus.yml. It starts a local server on port 9090.
PromQL is Prometheus Query Language used to retrieve and analyze time-series data. It supports filtering, aggregation, and mathematical operations to generate actionable insights.
Dashboards in Grafana are collections of visual panels that display metrics from data sources like Prometheus. They allow real-time tracking of application performance and system health.
Node Exporters are agents that expose hardware and OS metrics like CPU, memory, and disk usage in Prometheus format. They run on target machines and are scraped by Prometheus for monitoring.
Related Posts
Prometheus and Grafana Explained: Monitoring and Visualizing Kubernetes Metrics Like a Pro
- Blog
CI/CD Pipeline Failures Explained: Key Debugging Techniques to Resolve Build and Deployment Issues
- Blog
DevSecOps in Action: A Complete Guide to Secure CI/CD Workflows
- Blog
AWS WAF Explained: Protect Your APIs with Smart Rate Limiting
- Blog
How to Use AWS IAM Identity Center for Scalable, Compliant Cloud Access Control
- Blog
How to Choose Between In-Memory Data Stores and Caches for High-Performance Applications
- Blog