This post will collect information on how I have configured Prometheus in case there is a need to rebuild or make changes in the future.
At the moment I have the Prometheus server running on a Raspberry Pi 3 Model B. There is plenty of good information on how to quickly set up both a Prometheus server and the various node_ and other exporters for Prometheus out on the Prometheus Documentation Website and other locations which can be used for reference. I specifically want to focus on modifications to the default installation configuration which make it easier for me to quickly add additional collection targets without repeatedly altering the core /etc/prometheus/prometheus.yml for each target.
Sticking with the defaults, the Prometheus service is running on host ceed-chuck and available on port 9090. An instance of the node-exporter is also installed and running on the same host and is available via port 9100.
Enhancements to the Default Prometheus Configuration
Create a targets directory, /etc/prometheus/targets.d/.
Add node.yml config file the /etc/prometheus/targets.d/ listing target hosts and ports. In the following sample I am just adding a single target for the node_exporter on host ceed-chuck.
Modify /etc/prometheus/prometheus.yml file to include information from YAML file in new directory as part of configuration. Specifically, adding the following:
To the existing scrape_configs: section:
So that the /etc/prometheuse/prometheus.yml file ends up with a scrpes_config: section that now looks like this:
After doing the above, adding new targets for Prometheus server to scrape is a matter of:
Adding new host and port to the node.yml file in /etc/prometheus/targets.d/ directory.
Restarting the Prometheus service, sudo systemctl restart prometheus.
The next section is a snapshot of the state file I currently use to install Prometheus server.
Salt State for Prometheus Needful
app_prometheus file system layout
Chrls is currently a technology transformation leader at one of the oldest American investment banking services holding companies, headquartered in New York City. Previous roles at current employer include project manager, front office support, retail and enterprise global Windows server support. Currently residing in Tennessee on the Cumberland Plateau and enjoying life, family and open source.