Install Prometheus and Grafana
Install Prometheus, node_exporter and Grafana without Docker on Debian or Ubuntu, with dashboards, protected HTTPS access, updates and backups.
On this page
Your values
Enter them once and the commands in this guide update.
This guide explains how to install a complete monitoring platform, without Docker, on a VPS or dedicated server running Debian 12/13 or Ubuntu 22.04/24.04.
| Component | Purpose | Port | Listens on |
|---|---|---|---|
| node_exporter | Exposes system metrics | 9100 | 127.0.0.1 |
| Prometheus | Collects and stores metrics | 9090 | 127.0.0.1 |
| Grafana | Displays dashboards | 3000 | 127.0.0.1 |
| Nginx | Publishes Grafana over HTTPS | 80, 443 | Public |
Only Grafana is published, behind Nginx, over HTTPS and with an account. Prometheus and node_exporter are never exposed to the Internet.
Order a Server
To host your monitoring, HostMyServers offers several options:
Prerequisites
- A server running Debian 12/13 or Ubuntu 22.04/24.04, on the
amd64orarm64architecture - SSH access with a user that has
sudo - A domain name with an A record:
grafana.your-domain.commust point to the server's IP address - For Grafana, at least 512 MB of RAM and 1 CPU core according to its documentation, in addition to what Prometheus needs
- Disk space for the metrics: it depends on the number of series collected and the retention period
Prepare the server
sudo apt update
sudo apt upgrade -y
sudo apt install -y curl wget tar jq gnupg ufw nginx certbot python3-certbot-nginx apache2-utilsConfigure the firewall. Allow SSH before enabling UFW, and replace 22 if you use a different port.
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 22/tcp
sudo ufw allow 80,443/tcp
sudo ufw enablePorts 3000, 9090 and 9100 must not be opened.
Install Prometheus
Create the user and folders
sudo useradd --system --no-create-home --shell /usr/sbin/nologin prometheus
sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /var/lib/prometheusDownload and verify the archive
VERSION=3.13.5
ARCH=$(dpkg --print-architecture)
cd "$(mktemp -d)"
wget "https://github.com/prometheus/prometheus/releases/download/v${VERSION}/prometheus-${VERSION}.linux-${ARCH}.tar.gz"
wget "https://github.com/prometheus/prometheus/releases/download/v${VERSION}/sha256sums.txt"
sha256sum --check --ignore-missing sha256sums.txt
tar -xzf "prometheus-${VERSION}.linux-${ARCH}.tar.gz"
sudo install -m 0755 "prometheus-${VERSION}.linux-${ARCH}/prometheus" "prometheus-${VERSION}.linux-${ARCH}/promtool" /usr/local/bin/
prometheus --versionThe check must display OK for the downloaded archive.
Configure collection
sudo nano /etc/prometheus/prometheus.ymlglobal:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: prometheus
static_configs:
- targets: ["127.0.0.1:9090"]
- job_name: node
static_configs:
- targets: ["127.0.0.1:9100"]Check the file:
sudo chown -R root:prometheus /etc/prometheus
sudo chmod 750 /etc/prometheus
sudo chmod 640 /etc/prometheus/prometheus.yml
sudo promtool check config /etc/prometheus/prometheus.ymlCreate the systemd service
sudo nano /etc/systemd/system/prometheus.service[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file=/etc/prometheus/prometheus.yml \
--storage.tsdb.path=/var/lib/prometheus \
--storage.tsdb.retention.time=30d \
--web.listen-address=127.0.0.1:9090 \
--web.enable-admin-api
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
NoNewPrivileges=true
ProtectSystem=full
ProtectHome=true
PrivateTmp=true
[Install]
WantedBy=multi-user.target| Option | Effect |
|---|---|
--storage.tsdb.retention.time=30d |
Keeps 30 days of metrics (15 days by default) |
--web.listen-address=127.0.0.1:9090 |
Listens only on the local interface |
--web.enable-admin-api |
Enables the API used for snapshot backups |
Start the service:
sudo systemctl daemon-reload
sudo systemctl enable --now prometheus
sudo systemctl status prometheus --no-pagerInstall node_exporter
sudo useradd --system --no-create-home --shell /usr/sbin/nologin node_exporter
VERSION=1.13.0
ARCH=$(dpkg --print-architecture)
cd "$(mktemp -d)"
wget "https://github.com/prometheus/node_exporter/releases/download/v${VERSION}/node_exporter-${VERSION}.linux-${ARCH}.tar.gz"
wget "https://github.com/prometheus/node_exporter/releases/download/v${VERSION}/sha256sums.txt"
sha256sum --check --ignore-missing sha256sums.txt
tar -xzf "node_exporter-${VERSION}.linux-${ARCH}.tar.gz"
sudo install -m 0755 "node_exporter-${VERSION}.linux-${ARCH}/node_exporter" /usr/local/bin/
sudo nano /etc/systemd/system/node_exporter.service[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter --web.listen-address=127.0.0.1:9100
Restart=on-failure
NoNewPrivileges=true
[Install]
WantedBy=multi-user.targetsudo systemctl daemon-reload
sudo systemctl enable --now node_exporter
curl -s http://127.0.0.1:9100/metrics | head -n 5Check that Prometheus is collecting from both targets. Wait about fifteen seconds after startup:
curl -s http://127.0.0.1:9090/api/v1/targets | jq -r '.data.activeTargets[] | "\(.labels.job) \(.health)"'The expected result is:
node up
prometheus upInstall Grafana
Add the official repository
sudo mkdir -p /etc/apt/keyrings
sudo wget -O /etc/apt/keyrings/grafana.asc https://apt.grafana.com/gpg-full.key
sudo chmod 644 /etc/apt/keyrings/grafana.asc
echo "deb [signed-by=/etc/apt/keyrings/grafana.asc] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update
sudo apt install -y grafanaRestrict Grafana to the local interface
Grafana's initial account is admin / admin. Restrict where it listens before the first start:
sudo nano /etc/grafana/grafana.iniIn the [server] section, uncomment or add:
[server]
http_addr = 127.0.0.1
http_port = 3000Start the service and check that it responds:
sudo systemctl daemon-reload
sudo systemctl enable --now grafana-server
sudo ss -tlnp | grep 3000
curl -s http://127.0.0.1:3000/api/healthChange the administrator password
Change the password before publishing Grafana. From your own computer, open an SSH tunnel and leave that session open:
ssh -L 3000:127.0.0.1:3000 YOUR_USER@YOUR_SERVER_IPOpen http://localhost:3000, sign in with admin / admin, then set a strong password when Grafana asks for one. You can then close the tunnel.
Declare Prometheus as a data source
sudo nano /etc/grafana/provisioning/datasources/prometheus.yamlapiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://127.0.0.1:9090
isDefault: true
editable: falsesudo chown root:grafana /etc/grafana/provisioning/datasources/prometheus.yaml
sudo chmod 640 /etc/grafana/provisioning/datasources/prometheus.yaml
sudo systemctl restart grafana-serverPublish Grafana over HTTPS
Create the Nginx site
sudo nano /etc/nginx/sites-available/grafana.confmap $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
upstream grafana {
server 127.0.0.1:3000;
}
server {
listen 80;
listen [::]:80;
server_name grafana.your-domain.com;
location / {
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://grafana;
}
location /api/live/ {
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Host $host;
proxy_pass http://grafana;
}
}sudo ln -s /etc/nginx/sites-available/grafana.conf /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx
sudo certbot --nginx -d grafana.your-domain.com -m admin@your-domain.com --agree-tos --no-eff-email --redirect
sudo certbot renew --dry-runFor more details on certificates, see the Install an SSL Certificate with Let's Encrypt guide.
Finalize the Grafana configuration
sudo nano /etc/grafana/grafana.iniAdjust the following sections:
[server]
http_addr = 127.0.0.1
http_port = 3000
domain = grafana.your-domain.com
root_url = https://grafana.your-domain.com/
[security]
cookie_secure = true
cookie_samesite = lax
strict_transport_security = true
[users]
allow_sign_up = falsesudo systemctl restart grafana-serverImport a dashboard
- Open
https://grafana.your-domain.comand sign in. - Click Dashboards in the main menu.
- Click New, then Import dashboard.
- Enter the ID
1860(Node Exporter Full) and click Load. - Select the Prometheus data source, then click Import.
Verify the installation
On the server, no internal service must listen on a public address:
sudo ss -tlnp | grep -E ':(3000|9090|9100)\b'Each line must start with 127.0.0.1. From your own computer, these commands must fail:
curl -m 5 http://YOUR_SERVER_IP:3000/
curl -m 5 http://YOUR_SERVER_IP:9090/
curl -m 5 http://YOUR_SERVER_IP:9100/metricsFinally, https://grafana.your-domain.com must show the sign-in page, with a valid certificate.
To use the Prometheus interface, go through an SSH tunnel, then open http://localhost:9090:
ssh -L 9090:127.0.0.1:9090 YOUR_USER@YOUR_SERVER_IPProtect Prometheus with a password (optional)
Listening locally already prevents any access from the Internet. Add a password if other users or applications have access to the server.
First generate the password hash:
htpasswd -nBC 12 "" | tr -d ':\n'; echoCreate /etc/prometheus/web.yml with sudo nano, then paste the hash into it:
basic_auth_users:
admin: "BCRYPT_HASH"Also store the plain-text password, which Prometheus needs to collect its own metrics:
sudo install -o root -g prometheus -m 640 /dev/null /etc/prometheus/web-password
sudo nano /etc/prometheus/web-password
sudo chown root:prometheus /etc/prometheus/web.yml
sudo chmod 640 /etc/prometheus/web.yml
sudo promtool check web-config /etc/prometheus/web.ymlThen apply these three changes:
-
In
prometheus.service, add--web.config.file=/etc/prometheus/web.ymlto theExecStartcommand. -
In
prometheus.yml, complete theprometheusjob:YAML- job_name: prometheus basic_auth: username: admin password_file: /etc/prometheus/web-password static_configs: - targets: ["127.0.0.1:9090"] -
In the Grafana data source, add the credentials:
YAMLbasicAuth: true basicAuthUser: admin secureJsonData: basicAuthPassword: YOUR_PROMETHEUS_PASSWORD
Reload the configuration and restart the services:
sudo promtool check config /etc/prometheus/prometheus.yml
sudo systemctl daemon-reload
sudo systemctl restart prometheus grafana-serverFrom now on, add -u admin to the curl commands sent to port 9090.
Monitor another server
Install node_exporter on the second server, with --web.listen-address=0.0.0.0:9100 in its service. On that second server, allow only your Prometheus server:
sudo ufw allow from YOUR_SERVER_IP to any port 9100 proto tcpThen add the target in /etc/prometheus/prometheus.yml and reload Prometheus:
- job_name: node
static_configs:
- targets: ["127.0.0.1:9100", "SECOND_SERVER_IP:9100"]sudo promtool check config /etc/prometheus/prometheus.yml
sudo systemctl reload prometheusUpdate
Back up your data first and read the release notes.
Prometheus: repeat the download block with the new VERSION value, then:
sudo promtool check config /etc/prometheus/prometheus.yml
sudo systemctl restart prometheus
prometheus --versionnode_exporter: likewise, repeat its download and installation, then:
sudo systemctl restart node_exporter
node_exporter --versionGrafana: the package is updated with APT.
sudo apt update
sudo apt install --only-upgrade grafana
sudo systemctl restart grafana-serverBack up and restore
Prometheus
The Prometheus documentation recommends snapshots:
SNAPSHOT=$(curl -fsS -X POST http://127.0.0.1:9090/api/v1/admin/tsdb/snapshot | jq -r '.data.name')
sudo tar -czpf "/root/prometheus-${SNAPSHOT}.tar.gz" -C /var/lib/prometheus/snapshots "$SNAPSHOT"
sudo rm -rf "/var/lib/prometheus/snapshots/${SNAPSHOT}"
sudo tar -czpf "/root/prometheus-config-$(date +%F).tar.gz" /etc/prometheus /etc/systemd/system/prometheus.serviceTo restore a snapshot:
sudo systemctl stop prometheus
sudo mv /var/lib/prometheus /var/lib/prometheus.old
sudo mkdir /var/lib/prometheus
sudo tar -xzpf /root/prometheus-SNAPSHOT_NAME.tar.gz --strip-components=1 -C /var/lib/prometheus
sudo chown -R prometheus:prometheus /var/lib/prometheus
sudo systemctl start prometheusGrafana
By default, Grafana stores its accounts and dashboards in the SQLite database /var/lib/grafana/grafana.db. Stop the service while copying:
sudo systemctl stop grafana-server
sudo tar -czpf "/root/grafana-$(date +%F).tar.gz" /var/lib/grafana /etc/grafana
sudo systemctl start grafana-serverTo restore:
sudo systemctl stop grafana-server
sudo tar -xzpf /root/grafana-YYYY-MM-DD.tar.gz -C /
sudo chown -R grafana:grafana /var/lib/grafana
sudo systemctl start grafana-serverThese archives contain secrets: restrict their permissions with sudo chmod 600 /root/*.tar.gz and copy them off the server.
Troubleshooting
Prometheus does not start
sudo journalctl -u prometheus -n 50 --no-pager
sudo promtool check config /etc/prometheus/prometheus.yml
sudo ls -ld /var/lib/prometheuspermission denied on the data folder is fixed with sudo chown -R prometheus:prometheus /var/lib/prometheus. address already in use means another process is using port 9090: identify it with sudo ss -tlnp | grep 9090.
A target is down
Check the lastError field:
curl -s http://127.0.0.1:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health, lastError}'connection refused means the exporter is stopped or listens on a different address. A context deadline exceeded toward another server points more to the firewall.
502 Bad Gateway error on Grafana
sudo systemctl status grafana-server --no-pager
sudo journalctl -u grafana-server -n 50 --no-pager
curl -s http://127.0.0.1:3000/api/healthLogin loop or "origin not allowed" message
Check that domain and root_url exactly match the address used in the browser, and that Nginx passes the Host header. cookie_secure = true requires HTTPS access.
The dashboard shows no data
- Check that both targets are
up. - Widen the displayed time range to a few hours.
- In the dashboard, check that the selected data source,
joband instance are correct.
Lost Grafana administrator password
Reset it from the server, then change it again in the interface:
sudo -u grafana grafana cli --homepath /usr/share/grafana --config /etc/grafana/grafana.ini admin reset-admin-password TEMPORARY_PASSWORDThe disk is filling up
Reduce --storage.tsdb.retention.time, or add a limit with --storage.tsdb.retention.size=10GB, then run sudo systemctl daemon-reload and sudo systemctl restart prometheus. The Check Disk Space and Usage in Linux guide helps you locate what is using the space.
Going further
If you are looking for a simpler solution for a single server, see the Monitor Your Server with Netdata guide.