At some point, logs alone stop being enough. You start seeing short spikes in CPU, random latency jumps, or brief outages that don’t leave a clear trace in the logs. That’s where real monitoring pays off: numbers, charts, history, and alerts.

This guide walks through a complete, practical setup:

Run Prometheus with Docker Compose
Put Prometheus behind Traefik with TLS
Fix a common UI issue in newer Prometheus releases
Install Node Exporter on the host using systemd
Connect containerized Prometheus to host-installed Node Exporter
Put Node Exporter behind Traefik as well, so it’s reachable via HTTPS (secured with an IP whitelist)

What is Prometheus?

Prometheus is a time-series monitoring system. Exporters and services usually expose metrics on an HTTP endpoint such as /metrics. Prometheus periodically scrapes these endpoints, stores the data, and lets you query it using PromQL (and later build dashboards/alerts with Grafana/Alertmanager).

Prerequisites

Docker + Docker Compose installed on your server
Traefik already running and capable of issuing TLS certificates (e.g., certresolver=mytlschallenge)
A Docker external network named traefik
DNS records in place for:
- prometheus.mahdishadi.me
- nodeexporter.mahdishadi.me

Part 1: Run Prometheus behind Traefik with TLS

1) Create a project directory

mkdir -p /opt/monitoring/prometheus
cd /opt/monitoring/prometheus

2) Create `prometheus.yml`

Start simple: scrape Prometheus itself.

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["prometheus:9090"]

3) Create `docker-compose.yml` (Prometheus behind Traefik)

version: "3.8"

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus

    # Useful for debugging. If you want Traefik-only access, you can remove this.
    ports:
      - "9090:9090"

    labels:
      - traefik.enable=true
      - traefik.http.routers.prometheus.rule=Host(`prometheus.mahdishadi.me`)
      - traefik.http.routers.prometheus.entrypoints=web,websecure
      - traefik.http.routers.prometheus.tls=true
      - traefik.http.routers.prometheus.tls.certresolver=mytlschallenge

      # Optional security headers
      - traefik.http.middlewares.prometheus.headers.SSLRedirect=true
      - traefik.http.middlewares.prometheus.headers.STSSeconds=315360000
      - traefik.http.middlewares.prometheus.headers.browserXSSFilter=true
      - traefik.http.middlewares.prometheus.headers.contentTypeNosniff=true
      - traefik.http.middlewares.prometheus.headers.forceSTSHeader=true
      - traefik.http.middlewares.prometheus.headers.STSIncludeSubdomains=true
      - traefik.http.middlewares.prometheus.headers.STSPreload=true
      - traefik.http.routers.prometheus.middlewares=prometheus@docker

    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus

    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      # Helps if the new UI loads blank/spins on /query in some setups
      - "--enable-feature=old-ui"

    restart: unless-stopped
    networks:
      - traefik

volumes:
  prometheus-data:

networks:
  traefik:
    external: true

4) Start Prometheus

docker compose up -d

Access:

https://prometheus.mahdishadi.me

Part 2: If the UI is blank, verify Prometheus is healthy

If the UI is stuck/blank, check the server directly via API endpoints:

curl -sS http://127.0.0.1:9090/-/ready
curl -sS http://127.0.0.1:9090/api/v1/status/buildinfo | head
curl -sS 'http://127.0.0.1:9090/api/v1/query?query=up'

If you get “Ready” and the up query returns a result, Prometheus is healthy and it’s likely a frontend/UI issue.
Enabling --enable-feature=old-ui (already included above) typically fixes this scenario.

Part 3: Install Node Exporter on the host (systemd)

Node Exporter provides host-level metrics: CPU, memory, disk, network, load, etc.

1) Create a dedicated user

sudo useradd --no-create-home --shell /usr/sbin/nologin node_exporter

2) Check system architecture

uname -m

x86_64 → linux-amd64
aarch64 → linux-arm64

3) Download and install the binary (example: amd64)

cd /tmp
VER="1.8.1"

wget https://github.com/prometheus/node_exporter/releases/download/v${VER}/node_exporter-${VER}.linux-amd64.tar.gz
tar xvf node_exporter-${VER}.linux-amd64.tar.gz

sudo mv node_exporter-${VER}.linux-amd64/node_exporter /usr/local/bin/
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter
sudo chmod +x /usr/local/bin/node_exporter

(If you’re on ARM64, download node_exporter-${VER}.linux-arm64.tar.gz instead.)

4) Create a systemd service

sudo tee /etc/systemd/system/node_exporter.service > /dev/null <<'EOF'
[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter --web.listen-address=0.0.0.0:9100
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

5) Enable and start the service

sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter
sudo systemctl status node_exporter --no-pager

6) Test it locally

curl -sS http://127.0.0.1:9100/metrics | head

Part 4: Connect containerized Prometheus to host-installed Node Exporter

On Linux, host.docker.internal is not always available by default inside containers. The clean solution is to map it using Docker’s host-gateway.

1) Add `extra_hosts` to Prometheus in `docker-compose.yml`

extra_hosts:
  - "host.docker.internal:host-gateway"

Example (relevant section):

services:
  prometheus:
    ...
    extra_hosts:
      - "host.docker.internal:host-gateway"

Then recreate:

docker compose up -d --force-recreate

2) Add the Node Exporter job to `prometheus.yml`

  - job_name: "Mahdi Host"
    static_configs:
      - targets: ["host.docker.internal:9100"]

Final prometheus.yml:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["prometheus:9090"]

  - job_name: "Mahdi Host"
    static_configs:
      - targets: ["host.docker.internal:9100"]

Recreate Prometheus:

docker compose up -d --force-recreate

In Prometheus UI:

Status → Targets
The Mahdi Host job should be UP

Part 5: Put Node Exporter behind Traefik with HTTPS

Node Exporter does not provide TLS by itself. To expose it via HTTPS, put Traefik in front of it and let Traefik terminate TLS.

1) Ensure Traefik can reach the host

In Traefik’s own compose file (the one that runs Traefik), add:

services:
  traefik:
    extra_hosts:
      - "host.docker.internal:host-gateway"

2) Enable Traefik File Provider

This is the clean approach for services that are not Docker containers.

In Traefik’s compose:

services:
  traefik:
    volumes:
      - /opt/traefik/dynamic:/etc/traefik/dynamic
    command:
      - "--providers.file.directory=/etc/traefik/dynamic"
      - "--providers.file.watch=true"

Create the directory:

sudo mkdir -p /opt/traefik/dynamic

3) Create a dynamic config for Node Exporter (TLS + IP whitelist)

Create:

/opt/traefik/dynamic/nodeexporter.yml

http:
  routers:
    nodeexporter:
      rule: Host(`nodeexporter.mahdishadi.me`)
      entryPoints:
        - websecure
      tls:
        certResolver: mytlschallenge
      service: nodeexporter
      middlewares:
        - nodeexp-ipwhitelist

  services:
    nodeexporter:
      loadBalancer:
        servers:
          - url: "http://host.docker.internal:9100"

  middlewares:
    nodeexp-ipwhitelist:
      ipWhiteList:
        sourceRange:
          - "YOUR_PUBLIC_IP/32"

Get your public IP:

curl -s https://ifconfig.me

Replace YOUR_PUBLIC_IP/32 with the output, e.g. 1.2.3.4/32.

Recreate Traefik:

docker compose up -d --force-recreate

Access (Node Exporter’s main endpoint):

https://nodeexporter.mahdishadi.me/metrics

Node Exporter serves metrics on /metrics. Hitting / may not show useful output.

Part 6: Dealing with `403 Forbidden` on Node Exporter HTTPS

A 403 here is usually caused by the IP whitelist:

Your current IP is not included in sourceRange
You are on VPN (your public IP changed)

Update sourceRange with your actual public IP and reload/recreate Traefik.

Quick verification commands

Node Exporter directly on the host

curl -sS http://127.0.0.1:9100/metrics | head

Node Exporter via Traefik HTTPS

curl -sS https://nodeexporter.mahdishadi.me/metrics | head

Prometheus container scraping the host

docker exec -it prometheus sh -c 'wget -qO- http://host.docker.internal:9100/metrics | head'

Security note (important)

Node Exporter has no authentication. If exposed to the internet, always protect it:

IP whitelist (as shown), and/or
Basic auth, and/or
keep it internal-only and let Prometheus scrape it privately

Prometheus scraping does not require HTTPS; HTTPS is mainly for safe human/browser access.

When Logs Weren’t Enough: Setting Up Prometheus Behind Traefik (TLS)

Comments

More from this blog

Deploying Jitsi Meet Behind Traefik with Docker Compose (Custom SSL PEM/KEY + Host Authentication)

How I Put n8n Behind Traefik with Automatic HTTPS: A Real-World Traefik + n8n Setup

Want Seamless, Scalable Storage in Kubernetes? Here’s Why Longhorn is Your Best Bet!

Zero-Downtime Zabbix Migration: How I Moved from 6.4 to 7.0 Without Losing a Single Metric

What is Prometheus?

Prerequisites

Part 1: Run Prometheus behind Traefik with TLS

1) Create a project directory

2) Create `prometheus.yml`

3) Create `docker-compose.yml` (Prometheus behind Traefik)

4) Start Prometheus

Part 2: If the UI is blank, verify Prometheus is healthy

Part 3: Install Node Exporter on the host (systemd)

1) Create a dedicated user

2) Check system architecture

3) Download and install the binary (example: amd64)

4) Create a systemd service

5) Enable and start the service

6) Test it locally

Part 4: Connect containerized Prometheus to host-installed Node Exporter

1) Add `extra_hosts` to Prometheus in `docker-compose.yml`

2) Add the Node Exporter job to `prometheus.yml`

Part 5: Put Node Exporter behind Traefik with HTTPS

1) Ensure Traefik can reach the host

2) Enable Traefik File Provider

3) Create a dynamic config for Node Exporter (TLS + IP whitelist)

Part 6: Dealing with `403 Forbidden` on Node Exporter HTTPS

Quick verification commands

Node Exporter directly on the host

Node Exporter via Traefik HTTPS

Prometheus container scraping the host

Security note (important)

Command Palette

Comments

More from this blog

What is Prometheus?

Prerequisites

Part 1: Run Prometheus behind Traefik with TLS

1) Create a project directory

2) Create prometheus.yml

3) Create docker-compose.yml (Prometheus behind Traefik)

4) Start Prometheus

Part 2: If the UI is blank, verify Prometheus is healthy

Part 3: Install Node Exporter on the host (systemd)

1) Create a dedicated user

2) Check system architecture

3) Download and install the binary (example: amd64)

4) Create a systemd service

5) Enable and start the service

6) Test it locally

Part 4: Connect containerized Prometheus to host-installed Node Exporter

1) Add extra_hosts to Prometheus in docker-compose.yml

2) Add the Node Exporter job to prometheus.yml

Part 5: Put Node Exporter behind Traefik with HTTPS

1) Ensure Traefik can reach the host

2) Enable Traefik File Provider

3) Create a dynamic config for Node Exporter (TLS + IP whitelist)

Part 6: Dealing with 403 Forbidden on Node Exporter HTTPS

Quick verification commands

Node Exporter directly on the host

Node Exporter via Traefik HTTPS

Prometheus container scraping the host

Security note (important)

2) Create `prometheus.yml`

3) Create `docker-compose.yml` (Prometheus behind Traefik)

1) Add `extra_hosts` to Prometheus in `docker-compose.yml`

2) Add the Node Exporter job to `prometheus.yml`

Part 6: Dealing with `403 Forbidden` on Node Exporter HTTPS