Skip to main content
HashnodeMahdi Shadi

Command Palette

Search for a command to run...

When Logs Weren’t Enough: Setting Up Prometheus Behind Traefik (TLS)

Updated
•5 min read
When Logs Weren’t Enough: Setting Up Prometheus Behind Traefik (TLS)
M
Mahdi Shadi

At some point, logs alone stop being enough. You start seeing short spikes in CPU, random latency jumps, or brief outages that don’t leave a clear trace in the logs. That’s where real monitoring pays off: numbers, charts, history, and alerts.

This guide walks through a complete, practical setup:

  • Run Prometheus with Docker Compose

  • Put Prometheus behind Traefik with TLS

  • Fix a common UI issue in newer Prometheus releases

  • Install Node Exporter on the host using systemd

  • Connect containerized Prometheus to host-installed Node Exporter

  • Put Node Exporter behind Traefik as well, so it’s reachable via HTTPS (secured with an IP whitelist)

What is Prometheus?

Prometheus is a time-series monitoring system. Exporters and services usually expose metrics on an HTTP endpoint such as /metrics. Prometheus periodically scrapes these endpoints, stores the data, and lets you query it using PromQL (and later build dashboards/alerts with Grafana/Alertmanager).

Prerequisites

  • Docker + Docker Compose installed on your server

  • Traefik already running and capable of issuing TLS certificates (e.g., certresolver=mytlschallenge)

  • A Docker external network named traefik

  • DNS records in place for:

Part 1: Run Prometheus behind Traefik with TLS

1) Create a project directory

mkdir -p /opt/monitoring/prometheus
cd /opt/monitoring/prometheus

2) Create prometheus.yml

Start simple: scrape Prometheus itself.

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["prometheus:9090"]

3) Create docker-compose.yml (Prometheus behind Traefik)

version: "3.8"

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus

    # Useful for debugging. If you want Traefik-only access, you can remove this.
    ports:
      - "9090:9090"

    labels:
      - traefik.enable=true
      - traefik.http.routers.prometheus.rule=Host(`prometheus.mahdishadi.me`)
      - traefik.http.routers.prometheus.entrypoints=web,websecure
      - traefik.http.routers.prometheus.tls=true
      - traefik.http.routers.prometheus.tls.certresolver=mytlschallenge

      # Optional security headers
      - traefik.http.middlewares.prometheus.headers.SSLRedirect=true
      - traefik.http.middlewares.prometheus.headers.STSSeconds=315360000
      - traefik.http.middlewares.prometheus.headers.browserXSSFilter=true
      - traefik.http.middlewares.prometheus.headers.contentTypeNosniff=true
      - traefik.http.middlewares.prometheus.headers.forceSTSHeader=true
      - traefik.http.middlewares.prometheus.headers.STSIncludeSubdomains=true
      - traefik.http.middlewares.prometheus.headers.STSPreload=true
      - traefik.http.routers.prometheus.middlewares=prometheus@docker

    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus

    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      # Helps if the new UI loads blank/spins on /query in some setups
      - "--enable-feature=old-ui"

    restart: unless-stopped
    networks:
      - traefik

volumes:
  prometheus-data:

networks:
  traefik:
    external: true

4) Start Prometheus

docker compose up -d

Access:

Part 2: If the UI is blank, verify Prometheus is healthy

If the UI is stuck/blank, check the server directly via API endpoints:

curl -sS http://127.0.0.1:9090/-/ready
curl -sS http://127.0.0.1:9090/api/v1/status/buildinfo | head
curl -sS 'http://127.0.0.1:9090/api/v1/query?query=up'

  • If you get “Ready” and the up query returns a result, Prometheus is healthy and it’s likely a frontend/UI issue.

  • Enabling --enable-feature=old-ui (already included above) typically fixes this scenario.

Part 3: Install Node Exporter on the host (systemd)

Node Exporter provides host-level metrics: CPU, memory, disk, network, load, etc.

1) Create a dedicated user

sudo useradd --no-create-home --shell /usr/sbin/nologin node_exporter

2) Check system architecture

uname -m

  • x86_64 → linux-amd64

  • aarch64 → linux-arm64

3) Download and install the binary (example: amd64)

cd /tmp
VER="1.8.1"

wget https://github.com/prometheus/node_exporter/releases/download/v${VER}/node_exporter-${VER}.linux-amd64.tar.gz
tar xvf node_exporter-${VER}.linux-amd64.tar.gz

sudo mv node_exporter-${VER}.linux-amd64/node_exporter /usr/local/bin/
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter
sudo chmod +x /usr/local/bin/node_exporter

(If you’re on ARM64, download node_exporter-${VER}.linux-arm64.tar.gz instead.)

4) Create a systemd service

sudo tee /etc/systemd/system/node_exporter.service > /dev/null <<'EOF'
[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter --web.listen-address=0.0.0.0:9100
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

5) Enable and start the service

sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter
sudo systemctl status node_exporter --no-pager

6) Test it locally

curl -sS http://127.0.0.1:9100/metrics | head

Part 4: Connect containerized Prometheus to host-installed Node Exporter

On Linux, host.docker.internal is not always available by default inside containers. The clean solution is to map it using Docker’s host-gateway.

1) Add extra_hosts to Prometheus in docker-compose.yml

extra_hosts:
  - "host.docker.internal:host-gateway"

Example (relevant section):

services:
  prometheus:
    ...
    extra_hosts:
      - "host.docker.internal:host-gateway"

Then recreate:

docker compose up -d --force-recreate

2) Add the Node Exporter job to prometheus.yml

  - job_name: "Mahdi Host"
    static_configs:
      - targets: ["host.docker.internal:9100"]

Final prometheus.yml:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["prometheus:9090"]

  - job_name: "Mahdi Host"
    static_configs:
      - targets: ["host.docker.internal:9100"]

Recreate Prometheus:

docker compose up -d --force-recreate

In Prometheus UI:

  • Status → Targets

  • The Mahdi Host job should be UP

Part 5: Put Node Exporter behind Traefik with HTTPS

Node Exporter does not provide TLS by itself. To expose it via HTTPS, put Traefik in front of it and let Traefik terminate TLS.

1) Ensure Traefik can reach the host

In Traefik’s own compose file (the one that runs Traefik), add:

services:
  traefik:
    extra_hosts:
      - "host.docker.internal:host-gateway"

2) Enable Traefik File Provider

This is the clean approach for services that are not Docker containers.

In Traefik’s compose:

services:
  traefik:
    volumes:
      - /opt/traefik/dynamic:/etc/traefik/dynamic
    command:
      - "--providers.file.directory=/etc/traefik/dynamic"
      - "--providers.file.watch=true"

Create the directory:

sudo mkdir -p /opt/traefik/dynamic

3) Create a dynamic config for Node Exporter (TLS + IP whitelist)

Create:

/opt/traefik/dynamic/nodeexporter.yml

http:
  routers:
    nodeexporter:
      rule: Host(`nodeexporter.mahdishadi.me`)
      entryPoints:
        - websecure
      tls:
        certResolver: mytlschallenge
      service: nodeexporter
      middlewares:
        - nodeexp-ipwhitelist

  services:
    nodeexporter:
      loadBalancer:
        servers:
          - url: "http://host.docker.internal:9100"

  middlewares:
    nodeexp-ipwhitelist:
      ipWhiteList:
        sourceRange:
          - "YOUR_PUBLIC_IP/32"

Get your public IP:

curl -s https://ifconfig.me

Replace YOUR_PUBLIC_IP/32 with the output, e.g. 1.2.3.4/32.

Recreate Traefik:

docker compose up -d --force-recreate

Access (Node Exporter’s main endpoint):

Node Exporter serves metrics on /metrics. Hitting / may not show useful output.

Part 6: Dealing with 403 Forbidden on Node Exporter HTTPS

A 403 here is usually caused by the IP whitelist:

  • Your current IP is not included in sourceRange

  • You are on VPN (your public IP changed)

Update sourceRange with your actual public IP and reload/recreate Traefik.

Quick verification commands

Node Exporter directly on the host

curl -sS http://127.0.0.1:9100/metrics | head

Node Exporter via Traefik HTTPS

curl -sS https://nodeexporter.mahdishadi.me/metrics | head

Prometheus container scraping the host

docker exec -it prometheus sh -c 'wget -qO- http://host.docker.internal:9100/metrics | head'

Security note (important)

Node Exporter has no authentication. If exposed to the internet, always protect it:

  • IP whitelist (as shown), and/or

  • Basic auth, and/or

  • keep it internal-only and let Prometheus scrape it privately

Prometheus scraping does not require HTTPS; HTTPS is mainly for safe human/browser access.

#prometheus#traefik#nginx#tls#nodeexporter

More from this blog

Mahdi Shadi

8 posts

DevOps Engineer, Linux Lover, Chess Player