# When Logs Weren’t Enough: Setting Up Prometheus Behind Traefik (TLS)

At some point, logs alone stop being enough. You start seeing short spikes in CPU, random latency jumps, or brief outages that don’t leave a clear trace in the logs. That’s where real monitoring pays off: numbers, charts, history, and alerts.

This guide walks through a complete, practical setup:

* Run **Prometheus** with **Docker Compose**
    
* Put Prometheus behind **Traefik** with **TLS**
    
* Fix a common UI issue in newer Prometheus releases
    
* Install **Node Exporter** on the host using **systemd**
    
* Connect containerized Prometheus to host-installed Node Exporter
    
* Put **Node Exporter behind Traefik** as well, so it’s reachable via **HTTPS** (secured with an IP whitelist)
    

---

## What is Prometheus?

**Prometheus** is a time-series monitoring system. Exporters and services usually expose metrics on an HTTP endpoint such as `/metrics`. Prometheus periodically **scrapes** these endpoints, stores the data, and lets you query it using **PromQL** (and later build dashboards/alerts with Grafana/Alertmanager).

---

## Prerequisites

* Docker + Docker Compose installed on your server
    
* Traefik already running and capable of issuing TLS certificates (e.g., `certresolver=mytlschallenge`)
    
* A Docker external network named `traefik`
    
* DNS records in place for:
    
    * [`prometheus.mahdishadi.me`](http://prometheus.mahdishadi.me)
        
    * [`nodeexporter.mahdishadi.me`](http://nodeexporter.mahdishadi.me)
        

---

# Part 1: Run Prometheus behind Traefik with TLS

## 1) Create a project directory

```bash
mkdir -p /opt/monitoring/prometheus
cd /opt/monitoring/prometheus
```

## 2) Create `prometheus.yml`

Start simple: scrape Prometheus itself.

```yaml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["prometheus:9090"]
```

## 3) Create `docker-compose.yml` (Prometheus behind Traefik)

```yaml
version: "3.8"

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus

    # Useful for debugging. If you want Traefik-only access, you can remove this.
    ports:
      - "9090:9090"

    labels:
      - traefik.enable=true
      - traefik.http.routers.prometheus.rule=Host(`prometheus.mahdishadi.me`)
      - traefik.http.routers.prometheus.entrypoints=web,websecure
      - traefik.http.routers.prometheus.tls=true
      - traefik.http.routers.prometheus.tls.certresolver=mytlschallenge

      # Optional security headers
      - traefik.http.middlewares.prometheus.headers.SSLRedirect=true
      - traefik.http.middlewares.prometheus.headers.STSSeconds=315360000
      - traefik.http.middlewares.prometheus.headers.browserXSSFilter=true
      - traefik.http.middlewares.prometheus.headers.contentTypeNosniff=true
      - traefik.http.middlewares.prometheus.headers.forceSTSHeader=true
      - traefik.http.middlewares.prometheus.headers.STSIncludeSubdomains=true
      - traefik.http.middlewares.prometheus.headers.STSPreload=true
      - traefik.http.routers.prometheus.middlewares=prometheus@docker

    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus

    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      # Helps if the new UI loads blank/spins on /query in some setups
      - "--enable-feature=old-ui"

    restart: unless-stopped
    networks:
      - traefik

volumes:
  prometheus-data:

networks:
  traefik:
    external: true
```

## 4) Start Prometheus

```bash
docker compose up -d
```

Access:

* [`https://prometheus.mahdishadi.me`](https://prometheus.mahdishadi.me)
    

---

# Part 2: If the UI is blank, verify Prometheus is healthy

If the UI is stuck/blank, check the server directly via API endpoints:

```bash
curl -sS http://127.0.0.1:9090/-/ready
curl -sS http://127.0.0.1:9090/api/v1/status/buildinfo | head
curl -sS 'http://127.0.0.1:9090/api/v1/query?query=up'
```

* If you get “Ready” and the `up` query returns a result, Prometheus is healthy and it’s likely a frontend/UI issue.
    
* Enabling `--enable-feature=old-ui` (already included above) typically fixes this scenario.
    

---

# Part 3: Install Node Exporter on the host (systemd)

Node Exporter provides host-level metrics: CPU, memory, disk, network, load, etc.

## 1) Create a dedicated user

```bash
sudo useradd --no-create-home --shell /usr/sbin/nologin node_exporter
```

## 2) Check system architecture

```bash
uname -m
```

* `x86_64` → `linux-amd64`
    
* `aarch64` → `linux-arm64`
    

## 3) Download and install the binary (example: amd64)

```bash
cd /tmp
VER="1.8.1"

wget https://github.com/prometheus/node_exporter/releases/download/v${VER}/node_exporter-${VER}.linux-amd64.tar.gz
tar xvf node_exporter-${VER}.linux-amd64.tar.gz

sudo mv node_exporter-${VER}.linux-amd64/node_exporter /usr/local/bin/
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter
sudo chmod +x /usr/local/bin/node_exporter
```

*(If you’re on ARM64, download* `node_exporter-${VER}.linux-arm64.tar.gz` instead.)

## 4) Create a systemd service

```bash
sudo tee /etc/systemd/system/node_exporter.service > /dev/null <<'EOF'
[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter --web.listen-address=0.0.0.0:9100
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF
```

## 5) Enable and start the service

```bash
sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter
sudo systemctl status node_exporter --no-pager
```

## 6) Test it locally

```bash
curl -sS http://127.0.0.1:9100/metrics | head
```

---

# Part 4: Connect containerized Prometheus to host-installed Node Exporter

On Linux, `host.docker.internal` is not always available by default inside containers. The clean solution is to map it using Docker’s `host-gateway`.

## 1) Add `extra_hosts` to Prometheus in `docker-compose.yml`

```yaml
extra_hosts:
  - "host.docker.internal:host-gateway"
```

Example (relevant section):

```yaml
services:
  prometheus:
    ...
    extra_hosts:
      - "host.docker.internal:host-gateway"
```

Then recreate:

```bash
docker compose up -d --force-recreate
```

## 2) Add the Node Exporter job to `prometheus.yml`

```yaml
  - job_name: "Mahdi Host"
    static_configs:
      - targets: ["host.docker.internal:9100"]
```

Final `prometheus.yml`:

```yaml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["prometheus:9090"]

  - job_name: "Mahdi Host"
    static_configs:
      - targets: ["host.docker.internal:9100"]
```

Recreate Prometheus:

```bash
docker compose up -d --force-recreate
```

In Prometheus UI:

* **Status → Targets**
    
* The `Mahdi Host` job should be **UP**
    

---

# Part 5: Put Node Exporter behind Traefik with HTTPS

Node Exporter does not provide TLS by itself. To expose it via HTTPS, put Traefik in front of it and let Traefik terminate TLS.

## 1) Ensure Traefik can reach the host

In Traefik’s own compose file (the one that runs Traefik), add:

```yaml
services:
  traefik:
    extra_hosts:
      - "host.docker.internal:host-gateway"
```

## 2) Enable Traefik File Provider

This is the clean approach for services that are not Docker containers.

In Traefik’s compose:

```yaml
services:
  traefik:
    volumes:
      - /opt/traefik/dynamic:/etc/traefik/dynamic
    command:
      - "--providers.file.directory=/etc/traefik/dynamic"
      - "--providers.file.watch=true"
```

Create the directory:

```bash
sudo mkdir -p /opt/traefik/dynamic
```

## 3) Create a dynamic config for Node Exporter (TLS + IP whitelist)

Create:

`/opt/traefik/dynamic/nodeexporter.yml`

```yaml
http:
  routers:
    nodeexporter:
      rule: Host(`nodeexporter.mahdishadi.me`)
      entryPoints:
        - websecure
      tls:
        certResolver: mytlschallenge
      service: nodeexporter
      middlewares:
        - nodeexp-ipwhitelist

  services:
    nodeexporter:
      loadBalancer:
        servers:
          - url: "http://host.docker.internal:9100"

  middlewares:
    nodeexp-ipwhitelist:
      ipWhiteList:
        sourceRange:
          - "YOUR_PUBLIC_IP/32"
```

Get your public IP:

```bash
curl -s https://ifconfig.me
```

Replace `YOUR_PUBLIC_IP/32` with the output, e.g. `1.2.3.4/32`.

Recreate Traefik:

```bash
docker compose up -d --force-recreate
```

Access (Node Exporter’s main endpoint):

* [`https://nodeexporter.mahdishadi.me/metrics`](https://nodeexporter.mahdishadi.me/metrics)
    

> Node Exporter serves metrics on `/metrics`. Hitting `/` may not show useful output.

---

# Part 6: Dealing with `403 Forbidden` on Node Exporter HTTPS

A 403 here is usually caused by the IP whitelist:

* Your current IP is not included in `sourceRange`
    
* You are on VPN (your public IP changed)
    

Update `sourceRange` with your actual public IP and reload/recreate Traefik.

---

## Quick verification commands

### Node Exporter directly on the host

```bash
curl -sS http://127.0.0.1:9100/metrics | head
```

### Node Exporter via Traefik HTTPS

```bash
curl -sS https://nodeexporter.mahdishadi.me/metrics | head
```

### Prometheus container scraping the host

```bash
docker exec -it prometheus sh -c 'wget -qO- http://host.docker.internal:9100/metrics | head'
```

---

## Security note (important)

Node Exporter has no authentication. If exposed to the internet, always protect it:

* IP whitelist (as shown), and/or
    
* Basic auth, and/or
    
* keep it internal-only and let Prometheus scrape it privately
    

Prometheus scraping does not require HTTPS; HTTPS is mainly for safe human/browser access.