My homelab has grown to a good dozen Linux hosts: Proxmox nodes, Docker VMs, a few VPSs. For a while every host had something different collecting logs and metrics: an old OpenTelemetry Collector here, a Docker Loki logging driver there, nothing at all somewhere else. Now every host runs Grafana Alloy with almost the same config:
- Logs: journald → Loki
- Host metrics: built-in node exporter → Prometheus
- Container metrics (Docker hosts only): built-in cAdvisor → Prometheus
Everything is pushed. Prometheus doesn’t need a scrape job per host and the hosts don’t need any open ports. And no, you don’t need Mimir for that: Prometheus can receive remote writes itself.
The setup
- Loki at
192.168.1.64:3100 - Prometheus at
192.168.1.65:9090(installed in an LXC via the Proxmox helper script, so a plain systemd service) - Grafana on top of both
- Alloy installed natively (apt) on each Debian/Ubuntu host
1. Let Prometheus accept remote writes
This is off by default. Prometheus only accepts pushes on /api/v1/write if it’s started with this flag:
|
1 |
--web.enable-remote-write-receiver |
Add it to the Prometheus systemd unit (or the compose command:) and restart. Check that it’s active:
|
1 |
ps aux | grep prometheus | grep -o 'enable-remote-write-receiver' |
No output means the flag isn’t set, and Alloy will log write errors for every batch.
2. Install Alloy
|
1 2 3 4 5 6 |
sudo apt-get install -y gpg wget sudo mkdir -p /etc/apt/keyrings/ wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list sudo apt-get update sudo apt-get install -y alloy |
The package runs Alloy as the unprivileged user alloy, with its config in /etc/alloy/config.alloy.
3. Give the alloy user the right groups
|
1 2 3 |
sudo usermod -aG systemd-journal alloy # read the journal sudo usermod -aG docker alloy # only on Docker hosts sudo systemctl restart alloy |
Without systemd-journal, Alloy starts without complaint but ships almost no logs. It can only see its own entries.
4. The config
This is the base config every host gets. /etc/alloy/config.alloy:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 |
// ===== Grafana Alloy – base config for every Linux host ===== // ---------- METRICS -> Prometheus remote-write receiver ---------- // Alloy's own health prometheus.exporter.self "alloy_self" {} prometheus.scrape "alloy_self" { targets = prometheus.exporter.self.alloy_self.targets forward_to = [prometheus.remote_write.prom.receiver] scrape_interval = "30s" } // Host metrics (built-in node exporter) prometheus.exporter.unix "node" { filesystem { fs_types_exclude = "^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$" mount_points_exclude = "^/(dev|proc|sys|run)($|/)" } } prometheus.scrape "node" { targets = prometheus.exporter.unix.node.targets forward_to = [prometheus.remote_write.prom.receiver] scrape_interval = "15s" job_name = "node" } prometheus.remote_write "prom" { endpoint { url = "http://192.168.1.65:9090/api/v1/write" queue_config { capacity = 10000 max_shards = 4 max_samples_per_send = 2000 } } external_labels = { host = constants.hostname, env = "prod", } } // ---------- LOGS -> Loki ---------- loki.source.journal "journal" { max_age = "12h" relabel_rules = loki.relabel.journal.rules forward_to = [loki.write.loki.receiver] } // Turn journal fields into useful labels loki.relabel "journal" { forward_to = [] rule { source_labels = ["__journal__systemd_unit"] target_label = "unit" } rule { source_labels = ["__journal_priority_keyword"] target_label = "level" } } loki.write "loki" { endpoint { url = "http://192.168.1.64:3100/loki/api/v1/push" } external_labels = { host = constants.hostname, env = "prod", } } |
A few choices that turned out to matter:
- The host label is set in one place only, in
external_labelson both writers. Every metric and log line gets it automatically, so the same Grafana variable$hostfilters both. - journald is the only log source. On Debian/Ubuntu almost everything ends up in the journal anyway. When I also scraped
/var/log/syslog,auth.logandkern.log, every line arrived twice. - Same labels on every host (
host,env, plusjobper scrape). A host in testing getsenv = "testing"and can be filtered out of alerts.
Restart and watch the log:
|
1 2 |
sudo systemctl restart alloy journalctl -u alloy -n 50 --no-pager |
5. Docker hosts: container metrics via cAdvisor
Alloy has cAdvisor built in, so you don’t need an extra container. Add to the config:
|
1 2 3 4 5 6 7 8 9 10 |
// ---------- Container metrics (cAdvisor) -> Prometheus ---------- prometheus.exporter.cadvisor "containers" { docker_host = "unix:///var/run/docker.sock" } prometheus.scrape "cadvisor" { targets = prometheus.exporter.cadvisor.containers.targets forward_to = [prometheus.remote_write.prom.receiver] scrape_interval = "30s" job_name = "cadvisor" } |
The catch: the docker group gets Alloy access to the Docker socket, but not to the root-owned paths under /var/lib/docker that cAdvisor reads. The log fills with permission denied and “Failed to create existing container” errors, and you get no container metrics.
The fix is to run Alloy as root on Docker hosts. I write the systemd drop-in directly; systemctl edit silently didn’t save for me once:
|
1 2 3 4 5 6 7 8 |
sudo mkdir -p /etc/systemd/system/alloy.service.d sudo tee /etc/systemd/system/alloy.service.d/override.conf >/dev/null <<'EOF' [Service] User=root Group=root EOF sudo systemctl daemon-reload sudo systemctl restart alloy |
Check that it worked:
|
1 2 |
ps -o user= -p $(pgrep -x alloy) # should print root sudo journalctl -u alloy --since "1 minute ago" | grep -iE 'permission denied|Failed to create existing container' |
Empty output from the second command means cAdvisor is happy.
Running Alloy as root is a trade-off. Alloy already reads the whole journal and the Docker socket, so it’s privileged in practice anyway, but it’s still more access. If that bothers you, skip cAdvisor on that host, or run cAdvisor as its own privileged container on 127.0.0.1:8080 and let an unprivileged Alloy scrape it. My rule: cAdvisor = Alloy as root, every other host stays unprivileged.
6. Verify end to end
Metrics in Prometheus (Grafana Explore or curl):
|
1 |
curl -sG 'http://192.168.1.65:9090/api/v1/query' --data-urlencode 'query=up{host="myhost"}' |
You want one series per scrape job, each with value 1. Logs in Loki (Grafana Explore):
|
1 2 3 |
{host="myhost"} {host="myhost", unit="ssh.service"} {host="myhost", level=~"err|crit"} |
Dashboards that work with these labels out of the box: 1860 (Node Exporter Full) for hosts, 14282 or 19792 for cAdvisor. Set the dashboard’s host variable to the host label.
Gotchas
- Comments are
//, not#. Alloy’s config language is not YAML or shell. One#comment and the whole config fails to parse, and the service stays in “failed”. If Alloy won’t start after an edit, check this first:journalctl -u alloy -n 20. env("HOSTNAME")is empty under systemd. The variable exists in your shell, not in the service’s environment. Your logs then arrive withhost="". Useconstants.hostname.- Missing
systemd-journalgroup: Alloy runs fine but ships almost nothing. - Remote-write receiver not enabled in Prometheus: metrics never arrive, and the only hint is write errors in Alloy’s log.
- cAdvisor permission errors despite the docker group: see section 5.
- Duplicate logs when you add file scrapes on top of journald. Pick one source.
- Names are just names. Some of my hosts call the remote-write component
"mimir"from an earlier plan, even though it points at Prometheus. That works fine, butforward_tohas to use the name the component actually has (prometheus.remote_write.mimir.receivervs..prom.receiver). Copying a block between hosts with different names is a classic. - Leftovers from earlier experiments: on one host I found an OpenTelemetry Collector from an old test still running and sending data into the void.
systemctl list-units --type=service | grep -iE 'otel|promtail|agent'before you start.
Result
Every host now has the same small config, set up in five minutes. Logs and metrics share the same host label, so in Grafana I can jump from a CPU spike straight to the logs of the same host and minute. A new host is: apt install, groups, copy the config, restart. Docker hosts additionally get the cAdvisor block and the root override.