Chapter III · Self-hosted infrastructure

Service Isolation: One Service Per Container

Tested on
Proxmox VE 9.2 on a single Intel Ultra 9 node, 32GB RAM, unprivileged LXC plus two QEMU VMs. 15 service containers running. The stopped-on-demand build container tier described in earlier versions of this guide is retired (see below).
Last updated
2026-10-17

I run my homelab as a pile of small, boring, unprivileged LXC containers. One service each. Not one fat VM with a docker-compose monolith, not a single Debian box hand-fed twelve daemons. This guide is the discipline behind that choice: why I do it, where the blast radius gets drawn, and the handful of cases where a full VM is actually the right call.

This is the why, not the map. For the inventory of what runs where, see homelab-topology.md.

The Rule: One Service, One Container

Every long-lived service gets its own unprivileged LXC. DNS in one, the SIEM in another, the photo server in a third, the automation stack in a fourth. They share a kernel (that’s what makes LXC cheap) but nothing else: separate rootfs, separate network interface, separate resource caps, separate snapshot timeline.

The instinct early on is to consolidate. “Why spin up five containers when one Debian box could run all five daemons?” Because consolidation trades a one-time setup cost for a permanent operational tax, and the tax compounds every time something breaks.

What one-service-per-container actually buys you

That last point is the real reason. The threat model isn’t just “software crashes.” It’s “I am tired, it is late, and I am about to paste a command I half-understand into a root shell.” Isolation means that mistake costs me one service, not the whole lab. 🦞

Blast-Radius Thinking

Draw the boundary at the unit you’d want to restore independently. For me that’s “one service.” Ask three questions per container:

  1. If this is compromised, what else can it reach? A container with a NAS mount and SSH keys is a bigger prize than a stateless DNS resolver. Keep the credential-heavy services small and few, and don’t co-locate a public-facing service with your secrets. This is the container-level version of the agent-side rules in ../security/agent-security-hardening.md.
  2. If this dies, what dies with it? The answer should be “only this service.” If killing container A also kills container B, you’ve built a hidden monolith with extra steps. The classic trap here is shared bind-mounts (see the next section).
  3. If I have to roll this back, what state do I lose? Snapshot scope equals blast radius. A rollback reverts everything in that container’s rootfs to the snapshot point. Smaller containers mean rollbacks lose less.

The bind-mount trap

The fastest way to accidentally couple two “isolated” containers is a shared dependency that lives outside both of them. I learned this the expensive, silent way.

My photo server (its own LXC) bind-mounts a directory from a NAS share into the container so uploads land on bulk storage. One morning the NAS CIFS mount on the Proxmox host failed to come up after a reboot race. The host mount point existed but was empty, so the container’s bind-mount pointed at an empty directory. The photo server’s app container couldn’t create its upload path and exited. Postgres, the ML worker, and Redis in the same stack all stayed healthy and reported green. The phone app just silently stopped backing up. Nothing crashed loudly. The dependency that took the service down lived between the host and the container, in a place neither one’s health check was watching.

Two lessons baked in:

The fix was two lines (remount the NAS share, restart the one app container). The diagnosis took an hour because everything looked fine. Isolation didn’t fail here. It worked: only one service went down. But a hidden shared dependency is the seam where isolation leaks, so go find your bind-mounts and treat each one as a coupling you signed up for on purpose.

Per-Container Resource Caps

Isolation without caps is a polite fiction. If every container can burst to all 32GB of host RAM, one of them will, and you’re back to shared-fate. Set caps at create time and treat them as the contract.

# A small, stateless service: DNS resolver. Tiny on purpose.
pct create 100 local:vztmpl/debian-12-standard_amd64.tar.zst \
  --hostname dns \
  --cores 1 --memory 512 --swap 512 \
  --rootfs local-lvm:4 \
  --net0 name=eth0,bridge=vmbr0,ip=dhcp \
  --unprivileged 1 --onboot 1 --start 1
# A heavy service: SIEM. Gets real resources, still capped.
pct create 105 local:vztmpl/debian-12-standard_amd64.tar.zst \
  --hostname siem \
  --cores 4 --memory 8192 --swap 512 \
  --rootfs local-lvm:50 \
  --features nesting=1,keyctl=1 \
  --net0 name=eth0,bridge=vmbr0,ip=dhcp \
  --unprivileged 1 --onboot 1 --start 1

Adjust caps live when a service genuinely needs more, but do it deliberately:

pct set 105 --memory 8192     # bump RAM
pct set 105 --cores 4         # bump CPU
pct resize 105 rootfs +10G    # grow disk (can grow, cannot shrink)

Sizing principles I actually follow:

The Ephemeral Build-Container Pattern (Retired)

Earlier versions of this lab ran a tier of PR-build and smoke-test containers. Each was stopped by default and started only for the duration of a job, with a pristine snapshot rolled back before every run. That whole tier was destroyed in a September 2026 cleanup, and none of it exists today. I’m keeping the pattern here as history and as a recipe, because the isolation reasoning still holds and you may want it on a bigger box.

What it looked like: one stopped container per project that needed clean-room builds, plus a smoke-test container for MCP integration tests and a QEMU smoke template. The workflow was reset, start, build, capture, stop:

# Spin up a clean build CT for a PR, run the build, tear it back down.
ssh hypervisor "pct rollback <ctid> pristine && pct start <ctid>"
ssh hypervisor "pct exec <ctid> -- bash -lc '
  cd /root/project && git fetch origin && git checkout pr-branch &&
  pnpm install && pnpm build && pnpm test
'"
ssh hypervisor "pct stop <ctid>"

Why it was worth doing while it lasted:

What it cost, and why I don’t run it now:

If you do keep a tier like this, schedule the start, update, re-snapshot, stop cycle so the snapshots stay current, and keep the build NIC firewalled. The one container from that era that survived is the CI runner, which now runs as an ordinary always-on service container.

The ephemeral-state gotcha

Ephemeral containers have a sharp edge: state you want to keep can live in the part of the container that gets thrown away. I hit this with a Docker-based automation service whose SSH known_hosts file lived in the container’s writable layer rather than in a mounted data volume. Every container recreate wiped known_hosts, and SSH-using workflows started failing silently with host-key verification errors because the new layer didn’t trust any hosts yet.

The rule that falls out: if it’s ephemeral, treat anything outside an explicit persistent volume as gone on the next reset. For disposable build containers that was a feature. For services that recreate their app containers (Docker-in-LXC), it’s a trap, and it still applies to the automation and photo stacks today. Audit where each service keeps its must-survive state, and make sure it’s in a named volume or a bind-mount, not the disposable layer.

Snapshot Before Change

The cheap habit that makes everything above safe: snapshot the container before you change it.

pct snapshot 105 pre-maint-$(date +%Y%m%d-%H%M)
# ... do the risky thing: apt full-upgrade, config edit, version bump ...
# if it went sideways:
pct rollback 105 pre-maint-20261003-1430

Before each host-wide maintenance window I snapshot every service container that supports it, upgrade, and verify. The October 2026 windows used pre-fleet-update-20261003 and pre-vendor-update-20261003, one name per batch of containers. The snapshot timeline is per-container, so a botched upgrade on the monitoring box rolls back in seconds without touching anything else. That’s isolation paying off again: a per-service rollback is only possible because services aren’t sharing a rootfs.

Two caveats from the field:

When a Full VM Is Actually Justified

I default to LXC, but the discipline includes knowing when LXC is the wrong tool. Reach for a QEMU VM when:

What does not justify a VM: “it’s a big service,” “it has a database,” “it runs Docker.” LXC handles all of those fine. Heavy and isolated are different axes. My SIEM is 8GB and 4 cores in an LXC and it’s still isolated from everything else.

Verification

Run these on the Proxmox node to confirm the isolation posture matches what this guide promises:

# One service per container: the list should read like a service inventory
pct list
qm list

# Every service container is unprivileged and capped
pct config <CTID> | grep -E 'memory|cores|unprivileged'
# expected: memory: <cap>, cores: <cap>, unprivileged: 1

# The snapshot habit is real: a pre-change snapshot exists from the last window
pct listsnapshot <CTID>
# expected: at least one dated pre-update entry above "current"

If pct config shows no unprivileged: 1 on a service container, or pct listsnapshot comes back empty on a box you upgraded last week, the discipline has drifted from the design.

Gotchas

  1. CTID and VMID share one numbering pool. On Proxmox, pct list shows only LXC, qm list shows only VMs, and they draw from the same ID space. ID 110 can be a VM while 109 and 111 are containers. Always check both before assuming an ID is free or “doesn’t exist.”

  2. A bind-mount is a coupling, not a convenience. Every shared mount you bind into a container extends that container’s blast radius to include host-side state. Inventory them. The silent failures (empty mount after a boot race) are the ones that cost you an afternoon.

  3. Stopped containers still need upkeep, and they still cost disk. This is the lesson of the retired build tier. A pristine snapshot from six months ago builds against six-month-old toolchains, and the idle volume holds thin-pool space the whole time. Refresh on a schedule or don’t keep the tier.

  4. unprivileged 1 can’t always be toggled after create. Flipping a container between privileged and unprivileged isn’t a simple pct set. UID mapping changes how files on the rootfs are owned. Decide at create time. For services, the answer is always unprivileged.

  5. Resource caps are a contract you have to enforce. Setting --memory 512 caps the container, but a service that genuinely needs more will OOM-loop quietly inside its cap. Cap deliberately, then actually watch for the service hitting the ceiling instead of assuming the number you picked at 2am was right.

  6. Snapshot before change is a habit, not a feature you enable. Proxmox won’t snapshot for you. The discipline is yours: pct snapshot before every upgrade, config edit, or risky command. The five seconds it costs is the cheapest insurance in the lab.