Blame
|
1 | # Infrastructure |
||||||
| 2 | ||||||||
| 3 | ← [[Home]] |
|||||||
| 4 | ||||||||
|
5 | Proxmox host running Debian 12 Bookworm with LXC containers, a QEMU VM, and two ZFS pools. |
||||||
|
6 | |||||||
| 7 | --- |
|||||||
| 8 | ||||||||
| 9 | ## Storage Pools |
|||||||
| 10 | ||||||||
| 11 | | Name | Type | Total | Used | Notes | |
|||||||
| 12 | |------|------|-------|------|-------| |
|||||||
| 13 | | `tank` | ZFS | 3.77 TB | ~23% | Primary data — host path `/tank/`, mounted as `/mnt/tank/` inside containers | |
|||||||
| 14 | | `local-ssd` | dir | 239 GB | ~64% | SSD (PCT 102 rootfs) | |
|||||||
| 15 | | `local-lvm` | lvmthin | 354 GB | ~17% | LVM thin (PCT 108 rootfs) | |
|||||||
| 16 | | `local` | dir | 98 GB | ~34% | Proxmox local storage | |
|||||||
| 17 | ||||||||
|
18 | ### ZFS Notes |
||||||
| 19 | ||||||||
| 20 | - `tank/appdata` has `acltype=posixacl` enabled (2026-05-25). Use `setfacl` when a non-owner user needs access to files owned by a different UID: |
|||||||
| 21 | ```bash |
|||||||
| 22 | setfacl -R -m u:100000:rx /tank/appdata/nextcloud/data/data/nextcloud/files/obsidian |
|||||||
| 23 | setfacl -R -m u:1000:rx /tank/appdata/nextcloud/data/data/nextcloud/files/obsidian |
|||||||
| 24 | ``` |
|||||||
|
25 | |||||||
| 26 | --- |
|||||||
| 27 | ||||||||
| 28 | ## LXC Mount Points |
|||||||
| 29 | ||||||||
|
30 | All containers: `timezone: America/Toronto`, `onboot: 1`. |
||||||
| 31 | Default PUID/PGID: `1000:1000`. Exceptions: nextcloud/firefly `33:33`, guacamole/immich `0:0`. |
|||||||
| 32 | ||||||||
|
33 | ### PCT 101 (downloads — 192.168.2.190) |
||||||
| 34 | ||||||||
| 35 | | Mount | Host | Container | |
|||||||
| 36 | |-------|------|-----------| |
|||||||
| 37 | | mp2 | `/tank/appdata` | `/mnt/tank/appdata` | |
|||||||
| 38 | ||||||||
| 39 | ### PCT 102 (media-core — 192.168.2.191) |
|||||||
| 40 | ||||||||
|
41 | Privileged container (`unprivileged: 0`). Full ZFS tank pool mounted. |
||||||
|
42 | |||||||
| 43 | | Mount | Host | Container | |
|||||||
| 44 | |-------|------|-----------| |
|||||||
| 45 | | mp0 | `/tank/` | `/mnt/tank/` | |
|||||||
| 46 | ||||||||
|
47 | **Media library root:** `/mnt/tank/media/content/` — movies, tv, music, books, audiobooks, podcasts, comics, openbooks, roms, photos |
||||||
| 48 | ||||||||
| 49 | Any Docker container without the `media-core_` Swarm prefix is standalone. Standalone containers using `mode: host` ports can conflict with Swarm services — always check `pct exec 102 -- docker ps` before troubleshooting. |
|||||||
|
50 | |||||||
| 51 | ### PCT 104 (documents — 192.168.2.105) |
|||||||
| 52 | ||||||||
| 53 | Privileged container. Only `/tank/media` and `/tank/appdata/nextcloud` are mounted — not the full tank. |
|||||||
| 54 | ||||||||
| 55 | | Mount | Host | Container | Notes | |
|||||||
| 56 | |-------|------|-----------|-------| |
|||||||
| 57 | | mp0 | `/tank/media` | `/mnt/tank/media` | Media library | |
|||||||
| 58 | | mp1 | `/mnt/media-storage` | `/mnt/media-storage` | Separate media storage | |
|||||||
|
59 | | mp2 | `/tank/appdata/nextcloud` | `/mnt/tank/appdata/nextcloud` | Nextcloud data on ZFS (migrated 2026-05-25) | |
||||||
|
60 | | lxc.mount.entry | `/tank/docker` | `tank/docker` | Docker storage on ZFS, not rootfs | |
||||||
| 61 | ||||||||
| 62 | ### PCT 107 (debian — 192.168.2.81) |
|||||||
| 63 | ||||||||
|
64 | | Mount | Host | Container | Notes | |
||||||
| 65 | |-------|------|-----------|-------| |
|||||||
| 66 | | mp1 | `/tank/appdata/nextcloud/data/data/nextcloud/files/obsidian` | `/mnt/obsidian` | Obsidian vault (ro) | |
|||||||
| 67 | ||||||||
| 68 | **Standalone containers on PCT 107:** |
|||||||
| 69 | ||||||||
| 70 | | Container | Notes | |
|||||||
| 71 | |-----------|-------| |
|||||||
| 72 | | `archibus-scheduler-archibus-scheduler-1` | Custom scheduler, no published ports | |
|||||||
|
73 | |||||||
| 74 | ### PCT 108 (network — 192.168.2.82) |
|||||||
| 75 | ||||||||
|
76 | **Docker Swarm manager/leader.** Appdata on rootfs (local-lvm, 50 GB, ~55% used) — NOT on ZFS. Portainer compose files at `/mnt/tank/appdata/portainer/compose/` are on ZFS (mp0 mount). |
||||||
|
77 | |||||||
| 78 | | Mount | Host | Container | |
|||||||
| 79 | |-------|------|-----------| |
|||||||
| 80 | | mp0 | `/tank/` | `/mnt/tank/` | |
|||||||
| 81 | ||||||||
|
82 | - Portainer stacks: `/mnt/tank/appdata/portainer/compose/<id>/docker-compose.yml` |
||||||
| 83 | - Portainer data: `network_portainer_data` Docker volume (on rootfs) |
|||||||
| 84 | ||||||||
|
85 | ### PCT 109 (ai — 192.168.2.83) |
||||||
| 86 | ||||||||
|
87 | Privileged container. **Not a Swarm node.** `lxc.apparmor.profile: unconfined` (required so `docker build` can run `apparmor_parser`). `lxc.cgroup2.devices.allow: a`. |
||||||
|
88 | |||||||
| 89 | | Mount | Host | Container | Notes | |
|||||||
| 90 | |-------|------|-----------|-------| |
|||||||
|
91 | | mp0 | `/tank/appdata/nextcloud/data/data/nextcloud/files/obsidian` | `/mnt/obsidian` | Obsidian vault (read-write) for Hermes + n8n | |
||||||
| 92 | ||||||||
| 93 | Node.js 22 installed (via NodeSource) — required for `mcp-server-filesystem`. |
|||||||
| 94 | ||||||||
| 95 | ### PCT 300 (pterodactyl-panel — 192.168.2.136) |
|||||||
| 96 | ||||||||
| 97 | Pterodactyl Panel. Not a Swarm node. Services: `apache2`, `mariadb`, `redis-server`, `pteroq.service` (queue worker). |
|||||||
| 98 | ||||||||
| 99 | **Always use explicit env** to avoid inheriting `TMPDIR=/tmp/claude-0` from Proxmox host shell (breaks MariaDB): |
|||||||
| 100 | ```bash |
|||||||
| 101 | pct exec 300 -- env -i HOME=/root PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TMPDIR=/tmp LANG=en_US.UTF-8 bash -c "..." |
|||||||
| 102 | ``` |
|||||||
| 103 | ||||||||
| 104 | ### PCT 301 (pterodactyl-wings — 192.168.2.134) |
|||||||
| 105 | ||||||||
| 106 | Pterodactyl Wings daemon. Not a Swarm node. Game server data at `/var/lib/pterodactyl/volumes/<uuid>/`. |
|||||||
| 107 | ||||||||
| 108 | Local Docker registry at `192.168.2.134:5000` (container named `registry`, auto-restart). |
|||||||
|
109 | |||||||
| 110 | --- |
|||||||
| 111 | ||||||||
| 112 | ## GPU Passthrough |
|||||||
| 113 | ||||||||
|
114 | NVIDIA GPU passthrough configured for PCT 102, 104, and 109 via `lxc.cgroup2.devices.allow` and `lxc.mount.entry` in `/etc/pve/lxc/<vmid>.conf`. Key NVIDIA libs bind-mounted from the Proxmox host. |
||||||
|
115 | |||||||
|
116 | | Container | GPU | Use | |
||||||
| 117 | |-----------|-----|-----| |
|||||||
| 118 | | PCT 102 | NVIDIA (media-core) | Jellyfin transcoding, Immich ML | |
|||||||
| 119 | | PCT 104 | GTX 1050 | Immich machine-learning (CUDA) | |
|||||||
| 120 | | PCT 109 | NVIDIA | Whisper transcription | |
|||||||
|
121 | |||||||
|
122 | `nvidia-container-toolkit` installed in each. Default Docker runtime set to `nvidia` in `/etc/docker/daemon.json`. Runtime mode: **legacy** (not CDI) — CDI auto-detection fails in LXC because NVML can't init. |
||||||
|
123 | |||||||
|
124 | **To enable GPU in a container:** |
||||||
|
125 | ```yaml |
||||||
| 126 | environment: |
|||||||
| 127 | NVIDIA_VISIBLE_DEVICES: all |
|||||||
| 128 | NVIDIA_DRIVER_CAPABILITIES: all |
|||||||
| 129 | ``` |
|||||||
|
130 | No `runtime:` key needed — nvidia is already the default. |
||||||
| 131 | ||||||||
| 132 | **After a Proxmox host driver upgrade:** update the versioned lib filenames in `/etc/pve/lxc/<vmid>.conf` to match the new driver version, `chmod 755` the new versioned files, and reboot the affected container. |
|||||||
| 133 | ||||||||
| 134 | --- |
|||||||
| 135 | ||||||||
| 136 | ## Home Assistant VM |
|||||||
| 137 | ||||||||
| 138 | QEMU VM 200 (`haos16.3`), 2 cores, 4 GB RAM, 32 GB disk (local-lvm), OVMF/q35. IP: `192.168.2.129`. See [[Home Assistant]] for full details. |
|||||||
|
139 | |||||||
|
140 | **USB passthrough (in VM config):** |
||||||
| 141 | - `0a12:0001` — Bluetooth |
|||||||
| 142 | - `1a86:7523` — Zigbee stick |
|||||||
| 143 | - `303a:1001` — ESP32-C6 (added 2026-06-13) |
|||||||
|
144 | |||||||
| 145 | --- |
|||||||
| 146 | ||||||||
| 147 | ## Management Scripts (Proxmox host `/root/`) |
|||||||
| 148 | ||||||||
| 149 | | Script | Purpose | |
|||||||
| 150 | |--------|---------| |
|||||||
| 151 | | `create-app-folders.sh` | Create `/tank/appdata/<app>` dirs with `1000:1000` | |
|||||||
| 152 | | `setup-lxc-bind-mounts.sh` | Add ZFS bind mounts to LXC containers (`DRY_RUN=false` to apply) | |
|||||||
| 153 | | `create_lxc.sh <CTID> <HOST> <IP>` | Provision new Ubuntu 22.04 LXC | |
|||||||
| 154 | | `migrate-appdata-to-zfs.sh` | Migrate appdata from rootfs to ZFS dataset | |
|||||||
