Mac Studio — QwestlyCI External Drive & Self-Hosted CI Runner

💾 QwestlyCI external drive (Samsung PM951 NVMe, 256GB, USB)

External NVMe drive plugged into the Mac Studio, formatted GPT / APFS, volume name QwestlyCI, mounted at /Volumes/QwestlyCI (238GB usable). Reformatted from its original Windows layout on 2026-09-11.

What it does: hosts a Lima Ubuntu VM (qwestly-ci) running GitHub Actions organization self-hosted runners for the Qwestly org. Live and carrying real CI since 2026-09-11.

Started as candidate + qwestly-agent only; now eight repos (qwestly-workspace, api-python, candidate, public-site, qwestly-agent, qwestly-contracts, qwestly-hire, qwestly-internal) run tests, security scans, release automation, and Claude Code Review here. Vercel deployments stay on Vercel.

Reference doc: qwestly-workspace/docs/self-hosted-actions-runner.md (architecture, operator runbook, security model, gotchas). The original plan is archived at qwestly-workspace/plan/archive/2026/organization-self-hosted-actions-runner.md — keep it for the incident-by-incident detail, but the doc is what's current.

Why a VM instead of a container on the host: the runner's test jobs need Docker (a MongoDB service container), and Docker socket access is root-equivalent on whatever it's exposed to. A real VM keeps that root-equivalent access scoped to the guest kernel instead of this Mac. Why an external drive instead of the internal SSD: the internal drive only had ~56GB free; the VM disk (60GB) and its growth (Docker images, npm/pip caches) live on the external volume instead of eating into that.

Setup

# One-time: point Lima's data dir at the external volume
export LIMA_HOME=/Volumes/QwestlyCI/lima-home

Add that export to your shell profile before running any limactl command against this instance — otherwise Lima defaults to ~/.lima on the internal disk.

• VM config: /tmp/qwestly-ci.yaml (Ubuntu 26.04 LTS base + Docker rootful provisioning, home-directory mount explicitly disabled for isolation)
• Sizing: 4 vCPU / 8GB RAM / 60GB disk
• Disk image location: /Volumes/QwestlyCI/lima-home/qwestly-ci/disk
• Start/stop: limactl start qwestly-ci / limactl stop qwestly-ci
• Shell in: limactl shell qwestly-ci

Note: the one-time Ubuntu cloud image download cache (~/Library/Caches/lima, ~3.2GB) lands on the internal disk regardless of LIMA_HOME, but isn't needed once the VM exists — safe to delete after setup.

Runner instances

Six runner processes on the one VM, each its own systemd service and work directory under /home/github-actions/actions-runner[-N]. They split by label, which is the whole design:

Instance	Label	Takes
qwestly-ci-mac-studio	qwestly-ci	general work (tests, scans, releases)
qwestly-ci-mac-studio-2	qwestly-ci	general work
qwestly-ci-mac-studio-3	qwestly-ci-llm	Claude Code Review only
qwestly-ci-mac-studio-4	qwestly-ci-llm	Claude Code Review only
qwestly-ci-mac-studio-5	qwestly-ci-llm	Claude Code Review only
qwestly-ci-mac-studio-6	qwestly-ci-llm	Claude Code Review only

Only two take general work because 4 vCPUs are saturated by a single candidate test job — measured at 116s alone vs 150s (+29%) overlapping another, and that contention has already flaked a timing-sensitive test. More general capacity would make CI slower and flakier, not faster.

The four qwestly-ci-llm instances are free capacity by comparison: Claude Code Review spends a median 99% of its time waiting on the model API, so it costs RAM (~120–155MB idle each — corrected 2026-09-22, the ~75MB figure from the original 3/4 rollout was stale) and essentially no CPU. Reviews stopped queueing behind the test suite, then stopped queueing behind each other with 3/4; that queueing recurred as review volume grew (a candidate/qwestly-agent burst on 2026-09-22 delayed two candidate reviews 84s/118s behind instances 3/4), which is why 5 and 6 were added. Verified with a deliberate 4-repo burst landing one job on each of 3, 4, 5, 6 simultaneously, fully overlapping, no queueing. A seventh is fine if review demand grows further — bounded by RAM, not by the CPU the tests need.

# status / logs / restart (inside the VM, any instance)
sudo systemctl status actions.runner.Qwestly.qwestly-ci-mac-studio[-2|-3|-4|-5|-6].service
sudo tail -f /var/log/qwestly-ci/runner-jobs.log   # every job, START/END pairs

Status

Running, recovers on its own. A systemctl restart and a full VM reboot both came back with no manual intervention.

Current usage: VM disk 18G of 58G (31%), external volume 13G of 238G (6%), roughly ~120–155MB RAM per idle instance across all six (~0.8–0.9GB idle total — measured 2026-09-22 after adding instances 5/6, no regression from the two extra). Docker holds ~1.2GB images and ~5.7GB of local volumes, nearly all reclaimable — worth a periodic look, but nowhere near pressure.

The thing to remember: this ties Qwestly's CI to this machine being powered on with the external drive attached. That's an accepted interim tradeoff, not the end state. If the Mac is off or the drive is unmounted, affected jobs queue until it's back; the per-job rollback is a one-line runs-on swap to ubuntu-latest, which each workflow keeps commented beside the active line.

Auto-start on boot/login: ~/Library/LaunchAgents/com.qwestly.ci-vm-start.plist (added 2026-09-14, after a stuck PR #65 turned out to be the VM sitting Stopped with nothing bringing it back up). RunAtLoad, runs ~/Library/Scripts/com.qwestly.ci-vm-start.sh as user dominick, which waits up to 60s for /Volumes/QwestlyCI to mount then calls limactl start qwestly-ci (idempotent — no-ops cleanly if already running). Logs to ~/Library/Logs/qwestly-ci-vm-start.log. This only fires at login, not raw kernel boot, so it depends on this account auto-logging in — if that ever changes, this silently stops covering reboots again. The four runner systemd services inside the guest were already enabled at VM boot; what was missing was the host-level piece, i.e. nothing ever started the VM itself after a Mac reboot. Before this, a stopped VM was a silent, indefinite CI outage — no error, jobs just queued forever.

launchctl list com.qwestly.ci-vm-start        # check last run / exit status
launchctl unload/load ~/Library/LaunchAgents/com.qwestly.ci-vm-start.plist   # disable/re-enable

Two traps worth not rediscovering:

• docker system prune from a workflow is dangerous here — all six instances share one Docker daemon, so a global prune can kill another job's live service container.
• A runs-on label with no runner behind it queues forever rather than failing. Register an instance before routing anything to its label; move workflows off a label before removing one.

Maintenance timer (Docker cleanup + auto-update watch)

A qwestly-ci-maintenance.timer systemd timer runs inside the guest four times a day, pruning orphaned Docker volumes/images while idle and watching for a runner auto-update. Scripts, unit files, and the full writeup (why it exists, what it found, how to install/check on it) live at ~/Work/services/qwestly-ci-maintenance/.