Tales from the Server-Room: The 31 MB Log File That Ate My QNAP
Welcome to Tales from the Server-Room, a series about absurd IT incidents from a server room that is technically my living room.
There is no raised floor. The climate-control system is a window. The network operations centre is a sofa, and the incident commander is usually holding a coffee while staring suspiciously at a blinking box near the television.
This is where we solve very serious infrastructure problems with professional tools, careful verification, and the emotional maturity of two goblins who have been given shell access.
Today’s cast:
- Plex, requesting an urgent security update.
- QNAP, a small ARM-based NAS containing 16 TB of storage and several strong opinions.
- QVPN, an unemployed former VPN client with no clients, no traffic, and an uncompromising commitment to autobiography.
- Codex, my AI operations partner and co-owner of the metaphorical flashlight.
- Me, the adult allegedly responsible for all of this.
The five-minute mission
Plex had emailed server owners recommending an immediate upgrade to Plex Media Server 1.43.3 because it addressed multiple security issues. My server was still on 1.42.2, running as a native QNAP package.
The plan was simple:
- Download the correct ARMv8 package from Plex.
- Verify the checksum.
- Install it on top of the existing package.
- Confirm that Plex came back online.
In a conventional server room, this would require a change ticket, an approved maintenance window, and someone called Brian replying-all with “Any impact to production?”
In my server room, I looked at the sofa, confirmed that nobody was watching anything, and proceeded.
Instead of an update, I got a blue QTS screen that said Loading… forever.
Naturally, the Plex update turned into an investigation of an unrelated VPN package, a full system ramdisk, a misleading amount of free storage, and one of those bugs where every layer is technically alive but the whole thing is still dead.
The maintenance window had officially become an episode.
The corpse could still answer HTTP
The QTS login page loaded, accepted the login, and then stalled while opening the desktop. That distinction mattered: the web server itself was responding.
The browser console showed the real failure:
1
2
GET /cgi-bin/sysinfoReq.cgi 500 (Internal Server Error)
TypeError: Cannot read properties of undefined (reading 'lang')
The JavaScript error was only collateral damage. QTS expected a valid response from sysinfoReq.cgi, got an HTTP 500 instead, and then tried to read fields that did not exist.
So the question was no longer, “Why is the login screen stuck?”
It was, “Why can QTS serve the page but not answer a basic system-information request?”
The blue screen was not frozen. It was waiting for information from a colleague who had quietly died at their desk.
The doorbell worked, but nobody was home
I manage the NAS through a small operations repository called qmaestro. It contains status scripts and runbooks for Plex, QTS, Home Assistant, Tailscale, containers, storage, and network access.
The QTS health check showed that both web processes were alive:
1
2
3
apache_proxys HTTPS on port 443: running
apache_proxy HTTP redirect on port 8080: running
QTS login page: HTTP 200
Restarting the web server would have been satisfying. It is the sysadmin equivalent of slapping the side of a television and looking wise.
It would also have been cargo-cult troubleshooting. The failing component was behind the page, not the front door.
The broader NAS health check found this:
1
2
3
Filesystem Size Used Available Use%
QTS system root 277 MB 277 MB 0 MB 100%
Main data volume 16 TB 1.2 TB 14.8 TB 7%
Fourteen terabytes free, yet the system could not write a tiny CGI response. This is like owning an empty warehouse but being unable to enter because the doormat is full.
QNAP runs important parts of QTS from a small system ramdisk. The giant storage pool where the media lives can be almost empty while the operating system itself is completely out of space.
That explained more than the broken login. Other internal services were also complaining that they could not persist state.
Enter QVPN, the unemployed diarist
The largest abnormal file on the system root was:
1
/var/log/network/QVPN.log 31 MB
Thirty-one megabytes is adorable on a 16 TB NAS. On a 277 MB system filesystem, it is a home invasion.
The end of the log repeated the same pair of debug messages roughly every ten seconds:
1
2
[DEBUG] qvpnc.cc main:cmd = config
[DEBUG] global_options.cc cmd_global_opt:argc=3, argvs=config get gw_setting
QVPN kept waking up, getting asked for its status, and shitting debug lines all over the system disk until QTS ran out of room to breathe.
It had no active VPN connection. It had no useful traffic to carry. But every ten seconds it wrote another tiny entry announcing that it had once again checked the same setting.
QVPN was the infrastructure equivalent of an employee with no assigned work who submits a detailed timesheet for refreshing their inbox. It was, quite simply, the log-spamming asshole of the day.
Bag the evidence before hitting it with a shovel
I did not want to throw away the only evidence, so the log was copied to the large data volume before the active file was truncated:
1
2
3
4
5
6
mkdir -p /share/Public/qmaestro-recovery
cp /var/log/network/QVPN.log \
/share/Public/qmaestro-recovery/QVPN.log.before-root-cleanup
sha256sum /var/log/network/QVPN.log \
/share/Public/qmaestro-recovery/QVPN.log.before-root-cleanup
: > /var/log/network/QVPN.log
After confirming that both checksums matched, truncating the active log immediately freed about 31 MB:
1
2
QTS system root: 100% -> 89%
sysinfoReq.cgi: HTTP 500 -> HTTP 200
The QTS desktop could load again without restarting the NAS. The blue screen vanished, the desktop appeared, and somewhere in the living room a tiny fan continued spinning as though none of this had been its problem.
That was the actual fix for the blue loading screen. Clearing browser storage, disabling extensions, or repeatedly restarting Apache would never have addressed the cause.
Back to the side quest we started with
With QTS functional again, I returned to the task that started the adventure.
Plex’s official QNAP guidance is important here: install the new package on top of the existing installation. Never uninstall Plex first. On QNAP, the uninstall button is a tiny red suicide vest for your server data.
The package came from Plex’s official download endpoint, matched the NAS architecture, and matched Plex’s published checksum. I transferred it to the NAS and queued it with QNAP’s package manager:
1
2
3
4
5
6
7
scp PlexMediaServer-<version>-aarch64.qpkg \
<nas-user>@<nas>:/share/CACHEDEV1_DATA/Public/
ssh <nas-user>@<nas> \
'/sbin/qpkg_cli --manually \
/share/CACHEDEV1_DATA/Public/PlexMediaServer-<version>-aarch64.qpkg \
--keep'
The canonical path mattered. The friendly /share/Public alias worked over SSH, but QNAP’s package daemon claimed that the file did not exist when given that path. Using /share/CACHEDEV1_DATA/Public worked.
Because apparently a path can be correct for the shell and wrong for the service running on the same box. QNAP is an escape-room designer with a salary. Homelabs build character, mostly the kind that mutters at furniture.
The update stopped Plex briefly, installed the new package in place, and started it again. I verified more than the version label:
- The QPKG state was
enabled. - Plex was running under the expected system account.
- Port 32400 was listening.
- The identity endpoint returned HTTP 200.
- The identity response reported Plex Media Server 1.43.3.
- The existing server remained claimed and retained its machine identity.
Interrogating the suspect
The log was empty, but it immediately began growing again. Fixing the symptom without investigating that would only schedule another incident.
I was ready to disable QVPN, but first I checked its active profiles, server functions, interfaces, routes, and connections. You do the dependency check before taking a service behind the barn for one last affectionate pat.
The live QVPN configuration revealed two old OpenVPN client profiles. Both were disabled and disconnected. A leftover gateway-failover configuration still pointed at them, but the NAS had:
- No active OpenVPN, WireGuard,
tun, ortapinterface. - No VPN route replacing the normal LAN gateway.
- No service depending on those profiles.
The remote-access setup uses Tailscale, which is a separate QNAP package. Plex, Home Assistant, the Tailscale HTTPS endpoint, and a dedicated Ethernet share on the second NAS port did not depend on QVPN either.
The profile names looked suspiciously like an old Private Internet Access setup. It was the dusty remains of a project that had served some forgotten purpose before Tailscale took over remote access.
The current QVPN version probably deserved an update after its performance, but an unused service does not get rewarded with professional development. It gets disabled.
QVPN was disabled, not uninstalled. That kept the decision reversible while stopping the unused service and its runaway log.
The technical term for this procedure is “putting it on the naughty step.”
The final verification looked like this:
1
2
3
4
5
6
7
8
QVPN package: disabled
QVPN log growth: stopped
Plex 1.43.3: healthy
Home Assistant: healthy
QTS HTTPS: healthy
Tailscale HTTPS: listening
Secondary LAN: configured
Default route: unchanged
Why the robot was useful
The useful automation was not a magical “fix my NAS” button. It was accumulated operational context:
- The NAS model and CPU architecture were already documented.
- Plex was known to be a native QPKG rather than a container.
- The correct Plex health probes already existed.
- QTS ports and processes were documented, including a previous HTTPS outage.
- Tailscale and the secondary Ethernet setup had explicit verification checks.
- Credentials stayed outside the repository.
That context let an AI agent investigate through the qmaestro repository without rediscovering the entire homelab or poking random services until something moved.
Codex could see that Plex was a QPKG, that Tailscale was separate from QVPN, which paths were safe, which account Plex should run under, and how to test every dependent service. When one route failed, it could continue from evidence instead of guessing.
This is my preferred version of AI operations: not a chatbot confidently suggesting that I reinstall the universe, but a collaborator with access to the runbooks and enough patience to follow the smell of smoke.
The incident has now been added to the runbook as well. The next troubleshooting session starts with the answer this one had to earn.
Post-incident debrief from the sofa
Free storage is not the same as a healthy system filesystem
Always check every relevant mount. A mostly empty 16 TB data volume says nothing about a tiny root ramdisk.
Follow the first server-side failure
The JavaScript exception looked dramatic, but the HTTP 500 came first. Fix the backend response and the frontend error disappears with it.
Preserve evidence before cleanup
Copy and verify a runaway log before truncating it. Recovery should not destroy the clues needed to prevent recurrence.
Disabled configuration can still create work
The OpenVPN profiles were disconnected, yet QVPN was still installed, polled, and logging. “Not actively used” is not the same as “doing nothing.”
Verify dependencies before disabling infrastructure
I did not assume that QVPN was unrelated to Tailscale or the NAS gateway. I checked interfaces, routes, listeners, package state, and the dependent services before and after disabling it.
Automate context, not just commands
Scripts are useful. Runbooks, architecture notes, known failure modes, and explicit safety rules are what make those scripts safe to run at 17:30 when a five-minute update has become an unsolicited systems-administration workshop.
The final result was exactly what I wanted at the beginning: Plex patched, QTS working, QVPN sitting quietly in a disabled state, and no media data harmed.
It just took a brief detour through a 31 MB file with the confidence of a 16 TB one.
The server room returned to normal operation. The sofa resumed its duties. No incident report was presented to senior management because senior management was also me and wanted dinner.
Until next time, when another appliance in the living room decides it has become enterprise infrastructure.
AI disclosure: This incident was diagnosed and resolved collaboratively with Codex using my
qmaestrooperations repository. I reviewed the actions and kept the final operational decisions human-approved.