Ahoy!
Trust me on the greeting. There are pirates in this one. Sort of.
Before we start, a bit of anticipation: there’s a new rack in the lab. It’s orange and grey, it’s 3D printed, and the story of how it got here deserves a post all of its own, which is coming. Today is about something much smaller that lives inside it. Something with a screen.
It started with eight wires and a stupid amount of confidence.
GND, VCC, SCL, SDA, RST, DC, CS, BL. A 2.25" ST7789 bar TFT, 76×284 pixels, bought off AliExpress for a few euros to sit in the rack and tell me things I already have Grafana for, except glanceable, except physical, except mine. I’ve wired up displays before. I know what SPI is. This was going to be a twenty-minute detour before bed.
It was not a twenty-minute detour.
By the time the panel showed a clean “hello, world” I had rebuilt a userspace GPIO library from source, read the actual Python driver’s _init() sequence line by line, discovered that the controller I was fighting is mildly infamous on a specific corner of the Arduino forum, found and then had to un-find a plausible-but-fabricated part number an AI invented for me, and learned more about BJT switching characteristics than I’ve needed since a night class a decade ago that had nothing to do with electronics. Somewhere around hour three I started keeping a running list of things that turned out to be red herrings, mostly so I could feel something other than despair about how long the list was getting.
This is the story of that night. Mostly, though, it’s the story of what it taught me about debugging things that don’t want to be debugged, because the actual play-by-play makes for a genuinely boring read.
The backlight that wouldn’t stay lit
The first real wall was the backlight. I’d wire it up, run a script, and for about a hundred milliseconds the panel would flash to life. Then it went dark. Every time. No exception, no error, no complaint from Python. Just a blink and silence, like the display was teasing me.
What followed was a genuinely humbling sweep of hypotheses. Wrong GPIO pin. gpiozero picking the wrong backend. A dtoverlay I’d added for hardware PWM that turned out to be pure superstition. I’d told the Pi to reserve GPIO18 for PWM before I had any evidence I needed PWM at all, purely because that’s the textbook-correct pin. It sat there quietly claiming the pin out from under everything else for an embarrassing chunk of the night. I2S contention. Stale sysfs exports. I ran gpioinfo so many times I can recite its column layout from memory now.
Eventually I isolated the backlight completely: no display library involved, just raw gpiozero.PWMLED toggling the pin. It worked perfectly. Rock solid, every time. Which meant the pin was fine, the wiring was fine, and the bug lived somewhere in the interaction between “my code” and “the display library’s code,” which is always the worst place for a bug to live, because now you’re debugging someone else’s assumptions instead of your own.
The actual answer, when I finally read the driver’s source instead of guessing around it: the constructor grabs the backlight pin, sets it to INACTIVE as its default line state, and only then does the explicit off→on pulse I’d been watching. If you also pass a GPIO number for backlight= in the constructor, and something else in the request chain has any friction at all, you get exactly the symptom I had. A real pulse, then a line that quietly reverts to a state nobody’s actively driving anymore. The fix, once I found it, was almost insulting in its simplicity: stop asking the library to manage the backlight pin at all. Tie BL straight to ground at the panel, pass backlight=None, and let the physical wire be the only truth that matters.
Four hours to learn that sometimes the fix isn’t cleverer code. It’s less code, and a soldering-iron-shaped shortcut around an abstraction that was never earning its keep.
Reading GRAM like tea leaves
Getting light out of the panel felt like winning. It was not winning. It was the tutorial level.
The next problem was that nothing I drew showed up where I expected, or at all. I spent a long time convinced this was an offset problem (it partly was) before I found the real shape of it: the ST7789 controller has a full 240×320 addressable memory, and this panel’s visible glass is a narrow crop of that. 76×284 pixels sitting at some non-zero offset inside a much bigger frame buffer. Get the offset wrong and you’re not drawing “off screen,” you’re drawing into a part of memory that has no physical pixels wired to it at all. Worse: that memory persists. It’s not cleared by a soft reset, only by an explicit overwrite or a genuine power cycle. Every test I’d run that evening had left fossils behind. Half-frames from three tests ago, quietly waiting to bleed through the next time my offset math happened to overlap them.
I ended up writing a small function whose entire job is to blast zeroes across the full 240×320 range regardless of whatever window I’ve configured, purely so I’d stop debugging ghosts from my own previous attempts. It’s four lines. It probably saved me an hour of very confused screenshots.
The offset numbers themselves, when I finally found them, didn’t come from a datasheet. They came from two strangers on the internet, a year or more apart, who’d bought the same six-euro panel and independently arrived at the exact same two numbers after their own private version of this exact evening. One had posted a patch to an open GitHub issue. The other had left it in an Arduino forum thread that Google surfaces if you’re specific enough with your search terms. Neither of them knew the other existed. That’s not really a coincidence. It’s what community debugging is, at its best: a distributed, asynchronous log of everyone who’s ever fought a piece of hardware, waiting for the next person to search the right four words.
The AI told me a part number that doesn’t exist
Small, honest aside, because it happened mid-session and it’s worth admitting rather than editing out: at one point, stuck, I asked an LLM for help identifying the exact panel variant. It came back with a confident, specific-sounding model number. Plausible formatting, right general shape, wrong in a way that only became obvious once I searched for it and got nothing. The behavior it described was correct; the identifier wrapped around it was invented wholesale. I only caught it because I checked before acting on it.
I don’t think this is an argument against using AI for debugging. I used it for most of this night, including to help me write this post, and it was genuinely useful for reading through a driver’s source line by line at 1am when my own pattern-matching was fried. It’s an argument for treating anything an AI hands you with a specific proper noun in it (a part number, a library name, a version string) as a claim to verify, not a fact to build on. The general shape of an AI’s answer is often right. The specific-sounding details bolted onto it are exactly where confidence outruns evidence, in language models same as in humans who’ve been debugging for four hours straight.
Text that looked like a barcode
Once static images rendered correctly, I tried to scroll some text across it, and got something that looked less like a “Hello, world!” marquee and more like a torn barcode. Thin vertical slivers of green, roughly where letters should be, shuffling around frame to frame in a way that was clearly animating, just not into anything legible.
This one turned out to be two separate bugs wearing a trenchcoat. First: the library’s begin() method is, and I quote its own docstring, """Deprecated. Included in __init__.""" followed by a bare pass. All the actual initialization happens silently inside the constructor, which meant a call I’d been treating as meaningful, and reasoning about the timing of, was doing precisely nothing. Second, buried a level deeper: the constructor never actually calls its own reset() method. The RST pin gets configured and then just… left at whatever its default line state is. Which is INACTIVE. Same failure mode as the backlight, funnily enough. And on most panels, INACTIVE on RST means held in reset. Every _init() command was being shouted at a chip that was, electrically, not listening.
Reading a library’s source instead of trusting its docstrings turned out to be the theme of the entire night, and I want to be honest that this isn’t a novel insight. “Read the source” is debugging advice as old as source code. But there’s a difference between knowing that and actually doing it at hour five, when every instinct is screaming to try one more parameter combination instead of opening the file. The parameter sweep is often faster to start. Reading the source is almost always faster to finish.
Knowing when to stop guessing and go measure something
The thing I keep coming back to, days later, is how much of this night was really about the discipline of elimination: treating “I don’t know why this is happening” as a solvable state rather than an excuse to keep pattern-matching against half-remembered forum posts.
There’s a specific moment I’d point to as the hinge of the whole session: deep in the backlight chaos, a gpioset test appeared to show the pin failing to hold state, which looked exactly like a hardware fault. It wasn’t. gpioset, depending on the libgpiod version, releases the line the instant the command completes unless you explicitly tell it to hold. So what I was watching wasn’t a broken pin. It was a diagnostic tool’s own default behavior masquerading as the bug I was hunting. I only caught it by switching to a held request and watching the difference. That’s the whole game, really: every tool you use to investigate a problem has its own behavior, and if you don’t know that behavior cold, you’ll misattribute it to the system you’re actually trying to understand.
The corollary, and the thing I had to physically stop myself from forgetting more than once: at some point, more code stops being the answer. When I’d swept offsets, MADCTL values, SPI modes, and CS polarity (a real, wide parameter space) and gotten uniform silence across all of it, the right move wasn’t a sixth sweep. It was accepting that the bug had probably left software entirely, and going back to first principles on the wiring itself. In this case that meant reasoning carefully about what each pin should do rather than reaching for a multimeter I didn’t have on hand. But the instinct is the same one that says: if you’ve been staring at a stack trace for forty minutes, close the laptop and go look at the actual server.
What’s actually running now
All of that scar tissue is invisible in the final result, which is sort of the point. The script that’s now genuinely running 24/7 on probe, the little Pi that also happens to be my cluster’s quorum arbiter, polls SSH, ping, and a couple of Docker/LXC checks every twenty seconds and cycles through seven screens: cluster quorum state, per-node uptime, latency to every fleet node, IP addresses, internet reachability against two independent targets, and status for the two tunnels I actually depend on to reach any of this from outside my own LAN. Crossfades between screens so nothing jump-cuts. A staleness indicator if a poll cycle fails, so I never mistake “frozen” for “healthy.” A clean shutdown path that fades to black and physically kills the backlight through a transistor I wired in later, once I decided that leaving an LED backlight lit 24/7/365 in a room I sleep in wasn’t a hill worth dying on.
There’s something fitting here. probe is the one node in the cluster that exists purely to watch and report, never to do real work. Its entire job is noticing. Now it has a physical voice for exactly that. It was already the arbiter. Now it’s an arbiter with a face.
One more screen
Once the panel was solid I wanted one more page: live Docker container status, pulled from the Beszel instance already watching the cluster. Hub and agents, PocketBase underneath, one agent already sitting on a node from an earlier project. I assumed this would be the easy part, after everything else that night.
It was not brutal. It did have its own small arc.
First snag: a third-party dashboard’s own docs, the first result I found, stated flatly that reading Beszel’s data externally requires a superuser account. One search later, Beszel’s own documentation showed a regular authenticated user reading the exact same data just fine. A second-hand claim treated as gospel, overturned by going to the primary source. A small echo of the AI-hallucinated part number from earlier in this same night, the same shape of lesson wearing different clothes: check the thing that’s actually authoritative, not the thing that’s just confident.
So I did it properly. A dedicated read-only user in the Hub, shared access to the systems I actually needed and nothing else, credentials kept out of the script entirely via systemd’s EnvironmentFile= rather than a string sitting in plaintext next to my SSH targets. A smaller, calmer version of the credential-scoping instinct I’d already half-abandoned earlier that same day, when the whole display service ended up running as root out of pure pragmatism.
Then a small honesty problem with the data itself. I wanted memory shown as a percentage, the way the ping and uptime screens show clean, glanceable numbers. Went looking for a denominator to divide by. There wasn’t one. No per-container limit in the API, no host total exposed alongside it. I could have picked a plausible ceiling and faked a percentage that looked precise and meant something slightly wrong. Instead I just show the raw megabyte figure. Small decision, but it’s the same principle as the GRAM fossils and the MADCTL registers from earlier: don’t paper over what you don’t actually know with something that merely looks finished.
The last snag was the more interesting one, structurally. Every other screen in the rotation is a single static frame, handed to the outer loop, shown for SCREEN_DURATION, crossfaded into the next. A simple contract, and it had worked for seven screens without complaint. Then I pointed it at roughly forty containers, four per page, and did the math: one page advancing per full lap of the entire display meant something like ten minutes to see the full container list once. Technically working. Practically useless.
The fix broke the contract on purpose. The container screen now manages its own internal cycle: on its turn, it flips through every page of containers itself, crossfading page to page exactly like the outer loop does between screens, and only then hands control back. Every other screen is still one frame in, one frame out. This one is a screen that contains its own tiny show. Consistency is a means, not an end. The moment it stops serving the actual goal, it’s fine to bend it, deliberately, once, for the one part of the system that genuinely needs it, rather than everywhere, by accident.
And yes, every third-to-seventh screen cycle, three tiny pixel figures dance across it, because I spent an hour trying to hand-draw a Monkey Island tribute in an 18-pixel grid before realizing the actually correct engineering decision was to stop reconstructing an approximation of an asset I already had, and just play the real GIF. Look behind you. A three-headed monkey. Or two dancing ones, close enough. That one’s a small lesson too, in its way: sometimes the clever solution is worse than the boring one, and the boring one is sitting right there in your uploads folder.
If you’re fighting the same panel, here’s the script in full. It’s sold under a few names, but if yours reports as 76×284 with an ST7789-family controller, the odds are decent you’re in exactly this boat. It assumes a Pi 4, the st7789 Python package (lowercase import, the capitalized one is deprecated), gpiozero, and enough patience to adjust the SSH targets and container names to your own homelab:
import st7789
from PIL import Image, ImageDraw, ImageFont
import subprocess
import threading
import time
import re
import json
import signal
import atexit
import random
import gpiozero
import os
# ---------- config ----------
FLEET_NODES = [
{"name": "fleet1", "addr": "192.168.1.10"},
{"name": "fleet2", "addr": "192.168.1.11"},
{"name": "fleet3", "addr": "192.168.1.12"},
{"name": "fleet4", "addr": "192.168.1.13"},
]
SSH_TARGET = "192.168.1.10" # fleet1's addr — needs pvecm/corosync visibility
SSH_USER = "root"
SSH_KEY = "/home/user/.ssh/id_ed25519"
CLOUDFLARED_HOST = "192.168.1.10" # fleet1's addr — hosts LXC 102
CLOUDFLARED_LXC_ID = "102"
TAILSCALE_CONTAINER = "tailscale" # docker container name/ID on probe — verify with `docker ps`
POLL_INTERVAL = 20 # seconds between data refreshes
SCREEN_DURATION = 4 # seconds each screen is shown
FADE_STEPS = 8 # more steps = smoother fade, more SPI traffic per transition
BESZEL_URL = "http://192.168.1.14:8090"
BESZEL_EMAIL = os.environ.get("BESZEL_EMAIL")
BESZEL_PASSWORD = os.environ.get("BESZEL_PASSWORD")
_beszel_token = None
_beszel_page = 0
# ---------- display geometry (confirmed working) ----------
WIDTH, HEIGHT = 284, 76
OFFSET_LEFT, OFFSET_TOP = 18, 82
assert OFFSET_LEFT + WIDTH <= 320, f"CASET overflow: {OFFSET_LEFT}+{WIDTH} > 320"
assert OFFSET_TOP + HEIGHT <= 240, f"RASET overflow: {OFFSET_TOP}+{HEIGHT} > 240"
def clear_gram(display):
"""Blank the controller's full 240x320 GRAM, regardless of the
configured window/offsets. GRAM persists across soft resets —
only this or a real power cycle clears it."""
display.command(0x2A)
display.data(0x00); display.data(0x00)
display.data(0x00); display.data(0xEF)
display.command(0x2B)
display.data(0x00); display.data(0x00)
display.data(0x01); display.data(0x3F)
display.command(0x2C)
display.data(list(bytes(240 * 320 * 2)))
display = st7789.ST7789(
port=0, cs=0, dc=24, rst=25, backlight=None,
spi_speed_hz=4_000_000,
width=WIDTH, height=HEIGHT, rotation=0,
offset_left=OFFSET_LEFT, offset_top=OFFSET_TOP,
invert=False
)
display.reset()
display._init()
clear_gram(display)
# Backlight is switched via a transistor on GPIO18 (true hardware PWM),
# fully decoupled from the st7789 library's own GPIO handling — BL pin
# itself stays hardwired to GND at the panel, this switches the ground
# return path instead. Safe on GPIO18 because this circuit is driven
# entirely through gpiozero, outside the st7789 library's own GPIO calls
# (the library only touches this pin if backlight= is passed to its
# constructor, which we don't).
#
# Fixed at 0.8 rather than dynamically dimmed: this BJT's switching speed
# isn't fast enough to hold a clean, flicker-free duty cycle at low PWM
# levels — a lightly-driven transistor spends too much of a short pulse
# in transition rather than fully on/off. 0.8 sits comfortably above
# where that showed up. A MOSFET would remove this limitation if lower
# brightness is wanted later.
BL_SWITCH_PIN = 18
bl_switch = gpiozero.PWMLED(BL_SWITCH_PIN, frequency=1000)
bl_switch.value = 0.8
font_big = ImageFont.truetype("/usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf", 22)
font_small = ImageFont.truetype("/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf", 14)
font_tiny = ImageFont.truetype("/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf", 10)
BLACK_FRAME = Image.new("RGB", (WIDTH, HEIGHT), (0, 0, 0))
# ---------- shared state ----------
state = {
"quorate": None, # True / False / None (None = unknown/error)
"votes": "?/?",
"uptimes": {}, # name -> uptime string or "unreachable"
"pings": {}, # name -> (rtt_ms or None, loss_pct)
"internet": ((None, 100), (None, 100)), # (1.1.1.1 result, 8.8.8.8 result)
"external_ip": None, # public IP string, or None if fetch failed
"local_uptime": "?",
"cloudflared": "unknown",
"tailscale": "unknown",
"last_poll": None,
"last_poll_ok": False,
"beszel_containers": [],
}
state_lock = threading.Lock()
def run(cmd, timeout=5):
"""Run a shell command, return stdout on success or None on any failure."""
try:
r = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout)
return r.stdout if r.returncode == 0 else None
except (subprocess.TimeoutExpired, OSError):
return None
def fetch_quorum():
out = run([
"ssh","-i", SSH_KEY, "-o", "BatchMode=yes", "-o", "ConnectTimeout=4",
f"{SSH_USER}@{SSH_TARGET}", "pvecm status"
], timeout=8)
if out is None:
return None, "?/?"
quorate = "Quorate:" in out and "Yes" in out.split("Quorate:")[1].splitlines()[0]
m = re.search(r"Expected votes:\s*(\d+)", out)
expected = m.group(1) if m else "?"
m = re.search(r"Total votes:\s*(\d+)", out)
total = m.group(1) if m else "?"
return quorate, f"{total}/{expected}"
def fetch_uptime(addr):
out = run([
"ssh", "-i", SSH_KEY, "-o", "BatchMode=yes", "-o", "ConnectTimeout=3",
f"{SSH_USER}@{addr}", "uptime", "-p"
], timeout=6)
return out.strip().replace("up ", "") if out else "unreachable"
def fetch_local_uptime():
out = run(["uptime", "-p"], timeout=3)
return out.strip().replace("up ", "") if out else "?"
def fetch_local_ip():
"""Best-effort local (LAN) IP of probe itself, for the IPs screen."""
out = run(["hostname", "-I"], timeout=3)
return out.split()[0] if out else "?"
def fetch_ping(addr):
out = run(["ping", "-c", "3", "-W", "1", addr], timeout=6)
if out is None:
return None, 100
loss_m = re.search(r"(\d+)% packet loss", out)
rtt_m = re.search(r"= [\d.]+/([\d.]+)/", out) # avg from min/avg/max/mdev
loss = int(loss_m.group(1)) if loss_m else 100
rtt = float(rtt_m.group(1)) if rtt_m else None
return rtt, loss
def fetch_internet():
"""Ping two independent well-provisioned targets — divergence between
them is itself informative (a route/peer-specific issue vs. uplink dead)."""
return fetch_ping("1.1.1.1"), fetch_ping("8.8.8.8")
def fetch_external_ip():
out = run(["curl", "-s", "--max-time", "4", "https://api.ipify.org"], timeout=6)
if out and re.match(r"^\d+\.\d+\.\d+\.\d+$", out.strip()):
return out.strip()
return None
def fetch_cloudflared():
"""cloudflared runs inside LXC 102 on fleet1 — query via pct exec over SSH."""
out = run([
"ssh", "-i", SSH_KEY, "-o", "BatchMode=yes", "-o", "ConnectTimeout=4",
f"{SSH_USER}@{CLOUDFLARED_HOST}",
f"pct exec {CLOUDFLARED_LXC_ID} -- systemctl is-active cloudflared"
], timeout=8)
if out is None:
return "unreachable"
return out.strip() # "active", "inactive", "failed", etc.
def fetch_tailscale():
"""tailscale runs in a Docker container on probe itself — no SSH needed."""
out = run(["docker", "inspect", "-f", "{{.State.Status}}", TAILSCALE_CONTAINER], timeout=5)
if out is None:
return "unreachable"
container_status = out.strip() # "running", "exited", "restarting", etc.
if container_status != "running":
return container_status
ts_out = run(["docker", "exec", TAILSCALE_CONTAINER, "tailscale", "status", "--json"], timeout=5)
if ts_out is None:
return "running (ts unknown)"
try:
data = json.loads(ts_out)
return data.get("BackendState", "?") # "Running", "Stopped", "NeedsLogin", etc.
except json.JSONDecodeError:
return "running (ts parse error)"
LOCAL_IP = fetch_local_ip() # fetched once at startup — LAN IP doesn't change mid-run
def poll_loop():
while True:
quorate, votes = fetch_quorum()
uptimes = {n["name"]: fetch_uptime(n["addr"]) for n in FLEET_NODES}
pings = {n["name"]: fetch_ping(n["addr"]) for n in FLEET_NODES}
internet = fetch_internet()
external_ip = fetch_external_ip()
local_uptime = fetch_local_uptime()
cloudflared = fetch_cloudflared()
tailscale = fetch_tailscale()
beszel_containers = fetch_beszel_containers()
with state_lock:
state["quorate"] = quorate
state["votes"] = votes
state["uptimes"] = uptimes
state["pings"] = pings
state["internet"] = internet
state["external_ip"] = external_ip
state["local_uptime"] = local_uptime
state["cloudflared"] = cloudflared
state["tailscale"] = tailscale
state["last_poll"] = time.time()
state["last_poll_ok"] = quorate is not None
state["beszel_containers"] = beszel_containers
time.sleep(POLL_INTERVAL)
def _status_color(container):
status = container.get("status", "")
health = container.get("health", 0)
if not status.startswith("Up"):
return (255, 0, 0) # red — stopped/restarting/exited
if health not in (0, None):
return (255, 165, 0) # yellow — running but health check unhappy
return (0, 255, 0) # green — up, healthy
def screen_beszel(s):
"""Cycles through every container page in a single visit, crossfading
between pages and fading in from whatever was on screen before.
Hands back the last frame so the outer loop's own crossfade into the
next screen works exactly as it does for every other screen."""
global _last_frame
containers = sorted(s.get("beszel_containers") or [], key=lambda c: c.get("name", ""))
if not containers:
frame = Image.new("RGB", (WIDTH, HEIGHT), (0, 0, 0))
d = ImageDraw.Draw(frame)
draw_centered(d, "BESZEL", font_big, 8, (255, 165, 0))
draw_centered(d, "no data", font_small, 44, (150, 150, 150))
crossfade(display, _last_frame, frame, steps=FADE_STEPS)
_last_frame = frame
return frame
PER_PAGE = 4
PAGE_HOLD = 5 # seconds each page holds, after its fade completes
total_pages = max(1, (len(containers) + PER_PAGE - 1) // PER_PAGE)
row_h = HEIGHT / PER_PAGE
for page in range(total_pages):
chunk = containers[page * PER_PAGE:(page + 1) * PER_PAGE]
frame = Image.new("RGB", (WIDTH, HEIGHT), (0, 0, 0))
d = ImageDraw.Draw(frame)
for i, c in enumerate(chunk):
y = int(i * row_h) + max(1, int((row_h - 12) / 2))
color = _status_color(c)
d.ellipse([3, y + 2, 9, y + 8], fill=color)
name = c.get("name", "?")[:14]
cpu = f"{c.get('cpu', 0):.1f}%"
mem = f"{c.get('memory', 0):.0f}MB"
d.text((14, y), name, font=font_tiny, fill=(180, 180, 180))
d.text((WIDTH - 70, y), cpu, font=font_tiny, fill=(150, 150, 150))
d.text((WIDTH - 34, y), mem, font=font_tiny, fill=(150, 150, 150))
if i < len(chunk) - 1:
d.line([(0, int((i + 1) * row_h)), (WIDTH, int((i + 1) * row_h))], fill=(45, 45, 45))
d.text((WIDTH - 28, HEIGHT - 10), f"{page + 1}/{total_pages}", font=font_tiny, fill=(80, 80, 80))
crossfade(display, _last_frame, frame, steps=FADE_STEPS)
_last_frame = frame
time.sleep(PAGE_HOLD)
return _last_frame
def _beszel_authenticate():
out = run([
"curl", "-s", "-X", "POST", f"{BESZEL_URL}/api/collections/users/auth-with-password",
"-H", "Content-Type: application/json",
"-d", json.dumps({"identity": BESZEL_EMAIL, "password": BESZEL_PASSWORD})
], timeout=8)
if out is None:
return None
try:
return json.loads(out).get("token")
except json.JSONDecodeError:
return None
def _beszel_get(path):
"""Authenticated GET against the Hub, re-authenticating once on a
401/403 in case the cached token expired."""
global _beszel_token
if _beszel_token is None:
_beszel_token = _beszel_authenticate()
if _beszel_token is None:
return None
def _try():
out = run(["curl", "-s", f"{BESZEL_URL}{path}",
"-H", f"Authorization: {_beszel_token}"], timeout=8)
if out is None:
return None
try:
return json.loads(out)
except json.JSONDecodeError:
return None
data = _try()
if isinstance(data, dict) and data.get("status") in (401, 403):
_beszel_token = _beszel_authenticate()
if _beszel_token is None:
return None
data = _try()
return data
def fetch_beszel_containers():
"""Fetch every container record across every system Beszel tracks,
paginating as needed. No scoping — count varies day to day."""
items = []
page = 1
while True:
data = _beszel_get(f"/api/collections/containers/records?page={page}&perPage=100")
if not data or "items" not in data:
break
items.extend(data["items"])
if page >= data.get("totalPages", 1):
break
page += 1
return items
# ---------- rendering ----------
def draw_centered(draw, text, font, y, fill):
bbox = draw.textbbox((0, 0), text, font=font)
w = bbox[2] - bbox[0]
draw.text(((WIDTH - w) // 2 - bbox[0], y), text, font=font, fill=fill)
def draw_rows(frame, rows):
"""rows: list of (label, value, color). Stacks evenly across HEIGHT,
label left-aligned, value right-aligned, on the same line."""
d = ImageDraw.Draw(frame)
n = len(rows)
row_h = HEIGHT / n
for i, (label, value, color) in enumerate(rows):
y = int(i * row_h) + max(1, int((row_h - 12) / 2)) # vertically center in row
d.text((3, y), label, font=font_tiny, fill=(150, 150, 150))
vbbox = d.textbbox((0, 0), value, font=font_tiny)
vw = vbbox[2] - vbbox[0]
d.text((WIDTH - vw - 4, y), value, font=font_tiny, fill=color)
if i < n - 1:
d.line([(0, int((i + 1) * row_h)), (WIDTH, int((i + 1) * row_h))], fill=(45, 45, 45))
return frame
def screen_quorum(s):
# unchanged style — big centered status
frame = Image.new("RGB", (WIDTH, HEIGHT), (0, 0, 0))
d = ImageDraw.Draw(frame)
if s["quorate"] is None:
draw_centered(d, "Nexus QUORUM: UNKNOWN", font_big, 8, (255, 165, 0))
draw_centered(d, "ssh to fleet1 failed", font_small, 44, (150, 150, 150))
elif s["quorate"]:
draw_centered(d, "Nexus QUORUM", font_big, 8, (0, 255, 0))
draw_centered(d, f"votes {s['votes']}", font_small, 44, (150, 150, 150))
else:
draw_centered(d, "NOT QUORATE", font_big, 8, (255, 0, 0))
draw_centered(d, f"votes {s['votes']}", font_small, 44, (150, 150, 150))
return frame
def screen_uptimes(s):
frame = Image.new("RGB", (WIDTH, HEIGHT), (0, 0, 0))
rows = [("probe (local)", s["local_uptime"], (0, 200, 255))]
for node in FLEET_NODES:
name = node["name"]
up = s["uptimes"].get(name, "?")
color = (255, 0, 0) if up == "unreachable" else (0, 255, 0)
rows.append((name, up, color))
return draw_rows(frame, rows)
def screen_ping(s):
frame = Image.new("RGB", (WIDTH, HEIGHT), (0, 0, 0))
rows = []
for node in FLEET_NODES:
name = node["name"]
rtt, loss = s["pings"].get(name, (None, 100))
if rtt is None or loss == 100:
rows.append((name, "DOWN", (255, 0, 0)))
else:
color = (0, 255, 0) if loss == 0 else (255, 165, 0)
val = f"{rtt:.1f}ms" + (f" ({loss}%)" if loss else "")
rows.append((name, val, color))
return draw_rows(frame, rows)
def screen_ips(s):
frame = Image.new("RGB", (WIDTH, HEIGHT), (0, 0, 0))
rows = [("probe (local)", LOCAL_IP, (0, 200, 255))]
rows += [(node["name"], node["addr"], (0, 200, 255)) for node in FLEET_NODES]
return draw_rows(frame, rows)
def screen_internet(s):
frame = Image.new("RGB", (WIDTH, HEIGHT), (0, 0, 0))
ext_ip = s["external_ip"] or "unavailable"
(rtt_a, loss_a), (rtt_b, loss_b) = s["internet"]
rows = [("external ip", ext_ip, (0, 200, 255))]
for label, rtt, loss in [("1.1.1.1", rtt_a, loss_a), ("8.8.8.8", rtt_b, loss_b)]:
if rtt is None or loss == 100:
rows.append((label, "DOWN", (255, 0, 0)))
else:
color = (0, 255, 0) if loss == 0 else (255, 165, 0)
val = f"{rtt:.1f}ms" + (f" ({loss}%)" if loss else "")
rows.append((label, val, color))
return draw_rows(frame, rows)
def screen_cloudflared(s):
frame = Image.new("RGB", (WIDTH, HEIGHT), (0, 0, 0))
d = ImageDraw.Draw(frame)
status = s["cloudflared"]
if status == "active":
draw_centered(d, "CLOUDFLARED", font_big, 8, (0, 255, 0))
draw_centered(d, "tunnel active", font_small, 44, (150, 150, 150))
elif status == "unreachable":
draw_centered(d, "CLOUDFLARED", font_big, 8, (255, 165, 0))
draw_centered(d, "lxc unreachable", font_small, 44, (150, 150, 150))
else:
draw_centered(d, "CLOUDFLARED", font_big, 8, (255, 0, 0))
draw_centered(d, status, font_small, 44, (150, 150, 150))
return frame
def screen_tailscale(s):
frame = Image.new("RGB", (WIDTH, HEIGHT), (0, 0, 0))
d = ImageDraw.Draw(frame)
status = s["tailscale"]
if status == "Running":
draw_centered(d, "TAILSCALE", font_big, 8, (0, 255, 0))
draw_centered(d, "connected", font_small, 44, (150, 150, 150))
elif status == "unreachable":
draw_centered(d, "TAILSCALE", font_big, 8, (255, 165, 0))
draw_centered(d, "container unreachable", font_small, 44, (150, 150, 150))
else:
draw_centered(d, "TAILSCALE", font_big, 8, (255, 0, 0))
draw_centered(d, status, font_small, 44, (150, 150, 150))
return frame
SCREENS = [
screen_quorum,
screen_uptimes,
screen_ping,
screen_ips,
screen_internet,
screen_cloudflared,
screen_tailscale,
screen_beszel,
]
# ---------- GIF dance animation ----------
DANCE_GIF_PATH = "/home/user/monkeysdancing.gif" # adjust to wherever you copy it
def _load_gif_frames(path):
"""Loads all frames of a GIF, scaled to fit the panel (letterboxed,
aspect-preserved), pre-composited onto a black background. Returns a
list of (PIL.Image, duration_seconds) or None if the file is missing
or fails to load — caller must handle that gracefully."""
try:
im = Image.open(path)
except (FileNotFoundError, OSError):
return None
frames = []
src_w, src_h = im.size
# fit by height (the more constraining dimension for this GIF's aspect)
scale = HEIGHT / src_h
new_w, new_h = int(src_w * scale), int(src_h * scale)
new_w = min(new_w, WIDTH) # never exceed panel width
x_offset = (WIDTH - new_w) // 2
y_offset = (HEIGHT - new_h) // 2
try:
for frame_index in range(im.n_frames):
im.seek(frame_index)
duration_ms = im.info.get("duration", 100)
frame_rgba = im.convert("RGBA").resize((new_w, new_h), Image.NEAREST)
canvas = Image.new("RGB", (WIDTH, HEIGHT), (0, 0, 0))
canvas.paste(frame_rgba, (x_offset, y_offset), frame_rgba)
frames.append((canvas, duration_ms / 1000.0))
except Exception:
return None
return frames if frames else None
_dance_frames_cache = None
_dance_frames_loaded = False
def dance_animation(loops=2):
"""Plays the actual source GIF, scaled to fit the panel. Frames are
loaded once and cached — subsequent calls are just a cheap replay.
Returns the final frame so the caller can crossfade smoothly into
whatever comes next. Returns None (no-op) if the GIF can't be loaded,
so a missing/misplaced file degrades gracefully rather than crashing
the whole display loop."""
global _dance_frames_cache, _dance_frames_loaded
if not _dance_frames_loaded:
_dance_frames_cache = _load_gif_frames(DANCE_GIF_PATH)
_dance_frames_loaded = True
if _dance_frames_cache is None:
print(f"dance_animation: couldn't load {DANCE_GIF_PATH}, skipping")
if not _dance_frames_cache:
return None
last_frame = None
for _loop in range(loops):
for frame, duration in _dance_frames_cache:
display.display(frame)
last_frame = frame
time.sleep(duration)
return last_frame
# ---------- fade helper ----------
def crossfade(display, frame_a, frame_b, steps=FADE_STEPS):
"""Blend from frame_a to frame_b over `steps` intermediate frames.
Each step is a full SPI push, so total fade time scales with
steps * (frame transfer time) — at 4MHz that's roughly 90-100ms/step."""
for i in range(1, steps + 1):
alpha = i / steps
blended = Image.blend(frame_a, frame_b, alpha)
display.display(blended)
# ---------- clean shutdown ----------
_last_frame = BLACK_FRAME
_shutting_down = False
def shutdown(*_args):
global _shutting_down
if _shutting_down:
return
_shutting_down = True
try:
crossfade(display, _last_frame, BLACK_FRAME, steps=FADE_STEPS)
clear_gram(display)
bl_switch.off()
except Exception:
pass # never let cleanup itself crash the exit path
raise SystemExit(0)
signal.signal(signal.SIGINT, shutdown) # Ctrl+C
signal.signal(signal.SIGTERM, shutdown) # systemd stop / kill
atexit.register(lambda: shutdown() if not _shutting_down else None)
# ---------- main ----------
poller = threading.Thread(target=poll_loop, daemon=True)
poller.start()
print("waiting for first poll data...")
while state["last_poll"] is None:
result = dance_animation(loops=1)
if result is not None:
_last_frame = result
else:
# gif failed to load — don't busy-loop, just wait quietly
time.sleep(0.5)
try:
i = 0
cycles_until_dance = random.randint(3, 7)
while True:
with state_lock:
s = dict(state)
frame = SCREENS[i % len(SCREENS)](s)
if not s["last_poll_ok"]:
ImageDraw.Draw(frame).ellipse([WIDTH - 8, 2, WIDTH - 2, 8], fill=(255, 0, 0))
crossfade(display, _last_frame, frame, steps=FADE_STEPS)
_last_frame = frame
time.sleep(max(0, SCREEN_DURATION - (FADE_STEPS * 0.1))) # rough compensation
i += 1
cycles_until_dance -= 1
if cycles_until_dance <= 0:
# crossfade out of the current screen into the dance, then
# crossfade from the dance's last frame into whatever's next
last_dance_frame = dance_animation(loops=2)
if last_dance_frame is not None:
_last_frame = last_dance_frame
cycles_until_dance = random.randint(3, 7)
except KeyboardInterrupt:
pass
If your offsets don’t match mine (and they probably won’t, panel to panel), go back to first principles rather than guessing: draw a full-frame border with four differently-colored corners, push it, and nudge OFFSET_LEFT/OFFSET_TOP until all four corners are visible at once. It’s the single fastest diagnostic I found all night, and I found it embarrassingly late.
Next post: the story of the rack this little screen lives in. There’s a horse involved. Sort of.
Happy building ⚒