Debugging packpay itself: a real, working methodology
v0.72.103.244

Troubleshooting.org is for a user whose real circuit doesn't lay out well. This document is for a developer investigating packpay's own code – a reported bug, a suspicious result, a question about whether a past fix actually did what it claims. Every practice below is one this project has genuinely, repeatedly relied on, not aspirational advice – each is illustrated with a real, specific example from this project's own actual history, most from ResearchJournal.org's own 2026-09 entries.

A bug report, a code comment, an old docstring, a memory of "I already checked this" are all real, useful hints, never a substitute for looking. This project's own real, repeated pattern: read the actual, current source before acting on what anything else says about it.

A real, worked example: a field report claimed real_size_rows() used a component's own courtyard WIDTH unconditionally, with no orientation check. Confirmed directly by reading the function itself (fp.courtyard_size()[0], no conditional anywhere near it) before writing a single line of fix – and the direct reproduction that followed found the real problem was WORSE than the report described (five components violated bounds, not the two specifically named). Trusting the report's own scope alone would have left three real violations unfixed.

Before changing anything, write a short, direct Python script that exercises exactly the reported behavior – not a guess at what might trigger it, the EXACT scenario, using this project's own real functions directly (load_from_schematic(), layout(), render()), not through a CLI wrapper unless the CLI itself is what's suspect.

Where a bug report gives an exact reproduction recipe (circuit, config, exact call), use that recipe verbatim first, before simplifying or generalizing it – a recipe recorded precisely in FutureResearch.org exists so a future session doesn't have to reconstruct it from a vague description.

Python's own per-string hash() is randomized per process by default (PYTHONHASHSEED) – several of this project's own real placement decisions (set/dict iteration order in a few real code paths) depend on this, which is exactly why some bugs here have historically been "roughly 1-in-3 runs," not consistent.

Before concluding a non-deterministic bug is fixed (or before concluding it's absent), confirm randomization is genuinely active in the current environment – a real, one-line check:

$ python3 -c "print(hash('test'))"; python3 -c "print(hash('test'))"

Two different real values confirms genuine, per-process randomization; two identical values means PYTHONHASHSEED is fixed, and every run will be identically deterministic regardless of how many times it's repeated – a false sense of confidence either way if this isn't checked first. Then run enough real iterations that a real absence is actually meaningful: for an original rate of roughly 1-in-3, twenty clean runs in a row is (2/3)^20, under 0.03% – convincing; five clean runs, at 13%, is not.

Confirming a bug no longer reproduces is not the same as confirming WHY it doesn't. Where practical, revert the suspected fix, confirm the original symptom returns, then restore the fix – direct, causal evidence, not correlation.

A real, worked example (2026-09-06): a previously reported, intermittent crash on rp2040-basic-m1 no longer reproduced across thirty real runs. Rather than record "seems fixed," the actual cause was pinned down: reverting an unrelated, earlier fix (Finding 1's own resistor/diode sizing correction) alone still produced zero crashes – inconclusive on its own. Reverting a SECOND, separate fix (render()'s own self-verification, Finding 4) on top of that, recreating the exact original code state, brought the exact original crash back reliably. Restoring both, and reconfirming clean, is what actually demonstrated Finding 4 – specifically, not generally – was what closed it. The same real discipline applies in the other direction: before shipping any new, permanent regression test, deliberately break the real fix it's meant to guard and confirm the test actually fails – a test that has never been observed to fail is not yet confirmed to test anything.

breadboard_lib.py's own drawing state (body, bg, _occupied, and others) is documented as fresh per PROCESS, not per call – correct for every real, existing CLI tool (one call, one process), but genuinely wrong the moment a test script calls render() more than once in the same process. breadboard_lib.reset_module_state() exists specifically for this; a real, confirmed symptom of forgetting it is a spurious "label overlap" or "hole collision" between two components from two entirely DIFFERENT, unrelated circuits, each otherwise clean on its own.

This has recurred more than once in this project's own real test code, not just once (pptest_regression's own render sweep; separately, a new self-verification test built the same session) – worth checking specifically, by name, any time a new test calls render() or layout() more than once without an intervening process boundary.

A real, intentional improvement to placement can legitimately change which components end up unplaced, or which specific circuit/config combinations pass their own bounds check – this is not automatically a regression. pptest_regression's own EXPECTED_RESULTS table is deliberately a set of absolute, hand-confirmed values, not a saved snapshot, for exactly this reason – update it directly when a real, verified change alters an outcome, the same way any other real ground truth in this project gets updated after a verified, intentional change.

Never trust a single run when re-deriving these values: run the full sweep three separate times and confirm byte-for-byte stability before updating anything – this project's own established discipline throughout, not a one-off precaution. When investigating a value that changed for the worse alongside others that improved, look into it directly rather than assume it's an acceptable trade-off – a real, worked example: two circuit/config combinations flipped from passing to failing alongside a real sizing fix; direct investigation traced both to the SAME, already- documented, deliberate design choice elsewhere in the code (wire_row()'s own honest-overflow-over-silent-relocation behavior), not a new defect the fix introduced.

A fix that's correct for one case can be wrong if applied broadly without checking each instance it would also affect. A real, worked example: a field report suggested using min(courtyard_width, courtyard_height) for every component category, based on two categories (resistor, diode) where this was genuinely correct. Before applying it everywhere, every OTHER category's own real courtyard dimensions were measured directly – three had no real, confirmed alternate orientation at all, and one (a TO-220 package) had its own real "alternate" measuring LARGER, not smaller, than the current default. Applying the general rule anyway would have been an unverified guess dressed up as a principled fix, for categories it was never actually confirmed to help.

str_replace-style edits to ResearchJournal.org and FutureResearch.org, when the matched text is itself a section heading, silently drop that heading if the replacement doesn't explicitly reinclude it – confirmed, repeatedly, to be an easy mistake to make, not a rare one. After every edit to either document, grep for the expected heading pattern directly (for example, grep -n "^\* 2026-09" ResearchJournal.org) and confirm it's actually present before moving on – do not assume a successful edit preserved surrounding structure.

A change confirmed correct in the working tree can still fail once packaged, if a needed file was never added to EXTRA_DIST~/ ~ORG_DOCS~/~MAN_PAGES, or if a stale, auto-generated file (a .kicad_prl, an ERC report) was accidentally packaged alongside real, intended content. Before considering any delivery complete: extract the real, built archive into a genuinely separate directory, run ./bootstrap, ./configure, and make check there, from scratch – not in the working tree – and confirm the specific fix under discussion is directly, empirically observable from that pristine copy, not merely assumed present because it's present in the working tree.

make check is the sum of thirteen phases, check1 to check13, each of which can be run alone: make check7 runs the seventh and nothing else. On an ordinary machine make check runs them all, in order, as before. When a run is interrupted (a session that ends, a machine that goes to sleep), only the phase that was running has to be run again.

check1

the netlist loader's own suite (pptest_netlist) and the breadboard configuration's (pptest_breadboard_config, without its three extreme-scale checks, which run by hand with --extreme; it was left out of make check from 2026-09-15 to 2026-10-11 and check_test_phases now guards it).

check2

generates the two example circuits and confirms that each has no ERC error. The phases that need the examples depend on it, so make check5 alone generates them first.

check3

the build-instructions suite (pptest_build_instructions).

check4 to check13

the ten slices of pptest_regression, which runs the broad regression sweep, one check function after another. Slice K is pptest_regression --phases 10 --phase K.

The slices are contiguous stretches of the registry of check functions (CHECK_NAMES, in run order), cut so that each takes about the same time: CHECK_WEIGHTS holds the seconds each check took in a measured run, and only their relative sizes matter. Useful options of pptest_regression:

--list

prints the checks that would run, with their weights, and exits; with --phases N --phase K it shows one slice.

--only A,B

runs just these check functions, by name; the quickest way to repeat one check while working on it.

--timings

prints the seconds each check took; the source of CHECK_WEIGHTS. After adding checks, or if the slices become uneven, run the whole suite once with it and update the weights.

A new check function must be added to CHECK_NAMES, or it never runs: a test fails on any check function that is not in the registry. Changing the number of slices means changing the --phases number and the list of checkK targets in Makefile.am together; a test fails if a slice is left out.