Fear of Flashing: Building confidence in firmware through automated testing

While the phrase "If your system is working, do not update it" may be waning, it still pops up in firmware release notes. It represents the risk of various failures: e.g. bricking through a misapplied or interrupted update, or by regressions in the new firmware.
Testing using the same initial hardware and software present on customer systems and in manufacturing is the surest way to improve confidence in update and recovery procedures as well as other expected behaviors. This talk describes how our developer-first tools, control plane emulations, and fault insertion are used to test our Service Processor and Root of Trust firmware prior to full-stack integration testing.

Oxide Computer's systems incorporate a purpose-built pairing of a Service
Processor (STM32H753) and Root of Trust (LPC55S69).

As noted in previous talks by Oxide staff, the SP and RoT run the
open-source Hubris OS and are used in place of the usual BMC on the
compute sleds. They provide similar boot sequencing,
update, control, and telemetry services to our power shelf controllers
and network switches.

By design, there are no call-home functions in our products.
Unless we are invited into a customer's data center, the products we ship are
beyond our reach when handed off at the customer's loading dock. As we
scale up production, those invitations are expected to become more rare
and, ultimately, completely unnecessary.

Firmware updates have to be safe, predictable, and genuinely improving.
We build that confidence before a release ships, not at the customer's
site, by testing against the same hardware and firmware our customers and
our manufacturing line run.

This problem is not unique to us: any team shipping firmware to hardware
it can't easily reach has the same need for confidence before release.

This talk is about how we scale up that testing, with a stand-alone,
developer-first tool, sp-test, designed from the start to also run
unchanged in our hardware continuous-integration (HWCI) environment.

Most firmware testing starts as a one-off: a developer who already
knows their setup runs a command against the image they just built. The
sp-test tool's job is to turn that throwaway check into something
worth keeping, a test another developer can rerun to get the same
results. The test is complete enough to hand to our HWCI system as a
regression or other check. Making a personal test portable is where the
sp-test machinery comes in: pinning the released image to test against,
declaring the hardware and configuration a test needs so it is skipped
where it doesn't apply, and packaging results as a report someone else
can analyze or reproduce. A developer testing alone barely notices any
of it; it earns its keep when the test travels. The payoff is that the
same test, simple or complex, runs unchanged, whether on demand or in
response to a pull request in HWCI. A developer can put a job on scarce,
shared hardware and get back the same actionable results CI produces.

The tests worth sharing are often the hardest to run:

• the update path exercised the exact way our control plane drives it
• the code that runs only when something has already gone wrong
• failures provoked on purpose
• checks that debug interfaces never ship in production
• behavior that only real hardware reveals

The patterns are not specific to Hubris, or to Oxide: any team shipping
firmware could build the same kind of developer-first, shareable testing.
A live demo on an off-the-shelf ST Nucleo board makes it concrete:
hardware anyone can buy and try.