Ben Stoltz
Ben Stoltz is an engineer at Oxide Computer Company, where he works on
service processor and root of trust firmware: its security, its update
path, and, most recently, the testing infrastructure this talk covers.
Before Oxide, he worked on systems and infrastructure at Sun Microsystems,
Cisco, and Google.
Fear of Flashing: Building confidence in firmware through automated testing
While the phrase "If your system is working, do not update it" may be waning, it still pops up in firmware release notes. It represents the risk of various failures: e.g. bricking through a misapplied or interrupted update, or by regressions in the new firmware.
Testing using the same initial hardware and software present on customer systems and in manufacturing is the surest way to improve confidence in update and recovery procedures as well as other expected behaviors. This talk describes how our developer-first tools, control plane emulations, and fault insertion are used to test our Service Processor and Root of Trust firmware prior to full-stack integration testing.