Matt Keeter
Matt Keeter is an embedded software engineer at Oxide Computer Company, where he works on bare-metal firmware, debug tools, and system integration. He previously worked at Formlabs, a 3D printing startup, in a variety of roles – from desktop software to electrical engineering to tech lead – as the company grew from 12 to 600 people. His work has been featured on the Sparkfun and Hackaday blogs, and published in ACM Transactions on Graphics. He is based in Cambridge, MA; when not wrangling embedded systems, he enjoys rock climbing and writing weird GPU code.
Unplugging the Debugger: Live and post-mortem debugging in a remote system
When building an embedded system, there comes a time when you must unplug your debugger and seal the device in a box. Unfortunately, things can still go wrong after this point! The Hubris embedded OS has put a strong emphasis on debuggability, even to the point of writing Humility, an in-house debugger. However, Humility has historically assumed a physical connection to a running microcontroller. As we begin sealing up boxes, we’ve had to develop a range of tricks for mostly-seamless debugging of a network-connected system. Strategies range from forging inter-task IPC messages and calling arbitrary functions over the network, to (safely) reading and decoding memory from the running system, and even using a secondary microcontroller to take a full memory image of the main system for post-mortem debugging!