Unplugging the Debugger: Live and post-mortem debugging in a remote system
Main Track,
When building an embedded system, there comes a time when you must unplug your debugger and seal the device in a box. Unfortunately, things can still go wrong after this point! The Hubris embedded OS has put a strong emphasis on debuggability, even to the point of writing Humility, an in-house debugger. However, Humility has historically assumed a physical connection to a running microcontroller. As we begin sealing up boxes, we’ve had to develop a range of tricks for mostly-seamless debugging of a network-connected system. Strategies range from forging inter-task IPC messages and calling arbitrary functions over the network, to (safely) reading and decoding memory from the running system, and even using a secondary microcontroller to take a full memory image of the main system for post-mortem debugging!