Embedded Debugging: Methodologies, Tips, and Tricks
As embedded software developers, we often face complex bugs that can be challenging to reproduce and resolve.
In this post, I'll go over some of my methodologies, tips, and tricks in the hope to help you become a more efficient embedded debugger.
This is not a step by step guide, but a list of additional tools you can use.
Recreate the Bug.
The first and most crucial step in debugging is to recreate the bug exactly as it was documented.
Bugs that cannot be recreated are rarely solved.
This will both help you to understand the bug better and it will be used as a control test.Run Control Tests.
Before every debug session and after each major change, recreate the bug and make sure it still behaves as expected.
I can't count how many hours I've wasted for not noticing that I compiled the wrong version, used a different hardware, or not noticing that something small and silly has changed.Non-Consistent Bugs.
Non-consistent bugs are bugs that do not occur on every run, bugs that do not occur when you want them to, bugs that occur when they they want to.
The key to solving these types of bugs is to profile the probability of the bug to occur.
In other words, determine the number of times (or duration) the test needs to be run in order for the bug to occur at least once.
If you make a fix, and run the test this amount of times without the bug occurring, you can consider the bug resolved.
For long tests, try to automate this process whenever possible.Define Success Criteria.
When designing or running a test, make sure you understand what result is considered a success.
Most of the times this is trivial, but sometimes it is not.
This is especially important for long tests, where a small mistake in the test can make you rerun the test all over again and waste valuable time.Debug One Bug at a Time.
Each test you run should focus on a single bug or change.
Not following this rule may make it harder to identify which behavior is caused by which root cause, and cause the different changes you make interfere with each other.Prioritize Bugs.
When there are several bugs to solve, start debugging them in the chronological order they occur on runtime.
Solving the bug that occurs first may affect or even fix the bugs that occur after it.Standalone Tests.
When debugging a low-level entity like a UART device, an external library, or even a module you wrote, it's sometimes worth running a standalone test for it.
Meaning, start a new empty project in the IDE, import into it only the most crucial initializations, and write the code to test the HW/module right there in function main without the OS and all the noise of the application.
Do note that this can take some time, but for some cases it can be a good way to isolate the issue.
I use this method when I suspect there is an issue in a third party library or HW.Visual Code Inspection.
Reading the code visually and looking for coding mistakes is an obvious debug method.
However, here are a few additional pointers to pay attention to, especially in non-consistent bugs:Whenever you see a buffer, both local or global, ask yourself what variables follow this buffer in memory. In other words, what variables will be overrun if that buffer overflows (you can use the map file for this)?
Whenever you see an asynchronous (extern, private, or static) variable, ask yourself what value, if that variable holds, will cause the bug to occur?
When seeing uninitialized variables, make sure they are not being used before receiving a value.Breakpoints.
Breakpoints is another obvious debug method.
The only tip here is that the amount of breakpoints we can use is limited by the amount of breakpoints the hardware support.
If you need more breakpoints, you can either use a breakpoint assembly command (in ARM, it's asm BKPT)
Or you can create a function, set a breakpoint inside, and call that function from wherever you want to set the breakpoint at.
Obviously, this requires recompilation of the code.Variables Optimizations.
When pausing the run with the debugger and looking at variables, sometimes, you'll notice that the debugger cannot show the value of the variable. This usually occur due to the compiler's optimizations.
Since changing the optimization can sometimes alter the behavior of the bug, as a workaround , you can turn the variable you want to debug into a global volatile variable. This will prevent the optimization of the variable and you'll be able view it with the debugger.
just don't forget to clean up after yourself when done.Use Git as a Debug Tool.
Git can be used as a debug tool.
How many times have you pulled origin/master only to discover that it's broken? One way to debug it is to go back and test previous commits for the bug.
git has a helpful command for this: "git bisect" look it up.Commit Changes During Debug Sessions.
During a debug session, make a commit for every change you make.
Those are only temporary commits, they do not have to be descriptive or clean commits, don't be afraid to stuff them with all the debug prints global variables and other mayhem you inserted for debugging.
This way, whenever you want to go back to something you did previously, you can do it easily.Comment Out Code Sections.
For some bugs, specifically asynchronous bugs or memory issues, it's a good method to try commenting out sections of code and see when the bug ceases to occur.
This can help you isolate the problematic area and narrow down the root cause.Use global variables to view data without effecting runtime
Remember that prints and breakpoints may alter the behavior of the program.
This may cause the bug to occur differently, or not at all.
When you think that this might be the case, store in a global variable whatever data you want to see like parameters or make sure you reached a specific line of code (use the __LINE macro for this).
Then you can let the bug occur normally, without disruptions and either view/print those variables later.
I am sure that you, Ok, maybe not you, since you never write any bugs, but every other developer out there, has his own set of shenanigans he does during debugging, so why won't you write them up in the comments so we'll all become a bit wiser
#embedded #software #developer