Debugging small OS kernels (part 4)

network3-visop

network6

(Continued from part 3)

Clicking the step button from the Emacs menu drives us deeper into the network code:

Moving back to the Debian host, I step through the scheduler code.

scheduler3

In the graphic, (above) – the stepping cursor is not visible because I moved the scroll-bar on the source window in order to make the top of the function visible.  One really needs high monitor resolution here to see the whole series of functions at once, to keep in one’s head what’s happening.  I’ll see what I can do about that problem in future screen-shots.

Using Anjuta to debug the kernel

anjuta-kernel

For people who really like graphical IDE environments, it turns out that Anjuta can be used very easily to debug the Visopsys kernel. Just import the source tree as a makefile project, select remote debugging from the menu, and add the kernel as the “program” to debug. Then add source paths (I just pointed directly to the kernel source directory). Also, run a gdb command from the “GDB command” menu item in the debug menu, and add the symbol file. Add the entries for the target (localhost) and port (1234). Click “connect.” Very simple!

Anjuta might do a better job of keeping the source line cursor in sync with  execution, but Emacs ain’t bad either – IMO.  This sort of debugging environment is a great tool to get acquainted with a kernel – and to debug it on (almost) the same terms as a regular application.  It’s been seven or eight years since this level of functionality has been available in open source tools, but surprisingly few people make use of it.

Take a spin with the kernel, and just step it through its paces.  You’ll see where it goes, and about how often it goes there.  You’ll start to understand the basic “loop” of the kernel, and how it works its magic.  That “stuff down below” becomes less mysterious after this sort of enlightenment.  I watched the “loop” for awhile, as it ran.  I looked for interrupts, as that was the main reason I set up the environment in the first place (some people thought that perhaps interrupts were getting turned off, or otherwise going astray.  That idea may not to be the issue after some investigation.  The devs are working on it.)

So, just to name a few of the functions in the normal daily chore list of the kernel, observed by following breakpoints with single stepping commands:

scheduler(), markTaskBusy(), kernelMalloc(), spawnIdleThread(), kernelSysTimerSetupTimer(), chooseNextProcess(), kernelDebugOutput(),kernelInterruptHook(), kernelInitialize(), requestProcess(), kernelProcessingInterrupt(), kernelProcessorRestoreInts(), kernelLock(), kernelMemSet(), driverDetectBiosPnp(),  etc, etc.

When dealing with an arcane sort of problem like “interrupts” – its hard to just single step directly to the problem until you can do something to narrow the list of places you should look.  So, one needs to have a pretty good understanding of how the particular kernel in the experiment works.  Single stepping, and just following “where ever it goes” – is a good first milestone on the mission.  Another thing that could speed up the learning process is profiling.  If the kernel is recompiled with the profiling option (-pg) – then the gprof tool can be used to get an even better picture of where the kernel is spending its time.

The reason for this is simple. The problem (whether it’s interrupts or anything else) – most likely will arise under a specific set of circumstances, which (most likely) don’t exist the first hundreds of times you single step into the procedures (or thousands, or millions). The processor executes a whopping big number of functions every second. So, – narrowing the list is very important. The bad news is that fancy debuggers don’t solve the problem, but they can help if you know where to apply them. Once we understand the kernel basics, then we can use some educated guessing to place strategic breakpoints and “watch-conditions.” That approach can solve the problem!

From there on, it’s mainly a matter of repetition, but there’s less of it if the developer or experimenter debugs smartly with the utilization of this very nice set of tools.

Go to the next segment in the article

Note: Visopsys is a project of, and owned by J. Andrew McLaughlin, at http://www.visopsys.org. These pages are not affiliated with the author or website.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.