The point of this “Learning Linux” blog is to discuss learning more about Linux. In the next several blogs, at least, leading up to the training at the Collaboration Summit, we will look at topics that will be addressed in one of the training classes. To whet your appetite.

In this blog let’s look at something simple, yet apparently, not used nearly enough. Let’s look at configuring the Linux kernel with various options for debugging.

Linux kernel configuration is achieved by assigning values to macros These values are commonly just ‘y’ or ‘m’, for “yes” or “module”. The yes generally means to link that feature into the kernel image. The module, of course means to compile and link that option as a loadable kernel module.

The “make xconfig” configuration tool has a nice way to search for configuration macros and to see the names. Choose Option ? Show Name to see them. Use Edit ? Find to search for them. Below are a couple screen shots so you can better remember.

Note the faint pink ovals highlighting what to look forward. Simple things, but, sometimes unless someone points things like this out you never think to look. At least that is the way it often is for me.

Some kernel debugging features can be modules, some must be statically linked.

CONFIGURATION VARIABLES:  The configuration variables are stored in the .config file in the top level kernel source directory. If we do “grep DEBUG .config | wc - l” for 2.6.28 we see there are 133 configuration variables that have DEBUG in their name. Wow. Some of these are quite specialized, for example, FUJITSU_LAPTOP_DEBUG enables extra debug output from the fujitsu extras driver.

Many of the others, while potentially applicable to a wide range of developers, are appropriate only in those circumstances where the debug feature applies to the suspected bug. For example, DEBUG_SPINLOCK_SLEEP.

Let’s look at the use of the macro DEBUG_SPINLOCK_SLEEP in the kernel build and source. If we search for that name in the source tree with “grep -rl DEBUG_SPINLOCK_SLEEP | wc -l” we get a count of 204 occurrences. Naturally we won’t look at them all.

If we remove all of the arch sub-directories except for x86 there are only 9 references. It turns out that the vast majority of the references were actually kernel configuration files for non-x86 targets.

If we eliminate the Documentation files and x86 default configs we are only left with include/linux/kernel.h, kernel/sched.c and the file for “make xconfig” and other configuration tools: lib/Kconfig.debug.

Let’s look at each of these. In the Kconfig.debug file we see this entry:

• config DEBUG_SPINLOCK_SLEEP
• bool “Spinlock debugging: sleep-inside-spinlock checking”
• depends on DEBUG_KERNEL
• help

If you say Y here, various routines which may sleep will become very noisy if they are called with a spinlock held. We see that this option cannot be a module. That is the “bool” choice. We also see that one must select DEBUG_KERNEL in order to be able to select this option. The help documentation is quite brief, but it sounds like it could be quite helpful if we call a function that could sleep while holding a spinlock. Such a thing may lead to a deadlock. Bummer.

If we examine include/linux/kernel.h we see:

#ifdef CONFIG_DEBUG_SPINLOCK_SLEEP
void __might_sleep(char *file, int line);
/**
* might_sleep - annotation for functions that can sleep
*
* this macro will print a stack trace if it is executed in an atomic
* context (spinlock, irq-handler, …).
*
* This is a useful debugging help to be able to catch problems early and not
* be bitten later when the calling function happens to sleep when it is not
* supposed to.
*/
# define might_sleep() \
do { __might_sleep(__FILE__, __LINE__); might_resched(); } while (0)
#else
# define might_sleep() do { might_resched(); } while (0)
#endif

We see that the configuration macro is used to determine how the macro might_sleep() is defined. While we are looking at that, let’s more closely at __might_sleep().

We find __might_sleep() in kernel/sched.c. The function prints the following kind of information after checking, among other things, if it is being called in atomic context.

printk(KERN_ERR “BUG: sleeping function called from invalid context at %s:%d\n”, file, line);
printk(KERN_ERR “in_atomic(): %d, irqs_disabled(): %d, pid: %d, name: %s\n”, in_atomic(), irqs_disabled(), current->pid, current->comm);
debug_show_held_locks(current);
if (irqs_disabled())
print_irqtrace_events(current);
dump_stack();

SUMMARY: The Linux kernel has quite a number of configuration options to help with debugging. It can be valuable to examine the source to see their affect. The source code is also a truly reliable reference.  (Source: cpanelim)