Monday, January 26, 2015

Linux Kernel Debugging with VMware and GDB

Intro

There are many different ways you set up debugging the Linux kernel across platforms and using physical or virtual machines, using internal options such as KDB or KGDB, which are interfaces to the kernel's debug core, are the most notable couple. I'm going to show you how to utilize impressive visualization features with VMware to set up debugging for a Linux guest VM on a Mac OS X host machine using VMware and GDB. It's fairly straightforward, so if you're just starting out in kernel land, this is a chance for you to embrace a modern approach to Linux kernel debugging.

Environment

Host: Mac OS X Yosemite x64, VMware Fusion

Guest: Ubuntu Linux 14.04 x86

Note: Most of this should also be applicable to Windows host systems running VMware Workstation.

Step 1: Configure your host

Download the iso file for your target guest machine

Create a new virtual machine and install the guest OS

Shutdown the virtual machine and edit it's VMX file (if it's not a separate file, you can right-click the VM file and"Show Package Contents".

Add the following line at the end of the file:

debugStub.listen.guest32 = 1

Save and close the VMX file and boot your guest machine.

Step 2: Configure your guest

Make sure the system is completely up to date, eg. sudo apt-get update && sudo apt-get upgrade and reboot once complete.

Then open up a terminal and do the following:

codename=$(lsb_release -c | awk  '{print $2}')

sudo tee /etc/apt/sources.list.d/ddebs.list << EOF
deb http://ddebs.ubuntu.com/ ${codename} main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-security main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-proposed main restricted universe multiverse
EOF

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys
ECDCAD72428D7C01
sudo apt-get install linux-image-`uname -r`-dbgsym

This will update your apt sources in order to be able to pull down dbgsym for your current running kernel. We'll use these symbols later to make debugging much more effective.

Now you should see a new file was dropped:

/usr/lib/debug/boot/vmlinux-<kernel version>-generic

Copy this file to your host machine as we'll use it for symbols in GDB soon.

Step 3: Build and Connect GDB

The VMware debugging service for the guest VM is listening via host loopback on port 8832. We can use GDB to connect to the service and debug our virtual machine.

But first, we have do a custom build which supports ELF binaries in Terminal:

(If you don't have wget, install MacPorts)
wget http://ftp.gnu.org/gnu/gdb/gdb-7.8.tar.gz
tar xf gdb-7.8.tar.gz
cd gdb-7.8
./configure --build=x86_64-apple-darwin14.0.0 --target=x86_64-vfs-linux --with-python && make
make install
 
(gdb) target remote :8832
Remote debugging using :8832 
(gdb) symbol-file vmlinux-<kernel version>-generic
Reading symbols... done

Now you should be able to see details in the stack trace (bt) and if you encounter a kernel panic, you'll drop into the debugger!

Start Debugging

If you hit ctrl+c during boot, for example, then you can verify it's working.

^C
Program received signal SIGINT, Interrupt.
0xc1147b1c in copy_page (from=<optimized out>, to=0xfffb9000)
    at /build/buildd/linux-3.13.0/arch/x86/include/asm/page_32.h:47
47    /build/buildd/linux-3.13.0/arch/x86/include/asm/page_32.h: No such file or directory.
(gdb) i r
eax            0xfffb9000    -290816
ecx            0x400    1024
edx            0x1000    4096
ebx            0x1e8    488
esp            0xedb4be48    0xedb4be48
ebp            0xedb4be68    0xedb4be68
esi            0xecbe8000    -323059712
edi            0xfffb9000    -290816
eip            0xc1147b1c    0xc1147b1c <copy_user_huge_page+76>
eflags         0x210246    [ PF ZF IF RF ID ]
cs             0x60    96
ss             0x68    104
ds             0x7b    123
es             0x7b    123
fs             0xd8    216
gs             0xe0    224
(gdb) c
Continuing.

You can also intentionally trigger a kernel panic via the Magic SysRq sub-system. Using a root shell (sudo bash, not sudo this-cmd) on the guest.

echo c > /proc/sysrq-trigger

Misc

Looks like there's guide to even use IDA's debugger for kernel remotes.

References

http://askubuntu.com/questions/197016/how-to-install-a-package-that-contains-ubuntu-kernel-debug-symbols
https://lists.ubuntu.com/archives/kernel-team/2014-July/045843.html
http://stackframe.blogspot.com/2007/04/debugging-linux-kernels-with.html
http://ntraft.com/installing-gdb-on-os-x-mavericks/
http://webcache.googleusercontent.com/search?q=cache:HSCTIWJMsXQJ:xulei.me/blog/2012/03/13/set-up-vmware-fusion-for-linux-kernel/+&cd=1&hl=en&ct=clnk&gl=us