Tuesday, November 17, 2015

Why Security

How did you get into (computer) security?

I bet I've been asked this question a million times by curious acquaintances.

Or asked even more personally by non-techie family or friends: How do you learn to do stuff like that?

But those questions are easy to answer-- Simply replay the fond circumstances where you needed to be creative from memory. The more interesting question to me is this:

What do I enjoy about security?

But we're not going to go deep into that one either as I'd like to talk about my thoughts on a sub-field of security. To me, this area is the most fun and provides the platform for curiosity to overcome a thrill seeker: Bug hunting.

What is bug hunting? This is commonly described as actively looking at code for bugs. That code can be programming code or assembly or something else entirely.

For example, let's say I want to find a way to gain access to a server. The server is serving web pages with Apache web server. I know that Apache is open source, meaning anyone is free to download, read and modify their own copy of the software. If one can read and understand the programming language Apache is written in, one may start their bug hunt there.

Source code is in hand, next is identifying attack surface-- the difference places the web server parses or can be influenced by untrusted input, and so on and so on. The point is that there is a process to be followed here and many more than one. It's entirely up to the person trying to find the bug how they choose to approach it. But enough about what it is, let's ask an important question:

Why

Now why would anyone want to do it such a thing?

If you break it down to it's lowest practical level, bug hunting is the art of finding mistakes or invalidating assumptions-- more often than not, those that someone else made in code that a computer runs. Obviously everyone makes mistakes, not matter how smart they are and some of those mistakes allow for exploitation. Someone skilled enough at the keyboard can take that mistake, tie a ribbon on it and send it along like nothing happened. But something did happen and the program has a new feature: do whatever I tell you to do, which is often not what they were originally programmed to do.

CPUs, for all intents and purposes, have no distinction between right and wrong; they simply run the instructions given to them. The trick here is that mistakes in the code combined with how the CPU architectures work allow for persons (or robots, whatever) who did not write the original code, to modify it, the order in which it’s executed, or the resources it takes dependency on and change the expected outcome that the author intended.

Now, one might think, “How could anyone enjoy finding people's flaws and taking advantage of them?”. Well, now you've veered off on the wrong track. Let's get you back focused.

Bug hunters aren't finding flaws and using them against those developers in real life. Sure, programming is such a creative immersion that some might say there's a strong connection between a developer and their code. Yes, developers are often insulted when security folks find bugs in their code. But that's a whole different road to go down.

Philosophy (because I refuse to name this section “Ethics”)

Back to the question. If this work makes the people who do it seem they have debilitating personalities. Perhaps if you generalize it, they become thought of as overly-critical, arrogant or just out to make people look bad. I'm sure there are some exactly like that, but most don't fit that profile at all.

You see, the primary source of joy, excitement and achievement from finding bugs isn't about proving someone wrong or showing the world how dumb they are. It's about much more pleasant ideas.

One, the intellectual brain exercise of critical thinking.

Two, the paintbrush at your fingertips. A bug is an opportunity to create. Knowing about a bug doesn't mean you've exploited it-- once you understand it, you can write a program to exploit the program which contains the bug. And you can write the exploit any way you see fit. Really, you can write anything you want as long as it achieves the purpose. Now that's freedom.

Three, you can help fix the bug. Now, first and foremost, the bug at this point belongs to you. Sure, you can be almost certain someone somewhere else on Earth knows about it too, just because the math works out in 99% of the cases. But you are the one who decides what to do with the bug and/or exploit.

You could make the developers (or their company) aware of the bug.

You could make the community aware of the bug.

You could sell it; there's plenty of legitimate parties who perhaps have net-positive motives willing to purchase the rights to your research.

You could simply have no further ambition but sharing it with friends or keeping the details to yourself.

You could even combine two or more of the above in almost any order.

Some people believe you should not give away “work” to greedy corporations. Some think if you don't make the vendor aware of the bug, you're being irresponsible as now the wide user-base is not protected against others who may find it and exploit it maliciously. Some are still part of a scene where bugs have no monetary value. It's pointless to argue which way is the best as there's no right or wrong answer, it's completely subjective and it's logical for there to be several different values held.

Career-wise

There's so much ambiguity and context necessary when you talk about product security. “Code review” is a common way for developers to get feedback on eg. A new feature they've written by fellow colleagues. But when talking about security, a code review means reading the code to find potential security bugs.

Those who have this very mindset, which by this point you can understand is not learned, but developed (typically solo), often have an interesting challenge fitting into the team and organization they become a part of. They typically aren't traditional developers and aren't interested in learning more than they need to in order to find classes of bugs they're interested in or developing tools to find those bugs for them. They also are not traditional auditors, whose very existence is predicated on the idea that people will always make mistakes and there's no practical, non-human way to find and file them. Security folks are around, and often no longer around, due to this fact as well. They may also not come from a formal security background in academia, one exit off the Saltzer and Schroeder expressway, so the research role (as it's understood by the company) may not fit exactly either.

One may find themselves in the position to seek a well-paid job with a respectable company where they need to prove they can not only meet the bar with their development skills, but do that security thing the company hired them for exceptionally well (much better than the developers already employed there) to justify the hire.

Thoughts

10-15 years ago, the world had yet to experience the era of criminals proliferating the Internet. The hacking scene was a different groups of friends who enjoyed exchanging information and (mostly) harmlessly conquering the Internet. Now it's common for those veterans and newbies alike who have to deal with security, and all the interesting personalities that come with it, to assume that the criminals who steal, scam and cause real damage to people and businesses are “hackers”. Sure, some of them might hack. But there's still the ones who knew what the word stood for before. And there's those before us who knew it as something even different.

Are bug hunter's, hackers? Are criminals, hackers? Are employees at those HackFests, hackers (NO, and they’re insulting to real hackers who come across them)? Can anyone be a hacker once the marketing industry takes it and starts to ruin another cool word? Is the Wassenaar Arrangement the beginning of the end of complete freedom in bug hunting as we know it?

I'm not sure and nobody can be sure of anything, really. But that's what I believe, and now, you may think differently, too. Conventional wisdom is just that-- by convention, and it has it’s place. That doesn’t mean it’s the truth, though. And asking someone how they got into security can take them down the rabbit hole of creativity, self-dependence and fond memories that have yet to (and hopefully never will) completely fade away.

Monday, January 26, 2015

Linux Kernel Debugging with VMware and GDB

Intro

There are many different ways you set up debugging the Linux kernel across platforms and using physical or virtual machines, using internal options such as KDB or KGDB, which are interfaces to the kernel's debug core, are the most notable couple. I'm going to show you how to utilize impressive visualization features with VMware to set up debugging for a Linux guest VM on a Mac OS X host machine using VMware and GDB. It's fairly straightforward, so if you're just starting out in kernel land, this is a chance for you to embrace a modern approach to Linux kernel debugging.

Environment

Host: Mac OS X Yosemite x64, VMware Fusion

Guest: Ubuntu Linux 14.04 x86

Note: Most of this should also be applicable to Windows host systems running VMware Workstation.

Step 1: Configure your host

Download the iso file for your target guest machine

Create a new virtual machine and install the guest OS

Shutdown the virtual machine and edit it's VMX file (if it's not a separate file, you can right-click the VM file and"Show Package Contents".

Add the following line at the end of the file:

debugStub.listen.guest32 = 1

Save and close the VMX file and boot your guest machine.

Step 2: Configure your guest

Make sure the system is completely up to date, eg. sudo apt-get update && sudo apt-get upgrade and reboot once complete.

Then open up a terminal and do the following:

codename=$(lsb_release -c | awk  '{print $2}')

sudo tee /etc/apt/sources.list.d/ddebs.list << EOF
deb http://ddebs.ubuntu.com/ ${codename} main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-security main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-proposed main restricted universe multiverse
EOF

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys
ECDCAD72428D7C01
sudo apt-get install linux-image-`uname -r`-dbgsym

This will update your apt sources in order to be able to pull down dbgsym for your current running kernel. We'll use these symbols later to make debugging much more effective.

Now you should see a new file was dropped:

/usr/lib/debug/boot/vmlinux-<kernel version>-generic

Copy this file to your host machine as we'll use it for symbols in GDB soon.

Step 3: Build and Connect GDB

The VMware debugging service for the guest VM is listening via host loopback on port 8832. We can use GDB to connect to the service and debug our virtual machine.

But first, we have do a custom build which supports ELF binaries in Terminal:

(If you don't have wget, install MacPorts)
wget http://ftp.gnu.org/gnu/gdb/gdb-7.8.tar.gz
tar xf gdb-7.8.tar.gz
cd gdb-7.8
./configure --build=x86_64-apple-darwin14.0.0 --target=x86_64-vfs-linux --with-python && make
make install
 
(gdb) target remote :8832
Remote debugging using :8832 
(gdb) symbol-file vmlinux-<kernel version>-generic
Reading symbols... done

Now you should be able to see details in the stack trace (bt) and if you encounter a kernel panic, you'll drop into the debugger!

Start Debugging

If you hit ctrl+c during boot, for example, then you can verify it's working.

^C
Program received signal SIGINT, Interrupt.
0xc1147b1c in copy_page (from=<optimized out>, to=0xfffb9000)
    at /build/buildd/linux-3.13.0/arch/x86/include/asm/page_32.h:47
47    /build/buildd/linux-3.13.0/arch/x86/include/asm/page_32.h: No such file or directory.
(gdb) i r
eax            0xfffb9000    -290816
ecx            0x400    1024
edx            0x1000    4096
ebx            0x1e8    488
esp            0xedb4be48    0xedb4be48
ebp            0xedb4be68    0xedb4be68
esi            0xecbe8000    -323059712
edi            0xfffb9000    -290816
eip            0xc1147b1c    0xc1147b1c <copy_user_huge_page+76>
eflags         0x210246    [ PF ZF IF RF ID ]
cs             0x60    96
ss             0x68    104
ds             0x7b    123
es             0x7b    123
fs             0xd8    216
gs             0xe0    224
(gdb) c
Continuing.

You can also intentionally trigger a kernel panic via the Magic SysRq sub-system. Using a root shell (sudo bash, not sudo this-cmd) on the guest.

echo c > /proc/sysrq-trigger

Misc

Looks like there's guide to even use IDA's debugger for kernel remotes.

References

http://askubuntu.com/questions/197016/how-to-install-a-package-that-contains-ubuntu-kernel-debug-symbols
https://lists.ubuntu.com/archives/kernel-team/2014-July/045843.html
http://stackframe.blogspot.com/2007/04/debugging-linux-kernels-with.html
http://ntraft.com/installing-gdb-on-os-x-mavericks/
http://webcache.googleusercontent.com/search?q=cache:HSCTIWJMsXQJ:xulei.me/blog/2012/03/13/set-up-vmware-fusion-for-linux-kernel/+&cd=1&hl=en&ct=clnk&gl=us