vi and the Kinesis Advantage Pro keyboard

I use a Kinesis Advantage Pro keyboard wherever possible because I really enjoy the way that it feels to type on, and I believe that it's important to look after your hands.  I'm principally a Unix/Linux user, and use vi as my primary editor.  In addition, other software that I use regularly such as Google mail (the gmail web interface) uses vi keystrokes (specifically j and k) for navigating.  This has lead me to make a couple of customizations to the key layout on all my keyboards.

Customizing the Kinesis keyboard is much easier than creating a custom keymap on Linux or Windows.  The keys are remapped in the keyboard itself, which means that in the event somebody needs to use your computer, they can plug in a regular USB keyboard and it will function as they expect.

Firstly, I swap the Escape and Delete keys.  The Escape key is used heavily with vi, but is a tiny rubber key which is prone to being ignored when pressed by your pinkie.  In contrast, the Delete key is rarely used in Unix, and is a huge key with a proper switch operated by your left thumb.  Swapping the role of these two causes no loss of functionality, but makes using vi much easier as you no longer have to take your left hand off the keyboard to press Esc after every edit.

The second swap is the up and down arrow keys.  The reason for this is a little more subtle.  By default, the 'Up' arrow key is in line with the 'j' key, which in vi is used to move down.  The 'Down' arrow key is in line with the k key, which is used to move up.  Switching between vi and software that uses the arrow keys becomes very difficult, certainly for me.  By swapping the arrow keys, the same fingers are used to move up and down in both vi and every other application, so you don't have to mentally remap the functions every time you switch application.

One site that I've found very useful for stuff like this is RSIguy. There's a bunch of info there regarding selecting the best ergonomic keyboards for your needs.

Linux KVM virtualization and PCI-DSS

With the release of RedHat 6, KVM is now the default virtualization solution in both the RHEL and Ubuntu worlds.  With KVM virtualization, the Linux kernel itself acts as a hypervisor to manage the host hardware, allocating the resources to the guest virtual machines.  This is quite different to VMware, where a small, custom written hypervisor manages the host machine hardware, and management software runs in a Linux-like environment on that hypervisor.

This move to using a general purpose OS as the hypervisor has some significant advantages, as the full capabilities of Linux (e.g. RAID, encrypted storage, vast hardware support) can be leveraged when building a solution.  Also, relative to VMware, there can be significant cost advantages to using Linux.

However, in a high security environment, moving to a general purpose OS as the hypervisor can introduce additional risks which need to be mitigated.  A custom written hypervisor like VMware is designed to do one thing: run VMs.  In principle, as long as secure account management policies are followed, patches are installed in a timely manner, and management access is restricted to secure hosts, then the hypervisor is likely to be 'secure'.  Host environment security is mostly a function of securing the guest virtual machines themselves.

With a Linux KVM hypervisor, the situation can be very different.  Modern Linux distributions provide all sorts of software that are invaluable when deployed appropriately, but which would be poor candidates for installation on a host intended to be a dedicated hypervisor.  In this environment, any unnecessary services are just potential vulnerabilities to be exploited in gaining unauthorized access to the host.  Once an intruder gains access to the hypervisor, there are many tools which can be used to extract information from a running VM without security tools running inside the guest being aware that anything is happening.

To illustrate this, I've created the following scenario:

1) a host running Ubuntu 11.10 as a KVM hypervisor called 'kvm-host'

2) a VM running Ubuntu 11.10 called 'iso8583', simulating a transaction processor

3) a VM running Ubuntu 11.10 called 'switch' that will connect to iso8583 and send messages

On iso8583, the role of the processing software is simulated by the 'echo' service in inetd.  This is essentially the most trivial network server imaginable: you create a TCP connection to the service, and any data that you send is echoed back to you.  The data is not logged or processed in any other way, just received by the server and echoed back.

For this example, I'm assuming that our processing BIN is 412356, so all PANs (card numbers) will be of the form '412356' + 10 more digits.

We start by connecting from switch to iso8583 and sending a couple of messages (just fake PANs, in this case).  The 'netcat' utility is used to connect to the remote service, a PAN is sent to the processor, which is then echoed back:

18:05:43 switch:~> echo 4123560123456789 | nc iso8583 echo

4123560123456789

18:05:47 switch:~> echo 4123569876543210 | nc iso8583 echo

4123569876543210

Now, on kvm-host (the hypervisor), we dump a copy of the full memory of the virtual machine using the useful gdb tool 'gcore'.  Note that gcore produces a core dump of any process (including a virtual machine), without actually terminating the process:

# Get the PID of the VM called iso8583

18:06:05 kvm-host:~> pgrep -f iso8583

18170

# Now get a copy of the in-memory process

18:06:09 kvm-host:~> sudo gcore 18170

[Thread debugging using libthread_db enabled]

[New Thread 0x7f5b8542e700 (LWP 18244)]

[New Thread 0x7f5b89436700 (LWP 18241)]

[New Thread 0x7f5b8ac39700 (LWP 18239)]

[New Thread 0x7f5b87c33700 (LWP 18238)]

[New Thread 0x7f5b89c37700 (LWP 18236)]

[New Thread 0x7f5b86430700 (LWP 18216)]

[New Thread 0x7f5b87432700 (LWP 18214)]

[New Thread 0x7f5b88c35700 (LWP 18205)]

[New Thread 0x7f5b9d9d4700 (LWP 18180)]

0x00007f5ba2213913 in select () from /lib/x86_64-linux-gnu/libc.so.6

Saved corefile core.18170

The file core.18170 now contains a copy of the memory from within the VM - it's as if we lifted the DRAM chips out of a live system and extracted their contents to a file.  We now perform a trivial analysis of the core using the strings tool to extract all ASCII text strings from the dump, then look for anything that could be one of our PANs, i.e. anything of the form 412356+10 digits using egrep:

18:06:14 kvm-host:~> strings core.18170 | egrep '412356[[:digit:]]{10}'

4123569876543210

>4123560123456789

4123569876543210

4123569876543210

Sure enough, both PANs are there, even though the server software never attempted to log them to disk, and even though the process which was handling them exited the moment we disconnected.  This exposure would not be possible to catch by any software running inside the guest VM, since the exposure is occurring outside of the VM.  Therefore, the only way to catch this is by monitoring all actions taken on the hypervisor itself, and the only way to prevent it is to securely lock down the hypervisor.

Worse, if those had been real ISO8583 messages, then the full content of the message would likely be recoverable.  This includes what the PCI SSC considers to be 'Sensitive Authentication Data', defined as full track data, PIN block and CAV2/CVC2/CVV2/CID.  This is data which you're never allowed to store, and which this echo server software (rightly) doesn't attempt to save to disk.  But it's still in memory for some period of time until overwritten, and can be pulled silently from the hypervisor environment.

In a similar vein, any encryption keys which are used to perform software encryption within a VM would be present in the VM dump.  Finding these keys would be more tricky than simply using grep to look for a text string, but it would be possible.  The worst case scenario would involve walking through the image, looking for aligned blocks of data with the size of the key which could be valid keys (e.g. a randomly generated key is unlikely to contain a NULL byte) and then testing them.  This is still many orders of magnitude easier than attempting to break a key by brute force.

I actually consider myself to be a strong proponent of Linux, and it is not my intention to put anybody off using Linux as a hypervisor in a secure environment.  I am hoping to draw attention to the fact that a standard Linux distribution cannot and should not be used as another 'appliance' hypervisor.  The hypervisor is more critical to your security posture than most other infrastructure components, since a hypervisor compromise allows every system running on top of it to be silently compromised.  The hypervisor should be thoughtfully deployed as a first-class security citizen, and must be secured as any other host would be, including hardened configuration standards, FIM, logging of administrative actions, and all the rest.

If in doubt, ask your QSA (auditor) for an opinion.  Contrary to what some people believe, they are there to help!

Broken aptitude when running in xterm

With recent versions of Ubuntu, and using xterm as your terminal emulator, the package selection tool aptitude has a nasty habit of corrupting the display as it's used.  For example, running aptitude, then searching for "test" produces the following:

As the display is updated, some text remains which should have been overwritten with blank space, but isn't.  This makes the tool difficult to use, as you're left sorting out the real, current text from the gobbledygook remnants of previous screens.  The fix for this problem is to change the TERM environment variable to be xterm-color rather than the default xterm.  Unfortunately, this causes another issue because some tools (such as vim) have their own corruption issues when run with TERM set to xterm-color.

The solution is to put the following in your .bashrc:


if [ "$TERM" = "xterm" ]; then
        alias aptitude="TERM=xterm-color sudo aptitude"
else
        alias aptitude="sudo aptitude"
fi

The reason for the embedded sudo, and for the alias being defined even when TERM isn't xterm?  sudo doesn't process shell aliases or functions, and so sudo must be embedded in the alias.  Defining an alias even when TERM is already good is simply to preserve consistent behaviour, i.e. never needing to type sudo manually to invoke aptitude.

After doing this, aptitude now behaves correctly when searching:

Good stuff!

Concatenating PDFs

Concatenating PDF files should be pretty straightforward.  On Linux, there are several tools that can do this, including pdftk, pdf2ps and convert, which is a wrapper for Ghostscript.  Unfortunately, I had a batch of files that I wanted to concatenate for ease of use on my tablet, and none of these tools were working.  pdftk failed repeatedly with the useful error message:


Error: Failed to open PDF file: 

   input.pdf

Using pdf2ps did create a merged copy of the input files, but it was HUGE, consisting of bitmap images of the pages, losing the text in the process.  ImageMagick convert never ran to completion, since I terminated it after it had eaten over 3GB of memory, presumably rendering the text into images.

Ultimately, I was able to successfully create a high quality, merged copy of my files by resorting to manually invoking Ghostscript:

gs -q -sPAPERSIZE=letter -dNOPAUSE -dBATCH -sDEVICE=pdfwrite -sOutputFile=output.pdf *.pdf

The resulting file is actually smaller than the combined total of the input files, storing text as text, rather than horrid, pre-rendered bitmaps.  Ghostscript used a sane amount of memory, and it ran to completion in a sensible amount of time.

To Ghostscript, bravo!  To the others, a big "Why"?

Ubuntu branding #fail

Install Ubuntu 11.10 Server, add Xfce4 desktop environment and reboot.  Result?  Debian space theme branding on the grub and boot screens.  Quality Assurance, anyone?  Perhaps images from upstream packages need a little more vetting before importing...

debian space-themed boot screen/grub menu on Ubuntu 11.10 "Oneiric"

Linux ICMP redirects

It seems that Ubuntu 11.10 ships with a sample /etc/sysctl.conf which contains the following statement, intended to tell the system not to originate ICMP redirects when acting as a router:

# Do not send ICMP redirects (we are not a router)
net.ipv4.conf.all.send_redirects = 0

Unfortunately, (at least) with kernel 3.0 as shipped with Oneiric, even after setting this and activating with 'sysctl -p', it doesn't work.  Symptoms are noisy kernel log records such as:

[611513.083432] host 192.168.0.100/if2 ignores redirects for 8.8.8.8 to 192.168.0.1.

If you actually want to disable sending ICMP redirects, you have to explicitly set this per interface in /etc/sysctl.conf, by doing:

# Do not send ICMP redirects (we are not a router)

net.ipv4.conf.eth0.send_redirects = 0

net.ipv4.conf.eth1.send_redirects = 0

net.ipv4.conf.eth2.send_redirects = 0

etc.

Ubuntu service management

Running 'service --status-all' gives the following output:

 [ - ]  apparmor

 [ ? ]  apport

 [ + ]  apt-cacher-ng

 [ ? ]  atd

 [ + ]  bind9

(snipped)

Who on earth thought it was a good idea for the status characters to be symbols that have special meaning when they appear in a regex?  It makes doing 'service --status-all | grep ...' less trivial than it should be.

Ping!

In both Linux and the BSD variants, the default behavior of the ICMP echo-based ping command is to enter an infinite loop, sending a probe once per second.  This results in output something like:

ubuntu:~> ping 8.8.8.8

PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.

64 bytes from 8.8.8.8: icmp_req=1 ttl=52 time=35.9 ms

64 bytes from 8.8.8.8: icmp_req=2 ttl=52 time=37.3 ms

64 bytes from 8.8.8.8: icmp_req=3 ttl=52 time=36.2 ms

64 bytes from 8.8.8.8: icmp_req=4 ttl=52 time=37.0 ms

64 bytes from 8.8.8.8: icmp_req=5 ttl=52 time=36.4 ms

^C

--- 8.8.8.8 ping statistics ---

5 packets transmitted, 5 received, 0% packet loss, time 4004ms

rtt min/avg/max/mdev = 35.936/36.594/37.308/0.548 ms

In Solaris, on the other hand, the default output of the same command has always been to print '<machine> is alive', or 'no answer from <machine>' after a timeout.  Per the Solaris ping manual page, the Linux/BSD behavior is known as statistics mode, and has to be enabled by running ping with the -s flag.

Now, it seems that an easter egg has been added to the Solaris 11/Express ping program at build 33, remaining in later builds.  If you set the shell environment variable MACHINE_THAT_GOES_PING (I'm not joking!) to any non-null value, then the default ping behavior changes to statistics mode.  I've confirmed this as being the case on my b127 Solaris Express host.

Thanks to John Beck for the tip!

Thought for the (last) decade?

Just found a wonderful quote from the original, July 1974, Communications of the ACM paper on UNIX written by DMR and Ken. They say:

"Perhaps the most important achievement of UNIX is to demonstrate that a powerful operating system for interactive use need not be expensive either in equipment or in human effort"

This is perhaps something that SCO should have considered over a decade ago when they first decided to pursue Linux because (they felt) it was created far too quickly to have been done without stealing (what they believed was their) UNIX IP.

Oracle and ZFS shenanigans

With the latest Solaris 10 release or recommended patch cluster, there are significant updates provided for ZFS. By patching or reinstalling with Solaris 10 9/10, you can get close to the zpool version which was previously only available by using Solaris Express Community Edition, now Solaris 11 Express.

But there is a potential catch. Each incremental feature change to zpool capabilities causes the zpool version number to be incremented. You can see what versions are supported on your local install by typing:

zpool upgrade -v

which lists all available features on the current driver, along with their version number. After applying the most recent Solaris 10 patch cluster, you'll see the following:

cara:~> zpool upgrade -v

This system is currently running ZFS pool version 22.

The following versions are supported:

VER DESCRIPTION

--- --------------------------------------------------------

1 Initial ZFS version

2 Ditto blocks (replicated metadata)

3 Hot spares and double parity RAID-Z

4 zpool history

5 Compression using the gzip algorithm

6 bootfs pool property

7 Separate intent log devices

8 Delegated administration

9 refquota and refreservation properties

10 Cache devices

11 Improved scrub performance

12 Snapshot properties

13 snapused property

14 passthrough-x aclinherit

15 user/group space accounting

16 stmf property support

17 Triple-parity RAID-Z

18 Snapshot user holds

19 Log device removal

20 Compression using zle (zero-length encoding)

21 Reserved

22 Received properties

Note that version 21 is 'Reserved'. If you run the same command on a system running the Express kernel, version 21 shows as:

21 Deduplication

22 Received properties

23 Slim ZIL

The whole point of zpool versioning is that a pool with a given version number should be mountable on any system running ZFS where the kernel supports at least that version of the pool. Sun went to great lengths to enable this, even specifying that ZFS was endian-independent, where all writes would be done with the local byte order, but reads would be honored in either big or little endian order. You can move a pool from a SPARC to an x86 platform, and it works.

This was going to be a blog post about the evils of Oracle Corporation breaking this compatibility. Version 21 is deduplication on the Express version, but reserved on the release version. I was going to rant about the dangers of creating a filesystem utilizing deduplication using Solaris Express, then trying to import it into a release version of Solaris 10.

But I can't quite do that.

After performing an experiment, it seems that Solaris 10 can in fact correctly mount pools created on Express which have deduplication enabled. However, 10 won't continue to dedup newly written data, since that's not a supported feature. This at least makes sense as a compromise. Compatibility is preserved across pool versions, to the extent that you won't see any nasty side effects like kernel panics if you accidentally mount a deduped filesystem on a release version of Solaris 10. You won't get any further benefit from this unsupported feature, but it shouldn't kill you either.

So my only question is this: is the dedup feature left out of these updates because Oracle wants to provide a compelling reason to move to Solaris 11 (which may also feature significantly different license terms)? Or are they leaving it out because there's a concern about bugs which impact integrity, availability, or both in the current version of the software?

Time will tell.

Forcing a Linux Reboot

Linux zfs-fuse is an extremely useful piece of software, but this morning it crashed on me to the point where even 'reboot -f' was failing to reboot the server due to kernel confusion.

Fortunately, there is a way to use the Linux SysRq mechanism to force an immediate reboot. This won't sync the disks, and certainly won't wait for processes to terminate (which is why it works in this case), but it certainly saved me from going to the data center to manually intervene.

To do an emergency reboot on Linux, perform the following two steps as root:

echo 1 > /proc/sys/kernel/sysrq

echo b > /proc/sysrq-trigger

This causes an immediate reboot of the system. Of course, if the thing causing the problem was a corrupted root filesystem, the server may not boot, but that would be the case regardless :-)

More details on SysRq are available at the Linux Kernel site.