I have never really been a big fan of debuggers. Not that I think they aren’t useful, I just have never really had a use for them. I typically get all the debugging I need out of print or trace debugging (I heard it was called scaffolding once, but I can’t find any references on the internet to it.) Most of the code I have written on my own has been short simple exercises for school. I did one really big project that spanned 3 courses and 2 semesters, but I still did print debugging because it was what I was used to. It got the job done.

Since working at StorageCraft, the project I’m working on are orders of magnitude larger than anything I’ve worked on before. The code base was easy to jump into because the code I was working on is written in small chunks. I was able to get really good mileage out of a test-driven development module because the code was already working on Windows, and I just needed to port it to Linux with out introducing any bugs on the Windows side. When all my unit tests passed, and it was time to build the full product line and test it’s features, using a debugger saved me hours because I could easily set break points and run the code again, instead of adding a print statement, re-compiling, and then running it again. I was able to get Code::Blocks working on Linux for a GUI debugger.

On to the good part. Doing kernel development has been a lot of fun, but I couldn’t use my newly acquired debugging skills with GDB, so I went back to using printk to debug my kernel module. I am running RHEL6 Workstation on my machine. I obviously don’t want to actively develop kernel modules on my workstation, so I am using the live-cd creator tools to build a CentOS 6 iso with just enough software to build the kernel modules for me. I chose a live CD because it don’t have to worry about corrupting my root filesystem when I kernel panic. Every time I boot the machine, it comes up in a clean state, with full read-write access to the OS. Perfect, in my mind. I use KVM to boot the VM, and access the VM via the console, so I don’t have to depend on moving my mouse around to access the root login prompt on the VM. Everything from gnome-terminal.

Well, we’re to the point now where debugging the entire kernel module from printk statements is getting difficult. My co-worker figured out how to get KGDB working in a Gentoo VM. Using his instructions as a starting point, I found out the RHEL 6 has everything I needed just sitting there waiting for me to use it. Here’s the kickstart I’m using to generate a live CD:

# Kickstart file automatically generated by anaconda.
#sudo livecd-creator --verbose --fslabel=centos_kerneldev --cache=/var/tmp/livecache --config=kerneldev-ks.cfg

url --url=http://mirror/centos/6/os/x86_64
lang en_US.UTF-8
keyboard us
network --device eth0 --bootproto dhcp
rootpw  --iscrypted #################
firewall --service=ssh
authconfig --enableshadow --passalgo=sha512 --enablefingerprint
selinux --disabled
timezone --utc America/Denver
bootloader --timeout=3 --location=mbr --driveorder=vda --append="console=ttyS0,115200 kgdboc=ttyS1"
services --disabled kdump,libvirt-guests,lvm2-monitor,mdmonitor,postfix,rpcbind,rpcgssd,iptables,ip6tables,abrt
firstboot --disable

repo --name="local"  --baseurl=http://mirror/local/ --cost=50
repo --name="CentOS"  --baseurl=http://mirror/centos/6/os/x86_64/ --cost=100
repo --name="CentOS Updates"  --baseurl=http://mirror/centos/6/updates/x86_64/ --cost=1000
repo --name="EPEL"  --baseurl=http://mirror/fedora-epel/6/x86_64/ --cost=1000


echo '*.* @@rsyslog.hostname' >> /etc/rsyslog.conf
#I echo an NFS mount point into /etc/fstab for my source code here.

With I put a comment in the kickstart to remind me how to build the ISO :). I also put some other things in post specifically related to how I want to access my kernel module and source code from inside my VM. You don’t want to see any of that 🙂 I also have a local repository that I can drop things into if I want to override an upstream rpm with. For instance, I ran into a bug with the new version of GDB not working well with KGDB. So I downloaded the SRPM for my kernel (2.6.32-131.6.1.el6), added the patch, and rebuilt it. The bugzilla bug says it should be fixed in an upcoming kernel version, so eventually I won’t need to put my kernel packages in my personal repo anymore.

In my Kickstart, you’ll notice that I want to run a console on ttyS0, and kgdboc on ttyS1. We need to expose these both to the host OS. Here’s my qemu xml definition for one such VM:

<domain type='kvm' id='32'>
    <type arch='x86_64' machine='rhel6.1.0'>hvm</type>
    <boot dev='cdrom'/>
    <bootmenu enable='no'/>
  <clock offset='localtime'/>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw' cache='none' io='threads'/>
      <source file='/var/lib/libvirt/images/centos_kerneldev.iso'/>
      <target dev='hdc' bus='ide'/>
      <alias name='ide0-1-0'/>
      <address type='drive' controller='0' bus='1' unit='0'/>
    <controller type='ide' index='0'>
      <alias name='ide0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    <interface type='network'>
      <mac address='52:54:00:b0:f9:10'/>
      <source network='KernelDev'/>
      <target dev='vnet0'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    <serial type='pty'>
      <source path='/dev/pts/14'/>
      <target port='0'/>
      <alias name='serial0'/>
    <serial type='tcp'>
      <source mode='bind' host='' service='4555'/>
      <protocol type='raw'/>
      <target port='1'/>
      <alias name='serial1'/>
    <console type='pty' tty='/dev/pts/14'>
      <source path='/dev/pts/14'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='5900' autoport='yes'/>
    <sound model='ich6'>
      <alias name='sound0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
      <model type='cirrus' vram='9216' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>

You may find it easier to configure the VM from the gui, but taking screenshots and walking you through that interface just isn’t interesting to me. It should be easy for you to read the <serial> portions of the configuration file, and apply the options in the GUI.

Next, we need the host machine to have the source code and vmlinux image to debug. All I needed on RHEL6 was to install ‘kernel-debug-debuginfo’ which pulled in ‘kernel-debuginfo-common’ as a dependancy.

yum -y install kernel-debug-debuginfo

Now, the only concern here is making sure your kernel-debug-debuginfo pulls in the same version for the kernel you’re running inside your VM. This is one more reason why the local repository is useful. Just build or download the RPMs for the kernel version you need, and stick them all in the same place. Once there, install them from the same place in your VM and in your Host.

With all that in place, we’re finally ready to boot our VM. If you want to debug the kernel at boot, you need to add “kgdbwait” to the bootloader option in the kickstart (and rebuild the iso ofcourse). I don’t need to, so I let the VM come all the way up. Then I build and insmod my kernel module from the commandline (or a unit test script.)

insmod mymodule.ko
cat /sys/module/mymodule/sections/.text
echo g > /proc/sysrq-trigger

With these three commands, I have everything I need to finally run gdb. I need the output of the .text file to tell gdb where to load my kernel module’s symbols, otherwise breakpoints will not have the correct address. gdb has an -x parameter that you can use to automatically run some gdb commands. For instance, my gdb commands file looks likes this:

file /usr/lib/debug/lib/modules/2.6.32-131.6.1.el6.x86_64.debug/vmlinux
dir .
add-symbol-file mymodule.ko 0xffffffffa0398000
target remote localhost:4555

The add-symbol-file line’s 3rd argument is the output from the .text file from inside the VM. My unittest will probably start outputting a gdb commands file eventually, so I can just run ‘./unit_test.sh’ from inside my vm, which writes out a gdb commands file to an NFS share on my host machine, and then from my host, I can run gdb -x commands_file. I haven’t reached that point yet, but I really wanted to get what I have done so far written down.

One last note about break points. You can use gdb to set breakpoints right after running the ‘target remote’ command before running the ‘continue’ command. Alternatively, you can execute the function kgdb_breakpoint() from inside your code to trigger the break point. This is useful for when you want to break into your code when there is an exceptional case, instead of using BUG() to crash your kernel, and dump a call trace (which I have done A LOT).

PS3 + Mediatomb

So with school so recently out of the way, working full time has provided a little disposable income. I first splurged on a Wii, and got exactly what I wanted and what I expected. The wife doesn’t seem interested in letting me HomeBrew the Wii and install Linux, so it’ll just sit tight. Tonight, I bought my PS3, with the intent of replacing my Desktop that’s been attached to the TV for over 3 years now. That means I needed to fix Hulu and finalize my Mediatomb installation. I’m 1 for 2 tonight.

The internet is such a wonderful place, full of helpful people. It took all of 15 minutes to have my entire library available to the PS3. Even HD mkv video files work great (except when I’m connected via WiFi….) CentOS 6 has been great in that respect.

Hulu, not so much. It appears I do have Flash installed on my PS3, but Hulu refuses to be nice to me. I found one guy that said it was easy with squid. Lucky for me, I’m already running Squid (plus Dansguardian) on my network, and use iptables to redirect all outbound port 80 traffice through squid/dansguardian. I tried adding the two lines for the access_* parameters, but both are unrecognized by squid 3.1. It looks like I get to delve into some squid documentation tomorrow. For now, it’s time for bed.

Postfix+Dovecot with LDAP

I’ve been running a Postfix+Dovecot with MySQL as the backend for eSource for some time now. It’s on old CentOS 5, but it really did wonders for our ability to manage Virtual email accounts. We’ve been using postfixadmin to manage the MySQL database. Recently, I’ve been getting into LDAP, and have found some success migrating to CentOS 6.

The goal here is to simply replace MySQL with LDAP. (This isn’t a “tutorial”, so I can’t guarantee that copying and pasting anything will work. I frequently copy something real, and then massage the text to eliminate sensitive data. I may or may not be consistent with my changes.)

First, I have to duplicate my vmail user that does all the email transactions.

useradd -u 101 vmail -g mail -s /sbin/nologin -d /var/vmail

Then I copied over my postfix config, which is already configured for Dovecot and for MySQL. If you are starting from scratch, I pity you, and I’m sorry I don’t have time to start postfix from scratch. (On the plus side, the dovecot stuff below is “from scratch”.) So I left all the dovecot stuff the same, and found all the mysql entries and changed them to ldap in /etc/postfix/main.cf:

virtual_mailbox_domains         = proxy:ldap:$config_directory/ldap_virtual_domains_maps.cf
virtual_mailbox_maps            = proxy:ldap:$config_directory/ldap_virtual_mailbox_maps.cf
virtual_alias_maps              = proxy:ldap:$config_directory/ldap_virtual_alias_maps.cf

I picked up a really great tip from Postfix’s VIRTUAL_README page. It suggests, “The reader is strongly advised to make the system work with local files before migrating to network databases, and to use the postmap command to verify that network database lookups produce the exact same results as local file lookup.” I’ll say it right now, “postmap -v” saved me HOURS of troubleshooting. The LDAP README article is an excellent starting position and explains some of the basic configuration settings. My setup turned out to not include some of the attributes that postfix defaults to, so I had to make some adjustments. First, the virtual_domains list is what postfix uses to determine if it should handle the mail locally, or lookup the domain’s MX record, and send it over the internet. I created an LDAP “Organisational Unit” and named it “Domains”, and put all my domains underneath, like so:

dn: ou=Domains,dc=example,dc=com
description: Domains is used for Postfix as it's list of locally hosted doma
objectclass: organizationalUnit
objectclass: top
ou: Domains

And then add a domain:

dn: dc=example.com,ou=Domains,dc=example,dc=com
dc: example.com
objectclass: dNSDomain
objectclass: top

In LDAP, there are two main parts of the schema, the ObjectClass and the Attribute. The LDAP database is a tree, so everything is a parent or a child of another entry. ObjectClasses define which attributes are available, and if they are required or not. You can only add attributes to an entry if the ObjectClass lists it.

Now, in ldap_virtual_domains_maps.cf, I filter the query to be just dNSDomain objectclasses that have a “dc” attribute equivalent to the domain being looked up. The attribute that carries the value I’m interested in is “dc”.

server_host = ldap://localhost/
search_base = ou=Domains,dc=example,dc=com
version = 3
bind = no
query_filter = (&(ObjectClass=dNSDomain)(dc=%s))
result_attribute = dc

If you are familiar with SQL, the postfix variable “result_attribute” is like your “SELECT” clause (but your limited to only one column), and the postfix variable “query_filter” is an LDAP query_filter and is like your WHERE clause. In LDAP, the query_filter syntax here is doing an AND logical operation on both conditions. I don’t have a good reference for the filter syntax, but google does return some good results for “ldap filter syntax”.

With out having to even bother restarting/reloading postfix, you can test the configuration file with postmap, like this:

postmap -v -q example.com ldap:/etc/postfix/ldap_virtual_domains_maps.cf

If you leave out the -v, you either get a result (and a return value of 0), or no result (and a return value of 1). With -v, you get to see all the variables that postfix uses to query the LDAP server, including default values, in lines that begin with “cfg_get_str”, “cfg_get_int”, “cfg_get_bool”, and so on. What’s really nice is that the syntax is perfect, and you can copy and paste it into your .cf file. This is how I figured out how to use result_attribute correctly. Now that postfix knows which domains it is hosting email for, we teach it which mailboxes exist. We start with more LDIF:

dn: ou=People,dc=example,dc=com
objectclass: organizationalUnit
objectclass: top
ou: People


dn: cn=Kai Meyer,ou=People,dc=example,dc=com
cn: Kai Meyer
cn: Master of the Universe
gidnumber: XXX
givenname: Kai
homedirectory: /home/kaiuser
loginshell: /bin/bash
mail: kaiuser@example.com
maillocaladdress: kaiuser@example2.com
maillocaladdress: kaiuser@example3.com
o: eSource
objectclass: person
objectclass: organizationalPerson
objectclass: inetOrgPerson
objectclass: inetLocalMailRecipient
objectclass: posixAccount
sn: Meyer
telephonenumber: 555-555-5555
uid: kaiuser
uidnumber: XXX
userpassword: CENSORED

Notice, I have a “mail” and “maillocaladdress” attribute. “maillocaladdress” is a list of email aliases that will deliver to the “mail” address. With those LDAP entries in place, you can configure ldap_virtual_mailbox_maps.cf:

server_host = ldap://localhost/
search_base = ou=People,dc=example,dc=com
version = 3
bind = no
query_filter = (&(objectclass=inetOrgPerson)(mail=%s))
result_attribute = mail

So now I’m starting in the “People” tree (which means LDAP won’t even look in my other “Domains” tree) for all entries that are ObjectClass “inetLocalMailRecipient” and have a “mailLocalAddress” set to the lookup value. This is where email will actually attempt to deliver. IN my case above, it’s “kaiuser@example.com”. Again, before bothering postfix with a reload, you can do another postmap:

postmap -v -q kaiuser@example.com ldap:/etc/postfix/ldap_virtual_mailbox_maps.cf

Again, eliminate the -v if you just need a simple test. If you leave out the query_filter and result_attribute, postfix goes looking for objectclasses and attributes that don’t exist on my default openldap install, so again I’m modifying the default behavior to get the information I want it to look for. Hurray, mailboxes done. Now let’s do aliases. We don’t need any more LDAP entries. We created one with fields we can use for aliases already. Here’s ldap_virtual_alias_maps.cf:

server_host = ldap://localhost/
search_base = ou=People,dc=example,dc=com
version = 3
bind = no
query_filter = (&(objectclass=inetLocalMailRecipient)(mailLocalAddress=%s))
result_attribute = mail

Here, the result attribute is the same “mail”, but the filter is for “mailLocalAddress”. Doing another postmap for example2.com should return the email address for example.com, based on my LDIF above. Again, I wish I had LDAP attributes the way I want them (and I could modify/create schema to get it if I really really wanted to), but this is what I have to work with. So, postfix is done (for me anyway.) I don’t have any other MySQL lookups in my existing server. So now we need to plug in Dovecot.

CentOS 6 comes with Dovecot 2.0. I know that Dovecot2 is very different than Dovecot1, so I’m sorry if you’re stuck on 1. The changes to dovecot were actually quite minimal. Start in /etc/dovecot/conf.d/10-auth.conf, and un-comment the ldap include:

!include auth-ldap.conf.ext

Then copy the example ldap config to /etc/dovecot.conf:

cp /usr/share/doc/dovecot-2.0/example-config/dovecot-ldap.conf.ext /etc/dovecot

Then modify the following variables (they are all there either in comments or not):

hosts = localhost
auth_bind = no
ldap_version = 3
base = dc=example,dc=com
deref = never
scope = subtree
user_attrs =
user_filter = (&(objectclass=inetOrgPerson)(mail=%u))
pass_attrs = mail=user,userPassword=password
pass_filter = (&(objectclass=inetOrgPerson)(mail=%u))
default_pass_scheme = SSHA

With LDAP done, we just need to teach dovecot about how to dump the email the same way it was don the last machine, in /var/vmail/<email dress>. In /etc/dovecot/conf.d/10-mail.conf, add:

mail_location = maildir:/var/vmail/%u

Then we need to enable postfix-auth and auth-userdb correctly. In /etc/dovecot/conf.d/10-master.conf, change your “service auth” settings to this:

service auth {
  # auth_socket_path points to this userdb socket by default. It's typically
  # used by dovecot-lda, doveadm, possibly imap process, etc. Its default
  # permissions make it readable only by root, but you may need to relax these
  # permissions. Users that have access to this socket are able to get a list
  # of all usernames and get results of everyone's userdb lookups.
  unix_listener auth-userdb {
    mode = 0640
    user = vmail
    group = mail

  # Postfix smtp-auth
  unix_listener /var/spool/postfix/private/auth {
    mode = 0666
    user = postfix
    group = mail

  # Auth process is run as this user.
  #user = $default_internal_user

And we’re done! ( I think ). I may have missed a step here or there. Leave a comment if you are running into a problem. Chances are, it’s one I hit, and solved.

Next on the chopping block, SVN and SuPHP.

Kai vs LDAP

I’ve previously posted about the success I’ve had with my RHEL6 workstation joining the Windows Domain at the office. This has in-part inspired me to attempt to learn LDAP. I currently use a number of services that would be nice to have a unified authentication mechanism. I frequently use SSH (all the time) on multiple servers for administration work for eSource. I also run the eSource Mail server on Postfix/Dovecot + MySQL, using postfixadmin as my administrative tool. Lastly, my little SVN server that hardly gets any updates, especially now that I’m out of school. Unifying the Authentication across these three services would provide a great deal of flexibility for eSource, as well as my own personal stuff. So here we go.

First, my LDAP server will be CentOS 6 (another motivation for the move to LDAP is that I have to move mail, svn, and web services anyway.) It doesn’t take much to get a slapd service running, but you have to be careful. I thought it was as easy as editing slapd.conf, but it’s not….RHEL6 moved to a new slapd configuration format. Once I figured out where to stick the stupid password, slapd config was done. I installed phpldapadmin, and haven’t had any problems since. The trick has been learning what LDAP is all about. I’m still very confused, but I’m at last limping along. I’ve been able to successfully create an ldap entry, and use it to log in via ssh to my new server. Here’s an LDIF entry that I’ve exported from my running LDAP server, and slightly modified to obfuscate any information I’m concerned about. This isn’t a tutorial, so I can’t guarantee any of this will work cut-and-paste for you.

dn: cn=Kai Meyer,dc=example,dc=com
objectclass: person
objectclass: organizationalPerson
objectclass: inetOrgPerson
objectclass: posixAccount
objectclass: inetLocalMailRecipient
uid: kaiuser
givenname: Kai
sn: Meyer
cn: Kai Meyer
cn: Master of the Universe
telephonenumber: 555-555-5555
mail: therewasan@emailhere.com
maillocaladdress: therewasan@emailhere.com
userpassword: CENSORED
uidnumber: 1000
gidnumber: 1000
homedirectory: /home/users/kaiuser
loginshell: /bin/bash

I did other sorts of things, like create an Organizational unit, and add the object to the OU by modifying the “dn:” to include the OU after the cn. (If that made sense to you, you should probably be doing this for me.) The only other thing I needed to do to enable SSH + LDAP is to configure authentication for the machine to allow LDAP. This is CentOS 6, so I did it like so:

yum -y install nss-pam-ldapd
authconfig --enablemkhomedir --enableldap --enableldapauth --ldapserver=localhost --ldapbasedn="dc=example,dc=com" --updateall

It ended up looking like this:

[kai@example.com ~]$ ssh kaiuser@localhost
kaiuser@localhost's password:
Creating directory '/home/users/kaiuser'.
[kaiuser@example.com ~]$ pwd

Next up, Postfix/Dovecot + LDAP. Then after that, SVN + LDAP.

RHEL6 vs Windows Domain logins

I started my new job at StorageCraft this week. We produce backup and disaster recovery software for Windows servers and desktops. They have grown big enough to organically grow their development team, and pursue new markets. I think I’m under NDA not to disclose the nature of the projects I’m working on, but given my history, it shouldn’t be difficult to figure some of it out.

One of the nice perks they offered was to purchase books or software that I think would make me more effective at work. So I asked for a RHEL6 Workstation License, and was successful in justifying the purchase. Since I am one of only 4 brand-new employees that have any substantial experience with Linux at all, their entire infrastructure uses Active Directory for authentication. Workstation logins, Version Control, Issue Trackers, WiFi, the VPN, and Email all use it. As a sort of last hurrah as a Systems guy, I decided to get my RHEL workstation “on the domain.” What that really boils down to underneath is allowing PAM to delegate authentication out to the Domain Controllers, so things like GNOME and SSHD can authenticate users “on the domain.”

The process really just boils down to popping up a little GUI, joining the domain, and installing winbind-server. The only thing the GUI does that I had to update the smb.conf file for was to use the default Domain for logins. Gnome didn’t like having the backslash in the username, ie: “WDM\kai.meyer”. With winbind running properly, running the command “id kai.meyer” returns valid user information.

Once I figured out how to authenticate (running “ssh WDM\\kai.meyer@kai-rhel6” just felt wrong), what services were needed (winbind, smb, nmb), and which config files were used (winbind actually uses smb.conf), I feel like I could easily teach our IT guys how to deploy a RHEL6 Workstation for Developers, or a RHEL6 Desktop for other positions like Technical Support, and give them warm fuzzies about eliminating those pesky Windows Viruses.

One more benefit of having my RHEL6 workstation on the domain is configuring samba shares to use Domain Authentication to control permissions to files. My local files can be shared over the domain, and access read-write from any other Windows workstation that I’ve logged into with my account.

I realize all these benefits from joining the domain are fairly small in reality. All of the “features” it provides can be done in so many less-convoluted ways. What really makes it worth it is having the IT guys go, “You can do what?!?”