2008-07-25 05:44:00

Webmin: It is fixed if we say so

On July 25th I looked a bit into an older Webmin vulnerability, hoping to fix it for the older version of webmin shipped in pkgsrc. However, as it turned out, CVE-2008-0720 affected “almost every single search field». Looking a bit through the search CGIs I quickly found a cross site scripting issue as well as an URL parameter problem, so I looked into Webmin 1.400 for fixes where the problem was supposed to be fixed.

Being the nice person I am, I created a patch and sent it to the webalizer developers. The response was that there was no issue to be resolved since the referrer check would most likely catch the issues already. Thus, I decided to release information about the issue without further verification. The vendor is king, right?

Find the fixes in the current pkgsrc version.


Posted by Tonnerre Lombard | Permanent link | File under: security, programming

2008-07-23 00:04:25

Subversion — tales from a failed migration

Today our new programmer complained very loudly about that shitty CVS never showing him all the information he was looking for. He asserted that the best alternative was to use Subversion. Since he preferred not to work with CVS, we decided that instead of adding another animal to our version control zoo (currently featuring git as a legacy from older projects, CVS for revision keeping and Bazaar for projects with release versions and support periods), we would try to eliminate one.

Little did we know that this would have to be Subversion itself.

Data migration

First we migrated various CVS repositories to Subversion. The only automatted way to do this was cvs2svn (apart from the apparently-defunct Tailor) so we went for it. As it turned out, cvs2svn can only convert one project at a time, not an entire repository, so we ended up with different repositories for source, docs and other things for the same project.

As it turned out, cvs2svn could not convert all of our projects because some had files which were added, then later removed and re-added. This caused spontaneous explosions of the poor little tool, leaving our repository entirely unconverted. So we tried to migrate at least a subset of our repositories, ending up with a big bunch of partial-project repositories.

Commit mails

The first, and supposedly rather simple, hook to add were commit mails. Debian offered a package to achieve this, called svnmailer. This was basically the svn-mailer script from the Subversion site, which is Python code and has a bunch of dependencies, so we were glad we had the Debian package.

However, the script of course doesn't provide automatic activation for the various Subversion repositories cvs2svn created, so we had to add the hook and config files one by one (it seems that it also doesn't offer a sophisticated way to extract the repository name, so we couldn't use a generic mailer.conf).

After some debugging we got a mailer.conf to work which would send mail correctly, so we created a test repository and let it send some test mails via the commit hook. This turned out badly. Despite the fact that we had specified --debug in the commit hook, the script did not output anything and silently failed to send mail. Thus, we prefixed the entry in the post-commit hook with “strace -o /tmp/strace.out» (attention, symlink vulnerability!) and then suddenly it worked.

We then tried random stuff and found that not invoking sendmail directly but instead using the builtin SMTP client of the script would help, so we left it at that. Couple of hundreds of config files later, all repositories sent out commit mails to the relevant changes mailing lists.

However, it turned out quickly that the mails were, for some reason, double encoded to quoted-printable. Every “=” was replaced with a “=3D3D”, leaving an extraneous 3D in the output. Setting the config file parameter to «8bit» yielded singly encoded quoted-printable. Whatever, it was readable.

Web interface

The next step was the web interface. Everyone else, including our unfinished inhouse Open Source platform, used ViewVC, so we wanted to go for it too. However, it turned out that Debian did not ship ViewVC in the stable distribution, nor in a backport.

So we installed WebSVN, which I only recently patched in a pkgsrc-security hackathlon, and kept our fingers crossed. It looked terrible at first glance, required PHP to work and came with several charset problems (some of the umlauts on the site were ISO-8859-1 while some others were UTF-8). Additionally, it required a flat hierarchy of repositories in order to auto-discover them, so we had to put all repositories directly into /home/svn.

Then we went on to integrate the web interface into the commit mails as we were used to from CVS and Bazaar. After modifying a hundred repositories' mailer.conf files, it finally worked as well.

Permissions

Then we started to notice that the repository permissions were changed on every commit. We tried to set the SGID bit on the repositories as we were used to from CVS, but that didn't work, Subversion insisted on resetting the group ownership every time. As a solution, we put a chgrp -R cvs $REPO into the post-commit hook, but this error'd of course for everyone who was not the owner of the files, so we replaced it with chgrp -Rf cvs $REPO, which worked silently.

RT and Bugzilla integration

At this point things started to get down the slope. We also had to integrate our RT and Bugzilla scripts with Subversion. These scripts were meant to attach short versions of the commit mails (without diffs but with URLs and commands used to create diffs) to tickets and bugs mentioned in the commit messages.

This, however, required us to retrieve the commit message and all changes manually, using the svn commands. A lot of wild svn commands. However, before we finished this, another problem stepped in our way.

However, building full revision patches was rather easy; the CVS commit ID would simply have to be replaced with the Subversion revision ID.

User defined keywords

We also noticed that our merge scripts weren't working properly anymore across repositories. Of course, since we hadn't defined the custom keywords anymore which we used for the repositories. They basically work like this:

  • The Repository A uses the keyword $RepoA$ which is expanded locally like $Id$. They have a file ciss_pci.c with revision 1.52, containing the keyword
    $RepoA: src/sys/dev/pci/ciss_pci.c,v 1.52 2008-05-29 18:09:31 tonnerre Exp $
  • Project B imports the file into their code base, which is stored in RepoB with keyword $RepoB$, and customizes it slightly to work with their codebase. The file contains the keywords:
    $RepoA: src/sys/dev/pci/ciss_pci.c,v 1.52 2008-05-29 18:09:31 tonnerre Exp $
    $RepoB: src/sys/dev/pci/ciss_pci.c,v 1.2 2008-07-01 14:23:03 jmcneill Exp $
  • Project A updates their file with several patches to revision 1.58. It now contains the keywords:
    $RepoA: src/sys/dev/pci/ciss_pci.c,v 1.58 2008-07-04 22:54:41 tonnerre Exp $
  • Project B uses an apply script to automatically merge the changes between the Project A revisions 1.52 and 1.58 into their script, without having to look at the file except to clean up the merge conflicts.

Unfortunately, Subversion only supports a predefined set of keywords to be expanded, only one of which exports to the Id. Thus, it is impossible to synchronize files between different repositories. This makes it basically impossible to use Subversion for a lot of our projects which share code. This actually means most of our projects.

Vendor branches

Subversion also has a very bizarre notion of branches. Actually, there are no branches. There are only subdirectories in the branches directory (if you want to call it like that). Branches are created by copying files around directories. Relations between the branches basically don't exist.

What also doesn't exist are vendor branches. This makes it very hard to track projects coming from different vendors entirely, while applying local patches.

When using vendor branches, new versions of the original branch are always checked into the same vendor branch. Changes between the old vendor branch and the HEAD revision are created and merged into the new vendor branch, saved as the new HEAD revision and collisions are kept around to be fixed manually.

Subversion requires users to re-import the source into a new branch, diff against the branch and to apply the changes from the vendor into the branch with the user changes, rather than the other way around. This usually is tedious and requires a lot more work, especially in undoing the work in changes which fix the same problem (Rather than to just remove the change, one has to unapply the original modification and then apply the change to it again).

More than that, Subversion doesn't allow files from different revision to coexist; it isn't possible to have an older revision of specific files in a checkout. CVS allows this due to the fact that it tracks revisions per file. When debugging interactions between different changes, it may however sometimes be desirable to downgrade specific files to older revisions in order to exclude a change in chasing a bug. Subversion makes this unnecessarily hard.

Conclusions

So summing it up, the work involved in migrating from CVS to Subversion was rather exorbitant due to the different behavior in various places. Subversion focusses on full-tree changes to the point where it becomes hard to use otherwise. It also pushes the principle of simplicity ad absurdum, removing various very helpful features which one would expect from a version control system.

It also appears that Subversion focusses on projects which only ever use one single tree and don't share code with other projects, where everything is reimplemented. This may be on par with the vast majority of GNU projects, but doesn't sound adequate for the BSD world, where code is shared vividly between most projects. A good example for this is the Korn shell. The project doesn't have a web site or download location, but its code is shared between various source trees and keeps developing. As an example, a patch for UTF-8 support is nowadays part of “the tree» – if we can speak of one.

Under the aforementioned circumstances it seems to be fair to say that Subversion has its place in the world of tiny projects but stops working when many people are working together on code, or when code is shared publically. SVK – another product from Best Practical – tries to work around these limitations, but various problems remain unsolved.

Subversion still has a very long way to go before it can be on par with CVS and other similar version control systems. However, considering that some of the problems lie deep within the very design of Subversion, it is highly questionable whether this will ever happen. So we can only wait what the future will bring. All we can say so far is that right now it was not possible to migrate our projects to Subversion, and that we went back to CVS.

2008-08-17 20:41: I received a mail from the cvs2svn author stating that his tool can deal very well with the mentioned situation. His tool at least told me this was not the case, but I will re-verify and post the exact error message here if appropriate.

2008-09-14 17:44: The output is, along with a response to many more comments, in More notes on Subversion, a new article.


Posted by Tonnerre Lombard | Permanent link | File under: programming

2008-07-14 00:21:48

Goulash and programming — notes from the GPN

The last weekend was claimed by 4 different CCC events. I decided to go to the Goulash programming night of the Entropia e.V. Karlsruhe as a member of the Network Operating staff.

The former was nice, the latter was a bad choice in some way.

Ambience

I met a lot of old and new friends at the GPN. Due to some circumstances, it was also a chance to get in contact with a great amount of new people I had never met before. The University in the middle of the city was also a nice starting point for discoveries, shopping tours and simply going to the restaurant.

While the restaurant made me pay € 10.- too much for the food which had bad effects on my digestive system after a short while, the City Rallye Ralf created was clearly one of the best highlights of the entire event. However, there were also some two or three interesting lectures (I am hard to impress).

Network Operations

The network didn't run smoothly at all most of the time. The initial setup took slightly longer than intented, but that was mainly our fault since we tried to debug a VLAN config while filtering the probes. But later on the first day, the network was running.

Until the afternoon of the second day, all network traffic was running over a Cisco VPN to the VPN server, which was shaped to 1.5Mbit. This is enough usually for a single person, but with a hundred people who want internet access, the per-user bandwidth was unbearably low. So on the second day we got a technician of the University of Karlsruhe to provide us with a network cable to the University datacenter.

However, that's when the differential problems started to kick in. A couple of hours after activation, our first Intel e1000 died. We replaced the router since it was onboard hardware, and used the replacement machine. This machine had an e100 and four Sun Happy Meal cards. On the morning of the third day, both e100 ports died and everything ran over the Happy Meal card which is way more resilient with regard to voltage potential.

Unfortunately, the death of the e100 happened at a time when only me was around and I didn't have root passwords to the router itself, so I had to bridge around it for the users VLAN and reconfigure the name and DNS servers to work around the defunct admin VLAN. This worked more or less, but took away my capability to make contact with the rest of the admin staff who were at home. Unfortunately I also wasn't in possession of a telephone number of the staff members.

Another problem was that the VPN connection kept coming back and buggering up the routing, which was also a source of problems twice.

As a conclusion, we need high voltage resilient hardware next year, and we need to ensure that internal organization between the netadmins takes place before the first day of the event.


Posted by Tonnerre Lombard | Permanent link | File under: chaos

2008-07-13 12:58:58

pkgsrc-security hackathlon

Since agc called for an end of the pkgsrc freeze for the release of pkgsrc-2008Q2 tonight at 21:00 UTC, I had to put in a hackathlon to fix as many security problems as possible. This is an attempt to keep track of them.

websvn

pkgsrc has a very old version of WebSVN (1.x, current is 2.0) which comes with a whole bunch of cross-site-scripting issues (see also CVE-2007-3056). In order to fix them during the freeze I had to manually patch WebSVN because there wasn't enough time anymore to ask for a package upgrade of this dimension. Patches were mostly taken from upstream changeset 581. It seems however that both the programming and the English skills of the author are subterranean.

silc-client

The next item was the silc-client buffer overflow which salo refused to fix for more than a year. I had already created patches for it but they weren't applied for half a year now, so I decided to just go ahead and commit them (especially since wiz suggested so in a prior conversation in April). They basically update the silc-client package from 1.0.4.1 to 1.1.4, which also fixes various character set issues.

modular-xorg-server

Then our modular-xorg-server came with various vulnerabilities, namely those described in freedesktop.org bug 7447 and the release notes of xorg-server 1.4.2. Applying the patches in question fixed the problems, even though an upgrade would not appear to be a bad idea.

balsa

For relaxing I took a shot at an old one-liner in balsa. Gnome bug 4743662 has a neat little classic buffer overflow which was easily fixed by adding a range check to the buffer. Unfortunately my 10G slice allocated to packages started to run full here; Gnome is way too large these days.

bacula

The next station was the bacula information disclosure: passwords would be passed to various tools through the command line. The ”solution“ the bacula developers came up with was to document the fact. I guess since bacula is mainly a bunch of shell scripts, there isn't really a better way either, so I pulled their documentation.

vobcopy

A maintainer who clearly doesn't take anything seriously is the maintainer of vobcopy. I fixed an insecure temp file creation issue but it was rather hard to parse his changelog with all the smileys.

Perdition IMAP

Perdition IMAP was another easy one; a buffer overflow check in IMAP command tag deparsing had to be rearranged to remain effective if the tag contained a NULL byte.

pear-MDB2

An encounter I would rather have avoided was pear-MDB2. That little beast has an information disclosure vulnerability in the structure of the respective drivers, so they all have to be updated individually. This was a rather tedious task, and in addition to that it was PHP code...

Worse than that, the package even shipped with an XML file containing MD5 sums of all source files et cetera. This made it even harder to patch since the package.xml file had to be updated as well with every change, and it was a downloaded file.

Z shell

The Z shell featured a broken difflog script which created temporary files with predictable names. Most distributions simply removed the file from the distribution, but I decided to simply throw in a bit of File::Temp love. The result worked judging from my first small tests.

OpenAFS

After I told Gendalia a while ago that she should fix OpenAFS, I did it myself now. The fix consisted in an upgrade to a newer version.

wyrd

The last one handled today was wyrd, a simple privilege escalation in the ocaml code. The patch from Debian bug 466382 applied well.

WIP packages

Some pkgsrc-wip packages needed TODO entries, among those were jetty, e2fsprogs and Radiator.

Old tickets

A number of tickets were also simply outdated and had to be closed. The result of the day:

  Open tickets Stalled tickets
Before 113 27
After 69 3

I also left some crazy patterns, not in the sand but in the mailing list archives.

pkgsrc-security contest

The current stats in the (not serious) pkgsrc-security contest are:

salo tonnerre adrianp ghen wiz tron joerg 36 others
2038 334 238 153 107 92 63 327

Welcome in life, pkgsrc-2008Q2!


Posted by Tonnerre Lombard | Permanent link | File under: security, programming

2008-06-29 14:20:28

There's no crash like /bin/bash

I recently encountered a rather ugly bug in the nss_ldap module. The bug was rather easy to spot: if the nss_ldap.conf file was not readable, any call to getpwent(3) and similar functions would cause an immediate segmentation fault (access to a NULL pointer).

However, the way I learned about the problem was slightly weird. Some of the unprivileged users on my server used bash as their login shell, and I started noticing a lot of bash processes using up lots of CPU time in a tight loop. I attached a trace process to it and noticed that bash was caught in an endless loop calling the SEGV handler.

Inspecting the situation a bit closer, I noticed that bash, in its signal handler – which is defined for various different signals which are supposed to terminate the process – made calls to save the bash_history file, unset its signal handler and re-deliver the signal to itself.

There is a large number of things wrong with this code. Firstly, there is no protection against the same signal handler being called twice for different signals. If the second handler runs while the first is still running, this will lead to some cleanup functions being executed twice. Worse than that, if on non-BSD systems free() is called twice on the same buffer there is a possibility to get arbitrary code executed. However, according to the Bash people, this is not a problem because it would require the user to exploit his own shell.

Worse than that, bash also uses the getpwent API in the signal handler. This caused the segmentation fault handler to segfault, leading to an endless loop of segmentation faults.

Evidently, after the emacs X11 input handling problems, the GNU people still didn't learn that signal handlers are not the proper place to call non-reentrant functions.


Posted by Tonnerre Lombard | Permanent link | File under: programming