November 2008 Archives

2008-11-30 00:34:45

MySQL 5.1: few new features, but many new bugs

As Monty, who is apparently working at MySQL, wrote yesterday in his blog, there appears to be a rahter large number of quite devastating bugs left in MySQL 5.1 at the time of its release, many of which would have been declared release critical by open source projects.

In Oops, we did it again, Monty names a number of these bugs and outlines variety of bad management decisions which have lead to the current situation, and which apparently have lead to similar situations in the past.

To me, it does not really come as a surprise. As you may or may not know, I have been following the MySQL company, community and products for years, and haven't seen too many different types of releases so far. In any case, I strongly support the suggestion of Monty and also strictly recommend staying with MySQL 5.0 for now &mash; or to migrate to PostgreSQL, if possible.

2008-11-29 22:31:25

Trust based PGP key signing

As a visitor here at rgb2r is now moving to another planet a far-away country, the discussion came up again how to do PGP key signing properly. Originally, the requestor asked for passport based signing as done on most of the so-called PGP key signing parties. However, for the following reasons I will not sign passports and don't want passport based signatures.

Party signing

In the normal PGP key signing party based approach, all that is required to obtain a signature is a passport or identity card. The signer verifies that the signee looks like the person on the picture and then signs the key. For this, the signer does not have to know or trust the signee. The signer then submits the key to a key server and the signee can download it from there.

This method allows one to accumulate a lot of signatures over a short time, and to extend the network of signatures very quickly. However, it contains various flaws:

  • There is no trust relation between the signer and the signee, so the only thing that is attested by the signature is, that the signee has certain similarities with the documents he/she presented.
  • However, especially in multinational keysigning parties, these documents can rather easily be forged so that an untrained eye cannot detect the forgery.
  • There may be valid reasons not to want to show one's ID. For example if one's in possession of a diplomatic passport, one might not want to show it to everybody because it might be met with suspicion, or it might compromise the position of the owner of the passport to have everyone be aware of the status.
  • In some cases, people may have valid reasons not to disclose their real names, especially if they're working in IT security, if they are political activists or live in a totalitarian regime. But even under normal circumstances, it should be accepted if a person does not want to disclose his/her real name.
    Some people are not even known under their real name at all, and thus a signature on their real name would be rather pointless.
  • The associated addresses need not automatically belong to the signee.

There are probably many more problems this model is arising by its lack of trust dependencies or its focus on the passport as the sole base of trust, but these are the most apparent which came to mind.

Trust signatures

The so-called trust signature scheme follows an entirely different dogma. In this scheme, people knowing each other know enough will sign each other's key in order to testify that they are willing to testify that the signee is known as the person he/she claims to be. This of course implies a much greater level of familiarity between the signer and the signee.

Note that in this scheme, it is perfectly valid for the signer to sign a key belonging to a pseudonym — if that's what the signee is known as. That may sound suspicious to a person who's used to attesting passports, but in the end it is much more logical because in the end, the signer attests only what he knows about the other person, without relying on a paper he cannot verify.

Also, some more unrelated improvements come to play. In order to verify UIDs, the signer creates signatures for each individual UID and sends these individually to the mail address in the UID, in an encrypted mail. Then, the signer deletes the signature from the local keychain again. This sounds tedious, but there are good scripts to take care of that, such as, for example, caff from the pgp-tools package.

In this scheme, it is assured that the owner of the UIDs is at least aware of the signature that took place, and can decide whether or not to upload the signatures. On the other hand, a pseudonymous signer can participate without any loss of the trust relationship.

Posted by Tonnerre Lombard | Permanent link | File under: security, chaos

2008-11-26 12:31:18

MySQL build system command line wildness

Only seconds ago, the MySQL build system produced a ton of compiler command lines like this one:

gcc -shared -Wl,--whole-archive ndbapi/.libs/libndbapi.a common/transporter/.libs/libtransporter.a common/debugger/.libs/libtrace.a common/debugger/signaldata/.libs/libsignaldataprint.a mgmapi/.libs/libmgmapi.a common/mgmcommon/.libs/libmgmsrvcommon.a common/logger/.libs/liblogger.a common/portlib/.libs/libportlib.a common/util/.libs/libgeneral.a -Wl,--no-whole-archive -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lpthread -lcrypt -lnsl -lm -lpthread -Wl,-soname -Wl, -o .libs/

It seems that it really desperately wants to link against libpthread.

Posted by Tonnerre Lombard | Permanent link | File under: broken, programming

2008-11-25 18:34:20

Your daily pot of MySQL insanity: Threading

I really know I shouldn't look into MySQL source code, but since there are customers who really entrust their data to this piece of software, I sometimes have to.

The task today: Updating PHP on Etch to version 5.2.6. Sounds easy, right? Wrong. Because after you put up with disabling all the new security extension which prevent the worst PHP scripts you've ever seen from running, and after fiddling half a day with weird compile time detection scripts which don't work in a uniform way, i.e. discover different environments under different circumstances, thus leaving you with a PHP which is in a constant debate with itself where its extension directory really is, …

…you finally come to the point where you have a working PHP package which waits infinitely on a futex on exit. A command as simple as php -v hangs indefinitely.

Following the code path further up, there is main(), php_module_shutdown(), zend_shutdown() (Why this stack order?!), zend_hash_graceful_reverse_destroy(), zend_hash_apply_deleter(), module_destructor(), zm_shutdown_mysql() (Argh, there's our other worst offender), mysql_server_end(), my_end(), my_thread_global_end(), pthread_cond_destroy(), …

Digging through the code a bit, and digging through the glibc code some more, it becomes clear that this is indeed one of the good old pthread_exit() NPTL bugs from older glibc days. For once, it seems that Debian has also shipped beta software to the world. At least it wasn't some important system component, only the libc.

Looking around some more, I stumbled over the PHP bug #42625. It is not very helpful, though, it only proclaims:

 Problem will solve itself when you rename /lib/tls to

Thanks for the suggestion, dear PHP people, but I have to get some work done first. Of course the above workaround does its job, but it is simply not what you would like to see in your server environment. And I don't mean the name.

So I dug out a different patch for mysql which adds code to detect whether or not the glibc NPTL is in use. In fact, it gets the check entirely wrong by checking if the OS is Linux, while the bug is certain to appear on other operating systems running glibc as well. In that precise moment, though, I couldn't have cared less.

Digging through the related header files, some more horrid things turned up which made it perfectly clear that I should not have gone there. One comment was rather odd:

/*Irena: compiler does not like this: */
/*#define my_pthread_getprio(pthread_t thread_id) pthread_dummy(0) */

Yeah, Irena, I can imagine perfeclty well why the compiler didn't like that. After all, typed arguments aren't really supported in preprocessor macros. It can be hacked in using tricks, but they will always be tricks. However, now that you found out that the compiler doesn't like it, why commit it?

Only a number of lines further, there is some code to handle a nonexistent errno:

#ifndef ESRCH
/* Define it to something */
#define ESRCH 1

Maybe it is not the best idea after all to define ESRCH to EPERM.

Then finally I am at the code I want to modify, but there is some oddity as well (there's always one more oddity). The added thread types for the header file are not flags or anything, they just enumerate types of libraries. However, no enum is used. That's ok, but the numbering is slightly odd:

#define THD_LIB_OTHER 1
#define THD_LIB_NPTL 2
#define THD_LIB_LT 4

Why bitwise numbering if it is not a flag? Or can a threading library be LT and OTHER at a time?

Of course, that's all not dangerous, but it gives a very odd impression …

Posted by Tonnerre Lombard | Permanent link | File under: broken, programming

2008-11-25 11:56:56

Driving MySQL ad absurdum

I recently discovered that apparently there are database people with an insanely brilliant sense of humour: Drizzle.

Drizzle is a clone of MySQL 6.0 where any type of useful features has been removed. Its focus lies on Simply design for ease of installation and management (sic). There is a large list of features which have been removed:

  • Stored procedures, ok, they are not usually very useful in well-designed software, or maybe they are?
  • Query caches, so that queries will have to be reparsed and reexecuted every time, slowing down normal operation significantly
  • Prepared statements, slowing down repeated operations significantly and re-opening the doors to SQL injections
  • Views, ok, I admit I have started to dislike them myself
  • Triggers, because who cares about data integrity and the likes
  • Grants, who needs permission management anyway? Any attacker should be able to enjoy full damage capability
  • Some storage engines — I think this one slipped through, as it is not a hilarious idea

At the first glance I thought that someone was about to commit a devastating stupidity, but then, reading through the web sites and source code I am now convinced that this is an artistic act, aimed at making fun of the puritan MySQL community. Someone has proved a really great class of humour, especially in the way he imitates fresh students who fork software projects despite having no idea of what they're talking about. Also, the spelling mistakes look really authentic. I had a very good laugh.

Thank you a lot, Mr. Brian Aker, you're really a great commedian of the Open Source community!

Posted by Tonnerre Lombard | Permanent link | File under: programming

2008-11-17 13:11:03

"IT-Grundschutzhandbuch" against MySQL

The so-called IT-Grundschutzhandbuch of the German federal bureau of security in the information technology (BSI) openly pronounces itself against MySQL:

Zum Schutz der Datenbankintegrität muss die Datenbank-Software über ein vollständiges Transaktionssystem verfügen, welches dem ACID-Prinzip genügt.

(In order to assure data integrity, the database software must feature a full-blown transaction system which adheres to the ACID principle.)

As it is widely known, MySQLs transaction system, which was introduced only in version 4.0, does not satisfy the ACID criteria. Since the BSI also offers the BSI Grundschutzzertifizierung, which follows the guidelines laid out in the IT-Grundschutzhandbuch, this means that in order to obtain this certification, one cannot use MySQL — at least not as a database system.

As a side note, a number of German contract partners as well as numberous government agencies require their suppliers to have a BSI Grundschutzzertifizierung.

As a rather funny side note: according to a recent analysis by Alvar Freude, it is very likely that the new E-Petition site of the German parliament will fail not only the tests outlined in its own requirement specifications as well as accessibility guidelines, but also the very IT-Grundschutz certification.

Posted by Tonnerre Lombard | Permanent link | File under: news, standards

2008-11-14 14:06:59

Migration between operating systems or countries

I noticed today that sometimes there are serious parallels in migration between operating systems, just as there are between different countries.

Windows, for example, is somewhat like Germany. It will confront you with a large amount of bureaucracy in order to approve your immigration, and will sometimes refuse it claiming blatantly that you're a terrorist. If problems arise, you have to find someone to pay to fix them, who will usually be an immigrant. On the other hand, it lets go of you rather easily. If you attempt to immigrate into other operating systems, it may force you to choose either one.

MacOS (Versions 10, earlier and later) can be compared to the US. You have to be rich enough to immigrate in the first place. Your appartment comes with a record of the latest speech of the President himself. When you're finally there, everyone will keep iterating how everything is shiny, but looking out into the streets you quickly realize that there are many things seriously wrong. Despite borrowing various things from other countries (the Statue of Liberty is a glamorous example), they aren't really connected. Nevertheless, everyone will tell you how much they believe in their country. When emigrating, you're declared to be one of the greedy Germans, no matter where you go.

Solaris is like the british empire. When you immigrate, you first have to come to a much higher level. But then you are granted access to the most high-valued ressources of the entire Empire, and the Lords themselves will invite you to discuss their matters with them. When leaving, a message of regret is sent after you.

Linux is very much comparable to China. If you immigrate, you're greeted at first glance, but as soon as you start to critizise, you are outlawed. People will hate you, you may even be detained. People will try to prevent you from speaking at all. Once you try emigration, you will be chased until the end of the world by people trying to bring you back to the motherland.

OpenBSD, on the other hand, is a lot like Finland. You are welcome to join the country, but don't expect anyone to approach you with immigration forms. During your life, you will have to fix a lot of things on your own, and if you're doing that well, people will whisper words of thank to you when they're sure nobody listens. In the end, however, if you emigrate, everyone will say it's good riddance. (But most will not mean it.)

NetBSD has very strong parallels to Greece. You are welcomed warm-heartedly and people will bring cookies and other food and hard drinks. You're being asked by emigrated brits if you would care for tea. During your life, everyone keeps making jokes about problems, and one day someone comes up and fixes it. Since the majority of inhabitants is old and bearded, you use to meet for sitting on a bench and laughing benevolently about the mistakes of your own and other countries. People are assoicated with the things they do, if you need help in a specific area there is always a Mister or Miss Specific Area, who will be rather difficult to get hold of. Sometimes you're even referred to the wrong Mister Specific Area, because most of the time their name is Matt. When emigrating, people give you presents for the way and don't worry, as they know you will return either way.

Posted by Tonnerre Lombard | Permanent link | File under: general

2008-11-04 11:06:44

For all Django fans

There is now an ultimate must-have for all Django fans. OMG Ponies!

Unfortunately, it appears that this pony is not even free.

Posted by Tonnerre Lombard | Permanent link | File under: programming

2008-11-04 09:32:18

SQLAlchemy: With ancient methods against modern IT

There is a small number of projects the Python developers are especially proud of: Zope and SQLAlchemy.

The fact that the Python core team deliberately keeps breaking compatibility for Zope is probably bad enough, but at least SQLAlchemy appears to work well enough with the various versions. Well, but how does it work?

The answer is: bizarre.

Never trust the database to get sequences right!

The first thing I discovered was with PostgreSQL and sequences. Some tables use sequences for creating their primary key, which is then going to be a 64-bit integer. This is done automatically by either not naming the primary key column on insert, or by using the keyword DEFAULT.

Looking through the debug output, however, items like these can be spotted:

2008-11-04 09:26:24,258 INFO sqlalchemy.engine.base.Engine.0x..d0 select nextval('data_from_box_id_seq')

This is wrong on a number of levels. At this point, SQLalchemy manually queries the next value returned by the sequence for the id column. This value is consequently used in the insert statement for the id column. This breaks the atomicity of the insert statement — rather than letting PostgreSQL handle things internally as it's supposed to, the insert statement is artificially split into two statements. Of course, this also means that two SQL operations are performed, which almost doubles the network overhead (especially over encrypted connections).

The most likely reason why this is done is to have the row ID of the newly inserted row for use in the client side object representation. However, PostgreSQL has two much better mechanisms achieving the same: Firstly, the ID can be queried using the last_insert_id operation, or, even better, everything can be handled in one statement by using the RETURNING keyword:

INSERT INTO data_from_box VALUES (DEFAULT, something, else) RETURNING id;

Prepared statements are for the weak and timid

An even more bizarre affection manifests itself in the support for MySQL. Rather than to make use of the prepared statements, which are mandated by the API specification, SQLAlchemy attempts to reproduce prepared statements in the SQLAlchemy code, and to naturalize SQL statements manually. Then, the resulting statement is PREPAREd and EXECUTEd.

This of course constitutes a duplication of functionality, and more than this, the SQLAlchemy users are now prone to SQLAlchemy prepared statement naturalization bugs which cannot really occurr in SQL prepared statements (because they are never synchronized into a string but used as-is).

Overall, it is very hard to say what the SQLAlchemy developers had in mind, but it seems clear that it was nothing useful.

Posted by Tonnerre Lombard | Permanent link | File under: broken, programming