KDE4Daily Quickies: Goes Native; Lightly Traces Backs; Decides Our Fate in a Microsecond

Hello Planet - many thanks to jriddell for adding me :)

My name is Simon St James, and some of you may have heard of a project I run called KDE4Daily. This is largely running independently of me, now, so I'm looking into doing some "upstream" KDE development in the not too distant future, probably after the release of 4.1. Writing some docs for Plasma (I've never written docs before, so this will be a nice challenge) and helping out ppenz with his long, long list of Dolphin/ Konqueror TODOs will probably be the first thing I inflict on you all :)

Anyway, here's a trio of news snippets about KDE4Daily that I've been sitting on before getting syndicated:

Goes Native

A few people have asked whether it would be possible for Kubuntu 8.04 users (which forms the base of KDE4Daily) would be able to run KDE4 natively, and I'm pleased to note that the answer is "yes". Here it is running in Xephyr; I've also run it in its own X session, but this is slightly trickier.

The technique is based on the rough idea sketched out near the end of this document,
and in essence consists of making a new user called "kde4daily"; scraping out the scripts and the install from the VM image and dumping them into /home/kde4daily on your PC; installing the non-KDE package dependencies; starting up Xephyr and startkde'ing. Everything seems to work as expected: updates work the same as usual; kwin_composite worked in OpenGL mode when I ran it in its own X session, and XRender mode inside Xephyr; backtrace generation works with a little effort; etc.

I'm not quite sure what to do with it, though: some of the packages may or may not be something people particularly want on their desktop at the moment (e.g. mysqld for Akonadi server - actually, lazyweb - are there any security issues one needs to be aware of if one installs mysqld?), and of course, a native install can crash your X server if things go badly. On the plus side, the entirety of KDE4Daily is bunged into a single folder (except for the external package dependencies) so it can be easily removed if you decide you don't want it :) If people are interested in doing this, I'll see if I can put together a semi-official guide, but I'm fairly busy at the moment. Let me know :)

Lightly Traces Backs

This is a little long and technical, so I'll postpone it till the end.

Decides Our Fate in a Microsecond

As I've been planning for a while, now, human decisions have now been removed from the build process and it operates purely on a "If it compiles, ship it!" basis, which means I can bugger off at the weekend, say, or get run over by a bus, without (hopefully) everything screeching to a halt :) You can track the progress of builds here:

http://etotheipiplusone.com/kde4daily/buildprogress.html

It also operates as a "Dashboard Jr", and will display in angry red any compiler errors it encounters before re-trying again in a few hours.

Lightly Traces Backs (for real, this time)

One of the showstoppers for the initial release of KDE4Daily was getting proper, detailed backtraces working in a reasonably bandwidth efficient manner (both for you, the end user, and me, who has a very slow upload bandwidth shared with his housemates). The vast, vast volumes of debug info generated each build make this rather difficult, though, so in a rush, I knocked together a system that separates all the debugging info, binary diffs it against the last uploaded version, and uploads these much smaller diffs. In your KDE4Daily install, a request for a backtrace from DrKonqui via gdb will download the required pieces of debug info and re-assemble them, before offering them up to gdb for the formation of a nice, detailed backtrace. This process is rather slow, as you can imagine, but there was plenty of room for improvement, and I've spent quite a lot of time on trying to fix this.

The attempt was three-pronged: firstly, I got rid of the lazy and inefficient implementation that simply loaded the whole re-assembled debug data into memory before giving it to gdb in favour of a much more memory efficient buffer-and-stream solution. Secondly, I allow some of the debugging data to be pre-fetched during the kde4daily-update process when there is plenty of free memory (this functionality is only available on my harddrive at the moment). And thirdly, I delved deep into the guts of gdb to make it considerably more efficient when loading symbols for backtraces - with stock gdb, creating a backtrace (including Qt stack frames) for Konqueror took about 500MB of memory; with my patch, it peaks at about 200M, and generally hovers around 80MB or so. The patched gdb is called gdb-fast-bt-hack and handles all DrKonqui backtrace requests.

The basic effect of the patch is to allow a special mode, signified by passing the flag "--backtrace-only", that loads *only* the pieces of debugging needed for creating the stack frame, and unloads them when it has extracted the required stack frames. So not only does it take much less peak memory, but it also requests less pieces of debug data, which is expensive in the KDE4Daily system. Note that this will likely only work well if the debug info has been separated from the main executables/ libraries.

If you like horrifying hacks, you can find the patch here:

backtrace_only_gdb.diff

When gdb is attached, and if --backtrace-only has been passed to it, you can call "bt" and "bt" only (and only once!) to generate the backtrace.

Sadly, the time savings aren't as dramatic as I had expected in the VM - it seems I'm hitting some operations that are about an order of magnitude slower in the VM than in a real install. But it took me a whole weekend to do it so you're going to hear about it whether you like it or not ;)

Anyway, that's all for now - I hope to blog about some non-KDE4Daily-related stuff in the near future :)