Khrooty4Daily - Highly Experimental!

Experiment - Khrooty4Daily

Not much feedback here (but thanks to those wo did reply :)), but, hey - let's give it a go anyway!

I've cleaned up and uploaded a copy of Khrooty4Daily which you can find here. A few notes before people download it:

• I'm going in completely blind here, and have no idea if this will work across distros - it could easily simple fail for most people at the first hurdle.


• A chroot is not as insulated as a VM - in particular, processes started in the chroot are not separated from those started outside the chroot.


• Again unlike a VM, it's possible for apps in a chrooted environment to e.g. crash/ lockup your X session.


• Based on John's blog here, it seems at least possible that the chrooted KDE4Daily session won't run concurrently with an existing KDE4 session, which is a big shame, if true.


• I have some difficulty unmounting /dev/ from the chroot, apparently necessitating a reboot if I want to delete the chroot. Important Note: Under no circumstances should you attempt to delete the chroot without properly unmounting it!

Potential doom and gloom aside (I prefer to err on the side of caution :)), it would be great if at least a handful of people - preferably spanning a goodly range of distros - could download and try it and report their findings here :) As mentioned, I've not tried this before outside of Kubuntu 8.04, so if any step fails for you, please post here.

So, from the top, open a shell and navigate to where you want the chroot to be. You will need root access. Download it:

wget "http://home.kde.org/~kde4daily/Khrooty4Daily4.2-experimental.tar.bz2"

Check the md5sum:

md5sum Khrooty4Daily4.2-experimental.tar.bz2
dad7d49fe39e416ff36f3ab59bf23229 Khrooty4Daily4.2-experimental.tar.bz2

Extract *as root* (should take up <2.5 GB). The "p" flag is important, here: we want things that should have root ownership (e.g. kcheckpass) to have root ownership.

tar -xjpf Khrooty4Daily4.2-experimental.tar.bz2

Quick spot-check that the permissions are OK:

ls -l kde4daily-chroot/usr/bin/kcheckpass
-rwsr-sr-x 1 root root 14934 2008-12-06 10:33 kde4daily-chroot/usr/bin/kcheckpass

We now need to mount /sys, /proc and /dev in the correct places in the chroot - please refer to the warning about /dev, above. Again *as root*:

mount -o bind /proc kde4daily-chroot/proc
mount -o bind /sys kde4daily-chroot/sys
mount -o bind /dev kde4daily-chroot/dev

If all has gone well, you are now ready to chroot into your new Khrooty4Daily environment! *As root*, do:

chroot kde4daily-chroot /bin/bash

You will now be running as the root user inside your Khrooty4Daily chroot. You might want to change the password of kde4daily at this point. Let's switch to our kde4daily user:

su - kde4daily

The command-prompt should be red.


The story so far ...

Let's do a quick update:

kde4daily-update

Ok, now to try starting up a GUI session. I believe it's possible to run apps from the chroot transparently inside your current X session, or give Khrooty4Daily a full-blown X session of it's own or even, I would guess, to eventually add it as a selectable option to KDM, but I don't really feel comfortable with that, so I'll just use Xephyr for now. Install Xephyr from your distros packages, and start it up. I'll use screen :1 as I only have one display - you might need to use a different number. There may or may not be xhost authorisation required - please let me know :) Let's use a 1024x600 screen. *Outside* of your chrooted session, launch Xephyr:

Xephyr -screen 1024x600 :1

Now from within your chrooted session, do

export DISPLAY=localhost:1
startkde

The localhost appears to be necessary. The :1 should obviously match the :n you used when launching Xephyr. All being well, Khrooty4Daily should launch in your Xephyr window!


Success!

Obviously, if something has gone wrong before you get to this point, please let me know. If not, let's do something daring - let's compile Amarok!

In your Khrooty4Daily session, open up a Konsole and navigate to the KDE4DAILY_SOURCE_DIR (this is just a convention - you can bung it anywhere you want, but it's simpler this way for the time being.)

cd $KDE4DAILY_SOURCE_DIR

Amarok lives in the multimedia (extragear) module (http://websvn.kde.org/trunk/extragear/multimedia/). Due to the way the KDE build system is arranged, we'll have to check out and run cmake on the whole module, although we'll only be actually compiling Amarok itself. Check out the source; we'll put it into a directory called extragear-multimedia to differentiate it from ordinary multimedia:

svn co svn://anonsvn.kde.org/home/kde/trunk/extragear/multimedia extragear-multimedia

Let's make a corresponding build dir for doing out-of-source builds. Again, I'll use $KDE4DAILY_BUILD_DIR by convention:

mkdir $KDE4DAILY_BUILD_DIR/extragear-multimedia
cd $KDE4DAILY_BUILD_DIR/extragear-multimedia

Khrooty4Daily provides the cmakekde command as described on techbase, but this command implicitly does a make && make install on the whole module, whereas we want to do cmake on the whole module, but only make && make install inside the amarok dir. So I provided the onlycmakekde command. This is roughly analogous to the "./configure" step of autotools:

onlycmakekde $KDE4DAILY_SOURCE_DIR/extragear-multimedia

Ordinarily you'd have to hunt around for dependencies at this stage, but Khrooty4Daily is jam-packed with them :)


Almost too easy ...

Now we just need to compile amarok. Make yourself a nice cup of tea during this step - you've earned it! If you have a multi-core CPU, you might want to tweak the "make" parameters.

cd amarok
make VERBOSE=1 && make install

If all goes well (it did at the time of writing), Amarok should be installed and ready to launch! I can't get the Xine backend to work (almost certainly my fault) so I have to change the backend to GStreamer in The Multimedia section of System Settings.


Switching to the GStreamer Backend


Kind of!


Groovy!

Funky Music not included :)

To exit the chroot, simply

exit

from the chroot session until you get back to the shell on the host machine that you entered the chroot from. It doesn't appear to be necessary to unmount your /proc, /sys and /dev unless you're fed up with the chroot and want to delete it, so I always leave mine mounted. If and when you do decide to delete khrooty4daily, you will need to unmount it. From the directory containing khrooty4daily, and as root:

umount kde4daily-chroot/proc
umount kde4daily-chroot/sys
umount kde4daily-chroot/dev

As mentioned, the unmount of /dev might fail. DO NOT attempt to delete kde4daily-chroot until it has been cleanly unmounted! You may need to reboot in order to achieve this. When you are 100% sure that it is no longer mounted, simply delete the kde4daily-chroot directory - this step will need to be performed as root, so it's best to do it from the command-line.

And that's it! Assuming everything went OK, and you haven't followed the instructions too blindy and deleted khrooty4daily, you can check out source for other modules or apps from kde-apps.org and compile them, hack on them, install kdevelop3 from the repositories or kdevelop4 from SVN, whatever floats your coding boat, and get stuck in. Maybe sign up for Klassroom, or help out with the million and one other tasks that could all use an extra pair of hands :)

Good hunting!

Khrooty4Daily - Interest? Potential Difficulties?

Phew! It's been quite a few months since my last blog, and lots of stuff has happened - I finally got my PhD, so now I can work on KDE without any guilt, and as a result I've gotten my SVN account so that I can wreak some havoc(K)!

From my last blog:

"Anyway, that's all for now - I hope to blog about some non-KDE4Daily-related stuff in the near future :)"

Well, it looks like I suck, but hopefully soon I'll be able to blog about some of the (small amount of) upstream work I've been doing on trying to bring back some of the file management features to Konqueror that went missing during the port to KDE3 to KDE4, and hopefully, eventually, improve on them. Konqui fans - look out for a Konqui4 File Management - State of the Union that I'll be putting together with ppenz within the next month or so!

As for this blog: I've noticed, on IRC and the forums, that many people seem to be having problems getting a KDE development environment set up. Since KDE4Daily releases track the most recent Kubuntu release, and I couldn't be bothered to update my 8.04 Desktop and Laptop to 8.10, this time around KDE4Daily was entirely developed and tested within a self-contained chroot nicknamed, appropriately enough, Khrooty4Daily. It could of course have been performed within a VM, but chroots have a few advantages (and disadvantages) compare to VMs that make it more appropriate for development, IMO: not least, the comparative ease of transferring files to and from the host machine; sharing the hosts RAM; and of course running at native speed which is much more pleasant for compiling and debugging.


Khrooty4Daily offers an easy way to check out and compile KDE sources: all of the cmake stuff is there, the .bashrc macros are present and correct, there's a full and up-to-date KDE install ready to be run, tested and hacked on, and there is a very rich set of package dependencies provided which will allow you to compile pretty much anything from SVN with as many of the optional features as I could satisfy the requirements for. So I was toying with the idea of cleaning and tarring it up and providing it to the community, but since it's going to be a very large blob (probably someone between 1-1.5GB compressed as a rough estimate) that will take me ages to prepare and upload, I thought I'd scope out the level of interest for such a beast. Who thinks they would find this useful? It's not quite as completely insulated and self-contained as a VM, but it still provides a nice way of getting things running on your machine that you can just untar when you begin and rm -rf when you're done.

And also, I appeal to LazyWeb: what are the potential difficulties in providing a chroot environment? Will it work with all x86 distros? Since the root password for all Khrooty4Daily installs is the same and we are going to be mounting /dev/, /sys/ etc onto it, are there any security issues that people need to be aware of? Descriptions of any hitches at all that people have run into with using chroots would be much appreciated as I've only been able to test it in a Kubuntu 8.04 host so far!

KDE4Daily Quickies: Goes Native; Lightly Traces Backs; Decides Our Fate in a Microsecond

Hello Planet - many thanks to jriddell for adding me :)

My name is Simon St James, and some of you may have heard of a project I run called KDE4Daily. This is largely running independently of me, now, so I'm looking into doing some "upstream" KDE development in the not too distant future, probably after the release of 4.1. Writing some docs for Plasma (I've never written docs before, so this will be a nice challenge) and helping out ppenz with his long, long list of Dolphin/ Konqueror TODOs will probably be the first thing I inflict on you all :)

Anyway, here's a trio of news snippets about KDE4Daily that I've been sitting on before getting syndicated:

Goes Native

A few people have asked whether it would be possible for Kubuntu 8.04 users (which forms the base of KDE4Daily) would be able to run KDE4 natively, and I'm pleased to note that the answer is "yes". Here it is running in Xephyr; I've also run it in its own X session, but this is slightly trickier.

The technique is based on the rough idea sketched out near the end of this document,
and in essence consists of making a new user called "kde4daily"; scraping out the scripts and the install from the VM image and dumping them into /home/kde4daily on your PC; installing the non-KDE package dependencies; starting up Xephyr and startkde'ing. Everything seems to work as expected: updates work the same as usual; kwin_composite worked in OpenGL mode when I ran it in its own X session, and XRender mode inside Xephyr; backtrace generation works with a little effort; etc.

I'm not quite sure what to do with it, though: some of the packages may or may not be something people particularly want on their desktop at the moment (e.g. mysqld for Akonadi server - actually, lazyweb - are there any security issues one needs to be aware of if one installs mysqld?), and of course, a native install can crash your X server if things go badly. On the plus side, the entirety of KDE4Daily is bunged into a single folder (except for the external package dependencies) so it can be easily removed if you decide you don't want it :) If people are interested in doing this, I'll see if I can put together a semi-official guide, but I'm fairly busy at the moment. Let me know :)

Lightly Traces Backs

This is a little long and technical, so I'll postpone it till the end.

Decides Our Fate in a Microsecond

As I've been planning for a while, now, human decisions have now been removed from the build process and it operates purely on a "If it compiles, ship it!" basis, which means I can bugger off at the weekend, say, or get run over by a bus, without (hopefully) everything screeching to a halt :) You can track the progress of builds here:

http://etotheipiplusone.com/kde4daily/buildprogress.html

It also operates as a "Dashboard Jr", and will display in angry red any compiler errors it encounters before re-trying again in a few hours.

Lightly Traces Backs (for real, this time)

One of the showstoppers for the initial release of KDE4Daily was getting proper, detailed backtraces working in a reasonably bandwidth efficient manner (both for you, the end user, and me, who has a very slow upload bandwidth shared with his housemates). The vast, vast volumes of debug info generated each build make this rather difficult, though, so in a rush, I knocked together a system that separates all the debugging info, binary diffs it against the last uploaded version, and uploads these much smaller diffs. In your KDE4Daily install, a request for a backtrace from DrKonqui via gdb will download the required pieces of debug info and re-assemble them, before offering them up to gdb for the formation of a nice, detailed backtrace. This process is rather slow, as you can imagine, but there was plenty of room for improvement, and I've spent quite a lot of time on trying to fix this.

The attempt was three-pronged: firstly, I got rid of the lazy and inefficient implementation that simply loaded the whole re-assembled debug data into memory before giving it to gdb in favour of a much more memory efficient buffer-and-stream solution. Secondly, I allow some of the debugging data to be pre-fetched during the kde4daily-update process when there is plenty of free memory (this functionality is only available on my harddrive at the moment). And thirdly, I delved deep into the guts of gdb to make it considerably more efficient when loading symbols for backtraces - with stock gdb, creating a backtrace (including Qt stack frames) for Konqueror took about 500MB of memory; with my patch, it peaks at about 200M, and generally hovers around 80MB or so. The patched gdb is called gdb-fast-bt-hack and handles all DrKonqui backtrace requests.

The basic effect of the patch is to allow a special mode, signified by passing the flag "--backtrace-only", that loads *only* the pieces of debugging needed for creating the stack frame, and unloads them when it has extracted the required stack frames. So not only does it take much less peak memory, but it also requests less pieces of debug data, which is expensive in the KDE4Daily system. Note that this will likely only work well if the debug info has been separated from the main executables/ libraries.

If you like horrifying hacks, you can find the patch here:

backtrace_only_gdb.diff

When gdb is attached, and if --backtrace-only has been passed to it, you can call "bt" and "bt" only (and only once!) to generate the backtrace.

Sadly, the time savings aren't as dramatic as I had expected in the VM - it seems I'm hitting some operations that are about an order of magnitude slower in the VM than in a real install. But it took me a whole weekend to do it so you're going to hear about it whether you like it or not ;)

Anyway, that's all for now - I hope to blog about some non-KDE4Daily-related stuff in the near future :)

Anatomy of KDE4Daily

A few people have asked about making installable or LiveCD versions of KDE4Daily, so I've put together a kind of "Making of ... "for the project. You can find it here:

http://etotheipiplusone.com/kde4daily/docs/makingof/makingof.html

Hope people find it useful/ interesting! :)

Frist psot!

My Thesis is finally at the binders, so my self-imposed KDE contribution ban is finally at an end. Just in time for the Feature Freeze :)

Anyway, I exploited a loophole in my agreement with myself last October and made a small contribution to KDE in the form of KDE4Daily (my argument was that since KDE4Daily didn't actually involve sending any patches to KDE, it wasn't really much of a contribution. Since I was quite keen on the idea of working on KDE4Daily, I readlily conceded the point and allowed myself to work on it).

It proved to be quite popular, and several people asked if it would make a return again in the run up to 4.1. Since none of the KDE4 devs objected to the idea, I'm currently working on it now. Hopefully it will "go live" within the next 24 hours or so, but my terrible upload bandwidth is currently really putting the brakes on.

In the meantime, I thought I'd paraphrase an e-mail I sent to the kde-core mailing list about how the project went last time, and what its strengths and weaknesses were.

To re-cap, the KDE4Daily project had the following goals:

1) To provide low(-ish)-bandwidth (binary) daily updates (generally 20-50MB
per day) of an SVN snapshot of a reasonably wide range of KDE modules (
http://etotheipiplusone.com/kde4daily/docs/kde4-modules.txt )

2) To be able to generate useful backtraces, such as this sample:
http://etotheipiplusone.com/kde4daily/docs/konqueror-backtrace-sample.txt

(Note that inclusion of Qt debug info in backtraces was disabled by default
due to high memory usage when loaded into GDB.)

3) To be self-contained and distro and operating system neutral - KDE4Daily
was provided as a Qemu Virtual Image, and the updates were performed from
within the image itself.


Basically, it aimed to be a way of lowering the barrier of entry to people who
want to test KDE and file bugs and also to confirm that bugs claimed to be
fixed in SVN were indeed fixed.

Over its 7-week run, only ~5 daily updates were missed. The homepage asked
people to mention "kde4daily" in their bug reports, and a simple search
reveals ~54 such bugs:

http://bugs.kde.org/simple_search.cgi?id=kde4daily

Some of these are just "this bug is still occuring in revision 7xxxxxx"; and,
embarrassingly, were actually caused by KDE4Daily itself! The following
weaknesses in the project were uncovered:

1) Backtrace generation was incredibly time-consuming and resource intensive
due to the need to fetch and patch-in up-to-date debug info, which probably
discouraged many testers from generating backtraces.

2) VMs do not provide 3D acceleration/ compositing, so these aspects were not
tested.

3) The usage instructions neglected to detail how to enable sound, so sound
(Phonon, etc) was not tested.

4) Generally, KDE4Daily is a very homogeneous test environment: every
up-to-date install was effectively identical to all others, excepting the
user's personal data/ config files. Great for reproducing bugs; lousy for
shaking out those little corner cases that only 1 in 100 people will ever
stumble upon.

5) I think many end-users approached the project as a way to satisfy their
curiosity about KDE4.0.0 and to watch it progressing, rather than as a means
to help them file bug reports.

6) The partitioning scheme was very stupid and restrictive, and it seems that many people who actually wanted to work on KDE from within the KDE4Daily VM itself were thwarted by it.

Number 6) is already tackled with the 4.1 edition (people have a nice, 20GB, unpartitioned virtual HDD to mess around in :)) and I'm hoping to tackle at least #1 over the next few days (KDE4Daily 4.1 will hopefully have been released by then, but it can be automatically rolled out in an update) - more as and when it happens :)