Tag Archives: clang

Spring: cleaning up

Its the time of the year again, spring. Meaning, time for the big clean up. Not the house of course, we leave that to the people feeling themselves called to clean houses. I’m talking about code. Over time we, as in developers, come to the conclusion that this neat API we wrote is not as neat as it looked in the first place. So we redesign our API and BIC lovers as we are, mark methods that should not be used anymore as deprecated.

This is of course very nice, but how do we actually get an overview of the methods that are marked as deprecated, more important how do we get an overview of how often deprecated methods are still in use? Well, kdevcpptools to the rescue. As I blogged before I worked on a C++ query and transformation engine for KDevelop as part of my Msc thesis. After an awesome lot of stabilization work of Milian "code monkey" Wolf, the plugin really is in quite a nice state at the moment and I would say quite usable. In the rest of this post I’ll show how to use it by taking kdepimlibs as an example.

The first thing to do is to create a query file containing the queries for the deprecated functions. The query (and transformation) file is a relative simple XML file describing the queries and optionally transformations. There are some example files in the git repository, but don’t jump to that yet as I want to show a cool feature Milian implemented.  Lets find the deprecated functions first. Okay, no fancy tools for that (yet?), so just a plain ack-grep on "KDE_DEPRECATED" in kdepimlibs. This gives me a lot of files containing this macro. We startup KDevelop and create a new empty query file with the following content:
?xml version="1.0" encoding="UTF-8"?

(Yes, the formatting sucks, don’t know how to make drupal output XML in a sane way). Next, open one of the files that showed up in the results of your grep and go to one of the deprecated functions. Make sure that you have the kdevcpptools tool view enabled and right click the function. The context menu shows you this wonderful option: Copy XML query:

Copy the resulting XML from the clipboard into your file and there you go, its that easy. I did this for whole kdepimlibs, you can find the file here. First observations: we’ve 47 deprecated functions in whole kdepimlibs and we really have  a shitload of date string formatting functions in kdepimlibs, so…. if you need one…..

When you’re done putting in your queries in there, you can open the XML file in the kdevcpptools plugin. This enables you to query a project. I did this for kdepim, meaning, the result I get are all uses of deprecated functions from kdepimlibs in kdepim. This result was so to say pleasantly surprising. Have a look at the next picture:

In the lower part of the screen you see an overview of the results for kdepim. Vertical axis are files horizontal axis are queries. First observation here: there are surprisingly few uses of deprecated functions. (Note: we’re only talking here about functions, explicitly marked as deprecated). Second observation: though not particularly interesting for such few results, the curve you see in the total column looks promising. I.e. most uses in only a small part of the files. Meaning, only a couple of files need "major" changes. This must of course be seen in perspective, this reasoning is not particular interesting when you only have seven hits for a ~700KLOC code base. Would you had have a similar curve for ~8000 uses this would of course already make a lot more sense. The curve in the total column will actually tell you how the migration work is distributed over the code base you’re querying.

So, my initial idea was to do a bit of clean up in kdepim but it turned out that this is hardly needed with respect to this specific area. The good thing is that we now know that it isn’t needed. This means that as soon as KDE5 development kicks off we can get rid of these deprecated methods without a lot of work (in kdepim at least). I’d really encourage anyone interested in these kind of analysis on his/her code base, either from a lib perspective like I did, or from an application perspective (anyone interested in building up a query file for qt3support and query the various kde modules for it?) to checkout the plugin[1] and play with it. You can find me and Milian in KDevelop if you need help on getting started with the plugin.

Next step is using the transformation engine to do (at least part of) the migration work automatically. I hope to blog about that soon. But first a thesis to finish…..

[1] http://gitorious.org/kdevcpptools Continue reading

Tagged , , | Comments Off

Yet another *full blown* database [1]

(Warning: long post coming up. Read on if you are interested in performance of the various Akonadi database back ends).

Last weeks I’ve been looking (again) whether or not it was possible to create a working sqlite back end for Akonadi. The last time I tried was around august last year and by then sqlite just wasn’t able to meet the multi-threading requirements that Akonadi has for its database back ends. A couple of sqlite releases later things seem to have changed. I managed to clean up some problematic code paths in Akonadi server and voila, we’ve a sqlite back end that is on par (unit test wise) with the mysql back end.

There are some catches of course. The first being that the default sqlite driver that comes with Qt does not work. It uses the sqlite API in a non-blocking way. So I had to adjust that to make the driver consistent with the sqlite documentation which states that when sqlite calls return BUSY, you have to wait a bit and try again. This custom driver can be found in kdesupport/akonadi/qsqlite.

The next catch is related to performance. Though we did not had the numbers until now, it was expected that sqlite would perform worse compared to mysql. Given that we have another back end, postgresql and are working on yet another one: a file system back end, it seemed a good time to do some benchmarking. So I brushed up the item benchmark a bit and performed the benchmark for all back ends. The benchmark tests the performance of Akonadi over two dimensions. The first dimension is the number of items and the second dimension the size of the payloads. We used the Qt data driven test framework and added rows to it like: ${number-of-items}-${payload-size}. Then we benchmark the important actions of Akonadi, which are creation, modification, fetching and deletion of items. This enables us to see how the performance scales with the two dimensions.

Before getting to the results I first have to make a note about the file system back end. This one is designed to be a fall-back for large payloads. The idea is that database actions become too slow for large payloads. So at a certain offset Akonadi doesn’t store payloads in the database but in plain files. The benchmark for the file system back end is set up to always write the payload to the file system. This enables us to find out the offset that gives best of two worlds, i.e. fast performance for small payloads by using the database and fast performance for large files by using the file back end. (Note: currently the file system back end is disabled by default, there still are some issues with it that need to be sorted out).

Sooooo, show me the numbers, I hear you thinking. Well, here you go, lets start with item creation:

Item creation benchmark

The image show the results scaled logarithmic on the x-axis (time on y-axis, but relative due to logarithmic scaling on x-axis). As you can see the, the files system back end (akonadi-fs) is hardly influenced by the file size, only by the number of items. For the other back ends we see that file size has influence on the performance too, but roughly scale  linear. We also see here that sqlite does not perform as well as the others. Lets have a look at the absolute numbers:

Item creation benchmark scaled linearly

The y-axis now shows time in msecs. The graph now shows us clearly that when items get larger and the number of items grows too, sqlite is clearly outperformed by all other back ends. We also see that databases in general don’t cope well with large payloads, which is exactly the reason to provide a file system back end too. First conclusion, don’t use sqlite unless you have very strong restrictions on your environment. (Which is the case when running Akonadi on Windows CE for example, where the number of processes is extremely limited and which we are working on here at KDAB). Still not convinced about sqlite performance? Okay, lets have a look at one more. Item modification:

Item modification benchmark, scaled linearly

Again, we see that sqlite is outperformed by all other back ends as soon as the payload size  becomes large. When the payload size grows we also see that only the file system back end doesn’t start to grow exponentially like the database back ends do. So, sqlite works, might even work fast enough for you, but is definitely not fast enough for the general use case Akonadi was designed for in the first place: handle many large items. Again, unless you have very strict requirments on the environment where Akonadi is used.

The last thing I want to show is a benchmark with different payload sizes for 2500 items. This makes it easier to find the cutoff value for the file system back end. I.e. what payload size should be used to store an item using the file system back end in stead of the database? First the images (I only compared mysql and fs to have slightly clearer graphs, you can find the full results at the links posted at the end of the blog):

Benchmark for creation of 2500 items

For creation and modification the cutoff value seems to 8 KB. However, fetching, which is also an often performed operation, has a cutoff value of 512B. A good trade-off between those two is probably around 4 KB.

So that’s all for now. Short recap. The sqlite back end for Akonadi seems to work, though its about ~5 times slower. Also, there are already some problems reported, so it still should be considered as a work in progress. Work on the file system back end is ongoing but seems promissing and with the right offset and file system/database combination (i.e. mysql) we get best of both worlds. Thanks for reading!

Links to the full (interactive) results:

  1. Multiple item counts, multiple sizes
  2. 2500 items, multiple size

Update: For a better comparison between database and fs back ends I added another benchmark which also uses 2500 items but only goes up to 8 KB payloads. Check the results here:

2500 items, multiple size, only up to 8K
[1] The title is meant to be a pun. Every now and than people pop up on the ML who think that sqlite is not a *full blown* (whatever they mean with that) database. Let me ensure you, it is. It supports SQL at a similar level as mysql, it does transactions and multithreading, it just tends to be smaller (and here I mean the library itself, not the database) and it does not run in a seperate process but it therefore has its limitations too. Continue reading

Tagged , , | 25 Comments

All good things come to an end….

even so KPilot. After having been part of KDEPIM for years it has been moved to tags/unmaintained/4/kpilot today. For months now no real development happened, there was not even bug fixing. I guess we have to draw our conclusions from that. Jason already said he lost his interest after buying a new phone about a year ago, while I myself still do have a palm I actually don’t use it anymore. KPilot has been a fun ride for me. Adriaan de Groot once dragged me in when I requested a new feature. It where my first in KDE software development. It showed me the trickeries of syncing pim data between two devices and it got me in touch with Akonadi. I even think it was one of the first pim applications to make use of Akonadi. So this means that from KDE SC 4.4 on users won’t be able anymore to sync there palm with the PIM suite again. I’m sorry for the inconvenience for those who still did (I did see a bug report coming up every now and than). If anyone is still interested in getting it up and running again, the code can be checked out via svn co svn://anonsvn.kde.org/home/kde/tags/unmaintained/4/kpilot I’d like to thank Adriaan and Jason for guiding me on my first steps. For the fun time we’ve had working on KPilot or whining about the pity state of device synchronization on Linux in general (still today it seems that there’s not really a rock solid solution available). I’d also like to thank Robert and Doug for their contributions and of course the people who started KPilot in the first place like Dan Pilone and Rainhold Kainhofer. We’ll have to see what the future brings. But with all those new powerful devices, which are able to access various groupware solutions I doubt if anyone wants to really invest in a computer <-> phone/handheld solution. The curtain falls for KPilot, good bye old friend, fare well! Continue reading

Tagged , , | 20 Comments

Help us testing KPilot using your n810/n800/n770

Dear lazyweb (at least the KDE part of that, having an n810 Garnet VM supported device),

I just heard from Jason “vanRijn” Kaspaer that there is a VM for the n810 which makes it possible to run Palm applications on your n810. I just gave it a try and its just awesome! When you install the VM you instantly have the addressbook, calendar and todo applications available. As these are currently the only supported applications in KPilot this should be enough ;) . I tried to install, configured kpilot, pressed the hotsync button in the VM and guess what? It just seem to work. So how to turn your n810 in a PIM device which can be synced against trunk?

  1. Goto: Garnet VM and download the GVM on your n810 (mind you, you need to register).
  2. configure KPilot to look for device: net:any (kpilot -s -> General Setup -> Device -> Pilot Device
  3. Make sure you have some valid resource configured for the conduit you want to test (Note: first try to create a new resource in Akonadi and don’t use your production resources. Note 2: Don’t tell me I didn’t warn you. Note 3: If you use the resource bridge of Kevin Krammer you can even add it to the kde PIM like kaddressbook.)
  4. start adding some items on one or both sides
  5. Start KPilot
  6. press the hotsync button in the GVM (Note: you have to leave the specific application e.g. addressbook to see that button).

That’s it, happy syncing :)

If you have Issues please let us know and no, no screen shots today. Try out yourself! For issues please drop by in #kpilot, #kontact or #akonadi (whatever is most appropriate for the issue).

Update: Of course this works for every Nokia device that is supported by the Garnet VM. Thanks Lynoure Braakman for pointing out my narrow mindedness on this point =:). Continue reading

Posted in coding | Tagged , , | 7 Comments

Fixing EBN isues.

Study has started again, which means that my attention is (slowly) shifting from KDE hacking to all kind of study related things. Which is not that bad as I have some quite interesting courses (information security, advanced c++ programming and computer vision). Normally this means for me that I don’t have time (but more, no energy) to do very much KDE hacking. However I found some task which can be done relative easily in between. Bringing down EBN  issues. I have worked out a lot of issues in KPilot already and currently I’m giving a stab at KMail.

While it might seem very dull, which it is in some sense, it is also very satisfying. Not in the first place to see the numbers going down on EBN itself ;) . But when working on the issues you also see code transforming, becoming more uniform and in some cases more readable. I don’t know what it is but I just enjoy seeing beautiful code. (Might be due to the book Google send me this summer, and by hanging out too much with [ade] in the past).

One of the things I noted in KMail (but I’ve seen this also in KPilot) is a quite inconsistent use of header inclusion. This happens in two areas. The first is that in some header files no forward declarations are used even when that is possible (when only pointers or references of that type are declared).


#include <QtModule/QClass>

class MyClass {
QCLass *mVariable;
void method( const QClass& c );

The second thing is that the include lists are a real mess in some files. Sometimes even k includes and Q includes are mixed up. What I would like to see is having a preferred order for includes in each file. Let’s say: own headers, project headers, k headers, q headers and other headers. And the next thing I’d like to see would be to have the includes sorted alphabetically. These are in fact relatively small changes but I think most of the headers and cpp files can be designed to look that way. This really would make it easier for (new) developers to see in a glance which classes/methods are actually used and also to see if the class he wants to used is included already. Another little annoyance I did see was mixing up (between files): #include KMyClass and #include "kmyclass.h". I guess most of these things are not new for the readers of my blog and still we (including myself) make really a mess of things so now and then =:).
This wasn’t meant as some rant on the kmail coding style or something. Just wanted to share some thought about this in general. Please let me know if you have some useful comment on this subject. Currently there is unfortunately not a check for unneeded includes in headers in krazy, but this would be very nice. If you have perl skills please give winterz a hand on this. p.s. Even this can snoop a lot of time which should be spend on my study =:S. p.p.s. If you find issues in KMail due to EBN Fixes I did, please drop me a mail at: [b DOT broeksema AT home DOT nl] and I’ll try to fix or revert the change if necessary. p.p.p.s. The line edits in KMail will soon have that cool clear button…. Continue reading

Posted in coding | Tagged , , | Comments Off

OpenSync meeting in Berlin

This weekend I have been in Berlin at the KDAB office to meet with the OpenSync guys. I arrived Friday at around 20:00 in a really nice atmosphere. Volker and Tobias where already there as well as some OpenSync guys. Volker had enlightened the OpenSync guys about Akonadi by the time I arrived. We went for a dinner first and after that we got an introduction of OpenSync from Daniel Gollub. They are currently working on a new release so there’s a lot of API breaking going on currently. However most of the functionality seems to get in shape right now.
As a side note, breaking API seems to be part of their regular development work. I really hope that they can get to some stable API which won’t change with every release they do. This issue seems to be a bit of a pain currently.
I gave a short talk about the current status of KPilot and we looked a bit what OpenSync could mean for KPilot. It became (again) clear that KPilot has a couple of features for which OpenSync is really not designed. However the baseconduit framework I created is quite like OpenSync (at least in the functionality it has, the designs differ some). Currently I’ll not start on integration of OpenSync into KPilot because what we have now is working reasonably well. When however KitchenSync is ported to the new API and the opensync palm plugins too, then I might give it a try to delegate the actual syncing to KitchenSync/OpenSync.
Volker started working on an OpenSync Akonadi plugin. The basic code is already there but unfortunately not yet very useful due to missing/not working functionality in the OpenSync library.
Further more we got a lot of information from Michael Bell, who is working on the libsyncml library. He gave us a nice overview of the architecture and also a guide to get started. The guide contained a lot of details and pitfalls, so much that I loosed track a bit. Well I guess that’s the other side of the coin when you want to support every device out there.
All in all, it was a great atmosphere, really good coffee, a lot off cookies, a lot of hacking, a lot of strange music coming from the streets, a lot of good food and a lot of fun!

On the KPilot front: I’m currently working on a “not so nice thing to do”. Cleaning up the huge amount of krazy warnings. This is kind a dull job but it is annoying me for ages now. Currently KPilot is not the black sheep of kdepim anymore but I want to get that number as low as possible (0 preferably). I also still have some items on my to do list regarding the Akonadi code and the base framework. Hope I can get to that soon. Continue reading

Posted in coding | Tagged , , | 7 Comments

Wrapping up: GSoC and Akademy are over already.

Akademy is over, GSoC is over, so finally some time to cool down =:). Well, I guess this is more a sign that real life will take over very soon again. Within a week or two the college year will kick off and whatever more. However, I really have had a great time. It was again a real pleasure to work with vanRijn on my summer of code project. Like him I also am very pleased with the results of the summer of code. As mentioned earlier there is still some polishing left to do but I think that KPilot is in a reasonable good shape right now.

I cannot do anything else than confirm all the positive feedback given on Akademy. It was the second time I attended but this year it was just awesome! So much interesting talks, great ideas and wide visions for KDE. I also really enjoyed meeting all those cool people. It was just great to discuss code, problems in the Banlieus of Paris, eating habits, funny things in spoken language and what not more. So thank you Armin, Alex, Andre, David, Kevin, Patrick, Volker, Thomas and all of you who I have spoken but who I don’t remember right now. I truly enjoyed!

Akademy has got a tail for me as I was invited to join the KDE/OpenSync meeting in Berlin next weekend. I’m really exited to see what we can get out of that. Now KPilot is getting into shape and the amount of types of devices on my desk is growing I’m really looking forward to have a desktop which is prepared to sync all me personal stuff between those gadgets and my normal pc. I have a growing conviction that a desktop should be just able to sync when I pop in one of my devices. So lets see what can be done to make this happen.

note to self: I should start reading the OpenSync and KitchenSync code. Continue reading

Posted in coding | Tagged , , | 2 Comments

The future of KPilot

Now the Google Summer of Code approaches its end I’d like to share some thoughts on KPilot. First of these is that I was slightly surprised by the responses I got on an earlier blog. I know that there are still KPilot users but in some sense I did not really believe it. So if you are a user and want to use it in KDE4 please let us know. It is really encouraging when getting some nice message from someone out there . I know, we are not so good in keeping our website up to date, so you might also want to drop by in #kpilot on irc.freenode.org or at our personal websites. Furthermore, you can follow development activities at the KPilot ohloh site.

That said, what are the plans for KPilot? Our major goal is to release a new KPilot version with KDE 4.2. So that will be in six months or so. In the past couple of months we have done a lot of work to get KPilot syncing with akonadi and the results are promising. At the end of this GSoC (that is, after two weeks from now) we have “working” todo, contacts and calendar conduits. Working is quoted as polishing of the KPilot code as well as some kdepim/akonadi code is needed. How usefull KPilot will be at release time also depends a bit on the other kdepim applications as none of them (except mailody?) are ported to Akonadi yet if I’m correct. However at 4.2 that has changed (we hope) and therefore the following conduits will be released then:

  • calendar
  • contacts
  • file installer
  • memofile
  • time
  • todos

The results of this GSoC and the previous one is that most conduits share way more code than they did before. This should make the maintenance and bugfixing a bit easier for us.

We also slimmed down our focus. KPilot will be the application to synchronize your KDE PIM data with your palm device. This means that we will remove the old database viewers that where in it in the 3.5 series. The viewer code is old, not good working, does not have any real purpose and we do not have time/energy to work on that. In return of that you will get category synchronization for each record based conduit. I have been thinking a bit about the gui of KPilot but have no clear idea yet how it should look. So good ideas on this are welcome.

Besides the “getting things working again” again, the codebase needs a lot of clean up. But first things first. All in all, this should make KPilot should survive another KDE series. Continue reading

Posted in coding | Tagged , , | 17 Comments