Tag Archives: clang

Its the time of the year again, spring. Meaning, time for the big clean up. Not the house of course, we leave that to the people feeling themselves called to clean houses. I’m talking about code. Over time we, as in developers, come to the conclusion that this neat API we wrote is not as neat as it looked in the first place. So we redesign our API and BIC lovers as we are, mark methods that should not be used anymore as deprecated.

This is of course very nice, but how do we actually get an overview of the methods that are marked as deprecated, more important how do we get an overview of how often deprecated methods are still in use? Well, kdevcpptools to the rescue. As I blogged before I worked on a C++ query and transformation engine for KDevelop as part of my Msc thesis. After an awesome lot of stabilization work of Milian "code monkey" Wolf, the plugin really is in quite a nice state at the moment and I would say quite usable. In the rest of this post I’ll show how to use it by taking kdepimlibs as an example.

The first thing to do is to create a query file containing the queries for the deprecated functions. The query (and transformation) file is a relative simple XML file describing the queries and optionally transformations. There are some example files in the git repository, but don’t jump to that yet as I want to show a cool feature Milian implemented.  Lets find the deprecated functions first. Okay, no fancy tools for that (yet?), so just a plain ack-grep on "KDE_DEPRECATED" in kdepimlibs. This gives me a lot of files containing this macro. We startup KDevelop and create a new empty query file with the following content:
?xml version="1.0" encoding="UTF-8"?
  porting-description
    queries
   /queries
  /porting-description

(Yes, the formatting sucks, don’t know how to make drupal output XML in a sane way). Next, open one of the files that showed up in the results of your grep and go to one of the deprecated functions. Make sure that you have the kdevcpptools tool view enabled and right click the function. The context menu shows you this wonderful option: Copy XML query:

Copy the resulting XML from the clipboard into your file and there you go, its that easy. I did this for whole kdepimlibs, you can find the file here. First observations: we’ve 47 deprecated functions in whole kdepimlibs and we really have  a shitload of date string formatting functions in kdepimlibs, so…. if you need one…..

When you’re done putting in your queries in there, you can open the XML file in the kdevcpptools plugin. This enables you to query a project. I did this for kdepim, meaning, the result I get are all uses of deprecated functions from kdepimlibs in kdepim. This result was so to say pleasantly surprising. Have a look at the next picture:

In the lower part of the screen you see an overview of the results for kdepim. Vertical axis are files horizontal axis are queries. First observation here: there are surprisingly few uses of deprecated functions. (Note: we’re only talking here about functions, explicitly marked as deprecated). Second observation: though not particularly interesting for such few results, the curve you see in the total column looks promising. I.e. most uses in only a small part of the files. Meaning, only a couple of files need "major" changes. This must of course be seen in perspective, this reasoning is not particular interesting when you only have seven hits for a ~700KLOC code base. Would you had have a similar curve for ~8000 uses this would of course already make a lot more sense. The curve in the total column will actually tell you how the migration work is distributed over the code base you’re querying.

So, my initial idea was to do a bit of clean up in kdepim but it turned out that this is hardly needed with respect to this specific area. The good thing is that we now know that it isn’t needed. This means that as soon as KDE5 development kicks off we can get rid of these deprecated methods without a lot of work (in kdepim at least). I’d really encourage anyone interested in these kind of analysis on his/her code base, either from a lib perspective like I did, or from an application perspective (anyone interested in building up a query file for qt3support and query the various kde modules for it?) to checkout the plugin[1] and play with it. You can find me and Milian in KDevelop if you need help on getting started with the plugin.

Next step is using the transformation engine to do (at least part of) the migration work automatically. I hope to blog about that soon. But first a thesis to finish…..

[1] http://gitorious.org/kdevcpptools Continue reading →

(Warning: long post coming up. Read on if you are interested in performance of the various Akonadi database back ends).

Last weeks I’ve been looking (again) whether or not it was possible to create a working sqlite back end for Akonadi. The last time I tried was around august last year and by then sqlite just wasn’t able to meet the multi-threading requirements that Akonadi has for its database back ends. A couple of sqlite releases later things seem to have changed. I managed to clean up some problematic code paths in Akonadi server and voila, we’ve a sqlite back end that is on par (unit test wise) with the mysql back end.

There are some catches of course. The first being that the default sqlite driver that comes with Qt does not work. It uses the sqlite API in a non-blocking way. So I had to adjust that to make the driver consistent with the sqlite documentation which states that when sqlite calls return BUSY, you have to wait a bit and try again. This custom driver can be found in kdesupport/akonadi/qsqlite.

The next catch is related to performance. Though we did not had the numbers until now, it was expected that sqlite would perform worse compared to mysql. Given that we have another back end, postgresql and are working on yet another one: a file system back end, it seemed a good time to do some benchmarking. So I brushed up the item benchmark a bit and performed the benchmark for all back ends. The benchmark tests the performance of Akonadi over two dimensions. The first dimension is the number of items and the second dimension the size of the payloads. We used the Qt data driven test framework and added rows to it like: ${number-of-items}-${payload-size}. Then we benchmark the important actions of Akonadi, which are creation, modification, fetching and deletion of items. This enables us to see how the performance scales with the two dimensions.

Before getting to the results I first have to make a note about the file system back end. This one is designed to be a fall-back for large payloads. The idea is that database actions become too slow for large payloads. So at a certain offset Akonadi doesn’t store payloads in the database but in plain files. The benchmark for the file system back end is set up to always write the payload to the file system. This enables us to find out the offset that gives best of two worlds, i.e. fast performance for small payloads by using the database and fast performance for large files by using the file back end. (Note: currently the file system back end is disabled by default, there still are some issues with it that need to be sorted out).

Sooooo, show me the numbers, I hear you thinking. Well, here you go, lets start with item creation:

The image show the results scaled logarithmic on the x-axis (time on y-axis, but relative due to logarithmic scaling on x-axis). As you can see the, the files system back end (akonadi-fs) is hardly influenced by the file size, only by the number of items. For the other back ends we see that file size has influence on the performance too, but roughly scale  linear. We also see here that sqlite does not perform as well as the others. Lets have a look at the absolute numbers:

The y-axis now shows time in msecs. The graph now shows us clearly that when items get larger and the number of items grows too, sqlite is clearly outperformed by all other back ends. We also see that databases in general don’t cope well with large payloads, which is exactly the reason to provide a file system back end too. First conclusion, don’t use sqlite unless you have very strong restrictions on your environment. (Which is the case when running Akonadi on Windows CE for example, where the number of processes is extremely limited and which we are working on here at KDAB). Still not convinced about sqlite performance? Okay, lets have a look at one more. Item modification:

Again, we see that sqlite is outperformed by all other back ends as soon as the payload size  becomes large. When the payload size grows we also see that only the file system back end doesn’t start to grow exponentially like the database back ends do. So, sqlite works, might even work fast enough for you, but is definitely not fast enough for the general use case Akonadi was designed for in the first place: handle many large items. Again, unless you have very strict requirments on the environment where Akonadi is used.

The last thing I want to show is a benchmark with different payload sizes for 2500 items. This makes it easier to find the cutoff value for the file system back end. I.e. what payload size should be used to store an item using the file system back end in stead of the database? First the images (I only compared mysql and fs to have slightly clearer graphs, you can find the full results at the links posted at the end of the blog):

For creation and modification the cutoff value seems to 8 KB. However, fetching, which is also an often performed operation, has a cutoff value of 512B. A good trade-off between those two is probably around 4 KB.

So that’s all for now. Short recap. The sqlite back end for Akonadi seems to work, though its about ~5 times slower. Also, there are already some problems reported, so it still should be considered as a work in progress. Work on the file system back end is ongoing but seems promissing and with the right offset and file system/database combination (i.e. mysql) we get best of both worlds. Thanks for reading!

Links to the full (interactive) results:

1. Multiple item counts, multiple sizes
2. 2500 items, multiple size

Update: For a better comparison between database and fs back ends I added another benchmark which also uses 2500 items but only goes up to 8 KB payloads. Check the results here:

2500 items, multiple size, only up to 8K
[1] The title is meant to be a pun. Every now and than people pop up on the ML who think that sqlite is not a *full blown* (whatever they mean with that) database. Let me ensure you, it is. It supports SQL at a similar level as mysql, it does transactions and multithreading, it just tends to be smaller (and here I mean the library itself, not the database) and it does not run in a seperate process but it therefore has its limitations too. Continue reading →