Measure the code base

Every once in a while software needs to be ported. This seems to be more a rule than an exception. New versions of libraries are released, libraries are not maintained anymore or newer libraries outperforming the one currently used appear. So, there you’re sitting as a developer staring at the (potential huge) task lying in front of you.

Or imagine you’re the author of a library yourself. How do you determine how to split up your library in logical modules? Or where do you start when you want to optimize your library, given a couple of projects using your library?

These two scenarios are closely related to the research I do for my master thesis at KDAB. When doing work as described above, one would like to have tools that help him to estimate the work and where possible, to automate the work. For Java lots of tools are available for C++ however, it is a totally different ball game. Well, I guess most of you know why, parsing and analyzing C++ is, well, kind of difficult. There are good proprietary tools available, but part of my assignment was too evaluate how well a FOSS tool would do.

For several reasons, which I won’t outline here I chose to work with the KDevelop parser. It doesn’t do a full semantical analysis and unfortunately it doesn’t store all info I need during analysis but it gave me a descent starting point. The idea was to build a C++ query engine (i.e. find uses of type X, find uses of method X::y(), etc) and a transform engine (i.e. given an use of X::y(), transform it into foo<X>()). As said before, KDevelop doesn’t store all information needed to do a fully correct transformation but some useful transforms can already be done (more on that in another blog).

So, given a C++ query engine, a partly correct working transform engine and a (huge) porting job, what would we like to know and what can the plugin currently show? We start with putting in some queries which we can’t port (or even find) with our current scripts. We load the queries and run it on the project we want to port. First results:

In the tree you see an item for each file with as children results of the different queries run on the projects code base. Double clicking will bring you to the location in the editor. When a transform is defined for the query, selecting a query will enable the transform button and when transforms are applied you can view a diff. But well, as said the transform engine is a bit in a poor state still. However, at first we’re not directly interested in applying the transform. Remember, the customer didn’t sign any contract yet =;). So how do we know what to put in the contract in the first place? Lets enter the project view:

Here you see a more aggregated view of the results of the queries that where run. On the left side you see the files for which queries are run. On top you see which queries where run. Each cell contains the number of hits for a query in a file. Except for the first which shows the total of the row. Of course the results can be sorted for each separate column. So what can we learn from this picture? The first thing that is noticeable, when sorting on total is that the curve is broad at the top and getting narrow quite fast. Which means, given that the queries represent the amount of work to do, that most of the work has to be done in only a couple of files. Now, what kind of work do we need to do mostly? Lets sort by one of the queries:

Hey, that curve there on the right seems to correlate with the total curve. Hmm, so most work is due to query X. Aaaah, we know that this particular piece of code is [very easy|hard|very painful] to port so we need to bill our customer [a bit|a lot|an insane amount]. (But of course relative to the amount of file which are affected =;) ).

Up to now we’ve had the files as rows but it is also possible to switch the headers, which of course also changes the meaning of the image. Lets see:

Now we have the queries on the right and sorting on total gives you an overview which type/function/other construct is used most in the project you’re analyzing. Doing this for several projects using the library you’re maintaining/developing can help you in finding out which parts of your library you should optimize first, what the impact would be of changing the signature of that particular function or how to split up your library in modules when it gets too large.

Next thing I’ll be working on is visualizing the impact of transforms on the code base. Especially to show which parts of the documents are modified by the defined transform and if there are overlapping transforms. More on that later.

I didn’t release the code yet as its far from mature yet. However, if you feel brave and want to play with it, drop me a mail and I’ll send you the source. It will eventually become available under some free license (I should’ve read [ade] blogs better, I would have known which).

P.S. For who wants to know, these kind of views are called Table Lens views which are very powerfulf for finding trends.

Tagged , | 7 Comments

All good things come to an end….

even so KPilot. After having been part of KDEPIM for years it has been moved to tags/unmaintained/4/kpilot today. For months now no real development happened, there was not even bug fixing. I guess we have to draw our conclusions from that. Jason already said he lost his interest after buying a new phone about a year ago, while I myself still do have a palm I actually don’t use it anymore. KPilot has been a fun ride for me. Adriaan de Groot once dragged me in when I requested a new feature. It where my first in KDE software development. It showed me the trickeries of syncing pim data between two devices and it got me in touch with Akonadi. I even think it was one of the first pim applications to make use of Akonadi. So this means that from KDE SC 4.4 on users won’t be able anymore to sync there palm with the PIM suite again. I’m sorry for the inconvenience for those who still did (I did see a bug report coming up every now and than). If anyone is still interested in getting it up and running again, the code can be checked out via svn co svn://anonsvn.kde.org/home/kde/tags/unmaintained/4/kpilot I’d like to thank Adriaan and Jason for guiding me on my first steps. For the fun time we’ve had working on KPilot or whining about the pity state of device synchronization on Linux in general (still today it seems that there’s not really a rock solid solution available). I’d also like to thank Robert and Doug for their contributions and of course the people who started KPilot in the first place like Dan Pilone and Rainhold Kainhofer. We’ll have to see what the future brings. But with all those new powerful devices, which are able to access various groupware solutions I doubt if anyone wants to really invest in a computer <-> phone/handheld solution. The curtain falls for KPilot, good bye old friend, fare well!

Tagged , , | 20 Comments

krazy2 / ebn developments.

BBROEKSEMA ARE YOU DEAD?? NO MAN!!!

I just have been busy finishing my master. Which, I can say, is progressing nicely. I’m winding up my last two courses and preparing for my move to Berlin, where I’ll do my master thesis research at KDAB.

However, that’s not what I wanted to blog about. I wanted to write an update about the work I’m doing on the english breakfast network, krazy2 and the integration of a C++ parser in the framework. Due to one of the courses I followed I got interested in software maintenance, evolution and quality. Within KDE we have all kind of policies and measures in place to guarantee some kind of quality like coding standards, commit policies, program lifetime policies and what not more. Another framework we also have is the englishbreakfastnetwork which most of you probably know by now.

EBN shows the results of several tools that are developed like krazy2 and documentation sanitzers, which are ran against the KDE code base. So, what’s the actual use of this? Most of the checks are not that advanced. In this [1] paper it is even concluded that the presence of public coding standards and shared practices does not seem to have a significant influence on the quality in comparison to projects where this isn’t the case. So what is the use of krazy2/ebn in it’s current form than? I think that its main function, in its current form, is to provide a good point for starters who want to help out with easy coding/documentation tasks.

However I think that we can get more out of the EBN framework. To get more we also need to store a bit more. And that’s one of the things I’m currently working on. Currently a new tool is in development. It is (becomming) a framework which can run different tools on a software component. It’s a bit like the krazy2ebn script, but more flexible as it will support not only krazy2 but also other checkers to be plugged in. The framework will store not only results of the checkers (and their plugins) in a database but also the versions of the checkers/plugins that are used for a specific run. The main differences in storing the results will lie in the fact that the new framework will store detailed result information of all runs done. Storing all this information will enable us to not only see the current state (in terms of issues found by the tools) of the software, but will also enable us to relate checker evolution to the quality. It will also be possible to search for modules/submodules that shows tendency of bitrot (e.g. modules which have only a growing amount of issues over a specified period). This whole framework is in an early state and comments/suggestions are of course welcome.

Another thing I’ve been working on with a collegue student is integrating a C++ parser into the quality framework. I’ve played with several parsers and currently I’m using SolidFX from SolidSource IT [2]. This is a commercial parser but we can use it for free on EBN, due to the fact that one of my profs at the university is the author of it. It has some really nice features like:

It does full syntax analysis of several C/C++ dialects (including the one(s) supported by the gcc compiler suite); own preprocessing; and an almost full semantic analysis along the C++
Standard, including most of the template instantiation/specialization stuff. Tested on things like Boost, Python, Coin3D, wxWidgets, parts of the Linux kernel, also Windows code. There’s also a mechanism for error recovery in case of lexer/parser/type-checker errors due to wrong code/missing includes.[3]

Currently we’re Implementing a check using this parser which checks if a class overrides non-virtual methods of (one of its) base classes. We’re interested in implementing other checks, of which you think are useful and require a full fledged parser. So feel free to leave a comment or drop me an email. Please describe precisely what you want to check and also why. I’m also idling in #ebn btw. That’s all for now folks, thanks for reading.

[1] http://istlab.dmst.aueb.gr/~george/pubs/2007-PCI-GKSLVS/paper.pdf
[2] http://www.solidsourceit.com/products/SolidFX-static-code-analysis.html
[3] From the author, prof. Alex Telea

Tagged | 8 Comments

Some love for KGpg

For a course I did (Information Security) we had to create our own gpg key and do some signing and encrypting, well the normal stuff you seem to do with these things. As a side note, I did already did this twice some time ago but forgot the passwords of my private keys because I never used them so now I have two dangling accounts on the pgp servers *sigh*.

However, this made me have a look at KGpg which in turn made me cry. Heh, it seems in a not too good state. Having a bit of time, and being really annoyed by issues in the key server configuration page made me start hacking on it a bit. I fixed (hopefully) all issues in there. I have done this in trunk, maybe I should also back port this to 3.5 as the issues seems to be there also.

I don’t know if there is a maintainer currently and if there is I hope I didn’t pass you by. I did a quick search on bugs.kde.org but did not find any bugs related to this. Maybe I’ll take a stab to other issues I find too if I a) really start using gpg, b) have some spare time, c) find more issues (although that won’t be a constraint I think).

And last, it is not totally clear to me if there is any relation between kgpg and kleopatra, and if so what that relation is. If some kind soul could enlighten me on this….. that would be great.

Posted in coding | Tagged | 8 Comments

Code cosmetics and quality.

About two weeks ago I added the postfixop check to krazy. This checks for postfix uses of ++/– usage. Initially I constrained the check to for-loops. However this lead to a huge amount of issues because also postfix usage of ++/– operators on simple types where marked as an issue. After some discussion with Eckhart Woerner and Allen Winterz I relaxed the check more so that usage on simple types is also not taken in account anymore.

Although postfix usage on simple types won’t gain you any/much I still think it’s a good thing to consequently use prefix wherever appropriate. However that has more a cosmetic reason (which I why I took it out actually) then a perfomance reason. Consider for example the next piece of code (and yes I’ve seen something like this somewhere in the KDE codebase):

QList::Iterator it = list.begin();
for( int i = someNumber(); it != list.end(); ++it, i++ )
{
...
}

I realize that it’s partly a matter of taste if you like this kind of code snippets in your code base or not. However for people who never have had some in depth programming course it might look like if it is arbitrary what you choose.

Another thing I’ve been thinking about last weeks is "What is the importance of cosmetic changes in relation to code quality". When browsing through the source of KMail while fixing issues reported by krazy I see a lot of things which make me cry. For example defining variables which are used only ~25 lines later or in 3 different context. These kind of things make my fingers itch to rewrite such parts. Well, don’t be scared, I didn’t….. if it ain’t broken don’t fix it, right? But I still can’t find a good answer on the question: is that sort of code broken? In some sense it is in my opinion. When I as a newcomer to the KMail code look into some sections I really have to read that twice or sometimes even more times to get some vague idea what is happening over there. I think that is something that can be avoided (to some extend of course, as not everything in life is as easy as we would like it to be). All this made me think that it would be a nice to have something in addition to the normal krazy checks which checks for situations where some cosmetic change might improve the readability (and therefore the quality?) of the code.

Well, leave me a comment if you have some noteworthy ideas about these topics.

p.s. KMail only served as example here, I guess there are more places which have similar issues. It’s just that I was browsing through KMails code while these thoughts came up.

Posted in coding | Tagged , | 18 Comments

Help us testing KPilot using your n810/n800/n770

Dear lazyweb (at least the KDE part of that, having an n810 Garnet VM supported device),

I just heard from Jason “vanRijn” Kaspaer that there is a VM for the n810 which makes it possible to run Palm applications on your n810. I just gave it a try and its just awesome! When you install the VM you instantly have the addressbook, calendar and todo applications available. As these are currently the only supported applications in KPilot this should be enough ;) . I tried to install, configured kpilot, pressed the hotsync button in the VM and guess what? It just seem to work. So how to turn your n810 in a PIM device which can be synced against trunk?

  1. Goto: Garnet VM and download the GVM on your n810 (mind you, you need to register).
  2. configure KPilot to look for device: net:any (kpilot -s -> General Setup -> Device -> Pilot Device
  3. Make sure you have some valid resource configured for the conduit you want to test (Note: first try to create a new resource in Akonadi and don’t use your production resources. Note 2: Don’t tell me I didn’t warn you. Note 3: If you use the resource bridge of Kevin Krammer you can even add it to the kde PIM like kaddressbook.)
  4. start adding some items on one or both sides
  5. Start KPilot
  6. press the hotsync button in the GVM (Note: you have to leave the specific application e.g. addressbook to see that button).

That’s it, happy syncing :)

If you have Issues please let us know and no, no screen shots today. Try out yourself! For issues please drop by in #kpilot, #kontact or #akonadi (whatever is most appropriate for the issue).

Update: Of course this works for every Nokia device that is supported by the Garnet VM. Thanks Lynoure Braakman for pointing out my narrow mindedness on this point =:).

Posted in coding | Tagged , , | 7 Comments

Fixing EBN isues.

Study has started again, which means that my attention is (slowly) shifting from KDE hacking to all kind of study related things. Which is not that bad as I have some quite interesting courses (information security, advanced c++ programming and computer vision). Normally this means for me that I don’t have time (but more, no energy) to do very much KDE hacking. However I found some task which can be done relative easily in between. Bringing down EBN  issues. I have worked out a lot of issues in KPilot already and currently I’m giving a stab at KMail.

While it might seem very dull, which it is in some sense, it is also very satisfying. Not in the first place to see the numbers going down on EBN itself ;) . But when working on the issues you also see code transforming, becoming more uniform and in some cases more readable. I don’t know what it is but I just enjoy seeing beautiful code. (Might be due to the book Google send me this summer, and by hanging out too much with [ade] in the past).

One of the things I noted in KMail (but I’ve seen this also in KPilot) is a quite inconsistent use of header inclusion. This happens in two areas. The first is that in some header files no forward declarations are used even when that is possible (when only pointers or references of that type are declared).

So

#include <QtModule/QClass>

class MyClass {
...
QCLass *mVariable;
....
void method( const QClass& c );
...
}

The second thing is that the include lists are a real mess in some files. Sometimes even k includes and Q includes are mixed up. What I would like to see is having a preferred order for includes in each file. Let’s say: own headers, project headers, k headers, q headers and other headers. And the next thing I’d like to see would be to have the includes sorted alphabetically. These are in fact relatively small changes but I think most of the headers and cpp files can be designed to look that way. This really would make it easier for (new) developers to see in a glance which classes/methods are actually used and also to see if the class he wants to used is included already. Another little annoyance I did see was mixing up (between files): #include KMyClass and #include "kmyclass.h". I guess most of these things are not new for the readers of my blog and still we (including myself) make really a mess of things so now and then =:).
This wasn’t meant as some rant on the kmail coding style or something. Just wanted to share some thought about this in general. Please let me know if you have some useful comment on this subject. Currently there is unfortunately not a check for unneeded includes in headers in krazy, but this would be very nice. If you have perl skills please give winterz a hand on this. p.s. Even this can snoop a lot of time which should be spend on my study =:S. p.p.s. If you find issues in KMail due to EBN Fixes I did, please drop me a mail at: [b DOT broeksema AT home DOT nl] and I’ll try to fix or revert the change if necessary. p.p.p.s. The line edits in KMail will soon have that cool clear button….

Posted in coding | Tagged , , | Comments Off

OpenSync meeting in Berlin

This weekend I have been in Berlin at the KDAB office to meet with the OpenSync guys. I arrived Friday at around 20:00 in a really nice atmosphere. Volker and Tobias where already there as well as some OpenSync guys. Volker had enlightened the OpenSync guys about Akonadi by the time I arrived. We went for a dinner first and after that we got an introduction of OpenSync from Daniel Gollub. They are currently working on a new release so there’s a lot of API breaking going on currently. However most of the functionality seems to get in shape right now.
As a side note, breaking API seems to be part of their regular development work. I really hope that they can get to some stable API which won’t change with every release they do. This issue seems to be a bit of a pain currently.
I gave a short talk about the current status of KPilot and we looked a bit what OpenSync could mean for KPilot. It became (again) clear that KPilot has a couple of features for which OpenSync is really not designed. However the baseconduit framework I created is quite like OpenSync (at least in the functionality it has, the designs differ some). Currently I’ll not start on integration of OpenSync into KPilot because what we have now is working reasonably well. When however KitchenSync is ported to the new API and the opensync palm plugins too, then I might give it a try to delegate the actual syncing to KitchenSync/OpenSync.
Volker started working on an OpenSync Akonadi plugin. The basic code is already there but unfortunately not yet very useful due to missing/not working functionality in the OpenSync library.
Further more we got a lot of information from Michael Bell, who is working on the libsyncml library. He gave us a nice overview of the architecture and also a guide to get started. The guide contained a lot of details and pitfalls, so much that I loosed track a bit. Well I guess that’s the other side of the coin when you want to support every device out there.
All in all, it was a great atmosphere, really good coffee, a lot off cookies, a lot of hacking, a lot of strange music coming from the streets, a lot of good food and a lot of fun!

On the KPilot front: I’m currently working on a “not so nice thing to do”. Cleaning up the huge amount of krazy warnings. This is kind a dull job but it is annoying me for ages now. Currently KPilot is not the black sheep of kdepim anymore but I want to get that number as low as possible (0 preferably). I also still have some items on my to do list regarding the Akonadi code and the base framework. Hope I can get to that soon.

Posted in coding | Tagged , , | 7 Comments

Wrapping up: GSoC and Akademy are over already.

Akademy is over, GSoC is over, so finally some time to cool down =:). Well, I guess this is more a sign that real life will take over very soon again. Within a week or two the college year will kick off and whatever more. However, I really have had a great time. It was again a real pleasure to work with vanRijn on my summer of code project. Like him I also am very pleased with the results of the summer of code. As mentioned earlier there is still some polishing left to do but I think that KPilot is in a reasonable good shape right now.

I cannot do anything else than confirm all the positive feedback given on Akademy. It was the second time I attended but this year it was just awesome! So much interesting talks, great ideas and wide visions for KDE. I also really enjoyed meeting all those cool people. It was just great to discuss code, problems in the Banlieus of Paris, eating habits, funny things in spoken language and what not more. So thank you Armin, Alex, Andre, David, Kevin, Patrick, Volker, Thomas and all of you who I have spoken but who I don’t remember right now. I truly enjoyed!

Akademy has got a tail for me as I was invited to join the KDE/OpenSync meeting in Berlin next weekend. I’m really exited to see what we can get out of that. Now KPilot is getting into shape and the amount of types of devices on my desk is growing I’m really looking forward to have a desktop which is prepared to sync all me personal stuff between those gadgets and my normal pc. I have a growing conviction that a desktop should be just able to sync when I pop in one of my devices. So lets see what can be done to make this happen.

note to self: I should start reading the OpenSync and KitchenSync code.

Posted in coding | Tagged , , | 2 Comments