Information Bits

I am currently reading Gunter Dueck’s newest book and I found a reasonable explanation why it is currently so hard to find good Software Developers: he describes a interesting article by George A. Akerlof, The Market for “Lemons” Quality Uncertainty and the Market Mechanism, which simplified tells us why asymmetrically information can destroy a market. So my thoughts here are that this is maybe a reason why it is currently very hard to find good people: most companies have no knowledge about how to distinguish between different qualities of work and why it can make sometimes more impact as “process re-engineering”. For this reason, they do not know how they can evaluate the market and they expected on the over side that a lot of “lemons” are there, mostly they compare by price. The article tells us that in this situation the market loose the really valuable people, because they either have to sell them self on same low level or they have to leave the market at all (and become managers, consultants or freelancers).
May be this thought is a bit far fetched but it is definitely the best explanation I’ve found so far because the currently the market for above sub-standard Software Developers is in a imbalance.

These days a lot happens in the field of the Microsoft Research Project Singularity: the source is finally available from Codeplex! Why is this project exciting? It is a research playground to test ideas such as using virtual machines like the CLR on the level normally occupied by C or assembler (hey, device drivers in C# are definetly more readable). Also a lot of concepts such as contracts are inherited from Spec# and used for guarding most system services. Because until now only interviews take place (the last one on Software Engineering Radio with Markus Völter and Galen Hunt), so look at actual working code is amazing.
I’m sure anyone who is interested in novel operating system ideas and want not explore something like Minix should download the source. And I’m sure, also any other developer will get some new ideas from the source.

For efficient software development you need a reliable and very flexible Continuous Integration server. Most people know CruiseControl(.Net) but everyone who used it knows it is very flexible but has it’s limitations and well, it has a bit “uncool” fronted. After searching for a new one I tried out Jetbrains TeamCity once again. The first version had not provided enough new features to make a switch, but the current version 3.0 (and upcoming 3.1) has features not easily found elsewhere:

• The concept of build agents: you can install small Java based build agents on various platform (in my case a x64 Windows, a x64 Linux and a IA64 Linux based server), all managed by a single build server. Checkout can take place on the server or on any agent host if SVN is installed there.
• Remote builds: any user can trigger a remote build from his workstation without commit his code to SVN.
• Targets ANT, Maven2, NAnt, MSBuild, JUnit, TestNG, NUnit, Visual Studio 2003-2008, IntelliJ projects and simple shell scripts.
• The Web GUI is cool and workable (as well as the integration in Eclipse, Visual Studio or the Tray).
• Can integrate third party reports and integrate with any build script.
• And the professional version is free

So give TeamCity a try if the professional version is enough for your purpose. It solves a lot of problems very elegant and you definitely need less time to manage it.

Microsoft has released it’s Parallel Extensions to the .NET Framework 3.5 as CTP (you can found more about it on Joe Duffy’s Blog). The library provides some interesting features: one is a easy way to execute LINQ queries in parallel and simple loops. The other thing is that a simple to use Task execution library which, and this is the most interesting part, is not build up on the existing ThreadPool class instead a new Scheduler based on “Work-Stealing” is realized. Why is that important? Normally you use threads which have, although not so much, some overhead in creation and management. This plays no role for long running tasks but it is certainly not very effective for very small tasks as they exist if you want, as example, extend a sorting algorithm. On the other side you have often to sync the work of some parallel task so you use resettable events, in C#, and wait for them. The problem is that they are not cheap because they are using actively system resources, e.g. handles, which are somehow limited. The new library does not need event for this job, so the only resources used are the threads you assign to the TaskManager class. This is certainly not a production ready library but a very interesting one because it makes development of effective multi-core aware data structures and algorithms very easy. The Java world has already for some years a library like this: Doug Lea’s Concurrency Library has a Fork/Join framework which is also based on a “Work-Stealing” scheduler but I doubt a lot of people knew about it. The framework will be part of Java 7 which, sorry to say, will not be available until 2009.

The book Programming Collective Intelligence from Toby Segaran is a practical useful guide through the most common data mining, more exact classification, algorithms. The book covers the traditional algorithms as decision trees, naive Bayesian classifier, neural networks, clustering and not so common ones as support-vector machines  and non-negative matrix factorization. The last one was new for me. The first paper seems not to have surfaced before 2000 so it is relatively new technique and has shown very good results in the case of feature extraction of large numerical spaces. Also optimizing functions like simulated annealing and genetic algorithm are mentioned. Interestingly, there is also a small example for genetic programming.
All algorithms are explained with examples and small programs in Python. The only thing I’ve missed is a mathematical representation in addition to the explanation itself, but at least all utility functions are explained more formal in the appendix.

On SIGGRAPH 2007 Shai Avidan presented a amazing Demo how to resize, shrink and enlarge, images without loosing important content information. Everyone was amazed by this technology but the paper is available and now, in fact, there are some implementations of the algorithm, Seam Carving.
The results are really fascinating because the algorithm works without knowledge about the content of the image. So check out the link on Mike Swanson’s blog, he wrote also a first C# implementation.

If you are on the Java Side of Life you are certainly very jealous on the C# world because with the coming of LINQ you have a very nice and useful framework to resolve query tasks on any thinkable object structure (or XML, SQL, Amazon, whatever). Because Java is based on byte code and not on a flexible intermediate language as it is the case for the CLR, there will no big change in the near future for supporting new language features (but hopefully the generics will be repaired …). I personally think with the availability of JRuby we will see some cool extensions but at least for LINQ there is a new interesting project from Anders Norås: Quaere
This is a very nice project of an DSL with the API copied from LINQ and the use of ANTLR 3 to realize the same feeling and functionality. The projected is in it´s first steps, so there are some obstacles and not all is working as expected, but it is a nice peace of code and really worth reading. Hopefully it will mature …

Maybe Java will never have comparable structures as the expression tree which is the backbone of LINQ but maybe more elegant and useful APIs based on DSLs and JRuby will surface.

UPDATE: The new Quaere Home could be found at Codehaus.

Software development projects are always a challenge especially so if they are realized very fast and with the help of agile practices. By far the most important skills inside the team is the motivation to take the bait and the willingness for taking responsibility of his own work. Technical skills are necessary but by far less important. Skills are always teachable and every technical skill can be learned by everyone if he really wants it. But if you are not believing that you can successfully finish your work or it does not matter what you do, the project is nearly lost. Software projects are always a very uncertain territory and you have to change your path very often. For not getting lost it is essential that you know what your target is and you really want to reach it.

Currently I’ve the problem that my team has skilled and experienced members but the motivation to finish the project or to throw in more than the minimal acceptable amount of work. The project is already on a critical path but no one cares. The reason is in this case that trust in the success and the usefulness of the project was lost shortly after the start of the project (or never existed). The company management changed the path sometimes, feature creep is happening and there is definitely not much trust and expectation that the result will be finished in time and is valuable. So there a lot of external consultants, internal company struggles for influence and some unmotivated teams. So how could this problem solved? In my opinion only in that way that you have accept the current situation, appeal to the professional pride of the team members and try to finish all with less than the real existing amount of capabilities. After the finish line, full stop and rethink the whole eco system because it makes certainly no sense to go one step further.

This is the first time that I have the situation that the environment to realize a software project is so poisoned that the willingness to finish a project is nearly zero. I heard such stories from bog development shops as they could be found in large companies for internal software but it was new for me that you can paralyze also small teams completely. For the future I know now that the only way to solve the dilemma is to react on the first singes inside the team and fight back internal or external attacks on the moral. The last possible solution is to stop the project as soon as possible because otherwise it will become a tour de force …

I#ve found a interesting paper published at OOPSLA 2007 by Dan Grossman where he points to the similar patterns in the concept of Transactional Memory and Garbage Collection. The paper is not only well written by itself it is also a good source for finding more resources for both concepts. I also think that his claims are right: Transactional Memory will never need hardware support (as was similar claimed in the beginning for GC), that TM will need more time as expected to become mainstream and that it will not make concurrency programming much easier (but it will certainly help to prevent us from visiting the most silly traps again and again …)

I’ve stumbled over the Google Tech Talk of Brain Foote where he gives a talk about his famous paper Big Ball of Mud which he coauthored with Josef Yoder. The paper was published at the end of the last century, the Agile Movement starts and we all think that we have a great mess at most software development projects. The paper was and is a nice read and the question is if we are doing better now?
I think on one side we have at least the tools to be better: developmental testing (aka Unit Tests) are accepted everywhere, the Agile Practices are well established, Software testing in general is at least realised as need by most teams, refactoring is build in at most IDEs and patterns have matured. But is the code really become better, is it easier to integrate and extend software, have we found a way to dry out the swamp? As for code quality, I am not sure. Programmers today can produce the same smelly code as nearly one decade ago, they now also produce smelly unit test code (tests which are testing the wrong thing, which are depending on implicit preconditions or which are depending on time constraints)! The two major programming languages today, Java and C#, are object oriented and very powerful but most people never got the concepts provided by them and I fear it will not become better with the second comming of functional style programming features (LINQ). As the case for patterns, a lot of people read about them and at least the basic ones, the GoF patterns, are now common knowledge. What is missing is, if you have a problem conceptually to grasp, to look over the existing literature and see if you can find a pattern which solves your problem. What is also very disturbing today is that most software projects are one shot ones, so it doesn’t matter if they are bug free or functional, what only matters is that the last fantasiesed feature of someone who owns the project was implemented. To build up a sustainable process for developing software is not possible in most companies.
Software today becomes more complex every year but the ability to grasp the concepts either of the problem domain itself or the solutions needed stay the same. The brutal reality is that for the masses it doesn’t matter if software becomes better (hell, we have to reboot our mobile phones at least once a month!) and most development shops are not living long enough to have problems with their produced code base. For some people I think the awareness of most of the problems discussed has grown and sometimes they can make a positive impact, but for majority the ball of mud is as big as ever.