Evolution of Open Source Software – Survival of the Fittest – or: To b(uild a community before the hype cycle overtakes you) or not to be

JavaOne 2008 made one thing abundantly clear: everyone who owns a product, being it a commercial product or an open source product, desperately needs attention. A product that is not talked about might as well not exist. A necessity for making a product successful is momentum, a community, a positive spiral, a lot of buzz,some luck and stamina (patience, deep pockets, disappointment shield). If the product owners fail to achieve these elements, fail to build up a community beyond their inner circle, the product is almost certainly doomed. In a way, JavaOne 2008 was like a bazaar with many vendors clamoring for attention from their respective stalls. And I am not just talking about the Java Pavilion which has this physical appearance in the literal sense; many presentations were on (OSS) Projects, some in their very early stages, that quickly need visibility and at least the beginning of a community to ever become successful or even move beyond their current status.

Sun Microsystems, a company dealing with many challenges at this point, facing a lay off of up to 2000 employees and reporting a $30+ million loss in the previous quarter, is desperately seeking for business models for making money on open source software products. They bought MySQL AB for 1 billion dollar,  a company that is entirely based on a open source business model; and hope to learn to apply that same trick to for example GlassFish, OpenSolaris and of course Java itself.

Watching all these product owners at the Java Pavilion and during their talks in Technical Sessions and BOFs and thinking about Sun’s challenges, I started wondering about Open Source projects and products and later on software products in general. What makes them successful, where do they come from to begin with? What can you do as product owner to increase the chances of your product? What can you do as relative outsider to help grow a product or project you feel attracted to? When should you consider a project or product a lost case? In this article I try to address these questions. 

My first question really is targeted at the open source products – although it may apply to many commercial products as well – and it is: how does a product or project come into existence? How does it get started?

How does an open source product or project come into existence?

To me it seems that it is either a bottom up process or a top down process. Let’s start with the classic picture of the inspired individual whose genius ignites the spark that eventually leads to a world shattering product.

Bottom Up

A series of steps similar to these ones lead to a new open source project and product:

• Inspired and talented programmer has good idea, some extra energy and a lot of time on this or her hands (long flight, lonely hotel room, wife and kids away for the weekend,…) and in very short period of time creates a prototype that demonstrates a powerful concept

some well known examples:

Software consultant James Duncan Davidson created Ant on a transatlantic flight in 1998. Davidson was working on a cross-platform program build and running into problems using available build tools, so he created his own. Davidson said he named the tool "Ant" because it is a little thing that can build big things.

Gavin King began developing Hibernate in late 2001 when he found that
the popular persistence solution at the time-CMP Entity Beans-didn’t
scale to nontrivial applications with complex data models. Hibernate
began life as an independent, noncommercial open source project. Gavin King told in an interview (shortly after joining JBoss): 
I am the founder of the Hibernate project, an open source object/relational mapping solution for Java.  In Wikipedia it says: Hibernate was developed by a team of Java software developers around the world led by Gavin King.

Craig McClanahan: Struts grew out of a personal need (open source developers often call this scratching your own itch) to support the development of an application that I was responsible for, prior to joining Sun in 2000. .. there were few available implementations. Therefore, I set out to
create an application framework for my own use, particularly focused on
some key functional areas… The resulting framework served my needs quite well. As it became
obvious that my needs were not unique, I began the process of open
sourcing this idea of a web application framework at the Apache Software Foundation .

Spring is a layered Java/J2EE application framework, based on code published in Expert One-on-One J2EE Design and Development by Rod Johnson (Wrox, 2002). 

I guess that the origin of JavaFX, Chris Oliver’s hobby project, is very similar to these other ones: moment of true inspiration followed by a long series of late nights. 

• The original programmer is capable of getting the buzz going (either by himself or by hooking up with friends with a nack for it): communicating about what he has created, demonstrating it in the right places, showing a prototype or screenshots on a conference or in a weblog, building up some interest.

• The software is made available for download and try out with some sparse explanation and documentation; the creator wants to share what he has done, he may want to show of a little bit, he may want to solicit feedback, suggestions or help.

• At some point, a community starts to form: people download the software, try it out and give feedback. A forum is typically started to give the community a means to help itself, a multilateral process that also helps building a shared memory, rather than a one to many relationship. A Wiki is another means for a community to start sharing ideas, questions, comments, tips,… An important element to get the product going is the development and publication of documentation, tutorials, samples, to lower the threshold for the not so happy few. What the product also needs is a few evangelists: people who will announce the product in the main forums (TSS), in articles for internet magazines, on blogs etc. Evangelists, people who are trusted for their judgment or at least their writing skills, can help propel the still the fledgling product.
• If it has not happened by now, it is time for some additional developers to join the project. A one man band can not keep a serious open source project afloat for very long. Building up credibility requires the community described above – with resources such as Forum and Wiki, Tutorials and Documentation and Samples and bug tracking system – and a team of ‘core code committers’ as well. Several people joining forces to work on fixing known issues, testing the code and creating new features. And sharing the ownership and the responsibility.
• With the growing number of people involved as well as the required infrastructure (even though sites like Source Forge and Atlassian provide a lot of the essentials), the open source initiative becomes serious enough to require proper project management. It needs vision and strategy, goals, release schedule, transparent decision making, etc. The project must be seen as at least somewhat stable, mature and reliable. Delivering quality and making deadlines. And focusing on the less glamorous sides of software engineering such as portability (multi OS, multi app server, multi database and multi browser), internationalization, patch management, documentation, refactoring and code optimization etc.
• The next level to be achieved is the actual application of the product in production environment. When the first references of such situations start coming in, the credibility rises to a whole new level and the positive spiral is on its way.

The bottom up process is organic, people driven, largely unplanned, unstructured and even chaotic, creative and initially quite unstable.

In the next section, we will discuss how to keep it going.

The other route: Top-Down

Another way for open source products to come into being can be top down. Here we have a planned, structured, well managed and typically company sponsored process. There are at least two extremes in this top-down scenario:

• the open source product is built from the ground up, with open source in mind from the very beginning. An example is Alfresco (the document management system) and to a large extent Spring Framework;
• the product is turned open source after a previous existence in the commercial domain; examples range from the Ingres Database (Computer Associates), via Eclipse (Eclipse was originally a proprietary product developed by IBM as a successor of the VisualAge family of tools. Eclipse has subsequently been released as free/open source software under the Eclipse Public License.) to Apache MyFaces Trinidad (started life as ADF Faces), GlassFish (originally the Sun One Application Server) and EclipseLink (the open source continuation of Oracle TopLink).

Of course, a successful adoption as an open source product by a thriving community will quickly eradicate most memories and traces of the earlier commercial existence – such as is the case for Eclipse.

Why do corporations engage in the development of open source products? I can think of a number of motivations, that are not at all mutually exclusive, that will play a role:

* make money
– from providing pre-packaged, pre-built bundles
– from support, training, consultancy fees; documentation and books
– from providing updates and upgrades
– from providing ‘enterprise edition’ with additional features
– from selling supporting tools and extensions/pluginsdo marketing: building a profile from participation in open source projects

* connect with the community (a vendor known as closed, commercial can try to mollify, pacify and approach or even embrace a community by investing in open source projects thus supporting that community; IBM and Oracle could be regarded as examples)

* employee satisfaction: attract and retain staff by allowing them to work on open source projects

* out of idealism and principle: it would be cynical to not include it in the list, though I doubt it plays a major role

* implement corporate strategy
– help strengthen standards (JPA, JSF, ..) for idealistic or strategic reasons (for example Oracle’s donation of MyFaces Trinidad and TopLink Essentials, succeeded by EclipseLink)
– help strengthen open source offering to bring to a higher level required for some ulterior motive (e.g. Oracle Enterprise Linux, offering 24/7 enterprise level support demanded by customers as prerequisite for using Linux for enterprise database systems)
– dump (headache) product in a seemingly generous and elegant way (no serious money will ever be made of the currently commercial product, some customers may demand ongoing support, it will only cost more money to continue it, let’s make the sources available and everyone can support himself while it looks as if we make a generous contribution to the open source community) (I sort of got that feeling with the Ingres Database and perhaps something goes for Cloudscape as well)
– revitalize product (a product that has lost momentum as commercial product may well have a second life as open source product). Sun One Application Server rebranded as GlassFish is starting a whole new life. IceFaces is another example of this type of move.
– hurt competition (an organization may support an open source product it has no vested interest in, to hurt one or more of its competitors, either by direct competition or by muddying the waters)
– gain influence in a community that can be built upon to co-control development of standards and the overall direction the community is taking

 

What makes an open source product (continue to) thrive?

You would expect an Open Source project to be associated with nice concepts and terms like open, standards, trust, collaboration, transparent, fun, idealistic, quality, community, together, non-profit, growing, enthusiasm, young, unspoilt, contribution, democracy (en of course world pe

ace). And that is typically how they all begin.

However, many open source products and projects either fade into the background, linger on the edge of existence or become linked to words such as politics, egos, hidden agenda, influence, competition, envy, power, status, money, semi-open, control, dominance, conflict, etc.

Why is that? Well, almost nothing thrives on good intentions alone. Even though high spirits, great ideas, wonderful plans and lots of energy and enthusiasm get you started well enough as open source team, project or product, it does not last. At some point, there is to be something in it for the people involved, the ones doing the heavy lifting.

I have been thinking about reasons for people to spend a lot of time and energy on open source projects. Some of the motivators or objectives for these individuals that I could come up with:

• for the fun of it
• to really deliver on a promise or principle, out of a responsibility taken on at some point
• to strengthen the Curriculum Vitae
• to generate web traffic (and in turn make money of that)
• to build a profile and get speaking slots at conferences or articles published in JDJ and on JavaLobby
• to (have a topic and the credibility to) write a book
• to make money from training or consultancy
• to get a some great job offer
• to start a company (such as SpringSource – pka I21)

It seems that for any open source project to be successful in the long run, there must be a dedicated core circle of leaders pulling the weight, showing the way, making it happen and fostering, growing and inspiring the community as well. And there must be a reason for these people to actually to all that, next to day time jobs or other responsibilities.

I think that for any project to really scale and endure, a commercial gain is required and serious support from a company (or more than one) is a necessity. The most successful, more mature, open source products seem to all have one or more corporations who make several resources available for work on the product (Firefox, Linux, Eclipse, Apache MyFaces, ..) or even have taken on the leadership for the product (JBoss, SpringSource, ..). Of course such farreaching involvement of individuals and enterprises with their objectives will at times conflict with the spirit of true open source. And may weaken the role of the community at large, the transparency and openness. I am afraid that it cannot be avoided for most products.

This means by the way that the distinction between commercial products and open source offerings is frequently much smaller and more subtle than you might think at first glance. Using open source for serious business application sometimes requires enterprise licences that provide support that are just normal commercial products. The beauty of having the sources available is really a pretty much empty, meaningless benefit – as these sources usually are too complex and too poorly documented to really maintain or even fine tune yourself.  And besides: you do not want to spend your precious resources on generic software – if you have to maintain it yourself after all, the free software can become pretty expensive in the longer run.

The blog article "How To Spot A Successful Open Source Project" lists 9 conditions for successful open source projects. The number one would be my number one as well: a strong, living community. A project needs to get started, to build visibility and credibility, to solicit feedback and help, to create resources such as documentation, tutorials and samples, to maintain forums and wikis, to further promote the project and its product. Without a community, the project is invisible, no matter how good it is. And it will not take off. Note: this applies to commercial products just as much as to open source projects!

The top 9 essentials:

   1. A thriving community

   2. Transparency

   3. Employed developers

   4. Disruptive goals

   5. Civility

   6. A clear license

   7. A benevolent dictator

   8. Documentation

   9. Commercial support

CommunityOne

JavaOne 2008 was preceded on Monday by the CommunityOne day, an event focused on the community and what the community can do for some still under age open source projects such as jMaki, OpenSolaris, OpenDS and OpenSSO, OSGi, GlassFish and even more established ones such as NetBeans, Eclipse, MySQL, JBoss. Communities are also essential for the development and acceptance of new standards; the JCP and all its JSRs for example are what shape the future of Java and JEE, and are essentially community undertakings, just like most open source projects.

It is good to see and realize, as I did during CommunityOne and JavaOne, that as individual developers, we can still make substantial contributions to one or more communities and open source products. By using them, telling people about them, writing about them, providing feedback, perhaps publish a sample or tutorial or share some experiences. It does not take that much to get and keep a community going.
 

Resources 

Wikipedia: What is Open Source?  http://en.wikipedia.org/wiki/Open_source

What Makes An Open Source Project Successful?, Charles Babcock, InformationWeek April 14, 2007  http://www.informationweek.com/news/software/open_source/showArticle.jhtml?articleID=199000960

How To Spot A Successful Open Source Project, Dru  Posted 2/14/2007, http://blogs.ittoolbox.com/unix/bsd/archives/how-to-spot-a-successful-open-source-project-14492

How To Tell The Open Source Winners From The Losers, Charles Babcock, InformationWeek, February 3, http://www.informationweek.com/news/software/open_source/showArticle.jhtml?articleID=197002953&pgno=1

The Interview: Gavin King, Hibernate,  http://www.javaperformancetuning.com/news/interview041.shtml

How to Have a Successful Open Source Project  http://www.wikihow.com/Have-a-Successful-Open-Source-Project

One Company’s Search For The Perfect Open Source Software, Charles Babcock, InformationWeek February 3, 2007 http://www.informationweek.com/news/software/open_source/showArticle.jhtml?articleID=197002955  

Business Readiness Rating (OSS Watch)  http://www.oss-watch.ac.uk/resources/brr.xml
Interview: What are the grounds of a successful Open Source project ? – http://txpmag.com/interviews/interview-what-are-the-grounds-of-a-successful-open-source-project       

The secret of successful open-source companies, Part II, Matt Asay  July 11, 2007 – http://www.news.com/8301-10784_3-9743259-7.html
and part 1: http://weblog.infoworld.com/openresource/archives/2006/07/the_secret_of_s.html