The role of hierarchy in open source governance

Book: Multi-Stakeholder Governance and the Internet Governance Forum. Jeremy Malcolm. Terminus, 2008

I have long looked for an overview of the forms of governance of the open source projects, here we finally have it!!

Jeremy Malcolm has written a remarkably researched book about the history, present and future of internet governance.

Chapter four is especially interesting as it asks; what can internet governance learn from the governance of open source communities?

In doing so, Jeremy has to draw out the salient characteristics of peer governance, and in particular, the balance between control and decentralisation.

We are publishing the relevant excerpts from a draft of that chapter. We have asked Jeremy to comment on how this is applicable to internet governance, and he will likely contribute in the next few weeks.

Jeremy Malcolm:

The common conception of most open source software projects as being anarchistic is actually a myth. In most open source software development projects, anarchy is balanced with hierarchical control.

It is in fact common for open source software development projects to be governed by a “benevolent dictator for life” (or BDFL). These are found in projects ranging from the Linux operating system kernel itself, of which Linus Torvalds is the BDFL, Linux-based operating system distributions such as Ubuntu led by Mark Shuttleworth, application software such as the Samba networking suite coordinated by Andrew Tridgell, and programming languages such as Perl, PHP and Pythonin which Larry Wall, Rasmus Lerdorf and Guido van Rossum respectively act as project leaders in perpetuity.

The position of BDFL normally falls to the developer who initiated a project, though in the case of multiple original core developers, the phenomenon of a benevolent oligarchy for life is not unknown (for example Matt Mullenweg and Ryan Boren for the WordPress blog engine).

Linux

In the case of the Linux kernel, Torvalds who is perhaps the archetype of a BDFL, possesses ultimate authority to decide which contributions (“patches”) to the Linux operating system kernel should be accepted and which should be refused. Torvalds no longer personally manages the whole of the kernel and has delegated authority to a number of trusted associates to manage particular subsystems and hardware architectures, but it remains his authority to appoint these so-called “lieutenants” and to supervise their work. A document distributed with the Linux kernel source code that is subtitled “Care And Operation Of Your Linus Torvalds” describes him as “the final arbiter of all changes accepted into the Linux kernel.”

Thus contrary to what might be assumed from Raymond’s claim about “the Linux archive sites, who’d take submissions from anyone,” the Linux kernel development process is neither anarchistic nor consensual: if Torvalds does not like a patch, it does not go in to the kernel. This has often antagonised other kernel developers, one of them commencing a long-running thread on the kernel development mailing list by saying:

Linus doesn’t scale, and his current way of coping is to silently drop the vast majority of patches submitted to him onto the floor. Most of the time there is no judgement involved when this code gets dropped. Patches that fix compile errors get dropped. Code from subsystem maintainers that Linus himself designated gets dropped. A build of the tree now spits out numerous easily fixable warnings, when at one time it was warning-free. Finished code regularly goes unintegrated for months at a time, being repeatedly resynced and re-diffed against new trees until the code’s maintainer gets sick of it. This is extremely frustrating to developers, users, and vendors, and is burning out the maintainers. It is a huge source of unnecessary work. The situation needs to be resolved. Fast.

Torvalds’ initially unapologetic response recalls another classic example of his sardonic view of his position as BDFL, when announcing the selection of a penguin logo for Linux. Acknowledging the comments of those who had expressed reservations about it, Torvalds concluded with the quip, “If you still don’t like it, that’s ok: that’s why I’m boss. I simply know better than you do.”

Mozilla and OpenOffice.org

The Mozilla and OpenOffice.org projects, amongst the most successful and highest-profile of all open source projects, provide a slightly different example of hierarchical ordering in open source software development.In these cases, the authority is not that of an individual, but a corporation: originally Netscape Communications in the case of Mozilla, and Sun Microsystems in the case of OpenOffice.org.

As well as leading development, Netscape originally held the “Mozilla” trade mark (as Linus Torvalds does for “Linux” in various jurisdictions), and until 2001 required modifications to its source code to be licensed under terms that exclusively exempted it from the copyleft provisions applicable to other users. Similarly, Sun required (and continues to require) contributors to the OpenOffice.org project to assign joint copyright in their work to it.

This kind of collective hierarchical control over an open source software project can also be exercised by a civil society organisation. The non-profit Mozilla Foundation, for example, succeeded to the rights of Netscape, such as the trade mark and rights under the Netscape Public License. Membership of its governing body (or “staff”) is by invitation only.

Apache Software Foundation

Another example of such an organisation, also taken from one of the most prominent and successful open source projects, is the Apache Software Foundation (ASF), which is best known for the Apache HTTP Server which powers the majority of Web sites on the Internet.

The case of the ASF illustrates well that there are also various strata of developers underneath the BDFL. One study has categorised these into core members (or maintainers), active developers, peripheral developers, bug reporters, readers and passive users, and confirmed previous findings that the core developers are generally the smallest group but write the majority of the project’s code.[85] Whilst developers in lower strata are mostly self-selected,[86] in many projects, including those of the ASF, the core developers are selected by the BDFL, applying stringent meritocratic standards.

The Apache Software Foundation is a non-profit corporation governed by a board of nine directors who are elected by the Foundation’s members for one-year terms, and who in turn appoint a number of officers (63, in 2007) to oversee its day-to-day operations. As of 2007 there are 156 members of the ASF, each of whom was invited to join on the basis of their previous contributions to ASF projects, and whose invitation was extended by a majority vote of the existing members.

Each project overseen by the ASF (such as the HTTP Server Project) is in turn governed by a Project Management Committee (PMC) comprised of a self-selected group of ASF members. Whilst non-members can also contribute to project development, they are required to assign copyright in their contributions to the ASF. PMC chairs are officers of the ASF who have power to define the PMC’s rules (if any) and the responsibility to report to the board. The ASF itself claims that it

represents one of the best examples of an open organization that has found balance between structure and flexibility. We have grown from 200 committers to almost 800, and that number continues to grow on a daily basis. We have been able to create several software products that are leaders in their market. We have also been able to find balance between openness and economical feasibility. This has earned us respect from a range of people, from single individuals to multinational corporations. We hope to continue to provide inspiration for businesses, governments, education, and for other software foundations.

Debian

One final example of hierarchical ordering in open source software development is found in comparing two similar Linux-based operating system distributions, Debian GNU/Linux and Ubuntu. The Debian project was the first to be established, in 1993. Although not incorporated, an associated incorporated body Software in the Public Interest, Inc (SPI) was formed in 1997 to provide the Debian project (along with various other open source projects) with administrative and legal support. It does not take an active role in the development of the Debian distribution.

The Debian project is led by a Project Leader who is elected by the project’s members, known as its Developers (or Maintainers), for a one year term. The Project Leader presides over a Project Secretary, a Technical Committee of up to eight members, and various other technical positions, all of whom are appointed from amongst the Developers by the Project Leader (save that the Project Leader appoints a new Project Secretary jointly with the incumbent in that role). The process for becoming a Developer is a meritocratic one, the requirements of which are set out in a detailed policy document.[92] As at 2007 there are over 1100 Developers, although many of these are inactive.

The autonomy of the Project Leader is expressly limited by clause 5.3 of the Debian Constitutionwhich provides, “The Project Leader should attempt to make decisions which are consistent with the consensus of the opinions of the Developers” and “should avoid overemphasizing their own point of view when making decisions in their capacity as Leader.” The same applies to the Technical Committee, which is only to make decisions as a last resort, where “efforts to resolve it via consensus have been tried and failed” (clause 6.3.6). Clause 4.1 provides that any decision of the Project Leader may be overruled by a general resolution of the Developers, as may any decision of the Technical Committee by a two thirds majority resolution.

Ubuntu

This is in contrast to the position within the governance of the Ubuntu distribution. Mark Shuttleworth founded the distribution in 2004, and termed himself its SABDFL (self-appointed benevolent dictator for life). He exercises a casting vote on both of its main decision-making bodies: the Technical Board and the Ubuntu Community Council.

The Technical Board is a committee which makes decisions on technical issues relating to the distribution, such as which software it should include and how this should be packaged and installed. It acts by consensus, but only amongst its own members, and is not required to reflect the views of the Ubuntu community at large. Its members, currently four, are appointed by Shuttleworth subject to confirmation by a vote of all developers, and they serve a one year term.

The Community Council is responsible for overseeing the social structure of the Ubuntu community, including the creation of Teams and Projects with responsibility for particular tasks such as documentation and release management, and the appointment of new team leaders, developers and members. It also maintains a Code of Conduct to which all members of the Ubuntu community are required to adhere, and arbitrates disputes arising under it. The Community Council, also currently of four members, is appointed by Shuttleworth subject to confirmation by a vote of all developers and members, and sits for a two year term.

The developers or maintainers of Ubuntu are equivalent to those of Debian and nominated by a similar process, save that they are divided into two tiers: Masters of the Universe or MOTU, who have authority to maintain packages only in the unsupported “universe” and “multiverse” classes available for installation on an Ubuntu system, and Core Developers who are uniquely authorised to maintain packages in the fully supported “main” class which are installed by default by the Ubuntu distribution.

In addition, Ubuntu recognises as “members” those who have provided a significant contribution to the Ubuntu community, perhaps by providing support, writing documentation or reporting bugs, but who do not require the power to directly maintain packages. Members, like developers, are required to agree to the Code of Conduct and have the right to vote to confirm new appointments to the Ubuntu Community Council.

A survey of desktop users of Linux distributions conducted in 2005 by Open Source Development Labs (OSDL) revealed that in little over a year since its establishment in 2004, the Ubuntu distribution was already in use by more than 50 per cent of respondents. A prominent former Debian Developer who resigned in 2006 compared the Debian and Ubuntu distributions by saying,

There’s a balance to be struck between organisational freedom and organisational effectiveness. I’m not convinced that Debian has that balance right as far as forming a working community goes. In that respect, Ubuntu’s an experiment—does a more rigid structure and a greater willingness to enforce certain social standards result in a more workable community?

Exit-based empowerment

In fact of the examples given of open source projects in which a significant hierarchical structure exists or has existed—the Linux kernel, Mozilla, OpenOffice.org, Apache, and Ubuntu—all are the most widely-used open source projects in their class, and have large and active communities of developers.

How can this be reconciled with the earlier hypothesis that it was the very lack of hierarchy that empowered developers and attracted them to volunteer their services to open source projects?

Despite the fact that its significance to developers had earlier been downplayed, the answer is found in the open source licence. It is the open source license that enforces benevolence upon the dictator. It does this by ensuring that for any open source project, there is always relatively costless freedom of exit, in that any developers who feel they are being oppressed by a project leader can simply cease participating in the project, take its source code, and use it as the base for a new project of their own (known as a “fork” of the original project). This “exit-based empowerment” enjoyed by developers mitigates the power of the project leaders.

As Torvalds has put it,

I am a dictator, but it’s the right kind of dictatorship. I can’t really do anything that screws people over. The benevolence is built in. I can’t be nasty. If my baser instincts took hold, they wouldn’t trust me, and they wouldn’t work with me anymore. I’m not so much a leader, I’m more of a shepherd. Now all the kernel developers will read that and say, “He’s comparing us to sheep.” It’s more like herding cats.

The Linux kernel has, indeed, been forked numerous times. One prominent fork was that maintained by Red Hat Linux developer Alan Cox, who released a series of kernel source trees differentiated from Torvalds’ official kernel by the “-ac” tag, and which contained patches not yet accepted by Torvalds.[98] However since 2002, a technical solution to Torvalds’ backlog was found in the use of specialised revision control software (originally, ironically, a proprietary product called BitKeeper,and subsequently an open source equivalent called Git written by Torvalds himself). This has placated many of Torvalds’ critics, and resulted in the obsolescence of many former forks of the kernel.

Both Mozilla’s Firefox browser and the OpenOffice.org office suite have also been forked. The Debian project, for example, has replaced Firefox in its distribution with a forked version called Iceweasel, to escape the onerous trade mark licence conditions imposed by the Mozilla Foundation for the use of the Firefox name and logo.As for OpenOffice.org, a prominent fork called NeoOffice has been customised to integrate more smoothly with the Mac OS X operating system. Debian itself has also spawned a number of derivative distributions, although these are seldom termed forks. A federation of Debian-based distributions (though not including Ubuntu) is the DCC Alliance.

Admittedly, forking an open source project is not completely costless. Although the cost of the infrastructure required to host a new project is minimal (often even free), a new name and logo will be required (either because the originals are protected by trade marks as in the case of Firefox and OpenOffice.org, or simply because of the practical necessity to distinguish the two projects).

Usually the most significant cost however is that it will be necessary for the new project leader to establish a community of users and developers to support the project in the long term. This is defined by economic sociologists as the cost of developing social capital. Thus, the more successful the parent project is (and the more cohesive its communities of developer and users), the higher its social capital will be, the higher the transaction costs of a fork, and the more effectively that fork will have to differentiate itself from its parent in order to overcome those costs.

This is illustrated by the case of Samba-TNG which forked from the highly successful Samba project in 1999, seeking to differentiate itself by first offering the facility to replace a Microsoft Windows NT server as a Primary Domain Controller. However it struggled to build a development community comparable in size and expertise to that of its parent project, which in the meantime implemented its own version of Samba-TNG’s differentiating feature. In comparison, forks of less dominant and stable projects (such as the oft-criticised PHP-Nuke content management system) have been forked more often and more successfully.

This characteristic of the transaction costs associated with migration from one open source project to another provides a cohesive force against the unnecessary fragmentation of open source projects, that will only be overcome if enough developers become sufficiently dissatisfied to form a viable competing project (which the project leaders have an incentive not to allow to happen, lest they lose their base of developers). In comparison, developers within Microsoft Corporation face much higher transaction costs in replicating their work and their communities elsewhere if they are dissatisfied, if indeed it is possible for them to do so at all.

Thus it is from the unexpected source of the open source licence that a possible solution is found to the problem of how to ensure that an organisation acts benevolently rather than tyrannically, even though it may be structured in hierarchical rather than democratic form. The open source licence achieves this by providing an implicit check on the power of the organisation’s leadership, effectively conditioning its continued exercise of that power on the consent of its constituents.”

1 Comment The role of hierarchy in open source governance

  1. Pingback: Holistic ICT for EcoLiving » P2P Foundation & OVF Networks Mesh

Leave A Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.