(this post is a short synthesis of the essay I wrote for O’Reilly’s Open Government book: the full text is reachable by clicking here or on the titles of each section)
Introduction
(digital) Standards are the foundation for our everyday activities in ways we often don’t even imagine. Consider this question: are the pens that President Barack Obama’s used to sign the memoranda about transparency and Open Government and another about the Freedom of Information Act (FOIA) standard ones? Of course, almost nobody knows nor cares, but my point is “why, exactly, we don’t care and what could change this situation?” Here I’ll try to answer, because digital technology is essential for open government but also much less reliable and, in some ways, much less technically sophisticated than those pens.
The Digital Age Explained
Data of every kind are called digital when they are encoded in bits that hard drives can store and telecom network can transport regardless of what they mean. The cost and time savings enabled by this approach to information management are so big that the trend toward digitization is unstoppable.
However, converting data to bits involves a specification that says what each group of bits means and how they should follow one another. When they choose to, governments can mandate standard digital formats as compatibility requirements in public tenders, to provide real interoperability both now and in the future.
Standards and the Problems with Digital Technology
Even when hardware works perfectly, bit sequences are absolutely useless if you don’t know what they mean, and if the instructions you need to read or translate them are lost or too expensive to buy. These aren’t hypotheses. Almost all the files created by public and private businesses around the world are already encoded in a way that only one suite of programs, from one single, for-profit company, can read without compatibility problems. What if that company went belly up? Think it’s too big to fail? Isn’t this what everybody would have said in 2008 about Lehman Brothers, General Motors, or Chrysler?
George W. Bush left behind 100 trillion bytes of electronic records including data in “formats not previously dealt with” by the U.S. National Archives. If Obama’s pens were like the software used to create those data, anyone wanting to read the memoranda would have to buy the same kind of pen. Not much openness or freedom of information in that!
Why Has Digital Gone Bad So Often?
There are several reasons for the digital mess we’re in today, but a particularly important one is our ignorance as a society. Software is still so new in our culture that most of us (including many people who consider themselves “experts” because they spend lots of time using office suites, computer games, or social networks) haven’t actually realized yet the roles played by formats and protocols. Consider how people refer to office files. Nobody would talk about a handwritten letter by mentioning the name of the pen used to write it; saying “I sent you a Bic letter” or a “Mont-Blanc letter” would be a sure way to have everyone laugh at you. Yet most people regularly say “I’ll send you a PowerPoint” or “I need to check the figures in that Excel file,” which is the same thing, but with no embarrassment.
This is at least counterproductive, if not actually dangerous. Unlike any other product, badly used software hurts not just the people who used it badly, but everybody else who may need to interact with them both now and in the future. Once a movie, contract, or business report has been saved in a format that can be read by only one software program, you can forget copyright. That document now belongs to whoever developed that program. If you still want it, you must accept their conditions. That’s how Word/Excel/PowerPoint and AutoCAD became de facto monopolists in their respective markets: because their file formats, not the software itself, are secret.
That’s why I said that software is less sophisticated than pens, because pens create none of these problems. Software should (and can, even when it’s proprietary!!!) work in the same open way as pens, but in practice this will happen only if open digital formats and protocols become the only admissible technologies in certain contexts. I say “Formats and protocols” instead of software, because the former are often more important than the latter: we run software to manage data, not the other way around.
The Huge Positive Potential of Digital Technologies
Democracy implies openness. Software and digital data can help tremendously to provide openness in all fields of public activity, including critical ones in these days like independence from foreign oil or health care.
Publishing online without legal restrictions raw data such as maps, census records, weather surveys, agricultural statistics, court rulings, and agency budgets (while protecting citizens’ privacy, of course) reduces public expenses, stimulates the economy and increases the tax base. Besides, it also brings much more control by private citizens over their governments and closer cooperation with them.
In principle, the current U.S. administration is in favor of going digital this way. The Transparency and Open Government memo says:
Government should be transparent. Transparency promotes accountability and provides information for citizens about what their Government is doing. Information maintained by the Federal Government is a national asset. My Administration will take appropriate action, consistent with law and policy, to disclose information rapidly in forms that the public can readily find and use. Government should be collaborative.
The truth, however, is that these and many other things, including FOIA, will be technically possible only if by mandating the use of open, standard formats.
Free and Open Standards and Software: The Digital Basis of Open Government
Free/libre/open source software (FLOSS) is software that is available to everybody, both in executable form and as source code, without any royalty or legal restriction. Still, FLOSS is not enough to guarantee that owners of documents will always be able to read them, because the original source code might be lost or fail to work on newer computer systems. In such cases, files become unreadable not because of software licenses, but simply because their authors never bothered to demand that the programmers use fully documented file formats. That’s why it’s important to stick to really open standards that exist and are completely defined regardless of any specific software program, regardless of its license. Only standards such as these give real guarantees that our data remains ours.
Besides, file formats should be as few as possible and as stable as possible. FLOSS makes it always possible to convert data from one format to another, but why create the need for conversion if it isn’t absolutely necessary? Think of software as pens, and formats as alphabets. We went from quills to email in just a few centuries exactly because the alphabets remained practically unchanged, allowing each generation to learn and build on what already existed rather than rewrite every manuscript in a different way every few years. And it doesn’t matter if some pens are very expensive or patented exactly because the alphabet is independent from all pens.
Conclusion
Digitization is good, but (at least when government is concerned) only when it’s open in the ways described in this essay. Open data and file formats are mandatory to guarantee that public sector information is and remains really available to all citizens. Besides:
- Digital technology is legislation. A government that ignores this has abdicated part of its duty to protect individual freedom and equal opportunities.
- Open digital standards can save lots money with comparatively little effort…
- …but only if that effort is coordinated. That’s why it must be governments that set the example and constitute the critical mass that makes open standards and FLOSS accessible to everybody.
Final note: if you want to know more about the crucial role of file formats in today’s world, you are also welcome to download the slides of my seminar on how file formats can be used to favor (or hamper) innovation, active citizenship and really free markets