Last Thursday, G2010 — a conference on Government 2.0 — finally arrived. As others have said, it was a fantastic day, with many interesting people in attendance and on Twitter.

At the beginning of the year, Jeffrey Peel asked us to organise a panel for the day on digital engagement. It’s an area we have some experience of — we made TellThemWhatYouThink.org and ConsultationXML, and are helping the COI to deploy RDFa to describe consultations documents on central government websites. We had a lively panel and discussion about consultation and how to do it, which you can now watch:

And here are my feelings on the subject — in a bit more detail than we could fit in on the day.

Consultation is an area in which there is intense activity. Departments consult on almost everything they do, local government has a statutory obligation to consult residents on a wide variety of issues, and the rise of social media in government has brought into sharp focus the things that the Web makes possible. But despite all this activity — very much including innumerable panels at conferences — we’re not achieving the mass participation in policymaking that is, for many, the goal of digital engagement. So what’s wrong?

I think that some very big assumptions have been made about digital engagement, its potential, and the right way to do it. Everyone’s been excited by the possibilities, myself included, but I think we’ve failed to really look at the people we’re trying to engage, their level of interest, motivation and available time.

The reality is that formal consultation is simultaneously necessary (in that a deliberative, evidence-based policymaking process is valuable) and expensive in terms of the investment of time and energy that people must make to participate. We have conflicting goals: to reduce the barriers to participation — make it quicker and easier — while also maintaining an informed policymaking process. Formal consultation is far from perfect and we should work to make it better, but it’s not obsolete.

The solution we’ve adopted so far is to try and make it easier to dip one’s toes into a formal consultation. This has been valuable, and we’ve learned a lot from it, but I don’t think it’s workable.

Such approaches can substantially increase the number of responses that consultations receive, but they’re usually not the right kind of responses. A formal consultation doesn’t much benefit from large volumes of anecdotal correspondence about personal experiences. That kind of input is extremely valuable, but by the time a policy has reached formal consultation, it’s too late to use it. That kind of engagement has to happen earlier.

It also has to happen more often. It’s simply no good to pick a time — essentially arbitrarily — to ask people about their experience of, for example, public services. A consultation on the NHS probably wouldn’t be of much interest to me 5 years after my operation, but if I’m asked straight away, I’d be much more likely to respond. The issues would be fresh and immediate, and I wouldn’t have moved on with my life.

Those experiences happen all the time. They constitute the raw reality of our society and the value, or lack of it, that Government succeeds in generating for people. They are innumerable, chaotic, disorganised, neverending and personal: just the unstructured, unrepresentative things that you don’t want in a formal consultation, but that have the potential to create real, valuable change in the way ministers, Parliamentarians, policymakers, civil servants and front-line staff do their jobs.

We need to pick apart these strands. First, we must take formal consultation on to the web, away from paper and PDFs, and engage those people who are interested in investing their time and effort in the process. Second, we must embed into government a culture of engagement, so that those who have stories to tell can tell them to the right people at the right time. Engagement is not the exclusive province of web, press and comms teams. It’s everyone’s job, and everyone must make time for it.

After all, “engagement” is just another word for “talking to people and finding out what on earth’s going on”.

Who couldn’t get behind that?

The UK’s version of data.gov, ably put together by the Cabinet Office, has just launched in private beta. We got to have a sneak peak, and it’s great!

The site is a blend of the US’s equivalent, data.gov, and Directgov | Innovate. It’s got a listing of available data packages, powered by the Comprehensive Knowledge Archive Network, and user-generated lists of apps and new ideas. This is just right: the data you need, combined with a way to promote the things you make and a place to get ideas if you’ve got itchy typing fingers but lack inspiration.

It’s not perfect. Conspicuously missing is an organised way to browse data sets: but that’s coming, along with some other tweaks and twiddlings that’ll improve the site’s usability.

The site is powered by Drupal, with packages catalogued and hosted by CKAN. Meanwhile, data.gov.uk hosts a data store powered by Talis that can scale to 100 billion triples and is hosted on Amazon EC2. The system is federated, so departments can add and control their own data, lots of which is available as RDF, with the remainder downloadable in spreadsheet form.

Speaking of spreadsheets, they’ve even written an app that departments can deploy in-house to convert spreadsheets into RDF (kudos to John Sheridan!) which makes it much easier for departments to produce structured, linked data.

This is all working now, and was put together by the team at the Cabinet Office in the last three months. This is a massive achievement, and it sounds like it’s just the beginning: they have big plans. User submissions for new datasets. Metadata to describe provenance. More data sets on the site. More data as RDF. Organised browsing for packages. Source code releases. The list goes on.

This is such an encouraging thing to see. No expensive procurement exercises for clunky, bespoke sites: instead, we have the right tools for the job, joined together. Simple things that do one job well, combined to form a more complex whole. It’s the Unix philosophy in action.

This is how all Government IT should work.

Our hearty congratulations go out to the team at the Cabinet office, with special thanks to Richard Stirling for spilling some of the beans. I had lots of questions and nitpicks, and every single one of them was answered reassuringly.

They’ve got a plan, and it’s a good one.

Just discovered (via Twitter) that NHS Choices have released a new tool for users to rate hospitals. Fantastic. Government should embrace user generated content more often than it does, and feedback on Hospitals is a great application for it — but their implementation could do with a bit of tweaking.

First: the tool has been implemented as a new feature within the existing NHS choices website. Its design is clean, but very text heavy. This is not helped by the use of very long titles in the sidebar boxes that contain statistics — including the user-generated stuff — about the hospital you’re viewing. I suspect that they are this long because of a desire to make it ultra-clear which content is drawn from official statistics, and which is user-generated.

That’s a reasonable concern, but the site as it stands smacks of paranoia. I think that most people can tell which bits are official and which aren’t, even if the bits are in the same box. The distinction between “75% of people would recommend this hospital to a friend” and “1.33 MRSA infections for ever 10,000 bed stays” is obvious. There’s also a very strange chart in the user-generated sidebar: it appears in the middle of a sentence. I’d call myself a fairly seasoned web user, but even I found that confusing. It just looks broken.

Second: the comments aren’t prominent enough. They’re too far down the page, and relegated to a sidebar. They’re one of the most useful parts of the page — personal stories will always speak louder than dry statistics — so I’d give them a bit more importance, and put them in the main content area, beneath the hospital’s description. As well as being more prominent, it’s more consistent with how comments are usually presented.

If you click through to read all the comments, you find that they are presented in exactly this way on the next page, which is excellent: it displays all the comments in a way that makes it easy to absorb the ratings at a glance, summarises them right at the top of the page and has a prominent call to action for people who want to post feedback. Even more important than that, it has replies from the hospitals, which is fantastic. Government forays into the social web rarely ever result in real two-way communication. To see it being done is  encouraging.

In short — despite being a bit rough around the edges — this is a nice bit of work, and definitely a big step in the right direction.

I was slightly bemused when the Cabinet Office announced that it was going to create a new £160k position for the Director of Digital Engagement.

The job seemed like a tall order: a list of requirements that it would be hard for any one person to fulfill, and a very big job to do with very limited resources. It seemed like a strange move to make when creating two positions at £80k a piece would probably still attract very qualified people, and give you more time and knowledge for your money.

Nonetheless, I watched with interest, and now, a tad later than expected, the position has been filled by Andrew Stott. My initial reaction was along the same lines as Emma Mulqueeny’s — more bemusement — but actually, I think Andrew is a good choice. Not who I’d have expected, but good nonetheless. As numerous people have said, he is very qualified, does have a brain the size of a planet, and has lots of experience pushing through the kind of change that we need. More than that, though, he’s practical.

I worked with Andrew briefly in 2008. One of the things we were looking at at the time was the quasi-XML version of the Civil Service Yearbook, which has lots of useful data in it. As is usually the case, though, it wasn’t proper XML — it’s variously broken, inconsistent and badly written. We spent a satisfying ten minutes at the end of the day bemoaning such irritations, and the next morning Andrew showed up at the office having spent all the previous evening writing a bunch of code to take the nasty XML and make it into useful data.

That, I think, is indicative of the man.

People have been talking about the Ordnance Survey rather a lot recently.

It’s a very strange beast. It has mountains of really useful data: electoral boundaries, postcode databases and the locations of all sorts of buildings, not to mention roads, railways and green spaces. Unfortunately, it’s a quasi-independent body, which has to pay its own way. It charges heaps of money for access to all these datasets, and makes them available under strange and deeply inappropriate licences.

This has long been a problem for civic hackers, and by implication, everyone else: if you want to make a service that needs to turn a postcode into a geographic location, you need to use their data, and most of the time, you can’t. It costs an arm and a leg.

Fortunately, it’s clear to everyone that the OS needs to be significantly reformed. They own datasets like electoral boundaries, that are fundamental to the operation of a democracy. They’ll charge you if you want to use them, even if you’re the government and you’re organising an election. That’s just nuts, and we all know it. So what to do?

Thankfully, people have been working on it. The recent Power of Information Taskforce report had lots of good things to say: Recommendation 7 was almost perfect. Liberate postcodes and boundary data. Make basic mapping data available for free to all, for modest use. Simplify licensing. Its only real downside was the absence of any mention of derived data.

Put simply, if you use an OS map to create a database of something — like postboxes, hospitals or parks — then OS share copyright in that data. In this modern age of user generated content, that position is completely unsupportable. It’s a shameless grab for intellectual property, motivated by their desire to receive extra royalties. For a private company, that might be fair enough, but it’s reprehensible for a body that exists to provide a public service.

For anyone who thinks this is merely a theoretical problem, it may be of interest that all those lovely crime maps launched by police forces across the UK are probably in violation of the OS’s licensing terms.

This, then, was the background to the latest budget. Having heard that it would contain some new announcements, we were waiting for its publication with baited breath, and indeed: it promised reform. The following day, Ordance Survey published their new commercial strategy. It is underwhelming.

There are some good changes: more data, including boundary information, will be made available through OpenSpace, their API. They’re going to revise their definition of “commercial” so that sites that use their data can carry advertising without being required to pay for licences. But that, more or less, is it. The rest of the strategy revolves around converting people using free licences into ones that become financially sustainable so they can pay. Fair enough, but hardly groundbreaking.

The real problems remain. OS still own electoral boundary data and postcode boundaries & locations. They still decide if you’re commercial, and you still have to accept their onerous licences to use their data. OS still maintain a stranglehold on any data that they consider to be derived from theirs. They’ll still charge you royalties to use that data, of at least £1000 a year. You still won’t be able to do anything with that data that’s not acceptable under their licence, like adding it to OpenStreetMap.

This new strategy is progress, but only just. It is at best a fractional improvement upon what we had before.

A lot more needs to be done.

Here’s a story from FreeOurData which is, quite frankly, incredible. The Office of National Statistics, in preparing for the next census, has found that the postcode databases offered by the Royal Mail and Ordnance Survey aren’t accurate enough for their purposes. Their solution: to build their own database. This is fair. The postcode database is not amazingly accurate, and ONS have different requirements anyway.

Unfortunately, Royal Mail and Ordnance Survey make good money from selling the postcode databases to other organisations. These datasets are very valuable: you’ve probably made use of them whenever you’ve put your postcode into a website. Royal Mail and Ordnance survey did not — apparently — like the idea of ONS making another postcode database with which they’d presumably have to compete. So, rather than take that nice dataset and do useful things with it — like giving it back to us taxpayers — the ONS have pledged to build the database, use it for the census, and then destroy it.

Postcode databases are almost a holy grail. Of all the datasets in the country, liberating the postcode database for free reuse would probably create more value than any other. The thought of spending £12m on a new, super-accurate postcode database and then destroying it is wasteful, a huge missed opportunity and to be frank, completely idiotic.

We implore you: don’t do it.

The Power of Information Taskforce have been figuring out how to liberate public sector information, how to facilitate better use of the modern, social web in government, and how to support the efforts of those outside government who are doing worthy things. All in all, they’re a great bunch of people, doing great work. They’ve just published their draft report. This is notable for two reasons.

First, the report has been published using Commentariat, a Wordpress theme from the folks at DIUS that makes it easy to browse and comment on big documents. It is fantastic — it really works well — and you’d never know it was Wordpress, unless you checked. It’s a great example of Wordpress’s flexibility.

For me, this informal consultation exercise is characterised by its ease. I read the report online. When I had something to say, I could just fill out the comment box. I could read other people’s comments, which helped to clarify my own thoughts. The PoI team have been posting comments too: responding to people, thanking them for their feedback, letting us know when they’ve made changes. Brilliant. It’s a real conversation between people who genuinely want to seek out ideas. It stands in stark contrast to the process of formal consultation, which is stifled, slow, more or less one-way, and frequently happens after all the important decisions have already been made. This exercise couldn’t be more different: agile, easy, conversational and public. I have no doubt that the report will be better because of it.

Second, the report itself is great: it hits all the right boxes. Be active in other people’s networks. Make sure civil servants have decent ‘net access — it’s astonishing how many of them are so filtered that they’re almost useless. Support third sector developers. Invest in innovative new ideas, even if they’re high risk. Liberate geodata from burdensome licensing and fees. There are 25 recommendations in all, and they’re all great.

If you have some time, read the beta report and leave some comments. It’s a great document, and great opportunity to get your thoughts in front of people who will listen.

I was at the UKGovWebBarcamp last weekend, and among the talks I attended was one by the Directgov Innovate team. This team has been recently formed, and is a really good development. In their own words:

Directgov have created the innovate.direct.gov.uk developer network to inform the greater developer community about available resources, to provide a platform to connect with one another, and to showcase new ideas with the aim of supporting and encouraging innovation.

Over time we will provide content feeds and API’s allowing people to develop new and interesting ideas and applications for use by the greater community.

Among my questions was: what will this team actually do? I was very glad to hear that they plan to develop new sites, make APIs, make data available to people and create a community of developers who are interested in this field. Great. This is just what’s needed. What’s even better is that they’ve already delivered on that promise.

Yesterday, Tom Watson tweeted that we should have a site where people can check to see if their school is closed. Brian Hoadley and Paul Clarke at Directgov took up the challenge, and just a couple of days later, launched a new site: SchoolClosures.org.uk.

It’s pretty rough around the edges: there doesn’t seem to be much RSS support, and there’s no access to the underlying data, and — well — it doesn’t tell you whether your school is closed… but it is still useful, and it’s very impressive that it appeared so quickly, and with such little prompting.

Kudos to all involved — this is a fantastic and very encouraging start.

Simon Dickson picks up an interesting story from the BBC’s Editors’ blog about official releases of statistics.

Usually, when the Department of Children, Schools & Families releases new statistics, they’re given to the media in advance. The media need this lead time to be able to format all their articles and tables and make sure everything is correct and works properly: this is fair enough, given that they’re the data they’re working with lives in lots of Excel spreadsheets, with multiple sections, differing layouts and everything else you really don’t want if you’re tasked with this kind of job.

Given what they have to work with, the BBC’s anger is understandable, but perhaps misplaced: why are we still dealing with bunch of spreadsheets in the information age? Why isn’t there an API that allows this data to be queried, or at the very least, a standard data format that doesn’t change from year to year, and doesn’t reply on proprietary technologies that are hard to work with?

An API or standard data format would allow media organisations to write code which generates the statistics they need every year. They wouldn’t have to create new tools to be tweaked and tested every time there are new statistics. Better still, it would create a market for someone to create a tool that did this for them, saving them money. Even better than that, it would allow anyone who wants to do something innovative with these statistics to do so far more easily.

I think I’m not alone in saying that the case for releasing data properly — in reusable formats, to everyone, for free, whenever it is possible — has been made, has been heard and has been widely accepted as valid.

Why are we still fiddling with messy spreadsheets, and bemoaning the fact that we only have days to do what should take hours?

Maps to display levels of crime, nationwide, were promised by the Government last year. It’s great to see that they’ve finally launched. This is a step in the right direction.

Unfortunately, it’s only a small one. Some forces have collaborated, but most have their own maps — which seems very strange. Why not a central service? We have the National Crime Reporting Standards, so there must be some consistency in the data from individual forces. It seems rather strange to implement 40-odd websites that all do the same thing, not to mention inconvenient: people who live in areas that adjoin other police forces will have to check two or more sites to get a good idea of the levels of crime in their neighbourhood.

It also means that the quality of these sites is extremely variable: from the good, to the average, to the downright mediocre. Why didn’t the Met simply share — or even, sell — its nice Google Maps mashup to the other forces around the country? Why waste money reinventing the wheel?

We would also love to see an API, or at least some way to get these stats in a machine-readable format. We’d like to be able to take this data and play with it. We’d like graphs that compare levels of crime to population, mean income, number of CCTV cameras, number of police officers per person, proximity to alcohol retailers, or anything else that takes our fancy. Of course, not all this data is readily available, but getting hold of crime stats would be a great improvement.

Finally: there’s some speculation over whether these maps break the Ordnance Survey’s licensing terms. Said terms are extremely bad: expect programmers who work in this sector to go into histrionics if you mention them. The Home Office have said, more or less, that they’ll sort it out. Quite how, or what was decided, is unclear: Did the OS get a shedload of money, or did the government come to some arrangement? If it’s the latter, I hope it may go some way towards helping third-sector projects too.

In any case, despite all these gripes, this is a great development. Congratulations are due to all involved: hopefully, this is just the first of many such innovations to come.