Last Thursday, G2010 — a conference on Government 2.0 — finally arrived. As others have said, it was a fantastic day, with many interesting people in attendance and on Twitter.

At the beginning of the year, Jeffrey Peel asked us to organise a panel for the day on digital engagement. It’s an area we have some experience of — we made TellThemWhatYouThink.org and ConsultationXML, and are helping the COI to deploy RDFa to describe consultations documents on central government websites. We had a lively panel and discussion about consultation and how to do it, which you can now watch:

And here are my feelings on the subject — in a bit more detail than we could fit in on the day.

Consultation is an area in which there is intense activity. Departments consult on almost everything they do, local government has a statutory obligation to consult residents on a wide variety of issues, and the rise of social media in government has brought into sharp focus the things that the Web makes possible. But despite all this activity — very much including innumerable panels at conferences — we’re not achieving the mass participation in policymaking that is, for many, the goal of digital engagement. So what’s wrong?

I think that some very big assumptions have been made about digital engagement, its potential, and the right way to do it. Everyone’s been excited by the possibilities, myself included, but I think we’ve failed to really look at the people we’re trying to engage, their level of interest, motivation and available time.

The reality is that formal consultation is simultaneously necessary (in that a deliberative, evidence-based policymaking process is valuable) and expensive in terms of the investment of time and energy that people must make to participate. We have conflicting goals: to reduce the barriers to participation — make it quicker and easier — while also maintaining an informed policymaking process. Formal consultation is far from perfect and we should work to make it better, but it’s not obsolete.

The solution we’ve adopted so far is to try and make it easier to dip one’s toes into a formal consultation. This has been valuable, and we’ve learned a lot from it, but I don’t think it’s workable.

Such approaches can substantially increase the number of responses that consultations receive, but they’re usually not the right kind of responses. A formal consultation doesn’t much benefit from large volumes of anecdotal correspondence about personal experiences. That kind of input is extremely valuable, but by the time a policy has reached formal consultation, it’s too late to use it. That kind of engagement has to happen earlier.

It also has to happen more often. It’s simply no good to pick a time — essentially arbitrarily — to ask people about their experience of, for example, public services. A consultation on the NHS probably wouldn’t be of much interest to me 5 years after my operation, but if I’m asked straight away, I’d be much more likely to respond. The issues would be fresh and immediate, and I wouldn’t have moved on with my life.

Those experiences happen all the time. They constitute the raw reality of our society and the value, or lack of it, that Government succeeds in generating for people. They are innumerable, chaotic, disorganised, neverending and personal: just the unstructured, unrepresentative things that you don’t want in a formal consultation, but that have the potential to create real, valuable change in the way ministers, Parliamentarians, policymakers, civil servants and front-line staff do their jobs.

We need to pick apart these strands. First, we must take formal consultation on to the web, away from paper and PDFs, and engage those people who are interested in investing their time and effort in the process. Second, we must embed into government a culture of engagement, so that those who have stories to tell can tell them to the right people at the right time. Engagement is not the exclusive province of web, press and comms teams. It’s everyone’s job, and everyone must make time for it.

After all, “engagement” is just another word for “talking to people and finding out what on earth’s going on”.

Who couldn’t get behind that?

The UK’s version of data.gov, ably put together by the Cabinet Office, has just launched in private beta. We got to have a sneak peak, and it’s great!

The site is a blend of the US’s equivalent, data.gov, and Directgov | Innovate. It’s got a listing of available data packages, powered by the Comprehensive Knowledge Archive Network, and user-generated lists of apps and new ideas. This is just right: the data you need, combined with a way to promote the things you make and a place to get ideas if you’ve got itchy typing fingers but lack inspiration.

It’s not perfect. Conspicuously missing is an organised way to browse data sets: but that’s coming, along with some other tweaks and twiddlings that’ll improve the site’s usability.

The site is powered by Drupal, with packages catalogued and hosted by CKAN. Meanwhile, data.gov.uk hosts a data store powered by Talis that can scale to 100 billion triples and is hosted on Amazon EC2. The system is federated, so departments can add and control their own data, lots of which is available as RDF, with the remainder downloadable in spreadsheet form.

Speaking of spreadsheets, they’ve even written an app that departments can deploy in-house to convert spreadsheets into RDF (kudos to John Sheridan!) which makes it much easier for departments to produce structured, linked data.

This is all working now, and was put together by the team at the Cabinet Office in the last three months. This is a massive achievement, and it sounds like it’s just the beginning: they have big plans. User submissions for new datasets. Metadata to describe provenance. More data sets on the site. More data as RDF. Organised browsing for packages. Source code releases. The list goes on.

This is such an encouraging thing to see. No expensive procurement exercises for clunky, bespoke sites: instead, we have the right tools for the job, joined together. Simple things that do one job well, combined to form a more complex whole. It’s the Unix philosophy in action.

This is how all Government IT should work.

Our hearty congratulations go out to the team at the Cabinet office, with special thanks to Richard Stirling for spilling some of the beans. I had lots of questions and nitpicks, and every single one of them was answered reassuringly.

They’ve got a plan, and it’s a good one.