Posts Tagged ‘public sector information’
Indian Hack Day 4th Aug 10
The Hack Day that we participated in in India was great. A real experience. I wasn’t quite sure what to expect of it. In the end, there wasn’t a lot of time for much hacking, but we had fantastic conversations about India’s problems and projects.
Chief among the Indian projects was some amazing mapping and visualisation work. David McCandless has done a great write-up on the Guardian Data Blog. Another fabulous project was National Election Watch, who create public data on candidates by entering the details from their affadavits. This site has one of the most awesome features I’ve ever seen on a political website: a crime-o-meter for each candidate.
But the most interesting thing which came up was the widespread use of missed calls as a way to communicate. Entire services have sprung up built on the idea that you call a number, let it ring once, and then hang up. Which is free. And then the service calls you back. The lack of any privacy regulation also allows these services to enter agreements with the telecomms companies to obtain location data on callers.
This led me to wonder: could a system be made, powered by missed calls, to report incidents of police abuse? Like an ultra-light-weight Ushahidi? So, along with a couple of the Indian devs, I put together a mock-up:
The pins need to be converted into coloured areas, but hopefully that conveys the general idea: that people who experience problems with the police — like being assaulted, or forced to pay a bribe — could anonymously report it, allowing statistics to be built up about where police abuses take place. These could then be compared to the official statistics, or to the numbers of official complaints made. And could allow people to get a real sense of the scale and geography of the problem.
Such a system could work — despite most Indians’ lack of internet access — because it’s powered by mobile phones, which almost everyone has. Even more than that, by taking advantage of missed calls, it becomes free to use and to operate, unlike SMS. And missed calls are a practical approach, because they’re already known and used by Indians in other contexts.
Unfortunately, because we only had the morning, I couldn’t do much more than that basic mock-up. But if anyone in India is reading this, and would like to take over the project, let me know! I’d love to see it up & running.
Update: I’ve just done a podcast interview about the trip with Steph Gray of Helpful Technology.
Dextrous Web to join PM’s delegation to India 26th Jul 10
I’m honoured and delighted to have been asked to join the PM’s delegation for his visit to India next week, in order to participate in a hack day with Indian developers in Bangalore. This should be a great event — we’ll actually only have 4 hours for the hacking, but hopefully we’ll be able to take some of the lessons we’ve learned about civic hacking here at home and apply them to some Indian problems. Very exciting.
From my loose understanding, India certainly faces some interesting and unique challenges. But I don’t know what they are, or how best the web can solve them. I’m sure there’s a broad section of Indian society for whom web applications are not a very practical solution to any of the challenges they face. At least for the time being. So, what are the problems experienced by Indians who are digitally included? And how can we use the web to solve them?
Should we be focusing on applications for mobile phones? Or on low-cost devices like the curiously iPad-like $35 laptop?
Should we aim to produce a quasi-public-service like FixMyStreet, or tools for accountability and transparency, like Armchair Auditor?
What sort of public data is available in India, both officially and for scraping? And how might we be able to use it to influence Indian public policy for the better?
As always, the aim for the day is to have something tangible, useful and interesting that we can show to people. Even if it’s only a screenshot or a very raw prototype.
Very grateful for your ideas, thoughts and advice!
Have your say on MPs' Expenses 7th Jan 10
Just before Christmas 2009, the Independent Parliamentary Standards Authority approached us to see if we’d be interested in producing the online part of their first consultation on MPs’ Expenses. This was exciting, to put it mildly — so we leapt in with both feet.
As is often the case with this kind of work, the deadlines were short and the team rushed. The consultation was only five weeks long, and there was even less time to analyse the responses after it closed. With this in mind, we came up with a format that gave respondents a meaningful way to respond, while also making the responses as easy as possible to analyse.
Most of the questions have yes/no answers, and a few have options you can choose between. Each page has a box for free text responses, instead of each question — and the box is limited to 500 characters. We were a bit nervous about limiting input in this way, but hopefully it’s a good compromise. Very few people contacted us with extra things they wanted to say, so it hopefully most people didn’t feel constrained.
Given the number of questions, the length of the text and the constraints of the form, it was really important to make the site as intuitive as possible. The text boxes make it very obvious if you put too much in them. The final form at the end confirms that you really are finished, and the buttons are subtly differentiated using icons and colour. You’ll also find that using the sidebar menu to jump around is safe — even if you’re in the middle of writing a response. We’re using javascript to make sure the form gets submitted. And everything’s unobtrusive, accessible and standards-compliant.
The site was implemented using WordPress. It’s a fantastic platform to work with. There’s no doubt that WordPress’s amazing flexibility is what made it possible to launch this quickly — and on time, and on budget. Everyone seemed pleased with the way it came out (including a real user) so we’re pretty pleased!
The wraps come off data.gov.uk! 1st Oct 09
The UK’s version of data.gov, ably put together by the Cabinet Office, has just launched in private beta. We got to have a sneak peak, and it’s great!
The site is a blend of the US’s equivalent, data.gov, and Directgov | Innovate. It’s got a listing of available data packages, powered by the Comprehensive Knowledge Archive Network, and user-generated lists of apps and new ideas. This is just right: the data you need, combined with a way to promote the things you make and a place to get ideas if you’ve got itchy typing fingers but lack inspiration.
It’s not perfect. Conspicuously missing is an organised way to browse data sets: but that’s coming, along with some other tweaks and twiddlings that’ll improve the site’s usability.
The site is powered by Drupal, with packages catalogued and hosted by CKAN. Meanwhile, data.gov.uk hosts a data store powered by Talis that can scale to 100 billion triples and is hosted on Amazon EC2. The system is federated, so departments can add and control their own data, lots of which is available as RDF, with the remainder downloadable in spreadsheet form.
Speaking of spreadsheets, they’ve even written an app that departments can deploy in-house to convert spreadsheets into RDF (kudos to John Sheridan!) which makes it much easier for departments to produce structured, linked data.
This is all working now, and was put together by the team at the Cabinet Office in the last three months. This is a massive achievement, and it sounds like it’s just the beginning: they have big plans. User submissions for new datasets. Metadata to describe provenance. More data sets on the site. More data as RDF. Organised browsing for packages. Source code releases. The list goes on.
This is such an encouraging thing to see. No expensive procurement exercises for clunky, bespoke sites: instead, we have the right tools for the job, joined together. Simple things that do one job well, combined to form a more complex whole. It’s the Unix philosophy in action.
This is how all Government IT should work.
Our hearty congratulations go out to the team at the Cabinet office, with special thanks to Richard Stirling for spilling some of the beans. I had lots of questions and nitpicks, and every single one of them was answered reassuringly.
They’ve got a plan, and it’s a good one.
Data: Weekly Fuel Prices 11th Aug 09
James Darling and I met up for a chat about some data-related stuffs this afternoon, and came across this data on average fuel prices via the Office of National Statistics. This struck us both as being very useful (any hauliers out there who want to make some a nice visualisations?) so we threw together a script to convert it into (much) more useful formats.
Check out Weekly Fuel Prices, in more formats than you can shake a stick at, here.
Reforming Ordnance Survey 25th Apr 09
People have been talking about the Ordnance Survey rather a lot recently.
It’s a very strange beast. It has mountains of really useful data: electoral boundaries, postcode databases and the locations of all sorts of buildings, not to mention roads, railways and green spaces. Unfortunately, it’s a quasi-independent body, which has to pay its own way. It charges heaps of money for access to all these datasets, and makes them available under strange and deeply inappropriate licences.
This has long been a problem for civic hackers, and by implication, everyone else: if you want to make a service that needs to turn a postcode into a geographic location, you need to use their data, and most of the time, you can’t. It costs an arm and a leg.
Fortunately, it’s clear to everyone that the OS needs to be significantly reformed. They own datasets like electoral boundaries, that are fundamental to the operation of a democracy. They’ll charge you if you want to use them, even if you’re the government and you’re organising an election. That’s just nuts, and we all know it. So what to do?
Thankfully, people have been working on it. The recent Power of Information Taskforce report had lots of good things to say: Recommendation 7 was almost perfect. Liberate postcodes and boundary data. Make basic mapping data available for free to all, for modest use. Simplify licensing. Its only real downside was the absence of any mention of derived data.
Put simply, if you use an OS map to create a database of something — like postboxes, hospitals or parks — then OS share copyright in that data. In this modern age of user generated content, that position is completely unsupportable. It’s a shameless grab for intellectual property, motivated by their desire to receive extra royalties. For a private company, that might be fair enough, but it’s reprehensible for a body that exists to provide a public service.
For anyone who thinks this is merely a theoretical problem, it may be of interest that all those lovely crime maps launched by police forces across the UK are probably in violation of the OS’s licensing terms.
This, then, was the background to the latest budget. Having heard that it would contain some new announcements, we were waiting for its publication with baited breath, and indeed: it promised reform. The following day, Ordance Survey published their new commercial strategy. It is underwhelming.
There are some good changes: more data, including boundary information, will be made available through OpenSpace, their API. They’re going to revise their definition of “commercial” so that sites that use their data can carry advertising without being required to pay for licences. But that, more or less, is it. The rest of the strategy revolves around converting people using free licences into ones that become financially sustainable so they can pay. Fair enough, but hardly groundbreaking.
The real problems remain. OS still own electoral boundary data and postcode boundaries & locations. They still decide if you’re commercial, and you still have to accept their onerous licences to use their data. OS still maintain a stranglehold on any data that they consider to be derived from theirs. They’ll still charge you royalties to use that data, of at least £1000 a year. You still won’t be able to do anything with that data that’s not acceptable under their licence, like adding it to OpenStreetMap.
This new strategy is progress, but only just. It is at best a fractional improvement upon what we had before.
A lot more needs to be done.
The Office of National Statistics and Postcodes 12th Mar 09
Here’s a story from FreeOurData which is, quite frankly, incredible. The Office of National Statistics, in preparing for the next census, has found that the postcode databases offered by the Royal Mail and Ordnance Survey aren’t accurate enough for their purposes. Their solution: to build their own database. This is fair. The postcode database is not amazingly accurate, and ONS have different requirements anyway.
Unfortunately, Royal Mail and Ordnance Survey make good money from selling the postcode databases to other organisations. These datasets are very valuable: you’ve probably made use of them whenever you’ve put your postcode into a website. Royal Mail and Ordnance survey did not — apparently — like the idea of ONS making another postcode database with which they’d presumably have to compete. So, rather than take that nice dataset and do useful things with it — like giving it back to us taxpayers — the ONS have pledged to build the database, use it for the census, and then destroy it.
Postcode databases are almost a holy grail. Of all the datasets in the country, liberating the postcode database for free reuse would probably create more value than any other. The thought of spending £12m on a new, super-accurate postcode database and then destroying it is wasteful, a huge missed opportunity and to be frank, completely idiotic.
We implore you: don’t do it.
ConsultationXML is now Open Source 27th Feb 09
We’re terribly, fantastically pleased to announce that after a bit of wrangling, Steph Gray and I are able to release ConsultationXML as open source software under the GNU Affero license. The recent report on open source software in Government hinted that departments ought to try to release source code for the software they commission, and we’re delighted to be (we think!) the first to do so.
We’re not sure who will want to play with it yet. We hope that other departments will want to deploy and use the tool to improve their consultation offerings. It may be that people in the private sector will find some use for it. People have already used ConsultationXML for really neat things that we didn’t expect, so anything could happen: which is, of course, the point.
We ran this by the renowned geek-come-blogger-come-minister, Tom Watson, who had good things to say:
I think this is a great tool. We’ve just announced the Open Source, Open Standards and Re–Use report on the use of open source software in government, an element of which was to encourage government to contribute to the world of open source software, and this is the first practical expression of that goal.
For more information about ConsultationXML, and to download it, head over here.
Comment on the Power of Information Taskforce's report 11th Feb 09
The Power of Information Taskforce have been figuring out how to liberate public sector information, how to facilitate better use of the modern, social web in government, and how to support the efforts of those outside government who are doing worthy things. All in all, they’re a great bunch of people, doing great work. They’ve just published their draft report. This is notable for two reasons.
First, the report has been published using Commentariat, a WordPress theme from the folks at DIUS that makes it easy to browse and comment on big documents. It is fantastic — it really works well — and you’d never know it was WordPress, unless you checked. It’s a great example of WordPress’s flexibility.
For me, this informal consultation exercise is characterised by its ease. I read the report online. When I had something to say, I could just fill out the comment box. I could read other people’s comments, which helped to clarify my own thoughts. The PoI team have been posting comments too: responding to people, thanking them for their feedback, letting us know when they’ve made changes. Brilliant. It’s a real conversation between people who genuinely want to seek out ideas. It stands in stark contrast to the process of formal consultation, which is stifled, slow, more or less one-way, and frequently happens after all the important decisions have already been made. This exercise couldn’t be more different: agile, easy, conversational and public. I have no doubt that the report will be better because of it.
Second, the report itself is great: it hits all the right boxes. Be active in other people’s networks. Make sure civil servants have decent ‘net access — it’s astonishing how many of them are so filtered that they’re almost useless. Support third sector developers. Invest in innovative new ideas, even if they’re high risk. Liberate geodata from burdensome licensing and fees. There are 25 recommendations in all, and they’re all great.
If you have some time, read the beta report and leave some comments. It’s a great document, and great opportunity to get your thoughts in front of people who will listen.
Innovation in Government: SchoolClosures.org.uk 3rd Feb 09
I was at the UKGovWebBarcamp last weekend, and among the talks I attended was one by the Directgov Innovate team. This team has been recently formed, and is a really good development. In their own words:
Directgov have created the innovate.direct.gov.uk developer network to inform the greater developer community about available resources, to provide a platform to connect with one another, and to showcase new ideas with the aim of supporting and encouraging innovation.
Over time we will provide content feeds and API’s allowing people to develop new and interesting ideas and applications for use by the greater community.
Among my questions was: what will this team actually do? I was very glad to hear that they plan to develop new sites, make APIs, make data available to people and create a community of developers who are interested in this field. Great. This is just what’s needed. What’s even better is that they’ve already delivered on that promise.
Yesterday, Tom Watson tweeted that we should have a site where people can check to see if their school is closed. Brian Hoadley and Paul Clarke at Directgov took up the challenge, and just a couple of days later, launched a new site: SchoolClosures.org.uk.
It’s pretty rough around the edges: there doesn’t seem to be much RSS support, and there’s no access to the underlying data, and — well — it doesn’t tell you whether your school is closed… but it is still useful, and it’s very impressive that it appeared so quickly, and with such little prompting.
Kudos to all involved — this is a fantastic and very encouraging start.