Meta-data for data.gov.uk datasets

Published by:
Government Digital Service
Last updated:
25 September 2017
Topic:
Towns and cities
Licence:
Open Government Licence

Summary

A dataset of all the meta-data for all of the datasets available through the data.gov.uk service. This is provided as a zipped CSV or JSON file. It is published nightly.

Updates: 27 Sep 2017: we've moved all the previous dumps to an S3 bucket at https://dgu-ckan-metadata-dumps.s3-eu-west-1.amazonaws.com/ - This link is now listed here as a data file.

From 13/10/16 we added .v2.jsonl dump, which is set to replace the .json dump (which will be discontinued after a 3 month transition). This is produced using 'ckanapi dump'. It provides an enhanced version of each dataset ('validated', or what you get from packageshow in CKAN API v3 - the old json was the unvalidated version). This now includes full details of the organization the dataset is in, rather than just the ownerid. Plus it includes the results of the archival & qa for each dataset and resource, showing whether the link is broken, detected format and stars of openness. It also benefits from being json lines http://jsonlines.org/ format, so you don't need to load the whole thing into memory to parse the json - just a line at a time.

On 12/1/2015 the organizations of the CSV was changed:

  • Before this date, each dataset was one line, and resources added as numbered columns. Since a dataset may have up to 300 resources, it ends up with 1025 columns, which is wider than many versions of Excel and Libreoffice will open. And the uncompressed size of 170Mb is more than most will deal with too. It is suggested you load it into a database, ahandle it with a python or ruby script, or use tools such as Refine or Google Fusion Tables.

  • After this date, the datasets are provided in one CSV and resources in another. On occasions that you want to join them, you can join them using the (dataset) "Name" column. These are now manageable in spreadsheet software.

You can also use the standard CKAN API if you want to search or get a small section of the data. Please respect the traffic limits in the API: http://data.gov.uk/terms-and-conditions

More from this publisher

All datasets from Government Digital Service

Related datasets

Search

Data links

Link to the data Format File added Data preview
Archive of old CSV/JSON dumps XML 25 September 2017 Not available
Latest organizations JSON dump (nightly) JSON 24 November 2016 Not available
Directory of CSV/JSON dumps HTML 09 May 2016 Not available
CKAN API API 10 March 2014 Not available
Latest datasets JSON dump (nightly) JSON 06 May 2013 Not available
Latest datasets CSV dump (nightly) CSV 06 May 2013 Preview

Supporting documents

Link to the document Format Date added
CKAN API documentation HTML 10 March 2014
CKAN API terms and conditions HTML 10 March 2014
API tips HTML 10 August 2015
Dumps tips HTML 07 December 2015

Edit this dataset

You must have an account for this publisher on data.gov.uk to make any changes to a dataset.

Sign in