Knowing where your data came from

When someone asks where a report’s data came from, they should be able to check the answer easily. You may know the organization that published it. The person reviewing your work may also need to know which release you used and what your team changed along the way.

That explanation can become harder to follow as data moves between systems. A filename may carry the publisher’s name. A link may lead to its website. Neither tells the reader exactly what happened to the copy in front of them.

We’re building Commons to make sharing data easier to follow across organizations. Teams publish datasets for others to subscribe to. Recipients can see who publishes the dataset and check signed changes in their local copies.

Give the next person something they can check

The person receiving a dataset may never have spoken to its publisher. They need enough information to understand its origin and decide whether they can use it.

This is often called data provenance: the account of a dataset’s origin and the work that produced it. W3C’s guidance for publishing data on the web recommends describing both the data’s origins and the changes made to it. The organization publishing a dataset may be different from the one that originally collected it.

A useful explanation identifies the publication and the release being used. If another team combined or prepared the data, it should also explain that work. This gives the next person a starting point for checking the result, including questions that only the publisher can answer.

Check the data that arrives

Commons is infrastructure for coordinating data between organizations. It organizes shared records and files in publications called Stores. Each Store is attributed to its publisher, and each change carries the signature of an author the Store permits.

A recipient can use a content fingerprint to check that a record has not been altered since it was signed. These checks give a receiving team evidence about the data it has, even when its application reads from a local copy.

That matters because the location of a copy should not be the only clue to its source. A team using data in its own system still needs to verify what it received.

A signed record can still contain mistakes. Users still need to assess its quality and whether it fits their work. They also need to establish that they are following the intended publisher. Our Assurance guide explains the checks Commons provides and the decisions that remain with the people responsible for the data.

An accreditation register in practice

Consider a body publishing a register of accredited organizations. Another organization wants to use that register in an application. Its team needs to link a record to the publishing body and understand what the accreditation covers.

With Commons, the receiving team can check the signed changes in the publication it subscribes to. The publishing body explains what the fields mean and the scope of each record. The team using the register decides whether the information is current enough and applies to the activity it is reviewing.

This gives the team a clearer basis for its work. It can compare the received record with the signed publication, then assess what the record means for its decision.

Explain what happens between the source and the result

A team may combine that register with another dataset, reconcile organization names, and publish a prepared result. That creates another responsibility: explaining how the result relates to its inputs.

The signature on the prepared publication concerns what that team published. The team still needs to document its source publications and the preparation it performed. Keeping those references lets a reviewer examine the work behind the result.

We explored how teams can share that preparation in Making public data easier to reuse. A useful prepared dataset includes enough explanation for another team to assess it.

Be able to return to what you used

Questions about a source may come after a report is finished. To return to it later, the application or team needs to record the version it used and ensure that version is retained. Commons lets people inspect retained versions and compare them with current records.

We want it to be easier for someone joining the work later to understand the evidence behind it. Keeping the publication identifiable, recording the version used, and explaining preparation all help that person pick up the work.

Think about a dataset your team shares or depends on. If someone asked where a particular record came from, what could you show them? If this is a recurring part of your work, join the Commons waitlist for product updates and access announcements.