Making public data easier to reuse

Finding public data can leave you with another job before you can use it. You may need to match organization names against another source or work out what a blank value means. By the time the dataset is ready, you have made decisions that someone else will need to understand.

If another team needs a similar result, how much of that work can it build on? A prepared file helps. So does an explanation of how it was made and which questions remain unresolved.

We want the effort of preparing data to help the next team too. With Commons, a team can publish a prepared dataset for others to subscribe to. Capabilities can also make repeatable collection rules and checks reusable across participating datasets.

The decisions inside a prepared dataset

Consider two lists of organizations. One contains stable identifiers and full names. The other uses shortened names, with some entries that could refer to more than one organization.

Someone has to reconcile those records and keep uncertain matches visible. The resulting table might include the original name, the matched identifier, and a note about the basis for the match. Another team can then examine the work and decide whether it is suitable for its purpose.

That second team may need different coverage or a different reporting period. Where the preparation fits, it has a useful starting point. Its effort can go toward the remaining questions and the work it came to do.

Give the prepared result a publication of its own

Commons provides infrastructure for coordinating data across organizations. A team can use its existing tools or a custom ingestion capability to clean, combine, or enrich data it is permitted to share, then publish the result as its own maintained publication. Authorized subscribers receive local copies they can use in their applications.

The team maintaining that result should explain its sources and preparation choices. In the organization-list example, that means documenting which identifiers it used, how it handled uncertain matches, and what the result covers. Publishing the table gives other teams access to the work; this context helps them assess it.

The preparing team can also share proposed matches with the original publisher. The publisher can review the evidence and decide whether any corrections belong in its source. That preparation can help both the next user and the people maintaining the original data.

Agree on what the fields mean

Shared definitions make this easier. W3C’s CSV on the Web recommendations describe metadata for column meanings, date formats, and required values. A schema can be reused across tables, giving software a common way to interpret and check them.

The Humanitarian Exchange Language offers a practical example. OCHA’s explanation of HXL shows how common tags can identify corresponding columns even when spreadsheets use different headings or column orders. It also describes the remaining work: people still have to agree on codes and on how different names refer to the same organization or place.

Those agreements help a team understand what it can reuse. A familiar file format makes the data readable; definitions and documented choices make its contents easier to interpret.

Put shared rules into use

Some preparation repeats with each new contribution. A developer can package suitable import logic and validation rules in a Commons capability, attach it to a dataset, and reuse the package with other datasets that follow the same model.

A capability is a small, signed package that can combine an importer with validation rules and ways to read the data, such as a table or search. Teams can reuse the package with datasets whose inputs and requirements it supports. The Capabilities page explains how these parts work together.

In our example, an ingestion capability could map a contributor’s organization identifier into an agreed field. A schema could require that field and check its format. These rules need to be implemented for the dataset; the team still decides how to resolve an uncertain name match.

An ecosystem can use this approach to agree on a collection and validation model while participants keep different source systems and applications. A prepared dataset shares the result of work already done. A capability shares implemented rules and tools, such as an importer or validation check, that teams can apply to their own data.

Keep the result understandable as it changes

Shared preparation needs someone to maintain it. When a source changes, the team responsible for the prepared publication decides whether to rerun its work and release a new version. Commons then synchronizes published changes to connected subscribers. We explored that update path in When public data changes.

The explanation needs attention too. If the team changes its matching method or the coverage of its dataset, readers need to know.

Give another team a starting point

Think about a dataset your team has already prepared. What would another team need to know before using it? Which recurring checks could you share along with the result?

If your team prepares data that others could use, join the Commons waitlist for updates and access announcements. We’d like to hear which part of your work you could share, and what would help you maintain it.