When public data changes

When a publisher corrects a dataset, everyone using a copy has another decision to make: does this change affect our work?

For the publisher, that correction may mean updating an export and explaining which version others should use. A team receiving it has to work out what changed before refreshing its report or application. An organization that has cleaned, combined, and republished the data may need to revisit that preparation before deciding whether to release an update.

We’re building Commons because those organizations need a way to keep their work connected as data changes. Commons is data coordination infrastructure: publishers maintain a publication, and other organizations subscribe to local copies that receive its published updates.

The people behind the download

A public dataset has a working life before anyone downloads it. People gather contributions, resolve missing information, and decide what the records should mean. The publishing team has to explain enough of that work for someone outside the organization to use the result.

Another team may prepare it for a different purpose. It might reconcile identifiers across sources or turn a difficult file into a table others can use. That preparation involves judgment. For uses that depend on current data, the team also needs a way to follow changes in its inputs.

The person building a report or application then depends on both kinds of work. They need the source data, the explanation that goes with it, and a way to understand later changes. Government agencies, research groups, and industry networks can play several of these roles at once.

Sharing a dataset creates a relationship between these people. Keeping that relationship working is part of the cost of making public data useful.

Finding the data is one part of the job

Catalogs do valuable work by helping people discover what exists. Data.gov’s publishing guide explains how it collects dataset descriptions and access links from publishing agencies. The agencies host the underlying files and maintain the information in their listings. A catalog entry helps someone find a source; the source still needs an owner.

Once a file arrives, its format tells only part of the story. A CSV may be straightforward to open while leaving questions about units, missing values, or whether two identifiers refer to the same organization. Those answers come from definitions and the people who maintain them.

Even an update date needs context. DCAT, the W3C standard for describing datasets in catalogs, distinguishes a change to a dataset from a change to its catalog entry. A recently edited listing does not, by itself, tell us how recent its records are. That distinction matters when someone is deciding whether the data is current enough for their work.

Following a correction across organizations

Public data changes for good reasons. A publisher may correct an error or revise an estimate as better information becomes available. The Office for National Statistics makes this explicit in its API documentation: an edition of a dataset can have several versions following corrections, revisions, or new data. The dataset name and reporting period alone may not identify the information behind an earlier result.

For the people using that information, several events have to line up. The source changes. Its publisher releases an update. A consuming team receives it and decides which outputs need refreshing. Those events can happen at different times, under different people’s control.

There is a separate question when someone returns to an older report: which version did we use? If a download has been replaced or its address has disappeared, answering that question can take work before the analysis even begins. W3C’s publishing guidance recommends version history and persistent identifiers, including useful information at an old address when data is removed. These are practical ways to help the next person understand what happened.

The difficulty grows when each organization maintains a separate way to receive, compare, and redistribute the same publication. The publisher supports several delivery processes. Downstream teams repeat checks. An intermediary has to revisit its preparation before releasing a new result.

Keep the publication connected

With Commons, a publisher connects selected data from its existing system through an application or integration. It maintains a publication that authorized organizations can subscribe to, including organizations in different programs or networks.

Each subscriber holds a local copy. When the publisher releases a change, Commons synchronizes it to connected subscribers. Their applications query the copies they control, so everyday reads do not all have to return to the publisher’s infrastructure.

The practical aim is fewer separate delivery processes for the publisher to maintain and a common way for subscribers to receive published changes. It becomes easier to build on the source’s ongoing work as more organizations use it.

Retained versions also give teams something specific to refer back to. An application can record the version used for a report and revisit that retained input later. A previously received local copy can remain readable during a connection interruption, while its version makes clear which release it contains.

The team using the data still decides when to rerun a calculation or refresh a dashboard. That decision belongs with the application and the people responsible for its results. Commons connects the publication to the subscriber’s copy; the application connects that copy to its own work.

Our How it works guide explains the publication and subscription model in more detail.

Let the next team build on the work

Coordination also begins before publication. A program collecting data can agree on what contributors should provide and the checks each record needs to pass. Commons capabilities package those rules for reuse. Integrations map supported inputs into the agreed structure, so participating teams can keep their own tools.

The people involved decide what the fields mean. A shared capability helps apply that agreement across contributions. Other capabilities provide views, search, or tools for applications working with the dataset. The Capabilities page explains these extensions.

Prepared data can be shared too. A team that has cleaned or combined permitted data can publish the result for other organizations to subscribe to. When its sources change, that team decides whether to repeat the preparation and publish a new version. The next team has a maintained result to build on.

Catalogs, standards, and existing integrations remain part of this work. Commons focuses on the ongoing coordination between organizations that publish data and those that depend on it.

Start with one recurring handoff

Our ambition is to make Commons the infrastructure people turn to for data coordination across ecosystems. Organizations working with public data need that coordination across the entire life of a dataset.

A useful starting point is one dataset you repeatedly publish or depend on. What happens when it changes? Who needs to find out, and which part of the work starts again?

If that sounds familiar, join the Commons waitlist for product updates and access announcements. We’d like to understand the data you work with and where keeping it connected becomes difficult.