by Mindstellar
StarLive demo
A dotted world map with teal and coral location pins on cities, joined by thin curved lines.

Why we built our own places dataset, and gave it away

ShopClass needed countries, states and cities it could ship with no strings attached. Nothing fit, so we built one from Wikidata and released it as CC0.

By Navjot Tomer · · Open data

Every classifieds site asks the same question when someone posts a listing: where is it? A country, then a state or province, then a city. In ShopClass you import a country from Listings → Locations in the admin, or with one command:

Terminal window
php oc-cli.php location:update --country=IN

That data has to come from somewhere. This post is about where ours comes from, why we built it ourselves, and why we gave it away.

The dataset everyone uses has a catch

Most projects reach for dr5hn/countries-states-cities-database. It is a good dataset, and our own first attempt, a 2020 repository called geodata, was a cleaned-up copy of it.

The catch is the licence. It is ODbL, which is share-alike. Anyone who redistributes it owes attribution, and the derived database has to stay under the same licence, through every person who uses it after you.

For a website that is a footnote. For software like ShopClass, which ships the data into every install, it is a permanent obligation attached to a table of city names, and it passes on to every site owner. We tell people they own their ShopClass site outright. A licence that follows the city list around did not fit that.

Why not GeoNames or OpenStreetMap?

Both are great, and both would fill real gaps in our data. But GeoNames is CC-BY, which needs attribution, and anything derived from OpenStreetMap is ODbL again.

So we set one rule and kept it: no source with an attribution or share-alike requirement goes in, however convenient. Where Wikidata alone cannot fill a gap, the fallbacks follow the same rule. Time zones come from the IANA time zone database and boundaries from Natural Earth, both public domain. Flag emoji and country domains are computed from the ISO code.

Wikidata, the long way round

Wikidata is CC0, so anything built only from it can be CC0 too. The hard part is that it is huge. The dump we read is about 982 GB, around 8.2 billion statements.

The Wikidata dump, about 982 GB and 8.2 billion statements, is scanned once and built offline into a CC0 release of 255 countries, 4,314 divisions and 1,671,395 cities and towns. ShopClass sites import it, and placedb.org serves it as a free JSON API.

The build runs in two steps:

  1. Scan. One streaming pass over the whole dump keeps only what we need, about 1.8 GB. It cannot decide what a city is yet, because that depends on relationships spread across the entire dump.
  2. Build. A second step does all the judging, with no network access. It takes about 7 minutes, against roughly 90 for the scan on a 4-core machine.

The output is deterministic: the same Wikidata dump gives byte-identical files. A validation step fails the release if any single country loses places, because a global total can look fine while one country quietly loses most of its towns.

What counts as a city?

This turned out to be the hardest part. Wikidata does not draw the line where you would expect, so the build has to decide:

  • Municipalities count, even though Wikidata files them as administrative areas, not settlements. Without them, every commune in France goes missing.
  • Former divisions do not. Over half of Japan’s first candidates were abolished municipalities.
  • Suburbs count, neighbourhoods do not. An Australian address names its suburb, so a picker without suburbs cannot hold most Australian addresses.
  • Positive evidence wins. A rule that dropped neighbourhoods also deleted Bern and Basel, through a chain of Wikidata classes. Anything that is also a city, town, village or municipality now stays.
  • Places where people are confined are not places people live, so prison and internment camps are out. Refugee camps stay, on purpose: Kutupalong holds 598,195 people.

What we gave up

Being strict about the source has a cost, and we would rather show it than hide it:

  • 883,194 settlements are dropped because Wikidata has no coordinates for them. About two thirds are villages in China. Every place we ship has a latitude and longitude.
  • 163,759 settlements have no usable name, because their only names are in Arabic script, which we do not transliterate yet.
  • Vietnam is undercounted.

Every gap is written down, with numbers, in the limitations document.

Where it lives now

The result is 255 countries and territories, 4,314 states, provinces and regions, and 1,671,395 cities and towns. It is all public domain, so you can use it in anything, commercial products included, with no credit owed.

  • The pipeline and releases: mindstellar/location-data on GitHub. The code is GPLv3; the data is CC0.
  • In ShopClass: your site reads the latest release, so a fixed place name reaches you without waiting for a new ShopClass version. The locations guide shows how to import a country or use your own mirror.
  • As an API: placedb serves the same data as plain JSON, with no key, no account and CORS open, so a browser can call it directly. It also has prefix files for autocomplete, so a city search box can work without a backend.

If you build something with it, or find a wrong place, a missing region or something silly, please open an issue. We would love to hear about it.