Last year we expanded our OSM Extracts service to produce what we call GeoJSONL. This is a data format that goes by many names: newline-delimited GeoJSON, line-oriented GeoJSON, GeoJSONSeq, or GeoJSON Text Sequences. Whatever its name, the format enables effficient reading and efficient writing of vector geo-data features.
For more background, see our previous blog post on GeoJSONL and see this new overview of the format’s many names and its defining properties.
Over the last year, there’s been even more progress on tools for writing and reading GeoJSONL (or whatever your preferred name is for this format).
GeoJSONL support in HERE XYZ
HERE XYZ now supports GeoJSONL ingest.
Don’t know what HERE XYZ is? Here’s how we summarized the platform in a blog post last year:
HERE XYZ provides the full set of components that "modern" geo developers have come to expect: GeoJSON input and output, "slippy map" libraries, a RESTful API, a JavaScript SDK, a cross-platform CLI, base maps and styles. As we've been experimenting with the HERE XYZ beta, we've found it provides some unique combinations that make it especially well suited to combining open geodata APIs — especially in large quantities.
Using the latest version of the HERE CLI, you can parse and upload GeoJSONL files to the HERE XYZ platform.
Here’s an update of our “Use Crowdsourced Data from Interline OSM Extracts in an XYZ Web Map” tutorial that uses the GeoJSONL format.
Download a GeoJSONL extract from OSM Extracts:
Upload to XYZ and set tags:
Previously you might hit memory limits on your computer if you tried to upload an extract for a big metro region. Now, thanks to GeoJSONL’s efficient parsing, you should be able to upload any extract of any size!
The growing toolchain
There’s a long history of tools that work with this format, and it’s been growing over the last year. Here are some more highlights:
Command line On the command line, read .geojsonl and .geojsons files using GDAL’s drv_geojsonseq.
Desktop GIS GDAL also powers QGIS, so now you can load .geojsonl and .geojsons files into our favorite desktop GIS sofware.
Parse in Python: Fiona is a Python library that wraps GDAL and provides a “neat and nimble” interface. The next release will include support for drv_geojsonseq.*
Statistics and data science in R If you’re loading vector geo-data into R, try the geojson package on CRAN. See these instructions for how to load a .geojsonl file directly from Interline OSM Extracts.
Convert from a shapefile Install the shapefile NPM package and run shp2json --newline-delimited to convert a shapefile to GeoJSONL. See these instructions.
Convert to vector tiles Tippecanoe can take GeoJSONL as input and turn it in to MVT vector tiles. It also provides its own tippecanoe-json-tool command to convert a standard GeoJSON file to GeoJSONL format (see these instructions).
Stream in to a web browser Watch a GeoJSONL file stream into a web in this interactive JavaScript notebook.† If you have an Interline OSM Extracts API key and an Observable account, try this notebook to select from any of the 200+ cities and watch the stream arrive:
When to try GeoJSONL
If you like GeoJSON but find your data files are too big to hold in memory, give one of these tools a try.
If you need to run a bunch of vector geo-data through a MapReduce operation that, give this format a try. (It’s easy to divide a GeoJSONL file in to chunks before parsing its exact contents.)
If you want to download OSM Extracts from Interline and explore even the largest of extracts using HERE XYZ, give this a try.
Notes
* Shoutout to Sean Gillies, the lead creator of Fiona and instigator of RFC 8142 - GeoJSON Text Sequences. We may have differing views on the record separator character. Regardless, his packages like Fiona and Rasterio are fabulous and we’re happy users of them.
† Shoutout to Brett Camper, the author of this Observable notebook and a fellow alum of Mapzen.
