Skip to content

Add GeoArrow encoding as an option to the specification #189

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cholmes merged 13 commits into from
Mar 25, 2024
Merged

Add GeoArrow encoding as an option to the specification #189

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
cd8c0a2
first stab
paleolimbot Nov 17, 2023
3452b53
slightly better
paleolimbot Nov 17, 2023
c94bcaf
Apply suggestions from code review
paleolimbot Jan 30, 2024
3d28ebc
fix
paleolimbot Jan 30, 2024
146dcf0
don't mix
paleolimbot Jan 30, 2024
f8e0ae2
modify proposal
paleolimbot Feb 27, 2024
35a2919
update example JSON
paleolimbot Feb 27, 2024
2a41943
add back in bytearray bit
paleolimbot Feb 27, 2024
6dc23dc
separate WKB and geoarrow sections
jorisvandenbossche Mar 11, 2024
5a996f8
add explanation about corresponding Parquet types
jorisvandenbossche Mar 11, 2024
0fac197
optional -> required for non-top-level nesting
jorisvandenbossche Mar 11, 2024
88ae045
clarify nulls and optional/required fields
jorisvandenbossche Mar 25, 2024
ec154cc
Apply suggestions from code review
jorisvandenbossche Mar 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions examples/example_metadata_geoarrow.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
{
"geo": {
"columns": {
"geometry": {
"encoding": "geoarrow",
"geoarrow_type": "geoarrow.point",
"geometry_types": [
"Point"
]
Comment thread
kylebarron marked this conversation as resolved.
Outdated
}
},
"primary_column": "geometry",
"version": "1.1.0-dev"
}
}
2 changes: 1 addition & 1 deletion format-specs/compatible-parquet.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ The core idea of the compatibility guidelines is to have the output match the de

* The geometry column should be named either `"geometry"` or `"geography"`.

* The geometry column should be a `BYTE_ARRAY` with Well Known Binary (WKB) used to define the geometries, as defined in the [encoding](./geoparquet.md#encoding) section of the GeoParquet spec.
* The geometry column should be a `BYTE_ARRAY` with Well Known Binary (WKB) used to define the geometries, as defined in the [encoding](./geoparquet.md#encoding) section of the GeoParquet spec. Alternatively, the geometry column can be stored according to the Point, MultiPoint, MultiLineString, or MultiPolygon memory layouts with separated (struct) coordinates as specified in the [GeoArrow format](https://geoarrow.org/format).
Copy link
Copy Markdown
Contributor

@Kontinuation Kontinuation Jan 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean that geometry column with mixed types of geometries cannot be encoded as GeoArrow?

Copy link
Copy Markdown
Collaborator

@kylebarron kylebarron Jan 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not until apache/parquet-format#44 is merged (if ever)

Copy link
Copy Markdown
Collaborator Author

@paleolimbot paleolimbot Jan 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It can (or will be once we sort out the details of geoarrow/geoarrow#43 ), although it's unclear exactly how we'd do that in Parquet or if it would be useful in Parquet. In any case, it would be a future addition!

Copy link
Copy Markdown
Contributor

@Kontinuation Kontinuation Jan 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Columns with mixed geometry values are quite common for most query engines with geospatial support. Most of the time geometry columns have the umbrella type "geometry" or "geography", and it is not practical to first resolve the subtypes of the geometries before writing out parquet files. I'd look forward to a columnar encoding supporting mixed geometry types as well as geometry collections.

Copy link
Copy Markdown
Collaborator

@kylebarron kylebarron Jan 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Arrow and Parquet are two different specs. Arrow has a union type, which allows for mixed geometry types in a single column, while maintaining constant-time access to any coordinate. Parquet does not today have a union type, so it's impossible to to write the Geometry and GeometryCollection arrays in geoarrow/geoarrow#43 natively to Parquet.

GeoArrow implementations are able to statically know whether they have singly-typed geometries in a column, in which case they can write one of the 6 primitive types. GeoArrow implementations will have to fall back to WKB-encoded geometries for mixed-type columns. I don't see how this is something we could realistically change, unless we essentially re-implement union handling in a struct, which would be a big ask for implementors.

Copy link
Copy Markdown
Collaborator Author

@paleolimbot paleolimbot Jan 31, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All good points!

I think Kyle put the summary best:

In any case, I'd argue that a large majority of use cases will be solved by these 6 primitive types, and we can come back to union types in the future.

In the context of this PR, that would mean that the column option geoarrow_type could in the future be set to "geoarrow.mixed".

I don't think we anticipated that writing mixed geometries in geoarrow to Parquet would be the main use-case. If this is an important use, please chime in on geoarrow/geoarrow#43 with some details! We definitely don't want to represent mixed geometries in a way that prevents them being used.

The only way to know it is that scanning every single record of a big dataset first (get all geometry types), then in the second round, write the table to GeoParquet

This is only true if there's no ability for a user to supply any information about the encoding. If there is, write_geoparquet(..., encoding = geoarrow("geoarrow.point")) should do it. Typically the user does know this information (even if the database does not).

Copy link
Copy Markdown
Collaborator

@kylebarron kylebarron Jan 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This discussion is making me think that the GeoParquet spec should not be defined in terms of GeoArrow. Rather it should be defined as "native type" or "flat type" or similar. Then a sentence in prose can mention that it overlaps partially with the GeoArrow spec.

I'm also becoming convinced that the serialized form need not exactly overlap with GeoArrow. On the topic of mixed arrays specifically, as possibly the only one who has written an implementation of GeoArrow mixed arrays, I've become an even stronger proponent of using an Arrow union type for GeoArrow mixed arrays because of its ability to know geometry types statically. So I think the best way forward for GeoParquet (for a future PR) would be to discuss a "struct-union" approach for GeoParquet that is not the same in-memory representation as GeoArrow.

all database engines that didn't use Arrow as the in-memory format have no way to know if a geometry column is a mixed type or not. In other words, they don't know if they can use this GeoArrow encoding to GeoParquet files

I think changing nomenclature will also be clearer to non-arrow-based implementations that reading and writing this "native" encoding of GeoParquet is not dependent on using Arrow internally.

So my recommendation would be to take out most references to geoarrow from this PR. I.e., we don't want the metadata key called geoarrow_type if there's a possibility where the GeoParquet union type is not the same as the GeoArrow union type.

Copy link
Copy Markdown
Collaborator Author

@paleolimbot paleolimbot Jan 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually think the strength of this PR is the strong delegation to the GeoArrow specification: I don't think we should be developing two specifications, particularly since we have very little engagement on the GeoArrow spec already. We've spent quite a bit of time documenting the memory layouts for geoarrow in that specification and I don't think it would be productive to copy/paste those here and maintain them independently. I also don't think it would be productive to link to the GeoArrow specification for documentation of all the memory layouts but very pointedly not call it GeoArrow.

It may be that representing mixed geometry is not important in the context of GeoParquet (Maybe WKB is just as fast in the context of compression + IO + Parquet's list type? Have we checked?), or it may be that there is a common memory representation that makes sense for both specifications that will improve interoperability (although that would involve quite a lot of reimplementation on Kyle's end 😬 ).

I don't want us to loose track of the main point here, which is that this PR is mostly about enabling very efficient representations of single-type geometries, which are very commonly the types of files that you might want to put in a giant S3 bucket and scan efficiently.

Copy link
Copy Markdown
Collaborator

@jorisvandenbossche jorisvandenbossche Feb 12, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This discussion is making me think that the GeoParquet spec should not be defined in terms of GeoArrow. Rather it should be defined as "native type" or "flat type" or similar. Then a sentence in prose can mention that it overlaps partially with the GeoArrow spec.

We could go back to to the question about which "encoding" values to allow, and instead of a generic "geoarrow" option (with an additional "geoarrow_type" key to then be more specific), have actual encoding options "point", "linestring", "polygon", etc.
(i.e. like one of the options we initially discussed was also "geoarrow.point", "geoarrow.linestring", etc, but then just dropping the "geoarrow." prefix)

For the rest, it is mostly a question about how to document this: how to phrase this exactly in the specification, how strongly to tie it to geoarrow (or just reference as mostly similar), how much to duplicate the details of the memory layout, etc. But that's all "details" about the best way to document it, while keeping the actual specification (what ends up in the metadata in a file) agnostic to geoarrow.

I think I am somewhat convinced by @kylebarron's points on this, and like to idea of having the actual spec changes not use "geoarrow" (and then we can still debate how much to use the term in the explanation of it).

For example, as long as the specification would exactly overlap (or be a strict subset of geoarrow), we can still point to the geoarrow spec for the details to avoid too much duplication (and having to maintain two versions). And this is also easy to change then in the future if we would want to introduce differences.

I also don't think it would be productive to link to the GeoArrow specification for documentation of all the memory layouts but very pointedly not call it GeoArrow.

On the other hand, for an implementation of GeoParquet in some library that has nothing to do with Arrow (doesn't use an Arrow implementation under the hood), the "geoarrow" name is also somewhat uninformative, when strictly looking at it from a GeoParquet point of view.

Copy link
Copy Markdown
Collaborator Author

@paleolimbot paleolimbot Feb 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps it's worth crafting a new PR that uses the language you all are hoping for with a draft implementation? I don't currently have the bandwidth to do that but am happy to review!


* All data is stored in longitude, latitude based on the WGS84 datum, as defined as the default in the [crs](./geoparquet.md#crs) section of the GeoParquet spec.

Expand Down
19 changes: 12 additions & 7 deletions format-specs/geoparquet.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,7 @@ This is version 1.1.0-dev of the GeoParquet specification. See the [JSON Schema

## Geometry columns

Geometry columns MUST be stored using the `BYTE_ARRAY` parquet type. They MUST be encoded as [WKB](https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry#Well-known_binary).
Copy link
Copy Markdown
Collaborator

@jorisvandenbossche jorisvandenbossche Feb 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we keep the information of the first sentence somewhere? (i.e. that for a WKB encoding, the geometry column MUST be stores using the BYTE_ARRAY parquet type)

(you kept the "Implementation note" just below that also mentions BYTE_ARRAY, but that is not so specific as the above)

Copy link
Copy Markdown
Collaborator Author

@paleolimbot paleolimbot Feb 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done! We could also add some more in here about the Parquet physical description of how nesting works (but maybe in a future PR?)


Implementation note: when using the ecosystem of Arrow libraries, Parquet types such as `BYTE_ARRAY` might not be directly accessible. Instead, the corresponding Arrow data type can be `Arrow::Type::BINARY` (for arrays that whose elements can be indexed through a 32-bit index) or `Arrow::Type::LARGE_BINARY` (64-bit index). It is recommended that GeoParquet readers are compatible with both data types, and writers preferably use `Arrow::Type::BINARY` (thus limiting to row groups with content smaller than 2 GB) for larger compatibility.

See the [encoding](#encoding) section below for more details.
Geometry columns MUST be encoded as [WKB](https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry#Well-known_binary) or [GeoArrow](https://geoarrow.org/). See the [encoding](#encoding) section below for more details.

### Nesting

Expand Down Expand Up @@ -51,13 +47,14 @@ Each geometry column in the dataset MUST be included in the `columns` field abov

| Field Name | Type | Description |
| -------------- | ------------ | ----------- |
| encoding | string | **REQUIRED.** Name of the geometry encoding format. Currently only `"WKB"` is supported. |
| encoding | string | **REQUIRED.** Name of the geometry encoding format. Currently `"WKB"` and `"geoarrow"` are supported. |
| geometry_types | \[string] | **REQUIRED.** The geometry types of all geometries, or an empty array if they are not known. |
| crs | object\|null | [PROJJSON](https://proj.org/specifications/projjson.html) object representing the Coordinate Reference System (CRS) of the geometry. If the field is not provided, the default CRS is [OGC:CRS84](https://www.opengis.net/def/crs/OGC/1.3/CRS84), which means the data in this column must be stored in longitude, latitude based on the WGS84 datum. |
| orientation | string | Winding order of exterior ring of polygons. If present must be `"counterclockwise"`; interior rings are wound in opposite order. If absent, no assertions are made regarding the winding order. |
| edges | string | Name of the coordinate system for the edges. Must be one of `"planar"` or `"spherical"`. The default value is `"planar"`. |
| bbox | \[number] | Bounding Box of the geometries in the file, formatted according to [RFC 7946, section 5](https://tools.ietf.org/html/rfc7946#section-5). |
| epoch | number | Coordinate epoch in case of a dynamic CRS, expressed as a decimal year. |
| geoarrow_type | string | The [GeoArrow extension name](https://geoarrow.org/extension-types#extension-names) corresponding to the column's memory layout. This is required when `encoding` is `"geoarrow"` and must be omitted otherwise. |

#### crs

Expand All @@ -83,10 +80,18 @@ The optional `epoch` field allows to specify this in case the `crs` field define

#### encoding

This is the binary format that the geometry is encoded in. The string `"WKB"`, signifying Well Known Binary is the only current option, but future versions of the spec may support alternative encodings. This SHOULD be the ["OpenGIS® Implementation Specification for Geographic information - Simple feature access - Part 1: Common architecture"](https://portal.ogc.org/files/?artifact_id=18241) WKB representation (using codes for 3D geometry types in the \[1001,1007\] range). This encoding is also consistent with the one defined in the ["ISO/IEC 13249-3:2016 (Information technology - Database languages - SQL multimedia and application packages - Part 3: Spatial)"](https://www.iso.org/standard/60343.html) standard.
This is the memory layout used to encode geometries in the geometry column.

The preferred option for maximum portability is `"WKB"`, signifying Well Known Binary. This SHOULD be the ["OpenGIS® Implementation Specification for Geographic information - Simple feature access - Part 1: Common architecture"](https://portal.ogc.org/files/?artifact_id=18241) WKB representation (using codes for 3D geometry types in the \[1001,1007\] range). This encoding is also consistent with the one defined in the ["ISO/IEC 13249-3:2016 (Information technology - Database languages - SQL multimedia and application packages - Part 3: Spatial)"](https://www.iso.org/standard/60343.html) standard.

Note that the current version of the spec only allows for a subset of WKB: 2D or 3D geometries of the standard geometry types (the Point, LineString, Polygon, MultiPoint, MultiLineString, MultiPolygon, and GeometryCollection geometry types). This means that M values or non-linear geometry types are not yet supported.

Using the `"geoarrow"` encoding may provide better performance and enable readers to leverage more features of the Parquet format to accelerate geospatial queries (e.g., row group-level min/max statistics). When `encoding` is set to `"geoarrow"`, the column metadata must also specify `geoarrow_type` according to the [GeoArrow metadata specification for extension names](https://geoarrow.org/extension-types#extension-names) to signify the memory layout used by the geometry column.

Note that the current version of the spec only allows for a subset of GeoArrow: separated (struct) coordinates are required, only 2D or 3D geometries are permitted, and supported extension are currently `"geoarrow.point"`, `"geoarrow.linestring"`, `"geoarrow.polygon"`, `"geoarrow.multipoint"`, `"geoarrow.multilinestring"`, and `"geoarrow.multipolygon"`. This means that M values and serialized encodings are not yet supported.
Copy link
Copy Markdown
Collaborator

@kylebarron kylebarron Jan 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I see, this doc doesn't even allow interleaved coordinates as GeoParquet.

I'm sensitive to the complexity concerns of having too many options in the spec, but I see this as favoring the "support cloud-native remote queries" use case over the "efficient file format, but reading and writing whole tables" use case. It "feels" like there's still a strong pull in general towards storing interleaved coordinates across the geo ecosystem.

That said, the memcopy to and from separated coordinates is pretty fast, so I can tolerate this.

Copy link
Copy Markdown
Collaborator Author

@paleolimbot paleolimbot Jan 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The summary of why interleaved coordinates are not a good canidiate as of this writing are:

  • They don't give useful column statistics
  • Current tools are slow to read them compared to separated encodings (important for points)
  • NULL values randomly error in some cases

Demo of column statistics:

import geoarrow.pyarrow as ga
import pyarrow as pa
from pyarrow import parquet
import numpy as np

array_interleaved = ga.as_geoarrow(["POINT (0 100)", "POINT (2 102)"], coord_type=ga.CoordType.INTERLEAVED)
tbl_interleaved = pa.table([array_interleaved], ["geom"])

parquet.write_table(tbl_interleaved, "test_interleaved.parquet")


f = parquet.ParquetFile("test_interleaved.parquet")
f.metadata.row_group(0).column(0).statistics
<pyarrow._parquet.Statistics object at 0x1223e7600>
  has_min_max: True
  min: -0.0
  max: 102.0
  null_count: 0
  distinct_count: None
  num_values: 4
  physical_type: DOUBLE
  logical_type: None
  converted_type (legacy): NONE

Demo of slowness:

import geoarrow.pyarrow as ga
import pyarrow as pa
from pyarrow import parquet
import numpy as np

n = int(1e6)
array = ga.point().from_geobuffers(None, np.random.random(n), np.random.random(n))
array_interleaved = ga.as_geoarrow(array, coord_type=ga.CoordType.INTERLEAVED)
tbl = pa.table([array], ["geom"])
tbl_interleaved = pa.table([array_interleaved], ["geom"])

parquet.write_table(tbl, "test.parquet")
parquet.write_table(tbl_interleaved, "test_interleaved.parquet")

%timeit parquet.read_table("test.parquet")
#> 7.36 ms ± 2.21 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit parquet.read_table("test_interleaved.parquet")
#> 15.8 ms ± 49.2 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Demo of random errors for NULL values:

import geoarrow.pyarrow as ga
import pyarrow as pa
from pyarrow import parquet
import numpy as np

array_interleaved = ga.as_geoarrow(["POINT (0 1)", None], coord_type=ga.CoordType.INTERLEAVED)
tbl_interleaved = pa.table([array_interleaved], ["geom"])

parquet.write_table(tbl_interleaved, "test_interleaved.parquet")
parquet.read_table("test_interleaved.parquet")
#> ArrowInvalid: Expected all lists to be of size=2 but index 2 had size=0

These are probably all solveable/might be unique to Arrow C++-backed implementations, but I am not sure it is the best encoding to start with (and it does seem like a good idea to start with just one encoding to minimize burden on implementors).


Implementation note: when using WKB encoding with the ecosystem of Arrow libraries, Parquet types such as `BYTE_ARRAY` might not be directly accessible. Instead, the corresponding Arrow data type can be `Arrow::Type::BINARY` (for arrays that whose elements can be indexed through a 32-bit index) or `Arrow::Type::LARGE_BINARY` (64-bit index). It is recommended that GeoParquet readers are compatible with both data types, and writers preferably use `Arrow::Type::BINARY` (thus limiting to row groups with content smaller than 2 GB) for larger compatibility.

#### Coordinate axis order

The axis order of the coordinates in WKB stored in a GeoParquet follows the de facto standard for axis order in WKB and is therefore always (x, y) where x is easting or longitude and y is northing or latitude. This ordering explicitly overrides the axis order as specified in the CRS. This follows the precedent of [GeoPackage](https://geopackage.org), see the [note in their spec](https://www.geopackage.org/spec130/#gpb_spec).
Expand Down
6 changes: 5 additions & 1 deletion format-specs/schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
"properties": {
"encoding": {
"type": "string",
"const": "WKB"
"pattern": "^(WKB|geoarrow)$"
},
"geometry_types": {
"type": "array",
Expand Down Expand Up @@ -71,6 +71,10 @@
},
"epoch": {
"type": "number"
},
"geoarrow_type": {
"type": "string",
"pattern": "^geoarrow\\.(point|linestring|polygon|multipoint|multilinestring|multipolygon)$"
Copy link
Copy Markdown
Collaborator

@kylebarron kylebarron Jan 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not a json schema expert, but would we be able to make this conditionally required? It looks like dependentRequired meets what we need, though I don't know what version of json schema we're pinned to.

}
}
}
Expand Down