Conversation
|
@zeroshade can I get your thoughts on this? spark follows this pattern: https://spark.apache.org/docs/latest/sql-data-sources-protobuf.html#handling-circular-references-protobuf-fields but parquet can't do null types my idea, is if we get to depth = x, then we drop the field? e.g. for 2 |
|
@zeroshade otherwise, as this is used for json type fields, we could convert to a json string. and not need to think about recursive depths |
|
if the intent here is to use it for specifically JSON fields, then i would say we just use a string and utilize the Canonical JSON extension type so that it gets stored in parquet as a JSON typed field. |
zeroshade
left a comment
There was a problem hiding this comment.
Looking good, just a few nit picks
| "google.golang.org/protobuf/encoding/protojson" | ||
| "google.golang.org/protobuf/types/known/structpb" |
There was a problem hiding this comment.
can you shift these includes to be grouped with the others of the same prefix below?
| "fmt" | ||
| "google.golang.org/protobuf/types/known/structpb" |
There was a problem hiding this comment.
same as above, please place this with the sorted imports below
|
@tscottcoombes1 Running |
Rationale for this change
extra support for protobuf common types, converting to parquet
What changes are included in this PR?
support for protobuf struct and tests
Are these changes tested?
yes
Are there any user-facing changes?
no, just support for new type