[C++] Add filesystem / IO implementation for Google Cloud Storage

[asfimport]

See example jumping off point

https://github.com/tensorflow/tensorflow/tree/master/tensorflow/core/platform/cloud

Reporter: Wes McKinney / @wesm
Assignee: Carlos O'Ryan / @coryan

Related issues:

• [Plasma] Add GCS as an external store for Plasma (blocks)
• [C++] Refactor Abseil build in ThirdpartyToolchain (is blocked by)
• [C++] Add google_cloud_cpp_storage to ARROW_THIRDPARTY_DEPENDENCIES (is a parent of)
• [C++] GCS integration tests leak processes (is a parent of)
• [C++] GcsFileSystem tests should use PYTHON environment variable (is a parent of)
• [C++] Create GcsFileSystem skeleton (is a parent of)
• [C++] Add google-cloud-cpp to ThirdpartyToolchain (is a parent of)
• [Packaging][C++] Add google-cloud-cpp to conda-forge (is a parent of)
• [C++] Update vcpkg.json builtin-baseline (is a parent of)
• [C++] Add GCS library to vcpkg.json (is a parent of)
• [C++] Update crc32c dependency to 1.1.2 (is a parent of)
• [C++] Install GCS testbench for CI builds (is a parent of)
• [C++] Implement streaming reads for GCS FileSystem (is a parent of)
• [C++] Implement streaming writes for GCS FileSystem (is a parent of)
• [C++] Implement "random access" reads for GCS FileSystem (is a parent of)
• [C++] Update google-cloud-cpp dependency to v1.32.1 (or latest) (is a parent of)
• [C++] Add GCS library to conda files (is a parent of)
• [C++] Reduce memory use in GcsFileSystem::OpenInputStream (is a parent of)
• [C++] Implement GcsInputStream::ReadMetadata (is a parent of)
• [C++] Implement GcsFileSystem::CopyFile (is a parent of)
• [C++] Implement GcsFileSystem::DeleteFile (is a parent of)
• [C++] Implement GcsFileSystem::DeleteRootDirContents (is a parent of)
• [C++] Implement GcsFileSystem::DeleteDirContents (is a parent of)
• [C++] Implement GcsFileSystem::DeleteDir (is a parent of)
• [C++] Implement GcsFileSystem::CreateDir (is a parent of)
• [C++] Implement GcsFileSystem::GetFileInfo(const FileSelector&) (is a parent of)
• [C++] Implement generic filesystem tests for GCS (is a parent of)
• [C++] Decide if we implement GcsFileSystem::Move (is a parent of)
• [C++] Add factory function for GcsFileSystem (is a parent of)
• [C++] Support credential types in GcsFileSystem (is a parent of)
• [C++] Add size and mtime to GcsFileSystem::GetFileInfo (is a parent of)
• [C++] Implement max recursion for GcsFileSystem (is a parent of)
• [C++] Refactor GcsFileSystem tests helpers (is a parent of)
• [C++] Speed up GcsFileSystem tests (is a parent of)
• [C++] GcsFileSystem reduce test sizes (is a parent of)
• [Python] Add bindings for GCS filesystem (causes)
• [C++] Allow creating GCS filesystem from URI (causes)

_{Note: This issue was originally created as ARROW-1231. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
JSON API:
https://cloud.google.com/storage/docs/json_api/v1/

C++ Client library:
https://googleapis.github.io/google-cloud-cpp/
https://googleapis.dev/cpp/google-cloud-storage/latest/

[asfimport]

Antoine Pitrou / @pitrou:
It doesn't look like there are any locally-running GCS-compatible servers, or at least I haven't found any...

[asfimport]

Wes McKinney / @wesm:
Ah I'm really glad there's an official C++ library now

https://github.com/googleapis/google-cloud-cpp

Note that this library began after I opened this issue originally so it's good that we waited

https://github.com/googleapis/google-cloud-cpp/graphs/contributors

Probably this will need to be added to conda-forge. We'll have to have libcurl in our build toolchain also...

[asfimport]

Lei (Eddy) Xu:
Hey @wesm , do we have plan to implement this now?

We are very interested into this feature.

[asfimport]

Wes McKinney / @wesm:
I think we need some information about what is the recommended way (fastest, most robust) in C++ to use GCS. @brills or @emkornfield do you know what is the current state of the art (is it google-cloud-cpp?)

[asfimport]

Zhuo Peng / @brills:
I don't work on related stuff, but looking at our internal site, google-cloud-cpp seems to be right choice.

Micah might know more.

 

https://googleapis.dev/cpp/google-cloud-storage/latest/ seems to be the documentation for https://googleapis.github.io/google-cloud-cpp/ ?

[asfimport]

Micah Kornfield / @emkornfield:
I think Zhuo is correct.  I've reached out internally to the person I believe to be the owner to confirm.

[asfimport]

Frank Natividad:
Hi folks,

I'm confirming that the Cloud Storage library in https://github.com/googleapis/google-cloud-cpp is the current state of the art.

 

Cheers

[asfimport]

Antoine Pitrou / @pitrou:
Does it provide better performance than the S3 endpoint?

[asfimport]

Frank Natividad:
Hi Antoine, could you clarify what you mean by better performance than the S3 endpoint? 

[asfimport]

Antoine Pitrou / @pitrou:
Doesn't GCS provide a S3-compatible endpoint? Is it detrimental to use the AWS SDK as opposed to the native GCS APIs?

[asfimport]

Wes McKinney / @wesm:
Even if the performance of uploads and downloads is equivalent, I would guess that the SDK provided by the GCP development team will provide the most comprehensive access to GCS's features. And there are utilities (such as parallel uploading 1) developed with GCS's particular characteristics in mind

[asfimport]

Frank Natividad:
The XML API does exist and compatibility with S3 SDK is available but not for all operations. Not exhaustive, but three things I think about when telling you this:

• Permission management with IAM isn't available through the XML API and S3 SDK would need to rely on ACL management. The GCS team aligned with GCP permission management by supporting IAM policies at the Organization/Project/Bucket level. IAM policies is the canonical way to manage permissions on GCP. For example, Uniform Bucket Level Access was introduced to disable ACLs at Bucket and Object level and is required to support Conditional permission policies (IAM Conditions). ACL are still supported but not the canonical permission management and very specific to GCS.

• S3 API isn't fully supported through the XML API, for example, S3 multi-part uploads are not supported. Moreover, you'd need to develop around these limitations when you hit them.

• The GCS C++ library was design based on experience from user friction from existing libraries in other languages in mind.

  In this case, I would recommend the GCS C++ library if you're using C++ and Cloud Storage API over the S3 SDK.

  HTH

[asfimport]

Antoine Pitrou / @pitrou:
Thank you for the explanation. I agree we should use the GCS C++ SDK.
Note to self: there's a standalone GCS-compatible server at https://github.com/fsouza/fake-gcs-server, should be useful for testing.

[asfimport]

Clark Zinzow:
Where does this fit in y'all's priority queue at the moment? @pitrou  are you planning to take this on?  I saw that you did the S3 implementation.

I'm mostly interested in realizing the benefits of ARROW-8031 in Ray, but if no one plans on taking this issue in the next few weeks, I could try to find a few spare cycles to take on this issue (following the patterns set by Antoine's S3 implementation) and 8031 thereafter.