Prerequisites
Google Cloud Storage (GCS) is a file storage web service for storing and accessing data on Google Cloud Platform infrastructure. The service combines the performance and scalability of Google's cloud with advanced security and sharing capabilities.
Requirements and Permissions
Prior to setting up your GCS destination, there are some setup activities that you need to do to grant Crux access to writing data into your destination.
Grant Crux Access to Deliver Data
The Crux Service Account that will need access for the production environment is:
delivery-dispatch-core-service@crux-199318.iam.gserviceaccount.com
Granting access should be done at the bucket level, not at the project or organization level. Grant the Service Account noted above with Storage Legacy Bucket Writer and Storage Legacy Object Owner roles by selecting the relevant bucket, clicking the Permissions tab, and then clicking Grant Access.
This will now allow you to successfully create a GCS destination, with this GCS bucket, in the Crux External Data Platform.
Connect to Your Destination
To connect to your destination:
Provide a unique connection name that you can refer to at a later time.
Select GCS in the destination connection method.
Provide the GCS bucket name that you granted access to above.
Optionally, provide a folder path within this bucket that you would like to send your files. If left blank, a folder will be created based on the timestamp of the created destination.
Select the file types that you would like as part of delivery. Crux supports Avro, CSV, Parquet, and Raw formats.
Optionally, you may deliver uncompressed files if you would like. Select the Uncompressed delivery checkbox for this format. By default, Crux will send compressed versions of files.
Click Add connection to connect to the delivery channel.
Note: When testing a connection to a new destination, a test file is added to the target destination.
Connection fields
To create an GCS destination connection, configure the following connection settings:
Field | Description |
Connection name | Name your destination to help quickly reference it in the future. |
Connection method | Method for a destination connection. |
GCS bucket name | The bucket name associated with the Google Cloud Storage location. This bucket name is part of the domain name in the GCS cloud URL. |
Folder path (optional) | The destination folder path for the files. |
File types | Delivery options: Avro, CSV, Parquet, Raw |
Uncompressed delivery (optional) | If checked, delivers uncompressed files. Default: Compressed versions of the files. For example, gzip-compressed objects. |