Skip to main content

S3 Destination Connection

Follow our setup guide to connect your Amazon S3 bucket to Crux.

Jon Tam avatar
Written by Jon Tam
Updated over 2 years ago

Overview

Amazon S3, or Amazon Simple Storage Service, is an object storage service offered by Amazon Web Services that provides highly scalable, cloud storage service. Data, in S3, is stored in containers called buckets.

Connect to Your Destination

To connect to your destination:

  1. Provide a unique connection name that you can refer to at a later time.

  2. Select S3 in the destination connection method.

  3. Provide the bucket name associated with the S3 account

  4. Optionally, you may define the destination folder path within this bucket to send your files. If left blank, a folder will be created based on the timestamp of the created destination.

  5. Provide the Amazon Resource Name (ARN) Role associated with this bucket. This can be obtained in the properties section of your S3 bucket. Note: this AWS ARN Role will need the following permissions:

    • Versioning Property: enabled

    • Policy s3:PutObjectTagging

    • Policy s3:PutObjectAcl

    • Policy s3:PutObject

    • Policy s3:GetObject

    • Policy s3:DeleteObject

    • Policy s3:ListBucket

  6. Optionally, if you require External ID for your AWS IAM role, provide this value here. For more information on External IDs, see this page.

  7. Crux supports various S3 encryption types for your bucket. Toggle on S3 encryption to see the following options:

    • AES_256: this is standard for encrypting data with a 256-bit key size.

    • AWS_KMS: this allows you to provide a key set by the AWS Key Management Service (KMS).

  8. Select the file types that you would like as part of delivery. Crux supports Avro, CSV, Parquet, and Raw formats.

  9. Optionally, you may deliver uncompressed files if you would like. Select the Uncompressed delivery checkbox for this format. By default, Crux will send compressed versions of files.

  10. Click Add connection to connect to the delivery channel.

Note: When testing a connection to a new destination, a test file is added into the target destination.

Connection fields

To create an S3 destination connection, configure the following connection settings:

Field

Description

Connection name

Unique name of your connection that you may quickly refer to at a later time.

Connection method

Type of connection: S3

Bucket name

The bucket name associated with the AWS S3 account. The namespace is shared by all AWS accounts.

Folder path (optional)

Destination folder path for the files.

AWS ARN Role

Amazon Resource Names (ARNs) uniquely identify Amazon Web Services (AWS) resources. ARN roles are required when you need to specify a resource unambiguously across all of AWS.

External ID (optional)

External IDs are used to grant access from one AWS role to another.

S3 Encryption

Options: S3 Encryption type (AES_256, AWS_KMS), Key ID, Key

  • AES_256: Standard for encrypting data with a 256-bit key size.

  • AWS_KMS: Gives you centralized control over the cryptographic keys used to protect your data. (It is recommended to provide an alias of the key ID instead of the actual key ID.)

  • Key ID and Key: Access Key Id and Secret Access Key are used for programmatic (API) access to AWS services.

File types

Delivery options: Avro, CSV, Parquet, Raw

Uncompressed delivery (optional)

If checked, delivers uncompressed files. Default: Compressed versions of the files. For example, gzip-compressed objects.

Did this answer your question?