Send data to Amazon S3

Amazon Simple Storage Service (Amazon S3) stores customer data files of any size in any file formats.

This topic describes the steps that are required to send files to Amazon S3 from Amperity:

  1. Get details

  2. Add destination

  3. Add data template

Get details

Amperity can be configured to send data to Amazon S3. This may be done using cross-account role assumption (recommended) or by using IAM credentials.

Use cross-account roles

Amperity prefers to send data to customer-managed cloud storage. This approach ensures that customers can:

  • Use cross-account role assumption to manage access to data

  • Directly manage the files that are made available

  • Modify access without requiring involvement by Amperity; access may be revoked at any time by either Amazon AWS account, after which data sharing ends immediately

  • Directly troubleshoot incomplete or missing files

Amperity recommends to use cross-account role assumption to manage access to customer-managed cloud storage in Amazon S3. This allows managed security policies to control access to data.

Note

If you have already configured cross-account role assumption for an Amazon S3 data source you may use the “Amperity S3 Cross-Account Storage” credential for this destination. If you have not configured cross-account role assumption, ask your Amperity representative to help you with those configuration steps.

Use credentials

Amazon S3 requires the following configuration details:

  1. The IAM access key.

  2. The IAM secret key.

  3. The Amazon Resource Name (ARN) for a role with cross-account access. (This is the recommended way to define access to customer-managed Amazon S3 buckets.)

  4. The name of the Amazon S3 bucket to which Amperity will send data and its prefix.

  5. The public key to use for PGP encryption.

Optional workflows

The following sections describe additional workflows that are available. After Amperity sends data to an Amazon S3 bucket, you can configure downstream applications to consume that data and make it available to additional workflows.

Amazon Redshift

Amazon RedShift is a data warehouse located within Amazon Web Services that can handle massive sets of column-oriented data.

Amperity can be configured to send data to an Amazon S3 bucket, after which Amazon Redshift can be configured to load that data. Applications can be configured to connect to Amazon Redshift and use the Amperity output as a data source.

You may use the Amazon S3 bucket that comes with your Amperity tenant for the intermediate step (if your Amperity tenant is running on Amazon AWS). Or you may configure Amperity to send data to an Amazon S3 bucket that your organization manages directly.

The following applications can be configured to load data from Amazon Redshift as a data source:

AWS Lambda

AWS Lambda runs code for any type of application or backend service that can be configured to run automatically from within Amazon Web Services to support any downstream workflow.

For example, you can configure Amperity to send CSV data to Amazon S3, after which AWS Lambda processes that output prior to loading to Amazon Redshift using the COPY command .

Add destination

Configure Amperity to send files directly to Amazon S3.

To add a destination

  1. From the Destinations tab, click Add Destination. This opens the Add Destination dialog box.

  2. Enter the name of the destination and a description. For example: “Amazon S3” and “This sends query results to Amazon S3”.

  3. From the Plugin drop-down, select Amazon S3.

  4. The “iam-credential” credential type is selected automatically.

  5. From the Credential drop-down, select a credential that has already been configured for this destination or click Create a new credential, which opens the Create New Credential dialog box. For new credentials, enter a name for the credential, the IAM access key, and the IAM secret key. Click Save.

  6. Under Amazon S3 settings, add the name of the Amazon S3 bucket and the prefix. For example: “Amazon S3” and “upload”.

  7. From the File Format drop-down, select Apache Parquet (recommended), CSV, TSV, or PSV.

  8. Add a single character to be used as an escape character in the output file.

    Note

    If an escape character is not specified and quote mode is set to “None” this may result in unescaped, unquoted files. When an escape character is not specified, you should select a non-“None” option from the Quote Mode setting.

  9. Specify the encoding method. Encoding method options include “Tar”, “Tgz”, “Zip”, “GZip”, and “None”.

  10. Add the PGP public key that is used to encrypt files sent to Amazon S3.

  11. Set the quote mode.

    Note

    If the quote mode is set to “None” and the Escape Character setting is empty this may result in unescaped, unquoted files. When quote mode is not set to “None”, you should specify an escape character.

  12. Optional. Select Include success file upon completion to add a .DONE file to indicate when an orchestration has finished sending data.

    Tip

    If a downstream sensor is listening for files sent from Amperity, configure that sensor to listen for the presence of the .DONE file.

  13. Optional. Select Include header row in output files if headers are included in the output.

  14. Select Allow business users to use this destination.

    This allows users who have been assigned to the Amp360 User and/or AmpIQ User policies to use this destination with orchestrations and/or campaigns.

  15. Select Allow business users with limited PII access to send data.

    This allows users who have been assigned to the Restrict PII Access policy option to use this destination with orchestrations and/or campaigns.

  16. Click Save.

Add data template

A data template defines how columns in Amperity data structures are sent to downstream workflows. A data template is part of the configuration for sending query and segment results from Amperity to an external location.

To add a data template

Step 1.

From the Destinations tab, open the menu for a destination that is configured for Amazon S3, and then select Add data template.

This opens the Add Data Template dialog box.

Step 1

From the Destinations tab, open the menu for a destination that is configured for Amazon S3, and then select Add data template.

This opens the Add Data Template dialog box.
Step 2.

Verify business user access to queries and orchestrations and access to segments and campaigns.

A business user may also have restricted access to PII, which prevents them from viewing and sending customer profile data.

Step 2.

If business user access was not configured as part of the destination, you may configure access from the data template.

Important

To allow business users to use this destination with campaigns, you must enable the Make available to campaigns option. This allows users to send campaign results from Amperity to Amazon S3.

If you enable this option, the data extension settings require using campaign name and group name template variables to associate the name of the data extension to your campaign.
Step 3.

Verify all configuration settings.

Verify settings for the data template.

Note

When the settings required by Amazon S3 were are not configured as part of the destination, you must configure them as part of the data template before making this destination available to campaigns.
Step 4.

Review all settings, and then click Save.

Save the data template.

After you have saved the data template, and depending on how you configured it, business users can send query results and/or send campaigns to Amazon S3.