Send data to Google Cloud Storage¶

Google Cloud Storage is an online file storage web service for storing and accessing data on Google Cloud Platform infrastructure.

This topic describes the steps that are required to send files to Google Cloud Storage from Amperity:

Get details
Add destination
Add data template

Get details¶

Google Cloud Storage requires the following configuration details:

	A Google Cloud Storage service account key that is configured for the Storage Object Admin role.
	The name of the Google Cloud Storage bucket to which Amperity will send data and its prefix.
	The public key to use for PGP encryption.

How do I find Google Cloud Storage account information?

You can find all of this information after you log into Google Cloud Storage.

Service account¶

A service account must be configured to allow Amperity to send data to the Cloud Storage bucket:

A service account key must be created, and then downloaded for use when configuring Amperity.
The Storage Object Admin role must be assigned to the service account.

Service account key¶

A service account key must be downloaded so that it may be used to configure the destination in Amperity.

To configure the service account key

Open the Cloud Platform console.
Click IAM, and then Admin.
Click the name of the project that is associated with the Cloud Storage bucket to which Amperity will send data.
Click Service Accounts, and then select Create Service Account.
In the Name field, give your service account a name. For example, “Amperity GCS Connection”.
In the Description field, enter a description that will remind you of the purpose of the role.
Click Create.

Important

Click Continue and skip every step that allows adding additional service account permissions. These permissions will be added directly to the bucket.
From the Service Accounts page, click the name of the service account that was created for Amperity.
Click Add Key, and then select Create new key.
Select the JSON key type, and then click Create.

The key is downloaded as a JSON file to your local computer. This key is required to connect Amperity to your Cloud Storage bucket. If necessary, provide this key to your Amperity representative using Snappass .

SnapPass allows secrets to be shared in a secure, ephemeral way. Input a single or multi-line secret, along with an expiration time, and then generate a one-time use URL that may be shared with anyone. Amperity uses SnapPass for sharing credentials to systems with customers.

Example

{
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "client_email": "<<GCS_BUCKET_NAME>>@<<GCS_PROJECT_ID>>.iam.gserviceaccount.com",
  "client_id": "redacted",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/<<GCS_BUCKET_NAME>>%40<<GCS_PROJECT_ID>>.iam.gserviceaccount.com",
  "private_key_id": "redacted",
  "private_key": "redacted",
  "project_id": "<<GCS_PROJECT_ID>>",
  "token_uri": "https://oauth2.googleapis.com/token",
  "type": "service_account"
}

Service account role¶

The Storage Object Admin role must be assigned to the service account.

To configure the service account role

Open the Cloud Platform console.
Click Storage, and then Browser.
Click the name of the bucket from which Amperity will pull data.
Click the Permissions tab, and then click Add.
Enter the email address of the Cloud Storage service account.
Under Role, choose Storage Object Admin.

Important

Amperity requires the Storage Object Admin role for the courier that is assigned to pull data from Cloud Storage.
Click Save.

Add destination¶

Configure Amperity to send files directly to Google Cloud Storage.

Important

The bucket name must match the value of the <<GCS_BUCKET_NAME>> placeholder shown in the service account key example.

To add a destination

	Open the Destinations tab to configure a destination for Google Cloud Storage. Click the Add Destination button to open the Destination dialog box. Enter a name for the destination and provide a description. For example: “Google Cloud Storage” and “This sends files to Google Cloud Storage”. From the Plugin drop-down, start typing “google” to filter the list, and then select Google Cloud Storage.
	Credentials allow Amperity to connect to Google Cloud Storage. The credential type is set automatically. You may use an existing credential or you may add a new one. Select an existing credential from the Credential drop-down. – or – Select Create a new credential from the Credential drop-down. This opens the Credential dialog box. Enter the name for the credential, and then add a description. Google Cloud Storage has the following settings: The name of the bucket to which Amperity will send data. The service account key. The service account key is the contents of the JSON file downloaded from Cloud Storage. Open the JSON file in a text editor, select all of the content in the JSON file, copy it, and then paste it into the Service Account Key field.
	Each destination has settings that define how Amperity will deliver data to Google Cloud Storage. These settings are listed under the Settings section of the Destination dialog box. Complete the following SFTP Settings: The Object prefix. The object prefix is a string that is used to filter results to include only objects whose names begin with this prefix. When this value is set, the names of objects that may be returned in the response are relative to the root of the bucket. The File format. Select the file format – Apache Parquet (recommended), CSV, TSV, or PSV – from the drop-down list. Optional. The Escape character that is required by Google Cloud Storage. Note If an escape character is not specified and quote mode is set to “None” this may result in unescaped, unquoted files. When an escape character is not specified, you should select a non-“None” option from the Quote Mode setting. Optional. The Compression format. Encoding method options include “Tar”, “Tgz”, “Zip”, “GZip”, and “None”. Optional. The PGP public key that is used to encrypt files that are sent to Google Cloud Storage. Optional. The Quote mode that should be used within the file. From the drop-down, select one of “all fields”, “all non-NULL fields”, “fields with special characters only”, “all non-numeric fields” or “None”. Note If the quote mode is set to “None” and the Escape Character setting is empty this may result in unescaped, unquoted files. When quote mode is not set to “None”, you should specify an escape character. Optional. Select Include success file upon completion to add a .DONE file that indicates when an orchestration has finished sending data. Tip If a downstream sensor is listening for files sent from Amperity, configure that sensor to listen for the presence of the .DONE file. Optional. Select Include header row in output files if headers should be included in the output. Optional. Select Row number to include a row number column in the output file. Applies to CSV, TSV, PSV, and custom delimiter file types. When enabled, you may specify the name of the row number column in the output file. Optional. Select Exclude Parquet extension from the directory name for managing how Apache Parquet files are added to directories.
	Business users are assigned to the Amp360 User and/or AmpIQ User policies. (Amp360 User allows access to queries and orchestrations and AmpIQ User allows access to segments and campaigns.) A business user cannot select a destination that is not visible to them. Business users – including users assigned to the DataGrid Operator policy – may have restricted access to PII. What is restricted access to PII? Restricted PII access is enabled when the Restrict PII access policy option that prevents users who are assigned to that option from viewing data that is marked as PII anywhere in Amperity and from sending that data to any downstream workflow. You can make this destination visible to orchestrations and allow users with restricted access to PII to use this destination by enabling one (or both) of the following options: Note To allow business users to use this destination with campaigns, you must enable the Available to campaigns option within the data template. This allows users to send campaign results from Amperity to Google Cloud Storage. The other two settings may be configured within the data template instead of the destination.
	Review all settings, and then click Save. Important You must configure a data template for this destination before you can send data to Google Cloud Storage.

Add data template¶

A data template defines how columns in Amperity data structures are sent to downstream workflows. A data template is part of the configuration for sending query and segment results from Amperity to an external location.

To add a data template

	From the Destinations tab, open the menu for a destination that is configured for Google Cloud Storage, and then select Add data template. This opens the Add Data Template dialog box. Enter the name of the data template and a description. For example: “Google Cloud Storage” and “Send files to Google Cloud Storage.”.
	Verify business user access to queries and orchestrations and access to segments and campaigns. A business user may also have restricted access to PII, which prevents them from viewing and sending customer profile data. If business user access was not configured as part of the destination, you may configure access from the data template. Important To allow business users to use this destination with campaigns, you must enable the Available to campaigns option. This allows users to send campaign results from Amperity to Google Cloud Storage. If you enable this option, the data extension settings require using campaign name and group name template variables to associate the name of the data extension to your campaign.
	Verify all configuration settings. Note When the settings required by Google Cloud Storage were are not configured as part of the destination, you must configure them as part of the data template before making this destination available to campaigns.
	Review all settings, and then click Save. After you have saved the data template, and depending on how you configured it, business users can send query results and/or send campaigns to Google Cloud Storage.

Workflow actions¶

A workflow will occasionally show an error that describes what prevented a workflow from completing successfully. These first appear as alerts in the notifications pane. The alert describes the error, and then links to the Workflows tab.

Open the Workflows page to review a list of workflow actions, choose an action to resolve the workflow error, and then follow the steps that are shown.

	You may receive a notifications error for a configured Google Cloud Storage destination. This appears as an alert in the notifications pane on the Destinations tab. If you receive a notification error, review the details, and then click the View Workflow link to open this notification error in the Workflows page.
	On the Workflows page, review the individual steps to determine which step(s) have errors that require your attention, and then click Show Resolutions to review the list of workflow actions that were generated for this error.
	A list of individual workflow actions are shown. Review the list to identify which action you should take. Some workflow actions are common across workflows and will often be available, such as retrying a specific task within a workflow or restarting a workflow. These types of actions can often resolve an error. In certain cases, actions are specific and are shown when certain conditions exist in your tenant. These types of actions typically must be resolved and may require steps that must be done upstream or downstream from your Amperity workflow. Amperity provides a series of workflow actions that can help resolve specific issues that may arise with Google Cloud Storage, including: Invalid credentials
	Select a workflow action from the list of actions, and then review the steps for resolving that error. After you have completed the steps in the workflow action, click Continue to rerun the workflow.

Invalid credentials¶

The credentials that are defined in Amperity are invalid.

To resolve this error, verify that the credentials required by this workflow are valid.

Open the Credentials page.
Review the details for the credentials used with this workflow. Update the credentials for Google Cloud Storage if required.
Return to the workflow action, and then click Resolve to retry this workflow.