Send data to Azure Blob Storage

Note

This topic contains information about configuring a destination that sends query results to Azure Blob Storage using orchestrations. To configure a destination that sends audiences to Azure Blob Storage using campaigns see this topic .

Microsoft Azure Blob Storage can store data files of any size for any file format that is supported by Amperity.

Amperity can be configured to send Apache Parquet (recommended), CSV, JSON, NDJSON, PSV, or TSV files to any Azure Blob Storage container.

Get details

Review the following details before configuring credentials for Azure Blob Storage and before configuring Amperity to send Apache Parquet (recommended), CSV, JSON, NDJSON, PSV, or TSV files to any Azure Blob Storage container.

	Azure Blob Storage container details You will need to know the following details about the Azure Blob Storage container to which Amperity will send data. The name of the Azure Blob Storage container. A blob prefix is sometimes required. The bucket name must match the value of the <<GCS_BUCKET_NAME>> placeholder shown in the service account key example.
	Credential types and settings Amperity supports the following credential types for Azure Blob Storage: Connection string Shared access signature Storage URI
	Required configuration settings File format Configure Amperity to send Apache Parquet (recommended), CSV, JSON, NDJSON, PSV, or TSV files to any Azure Blob Storage container. Some file formats allow a custom delimiter. Choose the “Custom delimiter” file format, and then add a single character to represent the custom delimiter. Note All other Amperity file format settings for Azure Blob Storage are optional.

Use Azure Data Share

Azure Data Share is a simple and safe service for sharing data in any format and any size with Amperity. Azure Data Share requires no infrastructure setup or management and uses underlying Azure security measures as they are applied to both Azure accounts. Snapshot-based sharing of data can be automated and does not require a special access key.

Amperity prefers to send data to customer-managed cloud storage. This approach ensures that customers can:

• Use security policies managed in Azure Data Share to manage access to data

• Directly manage the files that are made available

• Modify access without requiring involvement by Amperity; access may be revoked at any time by either Azure account, after which data sharing ends immediately

• Directly troubleshoot incomplete or missing files

Amperity recommends to use Azure Data Share to manage access to customer-managed cloud storage in Azure. This allows managed security policies to control access to data.

Note

If you have already configured Azure Data Share for an Azure Blob Storage data source you may use the same process credential for this destination. If you have not configured Azure Data Share, ask your Amperity representative to help you with those configuration steps.

Configure credentials

Configure credentials for Azure Blob Storage before adding a destination.

Amperity supports the following credential types for Azure Blob Storage:

1. Connection string

2. Shared access signature

3. Storage URI

An individual with access to Azure Blob Storage should use SnapPass to securely share “azure-blob-connection-string”, “azure-blob-shared-access-signature”, or “azure-blob-storage-uri” details with the individual who will configure Amperity.

Connection string

A connection string includes the information that allows Amperity to authorize to your Azure Blob Storage account.

To configure credentials using a connection string

	From the Settings page, select the Credentials tab, and then click the Add credential button.
	In the Credentials settings dialog box, do the following: From the Plugin dropdown, select Azure Blob Storage. Assign the credential a name and description that ensures other users of Amperity can recognize when to use this destination. From the Credential type dropdown, select azure-blob-connection-string.
	The settings that are available for a credential are determined by the credential type. For the azure-blob-connection-string credential type, configure the following settings, and then click Save. The settings that are available for a credential are determined by the credential type. For the “azure-blob-connection-string”, “azure-blob-shared-access-signature”, or “azure-blob-storage-uri” credential type, configure settings, and then click Save. Connection string Required A connection string allows access to a location within your Azure Blob Storage storage account. A connection string is a combination of your storage account and your account access key. A connection string is similar to: DefaultEndpointsProtocol=https; AccountName=name;AccountKey=key Container Required A container organizes a set of blobs, similar to a directory in a file system. Your Azure Blob Storage account can include an unlimited number of containers. Each container can store an unlimited number of blobs. A container name must be a valid DNS name, as it forms part of the unique uniform resource identifier (URI) used to address the container or its blobs. The value of the blob within the URI must be configured as the value for the Container setting within Amperity.

Shared access signature

A service-level shared access signature (SAS) specifies which resources in your Azure Blob Storage account can be accessed, what permissions that access allows on resources in the container, and the length of time for which the SAS is valid.

Important

When Microsoft Azure is configured to use a shared access signature (SAS) to grant restricted access rights to Microsoft Azure storage resources, be sure to use the correct SAS token string for credentials within Amperity and that the SAS is assigned the following permissions within Microsoft Azure: READ, ADD, CREATE, WRITE, DELETE, and LIST.

To configure credentials using a shared access signature

	From the Settings page, select the Credentials tab, and then click the Add credential button.
	In the Credentials settings dialog box, do the following: From the Plugin dropdown, select Azure Blob Storage. Assign the credential a name and description that ensures other users of Amperity can recognize when to use this destination. From the Credential type dropdown, select azure-blob-shared-access-signature.
	The settings that are available for a credential are determined by the credential type. For the azure-blob-shared-access-signature credential type, configure the following settings, and then click Save. The settings that are available for a credential are determined by the credential type. For the “azure-blob-connection-string”, “azure-blob-shared-access-signature”, or “azure-blob-storage-uri” credential type, configure settings, and then click Save. Account name Required The name of your Azure Blob Storage storage account. Container Required A container organizes a set of blobs, similar to a directory in a file system. Your Azure Blob Storage account can include an unlimited number of containers. Each container can store an unlimited number of blobs. A container name must be a valid DNS name, as it forms part of the unique uniform resource identifier (URI) used to address the container or its blobs. The value of the blob within the URI must be configured as the value for the Container setting within Amperity. Shared access signature Required A shared access signature (SAS) grants limited access to containers and blobs in your storage account. The value of a SAS is the URI for the resource to which the SAS delegates access, followed by the SAS token.

Storage URI

Each Azure Blob Storage resource has a storage URI , which contains the name of the account and the name of the container in which blob storage is located.

To configure credentials using a storage URI

	From the Settings page, select the Credentials tab, and then click the Add credential button.
	In the Credentials settings dialog box, do the following: From the Plugin dropdown, select Azure Blob Storage. Assign the credential a name and description that ensures other users of Amperity can recognize when to use this destination. From the Credential type dropdown, select azure-blob-storage-uri.
	The settings that are available for a credential are determined by the credential type. For the azure-blob-storage-uri credential type, configure the following settings, and then click Save. The settings that are available for a credential are determined by the credential type. For the “azure-blob-connection-string”, “azure-blob-shared-access-signature”, or “azure-blob-storage-uri” credential type, configure settings, and then click Save. Container Required A container organizes a set of blobs, similar to a directory in a file system. Your Azure Blob Storage account can include an unlimited number of containers. Each container can store an unlimited number of blobs. A container name must be a valid DNS name, as it forms part of the unique uniform resource identifier (URI) used to address the container or its blobs. The value of the blob within the URI must be configured as the value for the Container setting within Amperity. Storage URI Required A URI for Azure Blob Storage that contains the name of the account and the name of the container in which blob storage is located. For example: https://myaccount.blob.core.windows.net/mycontainer

Add destination

Use a sandbox to configure a destination for Azure Blob Storage. Before promoting your changes, send a test audience, and then verify the the results in Azure Blob Storage. After the end-to-end workflow has been verified, push the destination from the sandbox to production.

To add a destination for Azure Blob Storage

	Open the Destinations page, and then click the Add destination button. To configure a destination for Azure Blob Storage, do one of the following: Click the row in which Azure Blob Storage is located. Destinations are listed alphabetically and you can scroll up and down the list. Search for Azure Blob Storage. Start typing “azure”. The list will filter to show only matching destinations. Select “Azure Blob Storage”.
	Select the credential for Azure Blob Storage from the Credential dropdown, and then click Continue. Tip Click the “Test connection” link on the “Configure destination” page to verify that Amperity can connect to Azure Blob Storage.
	In the “Destination settings” dialog box, assign the destination a name and description that ensures other users of Amperity can recognize when to use this destination. Configure business user access By default a destination is available to all users who have permission to view personally identifiable information (PII). Enable the Admin only checkbox to restrict access to only users assigned to the Datagrid Operator and Datagrid Administrator policies. Enable the PII setting checkbox to allow users with limited access to PII access to this destination. Restricted PII access is enabled when the Restrict PII access policy option that prevents users who are assigned to that option from viewing data that is marked as PII anywhere in Amperity and from sending that data to any downstream workflow.
	Configure the following settings, and then click “Save”. Blob prefix A prefix filters the list of blob storage objects to only blob names that match the prefix value within your instance of Azure Blob Storage. A prefix may be a character or a string. Compression The compression format to apply to the file. May be one of “GZIP”, “None”, “TAR”, “TGZ”, or “ZIP”. Escape character The escape character to use in the file output. Applies to CSV, TSV, PSV, and custom delimiter file types. When an escape character is not specified and the quote mode is “None” files may be sent with unescaped and unquoted data. When an escape character is not specified, you should select a non-“None” option as the quote mode. File format Required Configure Amperity to send Apache Parquet (recommended), CSV, JSON, NDJSON, PSV, or TSV files to any Azure Blob Storage container. Some file formats allow a custom delimiter. Choose the “Custom delimiter” file format, and then add a single character to represent the custom delimiter. Apache Parquet files only The extension for Apache Parquet files may be excluded from the directory name. Filename template A filename template defines the naming pattern for files that are sent from Amperity. Specify the name of the file, and then use Jinja-style string formatting to append a date or timestamp to the filename. Header Enable to include header rows in output files. PGP public key The PGP public key that Amperity will use to encrypt files. Quote mode The quote mode to use within the file. May be one of “all fields”, “all non-NULL fields”, “fields with special characters only”, “all non-numeric fields” or “None”. Unescaped, unquoted files may occur when quote mode is set to “None” and an escape character is not specified. Success file Enable to send a “.DONE” file when Amperity has finished sending data. If a downstream sensor is listening for files sent from Amperity, configure that sensor to listen for the presence of the “.DONE” file. Use Zip64? Enable to apply Zip64 data compression to very large files. Row Number Select to include a row number column in the output file. Applies to CSV, TSV, PSV, and custom delimiter file types. If Row Number is enabled you may use the Column name setting to specify the name of the row number column in the output file. The name of this column must be less than 1028 characters and may only contain numbers, letters, underscores, and hyphens. Default value: “row_number”.
	After this destination is configured: Use orchestrations to send query results Use orchestrations and campaigns to send audiences Use orchestrations and campaigns to send offline events

Workflow actions

A workflow will occasionally show an error that describes what prevented a workflow from completing successfully. These first appear as alerts in the notifications pane. The alert describes the error, and then links to the Workflows tab.

Open the Workflows page to review a list of workflow actions, choose an action to resolve the workflow error, and then follow the steps that are shown.

	You may receive a notifications error for a configured Azure Blob Storage destination. This appears as an alert in the notifications pane on the Destinations tab. If you receive a notification error, review the details, and then click the View Workflow link to open this notification error in the Workflows page.
	On the Workflows page, review the individual steps to determine which step(s) have errors that require your attention, and then click Show Resolutions to review the list of workflow actions that were generated for this error.
	A list of individual workflow actions are shown. Review the list to identify which action you should take. Some workflow actions are common across workflows and will often be available, such as retrying a specific task within a workflow or restarting a workflow. These types of actions can often resolve an error. In certain cases, actions are specific and are shown when certain conditions exist in your tenant. These types of actions typically must be resolved and may require steps that must be done upstream or downstream from your Amperity workflow. Amperity provides a series of workflow actions that can help resolve specific issues that may arise with Azure Blob Storage, including: Invalid credentials Invalid permissions
	Select a workflow action from the list of actions, and then review the steps for resolving that error. After you have completed the steps in the workflow action, click Continue to rerun the workflow.

Invalid credentials

The credentials that are defined in Amperity are invalid.

To resolve this error, verify that the credentials required by this workflow are valid.

1. Open the Credentials page.

2. Review the details for the credentials used with this workflow. Update the credentials for Azure Blob Storage if required.

3. Return to the workflow action, and then click Resolve to retry this workflow.

Invalid permissions

Microsoft Azure may be configured to use a shared access signature (SAS) to grant restricted access rights to Microsoft Azure storage resources.

What is a shared access signature (SAS)?

A shared access signature (SAS) grants limited access to storage resources in Microsoft Azure. A SAS may be constrained to access only specific storage resources, have specific permissions to those resources, and be configured to expire after a set amount of time. Every SAS is signed with a key.

The SAS is appended to the URI for a storage resource. The combined URI and SAS become a token that contains a set of query parameters that indiciate how a storage resource may be accessed. Use the SAS token to configure Amperity credentials to storage resources in Microsoft Azure.

An SAS token may have invalid permissions for any of the following situations:

1. The SAS token may be configured incorrectly within Amperity. For example: an extra character within or at at the end of the SAS token. Verify the string, and then make any updates that are required for the credentials within Amperity.

2. The permissions for the SAS token were configured incorrectly. Amperity requires an SAS token to be assigned the following permissions: READ, ADD, CREATE, WRITE, DELETE, and LIST.

3. The SAS token may have expired or the signing key associated with the SAS token may have been rotated.

   These situations will require generating a new SAS token, and then updating the credentials in Amperity.

Note

If the shared access signature was provisioned by Amperity, please use the “Report a problem” feature in Amperity to contact your Amperity Support team and ask for help resolving this workflow issue.

The “Report a problem” option is available from the    menu in the top navigation.

To resolve this error, determine the cause for the invalid permissions error.

1. Do one (or more) of the following:

   Verify that the SAS token was configured correctly within Amperity.

   Verify the permissions that have been assigned to the SAS token. This can be done from the Microsoft Azure Portal or by using Azure Storage Explorer . The policy for the SAS token must be assigned the following permissions: READ, ADD, CREATE, WRITE, DELETE, and LIST.

   Verify that the SAS token and/or the signing key associated with the SAS token is valid (and has not expired). If either have expired, generate a new SAS token (using a new signing key, if necessary).

2. After you have determined the cause of the invalid permissions error, make the appropriate updates within Microsoft Azure and/or the credentials for this destination within Amperity.

3. Return to the workflow action, and then click Resolve to retry this workflow.