Send data to Azure Blob Storage

Note

This topic contains information about configuring a destination that sends query results to Azure Blob Storage using orchestrations. To configure a destination that sends audiences to Azure Blob Storage using campaigns see this topic .

Microsoft Azure Blob Storage can store data files of any size for any file format that is supported by Amperity.

Amperity can be configured to send Apache Parquet (recommended), CSV, JSON, NDJSON, PSV, or TSV files to any Azure Blob Storage container.

Get details

Review the following details before configuring credentials for Azure Blob Storage and before configuring Amperity to send Apache Parquet (recommended), CSV, JSON, NDJSON, PSV, or TSV files to any Azure Blob Storage container.

	Azure Blob Storage container details You will need to know the following details about the Azure Blob Storage container to which Amperity will send data. The name of the Azure Blob Storage container. A blob prefix is sometimes required. The bucket name must match the value of the <<GCS_BUCKET_NAME>> placeholder shown in the service account key example.
	Credential types and settings Amperity supports the following credential types for Azure Blob Storage: Connection string Shared access signature Storage URI
	Required configuration settings File format Configure Amperity to send Apache Parquet (recommended), CSV, JSON, NDJSON, PSV, or TSV files to any Azure Blob Storage container. Some file formats allow a custom delimiter. Choose the “Custom delimiter” file format, and then add a single character to represent the custom delimiter. Note All other Amperity file format settings for Azure Blob Storage are optional.

Use Azure Data Share

Azure Data Share is a simple and safe service for sharing data in any format and any size with Amperity. Azure Data Share requires no infrastructure setup or management and uses underlying Azure security measures as they are applied to both Azure accounts. Snapshot-based sharing of data can be automated and does not require a special access key.

Amperity prefers to send data to customer-managed cloud storage. This approach ensures that customers can:

• Use security policies managed in Azure Data Share to manage access to data

• Directly manage the files that are made available

• Modify access without requiring involvement by Amperity; access may be revoked at any time by either Azure account, after which data sharing ends immediately

• Directly troubleshoot incomplete or missing files

Amperity recommends to use Azure Data Share to manage access to customer-managed cloud storage in Azure. This allows managed security policies to control access to data.

Note

If you have already configured Azure Data Share for an Azure Blob Storage data source you may use the same process credential for this destination. If you have not configured Azure Data Share, ask your Amperity representative to help you with those configuration steps.

Configure credentials

Configure credentials for Azure Blob Storage before adding a destination.

Amperity supports the following credential types for Azure Blob Storage:

1. Connection string

2. Shared access signature

3. Storage URI

An individual with access to Azure Blob Storage should use SnapPass to securely share “azure-blob-connection-string”, “azure-blob-shared-access-signature”, or “azure-blob-storage-uri” details with the individual who will configure Amperity.

Connection string

A connection string includes the information that allows Amperity to authorize to your Azure Blob Storage account.

To configure credentials using a connection string

	From the Settings page, select the Credentials tab, and then click the Add credential button.
	In the Credentials settings dialog box, do the following: From the Plugin dropdown, select Azure Blob Storage. Assign the credential a name and description that ensures other users of Amperity can recognize when to use this destination. From the Credential type drop-down, select azure-blob-connection-string.
	The settings that are available for a credential are determined by the credential type. For the azure-blob-connection-string credential type, configure the following settings, and then click Save. The settings that are available for a credential are determined by the credential type. For the “azure-blob-connection-string”, “azure-blob-shared-access-signature”, or “azure-blob-storage-uri” credential type, configure settings, and then click Save. Connection string Required A connection string allows access to a location within your Azure Blob Storage storage account. A connection string is a combination of your storage account and your account access key. A connection string is similar to: DefaultEndpointsProtocol=https; AccountName=name;AccountKey=key Container Required A container organizes a set of blobs, similar to a directory in a file system. Your Azure Blob Storage account can include an unlimited number of containers. Each container can store an unlimited number of blobs. A container name must be a valid DNS name, as it forms part of the unique uniform resource identifier (URI) used to address the container or its blobs. The value of the blob within the URI must be configured as the value for the Container setting within Amperity.

Shared access signature

A service-level shared access signature (SAS) specifies which resources in your Azure Blob Storage account can be accessed, what permissions that access allows on resources in the container, and the length of time for which the SAS is valid.

Important

When Microsoft Azure is configured to use a shared access signature (SAS) to grant restricted access rights to Microsoft Azure storage resources, be sure to use the correct SAS token string for credentials within Amperity and that the SAS is assigned the following permissions within Microsoft Azure: READ, ADD, CREATE, WRITE, DELETE, and LIST.

To configure credentials using a shared access signature

	From the Settings page, select the Credentials tab, and then click the Add credential button.
	In the Credentials settings dialog box, do the following: From the Plugin dropdown, select Azure Blob Storage. Assign the credential a name and description that ensures other users of Amperity can recognize when to use this destination. From the Credential type drop-down, select azure-blob-shared-access-signature.
	The settings that are available for a credential are determined by the credential type. For the azure-blob-shared-access-signature credential type, configure the following settings, and then click Save. The settings that are available for a credential are determined by the credential type. For the “azure-blob-connection-string”, “azure-blob-shared-access-signature”, or “azure-blob-storage-uri” credential type, configure settings, and then click Save. Account name Required The name of your Azure Blob Storage storage account. Container Required A container organizes a set of blobs, similar to a directory in a file system. Your Azure Blob Storage account can include an unlimited number of containers. Each container can store an unlimited number of blobs. A container name must be a valid DNS name, as it forms part of the unique uniform resource identifier (URI) used to address the container or its blobs. The value of the blob within the URI must be configured as the value for the Container setting within Amperity. Shared access signature Required A shared access signature (SAS) grants limited access to containers and blobs in your storage account. The value of a SAS is the URI for the resource to which the SAS delegates access, followed by the SAS token.

Storage URI

Each Azure Blob Storage resource has a storage URI , which contains the name of the account and the name of the container in which blob storage is located.

To configure credentials using a storage URI

	From the Settings page, select the Credentials tab, and then click the Add credential button.
	In the Credentials settings dialog box, do the following: From the Plugin dropdown, select Azure Blob Storage. Assign the credential a name and description that ensures other users of Amperity can recognize when to use this destination. From the Credential type drop-down, select azure-blob-storage-uri.
	The settings that are available for a credential are determined by the credential type. For the azure-blob-storage-uri credential type, configure the following settings, and then click Save. The settings that are available for a credential are determined by the credential type. For the “azure-blob-connection-string”, “azure-blob-shared-access-signature”, or “azure-blob-storage-uri” credential type, configure settings, and then click Save. Container Required A container organizes a set of blobs, similar to a directory in a file system. Your Azure Blob Storage account can include an unlimited number of containers. Each container can store an unlimited number of blobs. A container name must be a valid DNS name, as it forms part of the unique uniform resource identifier (URI) used to address the container or its blobs. The value of the blob within the URI must be configured as the value for the Container setting within Amperity. Storage URI Required A URI for Azure Blob Storage that contains the name of the account and the name of the container in which blob storage is located. For example: https://myaccount.blob.core.windows.net/mycontainer

Add destination

Use a sandbox to configure a destination for Azure Blob Storage. Before promoting your changes, send a test audience, and then verify the the results in Azure Blob Storage. After the end-to-end workflow has been verified, push the destination from the sandbox to production.

To add a destination for Azure Blob Storage

	Open the Destinations page, and then click the Add destination button. To configure a destination for Azure Blob Storage, do one of the following: Click the row in which Azure Blob Storage is located. Destinations are listed alphabetically and you can scroll up and down the list. Search for Azure Blob Storage. Start typing “azure”. The list will filter to show only matching destinations. Select “Azure Blob Storage”.
	Select the credential for Azure Blob Storage from the Credential drop-down, and then click Continue. Tip Click the “Test connection” link on the “Configure destination” page to verify that Amperity can connect to Azure Blob Storage.
	In the “Destination settings” dialog box, assign the destination a name and description that ensures other users of Amperity can recognize when to use this destination. Configure business user access By default a destination is available to all users who have permission to view personally identifiable information (PII). Enable the Admin only checkbox to restrict access to only users assigned to the Datagrid Operator and Datagrid Administrator policies. Enable the PII setting checkbox to allow users with limited access to PII access to this destination. Restricted PII access is enabled when the Restrict PII access policy option that prevents users who are assigned to that option from viewing data that is marked as PII anywhere in Amperity and from sending that data to any downstream workflow.
	Configure the following settings, and then click “Save”. Blob prefix A prefix filters the list of blob storage objects to only blob names that match the prefix value within your instance of Azure Blob Storage. A prefix may be a character or a string. Compression The compression format to apply to the file. May be one of “GZIP”, “None”, “TAR”, “TGZ”, or “ZIP”. Escape character The escape character to use in the file output. Applies to CSV, TSV, PSV, and custom delimiter file types. When an escape character is not specified and the quote mode is “None” files may be sent with unescaped and unquoted data. When an escape character is not specified, you should select a non-“None” option as the quote mode. File format Required Configure Amperity to send Apache Parquet (recommended), CSV, JSON, NDJSON, PSV, or TSV files to any Azure Blob Storage container. Some file formats allow a custom delimiter. Choose the “Custom delimiter” file format, and then add a single character to represent the custom delimiter. Apache Parquet files only The extension for Apache Parquet files may be excluded from the directory name. Filename template A filename template defines the naming pattern for files that are sent from Amperity. Specify the name of the file, and then use Jinja-style string formatting to append a date or timestamp to the filename. Header Enable to include header rows in output files. PGP public key The PGP public key that Amperity will use to encrypt files. Quote mode The quote mode to use within the file. May be one of “all fields”, “all non-NULL fields”, “fields with special characters only”, “all non-numeric fields” or “None”. Unescaped, unquoted files may occur when quote mode is set to “None” and an escape character is not specified. Success file Enable to send a “.DONE” file when Amperity has finished sending data. If a downstream sensor is listening for files sent from Amperity, configure that sensor to listen for the presence of the “.DONE” file. Use Zip64? Enable to apply Zip64 data compression to very large files. Row Number Select to include a row number column in the output file. Applies to CSV, TSV, PSV, and custom delimiter file types. If Row Number is enabled you may use the Column name setting to specify the name of the row number column in the output file. The name of this column must be less than 1028 characters and may only contain numbers, letters, underscores, and hyphens. Default value: “row_number”.
	After this destination is configured, users may configure Amperity to: Use orchestrations to send query results Use orchestrations and campaigns to send audiences Use orchestrations and campaigns to send offline events to Azure Blob Storage.