Send data to Azure Blob Storage

Azure Blob Storage is an object storage solution for the cloud that is optimized for storing massive amounts of unstructured data.

Important

Use this destination to send data from Amperity to Azure Data Lake Storage Gen1 or Azure Data Lake Storage Gen2.

This topic describes the steps that are required to send files to Azure Blob Storage from Amperity:

  1. Get details

  2. Add destination

  3. Add data template

Get details

Amperity can be configured to send data to Azure Blob Storage. This may be done using Azure Data Share (recommended) or by using Azure credentials.

Use Azure Data Share

Azure Data Share is a simple and safe service for sharing data in any format and any size with Amperity. Azure Data Share requires no infrastructure setup or management and uses underlying Azure security measures as they are applied to both Azure accounts. Snapshot-based sharing of data can be automated and does not require a special access key.

Amperity prefers to send data to customer-managed cloud storage. This approach ensures that customers can:

  • Use security policies managed in Azure Data Share to manage access to data

  • Directly manage the files that are made available

  • Modify access without requiring involvement by Amperity; access may be revoked at any time by either Azure account, after which data sharing ends immediately

  • Directly troubleshoot incomplete or missing files

Amperity recommends to use Azure Data Share to manage access to customer-managed cloud storage in Azure. This allows managed security policies to control access to data.

Note

If you have already configured Azure Data Share for an Azure Blob Storage data source you may use the same process credential for this destination. If you have not configured Azure Data Share, ask your Amperity representative to help you with those configuration steps.

Use credentials

Azure Blob Storage requires the following configuration details:

  1. The name of the container.

  2. The blob prefix.

  3. The credential details. (These vary depending on the chosen credential method.)

  4. The public key to use for PGP encryption.

Optional workflows

The following sections describe additional workflows that are available. After Amperity sends data to Azure Blob Storage, you can configure downstream applications to consume that data and make it available to additional workflows.

Synapse Analytics

Azure Synapse Analytics is a limitless analytics service that brings together enterprise data warehousing and analytics. Azure Synapse Analytics has four components: SQL analytics, Apache Spark, hybrid data integration, and a unified user experience.

Amperity can be configured to send data to Azure Blob Storage, after which Synapse Analytics can be configured to load that data. Applications can be configured to connect to Synapse Analytics and use the Amperity output as a data source.

You may use the Azure Blob Storage bucket that comes with your Amperity tenant for the intermediate step (if your Amperity tenant is running on Microsoft Azure). Or you may configure Amperity to send data to an Azure Blob Storage bucket that your organization manages directly.

The following applications can be configured to load data from Synapse Analytics as a data source:

Add destination

Azure Blob Storage is a destination that may be configured directly from Amperity.

To add a destination

  1. From the Destinations tab, click Add Destination. This opens the Add Destination dialog box.

  2. Enter the name of the destination and a description. For example: “Azure Blob Storage” and “This sends query results to Azure Blob Storage”.

  3. From the Plugin drop-down, select Azure Blob Storage.

  4. From the Credential Type drop-down, select a credential type: “azure-blob-connection-string” or “azure-blob-shared-access-signature”.

  5. From the Credential drop-down, select a credential that has already been configured for this destination or click Create a new credential, which opens the Create New Credential dialog box. For new credentials, enter the credential details, and then click Save.

    For new “azure-blob-connection-string” credentials, enter the name of the credential and the connection string.

    For new “azure-blob-shared-access-signature” credentials, enter the name of the credential, the account name, and the shared access signature.

  6. Under Azure Blob Storage settings, add the name of the container and blob prefix. For example: “Blob Storage” and “upload”.

  7. From the File Format drop-down, select Apache Parquet (recommended), CSV, TSV, or PSV.

  8. Add a single character to be used as an escape character in the output file.

    Note

    If an escape character is not specified and quote mode is set to “None” this may result in unescaped, unquoted files. When an escape character is not specified, you should select a non-“None” option from the Quote Mode setting.

  9. Specify the encoding method. Encoding method options include “Tar”, “Tgz”, “Zip”, “GZip”, and “None”.

  10. Add the PGP public key that is used to encrypt files sent to Azure Blob Storage.

  11. Set the quote mode.

    Note

    If the quote mode is set to “None” and the Escape Character setting is empty this may result in unescaped, unquoted files. When quote mode is not set to “None”, you should specify an escape character.

  12. Optional. Select Include success file upon completion to add a .DONE file to indicate when an orchestration has finished sending data.

    Tip

    If a downstream sensor is listening for files sent from Amperity, configure that sensor to listen for the presence of the .DONE file.

  13. Optional. Select Include header row in output files if headers are included in the output.

  14. Select Allow customers to use this destination.

  15. Select Allow orchestrations from users with limited PII access. (A user with limited PII access has been assigned the Restrict PII Access policy option.)

  16. Click Save.

Add data template

A data template defines how columns in Amperity data structures are sent to downstream workflows. A data template is part of the configuration for sending query and segment results from Amperity to an external location.

You have two options for setting up data templates for Azure Blob Storage:

  1. For use with campaigns

  2. For use with orchestrations

for campaigns

You can configure Amperity to send campaigns to Microsoft Azure Blob Storage. These results are sent from the Campaigns tab. Results default to a list of email addresses, but you may configure a campaign to send additional attributes to Azure Blob Storage.

To add a data template for campaigns

  1. From the Destinations tab, open the menu for a destination that is configured for Azure Blob Storage, and then select Add data template.

    This opens the Add Data Template dialog box.

  2. Enter the name of the data template and a description. For example: “Azure Blob Storage email list” and “Send email addresses to Azure Blob Storage.”

  3. Enable the Allow customers to use this data template option, and then enable the Make available to campaigns option. This allows users to send campaign results from Amperity to Azure Blob Storage.

  4. Verify all template settings and make any required updates.

  5. Click Save.

for orchestrations

You can configure Amperity to send query results to Microsoft Azure Blob Storage. These results are sent using an orchestration and will include all columns that were specified in the query.

To add a data template for orchestrations

  1. From the Destinations tab, open the menu for a destination that is configured for Azure Blob Storage, and then select Add data template.

    This opens the Add Data Template dialog box.

  2. Enter the name of the data template and a description. For example: “Azure Blob Storage customer profiles” and “Send email addresses and customer profiles to Azure Blob Storage.”

  3. Enable the Allow customers to use this data template option. This allows users to build queries, and then configure orchestrations that send results from Amperity to a configured destination.

  4. Optional. Enable the Allow orchestrations from customers with limited PII access option. This allows users who have been assigned the Restrict PII Access policy option to send results from Amperity.

  5. Verify all template settings and make any required updates.

  6. Click Save.