Send data to Azure Blob Storage

Azure Blob Storage is an object storage solution for the cloud that is optimized for storing massive amounts of unstructured data.

Important

Use this destination to send data from Amperity to Azure Data Lake Storage Gen1 or Azure Data Lake Storage Gen2.

This topic describes the steps that are required to send files to Azure Blob Storage from Amperity:

1. Get details

2. Add destination

3. Add data template

Get details

Amperity can be configured to send data to Azure Blob Storage. This may be done using Azure Data Share (recommended) or by using Azure credentials.

Use Azure Data Share

Azure Data Share is a simple and safe service for sharing data in any format and any size with Amperity. Azure Data Share requires no infrastructure setup or management and uses underlying Azure security measures as they are applied to both Azure accounts. Snapshot-based sharing of data can be automated and does not require a special access key.

Amperity prefers to send data to customer-managed cloud storage. This approach ensures that customers can:

• Use security policies managed in Azure Data Share to manage access to data

• Directly manage the files that are made available

• Modify access without requiring involvement by Amperity; access may be revoked at any time by either Azure account, after which data sharing ends immediately

• Directly troubleshoot incomplete or missing files

Amperity recommends to use Azure Data Share to manage access to customer-managed cloud storage in Azure. This allows managed security policies to control access to data.

Note

If you have already configured Azure Data Share for an Azure Blob Storage data source you may use the same process credential for this destination. If you have not configured Azure Data Share, ask your Amperity representative to help you with those configuration steps.

Use credentials

Azure Blob Storage requires the following configuration details:

1. The name of the container.

2. The blob prefix.

3. The credential details.

   These vary depending on the chosen credential method: connection string, shared access signature (SAS) token, or storage URI.

   When Microsoft Azure is configured to use a shared access signature (SAS) to grant restricted access rights to Microsoft Azure storage resources, be sure to use the correct SAS token string for credentials within Amperity and that the SAS is assigned the following permissions within Microsoft Azure: READ, ADD, CREATE, WRITE, DELETE, and LIST.

4. The public key to use for PGP encryption.

Add destination

Azure Blob Storage is a destination that may be configured directly from Amperity.

To add a destination

	Open the Destinations tab to configure a destination for Azure Blob Storage. Click the Add Destination button to open the Destination dialog box. Enter a name for the destination and provide a description. For example: “Azure Blob Storage” and “This sends files to Azure Blob Storage”. From the Plugin drop-down, start typing “az” to filter the list, and then select Azure Blob Storage.
	Credentials allow Amperity to connect to Azure Blob Storage. The credential type is set automatically. You may use an existing credential or you may add a new one. Select an existing credential from the Credential drop-down. – or – Select Create a new credential from the Credential drop-down. This opens the Credential dialog box. Enter the name for the credential, and then add a description. Azure Blob Storage has the following settings: The name of the container. The blob prefix. Credential details. These vary depending on the chosen credential method: connection string, shared access signature (SAS) token, or storage URI. The public key to use for PGP encryption. When finished, click Save.
	Each destination has settings that define how Amperity will deliver data to Azure Blob Storage. These settings are listed under the Settings section of the Destination dialog box. Complete the following Azure Blob Storage Settings: The Container and Blob prefix. The name of the container and blob prefix. For example: “Blob Storage” and “upload”. The File format. Select the file format – Apache Parquet (recommended), CSV, TSV, or PSV – from the drop-down list. Optional. The Escape character that is required by Azure Blob Storage. Note If an escape character is not specified and quote mode is set to “None” this may result in unescaped, unquoted files. When an escape character is not specified, you should select a non-“None” option from the Quote Mode setting. Optional. The Compression format. Encoding method options include “Tar”, “Tgz”, “Zip”, “GZip”, and “None”. Optional. The PGP public key that is used to encrypt files that are sent to Azure Blob Storage. Optional. The Quote mode that should be used within the file. From the drop-down, select one of “all fields”, “all non-NULL fields”, “fields with special characters only”, “all non-numeric fields” or “None”. Note If the quote mode is set to “None” and the Escape Character setting is empty this may result in unescaped, unquoted files. When quote mode is not set to “None”, you should specify an escape character. Optional. Select Include success file upon completion to add a .DONE file that indicates when an orchestration has finished sending data. Tip If a downstream sensor is listening for files sent from Amperity, configure that sensor to listen for the presence of the .DONE file. Optional. Select Include header row in output files if headers should be included in the output. Optional. Select Row number to include a row number column in the output file. Applies to CSV, TSV, PSV, and custom delimiter file types. When enabled, you may specify the name of the row number column in the output file. Optional. Select Exclude Parquet extension from the directory name for managing how Apache Parquet files are added to directories.
	Business users are assigned to the Amp360 User and/or AmpIQ User policies. (Amp360 User allows access to queries and orchestrations and AmpIQ User allows access to segments and campaigns.) A business user cannot select a destination that is not visible to them. Business users – including users assigned to the DataGrid Operator policy – may have restricted access to PII. What is restricted access to PII? Restricted PII access is enabled when the Restrict PII access policy option that prevents users who are assigned to that option from viewing data that is marked as PII anywhere in Amperity and from sending that data to any downstream workflow. You can make this destination visible to orchestrations and allow users with restricted access to PII to use this destination by enabling one (or both) of the following options: Note To allow business users to use this destination with campaigns, you must enable the Available to campaigns option within the data template. This allows users to send campaign results from Amperity to Azure Blob Storage. The other two settings may be configured within the data template instead of the destination.
	Review all settings, and then click Save. Important You must configure a data template for this destination before you can send data to Azure Blob Storage.

Add data template

A data template defines how columns in Amperity data structures are sent to downstream workflows. A data template is part of the configuration for sending query and segment results from Amperity to an external location.

To add a data template

	From the Destinations tab, open the menu for a destination that is configured for Azure Blob Storage, and then select Add data template. This opens the Add Data Template dialog box. Enter the name of the data template and a description. For example: “Azure Blob Storage” and “Send files to Azure Blob Storage.”.
	Verify business user access to queries and orchestrations and access to segments and campaigns. A business user may also have restricted access to PII, which prevents them from viewing and sending customer profile data. If business user access was not configured as part of the destination, you may configure access from the data template. Important To allow business users to use this destination with campaigns, you must enable the Available to campaigns option. This allows users to send campaign results from Amperity to Azure Blob Storage. If you enable this option, the data extension settings require using campaign name and group name template variables to associate the name of the data extension to your campaign.
	Verify all configuration settings. Note When the settings required by Azure Blob Storage were are not configured as part of the destination, you must configure them as part of the data template before making this destination available to campaigns.
	Review all settings, and then click Save. After you have saved the data template, and depending on how you configured it, business users can send query results and/or send campaigns to Azure Blob Storage.

Workflow actions

A workflow will occasionally show an error that describes what prevented a workflow from completing successfully. These first appear as alerts in the notifications pane. The alert describes the error, and then links to the Workflows tab.

Open the Workflows page to review a list of workflow actions, choose an action to resolve the workflow error, and then follow the steps that are shown.

	You may receive a notifications error for a configured Azure Blob Storage destination. This appears as an alert in the notifications pane on the Destinations tab. If you receive a notification error, review the details, and then click the View Workflow link to open this notification error in the Workflows page.
	On the Workflows page, review the individual steps to determine which step(s) have errors that require your attention, and then click Show Resolutions to review the list of workflow actions that were generated for this error.
	A list of individual workflow actions are shown. Review the list to identify which action you should take. Some workflow actions are common across workflows and will often be available, such as retrying a specific task within a workflow or restarting a workflow. These types of actions can often resolve an error. In certain cases, actions are specific and are shown when certain conditions exist in your tenant. These types of actions typically must be resolved and may require steps that must be done upstream or downstream from your Amperity workflow. Amperity provides a series of workflow actions that can help resolve specific issues that may arise with Azure Blob Storage, including: Invalid credentials Invalid permissions
	Select a workflow action from the list of actions, and then review the steps for resolving that error. After you have completed the steps in the workflow action, click Continue to rerun the workflow.

Invalid credentials

The credentials that are defined in Amperity are invalid.

To resolve this error, verify that the credentials required by this workflow are valid.

1. Open the Credentials page.

2. Review the details for the credentials used with this workflow. Update the credentials for Azure Blob Storage if required.

3. Return to the workflow action, and then click Resolve to retry this workflow.

Invalid permissions

Microsoft Azure may be configured to use a shared access signature (SAS) to grant restricted access rights to Microsoft Azure storage resources.

What is a shared access signature (SAS)?

A shared access signature (SAS) grants limited access to storage resources in Microsoft Azure. A SAS may be constrained to access only specific storage resources, have specific permissions to those resources, and be configured to expire after a set amount of time. Every SAS is signed with a key.

The SAS is appended to the URI for a storage resource. The combined URI and SAS become a token that contains a set of query parameters that indiciate how a storage resource may be accessed. Use the SAS token to configure Amperity credentials to storage resources in Microsoft Azure.

An SAS token may have invalid permissions for any of the following situations:

1. The SAS token may be configured incorrectly within Amperity. For example: an extra character within or at at the end of the SAS token. Verify the string, and then make any updates that are required for the credentials within Amperity.

2. The permissions for the SAS token were configured incorrectly. Amperity requires an SAS token to be assigned the following permissions: READ, ADD, CREATE, WRITE, DELETE, and LIST.

3. The SAS token may have expired or the signing key associated with the SAS token may have been rotated.

   These situations will require generating a new SAS token, and then updating the credentials in Amperity.

Note

If the shared access signature was provisioned by Amperity, please use the “Report a problem” feature in Amperity to contact your Amperity Support team and ask for help resolving this workflow issue.

The “Report a problem” option is available from the    menu in the top navigation.

To resolve this error, determine the cause for the invalid permissions error.

1. Do one (or more) of the following:

   Verify that the SAS token was configured correctly within Amperity.

   Verify the permissions that have been assigned to the SAS token. This can be done from the Microsoft Azure Portal or by using Azure Storage Explorer. The policy for the SAS token must be assigned the following permissions: READ, ADD, CREATE, WRITE, DELETE, and LIST.

   Verify that the SAS token and/or the signing key associated with the SAS token is valid (and has not expired). If either have expired, generate a new SAS token (using a new signing key, if necessary).

2. After you have determined the cause of the invalid permissions error, make the appropriate updates within Microsoft Azure and/or the credentials for this destination within Amperity.

3. Return to the workflow action, and then click Resolve to retry this workflow.