Send data using a Webhook

A webhook destination enables the development of custom integrations with any downstream application or workflow. This destination sends query results to AWS S3 storage and triggers an AWS Lambda function running in any AWS account.

A webhook destination tracks the AWS Lambda function runs and monitors for success using the same tooling as other Amperity-developed integrations. The AWS Lambda function is a template with which you can write custom code, and then send data to your preferred destination.

This topic describes the steps that are required to send query results to a custom destination using a webhook:

1. Use serverless destinations

2. Configure custom destination

3. Add webhook destination

4. Add data template

5. Extend the AWS Lambda function

How a webhook destination works

The webhook destination is a broad purpose destination that generates a file in NDJSON format, publishes to Amazon S3, and then provides a pre-signed URL from which that NDJSON file may be downloaded.

After the NDJSON file is generated, the destination will send a request to an AWS Lambda endpoint that exists outside of the Amperity virtual private cloud (VPC).

The Lambda function is built to process and publish the data however it needs. To ensure that Amperity can track the Lambda function, a public endpoint is available, to which regular updates from the Lambda function must be provided.

A webhook destination works like this:

1. Use a query to build a table with data that matches rows, columns, and/or data types that are required by the custom destination.

2. Configure a webhook destination to send data to a Lambda function that is running in AWS Lambda.

3. Deploy a Lambda function to AWS Lambda. The Lambda function may be deployed as a serverless application (recommended) or it may be a custom Lambda function that you have built using the open source toolkit provided by Amperity, and then uploaded to AWS Lambda.

4. Configure a webhook destination to send data to a function that runs in AWS Lambda.

5. The Lambda function sends validated results to the custom destination.

Serverless destinations

Amperity provides a AWS Serverless Application Repository that encapsulates all of the configuration to build your own custom destination.

This template enables two use cases:

1. A serverless application that is deployed directly to your AWS Lambda instance from the AWS Serverless Application Repository. This is the recommended use case when a serverless application is available for the destination.

2. A starting point from which you may build a custom Lambda function that is uploaded to your AWS Lambda instance.

Amperity also provides a set of Lambda functions developed using this model that you can use for the following destinations:

• Amazon Pinpoint

• Amazon Redshift

• Microsoft Dataverse

Custom destinations

A custom destination is configured using a webhook destination and Lambda function. The Lambda function must be able to perform all actions that are required to move the data from Amperity to the custom destination, including validations and/or pre-processing that is required by the custom destination, and including any services, such as Amazon S3, that may be needed in-between Amperity and the custom destination.

Note

Amperity sends data to the Lambda function in NDJSON format. This data is staged to an Amazon S3 bucket, after which it is made available to the Lambda function.

To manually configure a custom webhook destination, do each of the following steps:

1. Add basic function to your AWS Lambda console

2. Get Lambda function template

3. Upload function to AWS Lambda

4. Configure function in AWS Lambda

5. Add the API gateway

6. Set AWS Lambda environment variables

7. Attach policies

Important

Your custom destination may require additional steps and/or modifications to the steps described in this topic.

Add function to AWS Lambda

Add a basic function to your AWS Lambda console using only the default function without any customization.

To add a function to AWS Lambda

1. Log in to your Amazon AWS account.

2. Open the AWS Lambda console, and then click Create function.

3. Select Author from scratch.

4. Under Basic information, specify the function name, runtime, and architecture. For example:

   Function name “Amperity function for [custom destination]”

   Runtime “Python 3.9”

   Architecture “x86_64”

5. Click Save.

Get Lambda function template

Amperity provides a pre-built collection of Lambda functions that is available from an open-source repository on GitHub.

Note

Lambda functions are built using Python. Amazon AWS provides runtimes for Python that include an SDK, along with credentials from an AWS Identity and Access Management (IAM) role that you manage.

To get the Lambda function template

1. Clone the open source amperity-lambda-runner repository or download it as a ZIP file.

2. In the repo, navigate to “/src/lambdas/lambda_handlers/”, and then open “template.py”.

3. You may use this file as a starting point to support your desired workflow.

Upload function to AWS Lambda

You must upload the application code (and any dependencies) to AWS Lambda as a ZIP file archive.

To upload a Lambda function to AWS Lambda

1. Add the application code and dependencies to a ZIP file archive.

   Tip

   To build a ZIP file, run the following command from your amperity-lambda-runner directory:

   sh util/lambda-build.sh filename=application_name.py
   

   This will create a ZIP file that contains the following files: application_name.py, amperity_runner.py, and helpers.py, where “application_name” is the name of your Lambda function.

2. Open the AWS Lambda console, and then open the Code tab.

3. Click Upload from, and then click ZIP file.

4. Find the ZIP file archive that contains the application code (and any dependencies), and then click Upload.

5. Click Save.

Configure function in AWS Lambda

AWS Lambda must be updated for the name of the custom destination application and to specify a timeout value.

To configure the Lambda function in AWS Lambda

1. Open the AWS Lambda console, and then open the Code tab.

2. Under Runtime settings, select Edit.

3. Under Handler, add “app.application_name” (where “application_name” is replaced with the name of your application), and then click Save.

4. In the AWS Lambda console, open the Configuration tab.

5. Under General configuration, select Edit.

6. Set the Timeout value to “15 min 0 sec”, and then click Save.

Add the API gateway

Amazon API Gateway provides tools for creating and documenting web APIs that route HTTP requests to AWS Lambda functions. An API gateway is required to use webhook destinations.

To add an API gateway

1. Open the AWS Lambda console.

2. Open the Lambda function for the custom destination.

3. Click Add trigger.

4. From the drop-down menu, select “API Gateway”, and then select the following:

   Setting	Value
   Intent	Create a new API
   API type	REST API
   Security	API Key

5. Click Add.

   You will need these values when configuring the webhook destination in Amperity.

Set AWS Lambda environment variables

Use AWS Lambda environment variables to provide configuration settings and details to your custom Lambda function.

To set environment variables

1. Open the AWS Lambda console.

2. Open the Lambda function for Amazon Redshift.

3. Click Configuration.

4. On the Configuration page, click Environment variables, and then click Edit.

5. Click Add environment variable, and then define environment variables as key-value pairs.

6. Click Save.

Define policies

When custom destination workflows use services that exist entirely within Amazon AWS you can define inline policies or attach policies.

To define an inline policy

1. Open the AWS Lambda console.

2. Open the Lambda function for your custom destination.

3. Click Configuration.

4. On the Configuration page, under Execution role, and then click the name of the role to be used with this Lambda function.

5. Click Add permissions, and then click Create inline policy, and then define the values for the inline policy.

6. Click Review policy, add a name for the policy (like “Amperity Amazon Pinpoint”), and then click Create policy.

To attach policies

1. Open the AWS Lambda console.

2. Open the Lambda function for your custom destination.

3. Click Configuration.

4. On the Configuration page, under Execution role, and then click the name of the role to be used with this Lambda function.

5. Click Add permissions, and then click Attach policies, and then select the policies to be attached to this custom destination.

6. Click Attach policies.

Add destination

Use a webhook destination to send query results from Amperity to a Lambda function running in AWS Lambda in your instance of Amazon AWS.

To add a destination

1. From the Destinations page, open the menu for a destination that is configured for custom destination, and then select Add data template.

   This opens the Add Data Template dialog box.

2. Enter the name of the data template and a description. For example: “custom destination” and “Send query results to custom destination.”

3. Under Webhook settings, add the Label Name. For example: “custom destination”.

   Important

   For custom webhooks, Webhook Settings must specify all settings that are required by your Lambda function as JSON. For example:

   {
     "setting1" : "value",
     "setting2" : "value"
   }
   

   If your Lambda function does not have any settings OR when your Lambda function is deployed as a serverless application, use empty curly braces: “{}”.

4. Enable the Allow customers to use this data template option, and then enable the Make available to campaigns option. This allows users to send campaign results from Amperity to custom destination.

5. Verify all template settings and make any required updates.

6. Click Save.

Add data template

A data template defines how columns in Amperity data structures are sent to downstream workflows. A data template is part of the configuration for sending query and segment results from Amperity to an external location.

You can configure Amperity to send query results to a custom destination. These results are sent using an orchestration and will include all columns that were specified in the query.

To add a data template for orchestrations

	From the Destinations tab, open the menu for a destination that is configured for custom destination, and then select Add data template. This opens the Add Data Template dialog box. Enter the name of the data template and a description. For example: “custom destination” and “Send query results to custom destination.”.
	Verify business user access to queries and orchestrations and access to segments and campaigns. A business user may also have restricted access to PII, which prevents them from viewing and sending customer profile data. If business user access was not configured as part of the destination, you may configure access from the data template.
	Verify all configuration settings. Note Serverless destinations (such as Amazon Pinpoint, Amazon Redshift, and AWS Connect) use empty webhook settings: “{ }”. A custom webhook will often need to specify a JSON object. Note When the settings required by the custom Webhook destination were are not configured as part of the destination, you must configure them as part of the data template before making this destination available to campaigns.
	Review all settings, and then click Save. After you have saved the data template, and depending on how you configured it, business users can send query results to a custom destination to custom destination.

Extend the AWS Lambda function

Amperity provides an open source toolkit that you can use as a reference when building AWS Lambda functions. This toolkit contains example AWS Lambda functions, an AWS Lambda function you can use as a template, a helper class, and a tools that help you set up and configure your local environment.

To extend the AWS Lambda function using the Amperity open source toolkit

1. Open the Amperity open source toolkit: amperity-lambda-runner.

2. Follow the steps in the README to configure your environment.

3. Extend your AWS Lambda function to support your custom workflow using Python. You are not required to use the Amperity template as your starting point. You may replace the helper class with your own.

4. Build your AWS Lambda into a ZIP file archive, and then upload it AWS Lambda.

5. Test the AWS Lambda using a small file and verify the workflow runs as intended.

When the custom destination makes a request to your AWS Lambda function, you should receive a response similar to:

{
  'settings': {},
  'label_name': '',
  'access_token': '',
  'webhook_id': '',
  'callback_url': '',
  'data_url': ''
}