Pull from OneTrust¶

OneTrust is a consent and preference management platform. It records the consent and preferences of each data subject for each purpose, such as email or SMS marketing, and helps companies comply with privacy laws such as GDPR and CCPA.

OneTrust can send the current consent status of each data subject for each purpose to Amperity using the Get Optimized List of All Purpose Details for All Data Subjects endpoint of the OneTrust Universal Consent & Preference Management API. Amperity lands one NDJSON file with one record for each data subject and purpose. Each record has the data subject’s current status for that purpose, such as active or withdrawn.

Amperity reads purpose details, which hold each data subject’s current consent. Amperity does not read consent receipts or transactions.

OneTrust returns purpose details for a time period in which they were last updated. A run for a specific time period pulls the purpose details that were updated during that time period. A run that loads all data pulls the purpose details that were updated from the Backfill start date until the run starts.

Beta

The OneTrust connector is in beta. Contact your Amperity representative to learn more.

The steps that are required to pull the current consent status of each data subject for each purpose to Amperity from OneTrust:

Get details
Add courier
Get sample file
Add feed
Add load operations
Run courier
Add to courier group

Get details¶

OneTrust requires the following configuration details:

The Client ID and Client Secret for a OneTrust client credential.

Create a client credential in OneTrust under Global Settings > Access Management > Client Credentials, and give it the CONSENT_READ scope. Amperity only reads consent from OneTrust, so no other scope is required. Copy the client ID and the client secret.

The client credential must belong to the same OneTrust environment as the Tenant URL that you enter for the courier.
The Tenant URL, which is the host of your OneTrust environment. For example: app.onetrust.com, app-eu.onetrust.com, or yourcompany.my.onetrust.com.

Amperity connects to the Tenant URL over HTTPS. A URL without a scheme is treated as an HTTPS URL. A URL that begins with http:// is not accepted, because Amperity sends the client secret to this host.
The Backfill start, which is the first day of consent updates to pull when the courier loads all data. For example: 2024-01-01. The day starts at midnight UTC.

Choose a date that is early enough to include every data subject whose consent you need. A purpose detail that was last updated before the Backfill start is not pulled by a run that loads all data.

Tip

Use SnapPass to securely share configuration details for OneTrust between your company and your Amperity representative.

Add courier¶

A courier brings data from an external system to Amperity.

To add a courier

From the Sources page, click Add Courier. The Add Courier page opens.
Find, and then click the icon for OneTrust. The Add Courier page opens.
Enter the name of the courier. For example: “OneTrust”.

From the Credential field, select an existing credential or select Create a new credential.

To add a credential, enter the name of the credential, a description, and the OneTrust Client ID and Client Secret. Click Save.

When finished click Continue.
Enter the Tenant URL, and then select the Backfill start date.
Load Operations contains an example load operation for the purpose-consents file. Clear Load Operations, and then add the load operation after the feed for this courier is configured.
Click Save.

Get sample file¶

Newline-delimited JSON (NDJSON) is a data format for structured data that defines the structure of JSON data using lines as separators. Each line in a NDJSON file is a valid JSON value.

Every OneTrust file that is pulled to Amperity must be configured as a feed. Before you can configure each feed you need to know the schema of that file. Run the courier without load operations to bring sample files from OneTrust to Amperity, and then use each of those files to configure a feed.

Amperity names the file onetrust-purpose-consents.ndjson. Load operations refer to this file as purpose-consents.

To get a sample file

From the Sources tab, open the menu for a courier configured for OneTrust with empty load operations, and then select Run. The Run Courier dialog box opens.
Select Load data from a specific time period, and then choose a recent time period during which consent was updated in OneTrust.

A run for a specific time period returns only the purpose details that were updated during that time period. A sample file can be empty if no consent was updated, and an empty file cannot be used to configure a feed.

A run that loads all data pulls every consent update since the Backfill start, which can take a long time for a large OneTrust account. See How data is pulled.
Click Run.

The file that the courier pulls is available for selection as an existing source from the Add Feed dialog box.

Add feed¶

A feed defines how to load data into a domain table, including specifying required columns and columns with semantic tags for customer profile (PII) or transactions data.

The file contains the purpose details that OneTrust returns, with the fields that OneTrust returns for them. Amperity does not rename, add, or remove fields. Use the sample file, rather than the OneTrust documentation, to configure the feed.

The fields that identify and describe each record include:

identifier, the data subject’s identifier in OneTrust, such as an email address. The file does not include the type of identifier or other data elements for the data subject, such as a name or a phone number.
id, the identifier of the purpose.
name, the name of the purpose.
status, the data subject’s current consent status for the purpose. For example: ACTIVE, WITHDRAWN, or NO_CONSENT.
consentDate and withdrawalDate, when consent was given and when it was withdrawn.
expiryDate, when the consent expires.
lastUpdatedDate, when the record last changed. Amperity uses this field to select records for a time period.

OneTrust also returns nested arrays and objects with each record, such as topics and customPreferences, which hold the data subject’s preferences for the purpose. They are landed as OneTrust returns them.

OneTrust documents timestamps both with and without a time zone offset. Check the timestamps in the sample file before you configure the feed.

Select identifier and id together as the primary key, so that the feed has one record for each data subject and purpose. Use lastUpdatedDate as the last updated field. A record that Amperity pulls again updates the existing record in the domain table, so the domain table holds each data subject’s latest consent for each purpose.

To add a feed

From the Sources tab, click Add Feed. This opens the Add Feed dialog box.
Under Data Source, select Create new source, and then enter “OneTrust”.
Enter the name of the feed in Feed Name. For example: “Purpose Consents”.

Tip

The name of the domain table will be “<data-source-name>:<feed-name>”. For example: “OneTrust:Purpose Consents”.
Under Sample File, select Select existing file, and then choose from the list of files. For example: “onetrust-purpose-consents.ndjson”.

Tip

The list of files that is available from this dropdown menu is sorted from newest to oldest.
Select Load sample file on feed activation.
Click Continue. This opens the Feed Editor page.
Select the primary key.
Apply semantic tags to customer records and interaction records, as appropriate.
Under Last updated field, specify which field best describes when records in the table were last updated.

Tip

Choose Generate an “updated” field to have Amperity generate this field. This is the recommended option unless there is a field already in the table that reliably provides this data.
For feeds with customer records (PII data), select Make available to Stitch.
Click Activate. Wait for the feed to finish loading data to the domain table, and then review the sample data for that domain table from the Data Explorer.

Add load operations¶

After the feeds are activated and domain tables are available, add the load operations to the courier used for OneTrust.

Example load operations

Load operations must specify each file that is pulled to Amperity from OneTrust.

Refer to the file as purpose-consents. Do not add a truncate operation. A run for a specific time period, including every scheduled run, pulls only the purpose details that were updated during that time period. A truncate operation would replace the consent of every data subject in the domain table with only those updates.

For example:

{
  "df-A1B2C3": [
    {
      "type": "load",
      "file": "purpose-consents"
    }
  ]
}

To add load operations

From the Sources tab, open the menu for the courier that was configured for OneTrust, and then select Edit. The Edit Courier dialog box opens.
Edit the load operations for each of the feeds that were configured for OneTrust so they have the correct feed ID.
Click Save.

Run courier manually¶

Run the courier again. This time, because the load operations are present and the feeds are configured, the courier will pull data from OneTrust.

Important

Select Load all data for the first run. A run for a specific time period pulls only the purpose details that were updated during that time period, so a first run for a specific time period loads the consent of only some data subjects.

To run the courier manually

From the Sources tab, open the menu for the courier with updated load operations that is configured for OneTrust, and then select Run. The Run Courier dialog box opens.
Select the load option, either for a specific time period or all available data. Actual data will be loaded to a domain table because the feed is configured.
Click Run.

This time the notification will return a message similar to:
```
Completed in 5 minutes 12 seconds
```

Add to courier group¶

A courier group is a list of one or more couriers that run as a group. A courier group can act as a constraint on downstream workflows and can run automatically as part of a scheduled workflow.

To add the courier to a courier group

From the Sources tab, click Add Courier Group. This opens the Create Courier Group dialog box.
Enter the name of the courier. For example: “OneTrust”.
Add a cron string to the Schedule field to define a schedule for the orchestration group.

A schedule defines the frequency at which a courier group runs. All couriers in the same courier group run as a unit and all tasks must complete before a downstream process starts. Define a schedule using cron.

Cron syntax specifies the fixed time, date, or interval at which cron runs. Each line represents a job. 30 8 * * * represents “run at 8:30 AM every day” and 30 8 * * 0 represents “run at 8:30 AM every Sunday”.

For example:
```
┌───────── minute (0 - 59)
│ ┌─────────── hour (0 - 23)
│ │ ┌───────────── day of the month (1 - 31)
│ │ │ ┌────────────── month (1 - 12)
│ │ │ │ ┌─────────────── day of the week (0 - 6) (Sunday to Saturday)
│ │ │ │ │
│ │ │ │ │
│ │ │ │ │
* * * * * command to execute
```
Amperity validates the cron syntax and shows you the results. You may also use crontab guru to validate cron syntax.
Set Status to Enabled.
Specify a time zone.

A courier group schedule is associated with a time zone. The time zone determines the point at which a courier group’s scheduled start time begins. A time zone should be aligned with the time zone of system from which the data is being pulled.

Use the Use this time zone for file date ranges checkbox to use the selected time zone to look for files. If unchecked, the courier group uses the current time in UTC to look for files to pick up.

Note

The time zone that is chosen for an courier group schedule should consider every downstream business processes that requires the data and also the time zones in which the consumers of that data will operate.
Add at least one courier to the courier group. Select the name of the courier from the Courier dropdown. Click + Add Courier to add more couriers.
Click Add a courier group constraint, and then select a courier group from the dropdown list.

A wait time is a constraint placed on a courier group that defines an extended time window for data to be made available at the source location.

Important

A wait time is not required for a bridge.

A courier group typically runs on an automated schedule that expects customer data to be available at the source location within a defined time window. However, in some cases, the customer data may be delayed and is not made available within that time window.
For each courier group constraint, apply any offsets.

A courier can be configured to look for files within range of time that is older than the scheduled time. The scheduled time is in Coordinated Universal Time (UTC), unless the “Use this time zone for file date ranges” checkbox is enabled for the courier group.

This range is typically 24 hours, but may be configured for longer ranges. For example, it is possible for a data file to be generated with a correct file name and datestamp appended to it, but for that datestamp to represent the previous day because of how an upstream workflow is configured. A wait time helps ensure that the data at the source location is recognized correctly by the courier.

Warning

This range of time may affect couriers in a courier group whether or not they run on a schedule. A manually run courier group may not take its schedule into consideration when determining the date range. Only the provided input days to load data from are used as inputs.
Click Save.

Important

Each scheduled run pulls the time period since the schedule’s previous run time, whether or not that run succeeded. Consent that was updated during the time period of a failed or missed run is not pulled until you run the courier for that time period. Pulling the same time period again updates records rather than duplicating them, because the feed has a primary key.

Leave Only retrieve files dropped in the past day? cleared unless the courier group runs daily. When it is selected, each scheduled run pulls only the 24 hours before the run, so a schedule that runs less often than daily skips the consent that was updated before those 24 hours.

How data is pulled¶

Time periods are split. OneTrust documents a maximum time period of 30 days for each request for purpose details. Amperity splits a longer time period into consecutive parts of up to 30 days and reads them in order. Consecutive parts share the second at which one ends and the next begins, so a record that was updated in that second can be pulled twice. The feed’s primary key updates that record rather than duplicating it.

Requests are paced. Amperity reads up to 50 records in each request and paces its requests to about 270 a minute, below the Consent API quota of 300 requests a minute in the OneTrust rate limits . OneTrust applies rate limits to the whole account, so Amperity shares them with your other OneTrust integrations. At this pace a run reads at most about 800,000 records an hour, so a run that loads all data from an early Backfill start can take several hours.

Runs succeed or fail as a whole. A request that is rate limited or times out is retried. If OneTrust keeps failing, or the run stops before every record is read, the courier run fails rather than load part of the consent.

Troubleshoot errors¶

The following errors may occur when a courier is saved, when it runs, or when a credential is tested.

Error	Resolution
Tenant URL must be your OneTrust environment’s host, such as app.onetrust.com.	Shown when the courier is saved. Enter the host of your OneTrust environment, such as `app.onetrust.com`. A URL that begins with `http://` is not accepted.
Backfill start must be a date, such as 2024-01-01.	Shown when the courier is saved. Select a date for the Backfill start.
OneTrust rejected the client ID and secret.	The client ID or client secret is incorrect, or the client credential belongs to a different OneTrust environment than the Tenant URL. Check both values and the Tenant URL.
OneTrust returned no access token.	The Tenant URL is not the host of your OneTrust environment. Correct the Tenant URL.
OneTrust rejected the access token.	The client credential is no longer valid in OneTrust. Check that the client ID and client secret are current, and update the credential.
OneTrust denied access to consent data.	The client credential does not have the CONSENT_READ scope. Add the scope to the client credential in OneTrust.
OneTrust has no consent API at the Tenant URL.	The Tenant URL points to a host that does not serve the OneTrust API. Use the host of your OneTrust environment, such as `app.onetrust.com`.
OneTrust kept rate limiting requests after several retries.	The rate limit is shared with your other OneTrust integrations. Avoid scheduling runs that overlap with other heavy use of the OneTrust API, and then run the courier again.
OneTrust kept timing out or failing after several retries.	A problem on the OneTrust side. Run the courier again after OneTrust recovers.
Could not reach OneTrust.	Check the Tenant URL. Amperity retries the run.
The run’s start must be before its end.	The Backfill start is in the future. Choose a date in the past.
OneTrust rejected the request, and retrying will not change that.	Contact your Amperity representative.
OneTrust returned an access token without a valid expiry, the file landed before every record was read, or the OneTrust connector failed unexpectedly.	Contact your Amperity representative.

Choose topic collections

Pull from OneTrust¶

Get details¶

Add courier¶

Get sample file¶

Add feed¶

Add load operations¶

Run courier manually¶

Add to courier group¶

How data is pulled¶

Troubleshoot errors¶