About scheduled workflows¶
A workflow is the series of steps that your data follows as it moves through Amperity. Information about a workflow, including details and individual tasks, is available from the Workflows page.
For example, many workflows are end-to-end process that:
Use a courier to pull data to Amperity.
Standardize data using semantic tags and feeds.
Add data to domain tables.
Build databases.
Run queries and segments to return results and audiences for use with downstream workflows.
Send results to destinations using orchestrations and campaigns.
An automated workflow is configured to run these steps automatically, and then send workflow alerts when warnings and errors within a workflow are detected.
Automation settings¶
A workflow can be configured to run at defined intervals, such as daily or weekly, and then can be configured to run specific couriers, bridges, and activations.
Set to active¶
A courier group must be activated in order for it to run on an end-to-end schedule that pulls data to Amperity using couriers, syncs data using bridges, runs Stitch, refreshes databases, and then runs any orchestration, orchestration group, campaign, or profile API endpoint that is associated with the courier group.
A courier group that is deactivated may be run manually.
Name¶
Give the workflow a clear and succinct name that is meaningful. This name will appear in various locations, including other locations within the Amperity user interface, error messages related to the workflow, and potentially in downstream applications and workflows.
Define schedule¶
A schedule defines the frequency at which a courier group runs. All couriers in the same courier group run as a unit and all tasks must complete before a downstream process can be started. The schedule is defined using cron.
Cron is a time-based job scheduler that uses cron syntax to automate scheduled jobs to run periodically at fixed times, dates, or intervals.
Cron syntax specifies the fixed time, date, or interval at which cron will run. Each line represents a job, and is defined like this:
┌───────── minute (0 - 59)
│ ┌─────────── hour (0 - 23)
│ │ ┌───────────── day of the month (1 - 31)
│ │ │ ┌────────────── month (1 - 12)
│ │ │ │ ┌─────────────── day of the week (0 - 6) (Sunday to Saturday)
│ │ │ │ │
│ │ │ │ │
│ │ │ │ │
* * * * * command to execute
For example, 30 8 * * * represents “run at 8:30 AM every day” and 30 8 * * 0 represents “run at 8:30 AM every Sunday”. Amperity validates your cron syntax and shows you the results. You may also use crontab guru to validate cron syntax.
Amperity uses cron syntax to schedule the time at which a courier group is available for transferring files from a customer data source location to Amperity. A courier group that is scheduled runs automatically. Schedules are in UTC.
Note
Scheduling a courier group is optional. When a courier group is not assigned a schedule, it may be run manually on an ad hoc basis.
A courier group that is scheduled to run on a daily basis will check for files on a daily basis.
A courier group that runs less frequently, such as weekly, bi-weekly, monthly, or quarterly, will (by default) look for files on each day that has passed since the last time the courier group ran.
When a courier group is configured to run less frequently, you can also configure that courier group to only look for files on a specific day.
To use this approach, it is recommended to configure your upstream systems to make their files available to the courier group on a schedule that ensures they will be available to the courier group within a 24-hour window. This window is based on the schedule that is defined for the courier group.
Configure the courier group to run at the end of that 24-hour window, and then enable the Only retrieve files dropped in the past day? option, which forces the courier group to only look for files that were made available within the previous 24 hours.
Tip
Daylight savings time can affect a schedule. Be sure to set the schedule to be stable and not require changes over time. For example: if a schedule is set to 12:30 AM, and then you fall back, the schedule may become 11:30 PM (fall back) or 1:30 AM (spring forward).
Configure alerts¶
The following alerts can be configured:
A workflow succeeds.
A workflow’s runtime has exceeded a configurable number of hours.
Set the run type¶
A courier group can be configured with any of the following run types:
- Full
A full workflow refreshes domain tables, runs Stitch, refreshes your customer 360 database, and then runs every activation that is configured to run as part of this courier group workflow.
- Refresh
A refresh workflow refreshes domain tables, runs Stitch, refreshes your customer 360 database, but does not run any activations.
Important
Use partial workflows in sandboxes to ensure that data in your sandbox is not inadvertently sent to downstream destinations.
- Source
A source workflow refreshes domain tables, but does not run Stitch.
Couriers¶
A courier group is a list of one (or more) couriers that are run as a group, either ad hoc or as part of an automated schedule. A courier group can be configured to act as a constraint on downstream workflows.
Note
A courier group is typically configured to run automatically on a recurring schedule. All couriers within a courier group run as a unit; couriers with required files must complete before any downstream processes, such as Stitch or database generation, can be started.
For each courier with required files, Amperity determines if those files have updates, and then pulls updated files to Amperity. Depending on the run type, Amperity may then run Stitch and generate (or refresh) a customer 360 database. Orchestrations, recurring campaigns, and Profile API indexes may be configured to run as part of a courier group after the customer 360 database is refreshed.
Duration thresholds¶
A runtime alert is a type of workflow alert that is sent when a courier group has run longer than a configured amount of time. A runtime alert is sent to any email address or Slack channel that is configured for the Courier group workflow type.
To configure a workflow alert for a courier group
From the Sources page, open the menu for a courier group, and then select Edit.
Enable the Alert when runtime exceeds checkbox, and then set the number of hours (or minutes) at which, when this amount of time is exceeded, a workflow alert will be sent.
Click Save.
Missing files¶
A courier group can be configured to send email and Slack alerts when files are missing, and then:
Notify and continue¶
A courier group can be configured to send workflow alerts when one (or more) files are missing, and then continue processing if files are missing.
Tip
Some files are not considered essential to the daily Amperity run. The reasons why a particular file may be considered non-essential will vary from tenant to tenant, but they may include situations like:
A data source is mostly static
A data source does not contain PII that will affect the quality of the Amperity ID.
A data source is associated with a workflow that often misses the configured Amperity wait time period.
To send a workflow notification, and then continue the workflow
From the Sources page, open the menu for a courier group, and then select Edit.
On the Couriers tab, under the name of a courier group, set Notify when missing? to enabled, and then set Stop when missing? to disabled.
Click Save.
Notify and stop¶
A courier group can be configured to send workflow alerts when one (or more) files are missing, and then stop processing if files are missing.
To send a workflow notification, and then stop the workflow
From the Sources page, open the menu for a courier group, and then select Edit.
On the Couriers tab, under the name of a courier group, set Notify when missing? to enabled, and then set Stop when missing? to enabled.
Click Save.
Wait for missing files¶
When files are missing or late, in addition to sending an email alert and either continuing or stopping the workflow, Amperity will continue to attempt to find the these files. Use the Wait setting to configure amount of time Amperity should wait:
Files can be missing for any number of reasons, including by delays that may have occurred in upstream workflows that exist outside of Amperity. And in many situations a file is late, not missing.
Bridges¶
A bridge may be configured to run as part of a scheduled workflow. Use the Bridges tab in the Scheduled workflows dialog to add one or more bridges.
Activations¶
Activations are orchestrations, campaigns, and Profile API indexes that are configured to be run as part of a scheduled workflow.
Orchestrations¶
An orchestration can be configured to run after a courier when the run mode is set to Automatic and an active courier group is selected from the dropdown menu.
Profile API indexes¶
A Profile API index can be configured to be refreshed as part of a scheduled workflow when the schedule is set to Run after courier group and an active courier group is selected from the dropdown menu.
Recurring campaigns¶
A recurring campaign can be configured to run as part of a scheduled workflow when the schedule is set to Run as part of workflow and an active courier group is selected from the dropdown menu.
