Import Guide
When it is not possible to establish a real-time data stream from a source, such as an offline system, historical orders, or certain CRM providers, Kevel Audience supports data onboarding through secure file uploads or specific third-party API integrations.
Quick-Start
Upload Files
The first step towards importing data into the CDP is uploading your files. We recommend using the S3 bucket dedicated to your CDP. You can learn more about this process on the Upload Files documentation page.
Setup the Import
The second step is to configure the import in the Audience dashboard (in the Collect > Data Imports area), by clicking the New Import button.
From here, you can configure these types of imports:
- Order imports - Import order data into the CDP.
- User imports - Import CRM data into the CDP.
- Product imports - Import product information into the CDP.
Before starting an import, make sure that:
- Your files are properly encoded as UTF-8 or that the appropriate encoding is selected when importing;
- The IDs (user IDs, order IDs, product IDs) in the imported files match the ones sent by events;
- Required fields are present in all lines;
- The fields have the correct keys (case sensitive).
Dry Run
A Dry Run of an import is a simulation of the import, where everything in the import pipeline runs except for storing the imported data. This means the files are downloaded and read, transformations (if any) are applied, and the values are validated. The results of a dry run, which include errors and some statistics on the values on the files, are displayed in the same way as the results of an actual import, save for a banner that informs that it is a dry run.
It is recommended to dry run before performing the import to ensure data will be imported as expected.
From the dry run results, it is possible to begin the actual import with the same configurations by clicking "Clone and import".
Advanced Configurations
Import Schedules
Audience users can configure an offline import to execute and repeat on a given schedule.
The schedule can be specified according to years, months, days and hours. You can build the schedule by choosing appropriate values on the dropdown boxes and see the resulting description below. All dates and times are in UTC timezone.
Please note that, if an import gets triggered while another instance pertaining the same schedule is still running, the new one will be skipped.
You can also verify the status of your periodically configured imports.
File Sources
We support importing files with the following protocols: http, https, s3 and sftp.
We have support for importing files compressed using gzip. By default, we'll try to detect the
necessary decompression method from the file name extension.
For convenience, each Kevel Audience system has a pre-created AWS S3 bucket that the client can use to upload files to be imported securely later. Kevel Audience will supply the credentials for the bucket to the client during onboarding.
In order to use a s3 bucket other than then provided one, or an authenticated sftp you should contact
Kevel support.
Folder Sources
Audience also supports importing files from folders via S3 or SFTP. Folder imports can be run on-demand or scheduled, making it easier to upload multiple files from the same location.
As shown below, to set up a folder import, users need to specify a folder path using one of the supported protocols (s3 or sftp)
and indicate the file extension for the files to be imported. Only files located at the root of the folder that match the
specified file extension will be imported, in alphabetical order. As with individual file imports, the decompression
method will be automatically detected based on the file extension.
Shortly after the import starts, users can verify which files will be imported from each folder.
In scheduled folder imports only files with a name that hasn't yet been imported will be considered on each import run. So, each time an import gets triggered, any files that were already imported in previous runs of that same scheduled import will be skipped.
Files in the same folder are distinguished by their names and imported alphabetically. Therefore, to guarantee that
files are uniquely named and imported in the expected order, it is recommended to choose a naming format based on the file's creation date.
Such a strategy will also allow users to easily determine which files can be safely discarded from the folder.
One possible approach is to prefix the name of the files with its creation date using the format
YYYYMMDDHHMMSS[-N], where N is an optional version number to distinguish files with identical timestamps
(e.g.: 20211006090000-1.csv, 20211006090000-2.csv, etc).
File Encodings
You can specify the file encoding of the files to be imported. Most files should use the very common "UTF-8" file encoding. However, when exporting files from specific CRM platforms or third-party providers, the files can be encoded with another format, which results in garbled text or in an error when importing into the CDP. To address this, file encoding auto-detection can be selected when importing data into the CDP. This process relies on heuristics and is not 100% accurate for all file encodings. If a user knows the correct encoding for the file, or auto-detection is failing, it is also possible to explicitly specify the file's encoding.
Data Transformation
Data transformation allows you to rename fields and map values without editing the original file.
The most basic transformations are:
- Rename field (e.g. renaming the field "number_of_orders" to "completedOrders". The "number_of_orders" field will disappear).
- Copy field (e.g. copying the field "number_of_orders" to "completedOrders". The "number_of_orders" field will be unaffected).
- New field (e.g. adding the field "completedOrders" with the value 0 set for all lines)
- Delete field (e.g. Deletes the field "completedOrders")
For the previous (Rename/Copy) transformations, you can also specify some value transformations to apply (i.e. transformations that will affect the values of the fields):
Mapping Field Values
Accepts an exhaustive list of from -> to values to transform. This transformation will iterate over the provided mappings and if the field's values matches the from value, it will be replaced with the to value. If no value matches, the field value will remain unaltered.
Example:
Given the mappings "1" -> "one" and "2" -> "two":
| Original value | Resulting value | Description |
|---|---|---|
"1" | "one" | Value is transformed |
"2" | "two" | Value is transformed |
"other" | "other" | Value is unaltered because it doesn't match any of the from values |
1 | 1 | Value 1 is different from value "1" (number instead text) so it doesn't match the "1" -> "one" mapping |
When dealing with CSV files, when this transformation takes place, all field values are interpreted as text.
Extracting a Custom Timestamp
Allows parsing a non-standard DateTime format into a ISO 8601 format. Requires the user to specify the custom DateTime pattern which the file to import is using.
Example:
Given the pattern yyyy/MM/dd hh:mm, the value 2021/01/21 14:25 will be converted to the
ISO 8601 format 2021-01-21T14:25:00Z.
Casting Field Value Types
Allows casting values to a specific type. This is useful mostly for User Imports where the values are directly set as user attributes and the type (most commonly, whether we're dealing with text, or a number) is relevant (e.g. to create segments with numeric rules with that attribute). On the other hand, while dealing with Order Imports or Product Imports, users don't need to cast the fields to the correct types. This happens because these types of imports have well-defined schemas that will implicitly perform the transformation to the data type.
The supported cast types are: string, integer, float and bool.
Example:
| Original value | Cast to String | Cast to Integer | Cast to Float | Cast to bool |
|---|---|---|---|---|
"2.4" | "2.4" | 2 | 2.4 | true |
"0" | "0" | 0 | 0.0 | false |
"car" | "car" | Error | Error | Error |
"false" | "false" | Error | Error | false |
3.8 | "3.8" | 3 | 3.8 | true |