Offline Imports

When it is not possible to establish a real-time data stream from a source, such as an offline system, historical orders, or certain CRM providers, Kevel Audience supports data onboarding through secure file uploads or specific third-party API integrations.

Quick-Start

Upload Files

The first step towards importing data into the CDP is uploading your files. We recommend using the S3 bucket dedicated to your CDP. You can learn more about this process on the Upload Files documentation page.

How to Import

Offline imports can be configured using the Dashboard under Collect > Data Imports, by clicking the New Import button.

From here, you can configure these types of imports:

• Order imports - Import order data into the CDP.
• User imports - Import CRM data into the CDP.
• Product imports - Import product information into the CDP.

caution

Before starting an import, make sure that:

• Your files are properly encoded as UTF-8 or that the appropriate encoding is selected when importing;
• The IDs (user IDs, order IDs, product IDs) in the imported files match the ones sent by events;
• Required fields are present in all lines;
• The fields have the correct keys (case sensitive).

Dry Run

A Dry Run of an import is a simulation of the import, where everything in the import pipeline runs except for storing the imported data. This means the files are downloaded and read, transformations (if any) are applied, and the values are validated. The results of a dry run, which include errors and some statistics on the values on the files, are displayed in the same way as the results of an actual import, save for a banner that informs that it is a dry run.

It is recommended to dry run before performing the import to ensure data will be imported as expected.

From the dry run results, it is possible to begin the actual import with the same configurations by clicking "Clone and import".

Orders

Export your CRM orders and import them into the CDP.

Use Cases

Importing orders into the CDP has 3 main use cases, as described below.

Bootstrap Your System with Historic Order Data

Importing orders plays an important role when bootstrapping a system. While Kevel Audience builds user profiles with attributes produced from the users' interaction with the website, businesses usually have a lot of information on orders placed before events are collected. These orders can add valuable information to produce user attributes. The attributes that benefit the most from a bootstrap from past orders are the ones related to the next purchase predictions, RFM and lifetime value. The attributes related to the likelihood to buy are produced using different types of events from the users' interaction with the website (besides the actual purchases), so they don't benefit much from an import of orders previously unseen in the system.

Ensure the Orders Used by the CDP Are in Sync with Your CRM Orders

Orders can change their state (for example, becoming canceled or refunded) without an explicit event from the user on the website. In those scenarios, periodic import of orders from your CRM backend can help Kevel Audience have a more accurate view of existing orders.

Complement Order Information Collected by Tracking Events with Details Private to the CRM Backend

Sometimes, the data available when producing events to be collected by Kevel Audience lacks some details that are private and, therefore, not directly accessible by the website. Typical scenarios are information missing from specific line items (like product-related data) or incomplete [payment information] (/collect/events/orderPlace#order) due to actions that happen after the event is triggered. To ensure that Kevel Audience has access to relevant order details, CDP users should set up a periodic import of orders.

Order Events from Imports

Order imports generate order events. If the order is unknown, an orderPlace is generated. Otherwise, if the import provides new information, for example, the status of a line item is updated, an orderUpdate event is generated.

Considering the provided product ID, events generated from order imports are enriched if a product feed is available. Only omitted attributes are added from the product feed. Attributes available on the import are kept on the event and are not replaced by data from the product feed.

Formats

We currently have support for the following 1st and 3rd party formats:

Supported schemas

• Kevel Audience CSV
• Shopify CSV

Kevel Audience CSV

This format is intended for order imports using CSV, where each line is an Order is serialized to CSV. You can download an example file here.

Relevant notes:

• The export file should be encoded using UTF-8.
• All line items for the same order must be listed sequentially. Otherwise, only the last sequential line items of the order will be persisted;
• Given an order with an arbitrary number of rows, only the first row is used to extract Order details and all rows are used to extract Line item details. This means that Orders with more than 1 Line item do not need to populate the order columns for all columns and can be left empty.
• If the provided userId is not hashed (E.g. idType=email), you can configure the import to hash it (to a idType=email_sha256) beforehand.
• The userId field is validated accordingly to the type specified in userIdType field. You can find more information about user ID types in the User Identifiers documentation.

Column	Required	Type	Description	Details	Example
orderId	Required	String	Order identifier	Must be unique by Order	or123
userIdType	Required	String	The type of user identifier (e.g. "email")	Check the User Identifiers documentation	email
userId	Required	String	The user identifier		john@email.com
currency	Required	String	The currency used in monetary values	Check the supported currencies	EUR
total	Required	Double	Total value of the order with all discounts already considered, including subtotal, tax and shipping	total = subtotal + tax + shipping	102.5
subtotal	Required	Double	The value of the order, when tax and shipping are excluded	subtotal = total - tax - shipping	85
tax	Required	Double	The total applied tax value for the order		11.5
shipping	Required	Double	The total shipping costs for the order		6
discount	Required	Double	The total discounted value for the order		5
refund	Required	Double	The total refunded value for the order		0
paymentMethod		String	The payment method		Visa
shippingMethod		String	The shipping method		UPS
shippingCountry		String	The destination country		France
source		String	Source from where the order was placed. Examples are web, phone call, POS.
promotions		List of Strings	Vouchers applied in the purchase.	Strings separated by ;	WINTERSALE
createdAt	Required	DateTime	When the order was created	Date format: ISO-8601	2020-09-16T10:38:06+0100
lastUpdated		DateTime	When the order was last updated	Date format: ISO-8601	2020-09-16T20:13:31+0100
custom_<name...>		String	Attribute name and value to be imported as custom field	These columns can be specified multiple times with different names with the "custom_" prefix. The imported order will have those attribute names (without the "custom_" prefix) and respective values as custom fields. For example, "custom_color" will be mapped to "customFields.color" in the order.
lineItem_id		String	A unique identifier for the line item within the order		li125zc
lineItem_productId	Required	String	Product identifier		p125zc
lineItem_itemGroupId		String	Identifier for a group of products that come in different versions (variants)		p125
lineItem_name		String	Product name		My Product
lineItem_sku	Required	String	Product SKU		p125zc-5
lineItem_brand		String	Product brand		Nike
lineItem_category		String	Product category		Men > Shirts > Short Sleeve Shirts
lineItem_categoriesList		List of Strings	A hierarchically sorted list of the sub-categories that this product belongs to. The first item on this list should be the most general sub-category, while the last should be the most specific. Any item on this list should correspond to a sub-category of the item immediately preceding it.	Strings separated by ;	Men;Shirts;Short Sleeve Shirts
lineItem_currency	Required	String	The currency used in monetary values	Check the supported currencies	EUR
lineItem_total	Required	Double	Total value of the line item with all discounts already considered, including subtotal, tax and shipping	total = subtotal + tax + shipping	102.5
lineItem_subtotal	Required	Double	The value of the line item, when tax and shipping are excluded	subtotal = total - tax - shipping	85
lineItem_tax	Required	Double	The total applied tax value for the line item		11.5
lineItem_shipping	Required	Double	The total shipping costs for the line item		6
lineItem_discount	Required	Double	The total discounted value for the line item		5
lineItem_refund	Required	Double	The total refunded value for the line item		0
lineItem_quantity	Required	Double	Quantity of items purchased from this line item		1.0
lineItem_promotions		List of Strings	Vouchers applied in the purchase or displayed together with the product	Strings separated by ;	WINTERSALE
lineItem_adult		Boolean	Whether the product includes sexually suggestive content.	Must be true or false. Defaults to false	false
lineItem_subscriptionDuration		Long	If the purchase relates to a subscription, captures the subscription duration, in milliseconds.		15552000000
lineItem_size		String	Product size		S
lineItem_sizeSystem		String	Country of the size system used by your product.		EU
lineItem_status	Required	String	Line item status. placed the order has been placed but the item is yet to be paid paid the item has been paid but not yet delivered fulfilled the item has been paid and delivered refunded the item has been refunded cancelled the order has been modified to exclude this line item	Supported status: placed, paid, fulfilled, refunded, cancelled	fulfilled

This example file shows how each field can be used.

Shopify CSV

Kevel Audience supports CSV order exports from Shopify. The documentation for generating a Shopify export can be found here. Please ensure you select Export orders to export all order information instead of Export transaction histories, which only exports the transactions.

Relevant notes:

• Shopify uses the Name column to distinguish between different orders inside the CSV file, so it must be unique per Order;
• All Line items for the same Order must be listed sequentially;
• Given an order with an arbitrary number of rows, only the first row is used to extract Order details and all rows are used to extract Line item details. This means that Orders with more than 1 Line item do not need to populate the order columns for all columns and can be left empty.
• If the provided userId is not hashed (e.g. idType=email), you can configure the import to hash it (to a idType=email_sha256) beforehand.
• The email field is validated as an email.
• It's important that the extracted order identifier from the shopify import matches the orderId provided via tracking events. When importing, you'll be able to select which field to use as the order identifier between the Id and Name columns.

Kevel Audience will consider the following attributes while importing a Shopify file:

Column	Required	Type	Details
Name	Required	String	Must be unique by Order and can be the same as Id (Can be used as an order identifier)
Email	Required	String
Id	Required	String	(Can be used as an order identifier)
Currency	Required	String	Check the supported currencies
Total	Required	Double
Subtotal	Required	Double
Taxes	Required	Double
Payment Method		String
Financial Status	Required	String	Supported status: authorized, paid, partially_paid, pending, refunded, partially_refunded, voided
Refunded Amount	Required	Double
Discount Amount	Required	Double
Discount Code		String
Shipping Method		String
Shipping Country		String
Shipping	Required	Double
Source		String
Created at	Required	DateTime	Date format: yyyy-MM-dd HH:mm:ss Z
Paid at		DateTime	Date format: yyyy-MM-dd HH:mm:ss Z
Fulfilled at		DateTime	Date format: yyyy-MM-dd HH:mm:ss Z
Lineitem quantity	Required	Double
Lineitem name	Required	String
Lineitem price	Required	Double
Lineitem discount	Required	Double
Lineitem fulfillment status	Required	String	Supported status: pending, partial, restocked, fulfilled
Lineitem sku	Required	String
Vendor		String

Users

Export your CRM data and import them into the CDP. Please note that attributes imported through files typically take 4 hours to be available in the Segment Builder UI and in the Custom Attributes UI.

Use cases

• Import CRM data to use in activations;
• Complement the users' profile with their CRM data.

Info

We automatically lowercase the following User IDs from the default ID Types list:

• Email
• Email SHA256
• Email MD5
• Unreliable Email SHA256
• Unreliable Email MD5
• Phone Number SHA256

Additionally, if a transformation is required for any ID type before being imported (for example hashing plaintext emails), we also lowercase and trim the original ID before transforming it.

We support importing CRM data from different Sources.

File and Folder Sources

The following formats are supported:

Supported schemas

• CSV
• JSON lines

The import will import every field as a custom User Attribute of the User Profile. Some notes:

• The file must have at least one field which identifies the user (See User Identifiers). You can select which field contains the User ID and the associated User ID type.
• By default, the attribute merge strategy is merge, which means the imported attributes will be merged with the existing attributes so that non-imported attributes remain unaffected in the User Profile.
• All attributes will be imported as text if a CSV file is imported. This restricts the ability to segment the user using numeric rules (e.g., numberOfOrders > 20 won't work). As a workaround, you can apply castTo transformations to known numeric fields. The attributes will use the original JSON types if a JSON-lines file is imported.

E-goi Contacts Source

This third-party integration loads user data from an E-goi Contact List and imports it as attributes and user IDs. If the contact information contains an email, it will be transformed into sha256 representation and merged into any existing profiles. The E-goi contact ID is also imported and allows you to use the E-goi Destination, which will use that ID to attach tags to activated users.

To configure this import you should first select E-goi Contacts from the Source Type dropdown and then provide an E-goi API key with permissions to List All Contacts and Get My Account Info, as well as the List ID from which you want to download and import contact data. If you want to import only a subset of Contacts from that List that belong to a given Segment, you may fill in the Segment Id input. Otherwise, you may leave that field empty.

An API key can be created in your Integrations dashboard by choosing to Create a New Key and make sure the necessary permissions are selected:

The set of required permissions are the following:

• Get all contacts
• Get all contacts by segment
• Get my account info

The list id can be taken from your E-goi dashboard and should be the numeric ID before its title:

The segment id can be taken from you List view on the E-goi dashboard, and should correspond to the numeric value that prefixes the segment name:

Notice that the user import from this type of Source does not allow you to specify any transformations.

E-goi Attribute Mapping

This source imports a subset of the available E-goi Contact fields, and maps them either to CDP user IDs or to user attributes. User attributes are mapped from E-goi original fields using the egoi.account<ACCOUNT-ID>.list<LIST-ID>.<E-GOI-ATTRIBUTE> format. <ACCOUNT-ID> is the E-goi account ID to which the provided API key refers to and is automatically determined, so you don't have to provide it explicitly. <LIST-ID> corresponds to the List ID provided in the form and <E-GOI-ATTRIBUTE> corresponds to the original Contact field this source will import into the CDP.

For example, importing the status field from an E-goi Contact might result in egoi.account123.list321.status, if the API key used belongs to the E-goi account 123 and the user submitted 321 as the List to import.

The following describes the fields that are consumed from the E-goi Contacts and how they are mapped into User Attributes:

E-goi Contact Field	CDP User ID or Attribute	Description
contact_id	User ID: Internal E-goi User ID Type	The contact_id field is mapped into an internal user ID type whose main purpose is to be used by the E-goi Destination
email	User ID: Email SHA256	The E-goi plain-text email field, if available, hashed and mapped to a SHA256 User ID
status	Attribute: egoi.account<ACCOUNT-ID>.list<LIST-ID>.status	The Contact status
consent	Attribute: egoi.account<ACCOUNT-ID>.list<LIST-ID>.consent	The Contact consent
email_status	Attribute: egoi.account<ACCOUNT-ID>.list<LIST-ID>.emailStatus	The Contact email status, if an email is available
cellphone_status	Attribute: egoi.account<ACCOUNT-ID>.list<LIST-ID>.cellphoneStatus	The Contact cellphone status, if a cellphone is available
phone_status	Attribute: egoi.account<ACCOUNT-ID>.list<LIST-ID>.phoneStatus	The Contact phone status, if a phone is available

Please refer to E-goi Contact API documentation for more information about the existing Contact fields and their possible values.

Product Data

Export your products' information and import them into the CDP.

Use cases

• Augment product-related tracking events.

We have support for importing product information in the following formats:

Supported schemas

• Google Product data following the RSS v2.0 or Atom v1.0 specifications.

Importing product data allows for product-related tracking events to be complemented with the products' information even if the original event did not include them.

For more information, see the Product Feed documentation.

Product Import Results

A completed product import presents details in four main sections.

Firstly, there are the configuration parameters of the report and its status. This section is present for product imports of all statuses (Queued, In Progress, Completed, Cancelled, and Failed), whereas the remaining only exists for completed product imports. (1)

Then, there is an overview of the imported products. The "File Items Count" represents the total number of products present in the file, and the "Successfully Imported Products Overview" is a table detailing the total number of valid products in the file, as well as how many of them were (or will be, if it is a dry run) added, updated, or remained unchanged. (2)

There is also a sample table, which allows you to inspect a sample of the imported products. When expanded, it displays a table with the first and last five elements read from the file, which can be clicked to open a detailed overview. (3)

Finally, there is a summary table of all the issues on the import. It provides hints on why a value for a column was considered an issue and how many items have the issue, as well as what that means in the total number of items in percentage. Import errors invalidate the entry, resulting in it being discarded during the import. (4)

Advanced Configurations

Import Schedules

Audience users can configure an offline import to execute and repeat on a given schedule.

The schedule can be specified according to years, months, days and hours. You can build the schedule by choosing appropriate values on the dropdown boxes and see the resulting description below. All dates and times are in UTC timezone.

Please note that, if an import gets triggered while another instance pertaining the same schedule is still running, the new one will be skipped.

You can also verify the status of your periodically configured imports.

File Sources

We support importing files with the following protocols: http, https, s3 and sftp.

We have support for importing files compressed using gzip. By default, we'll try to detect the necessary decompression method from the file name extension.

info

For convenience, each Kevel Audience system has a pre-created AWS S3 bucket that the client can use to upload files to be imported securely later. Kevel Audience will supply the credentials for the bucket to the client during onboarding.

caution

In order to use a s3 bucket other than then provided one, or an authenticated sftp you should contact Kevel support.

Folder Sources

Audience also supports importing files from folders via S3 or SFTP. Folder imports can be run on-demand or scheduled, making it easier to upload multiple files from the same location.

As shown below, to set up a folder import, users need to specify a folder path using one of the supported protocols (s3 or sftp) and indicate the file extension for the files to be imported. Only files located at the root of the folder that match the specified file extension will be imported, in alphabetical order. As with individual file imports, the decompression method will be automatically detected based on the file extension.

Shortly after the import starts, users can verify which files will be imported from each folder.

Scheduled Folder Imports

In scheduled folder imports only files with a name that hasn't yet been imported will be considered on each import run. So, each time an import gets triggered, any files that were already imported in previous runs of that same scheduled import will be skipped.

Files in the same folder are distinguished by their names and imported alphabetically. Therefore, to guarantee that files are uniquely named and imported in the expected order, it is recommended to choose a naming format based on the file's creation date. Such a strategy will also allow users to easily determine which files can be safely discarded from the folder. One possible approach is to prefix the name of the files with its creation date using the format YYYYMMDDHHMMSS[-N], where N is an optional version number to distinguish files with identical timestamps (e.g.: 20211006090000-1.csv, 20211006090000-2.csv, etc).

File Encodings

You can specify the file encoding of the files to be imported. Most files should use the very common "UTF-8" file encoding. However, when exporting files from specific CRM platforms or third-party providers, the files can be encoded with another format, which results in garbled text or in an error when importing into the CDP. To address this, file encoding auto-detection can be selected when importing data into the CDP. This process relies on heuristics and is not 100% accurate for all file encodings. If a user knows the correct encoding for the file, or auto-detection is failing, it is also possible to explicitly specify the file's encoding.

Data Transformation

Data transformation allows you to rename fields and map values without editing the original file.

The most basic transformations are:

• Rename field (e.g. renaming the field "number_of_orders" to "completedOrders". The "number_of_orders" field will disappear).
• Copy field (e.g. copying the field "number_of_orders" to "completedOrders". The "number_of_orders" field will be unaffected).
• New field (e.g. adding the field "completedOrders" with the value 0 set for all lines)
• Delete field (e.g. Deletes the field "completedOrders")

For the previous (Rename/Copy) transformations, you can also specify some value transformations to apply (i.e. transformations that will affect the values of the fields):

Mapping Field Values

Accepts an exhaustive list of from -> to values to transform. This transformation will iterate over the provided mappings and if the field's values matches the from value, it will be replaced with the to value. If no value matches, the field value will remain unaltered.

Example:
Given the mappings "1" -> "one" and "2" -> "two":

Original value	Resulting value	Description
"1"	"one"	Value is transformed
"2"	"two"	Value is transformed
"other"	"other"	Value is unaltered because it doesn't match any of the from values
1	1	Value 1 is different from value "1" (number instead text) so it doesn't match the "1" -> "one" mapping

note

When dealing with CSV files, when this transformation takes place, all field values are interpreted as text.

Extracting a Custom Timestamp

Allows parsing a non-standard DateTime format into a ISO 8601 format. Requires the user to specify the custom DateTime pattern which the file to import is using.

Example:
Given the pattern yyyy/MM/dd hh:mm, the value 2021/01/21 14:25 will be converted to the ISO 8601 format 2021-01-21T14:25:00Z.

Casting Field Value Types

Allows casting values to a specific type. This is useful mostly for User Imports where the values are directly set as user attributes and the type (most commonly, whether we're dealing with text, or a number) is relevant (e.g. to create segments with numeric rules with that attribute). On the other hand, while dealing with Order Imports or Product Imports, users don't need to cast the fields to the correct types. This happens because these types of imports have well-defined schemas that will implicitly perform the transformation to the data type.

The supported cast types are: string, integer, float and bool.

Example:

Original value	Cast to String	Cast to Integer	Cast to Float	Cast to bool
"2.4"	"2.4"	2	2.4	true
"0"	"0"	0	0.0	false
"car"	"car"	Error	Error	Error
"false"	"false"	Error	Error	false
3.8	"3.8"	3	3.8	true