Triggering a Customer Insights - Data process—whether through a schedule or the 'Refresh all' option in the user interface—initiates a batch run. While the batch may initially seem like a black box, it actually follows a well-defined and logical execution flow. This blog post unpacks that flow, providing insights into how the various stages of data processing are orchestrated behind the scenes.
In this blog, I’ll walk through the key jobs that execute during a Customer Insights – Data (CI-D) batch run: Ingestion, Data Preparation, Profile Unification, Activity Unification, Measures, Segments, and Exports.
It is important to highlight that CI-D uses a fair scheduling mechanism. This means jobs are distributed across the available compute resources to ensure balanced execution. So, even if a job is ready to run, it may remain queued briefly depending on system load and available capacity.
Below, you’ll find a visual representation of the job execution sequence. The dotted lines indicate dependencies between jobs. Please note: this illustration is meant to depict execution order and dependencies — not the actual duration of each step.
🔄 Data Ingestion
Data Ingestion, also referred to as Data Sources, is the first step triggered in a CI-D batch run. These jobs have no upstream dependencies within CI-D, meaning they initiate as soon as the batch starts. However, you must ensure that the required data is available at the source before the batch begins. The time to complete ingestion will vary across data sources, depending on the volume and structure of the data.
In the illustrated example, Data Source 1 ingests a single table containing profile data, Data Source 2 ingests a single table with transactional data, and Data Source 3 ingests two tables—one for profiles and one for transactions. A data source ingestion is considered complete only when all tables within that source have finished processing.
🛠️ Data Preparation
Data Preparation follows the ingestion step for each data source. As soon as a data source finishes ingesting, its preparation job is eligible to begin. This step is independent and has no downstream dependencies. Its main functions are:
- Generating statistics.
- Identifying problematic records, if any and create diagnostics tables with error details for them.
👤 Profile Unification
Profile Unification is the process of matching and merging customer data from different sources to create unified customer profiles, which are then hydrated into Dataverse.
This job can begin only after ingestion completes for all data sources participating in profile unification. In the illustrated example, Data Source 1 and Data Source 3 provide profile data, so unification starts after both the data sources have completed successfully. Notably, even if only one of the two tables in Data Source 3 is required for unification, the process still waits for the entire ingestion job for that data source to finish.
The process includes:
- Match
- Merge
- Customer Profile - responsible for loading/hydrating the unified profiles into Dataverse
- Search Preparation
- Search
📊 Activity Unification
Activity Unification unifies or connects transactional data from various sources to customer profiles; the unifiedactivity table is then hydrated into Dataverse.
Since activity unification requires customer profiles, it can only begin after:
- The Merge step of Profile Unification is complete meaning customer profiles are created
- All participating transactional sources have been ingested
In the scenario above, Data Source 2 and Data Source 3 are involved in activity unification..
If your data is semantically mapped (e.g., to SalesOrder or SalesOrderLine), the process also generates semantic tables for downstream use.
The online activity step is responsible for hydrating the unified activity table into Dataverse. Under normal circumstances, it runs in incremental mode, where CI-D calculates the upserts and deletes, and only the changes are pushed. However, if the job is triggered following a failure or a change in activity configuration, it runs in 'full' mode, which is executed asynchronously.
📈 Measures
Measures calculate KPIs and insights over ingested and unified data. The execution depends on the type of measure. If you have business measures where CustomerId or Customer Profile is not used as a dimension, their execution can begin as soon as the ingestion of the relevant data sources is complete. Measures that rely on CustomerId or Customer Profile will start after customer profiles are created. Likewise, measures dependent on unified activities or other downstream tables will begin once those processes have completed.
🎯 Segments
Segments identify and group customer profiles that meet specific criteria defined in the segment definition. Since this requires unified customer profiles, segmentation can begin only after the Merge step of profile unification. If a segment relies on activity or measure data, it must also wait for those respective processes to complete.
📤 Exports
Customer Insights - Data supports a wide range of export destinations including first-party services (e.g., Azure Data Lake Storage Gen2) and third-party platforms (e.g., Facebook Ads manager). Exports are primarily categorized into two types: data-out exports and segment exports.
- Data-out exports involves exporting data tables, typically to first-party destinations such as Azure Data Lake Storage Gen2. These exports can begin as soon as relevant table is ready—for example, a table created during profile unification like ConflationMatchPairs or Deduplication_... Data-out exports can also include segments.
- Segment exports generally export to third-party platforms like Facebook Ads Manager, LinkedIn Ads, DotDigital. Since these exports rely on segments, the export can only begin once the segment has completed execution and the necessary table is available.
Generally if an export contains multiple tables, the process will start as soon as one of the eligible tables is available and will continue to run until the last table is available and has been exported.
