How to Set up Synapse Link for Dataverse to Enhance System Integration
With news that Microsoft is deprecating Export to Data Lake, a popular upgrade to replace this system has emerged in Synapse Link for Dataverse.
One of the primary ways companies can adapt to this is by using Azure Synapse Link for Dataverse. Understanding how to set up Synapse Link for Dataverse allows you to leverage its capabilities for optimized system integration.
In this blog, we'll explore how to set up Synapse Link for Dataverse to achieve an optimized integration by:
- Leveraging Synapse Link for Dataverse with Dynamics 365 Customer Engagement (D365 CE) data, including setup steps and table definition.
- Setup and configuration in Dynamics 365 Finance and Operations (D365 Finance and Operations) within Synapse Link for Dataverse, demonstrated through a Power BI report.
- Discuss the deprecation of Export to Data Lake while highlighting the substantial benefits and specifics of leveraging two potential upgrade paths, specifically Synapse Link for Dataverse.
Understanding Synapse Link for Dataverse
First, let’s delve into the four key components of Azure Synapse Link for Dataverse to build a foundational understanding of how these elements are tied together. These components form the backbone of a comprehensive data integration and analytics environment:
- Dataverse: This low-code data platform, part of the Microsoft Power Platform, is instrumental in building and managing business applications like Dynamics 365 Customer Service. Dataverse provides a secure repository for business data and supports effective application development and management.
- Azure Data Lake Storage: As a cloud-based data lake solution, Azure Data Lake Storage excels in storing data of any format and enabling complex analytical workloads. It acts as a centralized repository, facilitating data capture and analysis through various frameworks and tools.
- Synapse: Positioned as a cloud-based analytics service, Synapse empowers organizations to process data efficiently and collaborate on analytics projects. It unlocks the potential of data, enabling organizations to generate valuable insights to drive informed decision-making.
- Synapse Dataverse Link: Serving as a crucial bridge between Dataverse and the Synapse platform, Synapse Link for Dataverse facilitates seamless data exchange and analysis. It enables continuous replication of tables from Dataverse to Azure Data Lake Storage Gen2 and Azure Synapse within your Azure subscription.
Key Features of Synapse Link for Dataverse
Next, let’s look at the Synapse Link for Dataverse. It is designed to enhance data integration and analytics capabilities to empower organizations to leverage data assets efficiently. These features include:
- Integration: This feature allows for real-time data analysis within Dataverse as it is created or updated, eliminating. That eliminates the need for manual data replication or synchronization.
- Simplified Data Access: Users can seamlessly access Dataverse data from Azure Synapse Analytics, streamlining data access without complex data pipelines or transformations.
- Unified Analytics: Synapse Link for Dataverse integrates Dataverse data with other sources within Azure Synapse Analytics. This enables comprehensive analytics and organizational reporting.
- Performance Optimizations: Built-in optimizations within Azure Synapse Analytics ensure efficient querying and analysis of Dataverse data.
- Cost-Effective Data Storage: Historical data can be stored cost-effectively in Azure Data Lake, allowing organizations to manage and analyze large datasets economically over time.
- Scalability: Synapse Link for Dataverse enables seamless scaling of analytics workloads to accommodate growing data volumes and evolving business needs.
Common Scenarios for Synapse Link for Dataverse
Stoneridge experts have identified several powerful applications for Azure Synapse Link for Dataverse that showcase its versatility and capabilities. These scenarios highlight how Synapse can drive value across different aspects of business operations:
- Sales Forecasting: By analyzing historical sales data from D365 alongside external market trends, organizations can leverage Synapse to forecast future sales performance accurately. This enables proactive decision-making and strategic planning based on data-driven insights.
- Customer Segmentation: Synapse Link for Dataverse facilitates customer segmentation by utilizing demographics, purchasing behavior, and interaction history from D365. This segmentation empowers targeted marketing campaigns and personalized customer engagement strategies.
- Supply Chain Optimization: Synapse Link for Dataverse plays a pivotal role in supply chain optimization. It does this through advanced analytics to identify inefficiencies, mitigate risks, and enhance overall supply chain performance. It also consolidates and analyzes critical data related to:
- Inventory levels
- Demand forecasts
- Supplier performance
- Logistics
- Financial Reporting and Analytics: Synapse Link for Dataverse seamlessly integrates with D365 Finance and Operations to pull data for advanced financial analysis, forecasting, and budgeting.
Regardless of your organization's goal, Synapse is a powerful tool for you to leverage data for informed decision-making.
The diagram above illustrates the components of Azure Synapse Link for Dataverse. On the left side, you have data producers, which generate or create data. This includes Dynamics apps like Customer Engagement or Finance and Operations or other apps using Dataverse storage. It could also include external data sources integrated into Azure Data Lake.
Synapse Link for Dataverse extracts data from these applications, continuously exporting both data and metadata. It tracks changes such as inserts, updates, and deletes in the environment. Any modifications to application structures (like adding or removing columns) are reflected in the metadata exported by Azure Synapse Link for Dataverse.
While data can be pulled directly from Azure Data Lake, Azure Synapse Analytics takes this a step further by providing efficient access to data through standard SQL queries and native tools like Apache Spark and Synapse pipelines for data transformation.
Synapse Link for Dataverse acts as the bridge between data producers and data consumers. For instance, Power BI can consume data from Azure Synapse Analytics, but other tools like Azure Databricks or Data Factory can also be used for data consumption and processing. This integration streamlines data workflows, enabling organizations to leverage insights efficiently.
Prerequisites Required for Azure Synapse Dataverse Link Configuration
However, before configuring Azure Synapse Link for Dataverse, it's essential to meet certain prerequisites to ensure seamless and efficient data integration and processing. These prerequisites include:
- Azure Data Lake Storage Gen2:
- Have an Azure Data Lake Storage Gen2 account.
- Obtain Owner and Storage Blob Data Contributor role access.
- Enable the Hierarchical namespace feature.
- Azure Synapse Analytics:
- Set up a Synapse Workspace.
- Obtain Synapse Administrator role access within Synapse Studio.
- Provision the Synapse workspace in the same region and resource group as your Azure Data Lake Storage Gen2 account.
- Dataverse:
- Create the Azure Data Lake Storage Gen2 account and Synapse workspace in the same Microsoft tenant as your Power Apps tenant.
- Ensure the Azure Data Lake Storage Gen2 account and Synapse workspace are in the same region as your Power Apps environment.
- Obtain the Dataverse System Administrator security role.
Ensuring you meet these prerequisites before configuring Azure Synapse Dataverse Link will streamline the integration process.
The Deprecation of Export to Data Lake: Two Upgrade Paths
It is important to note that the "Export to Data Lake" feature exclusive to Finance and Operations was deprecated - effective October 15th, 2023. Users have until November 1st, 2024, to transition away from it. If you're currently relying on this feature, it's important to determine an upgrade path. Here are the two paths ensuring continuity and enhanced capabilities for your business intelligence processes that align with Microsoft's data and analytics strategies:
Azure Synapse Link for Dataverse
If you are upgrading from BYOD or Export to Data Lake, Microsoft recommends upgrading to this solution to ensure more effective data integration and collaboration across Microsoft applications. It is similar to how Export to Data Lake operates.
Microsoft Fabric
If you're at the beginning stages of your business intelligence journey, Microsoft recommends exploring Microsoft Fabric. This data analytics platform is designed to cover every aspect of analytics, from data movement to data science, real-time analytics, and business intelligence.
Microsoft is transitioning towards unified solutions, and Fabric embodies this shift by offering a Lake-centric and open approach.
Transitioning from “Export to Data Lake” to “Synapse Link for Dataverse”
Let's take a closer look at transitioning from the Export to Data Lake feature to Synapse Link for Dataverse.
The Synapse Link for Dataverse expands integration capabilities by incorporating Dynamics 365 Finance and Operations data. In the past it was exclusive to Dynamics 365 Customer Engagement. This integration combines Finance and Operations and Customer Engagement data within Synapse, replacing the isolated Export to Data Lake feature exclusive to Finance and Operations.
The unified experience of integrating Dynamics data through Synapse Link for Dataverse uses the common data model to store data from various Dynamics applications. Unlike the previous CSV format used with Export to Data Lake, Synapse Link leverages PySpark and the efficient Parquet file format, enabling efficient storage and supporting all CRUD transactions seamlessly. Additionally, Synapse Link offers continuous replication of both standard and custom tables within the database.
Exploring Key "Synapse Link For Dataverse" Capabilities
Azure Synapse Link for Dataverse unlocks a range of powerful methods tailored for integrating and analyzing Finance and Operations. Here's a breakdown of its primary features:
- Flexible Entity and Table Selection: Choose from a comprehensive selection of both standard and custom Finance and Operations entities and tables to include in your data integration and analytics workflows.
- Continuous Data Replication: Enjoy continuous replication of entity and table data, supporting Create, Update, and Delete (CUD) transactions to ensure data accuracy and timeliness.
- Streamlined Environment Linkage: Link or unlink your environment to Azure Synapse Analytics and/or Data Lake Storage Gen2 directly within your Azure subscription.
- Integrated Data Exploration: Explore and analyze selected data using Azure Synapse without the need for external configuration tools.
- Comprehensive Feature Support: Benefit from full support for all Azure Synapse Link for Dataverse features, including:
- Availability in all regions
- Saving data as Parquet Delta files
- Compatibility with restricted storage accounts.
- Enhanced Table Limits: Unlike the Export to Data Lake service, Azure Synapse Link for Dataverse offers expanded table limits, providing greater flexibility and scalability.
- Optimized Data Storage: Data from Finance and Operations is automatically saved in Parquet Delta Lake format, ensuring faster query response times and improved performance for analytics tasks.
These capabilities empower organizations to leverage Finance and Operations data efficiently within Azure Synapse, driving enhanced insights and operational efficiency.
Comparing Export to Data Lake with Synapse Link: A Closer Look
Here is a quick side-by-side comparison of Export to Data Lake vs. Synapse Link Export to your Data Lake (BYOL): .
One notable difference highlighted is the management complexity. With Export to Data Lake, managing resources often involves additional tools like Key Vaults. Conversely, Synapse Link simplifies that process by eliminating the need for these resources. Both solutions offer similar data pipeline capabilities, enabling near real-time data flow within a 15-minute interval after enabling table changes. However, it's important to note that the initial table setup time with Synapse Link may vary based on workload demands. This comparison underscores the benefits and efficiencies gained by transitioning to Synapse Link.
Important Custom Tables Requirement in Synapse Link for Dataverse
In the latest update to Synapse Link for Dataverse (version 10.0.38), Microsoft has expanded the availability of change tracking to include 2750 base tables by default. However, for custom tables, ISV tables, data entities, and any base tables where the property is not enabled, you must set it to “yes”. This small property change is required to ensure these tables appear in Synapse Link. This adjustment allows organizations to integrate their custom and specific data tables into the Synapse Link environment.
Access Near Real-time Data and Snapshot Data
After configuring Azure Synapse Link, two versions of table data are synchronized within Azure Synapse Analytics and/or Azure Data Lake Storage Gen2:
- Near Real-Time Data: This feature efficiently syncs data from Dataverse via Azure Synapse Link by detecting changes since initial extraction or the last synchronization.
- Snapshot Data: Snapshot data offers a read-only copy of near real-time data updated at regular intervals, typically every hour, ensuring users have a consistent and controlled view for reporting and analytical purposes.
In our Azure Synapse workspace, synchronized table data exists in two versions: the actual table and a partitioned version named "edit table." It's crucial to note that the actual table (without the underscore partitioned) represents near real-time data, capturing changes flowing from Finance and Operations. The partitioned table reflects snapshots, determined by the chosen partition in Synapse Link for Dataverse before table replication activation.
Property Change on Tables:
- Developers can enable change tracking on base tables through an extension.
- Custom tables can have this change enabled via customization.
- In later application versions, this property is enabled by default, though older custom tables may require manual adjustment.
This property change introduces a field called "FISroversion" to the table alongside the Rec ID. Synapse Link for Dataverse utilizes these fields to track changes upon table activation in the Maker's portal, facilitating automatic index creation without the need for manual setup in Visual Studio, unlike Export to Data Lake. This streamlines change tracking and synchronization processes effectively.
Why transition from Export to Data Lake?
Making the switch from Export to Data Lake to Synapse Link for Dataverse offers a range of compelling benefits. The below-highlighted enhancements offer improvements over the limitations experienced with Export to Data Lake and can improve your organization's overall data management, including:
- Unified Data Access: Synapse Link for Dataverse provides a simplified user interface that consolidates data from all Dynamics applications, removing the isolation seen with Export to Data Lake.
- Simplified Configuration: Setting up Synapse Link requires fewer objects compared to Export to Data Lake, streamlining the configuration process.
- Parquet File Format: With Synapse Link, the ability to save files in Parquet format leads to faster query times, and smaller file sizes, and resolves update conflicts that were common with CSV files in Export to Data Lake.
- Enhanced Security: Synapse Link allows for the setup of firewall storage accounts, improving data security.
- Simplified Entity Materialization: Materializing Finance and Operations entities into the Synapse workspace is more straightforward, eliminating the need for complex entity recreations and logic transfers.
- Elimination of Table Limitations: Synapse Link removes table limitations, requiring only the enabling of a specific property to include desired tables.
- Global Availability and Field Support: Synapse Link is available in all regions and supports memo and large string fields, addressing previous limitations encountered with Export to Data Lake.
- Improved Data Handling: Enum values from Finance and Operations are now translated into text and integer formats within Synapse Link, resolving issues experienced with Export to Data Lake.
Transitioning from Export to Data Lake to Synapse Link for Dataverse presents a strategic opportunity for organizations to enhance their data management and analytics capabilities. By leveraging Synapse Link for Dataverse, businesses can consolidate data across Dynamics applications, streamline configuration processes, and benefit from improved data handling, security, and accessibility.
The shift not only addresses limitations associated with Export to Data Lake but also unlocks a range of features that empower organizations to derive valuable insights and make informed decisions.
Reach Out to Stoneridge Software For Help With the Transition
If you're currently using Export to Data Lake and looking to transition to Azure Synapse Link for Dataverse or Fabric, please reach out to us at Stoneridge Software. Our team is here to support you in implementing this new solution. Alternatively, if you're just beginning your business intelligence journey, we're eager to assist you in getting started.
Under the terms of this license, you are authorized to share and redistribute the content across various mediums, subject to adherence to the specified conditions: you must provide proper attribution to Stoneridge as the original creator in a manner that does not imply their endorsement of your use, the material is to be utilized solely for non-commercial purposes, and alterations, modifications, or derivative works based on the original material are strictly prohibited.
Responsibility rests with the licensee to ensure that their use of the material does not violate any other rights.