How to Prepare for the Apache Spark Update
For clients using Dynamics 365 Finance and Operations connected to Azure Synapse Link for Dataverse, you will soon have to upgrade to Apache Spark 3.4.
Microsoft is decommissioning Apache Spark 3.3, meaning that an upgrade is imminent. While the Power Apps maker portal will tell you when it’s time to update, there are still some steps you must follow to properly set up the new Spark Pool in Azure Synapse and how to update your Azure Synapse link to reflect the new Spark Pool.
You must do this upgrade any time before Microsoft retires Apache Spark Pool 3.3.
The Steps to Follow to Complete the Upgrade
There are three step-by-step processes to follow to upgrade your Apache Spark Pool when the time comes.
1. Identifying When to Upgrade Apache Spark Version
- You’ll know when to upgrade to the new Apache Spark version when you receive the notification in the Tables view of Azure Synapse Link.
- Log in to the Power Apps maker portal.
- Select your Environment from the Environment menu connected to Finance and Supply Chain Management and Synapse.
- Go to the Azure Synapse Link.
- Find the Link you want to use, click on the 3 dots (…) to the right of the name, and from the drop-down menu select Tables.
- When that opens, wait and you’ll get an informational bar in yellow that will tell you your current version of Spark Pool is being retired and needs to be upgraded by a certain time.
- Next, click on the Details tab at the top and look for the “Azure Synapse Analytics Workspace” to tell you what workspace where you’ll need to create the new Spark Pool.
2. Creating the New Apache Spark Pool
- Open a new tab on your browser and navigate to the Azure Portal.
- Click on Azure Synapse Analytics – if you don’t see it in the list click on More Services and find it in that list.
- Select the Synapse Analytics space that handles your Finance and Operations and Dataverse connection.
- Once you are in the Analytics space, expand the Analytics pool section and select Apache Spark pools. You’ll see your current spark pool and click on the name.
- In the spark pool, select properties and take notes of the “Node size,” “Number of Nodes,” and “Automatic Pausing.” Your new spark pool must have at least these settings (particularly the Node size and Number of Nodes). You can add more nodes or switch to a better size, but you cannot go less (at least during this upgrade to the new version).
- Go to the Apache Spark pools page and click “+New”.
- Provide a name for the new spark pool, choose the Node size, and Number of Nodes, and then click “Next: Additional Settings.”
- On the Additional settings page set the Number of minutes idle and then choose the Apache Spark version (currently it’s 3.4) and then click “Review + Create.”
- On the “Review + Create” screen, validate your selections and click “Create”.
- When the deployment finishes, you will be notified.
3. Switch to the New Spark Pool
- Go back to the Power Apps maker portal.
- Select your Environment from the Environment menu connected to Finance and Operations and Synapse (it should be the same environment as in the first section.)
- Go to the Azure Synapse Link.
- Find the Link you want to use and click on the three dots (…) to the right of the name. From the drop-down menu, select “Tables”.
- When that opens again, you’ll wait and get the same informational bar in yellow as before. That will tell you your current version of Spark Pool is being retired and needs to be upgraded by a certain date.
- In the top menu, click “Upgrade to Apache Spark X.X with Delta Lake X.X."
- It’ll take a few seconds for the upgrade window to appear. Once it does, select the drop-down menu and select the Spark Pool you created in the second section. You should see the Spark Pool name in the drop-down. After selecting it, click “Update.”
- After it updates to the new Spark Pool version, you’ll no longer see the Yellow Message or the Upgrade Spark Pool option in the top menu.
- Next time the Dataverse connects to Synapse, it will use the new Spark Pool. After being idle for whatever time you set after setting up the original spark pool, your old spark pool should become inactive. Eventually, you will be able to delete it.
Get in touch with Stoneridge Today to Learn More About This Process
Our experts always ensure that your software solutions are updated and that you are always operating the latest version. If you need help with this process or need assistance optimizing other areas of your Dynamics 365 Finance and Operations environment, contact our team today.
Under the terms of this license, you are authorized to share and redistribute the content across various mediums, subject to adherence to the specified conditions: you must provide proper attribution to Stoneridge as the original creator in a manner that does not imply their endorsement of your use, the material is to be utilized solely for non-commercial purposes, and alterations, modifications, or derivative works based on the original material are strictly prohibited.
Responsibility rests with the licensee to ensure that their use of the material does not violate any other rights.