How to Prepare for the Apache Spark Update

By Kristofer Schlieper | January 24, 2025

For clients using Dynamics 365 Finance and Operations connected to Azure Synapse Link for Dataverse, you will soon have to upgrade to Apache Spark 3.4.

Microsoft is decommissioning Apache Spark 3.3, meaning that an upgrade is imminent. While the Power Apps maker portal will tell you when it’s time to update, there are still some steps you must follow to properly set up the new Spark Pool in Azure Synapse and how to update your Azure Synapse link to reflect the new Spark Pool.

You must do this upgrade any time before Microsoft retires Apache Spark Pool 3.3.

The Steps to Follow to Complete the Upgrade

There are three step-by-step processes to follow to upgrade your Apache Spark Pool when the time comes.

1. Identifying When to Upgrade Apache Spark Version

  1. You’ll know when to upgrade to the new Apache Spark version when you receive the notification in the Tables view of Azure Synapse Link.
  2. Log in to the Power Apps maker portal.
  3. Select your Environment from the Environment menu connected to Finance and Supply Chain Management and Synapse.
    Apache Spark update selecting an environment
  4. Go to the Azure Synapse Link.
    Power Apps Azure Synapse Link
  5. Find the Link you want to use, click on the 3 dots (…) to the right of the name, and from the drop-down menu select Tables.
    Apache Spark update select tables
  6. When that opens, wait and you’ll get an informational bar in yellow that will tell you your current version of Spark Pool is being retired and needs to be upgraded by a certain time.
    Information bar need to update apache spark
  7. Next, click on the Details tab at the top and look for the “Azure Synapse Analytics Workspace” to tell you what workspace where you’ll need to create the new Spark Pool.
    Azure synapse link workspace

2. Creating the New Apache Spark Pool

  1. Open a new tab on your browser and navigate to the Azure Portal.
  2. Click on Azure Synapse Analytics – if you don’t see it in the list click on More Services and find it in that list.
    More services
  3. Select the Synapse Analytics space that handles your Finance and Operations and Dataverse connection.
    Azure synapse link F&O Dataverse connection
  4. Once you are in the Analytics space, expand the Analytics pool section and select Apache Spark pools. You’ll see your current spark pool and click on the name.
    Spark pool name
  5. In the spark pool, select properties and take notes of the “Node size,” “Number of Nodes,” and “Automatic Pausing.” Your new spark pool must have at least these settings (particularly the Node size and Number of Nodes). You can add more nodes or switch to a better size, but you cannot go less (at least during this upgrade to the new version).
    Apache spark update nodes
  6. Go to the Apache Spark pools page and click “+New”.
    Creating new apache spark pool
  7. Provide a name for the new spark pool, choose the Node size, and Number of Nodes, and then click “Next: Additional Settings.”
    apache spark update node additional settings
  8. On the Additional settings page set the Number of minutes idle and then choose the Apache Spark version (currently it’s 3.4) and then click “Review + Create.”
    Apache spark update review and create new spark pool
  9. On the “Review + Create” screen, validate your selections and click “Create”.
    Finalizing creation of new apache spark pool
  10. When the deployment finishes, you will be notified.
    Notification of new spark pool created

3. Switch to the New Spark Pool

  1. Go back to the Power Apps maker portal.
  2. Select your Environment from the Environment menu connected to Finance and Operations and Synapse (it should be the same environment as in the first section.)
    Select environment connected to Finance and Operations and synapse
  3. Go to the Azure Synapse Link.
    Go to azure synapse link
  4. Find the Link you want to use and click on the three dots (…) to the right of the name. From the drop-down menu, select “Tables”.
    Select tables
  5. When that opens again, you’ll wait and get the same informational bar in yellow as before. That will tell you your current version of Spark Pool is being retired and needs to be upgraded by a certain date.
    Yellow bar again
  6. In the top menu, click “Upgrade to Apache Spark X.X with Delta Lake X.X."
    Upgrade to apache spark 3.4 with delta lake 2.4
  7. It’ll take a few seconds for the upgrade window to appear. Once it does, select the drop-down menu and select the Spark Pool you created in the second section. You should see the Spark Pool name in the drop-down. After selecting it, click “Update.”
    Select spark pool you created Apache Spark update
  8. After it updates to the new Spark Pool version, you’ll no longer see the Yellow Message or the Upgrade Spark Pool option in the top menu.
    Refresh
  9. Next time the Dataverse connects to Synapse, it will use the new Spark Pool. After being idle for whatever time you set after setting up the original spark pool, your old spark pool should become inactive. Eventually, you will be able to delete it.

Get in touch with Stoneridge Today to Learn More About This Process

Our experts always ensure that your software solutions are updated and that you are always operating the latest version. If you need help with this process or need assistance optimizing other areas of your Dynamics 365 Finance and Operations environment, contact our team today.

Related Posts


Under the terms of this license, you are authorized to share and redistribute the content across various mediums, subject to adherence to the specified conditions: you must provide proper attribution to Stoneridge as the original creator in a manner that does not imply their endorsement of your use, the material is to be utilized solely for non-commercial purposes, and alterations, modifications, or derivative works based on the original material are strictly prohibited.

Responsibility rests with the licensee to ensure that their use of the material does not violate any other rights.

Start the Conversation

It’s our mission to help clients win. We’d love to talk to you about the right business solutions to help you achieve your goals.

Subscribe To Our Blog

Sign up to get periodic updates on the latest posts.

Thank you for subscribing!