How to Resolve the “Maximum number of vCores exceeded” error in your Apache Spark Pool jobs
If you are using a Spark Pool to take advantage of the Delta/Parquet conversion with Azure Synapse Link, one error you might encounter is ‘Maximum workspace capacity exceeded’. If you are unfamiliar with the Delta/Parquet format, it has multiple benefits over CSV. Query times are faster and you can take advantage of compression.
In the Azure subscription where the Azure Synapse Link, Storage Account, Azure Synapse Workspace, and Spark Pool were created, a quota is sometimes defined at the Azure subscription level. This can impact your conversion and when it impacts your conversion, you might see the error above.
To monitor your Spark Pool, you go to Monitor > Apache Spark applications in your Synapse Analytics workspace.
This screen will provide you with a list of all Apache Spark Pool jobs executed. You will notice our jobs have DeltaLakeConversion in the application name. This screen will give you the Spark Pool name, the status, and other information.
Since the conversion job fails, you will not see the activated tables in the Synapse workspace or the storage account in the deltalake folder.
It is important to note conversions run in parallel for pending CSV table data. These conversions use the vCores available. When you set up your Spark Pool, you select your node size. You also select the number of nodes. The number of vCores is per node. Based on my settings, I can have 10 nodes, and using the Small node size of 4 vCores, I can process about 40 (10x4) tables in parallel.
In my Spark Pool settings, I have Autoscale and Dynamically allocate executors set to enabled. As demand comes in, the number of nodes and executors will increase.
To fix the error ‘Maximum workspace capacity exceeded’, I had to reduce the number of nodes in my Spark pool settings. My Spark pool conversions are now succeeding.
Under the terms of this license, you are authorized to share and redistribute the content across various mediums, subject to adherence to the specified conditions: you must provide proper attribution to Stoneridge as the original creator in a manner that does not imply their endorsement of your use, the material is to be utilized solely for non-commercial purposes, and alterations, modifications, or derivative works based on the original material are strictly prohibited.
Responsibility rests with the licensee to ensure that their use of the material does not violate any other rights.