Developing Regression Testing Strategy in D365 Finance and Supply Chain
Regression testing encompasses activities aimed at ensuring application updates do not cause existing functionality to be compromised. A regression has occurred when new code interferes with or breaks existing code. An example of this might be the addition of a required field on a screen causing downstream processes (where this field was not previously an input) to stop working.
Regressions can be introduced by ongoing enhancements you are making to your application, ISV updates or by Microsoft OneVersion updates. Microsoft regression tests their updates before they are released, so we usually do not have to worry about testing every new feature they release. In the case of One Version updates, the areas most prone to regressions include customizations and integrations, but you may also choose to regression test areas of medium-high importance to your business even if they are not customized. ISV’s are responsible for ensuring their code is stable with base D365, but if you have made changes that affect the same code areas then this will need regression testing in your system.
In earlier versions of AX2012 or AX2009, updates were mostly limited to hotfixes and occasionally new releases about every 2 years. New releases included numerous new features that required intensive resources to migrate and test and in some cases was treated as a re-implementation rather than an upgrade.
With the advent of Dynamics 365, Microsoft has increased the update cadence to up to 8 per year, of which you only can skip up to 3. This rapid update cadence means we get to enjoy new features more often and limits the number of previous versions that Microsoft must support, freeing up resources to build additional features. The frequent release cadence means that testing must happen more frequently, thus, having a regression testing plan has become more important than ever.
A note on regressions due to enhancements
This document is primarily focused on developing a testing strategy to mitigate risk seen during OneVersion updates. It is assumed that any custom development will be properly evaluated for testing requirements during the design process and that these tests will be executed as part of the release process of that particular effort.
Laying out the testing strategy
The first step to creating regression tests is to identify your testing strategy. This includes identifying and prioritizing critical areas of the application that should be tested every time a code update is applied (either from an ISV or Microsoft OneVersion updates). It is not recommended to create regression tests that cover the entire system; creating and maintaining this number of tests would become burdensome over time, especially as new features are added.
As you identify these areas, categorize and prioritize them. Categorization allows you to narrow down tests that need to be performed based on which areas of the application are changed. For example, if a change is made to the sales order screen you can easily find all test cases that relate to the sales and marketing module.
Prioritization has two functions. One, during test writing it focusing efforts on the more critical tests first, which results in more thorough tests. Second, if there are many tests to execute it could take hours or days for testing to complete. When tests are executed in priority order, issues relating to more important areas are revealed earlier in the process, thus allowing you more time to respond to them.
The areas that we focus on testing are those where there is a high risk of regression or where the consequences of failure are medium to high. Areas that have a high risk of regression include integrations and complex or key customizations. For risk analysis, analyze each area of the application and ask yourself these questions:
- What could a change to this area have an impact on?
- How important is a fault in the impacted area? Some examples are:
- Does this area experience a high volume of transactions?
- Would sensitive or important information be compromised, particularly if an error would be hard to detect or correct in the production environment (e.g. where a month end process would be the first opportunity to catch an error).
- Is it governed by regulatory requirements where a regression could result in fines or penalties?
- Should we test only what has been affected, if so, how much? For example, if the sales order form is changed, should test be executed from sales order creation all the way through invoicing and payment?
- Only the affected area?
- Other areas affected?
- Whole system?
In order to perform this analysis, you need a team of users who are proficient in both the business and the application. After performing this analysis, you may find there are parts of the system where it is not possible to regression test. For example, integrations with an ISV that does not have a test environment. Carefully consider the risk involved and develop a mitigation strategy should problems arise.
Also, ensure that this testing strategy is updated as new customizations are developed.
Develop the tests
Writing regression tests can be done similarly to how UAT tests were written, in fact in many cases the tests can be copied from UAT tests. However, because of the short time frame between deployment from Test to Production, and the reduced testing staff versus during implementation, take care to ensure all outlying scenarios and expected results are properly identified. Writing good software tests is a subject unto its own which we’ll cover later.
Ideally, these tests will be created and tracked in Azure DevOps, however there are many platforms which support software testing.
If automated testing is to be employed, then certain additional techniques must be employed to ensure successful execution which will be covered later this blog series.
Execute the tests
Microsoft has release notes that can be accessed here. These notes detail all the feature and platform changes taking place in each release for the next 2 or 3 updates. Read through this and thoroughly assess whether these changes affect any of the previously identified risk areas.
If you are using DevOps as your test management tool, you can add these tests to a new test plan as soon as the changed features are announced and assign them to the testers. Then when the update is released testing can commence immediately. Test results should be recorded in DevOps and bugs created when failures occur. If any tests fail, your support team should be notified immediately so the update can be paused until a resolution can be found.
D365 Finance & Supply Chain Regression Testing Conclusion
Microsoft Dynamics 365 gives you many unique advantages: the ability to work in the cloud from anywhere, less maintenance of hardware and more frequent feature releases. With the more frequent update cadence, the importance of having a regression testing strategy cannot be overstated. Getting prepared for the inevitable updates should be high on your priority list during implementation to ensure the continuity of your business processes as you receive new features.
Under the terms of this license, you are authorized to share and redistribute the content across various mediums, subject to adherence to the specified conditions: you must provide proper attribution to Stoneridge as the original creator in a manner that does not imply their endorsement of your use, the material is to be utilized solely for non-commercial purposes, and alterations, modifications, or derivative works based on the original material are strictly prohibited.
Responsibility rests with the licensee to ensure that their use of the material does not violate any other rights.