In order to perform a correct regression test you need first understand what it means to test in this way. Regression testing is the practice of verifying correctness based off previously passed tests. Meaning, if it worked before, it should work now.
The types of tests that are usually included in this suite are pretty much all the tests that have been written. For a large software suite, this is usually handed off to an automated system that performs the tests on a schedule and returns the results. For the purpose of this post, I’m going to focus on what you should look for.
Quick Examples
If you’re working in an application that requires the user to authenticate themselves, then one of the simplest actions that user must take is to sign in or log in (terminology varies on application). This test should be run constantly, and usually is with most software applications where there is active development and testing going on. There is an obvious problem if this feature is broken.
Speaking of secured applications, another simple test to run is to verify that each page that is required to be protected, actually is protected, and not accessible by non-authenticated users. This is another favorite addition to an automated system being that it requires the constant checking of each page within the application.
What to add to the test plan
The instant reaction to me would be to add every test that you can think of on every page. While this is a really good idea and most likely will produce the highest number of results each time the suite is run, the amount of maintenance to keep each of the tests current to new changes and new expected behaviors would take a great deal of time.
Instead, you should look at adding a larger number of major actions to the test plan, and some important minor ones. This will keep the maintenance time down and still produce good results. Any specific features that have been added to your application since the last release should also have their own new set of test cases added to the regression plan and be included in regular maintenance for future releases.
One other important list of tests to add were bugs submitted from the last release. These bugs should have steps already written to reproduce a bug, and you want to be sure that you don’t keep fixing the same bug.
Verifying Custom Code
This section is specific to applications that are rolled out to multiple clients, each of which usually have their own requests for functionality. Obviously with the ability to have custom modifications in the same area of the application where on other instances the functionality works differently, having a test instance that includes all these custom mods is not a good idea.
This creates the need for multiple instances to be tested against. One for each of the custom modifications that are done for clients. Of course this assumes that you will be upgrading the client instances along with regular upgrades. if not, then this section is really not necessary. However keep in mind that if you have a client on an older version, especially a very old version, you are going to need to upgrade them at some point. The easier option is to keep them up to date. Otherwise you’ll run into a number of issues with the custom code a client originally had designed, and changes to the application around that code can cause it to no longer function, or even conflict with current code.
Database Testing
More than likely, your application requires the use of a database of some form. One of the most common ways that applications fail when released to the public is when more users than when tested try to access the appication at the same time, or when a table gets larger and accessed more often than during initial testing. The right amount and type of testing is essential when regression testing as well. Changes to functionality can cause querys to increase or change into something that didn’t happen before. You need to run the query’s that your application is running and verify that the result time is the same (or faster) than the previous release.
Time Tracking
If your application lives on the web, page load times are essential to any testing plan. Correct usage of caching and database indexing create a faster response from your application to the end-user. Make sure to track the amount of time it takes for a page to load. This can easily be done through automated testing. The most important part of this measurement is the time it takes your application to deliver a response to the browser to build the page. YSlow and Firebug can do this measurement for you. Firebug’s Net tab will show you each of the components and their time to download. The first one in that list is the time it took for your application to get a request, process it, and spit the results back out to the browser.
Frequency of tests
In a perfect word this would be run every time there is a commit, but that’s unrealistic. A dependancy to this is also mode of testing. If you are running an automated suite this could potentially be done every night with a checkout (and compile if necessary) of the latest commited code. If you don’t have automation setup you should regression test with every release using release candidates.
——
Regression testing is a very important and key part of the software release cycle. Without it, all the new features in an application would work, but that would be meaningless if you broke the login page.
–dez