We considered a project which was reported to contain leakage bugs and checked for the following,

• Whether any functional requirement was missed out by us during testing.
• Whether any functional requirements were missed while creating the test cases.
•  Nature of the leakage bugs.

No issues were found in the first two cases which lead to the conclusion that leakage bugs were creeping into the application because of the nature of the bugs itself and the possibility of finding these types of bugs in test case driven approach is very less.

Let’s see an example: Consider a functional requirement “User should be able to rename the file”, for this we might have a test case with the objective “Verify that users are allowed to rename the file”. In the real world every day, the test manager or leader will allocate 30-40 test cases to a tester for execution.  Assume this test case is one among the allocated list. Now, the tester will take the file, say “File.txt” and will try to rename it as “filerenamed.txt”. In most of the cases, the tester will mark the test case as pass.

Let see what are the possible leakage bugs and other test scenarios here

• System might crash if the tester uses some special character while renaming the file.
• System might crash if the tester tried to rename the file by putting some large name.
• System might crash if the tester tried to rename the file by putting an already existing name.
• Whether the user will be able to open the file after renaming it.
• What happens if the user changes the file format while renaming?
• If the system displayed a warning message for special characters or for an already existing name, then by clicking the ‘Ok’ button in the warning pop up whether the tester is able to rename the file by putting a valid name.
• Whether the tester checked all the possible special characters or not.
• What happens if the tester renamed the file with a blank space?
• Whether the tester is able to rename the file by paste functionality.

This list will grow depending on the testing scenario. A tester who is not very experienced will simply run the test case with a valid filename. Even an experienced tester might execute just 2 or 3 other scenarios as 39 other test cases needs to be executed the same day.

As a remedy for this issue our team introduced Exploratory testing approach to our testing process.

So let’s see what Exploratory testing is,

• ET is any testing to the extent that the tester actively controls the design of the tests as those tests are performed and uses information gained while testing to design new and better tests      –   James Bach
  

• ET Allows  the personal freedom and responsibility of the individual tester to continually optimize the value of his/her work by treating  the test-related learning, test design, test execution, and test result interpretation  – Cem Kaner
  

• ET is not a technique; we can say it is a testing approach or a testing style.

• ET doesn’t use any comprehensive documents like test case document.

• ET completely relies on the capability of the testers.

• ET can provide a good feedback of any build or any application in a minimal span of time.

• ET gives enough opportunities to testers for conducting good investigation, reasoning, thinking, observing and doing creative things while testing.

One way of tackling the leakage bugs could be to pen down each of these different possibilities as separate test cases and execute it in the regular way. But, this would create quite a bit of an overhead.  In a way, ET can be looked upon as an alternate way of covering these scenarios in a cheaper manner banking on the experience, system knowledge and imagination of the person who is executing the test cases as opposed to the assured coverage from the orthodox test case based approach.

How it is conducted.  

 ET approach will be conducted in sessions and the first step is to decide on a charter, where session is the basic testing work unit. By “charter”, we mean that each session is associated with a mission which is similar to the test objective in test case driven approach, also it will tell what the tester is going to do in that session. In the above example we can use the same test case objective  “Verify that users are allowed to rename the file” as charter. The charter may be chosen by the tester himself, or assigned by the test lead or test manager. Sometimes charters are written down or else the functional test cases are used as charters. Once the session is started, the testers will concentrate on the charter and will try to explore all possible test scenarios of the charter. There won’t be any test case development phase hence the tester can utilize the entire time to explore the hidden and dangerous bugs. The only possible output from a session of ET is a set of bug reports.

It’s no surprise if doubts are raised about relying completely on a tester.  Or how the managers or leads manage their team? Or how the work is accountable? Thus the testing master James Bach and his team came up with technique called Session Based Test Management (SBTM) which can be used to manage ET in a structured way.

Both ET and Test case development approach has its own benefits. So we can’t limit our testing process to any one of these approaches. It is better to have a right mix of both approaches for ensuring the quality of any application and we always offer both approaches to our clients.

Some more related topics are on the way like ET with SBTM, Skill set of ET tester …etc. Please stay connected. 

In such a scenario, the onus is on the testing process to ensure that the functionality additions of the current release has not regressed the existing functionalities. When frequent releases are made, it is not enough that we test the requirements pertaining to that single release, but it should be ensured that the application on a whole is stable and that any changes has not lead to subsequent errors somewhere else in the system. Simply put, regression testing means re-testing the application as and when required.

Why is it important?

Making changes to an already tested application can lead to the following issues,

• Any new change in functionality may break the current functionality.
• Changes at one part of the application could affect the system elsewhere.
• Fixing a bug may have resulted in other unforeseen bugs.

All these create a serious impact on the quality of the product. Therefore by doing proper regression testing at every phase of development, we can ensure true progress of the product. However, the challenge to regression testing is the increasing overheads. In an utopian world of unlimited resources, we would want to do thorough testing of the entire application for every miniscule change done.  Thus a huge system will carry a large overhead of regression testing along its maintenance lifecycle. We should recognize this and have a conscious plan to tackle this issue.

Regression testing Process:

Now that we can’t afford to do all the testing each time the application is touched, we need to have scientific ways of ensuring maximum returns from the available testing effort for the release. There are mainly two parts to this process – i) designing the regression test suite for the release and ii) executing it.

Designing regression test suite

Typically, the regression test suite would consist of a representative selection of the test cases for the system. This representative set of test cases needs to be skewed towards the areas of the system that are more affected by the changes in the release. Modularization of test cases helps a lot in designing the regression test suite.  If we have maintained a dependency matrix between the different modules of the application, one could identify the modules (at least the ones which we need to concentrate on) that need representation in the regression test suite, based on the modules to which the changes of the release belong. 

The test case documentation should carry information as to whether it is marked as a representative test case of the module it belongs to. This would help in picking up the test cases that represent the modules affected. Apart from the test cases that represent the functionality within the modules there needs to be ones which represent inter-modular behavior.  These need to be added separately based on the affected modules. In addition to the test cases representing the affected modules and their inter-modular behavior, there needs to be representation across the other modules also. We could maintain a base regression set which cuts across modules and ensures sanity of the entire application and ensure that all regression test suites would contain this.

Execution of regression test suite

Having designed the regression test suite,  we need to execute the same. Automation is an option which needs to be seriously considered for elaborate / frequent regression testing scenarios as we could get considerable return on the investment we do for the recording of the scripts.

We will look at automation for web application regression testing in an upcoming post. Do stay connected…

• Critical Error
• Lack of Functionality / Logical Error
• Not Living up to Specification
• Spelling or Grammar Error
• Design Problem

• SQL injection
• Bot login prevention
• JavaScript loopholes
• Password protection
• Document privacy
• Reader privacy
• IP restrictions
• Credit card encryption

• W3C standards
• Web accessibility
• Section 508 compliance
• AAA compliance
• XHTML compliance

• Internet Explorer 6
• Internet Explorer 7
• IE 8
• Firefox 1.5 browser
• Safari browser for Mac and Windows XP
• IE 5.2 for Macintosh
• Netscape browser 7.x
• Opera 8.x
• Android and iPhone tablet browsers
  

• Portal development standards
• Web services testing
• Web load performance testing
• Automated web testing
• Web site optimization
• Website coder matrices

• Functional specification of the project
• Client / Admin login access on staging server

• Pay per bug
• Fixed hour testing
• Agile testing for large projects

Write to us to know about skill sets and the services that we can provide to you.