Tuesday, December 20, 2011

Top 20 practical software testing tips you should read before testing any application.


I wish all testers read these software testing good practices.Read all points carefully and try to implement them in your day-to-day testing activities. This is what I expect from this article. If you don’t understand any testing practice, ask for more clarification in comments below. After all you will learn all these testing practices by experience. But then why not to learn all these things before making any mistake?

Here are some of the best testing practices I learned by experience:
1) Learn to analyze your test results thoroughly. Do not ignore the test result. The final test result may be ‘pass’ or ‘fail’ but troubleshooting the root cause of ‘fail’ will lead you to the solution of the problem. Testers will be respected if they not only log the bugsbut also provide solutions.
2) Learn to maximize the test coverage every time you test any application. Though 100 percent test coverage might not be possible still you can always try to reach near it.
3) To ensure maximum test coverage break your application under test (AUT) into smaller functional modules. Write test cases on such individual unit modules. Also if possible break these modules into smaller parts.
E.g: Lets assume you have divided your website application in modules and ‘accepting user information’ is one of the modules. You can break this ‘User information’ screen into smaller parts for writing test cases: Parts like UI testing, security testing, functional testing of the ‘User information’ form etc. Apply all form field type and size tests, negative and validation tests on input fields and write all such test cases for maximum coverage.
4) While writing test cases, write test cases for intended functionality first i.e. for valid conditions according to requirements. Then write test cases for invalid conditions. This will cover expected as well unexpected behavior of application under test.
5) Think positive. Start testing the application by intend of finding bugs/errors. Don’t think beforehand that there will not be any bugs in the application. If you test the application by intention of finding bugs you will definitely succeed to find those subtle bugs also.
6) Write your test cases in requirement analysis and design phase itself. This way you can ensure all the requirements are testable.
7) Make your test cases available to developers prior to coding.Don’t keep your test cases with you waiting to get final application release for testing, thinking that you can log more bugs. Let developers analyze your test cases thoroughly to develop quality application. This will also save the re-work time.
8 ) If possible identify and group your test cases for regression testing. This will ensure quick and effective manual regression testing.
9) Applications requiring critical response time should be thoroughly tested for performance. Performance testing is the critical part of many applications. In manual testing this is mostly ignored part by testers due to lack of required performance testing large data volume. Find out ways to test your application for performance. If not possible to create test data manually then write some basic scripts to create test data for performance test or ask developers to write one for you.
10) Programmers should not test their own code. As discussed in our previous post, basic unit testing of developed application should be enough for developers to release the application for testers. But you (testers) should not force developers to release the product for testing. Let them take their own time. Everyone from lead to manger know when the module/update is released for testing and they can estimate the testing time accordingly. This is a typical situation in agile project environment.
11) Go beyond requirement testing. Test application for what it is not supposed to do.
12) While doing regression testing use previous bug graph (Bug graph – number of bugs found against time for different modules). This module-wise bug graph can be useful to predict the most probable bug part of the application.
13) Note down the new terms, concepts you learn while testing. Keep a text file open while testing an application. Note down the testing progress, observations in it. Use these notepad observations while preparing final test release report. This good habit will help you to provide the complete unambiguous test report and release details.
14) Many times testers or developers make changes in code base for application under test. This is required step in development or testing environment to avoid execution of live transaction processing like in banking projects. Note down all such code changes done for testing purpose and at the time of final release make sure you have removed all these changes from final client side deployment file resources.
15) Keep developers away from test environment. This is required step to detect any configuration changes missing in release or deployment document. Some times developers do some system or application configuration changes but forget to mention those in deployment steps. If developers don’t have access to testing environment they will not do any such changes accidentally on test environment and these missing things can be captured at the right place.
16) It’s a good practice to involve testers right from software requirement and design phase. These way testers can get knowledge of application dependability resulting in detailed test coverage. If you are not being asked to be part of this development cycle then make request to your lead or manager to involve your testing team in all decision making processes or meetings.
17) Testing teams should share best testing practices, experience with other teams in their organization.
18) Increase your conversation with developers to know more about the product. Whenever possible make face-to-face communication for resolving disputes quickly and to avoid any misunderstandings. But also when you understand the requirement or resolve any dispute – make sure to communicate the same over written communication ways like emails. Do not keep any thing verbal.
19) Don’t run out of time to do high priority testing tasks.Prioritize your testing work from high to low priority and plan your work accordingly. Analyze all associated risks to prioritize your work.
20) Write clear, descriptive, unambiguous bug report. Do not only provide the bug symptoms but also provide the effect of the bug and all possible solutions.
Don’t forget testing is a creative and challenging task. Finally it depends on your skill and experience, how you handle this challenge.
Over to you:
Sharing your own testing experience, tips or testing secrets in comments below will definitely make this article more interesting and helpful!!

Bug Reporting


Bug Reporting - Defect Profile Document


Following are the elements consisting in the DPD

1)Bug Title : Title of the bug

2)Bug Summary : Brief description of the deviated behavior

3)Severity,Priority: Seriousness and priority of the Bug

4)Rank: Depending upon the Severity and Priority Rank is given to the Bug to be fixed

5)Components: Occurring in component like UI etc

6)Reporter: Tester who finds and documents the Bug

7)Assignee: Developer to whom Bug is assigned

8)Fix versions: Expected to fix the Bug in Version or Build of the application

9)Build reported against: Current build number of the application on which
    Bug is identified

10)Attachment: Screenshot of the Bug

11)Description: Whole description of the Bug

12)Fix Date: End date

13)Comments: Comments by developer or Tester

Bug logging process - Jira tool

 Please go through the following screen shots step by step :

1.   

2.


3. 



ISTQB Sample Question and Answers- #1

Please scroll down to find the answers....

Q1. An input field takes the year of birth between 1900 and 2004. 
The boundary values for testing this field are:
a.) 0,1900,2004,2005
b.) 1900, 2004
c.) 1899,1900,2004,2005
d.) 1899, 1900, 1901,2003,2004,2005


Q2. Which one of the following are non-functional testing methods?
a. System testing
b. Usability testing
c. Performance testing
d. Both b & c



Q3. Which of the following tools would be involved in the automation of regression test?
a. Data tester
b. Boundary tester
c. Capture/Playback
d. Output comparator.

Q4. Incorrect form of Logic coverage is:
a. Statement Coverage
b. Pole Coverage
c. Condition Coverage
d. Path Coverage

Q5. Which of the following is not a quality characteristic listed in ISO 9126 Standard?
a. Functionality
b. Usability
c. Supportability
d. Maintainability

Q6. To test a function, the programmer has to write a _________, which calls the function to be tested and passes it test data.
a. Stub
b. Driver
c. Proxy
d. None of the above

Q7. Boundary value testing
a. Is the same as equivalence partitioning tests
b. Test boundary conditions on, below and above the edges of input and output equivalence classes
c. Tests combinations of input circumstances
d. Is used in white box testing strategy

Q8. Pick the best definition of quality
a. Quality is job one
b. Zero defects
c. Conformance to requirements
d. Work as designed

Q9. Fault Masking is
a. Error condition hiding another error condition
b. Creating a test case which does not reveal a fault
c. Masking a fault by developer
d. Masking a fault by a tester

Q10. One Key reason why developers have difficulty testing their own work is:
a. Lack of technical documentation
b. Lack of test tools on the market for developers
c. Lack of training
d. Lack of Objectivity

Q11. During the software development process, at what point can the test process start?
a. When the code is complete
b. When the design is complete
c. When the software requirements have been approved
d. When the first code module is ready for unit testing

Q12. In a review meeting a moderator is a person who
a. Takes minutes of the meeting
b. Mediates between people
c. Takes telephone calls
d. Writes the documents to be reviewed

Q13. Given the Following programIF X <>= ZTHEN Statement 2;ENDMcCabe’s Cyclomatic Complexity is :
a. 2
b. 3
c. 4
d. 5

Q14. How many test cases are necessary to cover all the possible sequences of statements (paths) for the following program fragment? Assume that the two conditions are independent of each other : -…………if (Condition 1)then statement 1else statement 2fiif (Condition 2)then statement 3fi…………
a. 2 Test Cases
b. 3 Test Cases
c. 4 Test Cases
d. Not achievable


15. Acceptance test cases are based on what?

a. Requirements
b. Design
c. Code
d. Decision table

Q16. How much testing is enough?
a. This question is impossible to answer
b. This question is easy to answer
c. The answer depends on the risk for your industry, contract and special requirements
d. This answer depends on the maturity of your developers

Q17. A common test technique during component test is
a. Statement and branch testing
b. Usability testing
c. Security testing
d. Performance testing

Q18. Statement Coverage will not check for the following:
a. Missing Statements
b. Unused Branches
c. Dead Code
d. Unused Statement

Q19. Independent Verification & Validation is
a. Done by the Developer
b. Done by the Test Engineers
c. Done By Management
d. Done by an Entity Outside the Project’s sphere of influence

Q20. Code Coverage is used as a measure of
a. Defects
b. Trends analysis
c. Test
d. Time Spent Testing

Answers:--
1. C
2. D
3. C
4. B
5. C
6. B
7. B
8. C
9. A
10. D
11. C
12. B
13. B
14. A
15. A
16. C
17. A
18. A
19. D
20. C

Effective way of reporting Bug- Defect report


According to the standard way of Defect profile document format:

Bug Title: 
Title should be short, precise and point the issue correctly.


Bug Description: 
Though the title reflects the issue in short, define the bug in few lines
which gives brief description


Steps to reproduce:
Here clearly mention the steps where you face the error and what are
the steps to be reproduced for it. Mention the steps right from invoking
a browser , entering application url to the place where issue arised.


Screen shots:
Screen shots play an important role for effective bug report.Try to attach
 the screen shots which make others to understand easily on looking it.
Sometime the issue may be over looked while reproducing, hence attach
the screen shots for necessary bugs.
Make an habit of it.

Details:
Some data like environmental details may also help developers to understand
what exactly is the cause by knowing the environment. Environment details
such as Operating System, Browser and its version etc.. Some browsers may
not support some features and some issues may also does not occur in some browsers.
Hence this information helps developers to resolve issue sooner.

Shot directly:
Never report against any single person directly as XXX person developed this .

Priority:
When bug should be fixed? Priority is generally set from P1 to P5. P1 as “fix the bug with highest priority” and P5 as ” Fix when time permits”.
Severity:
This describes the impact of the bug.
Types of Severity:
  • Blocker: No further testing work can be done.
  • Critical: Application crash, Loss of data.
  • Major: Major loss of function.
  • Minor: minor loss of function.
  • Trivial: Some UI enhancements.
  • Enhancement: Request for new feature or some enhancement in existing one.



Current Result: 
What is the actual result on running above steps i.e. the bug behavior


Expected:
How application should behave on above mentioned steps.



Environment

This helps to answer question that the assignee may have.

Product: In which product you found this bug.
Version: The product version if any.
Component: These are the major sub modules of the product.
Platform: Mention the hardware platform where you found this bug. The various platforms like ‘PC’, ‘MAC’, ‘HP’, ‘Sun’ etc.
Operating system: Mention all operating systems where you found the bug. Operating systems like Windows, Linux, Unix, SunOS, Mac OS. Mention the different OS versions also if applicable like Windows NT, Windows 2000, Windows XP etc.



Simple Language :
Use simple language while explaining about any issue which is understood by anyone

Setup framework :Selenium, Maven, TestNG, Java, Eclipse IDE


Selenium Framework(Selenium,Eclipse,Maven,TestNG, Java)

Resources/Checklist:
JDk
Eclipse IDE
Selenium server
Selenium-java-client driver
Starting and stoppig Selenium server
TestNG in Eclipse IDE
Maven in Eclipse IDE


1. JDK:
Download the latest version of JDK from Java official web site (www.oracle.com)
in your computer and set the system environmental variables accordingly.
select the JDK package with respect to your system OS.

2. Eclipse IDE:
Download the latest version of Eclipse IDE from eclise official website : 
 http://www.eclipse.org/downloads/

3. Selenium server and client driver:
Download the Selenium server and Selenium java client driver of latest version.
http://seleniumhq.org/download/

find the attached screen shot

4. Start and Stop server

After downloading the server unzip it.
Open the cmd prompt and point to the location where you have installed the server and type the command
java -jar selenum-standaloner-server.jar and press "Enter" button.

The server will start in the port 4444 by default where you can find the logs while running the tests.


5. TestNG and Maven integration for Eclipse:

GoTo Help>>Eclipse Market place>> search for TestNG
TestNG integration for eclipse will appear first in list.

Same process is followed for Maven plugin

Security Testing


Security Testing is a type of software testing in which testing 
is performed on an application to check if the security is maintained 
in such a way that the valid users are able to access, invalid users 
are unable to access and the vital information is protected from 
destructive agents like viruses and also protect from hackers.
This type of software testing can be done in many ways with
several objectives in many areas.

Some of them are given below.

Loging in to application:
Security testing is performed on login page to test valid user is able to
access and invalid user is unable to access.

Illegal access of web page:
Here, testing is performed to test the acess of the web page with URL
with out login and see the security is maintained.

Firewall:
Firewall is a means of security usually is established before the servers
where in vital information is stored. Security testing is performed to check if the firewall
is working as per the administrative setting to allow the desire requests
and not to allow the undesired requests. In other words it must have capability
to block destructive agents like viruses for the sake of protection.

Security should be maintained to avoid the following :
Checklist for security testing

A1: Injection
A2: Cross-Site Scripting (XSS)
A3: Broken Authentication and Session Management
A4: Insecure Direct Object References
A5: Cross-Site Request Forgery (CSRF)
A6: Security Misconfiguration
A7: Insecure Cryptographic Storage
A8: Failure to Restrict URL Access
A9: Insufficient Transport Layer Protection
A10: Unvalidated Redirects and Forwards


Security testing is must for Banking & financial domain projects.
This type of testing is optional for all other domains and will be opted by clients.

Test case:

Test Case is a commonly used term for a specific test. This is usually the smallest unit of testing. A Test Case will consist of information such as requirements testing, test steps, verification steps, prerequisites, outputs, test environment, etc.


A Test Case is:
- A set of inputs, execution preconditions, and expected outcomes developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement.

- A detailed procedure that fully tests a feature or an aspect of a feature. Whereas the test plan describes what to test, a test case describes how to perform a particular test. You need to develop a test case for each test listed in the test plan.



Test cases should be written by a team member who understands the function or technology being tested, and each test case should be submitted for peer review.

Organizations take a variety of approaches to documenting test cases; these range from developing detailed, recipe-like steps to writing general descriptions. In detailed test cases, the steps describe exactly how to perform the test. In descriptive test cases, the tester decides at the time of the test how to perform the test and what data to use.

Detailed test cases are recommended to test a software because determining pass or fail criteria is usually easier with this type of case. In addition, detailed test cases are reproducible and are easier to automate than descriptive test cases. This is particularly important if you plan to compare the results of tests over time, such as when you are optimizing configurations. Detailed test cases are more time-consuming to develop and maintain. On the other hand, test cases that are open to interpretation are not repeatable and can require debugging, consuming time that would be better spent on testing.

When planning your tests, remember that it is not feasible to test everything. Instead of trying to test every combination, prioritize your testing so that you perform the most important tests — those that focus on areas that present the greatest risk or have the greatest probability of occurring — first.

Once the Test Lead prepared the Test Plan, the role of individual testers will start from the preparation of Test Cases for each level in the Software Testing like Unit Testing, Integration Testing, System Testing and User Acceptance Testing and for each Module.

General Guidelines to Prepare Test Cases

As a tester, the best way to determine the compliance of the software to requirements is by designing effective test cases that provide a thorough test of a unit. Various test case design techniques enable the testers to develop effective test cases. Besides, implementing the design techniques, every tester needs to keep in mind general guidelines that will aid in test case design:

a.  The purpose of each test case is to run the test in the simplest way possible. [Suitable       techniques - Specification derived tests, Equivalence partitioning]
b.  Concentrate initially on positive testing i.e. the test case should show that the software does what it is intended to do. [Suitable techniques - Specification derived tests, Equivalence partitioning, State-transition testing]
c. Existing test cases should be enhanced and further test cases should be designed to show that the software does not do anything that it is not specified to do i.e. Negative Testing [Suitable techniques - Error guessing, Boundary value analysis, Internal boundary value testing, State transition testing]
d. Where appropriate, test cases should be designed to address issues such as performance, safety requirements and security requirements [Suitable techniques - Specification derived tests]
e. Further test cases can then be added to the unit test specification to achieve specific test coverage objectives. Once coverage tests have been designed, the test procedure can be developed and the tests executed [Suitable techniques - Branch testing, Condition testing, Data definition-use testing, State-transition testing]

Test Case Template

To prepare these Test Cases each organization uses their own standard template, an ideal template is providing below to prepare Test Cases.
Common Columns in Test cases that are present in all Test case formats

Fig 1: Common Columns in Test cases that are present in all Test case formats

Low Level Test Case format


Fig 2: A very details Low Level Test Case format

The Name of this Test Case Document itself follows some name convention like below so that by seeing the name we can identify the Project Name and Version Number and Date of Release.

DTC_Functionality Name_Project Name_Ver No 

DTC – Detailed Test Case
Functionality Name: For which the test cases is developed
Project Name: Name of the Project
Ver No: Version number of Software
(You can add Release Date also)

The bolded words should be replaced with the actual Project Name, Version Number and Release Date. For eg. Bugzilla Test Cases 1.2.0.3 01_12_04

On the Top-Left Corner we have company emblem and we will fill the details like Project ID, Project Name, Author of Test Cases, Version Number, Date of Creation and Date of Release in this Template.

And we will maintain the fields Test Case ID, Requirement Number, Version Number, Type of Test Case, Test Case Name, Action, Expected Result, Cycle#1, Cycle #2, Cycle#3, Cycle#4 for each Test Case. Again this Cycle is divided into Actual Result, Status, Bug ID and Remarks.

Test Case ID:

To Design the Test Case ID also we are following a standard: If a test case belongs to application not specifically related to a particular Module then we will start them as TC001, if we are expecting more than one expected result for the same test case then we will name it as TC001.1. If a test case is related to Module then we will name it as M01TC001, and if a module is having a sub-module then we name that as M01SM01TC001. So that we can easily identify to which Module and which sub-module it belongs to. And one more advantage of this convention is we can easily add new test cases without changing all Test Case Number so it is limited to that module only.

Requirement Number:
It gives the reference of Requirement Number in SRS/FRD for Test Case. For Test Case we will specify to which Requirement it belongs to. The advantage of maintaining this one here in Test Case Document is in future if a requirement will get change then we can easily estimate how many test cases will affect if we change the corresponding Requirement.

Version Number:

Under this column we will specify the Version Number, in which that particular test case was introduced. So that we can identify finally how many Test Cases are there for each Version.

Type of Test Case:

It provides the List of different type of Test Cases like GUI, Functionality, Regression, Security, System, User Acceptance, Load, Performance etc., which are included in the Test Plan. So while designing Test Cases we can select one of this option. The main objective of this column is we can predict totally how many GUI or Functionality test cases are there in each Module. Based on this we can estimate the resources.

Test Case Name:

This gives more specific name like particular Button or text box name, for which that particular Test Case belongs to. I mean to say we will specify the Object name for which it belongs to. For eg., OK button, Login form.

Action (Input):

This is very important part in Test Case because it gives the clear picture what you are doing on the specific object. We can say the navigation for this Test Case. Based the steps we have written here we will perform the operations on the actual application.

Expected Result:

This is the result of the above action. It specifies what the specification or user expects from that particular action. It should be clear and for each expectation we will sub-divide that Test Case. So that we can specify pass or fail criteria for each expectation.

Up to the above steps we will prepare the Test Case Document before seeing the actual application and based on System Requirement Specification/Functional Requirement Document and Use Cases. After that we will send this document to the concerned Test Lead for approval. He will review this document for coverage of all user Requirements in the Test Cases. After that he approved the Document.

Now we are ready for testing with this Document and we will wait for the Actual Application. Now we will use the Cycle #1 parts.

Under each Cycle#1 we are having Actual, Status, Bug ID and Remarks.

Number of Cycles is based on the Organization. Some organizations document Three Cycles some organizations maintain the information for Four Cycles.
But here I provided only one Cycle in this Template but you have to add more cycles based on your requirement.

Actual:

We will test the actual application against each Test Case and if it matches the Expected result then we will say it as “As Expected” else we will write the actually what happened after doing those action.

Status:
It simply indicates Pass or Fail status of that particular Test Case. If Actual and Expected both mismatch then the Status is Fail else it is Pass. For Passed Test Cases Bug ID should be null and for failed Test Cases Bug ID should be Bug ID in the Bug Report corresponding to that Test Case.

Bug ID:

This is gives the reference of Bug Number in Bug Report. So that Developer/Tester can easily identify the Bug associated with that Test Case