TESTING TOOLS HANDBOOK

Introduction What is QA? Testers QA flow     QA General Levels of QA Functionality testing Compatibility testing Integration testing Performance testing Security testing Stress testing Fail Over testing Negative testing Release Level Definitions Conceptual Alpha Beta Gold Release Candidates 3rd Party testing Benefits of outside testing houses (imitating Customer feedback) Web Testing The HTML3 – HTML4 basics that help Being creative in ways of breaking things Browser Matrix Testing Why? Netscape Internet Explorer AOL What is a good Browser Matrix? What to Look for Connection – speed, error types Load time - 50KB rule Java console Version JavaScript is not Java Global Functions or SSI Database writing and reading Cookies Images Bookmarks Links, pages and link checkers Forms Resizing Video and 3d plug-in rendering Sound Resolutions Text problems – Font consistency, spelling errors, grammar errors, and CSS. Alt Text Alignment Plug-ins Consistency Copyrights Search Engines

Bug Tracking Why? (Communication / speed) Bug databases Process / Flow Writing bugs Severity and definitions Format Do’s and Don’ts Test Scripts Why? How to write Lab Setup Hardware OS Internet Connections Clean VS Dirty systems Ghost (how to make it work for you)

The Quality Assurance Web Tester’s Handbook

Dedication
To my Intel IWOS QA team…Don’t ever ask me a QA question again, it’s all in this handbook!

Introduction

This handbook has been written for the new QA web testers out there. Welcome to web QA! We all have a little bit of QA in us. If your not new to QA, I hope that you can at least learn one thing from it all and use it to your benefit. My suggestion is read it all the way through and then use it as your checklist for QA’ing your product. This is more of a fact and tips book or maybe a “Cliff Notes” handbook, so don’t expect to many details (maybe the next addition). In attempt to not make this handbook to long or extremely boring the following pages focus on the role of an actual QA tester and do not really get into QA management.

 In the following pages I have several examples and facts on web testing but keep in mind this does not cover every scenario. You’ll probably think of a ton of things that could be added. But I just want to briefly focus on the basic points of Web QA. Hopefully by putting all this stuff from my head into writing I can erase that QA mental checklist and add other stuff to my brain…maybe some more programming languages? Who knows? I just need more hard drive space up there and upgrading isn’t an option yet. Johnny Pneumonic where are you when I need you?

What is QA?

QA stands for Quality Assurance and is defined as: a planned and systematic pattern of all actions necessary to provide adequate confidence that the product optimally fulfils customer's expectations. This QA process has multiple levels, some handled by specific teams.  Each with a specific role in the process but all working together to achieve these nine items of quality measurement criteria:

1.   Correctness factor – Is the software achieving the things we wanted it to?

2.   Accuracy factor – Is the software achieving all it’s goals?

3.   Efficiency factor – Is the software running at peak performance levels?

4.   Reliability/ safety factor – is the software safe use?

5.   Maintenance factor – Is it possible to fix or take care of the software?

6.   Flexibility factor – Is the software easy to change and is it possible to reuse some of its parts?

7.   Testing factor - Is it possible to test the software.

8.   Computability factor – Is it possible to transfer between computers systems and openness to external environment?          

3^rd Party QA

QA does not have to be done in house. Many 3^rd party QA companies are available and have knowledgeable people (saves time, gets rid of the in house learning curve) and can usually provide instant manpower for small or large applications. 3^rd party test houses usually have a wide hardware matrix to choose from if that is a factor in the testing needed (saves money).

What does a QA tester look like?

To fit the role of QA tester a person must love computers, have no social life, enjoy video games, wear glasses, smell of coffee, and have a nerdy laugh.
Or…

After being in the biz a little while and managing teams I feel the perfect Web QA Tester is somebody who is an up and coming programmer or a programmer period. Understanding the code underneath the web app you are working on is huge benefit and gives the tester great hacking/breaking skills. Most importantly you must love working with computers. Computers must be your best friend. If you can’t work with them all day WITH patience then you might be better off working in a different field. Hey, I heard Burger King was hiring!

QA Levels

The QA process is divided to provide and show proper QA coverage of a product. This allows teams to handle/focus on different areas individually, until all levels are complete. Once complete, the application/product QA process is complete.

Functional testing

Testing that ensures the application is working as planned. This includes testing for compatibility with hardware, OS, screen resolutions, transfer rates, and browsers. Looking at the functionality of an e-commerce site might raise such questions as:

·         Are all purchasing process working correctly?

·         Are all the item prices being calculated correctly?

·         Is the site responsive?

·         Is it taking orders?

·         Are the financial modules performing the right tasks?

·         Is the inventory being updated correctly?

·         Is your distribution working as promised?

Usability testing
Testing to make sure that the end user experience is positive and that the user interface is properly designed. High usability means a site is:

·         Easy to learn and remember (consistent).

·         Efficient.

·         Visually pleasing and fun to use.

·         Quick to recover errors.

Compatibility testing
Testing that ensures the application works appropriately on supported operating systems and browsers. Possible scenarios:

• Double byte characters.
• Differences between browsers.
• Differences between OS’.

Link testing/Page audits

Testing that ensures all links of the application pages go to the correct locations.

·         Manual link testing

·         Link testing using a program like Linkbot Pro.

Regression

Testing that verifies that past bugs are fixed and old bugs stay fixed.

Localization testing

Testing that ensures that the English version of the application can handle input from different language browsers and systems.

• Double byte characters.
• Language specific characters.
• Language writing styles.

Integration testing/Build Verification Testing (BVT)

Integration testing ensures that currently updated code flows/works with the old code. This happens when partial build/sections are integrated into the application. A BVT is a check on a whole site after a new build has been released, to ensure bug fixes have not affected other parts of a site.

 

Performance testing

Testing that ensures that the application works well using different connection speeds. A maximum time limit for page load should be set.

• 56K modem testing.
• LAN testing.
• Cable/DSL testing.
• Proxy testing.

Stress/Load testing
Testing that ensures the application works during peak load conditions. The maximum amount of simultaneous users that can use the application before the application breaks will be determined.

Fail Over testing

Testing that ensures that online servers back up plan initiates when the primary servers fail.

Negative testing
Testing that ensures the application gives appropriate messages and/or handles invalid input. The test cases should include boundary conditions as well as invalid input.

Information security testing
Testing that ensures the application is secure and unbreakable by hackers. This includes application level security as well as Network Security testing.

Possible risks:

• Spoofing – Copying of existing pages to create illegitimate sites that appear to be published by established organizations. In fact, con artists have illegally obtained credit card numbers by setting up professional-looking storefronts that mimic legitimate business.
• Unauthorized disclosure – When transaction information is transmitted, hackers can intercept the transmissions to obtain customers confidential information.
• Unauthorized action – A competitor or disgruntled customer can alter your web site so that it refuses services to potential clients, or malfunctions.
• Data Alteration – The content of a transaction can be intercepted and altered en route, either maliciously or accidentally. User name, Credit card numbers, and dollar amounts are all vulnerable to such alteration.

Release Levels

Conceptual

QA will hardly ever see an application during this level. Most of the time the app only exists on a whiteboard somewhere. If QA were to approach the application on this level it would mostly be with menu design and usability.

Alpha

·         Development/Testing site available.

·         No Catastrophic PR’s.

·         Ability to get to pages (may not be with correct links).

·         Base test plan complete

·         List of countries/languages supported.

·         Site is 33% complete??????

Beta

·         Feature complete (all further changes scheduled with Release notes).

·         All links work from the main page

·         Tested with primary rev of IE and Netscape.

·         Test case document done.

·         Site is 85% complete.??????

Gold/Release Candidates

·         All links work (pass link checker).

·         No show stopping or High PR bugs.

·         All supported browsers are tested.

·         Full grammar and spelling check done.

·         Stress testing is complete.

·         Site is 100% complete.

Web Testing

The HTML3 – HTML4 basics that help –Tags. Frames, layers, tables

Knowing what’s going on behind the scenes gives a QA tester more power in finding and writing bugs.

A little bit of HTML or JavaScript knowledge will take you a long way. Being able to portray a bug with sections of code or file names makes it really easy on the developer. Even if you are able to visually portray the site using HTML terms is very helpful. For example: “In the top left FRAME there is a table with 2 columns, inside the second table column is a broken image”. The developer will love you if you can portray the Web site bugs in this way. Guaranteed! The problem is you can’t really report this way if you don’t know HTML or JavaScript. The best thing to do is get your hands dirty. Try a little coding yourself. It’s not as hard as you might think. I promise that when you have just a small grasp of these languages you will take Web QA testing to next level (Database….Ooooooh GAWD!).

Creativity in ways of breaking software
You can follow all the suggestions in this handbook but there will always be new ways of testing or breaking things. You’ve got to pretty much leave it up to your imagination. Think like a hacker (that’s where HTML or scripting knowledge is useful).

Crying Wolf

One of the most common mistakes I’ve seen from new QA tester is what I call the “Cry Wolf Syndrome”. Medically I think, this syndrome is caused by an over stimulation of adrenaline on the brain causing testers to take a new found bug and run screaming to the nearest manager without first verifying the bug exists on other PC’s. Although this is a severe and problematic condition it is curable with a little self-discipline. Although it can be exciting to find a huge “showstopping” bug, if the bug is not verified on another machine or the details gathered you may be “Crying wolf” or just wasting peoples time with something that doesn’t apply to the situation. Here is an example scenario, starring Joe the rookie QA Tester:

While QAing for Microsoft, Joe was one day casually surfing the beta Microsoft.com site for bugs when his browsers GPF’d and smoke began to billow from the back of his computer. Joe jumped up and exclaimed “Microsoft has succeeded in blowing up computers via the web, I’ve got to tell the QA manager”. Joe hastily tracked down the extremely busy manager and asked for a few minutes of his time. Joe’s manager (being extremely busy) took a moment to test the proposed bug on his personal computer. Upon entering the site and same area, they both cringed waiting for the GPF and smoke but nothing happened. Joe’s manager became angry and exclaimed “Thank you for wasting my time, did you verify this before you came to me”? Joe answered “No” in a shaky, meek voice. Needless to say Joe learned his lesson and still to this day triple checks his bugs and gathers all details before reporting or submitting the bug to anybody else.

Browser Matrix Testing

As of 2/9/00, 33.6% of Internet users have Netscape for a browser, giving Internet Explorer 52.9% control, which leaves 13.5% belonging to miscellaneous browsers. Unfortunately Web QA testing has been made quite a bit harder due to the Netscape and Internet Explorer war. Neither will agree to HTML4 standard which affects many things like: Netscape uses JavaScript, while IE uses their own version -Jscript. Due to examples like this developers are forced to write browser specific code to make each browser function as expected. Not only do they have JavaScript differences, but they both work differently when it comes to rendering images, creating font details with CSS (Cascading Style Sheets), window resizing/refreshing, table rendering, frame rendering, etc…  The list goes on and on so this is why we have to set and test a specific range of browsers. Keep in mind the following matrix is intended for Windows operating systems (the headache gets a lot bigger if we start testing MAC or UNIX).

This matrix applies to a site that is 4+ compatible:

IE 4.01 - JavaScript 1.2 compatible

IE 5.0 - JavaScript 1.2 compatible

IE 5.5 - JavaScript 1.2 compatible

AOL 4 (IE4) - JavaScript 1.2 compatible

AOL 5 (IE5) - JavaScript 1.2 compatible

Netscape 4.05 JavaScript 1.2

Netscape 4.08 JavaScript 1.3 

Netscape 4.51 JavaScript 1.3

Netscape 4.7 JavaScript 1.3

This matrix applies to a site that is 3+ compatible:

IE 3.01 – JavaScript 1.1 compatible

IE 4.01 - JavaScript 1.2 compatible

IE 5.0 - JavaScript 1.2 compatible

IE 5.5 - JavaScript 1.2 compatible

AOL 4 (IE4) - JavaScript 1.2 compatible

AOL 5 (IE5) - JavaScript 1.2 compatible

Netscape 3.04 JavaScript 1.1

Netscape 4.05 JavaScript 1.2

Netscape 4.08 JavaScript 1.3 

Netscape 4.51 JavaScript 1.3

Netscape 4.7 JavaScript 1.3

Sometimes the browser matrix gets even easier, this is when the developer decides that the site will only support the latest version of each browser. This cuts the list down to 3; IE 5.5, Netscape 4.7, and AOL5.

If Netscape does not have any of these browsers easily accessible off of their site unfortunately they do not support them any more. So this must be taken into consideration by the developer and the tester during the Alpha stage. How does a tester find old browsers? Netscape and IE are pretty good about archiving 4.0 and up. If you surf the site pretty heavy you can usually find what you need. If you can’t find what you need then the next best alternative is find out the name of the install executable and do an FTP search for it. Still no avail? I hope you have some friends in high places! If these options don’t turn up anything for you then you’re usually out of luck.

What to Look for

When testing it is very easy to become overwhelmed and confused with the site and it’s bugs (especially when testing in multiple languages). The secret to success is using a checklist (like the one I’ve provided below) and checking each item on every individual page working from top to bottom left to right.  For example: The first item on your checklist is looking for image problems. Enter the site and surf the entire site looking for image problems (top to bottom left to right), reporting them as you find them. When images are done, move on to links…etc… etc… This may sound time consuming, and it is, but I promise this method will give you one of the highest bug counts on your team, along with a well QA’d Web site.

The following list contains suggestions of things to look for. Take these examples/scenarios and dream up more for the website your working on. All web sites are different which makes breaking them that much more fun!

Connection

A rule of thumb for developers as of 1999 is to keep a page below 50Kb to keep load time minimal, to keep the users attention. But sometimes a well-built page can still produce errors because of slow connection speeds or overwhelmed servers. A huge 500KB page will usually load, taking some time but a page on a slow server will sometimes time out and report errors. Errors such as DNS Server Error, or 500 Bad Request. This is a bug to be reported by QA. It’s definitely not a QA web page bug but a server performance bug. These types of problems are usually analyzed to the fullest extent by a QA Stress Testing team.

Content

Is the content meet the site criteria? Think of this as a movie rating system. ‘R’ or ‘PG” content isn’t going to fly on a ‘G’ site.

·         Look for the ability to post bad language on G or PG sites via login names, chat rooms, message boards, or top 10 game score postings.

·         Look for possibly offensive pictures. Naked statues or provocative poses.

JavaScript
JavaScript is not Java. Java is not coffee. Coffee is not the answer. NO-DOZE babeh!  Think of JavaScript as a slimmed down version of Java built into the browser. Usually when a JavaScript error occurs in one browser it will occur in other browsers that use the same version of JavaScript. For example looking at the following list we could deduce that if we found a JavaScript error within IE4, more than likely the error can also be found in Netscape 4.x or lower, but probably not in Netscape 4.08.

Versions:
Netscape 2.0 – JavaScript 1.0

Netscape 3.0 – JavaScript 1.1

Netscape 4.x – JavaScript 1.2

Netscape 4.06+ - JavaScript 1.3

Internet Explorer 4 – JavaScript 1.2

Internet Explorer 5 – JavaScript 1.2

Java on the other hand, can be processed by a browser via a Java Applet. The same type of version problems occur with Applets except there are far more versions of Java. Fortunately we see a lot less Java Applets than JavaScript.

Navigation

The world is flat! If you walk to the edge YOU WILL FALL OFF. The World Wide Web that is! Using a page or browser navigation menu can sometimes leave the user stuck in an unwanted place. Things to look for with web site navigation:

• Back buttons that don’t take the user back to the expected page.
• Forward buttons that move the page forward to page previously viewed.
• Home buttons that do not take the user to the root of the site “Home”.
• Bookmarks that when used do not take the user to the desired page.
• Refresh button that refreshes the current page. Sometimes refreshing takes user back to HOME.
• Can you reach a certain page from any point on the Web site? Does it have a global menu?

Cookies

Ahhhh…cookies… My favorite snack. No not Oreos. Web cookies. I like to break them and give them back to developers! Cookies are everywhere in web pages! Developers use this information to track information about you on their site. Simple things like dates, membership info, form info, or pages you’ve visited.

Ways to create cookie problems:

• Mess with system dates (most cookies are set to expire on a certain date)
• Delete the cookie and revisit the site. Do you have to re-sign up for membership?
• Open the cookie. Is it encrypted? (Non-encrypted can be a security problem).

 

Images

Image problems are pretty simple because they are visual and obvious.

Things to look for with images:

• Broken images
• Poor image quality
• Images that are to large in file size (long load time).
• Misaligned images
• Broken links within image maps.
• Broken links within Flash movies
• ALT text inconsistency or misspelling.
• Repeating background images at high resolutions
• Image maps with links

 

Bookmarks

Bookmarks are fun! HEEHEE! I want one with a pretty unicorn on it! Uh…oh. Where was I? Bookmarks are sort of a security threat on membership sites. So how do we test the? Start with the question: Can you bookmark a page that a user shouldn’t enter without going through a certain process?

Something to look for with bookmarks:

• Can the user bookmark a page after logging into the site? Test this by deleting cookies and re-entering the site.
•  Can the user bookmark a page that gives a special offer to get the special offer or free item multiple times?

 

Forms

Forms are pretty tricky and time consuming. Problems can occur in a variety places. Usually forms will submit to some sort of database/cookie or send an email. The testers job is to verify that the information input into the form is captured into the proper location and then (if available) is displayed or “confirmed” with the user. Privacy/Security is also a huge issue with form submission. Forms almost always contain sensitive information. It is QA’s job to make sure that information stays private.

What to test for in forms (most results occur after form submission):

·         Does the input information write to the database or cookie correctly?

·         Is the cookie encrypted if needed?

·         Is the transmission of the form encrypted?

·         Is the form on a secure server? (https://)

·         Does the database or cookie return the previously input information correctly on the confirmation page?

·         Does the database or application you’re submitting the form to understand ASCII characters?

·         Does the form accept a space as a character?

·         Can you enter invalid email addresses? (Tom.at.aol.com)

·         Does the form accept email addresses with 2 letter extensions (tom@bravo.cc)

·         Does the form accept and process capital letters?

·         In password fields, can the user enter a password and different confirmation password and submit the form without an error?

·         Can spaces be used in the password field?

·         What is the character limit of the field? Does it all save to the database? What happens when you input 100, 500, or 1000 characters?

·         How do check/radio boxes affect the submission of the form? Does it affect database entries?

·         Is it possible to overwrite existing data in the database or cookie on submission of the form?

Security Settings

Playing with the browser’s security settings can make a web site not function as intended. Most web sites are designed around default security settings, but the tester needs to look at what happens when the security settings are set to high. A good web site will detect these settings and dump the user on a page stating what the problem is and how to go about fixing it. Examples:

• Turn off cookies, when an attempt to write a cookie occurs, what happens?
• Turn off JavaScript, when JavaScript attempts to load, what happens?
• Turn off JAVA, when an attempt to load a Java Applet occurs, what happens?
• Turn off ActiveX, when an attempt to load an ActiveX control occurs, what happens?

Video and 3d plug-in rendering

• Does video stream at an acceptable rate?
• Is video pausing minimal?
• Are the video players controls responsive?
• Do the textures on 3D models display correctly?
• Is 3D animation smooth?
• Is sound uninterrupted and smooth?
• Is default sound volume acceptable?

Links, pages and link checkers
Manual link checking is very important. All though link checking tools can check for 404 not found pages and various other errors, they can not tell you if a page goes to the correct URL.
Things to look for:

• Does the link open the correct URL as labeled?
• Does the link open the correct URL in the proper frame or window?

 

Sound

Sound testing with the web is pretty basic. Does it work? Does it work well?

• Is the user able to turn off/on sound? (Flash – plug-in)
• Does the sound pause or skip because of download speed or image rendering?
• Is default sound volume acceptable?

Screen Resolutions

Resolutions affect the way a page is viewed. Most pages are designed for a minimum resolution. So before reporting bugs know what that minimum resolution is.

Things to look for with resolution settings:

• If a web page is to large for a certain resolution, is it scrollable or does it get cut off?
• At what resolution does the background start to repeat?

Browser Resizing

Sometimes when a browser is resized it will not resize the web page correctly, leaving pages cut off or improperly rendered. This is a very simple test. Simply resize the browser window by dragging the resize handle, minimize, or maximize the window.

Text problems

Text problems can be found within many parts of web page. You have the plain HTML text, text within Java, text within Flash or a plug-in, or text within an image.

Things to look for with text:

• Spelling/Grammar errors. Copy the text and run it through a spell/grammar checker.
• Left, right, center justification. Does everything line up correctly?
• Do paragraphs appear to have a right margin?
• Some text is in the form of an image, which sometimes causes distortion.
• Is the font type and size consistent throughout the site?

Alt Text

ALT text only applies to images. It is a mouse over effect built into the browser. Things to look for:

• Does the ALT text match or correspond with the image?
• Spelling/grammar errors.
• Inconsistency of the ALT tags. They appear with some images but not others

Search Engines

• Can the – operator be used to prohibit words? Example: Santa –elves -reindeer
• Can the + operator be used to prohibit words? Example: Santa +toys +Rudolph
• Can wildcards be used? Example: *at   = possibly cat, rat, bat,
• Can specific phrases be used? Example: “Santa’s Reindeer”
• Are results displayed in a numbered order, even when entering the next results page?

Plug-ins

• If a required plug-in is not installed does the site prompt or warn you about installing it?
• If an older version of the plug-in is installed but the newer version is needed, does the site prompt the user to upgrade?

Consistency

Multiple people working on one site can affect the consistency. This usually happens because guidelines have not been set. This can create small or large problems.

• Does the menu stay in a consistent location?
• Is the font type and size consistent throughout the site?
• Are framesets consistent?
• Is copyright and legal information consistent?

Copyrights and Legal info

• Is the Copyright date/year current?
• Is a separate legal info page available?

Bug Tracking

No this section has nothing to do with hunting down ants with a magnifying glass. Bug tracking can be done via email, spread sheets or word documents but works best done with a database program. Because of a database’s effectiveness, the following list focuses on tracking the progress of web bugs within a database. Using a bug-tracking database gives QA the advantage of:

• More Control over change process.
• Highlights areas of concern.
• Helps to prevent bug duplication.
• Highlights areas that need testing – Locates problems in a specific set of ranges via queries.
• Regular build process promotes control.
• Methodical change request process tracking.
• Database is central repository for “Change Requests”.
• Better communication to help focus improvements in: What happens with bugs, management tracking process, and the history of problems.

Programs such as PVCS Tracker allow users to do the above mentioned. It gives users a professional environment where many people can work from the same database without affecting others. It allows bug checking in and out, submitter/owner assignment and email forwarding capabilities. Simply, PVCS is a very powerful tool that makes a QA tester’s bug reporting easier.

How to write a bug

The purpose of a bug is to convey a problem. Bug reports need to be concise and need to portray the severity of the problem. Keeping a general format and writing style for your bugs makes everybody’s life a little easier. Being consistent in these areas allows people who read your bugs on a daily basis to easily dissect your report because the format/style is the same the only thing that differs is the actual content or problem. Here is a picture of the “Perfect” bug report.

Steps in writing a bug report:

1.   Check the current database to ensure that the problem has not already been reported.

2.   Bug Title: State the problem in simple terms. A good approach is to emulate a newspaper article title. (Location + Operating Condition + Action = Symptom). Example: “On test.htm, In Internet Explorer when clicking on the blue button a JavaScript error occurs”. It doesn’t have to be ultra pretty, or in that specific order, just get all the info you can in one sentence.

3.   Description: Description is actually broken up into two parts; the actual detailed description of the bug and the listed steps to reproduce. Which pretty much sums it up. The description is the area to list the details of the bug (how, what, where, when) and the steps to reproduce include steps to recreate the bug from a starting location.

4.   Other information: This information is easiest via drop down boxes when it is actually built into the form you input your bugs into, because you’ll find yourself repeating this information over and over again. Fields may include:  Severity, status, language, component, category, operating system, browser, owner and submitter.

Tips on writing a bug report

·         Determine the fewest number of simple steps to reproduce the bug.

·         If the bug involves a detailed error message (GPF, JavaScript, or Database error) cut and paste the error message into the bug description.

·         It’s better to repeat information in the Title, Description and steps than not to have enough information.

·         If the end user effect is unclear, state it in simple terms at the end of the report.

·         Steps to reproduce, stated in simple form will allow anyone to recreate the problem.

·         If the problem is specific to on language, state the language as the first three letters of the Title.

·         Abusive language or opinion is the best way to make sure your bugs are ignored. Don’t go there.

·         If you are testing on multiple platforms/OS’ identify the one you are using.

·         Double-check your own spelling/grammar.

·         If possible provide a work around or suggestion.

How to determine the severity of a bug?

The 4 severities of a bug are low, medium, high and showstopper. All bugs are important but some are not as easily noticed than others. Developers start with show stopping bugs and work there way down. This is done in case deadlines are not met, site has to be launched. If the Showstopper and High bugs are resolved usually the site is pretty stable. But a lot of medium or low bugs may make the user’s experience unpleasant. We won’t talk about that because I may get shot by the developers.

Rules on rating a bug as “Showstopper”:

·         Product does not meet customer requirements.

·         Schedule impact risks FCS.

·         Stops progress for 25% or greater of the user base.

·         Does not meet legal requirements (Trademark and branding).

·         Reduces functionality and one or more of the following secondary failures exists on 10% or greater of the user base:

o   Causes the system to hang.

o   Causes a GPF fault or blue screens.

o   Cause loss of data or conveys wrong information to the user.

o   Product installation doe not install/load correctly.

Real World defect Severity examples for “Showstopper”:

·         404 Not found error.

·         403 Forbidden error.

·         Password protected URLs or sites.

·         VB Script errors preventing the user from going through the site.

·         JavaScript errors preventing the user from going through the site.

·         ODBC Database errors.

·         Image or text links that take the user to the wrong place.

·         Spelling or grammatical errors that may result in a lawsuit.

·         After plug-in installation the related plug-in media will not load.

·         Page is not scrollable but needs to be.

·         Language translation is incomplete.

·         Inputting 10,000 characters into a form text-box crashes the server.

·         Links to download do not download files.

·         Something that causes serious performance degradation under load.

·         Deleting cookies makes the user go into an infinite redirect loop when trying to log in.

·         System cannot handle projected user loads due to programming errors.

·         Hardware errors.

Rules on rating a bug as “High”

·         Stops progress for 10% or greater of the user base.

·         Schedule impact risks FCS.

·         Work around does exist.

·         Is a high annoyance/impact to user and the frequency of the occurrence is high

·         Reduces functionality.

Real World defect Severity examples for “High”:

·         Broken image.

·         JavaScript form checking doesn’t work.

·         Spelling or grammatical error.

·         Buttons that do not function as expected.

·         Text does not fit on page.

·         Windows do not operate as expected.

·         Image alignment is off/bad.

·         3D distortion within a plug-in.

·         Sound skipping or pausing.

·         Form text boxes will not hold enough characters.

·         Passing sensitive user info is unencrypted.

Rules on rating a bug as “Medium”

·         Problem can be worked around.

·         Annoyance/impact to the user, but the frequency of occurrence is low.

Real World defect Severity examples for medium:

·         ALT tags

·         Distorted images.

·         Page titles mislabeled.

·         Text describes that is not really there: “Click button below”, but no button.

Rules on rating a bug as “Low”

·         No schedule impact.

·         No loss of functionality.

·         Low annoyance/impact to the user.

Real World defect Severity examples for “Low”:

·         Proxy error.

• Button mouse-over does not function.
• Link appears twice (and it’s not supposed to).
• Small Typos.
• Usability problems.
• Cosmetic issues.
• Unexpected behavior/invalid test cases.

 

Process / Flow

The bug flow process is best understood visually. The following flow chart depicts the flow of a bug after submission (example 1). The second chart (example 2) depicts the flow of bug owners and bug status as it changes hands.

 

 

 

 

Example 1

 

Example 2

Test Scripts/Test Cases

Why?

Test scripts are written for testers to follow a certain process of testing to ensure that specific portions of a web site are covered. A good test script is actually a very detailed site map. A site map that even covers things that may not be apparent to user of the site, such as cookies, plug-in installations, Java installations, or database reading and writing. Once a test script is written to cover the features of a site, it can be handed on to others and they will not have to “re-discover” all the features of the site. With a finished test script, browser matrix testing can be quick and efficient. In summary it is a time saving document that ensures that all aspects of a site have been covered. A test script is also known to be a proficient “Butt coverer”. It’s a document that proves that all portions of a site were properly covered in case question should ever arise on errors that may be found after release.

How to write

Like mentioned previously a test script is a document that covers all or specific portions of a site. Your goal is to help other testers cover functionality and compatibility of a site with one big sweep. So where does one start? The first thing to do is familiarize your self with the site. Understand how the site flow works, how the site keeps track of you as a user (cookies/database), and what items are needed to view the site as intended (minimum resolutions, plug-ins, and browser requirements).

 Once you have a general feel for the site its time to start breaking it down into separate components. We do this by starting with the menu. The site is already divided for you via the menu and submenus. With this you can have a section of test script covering each menu/submenu area. Visualize your script as a site map.

Once you’ve divided the menu areas you can dissect each page working from left to right, top to bottom, documenting each item and what the expected behavior is when the item is clicked on. This is done for each page. This method covers the content in each page but does not test the flow of the site. So the last part of your test script should cover the special situations that may while navigating or entering the site. Situations like: Entering the site, registering as a member, receiving a cookie, exiting the site and reentering to verify the cookie allows you in as previous member. There are usually many special situations that need to be covered. Sometimes the situations seem endless and really only need to be verified once, so it not worth documenting in the test script to have many other people re-verify. This is a decision of the test case writer, but he/she should ask himself a basic list of questions to help decide whether it should be documented in the test script:

1)   Will this scenario possibly affect different browsers?

2)   Is the scenario really extensive and difficult? If it is, the possibility of error grows.

3)   Is this standard behavior for site navigation?

4)   Does the scenario utilize something on the back end (database, streaming media)?

5)   Doe the scenario utilize something other than standard HTML (JAVA, XML, PERL)?

6)   Would the scenario be a technical support call generator?

If you can answer yes to any of these you best bet is to include the scenario in the test script. With this basic list and some feelings from your gut, a test scriptwriter can usually make the proper call whether or not something should be included. Here is an example test script: INSERT SCRIPT OR URL HERE

Lab Setup

Hardware

Hardware plays two major roles in web testing: web connection and video.  Machine configurations should be a variety of connection of speeds and video configurations. Why different connection speeds? Different types of connection speeds are used to test user experience with slow VS fast and also Time testing.

With video cards a high-powered card that handles the largest resolutions is the ultimate setup. The more resolutions that can be tested on one machine the better. The only issue I’ve seen that concerns video cards on the web is the use of 3D plug-ins. Some cards do not render textures correctly.

Browsers

Every QA team has limited computer resources. The more computers the better, but lets face it having 30 different computers with different browser configurations is almost impossible because of cost of computers. But a starting point is 3. Computer one with IE4 and Netscape 3.04. Computer 2 with IE 5 and Netscape 4.01. Computer 3 with IE 5.01 and Netscape 4.51. This is a decent browser matrix and the tester has the power to upgrade the browser on each machine in an effort to solve a bug or problem. Or just move to the next machine to see if the bug is apparent on a different browser version.

Internet Connections

The speed of internet connection can affect a variety of things. A slow connection can create severe lag in video, or cause page requests to time out. Giving you false errors. When testing it is best to test as a consumer might. Matching the general connection speed of the public is the best bet. As of 01/20/00 we test with 56K. Cable and DSL are slowly taking over though!

Usually testing from larger corporations users are forced to test via the LAN. This creates problems in its self. Problem number one being; usually all http requests go through a proxy server. Sometimes proxy servers get over loaded which returns browser errors. Example errors caused by proxy servers are: 500 Bad Server Request, and 400 Bad Request.  Problem number two is speed. Although your company may have a fat Internet pipe to the building, 100 people surfing the web at lunchtime can bring your connection speed to a crawl, creating an invalid test.

OS

Operating system is important because each has it’s own default browser that may be different than the downloadable version. Examples:

Win95a IE 3.0
Win95b IE 4.0

Win95C IE 4.0

Win98 IE 4.01

Win98SE IE 5.0

Win2000 IE 5.5

Other instances that deal directly with operating systems are web pages that do an OS check and interact with the user using this information.

The importance of current driver releases:

Having the latest driver versions for all hardware in your PC is important. Having the latest and greatest creates one less possible variable when it comes to troubleshooting a problem.

Scenario #1:

Betty installed the Beatnik plug-in for her browser – IE4, but when she visits a site with some heavy Beatnik 3D work the textures on the 3D object don’t render.

Resolution:

9 times out of 10, dealing with video or sound distortion issues are related to old drivers. A simple hardware driver upgrade will usually solve the problem. If not…Upgrade DirectX and DirectX Media! If this doesn’t solve the problem is probably because your video card doesn’t support the 3D format that the plug-in uses.

“Clean” VS “Dirty” systems

The definition for a “Clean System” is as follows: “A system that has nothing but the default installation of the OS on the drive. No other programs installed (besides drivers)”. A clean system gives the user the benefit of an OS that has never had anything put on it which creates the most valid test case possible. If a problem is encountered the variables to test are very small. Usually the OS or the browser itself is the culprit.

Scenario #1:

Joe enters http://www.testme.com with a “Clean System” and encounters no errors. Betty enters the same site with the same configuration but encounters a JavaScript error. The only difference is that Betty has surfed the internet with her browser for the last 3 days.

Resolution:

We can probably count on the problem being something to do with his “Dirty System”. Like a cookie, or changed security settings. In order to verify that it is the “Dirty System”, retest the site on identical another “Clean system”. If it passes, then Joe’s buddy needs to use the process of elimination to figure out what his machine has that Joe’s doesn’t.

The definition for a “Dirty System” is as follows: “A system that has had other programs (besides the OS) installed on it, or has previously surfed the internet”. An error that is encountered with a “Dirty system” is harder to verify because there are more variables that may be causing the errors found. To test and diagnose an encountered problem the tester will usually have to use the process of elimination to nail down the culprit. But testing with a “Dirty system” is an essential part of QA because that’s what the real world user would have.

Scenario #1:

Joe has installed Norton Antivirus and Microsoft Office 2000 onto his system and every since he has not been able to log into his favorite web site.

Resolution:

Obviously one of the two programs is the culprit if this was the only thing done since Joe last logged onto his favorite web site. The way to resolve this is to use the process of elimination by uninstalling each one until Joe can log back into his favorite web site. Joe may also have to reregister with his favorite site to refresh any info that the web site dumps onto his system.

Symantec’s Ghost (how to make it work for you)

Ghost is a simple disk image program that allows the user to take an image (copy) of a disk or partition and compress it into one file. This program has cut down OS rebuilding time to almost nothing. Picture this:

A fresh install of Windows 98 takes about 250MB of hard disk space. Using ghost to take an “image” of the hard disk and then compressing it makes a ghost image file of about 150MB. It gets even better though. When you’ve decided to restore your OS with this image file it will take Ghost about 3 minutes to have your OS up and running. Another Ultra-Cool fact is that you can use the same ghost images for identical machines. So you can have identical machines with only one Ghost image (huge time saver). Lets take a look at some scenarios where Ghost can benefit the QA tester.

Scenario#1:

Joe was doing a little web site QA’ing with a “dirty” OS and encountered an error that seemed to be browser related.

Resolution:

Just to make sure the error is Browser related Joe needs to test for the error on a “Clean” OS/Browser. With this in mind Joe restores his OS with his “Clean” ghost image and his OS is up and running in around 3 minutes to conduct the test.

Scenario#2:

Betty is surfing the web and downloads an ActiveX control through her browser that corrupts her Windows 95 registry. Does this ActiveX control corrupt Windows 98’s registry or Windows 2000’s registry?

Resolution:

Since Betty is a good tester she took the time and setup multiple Ghost images all the OS’. She uses Ghost to restore the computer to Windows 98, tests the problem and then uses Ghost again to test the problem in Windows 2000. All within 15 minutes, on one machine! With a “Clean”  OS/Browser!

Setting up storage for your Ghost images

The ultimate setup has proven to be on a partitioned drive. Keep in mind Ghost can copy single partitions not just whole drives. Simply put: Put the OS on drive C: and all the Ghost images on drive D: . With this setup you can exit to DOS run ghost from the D: drive and refresh the C: drive. Other options for setup are to have Ghost images burned to CD and no partitioned drive. The only problem with this setup is that if you ever want to fix or update the Ghost image (which happens a lot), you would have to burn a new CD with your new Ghost image. A third option for setup is to store the image files on a separate machine somewhere on a network. But this method has a few problems. The first problem being Network lag can create faulty ghost images and the 2^nd problem being that you must set up your machine to run on the network in DOS, which is a pain!

Batch files

A “Batch” of “Ghosts”? OOOOh! How eery! Ghost is great because it has it’s own little language that can be used in batch files. With a properly programmed batch file you can skip the whole Ghost menu and refresh setup process. Creating a batch file with a desktop shortcut pointing to it gives you the power of refreshing your system to a new OS with one mouse click! What makes this Ultra-Ultra Cool is that you can have multiple icons on the desktop pointing to different OS versions or languages.

Tweaking the OS to your liking before Ghosting the partition

It seems no matter how good you think your OS setup there is something you will always forget to change to your liking. Here’s a don’t forget list:

View all files in Explorer

English Keyboard layout (for international OS’)

Shortcuts to cache-cookie folders

Desktop backgrounds labeling OS and Browser configuration

Turning off Windows Power Management

Turning on the Java Console in IE

Browser proxy settings

Daylight savings disabled (when the time changes Windows prompts you to change every time you restore).

Set resolution and color depth to preferred level.

Enable QuickRes in the taskbar.

Put all your shortcuts to batch files/ghost images on the desktop.

Map heavily used network drives.

Empty the Recycle Bin.

Configure the desktop the way YOU like it.

Clear cookies (Makes browser ultra-clean and shrinks Ghost file size)

Clear temp file (Makes browser ultra-clean and shrinks Ghost file size)

Clear history in IE (Makes browser ultra-clean and shrinks Ghost file size)

Set browser home page to something useful.

How to automate a Ghost Refresh

1)   Using a text editor input this command: “ghost.exe –clone,mode=pload, src=d:\ghost~1\br98.gho”  (command lines can be found within the Ghost manual)

2)   Save the file as “br.bat” and put in your “D:\ghost images” directory.

3)   Create a desktop shortcut to this batch file.

4)   Right mouse click the newly made desktop icon and click properties.

5)   Click the ”Program” tab.

6)   Click the “Advanced” button.

7)   Check the box “MS-DOS mode”.

8)   Check “Use current MS-DOS configuration”.

9)   Click “OK” twice.

Now the Icon will run the batch file you’ve created in MS-DOS mode! Now you’ve got to create the BR98.GHO image that the batch file is pointing to. So use Ghost to copy the “Clean” OS partition (drive C:) to the “D:\Ghost Images” directory.

So think about this lab setup and the power you would have as a QA tester:

1)   A “D:\” drive with Ghost images of Windows 3.11, Windows 98, Windows 98SE, Windows NT 4, Windows 2000, Windows Millenium, and Linux.

2)   An image of Windows 98 with Netscape Navigator and IE 5.5

3)   An Image of Windows NT4 with IE5 and Netscape Navigator.

The options are endless, just set up the drive to make your testing convenient! And all just one mouse click away from your desktop. All you have to do is build and save the OS with Ghost one time!