|
Add to del.icio.us
Digg it!
Add to reddit
Add to MyWeb
Using Web Server Logs to Direct Web Site Testing
By Karen N. Johnson
Please note: This article was adapted from an article that appeared in STQE magazine, January/February 2001.
Introduction
How often have you wished you knew how your customers really use your Web site? That information actually exists - it is just lying untapped in your Web server's logs. Those records reveal the reality of how your site is used, and can provide a great deal of data if you know how to mine it. Your quality assurance team can use this information to organize browser testing based on user statistics, improve testing coverage of your Web site, and plan more realistic load testing.
Let us take a look at what information can be pulled from Web server logs and how server log information can be used. E-commerce sites are dependent on their company site to keep business alive and thriving. One tactic to keep quality high is to use the Web server logs to help the development staff and other internal departments stay in touch with the customer experience.
Narrow Your Focus
There is always too much testing to get through in the time available for testing. Add up all the combinations of operating systems and browser versions your customers may be using, and it is not possible to test all the features of your Web site in every configuration. Fortunately, you can apply two high-level strategies to cut to the core of what needs to be tested. One strategy is to identify which browser and operating system combination is the most popular among your customers (you can determine the most typical configurations from your Web server logs).
A second strategy is to get organized about what aspects of your Web site are vulnerable to variances in either the operating system or the browser version (or the combination of the two) and plan testing accordingly. This second strategy might be referred to as risk-based testing. By determining the most common configurations and focusing your testing efforts based on risk, your quality assurance team can optimize their testing efforts.
For example, customers may use hundreds of combinations of operating systems and browsers but by using server log information you can assess the most popular combinations. When testing time is limited and testing effort has to be cut, QA can focus on the primary configurations. The time spent on remaining combinations can be based on customer usage statistics. This efficient use of testing coverage would not be possible without using the knowledge gained from the Web server logs.
Testing against a variety of browser and operating system combinations can be focused based on server log information. For general browser trends, you can find information on the W3 schools web site http://www.w3schools.com/browsers/browsers_stats.asp.
Even if we restricted testing to the top environments, we still would not have time to run all tests against each of the configurations. Fortunately, we have not needed to. Some testing is not browser specific. For instance, suppose a new data entry field to an existing online registration page, and the data in that field was verified on the server (rather than, for example, by using JavaScript in the browser). The verification and new "write" operation to the database happen entirely on the server, so we might conclude that the new field would not be impacted by the user's operating system or browser version. Verifying the new field in one test environment might be sufficient. Identifying what aspects of your site only need to be tested in one environment is a tremendous timesaving strategy.
But some Web testing needs to be performed on multiple environments because the operating system and browser combination can have an impact on whether the new feature works correctly. These are the testing activities that have to be assessed more carefully, since this testing
impacts the QA time significantly.
So what types of features require testing in multiple browser and operating system combinations? I have never found a standard list that answered that question, so I invented one of my own, based on my testing experiences. I use this seven-item list as a point of conversation between Development and Quality Assurance departments when discussing the testing strategy for each release. Use this list as a starting point; I would recommend adding to the list based on your own testing experiences.
- JavaScript executes code in the browser: JavaScript code is browser specific, so you will need to identify all the places JavaScript is used within your site (e.g., button clicks, image clicks, field validation, and form validation). If your developers have not specified where JavaScript is used, you can check the source to search for it: access a page, use the browser menu option to View Source, then use Find and search for JavaScript. You will especially want to check pages that have forms, buttons, and images. Your team needs to test each use of JavaScript on each of the browser and OS combinations identified and should also test with JavaScript disabled in the browser.
- Forms: Consider the forms you use carefully. If your forms use JavaScript for form validation, those forms need to be tested on each of the operating system/browser combinations. (The general warning on JavaScript is not enough; you have to pay special attention to forms, because every form adds a twist. Focusing only on JavaScript might tempt you to think only of clickable images and field-level validation.)
- Heavy HTML formatting: Pages heavy with HTML formatting may appear differently in different browsers. Check formatting against each of the popular browsers that you have identified. It is a quick look/see, not a major time-consuming test.
- User Interface: If your site uses Cascading Style Sheets, Active X Components, other newer technologies, or browser plug-ins, your testing coverage needs to watch these areas of your site. You need to test in browsers that support the newer technologies, as well as in browsers that do not.
- Security: If your site has a secure checkout or a credit card page, you will want to verify that users can access those pages in secure mode. Since security is a critical component of any e-commerce site (and because secure pages can behave differently based on operating system and browser combinations), secure pages need to be checked carefully on each environment.
- Pop-up windows: If your site uses pop-up windows, you will need to test these carefully on each browser.
- Cookies: If your site uses cookies, testing coverage needs to cover one version each of the identified browsers. Testing with cookies disabled should also be covered.
In addition to new features for a release, I have also planned regression testing for releases. I have created a coverage matrix that shows which features must be tested against which environments. What do they do if we do not have time to complete the entire matrix - or even the core environments? In those situations the team focuses on a combination of popular and fragile environments.
Recognizing Site Patterns
Log statistics can help you determine realistic site patterns. The logs' data reveals which URLs are visited the most frequently, and which URL's users are least "sticky" - those pages most likely to be the ones from which your users leave to surf other pages. Your team can review those "high exit" pages and do an assessment. Do you need to improve graphics or content on that page? Can you verify that performance is acceptable for those URLs? Are there any other usability issues?
At one company, we had a logging mechanism that enabled us to track which features were used most often. We used this information to determine which features and functions users prefer. The most commonly used features received the most testing attention.
Realistic Load Tests
You can also use the server logs to determine realistic load tests, by quantifying your site's daily statistics - such as the number of visitors per day, or the traffic patterns by time of day and day of week. When planning load tests, this information will help your team know what peak loads your site needs to handle. You can use the statistics to know which URLs are visited most often, and include those URLs in your load test user scenarios.
The statistics may also reveal the length of a typical user session, which can help to determine how long to sustain the load for performance testing.
Improving Searches
If your site uses a search facility, data from your server logs can go a long way toward improving your customers' experience by helping you identify common search criteria. Your quality assurance team can then use this information as they try to mimic your user audience. Do most user sessions include one or more searches? Then include searching in a typical user scenario.
What is the typical user behavior after a search? Do your customers typically purchase the product they find? Do they review the product details? If your site is a content-only site, do your users typically drill down for more information after a search? Take a look at your server logs' data to learn your users' behavior and build your testing scenarios to mirror the user experience.
Even unsuccessful searches can provide quality assurance teams with valuable information. Use the log information to determine which searches fail, or which searches return too many matches. When testing your search facility, include a search that has a match, a search with too many matches, and a search with no matches. Take a proactive approach to the failed searches; you may want to have your content department add words to the dictionary of available search words.
Your logs can also reveal the performance levels that your users experience as they perform their searches. Your development group can add a time stamp to each search query and a time stamp to each result. This information enables Quality Assurance and Development to be aware of current and realistic search performance, not just what is projected or estimated. This information can be used to ensure that appropriate hardware is in place for your production site. Your quality assurance team can use this performance information to test any changes made to the search facility, to ensure there is no decrease in search performance.
One example of how we used server logs as a guide for focusing resources was to assess the most common features of the site to build our performance testing. While we certainly cared about performance throughout the site, we knew - because of the logs – what functionality was used by nearly every customer. We learned how to mine untapped data from server logs, so that we didn't have to make guesses about customers' online experience and site needs.
Conclusion
As can be seen, a web server's logs can be used as a road map to guide software testing activity. By using such a "road map", your testing efforts will focus on the things that really matter and make a difference. Effective use of the logs can also reduce testing time while improving testing coverage because of the focus it brings to testing efforts.
About the Author
Karen N. Johnson is an independent software test consultant. She is a frequent speaker at software testing conferences and is an active participant in several software testing workshops. She serves as a Director on the Board for the Association for Software Testing and is a panel expert on Tech Target's web site www.searchsoftwarequality.com. For more information about Karen, visit http://www.karennjohnson.com.
Other Articles by this Author
Download free articles, white papers, templates and more!
|
