Book Review: How We Test at Microsoft

I’ve been intending to write a book review of How We Test Software At Microsoft, by Alan Page, Ken Johnston, and Bj Rollison, but for whatever reason I just never found the time, until now.

In general, I like this book a lot. It’s a nice blend of the tactical and the strategic, of the pragmatic and the theoretic, and it covers a lot of ground in a very readable fashion. It’s hard to imagine anybody who is seriously interested in software testing who wouldn’t find something that interested them in the book.

To give you a very high-level overview of the book, here’s the “Contents at a Glance”:

About Microsoft

  1. Software Engineering at Microsoft
  2. Software Test Engineers at Microsoft
  3. Engineering Life Cycles

About Testing

  1. A Practical Approach to Test Case Design
  2. Functional Testing Techniques
  3. Structural Testing Techniques
  4. Analyzing Risk with Code Complexity
  5. Model-Based Testing

Test Tools and Systems

  1. Managing Bugs and Test Cases
  2. Test Automation
  3. Non-Functional Testing
  4. Other Tools
  5. Customer Feedback Systems
  6. Testing Software Plus Services

About the Future

  1. Solving Tomorrow’s Problems Today
  2. Building the Future

Now let’s take a deeper look at a few of the areas the book covers, in more detail.

Not “those squeegee guys that wash your windows”

The section Software Test Engineers at Microsoft describes the organizational approach that Microsoft takes to software testing. I think that Microsoft doesn’t get enough credit in areas such as these. Although there are other high-tech companies that are much larger than Microsoft (e.g., IBM, HP, Cisco, Intel) Microsoft is different from these other companies because they are purely a software company (well, OK, they have a very small hardware organization, but it’s nothing like the others in that list). I think Microsoft has, far and away, the most sophisticated understanding of how to do very-large-scale software engineering, and I also think that they have one of the most sophisticated approaches to software testing. At a previous job, some of my co-workers did contract software engineering for Microsoft in their test organization, and it was very interesting to get a peek behind the curtain at how Microsoft works.

The book discusses some of the major tasks and activities that a Microsoft SDET (Software Development Engineer in Test) gets involved with:

  • Develop test harness for test execution
  • Develop specialty tools for security or performance testing
  • Automate API or protocol tests
  • Participate in bug bashes
  • Find, debug, file, and regress bugs
  • Participate in design reviews
  • Participate in code reviews

This is a challenging role, and it’s pleasing to see Microsoft giving it the respect it deserves.

“The happy path should always pass”

The section A Practical Approach to Test Case Design is one of the strongest in the book, and is just jam-packed with useful, hard-won advice for the practical tester. It contains information on:

  • Testing patterns
  • Test estimation
  • Incorporating testing earlier in the development cycle
  • Testing strategies
  • Testability
  • Test specifications
  • Positive and negative testing
  • Test case design
  • Exploratory testing
  • Pair testing

It’s not an exaggeration to suggest that a professional tester might find it worth getting this book for this section alone, and might well find himself (or herself) returning to re-read this section every year or two just to re-focus and re-center yourself around this level-headed, thorough approach. I particularly enjoy the section’s pragmatic assessment:

There isn’t a right way or a wrong way to test, and there are certainly no silver bullet techniques that will guarantee great testing. It is critical to take time to understand the component, feature, or application, and design tests based on that understanding drawn from a wide variety of techniques. A strategy of using a variety of test design efforts is much more likely to succeed than is an approach that favors only a few techniques.

“The USB cart of death”

The two sections Non-Functional Testing and Other tools are also, in my opinion, particularly strong sections, perhaps surprisingly so since they don’t at first glance look like they should be as informative as they actually are.

Non-Functional Testing talks about a collection of “ilities” that are challenging to test, and that are often under-tested, particularly since it is hard to test them until relatively late in the process:

Areas defined as non-functional include performance, load, security, reliability, and many others. Non-functional tests are sometimes referred to as behavioral tests or quality tests. A characteristic of non-functional attributes is that direct measurement is generally not possible. Instead, these attributes are gauged by indirect measures such as failure rates to measure reliability or cyclomatic complexity and design review metrics to assess testability.

Here we find a variety of very useful sub-sections, including “How Do You Measure Performance?”, “Distributed Stress Architecture”, “Eating Our Dogfood”, “Testing for Accessibility”, and “Security Testing”. Sensibly, many of these sections give some critical principles, philosophies, and techniques, and then are filled with references to additional resources for further exploration. For example, the “Security Testing” section mentions four other entire books specifically on the subject of software security:

  • Hunting Security Bugs
  • The How to Break… series
  • Writing Secure Code
  • Threat Modeling

These are quite reasonable suggestions, though I wish they’d included suggestions to read Ross Anderson’s Security Engineering or some of Bruce Schneier’s work.

Other Tools is unexpectedly valuable, given such a worthless section title. This section makes three basic points:

  • Use your build lab and your continuous integration systems
  • Use your source code management (SCM) system
  • Use your available dynamic and static analysis tools

Of course, at my day job we’re proud to provide what we think is the best SCM system on the planet, so the topics in this section are close to my heart, but even when I wasn’t working for an SCM provider I thought that techniques such as the ones provided in this section are incredibly useful. I’ve spent years building and using tools to mine information from build tools and CI systems, and I’ve found great value in static analysis tools like FindBugs for Java; at my day job we’re big fans of Valgrind. Lastly, I like the fact that this section observes that “Test Code is Product Code”, and so you need to pay attention to design, implementation, and maintenance of your tests, just as you pay attention to these same topics for your product code.

“Two Faces of Review”

Oddly buried in the About the Future section at the end of the book is an all-too-short section on code reviews. The section contains some suggestions about how to organize and structure your code reviews, how to divide and allocate reviewing responsibilities among a team, and how to track and observe the results of your code review efforts so that you can continue to improve them. And I particularly liked this observation:

For most people, the primary benefit of review is detecting bugs early. Reviews are, in fact, quite good at this, but they provide another benefit to any team that takes them seriously. Reviews are a fantastic teaching tool for everyone on the team. Developers and testers alike can use the review process to learn about techniques for improving code quality, better design skills, and writing more maintainable code.
Conducting code reviews on a regular basis provides an opportuni­ty­ for everyone involved to learn about diverse and potentially superior methods of coding.

Here, I have to take a brief aside, to relate a short story from work. Recently, my office conducted a two-day internal technical conference. The software development staff all went offsite, and we got together and discussed a variety of topics such as: agile programming, domain-specific languages, cloud computing, and other trendy stuff. But one presentation in particular was perhaps unexpected: our President, Christopher Seiwald, reprised a presentation he’s given many-a-time before: The Seven Pillars of Pretty Code. If you’ve never seen the presentation, give it a try: it’s quite interesting. But the important part, I think, is that the company felt strongly enough about the importance of code, and of code review, to get everybody, including the company president, together to spend an hour discussing and debating what makes great code, how to write great code, and so forth.

Is How We Test Software At Microsoft a great book? No, it’s not. It’s too long, and the presentation style bounces around a lot (not surprising for a book with many different authors), and the book is a bit too encyclopedic. I wish that the authors had covered fewer topics, perhaps only one-half to two-thirds of the overall topics, but had covered them in more detail. And the book’s unapologetically-Microsoft-only approach can be frustrating for people outside of Microsoft who are interested in how to apply these techniques in other situations: what if you’re trying to test low-memory handling on a Linux system, rather than Windows? What should you be thinking about when looking at character set issues in Java? etc. And I think that the book gives rather too much credit to some approaches that I’m not particularly fond of, such as code-complexity measurement tools, model-based test generators, and the love-hate relationship I have with code coverage tools.

But those are fairly minor complaints. If you are a professional tester, or a professional software developer, or even an amateur software developer, you’ll find that this book has a lot of ideas, a lot of resources, and a lot of material that you can return to over and over. Don’t make it the first software engineering book that you buy, but consider putting it on your personal study list, somewhere.
 

Bryan Pendleton

Bryan Pendleton is a software engineer living in Northern California. He works for Perforce Software as a server developer, and is also a committer on the Derby project at the Apache Software Foundation. Visit his blog at Journal of a Programmer.

Bryan Pendleton
Bryan Pendleton is a software engineer living in Northern California. He works for Perforce Software as a server developer, and is also a committer on the Derby project at the Apache Software Foundation.

The Related Post

Think you’re up for a challenge? Print this word search out! See if you can find all the words and learn a few new software testing terms in the process. To see how you’ve done, check your answers in the answer key below. *You can check the answer key here.
Plan your Test Cases with these Seven Simple Steps What is a mind map? A mind map is a diagram used to visually organize information. It can be called a visual thinking tool. A mind map allows complex information to be presented in a simplified visual format. A mind map is created around a single ...
Companies generally consider the software they own, whether it is created in-house or acquired, as an asset (something that could appear on the balance sheet). The production of software impacts the profit and loss accounts for the year it is produced: The resources used to produce the software result in costs, and methods, tools, or ...
Most have probably heard the expression ‘less is more‘, or know of the ‘keep it simple and stupid‘ principle. These are general and well-accepted principles for design and architecture in general, and something that any software architect should aspire to. Similarly, Richard P. Gabriel (a major figure in the world of Lisp programming language, accomplished poet, and currently ...
Trying to understand why fails, errors, or warnings occur in your automated tests can be quite frustrating. TestArchitect relieves this pain.  Debugging blindly can be tedious work—especially when your test tool does most of its work through the user interface (UI). Moreover, bugs can sometimes be hard to replicate when single-stepping through a test procedure. ...
“Combinatorial testing can detect hard-to-find software faults more efficiently than manual test case selection methods.” Developers of large data-intensive software often notice an interesting—though not surprising—phenomenon: When usage of an application jumps dramatically, components that have operated for months without trouble suddenly develop previously undetected errors. For example, newly added customers may have account records ...
Test plans have a bad reputation, and perhaps, they deserve it! There’s no beating around the bush. But times have changed. Systems are no longer “black boxes” where QA Teams are separated from design, input, and architecture. Test teams are much more technically savvy and knowledgeable about their systems, beyond domain knowledge. This was an old ...
Dr. Cem Kaner – Director, Center for Software Testing Education & Research, Florida Institute of Technology PC World Vietnam: What did you think of VISTACON 2010? Dr. Kaner: I am very impressed that the event was very professionally organized and happy to meet my old colleagues to share and exchange more about our area of ...
I’ve been reviewing a lot of test plans recently. As I review them, I’ve compiled this list of things I look for in a well written test plan document. Here’s a brain dump of things I check for, in no particular order, of course, and it is by no means a complete list. That said, if you ...
MARCH 2016_ TEST DESIGN ISSUE
Alexa Voice Service (AVS): Amazon’s service offering for a voice-controlled AI assistant. Offered in different products. Source: https://whatis.techtarget.com/definition/Alexa-Voice-Services-AVS Autopilot Short for “automatic pilot,” a device for keeping an aircraft on a set course without the intervention of the pilot. Source: https://en.oxforddictionaries.com/definition/us/automatic_pilot Blockchain Infrastructure: A complex, decentralized architecture that orchestrates many systems running asynchronously over the ...
Karen N. Johnson began as a technical writer in 1985 and later switched to software testing in 1992. She maintains a blog at TestingReflections, a collaborative site where she is featured as a main contributor. In her latest entry, she discusses search testing with different languages. Here is an excerpt from her blog: “I started ...

One thought on “Book Review: How We Test at Microsoft

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news

Subscribe