I just got back from the 8th International Conference on Software QA and Testing for Embedded Systems, Oct 21 – 23. This was my first time at this conference and I’d give it a high grade for the overall quality of the content and speakers, and in particular, for the people who organize this conference. They are first class. Thanks to my new friends at SQS S.A., Spain — Sander Hanenberg, Jesús M. de la Maza and Begoña Laibara — for treating me well during the conference.
My keynote “Automation Coverage Less than 50%? Don’t Do It!” explored the common excuses for failing to achieve high-volume test automation output. It also pin-pointed underlying barriers to test automation. My key messages were:
- You must achieve high-volume test automation that exceeds at least 50% of automatable test cases or you will not get good return-on-expenses.
- To get efficient in optimizing your test volume, you need to have a good handle on managing the high rate-of-change (interface changes, functionality changes, etc.). This will help minimize the maintainability effort.
- Your automation technology must be easily extensible to allow supporting new platforms and new object recognition capability. This will help extending your automation technology to support any new platform that your company needs to support.
- Then ultimately, your tests must be highly reusable and scalable. This helps you get to high volume of tests.
Here is the summary of the key takeaways from the talk:
- You must fully understand your automation cost of ownership.
- You should not underestimate the challenge of keeping maintenance costs low (it’s not easy but you must keep it low).
- You need to get efficient—you must optimize your volume of test to exceed 50% coverage or your return value will be marginal.
- Efficiency is key, and it will come from excellent test design and automation methodology (e.g., action-driven), and a well-architected framework technology.
- You must minimize programming tests.
- High scalability comes from high reusability of common “actions” and team-based staffing model.
- You must provide high visibility or transparency to your automation program to give management full control of measurability, which ultimately leads to manageability.
Coincidently, Jamie Tischart of McAfee also spoke at this conference on the topic called “Fusion Testing” (I think Jamie came up with this term) which is a Lightweight Testing Approach of his own that combines classic testing techniques with lightweight principles like Agile and XP to increase test execution by better utilizing test resources. He talked about the needs of both exploratory testing and high-volume test automation, and how you must do both effectively. Ironically, Jamie’s team at MXLogic-McAfee has been living this for years and they have also formed a strategic partnership with LogiGear, now in its 4th year, to help implement this program with high-volume test automation, and been benefiting from the result. This was a real treat for me because to speak about a subject is one thing; but to hear about the success from the work that we contribute is actually a whole different level of satisfaction.
Another new friend I have made is Chris C. Schotanus who was Hans’ colleague at CMG. Chris recently completed another book, “TestFrame: An Approach to Structured Testing” which he was kind enough to give me a copy—thanks Chris. I am sure it will be a great read…so it will be my next read. I’ll let you know how it goes.
To top things off, Heather and I also got a chance to go see an UEFA EUROPA LEAGUE football game (soccer for us in America) between Athletic Club (Bilbao, Spain) and CD Nacional (Madeira, Portugal) which the Athletic Club won 2–1. It was great fun!