Leverage Your Existing Experience for Automotive Software Testing

From automotive Software Testing standards, testing techniques, and process, this article is an in-depth guide for those looking to transfer their existing skills to this exciting industry.

For the Software Car, autonomous driving gets most of the hype, but most overlook the fact that there is so much more to Software Testing for the automotive industry. That’s why we decided to write an article that discusses how businesses or individuals that are looking to break into automotive can better understand the industry as a whole and leverage their existing skillsets to effectively test in automotive.

The State of it All

Automotive is one of the largest global industries, and is arguably the most disrupted and reshaped by technology today. According to Wired Magazine,

“The growing role of complex software in a wide range of products provides increased functionality and value for consumers […] this is especially true in the auto industry as cars grow more reliant upon software to automate and manage everything from advanced drivetrains to elaborate infotainment systems.

Given the increasing prevalence of software, it’s no mystery why Toyota signed a deal with Microsoft to bring telematics to cars from the cloud. Microsoft worked with Ford on the groundbreaking Sync system. Google, IBM and Cisco also are moving into the automotive space. According to one study, 90% of the innovation we’re seeing within the auto industry is driven by advancements in software and gadgetry.” 

Cars today have significantly more software in them than they did 5 years ago, and this trend is only going to increase as more businesses work to add more software services to the automotive environment, from obvious entertainment and communication to security gathering significant data to automate and personalize your driving experience—and businesses are all looking to grab a piece of the pie. From major systems running the car, to navigation, to third party application integrations, and growing numbers of sensors, there’s plenty of testing tasks to go around. When one looks at the testing of a car and the essential software components, you’ll notice a lot of testing skills that are transferable from other industries.

Understanding the Big Picture

It’s important to think of automobiles from the start as big, complex systems of devices and specific use hardware. Each of the major car systems has a specific and small OS (operating system) built to enable the device and not much more. Then there is firmware, then drivers, and finally applications. At last, it all gets integrated, and some of the systems need to talk to each other occasionally.

Understanding Some Definitions

This article covers a lot of terms unfamiliar with testers, refer to the glossary for additional definitions.

OSs and the Embedded Systems Model

See Tammy Noergaard’s definition in the column to the right.

Breaking Down an Embedded System

Wikipedia states an embedded system as “a computer system with a dedicated function within a larger mechanical or electrical system, often with real-time computing constraints. It is embedded as part of a complete device often including hardware and mechanical parts. Embedded systems control many devices in common use today. 98% of all microprocessors are manufactured as components of embedded systems.”

Firmware: What is it?

Found on Wikipedia: “In electronic systems and computing, firmware is a specific class of computer software that provides the low-level control for the device’s specific hardware. Firmware can either provide a standardized operation environment for the device’s more complex software (allowing more hardware-independence), or, for less complex devices, act as the device’s complete operating system.” Fossbytes breaks down the differences between firmware, drivers, and software quite eloquently:

“Short Bytes: The principal difference between a firmware, driver, and software is their design purpose. Firmware is a program which gives life to the device hardware. A driver is a middle man between the OS and the hardware component. And a software makes the use of the hardware in the best possible ways.” –Fossbytes.com

Testing Strategy for this System

Hardware, devices, and low-level functionality have always been a unique craft, compared to the mainstream OS UI-driven Software Testing and Test Automation that most testers do. These low-level systems are built uniquely for limited variety or functionality with very few specific uses.

The systems can be large, far-reaching, and important; being safety-critical, most of these low-level systems have limited availability for users (aka the driver) to do things wrong, enter wrong or random data, or try and do things the firmware was not built for it to do.

The power system, or the driveshaft system, is not like a business productivity app or an ecommerce system. They are way narrower in scope. The testing of these system types tends to be very technical with OS or firmware tools as a specific test interface. For example, the braking system will have no public UI. However, the cameras will. The security system setup and configuration will have a public UI but many other functions do not.

Examples: System Integration

These low-level software systems have been around a few years. They are stable enough that there are many fully-functioning and fully-used autonomous driving systems. Their use in the mainstream consumer world is limited today, but we know this will change very soon. The bigger unknown from a software and Software Testing perspective today is the integration of so many middle layer applications, devices, and services. The data collection systems for cars is a massive field that is being played out now, but has not yet been fully realized or tested. The collection of data and its transmission, privacy, algorithms, predicting behaviors, the modeling, building, and testing of these systems has nothing to do with low-level autonomous driving.

Yet it does have to do with new devices, telematics, security, privacy, and infotainment and distraction. Every experienced tester will have skills to leverage in this brave new world.

Regulated: Testing and Guidelines


It’s worthwhile to discuss testing regulated systems-it may or may not impact how you test-at the very least, it will change what is documented, how it is documented and archived for the purpose of a compliance audit.  Standards like Misra C, A-Spice, and auto standards like ISO-26262 and AutoSAR are examples of regulations used in the automotive industry.

Transferable Skills for Automotive Newbies

Transferable Skill: Automaker Car Applications and 3rd Party Integrations


Customers nowadays are demanding the convenience of having a mobile app to access information on their car. This includes tasks such as unlocking and locking their car or even scheduling maintenance appointments via mobile app. Those with mobile application testing experience can readily make the transition to automotive. One of our automotive projects had an application which allowed the user to connect their car and share information in real-time with a mobile phone, tablet, or laptop. This app detailed information on vehicle maintenance, diagnostics, and more. It also had a 3rd-party integration with Amazon’s Virtual Assistant, Alexa. For this project, LogiGear performed API testing using a Javascript (JS) framework, and end-to-end testing; and, at the clients’ request, we set up a timeline for this to fit within their Continuous Delivery (CD) pipeline. This was accomplished through using a C# framework with Specflow. For the integration with Alexa, we used speech-to-text libraries (e.g. Google/IBM voice services) and text-to-speech (Windows services) to verify the Amazon Alexa Skills performed correctly for the Application Under Test (AUT). To see a real demo of how this can work, this article, written by our own Do Nguyen, discusses how to Automate Voice Apps Testing in-depth. We also used Jenkins to ensure the JS framework would fit within their CD pipeline. As you can see, this area of focus could definitely be a great fit for someone who has testing experience in the mobile arena.

Transferable Skill: Infotainment and Sound Systems

Another area of testing in the car is infotainment and the sound system. This is another area that is applicable to those who have already tested some aspects of sound system. Testing acoustics, sound devices, etc., can be done through desktop testing. Another of LogiGear’s projects included testing a comprehensive measurement system for both analogue and digital audio generation and analysis, including digital audio carrier parameters, acoustic transducers, and Windows sound devices. Measuring audio generation is no different than running a test to make sure that the stereo system in the car plays when it’s supposed to. Other scenarios for testing in the car include making sure the software to increase/decrease volume performs as expected. For this, it involved mostly desktop testing, with customized Win32/MFC controls. Testing was done through a C# framework with a Python Harness to call published APIs for testing.

Transferable Skill: Sensors

Testing sensors is prevalent throughout the following industries: oil and gas, healthcare, large scale equipment manufacturing, the electrical and electronics sector, aerospace, and—you guessed it—automotive. Whether it’s the sensor on a piece of factory equipment, or in the check engine light of a car, automating the testing for sensors is easily remedied with a scriptless Test Automation tool like TestArchitect.


For this example, we’ve configured TestArchitect with a custom Python Harness. TestArchitect’s extensibility into Python or C++ allows for this to be more easily configured with the machine. TestArchitect becomes the engine for driving Test Automation, it sends a call request to Python or C++ and the external device (aka sensor) and the request for the action (e.g. turn on/off a sensor) is performed. TestArchitect can validate the data on the AUT by UI testing in real-time, and TestArchitect can communicate via connectors to control and validate the DUT (Device Under Test) in real-time. What we mentioned above is controlling, measuring, and testing the sensors.

By using the sensors themselves as our test equipment, we can solve many challenges in Automation Testing for hardware creatively. For example: Using pressure sensors to do leak testing or using temperature sensors to do safety testing can be just a few examples of ways to use our sensors as test equipment. To see a more in-depth explanation of how this is done, refer to TestArchitect Corner.

Transferable Skill: Embedded Devices/ECUS


Testing embedded devices is a transferable skill, especially when one considers that an Electronic Control Unit (ECU) is simply an embedded system in automotive electronics that controls one or more of the electrical systems or subsystems in a vehicle and can be tested using a leading tool. Automation can be done for ECUs using a tool like TestArchitect.

One of our client’s products is a development and testing software tool for this specific industry. The software is primarily used by automotive manufacturers and electronic control unit (ECU) suppliers for development, analysis, simulation, testing, diagnostics and start-up of ECU networks and individual ECUs.

Its widespread use and large number of supported vehicle bus systems makes it especially well-suited for ECU development in conventional, gasoline vehicles, as well as hybrid and electric vehicles.

The simulation and testing facilities in the tool are performed with the programming language CAPL.

Summary

When one strips away buzzwords and hype, it’s easier to get at the heart of testing a car. It’s important to think of automobiles from the start as big, complex systems of devices, specific use hardware, and of the services. We have covered how testers that are familiar with automated testing, testing mobile applications, ECUs, infotainment or sound systems, or testing sensors, can easily transfer these skills to working on car software systems. 

Whether you’re a business looking to break into this arena, a manager trying to staff a project, or a tester looking to make the transition, there are plenty of areas for you to “jump in head first,” so-to-speak. You may already have the skills you need to start testing the software car!

Guideline References:

A-Spice: http://www.plays-in-business.com/automotivespice/
https://www.prosource.be/blogs/news/automotive-spice-versus-cmmi-method-comparison

Misra C: https://en.wikipedia.org/wiki/MISRA_C
https://www.perforce.com/resources/qac/misra-c-cpp
https://www.embedded.com/electronics-blogs/beginner-s-corner/4023981/Introduction-to-MISRA-C

Regulation ISO 26262: https://www.perforce.com/blog/qac/what-iso-26262-overview
https://www.iso.org/standard/43464.html, https://en.wikipedia.org/wiki/ISO_26262

Regulation AutoSAR: https://en.wikipedia.org/wiki/AUTOSAR
https://www.nxp.com/support/developer-resources/run-time-software/automotive-software-and-tools/autosar-:AUTOSAR-HOME
https://www.engineersgarage.com/articles/autosar-automotive-open-systems-architecture

Michael Hackett
Michael is a co-founder of LogiGear Corporation, and has over two decades of experience in software engineering in banking, securities, healthcare and consumer electronics. Michael is a Certified Scrum Master and has co-authored two books on software testing. Testing Applications on the Web: Test Planning for Mobile and Internet-Based Systems (Wiley, 2nd ed. 2003), and Global Software Test Automation (Happy About Publishing, 2006). He is a founding member of the Board of Advisors at the University of California Berkeley Extension and has taught for the Certificate in Software Quality Engineering and Management at the University of California Santa Cruz Extension. As a member of IEEE, his training courses have brought Silicon Valley testing expertise to over 16 countries. Michael holds a Bachelor of Science in Engineering from Carnegie Mellon University.
Long Trinh
Mr. Long Trinh is a Manager, for the Test Center of Excellence (TCoE) at LogiGear Corporation. As TCoE Manager, he is responsible for providing the plans, strategies and approaches to accomplish all Proof of Concept (PoC) on time, researching new technologies and solutions per demand from marketing, serving all related DevOps activities from organization and clients, test solution providers for all internal projects per requests from DMs, STEMs as well as provides training directions and materials for Organization.

The Related Post

Learn how to leverage TestArchitect and Selenium for turnkey, Automated Web testing. TestArchitect lets you create, manage, and run web-based automated tests on different types of browsers—using either a WebDriver or non-WebDriver technique. In this article, we will explore employing WebDriver for testing a web-based application with TestArchitect. TestArchitect with WebDriver is a tool for automating ...
Introduction A common issue that I come across in projects is the relationship between test automation and programming. In this article I want to highlight some of the differences that I feel exist between the two.
There is no one recipe to make big testing a big success. It takes planning and careful execution of the various aspects, like test design, infrastructure and organization – a mix that can be different for each situation in which you may find yourself. In writing about big testing, the first question that comes up ...
“Testing Applications on the web” – 2nd EditionAuthors: Hung Q. Nguyen, Bob Johnson, Michael HackettPublisher: Wiley; edition (May 16, 2003) This is good book. If you test web apps, you should buy it!, April 20, 2001By Dr. Cem Kaner – Director of Florida Institute of Technology’s Center for Software Testing Education & Research Book Reviews ...
Utility: A program that performs a specific task related to the management of computer functions, resources, or files, as password protection, memory management, virus protection, and file compression. Tool: A program or application that software development teams use to create, debug, maintain, or otherwise support other programs and applications. The term usually refers to programs that can be combined together ...
I got some comments on my post “Test Everything all the Time” — most notably people commenting that it’s impossible to test “everything”. I can’t agree more. The intention of the post was to make the point that we need to be able to test “everything we can” all the time. That is, you should ...
September Issue 2019: Advancing Automation
LogiGear Magazine – March 2011 – The Agile Test Automation Issue
Test Automation is significant and growing-yet I have read many forum comments and blog posts about Test Automation not delivering as expected. It’s true that test automation can improve reliability while minimizing variability in the results, speed up the process, increase test coverage, and ultimately provide greater confidence in the quality of the software being ...
Mobile testers need to take a different approach when it comes to Test Automation.
*You can check the answer key here
Source: From I.M.Testy (BJ Rollison’s blog) I just finished reading Implementing Automated Software Testing by E.Dustin, T. Garrett, and B. Gauf and overall this is a good read providing some well thought out arguments for beginning an automation project, and provides strategic perspectives to manage a test automation project. The first chapter made several excellent ...

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay in the loop with the lastest
software testing news

Subscribe