From automotive Software Testing standards, testing techniques, and process, this article is an in-depth guide for those looking to transfer their existing skills to this exciting industry.
For the Software Car, autonomous driving gets most of the hype, but most overlook the fact that there is so much more to Software Testing for the automotive industry. That’s why we decided to write an article that discusses how businesses or individuals that are looking to break into automotive can better understand the industry as a whole and leverage their existing skillsets to effectively test in automotive.
The State of it All
Automotive is one of the largest global industries, and is arguably the most disrupted and reshaped by technology today. According to Wired Magazine,
“The growing role of complex software in a wide range of products provides increased functionality and value for consumers […] this is especially true in the auto industry as cars grow more reliant upon software to automate and manage everything from advanced drivetrains to elaborate infotainment systems.
Given the increasing prevalence of software, it’s no mystery why Toyota signed a deal with Microsoft to bring telematics to cars from the cloud. Microsoft worked with Ford on the groundbreaking Sync system. Google, IBM and Cisco also are moving into the automotive space. According to one study, 90% of the innovation we’re seeing within the auto industry is driven by advancements in software and gadgetry.”
Cars today have significantly more software in them than they did 5 years ago, and this trend is only going to increase as more businesses work to add more software services to the automotive environment, from obvious entertainment and communication to security gathering significant data to automate and personalize your driving experience—and businesses are all looking to grab a piece of the pie. From major systems running the car, to navigation, to third party application integrations, and growing numbers of sensors, there’s plenty of testing tasks to go around. When one looks at the testing of a car and the essential software components, you’ll notice a lot of testing skills that are transferable from other industries.
Understanding the Big Picture
It’s important to think of automobiles from the start as big, complex systems of devices and specific use hardware. Each of the major car systems has a specific and small OS (operating system) built to enable the device and not much more. Then there is firmware, then drivers, and finally applications. At last, it all gets integrated, and some of the systems need to talk to each other occasionally.
Understanding Some Definitions
This article covers a lot of terms unfamiliar with testers, refer to the glossary for additional definitions.
OSs and the Embedded Systems Model
See Tammy Noergaard’s definition in the column to the right.
Breaking Down an Embedded System
Wikipedia states an embedded system as “a computer system with a dedicated function within a larger mechanical or electrical system, often with real-time computing constraints. It is embedded as part of a complete device often including hardware and mechanical parts. Embedded systems control many devices in common use today. 98% of all microprocessors are manufactured as components of embedded systems.”
Firmware: What is it?
Found on Wikipedia: “In electronic systems and computing, firmware is a specific class of computer software that provides the low-level control for the device’s specific hardware. Firmware can either provide a standardized operation environment for the device’s more complex software (allowing more hardware-independence), or, for less complex devices, act as the device’s complete operating system.” Fossbytes breaks down the differences between firmware, drivers, and software quite eloquently:
“Short Bytes: The principal difference between a firmware, driver, and software is their design purpose. Firmware is a program which gives life to the device hardware. A driver is a middle man between the OS and the hardware component. And a software makes the use of the hardware in the best possible ways.” –Fossbytes.com
Testing Strategy for this System
Hardware, devices, and low-level functionality have always been a unique craft, compared to the mainstream OS UI-driven Software Testing and Test Automation that most testers do. These low-level systems are built uniquely for limited variety or functionality with very few specific uses.
The systems can be large, far-reaching, and important; being safety-critical, most of these low-level systems have limited availability for users (aka the driver) to do things wrong, enter wrong or random data, or try and do things the firmware was not built for it to do.
The power system, or the driveshaft system, is not like a business productivity app or an ecommerce system. They are way narrower in scope. The testing of these system types tends to be very technical with OS or firmware tools as a specific test interface. For example, the braking system will have no public UI. However, the cameras will. The security system setup and configuration will have a public UI but many other functions do not.
Examples: System Integration
These low-level software systems have been around a few years. They are stable enough that there are many fully-functioning and fully-used autonomous driving systems. Their use in the mainstream consumer world is limited today, but we know this will change very soon. The bigger unknown from a software and Software Testing perspective today is the integration of so many middle layer applications, devices, and services. The data collection systems for cars is a massive field that is being played out now, but has not yet been fully realized or tested. The collection of data and its transmission, privacy, algorithms, predicting behaviors, the modeling, building, and testing of these systems has nothing to do with low-level autonomous driving.
Yet it does have to do with new devices, telematics, security, privacy, and infotainment and distraction. Every experienced tester will have skills to leverage in this brave new world.
Regulated: Testing and Guidelines
It’s worthwhile to discuss testing regulated systems-it may or may not impact how you test-at the very least, it will change what is documented, how it is documented and archived for the purpose of a compliance audit. Standards like Misra C, A-Spice, and auto standards like ISO-26262 and AutoSAR are examples of regulations used in the automotive industry.
Transferable Skills for Automotive Newbies
Transferable Skill: Automaker Car Applications and 3rd Party Integrations
Customers nowadays are demanding the convenience of having a mobile app to access information on their car. This includes tasks such as unlocking and locking their car or even scheduling maintenance appointments via mobile app. Those with mobile application testing experience can readily make the transition to automotive. One of our automotive projects had an application which allowed the user to connect their car and share information in real-time with a mobile phone, tablet, or laptop. This app detailed information on vehicle maintenance, diagnostics, and more. It also had a 3rd-party integration with Amazon’s Virtual Assistant, Alexa. For this project, LogiGear performed API testing using a Javascript (JS) framework, and end-to-end testing; and, at the clients’ request, we set up a timeline for this to fit within their Continuous Delivery (CD) pipeline. This was accomplished through using a C# framework with Specflow. For the integration with Alexa, we used speech-to-text libraries (e.g. Google/IBM voice services) and text-to-speech (Windows services) to verify the Amazon Alexa Skills performed correctly for the Application Under Test (AUT). To see a real demo of how this can work, this article, written by our own Do Nguyen, discusses how to Automate Voice Apps Testing in-depth. We also used Jenkins to ensure the JS framework would fit within their CD pipeline. As you can see, this area of focus could definitely be a great fit for someone who has testing experience in the mobile arena.
Transferable Skill: Infotainment and Sound Systems
Another area of testing in the car is infotainment and the sound system. This is another area that is applicable to those who have already tested some aspects of sound system. Testing acoustics, sound devices, etc., can be done through desktop testing. Another of LogiGear’s projects included testing a comprehensive measurement system for both analogue and digital audio generation and analysis, including digital audio carrier parameters, acoustic transducers, and Windows sound devices. Measuring audio generation is no different than running a test to make sure that the stereo system in the car plays when it’s supposed to. Other scenarios for testing in the car include making sure the software to increase/decrease volume performs as expected. For this, it involved mostly desktop testing, with customized Win32/MFC controls. Testing was done through a C# framework with a Python Harness to call published APIs for testing.
Transferable Skill: Sensors
Testing sensors is prevalent throughout the following industries: oil and gas, healthcare, large scale equipment manufacturing, the electrical and electronics sector, aerospace, and—you guessed it—automotive. Whether it’s the sensor on a piece of factory equipment, or in the check engine light of a car, automating the testing for sensors is easily remedied with a scriptless Test Automation tool like TestArchitect.
For this example, we’ve configured TestArchitect with a custom Python Harness. TestArchitect’s extensibility into Python or C++ allows for this to be more easily configured with the machine. TestArchitect becomes the engine for driving Test Automation, it sends a call request to Python or C++ and the external device (aka sensor) and the request for the action (e.g. turn on/off a sensor) is performed. TestArchitect can validate the data on the AUT by UI testing in real-time, and TestArchitect can communicate via connectors to control and validate the DUT (Device Under Test) in real-time. What we mentioned above is controlling, measuring, and testing the sensors.
By using the sensors themselves as our test equipment, we can solve many challenges in Automation Testing for hardware creatively. For example: Using pressure sensors to do leak testing or using temperature sensors to do safety testing can be just a few examples of ways to use our sensors as test equipment. To see a more in-depth explanation of how this is done, refer to TestArchitect Corner.
Transferable Skill: Embedded Devices/ECUS
Testing embedded devices is a transferable skill, especially when one considers that an Electronic Control Unit (ECU) is simply an embedded system in automotive electronics that controls one or more of the electrical systems or subsystems in a vehicle and can be tested using a leading tool. Automation can be done for ECUs using a tool like TestArchitect.
One of our client’s products is a development and testing software tool for this specific industry. The software is primarily used by automotive manufacturers and electronic control unit (ECU) suppliers for development, analysis, simulation, testing, diagnostics and start-up of ECU networks and individual ECUs.
Its widespread use and large number of supported vehicle bus systems makes it especially well-suited for ECU development in conventional, gasoline vehicles, as well as hybrid and electric vehicles.
The simulation and testing facilities in the tool are performed with the programming language CAPL.
Summary
When one strips away buzzwords and hype, it’s easier to get at the heart of testing a car. It’s important to think of automobiles from the start as big, complex systems of devices, specific use hardware, and of the services. We have covered how testers that are familiar with automated testing, testing mobile applications, ECUs, infotainment or sound systems, or testing sensors, can easily transfer these skills to working on car software systems.
Whether you’re a business looking to break into this arena, a manager trying to staff a project, or a tester looking to make the transition, there are plenty of areas for you to “jump in head first,” so-to-speak. You may already have the skills you need to start testing the software car!
Guideline References:
A-Spice: http://www.plays-in-business.com/automotivespice/
https://www.prosource.be/blogs/news/automotive-spice-versus-cmmi-method-comparison
Misra C: https://en.wikipedia.org/wiki/MISRA_C
https://www.perforce.com/resources/qac/misra-c-cpp
https://www.embedded.com/electronics-blogs/beginner-s-corner/4023981/Introduction-to-MISRA-C
Regulation ISO 26262: https://www.perforce.com/blog/qac/what-iso-26262-overview
https://www.iso.org/standard/43464.html, https://en.wikipedia.org/wiki/ISO_26262
Regulation AutoSAR: https://en.wikipedia.org/wiki/AUTOSAR
https://www.nxp.com/support/developer-resources/run-time-software/automotive-software-and-tools/autosar-:AUTOSAR-HOME
https://www.engineersgarage.com/articles/autosar-automotive-open-systems-architecture
