Challenges in Automotive Testing

Even the highest quality organizations have tradeoffs when it comes to their testing coverage.

In Japan, Europe, and the United States, automotive manufacturers are aiming to enhance automotive functions by using software; in Japan in particular, Toyota, Nissan, Honda, Mazda, and Subaru are all adding endless amounts of software to their vehicles in the form of convenience and safety features. In the automotive industry, many companies are original equipment manufacturers (OEM), meaning they make most (if not all) of their automotive parts in-house. While, Sony, Panasonic, and other electronic manufacturing companies are researching ways to increase their sales and profit, automotive companies’ business is growing. According to statista.com, in 2016, the sales value of the automotive software market in Japan was valued at ¥525 billion Japanese yen ($4.8 billion USD); it is expected to reach ¥747.5 billion Japanese yen ($6.8 billion USD) in 2020 and ¥995 billion Japanese yen ($9 billion USD) by 2030. Japanese automotive brands have a history of being the most reliable vehicles; thus, they will want to ensure their software is reliable as well.

According to J.D. Power’s 2019 U.S. Vehicle Dependability Study, which ranks brands based on problems per 100 vehicles, two out of the top three brands are Japanese: Lexus is number 1 and Toyota is number 3. Infiniti and Subaru make it above the industry average and Nissan is just 1 ranking below said average.

The Tradeoffs of Coverage and Risk

Even the highest quality organizations have tradeoffs. As old as Software Testing, automotive Software Testing has the same issues. Technology aside, companies have to balance coverage and risk with cost and time. High risk needs more testing, more people, more time, and more coverage. Less risk often means less testing, less time, and less coverage. No one wants to make these tradeoffs-especially in a safety-critical product. In a regulated industry, these tradeoffs are often standardized levels demanded for regulatory compliance.

With shift-left, more Continuous Delivery (CD), and modern development techniques like test-driven development (TDD),  there is more unit testing and more meaningful measures of unit test code coverage. The tradeoff may become coverage at what level. Test coverage at the unit level, API or service level, or testing at the UI level. The following is an example of how a Japanese software company balances coverage and risk on an automotive project.

The level of quality demanded varies from project to project. Testing car navigation may require a different level of quality than an electronic control unit (ECU) because car navigation does not have the direct power to kill people.

On the other hand, ECU testing—a system that can kill—requires a higher quality, such as 100% C1 (code-level Branch Coverage) coverage rate, sometimes requiring MC/DC coverage rate. The aforementioned testing terms form a basic idea and methodology toward quality assurance for automotive software.

In one automotive project, we had to balance and think about the size of software and the relationship between ECU and the car navigation system (see Figure below).

· Software size of car navigation = Huge (using Linux, Car Play, Android Auto)

· Software size of ECU = Small (Using automotive specific OS or non-OS)

Based on risk:

· It is okay to cover 100% coverage testing with small ECU software.

· It may be okay to cover less than 20% code-level coverage testing with huge car navigation software.

Figure – Relationship between ECU and the car’s navigation system

Treated separately, it is okay to not communicate between ECU and car navigation. When the ECU communicates too much with car navigation, there could be critical quality problems. But, automotive software is becoming increasingly complicated every year.

And the size of data communication between the ECU and the car navigation is increasing. An ECU may use Open Source Software (OSS). The ECU may also use virtualized software. The ECU may communicate to the internet or cloud directory. ECU software may be updated by internet.

By themselves, the pieces can be tested. The integration and many workflow paths can be very complicated and leave data under various loads—calling for different testing than code coverage.

The Other Side of the Equation: More People, More Time

We are test engineers. We need to prepare ourselves to strategize such kind of complicated automotive Software Testing tradeoffs between coverage and risk. Currently, there are too many developers compared to testers working for automotive system software.

I worked at Microsoft’s headquarters in Seattle. At that time on my team, more than 300 development and testing engineers worked together to develop Windows OS.

From my experience, I can say the team ratio is that there are many more developers compared to testers working on automotive software than the entirety of the Windows OS.

Interestingly, some companies are afraid to increase the size of the software test teams.

I believe that software engineers for the automotive industry need to prepare and improve their effectiveness and efficiency for production. Both developer-level unit testing skills are needed as well as integration level testing. Test Automation is key to increase productivity.

Summary

How much testing is enough? This is a question as old as software itself. How much gets covered in the test effort is a balance of risk, cost, and schedule. These days, with shift-left mindset of more developer testing, how much test coverage happens at each level is the new question.

To achieve the level of quality and testing needed for safety-critical applications, highly skilled engineers are needed for both developer level testing and all aspects of Test Automation.

Juichi Takahashi
Juichi Takahashi is CEO of LogiGear Japan. Prior to LogiGear, Dr. Takahashi held engineering and technical leadership positions in software testing and QA for prominent software/IT and product companies including Microsoft, SAP, and Sony. Throughout his career, he has gained extensive experience in testing and quality engineering. Dr. Takahashi is the author of the top-selling testing book in Japan, “Software Testing for Beginners.”

The Related Post

How to do UI test automation with the fewest headaches I’m currently interviewing lots of teams that have implemented acceptance testing for my new book. A majority of those interviewed so far have at some point shot themselves in the foot with UI test automation. After speaking to several people who are about to do ...
Source: From I.M.Testy (BJ Rollison’s blog) I just finished reading Implementing Automated Software Testing by E.Dustin, T. Garrett, and B. Gauf and overall this is a good read providing some well thought out arguments for beginning an automation project, and provides strategic perspectives to manage a test automation project. The first chapter made several excellent ...
Regardless of your current state of tools, building an effective Continuous Integration suite of significant automated regression tests is the key to moving to a higher level of confidence in today’s development world. In the evolution timeline of software development tools, new tools have recently proliferated. We have all been sold on collaboration, transparency and ...
Has this ever happened to you: You’ve been testing for a while, perhaps building off of a branch, only to find out that, after all of this time, there is something big wrong. It’s a bad build and now you have to go backwards, fix something, and get a new build. Basically, you just wasted ...
LogiGear Magazine – March 2011 – The Agile Test Automation Issue
Elfriede Dustin of Innovative Defense Technology, is the author of various books including Automated Software Testing, Quality Web Systems, and her latest book Effective Software Testing. Dustin discusses her views on test design, scaling automation and the current state of test automation tools. LogiGear: With Test Design being an important ingredient to successful test automation, ...
An automation framework is a way to organize your code in meaningful manner so that any person who is working with you can understand what each file contains. Automation frameworks differ based on how you organize your code – it can be organized based on your data, so that any person who wants to use ...
Cross-Browser Testing is an integral part of the Software Testing world today. When we need to test the functionality of a website or web application, we need to do so on multiple browsers for a multitude of reasons.
I recently came back from the Software Testing & Evaluation Summit in Washington, DC hosted by the National Defense Industrial Association. The objective of the workshop is to help recommend policy and guidance changes to the Defense enterprise, focusing on improving practice and productivity of software testing and evaluation (T&E) approaches in Defense acquisition.
Picture a series of sprints: There are a variety of features being developed, with an eye towards having automated tests related to those features. Work starts to move along and Test Automation work likewise goes along with it. However, at some point, there invariably is that moment, usually in the middle of the project, where ...
Two dominant manual testing approaches to the software testing game are scripted and exploratory testing. In the test automation space, we have other approaches. I look at three main contexts for test automation: 1. Code context – e.g. unit testing. 2. System context – e.g. protocol or message level testing. 3. Social context – e.g. ...
One of the basic challenges with test automation is adoption. I can’t tell you how many times I’ve cataloged licenses for a company and found out they already have many different automation software packages, none of which is being used. Traditionally I’ve been told that is because the tools don’t work and that the teams ...

Leave a Reply

Your email address will not be published.

Stay in the loop with the lastest
software testing news

Subscribe