Market trend towards SW-defined car has changed rules and boundary conditions for mobility business. SPACE – Software and Services, Personalization, Autonomous Driving, Connected and Electrified Cars is transforming the traditional business models. The E&E architecture of modern cars is changing to keep pace with the market expectations. From domain centric vehicle architecture the trend is towards vehicle centric E&E architecture and vehicle cloud computing (Fig. 1). The global vehicle production is not expected to increase significantly year-on-year, even by the pre-COVID estimates. Therefore, SaaP and data-based services (XaaS) will capture significant portion of the business turnover. In Software engineering space trends like decoupling of SW and HW, OTA, and DevOps are becoming more common.
Figure 1 : Evolution of Vehicle E&E Architecture
The Automotive Software Verification and Validation ( V&V ) is also significantly evolving to meet the changing customer expectation, agile development, demand for faster time to market, and pressure to reduce R&D costs. There is a huge focus on front –loading and continuous deployment of software.
In this article we will examine major trends influencing automotive SW V&V today and evolution of solutions.
The major trends in automotive SW V&V currently are depicted in figure 1.
Figure 2 : Trends in Automotive SW V&V
Standardization:
There is ever increasing demand for standardization of V&V process, methods and tools. On one hand customers are demanding compliance to Test Maturity Model Integration ( TMMI ), ASPICE and the like. On the other hand, test strategies have to be defined keeping in mind functional safety (ISO 26262), SOTIF and country specific ENCAP and RDE regulations. We also see development of new standards by organizations like VDA, SAE and ISO. A typical example is the ongoing standardization of Software-In-Loop process and methods by VDA which is supported by many Auto-OEMs and Tier1s. The initiative is led by Bosch. In general standardization helps the business immensely and is supported by major industry players.
Virtualization:
There are certain situations which are very challenging or expensive to validate in field or in Hardware-In-Loop ( HiL ) systems. Eg. : Miles accumulation for autonomous driving, real driving emission (RDE) or near crash scenarios. Virtualized validation using Software-In-Loop ( SiL ) is an answer to this challenge. It can help in faster-than real time simulation, cost-saving by enabling front-loading of projects, helps test development in virtualized environment and later validation in HiL. It saves on validation in expensive lab set ups and proto vehicles in later phases of the product development. There are different use cases. While SiL is not a complete replacement of HiL or vehicle validation, it reduces the dependency significantly.
But there are challenges. First and foremost: Mindset. How to convince embedded systems engineers to test is virtualized environment and convince project managers to add the extra-activity of virtual ECU ( vECU ) generation in the product plan ? Secondly, how to establish methods for qualification of the SiL tool chain and the SW product? Thirdly how to establish the SiL validation set up ? It requires creation and integration vECUs with middleware along with plant, environment and sensor models from different suppliers. And then run the tests seamlessly with a test automation framework. Later the whole set up becomes part of a DevOps pipeline. Additionally there is need for IP protection.
Significant work is happening across the automotive industry to solve this puzzle. VDA initiative to standardize SiL is one such example. Many OEMs have already established component SiL test setups and using them for V&V. There are also agreed methods to deliver certain SW requirements with only SiL testing. Test coverage can be achieved by testing in SiL and HiL environments together. Reference architectures are being built to operate and deliver SIL both on premise and on cloud. Current efforts are on to build a System-SIL to test multiple vECUs in a system-of-system testing environment.
Automotive DevSecOps
While vehicle architecture is evolving towards increasing centralization, the need for SW V&V does not stop at start-of-production (SOP). Introduction of new features, new traffic regulations, system updates, security updates, troubleshooting etc. requires V&V for the new SW releases even after production. The traditional customer value creation cycle with slow feedback mainly through manual interaction is being replaced with a data enriched and faster automatic feedback cycle. There is rapid continuous improvement through data driven, agile feedback cycles. With the systems being connected, the cyber-security threats are ever-increasing and strategies need to be in place for a continuous monitoring and protection.
So the traditional waterfall model is b replaced with a continuous-integration-continuous-testing-continuous-delivery-continuous-deployment model, in short ContinuousX. Implementation of a Continuous testing strategy is essential for the success of this model.
DevOps is a composition of enhanced “engineering” practices that reduce lead time and increase the frequency of delivery. The primary goal of DevOps is to ensure Operations team members are engaged and collaborating with Development from the very beginning of a project / product development. DevOps is a cultural shift that merges operations with development and demands a linked toolchain of technologies to facilitate collaborative change. It requires pushing past departmental lines for more effective planning, design, and release of projects / products. DevSecOps is a strategy that extends DevOps efficiencies to software security. ( src. : Building a DevSecOps Culture – from a Technical Perspective – Tech at GSA )
This approach really blurs the traditional boundaries of development and testing in SW development. The focus is on feature ownership in a team and frequent deliveries. All the major automotive OEMs have initiated this journey and Tier1s are following suite.
Data Driven Engineering
We have seen the transformation of the traditional value creation cycle into a data driven fast feedback cycle. This opens up the enormous opportunities to establish a data driven lifecycle in mobility. Here we are referring to mostly two types of data:
- Data available from current and past SW developments. It can be requirements documents, source code, test cases, test reports, other reports, executables etc.
- Data generated from field during a product’s life cycle and is incorporated into the product development for new feature creation and verification.
We see a huge push to use the first type of data to deliver quality software. This data can be used to perform risk based testing and defect prediction, static code analysis with code patterns, automated data labeling and identification of edge scenarios from years of field acquired data. Also generation of synthetic scenarios, combining real and synthetic scenarios are being used more and more to catch defects at an early stage of development.
The field-recorded data on the other hand is used for both pre- and post-SOP activities. In pre-SOP phase data is used for design optimization, remote validation and field validation during ramp up. Post-SOP data is used for field quality management and product performance enhancement.
Artificial Intelligence and machine learning techniques are used to make use of data for creating insights during development and operations.
However there are challenges in bringing these methods to mainstream. The existing deterministic methods are not easy to replace. But industry is already experiencing early successes of data driven techniques in very traditional activities like static code analysis and power train calibration.
Here our suggestion will be to go for use-case driven approach and establish early success stories.
Please note, in this article we have rather focused on use of data and intelligent techniques for process and methods of V&V. The validation of cognitive systems is a huge subject in itself and needs to be addressed separately.
Intelligent Automation
Virtualization, DevSecOps and data driven engineering requires high level of automation to derive their true potential. Test automation has been well established in automotive SW validation for many years. The current trend is to automate not just the test execution but the entire test lifecycle. One of the major trends that we notice is to combine an intelligent core with a domain wrapper to derive full potential of the solution.
Let us elaborate a very specific use case for validating an automotive product. Here we have the challenge of high manual effort to perform unit test and schedule pressure to perform test coverage. The core of the solution is an AI engine using natural language processing and image recognition algorithms. This engine is able process the requirements documents consisting of text and images and generate high level test specifications. Then a domain wrapper, a post-processing script, is run to apply domain specific test methods on the high level test specifications. Lastly, as test script generator is used to create the final scripts which are run on the target system.
Figure 3 : Automated Requirement-to-Test Case Generation pipeline
The initial training effort of the model was the biggest challenge in this solution. But once done, the productivity improvement is really high and long lasting.
System-of-Systems Validation
The INCOSE Systems Engineering Body of Knowledge (SEBoK) states that:
“The term “system of systems” (SoS) is commonly used, but there is no widespread agreement on its exact meaning, or on how it can be distinguished from a conventional system.”
This is an accurate description of the state of understanding of the term. Despite this there are useful descriptions of SoS such the ISO/IEC/IEEE 15288 Annex G definition:
“A system of systems (SoS) brings together a set of systems for a task that none of the systems can accomplish on its own. Each constituent system keeps its own management, goals, and resources while coordinating within the SoS and adapting to meet SoS goals”.
The US Department of Defense defines a SoS as:
“An SoS is defined as a set or arrangement of systems that results when independent and useful systems are integrated into a larger system that delivers unique capabilities”
All the above definitions talks systems from operations context. The human driven non-networked vehicles in that sense are not system-of-systems, even though they are very complex. The vehicle is driver operated and not typically supported directly during operation by either the component developing organizations, the OEM or a mobility provider. In the context of autonomous vehicle operations, we will fulfill many of the characteristics of SoS discussed above. A networked vehicle providing or using information from V2X, backbone services and perhaps satellite systems in conjunction with networked, embedded systems fulfills the criteria for geographically distributed and independently operated systems.
The validation of such systems require a different approach. First we identify the Device-Under-Test ( DUT ). This can be a complete vehicle E&E network or a set of subsystems delivering a specific function (eg. Lane Keep Assist or Parking Assist functions).We follow a test scenario based validation for a system-of-systems. The scenario libraries are used as reference and parameterized for a specific test case. The scenario creation is enabled by data driven engineering techniques described previously.
Field Based Validation: An Integrated Validation Approach
While major industry trends for Automotive SW V&V are evolving, an integrated solution approach is also being developed to handle the challenges of SPACE – Field Based Validation.
Field based Validation is “Validation of products utilizing information and experience from the target environment where a product is usually being used and under conditions which a product is usually being used. This allows Whitebox tests as well as Blackbox tests and is in addition a tool for proving evidence of specific V&V criteria in the field.” – Robert Bosch Center of Competence for SW V&V
Multiple use cases have been developed and is being practiced:
• Continuous System Improvement, including recording + re-insertion of additional situations for simulation
• Proving quality criteria / metrics for system release
• Runtime Safety Monitors
This method captures the essence of Automotive DevSecOps using virtualization and data driven engineering, where the data captured from real vehicles in field is fedback into the development system. This data in turn is used to create test scenarios which are used to verify SW in a virtualized environment.
Figure 4 : Field Based Validation
Conclusion
As businesses are getting disrupted in many ways, it is necessary for V&V practitioners to evolve test strategies and incorporate new techniques to handle business challenges. A V&V engineer should continuously upskill oneself to become a full-stack engineer. In this way organizations will be able to deliver quality products and solutions under challenging circumstances. After all “Fight for good software is not a sprint, but a marathon!” – Dr. Stefan Hartung, Head of Bosch Mobility Business Sector.
Author
Anupam Gupta
Head of Engineering and Lead, CoE, Verification & Validation
Robert Bosh Engineering & Business Solutions
Published in Telematics Wire
