The automotive industry, which can be viewed as lethargic at times, is undergoing a shift with the consumer electronics world having more of an impact on the purchase decisions. One of the areas which have seen substantial growth, with increased importance from manufacturers, is the in-vehicle infotainment experience. Poor user experience, sparse feature sets and an abundance of software bugs used to be an acceptable evil for consumers, but this is no longer the case. Being able to effectively benchmark, target set and validate a pre-production solution is more important now than it ever was.
The ability to make informed decisions which impact on the final customer experience is vital. It is too easy to allow confirmation bias and preconceived viewpoints to dictate business decisions which will inevitably lead to poor performance.
Testing methodology choice is important and should be based on the current stage in development, the output needed, timeline and budgetary constraints. Outlined below is an overview of the core types of evaluation methods with a cost indication. Another point to consider is what parts of the user experience are being evaluated, broadly grouped into three domains:
- Functionality: does the system provide the features needed by the end user?
- Ergonomics: has the HMI approach considered the user’s tasks and environment?
- Usability: have the features and UI been implemented well?
| Evaluation method | Overview | Cost |
| High level expert UX evaluation | A “quick and dirty” assessment used to highlight product positioning and main concern points | Low |
| Detailed expert evaluation | A detailed assessment of all system components to outline improvement areas with detailed recommendations | Medium |
| Quick ergonomics assessment | An assessment which is used to highlight whether there are any major ergonomic issues to a design | Low |
| Detailed ergonomics assessment | Computer aided assessments which engineering specifications and outputs | Very high |
| Biometrics assessment | Quantifiable assessments used to provide indisputable data on physiological inputs | Medium |
| Consumer diary exercises | Longitudinal research which can last months to capture the consumer experiences at each touchpoint | Medium to High |
| Consumer concept testing | Robust validation of consumer preferences and take up rates to build business cases against | Low to Medium |
| Co-creation with consumers | Immersive workshopping sessions where clients and consumers interact for rapid prototype development | Medium |
| Depth interviews | An immersive one-on-one consumer interview to explore consumer reactions to ideas, concepts or new innovations | Low |
| Consumer behavioural observations | A holistic view of how consumers behave within an environment (such as a dealership) | Medium |
To maximize research budgets the right tool must be used at the right time and the decision of when not to use a methodology must also be considered. Some common research methodologies are explained below taking into account what they are, how they are carried out and when they should be used.
Quick ergonomics assessments
What is it? An assessment by an ergonomics expert to validate whether a solution (from outside to inside the car) has any major concerns and what they could be. This evaluation takes the form of a technical analysis of physical access to the vehicle’s HMI hardware, rated by the expected level of customer satisfaction for users in the 5th to the 95th percentile.
When to use it? To benchmark a current solution on the market which has concerns or to evaluate a pre-production design. It can be used to provide further depth and supporting evidence to the usability of vehicle’s HMI. It provides a perspective that is separate from the usability of the solution’s information architecture and provides analysis and insight with regard to packaging and layout. It can also reveal common issues that remain a challenge to overcome, or unique mistakes that provide “lessons learned” for other OEMs.
When to not use it? When a detailed, and expensive, computer simulated ergonomics assessment is needed. It should be noted that an “quick” ergonomic assessment should not be used on its own and should not be used to provide a definitive, clear-cut answer to why usability is considered particularly good or bad for a particular solution. Its use should be as a supplementary element of a larger usability evaluation, or to benchmark similar vehicles against each other from a pure ergonomic point of view.
How is it carried out?
Evaluation covers the natural user journey for interactions with their car; from vehicle entry, to getting into a comfortable seating position, static use of HMI, dynamic use of HMI and finally exiting the car.
Ratings are applied considering the condition of the HMI being evaluated against the tester’s stature and whether or not they encounter physical difficulty operating that control. That performance is then considered against general user expectation for that vehicle in its segment; for example some sports cars may come with inherent difficulties but that is to be expected for that type of vehicle and would not be marked-down unfairly.
The evaluations are split into individual line items that cover common, over 30 everyday use cases such as:
- Unlock the doors
- Open the boot
- Operate the door handle/release mechanism
- Move the seat to a comfortable position
Example high level feedback report
Cost effective consumer viewpoints
What is it? Short, quick and cost-effective consumer feedback sessions. This approach is the opposite of complicated longitudinal research methods such as ethnography (the study of users in their own environment through expert observation) where the research findings capture the consumer viewpoint for immediate input.
When to use it? Always. End consumer viewpoints must be included in any product development to increase the chance of success.
When to not use it? When the methodology to be used will not provide a true picture of what the end user will experience. There are many methodologies which may appear perfect from a purely academic perspective but will provide misleading results. This in turn provides erroneous data to decision makers and impacts on any business decisions.
How is it carried out?
Various approaches need to be carried out at different stages of development. Here are three such stages and methods:
Vision creation and user needs: Consumer diary exercises: To capture the daily lives of the customer, and their regular interactions with product touchpoints, using moderation to dig into how those touchpoints are impacting customer perception.
Proof of concept validation: Consumer concept testing: Best used for measuring and validating customer preferences for a product/feature. This enables detailed profiling on who the prospective customers are, how many of them are likely to buy, at what price, all to aid business decision making.
Fast prototyping design: Co-creation with consumers: Immersive and multi-session workshops whereby the consumer is invited to test and feedback to a prototype in the morning, the feedback is then acted on over lunch/break, then retested in the afternoon to rapidly advance a prototype’s development.
Further consumer study examples
Biometrics testing
What is it? Evaluations carried out using biometric equipment to understand human emotions, behaviours and what they are focused on.
When to use it? When a more objective and quantifiable evaluation methodology is needed. This can be to quantify the level of driver distraction of a solution or to understand where the main frustration or excitement comes from an infotainment system, companion app or website.
When to not use it? When a “quick and dirty” methodology can be carried out. Biometrics evaluations take time, specialist equipment and detailed analysis. Because of the data which can be extracted from these studies the value for money is high however the amount of investment needed is also high.
How is it carried out? Test respondents are setup with equipment which measure various inputs with the most common being the following:
- Eye tracking: used to understand where the respondent is looking, for how long and how many times they revisit an area. Perfect to calculate eyes off road metrics.
- Galvanic Skin Response (GSR): used to understand the intensity of an emotion or the amount of stress being experienced. Can be used to understand whether specific use cases carried out or conditions have increased positive or negative sentiment.
Pupil dilation: When taking into the correct context measures such as pupil dilation spikes can be used to provide an indication of cognitive load. Can be used to quantify the cognitive load for driver interactions.
Authors:
Adam O’Shaughnessy,
Adam leads the User Experience testing practice at SBD and authors the industry leading monthly “UX Evaluation” series of reports which objectively evaluates the latest vehicles and systems to create a comparative view of the infotainment marketplace. He has worked with more than 30 automotive OEM clients (and tested 50+ unique systems) on studies ranging from expert testing, to consumer usability and focus groups, to dealership activation, on-boarding, and training initiatives
Sasangk Sarmah
Sasangk is the Operations Head of SBD India office at Bengaluru. He drives all Connected Car related research initiatives and authors one of the most comprehensive report in connected car forecast that tracks some of the key technology trends in over 13 regions. Sasangk has led several strategically important projects for automotive OEMs in India on topics ranging from market sizing to forecast to consumer research.
Published in Telematics Wire
