Driver’s AI Assistant for Connected Vehicle

A lot of work has been done in the field of Autonomous vehicle. Combination of various technical strategies are put in place to enable the same. One of the commonly used strategy is having an in-vehicle processing capability using a powerful computer surrounded by various sensors installed in the vehicle to sense the environment of the vehicle. 

In order to have a better knowledge of these surroundings and improved navigational capability the vehicle is also connected with a satellite and can communicate and exchange information with other vehicles. All this information is processed in real time by the on-board processor to provide autonomous capability to the vehicle.

Though the research will continue to make autonomous vehicles more intelligent having enough capability to drive in any type of surrounding, for all practical purposes we cannot ignore the importance of drivers driving various categories of vehicles. In fact they are the ones who have done the job for us till now and will keep on doing the same for years to come.

Therefore, it is equivalently important to improve technology to assist drivers for better driving experience and to improve overall safety. As per various studies, we can list down few of the many reasons for an accident:

• Drunk driving
• Mobile phone
• Speeding 
• Weather conditions
• Driver fatigue

These can be addressed up to a great extent through an intelligent connected vehicle with a combination of information received through satellites, vehicle’s current and surrounding geographical locations, virtual assistant to communicate with the driver, and audio notifications and in-cabin monitoring of the driver. 

The enablement of various features to assist drivers and improve driver’s experience is explained below:

• Virtual Assistant: In most of the commercial vehicles, normally, the drivers travel alone without company. During long journeys it becomes a monotonous and a boring job for them. This tempts the driver to use their mobile phones to check incoming messages and notifications or for communicating with someone. Hence, providing a virtual in-vehicle assistant to drivers to interact will give them the sense of companionship during their journey. 

This virtual assistant can also be connected to the vehicle diagnostics to provide vehicle health to the driver as an when required. The important checks such as vehicle engine status, air pressure, filter oil, door locks, and seat belt can be communicated to the driver. All the communication to/ from the driver will be controlled and routed through it. Having AI enabled capabilities, this assistance will improve over a period of time and will develop itself into a unique personality aligned to driver’s interests. 

Having a voice enabled driver virtual companion may reduce driver’s distractions due to use of mobile phones, thus reducing the possibility of accidents.

This virtual assistant can also help driver to remain connected to the base location, therefore any eventuality can be communicated back to them immediately to channelize assistance, if required.

• Connected Vehicle: A connected vehicle will let the vehicle’s driver to be aware of the geographic conditions of the terrain where he is currently driving. Since the centralised systems have plethora of data regarding in-route weather conditions, traffic congestions, accidents; the information can help the drivers to make informed decisions, whether to take a break or do a route diversion to avoid unfavourable weather conditions depending on the type of cargo and the criticality of reaching at the destination.

The information of the vehicle being driven at any road based on its current geo-location and the applicable speeding restrictions on that road can help to apply restrictions on the driver to not breach the maximum speed at the road. This will enable variable maximum speed allowed for a vehicle depending on which road the vehicle is being driven.

The vehicle will only have a limited amount of information and computing capability, however the centralised systems will have enormous data and the processing capabilities. This will help in providing the connected vehicles with computed information thus extending and assisting their capabilities enormously.

It can also facilitate intra-vehicle communication with nearby vehicles which can assist each other, if required.

• In-vehicle monitoring: Another feature to be considered in connected vehicle is in-vehicle monitoring by installing camera to monitor the driver. In order to protect privacy, the telecast of the video should be restricted; instead the processing of the captured images/ video stream should be done locally and only the computed result should be sent to the central system. 

The deep enforced learning video analytical algorithms applied on the in-vehicle video feed can be used to understand the driver’s physical state related to fatigue and also to analyse whether the driver is drunk or not.

The driver’s visual to:

• Lack of co-ordination
• Loss of balance
• Drowsiness 

could be analysed to ascertain whether the driver is drunk or not. A confidence score of 90% or above relating to the said physical conditions can be used by the virtual driver assistant to restrict the driver from driving. It can achieve this by locking the driver to mobilise the vehicle and/ or inform the centralised system about the physical state of the driver.

The system will perform fitness level verification of the fitment of the driver for driving the vehicle. Below are two examples showing the fitness verification of a driver based on his/her physical condition.  

Any continuous efforts to attempt to drive the vehicle can also be reported to concerned authorities by the on-board computer. 

Managing driver’s fatigue is also very important for ensuring safety and security of the driver and others. It is one of the main reasons for accident as well. Transport authorities of various countries across the globe take driver fatigue very seriously and any failure to comply results in heavy fines to driver and the organisation operating vehicles.

The fatigue of the driver can be identified by following physical symptoms:

• Drowsiness
• Unable to keep eyes open
• Yawning
• Shaking head

These physical symptoms can also be identified by analysing the video feed of the in-vehicle camera by vehicle computer to notify the driver to take appropriate breaks. With the available geo-location information, it is able to suggest the driver of the near-by locations suitable for taking rest.

In countries like Australia, the drivers must observe the NHVR (National Heavy Vehicle Regulation) compliance fatigue rules. Every Australian state has its own set of rules, and it is mandatory for drivers to comply the same and provide the evidence if asked by RTO (Road transport authority) failing which heavy penalties are applicable. 

For example, as per BFM (Basic Fatigue Management) rule applicable in NSW, Australia, conditions as per table (iii) must be satisfied.

Time	Work	Rest
In any period of	A driver must not work for more than a maximum of…	And must have the rest of that period off work with at least a minimum rest break of…
6 Hrs 15 min	6 hours work time	15 continuous minutes rest time
9 hours	8 Hrs 30 min work time	30 minutes rest time in blocks of 15 continuous minutes
12 hours	11 hours work time	60 minutes rest time in blocks of 15 continuous minutes
24 hours	14 hours work time	7 continuous hours stationary rest time
7 days	36 hours long/night work time	No limit has been set
14 days	144 hours work time	24 continuous hours stationary rest time taken after no more than 84 hours work time and 24 continuous hours stationary rest time and 2 x night rest breaks# and 2 x night rest breaks taken on consecutive days.

Despite the fatigue from driver to driver, these guiding principles are in place to address driver fatigue to the best possibility. This means that the in-vehicle assistant should be able to analyse fatigue symptoms, help in aligning fatigue management as per individual driver’s needs and advice driver accordingly so that the driver can take proper rest and is also in compliance to the state specified fatigue rules and guidelines.

Challenges and Considerations

Having discussed the features required in a vehicle for driver’s assistance to improve driver’s experience, safety and security of the driver, various challenges can arise which need to be addressed as well. These include and are not limited to:

• Connectivity: The features such as sharing real time geographic location, get accurate weather information, information of near-by places and vehicles require a continuous connection of the vehicle to central server with a proper connectivity. This means a continuous connectivity with appropriate bandwidth is required for optimal operation of connected vehicle. The upcoming advancement in telecommunication such as 5G will play a significant role in providing smooth connectivity to the vehicle to perform all the operations smoothly while moving.
• Operating Cost: Up to now the major operating cost of the vehicle is related to periodic servicing of vehicles, however, for connected vehicle offering the other costs related to connectivity, subscriptions to services such as weather information, navigation assistance, terrain maps, communicating to nearby vehicles, emergency assistance etc. will also form the part of operating cost expenses.
• Surrounding Noise: The voice enabled interaction capability need to be improvised further so that the drivers can interact with the virtual assistant in a noisy environment. The surrounding noise need to be filtered out so that it is able to accept driver’s commands without any extra efforts. The idea is to provide driver an experience of companionship within the vehicle driven by him/her.
• AI Improvement: The AI assistant capability also needs to be improved so that it can sense the driver’s mood and can strike communication. Instead of only working on the driver’s commands and fulfilling the request, it should have conversational capabilities and also learn and adjust as per interaction with the driver.
• Driver Monitoring: The vehicle needs to be fitted with in-vehicle camera so that it can monitor driver’s physical state and can notify in case of any change in physical behaviour of the driver. The vehicle will also have the capability to identify and segregate the condition of the driver from fatigue, influence of alcohol, illness and thus should be able to respond appropriately or request assistance. 

This may be achieved in practice by deep reinforcement learning. It is a fairly new technology which combines the famous Deep learning, which is a supervised learning technique engaging several layers of neurons, and the Reinforcement learning. Now, as we know deep learning methods can efficiently solve complex high-dimensional data such as images (engaging computer vision), with less manual feature engineering, it probably makes a perfect fit for the problem set.

On the other hand, reinforcement learning, follows the natural human behaviour of trial and error, learning from its previous experience. So, in this case, it guarantees an optimum algorithm over time, i.e., the algorithm will learn from its own history, taking actions to change the state based on a policy and provide better results, decreasing the probability of an accident with every iteration.

Tech Giants like Tesla, Google, and Uber etc., are also employing the same tech (Deep reinforcement learning) for their respective projects of Autonomous cars.

To summarise, with the advancement in connected vehicle space, and in-vehicle AI enabled assistance, the vehicle can have a personality and be able to assist the driver in performing duties more efficiently, thus significantly improving driver’s experience and safety.

Author:

Abhishek Kumar Arya is working as Practice Head with Tech Mahindra, and has more than two decades of experience in IT consulting, Enterprise Architecture establishment, technical architecture review and design, solution design, development and support of mission critical business applications. 

Published in Telematics Wire