Further work - 1

In addition to the work with the sensors, my main task is now to begin designing the user study. I have to find some way to frustrate the driver. I already discuss some ideas with Ronnie, some were goods other less. The most important is that I have to be able to easily implement the task and the frustration has to be related to the driving experience and not to the software.

I will also continue to read some papers, trying to find more information about the relation between body posture on a seat and emotion.

In the same time I will try to look a bit more the weka library (open source library developed in Java) as advised by Jainlong.

External link

Weka

Sensors - 1

In order to collect a maximum amount of data, different kinds of sensors will be used for the experiment (some of them will not be used for the analysis, at least non for mine). 

The sensors have to respect some constraint, the most important is that they have to be non-intrusive. We have to keep in mind that our system could be used in a normal (future) car and works for anybody. Then everything has to be integrated in the car (for a low cost) without any constraint for the user.

Then the sensors, I will use are:

• eye tracking system (can be included in the dashboard). 

• affectiva q-sensor: which provides skin conductance, temperature and movement of the arm data. It is the most intrusive sensor that we have since driver need to wear a wristband.

• seat sensors: including pressure and position sensors, it is part of my project to choose how to use them to track the position of the driver. These sensors will be integrated in the seat or the dashboard.

• we may use other sensors as  pedal pressure sensor, microphone, ...

• the experiment will be recorded but the video is not part of the machine learning analysis.

Work on the sensors

• I "played" with the seat sensors to see how to use it and how hey react. I find out that the distance sensors are not robust when the object is far away (20-30cm) even if the official working range distance is 4-30 cm.

• I wrote a small program using the phidgets API in order to record the data generated by the seat sensors. The program is working (a demo has been done to Ronnie) but I will improve it in the next weeks to make it more efficient.

Further work

• Improve the acquisition program for example make it robust to crash.

• Find out how to integrate the sensors in the seat (or the car) in order to collect the most relevant data.

External links

Affectiva sensor 

Pressure sensor

Distance sensor

Literature review - 1

Since I do not have any particular knowledge in cognitive science. My first goal was to learn about this domain. So I read a lot of scientific papers about emotions; Which physiological or behavioral signals we can use, how to analyze these signals, how we can condition someone to make him feel a specific emotion... Obviously, I mainly focused my researches on frustration and drivers mental state but also on the body posture (since my machine learning analysis should focus in priority on the driver's position on the seat).

The first article justifies that we care about drivers' frustration. In this paper, Shinar [1] analyzes the frustration-aggression model introduced by Dollard et al. [2] and some other studies models which states that drivers' aggression are more related to the driver's culture and establish that most of the "real" aggression are due to the driver's frustration.

Several interesting papers dealing with frustration monitoring are written in the context of learners using AutoTutor (for examples: Kapoor et al. [3] and D'Mello et al. [4]). Even if they do not speak about drivers, these papers are really useful for me, because they show some methods used to detect the learner's frustration with some sensors very similar to the one I will use. In particular, [4] used the body posture of the learner on its seat to predict his mental state.

Other papers are dealing with drivers' frustration but not with body posture. Bořil et al. [5] use only a speech analysis (not sure that I use this kind of data) to predict drivers' mental state on a simulator (as in my case). While Malta et al. [6] deal with pedal actuation, Electrodermal activity (EDA) (I could use this one) and speech recognition.

In all these papers the results show that we could be able to efficiently monitor the drivers mental state in real life.

References

[1] Aggressive driving: the contribution of the drivers and the situation, D. Shinar, 1998

[2]  Frustration and aggression, J. Dollard, L. W. Doob, N. P. Miller, O. H. Mowrer, R. R. Sears, 1939 

[3] Automatic prediction of frustration, A. Kapoor, W. Burleson, R. W. Picard, 2006

[4] Automatic detection of learner's affect from gross body language, S.D'Mello, A. Graesser, 2009

[5] Analysis and detection of cognitive load and frustration in drivers’ speech, H. Bořil, S. O. Sadjadi, T. Kleinschmidt, J. H. L. Hansen

[6] Analysis of real-world driver’s frustration, L. Malta, C. Miyajima, N. Kitaoka, K. Takeda

External Link

AutoTutor

Project Starts !

I'm starting my 5 months internship at NICTA (Sydney, Australia) as member of the Machine Learning group. I take part in the Driver Mental State Monitoring (DMSM) project. My task is to focus on the frustration, I will design the experiments to collect the data and then I will do the machine learning analysis in order to elicit the drivers' frustration.

This master thesis is part of my MSc. in Computer Science and Engineering (study line: Efficient and intelligent software) at Danmarks Tekniske Universitet (DTU). I will be supervised by Morten Mørup (DTU), Ronnie Taib (NICTA) and Jianlong Zhou (NICTA).

 

Project's Description

The Driver Mental State Monitoring (DMSM) project is a research collaboration between NICTA and Germany’s Fraunhofer research institute. It focuses on the real-time assessment of drivers’ mental states through behavioural and physiological inputs, in view of improving user experience, safety and training for the automotive industry and end-users.

The project designs tasks and scenarios to elicit specific mental states, for example giving unachievable goals to a driver creates frustration. Using a driving simulator and virtual environment, user studies are carried out to collect physiological and behavioural response from the users subjected to these tasks. The objective of this training period is to analyse the data streams and establish correlations with the designed mental states. The student will learn how to pre-process data, which may include sources as varied as eye tracking, speech, galvanic skin response, posture (through body position) etc.

Using machine learning, the student will create models of interaction, which can then be used to assess the mental states in real-time.

Expected Outcomes: pre-processed data; data features determined; models of some physiological/behavioural responses produced.

Structured Training / Study Plan

Week 1-3

Introduction to the DMSM project, physiological and behavioural signals.

Week 4-6

User study design, including a task addressing a specific mental state (to be defined, but potentially frustration). At the same time: introduction to classical machine learning methods.

Week 7-10

User study implementation, including instrumentation of the simulator, software development/adaptation to include the new task, and documentation of the methodology. At the same time: introduction to advanced machine learning, e.g. non-parametric methods.

Week 11-12

User study being run, including pilot, and data collected.

Week 13-16

Data pre-processing, feature definition and extraction.

Week 17-20

Machine learning analysis, construction of models of interaction.

Week 21-22

Documentation of findings, presentation to research team.

External links:
DTU
NICTA