Pilot experiment - 2

The Dataset

A lot of work have been done since the last update. First the idea of averaging across the subjects was actually not that good. In fact we were defining an "average subject" and using this average user for the training and the test which can explain that the KNN algorithm achieved quite good performance.
An other averaging method was to average the data for each subject over the 10 repeats then we had 11(actions) * 5(subject) = 55 data. This approach is much more realistic and allows us to perform user independent analysis but the problem is that we have now very few data. 

Finally I decided to not average the data and then I used 11(actions) * 5(subject) *10 (repeats)= 550 data. This data are separated into training (4subjects = 440 data points) and test set (the remaining subject 110 data points) our analysis is then user independent. 

Features

I define new features for each sensor and each action:

• The number of sensor’s value changes
• The number of peaks 

With this two new features I add 10 attributes to my problem, therefore I have 15 attributes for each data points. All these attributes are normalized.

Data Analysis

 I ran three different algorithms:

•  A bayesian neural network written by Hans Bruun Nielsen at DTU and based on MacKay [1] article.I uses a 10-fold cross validation to choose the number of hidden units (the number of hidden unit is 1). I reached an accuracy of 33%. The number of hidden units used for each subject is 18, 17, 10, 9 and 20 (20 is the maximum tested). And the accuracy for each subject is 33, 44, 29, 26, and 30 %.
  We can see (below) here that the class 7 has a precision of 1 but a very small recall (we predict it only three times). We also see that the classes 8 and 9 perform very well compared to the others.

	class 1		class 2	class 3	class 4	class 5	class 6	class 7	class 8	class 9	class 10	class 11	macro average	Standard deviation
precision	0.282		0.688	0.474	0.308	0.423	0.148	1.000	0.851	0.643	0.176	0.124	0.465	0.296
recall	0.220		0.220	0.180	0.240	0.220	0.180	0.060	0.800	0.720	0.460	0.280	0.325	0.235
F-Score	0.247		0.333	0.261	0.270	0.289	0.162	0.113	0.825	0.679	0.254	0.172	0.328	0.221

•  A Support Vector Machine using LIBSVM written by Chih-Chung Chang and Chih-Jen Lin. I didn't use any cross validation since I'm not setting any parameter. I achieved an accuracy of 33% .  And the accuracy for each subject is 35, 50, 23, 30, and 25 %.
  
  As we can see (below) the class 6 (long turn left) performs a bit more badly. One the other hand as expected the class 8 and 9 (lean forward and sit backward) achieve here again very good results.

	class 1	class 2	class 3	class 4	class 5	class 6	class 7	class 8	class 9	class 10	class 11	macro average	Standard deviation
precision	0.200	0.243	0.250	0.300	0.283	0.133	0.286	0.690	0.576	0.342	0.133	0.312	0.173
recall	0.180	0.180	0.120	0.300	0.260	0.080	0.240	0.800	0.680	0.520	0.220	0.325	0.236
F-Score	0.189	0.207	0.162	0.300	0.271	0.100	0.261	0.741	0.624	0.413	0.165	0.312	0.203

•  An Infinite Gaussian Mixture Model using variational inferences that I wrote by myself using the method described in Bishop [2]. In order to make it a classifier I trained one model using the data of only each class at a time (in the training set) and then I looked to which model the tested data is more likely to belong to. So for each subject I have to train 11 models. I achieved an accuracy of only 10.5% (when chance is around 9%) which is very poor. I think that the explanation it that  I have only 44 data points to train each model which is very few. With more data points this method could be much more efficient.

 References

[1] A Practical Bayesian Framework for Backpropagation Network, D. J.C. MacKay, 1992

[2] Pattern Recognition and Machine Learning, C. M. Bishop, 2007

External Links 

 LIBSVM