Python Code Training sample1

Data Analysis Instruction
for EEG (g.tec), eye-tracking
(Tobii) and motion capture
(Perception Neuron)
H. Wagatsuma＠Kyutech
Instruction meeting for EEG team in Wagatsuma Lab@Kyutech
1
Started: 2020/1/6
Revised:2020/1/26

脳波データ説明資料
我妻研＠Kyutech
我妻研ミーティング(2020/1/6)
2

Protocols on EEG:
1. EEG data confirmation→ ch pos, time, subjects and so on
2. Epoch segmentations depending on events
• Tobii apps → movie with gaze maker → Human operation (get event
timing as H:M:S: ms）
• EEG verification with Tobii trigger （1/512; sampling rate）
• Estimate Tobii start timing (to get the subtraction time with g.tec
00:00:00 starting time
3. Noise removal （EEG lab or MCA）
4. Data analysis 1（averaging with all segments; ERP ）
5. Data analysis 2（FFT or others）
3

処理手順
1. EEG （脳波データ）確認→ ch配置、時間、実験協力者など
2. エポッキング（イベントごとのデータ区間切り出し）作業
• Tobiiの映像からのイベント時間推定（時:分:秒: ms）
• EEG データにおける単位時間との照合（1/512; sampling rate）
• Tobii記録開始時間とEEG記録開始時間の差を考慮した開始時間の計算
3. ノイズ除去処理（あとで確認）
4. データの統計処理（平均化）
5. 脳波解析（ERPなど）
4

1. EEG data confirmation
• → ch pos, time, subjects and so on
AutoCarExp_sub1_4122019_04_12_2019_13_44_02_0000.mat
AutoDrivingExpDay2_sub1_5122019_05_12_2019_13_36_09_0000.mat
AutomatedDrivingExp_sub1_4122019_04_12_2019_13_49_59_0000.mat
AutomatedDriving_sub2_4122019_04_12_2019_14_39_41_0000.mat
DriveSim_Subj2_26112019_26_11_2019_16_54_35_0000.mat
DriveSim_sub1_26112019_26_11_2019_12_54_33_0000.mat
DriveSim_sub1_26112019_26_11_2019_12_54_33_0001.mat
currentCFG_g.SCOPE_f47fe32382a59a51e350d6b98bbc5d17_0.cfg
AutoCarExp_sub1_4122019_
AutoDrivingExpDay2_sub1_5122019_
Data name should be
defined uniquely and
systematically for reading
with programming codes
Next page
5

1. EEG （脳波データ）確認
• → ch配置、時間、実験協力者など
AutoCarExp_sub1_4122019_04_12_2019_13_44_02_0000.mat
AutomatedDrivingExp_sub1_4122019_04_12_2019_13_49_59_0000.mat
DriveSim_sub1_26112019_26_11_2019_12_54_33_0000.mat
DriveSim_sub1_26112019_26_11_2019_12_54_33_0001.mat
AutoCarExp_sub1_4122019_
AutoDrivingExpDay2_sub1_5122019_
名前がばらばらだと解析用
プログラムで問題が発生す
るので統一する次ページ
6

1.1 EEG data definition (1/3)
7
DriveSim
AutoDrivingExp
Filename definition
+ _Subj* _5122019_05_12_2019
_13_36_09_0000.mat+
*:sub no.
(1,2,…)
or
gtec(MATLAB)auto
serial code
subheader
_Day* +
date

1.1 EEG EEG data definition (2/3)
8
AutoDrivingExp_Day1_Subj1_4122019_04_12_2019_13_44_02_0000.mat
DriveSim_Day1_Subj1_26112019_26_11_2019_12_54_33_0000.mat
Renamed files：
/Volumes/Elements/Automated_Driving_System_Sato_Lab_20191204_05_Data
Analysis/EEG

1.1 EEG データファイル整理(1/3)
9
DriveSim
AutoDrivingExp
ファイル名の定義：
+ _Subj* _5122019_05_12_2019
_13_36_09_0000.mat+
*:実験協力者番号
(1,2,…)
or
gtec(MATLAB)自動
割り当てシリアル番号
協力者番
号
ヘッダー
_Day* +
日付

1.1 EEG データファイル整理(2/3)
10
ファイル名変更後のリスト：
/Volumes/Elements/Automated_Driving_System_Sato_Lab_20191204_05_Data
Analysis/EEG

11
>> b={}
b =
0×0 empty cell array
>> b{1}='abc'; b{2}='dbc';
>> b{3}='kdjkasd'
b =
1×3 cell array
{'abc'} {'dbc'} {'kdjkasd'}
>> b
b =
2×3 char array
'abc'
'dbc'
>> b{3}='kdjkasd'
Unable to perform assignment because brace indexing is not
supported for variables of
this type.
NG matrix
OK matrix
Filenameの扱いにはcellを使う
Point!
cell is better for treating Filename(string )
rather than matrix.

How to connect g.tec recording files
12

How to get a file list: fread_Plus.m (1/2)
13
defaultPath=pwd;
DataFolderPath='/Volumes/Elements/Automated_Driving_System_Sato_Lab_
20191204_05_DataAnalysis/EEG';
fnlist=struct2cell(dir);
merge_f='merged';
cd(fullfile(DataFolderPath,merge_f));
close all;
flist = dir; rfields = {'date','bytes','isdir','datenum'};
flist=rmfield(flist,rfields);
tag=cell(size({flist(:).name})); tag(:)={'mat'};
flag=cellfun(@strfind,{flist(:).name},tag,'UniformOutput', false);
flist2 = struct2cell(flist(~cellfun(@isempty,flag)));
flist3=flist2(1,:);
Set of the Default Path to return the same place later
Extract
filenames
with ”.mat”
See detail explanation in the next page

15
flist = dir;
rfields =
{'date','bytes','isdir','datenum’};
flist =
78×1 struct array with fields:
name
folder
date
bytes
isdir
datenum
>> flist
flist =
78×1 struct array with fields:
name
folder
Nothing anymore!
Remove specific fields

16
tag=cell(size({flist(:).name}));
tag(:)={'mat'};
flag=cellfun(@strfind,{flist(:).name},tag,'UniformOutput', false);
flag =
1×10 cell array
Columns 1 through 7
{0×0 double} {0×0 double} {[55]} {[55]} {[55]} {[55]} {[55]}
Columns 8 through 10
{[50]} {[50]} {0×0 double}
'AutoDrivingExp_Day1_Subj1_4122019_04_12_2019_13_44_02.mat'
'DriveSim_Day1_Subj1_26112019_26_11_2019_12_54_33.mat'
'DriveSim_Day1_Subj2_26112019_26_11_2019_16_54_35.mat'
'figure'
>> flist(:).name
'.'
'..'
{3}
{1}
{2}
{4}
{5}
{6}
{7}
{8}
{9}
{1０}
{3}{1} {2} {4} {5} {6} {7}
{8} {9} {1０}
[55]
Position of “mat” in this sentence

17
flist3=flist2(1,:);
(~cellfun(@isempty,flag))
ans =
1×10 logical array
0 0 1 1 1 1 1 1 1 0
flist([3,4,5,6,7,8,9]).name
1 2 3 4 5 6 7 8 9 10
flist3 =
1×7 cell array
Columns 1 through 3
{'AutoDrivingExp_…'} {'AutoDrivingExp_…'} {'AutoDrivingExp_…'}
Columns 4 through 7
{'AutoDrivingExp_…'} {'AutoDrivingExp_…'} {'DriveSim_Day1_S…’} {'DriveSim_Day1_S…'}
1,2,3,4,5,6,7

Merge program:
18
Extract
filenames
with ”.mat”
defaultPath=pwd;
DataFolderPath='/Volumes/Elements/Automated_Driving_System_Sato
_Lab_20191204_05_DataAnalysis/EEG';
cd(DataFolderPath);
fnlist=struct2cell(dir);
merge_f='merged';
gtecLen=4; % gtec simlink automatically generated serial number
for connecting files like
%"temp_26112019_26_11_2019_16_54_35_0001.mat" with 4 digit at
last
if ~isfolder(merge_f); mkdir(merge_f); end
% 'folder' exist in MATLAB 2018
% flist = dir; rfields =
{'date','bytes','isdir','datenum','folder'};
flist = dir; rfields = {'date','bytes','isdir','datenum'};
tag=cell(size({flist(:).name})); tag(:)={'mat'};
flag=cellfun(@strfind,{flist(:).name},tag,'UniformOutput',
false);
flist3=flist2(1,:);
tag2=cell(size(flist3)); tag2(:)={'_'};
flist4=cellfun(@strsplit,flist3,tag2,'UniformOutput', false);

19
flist5={};
serialID={};
serialID_all={};
for i=1:size(flist4,2)
flist3{i}
tempCell=flist4{i};
flist5{i}=tempCell(1:end-1);
lastPos=size(flist5{i},2)+1;
lastItem=strsplit(tempCell{lastPos},'.');
flist5{i}{lastPos}=lastItem{1};
flist5{i}
serialID=cell2mat(flist5{i}(4:10));
serialID_all{i}=serialID;
end
flag4Connect=[];
tag2=cell(size(serialID_all)); tag2(:)={serialID_all{i}};
flag4Connect(i,:)=cell2mat(cellfun(@strcmp,serialID_all,tag
2,'UniformOutput', false));
end
cd(defaultPath);
See detail explanation in the next page

20
flist5={};
serialID={};
serialID_all={};
flist3{i}
tempCell=flist4{i};
flist5{i}=tempCell(1:end-1);
lastPos=size(flist5{i},2)+1;
lastItem=strsplit(tempCell{lastPos},'.');
flist5{i}{lastPos}=lastItem{1};
flist5{i}
end
i=1
>> flist3{i}
'AutoDrivingExp_Day1_Subj1_4122019_04_12_2019_13_44_02_0000.mat'
tempCell =
1×11 cell array
{'AutoDrivingExp'} {'Day1'} {'Subj1'} {'4122019'} {'04'} {'12'}
{'2019'} {'13'} {'44'} {'02'} {'0000.mat'}
lastItem =
{'0000'} {'mat'}

21
flist5{i}
=
{'AutoDrivingExp'} {'Day1'} {'Subj1'} {'4122019'} {'04'} {'12'}
{'2019'} {'13'} {'44'} {'02'} {'0000'}
serialID =
'412201904122019134402'
1 2 3 4 5 6
7 8 9 10

22
flag4Connect=[];
tag2=cell(size(serialID_all)); tag2(:)={serialID_all{i}};
flag4Connect(i,:)=cell2mat(cellfun(@strcmp,serialID_all,ta
g2,'UniformOutput', false));
end
>> flag4Connect
flag4Connect =
1 0 0 0 0 0 0 0 0 0 0
0 1 0 0 0 0 0 0 0 0 0
0 0 1 1 0 0 0 0 0 0 0
0 0 1 1 0 0 0 0 0 0 0
0 0 0 0 1 0 0 0 0 0 0
0 0 0 0 0 1 0 0 0 0 0
0 0 0 0 0 0 1 1 0 0 0
0 0 0 0 0 0 1 1 0 0 0
0 0 0 0 0 0 0 0 1 1 1
0 0 0 0 0 0 0 0 1 1 1
0 0 0 0 0 0 0 0 1 1 1

23
>> flist3{:}
'DriveSim_Day1_Subj1_26112019_26_11_2019_12_54_33_0000.mat'
1 0 0 0 0 0 0 0 0 0 0
0 1 0 0 0 0 0 0 0 0 0
0 0 1 1 0 0 0 0 0 0 0
0 0 1 1 0 0 0 0 0 0 0
0 0 0 0 1 0 0 0 0 0 0
0 0 0 0 0 1 0 0 0 0 0
0 0 0 0 0 0 1 1 0 0 0
0 0 0 0 0 0 1 1 0 0 0
0 0 0 0 0 0 0 0 1 1 1
0 0 0 0 0 0 0 0 1 1 1
0 0 0 0 0 0 0 0 1 1 1
1
2
3
4
5
6
7
8
9
10
11

24
flag4ConnectE=flag4Connect-eye(size(flag4Connect));
checkList=sum(flag4Connect);
checkListE=sum(flag4ConnectE);
for i=1:length(checkList)
newDataName0=makefname(flist5{i}(1:10));
newDataName=[newDataName0,'.',tag{1}];
if checkListE(i)==0
disp(['one file: ',num2str(i)]);
load(fullfile(DataFolderPath,flist3{i}));
ynew=permute(y,[1,3,2]);
size(ynew)
save(fullfile(DataFolderPath,merge_f,newDataName),'ynew','SR');
ynew=[];
elseif checkListE(i)>0
disp(['multi files: ',num2str(i)]);
disp(checkListE);
y2new=[];
cID=find(flag4Connect(i,:)>0);
for j=1:checkList(i)
fnum=j-1; snum=num2str(fnum);
s2num=[repmat('0',[1,gtecLen-length(snum)]),snum];
fname_tmp=[newDataName0,'_',s2num,'.',tag{1}];
load(fullfile(DataFolderPath,fname_tmp));
display(fname_tmp)
display(flist3{cID(j)});
if ~strcmp(fname_tmp,flist3{cID(j)}); disp('error!!'); end
checkListE(cID(j))=-1;
y2new=[y2new,permute(y,[1,3,2])];
end
ynew=y2new;
size(y2new);
ynew=[];
end
end
Files to be
connected
This is
single file

25
flist3{i} =
i=1
size(y)
ans =
18 1 102753
size(ynew)
ans =
18 102753
permute(y,[1,3,2]);
1 2 3
1 2
Difficult to plot in the graph
easy to plot in the graph

26
flag4ConnectE=flag4Connect-eye(size(flag4Connect));
checkList=sum(flag4Connect);
checkListE=sum(flag4ConnectE);
for i=1:length(checkList)
newDataName0=makefname(flist5{i}(1:10));
newDataName=[newDataName0,'.',tag{1}];
if checkListE(i)==0
disp(['one file: ',num2str(i)]);
size(ynew)
ynew=[];
elseif checkListE(i)>0
disp(['multi files: ',num2str(i)]);
disp(checkListE);
y2new=[];
cID=find(flag4Connect(i,:)>0);
for j=1:checkList(i)
fnum=j-1; snum=num2str(fnum);
s2num=[repmat('0',[1,gtecLen-length(snum)]),snum];
fname_tmp=[newDataName0,'_',s2num,'.',tag{1}];
load(fullfile(DataFolderPath,fname_tmp));
display(fname_tmp)
display(flist3{cID(j)});
if ~strcmp(fname_tmp,flist3{cID(j)}); disp('error!!'); end
checkListE(cID(j))=-1;
y2new=[y2new,permute(y,[1,3,2])];
end
ynew=y2new;
size(y2new);
ynew=[];
end
end
1 2 3
matrix matrix matrix
time
Connected Matrix

Problem:
same time stamp in different matrices
27
1 2 3
matrix matrix matrix
time
Connected Matrix
7: 'DriveSim_Day1_Subj1_26112019_26_11_2019_12_54_33_0000.mat’
8: 'DriveSim_Day1_Subj1_26112019_26_11_2019_12_54_33_0001.mat'
1 2 3 1 2 3 1 2 3
1 2 3 1 2 3 1 2 3
1 2 3 4 5 6 7 8 9
Check this
(wrong)
(right)

28
>> ynew1(1,1:10)
ans =
Columns 1 through 8
0 0.0020 0.0039 0.0059 0.0078 0.0098 0.0117 0.0137 0.0156 0.0176
>> ynew2(1,1:10)
ans =
Columns 1 through 8
0 0.0020 0.0039 0.0059 0.0078 0.0098 0.0117 0.0137 0.0156 0.0176
i=7;
ynew1=permute(y,[1,3,2]);
i=8;

29
>> format long;ynew1(1,1:10)
ans =
Columns 1 through 4
0 0.001953125000000 0.003906250000000 0.005859375000000
Columns 5 through 8
0.007812500000000 0.009765625000000 0.011718750000000 0.013671875000000
0.015625000000000 0.017578125000000

30
temp=ynew2(1,end-10:end);disp(sprintf('%5.10f¥r¥n',temp));
1317.1523437500
1317.1542968750
1317.1562500000
1317.1582031250
1317.1601562500
1317.1621093750
1317.1640625000
1317.1660156250
1317.1679687500
1317.1699218750
1317.1718750000

31
format long; diff(temp)
ans =
Columns 1 through 4
0.001953125000000 0.001953125000000 0.001953125000000 0.001953125000000
Columns 5 through 8
0.001953125000000 0.001953125000000 0.001953125000000 0.001953125000000
0.001953125000000 0.001953125000000
>> 1/0.001953125000000
ans =
512

Result of the data analysis
32
7 8
matrix matrix
time
Connected Matrix
7: 'DriveSim_Day1_Subj1_26112019_26_11_2019_12_54_33_0000.mat’
8: 'DriveSim_Day1_Subj1_26112019_26_11_2019_12_54_33_0001.mat'
0 1 2 0 1 2
0 1 2 0 1 2
0 1 2 3 4 5
Check this
(wrong)
(right)
>> ynew1(1,1)
ans =
0
>> ynew2(1,1)
ans =
0
Data is same or different?
0 1 2
0 1 2
0 1 2
0 1 2
7
8
Problem①
start “0” or “1”?
Problem②
data is same or not?
Problem③
what time difference?
d_t
Answer①
start “0”

33
>> first_end
first_end =
-14.446052551269531
-26.748914718627930
-16.038101196289062
-11.351375579833984
-15.261199951171875
1.580409526824951
1.644356608390808
11.256268501281738
-12.424040794372559
-8.702564239501953
64.684532165527344
-15.415493965148926
-20.162462234497070
5.687370300292969
34.156707763671875
-3.601672172546387
0
>> second_first
second_first =
-16.671512603759766
-28.687063217163086
-19.167751312255859
-14.604801177978516
-18.160932540893555
-0.071848385035992
1.512522339820862
9.683222770690918
-13.267334938049316
-9.264326095581055
62.131206512451172
-16.077695846557617
-21.063846588134766
7.599884033203125
36.985198974609375
-5.578239917755127
0
first_end=ynew1(2:end,end); second_first=ynew2(2:end,1);
Problem②
data is same or not?
Answer②
different!

34
>> > timestep_first=ynew1(1,:); d_t=diff(timestep_first);
>> min(d_t)
ans =
0.001953125000000
>> max(d_t)
ans =
0.001953125000000
>> max(d_t)-min(d_t)
ans =
0
>> mean(d_t)
ans =
0.001953125000000
0 1 2
0 1 2
Problem③
what time difference?
d_t
No fluctuation
It is okay to set d_t as one sample or mean

35
i=7;
i=8;
timestep_first=ynew1(1,:);
d_t=diff(timestep_first);
if abs(max(d_t)-min(d_t))>0 disp('sampling rate is fluctuating!!'); end
m_dt=mean(d_t);
new_ts=ynew2(1,:)+ynew1(1,end)+m_dt;
ynew2(1,:)=new_ts;
ynew3=[ynew1 ynew2];
size(ynew1)
size(ynew3)
figure(1); clf; plot(ynew1(1,:),ynew1(2:17,:));
rangeV=600;
figure(4); clf; plot(ynew1(1,:),ynew1(2:17,:)); set(gca,'ylim',[-
rangeV,rangeV]);
rangeV,rangeV]);
rangeV,rangeV]);
Dropbox/Wlabフォルダ
/2019_B4/Wlab_Mishima/_MAT
LAB_coding/MATLAB4EEG_cod
es2/check_code_merge_Normal.
m

36
/Users/hiro/Dropbox/Wlabフォルダ
/2019_B4/Wlab_Mishima/_MATLAB_cod
ing/MATLAB4EEG_codes/getfilelistMerg
e_Advanced.m
y2new2=permute(y,[1,3,2]);
if ~isempty(y2new)
timestep_first=y2new(1,:);
d_t=diff(timestep_first);
if abs(max(d_t)-min(d_t))>0 disp('sampling rate is fluctuating!!'); end
m_dt=mean(d_t);
new_ts=y2new2(1,:)+y2new(1,end)+m_dt;
y2new2(1,:)=new_ts;
end
y2new=[y2new,y2new2];

37
/Dropbox/Wlabフォルダ
/2019_B4/Wlab_Mishima/_MATLAB_coding/MATLAB4EEG_codes/fread_A
dvanced.m

Reading program: fread_Plus.m (2/2)
38
flist3
pl1={}; pl2={};
fignum=1;
initfigx=100; fig_w=1500;
initfigy=100; fig_h=600;
for i=1:length(flist3)
load(flist3{i});
figure(fignum); clf
ftag=strsplit(flist3{i},'.');
ftag2=strrep(ftag{1},'_','_');
plot(ynew(1,:),ynew(2:17,:));
title(ftag2);
set(fignum,'Position',[initfigx,initfigy,fig_w,fig_h]);
set(fignum,'name',ftag{1});
initfigy=initfigy+100;
fignum=fignum+1;
figure(fignum); clf
pl1{i}=subplot(2,1,1);
title(ftag2);
plot(ynew(1,:),ynew(18,:));
linkaxes([pl1{i},pl2{i}],'x')
set(fignum,'name',[ftag{1},'_wTrig']);
fignum=fignum+1;
end
cd(defaultPath);
datafname=fullfile(DataFolderPath,merge_f,'figure');
save_fig;

Merge program: fread_Plus.m
39
>> fread_Plus
flist3 =
1×7 cell array
Columns 1 through 3
{'AutoDrivingExp_…'} {'AutoDrivingExp_…'} {'AutoDrivingExp_…'}
Columns 4 through 6
{'AutoDrivingExp_…'} {'AutoDrivingExp_…'} {'DriveSim_Day1_S…'}
Column 7
{'DriveSim_Day1_S…'}
This is output

Merge program: fread_Plus.m (2/2)
41
flist3
pl1={}; pl2={};
fignum=1;
initfigx=100; fig_w=1500;
initfigy=100; fig_h=600;
for i=1:length(flist3)
load(flist3{i});
figure(fignum); clf
ftag=strsplit(flist3{i},'.');
ftag2=strrep(ftag{1},'_','_');
title(ftag2);
set(fignum,'name',ftag{1});
fignum=fignum+1;
figure(fignum); clf
title(ftag2);
plot(ynew(1,:),ynew(18,:));
linkaxes([pl1{i},pl2{i}],'x')
set(fignum,'name',[ftag{1},'_wTrig']);
fignum=fignum+1;
end
cd(defaultPath);
datafname=fullfile(DataFolderPath,merge_f,'figure');
save_fig;

How to get Tobii data (eye-tracking event)
42

How to read json data: Tobii data analysis
43
References:
https://qiita.com/kikuchiTakuya/items/53990fca06fb9ba1d8a7
Last login: Sun Jan 26 15:32:37 on ttys005
[HiroWaga-Retina-MacBook-Pro-2:~]
hiro% cd /Volumes/Elements2/Automated_Driving_System_Sato_Lab_20191204_05_DataAnalysis/Tobii
[HiroWaga-Retina-MacBook-Pro-
2:Elements2/Automated_Driving_System_Sato_Lab_20191204_05_DataAnalysis/Tobii]
hiro% Jupyter Notebook
[I 15:34:22.711 NotebookApp] Serving notebooks from local directory:
/Volumes/Elements2/Automated_Driving_System_Sato_Lab_20191204_05_DataAnalysis/Tobii
[I 15:34:22.711 NotebookApp] The Jupyter Notebook is running at:
[I 15:34:22.711 NotebookApp]
http://localhost:8888/?token=c48f7978b541eada07ddadd2bd206b24e32ee62270f33d1b
[I 15:34:22.712 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice
to skip confirmation).
[C 15:34:22.723 NotebookApp]
Copy/paste this URL into your browser when you connect for the first time,
to login with a token:
http://localhost:8888/?token=c48f7978b541eada07ddadd2bd206b24e32ee62270f33d1b
[I 15:34:23.052 NotebookApp] Accepting one-time-token-authenticated connection from ::1

47
[HiroWaga-Retina-MacBook-Pro-2:~]
hiro% cd
/Volumes/Elements2/Automated_Driving_System_Sato_Lab_20191204_05_DataAnalysis/Tobii/
recordings_Driving_Simulator/a6nbbtq/segments/1
[HiroWaga-Retina-MacBook-Pro-2:a6nbbtq/segments/1] hiro% ls
calibration.json fullstream.mp4 livedata.json.gz mems.tslv.gz
et.tslv.gz livedata.json md5sums segment.json
[HiroWaga-Retina-MacBook-Pro-2:a6nbbtq/segments/1] hiro% ls -l -a
total 4065096
drwxrwxrwx 1 hiro staff 4096 1 26 15:15 .
drwxrwxrwx 1 hiro staff 0 12 2 16:11 ..
-rwxrwxrwx 1 hiro staff 312 11 26 07:58 calibration.json
-rwxrwxrwx 1 hiro staff 7658056 11 26 08:49 et.tslv.gz
-rwxrwxrwx 1 hiro staff 1954842114 11 26 08:49 fullstream.mp4
-rwxrwxrwx 1 hiro staff 97983050 11 26 08:49 livedata.json
-rwxrwxrwx@ 1 hiro staff 14667384 11 26 08:49 livedata.json.gz
-rwxrwxrwx 1 hiro staff 0 11 26 08:49 md5sums
-rwxrwxrwx 1 hiro staff 6160155 11 26 08:49 mems.tslv.gz
-rwxrwxrwx 1 hiro staff 333 11 26 08:49 segment.json
[HiroWaga-Retina-MacBook-Pro-2:a6nbbtq/segments/1] hiro%

49
[HiroWaga-Retina-MacBook-Pro-2:a6nbbtq/segments/1] hiro% pip3 install jsonlib-python3
Collecting jsonlib-python3
Downloading
https://files.pythonhosted.org/packages/f2/ec/f632f40dc465923baa25f81f4497e22bf801ba1e48
5b4ff09055a49c7ed4/jsonlib-python3-1.6.1.tar.gz (43kB)
|████████████████████████████████| 51kB 922kB/s
Building wheels for collected packages: jsonlib-python3
Building wheel for jsonlib-python3 (setup.py) ... done
Created wheel for jsonlib-python3: filename=jsonlib_python3-1.6.1-cp37-cp37m-
macosx_10_13_x86_64.whl size=36475
sha256=976657ea6d0f8a459b17d5f8c8ddefb26240b13f5234c3557387e504f3b83464
Stored in directory:
/Users/hiro/Library/Caches/pip/wheels/05/32/40/4816ec2597eb6584df06e7a5700647af098056db7
f3bc32536
Successfully built jsonlib-python3
Installing collected packages: jsonlib-python3
Successfully installed jsonlib-python3-1.6.1
[HiroWaga-Retina-MacBook-Pro-2:a6nbbtq/segments/1] hiro%
Python json module install

50
[HiroWaga-Retina-MacBook-Pro-2:~] hiro% cd
/Volumes/Elements2/Automated_Driving_System_Sato_Lab_20191
204_05_DataAnalysis/Tobii
[HiroWaga-Retina-MacBook-Pro-
2:Elements2/Automated_Driving_System_Sato_Lab_20191204_05_
DataAnalysis/Tobii]
hiro% Jupyter Notebook

51
json file sample
sample.json

52
sample.json {
"section1":{
"key":"key1",
"number": 1
},
"section2":{
"key":"key2",
"number": 2
}
}
try:
# ローカルJSONファイルの読み込み
with open('sample.json', 'r') as f:
data = json.load(f)
print(data)
except json.JSONDecodeError as e:
print('JSONDecodeError: ', e)
{'section1': {'key': 'key1', 'number': 1}, 'section2': {'key': 'key2', 'number': 2}}
json module successfully installed with
pip install jsonlib-python3
and the code can read “sample.json”
# local JSON file reading

53
{"ts":365954350,"s":0,"l":423305,"gp":[0.4583,0.3033]}
try:
# ローカルJSONファイルの読み込み
with open('sample.json', 'r') as f:
data = json.load(f)
print(data)
except json.JSONDecodeError as e:
print('JSONDecodeError: ', e)
json module successfully installed with
pip install jsonlib-python3
and the code can read “sample.json”
and single line of livedata.json.
# local JSON file reading

54
But there is an error for reading
“livedata.json”

55
It means that “livedata.json” is not single json file, and every single line
has to be loaded as individual json file.
結論：
Tobiiの生成したjson file (livedata.json)は１ファイルで１つのjsonファ
イルでなく、１行が1ファイルと同じ構成になっている。
→ つまり、１行ずつ読み込んで, json変換する必要がある。
→ json.load(ファイルポインタ)でなく、json.loads(文字列)を用いる
It requires that json.loads(json strings) not json.load(file pointer)

56
https://qiita.com/YKInoMT/items/5695f6300a75cd6b2242
Pythonでファイルを一行ずつ読んで処理する
https://qiita.com/Morio/items/5170c103647ef3a4aa69
【Python入門】JSONをパースする方法
JSONファイルから読み込む
JSONファイルとして保存されているデータを読み込む方法です。
データの読み込みにはjsonモジュールのload関数を使います。
import json #必ず必要
変数1 = open(‘読み込むJSONファイルのパス’ , ‘r’)
変数2 = json.load(変数1)

57
https://qiita.com/Morio/items/5170c103647ef3a4aa69
【Python入門】JSONをパースする方法
JSONファイルから読み込む
JSONファイルとして保存されているデータを読み込む方法です。
データの読み込みにはjsonモジュールのload関数を使います。
import json #必ず必要
変数1 = open(‘読み込むJSONファイルのパス’ , ‘r’)
変数2 = json.load(変数1)
JSONデータを読み込むには、JSON形式の文字列を辞書型に変換する方法も
あります。JSON形式の文字列の変換にはjsonモジュールのloads
関数を使います。load関数と似ているので間違えないようにしましょう。
import json
変数1 = json.loads(変数2) #変数2はJSON形式の文字列

59
https://www.craneto.co.jp/archives/1309/
Python3 で CSV の読み書きをする方法 – Shift_JIS と UTF-8 対応サンプルコー
ド付

60
# 標準モジュール csv のインポート
import csv
try:
# 書き込み UTF-8
with open('sample_person.csv', 'w') as csvfile:
writer = csv.writer(csvfile, lineterminator='¥n')
writer.writerow(['連番ID', '姓', '名'])
writer.writerow([1, '佐藤', '葵'])
writer.writerow([2, '鈴木', 'さくら'])
writer.writerow([3, '高橋', '陽菜'])
# 書き込み Shift_JIS
with open('sample_person_sjis.csv', 'w',
encoding='shift_jis') as csvfile:
writer.writerow(['連番ID', '姓', '名'])
writer.writerow([1, '田中', '大翔'])
writer.writerow([2, '伊藤', '蓮'])
writer.writerow([3, '渡辺', '悠真'])
# 起こりそうな例外をキャッチ
except FileNotFoundError as e:
print(e)
except csv.Error as e:
print(e)
連番ID,姓,名
1,田中,大翔
2,伊藤,蓮
3,渡辺,悠真
serialID,last
name,first name
1,Tanaka,taisho
2,Ito,ren
3,Watanabe,yuma

61
# module csv import
import csv
try:
# Writing file by UTF-8
with open('sample_person.csv', 'w') as csvfile:
writer.writerow([' serial ID', ' last name ', ' first name '])
writer.writerow([1, ‘Sato', ‘aoi'])
writer.writerow([2, ‘Suzuki', ‘sakura'])
writer.writerow([3, ‘Takahashi‘, ‘yuna'])
# Writing file by Shift_JIS
with open('sample_person_sjis.csv', 'w',
encoding='shift_jis') as csvfile:
writer.writerow([‘ serial ID’, ‘ last name ’, ‘ first name ’])
writer.writerow([1, ‘Tanaka’, ‘taisho'])
writer.writerow([2, ‘Ito', ‘ren'])
writer.writerow([3, ‘Watanabe', ‘yuma'])
# Exceptions
print(e)
print(e)
連番ID,姓,名
1,田中,大翔
2,伊藤,蓮
3,渡辺,悠真
serialID,last
name,first name
1,Tanaka,taisho
2,Ito,ren
3,Watanabe,yuma

62
import csv
try:
# utf-8 のCSVファイル
with open('sample_person.csv', 'r') as csvfile:
csv_reader = csv.reader(csvfile, delimiter=',',
quotechar='"')
for row in csv_reader:
# print(row)
print(','.join(row))
# utf-8 のCSVファイル - ヘッダ行をスキップ
with open('sample_person.csv', 'r') as csvfile:
quotechar='"')
next(csv_reader) # ヘッダ行をスキップ
# shift-jis のCSVファイル
with open('sample_person_sjis.csv', 'r', encoding='shift_jis')
as csvfile:
quotechar='"')
print(e)
print(e)
# utf-8 CSV file
# utf-8 CSV file – header line skip
# header line skip
# shift-jis CSV file
# Exceptions

jupyter notebook インデント
63
https://mytechnote.hatenadiary.org/entry/20090627/1246115484
2009-06-27
複数行をいっぺんにインデント
これ知らなかったんだけども。
やり方は簡単。
以下手順。
1.Shiftキーをおしながらインデントしたい行を１行もしくは複数行選択。
2.Tabキーを押すとあらびっくりで複数行がいっぺんにタブインデントされ
ます。
3.複数行を選択した状態でShiftキーをおしながらTabキーをおすと複数行
がいっぺんにマイナスインデントされます。
Indent short cut key

64
if文の基本（if, elif, else）
if文の基本的な形は以下のようになる。
if 条件式1:
`条件式1がTrueのときに行う処理`
elif 条件式2:
`条件式1がFalseで条件式2がTrueのときに行う処理`
elif 条件式3:
`条件式1, 2がFalseで条件式3がTrueのときに行う処理`
...
else:
`すべての条件式がFalseのときに行う処理`
https://note.nkmk.me/python-if-elif-else/
Pythonのif文による条件分岐の書き方

65
Python If ... Else
❮ PreviousNext ❯
Python Conditions and If statements
Python supports the usual logical conditions from
mathematics:
•Equals: a == b
•Not Equals: a != b
•Less than: a < b
•Less than or equal to: a <= b
•Greater than: a > b
•Greater than or equal to: a >= b
These conditions can be used in several ways, most
commonly in "if statements" and loops.
An "if statement" is written by using
the if keyword.
Example
If statement:
a = 33
b = 200
if b > a:
print("b is greater than a")
https://www.w3schools.com/python/pyt
hon_conditions.asp

66
import csv
try:
with open('tobii_gp.csv', 'w') as csvfile:
writer.writerow(['ts', 'gp', 'l', 's'])
f = open('livedata_a6nbbtq_rep2.json', 'r')
line = f.readline()
while line:
# print(line.strip())
line = f.readline()
if len(line)>0:
data = json.loads(line)
if "gp" in data.keys():
val1 = data["ts"]
val2 = data["gp"]
val3 = data["l"]
val4 = data["s"]
writer.writerow([val1,val2,val3,val4])
# print(data)
f.close()
print(e)
print(e)
print("successfully completed!!")

67
https://qiita.com/yoshiyakato/items/14f88a81bbbffb73520d
pythonで時刻文字列をパースする３つの方法【備忘録】
Python, splitでカンマ区切り文字列を分割、
空白を削除しリスト化
https://note.nkmk.me/python-split-strip-list-join/
https://qiita.com/sai-san/items/3d986fb0c674c2d44859
【Python基礎】list型(リスト型)の使い方
Parse
split
list

69
import csv
try:
with open('tobii_gp_xy.csv', 'w') as csvfile:
# writer.writerow(['ts', 'gp', 'l', 's'])
writer.writerow(['ts', 'gp_x','gp_y', 'l', 's'])
line = f.readline()
while line:
line = f.readline()
if len(line)>0:
val1 = data["ts"]
val2 = data["gp"]
val3 = data["l"]
val4 = data["s"]
writer.writerow([val1,val2[0],val2[1],val3,val4])
# print(data)
f.close()
print(e)
print(e)

71
import csv
csvfile_gp = open('tobii_gp_xy2.csv', 'w')
csvfile_gy = open('tobii_gy.csv', 'w')
csvfile_acc = open('tobii_acc.csv', 'w')
writer_gp = csv.writer(csvfile_gp, lineterminator='¥n')
writer_gp.writerow(['ts', 'gp_x','gp_y', 'l', 's'])
writer_gy = csv.writer(csvfile_gy, lineterminator='¥n')
writer_gy.writerow(['ts', 'gy_x','gy_y','gy_z', 's'])
writer_acc = csv.writer(csvfile_acc, lineterminator='¥n')
writer_acc.writerow(['ts', 'ac_x','ac_y','ac_z', 's'])
line = f.readline()
while line:
line = f.readline()
if len(line)>0:
val1 = data["ts"]
val2 = data["gp"]
val3 = data["l"]
val4 = data["s"]
writer_gp.writerow([val1,val2[0],val2[1],val3,val4])
elif "gy" in data.keys():
val1 = data["ts"]
val2 = data["gy"]
val4 = data["s"]
writer_gy.writerow([val1,val2[0],val2[1],val2[2],val4])
elif "ac" in data.keys():
val1 = data["ts"]

72
writer_gy = csv.writer(csvfile_gy, lineterminator='¥n')
writer_acc = csv.writer(csvfile_acc, lineterminator='¥n')
line = f.readline()
while line:
line = f.readline()
if len(line)>0:
val1 = data["ts"]
val2 = data["gp"]
val3 = data["l"]
val4 = data["s"]
val1 = data["ts"]
val2 = data["gy"]
val4 = data["s"]
val1 = data["ts"]
val2 = data["ac"]
val4 = data["s"]
writer_acc.writerow([val1,val2[0],val2[1],val2[2],val4])
# print(data)
f.close()

73
https://qiita.com/Morio/items/38701038ad098dd5dc3d
【Python入門】split関数で文字列の分割の仕
方
Python文字列操作マスター
https://qiita.com/tomotaka_ito/items/594ee1396cf982ba9887
https://www.headboost.jp/python-connect-strings/
Pythonの文字列を連結・結合する方法｜+演算子, format, join
https://minus9d.hatenablog.com/entry/2014/12/26/232423
Python3でmkdir -pする(深いディレクトリを作る)
import os
os.makedirs(os.path.join("very", "deep", "dir"), exist_ok=True)
exist_ok=True
と書くことによって、たとえ既に very/deep/dir ディレクトリが作成済で
あっても例外が出ません。

74
[Python]ファイル/ディレクトリ操作
Python
【Python入門】for文を使った繰り返し文の書き
方
https://qiita.com/Morio/items/e8aed85346c0055beea7
https://qiita.com/supersaiakujin/items/12451cd2b8315fe7d054
https://programming-study.com/technology/python-for-index/
【Python入門】for文でデータのindexを取得する方法
1
2
3
4
list = ['python', 'Hello', 'world']
for i, j in enumerate(list):
print('{0}:{1}'.format(i, j))
0:python
1:Hello
2:world

75
https://qiita.com/HajimeKawahara/items/02c288667f0a893e8761
Python 文字リスト（配列）の操作
ここでは、tag=['svd3' 'svd3' 'svd3' 'svd3' 'svd5' 'svd5' 'svd5' 'svd3' 'svd3'
'svd3'
'svd3']というndarrayの要素それぞれの末尾に"iseg"という語を追加した
い
import numpy as np
num=len(tag)
iseg= np.array(["iseg"]*num) tag=np.core.defchararray.add(tag,iseg)
print tag
実行結果：
['svd3iseg' 'svd3iseg' 'svd3iseg' 'svd3iseg' 'svd5iseg' 'svd5iseg' 'svd5iseg'
'svd3iseg' 'svd3iseg' 'svd3iseg' 'svd3iseg']
Pythonでリスト(配列)を初期化する方法:入門編
https://it-ojisan.tokyo/python-list-init/

77
# PROCESS1: duplicate livedata.json "livedata.json" -(copy)-> "livedata_a6nbbtq.json"
(livedata_(target_tag)).json"
import os
import shutil
#input_path='/Volumes/Elements2/Automated_Driving_System_Sato_Lab_20191204_05_DataAnal
ysis/Tobii/recordings_Driving_Simulator/a6nbbtq/segments/1'
input_path='/Volumes/Elements2/Automated_Driving_System_Sato_Lab_20191204_05_DataAnaly
sis/Tobii/recordings_Driving_Simulator/'
target_tag='a6nbbtq'
# target_tag='cwnxiwl'
original_name='livedata.json'
#os.makedirs(os.path.join(input_path,target_tag,'segments', '1'), exist_ok=True)
input_file=os.path.join(input_path,target_tag,'segments', '1',original_name)
# print(os.path.exists(input_file))
# print(os.path.isfile(input_file))
dup_file=input_file.split('.')[0]+'_'+target_tag+'.json'
# print(dup_file)
shutil.copyfile(input_file,dup_file)

80
# PROCESS2: livedata.json -> extract 'gp' 'gy' 'ac' data in csv format
import json
import urllib.request
import csv
input_file_name=dup_file
output_file_tag=input_file_name.split('.')
file_postfix=['gp','gy','ac']
outfile_name=['']*len(file_postfix)
for i, name in enumerate(file_postfix):
# temp_file_name[i]=dup_file.split('.')[0]+'_'+str(name)+'.csv'
outfile_name[i]='{0}{1}{2}{3}'.format(dup_file.split('.')[0],'_',name,'.csv')
# write file based on UTF-8 code
# csvfile_gp = open('tobii_gp_xy2.csv', 'w')
# csvfile_gy = open('tobii_gy.csv', 'w')
# csvfile_acc = open('tobii_acc.csv', 'w')
csvfile_fp=[0]*len(file_postfix)
for i,name in enumerate(file_postfix):
csvfile_fp[i]= open(outfile_name[i], 'w')
writer_gp = csv.writer(csvfile_fp[0], lineterminator='¥n')
writer_gp.writerow(['ts', 'gp_x','gp_y', 'l', 's'])
writer_gy = csv.writer(csvfile_fp[1], lineterminator='¥n')
writer_acc = csv.writer(csvfile_fp[2], lineterminator='¥n')
f = open(dup_file, 'r')
line = f.readline()

81
f = open(dup_file, 'r')
line = f.readline()
while line:
line = f.readline()
if len(line)>0:
val1 = data["ts"]
val2 = data["gp"]
val3 = data["l"]
val4 = data["s"]
val1 = data["ts"]
val2 = data["gy"]
val4 = data["s"]
val1 = data["ts"]
val2 = data["ac"]
val4 = data["s"]
writer_acc.writerow([val1,val2[0],val2[1],val2[2],val4])
# print(data)
f.close()

Python Code Training sample1

Recommended

Recommended

More Related Content

Similar to Python Code Training sample1

Similar to Python Code Training sample1 (20)

More from Hiroaki Wagatsuma

More from Hiroaki Wagatsuma (13)

Recently uploaded

Recently uploaded (20)

Python Code Training sample1