RecoFit: Using a Wearable Sensor to Find, Recognize, and Count Repetitive Exercises

RecoFit: Using a Wearable
Sensor to Find, Recognize,
and Count Repetitive
Exercises
+ CHI 2014
- Dan Morris et al.
/ 맹욱재
x 2016 Spring

UX Lab, Graduate School of Convergence Science and Technology, Seoul National Univ. 2
ABSTRACT
해결책 RecoFit: 관성 센서로 반복 운동(근력, 맨몸)을 자동으로 추적하는 팔에 차는 장치
목적 사용자가 운동 중 조작/운동 종류 선택 없이, 실시간, 운동 후 피드백 제공
방법 1) 분할(segmenting) 운동과 임시 멈춤 구간 구분
2) 인지(recognizing) 어떤 운동을 하고 있는지
3) (횟수)계산 counting 반복 운동
평가 최종 시스템 평가를 위해 114명 참가자의 146 세션 training data를 교차검증
(cross-validation)
결과
정확도
운동/비운동 구분 precision > 95% recall > 95%
운동 종류 구분 (
순환 운동 구성 수)
99% (4) 98% (7) 96% (13)
운동 횟수 측정 93%

INTRO
규칙적인 운동의 여러 장점이 있지만[11], 꾸준히 지속하기 어렵다.[20]
-> 대부분 권장 운동량을 채우지 못한다.[5]
=>만보계로 걸음수를 자동 트래킹하면 더 걷게 된다[4]
https://www.flickr.com/photos/neoadam/15837373658
https://www.youtube.com/watch?v=4LpYyULx2T4
https://commons.wikimedia.org/wiki/
운동 데이터 전송
만보계 콘솔 액세서리 유산소 운동 기계
실내 운동 + TV걷기 & 뛰기
기회를 인식 가전제품 회사가 신체 활동 측정 기기를 출시했다.

문제점
상용화, 보편화된 기기들이 놓친 2가지 운동 영역
1) 근력 운동(weight training) 2) 맨몸 운동(calisthenics)
일부 사용자에겐 생활 방식, 취향으로 인해 근력 운동이 유산소 운동보다 지속하기 좋음
균형 잡힌 운동을 위해 근력운동은 필수며, 미 질병관리본부의 성인 최소 권장량은 주 2
회[6]
권장량에 대한 순응도(compliance)가 유산소 운동량보다 적음[5]
INTRO
calisthenics : 기구 없이 하는 근력 운동 ex) 윗몸일으키키, 팔굽혀펴기, 팔벌려높이 뛰기
Fitbit, Fuelband가 운동중 사용되며, 전체 활동에 대한 칼로리 소모량을 제공하지만
횟수, 세트수, 시간 등의 근력 운동과 관련된 high-fidelity 정보를 주는 장치는 없음
카메라 같은 센서는 동작의 다양성이나 근력 운동의 복잡성을 처리하기 어려움
웨어러블로 근력 운동을 자동 트래킹해서
GPS watch로 달리기를 시작할 때 “set it and forget it” 된 것처럼
근력 운동 중의 조작 없이 실시간, 운동후 분석 피드백을 제공함
목적

운동 분석
REALTED WORK
RecoFit VS What can an arm holster worn smart phone do for activity recognition?
smart phone worn on arm RecoFit
공통점
1) segmentation, recognition, counting로 나눠 접근 2)
segmentation using autocorrelation features
# of participant 7 104
thresholds for
segmentation
heuristic learned
sensor placement
variation
X
dimensionality reduction
orientation-invariant
repetition counting X false peak rejection
real-time X O
결과
segmentation 85% > 95%
recognition (# of
exercises)
94% (10) subject-
independent training
96%(13)

myHealthAssistant
Reference, 나눔명조OTF, Regular, 15pt
운동 분석
REALTED WORK
myHealthAssistant RecoFit
sensor(position)
three accelerometers
(hand, arm, leg) inertia sensor(arm)
classifier
Bayesian classifier trained
on the mean and variance
on each accelerometer
axis
HMM, SVM
counting
combination of autocorrelation-based period estimation
and peak counting on one of the accelerometer axes
결과
segmentation X > 95%
recognition (# of
exercises)
92% (13) subject-specific
training
96%(13)
To handle non-axis- aligned movements and more complex temporal patterns (e.g. secondary peaks within
repetitions, preparatory movements) that are common in natural exercise behavior

Significant contribution: exercise segmentation, finding exercise amidst periods of
non-exercise. gesture recognition is similar problem. “gesture spotting”
https://upload.wikimedia.org/wikipedia/commons
Gesture Spotting
REALTED WORK
Autocorrelations and Periodicity
key contributions: autocorrelation function to find regions of self-similar,
repetitive exercise. highly periodic signals
ex) tracking the pitch of a musical signal, finding abnormalities in EKG signals

Exercise looks very similar to non-exercise
WHY IS AUTOMATIC EXERCISE ANALYSIS HARD?
The most challenging problem: actually exercising time 10% ~ 100% of workout session.
Between exercises, one walks around the workout space, socializes, stretches, rests,
drinks water, selects and retrieves equipment, etc.
Distinction obvious to a human observing, but to a wearable sensor much less clear
Magnitude alone is rarely informative
most exercises are quite slowly, strive to avoid jerky movements - high acceleration.
Amplitude of acceleration during exercise consistent with that during non- exercise
non-exercise stretches > the magnitude of most exercises
“easy cases” high-velocity exercises like jumping jacks
motion magnitude > typical non-exercise magnitude.
exception
“normal cases”
motion magnitude = typical non-exercise magnitude.

Limit of Gyroscopes
Slow pushups almost no translation of an arm-worn sensor
slow rotation of a sensor = the noise of gyroscopes.
Primary observable phenomenon - not energy on any sensed axis,
but repetitive change in the gravity axis by accelerometer,
shoulder presses, non-rotational exercise:not observed at all
This fundamental diversity characterizing exercises motivates machine learning
approach to segmentation
https://upload.wikimedia.org/wikipedia/commons/e/e2/3D_Gyroscope.png

exercise is typically more periodic(repetitive) than non-exercise
-> features; autocorrelation derive metrics of repetitiveness.

Another challenge: walking is the most common non-exercise activity performed during
a workout. Walking is extremely periodic,
almost impossible to heuristically describe systematic differences between walking and
exercise
-> machine learning to segmentation
dynamic stretching – repetitive to loosen joints or muscles, not intended as exercises per
set - is quite common during a workout. tremendous challenge to robust exercise
segmentation: separating “exercise” from “dynamic stretching” is almost a question of
semantics, but one that significantly impacts UX.
it is extremely rare for an individual to consistently perform the same dynamic stretching
movement – without changing orientation – for more than a few seconds, which
supports our use of self-similarity as a core of our feature set, and motivates the
temporal smoothing approach

robust segmentation critical to the practicality of automatic exercise analysis. At the
highest level, false positives (when the system tracks an exercise that the user did not
actually perform) or false negatives (when the system fails to “credit” the user for an
exercise) are potentially disastrous from UX perspective.
when segmentation is “correct”, the boundaries need to be precise to enable robust
performance at subsequent stages of our pipeline.
reliable counting relies heavily on accurate segmentation to ignore preparatory and post-
exercise movements, such as lying down to perform pushups, or putting weights down
after biceps curls.

Variability in form
Aim to require no user-specific training, variability in users’ interpretations of exercise
descriptions, and their ability to consistently execute a particular form, has a tremendous
impact on recognition accuracy
Pushups, consistent definition to most potential users, exhibits wide variation in arm
posture, pace of repetition, the temporal “shape” of the movement.
And less familiar exercises often exhibit an entirely more challenging level of variability,
where users interpret the fundamental form of the exercise differently.
End-users have some familiarity with the available exercises, not typically watch a
proscriptive video or have access to a coach who would refine their form
-> no way to address the problem of variation in form other than large-scale training data
collection with enough flexibility to elicit such variation. users in both our training data
collection and our evaluation study were given instructions
“reasonable familiarity”, but allow enough interpretation to elicit natural variation.
Instructions contained an illustrative image and a high level description for each
exercise, and experimenters did not coach or correct form during data collection

Temporal Irregularities
Simple peak-picking or zero-crossing approach will yield an accurate count
jumping jacks - high-amplitude activities
typical set of squats - “double peak” for each count,
1) autocorrelation-based period estimation
2) peak counting
highlight very challenging cases.

Unpredictable device orientation
Form factor challenge: in real world use, an arm-worn sensor naturally rotate differently
around different users’ arms, due to preference and natural fit.
“trust” the axis pointing along the arm (ex: watch always has its face pointed out, in a
readable orientation), but that the device might rotate arbitrarily around the arm.

Hardware
TRAINING DATA COLLECTION
Right forearm, inertial sensor = 3-axis accelerometer + 3-axis gyroscope.
battery, Bluetooth radio, transmitting to PC at 50Hz

“Natural” Environment and Procedure Design
TRAINING DATA COLLECTION
it quickly became clear that early success was the result of “robotic” behavior among
training participants: segmentation dropped to precision/recall levels close to 50% when
users were in a more natural environment
it became clear that real-world deployment required a data set that was both larger and
more natural. The practicalities of labeling activities and scaling to over 100 participants
prevented us from operating in an actual gym, so we retro-fitted a large lab space to
resemble a home gym, with appropriate décor (wallpaper, curtains, etc.), video and
audio entertainment under participants’ control, a couch for rest periods, and no
computers or experimenters visible to participants.
importance of encouraging natural variability in training data.
smaller data set (30 participants), collected in a space-constrained laboratory
environment that did not aesthetically resemble a gym, with clear instructions regarding
sequencing and form.

Overview of Pipeline
THE RECOFIT SYSTEM
whether the user is
exercising
RecoFit’s 6-axis data at 50Hz
Which particular exercise type?
segmentation recognition
counting
How many times?

Segmentation
THE RECOFIT SYSTEM

Counting
THE RECOFIT SYSTEM
Counting algorithm depends on the label (e.g. “jumping jacks”), raw accelerometer data
corresponding to an exercise. Empirically, gyroscope not helpful for counting.
first compute a set of candidate peaks (local maxima).
Sort these peaks based on amplitude and loop through this sorted list,

RESULT

RESULT
Recognition accuracy is assessed in the
context of a circuit, and inevitably the
choice of circuit affects accuracy. A
larger number of activities or high
similarity among activities will reduce
accuracy.

DISCUSSION AND FUTURE WORK
Intensive Strength-Training and Periodicity Breakdown
Mechanical and Form Factor Considerations
User Experience Considerations

Implications for Generalized Activity Recognition
(1) When analyzing periodic signals, the use of independent learned models for
periodicity identification and activity recognition can increase robustness.
(2) Dimensionality reduction can increase robustness to variation in device
placement and behavioral orientation.
(3) We provide specific novel features to capture self-similarity for human motion
applications, relevant to fitness, pe- dometry, physical therapy, etc.

RecoFit: Using a Wearable Sensor to Find, Recognize, and Count Repetitive Exercises

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (20)

Similar to RecoFit: Using a Wearable Sensor to Find, Recognize, and Count Repetitive Exercises

Similar to RecoFit: Using a Wearable Sensor to Find, Recognize, and Count Repetitive Exercises (20)

Recently uploaded

Recently uploaded (20)

RecoFit: Using a Wearable Sensor to Find, Recognize, and Count Repetitive Exercises

Editor's Notes