Motion sensor and gyroscope

1. CS144r/244r
Network Design Project
on
Secure and Intelligent Internet of Things
(wearables and motion sensing)
2/10/2014
Instructor: Professor HT Kung
Harvard School
of Engineering and Applied Sciences

2. Announcements
• Welcome our 3rd TF: Surat Teerapittayanon
• Next set of questions due Tuesday night (Feb 11) will be posted
this evening
– They will cover materials in today’s class and readings for
Wednesday’s class. For the latter, see the course website this
evening
• We have received various microcontrollers and sensors for
course projects
• We will hold a “Make Your Own Nest” information meeting
today at 4:15pm after the class in Science Center B-10.
Everyone is welcome to attend the meeting
• We will announce instruction labs on breadboarding techniques.
These labs will be mandatory for students who do not have
previous experience in this area. (This requirement will be
waived for others. These students should let TFs know)
• We will have no class on Monday, Feb 17, which is a University
Holiday (President's Day). Tsung-Han Lin will give an overview
on popular machine learning methods on Wed, Feb 19.
Updated course schedule will be posted
2

3. What Is New in the News?
2. “Want to remotely control a car? $20
in parts, some oily fingers, and you're
in command” (2/6/2014)
• “… a circuit board using untraceable,
off-the-shelf parts worth $20 can give
wireless access to the car's controls
while it's on the road.”
• The Controller Area Network (CAN) that
car manufacturers use currently has little
security protection, unfortunately 3
1. “Government wants you to
broadcast your driving data—
eventually” (2/3/2014)
• The US Department of Transportation’s
National Highway Traffic Safety
Administration (NHTSA) has announced
that it’s finally ready to consider
regulations that might require “light
vehicles” to communicate with each other
about their speed, direction of travel, and
location in order to prevent collisions.”
• Challenges: privacy concerns, radio
spectrum sharing, and cost ensuring
the security and integration with
highway infrastructure

4. One major problem is to locate 𝑥’l and 𝑥’r : The correspondence problem
Distinguishing textual features around the object will help. We can then use
neighborhood block matching to locate 𝑥’l and 𝑥’r
Recap: Principle of Stereo Cameras:
Deriving Depth from Left and Right Images
4
,
( )
elimate
(
'
l
'
r
' '
l r
x x
=
z f
x - b x
=
z f
f b
x z =
x - x )


𝑥’l 𝑥’r
Focal Length 𝑓
Object at (𝑥,𝑦,𝑧)
𝑧
Left Camera Center (Reference Point)
𝑏 (Baseline)
Left
Image
plane
Right
Image
plane
𝑥
𝑥 − 𝑏

5. 3. infer
body joint
positions
2. infer
body parts
per pixel
1. capture
depth image
4. track skeleton
Recap:
The Kinect Pose Estimation Pipeline
Infrared dots
projected onto objects In training tree decision trees, it suffices to use synthetic
training data. User-specific online training is not required 5

6. Recap: IoT Devices
• Major Players: Raspberry Pi, Ardruino
Family, Intel Galileo/Edison, BeagleBone
• Based on ARM, MIPS and x86
• All have networking: WiFi, Bluetooth,
Ethernet
• Targeted for various low-end, mid-end or hi-
end applications

7. Recap: Power-Performance
and Energy-Performance Trade-offs
• Note super-linear trade-off
curves (red)
• Where do we pay in
power/energy for high
performance under the same
clock frequency?
– Larger caches
– Larger branch prediction
tables
– Faster TLB (Translation
Lookaside Buffer)
– Wider instruction and data
paths
– OOO execution
– Larger active chip area (and
higher leakage)
• These features have little to
do with CISC or RISC
• A central issue with IoT
devices is about making
proper reduction on all these
energy consuming features
7
Note that in Fig. 15 A8 moves to a higher location (why?)

8. Today’s Topic:
Wearables and Motion Sensors
• Smart wearables (smart watches, smart
bands, etc). They have usage in various areas
including:
– Fitness tracking, digital assistant to the user, interact
with the environment, accessories to smartphones,
and learning about the person
• Motion sensors (accelerometers, gyroscopes,
compass, etc.), which are essential enabling
technology for smart wearables. They have
usage in various areas including:
– Gaming, handsets, digital still cameras, TV remote
controls, medical devices, sports and fitness
equipment, augmented reality, digital entertainment
and social networking
8

9. Readings for Today’s Topics
1. “Network Access Using Short-range
Connectability,” Apple’s US patent
application, 2013
2. “An Overview of Motion Processing
Solutions for Consumer Products,”
InvenSense, 2010
3. “Accelerometers and How they Work,”
Texas Instruments
4. Device teardown: Google glass, Pebble
smart watch and Fitbit tracker, 2013-14
9

10. On Predicting “The Next Big Thing”
• Likely, these IoT devices will eventually get right
in a big way
– Analogous to “mp3 player  iPod” and “pda 
iPhone”
– Then market explosion will follow
• However, on predicting “the Next big thing,” we
note that it is usually difficult to foresee the
coming of a killer device until it has already
arrived. We should just be happy if some early
versions are “good enough to criticize” as Alan
Kay would say
• Instead of predicting the future, it probably
makes more sense just trying to invent it, like
we do in this course
10

11. Some Notable Wearables
• Glasses
– Google glass and various competitors
• Smart watches in two flavors (note that Apple
and Google are absent from today’s market!)
1. Fully-fledged standalone phones on a wrist
• Feature watch phone (e.g., LG GD510)
• Smart watch phone (e.g., Neptune Pine based on
Snapdragon S3)
2. Companion watches for smartphones
• E.g., Pebble, Samsung Galaxy Gear, Sony SmartWatch
2, Qualcomm Toq, Martian
• These seem to be the current market focus (see Apple’s
US patent application)
• Smart bands
– Fitness tracking (e.g.,Nike FuelBand and Fitbit
Force) 11

12. Evaluation Criteria for Companion
Smart Watches
• Android compatibility (all)
• iPhone compatibility (Pebble, Martian)
• Voice assist (Martian)
• Size and weight (Pebble >> Martian)
• Screen size and resolution (Galaxy Gear >> Martian)
• Screen type (Galaxy Gear, Sony SW2, Toq: color and touch)
• Phone calls (Galaxy Gear, Martian)
• Camera (Galaxy Gear 1.9MP)
• Fitness tracking (all except Martian)
• Battery life (Galaxy Gear a day and others multiple days)
• Bluetooth (all)
• Wireless charging (Toq---using Qualcomm WiPower)
• Water resistance (Pepple >> others)
• NFC (SW2)
• Pricing (Pebble--$150, Toq--$350) 12

13. Apple’s US Patent Application on
Network Access Using Short-range Connectability
• Three phases of making short-range
connection (no real surprises here)
1. Discovery
2. Connection establishment (and teardown later)
3. Data transfer
• Challenging issues:
– Power management (sleep/wakeup)
– Security (mutual authentication, low-power
encryption and key management)
– Interoperability with a vast array of devices
13

14. Some Useful Things To Know About Patents
• A patent application has two main parts
– Claims: A list of self-contained assertions of the
new methods and apparatuses. No diagrams are
allowed in claim statements
– Description: It provides contexts of usage and
embodiment examples for the claims. They include
system diagrams and operations flowcharts
• Fundamental patents are those which are
difficult to sidestep. They usually involve great
insights and deep knowledge
• However, patent trolls who are interested in
patent infringement suits rather than making
things are an increasing problem. These are
bad guys 14

15. Figuring Out What A Person Is Doing
(We Don’t Want Instrumenting a User Like This!)
15

16. Motion Sensors
• Inertial sensors such as MEMS
accelerometers and gyroscopes
– Microelectromechanical systems (MEMS) contain
miniaturized structures, sensors, actuators, and
microelectronics
– Google Glass uses the Invensense MPU-6050 chip
which integrates a 3-axis gyroscope, 3-axis
accelerometer and a Digital Motion Processor
(DMP) in a small 4x4x0.9mm package
• Inertial sensors and other sensors such as
image sensors and GPS play complementary
roles, and can be used together (so-called
sensor fusing) in consumer products
16

17. Example Usages of Motion Sensors
17
Capture body motion
without cameras
Gesture-based TV
remote control
Gesture-based
laptop control
Using GPS and heading
data, photos to be
viewed in a map view
Augmented reality

18. Inertial Sensors
We discuss two widely used MEMS inertial sensors
• Accelerometers
– Measure acceleration in x, y, z directions
• Gyroscopes (or simply, gyros)
– Measure angular velocity in yaw, pitch, and roll
directions
18

19. Accelerometer: Background
• Velocity (𝑣) is speed and
direction so any time
there is a change in either
speed or direction there is
acceleration (𝑎)
• Acceleration metric is
reported in units of g
– Earth’s gravity: 1g
– Bumps in road: 2g
– Space shuttle: 10g
– Death or serious injury: 50g
19
2
2
x
a
t



x
v
t




20. MEMS Accelerometer: Principle
Force 𝐹 = 𝑚𝑎 = 𝑘Δ𝑥. From measured Δ𝑥,
we can compute acceleration:
20
k
a x
m
 

21. Use of an Accelerometer in Determining
the Orientation of a Smartphone
• Device orientation
detected by accelerometer
relies on a constant
gravitational pull of 1g (9.8
m/s2) downwards
• The forces experienced by
the smartphone on 3 axes
can be used to determine
the orientation of the
phone, unless the surface
plane of the device is
totally parallel to the
ground. (Acceleration
vectors are provided at 50
Hz)
• An app can display a
virtual carpenter's level 21
• X=0, Y=0, and Z=-1
means phone is face up
• X=0, Y=-1, Z=0 means
phone is standing up
• …

22. Acquiring the Tilt Angle
22
By measuring the vertical value of gravity, we
can acquire the tilt angle of the accelerometer

23. Device Stable Near +1g or -1g Positioning
But Instable Near 0g
+1g Position 0g Position
Note that with a 1∘
error, the device is .9998/.017 = 59 times
more sensitive at 0g position compared to 1g position 23

24. Gyroscope
• A gyroscope measures angular velocity (the rate of change in
orientation angle). Angular change can then be derived by
integration
• Must first initialize the sensor position with a known value (possibly
from the accelerometer), then measure the angular velocity
(ω) around the X, Y and Z axes at measured intervals (Δt)
– ω × Δt = change in angle
– The new orientation angle is the original angle plus this change
• This is integrating - adding up many small computed intervals - to
find orientation
– Repeatedly adding up increments of ω × Δt results in small
systematic errors becoming magnified over time
– Gyroscopic drift---over long timescales the gyroscope data will
become increasingly inaccurate

25. MEMS Gyroscope: Basic Principle
• Uses Coriolis effect to
transform an angular velocity
into a displacement
• The Coriolis force acts
perpendicular to the rotation
axis and to the velocity of the
body in the rotating frame
– Fc= -2m Ω x v
• The displacement induces a
change in capacitance
between the mass and the
housing, thus transforming
the angular rate input to the
gyroscope into an electrical
output
• MEMS gyroscopes use
vibrating objects, which
also tend to preserve the
direction of vibration

26. Gyroscope
• Each gyroscope
measures the rotation
around one axis
• Axz – is the angle
between the Rxz
(projection of R on XZ
plane) and Z axis
• Ayz – is the angle
between the Ryz
(projection of R on YZ
plane) and Z axis
• Gyroscopes measure
the rate of change of
these angles

27. Accelerometers vs. Gyroscopes
• Accelerometers
– Measure specific force of the body frame wrt
the inertial frame in the body frame coordinates
• Need to subtract the acceleration due to gravity to
obtain the motion induced quantity
– In general, all points on a rigid body do NOT
experience the same linear velocity
• Gyroscopes
– Measure the inertial angular velocity
• Essentially, the rate of change of orientation
– All points on a rigid body experience the same
angular velocity 27

28. Sensor Fusion
• An accelerometer measures inertial force, such as
gravity (and ideally only by gravity), but it might also
be caused by acceleration (movement) of the
device. Even if the accelerometer is relatively stable,
it is very sensitive to vibration and mechanical noise
• A gyroscope is less sensitive to linear mechanical
movements, the type of noise that accelerometer
suffers from. Gyroscopes have other types of
problems like drift (not coming back to zero-rate
value when rotation stops)
• Averaging the data that comes from accelerometers
and gyroscopes can produce a better estimate of
orientation than obtained using accelerometer data
alone

29. Sensor Fusion
with Ccomplimentary Filters
• Low pass filter is applied to accelerometer
(ignore high-frequency vibrations)
• High pass filter is applied to gyroscope (ignore long-term drifts)