This paper introduces MagPen, a magnetically driven pen interface that works both on and around mobile devices. The proposed device introduces a new vocabulary of gestures and techniques that increase the expressiveness of the standard capacitive stylus. These techniques are: 1) detecting the orientation that the stylus is pointing to, 2) selecting colors using locations beyond the screen boundaries, 3) recognizing different spinning gestures associated with different actions, 4) inferring the pressure applied to the pen, and 5) uniquely identifying different pens associated with different operational modes. These techniques are achieved using commonly available smartphones that sense and analyze the magnetic field produced by a permanent magnet embedded in a standard capacitive stylus. This paper explores how magnets can be used to expand the design space of current pen interaction, and proposes a new technology to achieve such results.

1. magPen
Magnetically Driven Pen Interaction
On and Around Conventional Smartphones
Presenter: S.J. Hwang
Ph.D. Candidate | 6thSemester

2. BACKGROUND
Touchscreensprovideintuitiveandeffectiveinput.However,flattouchscreenshaveseveralinteractionissues:smallinteractionarea,fatfingerproblem,andvirtualcontrolslacktactileclues.

3. BACKGROUND
Pen-like interfaces provide users with higher precision and less occlusion, as well as tactile clues already familiar to most users (Engelhard, 2008)
Pen-like interface

4. BACKGROUND
UnmousePad
(Rosenberg et al., 2009)
Papiercraft
(Liao et al., 2005)
N-Tring
(Engelhard et al., 2008)
Pen interaction techniques

5. BACKGROUND
Rolling
Expansion of the design space of pen interaction
Tilting
Gripping
Contacting
Pen Rolling
(Bi et al., 2008)
Tilt Menu
(Tianet al., 2008)
MTPen
(Song et al., 2011)
Conté
(Vogeletal., 2011)

6. BACKGROUND
Technique (Author, Year)
surface customization
pen customization
P/A (styli)
Papiercraft(Liao et al., 2005)
dot patterns
IR camera
Active
UnmousePad(Rosenberg et al., 2009)
a force sensor grid
-
Passive
N-Tring(Engelhard et al., 2008)
an electrostatic grid
capacitive probes
Active
Pen Rolling (Bi et al., 2008)
Cameras
Viconmarkers
Passive
Tilt Menu (Tianet al., 2008)
-
a tilt sensor
Active
MTPen (Song et al., 2011)
-
multi-touch sensors
Active
Conté(Vogeletal., 2011)
A camera
IR LEDs
Active
Althoughthesemethodshavesuccessfullyenrichedusers’experienceswithstyli, theyrequireeitherspecialsensorysurfacesorpencustomizationwithmechanicalparts.
Previous methods for enabling pen interaction

7. BACKGROUND
Magnetsareverycheap,small,androbust.Itdonotrequirepower,andcanbeeasilydetectedwirelesslyusingamagnetometeralreadyembeddedinmodernphones.Moreover,theydonoteasilydegradeovertime.
Magnetic Input Techniques

8. BACKGROUND
Magnetic Input Techniques
Abracadabra
(Harrison et al., 2009)
MagiWrite
(Ketabdaret al., 2010)
Digital Music Performance (Ketabdaret al., 2011)
Magnetic Appcessories(Bianchi et al., 2013)
GaussSense(Liang et al., 2012)

9. BACKGROUND
Technique (Author, Year)
Task
Limitation
Abracadabra
(Harrison et al., 2009)
accurate selections for small screens
No tactile clue
MagiWrite
(Ketabdar et al., 2010)
writing system
No tactile clue
Digital Music Performance
(Ketabdaret al., 2011)
musical performance
No tactile clue
GaussSense
(Liang et al., 2012)
Pen interaction
Need additional mechanicalhardware.
(a board with 192 magnetic sensors and USB connection).
Malfunctioning for ferromagnetic materials.
Magnetic Input Techniques
Whilethesestudiesdefinitelyimproveusability,somelacktactileclues(i.e.,drawinginthe“air”asopposedtodrawingona“surface”)andsomeneedspecialsensoryhardware.

10. OUR METHOD
a capacitive pen
1. provide tactile clues.
+
a passive magnet
2. increase input expressiveness and expand input area (e.g., touchscreen).
+
a magnetometer
3. using a magnetometer that already installed on current mobile devices.

11. OUR METHOD
Hardware
MagPen is a capacitive pen augmented with a magnet. It is made from a plastic
cylinder (8-12ø, 12-15cm) covered with conductive tape and physically connected to
two conductive rubber tips (8 mm) at the pen’s extremes.
N
S
a coin-shaped magnet
conductive tape
a plastic cylinder
a conductive rubber tip
a conductive rubber tip

12. OUR METHOD
Hardware
MagPencomes in three different forms: basic, identifiable, and pressure sensitive type.

13. OUR METHOD
Hardware –basic type
120mm
60mm
8ø
3T/5ø x 1, 500 uTesla
Forthebasicpen,asinglecoin-shapedpermanentmagnet(3T/5ø, about500uT)isembeddedinthecenterofthepen,withthemagnetfacingparalleltothescreen.

14. OUR METHOD
Hardware –identifiable type
Fortheidentifiablepen,weattachedmagnetswithdifferentmagneticproperties(intensityandposition).

15. OUR METHOD
Hardware –pressure-sensitive type
For the identifiable pen,we designed the pressure-sensitive pen to adapt its length depending on the pressure being applied to the tip. When pressure is applied to the tip, the inner spring is squeezed, bringing the magnet to the device (a change of about 250 mm).

16. OUR METHOD
Software
-
WeimplementedamagPentoolkitontheAndroid4.0.4.(SHV- E160S,SamsungGalaxyNote).
-
Wealsobuilta3DvisualizerusingOpenGLtoexaminechangesinthemagneticfield.
-
Theinertialthree-axismagnetometersensedthemagneticfieldatasamplingfrequencyof100Hz.
-
Usedabuffertoreducesamplingnoise;themovementvectorfromthelastreadingtoitspredecessorwascomputedandnormalized.

17. OUR METHOD
a) detecting the orientationthat the stylus is pointing to.
b) selecting colors using locations beyond screen boundaries.
c) recognizing different spinning gestures associated with different actions.
d) inferring the pressurebeing applied to the pen.
e) identifying various pens associated with different operational modes.
MagPenInteractions
To verify some of these techniques, five participants (2 women) aged between 28 and 34 (average 31.4, SD 2.4) were involved.

18. OUR METHOD
(A) Determining Pen Orientation
Fiveparticipantstrainedourmachinelearningsoftwarebyfreedrawingwithbothsidesofthesimplepenfor10seconds.10%crossvalidationshowedthatourapplicationcorrectlydeterminedwhichofthetwotipswasbeingusedupto99% (99.8%,K=0.99usingJ48DecisionTree).
by using the bipolar property of magnets

19. OUR METHOD

20. OUR METHOD
(B) Dragging Gesture on the Device Frame
Itispossibletoestimatethepositionofthepenoutsideofthescreenframe.Theadvantageofthistechniqueisthatitalleviatesocclusiononthetouchscreen, resultinginalargerinteractionarea.
by using a vector calculation that compares the incoming magnetic value to the values stored when calibrating.

21. OUR METHOD

22. OUR METHOD
(C) Spin Gesture Recognition
Spinning the pen between the fingers is a common activityamong many users who are concentrating. We tried to explore how this common activity could be encoded in a rich vocabulary of recognizable gestures

23. OUR METHOD

24. OUR METHOD
(C) Spin Gesture Recognition
Three different spinning gestures were classified using a machine-learning approach (J48 decision tree) with a set of features that included the average movement speed, speed variance, size of the gesture, shape, and trajectory.

25. OUR METHOD
(C) Spin Gesture Recognition
Toverifyourprototype,weconductedapilotstudyandsampled10trialsforeachgesturefromamongfiveparticipants(5participants×10trials×3gestures).Wethenconducteda10%cross-validationandfoundthatthegestureswereclassifiedcorrectlywith99.3%accuracy(K=0.99).
Charge
Twisted Sonic
Zigzag
http://en.wikipedia.org/wiki/Pen_spinning

26. OUR METHOD

27. OUR METHOD
(D) Identifying Different Pens

28. OUR METHOD
(D) Identifying Different Pens
Themagneticintensitiesforthethreepens,andthegapsamongthem,decreaseasthedistanceincreases.Weusedthesecurvesasstandardstoidentityeachpen.
250 uT
70mm

29. OUR METHOD
(D) Identifying Different Pens
The participants were asked to make 10 horizontal strokes, starting from the top and going to the bottom for each pen (5 participants ×10 trials ×3 pens). The results showed that the penswere classified correctly with 92.6%accuracy and that all errors occurred at distances of more than 83.5 mm.
Error occurred
Error occurred

30. OUR METHOD

31. OUR METHOD
(E) Determining Applied Pressure

32. OUR METHOD
(E) Determining Applied Pressure
We used a relative position between the two curves as a coarse proxy of pressure (The farther the distance is, the denser the pressure levels).

33. OUR METHOD

34. LIMITATION
-
The magnetism involved might damage the objects when the objects are very close to the magnets.
-
The magnetometer we used occasionally reports a large offset in the data or a stuck pointing in one direction. In this case, user should wave the device in figure eight to get back on track.
However, magnets do not affect flash memories on modern mobile devices or IC chips on credit cards.
This technical glitch, however, will be undoubtedly solved as sensor technology improves.

35. CONCLUSION
-
We have shown how MagPenprovides users with high input expressiveness, a wide input area, and rich tactile clues without requiring power or wireless connections.
-
We believe that MagPenwill open a large area for novel pen interaction designs.
-
Future work will explore how to further extend the additional degrees of freedom (e.g., pen hovering or tilting).
-
We plan to run several user studies to gauge the feasibility of this interface and determine how it compares with previous, more complex hardware solutions

36. THANK YOU

37. Q&A
Spin Gesture Recognition
Three different spinning gestures were classified using a machine-learning approach (J48 decision tree) with a set of features that included the average movement speed, speed variance, size of the gesture, shape, and trajectory.

38. Q&A

39. EXAMPLE USE