Everybody wants a Magic Mirror these days, it’s true. We couldn’t steal one , so we made our own using a Raspberry Pi, hooked it up to the internet and made use of several online Api’s to populate our app with user specific information. The mirror can tell you how long you’ll be in traffic, whether you’ll need to bring an umbrella and information about cool events happening soon. It will also send a nice compliment every now and then to make those Monday mornings more bearable. Want to share your smart mirror with others? (we don’t). No problem. Just let the mirror scan your face and it will recognize you the next time it sees you. In this session, we will talk about how we built a magic mirror with a Raspberry Pi running an UWP app on Windows 10 IoT Core. The app is built to consume web services and display data on the reflection of a mirror. Learn more about how to build true Universal Windows Platform(UWP) apps using the new .Net Standard 2.0 libraries and see how Cognitive Services can improve the user friendliness of your app.

1. Magic Mirror
Beyond reflection
Tom Vandevoorde
Michiel Wouters

2. 2
Today’s topics
 Why a Magic Mirror?
 What’s important is what's on the inside!
 The App
 Cortana
 Facial Recognition
 How did we do it?
 GeoPosition

3. Why Magic Mirror?

4. 4
Why?
You know the feeling...
 Morning rush
 No time to check traffic, weather, latest news...
 The average person spends between 10 and 45 minutes
every day in front of a mirror.
 Because we can
Looking up information costs time. Getting ready for work
takes time. Why not combine them?

5. 5
Concept
Your reflection like any other mirror
But:
 Connected to the internet
 Date and time
 Weather, news and traffic information
 Motivational message
 Making everyday objects smart

6. 6
Concept
and we went even further…
 Face detection
 Voice recognition and voice commands
 User profiles
Your reflection is not the limit!

7. 7
Future
 Smart home connection
- Speech & Voice commands
- Person detection
- Blends in with the interior design
 Extra's
- Delivery tracking information
- Social media
- Music player
- Make-up tutorial and simulation

8. Demo

9. Hardware: What’s important is
what’s on the inside!

10. 10
Overview

11. 11
Mirror with monitor Reflective coating
Glass
LIGHT
DARK
AREA
REFLECTIONS
LIGHT

12. 12
Raspberry Pi 3b
 Quad-code 1200MHz 64bit
ARMv8-A
 1GB SDRAM
 Ethernet, Wi-Fi and Bluetooth
 4x USB 2.0
 MicroSD slot
 Stereo output jack
 HDMI
 40 pin GPIO
 Runs an OS (develop for that
OS)

13. 14
27" HD LCD monitor USB hub
Hardware
Only VA Panel needed
Easy to remove bezel
Extra connections
Provides power to other devices

14. 15
Microsoft LifeCam HD-3000 HP LCD Soundbar
Hardware
720p HD
Integrated microphone
Windows IoT Core compatible
USB powered
Audio Jack
Fits below camera

15. 16
Hardware
 3D printed mount
 Fits Raspberry Pi 3b
 Custom printed mounting
pins
 3D printing models:
 https://www.thingiverse.co
m
Thanks Stijn​!

16. 17
Hardware. Open says me.

17. 18
Hardware: Down to the guts
S
o
u
n
d
b
a
r
cam
Raspberry
Pi 3
Power
&
Network

18. The App

19. 20
The App
 C# .Net UWP app
 Emphasis on online services
 .Net Standard

20. 21
Universal Windows Platform
 Windows Store
 Capabilities Extensions
 Cross platform
 Scales according
to screen size

21. 22

22. 23
.Net Standard Library
 New Standard
 Goal: establish uniformity in .NET
- Compatible with all modern .Net projects
 Effectively replacing Portable Class Libraries
- Not compatible with .Net core

23. 24
.Net Standard

24. 25
Flow
Data
Call Api
Receive Json
Convert to
POCO
Business
AutoMapper
Calculate
unmappable
values
Presentation
ViewModel
loads models
Data is
Displayed

25. Cognitive Services

26. 27
Cognitive Services
 What are cognitive services?
 Vision. Speech. Knowledge.
 Self driving cars. Smart lighting. Intelligent thermostats.
 A.I. will continue to grow and play an ever important
role in our daily lives

27. 28
Cognitive Services
Vision Speech Language Knowledge Search
Computer Vision API Translator Speech API Language
Understanding
Intelligent Service *
Recommendations API
*
Bing Autosuggest API
Content Moderator Bing Speech API Bing Spell Check API Knowledge
Exploration
Service *
Bing News Search API
Custom Vision Service
*
Speaker Recognition
API *
Web Language Model
API *
Entity Linking
Intelligence Service
API *
Bing Web Search API
Face API Custom Speech
Service *
Text Analytics API Academic Knowledge
API *
Bing Entity Search API
*
Emotion API * Translator Text API QnA Maker API * Bing Image Search API
Video Indexer * Linguistic Analysis API
*
Custom Decision
Service *
Bing Video Search API
Bing Custom Search
API

28. 29
Face API
 Face Detection
- Find faces in an image
- Get face attributes

29. 30
Face API
 Face Verification
- Compare 2 faces
- Given 2 faces
- Returns whether faces are similar
- Confidence percentage
 Face Identification
- Identify faces, who is who
- Tag people and groups

30. 31
Face API
 Find similar faces
- Given collection of faces
- Given face to search for
- Returns a collection of similar faces
 Face Grouping
- Organize unidentified faces
- Given collection of faces
- Returns groups based on similarity

31. 32
Face API
 Hosted on Azure

32. 33
Cortana
 Personal Assistant
 Voice commands
 Machine learning
 Types of commands:
- Navigation
- Perform an action

33. Cortana Demo

34. Implementing Cortana

35. 36
Set up Cortana
 Set up VCD File
 Register Vcd in in app.xaml.cs
 Register Cortana as extension
 Implement background task

36. 37
1. Voice Command Defenitions file
 Windows 10 Platform
 Voice Command File
 Voice Commands
 Localization
 Must have user’s culture

37. 38
CommandSet

38. 39
Navigation Commands

39. 40
Action Commands

40. 41
PhraseList
 Hard coded values
 Less chance on communication errors
 Can be modified at runtime
 Label matches value in curly braces

41. 42
PhraseTopic
 Anything goes
 More prone to communication errors
 Use for input with a wide range of options

42. 43
2. Modify App.xaml.cs
 OnLaunched
- Install Vcd Commands
 OnActivated
- Catch Cortana commands

43. 44
Install VCD during OnLaunched

44. 45
Handle navigation Commands in OnActivated

45. 46
3. Register Cortana as Extension in AppManifest

46. 47
4. Voice Command Service
 Win RT
 Inherit IBackgroundTask
 Handle events thrown from VCD
 Only for Action Commands

47. 48
Voice command Service

48. 49
Cortana Summary
VCD
App.xaml
AppManifest
Win RT Background Task

49. 50
Facial Recognition

50. 51
Face Detection
 Runs in background
 Face detection works
offline
 Default profile
- Random name from a list
- Random addresses from a
list
- Can be changed later

51. 52
Face Detection: camera feed

52. 53
Face Detection: Finding a face

53. 54
Face Detection: getting a picture
 Face detection
- JPEG, PNG, GIF (1 frame) or BMP
- Between 1KB and 4MB
- Face should be between 36x36 and 4096x4096 pixels
- Maximum of 64 faces
- Works best with frontal or near-frontal faces

54. 55
Face Detection: getting a picture

55. 56
Face Detection: preparing the Face Api Service

56. 57
Face Detection: authorize the api call / create the person group

57. 58
Face Detection: detecting a face

58. 59
Face Detection: identifying a face

59. 60
Face Detection: identifying a face

60. 61
Face Detection: when we don't know the person

61. 62
Face Detection: and then... or when we do know the person

62. 63
Face Detection: training the personGroup

63. 64
Face Detection: get to know people
 Problem
- Identification does not work with untrained face
 Solution
- Provide the api with a list of faces and tell it who it is
- Then start training it
 Another problem
- FaceId's expire after 24 hours
- You would need to keep a folder of pictures and re-train
- Privacy concerns

64. How did we build it?

65. 66
How did we build it?
A basic app was created
 Universal Windows Platform
 Date and time
 Weather information
 Traffic information
 Settings page
It worked on a Raspberry Pi

66. 67
How did we build it?
We tried it out on a computer screen
with the mirror on top

67. 68
How did we build it?
We built an enclosure and filled it up with the hardware

68. 69
How did we build it?
It was messy at first

69. 70
How did we build it?
But everything fits well
Except for the cables hanging out

70. 71
How did we build it?
Back to the software
 We added a live camera feed
 hooked up the face API
 and generate a random username
for a face that was detected.

71. 72
How did we build it?
 We had some problems

72. 73
How did we build it?
Then we added a newsfeed in a ScrollViewer

73. 74
How did we build it?
 Painting the enclosure made it look pretty

74. 75
How did we build it?
The default name, given when your face is detected is
changed to a name from a character of Game of Thrones.

75. 76
How did we build it?
 and the cable issue was resolved
Power Connector
Ethernet Connector

76. 77
 Through many late night programming sessions

77. 78
How did we build it?
We finally released a first
version of the app at the
Ordina Innovation Day
You might have seen it.
(It probably saw you)
Unfortunately without
a network connection

78. 79
How did we build it?
 So we added WiFi support
 And we added Cortana commands
 Then we talked at Unite in front of a great crowd

79. 80
GeoLocation

80. 81
GeoLocation
 Get user’s location
 Gps, Wifi, Mobile Network
 UWP
 Privacy

81. 82
How to use GeoLocation
 Register Location in AppManifest

82. 83
How to use GeoLocation
 Request location access

83. 84
How to use GeoLocation
 Get user coordinates

84. Final words

85. 86
Total Cost
 Raspberry Pi 3 – 55 €
 Mirror – 75 €
 Monitor – 180 €
 Camera – 50 €
 Speaker – 30 €
 Building your own Magic Mirror - Priceless

86. 87
Last words
 App is Open-Source
 Feel free to add your own services
 https://github.com/MichWouters/magic-mirror

87. 88
Tom Vandevoorde Michiel Wouters
Your Hosts
- Hardware
- Facial Cognitive Services
- App
- Voice Command Services

88. 89
Thank you for your attention!
Time for Drinks!

89. The Dto: An
unexpected journey

90. DataLayer

91. 92
Data Access
 Retrieve data from a source
- Online Api’s
- Local repositories
 Json
 Return data unmodified
 Return object: Entity
- Map of a database table

92. 93
Call API

93. 94
Call API

94. DataAccess.Entities

95. Business Layer

96. 97
Business Layer
 Perform calculations
 Filter out unnecessary data
 Loads and writes User Settings
 Convert Json to useful ouput

97. AutoMapper

98. 99
Automapper
 Map data from Entity to Model
 Transfers data from object A into object B
- Implicit mapping if parameters have the same name:
- Explicit mapping can be set up
- Useful for communication between layers

99. 100
How to use AutoMapper
 Install from Nuget
 Set up Automapper profile
- Must inherit from AutoMapper.Profile
- Mapping in constructor
- CreateMap...ReverseMap()
 Instantiate Mapper object using Configuration class
 Call Mapper

100. 101
Set up Configuration class

101. 102
How to use AutoMapper
 Instantiate Mapper object using Profile

102. 103
How to use AutoMapper
 Call Mapper object

103. 104
Presentation Layer

104. 105
Dispatcher Timer