The document discusses building a tool to transform hand-drawn designs into HTML code using AI services. It outlines splitting the problem into handwriting recognition, interpreting UI controls and positioning. It then introduces Microsoft Cognitive Services like Vision Service for text and handwriting detection and Custom Vision Service for custom object detection, describing how to use the APIs with sample code and demos. The document advocates that AI can help developers build applications by leveraging pre-trained models and services rather than requiring deep AI expertise.

2. Developers are lazy

3. Developers are efficientlazy

4. What are we trying to achieve?
• Transform any hands-drawn design into a HTML code

5. How can we build this?
Split up the problem
Handwriting recognition
Recognize UI control
Interpret positioning
Can we code this?
How to interpret graphical
input?
Many variations in
handwriting
Edge detection, pattern
matching, if-then-else
Can we use AI?
DIY model
Choose existing models
Sample labeled data
Model training

6. AI…OK, give it to me!
https://towardsdatascience.com/r-cnn-fast-r-cnn-faster-r-cnn-yolo-object-detection-algorithms-36d53571365ehttps://towardsdatascience.com/handwriting-recognition-using-tensorflow-and-keras-819b36148fe5
http://www.fki.inf.unibe.ch/databases/iam-handwriting-database

7. But I’m a developer…

8. AI…I’m not OK…
• How do I know about these models?
• What’s a neural network anyway?
• Where do I get the data to train these models?
• How do I integrate these models into my app?

9. A3S

10. AI As A ServiceA3S

11. Cognitive Services
Infuse your apps, websites, and bots with human-like intelligence
https://microsoft.com/cognitive

12. Vision Service
• Text and handwriting detection
• Text and position
• Off the shelve – requires no
training
• Asynchronous API

13. Vision Service – Getting started
1. Create an account
• Pricing tier
• Region
2. Get the subscription key
3. Set HTTP headers
• Ocp-Apim-Subscription-Key
• Content-Type
4. Invoke method
• RecognizeText
• GetTextOperationResult

14. Demo: Vision service

15. Custom Vision Service
• Object detection on your data
• Objects and position
• Bring your own data – limited dataset
due to transfer learning
• Asynchronous API
• Training
• Prediction

16. Custom Vision Service – Getting started
1. Create an account
2. Upload and tag images
3. Train Model
4. Get the training & prediction
keys
5. Set HTTP headers
• Training-Key | Prediction-Key
6. Invoke method
• TrainProject
• GetIteration
• DetectImage

17. Demo: Custom Vision service

18. https://sketch2code.azurewebsites.net/

19. Recap
• AI can help you turn data into programs
• AI is not only for Data Scientists
• Bring your own Model
• Bring your own Data
• Bring your own HTTP request
• Get started at https://aka.ms/jsconfbe/cognitivedocs

20. “I choose a lazy person to do
a hard job. Because a lazy
person will find an easy way
to do it.”