WebKit is everywhere, on desktops, mobiles, - and as an integrated part of the Qt platform, it is on Linux-based MeeGo as well. n this session we will quickly introduce the WebKit project, and look at why it has become so popular on the mobile platform. We will look at some of the mobile adaptions normally done by the different WebKit ports and at the work we have been doing in Qt WebKit to make it more fit for mobile usage. Some of the technologies we will touch, includes: frame flattening, tiling (including challenges), viewport meta information, touch events and fuzzy hit testing. We hope the session will make people aware of the importance of the Qt WebKit project and its mobile efforts; create discussions and use-cases that we can use for integrating even better with, in particular, open source projects targeting mobile usage and the open source community in general.

1. Qt WebKit goes Mobile
Kenneth Rohde Christiansen
FOSDEM 2011

2. Introduction
Welcome everyone!
2

3. Introduction
• Name is Kenneth Rohde Christiansen
• Works in the Nokia, Application and Service Frameworks subentry
(basically Qt Development Frameworks + browser/web teams)
• Among other, part of the Qt team developing and adapting WebKit for Nokia
3

4. Relevance to FOSDEM
• Qt and Qt WebKit are fully open source projects, and we actively encourage
contributions.
• Long time Linux support, now also targeting the open source MeeGo project
4

5. Today
Today we will identify what is needed
to make a great mobile browser
5

6. Today
… and how to accomplish that
6

7. Rendering Engine
7

8. First we need a great, flexible Web Engine
WebKit
is our recommendation
8

9. Rendering Engine
Why ?
• WebKit is already a part of Qt (and MeeGo)
• It is highly modular given it is not a browser, but an API for a.o. making one
• That is what I know about ;-)
9

10. Rendering Engine
But the most important reason:
WebKit is the engine powering most of the mobile web browser in use
That means:
It is proven technology – it can be done
Less bootstrapping – a lot of things already in place
10

11. Targeting a small form factor
and different input
11

12. Adaptions
• Nokia’s track record not that great, or is it?
• First to bring WebKit to mobile phones and did quite well at the point
Lot has changed since then
• Availability of 3G and Wi-fi
• Reduced data cost / plans
• More powerful devices – with larger screens
• Totally new interaction model; touch!
12

13. Adaptions
This means that, people today actually make use of their data plans
The better connection and faster devices means that we can load and process
normal pages and are not limited to WAP pages as before.
ie. brings the real internet to devices
13

14. Adaptions
We have lots of challenges:
• Small form factor – things doesn’t fit, becomes too small
• Load and layout/paint is slow compared with desktop
• Touch interaction requires a very responsive user experience
• Fingers are huge – content is small
• Web apps… let’s talk about that later
14

15. Adaptions: small form factor
• We need good support for zooming
Some great ideas exists
• Double tap to zoomable area (vertically centered where you clicked); zoom fully out
if trying to zoom to a similar target.
• Pinch zoom, preferable with moving – pinch and move simoustainly.
15

16. Double tap to zoomable area
16

17. Pinch zoom
17

18. Adaptions: small form factor
Did that solve our issues?
Still too small text, or text being cut when zoomed in too much.
Solutions?
Android: text reflowing
iOS: enforce min. font sizes depending on the width of the zoomable area (eg. <div>).
18

19. Text reflowing
Not my personal favorite
• Hard to get right – lots of bugs related to that
on Android
• Relayouting is expensive
• “interrupted” animation
Text reflowing on Android
19

20. Adaptions: big, imprecise fingers
• Problem: We need to interact with the content
• We have a very little screen size, so we miss overview and a way to scroll to the
important area, fast.
• Solution?
• For news sites with columns: Double tap to area / back out, shows overview and
makes you go to an area fast.
• For other pages we need another solution:
• Panning; pan around with your finger
• Added kinetics; when you lift you finger it will continue your motion with kinetics
20

21. Adaptions: big, imprecise fingers
Give visual feedback:
• Scroll indicators (fade out when not in use; focus is contents)
• A bounce back or similar way to show the user reached the boundaries
Deal with imprecise fingers:
• Ignore kinetics for minor moves – We do not want a finger ‘click’ to start kinetic
panning
• Lock panning in the horizontal or vertical direction
• More complicated: Improve hit testing to look for area close by.
21

22. Hit testing
Fingers click an area, not a single point as a mouse
• We need heuristics to guess where the user tries to click: This depends on
finger geometry.
• We need the hit testing to look for clickable things close to the actual click.
22

23. Frame flattening
• Panning depends on the whole web content being one
• Thus, we need to eliminate sub-scrollable areas
• Expand iframes to the size of their contents (done of iOS, etc)
• Expand framesets to become one flattened area
Examples coming up ;-)
23

24. Frame flattening
No one wants
to interact with
a page like this
24

25. Frame flattening
Now what
about this?
This is easily pannable
25

26. Conclusion
Touch is a very nice way to interact with web contents.
It feels quite natural and easy.
But there is a one big but…
You feel if the page blocks
26

27. Now why is that?
Web browsers actually stall a lot
• During loading
• While laying out the content
• While painting
• While running JavaScript
You can feel this on the desktop, but desktops are fast, and more important,
you don’t touch the contents
Big screen, less need for scrolling, not pinch zoom, etc
27

28. Touch requires a non-blocking UI
28

29. Back to the drawing board
• Do we need to redesign the engine?
• Or is there a simpler idea
Redesigning the engine will be very complex and there is not much we can do
with regard to loading.
29

30. Solution
Separate UI and web engine in different threads or processes
30

31. Two ideas
1. Store painting sequences and replay them on the UI.
2. Share painting area (shared memory, or memory mapped IO)
Quite complicated.
We want a non blocking UI, so everything has to have an asynchronous API.
31

32. Process/thread separation
In a way we need the process separation just like e.g. Chrome, but not for
security reasons (that is an extra bonus) but for the sake of a non-blocking user
experience.
Example: Fixed elements cannot be handled/painted by the web process as it
would be invoked for each scrolling step, meaning it would block:
We need to decouple the dependency on the web process for all kinds of interaction
(panning, pinching, rotation, etc)
32

33. Asynchronous Tiling
Basic idea:
Imagine that the contents was just one big image. Then we could easily ensure
60 fps panning, pinch zoom, etc.
That is the basic idea!
33

34. Basics
Viewport
Full size tiles
Edge tiles
34

35. Basics
Viewport
Full size tiles
Edge tiles
Tiling means that we device our contents into tiles and we handle scrolling
etc. on the UI side.
Basically you can think of the contents being one big tiled image on the UI
side. In practice an intelligent image which doesn’t need to paint tiles not
partially visible.
35

36. Basics
When a tile becomes visible:
• If painted, blit it.
• If unavailable, show some kind of indicator as background image, ie. checker
boards (known from graphics apps).
When web page area change:
• If visible, update the changed area of the tiles (all tiles need to be in sync) – you
can mark dirty on non-visible tiles.
• The dirty regions can eventually be merged smartly.
36

37. The magic number
We want to ensure 60fps for panning, pinch zoom
and rotation.
Repaints, loading and JavaScript makes it hard to keep that promise.
So we need to:
• Suspend JavaScript execution while panning, pinch etc.
• Defer loads
• Ignore repaints from the web process.
37

38. Pinch zooming
When zooming painting is involved (we change the scale)
We cannot control how long it will take to paint – painting web contents is
basically painting rectangles, lines etc… very vector graphics like.
Solution:
• Freeze the backing store (do not repaint tiles; the page is suspended)
• Scale the tiles (basically scaling an image; zooming in will show pixelated contents)
• Unfreeze on end, repainting all tiles at the right scale
38

39. Rotation
Rotation is a bit tricky.
For normal pages we basically multiply/divide with the device width/height
factor, so that you look at the same contents.
39

40. Rotation, layouting
The iPhone gave page authors some ways to control how the page is laid out
For instance, the page author can decide that the contents has a max scale:
Example: Google.com has a min and max scale of 1.0, so instead of applying the
difference in the width in landscape and portrait, we just rotate!
40

41. Rotation, layouting continued
Now what happens, this doesn’t look good!
41

42. Relayout is the solution
We need to relayout with a
different width
42

43. Rotation
Recall that layouts are expensive, so if we layout during the rotation we cannot
ensure 60 fps.
Thus, only layout after rotation has finished; with the above as intermediate step
(show checkerboards)
43

44. Rotation
Summing up:
Rotation starts: freeze backing store / suspend JS, paint, loads
During rotation: Keep old contents, show checkerboards where no contents is
After rotation: Relayout (if needed) and unfreeze / resume.
44

45. Web Apps!
Let’s not get into why web apps makes sense, or when then do. That would be
another presentation J
If interested, check out: http://tinyurl.com/4hx36lt which is my presentation:
“Connecting technology for great experiences” which touches QML and Web for
application developers.
Web Apps makes sense in many cases, so how can
we support them?
45

46. Web Apps
How should web apps behave?
• Basically like real apps
• A one to one scale – one css pixel is one device pixel – no scaling
• No pinch zoom of the whole apps, it is an app not a page!
• It has to be able to deal with input
46

47. Touch events
Mobile browsers support touch events in addition to mouse events
This allows apps to take advantage of multi touch etc. We need to support that.
This means that each event has to go to the web process and be treated before
we can use it on the UI for panning etc. A bit slower.
Trick: Only send events to the web process if the page is actually using touch
events (have listeners registered).
47

48. Viewport meta tag
Introduced by the iPhone.
• Layouts with a width of 980 by default.
• Works with most sites, but some break. ie. www.apple.com which needs a
width > 1000 or so.
• Added a meta tag to give control to the web authors
48

49. Viewport meta tag
Features:
• Define layout width (and height, though seldom used in practice)
• Define initial, minimum and maximum scale
• Define non-user-scalable (basically remove pinch zoom)
Nota bene: It is defines given portrait mode.
<meta contents=“viewport” value=“width=320, maximum-scale=1”>
Only comma allowed as separators, though some Android versions allowed
semicolon.
49

50. Avoid hardcoded values
There was a need to avoid hardcoding values such as the the iPhone width of
320.
Introduced device-width, device-height
<meta contents=“viewport” value=“width=device-width, maximum-scale=1”>
50

51. DPI, a pixel is not a pixel
Facts:
• Most contemporal mobile pages were designed with the iPhone in mind
• Buttons, hit areas are adapted to the sizes and DPI of the iPhone which is 160.
First Android phones has the same DPI, but some low end models had lower and
almost all current models have a DPI of 240.
For pages designed for the iPhone, they need to be scaled up by 1.5 for text, hit
areas to not be too small. (240 / 160 = 1.5)
Android defines a pixel at 160 as a DIP = density independent pixel.
51

52. DPI and the meta tag
Android added the target-densitydpi extension
<meta contents=“viewport” value=“width=device-width”>
on a 240 DPI device uses 320 as device-width. On the other hand
<meta contents=“viewport” value=“width=device-width, target-densitydpi=device-dpi”>
uses 420.
52

53. Summing up
Making a browser for touch devices has different requirements than that of a
desktop browser.
Focus is touch interaction, requires touch events, engine/application separation,
and many tricks to make things work smoothly.
Web apps requires touch events and the ability to control whether pinch zoom
etc is allowed and how the contents should be laid out.
53

54. Questions?
I hope you enjoyed the show and
have some good questions!
Feel free to send questions to
kenneth@webkit.org
and you are welcome to join
#qtwebkit at irc.freenode.net
54