Qt WebKit goes MobileKenneth Rohde ChristiansenFOSDEM 2011
Introduction        Welcome everyone!2
Introduction• Name is Kenneth Rohde Christiansen• Works in the Nokia, Application and Service Frameworks subentry    (basi...
Relevance to FOSDEM• Qt and Qt WebKit are fully open source projects, and we actively encourage  contributions.• Long time...
Today    Today we will identify what is needed    to make a great mobile browser5
Today    … and how to accomplish that6
Rendering Engine7
First we need a great, flexible Web Engine    WebKit    is our recommendation8
Rendering Engine         Why ?• WebKit is already a part of Qt (and MeeGo)• It is highly modular given it is not a browser...
Rendering EngineBut the most important reason:WebKit is the engine powering most of the mobile web browser in useThat mean...
Targeting a small form factor     and different input11
Adaptions• Nokia’s track record not that great, or is it?• First to bring WebKit to mobile phones and did quite well at th...
AdaptionsThis means that, people today actually make use of their data plansThe better connection and faster devices means...
AdaptionsWe have lots of challenges:     •  Small form factor – things doesn’t fit, becomes too small     •  Load and layo...
Adaptions: small form factor• We need good support for zooming     Some great ideas exists     •  Double tap to zoomable a...
Double tap to zoomable area16
Pinch zoom17
Adaptions: small form factorDid that solve our issues?     Still too small text, or text being cut when zoomed in too much...
Text reflowingNot my personal favorite• Hard to get right – lots of bugs related to that  on Android• Relayouting is expen...
Adaptions: big, imprecise fingers• Problem: We need to interact with the content     •  We have a very little screen size,...
Adaptions: big, imprecise fingersGive visual feedback:• Scroll indicators (fade out when not in use; focus is contents)• A...
Hit testingFingers click an area, not a single point as a mouse• We need heuristics to guess where the user tries to click...
Frame flattening• Panning depends on the whole web content being one• Thus, we need to eliminate sub-scrollable areas• Exp...
Frame flattening     No one wants     to interact with     a page like this24
Frame flattening           Now what           about this?     This is easily pannable25
ConclusionTouch is a very nice way to interact with web contents.It feels quite natural and easy.But there is a one big bu...
Now why is that?Web browsers actually stall a lot     •  During loading     •  While laying out the content     •  While p...
Touch requires a non-blocking UI28
Back to the drawing board• Do we need to redesign the engine?• Or is there a simpler ideaRedesigning the engine will be ve...
Solution     Separate UI and web engine in different threads or processes30
Two ideas1.  Store painting sequences and replay them on the UI.2.  Share painting area (shared memory, or memory mapped I...
Process/thread separationIn a way we need the process separation just like e.g. Chrome, but not forsecurity reasons (that ...
Asynchronous TilingBasic idea:Imagine that the contents was just one big image. Then we could easily ensure60 fps panning,...
Basics         Viewport         Full size tiles         Edge tiles34
Basics                                                                   Viewport                                         ...
BasicsWhen a tile becomes visible:     •  If painted, blit it.     •  If unavailable, show some kind of indicator as backg...
The magic numberWe want to ensure 60fps for panning, pinch zoomand rotation.Repaints, loading and JavaScript makes it hard...
Pinch zoomingWhen zooming painting is involved (we change the scale)We cannot control how long it will take to paint – pai...
RotationRotation is a bit tricky.For normal pages we basically multiply/divide with the device width/heightfactor, so that...
Rotation, layoutingThe iPhone gave page authors some ways to control how the page is laid outFor instance, the page author...
Rotation, layouting continued                        Now what happens, this doesn’t look good!41
Relayout is the solution                           We need to relayout with a                           different width42
RotationRecall that layouts are expensive, so if we layout during the rotation we cannotensure 60 fps.Thus, only layout af...
RotationSumming up:Rotation starts: freeze backing store / suspend JS, paint, loadsDuring rotation: Keep old contents, sho...
Web Apps!Let’s not get into why web apps makes sense, or when then do. That would beanother presentation JIf interested, ...
Web AppsHow should web apps behave?• Basically like real apps• A one to one scale – one css pixel is one device pixel – no...
Touch eventsMobile browsers support touch events in addition to mouse eventsThis allows apps to take advantage of multi to...
Viewport meta tagIntroduced by the iPhone.• Layouts with a width of 980 by default.• Works with most sites, but some break...
Viewport meta tagFeatures:• Define layout width (and height, though seldom used in practice)• Define initial, minimum and ...
Avoid hardcoded valuesThere was a need to avoid hardcoding values such as the the iPhone width of320.Introduced device-wid...
DPI, a pixel is not a pixelFacts:• Most contemporal mobile pages were designed with the iPhone in mind• Buttons, hit areas...
DPI and the meta tagAndroid added the target-densitydpi extension <meta contents=“viewport” value=“width=device-width”>on ...
Summing upMaking a browser for touch devices has different requirements than that of adesktop browser.Focus is touch inter...
Questions?     I hope you enjoyed the show and     have some good questions!        Feel free to send questions to        ...
Upcoming SlideShare
Loading in …5
×

Qt WebKit going Mobile

4,828 views

Published on

WebKit is everywhere, on desktops, mobiles, - and as an integrated part of the Qt platform, it is on Linux-based MeeGo as well.

n this session we will quickly introduce the WebKit project, and look at why it has become so popular on the mobile platform. We will look at some of the mobile adaptions normally done by the different WebKit ports and at the work we have been doing in Qt WebKit to make it more fit for mobile usage.

Some of the technologies we will touch, includes: frame flattening, tiling (including challenges), viewport meta information, touch events and fuzzy hit testing.

We hope the session will make people aware of the importance of the Qt WebKit project and its mobile efforts; create discussions and use-cases that we can use for integrating even better with, in particular, open source projects targeting mobile usage and the open source community in general.

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
4,828
On SlideShare
0
From Embeds
0
Number of Embeds
18
Actions
Shares
0
Downloads
71
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

Qt WebKit going Mobile

  1. 1. Qt WebKit goes MobileKenneth Rohde ChristiansenFOSDEM 2011
  2. 2. Introduction Welcome everyone!2
  3. 3. Introduction• Name is Kenneth Rohde Christiansen• Works in the Nokia, Application and Service Frameworks subentry (basically Qt Development Frameworks + browser/web teams)• Among other, part of the Qt team developing and adapting WebKit for Nokia3
  4. 4. Relevance to FOSDEM• Qt and Qt WebKit are fully open source projects, and we actively encourage contributions.• Long time Linux support, now also targeting the open source MeeGo project4
  5. 5. Today Today we will identify what is needed to make a great mobile browser5
  6. 6. Today … and how to accomplish that6
  7. 7. Rendering Engine7
  8. 8. First we need a great, flexible Web Engine WebKit is our recommendation8
  9. 9. Rendering Engine Why ?• WebKit is already a part of Qt (and MeeGo)• It is highly modular given it is not a browser, but an API for a.o. making one• That is what I know about ;-)9
  10. 10. Rendering EngineBut the most important reason:WebKit is the engine powering most of the mobile web browser in useThat means: It is proven technology – it can be done Less bootstrapping – a lot of things already in place10
  11. 11. Targeting a small form factor and different input11
  12. 12. Adaptions• Nokia’s track record not that great, or is it?• First to bring WebKit to mobile phones and did quite well at the pointLot has changed since then • Availability of 3G and Wi-fi • Reduced data cost / plans • More powerful devices – with larger screens • Totally new interaction model; touch!12
  13. 13. AdaptionsThis means that, people today actually make use of their data plansThe better connection and faster devices means that we can load and processnormal pages and are not limited to WAP pages as before.ie. brings the real internet to devices13
  14. 14. AdaptionsWe have lots of challenges: •  Small form factor – things doesn’t fit, becomes too small •  Load and layout/paint is slow compared with desktop •  Touch interaction requires a very responsive user experience •  Fingers are huge – content is small •  Web apps… let’s talk about that later14
  15. 15. Adaptions: small form factor• We need good support for zooming Some great ideas exists •  Double tap to zoomable area (vertically centered where you clicked); zoom fully out if trying to zoom to a similar target. •  Pinch zoom, preferable with moving – pinch and move simoustainly.15
  16. 16. Double tap to zoomable area16
  17. 17. Pinch zoom17
  18. 18. Adaptions: small form factorDid that solve our issues? Still too small text, or text being cut when zoomed in too much.Solutions? Android: text reflowing iOS: enforce min. font sizes depending on the width of the zoomable area (eg. <div>).18
  19. 19. Text reflowingNot my personal favorite• Hard to get right – lots of bugs related to that on Android• Relayouting is expensive• “interrupted” animation Text reflowing on Android19
  20. 20. Adaptions: big, imprecise fingers• Problem: We need to interact with the content •  We have a very little screen size, so we miss overview and a way to scroll to the important area, fast.• Solution? •  For news sites with columns: Double tap to area / back out, shows overview and makes you go to an area fast. •  For other pages we need another solution: •  Panning; pan around with your finger •  Added kinetics; when you lift you finger it will continue your motion with kinetics20
  21. 21. Adaptions: big, imprecise fingersGive visual feedback:• Scroll indicators (fade out when not in use; focus is contents)• A bounce back or similar way to show the user reached the boundariesDeal with imprecise fingers:• Ignore kinetics for minor moves – We do not want a finger ‘click’ to start kinetic panning• Lock panning in the horizontal or vertical direction• More complicated: Improve hit testing to look for area close by.21
  22. 22. Hit testingFingers click an area, not a single point as a mouse• We need heuristics to guess where the user tries to click: This depends on finger geometry.• We need the hit testing to look for clickable things close to the actual click.22
  23. 23. Frame flattening• Panning depends on the whole web content being one• Thus, we need to eliminate sub-scrollable areas• Expand iframes to the size of their contents (done of iOS, etc)• Expand framesets to become one flattened areaExamples coming up ;-)23
  24. 24. Frame flattening No one wants to interact with a page like this24
  25. 25. Frame flattening Now what about this? This is easily pannable25
  26. 26. ConclusionTouch is a very nice way to interact with web contents.It feels quite natural and easy.But there is a one big but…You feel if the page blocks26
  27. 27. Now why is that?Web browsers actually stall a lot •  During loading •  While laying out the content •  While painting •  While running JavaScriptYou can feel this on the desktop, but desktops are fast, and more important,you don’t touch the contentsBig screen, less need for scrolling, not pinch zoom, etc27
  28. 28. Touch requires a non-blocking UI28
  29. 29. Back to the drawing board• Do we need to redesign the engine?• Or is there a simpler ideaRedesigning the engine will be very complex and there is not much we can dowith regard to loading.29
  30. 30. Solution Separate UI and web engine in different threads or processes30
  31. 31. Two ideas1.  Store painting sequences and replay them on the UI.2.  Share painting area (shared memory, or memory mapped IO)Quite complicated.We want a non blocking UI, so everything has to have an asynchronous API.31
  32. 32. Process/thread separationIn a way we need the process separation just like e.g. Chrome, but not forsecurity reasons (that is an extra bonus) but for the sake of a non-blocking userexperience.Example: Fixed elements cannot be handled/painted by the web process as itwould be invoked for each scrolling step, meaning it would block:We need to decouple the dependency on the web process for all kinds of interaction(panning, pinching, rotation, etc)32
  33. 33. Asynchronous TilingBasic idea:Imagine that the contents was just one big image. Then we could easily ensure60 fps panning, pinch zoom, etc.That is the basic idea!33
  34. 34. Basics Viewport Full size tiles Edge tiles34
  35. 35. Basics Viewport Full size tiles Edge tiles Tiling means that we device our contents into tiles and we handle scrolling etc. on the UI side. Basically you can think of the contents being one big tiled image on the UI side. In practice an intelligent image which doesn’t need to paint tiles not partially visible.35
  36. 36. BasicsWhen a tile becomes visible: •  If painted, blit it. •  If unavailable, show some kind of indicator as background image, ie. checker boards (known from graphics apps).When web page area change: •  If visible, update the changed area of the tiles (all tiles need to be in sync) – you can mark dirty on non-visible tiles. •  The dirty regions can eventually be merged smartly.36
  37. 37. The magic numberWe want to ensure 60fps for panning, pinch zoomand rotation.Repaints, loading and JavaScript makes it hard to keep that promise.So we need to: •  Suspend JavaScript execution while panning, pinch etc. •  Defer loads •  Ignore repaints from the web process.37
  38. 38. Pinch zoomingWhen zooming painting is involved (we change the scale)We cannot control how long it will take to paint – painting web contents isbasically painting rectangles, lines etc… very vector graphics like.Solution: •  Freeze the backing store (do not repaint tiles; the page is suspended) •  Scale the tiles (basically scaling an image; zooming in will show pixelated contents) •  Unfreeze on end, repainting all tiles at the right scale38
  39. 39. RotationRotation is a bit tricky.For normal pages we basically multiply/divide with the device width/heightfactor, so that you look at the same contents.39
  40. 40. Rotation, layoutingThe iPhone gave page authors some ways to control how the page is laid outFor instance, the page author can decide that the contents has a max scale:Example: Google.com has a min and max scale of 1.0, so instead of applying thedifference in the width in landscape and portrait, we just rotate!40
  41. 41. Rotation, layouting continued Now what happens, this doesn’t look good!41
  42. 42. Relayout is the solution We need to relayout with a different width42
  43. 43. RotationRecall that layouts are expensive, so if we layout during the rotation we cannotensure 60 fps.Thus, only layout after rotation has finished; with the above as intermediate step(show checkerboards)43
  44. 44. RotationSumming up:Rotation starts: freeze backing store / suspend JS, paint, loadsDuring rotation: Keep old contents, show checkerboards where no contents isAfter rotation: Relayout (if needed) and unfreeze / resume.44
  45. 45. Web Apps!Let’s not get into why web apps makes sense, or when then do. That would beanother presentation JIf interested, check out: http://tinyurl.com/4hx36lt which is my presentation:“Connecting technology for great experiences” which touches QML and Web forapplication developers.Web Apps makes sense in many cases, so how canwe support them?45
  46. 46. Web AppsHow should web apps behave?• Basically like real apps• A one to one scale – one css pixel is one device pixel – no scaling• No pinch zoom of the whole apps, it is an app not a page!• It has to be able to deal with input46
  47. 47. Touch eventsMobile browsers support touch events in addition to mouse eventsThis allows apps to take advantage of multi touch etc. We need to support that.This means that each event has to go to the web process and be treated beforewe can use it on the UI for panning etc. A bit slower.Trick: Only send events to the web process if the page is actually using touchevents (have listeners registered).47
  48. 48. Viewport meta tagIntroduced by the iPhone.• Layouts with a width of 980 by default.• Works with most sites, but some break. ie. www.apple.com which needs a width > 1000 or so.• Added a meta tag to give control to the web authors48
  49. 49. Viewport meta tagFeatures:• Define layout width (and height, though seldom used in practice)• Define initial, minimum and maximum scale• Define non-user-scalable (basically remove pinch zoom)Nota bene: It is defines given portrait mode.<meta contents=“viewport” value=“width=320, maximum-scale=1”>Only comma allowed as separators, though some Android versions allowedsemicolon.49
  50. 50. Avoid hardcoded valuesThere was a need to avoid hardcoding values such as the the iPhone width of320.Introduced device-width, device-height <meta contents=“viewport” value=“width=device-width, maximum-scale=1”>50
  51. 51. DPI, a pixel is not a pixelFacts:• Most contemporal mobile pages were designed with the iPhone in mind• Buttons, hit areas are adapted to the sizes and DPI of the iPhone which is 160.First Android phones has the same DPI, but some low end models had lower andalmost all current models have a DPI of 240.For pages designed for the iPhone, they need to be scaled up by 1.5 for text, hitareas to not be too small. (240 / 160 = 1.5)Android defines a pixel at 160 as a DIP = density independent pixel.51
  52. 52. DPI and the meta tagAndroid added the target-densitydpi extension <meta contents=“viewport” value=“width=device-width”>on a 240 DPI device uses 320 as device-width. On the other hand <meta contents=“viewport” value=“width=device-width, target-densitydpi=device-dpi”>uses 420.52
  53. 53. Summing upMaking a browser for touch devices has different requirements than that of adesktop browser.Focus is touch interaction, requires touch events, engine/application separation,and many tricks to make things work smoothly.Web apps requires touch events and the ability to control whether pinch zoometc is allowed and how the contents should be laid out.53
  54. 54. Questions? I hope you enjoyed the show and have some good questions! Feel free to send questions to kenneth@webkit.org and you are welcome to join #qtwebkit at irc.freenode.net54

×