Qt WebKit going Mobile

Qt WebKit goes MobileKenneth Rohde ChristiansenFOSDEM 2011

Introduction Welcome everyone!2

Introduction• Name is Kenneth Rohde Christiansen• Works in the Nokia, Application and Service Frameworks subentry (basically Qt Development Frameworks + browser/web teams)• Among other, part of the Qt team developing and adapting WebKit for Nokia3

Relevance to FOSDEM• Qt and Qt WebKit are fully open source projects, and we actively encourage contributions.• Long time Linux support, now also targeting the open source MeeGo project4

Today Today we will identify what is needed to make a great mobile browser5

Today … and how to accomplish that6

Rendering Engine7

First we need a great, flexible Web Engine WebKit is our recommendation8

Rendering Engine Why ?• WebKit is already a part of Qt (and MeeGo)• It is highly modular given it is not a browser, but an API for a.o. making one• That is what I know about ;-)9

Rendering EngineBut the most important reason:WebKit is the engine powering most of the mobile web browser in useThat means: It is proven technology – it can be done Less bootstrapping – a lot of things already in place10

Targeting a small form factor and different input11

Adaptions• Nokia’s track record not that great, or is it?• First to bring WebKit to mobile phones and did quite well at the pointLot has changed since then • Availability of 3G and Wi-fi • Reduced data cost / plans • More powerful devices – with larger screens • Totally new interaction model; touch!12

AdaptionsThis means that, people today actually make use of their data plansThe better connection and faster devices means that we can load and processnormal pages and are not limited to WAP pages as before.ie. brings the real internet to devices13

AdaptionsWe have lots of challenges: •  Small form factor – things doesn’t fit, becomes too small •  Load and layout/paint is slow compared with desktop •  Touch interaction requires a very responsive user experience •  Fingers are huge – content is small •  Web apps… let’s talk about that later14

Adaptions: small form factor• We need good support for zooming Some great ideas exists •  Double tap to zoomable area (vertically centered where you clicked); zoom fully out if trying to zoom to a similar target. •  Pinch zoom, preferable with moving – pinch and move simoustainly.15

Double tap to zoomable area16

Pinch zoom17

Adaptions: small form factorDid that solve our issues? Still too small text, or text being cut when zoomed in too much.Solutions? Android: text reflowing iOS: enforce min. font sizes depending on the width of the zoomable area (eg. <div>).18

Text reflowingNot my personal favorite• Hard to get right – lots of bugs related to that on Android• Relayouting is expensive• “interrupted” animation Text reflowing on Android19

Adaptions: big, imprecise fingers• Problem: We need to interact with the content •  We have a very little screen size, so we miss overview and a way to scroll to the important area, fast.• Solution? •  For news sites with columns: Double tap to area / back out, shows overview and makes you go to an area fast. •  For other pages we need another solution: •  Panning; pan around with your finger •  Added kinetics; when you lift you finger it will continue your motion with kinetics20

Adaptions: big, imprecise fingersGive visual feedback:• Scroll indicators (fade out when not in use; focus is contents)• A bounce back or similar way to show the user reached the boundariesDeal with imprecise fingers:• Ignore kinetics for minor moves – We do not want a finger ‘click’ to start kinetic panning• Lock panning in the horizontal or vertical direction• More complicated: Improve hit testing to look for area close by.21

Hit testingFingers click an area, not a single point as a mouse• We need heuristics to guess where the user tries to click: This depends on finger geometry.• We need the hit testing to look for clickable things close to the actual click.22

Frame flattening• Panning depends on the whole web content being one• Thus, we need to eliminate sub-scrollable areas• Expand iframes to the size of their contents (done of iOS, etc)• Expand framesets to become one flattened areaExamples coming up ;-)23

Frame flattening No one wants to interact with a page like this24

Frame flattening Now what about this? This is easily pannable25

ConclusionTouch is a very nice way to interact with web contents.It feels quite natural and easy.But there is a one big but…You feel if the page blocks26

Now why is that?Web browsers actually stall a lot •  During loading •  While laying out the content •  While painting •  While running JavaScriptYou can feel this on the desktop, but desktops are fast, and more important,you don’t touch the contentsBig screen, less need for scrolling, not pinch zoom, etc27

Touch requires a non-blocking UI28

Back to the drawing board• Do we need to redesign the engine?• Or is there a simpler ideaRedesigning the engine will be very complex and there is not much we can dowith regard to loading.29

Solution Separate UI and web engine in different threads or processes30

Two ideas1.  Store painting sequences and replay them on the UI.2.  Share painting area (shared memory, or memory mapped IO)Quite complicated.We want a non blocking UI, so everything has to have an asynchronous API.31

Process/thread separationIn a way we need the process separation just like e.g. Chrome, but not forsecurity reasons (that is an extra bonus) but for the sake of a non-blocking userexperience.Example: Fixed elements cannot be handled/painted by the web process as itwould be invoked for each scrolling step, meaning it would block:We need to decouple the dependency on the web process for all kinds of interaction(panning, pinching, rotation, etc)32

Asynchronous TilingBasic idea:Imagine that the contents was just one big image. Then we could easily ensure60 fps panning, pinch zoom, etc.That is the basic idea!33

Basics Viewport Full size tiles Edge tiles34

Basics Viewport Full size tiles Edge tiles Tiling means that we device our contents into tiles and we handle scrolling etc. on the UI side. Basically you can think of the contents being one big tiled image on the UI side. In practice an intelligent image which doesn’t need to paint tiles not partially visible.35

BasicsWhen a tile becomes visible: •  If painted, blit it. •  If unavailable, show some kind of indicator as background image, ie. checker boards (known from graphics apps).When web page area change: •  If visible, update the changed area of the tiles (all tiles need to be in sync) – you can mark dirty on non-visible tiles. •  The dirty regions can eventually be merged smartly.36

The magic numberWe want to ensure 60fps for panning, pinch zoomand rotation.Repaints, loading and JavaScript makes it hard to keep that promise.So we need to: •  Suspend JavaScript execution while panning, pinch etc. •  Defer loads •  Ignore repaints from the web process.37

Pinch zoomingWhen zooming painting is involved (we change the scale)We cannot control how long it will take to paint – painting web contents isbasically painting rectangles, lines etc… very vector graphics like.Solution: •  Freeze the backing store (do not repaint tiles; the page is suspended) •  Scale the tiles (basically scaling an image; zooming in will show pixelated contents) •  Unfreeze on end, repainting all tiles at the right scale38

RotationRotation is a bit tricky.For normal pages we basically multiply/divide with the device width/heightfactor, so that you look at the same contents.39

Rotation, layoutingThe iPhone gave page authors some ways to control how the page is laid outFor instance, the page author can decide that the contents has a max scale:Example: Google.com has a min and max scale of 1.0, so instead of applying thedifference in the width in landscape and portrait, we just rotate!40

Rotation, layouting continued Now what happens, this doesn’t look good!41

Relayout is the solution We need to relayout with a different width42

RotationRecall that layouts are expensive, so if we layout during the rotation we cannotensure 60 fps.Thus, only layout after rotation has finished; with the above as intermediate step(show checkerboards)43

RotationSumming up:Rotation starts: freeze backing store / suspend JS, paint, loadsDuring rotation: Keep old contents, show checkerboards where no contents isAfter rotation: Relayout (if needed) and unfreeze / resume.44

Web Apps!Let’s not get into why web apps makes sense, or when then do. That would beanother presentation JIf interested, check out: http://tinyurl.com/4hx36lt which is my presentation:“Connecting technology for great experiences” which touches QML and Web forapplication developers.Web Apps makes sense in many cases, so how canwe support them?45

Web AppsHow should web apps behave?• Basically like real apps• A one to one scale – one css pixel is one device pixel – no scaling• No pinch zoom of the whole apps, it is an app not a page!• It has to be able to deal with input46

Touch eventsMobile browsers support touch events in addition to mouse eventsThis allows apps to take advantage of multi touch etc. We need to support that.This means that each event has to go to the web process and be treated beforewe can use it on the UI for panning etc. A bit slower.Trick: Only send events to the web process if the page is actually using touchevents (have listeners registered).47

Viewport meta tagIntroduced by the iPhone.• Layouts with a width of 980 by default.• Works with most sites, but some break. ie. www.apple.com which needs a width > 1000 or so.• Added a meta tag to give control to the web authors48

Viewport meta tagFeatures:• Define layout width (and height, though seldom used in practice)• Define initial, minimum and maximum scale• Define non-user-scalable (basically remove pinch zoom)Nota bene: It is defines given portrait mode.<meta contents=“viewport” value=“width=320, maximum-scale=1”>Only comma allowed as separators, though some Android versions allowedsemicolon.49

Avoid hardcoded valuesThere was a need to avoid hardcoding values such as the the iPhone width of320.Introduced device-width, device-height <meta contents=“viewport” value=“width=device-width, maximum-scale=1”>50

DPI, a pixel is not a pixelFacts:• Most contemporal mobile pages were designed with the iPhone in mind• Buttons, hit areas are adapted to the sizes and DPI of the iPhone which is 160.First Android phones has the same DPI, but some low end models had lower andalmost all current models have a DPI of 240.For pages designed for the iPhone, they need to be scaled up by 1.5 for text, hitareas to not be too small. (240 / 160 = 1.5)Android defines a pixel at 160 as a DIP = density independent pixel.51

DPI and the meta tagAndroid added the target-densitydpi extension <meta contents=“viewport” value=“width=device-width”>on a 240 DPI device uses 320 as device-width. On the other hand <meta contents=“viewport” value=“width=device-width, target-densitydpi=device-dpi”>uses 420.52

Summing upMaking a browser for touch devices has different requirements than that of adesktop browser.Focus is touch interaction, requires touch events, engine/application separation,and many tricks to make things work smoothly.Web apps requires touch events and the ability to control whether pinch zoometc is allowed and how the contents should be laid out.53

Questions? I hope you enjoyed the show and have some good questions! Feel free to send questions to kenneth@webkit.org and you are welcome to join #qtwebkit at irc.freenode.net54

Qt WebKit going Mobile

by Kenneth Rohde Christiansen on Feb 22, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

1 Embed 1

Statistics

Qt WebKit going Mobile — Presentation Transcript