Multitouching your apps


Published on

Multitouch on Linux, more specifically GTK+/GNOME

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Multitouching your apps

  1. 1. Multitouching your apps
  2. 2. Multitouch ● Touchscreen interaction with more than one appendage ● Getting more common
  3. 3. And not only mobile devices...
  4. 4. Multitouch ● Recognition of the shapes of pressure points
  5. 5. The good ol' times ● Single mouse pointer ● Single keyboard focus ● Nothing new since the 80s
  6. 6. So what now? ● Collaborative interfaces ● Multiple users ● Multiple pointer/keyboard pairs ● Multitouch ● Single user ● Single keyboard ● Multiple pointers
  7. 7. And any combination...
  8. 8. Device hierarchy ● Master: Devices visible in the screen ● Slave: Hardware devices ● Attached: Control a Master device ● Floating: Not attached to any Master device
  9. 9. Device hierarchy typedef enum { GDK_DEVICE_TYPE_MASTER, GDK_DEVICE_TYPE_SLAVE, GDK_DEVICE_TYPE_FLOATING } GdkDeviceType; GList * gdk_device_manager_list_devices (GdkDeviceManager *device_manager, GdkDeviceType type); GdkDevice * gdk_device_get_associated_device (GdkDevice *device); GdkDeviceType gdk_device_get_device_type (GdkDevice *device);
  10. 10. Device hotplugging ● Signals in GdkDeviceManager ● device-added ● device-removed ● device-changed
  11. 11. Event interaction ● Events generated by input devices ● Button press/release ● Key press/release ● Grab broken ● … ● These events will contain information about the device that generated them
  12. 12. GdkDevice* gdk_event_get_device (const GdkEvent *event); /* EXPERIMENTAL */ GdkDevice* gdk_event_get_source_device (const GdkEvent *event);
  13. 13. Enabling events ● A GdkWindow only obeys Master devices by default ● Slave devices need explicit events activation void gdk_window_set_device_events (GdkWindow *window, GdkDevice *device, GdkEventMask event_mask); /* EXPERIMENTAL */ void gdk_window_set_source_events (GdkWindow *window, GdkInputSource source, GdkEventMask event_mask);
  14. 14. Enter/Leave events ● Emitted whenever a pointer enters or leaves a GdkWindow ● For backwards compatibility, only a pair of events is emitted if several pointers are within the window area ● First pointer entering → GDK_ENTER_NOTIFY ● Last pointer leaving → GDK_LEAVE_NOTIFY
  15. 15. Enter/Leave events ● Each pointer emits its own Enter/Leave events ● Widgets need to take this into account if these functions are called void gdk_window_set_support_multidevice (GdkWindow *window, gboolean support_multidevice); void gtk_widget_set_support_multidevice (GtkWidget *widget, gboolean support_multidevice);
  16. 16. Where are thou, pointer? ● The old calls are non deterministic ● Basically, the position of a “favourite” pointer is obtained, the client pointer
  17. 17. void gdk_display_get_device_state (GdkDisplay *display, GdkDevice *device, GdkScreen **screen, gint *x, gint *y, GdkModifierType *mask); GdkWindow * gdk_display_get_window_at_device_position (GdkDisplay *display, GdkDevice *device, gint *win_x, gint *win_y); GdkWindow * gdk_window_get_device_position (GdkWindow *window, GdkDevice *device, gint *x, gint *y, GdkModifierType *mask);
  18. 18. Pointer and keyboard grabs ● Capture events from the device, and confine them to a window ● Menus ● Drag and Drop ● Blocks interaction with further applications
  19. 19. GdkGrabStatus gdk_device_grab (GdkDevice *device, GdkWindow *window, GdkGrabOwnership grab_ownership, gboolean owner_events, GdkEventMask event_mask, GdkCursor *cursor, guint32 time_); void gdk_device_ungrab (GdkDevice *device, guint32 time_);
  20. 20. Grab Ownership Defines how the device owns control over the grabbed window ● None: Any other devices are free to interact ● Window: The GdkWindow is effectively blocked for any other device ● Application: The full application is blocked
  21. 21. Grab Ownership ● Several devices may hold a grab at the same time, on the same or different windows ● The device with most restrictive ownership blocks the others
  22. 22. GTK+ grabs ● Similar to GDK grabs, but don't block interaction with other applications ● Modal dialogs
  23. 23. void gtk_device_grab_add (GtkWidget *widget, GdkDevice *device, gboolean block_others); void gtk_device_grab_remove (GtkWidget *widget, GdkDevice *device); /* To be used in grab-notify handler */ gboolean gtk_widget_device_is_shadowed (GtkWidget *widget, GdkDevice *device);
  24. 24. Multitouch ● Each finger gets a device ● Devices emit individual events ● There is a need to gather and correlate them ● Obtain an event packet ● Calculate the distance and relative angle between 2 events ● GtkDeviceGroup and multidevice-event signal
  25. 25. /* EXPERIMENTAL */ GtkDeviceGroup * gtk_widget_create_device_group (GtkWidget *widget); void gtk_widget_remove_device_group (GtkWidget *widget, GtkDeviceGroup *group); void gtk_device_group_add_device (GtkDeviceGroup *group, GdkDevice *device); void gtk_device_group_remove_device (GtkDeviceGroup *group, GdkDevice *device); GList * gtk_device_group_get_devices (GtkDeviceGroup *group);
  26. 26. /* EXPERIMENTAL */ typedef enum { GTK_EVENT_DEVICE_ADDED, GTK_EVENT_DEVICE_REMOVED, GTK_EVENT_DEVICE_UPDATED } GtkMultiDeviceEventType; struct _GtkMultiDeviceEvent { GtkMultiDeviceEventType type; guint n_events; GdkEventMotion **events; GdkEventMotion *updated_event; GdkDevice *updated_device; };
  27. 27. gboolean gdk_events_get_distance (GdkEvent *event1, GdkEvent *event2, gdouble *distance); gboolean gdk_events_get_angle (GdkEvent *event1, GdkEvent *event2, gdouble *angle); gboolean gdk_events_get_center (GdkEvent *event1, GdkEvent *event2, gdouble *x, gdouble *y);
  28. 28. Gestures ● Definition: “The act of moving the limbs or body as an expression of though or emphasis” ● Not really computable ● Gestures are a set of finger movements over time, reduced to an expression with a meaning, ● The logic to get an expression out of a gesture is highly heuristic
  29. 29. Gestures (2) ● Gestures are something extending and complementing multi-touch, rather on top than part of it.
  30. 30. What's left? ● Consolidating multidevice-event ● Gesture support ● Further multidevice support for stock widgets
  31. 31. Where's all this? ● Now in GTK+ master ● Methods marked as “experimental”: git clone -b xi2-playground git://
  32. 32. Questions?
  33. 33. Thank you!