Multitouching your apps

Multitouching your apps
carlos@lanedo.com

Multitouch
● Touchscreen interaction with more than
one appendage
● Getting more common

And not only mobile devices...

Multitouch
● Recognition of the shapes of pressure
points

The good ol' times
● Single mouse pointer
● Single keyboard focus
● Nothing new since the 80s

So what now?
● Collaborative interfaces
● Multiple users
● Multiple pointer/keyboard pairs
● Multitouch
● Single user
● Single keyboard
● Multiple pointers

Device hierarchy
● Master: Devices visible in the screen
● Slave: Hardware devices
● Attached: Control a Master device
● Floating: Not attached to any Master device

Device hierarchy
typedef enum {
GDK_DEVICE_TYPE_MASTER,
GDK_DEVICE_TYPE_SLAVE,
GDK_DEVICE_TYPE_FLOATING
} GdkDeviceType;
GList * gdk_device_manager_list_devices (GdkDeviceManager *device_manager,
GdkDeviceType type);
GdkDevice * gdk_device_get_associated_device (GdkDevice *device);
GdkDeviceType gdk_device_get_device_type (GdkDevice *device);

Device hotplugging
● Signals in GdkDeviceManager
● device-added
● device-removed
● device-changed

Event interaction
● Events generated by input devices
● Button press/release
● Key press/release
● Grab broken
● …
● These events will contain information
about the device that generated them

GdkDevice* gdk_event_get_device (const GdkEvent *event);
/* EXPERIMENTAL */
GdkDevice* gdk_event_get_source_device (const GdkEvent *event);

Enabling events
● A GdkWindow only obeys Master devices
by default
● Slave devices need explicit events
activation
void gdk_window_set_device_events (GdkWindow *window,
GdkDevice *device,
GdkEventMask event_mask);
/* EXPERIMENTAL */
void gdk_window_set_source_events (GdkWindow *window,
GdkInputSource source,
GdkEventMask event_mask);

Enter/Leave events
● Emitted whenever a pointer enters or
leaves a GdkWindow
● For backwards compatibility, only a pair
of events is emitted if several pointers
are within the window area
● First pointer entering → GDK_ENTER_NOTIFY
● Last pointer leaving → GDK_LEAVE_NOTIFY

Enter/Leave events
● Each pointer emits its own Enter/Leave
events
● Widgets need to take this into account if
these functions are called
void gdk_window_set_support_multidevice (GdkWindow *window,
gboolean support_multidevice);
void gtk_widget_set_support_multidevice (GtkWidget *widget,
gboolean support_multidevice);

Where are thou, pointer?
● The old calls are non deterministic
● Basically, the position of a “favourite”
pointer is obtained, the client pointer

void gdk_display_get_device_state (GdkDisplay *display,
GdkDevice *device,
GdkScreen **screen,
gint *x,
gint *y,
GdkModifierType *mask);
GdkWindow * gdk_display_get_window_at_device_position
(GdkDisplay *display,
GdkDevice *device,
gint *win_x,
gint *win_y);
GdkWindow * gdk_window_get_device_position (GdkWindow *window,
GdkDevice *device,
gint *x,
gint *y,
GdkModifierType *mask);

Pointer and keyboard grabs
● Capture events from the device, and
confine them to a window
● Menus
● Drag and Drop
● Blocks interaction with further
applications

GdkGrabStatus gdk_device_grab (GdkDevice *device,
GdkWindow *window,
GdkGrabOwnership grab_ownership,
gboolean owner_events,
GdkEventMask event_mask,
GdkCursor *cursor,
guint32 time_);
void gdk_device_ungrab (GdkDevice *device,
guint32 time_);

Grab Ownership
Defines how the device owns control over
the grabbed window
● None: Any other devices are free to interact
● Window: The GdkWindow is effectively
blocked for any other device
● Application: The full application is blocked

Grab Ownership
● Several devices may hold a grab at the
same time, on the same or different
windows
● The device with most restrictive
ownership blocks the others

GTK+ grabs
● Similar to GDK grabs, but don't block
interaction with other applications
● Modal dialogs

void gtk_device_grab_add (GtkWidget *widget,
GdkDevice *device,
gboolean block_others);
void gtk_device_grab_remove (GtkWidget *widget,
GdkDevice *device);
/* To be used in grab-notify handler */
gboolean gtk_widget_device_is_shadowed (GtkWidget *widget,
GdkDevice *device);

Multitouch
● Each finger gets a device
● Devices emit individual events
● There is a need to gather and correlate
them
● Obtain an event packet
● Calculate the distance and relative angle
between 2 events
● GtkDeviceGroup and multidevice-event signal

/* EXPERIMENTAL */
GtkDeviceGroup * gtk_widget_create_device_group (GtkWidget *widget);
void gtk_widget_remove_device_group (GtkWidget *widget,
GtkDeviceGroup *group);
void gtk_device_group_add_device (GtkDeviceGroup *group,
GdkDevice *device);
void gtk_device_group_remove_device (GtkDeviceGroup *group,
GdkDevice *device);
GList * gtk_device_group_get_devices (GtkDeviceGroup *group);

/* EXPERIMENTAL */
typedef enum
{
GTK_EVENT_DEVICE_ADDED,
GTK_EVENT_DEVICE_REMOVED,
GTK_EVENT_DEVICE_UPDATED
} GtkMultiDeviceEventType;
struct _GtkMultiDeviceEvent
{
GtkMultiDeviceEventType type;
guint n_events;
GdkEventMotion **events;
GdkEventMotion *updated_event;
GdkDevice *updated_device;
};

gboolean gdk_events_get_distance (GdkEvent *event1,
GdkEvent *event2,
gdouble *distance);
gboolean gdk_events_get_angle (GdkEvent *event1,
GdkEvent *event2,
gdouble *angle);
gboolean gdk_events_get_center (GdkEvent *event1,
GdkEvent *event2,
gdouble *x,
gdouble *y);

Gestures
● Definition: “The act of moving the limbs or
body as an expression of though or
emphasis”
● Not really computable
● Gestures are a set of finger movements
over time, reduced to an expression with a
meaning,
● The logic to get an expression out of a gesture
is highly heuristic

Gestures (2)
● Gestures are something extending and
complementing multi-touch, rather on top
than part of it.

What's left?
● Consolidating multidevice-event
● Gesture support
● Further multidevice support for stock
widgets

Where's all this?
● Now in GTK+ master
● Methods marked as “experimental”:
git clone -b xi2-playground git://git.gnome.org/gtk+

Multitouching your apps

More Related Content

Viewers also liked

Similar to Multitouching your apps

More from FSCONS

Recently uploaded

Multitouching your apps