Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

A new interface between smart device and web using html5 web socket and qr code


Published on

My graduation thesis.

Published in: Engineering
  • Be the first to comment

A new interface between smart device and web using html5 web socket and qr code

  1. 1. A New Interface Between Smart Device and Web using HTML5 WebSocket and QR-Code Ikwhan Chang Department of Computer Science and Engineering Chung-Ang University Seoul, South Korea Abstract—In the era of smart device, there are only few interactions between smart device and Web on desktop. For Web browsing, the action of smart device is more diverse than original interface of the desktop. For example, we can touch or swipe via mobile device, which are not allowed by keyboard and mouse. The main thesis question of original computer-Web interface is why we only use keyboard and mouse for Web browsing. In this thesis, I propose a new way to interface between smart device and Web on desktop, studying a HTML5 WebSocket and the optimization of back-end , making a simple remote game demo, providing a new tools for remote Web controlling via QR-Code and smart device. I. INTRODUCTION In Sep 2014, Apple announced Apple iWatch [1], a smart watch that can run application for check health status and more functions do. Likewise, Samsung, a massive world’s well- known mobile device manufacturing company, was also showed the newest mobile device such as Galaxy S5, Galaxy Gear Fit. Especially, Galaxy Gear Fit is focused on health-care society, collecting the owner’s information: Heartbeat or Walk Steps and of course the owner can check their mobile information like SMS, Phone, Calendar even though they cannot open their host device (in this case, iPhone). Moreover, many companies will join that competitive markets kind of wearable smart device. In the era of high technology, most of IT companies focus on new technologies such as Cloud Computing or N-Screen, which means most of people has two or more smart devices and they do not want to make their data repeatedly for sharing in their devices so they want to OSMU (one-source multi-use) their personal data. I focused on these new technologies. In particular, the Cloud Computing and N-Screen, especially Hybrid Application, is the optimal venue for us to accomplish my research on human-centered interface and its applications. While I took a class since freshman, the coursework of my department carry me to study on two related skills: 1) Java Application on Socket Server 2) Android and Mobile Web Application To achieve these study goals, I first made some server- sided Java application such as messenger or online game. So, I was already familiar with the communication between client and server. Secondly, I made Android application based on Apache Cordova, which can be used by developing Hybrid Application. Thus, I knew how to implement mobile web as well as what is consist of HTML such as Tags, DOM, Script or CSS. While I made my own server, I knew that there are a lot of server-sided technologies such as Apache Tomcat [2], NginX [3], Node.js [4] or Netty [5]. Since I learned Java Language, I already was familiar with Tomcat and Netty. In course of Java Programming, I tried to make web server. In my case, I made online game with Facebook OAuth login [6]. For his game, he should implement Java socket server, which provide the channel of packet between client and server. Since he spent at least two months for implementing server, I have the technology of server-sided and then I just want to make some practical case by his server-sided skills. Thus, I decided to make the I/O server using some minor server-sided technology such as NginX or NodeJS. Currently, there is some issue about the communication between human and device. It called Human-centered Computing. At the Fluid Lab, MIT MediaLab, they conducted the ‘THAW’ projects, a novel interaction system that allows a collocated large display and small handheld devices to seamlessly work together [7]. The smartphone acts both as a physical interface and as an additional graphics layer for near- surface interaction on a computer screen. In this thesis, I propose a new way to interface between smart device and Web. By proposing, I will demonstrate a simple soccer game that will be based on these trends. This game will be remote controlled using user’s own smart devices such as Galaxy or iPhone and they can see their game play view through web pages in their laptop or PC. II. BACKGROUND I was worked as software engineer for ten years. Since I have handled HTML5, JavaScript and jQuery, I decide to use these skills for frontend, as well as I want to handle the packet that included the behavior of user. By utilize the packet, I want to provide some entertaining application and thus I decide to
  2. 2. make remote soccer game. The game is based on HTML/JavaScript and customized Javascript Library. I used ‘’ [8] because provide WebSocket protocol to push/notification between client and server. Nowaday, the performance of websocket is very well because most of web browser provides that technology in the web standard for push/notification. Moreover, it is much faster than Polling method. [9] Also, there are just few packets since the game has just two players. The most important thing, however, is user can REMOTE control their game using their smart devices for playing game that will be played in web pages. The pushing notification between server and client was implemented by ‘’, a web-socket technology of web. ‘Node.js’ is main web server. Also, the remote controller was ran on any kind of smart device (Android or iOS) since I made web page using HTML5’s Canvas and jQuery mobile, QR-Code library for jQuery. Controller is actually not a app, but mobile web. At first time, the operating system of my bear-wearer server(customized server) was Windows 2013 Server Enterpriser R2. But I found that there was some slow performance if I use that O/S because the performance of and Node.js is resonated in Linux-based O/S. Since I already installed all of the server-sided software such as Nginx and MariaDB, there were huge jobs to move from Windows to Linux. Plus, since the developer of KineticJS [10] was disappeared, KineticJS has no more update. So, I have to use pure-HTML5 Canvas Animation rather than use KineticJS. Server & Client Specification is as below. A. Server • for push notification • NginX • Node.js + Express.js [11] : for web template • JSON [12] : data format B. Client • Adobe Dreamweaver CC • Adobe Edge for HTML5 web app[13] • KineticJS for Graphic Library • jQuery C. Cooperation System • Jenkins : Continuous Integration [14] • Github : Repository • JIRA : Issue tracker, sharing TODO [15] D. Tools • IntelliJ IDEA [16] • Adobe Edge • Adobe Photoshop: for UI E. Etc • Ngxqrctl.js: This is the NGX QR Controller, Javascript Library (jQuery Plugin). [17] • NGXRCS (NGX Remote Control Server) III. BODY OF PROJECT A. WebSocket [18] Since the communication between server and client was mainly made by HTML5 WebSocket, separating parts of application such as webserver, game and controller is based on HTML5 WebSocket. The HTML5 WebSockets specification defines an API that enables web pages to use the WebSockets protocol for two-way communication with a remote host. It introduces the WebSocket interface and defines a full- duplex communication channel that operates through a single socket over the Web. HTML5 WebSockets provide an enormous reduction in unnecessary network traffic and latency compared to the unscalable polling and long-polling solutions that were used to simulate a full-duplex connection by maintaining two connections. HTML5 WebSockets account for network hazards such as proxies and firewalls, making streaming possible over any connection, and with the ability to support upstream and downstream communications over a single connection, HTML5 WebSockets-based applications place less burden on servers, allowing existing machines to support more concurrent connections. The following figure shows a basic WebSocket-based architecture in which browsers use a WebSocket connection for full-duplex, direct communication with remote hosts. One of the more unique features WebSockets provide is its ability to traverse firewalls and proxies, a problem area for many applications. Comet-style applications typically employ long-polling as a rudimentary line of defense against firewalls and proxies. The technique is effective, but is not well suited for applications that have sub-500 millisecond latency or high throughput requirements. Plugin-based technologies such as Adobe Flash, also provide some level of socket support, but have long been burdened with the very proxy and firewall traversal problems that WebSockets now resolve. A WebSocket detects the presence of a proxy server and automatically sets up a tunnel to pass through the proxy. The tunnel is established by issuing an HTTP CONNECT statement to the proxy server, which requests for the proxy server to open a TCP/IP connection to a specific host and port. Once the tunnel is set up, communication can flow unimpeded through the proxy. Since HTTP/S works in a similar fashion, secure WebSockets over SSL can leverage the same HTTP CONNECT technique. Note that WebSockets are just beginning to be supported by modern browsers (Chrome now supports WebSockets natively). However, backward-
  3. 3. compatible implementations that enable today's browsers to take advantage of this emerging technology are available. WebSockets—like other pieces of the HTML5 effort such as Local Storage and Geolocation—was originally part of the HTML5 specification, but was moved to a separate standards document to keep the specification focused. WebSockets has been submitted to the Internet Engineering Task Force (IETF) by its creators, the Web Hypertext Application Technology Working Group (WHATWG) [19]. Authors, evangelists, and companies involved in the standardization still refer to the original set of features, including WebSockets, as "HTML5." Fig.1 The flow of WebSockets Fig. 2 Main technology B. Protocol The WebSocket protocol was designed to work well with the existing Web infrastructure. As part of this design principle, the protocol specification defines that the WebSocket connection starts its life as an HTTP connection, guaranteeing full backwards compatibility with the pre-WebSocket world. The protocol switch from HTTP to WebSocket is referred to as the WebSocket handshake. C. Game Rule The remote soccer game has several rules that I set up from the first. Two players can play this game using one ball and each player has their own goal post. The ball only can move inside the rectangle area. Whenever the ball collide with users or wall (rectangle area), they move on the law of reflection. It is similar logic with billiard. Users and ball are moving on the any coordinated by x-y. If the ball goes into the opposite's goalpost during the game, the user who got the goal earns a single point. The game keeps going on after 3 minutes. Therefore, I used the time function and server-client system, which can make users play at the same time on the ground. D. Architecture There are four separated parts of implementation. 1) Soccer Game (HTML5 Canvas View) 2) Controller (HTML5 Canvas + jQuery Mobile) 3) Server (NginX + NodeJS + WebSocket) 4) QR-Controller Library Since ‘’ is main library, all parts of the project loaded the library into their raw source code. This library uses of the communication between server and client. Moreover, since is a part of NodeJS, I can easily make the web application using JavaScript. Fig.3 System environment IV. IMPLEMENTATION & EVALUATION A. Web Application Server First, I mainly strive to make web server. At first time, the operating system of server was Windows 2012 R2. However, I have changed my server since there are several issues such as worm virus or the speed of server. Finally, the server was changed from Windows to CentOS. The web routing server is ‘NginX’. In reality, I was familiar with ‘Apache Tomcat’ or ‘Netty’. However, since I want to know some latest technology of web, we decide to use ‘NginX’ for routing, as well as ‘NodeJS’ for the web application server. var app = require('express')(); var http = require('http').Server(app); var io = require('')(http); app.set('view engine', 'jade'); app.get('/c/:siteID', function(req, res){ res.render('controller', { siteID: req.params.siteID}); }); var devices = []; var sites = []; var clients = []; io.on('connection', function(client) { client.on("join", function(type,siteID) {
  4. 4. if(type == "site"){ sites.push(client); client.emit('join ok',; console.log(; } if(type == "device") { for(var i in sites){ if(sites[i].id == siteID){ var tdevice = {"site":sites[i], "client":client}; devices.push(tdevice); break; } } } }); client.on("exit", function(type,siteID) { if(type == "site"){ sites.push(client); for(var i in sites) { if(siteID == sites[i].id){ sites.splice(i, i); break; } } } if(type == "device") { for(var i in devices){ if(devices[i] =={ devices.splice(i, i); break; } } } }); // Simple Moving client.on('move control', function(event) { for(var i in devices) { if(devices[i] =={ devices[i].site.emit('move', event); } } }); // Button Down client.on('button down', function(event) { for(var i in devices) { if(devices[i] =={ console.log(event); devices[i].site.emit('btnDown', event); } } }); // SWipe Left client.on('swipeleft', function() { console.log("swipeleft"); for(var i in devices) { if(devices[i] =={ devices[i].site.emit('swipeleft'); } } }); // SWipe Right client.on('swiperight', function() { console.log("swiperight"); for(var i in devices) { if(devices[i] =={ devices[i].site.emit('swiperight'); } } }); // SWipe Up client.on('swipeup', function() { console.log("swipeup"); for(var i in devices) { if(devices[i] =={ devices[i].site.emit('swipeup'); } } }); // SWipe Down client.on('swipedown', function() { console.log("swipedown"); for(var i in devices) { if(devices[i] =={ devices[i].site.emit('swipedown'); } } }); http.listen(3000, function() { console.log('listening on *:3000'); }); Fig. 4 Source code of NodeJS B. Controller For control the ball of player, I need to move the character remotely using ‘’ as well as user-friendly design. Therefore, I decide to use jQuery Mobile, which provide to easily make mobile web. Moreover, I used ‘Express.js’, which is a part of NodeJS and I can use that kind of template engine with easily install. For Example, if some user installed NodeJS in order to web application server, there is several options for endpoints. Although most of user use the normal HTML, I use ‘Express.js’ since I want to template engine for remove duplicated HTML. The final result of the controller is as follow: doctype html
  5. 5. html head title NGX Controller meta(charset="utf-8") meta(name="viewport",width=device-width,initial- scale=1.0,maximum-scale=1.0,minimum-scale=1.0,user-scalable=0) script(type='text/javascript',src='// 1.11.1.min.js') link(rel="stylesheet",href='') script(type='text/javascript',src='//') style include style.css body div(data-role="page",id="ngxSwipeDiv") div(data-role="header",data-theme="b") h1 The NGX Controller a(href="#popupDialog" data-rel="popup" data-position- to="window" id="showMenu",ui-btn-right ui-btn ui-btn-b ui-btn- inline ui-mini ui-corner-all ui-btn-icon-right ui-icon-check) Help div(data-role="popup" id="popupDialog" data-overlay- theme="b" data-theme="b" data-dismissible="false" style="max- width:400px;") div(data-role="header" data-theme="a") h1 Help div(role="main" class="ui-content") h3 Swipe p Swipe is simple swipe function. a(href="#" class="ui-btn ui-corner-all ui-shadow ui-btn- inline ui-btn-b" data-rel="back") Close div(data-role="main",class="ui-content",data-theme="b") canvas(class="ngxCtlPanel",id="ngxSwipePanel",style="border: 1px solid #4d4d4d;text-align:center;") div(data-role="footer") div(data-role="navbar" data-iconpos="bottom") ul li a(href="#ngxSwipeDiv",data-icon="grid",class="ui-btn- active") Swipe li a(href="#ngxMoveXyDiv",data-icon="star") Move XY li a(href="#ngxWsadDiv",data-icon="gear") WSAD div(data-role="page",id="ngxMoveXyDiv") div(data-role="header",data-theme="b") h1 The NGX Controller a(href="#popupDialog" data-rel="popup" data-position- to="window" id="showMenu",ui-btn-right ui-btn ui-btn-b ui-btn- inline ui-mini ui-corner-all ui-btn-icon-right ui-icon-check) Help div(data-role="main",class="ui-content",data-theme="b") canvas(class="ngxCtlPanel",id="ngxWsadPanel",style="border: 1px solid #4d4d4d;text-align:center;") div(data-role="footer") div(data-role="navbar" data-iconpos="bottom") ul li a(href="#ngxSwipeDiv",data-icon="grid") Swipe li a(href="#ngxMoveXyDiv",data-icon="star",class="ui-btn- active") Move XY li a(href="#ngxWsadDiv",data-icon="gear") WSAD div(data-role="page",id="ngxWsadDiv") div(data-role="header",data-theme="b") h1 The NGX Controller a(href="#popupDialog" data-rel="popup" data-position- to="window" id="showMenu",ui-btn-right ui-btn ui-btn-b ui-btn- inline ui-mini ui-corner-all ui-btn-icon-right ui-icon-check) Help div(data-role="main",class="ui-content",data-theme="b") div(id="divButtons") input(type="button",data-inline="true",value="") input(type="button",data- inline="true",value="W",class="btnWsad") br input(type="button",data- inline="true",value="A",class="btnWsad") input(type="button",data- inline="true",value="S",class="btnWsad") input(type="button",data- inline="true",value="D",class="btnWsad") div(data-role="footer") div(data-role="navbar" data-iconpos="bottom") ul li a(href="#ngxSwipeDiv",data-icon="grid") Swipe li a(href="#ngxMoveXyDiv",data-icon="star") Move XY li a(href="#ngxWsadDiv",data-icon="gear",class="ui-btn- active") WSAD script(src='/') script(type='text/javascript'). var socket = io(); socket.emit('join', 'device', '#{siteID}'); $(window).unload(function() { socket.emit('exit', 'device', '#{siteID}'); }); script include script.js Fig. 5 Controller implemented by Express.js C. QR-Code Library For using the remote soccer game, user need to connect via QR-Code for make their smartphone. For make that function, I first using the ‘Coffeescript[20]’, a little language that compiles into JavaScript, for make jQuery library. # # 2014 Matthew LAB Product # NullPointException QR Controller # Programming by Matthew, Chang( # If you have any sort of question, let me know my email address # # # =============== Program Usage =============== # This Coffeescript script is based on jQuery Plugin # and it will send user's data for customized server url # user need to setting initialize variables(position, surl, debug)
  6. 6. # the default value of surl is : # ============================================= # # Getting a familiar reference to jQuery $ = jQuery # An abstract class that provide jQuery plugin setup functionalities. class jQueryPlugIn # Redefine this dictionary to specify default options @defaultOptions: {} # The default constructor calls the initialize method and set the jQuery element. # Remember to call super in subclasses if you want to maintain this behaviour. constructor: (@element, options) -> @initialize options # Method to initialize the plugin instance with the given options # This method could be called initialize: (@options) -> # Install a class as a jQuery plugin. Assuming that myClass extends jQueryPlugIn it can than be installed with: # myClass.installAsjQueryPlugIn() @installAsjQueryPlugIn: (pluginName = @name) -> pluginClass = @ $.fn[pluginName] = (options, args...) -> options = $.extend pluginClass.defaultOptions, options or {} if $.type(options) is "object" return @each () -> $this = $(this) instance = $ pluginName if instance? if $.type(options) is "string" instance[options].apply instance, args else if instance.initialize? instance.initialize.apply instance, [options].concat args else plugin = new pluginClass $this, options, args... $ pluginName, plugin $this.addClass pluginName $this.bind "destroyed.#{pluginName}", () -> $this.removeData pluginName $this.removeClass pluginName $this.unbind pluginName plugin.destructor() plugin # NPE QR Control Class class npeQrCtl extends jQueryPlugIn Socket=0 @defaultOptions: position: 'bottom' width:50 height:50 color:"#3a3" background:"#fff" surl: '' debug: false constructor: (@element, options) -> super @element, options ext = this #Init Socket $.getScript @options.surl+"", (data, textStatus, jqxhr) -> #Setting functions Socket = io.connect(options.surl) Socket.emit('join','site'); Socket.on 'join ok', (siteID) -> #Create QR-Code ext.element.qrcode({ "render": "div", "width": ext.options.width, "height": ext.options.height, "color": ext.options.color, "background":ext.options.background, "text": ""+encodeURIComponent(siteID) }); return Socket.on 'move', (event) -> #ext.moveCircle event ext.element.trigger('move',event) return #Socket.on 'gyro move', (event) -> #ext.moveCircle event # ext.element.trigger('gyro_move',event) # return Socket.on 'btnDown', (event) -> ext.element.trigger('btnDown',event) return Socket.on 'swipeleft', () -> ext.element.trigger('swipeleft') return Socket.on 'swiperight', () -> ext.element.trigger('swiperight') return Socket.on 'swipeup', () -> ext.element.trigger('swipeup') return Socket.on 'swipedown', () -> ext.element.trigger('swipedown') return return initialize: (@options) -> super @options return # Installing the plugin as 'SimpleTabs' npeQrCtl.installAsjQueryPlugIn() Fig.6
  7. 7. For use this library for remote controlling via their smartphone, web developer need to allocate some HTML document object models such as DIV layer or tags. Afterward, user need to initialize the function of ‘npe-qrctl’ and then user need to bind some action of library such as move, swipe or button down. Fig.7 is shown how developer can initialize library and Fig.8 is shown how developer can handle the behavior of user’s touch move. Table.1 shows the list of remote event. $( "#npe_area" ).npeQrCtl({ 'position':'top', 'width':10, 'height':10 }); Fig.7 Initialize to npe-qrctl.js $('#npe_area').bind('move',function(event,x,y){ $.moveCircle(x,y); }); Fig.8 Example code of the event of move action Event Name Argument Description move e=(x,y) User's touching X,Y coordinate btnDown e=('W' or 'S' or 'A' or 'D') WSAD Button swipeleft Null Is left swiped swiperight Null Table.1 List of user’s action D. Game Intro and Join For game intro, I used Adobe Edge Animation, a tool for HTML5 and CSS3 transitions. Therefore, user can see some animation when they run the ‘index.html’ file of game. Moreover, for the connection of user via QR-Code, I attached the npe-qrctl.js into two DIV tags. Fig.10 is shown in game intro. Player can use his own device via QR-Code. Fig.11 is shown the change of QR-code after user connected. Finally, if there are two users, it is necessary to set the score of game from any users. Fig.12 is shown how can user set the score of game via NPE Controller. After set the score, if user touch the start button, game will be started automatically. Fig. 9 Adobe Edge Animation for game intro Fig.10 The intro view of game Fig.11 Changed color of QR-code after user connected Fig.12 How can user set the score of game via QR- Controller E. Game Main There is one HTML5 canvas tag for drawing game. For implementation the main views of game, I first add the background SVG [21], an XML-based vector image format for two-dimensional graphics with support for interactivity and animation, and then loaded SVG files of two character of players and finally loaded SVG file of ball. In reality, most of my game logic is controlled by HTML5 animation status. Fig.13 shows that how can I rendered game resources into the HTML5 canvas. By animate, the logic of game is handled in animation function such as the movement of user. Fig.14 shows the game main. $.animate = function (startTime) { // update var time = (new Date()).getTime(); if (pause[0] == 0 || pause[1] == 0) { var timeDiff = time - startTime; readyTime -= timeDiff;
  8. 8. if (player1_score >= scorePerGame) { if (readyTime > 0) { $.drawResult('WIN', 'LOSE'); } } else if (player2_score >= scorePerGame) { if (readyTime > 0) { $.drawResult('LOSE', 'WIN'); } } else if (isStart == 0) { if (readyTime > 2000) { $.drawAll_s('3'); } else if (readyTime > 1000) { $.drawAll_s('2'); } else if (readyTime > 0) { $.drawAll_s('1'); } else if (readyTime > -500) { $.drawAll_s('Start!'); } else { isStart = 1; $.drawAll(); } } else{ $.collision(); $.collision_wall(); // pixels / second circle_x[2] += puckSpeed_x * timeDiff / 1000; circle_y[2] += puckSpeed_y * timeDiff / 1000; $.drawAll(); } } // request new frame requestAnimFrame(function () { $.animate(time); }); } Fig.13 HTML5 Animation loop Fig.14 Game main F. End of Game After game is finished, score will be appeared upper the screen and finally, there is appeared in the game screen about win or lose the game. Fig.14 is shown how score is appeared after some player got the score. Fig.14 After right player get the score V. DISCUSSION The game has a lot of advantages because of using HTML5. That makes game's responsive speed fast and can use diverse platform. Especially, I was using the QR-code that any website have recently. However, the project that I made also has some possible problem. At first, the start resume is unstable and the function that zoom in/out sometimes can make errors. Mostly, the program is very light and simple. It means this program barely makes the soft error. I have planned to implement multi-play. But it took a lot of time building server and client. That's why I only implemented double play game. If I want to make the game more fun, then I can add users who can enjoy the game together. For example, the game view is divided by four section and each sections are used by each users. After then, some two players can be a one team and the rest can be one. In addition, it is possible to add other functions such as item, magic and character system. Each user can select their character or avatar and each has their own magic (skill) and item that can get from the field and use. Even they can use two balls and play with computer if I use artificial intelligence technique VI. CONCLUSION By implement the soccer game, I needed to optimize the channel between server and client. The most important thing that I learned is that if I want to change the interface of computer infrastructure, I need more complete research before I conduct to make the project. REFERENCES [1] Apple Watch release date, price and specs: Apple Watch Edition could cost more than £3,000, TechAdvisor, advice/apple/3443668/apple-iwatch-release-date-rumours-specs- uk-price-features-spring/ [2] [3] [4] [5]
  9. 9. [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] airhockey/tree/master [18] W3C: The WebSocket API, websockets-20091222/ [19] [20] [21]