Unraveling Multimodality with Large Language Models.pdf
Improved learning through remote desktop mirroring control
1. Improved Learning through Remote Desktop
Mirroring Control
Khong-Neng Choong, Chrishanton Venthanayagam and Putri Shahnim Khalid
MIMOS, Kuala Lumpur, Malaysia
Email: {kn.choong, chrishanton.v, shahnim.khalid}@mimos.my
Abstract—The potential of wireless technology to improve
teaching and learning is so effective that it is currently at the
forefront of technological advancements in education.
Besides utilizing laptops and mobile devices, content
mirroring (e.g. Apple Airplay and Miracast) has also been
an important application technology in supporting better
learning. This paper describes a wireless presentation
system which directs and controls wireless presentation
stream called the Wireless Stream Management System
(WSMS). Unlike Airplay and Miracast, WSMS is designed
to allow the moderator (teacher) to remotely manage and
control selected real-time screen mirroring source for
instant content amendment/correction in the effort of
supporting collaborative learning.
Index Terms—wireless presentation, seamless presentation,
presentation management, collaborative learning, screen
capturing, display sharing.
I. INTRODUCTION
The potential of wireless technology to improve
teaching and learning is so effective that it is currently at
the forefront of technological advancements in education.
These technologies include the continuous enhancement
of mobile device capabilities, omnipresence of wireless
connectivity, use of multimedia content and collaborative
application tools to improve teaching/learning interaction.
The key objective here is to utilize these technologies to
aid and enhance the delivery of course content for better
understanding and experience of the students.
Presentation is a common routine being in classroom,
seminars, conference sessions, meetings or other
interactive forums using shared displays and mobile
devices for collaboration purposes [1, 2]. Presentation
generally involves presenters connecting their laptops to
projectors, which could be done wirelessly using wireless
projector, or with wire in the traditional manner.
Connecting individual laptop one at a time to the
projector is manageable if the presentation session
involves single laptop and single screen output per
presentation, such as in typical meeting and lectures. But
it would be a challenge in a smart classroom environment
given both the teacher (moderator) and students have to
constantly interact by switching their laptop content onto
the common projector screen for presentation and
discussion.
As of today, there are many electronic communication
tools that help users to conduct meeting and presentation.
This includes online chat, video conferencing, remote
desktop sharing, application sharing etc. Despite these
long list of tools, the fundamental problems in handling
meetings and presentations have not been addressed. In
general, these problems can be categorized as follows:
A. Limitation of physical connection
The need to plug and unplug VGA cable across
different laptops for various individual presentations. The
length of VGA cable further determines the location and
hence limit the mobility of the presenter. The unhealthy
practice of attaching and detaching USB drives
(potentially from various unsecure sources) to/from the
PC connected to the LCD/projector, causing unnecessary
virus infections to the PC, and also causing delay to the
presentation.
B. Inflexible meeting control
Presenters must listen to the session chair (teacher) on
when to start connecting to the projector in order to show
their laptop content. Absence of easy-to-use meeting flow
control capability for session chair to start/stop, direct and
queue-up presentation from one central location.
C. Limited interaction with participants
Some meeting participants are not able to see the
projected contents on the screen due to their seating
location, viewing distance and blocking by front
participants. Participants who are late are not able to
access the materials they want (e.g. the first few slides
that has been presented). Less vocal participants are
unable to interact with the presenter or session moderator
throughout the presentation session.
The above top 2 issues can cause problems and often
delay the setting up of presentation. Worst, it could even
introduce technical issues where no output is shown on
the LCD/projector due to hardware incompatibility or
different screen resolutions. In this paper, we introduce a
wireless mirroring system which tackles the above issues.
This paper focuses on enabling a remote management
feature on top of the wireless mirroring system, which
allows the moderator (teacher) to remotely manage and
take over selected real-time screen mirroring source for
on-the-spot guidance as an effort to support collaborative
learning.
This paper is organized as follows. Section 2 provides
a detailed description of the proposed system. Section 3
describes the system architecture and components.
Section 4 describes the testbed setup and performance
results, while Section 5 highlighted some related works.
Section 6 concludes the paper.
2. II. WIRELESS STREAM MANAGEMENT SYSTEM
Figure 1 shows the high-level functional overview of
our proposed system named the Wireless Stream
Management System (WSMS). WSMS uses wireless
connectivity (Wi-Fi) to address the physical limitation of
conventional presentation setup. It supports two modes of
presentation, namely (1) the democratic mode where any
presenter can take over the central screen whenever they
want, and (2) the moderated mode where a moderator is
allowed to manage and select which presenter can stream
his/her device screen to the central screen. WSMS is also
designed to distribute the current presentation content as
lightweight images (less than 1 MB in size) to
participants. This allows participants to zoom-in on the
image for clearer visual clarification, and save to their
local device storage for offline reading.
Figure 1. Wireless Streaming Management System
Besides the above features, WSMS is also supporting
remote device control over the network. This capability is
especially useful for teaching session which involves
practical hands-on such as lab assignment or exercise
where students are expected to repeat or utilize what was
taught to complete an assignment in the class using their
individual PC/laptops. Very often, students will
encounter problems, and lab assistants are usually the
helpers to the teacher/lecture who move around the class
to resolve issues for the students. This process takes time
and effort if there are lots of students. Students may be
experiencing the same or similar issues, but unfortunately
due to lack of technology, lab assistants are required to
repeatedly resolve these similar issues for multiple
students.
A proper technological approach would be to get all
students’ laptops connected to the WSMS so that the
teacher could log-in as the moderator to select any
student’s laptop to be projected on the main
screen/projector as shown in Step 1 at Figure 2. The
tablet at the lower left of Figure 2 shows a list of 3
student laptop icons where the first laptop is currently
mirroring/projecting its laptop’s content.
The teacher could then remotely control and manage
the student’s laptop to demonstrate how to resolve their
problems for the benefits of the entire class as shown in
Step 2 at Figure 2. With this approach, the teacher alone
could manage the entire practical session without the
need of any lab assistant. Moreover, the demonstration
showed on the projector would be seen by other students
who were experiencing the same or similar problems.
With WSMS, any student could have their laptop’s screen
projected easily onto the main screen/projector for the
teacher to comment or for other students to ask questions.
Hence, it would lead to an effective collaborative learning
experience.
Figure 2. Sequence of Moderator Flow Control
III. SYSTEM DESCRIPTION
WSMS is an application that was developed based on
the Seamless Presentation (SP) core modules as reported
in [3], which supports various screen mirroring delivery
models such as 1-to-1, 1-to-Many and 1-to-1-to-Many.
The latest version has incorporated the capability of
distributing presented content as screenshot images to
multiple receiving devices which run web browsers.
These combinations of content mirroring, delivery and
distribution are capable of supporting multiple usage
scenarios.
A. System Architecture
Figure 3 shows the system architecture of the entire
system. The system comprises of four key components
namely: Wireless Screen Sender (WSS) which is the
content origin, the Wireless Screen Receiver (WSR)
which is the corresponding receiver of content delivered
by WSS, the Wireless Screen Administrator (WSA) that
is responsible for assigning and managing the
presentation session and the Wireless Screen Controller
(WSC) as the central point of connection for WSS, WSR
and WSA besides coordinating the presentation session.
WSMS is built on top of the existing system as described
in [3] with additional VNC-based components, as
denoted by the shaded boxes.
B. System Components
There are several modules within each component. This
section discusses these components together with their
respective modules in detail:
WSS is consists of:
a) Application GUI as the interface for the initial
configuration input such as the presentation
content type and the group name
3. b) Core, which provides the main functionalities
such as performing screen capturing, encoding
and transmission to the WSC and also for
providing connection security. Detailed
description of the Core can be found in [3]
c) Screen Manager that comprises of the Session
Manager for coordinating the screen capturing,
encoding and transmission process, and the
Registry Client as the initial registration to the
WSC
d) VNC Server is a 3rd
party component which
shares screen and control of its local device, to
the VNC client over the network
WSR has a set of corresponding modules to the WSS:
a) Application GUI as the interface for setting up
the configuration such as the custom name.
b) Core, which provides the main functionalities
such as receiving, decoding and display contents
received from WSS via WSC
c) Screen Player, which consists of the Session
Manager for coordinating the receiving,
decoding and displaying of screen content from
the Screen Manager of the WSS, and the
Registry Client for registration to the WSC
WSA which works within the web browser context runs
the Session Management Module which consists of three
sub-modules:
a) Web UI as the user interface for the WSA to set
up, assign and manage the presentation sessions
b) UI Controller as the UI event handling. It runs
scripts to support VNC viewer web application
c) Session Agent which is used by the UI
Controller to forward WSA’s command to the
WSC as well as receiving status information and
updates from WSC
d) VNC Viewer Web App is a 3rd
party component
which is loaded from the VNC Proxy Server to
interact with the VNC Server
WSC as the session coordinator requires a Web Server to
deliver the Session Management Module to the WSA.
Besides that, it requires the Registry Module to enable the
coordination of the presentation session. In details, the
Registry Module comprises of:
a) Session Proxy as the gateway of communication
between the Session Agent and the Registry
Manager. The commands received from the
WSA is parsed and translated into a form of
message that is recognized by the Registry
Manager and vice versa.
b) Registry Storage for storing the information of
the WSS and the WSR.
c) Registry Manager for handling WSA’s
command as well as the registration process for
the WSS and the WSR
VNC Proxy Server, which interfaces websocket
communication from the VNC viewer web application in
WSA, to the RFB (Remote Frame Buffer) protocol with
the VNC server residing in the WSS.
Figure 3. System Architecture
4. IV. TESTBED SETUP
A. Network Architechture
A testbed as shown in Figure 4 has been set up to verify
the functionality and performance of WSMS. This testbed
is made up of one WSR and three WSSs, connected
wirelessly to one WSC which serves Wi-Fi hotspot. WSC
also serves the administrative webpage WSA to run on
browser device. A network monitoring tool named PRTG
[4] is used to collect statistics from devices running in the
network.
In this experiment a 3D3 Alix board [5] is used to host
the WSC component. This system board is equipped with
Ubuntu 12.04 OS and running in AP mode. All nodes
should connect to this board and ultimately get registered
with the Registry Module.
Figure 4. Testbed Setup
WSA on the other hand, is a web-based component
which retrieves the Session Management Module from
WSC. Therefore, it can be running on any machine or
device equipped with a web Brower. In our experiment a
Windows-based laptop runs as the WSA. Once any of
WSS node connects to WSC, WSA will be notified and
the web UI will be updated with the newly joined node
icon and information.
Figure 5 shows the user interface of the WSA. The
interface is very simple and direct. All connected and
registered WSS shall be represented with individual
square boxes with their names within. Whenever the
moderator selects a WSS box, an animation is played
within the box indicating an active session. Selecting the
active WSS will terminate the ongoing session. This is a
typical control for moderator to start and stop any WSS
via the moderator UI. Above each square box, there is an
extra button named “control”. Pressing this button allows
the moderator to remotely control the respective WSS.
Figure 5. User Interface of WSA
B. System Performance
Performance measurement for such a wireless-based
system is generally broken into 3 parts as follows:
• Network bandwidth utilization,
• Network latency and
• CPU utilization
This combination of tests should serve as a good
performance indicator on the overall user experience for
conducting remote desktop control.
To conduct the network bandwidth utilization, we first
determine the bandwidth available in the WiFi network
using iPerf [6]. This is followed by running PRTG on the
moderator laptop to measure the bandwidth used on
establishing the remote control VNC session for 3
different scenarios for a fixed period of 15 min as in
Table 1.
Table 1. Test Scenarios
Scenario Description
A Presenter streams PowerPoint slides
B Presenter streams low motion video
C Presenter streams high motion video
The total bandwidth available and WiFi link speed
measured during the experiment are as shown in Table 2.
Table 2. Network Measurement
Bandwidth
Available
(Mbps)
WiFi Link
Speed
(Mbps)
19.5 54
Figure 6 shows the maximum bandwidth utilization for
all 3 scenarios, from 0.47, 1.082 to 1.835 mbps
respectively. This matches logically to the nature of the
content from minimum screen changes in presentation
slides to drastic screen change as in videos.
5. Figure 6. Bandwidth Utilization
Measuring packet round trip time is essential to
understand the latency in the network. In general, ping
result of less than 50 ms means the network is really
good, less than 100 ms indicates good to average
performance, 150 ms means it is starting to have
problems with interactive online games, and above 150
ms means users may experience lagging.
Figure 7 shows the network latency collected for all 3
scenarios. Scenario C showed the highest latency at a
maximum of 173 ms which may due to certain high
movement scenes therefore utilizing higher bandwidth.
This has caused some performance degradation to
scenario C.
Figure 7. Network Latency
Table 3 tabulates the actual recorded values of network
latency for all 3 scenarios. There were however no issues
with CPU having enormous workload for all 3 scenarios.
Table 3. Network Latency and CPU workload
Scenario Network Latency (ms) CPU workload
Maximum Average Maximum
A 4.6 3 18%
B 16.1 6 19%
C 173 23 23%
V. RELATED WORKS
Various works and products can be found offering
similar basic features as our WSMS. However, each of
these works is limited to basic screen mirroring, there is
still an absence of session management and remote
control capability.
Wireless projector [7, 8] is the most common and
widely used product in the market. Even though it is a
self-contained device, it has many shortcomings. It is not
a straight-forward process for connecting laptops to the
wireless projectors as each brand comes with its own
configuration and execution steps. As mentioned, there is
no session management as described in WSMS.
A closer product in terms of basic functionality is
Google Chromecast [9] which supports browser-content
mirroring. It provides direct 1-to-1 streaming model and
is capable of running across any OS platforms. The main
limitation is that only contents within the browser could
be mirrored across. Again, it has no session management
as proposed in WSMS.
A screen sharing system in [10] has studied screen
contents for variety of usage scenarios and developed a
practical transformation that improved compression rates
which adapts to different screen content. This system runs
natively on Mac OS X, iOS and Windows platforms.
Unfortunately, it stops as a research project without
further development into product or service that is made
available in the market. Nevertheless, it is still lacking
session management as discussed in this paper.
VI. SUMMARY AND FUTURE WORKS
This paper proposes and describes a wireless
collaborative system for interactive learning called
WSMS, which allows the moderator (teacher) to remotely
manage and take over selected real-time screen mirroring
source (student’s laptop/device) for instant content
amendment/correction in the effort of supporting
collaborative learning. WSMS is developed on top of a
streaming platform called Seamless Presentation (SP) as
reported in [3]. Through SP platform, various mirroring
delivery models could be supported. Explained in this
paper is one of the recent models.
Future works include enabling more functionality. First
is to show desktop activities of all participants’
laptops/devices presented as image thumbnails which are
updated at fixed intervals. This allows moderator to easily
monitor each and every participant from a central point of
control. Second is to develop some image processing
algorithm to determine whether there are any
participants’ laptops/devices screens which are different
than the majority participants’ screens. This is to help the
moderator to ensure all participants are performing the
same tasks or focusing on the same piece of content as
instructed by the moderator.
REFERENCES
[1] Christian Berkhoff, Sergio F. Ochoa, José A. Pino, Jesus Favela,
Jonice Oliveira, Luis A. Guerrero, Clairvoyance: “A framework to
integrate shared displays and mobile computing devices”, Future
Generation Computer Systems, Volume 34, May 2014, Pages 190-
200, ISSN 0167-739X.
[2] Sungwon Nam, Khairi Reda, Luc Renambot, Andrew Johnson,
Jason Leigh: “Multiuser-centered resource scheduling for
collaborative display wall environments”, Future Generation
Computer Systems, 30 August 2014, ISSN 0167-739X.
[3] K.N. Choong, D. Naghshbandi and M.S Hisham, ‘Seamless
Presentation: A screen sharing system for WiFi network,” in Proc.
Int’l Conference on Communication, Networks and Satellite,
Jakarta, Indonesia, 2014.
[4] PRTG. [Online]. Available: https://www.paessler.com/prtg
6. [5] PC Engines. [Online]. Available: http://www.pcengines.ch/
alix3d3.htm.
[6] iPerf. [Online]. Available: https://iperf.fr/
[7] X. Li, S. Wang, J. Tan, J. Ye, X. Si and M. Li, “Automatic
wireless embedded projector player system,” in Proc. Int’l
Conference on Multimedia Technology, Ningbo, China, 2010.
[8] I. Dvir and Y. Medan, “Method and system for remote video
display through a wireless projector,” WO 2002033687 A2, April
25, 2002.
[9] Google Chromecast. [Online]. Available: http://www.google.com/
intl/en/chrome/devices/chromecast/
[10] S. Chandra, J. Boreczky and L.A. Rowe, “High performance
many-to-many intranet screen sharing with DisplayCast”, ACM
Transactions on Multimedia Computing, Communications, and
Applications, Volume 10, Issue 2, 2014.