Development Prototype Design of Virtual Assembly Application-Based Leap Motion

International Journal of Advanced Engineering, Management and Science (IJAEMS) [Vol-3, Issue-5, May- 2017]
https://dx.doi.org/10.24001/ijaems.3.5.14 ISSN: 2454-1311
www.ijaems.com Page | 497
Development Prototype Design of Virtual
Assembly Application-Based Leap Motion
Arif Krisbudiman1
, Gandjar Kiswanto2
1
Engineering Staff, Gedung Teknologi II No 250-251, Kawasan PUSPIPTEK, Tangerang Selatan, Banten – 15314, Indonesia
2
Lecturer, Department of Mechanical Engineering, Universitas Indonesia, Kampus Baru UI – Depok, Indonesia
Abstract— Innovation in design engineering practice is
very important in the world of manufacturing in the
increasingly competitive global market. Prototyping and
evaluation measures are inseparable from the design
process in the manufacture of a product. And made one of
many physical prototypes require very expensive and time
consuming, so the technology of Virtual Reality (VR) is
needed, so the industry can quickly and precisely in the
decision. VR technology combines a human being with a
computer environment visually, touch and hearing, so that
the user as if into the virtual world. The goal is that users
with hand movements can interact with what is displayed
on the computer screen or the user can interact with the
environment is unreal to be added into the real world. VR
is required for simulations that require a lot of interaction
such as prototype assembly methods, or better known as
the Virtual Assembly. Virtual Assembly concept which was
developed as the ability to assemble a real representation
of the physical model, the 3D models in CAD software by
simulating the natural movement of the human hand. Leap
Motion (accuracy of 0.01mm) was used to replace
Microsoft's Kinect (accuracy of 1.5cm) and Motion Glove
with flex sensors (accuracy of 1°) in several previous
research. Leap mot ion controller is a device that captures
every movement of the hand to then be processed and
integrated with 3D models in CAD software. And
simulation of assembly process virtually in CA D software
with hand gestures detected by the leap mot ion, assembly
parts can be driven either in translation or rotation,
zooming and adding the assembly constraint. It also can
perform mouse functions (such as left-click, middle-click,
right-click and move the mouse cursor position) to a
virtual assembly process simulation on CAD software.
Keywords— 3D CAD Model, API, leap motion controller,
virtual assembly, virtual reality
I. INTRODUCTION
Innovation is very important for companies to be
successful in today's global marketplace. Competitive
advantage can be achieved either by doing a revolution in
design engineering practice. It must cover all aspects of the
design of a product (such as ergonomics, manufacturing,
maintenance, product lifespan, etc.) during the early stages
of product manufacture. Prototyping and evaluation
measures are inseparable from the process of design or
manufacture of the product. Although the computer
simulation modelling practices such as CAD is currently
widely used in the different stages (eg FEA simulation
software), makes one of many physical prototypes needed
a very expensive and long time. The new
technology is needed so that the industry can quickly and
precisely in the decision. The concept of virtual reality
(VR) has grown over (State the objectives of the work and
provide an adequate background, avoiding a detailed
literature survey or a summary of the results) the past two
decades. VR has changed the way engineers and scientists
to the computer to perform mathematical simulations, data
visualization, and decision making (Bryson, S., 1996;
Eddy, J. and Kemper E., Lewis, 2002; Xianglong, Y.,
Yuncheng, F., Tao, L. and Fei, W., 2001; Zorriassatine, F.,
Wykes, C., Parkin, R. and Gindy, N., 2003). VR
technology combines multiple human environments with a
computer to provide a variety of sensations (visual, haptic,
auditory, etc.) that makes the user as if into the virtual
world. It allows the user to natural human movement can
interact with what is displayed on the computer screen.
The aim is that the user can interact with the virtual
environment you want to create or add in the real world.
This makes it the perfect tool for the VR simulation
activities that require a lot of user interaction such as
prototype assembly methods, or better known as the virtual
assembly. Jayaram et al., in 1997, a virtual assembly
defines as "the use of computer tools to make or assist the
engineers to take decisions in the assembly through
analysis, predictive modelling, visualization, and
presentation of data without any physical product or
process support tool significantly" (Jayaram, S.,
Connacher, Hugh I. and Lyons Kevin W., 1997). This
definition emphasizes the use of computer equipment and
visualization techniques that facilitate decision-making in
the assembly process with the addition of the virtual
environment to the real interaction as an integrated part of
the virtual assembly simulation.

Then the definition by Kim and Vance in 2003, described
the virtual assembly is "the ability to assemble CAD
models of components in 3D, user environment and
natural human motion" (Kim Chang E. and Vance, Judy
M., 2003). This illustrates that the incorporation of 3D
environments, VR and natural human interaction are an
important part of the virtual assembly. Virtual assembly
definition has evolved with VR technology, after the
visualization problems can be solved, a new definition is
added natural human interaction as a challenge to the
virtual assembly simulation. And the development of VR
technology is more advanced then expands previous
definitions to provide a more comprehensive picture of the
virtual assembly.
Virtual assembly in th is paper is defined as the ability to
assemble a real representation of the physical model, the
3D models in CAD through a realistic simulation of the
environment as the natural movement of the human hand
in the assembly process. Planning the assembly process is
an important step in the development of products which
includes details of assembly operations that describes the
steps the most quickly, accurately and efficiently. And the
cost of the assembly process is often a large part of the
total cost of the product (Boothroyd, G. and Dewhurst, P.,
1989). Thus, it is very important to develop proper
assembly in the initial design planning stages. A good
assembly plan by combining consideration for minimum
assembly time, low cost, ergonomics and operator safety.
And the design of the assembly process that will either be
able to imp rove the efficiency, quality, reduce costs and
shorten product time to market.
Expert assembly planners is still using the traditional
approach in which they have to see the 3D model in a
CAD of parts to be assembled on a 2D screen of a
computer to check the part geometry and determine the
assembly sequence for the new product. Another method
for planning the assembly, including doing some
experiments with a physical prototype assembling and find
the best sequence of assembly. This causes the assembly
becomes more complicated tasks, such methods tend to be
more time-consuming, expensive and prone to errors.
Planning assembly with the aid of computer simulations
became popular in solving existing problems. Much
research has been done to develop an algorithm to generate
a suitable sequence assembly process or the most optimal.
But there are some things that still need to be addressed,
because of the increased number of components, the
assembly sequence increases exponentially making it
difficult to characterize the criteria in selecting the most
appropriate assembly sequence for specific products
(Dewar, R. G., Carpenter, I. D., Ritchie, J. M. and
Simmons, J. E. L., 1997).
Modern CAD systems also provide the ability to build a
CAD assembly model. However, CA D systems are still
using 2D environments such as a keyboard and mouse as
well as the user must manually choose the surface, axis
lines or the edges of the components to be assembled.
Thus it does not reflect the human interaction with the
component parts of complex shape. And it becomes
difficult to predict what will appear either in the process of
assembly, maintenance or during operation, for example,
ensuring accessibility to replace components during
maintenance and the effect of changing the assembly
sequence. It is also in addressing issues related to
ergonomics as the ease in the assembly process.
VR technology plays an important role in the simulation of
the interaction between humans and computers in 3D by
making as if the user is logged into the virtual world
displayed on the computer screen. Simulating a virtual
assembly process is to produce the most suitable process in
assembling a product so as to reduce the high cost and
time required for the process of making physical
prototypes.
Fig.1: Application of virtual assembly simulation
Using VR technology, the simulation can be done and
different assembly sequences can be analyzed by utilizing
existing CAD data. The ability of the virtual assembly can
be applied to produce a competitive product designs in
various aspects, such as process design, ergonomics,
workstation layout, assembly design, fixtures and
equipment, training, and maintenance (Figure 1).
This study combines motion detection technique hand to
interact with 3D models in CAD software in its application
to the virtual assembly. And approaches for detecting the
movement is with method System Machine Vision (SMV)
and image processing techniques. In recent years there are
some inexpensive sensor devices such as Microsoft's
Kinect (1.5cm accuracy) and Leap Motion (accuracy
0.01mm), resulting in a virtual assembly has grown not
only limited to using a mouse, keyboard and touchscreen.
There was also a motion that uses the virtual assembly g

love with flex sensors (accuracy of 1°) in interaction with
3D models in CAD software. And in this study used a leap
motion which has a higher degree of accuracy and include
Software Development Kit (SDK) and Application
Programming Interface (API) for interacting with other
applications, making it possible to simulate a virtual
assembly process.
II. METHODOLOGY/ EXPERIMENTAL
The method used in the design of a prototype application is
to use a Virtual Assembly leap motion controller as a
detector of hand movements. And by utilizing the SDK of
leap motion and API Object Model from Autodesk
Inventor, created and developed a prototype design of the
application of Virtual Assembly by using software, namely
Microsoft Visual Studio Professional 2013, to simulate
assembly in CAD software, the Autodesk Inventor
Professional 2014 - Student Version (Figure 2).
Fig.2: Conceptual design of the virtual assembly system
Fig.3: Virtual Assembly System Design
Leap hand gestures detected by the motion controller and
the main data used are the position and movement of the
point of the finger (finger positioning). The data will be
integrated with assembly parts in the application of
Autodesk Inventor, use other applications built with C #
programming language to software Visual Studio 2013.
The application was made by utilizing Leap Motion SDK
and Inventor's API Object Model so as to relate the data
from leap mot ion with Data 3D CAD models from
Autodesk Inventor application. The technical
specifications of the computer used is as follows:
1. Processor: Intel (R) Core (TM ) i5-3210M CPU
@ 2.50GHz
2. RAM: 8.00 GB
3. System type: 64-bit Operating System (Windows
7 Home Basic).
In the simulation of assembly process virtually in
Autodesk Inventor application is divided into four major
parts, namely:
1. Moving parts assembly both translational
2. Move the rotation assembly, both parts
3. Zoom in or out assembly part (zooming process)
4. Adding assembly constraints.
And the process can be run with the command through
gestures or called as a command gesture. Command hand
movements may also perform mouse functions, such as:
left-click, middle-click and right-click and move the
mouse cursor position.
Command hand movements are made, among others:
1. Left-Click: use the right hand with the middle finger,
forefinger and thumb open and left-click to do is to
close and open the thumb (the tip of the index finger as
a mouse cursor position).
2. Middle-Click: use the right hand with the middle
finger, forefinger and thumb open and to middle click
is to open and close the ring finger (index finger as a
mouse cursor position).
3. Right-Click: use the right hand with the middle finger,
forefinger and thumb open and to do the right-click
with the closing and opening of the middle finger
(index finger as a mouse cursor position).
4. Moving in translational assembly parts are selected
(Free Move): use the right hand with the middle finger,
forefinger and thumb open, then point the mouse cursor
to the parts that will be selected to be moved. Then
close the thumb to select (left-click) and the selected
part can be moved in accordance with movement of the
hand movement.
5. Moving in translational all assembly parts
simultaneously (Pan): use the right hand with the
middle finger, forefinger and thumb open, then middle-
click to open the command ring finger. After that all
assembly parts can be driven simultaneously in
accordance with the displacement movement of the
hand.
6. Moving in rotation (Orbit View): use the right hand
with the index finger, middle finger, ring finger and

little finger open. After that all assembly parts can be
driven at the same rotation in accordance with the
movement of the hand.
Zoom in or out (zooming process): use the right hand with
the thumb open. After that hand movement away from the
user to zoom out and approaching from the user to zoom
all the assembly parts.
The steps undertaken in the development of prototype
design application-based Virtual Assembly leap motion, as
follows:
1. Making an application with Visual Studio using C #
programming language to connect to leap motion
controller.
2. The application development with Visual Studio using
C # programming language to read data from the leap
mot ion movement of the hand.
3. Replace the mouse cursor position with the position of
the user's finger is detected by the motion leap.
4. Create a movement of the hand which can perform
certain commands, and replace the function of the
mouse (left-click, middle-click and right-click).
5. Integrate data from leap motion with the 3D model data
in CAD software, the Autodesk Inventor.
6. Creating a movement that can perform certain
commands on CAD software (command gesture).
7. Creating a Graphic User Interface (GUI) to display
related information in performing a virtual assembly
process simulation using motion leap in CAD software.
In Figure 4 above can be seen information obtained from
the leap motion, such as: the position of the fingers, hand
movements (pitch, roll and yaw), number of hands (right
or left), the number of tools and the type of gesture. And
other information that informs the assembly document on
the application of Autodesk Inventor have been opened
beforehand, while in the LOG contains real time
information to the commandments of what has been done.
Fig.4: Graphic User Interface
Commands include: Left-Click, Middle-Click, Right-
Click, Free Move, Pan, Orbit View, Zoom In, Zoom Out
and Assembly Constraint.
Development of Virtual Assembly prototype design
applications using procedures and algorithms are
programmed using the programming language C #.
Algorithms and procedures are as follows:
1. Connection to leap motion controller
algorithm:
using Leap.dll
New controller
Initialization controller
Connection controller
Turn all gestures (circle, key tap, tap and swipe the
screen)
The reading of the data of each frame of a leap
motion
procedure:
a. Make sure the leap motion controller is
connected to a computer through a USB port
b. Make sure the indicator light from the flame
leap motion has been colored green, orange if
service mark of leap motion and the road yet
to run: on the list view select Leap Services 
Service  click Start.
c. Three files, namely: Leap.dll, Leap CSharp.dll
and Leap CSharp.NET4.0.dll must copy first
into the project folder (PROJECT  Add
Existing Item ... or Shift + Alt + A).
d. Leap CSharp.NET4.0.dll should be inserted
first into the reference project (PROJECT 
Add Reference ...).
2. Reading the data of hand gestures
Algorithm:
Reading data from the previous frame of leap motion
The reading of the current data frame of leap motion
Reading the data of hand from leap motion
Reading the data of gesture from leap motion
Procedure:
a. Reading data for each frame hand is detected
by motion leap.
b. Data reading gesture was to some 10,000
frames per second which is detected by the
motion leap.
3. Data reading position of the finger
Algorithm:
The reading of the data of each frame from leap
motion
Reading the data of hand-motion from leap motion
Reading the data of finger-motion from leap motion
Procedure:
a. Reading the data of right hand fingers, if
only detected a right hand
b. Reading the data of left hand fingers, if only
detected a left hand alone.

c. Reading the data of right hand fingers, if it is
detected more than one hand.
4. The position of the fingers used as the position of the
mouse cursor
Algorithm:
Reading the data of each frame from leap motion
Using user32.dll
Setting the mouse cursor position (X, Y) with the
position of one finger is detected by the motion leap
Procedure:
a. Setting the mouse cursor position changes if the
speed of movement of the fingers (Tip
Velocity>25), to eliminate vibration fingers.
b. Conversion unit of the position of the fingers
(mm) into the mouse cursor units (pixels).
5. The command hand movements replace the functions
of the mouse (left-click, middle-click and right-click).
Algorithm:
Reading the data of each frame from a leap motion
Using user32.dll
Using mouse_event (Move, LeftDown, LeftUp,
RightDown, RightUp, MiddleDown, MiddleUp,
Xdown, XUP, Wheel, VirtualDesk, Absolute)
Procedure: Changes in the finger motion is read by a
determined leap into commands to perform the
functions of the mouse (left-click, middle-click and
right-click).
6. The integration of data from leap motion with the 3D
model data in CAD software.
Algorithm:
Using user32.dll
Using reference Autodesk.Inventor.interop
Reading data from Autodesk Inventor 3D models
Procedure: Autodesk.Inventor.interop.dll should be
inserted first into the reference project (PROJECT 
Add Reference ...).
7. Creating a movement that can perform certain
commands on CAD software (command gesture).
Algorithm:
Using user32.dll
Using mouse_event (Move, LeftDown, LeftUp,
RightDown, RightUp, MiddleDown, M iddleUp,
Xdown, XUP, Wheel, VirtualDesk, Absolute)
Using reference Autodesk.Inventor.interop
Using class Camera.cs (TranslateView function,
ChangeView and Zoom)
Procedure: Hand movements are read by leap
motion was determined to be a command to perform
certain commands on the application of Autodesk
Inventor (Free Move, Orbit View, Pan, Zoom and
insert assembly constraints).
8. Creating a Graphic User Interface (GUI).
Algorithm:
Display design (layout) GUI desired
Reading data from Autodesk Inventor
Displaying all desired data as output information
Procedure: Data from the leap motion with regard to
the position of the fingers, hand movements (pitch,
roll and yaw), number of hands (right or left), the
number of tools and the type of gesture and
information whether the leap motion controller has
been connected properly to the computer before the
application of Virtual Assembly run.
Data from Autodesk Inventor related to the commands that
are being done on CAD software, such as: Left Click,
Middle Click, Right Click, Free Move, Pan, Orbit View,
Zoom In, Zoom Out and Assembly Constraint and
information whether the document assembly has been
opened before the application is run Virtual Assembly.
III. RESULTS
Virtual simulation experiment Assembly is done by using
multiple assembly files (*.Iam), which can be divided into:
1. Assembly file 1, consists of 5 parts
2. Assembly file 2, consists of seven parts
3. Assembly file 3, consists of nine parts
Virtual assembly process simulation in Autodesk Inventor
using motion leap and applications made, among others:
1. Replacing a mouse function, namely move the
mouse cursor position with hand gestures detected
by the motion leap. The movement is useful for
directing, selecting and moving parts in translation
in a virtual simulation of assembly process.

Fig.5: Command left click mouse
2. Replacing a mouse function, that is to left-click,
middle-click and right-click. The hand gesture
commands useful for selecting and moving parts in
the translation.
Fig.6: Command Pan (Translation)
3. Mobilize assembly part in translation and rotation
with command hand movements in a virtual
simulation of assembly process.
Fig.7: Command Orbit View (Rotation)
4. Zoom in or out (zooming process) of the
assembly part
Fig.8: Command Zoom In
Fig.9: Command Zoom Out
5. Adding assembly constraints between two parts
for virtual assembly process.

Fig.10: Command Mate Constraint
IV. DISCUSSION
Based on the simulation results that the assembly part in
Autodesk Inventor can move, enlarged and reduced by
using a hand gesture commands are detected by a motion
leap. As for adding assembly constraints still use the
"Place Constraint" in Autodesk Inventor, but still uses the
motion leap-click menu and selection of spare parts
assembly (surface, edge or axis) which will be used as
constraints. This was due not yet succeeded in choosing
the assembly spare parts directly using motion leap
through hand motions, because the function of the
selection of spare parts of the API can only be used with a
mouse (as the standard default). And simulations were
done using different assembly file, shows that with more
and more part in the assembly file the movement more
visible dashed.
V. CONCLUSION
Based on the research results can be concluded as follows:
1. Data reading leap motion (movement of the hand, the
finger position and gesture) through other
applications that are programmed in C # has been
successfully carried out.
2. Through the development of a prototype application
that created virtual assembly has managed to perform
the command function mouse (left-click, middle-
click, right-click and move the mouse cursor
position).
3. Based on the virtual assembly simulation done by
hand movement has succeeded in moving the
assembly part by translation, rotation, zoom and
adding assembly constraint.
4. In adding assembly constraints still use the menu on
the Autodesk Inventor leap motion due to limitations
in choosing the parts section directly by using the
command hand movements.
ACKNOWLEDGEMENTS
For research on Virtual Assembly theme in the world of
manufacturing, the authors suggest the following research:
1. Virtual Assembly technology combined with
Augmented Reality (AR).
2. The use of more than one leap motion with the
program algorithms to simulate a Virtual Assembly.
3. Selection of spare parts directly using hand gestures
via motion leap can be made in virtually any
assembly process simulation on CAD software.
REFERENCES
[1] Boothroyd, G. and Dewhurst, P., (1989), Product
Design for Assembly. New York: McGraw-Hill, Inc.
[2] Bryson, S., (1996), Virtual Reality in Scientific
Visualization, vol. 39, pp. 62-71, Communications of
the ACM.
[3] Dewar, R. G., Carpenter, I. D., Ritchie, J. M. and
Simmons, J. E. L., (1997), Assembly Planning in a
Virtual Environment, presented at Proceedings of
Portland International Conference on management of
Engineering and Technology (PICMET 97), Portland,
OR.
[4] Eddy, J. and Kemper E., Lewis, (2002), Visualization
of Multidimensional Design and Optimization Data
using Cloud Visualization, ASME Design
Engineering Technical Conferences and Computers
and Information in Engineering Conference
(DETC2002/DA C-34130), Montreal, Canada.
[5] Jayaram, S., Connacher, Hugh I. and Lyons Kevin W.,
(1997), Virtual Assembly using Virtual Reality
Techniques, Computer Aided Design, vol. 29, pp.
575-584.
[6] Kim Chang E. and Vance, Judy M., (2003), Using
Vps (Voxmap Pointshell) As The Basis For
Interaction in a Virtual Assembly Environment,
ASME Design Engineering Technical Conferences
and Computers and Information in Engineering
Conference (DETC2003/ CIE-48297), Chicago, IL.
[7] Xianglong, Y., Yuncheng, F., Tao, L. and Fei, W.,
(2001), Solving Sequential Decision-making Problems
Under Virtual Reality Simulation System, Winter
Simulation Conference Proceedings, Arlington,
Virginia.
[8] Zorriassatine, F., Wykes, C., Parkin, R. and Gindy, N.,
(2003), A Survey of Virtual Prototyping Techniques
for Mechanical Product Development, Institution of
Mechanical Engineers, Part B: Journal of Engineering
Manufacture, vol. 217, pp. 513-530.

Development Prototype Design of Virtual Assembly Application-Based Leap Motion

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Development Prototype Design of Virtual Assembly Application-Based Leap Motion

Similar to Development Prototype Design of Virtual Assembly Application-Based Leap Motion (20)

Recently uploaded

Recently uploaded (20)

Development Prototype Design of Virtual Assembly Application-Based Leap Motion