1) The document discusses advances and future scenarios in agricultural robotics, including the use of virtual reality, haptics, and hand gestures. 2) It describes current agricultural robots that use sensors and automation to perform tasks like field scouting, weeding, spraying and harvesting. These robots collect large amounts of farm data. 3) The document predicts that in a few decades, farmers may no longer need to enter fields, as intelligent, interconnected robots will perform all agricultural labor with the help of technologies like computer vision, artificial intelligence and machine learning. Robots will use collected data to make precise decisions that maximize farm outputs and profits.

1. Virtual Reality, Haptics, and Hand Gestures for Agricultural
Robotics and Digital Farming
“The Future Scenario”
Redmond R. Shamshiri, Ph.D.
Prof. Dr.-Ing. Cornelia Weltzien
Berlin, Germany
November 29th, 2018
Arbeitskreis Landwirtschaft, Autonomie und Robotik in der
Landwirtschaft: Wie weit ist die deutsche Stall- und Feld-robotik?

2. 03.12.2018 2
Some of the possible solutions to gain higher profit in agriculture
What future scenarios are conceivable?
Will the farmer refrain from entering the field in a few decades?
Research and development
Where and how successfully Ag-Robotics have been used?
How intelligent and interconnected they are?
Mobile (Field robots) Non-mobile Unmanned Aerial Systems Multi-robot, Hybrid
Industrial-age Agriculture Information-age agriculture
Robot-Human Interaction for Digital Farming
PART 1: Introduction and Background
PART 2: Current Status and advances in Ag-Robotics
PART 3: Future Scenario

3. 03.12.2018 3
Inputs
Profit
To Maximize the profit:
1- Same output with lower input
2- Higher output with the same input
3- Higher output with lower input 
 Soil and water
 Fertilizer
 Pesticide / herbicide
 Machinery/ha
 Manpower/ha
 Manual operation

4. 03.12.2018 4
Classic response to
higher yield
Industrial-age agriculture
Modern Response to
higher yield
Bigger Machines
Information-age agricultureSmarter Machines
Digital Agriculture
Data Analysis
Artificial Intelligence
Machine Learning
Autonomous Systems
Hot Topics

5. 03.12.2018 5
How Precision Agriculture Responses to a Yield Map?
1) Simple: Find the [x] deficiency, then apply more [x]
2) Advanced: Understand the reason of the problem and apply the correct remedy
Learn the Problem
and Deliver
Right amount
&
Right shape
of solution to it.

6. 03.12.2018 6
Video is courtesy of Russell Moss, Lecturer in viticulture at Cornell University
Example: Normalized Difference Vegetation Index (NDVI)

7. 03.12.2018 7
Automation
& Control
Internet
of
Things
Big Data
Unmanned
Aerial
Vehicles
Artificial
Intelligence
Smart
Sensors
Virtual Farms
Wireless
Sensor
Networks
Deep
Learning
Precision
Management
Agricultural
Robotics
A solution to manual
operations, labor cost and
labor shortage

8. 03.12.2018 8
Replacing labors with robotsAgricultural Robotics
Stats and analytics
Cloud Apps
 Notifications
Image credit: www.AdaptiveAgroTech.com
Farmer
Agro-
Industry
Autonomous
Scouting
Smart
Data
Robotic
seeding
Robotic
planting
Robotic
weeding
Robotic harvesting
Field
Agents
Cloud
Advisory System

9. 03.12.2018 9
Platform Design
o Vehicle Selection
o Sensor Selection
o Data Acquisition
Data Management
o Data Modeling
o Data Storage
o Data Access
Data Processing
o Quality Control
o Geo-processing
o Outlier Removal
Field Design & Layout
o Experimental design
o Plot Map Development
o GIS Shapefiles
Data Enhancement
o Simulation Modeling
o Model Inversion
o Parameter Estimation
 Collect data in the field using Data Collection Equipment (Ag-Robotics and Sensors)
 Manage, Process, and Enhance data
 Make Data useful for Ag&Bio, Genetic, etc..etc applications
Ag&Bio Applications
o QTL Mapping
o GWAS
o Breeding Selection
Definition
Example

10. 03.12.2018 10
Another good Example of a Phenotyping Robot

11. 03.12.2018 11
Data collection
Data processing
Decision making
Control action
Internet
Power source Greenhouse sensors GSM & IoT modules
Data collection Data transmission to cloud
Data analysis
and storage
Data sharing and decision making
SMS
App Notifications
Emails
Stats and analytics
0 2 4 6 8 10 12 14 16 18 20 22 24
0.5
1
1.5
2
2.5
Time (Hours)
VPD(kPa)
VPD, Seremban
VPD, Kota Kinabalu
Sensors are the product of
Adaptive AgroTech Consultancy Int
www.AdaptiveAgrotech.com

12. 03.12.2018 12
We are in the 21st century, yet million tons of fruits and vegetables for fresh market are picked-off manually!
Courtesy of Queensland Univ of Technology Abundant Robotics SWEEPER Robot EU
We need a reliable and robust robotic fruit harvesting platform for
efficient, cost-effective and bruise-free fruit picking.

13. 03.12.2018 13
(Source: ATB-Potsdam, Credit: Dr.-Ing. Volker Dworak)

14. 03.12.2018 14

15. 03.12.2018 15
Global Ag-Robot Market in 2016: USD 3.5 BILLION
Growing at a compound annual growth rate (CAGR) of 21.5% during the forecast period of 2017-2024
Top U.S. companies
1. Deere & Company
2. Trimble Inc.
3. AGCO Corporation
4. Agjunction, Inc.
5. BouMatic Robotics, B.V.
6. Lely Holding S.À.R.L
7. AG Leader Technology
8. Topcon Positioning Systems, Inc.
9. AG Eagle LLC
10. Agribotix LLC
11. Autocopter Corp
12. Blue River Technology
13. Auroras S.R.L.
14. Grownetics Inc.
Ag-Robotic market offers:
Hardware, Software, Services
UAV/Drones
Milking Robot
Driverless Tractors
Automated Harvesting Systems
Harvest Management
Field Farming
Dairy Farm Management
Soil Management
Irrigation Management
Pruning
Weather Tracking & Monitoring
Inventory Management
Market Drivers And Restraints:
• Increased global food demand,
• Rising government initiatives with automation,
• Increasing adoption of new technologies in farming,
• High capital investments,
• Lack of awareness and technical knowledge
Source: Data Bridge Market Research

16. 03.12.2018 16
Dynamic system
Mobile (Field robots)
Non-mobile Unmanned Aerial Systems
Multi-robot, Hybrid
• Autonomous navigation
• Field Scouting
• Plant Phenotyping
• Weeding
• Targeted spraying
• Seeding
• Fertilizer
• Transplanting
• Delicate handling
• Multi-purpose
• Harvesting
• Milking robots
• Quality assessment
• Sorting robots
• Phenotyping
• Post-harvesting
• Handling
• Packaging
• Pruning
• In-field quality assessment
• Data collection platform
• Remote sensing platform
• Mapping application
• Spraying
• Inventory management
• Yield monitoring
• Disease detection
Multi-Rotor, Fixed wing

17. 03.12.2018 17
Quality assessment
for seed sorting
Quality assessment
for seedling sorting
Quality assessment in
Greenhouse cultivation
UAS-based quality
assessment in field
Precision data
collection in field
Autonomous
navigation
Robotic harvesting
Quality assessment
for fruit sortingAgro-food processing
Robotic packaging
Robotics for Agro-Food
Robot animations are courtesy of Wageningen UR

18. 03.12.2018 18
PART 2
Advances in Agricultural Robotics

19. 03.12.2018 19
VINESCOUT: Claimed to be one of the most sophisticated orchard robot
 Photovoltaic powered
 Monitoring key wine vineyard parameters,
(i) water availability, (ii) the temperature of the leaves, (iii) plant robustness
 Robust autonomous navigation using 3-D images with LiDAR and
ultrasounic sensors
 Built-in artificial intelligence for improved row-end turning
 Operates days and nights
 20 measurement/hour, more than 3000 data value/hour
 Produce information for irrigation, harvesting date, yield distribution
http://vinescout.eu/web/
Wageningen UR 2018 orchard robot
- Developed for rugged terrain
- Two LiDAR scanners
- Bi-directional driving
- Two headland turn sequences
- Obstacle avoidance
- 8 hour non-stop operating time at 4
km/h
https://youtu.be/nGWCTSavlAQ
BoniRob

20. 03.12.2018 20
PROBOTIQ Fendt 211V X-pert
System von PRECISION MAKERS
Source: https://youtu.be/fiX1cHr_OWA
Mobius for Orchard Vineyard
Source: https://www.asirobots.com/farming/orchard-vineyard/
Source: https://youtu.be/sZCArRruLz8
Autonomous orchard
spraying with GOtrack GPS
Platform layer:
• Tractor selection (John Deere, Fendt, Kharkiv, Solectrac)
• Implement selection (Sickle bar mower, sprayer, etc)
Sensors layer
• GNSS, DGPS, IMU, Camera and vision sensors, RADAR, Ultrasonic
• Laser and LiDAR, etc
Autonomous navigation layer
• Hardware/software interface, Localization and Mapping
• Obstacle avoidance algorithms,
• Waypoint and trajectory following

21. 03.12.2018 21
1. Position based and Angular-based
o GNSS, RTK, DGPS,
o IMU (Inertial measurement Unit)
2. Vision based
o Regular RGB, or Fisheye
o Night vision
o 3D cameras
o Other: i.e., Microsoft Kinect (RGB+IR)
3. Radio Detection and Ranging (RADAR)
o Short range and long range RADAR
o Ultrasonic and Proximity sensors
4. Laser-based
o LiDAR scanners
o Point by point ToF (Time of Flight)
6. Dead reckoning (relative)
o Wheel odometry (i.e. shaft encoder)
o Accelerometers and Gyroscopes
o Mid and high accuracy INS (inertial navigation systems)
5. Infrared
o Thermal camera
o Range finders
o Time Of Flight
 Sensor fusion algorithms
Radar
LongRangeRADAR_ARS300
Mid Accuracy AHRS and INS/GNSS High Accuracy AHRS and INS/GNSS
SwissRanger (TOF)
Sharp IR range finder
Thermal cameras
TeraRanger One ToF
Point Grey
Night vision 3D camera
Kinect
Fisheye

22. 03.12.2018 22

23. 03.12.2018 23
 Sensing inputs and coverage
 Precision, accuracy, robustness, and response time
SunBot
E-Tractor
 Automation and Safety

24. 03.12.2018 24
• https://ec.europa.eu/eip/agriculture/en/find-
connect/projects/emissionsfreie-strauchbeerenproduktion-sunbot
• https://eip-agri.brandenburg.de/cms/media.php/lbm1.a.3310.de/EIP-
SunBot_210818.pdf

25. 03.12.2018 25
Courtesy of Wageningen UR and http://sweeper-robot.eu/Courtesy of CROPS http://www.crops-robots.eu/
Courtesy of http://www.ffrobotics.com/
Courtesy of Univ of Florida. Image source: http://ncr.mae.ufl.edu/index.php?id=research/citrus_harvesting sites.google.com/site/cvhanaian/research Courtesy of Energid Citrus Picking System
The arm of Robotic Harvesting's machine picking
strawberries.
Courtesy of Queensland Univ of Technology
 Published RESEARCH on Robotic Harvesting is Huge
 There is currently no commercial harvesting robot in
the market
 SWEEPER aims to introduce the first commercial
robot for harvesting of sweet pepper

26. 03.12.2018 26

27. 03.12.2018 27
Source: https://www.popco.net/zboard/view.php?id=dica_news&no=13777&pcl=mpopco
3d Print an Artificial
Muscle Robot Hand
https://www.instructables.com/id/3d-
Print-An-Artificial-Muscle-Robot-Hand/

28. 03.12.2018 28

29. 03.12.2018 29
PART 3
Future Scenario in Agricultural Robotics

30. 03.12.2018 30
Data acquisition
Filtering, Perception
{Simulation + Real World}
SENSE
Image processing, Control
algorithms, Decision making
{Simulation + Real World}
THINK
Inverse and Forward Kinematics, Fruit tracking,
Visual Servoing, Motor and Joints control, Actuators
activation, Yield estimation, Vegetation Index,
Variable rate, etc…{Simulation + Real World}
ACT
Exchanging “acting” between real-world and simulation
Implication for digital robotic harvesting
Exchanging “sensing-and-thinking” between real-world and simulation
Implication for field and crop management
Two Simple Live Demonstration

31. 03.12.2018 31

32. 03.12.2018 32
Source: Redmond R. Shamshiri, AdaptiveAgroTech Consultancy Int
Source: SWEEPER EU H2020 project
consortium –(www.sweeper-robot.eu)
In this experiment the robot is trying different approaching directions using visual
servo control to locate and move towards the fruits that are not blocked by leaves.

33. 03.12.2018 33
Actual Simulated Actual Simulated a b

34. 03.12.2018 34
Acting simulation for robotic harvesting of citrus with multiple linear manipulators. Source: Energid Technologies
Using simulation to adapt multiple linear manipulators for harvesting of sweet pepper. Source: Redmond@AdaptiveAgroTech.com

35. 03.12.2018 35

36. 03.12.2018 36
Jaco Arm
Velodyne sensor
Kinect
Husky
J

37. 03.12.2018 37
Navigation Sensing and actuation Image from stereo sensors
Joints
Servo control

38. 03.12.2018 38
Digital Farming
o V-REP
o MATLAB
o ROS
o MicroController
o Sensors:
 Ultrasonic
 Infrared
 PSD
 RGB
Demo 1: An artificial hand in the simulation world responds to the
operators hands in the real world

39. Source for the raw mesh model: GrabCad
Step 1: Creation of 3D models for simulation
Step 2: Designing mechanism, setting positions for gears, joints, pins, etc

40. Step 3: Setting correct axis for the joints and members
Step 4: Building IK tasks for the arm and each fingers

41. Step 5: Improving the simulation
Adjusting inertia and materials properties for more realistic effects
Re-programming the IK with different constraints and simulation engines

42. Step 6: Experiments and calibrations
Exchanging data between real and simulated sensors

43. Final simulation set-up
The artificial arm in the simulation world responds to the operator’s hand in the real world

44. Video #1: Exchanging “ACTING” between real-world and simulation-world
Video is available at https://youtu.be/1IA4Lk-aHzc
Video Source: Redmond@AdaptiveAgroTech.com
03.12.2018 44

45. 03.12.2018 45
Video Source: Eric Rohmer Youtube
Video source: robokid88

46. 03.12.2018 46
Video source: robokid88

47. Simulated
Environment
Image Processing in MATLAB and V-REP
Control signals are send to the real world, generating new image scene
Exchanging live stream images and sensor data between simulation-world
(i.e. virtual orchards) and real-world
Implications: Fruit detection, Yield monitoring, Inventory management, Health
assessment, Vegetation index
Demo #2: Exchanging “SENSING AND THINKING” between real-world and simulation
MATLAB Window

48. Video #2: Exchanging “SENSING AND THINKING” between real-world and simulation
Video is available at https://youtu.be/1IA4Lk-aHzc
03.12.2018 48
Video Source: Redmond@AdaptiveAgroTech.com

49. Virtual Reality: Creation of Virtual Orchards: LiDAR or Photogrammetry
Software: 3DF Zephyr
Image source: https://all3dp.com/1/best-photogrammetry-software/
Image source: https://www.truepointscanning.com/truepoint-provides-point-cloud-and-3d-mesh-of-trees
Image: AdaptiveAgroTech.com
https://forum.sketchfab.com/t/exporting-colored-point-cloud-from-capture-reality-and-meshlab/13928

50. Examples of virtual orchard
GIF Image source: http://www.kvpr.org/post/3d-orchards-uc-researcher-turns-farms-virtual-reality
03.12.2018 50

51. Demo #3: Digital Farming for Orchard management
Video is available at https://youtu.be/1IA4Lk-aHzc
03.12.2018 51
Video Source: Redmond@AdaptiveAgroTech.com

52. 03.12.2018 52
Challenges of commercialization?
Despite of all research, there are several applications for which Ag-Robotics
have not reached the full-scale commercialization stage. Example: Robotic Harvesting
Why? Which section(s) do you think require more investment?
Body
structure
Sensory
system
Planning
algorithms
Control
systems
Actuating
mechanism

53. Acknowledgement
The technical and consultation support from
Dr. Jörn Selbeck
Dr.-Ing. Volker Dworak
Dr. rer. nat. Michael Schirrmann
Are duely acknowledged.
https://florida.academia.edu/Redmond
https://www.researchgate.net/profile/Redmond_Shamshiri
https://www.linkedin.com/in/redmond-r-shamshiri-051b7930/