Recommended
More Related Content
What's hot
What's hot (20)
Similar to Robotic Harvesting Simulation Accelerates Fruit Picking Robot Design
Similar to Robotic Harvesting Simulation Accelerates Fruit Picking Robot Design (20)
More from Redmond R. Shamshiri
More from Redmond R. Shamshiri (16)
Recently uploaded
Recently uploaded (20)
Robotic Harvesting Simulation Accelerates Fruit Picking Robot Design
- 1. Robotic Harvesting of Fruiting Vegetables, “Acceleration by Simulation” Redmond Ramin Shamshiri Muhammad Razif Mahadi, Desa Ahmad, Siti Khairunniza Bejo, Samsuzana Abd Aziz, Hasfalina Che Man Smart Farming Technology Research Center, Univ Putra Malaysia www.AdaptiveAgroTech.com Contact Info 11 -12 SEPTEMBER 2017 Delf Univ of Technology The Netherlands Presented at the Acceleration Workshop Robotics & Crop Sensing in Greenhouses, 11-12 September, 2017. Delf University of Technology, The Netherlands Organized by
- 2. Statement of the problem: Hand Harvesting in the 21st Century! © Redmond Ramin Shamshiri We are in the 21st century, yet million tons of fruits and vegetables for fresh market are picked-off manually! Question: How can we accelerate designing of a reliable and robust robotic fruit harvesting platform for efficient, cost-effective and bruise-free fruit picking.
- 3. Review: Advances in Automated (Robotic) Harvesting. “Different hardware setup” © Redmond Ramin Shamshiri Courtesy of Wageningen UR and http://sweeper-robot.eu/Courtesy of CROPS http://www.crops-robots.eu/ Courtesy of Univ of Florida. Image source: http://ncr.mae.ufl.edu/index.php?id=research/citrus_harvesting Source: https://sites.google.com/site/cvhanaian/research Courtesy of Energid Citrus Picking System Courtesy of http://www.ffrobotics.com/
- 4. Review: Advances in Automated (Robotic) Harvesting. “Different control strategy” © Redmond Ramin Shamshiri Source Link Courtesy Agricultural machinery maker Shibuya Seiki The arm of Robotic Harvesting's machine picking strawberries. http://mathscinet.ru/robocenter/ Cucumber harvester. Courtesy of Wageningen UR. Source: https://www.wageningenur.nl/en/show/Cucumber-harvesting-robot.htm Courtesy of Queensland Univ of Technology
- 5. Question: Which design is more efficient? How many DoF? What VSC algorithm? © Redmond Ramin Shamshiri versus
- 6. Finding the optimum solution. (1): Analytical techniques © Redmond Ramin Shamshiri Solution 1: Extensive analytical techniques. Is it accurate and reliable? How about finding optimum places for sensors? Testing algorithms? Visualizing performances?
- 7. © Redmond Ramin Shamshiri 1. Robot Virtual Worlds 2. RoboDK 3. Microsoft Robotics Developer Studio 4. Webots 5. Workspace 6. V-REP 7. LabVIEW 8. OpenHRP3 9. Player 10. Simbad 11. Algodoo 12. RoboWorks 13. RobotStudio 14. Gazebo 15. Actin Simulation 16. Blender 17. WorkcellSimulator 18. robotSim:Edu 19. 3DSimulate 20. Roboguide 21. RoboLogix 22. Ezphysics 23. SimRobot Finding the optimum solution. (2): Simulation approach
- 8. Simulation for Acceleration © Redmond Ramin Shamshiri Experiments with actual hardware setup in robotic harvesting are not always feasible due to time constraints, unavailability of equipment and experiment costs (i.e. sensors, cameras, and the robot manipulator) Some hardware setups may result in actuator saturation, or create unsafe situation to the operators and/or plants system. Simulation can provide a cheap and safe experiment platform with a faster approach for development, testing and validating control strategies and algorithms.
- 9. Using simulation for innovative designs, Thinking outside the box: © Redmond Ramin Shamshiri Eliminating the redundant vision feedbacks and mission time Linearizing the problem Creating Sub-Problems (smaller task) Manual fruit detection, Automated harvesting Re-Thinking the Manipulator Re-Thinking the Gripper Re-Thinking the image-processing Re-Thinking the entire strategy Camera View Camera View Professional Robot, 20-60,000$/unit Multiple simple robots, below 3000$/unit
- 10. Example of an innovative idea for simulation © Redmond Ramin Shamshiri Can we use multiple linear actuators with low-cost webcams for harvesting? We have provided a simulation for this idea at the end of the presentation Suction Gripper
- 11. Example of Fruit detection and tracking using low-cost webcams © Redmond Ramin Shamshiri Video Demo Real-time, robust and rapid red-pepper fruit detection with Matlab, © Redmond Ramin Shamshiri: https://youtu.be/rFV6Y5ivLF8
- 12. Simulation Recipe © Redmond Ramin Shamshiri Computer simulation of a complete robotic harvesting task requires: 1. CAD file setup including good replication of the plants-and-fruit scene, robot manipulators, etc 2. Software setup and calculation modules (i.e., inverse kinematics) for the manipulator candidates and platforms 3. Sensors and experiments setup 4. Good algorithms for data processing, i.e. Simulation of visual servo control task for the robot and gripper 5. Some people with passion
- 13. Simulation Recipe: CAD files, Fruit and plant model © Redmond Ramin Shamshiri Source of CAD files: Some self designed, some Internet orders and purchased
- 14. Simulation Recipe: CAD files, Fruit and plant model © Redmond Ramin Shamshiri
- 15. Simulation Recipe: CAD files, Robot manipulators © Redmond Ramin Shamshiri Jaco Arm KUKA LBR 4 KUKA LBR iiwa 14 R 820 KUKA LBR iiwa 7 R 800 UR 10 UR 5Mico Arm FANUC LR Mate 200iD NOVABOT (Has not been manufactured yet) Courtesy of Arjen Hoti @GrabCad Kinova robotics
- 16. Example: NOVABOT © Redmond Ramin Shamshiri The CAD file is Courtesy of Ajit Hoti (Published Aug.10.2017) Simulation file is courtesy of Adaptive AgroTech
- 17. Simulation of NOVABOT for Robotic harvesting of sweet pepper © Redmond Ramin Shamshiri
- 18. Simulation Recipe: CAD files, lower-cost and conceptual design robots © Redmond Ramin Shamshiri PhantomXPincher Can they go head-to-head with professional robots ? C6 6-Axis Servos Control Palletizing Robotic Arm Model for Arduino UNO/ MEGA2560 Price: ~ USD160 They cost 10-500 times less than professional robots
- 19. Example: Simulating a conceptual design © Redmond Ramin Shamshiri
- 21. Simulation Recipe: Software Setup, V-REP, ROS, MATLAB © Redmond Ramin Shamshiri 3D laser scanner Kinect Hokuyo URG-04LX-UG01 Arm camera RGB vision sensor Proximity sensor Fixed Camera A 180° scanning path Cameras/sensors floating views and graphs Plant model Video Demo V-REP, ROS Env Setup ©Redmond Ramin Shamshiri https://youtu.be/tKagjNQ9FW4
- 22. Simulation Recipe: Calculation modules, Inverse Kinematics, minimum distance, etc © Redmond Ramin Shamshiri Minimum distance calculation collision detection path planning Remote API Serial port Plugins ROS nodes Control Mechanisms Embedded Child Script Control mechanism inverse kinematics
- 23. Simulation Recipe: Sensors, i.e., Vision sensors, Laser Scanners, etc © Redmond Ramin Shamshiri Fish-eye RGB Axis 212 PTZ sensor (left, Source) and V-REP perspective vision-sensor (right) Microsoft Kinect, Fast Hokuyo URG-04LX-UG01” scanning Laser Rangefinder, and “3D Laser Scanner Fast” rangefinder An example of Infrared Proximity Sensor Long Range - Sharp GP2Y0A02YK0F, and Bottom: different proximity sensors types in V-REP, respectively from left to right: Ray-type, pyramid-type, cylinder-type, disk-type and cone-or randomized ray-type proximity sensors
- 24. Simulation Recipe: Sensors and experiment setup i.e., 360-degree scanning experiment © Redmond Ramin Shamshiri Simulated scan of a red sweet pepper fruit. Camera view at 30 degrees increments around the fruit model and leaves. The robot moves in the x-y plane for finding best angle of attack Fruit/plant 360-degree scan experiment in x-y plane, and the right image is a top view of the scanning setup. The RGB sensor mounted on the robot tip moves on the horizontal plane x-y, to find the best view of the fruit
- 25. Simulation Recipe: Sensors and experiment setup i.e., vertical scanning experiment © Redmond Ramin Shamshiri Scanning around fruit/plant in x-y-z plane, the manipulator is “twisted” to provide different viewpoints for the end-effector camera Vertical scan of fruits/plants in x-y-z plane, the end-effector camera is moved and rotated in both x-z and y-z directions
- 26. Experiment with sensors? Environment scanning, reconstruction of fruit/plant © Redmond Ramin Shamshiri Depth image to work image Extract coordinates from work image Work image to output image Filter used in the “Fast Hokuyo URG- 04LX-UG01” sensor “Fast Hokuyo URG-04LX-UG01” scanning Laser Rangefinder “Hokuyo URG-04LX-UG01
- 27. © Redmond Ramin Shamshiri Depth sensors Microsoft Kinect Depth image to work image Depth image to work image Depth image to work image Experiment with sensors? Environment scanning, reconstruction of fruit/plant
- 28. © Redmond Ramin Shamshiri Original depth image to work image Extract coordinates from work image Intensity scale work image Work image to output image 3D Laser Scanner Fast” rangefinder Experiment with sensors? Environment scanning, reconstruction of fruit/plant
- 29. Simulation Recipe: Algorithms, i.e., Simulating Visual Servo Control © Redmond Ramin Shamshiri Objective: A robot end-mounted camera is used to position the robot arm in a plane orthogonal to the axis, such that the fruit to be harvested is centered in the camera’s field of view. Control law Inverse Kinematics Feature extractionPose determination Joint Controllers Power amplifiers + - 𝑪 𝒙 𝒅 𝑪 𝒙 Position-based eye-in-hand visual servo structure Image processing Feature extraction Pose determination Inverse kinematicsHardware setup Reference values CONTROLLER The system has no trajectory generator, instead a feedback loop, closed visually, is used to control the robots arm position. (Two control laws were simulated: join angles and joint velocity) Input Output
- 30. Algorithms, i.e., Joint position vs. Joint velocity control for VSC © Redmond Ramin Shamshiri Joints position control law (Localization) 1 2 3 4 5 6 Joints velocity control law (Tracking) Robot and camera with eye-in-hand configuration setup. The RGB camera was mounted on the tip of the FANUC LR Mate 200iD robot manipulator.
- 31. Algorithms, i.e., Deep learning for Maximum visibility © Redmond Ramin Shamshiri Z 65% Visible 85% Visible 98% Visible What about hand- in-eye approach?
- 32. Algorithms, i.e., Optimization for minimum time < 6s © Redmond Ramin Shamshiri X Y Z Desired Time: 4 to 6 sec!!
- 33. Simulating Conceptual design: Google search © Redmond Ramin Shamshiri Image source: http://www.freshplaza.com/2012/0816/paprikarobot.jpg Image: Courtesy of Vision Robotics. http://www.wired.com/2007/06/robo-picker/ Courtesy of Hanaian. https://sites.google.com/site/cvhanaian/research Vision Robotics.
- 34. Simulating two simple mechanisms © Redmond Ramin Shamshiri Arrays of Linear actuators vs. multiple SCARA robot Support Cylinder Piston Adjustable extension Suction cup
- 35. Simulating arrays of robots? A: Linear actuators, Are they promising? © Redmond Ramin Shamshiri
- 36. Arrays of robots? A: Linear actuators © Redmond Ramin Shamshiri Adjustable width
- 37. Arrays of robots? A: Linear actuators © Redmond Ramin Shamshiri
- 38. Arrays of robots? A: Linear actuators © Redmond Ramin Shamshiri
- 39. Arrays of robots? A: Linear actuators © Redmond Ramin Shamshiri
- 40. Simulating arrays of robots? B: SCARA manipulators, Are they promising? © Redmond Ramin Shamshiri Farmer’s price robotic arm. Can we have it at laptop cost? Suction cup
- 41. Arrays of robots? B: SCARA manipulators © Redmond Ramin Shamshiri
- 42. Arrays of robots? B: SCARA manipulators © Redmond Ramin Shamshiri
- 43. Arrays of robots? B: SCARA manipulators © Redmond Ramin Shamshiri