Talk given at ESA Baltimore, August 13, 2015

1. NextGen tools for scaling from
seeds to traits to ecosystems
Tim Brown, Research Fellow, Borevitz Lab
ARC Centre for Plant Energy Biology, Australian National University
Joel Granados, Chuong Nguyen, Kevin D. Murray, Riyan
Cheng, Cristopher Brack, Justin Borevitz,

2. In the next 100 years we will face challenges of
unprecedented scale and complexity

3. Humans are causing a 6th mass extinction
Since 1500, ~30 -90% species declines globally 1,2
1) Dirzo, Rodolfo, et al. 2014. DOI: 10.1126/science.1251817
2) http://worldoceanreview.com/en/wor-2/fisheries/state-of-fisheries-worldwide/

4. More than 7 billion people
on the planet for the next 100 years
• >10 billion people on the planet by 2050

5. Feeding 10 billion people will be very hard
• We must grow more food in the next 75 years than all the food
produced in human history 1
• Requires a 38% greater yield increase over historical gains, every year
for the next 40 years2
1) “Seeds of Doubt” New Yorker, Aug 25, 2014
2) Tester & LeGrange. 2010. Science:327(818)

6. Terraforming
“To alter the environment of a planet to make it capable
of supporting terrestrial life forms.”
We are currently unterraforming the earth at an exceptionally fast rate
To meet the challenges of the coming century we need to restore and
re-engineer the environment to support >7 billion people for the next
100 years in the face of climate change, while maintaining biodiversity
and ecosystem services
These ecological challenges are too hard to be solved
with existing data and methods

7. The current resolution of field ecology is very limited
• Low spatial & time resolution data
• Limited sensors types
• Sampling is often manual and subjective
• Observations not-interoperable or proprietary; little or no data sharing
• Sample resolution is “Forest” or “field” not Tree or Plant
• Very little data from the last century of ecology is available for reuse
This slows our rate of knowledge discovery
© Tim Brown

8. Research focus
How can we use new technology to quantify environmental
processes in high resolution for scientists (and the public) in a
format that they can use?
(1) Lab phenomics
• “Phenotype” (plant behaviors) + “Genomics” (plant genomes)
• Identifying the genetic basis of plant growth and development
(2) Field ecology
• High resolution monitoring of ecosystems

9. Genotype x Environment = Phenotype
Phenotype (and ecosystem function) emerges as a cross-scale
interaction between genotypes and the environment they experience.
The degree to which we can measure all three components is the
degree to which we can understand ecosystem function
Requires LOTS of complex data!

10. The challenge – Measure everything all the time
How do we go from doing the science at
the scale of one point per forest to
multilayer data cubes for every tree or
leaf?
10/
20

11. This isn’t impossible
• Google didn’t exit 17 yrs ago and now it indexes 30 trillion
web pages
• You can now ASK google almost anything and get a pretty
good answer
• 1.8 billion (mostly geolocated) images are uploaded to
social media every day (2014; was 500m in 2013)1
We need this level of resolution (and google-like tools) for
ecological knowledge
1. Meeker, 2013, 2014

12. Lab vs field phenotyping
Lab: High precision measurement and control but low realism
youtu.be/d3vUwCbpDk0

13. Lab vs field phenotyping
Field: Realistic environment but low precision measurements
In the field we have real environments but the complexity (and bad lighting!) reduces
our ability to measure things with precision
youtu.be/gFnXXT1d_7s

14. Lab phenotyping
Normal lab growth conditions aren’t very “natural”
Kulheim, Agren, and Jansson 2002
Real World
Growth Chamber

15. Borevitz Lab Approach
• Create more “natural” Lab conditions with precision LED’s
and temperature control
• Measure more precisely in the Field
© Suzanne Marselis
enviro-net.org

16. Growth cabinets with dynamic “semi-realistic” environmental &
lighting conditions
• 8 & 10-band controllable LED lights to control light spectra, intensity
• Python scripts control chamber Temp/Humidity in 5 min intervals
• Grow plants in simulated regional/seasonal conditions & simulate climate
• E.g. coastal vs inland | Late Spring or Early Fall
• Expose “cryptic” phenotypes
• Repeat environmental conditions
• Between studies and collaborators
• Simulate live field site climate
SpectralPhenoClimatron (SPC)
Spectral response of Heliospectra LEDs. (L4A s20: 10-band)

17. Real-time monitoring, analysis and data visualization
• Phenotype 2,000 plants in real-time 24/7
• 2 DSLR cameras / chamber * 7 chambers
• JPG + RAW imaging every 10 minutes  processing server
• Automated Image analysis pipeline to extract
growth data from 150,000 pot images a day
• Detect color checker
• Correct color and lens distortion
• Detect pots
• Segment each image
• Leaf detection and tracking
Corrected
Segmented
Original

19. Goal: High Throughput Phenotyping for the masses
Most high throughput phenotyping systems cost millions of $$
Our system:
• Low cost:
• Off the shelf cameras controlled by $35 linux pc’s (<$5K/chamber)
• Online project management, visualization and analysis tool
• Open source, cloud-based analysis & bioinformatics pipelines
• “Develop once and everyone can use it”

20. NextGen Field Ecology – Where’s my PCR?
• Field ecology is like genetics before PCR and high throughput
sequencing
• Back in the ’80s & early 90s people would get a PhD just sequencing
a single gene.
• Humane Genome Project cost $2.7billion USD and took 13 years
• Now (20 years in) we buy sequencing machines on eBay and light
sequence whole genomes for <$50 a plant in just a few weeks

21. Genetics -> Genomics -> Phenomics
20 years of technical advances have turned genetics into genomics into
phenomics and yielded the ability to address fundamental and very
complex questions
Now state of the art phenomics is pretty high resolution
• Precision environmental controls
• 3D time-lapse models of every plant growing with each pixel in all
spectra mapped to the 3D “data cube” model of each plant
• Genome sequence for every plant
• Automated bioinformatics pipeline for trait extraction

22. The real world is way more complex than plants in the lab
We need equally complex datasets and models to
understand real world ecosystems
• The questions we ask have often been defined more by what data
we can get than by what the best question would be

23. So how do we measure everything all the time?
• Persistent 3D, time-series multi data-layer ecosystem
models tracking every tree

24. “NextGen” field monitoring
Within 5-10 years we will have similar data to lab phenomics
• Automated time-series (weekly/daily) aerial (UAV) scans measure
• RGB; Hyperspectral; Thermal; LiDAR
• Centimeter resolution 3D model of every tree on field site
• Gigapixel imaging to track phenology in every tree/plant, hourly
• Automated computer visions analysis for change detection
• LIDAR (laser) scanning
• DWEL / Zebedee – high resolution ground-based 3D scans
• Dense point clouds of 3D structure
• Microclimate sensor networks
• PAR
• Temp, Humidity
• Soil moisture @ multiple depths
• mm resolution dendrometers
• Full genome for every tree on site (<$5/tree)
Environment
Phenotype
Genetics

25. National Arboretum Phenomic & Environmental Sensor Array
National Arboretum, Canberra, Australia
ANU Major Equipment Grant, 2014
Collaboration with:
• Cris Brack and Albert Van Dijk (ANU Fenner school) ; Borevitz Lab

26. National Arboretum Phenomic & Environmental Sensor Array
• Ideal location
• 5km from ANU (64 Mbps wifi) and near many research institutions
• Forest is only ~4 yrs old
• Chance to monitor it from birth into the future!
• Great site for testing experimental monitoring systems prior to
more remote deployments
26/
20

27. National Arboretum Sensor Array
• 20-node Wireless mesh sensor network (10min sample interval)
• Temp, Humidity
• Sunlight (PAR)
• Soil Temp and moisture @ 20cm depth
• uM resolution denrometers on 20 trees
• Two Gigapixel timelapse cameras:
• Leaf/growth phenology for > 1,000 trees
• Campbell weather stations (baseline data for verification)
• All data live online in realtime
• LIDAR: DWEL / Zebedee
• UAV overflights (bi-weekly/monthly)
• Georectified image layers
• High resolution DEM
• 3D point cloud of site in time-series
Total Cost ~<$250K AUD

28. Arboretum Video
https://www.youtube.com/watch?v=YanOqSlW7yE

29. High resolution tools for the field
1. Gigapixel imaging
2. UAV’s (drones)

30. -
Golfer, 7km distant
Monitor daily change in every plant in your field site
Gigapixel imaging

31. 20 gigapixel image of Canberra, Australia from the Black Mountain Telstra Tower
Zoom in to the National Arboretum

32. Midsummer
Zoom in to the Each forest at the Arboretum

33. Low cost sequencing let’s us genotype every individual tree and identify genetic loci that correlate
with observed phenotypic differences between trees.
We can do this for all trees at the arboretum within view of the camera.
Fall Color change shows differing rates of fall senescence in trees
Late fall

34. UAV’s (drones) for monitoring
• $2-4K airframe (DJI, Aeronavics) + 10-20MP digital
camera (~1kg payload)
• Processing software $700 - 2,000 USD (Agisoft; Pix4D)
• 3D models of field site (cm resolutions are possible)
• Orthorectified image and map layers
• LAS / point cloud data
• Automated pipeline: Flight -> tree data
• Tree Height; Volume, foliage density
• RGB color
• GPS location
• DEM of site
34/
20
View 3D model online:
http://bit.ly/ARB3Dv1

35. Software outputs DEM and point cloud data
• Processing script for tree data:
• GPS, Height, 3D volume, top-down area, RGB phenology data
• Straight to google maps online

36. Ultra-high resolution ground-based laser
• DWEL (CSIRO); Echidna (handheld; $25K LiDAR)
• Multiband Lidar with full point returns
• ~30 million points in a 50m2 area (vs 5-10 pts/m for aerial)
Data: Michael.Schaefer@csiro.au

37. 3D trees rendered from LiDAR data
Image: Stu Ramsden, ANU Vislab

38. So what do we do with all this data?
The challenge is no longer to gather the data, the challenge is how
we do science with the data once we have it
• A sample is no longer a data point
• Example: Gigavision data
Sample:
• Camera hardware: 900 images (per hour)
• Automate stitching into panoramas (5-20,000 tiled images/pano/hr)
• Need to align time-series images to each other and to the real world
(despite hardware failures, camera upgrades)
• How to visualize a time-series of 22 million images/year?
• Computer vision analysis
• Automated feature detection and phenophase detections
To Data:
• “Phenophase” transition and growth data from 1,000 trees
• And then how do you even analyze that?!

39. Virtual 3D Arboretum Project
• Goal:
• Use modern gaming software to explore new methods for
visualizing time-series environmental data
• Historic and real-time data layers integrated into persistent 3D
model of the national arboretum in the Unreal gaming engine
• Collaboration with
• ANU Computer Science Dept. TechLauncher students
• Stuart Ramsden, ANU VISlab

41. Tips for managing big data
1. This is hard! Don’t feel bad. No one else has much of an idea how
to do it either.
2. Where we are now in dealing with big data is like dealing with
numbers before excel
3. Learn to program; hire computer scientists whenever possible
4. Collaborate but don’t be afraid to make executive decisions:
1. There are many ways to solve the same problem
5. Make a robust data management plan; don’t collect data until you
have a plan to organize it
1. BUT don’t be afraid to jump in and fail. Learning is iterative and there aren’t
actually consensual solutions to most of these problems yet
6. Virtual machines and cloud storage are your friends
1. Don’t manage any hardware you don’t have to
2. Convince your university to invest in computing infrastructure and good IT
support
7. Share code – If you solve a problem give the solution to others!

42. Thanks and Contacts
Justin Borevitz – Lab Leader Lab web page: http://borevitzlab.anu.edu.au
• Funding:
• Arboretum ANU Major Equipment Grant
• ARC Center of Excellence in Planet Energy Biology | ARC Linkage 2014
• Arboretum
• http://bit.ly/PESA2014
• Cris Brack, Albert VanDijk, Justin Borevitz (PESA Project PI’s)
• UAV data: Darrell Burkey, ProUAV
• 3D site modelling:
• Pix4D.com / Zac Hatfield Dodds / ANUVR team
• Dendrometers & site infrastructure
• Darius Culvenor: Environmental Sensing Systems
• Mesh sensors: EnviroStatus, Alberta, CA
• ANUVR Team
• Zena Wolba; Alex Alex Jansons; Isobel Stobo; David Wai
• TraitCapture:
• Chuong Nguyen; Joel Granados; Kevin Murray; Gareth Dunstone; Jiri Fajkus
• Pip Wilson; Keng Rugrat; Borevitz Lab
• Contact me:
• tim.brown@anu.edu.au
• http://bit.ly/Tim_ANU
http://github.com/borevitzlab