A presentation on my work to the Robert Mahony's lab at the ARC Centre of Excellence for Robotic Vision at ANU. Video here: http://youtu.be/IGPZSZn_zzw

1. Scaling from seeds to ecosystems
Computer Science and engineering challenges for
NextGen ecosystem monitoring and genomics
Tim Brown, PhD
Research Fellow, Borevitz Lab
ARC Centre for Plant Energy Biology
Australian National University

2. In the next 100 years we will face challenges of
unprecedented scale and complexity
• Massive Biodiversity loss (since 1500):
• ~30% decline in land animals; ~70% of invertebrates (insects, etc.) show >45% declines4
• > 90% of the world’s fish stocks are fully exploited or overexploited 6
• > 7 billion people on the planet for at least the next century
• 11 billion people on the planet by 2100 1
• We must grow more food by 2100 than all the food produced in human history2
• Global climate will warm by 4-6°C by 2100
• No one under the age of 24 has lived in a year when the earth was cooler than any time in the last 2,000 years
• 53% of Eucalyptus species will be out of their native ranges 21003
• Sea levels will rise by 0.5 – 1.1m4
• By 2100, extreme sea level events that happened every few years now, are likely to occur every few days
1) Gerland, et al, 2014, [DOI:10.1126/science.1257469];
2) “Seeds of Doubt” New Yorker, Aug 25, 2014
3) Hughes et al. Global Ecology and Biogeography Letters (1996): 23-29.
4) “Climate Change Risks to Australia’s Coast”, 2009
5) Dirzo, Rodolfo, et al. 2014. DOI: 10.1126/science.1251817
6) http://worldoceanreview.com/en/wor-2/fisheries/state-of-fisheries-worldwide/
2/2
0

3. (re) Terraforming the earth
Terraforming:
“To alter the environment of (a celestial body) in order to make it
capable of supporting terrestrial life forms.”
• We are currently un-terraforming the earth at an exceptional
rate.
• To meet the challenges of the coming century we need to think
about how to restore and re-engineer the environment to
support >7 billion people for the next 100 years in the face of
climate change

4. The grand challenge of the 21st century
Sustain, manage and create ecosystems that
• optimize benefit to other living things
• provide sustainable ecosystem services for humans
• in the face of over-population, resource scarcity, climate
change and other major negative humans impacts
We no longer have the luxury of trusting low resolution
ecological models
4/2
0

5. The current resolution of field ecology is very limited
• Low spatial/time resolution data
• Limited sensors other than weather
• Sampling is manual; subjective
• Observations not-interoperable
• little or no data sharing
• often proprietary
• Repeat experiments to verify results are often
at different site by different observers
• Paradigm for sample resolution is “Forest” or
“field” not Tree or Plant
Very little data from the last century of ecology is available for reuse

6. Research focus:
How can we use new technology to capture what is going on in the
environment, quantify it and provide it to scientists (and the public) in a
format that they can use?
(1) Lab phenomics
• “Phenotype” (plant behaviors) + “Genomics” (plant genomes)
• Identifying the genetic basis of plant growth and development
(2) Field ecology
• High resolution monitoring of ecosystems
How do we continuously monitor thousands of plants at high
resolution and low cost?

7. Why Phenomics matters
• Maximize carrying capacity
• Improve crop yields and land use efficiency
• Maintain and restore ecosystem services
• Requires advanced environmental monitoring and high-
accuracy ecosystem models
• Maintain and restore natural areas and biodiversity
• Requires accounting for significant projected climate change
impacts
• Climate ready seed selections (yield stability)
• Climate ready forests
• Inform conservation and management decisions for how to
spend scarce conservation dollars (i.e. identify most at-risk
ecosystems)

8. Genotype x Environment = Phenotype
Phenotype (and ecosystem function) emerges as a cross-scale
interaction between genotypes and the environment they experience.
The degree to which we can measure all three components is the
degree to which we can understand ecosystem function

9. Borevitz Lab Approach
• Measure more precisely in the Field
• Create more “natural” Lab conditions
© Suzanne Marselis
enviro-net.org

10. Lab vs field phenotyping
Field: Realistic environment but low precision measurements
In the field we have real environments but the complexity (and bad lighting!) reduces
our ability to measure things with precision
youtu.be/gFnXXT1d_7s

11. Lab vs field phenotyping
Lab: High precision measurement and control but low realism
youtu.be/d3vUwCbpDk0

12. Lab phenotyping
Normal lab growth conditions aren’t very “natural”
Kulheim, Agren, and Jansson 2002
Growth Chamber
Real World

13. SpectralPhenoClimatron
Dynamic “semi-realistic” environmental & lighting conditions
• Plant growth cabinets with 8 & 10-band controllable LED’s
• Control light spectra and intensity, Temperature and Humidity at 1-min intervals
• JPG + RAW imaging every 10 minutes
Tools for the lab
L4A s20: 10-band
L4A s10: 8-band LED lights
Spectral response of Heliospectra LEDs

14. Climatron
• Grow plants in simulated regional and seasonal
conditions
• E.g. coastal vs inland | Late Spring or Early Fall
• Simulate full or short season, climate shifts,
drought conditions, etc.
• Expose “cryptic phenotypes”
• Repeat environmental conditions
• Between studies and collaborators
• Field sites
• Real-time monitoring, analysis and data
visualization
• Image analysis pipeline extracts growth data from
120,000 pot images a day
Corrected
Segmented
Original

15. High Throughput Phenotyping for the masses
• Off the shelf low-cost cameras (<$5K/chamber) controlled by
raspberry pi’s
• Open source analysis & bioinformatics pipelines
• Online project management, visualization and analysis tools
• Cloud-based image processing solutions (Partnering with NCI)
• “Develop once and everyone can use it”

16. NextGen Ecology – Where’s my PCR?
• Field ecology is like genetics before PCR and high throughput
sequencing
• Back in the ’80s & early 90s people would get a PhD just sequencing
a single gene.
• Humane Genome Project cost $2.7billion USD and took 13 years
• Now (20 years in) we buy sequencing machines on eBay and light
sequence whole genomes for <$50 a plant in just a few weeks

17. Genetics -> Genomics -> Phenomics
20 years of technical advances have turned genetics into genomics into
phenomics and yielded the ability to address fundamental and very
complex questions
State of the art phenomics is pretty high resolution
• Precision environmental controls
• 3D time-lapse models of every plant growing with each pixel in all
spectra mapped to the 3D “data cube” model of each plant
• Genome sequence for every plant
• Automated bioinformatics pipeline for trait extraction
But ecosystems are more complex than plants in growth chambers…
We need much higher resolution capacity in the field

18. The real world is way more complex than plants in the lab
Don’t we need equally complex datasets and models to understand real world
ecosystems?
• This hasn’t traditionally been possible
• The questions we ask are often defined more by what data we can get than by what
the best question would be
• What questions could we ask if we had a 10-year high-resolution dataset of our field
sites?
18/20

19. The challenge – Measure everything all the time
• Persistent 3D, time-series multi data-layer ecosystem
models tracking every tree
19/
20

20. • 30 years ago was ~1980 (everything was analog)
• The first space shuttle (1981) 1MB of RAM1 (your phone has 2000x this)
• 4G is the same bandwidth that MODIS uses to download a full satellite image of the
earth every day (MODIS: 6.1Mbps avg; Verizon 4G: 5-12Mbps 2,3)
• Even 20 years ago: no Web, WiFi, cell phones, Google anything
• Facebook is barely 10yrs old with 1.32 bn active monthly users (more people
than any country but China)
• The first iPhone was only released in 2007
New technology is changing everything
1.http://www.popsci.com/node/31716
2.http://modis.gsfc.nasa.gov/about/specifications.php
3.https://www.lte.vzw.com/AboutLTE/VerizonWirelessLTENetwork/tabid/6003/Default.aspx
1980: 60 min of music 2010: 9,000 min of music

21. New technology is changing everything
• 1.8 billion (mostly geolocated) images are uploaded to
social media every day1
• Consider: 75% of cars may be self-driving by 20402 –
continuously imaging, laser scanning and 3D modelling
their immediate environment: 6.2 billion miles3 of
roadside environments in US, imaged in 3D daily!
1.Meeker, 2014
2.http://www.ieee.org/about/news/2012/5september_2_2012.html
3.http://www.fhwa.dot.gov/policyinformation/travel_monitoring/13dectvt/figure1.cfm
21/
20

22. 40 Gadgets replaced by the smartphone
© wired.com

23. On demand super-computing in our pockets
Cloud computing backed by ubiquitous high speed
bandwidth changes everything

24. National Arboretum Phenomic & Environmental Sensor Array
ANU Major Equipment Grant, 2014
Collaboration with
Cris Brack, Albert Van Dijk (Fenner school) and Borevitz Lab
24/
20

25. “NextGen” monitoring
Within ten years we will have similar data to lab phenomics
• Automated time-series (weekly/daily) aerial (UAV) scans measure:
• RGB; Hyperspectral; Thermal; LiDAR
• Centimeter resolution 3D model of every tree on field site
• Gigapixel imaging to track phenology in every tree/plant
• Automated computer visions analysis for change detection
• LIDAR scanning
• DWEL / Zebedee – high resolution 3D scans (ground based)
• Dense point clouds of 3D structure
• Microclimate sensor networks
• PAR
• Temp, Humidity
• Soil moisture @ multiple depths
• mm resolution dendrometers
• Full genome for every tree on site (<$5/tree)
Environment
Phenotype
Genetics
25/
20

26. National Arboretum Phenomic & Environmental Sensor Array
• Ideal location
• 5km from ANU (64 Mbps wifi)
• Many institutions close by
• ANU, CSIRO, U. Canberra , etc
• Brand new forest to monitor from birth into the future
• Great site for testing “NextGen” monitoring systems prior to more
remote deployments
26/
20

27. National Arboretum Sensor Array
• 20-node Wireless mesh sensor network (10min sample interval)
• Temp, Humidity
• Sunlight (PAR)
• Soil Temp and moisture @ 20cm depth
• uM resolution denropmeters on 20 trees
• Two Gigapixel timelapse cameras:
• Leaf/growth phenology for > 1,000 trees
• Campbell weather stations (baseline data for verification)
• All data live online in realtime
• LIDAR: DWEL / Zebedee
• UAV overflights (bi-weekly/monthly)
• Georectified Google Earth/GIS image layers
• High resolution DEM
• 3D point cloud of site in time-series
Total Cost ~$200K 27/
20

28. 28/
20

29. Tools for the field
-
Golfer, 7km distant
Gigapixel Imaging

30. (Single 15MP image)
Area: ~7ha
Area: ~1m2
The Gigapan and
Gigavision systems
allow you to capture
hundreds or
thousands of
zoomed-in images in
a panorama.
Images are then
“Stitched” into a
seamless panorama.
The super-high
resolution of the final
panorama lets you
monitor huge
landscape areas in
great detail.
Gigapixel Imaging – How it works

31. Australian time-series monitoring
• Canberra, Telstra Tower
• National Arboretum
Gigapixel imaging lets us monitor seasonal change for huge areas

32. 20 gigapixel image of Canberra, Australia from the Black Mountain Telstra Tower
Zoom in to the National Arboretum

33. Midsummer
Zoom in to the Each forest at the Arboretum

34. Low cost sequencing let’s us genotype every individual tree and identify genetic loci that correlate
with observed phenotypic differences between trees.
We can do this for all trees at the arboretum within view of the camera.
Fall Color change shows differing rates of fall senescence in trees
Late fall

35. Gigavision
• Control Software (Chuong)
• Need to automate stitching
• Need automated change detection and phenotyping for
every tree (Much bigger project)
35/
20

36. UAV’s for monitoring
• $4-5K airframe (aeronavics or similar) w/ 20MP digital
camera (~1kg payload).
• $1,800 AUD – Pix4D Software (non-commercial license)
• 3D models of field site (cm resolutions are possible)
• Orthorectified image and map layers
• LAS / point cloud data
• Automated pipeline: Flight -> tree data
• Tree Height; Volume, foliage density
• GPS location
• DEM of site
36/
20

37. Other data from the site:
37/
20
NextGen Ground based LiDAR: Zebedee

38. Merge aerial and understory scans
38/
20

39. DWEL (CSIRO NextGen laser
• Multiband Lidar with full point returns
• ~30 millions point in a 50m2 area (vs 5-10 pts/m for
aerial)
39/
20

40. So what do we do with all this data?
The challenge is no longer to gather the data, the challenge is
how we do science with the data once we have it
• A sample is no longer a data point
• Example: Gigavision data
From Sample:
• Camera hardware: 1200 images (per hour)
• Automate stitching into panoramas (5,000 tiles images / pano /hr)
• Need to align time-series images to each other and to the real
world (despite hardware failures, camera upgrades)
• How to visualize a time-series of 22 million images/year?
• Computer vision analysis
• Automated feature detection and phenophase detections
To Data:
• “Phenophase” transition and growth data from 1,000 trees
• And then how do you even analyze that?!
40/
20

41. Virtual 3D Arboretum Project
• Goal:
• Explore new methods for visualizing time-series environmental
data
• Historic and real-time data layers integrated into persistent 3D
model of the national arboretum in the Unreal gaming engine
• Collaboration with
• CS/TechLauncher students
• Stuart Ramsden, ANU VISlab

42. Classified point cloud data to 3D model
• Image: Stuart Ramsden (w/Houdini); Data: Suzanne Marselis

43. Unreal Arboretum project

44. Unreal Arboretum project
• Time slider for watching data change over time
• Temperature shown as dynamically colored particle cloud,
positioning deformed by prevailing wind data
• Tree color can be set to tree height and growth data for
instrumented trees
• All trees on site adjust to measured height, volume and
foliage density
• Ground color scales to match moisture
• User can fly to specific locations with higher density
sampling to see additional layers (gigapixel timelapse;
DWEL Lidar sites, etc)

45. Conclusions
• What we do in the next 50-100 years will have a critical
impact on the earth and its inhabitants for centuries to
come
• Scaling environmental monitoring to the prevision
required for meeting the challenges of the next century is
in large part an engineering and computer science
problem.
• Better optimizing transfer of engineering and CS research
from lab to real world applications can have a major
impact on how well we address these challenges.

46. Thanks and Contacts
Justin Borevitz – Lab Leader Lab web page: http://borevitzlab.anu.edu.au
• Funding:
• Arboretum ANU Major Equipment Grant
• ARC: Center of Excellence in Planet Energy Biology | Linkage 20014
• Arboretum
• http://bit.ly/PESA2014
• Cris Brack, Albert VanDijk, Justin Borevitz (PESA Project PI’s)
• UAV data: Darrell Burkey, ProUAV
• 3D site modelling:
• Pix4D.com / Zac Hatfield Dodds / ANUVR team
• Dendrometers & site infrastructure
• Darius Culvenor: Environmental Sensing Systems
• Mesh sensors: EnviroStatus, Alberta, CA
• TraitCapture:
• Chuong Nguyen; Joel Granados; Kevin Murray; Gareth Dunstone; Jiri Fajkus
• Pip Wilson; Keng Rugrat; Borevitz Lab
• Contact me: tim.brown@anu.edu.au / ​http://bit.ly/Tim_ANU
46/
20