Turning	
  agro-­‐biodiversity	
  open	
  data	
  to	
  
real	
  services:	
  an	
  intro	
  to	
  technologies	
  
Gianni...
AIM	
  OF	
  THE	
  COURSE	
  
How	
  to	
  get	
  as	
  much	
  as	
  possible	
  from	
  data?	
  
Which	
  technologies	
  and	
  how	
  they	
  can	
...
Course	
  syllabus	
  
•  Intro	
  
•  Technologies	
  
•  Sharing	
  data	
  
•  Linking	
  data	
  
•  Architectural	
  ...
ARE	
  WE	
  READY	
  TO	
  BUILD	
  REAL	
  
SERVICES	
  USING	
  AGRO	
  AND	
  
BIODIVERSITY	
  OPEN	
  DATA?	
  
What	
  we	
  need	
  
data	
   technology	
  
and	
  of	
  course	
  a	
  good	
  idea	
  
As	
  for	
  the	
  data	
  
Thanks	
  Nikos	
  	
  
“Intro	
  to	
  green	
  data	
  ecosystems:	
  why	
  is	
  a	
  dat...
TECHNOLOGIES:	
  THERE	
  IS	
  REALLY	
  
GREAT	
  STAFF	
  OUT	
  THERE	
  
Infrastructure	
  
•  We	
  have	
  the	
  cloud	
  
•  Cloud	
  is	
  geJng	
  cheaper,	
  beLer	
  and	
  more	
  
devel...
Storage and Processing Monitoring/Management/Allocation layer
Virtualization of Infrastructure Layer
Virtual Machines
Virt...
What	
  does	
  this	
  mean?	
  
Quick	
  prototyping	
  
Get	
  your	
  running	
  instance	
  in	
  minutes	
  
REST	
  
•  AbstracFng	
  the	
  architectural	
  elements	
  
•  Close	
  to	
  the	
  way	
  we	
  are	
  working	
  in	...
REST	
  clients	
  
•  easy	
  to	
  be	
  	
  developed	
  with	
  any	
  language	
  
•  check	
  before	
  start	
  dev...
Data	
  exchange	
  formats	
  
•  XML,	
  RDF	
  or	
  JSON?	
  
•  They	
  all	
  work	
  but	
  we	
  always	
  have	
 ...
Sensors	
  
•  Are	
  everywhere	
  
•  Two	
  categories	
  
– The	
  one	
  that	
  we	
  hold	
  i.e.	
  from	
  smartp...
get	
  data	
  from	
  a	
  smartphone	
  
•  sensing	
  data	
  
– MoFon	
  sensor/accelerometer	
  
– Proximity	
  
– gy...
GeJng	
  real	
  Fme	
  weather	
  data	
  
GeJng	
  real	
  Fme	
  weather	
  data	
  
	
  
h:p://api.openweathermap.org/data/2.5/weather?q=London,uk	
  	
  
Big	
  data	
  technologies	
  
•  A	
  trend?	
  	
  
•  Some	
  of	
  them	
  
– NoSQL	
  dbs	
  e.g.	
  MongoDB,	
  Cou...
Natural	
  Language	
  Processing	
  
•  Dbpedia	
  Spotlight	
  
•  EnFty	
  recogniFon	
  (idenFfy	
  names	
  and	
  pl...
Image	
  processing	
  
APIs	
  for	
  mobile	
  apps	
   Open	
  Source	
  Libraries	
  	
  
how	
  image	
  processing	
  can	
  be	
  used?	
  
•  Similar	
  to	
  how	
  Google	
  Goggles	
  works	
  but	
  train...
Machine	
  Learning	
  
how	
  machine	
  learning	
  can	
  be	
  used?	
  
Provide	
  a	
  recommendaFon	
  service	
   Mine	
  and	
  Cluster	
...
JavaScript	
  
•  can	
  do	
  almost	
  everything	
  
•  even	
  image	
  processing	
  e.g.	
  PixasFc	
  for	
  image	...
AnalyFcs	
  
•  Track	
  the	
  users	
  and	
  events	
  of	
  your	
  applicaFon	
  
– ElasFc	
  Search,	
  MongoDB	
  
...
Follow	
  a	
  well	
  accepted	
  framework	
  
•  Model	
  View	
  Controller	
  (MVC)	
  
•  There	
  are	
  many	
  ni...
SHARING	
  DATA	
  
  	
   	
   	
  
	
   	
   	
   	
  	
  
Publisher
Date Catalog
Subject
ID
Author
Title
we actually share metadata
the	
  value	
  is	
  sFll	
  in	
  metadata	
  
•  Metadata	
  connect	
  us	
  to	
  the	
  real	
  data	
  	
  
– Searc...
AGGREGATING	
  METADATA	
  
…	
  the	
  power	
  is	
  at	
  the	
  back	
  end	
  
CulQvaQon	
   HarvesQng	
   Blossom	
  
Unorganized	
  content	
  ...
Why	
  to	
  aggregate?	
  
•  Enrich	
  metadata	
  
•  Provide	
  developer	
  friendly	
  APIs	
  that	
  can	
  be	
  ...
LINKING	
  DATA	
  
Why	
  we	
  sFll	
  have	
  data	
  silos?	
  
•  CompeFng	
  metadata	
  standards	
  (e.g.	
  DC,	
  IEEE	
  LOM)	
  
•...
What	
  is	
  Linked	
  Data?	
  
•  A	
  set	
  of	
  principles	
  and	
  technologies	
  for	
  the	
  Web	
  
of	
  Da...
Seman&c	
  Web	
  /	
  Linked	
  Data	
  Technologies	
  by	
  Mathieu	
  d'Aquin	
  on	
  Sep	
  11,	
  2013	
  
ApplicaF...
Mash	
  up	
  and	
  Linked	
  Data:	
  FAO	
  case	
  
How	
  it	
  works?	
  
•  Based	
  on	
  AGROVOC	
  
•  Using	
  AGROVOC	
  mappings	
  to	
  other	
  ontologies	
  e.g....
Why	
  to	
  rely	
  on	
  linked	
  data	
  
•  Easy	
  to	
  be	
  processed	
  e.g.	
  microdata,	
  rdf	
  
•  Get	
  ...
and	
  even	
  more	
  …	
  
•  Data	
  that	
  you	
  can	
  get	
  from	
  World	
  Bank	
  based	
  on	
  ISO3	
  
coun...
DECOUPLING	
  FRONT	
  END	
  FROM	
  
BACK	
  END	
  
how	
  it	
  works	
  
Template	
  customizaQon	
  
html,	
  css,	
  Ajax,	
  JS	
  
Data	
  mash	
  up	
  
Search	
  API	...
how	
  it	
  works	
  
Template	
  customizaQon	
  
html,	
  css,	
  Ajax,	
  JS	
  
widget	
  in	
  Facebook	
  page	
  
...
AUTHENTICATION	
  
Use	
  authenFcaFon	
  APIs	
  
NATIVE	
  OR	
  WEB	
  APP?	
  
NaFve	
  vs	
  Web	
  app	
  
•  Depends	
  on	
  the	
  business	
  case	
  
•  Responsive	
  web	
  app	
  are	
  cross	...
BUILDING	
  A	
  BASIC	
  APP	
  IN	
  HOURS	
  
THE	
  IDEA	
  
A	
  mobile	
  app	
  for	
  discovering	
  green	
  
learning	
  resources	
  
•  Similar	
  apps	
  
– OER	
  commons	
 ...
Which	
  data	
  
•  Learning	
  Resources	
  about	
  environment,	
  
agriculture	
  and	
  biodiversity	
  
•  API	
  e...
Recipe	
  
•  Mobile	
  app	
  SDK	
  e.g.	
  XCODE	
  
•  Mobile	
  simulator	
  
•  Some	
  reading	
  
•  Similar	
  ex...
Development	
  is	
  friendly	
  
Defining	
  the	
  workflow	
  is	
  a	
  game	
  
ConnecFng	
  to	
  the	
  data	
  
Use	
  NSJSONSerializaFon	
  Class	
  to	
  get	
  the	
  data	
  
and	
  finally	
  
SuggesFons	
  
•  focus	
  on	
  the	
  idea	
  and	
  business	
  value	
  
•  re-­‐use	
  the	
  workflow	
  of	
  succes...
 
	
  
thank	
  you!	
  
stoitsis@agroknow.gr	
  	
  
Upcoming SlideShare
Loading in...5
×

Intro to-technologies-Green-City-Hackathon-Athens

206

Published on

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
206
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Intro to-technologies-Green-City-Hackathon-Athens

  1. 1. Turning  agro-­‐biodiversity  open  data  to   real  services:  an  intro  to  technologies   Giannis  Stoitsis   Agro-­‐Know  Technologies    
  2. 2. AIM  OF  THE  COURSE  
  3. 3. How  to  get  as  much  as  possible  from  data?   Which  technologies  and  how  they  can  be  used?  
  4. 4. Course  syllabus   •  Intro   •  Technologies   •  Sharing  data   •  Linking  data   •  Architectural  aspects   •  Building  a  simple  case   •  Some  suggesFons  
  5. 5. ARE  WE  READY  TO  BUILD  REAL   SERVICES  USING  AGRO  AND   BIODIVERSITY  OPEN  DATA?  
  6. 6. What  we  need   data   technology   and  of  course  a  good  idea  
  7. 7. As  for  the  data   Thanks  Nikos     “Intro  to  green  data  ecosystems:  why  is  a  data-­‐ powered  tech  start  up  for  agriculture  &   biodiversity  going  to  be  profitable?”  
  8. 8. TECHNOLOGIES:  THERE  IS  REALLY   GREAT  STAFF  OUT  THERE  
  9. 9. Infrastructure   •  We  have  the  cloud   •  Cloud  is  geJng  cheaper,  beLer  and  more   development  friendly     •  OpFons   –  Amazon   –  Rackspace   –  Linode   –  MicrosoQ   –  Google   –  Okeanos   –  Other  Private  or  even  yours  (e.g.  OpenStack,  OpenNebula)  
  10. 10. Storage and Processing Monitoring/Management/Allocation layer Virtualization of Infrastructure Layer Virtual Machines Virtualization of Infrastructure LayerVirtualized Infrastractures Management Layer GUI tools and APIs Cloud provider A Cloud provider B Cloud provider B
  11. 11. What  does  this  mean?   Quick  prototyping   Get  your  running  instance  in  minutes  
  12. 12. REST   •  AbstracFng  the  architectural  elements   •  Close  to  the  way  we  are  working  in  the  web   •  Standard  hLp  methods   –  GET   –  PUT   –  POST   –  DELETE   •  easy  to  be  built  especially  with  MVC  frameworks   e.g.  Rails     •  easy  to  be  consumed  with  any  language     GET  hLp://www.example.com/User/1234   POST  hLp://www.example  .com/User/  ‘{“user”:”{“”id”:”55”,  “name”:”John”}’   DELETE  hLp://www.example.com/User/1234  
  13. 13. REST  clients   •  easy  to  be    developed  with  any  language   •  check  before  start  developing  
  14. 14. Data  exchange  formats   •  XML,  RDF  or  JSON?   •  They  all  work  but  we  always  have  preferences   •  Why  we  love  JSON?   – can  be  parsed  by  any  front  end  (web,  mobile)   – simple   – can  be  used  to  represent  data  models   – beLer  in  processing  e.g.  transformaFon   – used  widely  by  all  document  based  repositories   (e.g.  MongoDB,  elasFcsearch)  
  15. 15. Sensors   •  Are  everywhere   •  Two  categories   – The  one  that  we  hold  i.e.  from  smartphones  today   from  watch  tomorrow   – The  one  installed  at  sites  e.g.  fields   •  We  can  get  and  use  data  from  both  
  16. 16. get  data  from  a  smartphone   •  sensing  data   – MoFon  sensor/accelerometer   – Proximity   – gyroscope  (for  gaming)   •  GPS:    Core  LocaFon  Framework  to  get  the   coordinates  and  tracking  user’s  locaFon  
  17. 17. GeJng  real  Fme  weather  data  
  18. 18. GeJng  real  Fme  weather  data     h:p://api.openweathermap.org/data/2.5/weather?q=London,uk    
  19. 19. Big  data  technologies   •  A  trend?     •  Some  of  them   – NoSQL  dbs  e.g.  MongoDB,  CouchDB   – elasFcsearch   – MapReduce  framework  e.g.  run  processes  over  a   big  amount  of  data   – Hadoop   – Cassandra   – and  more  ….  
  20. 20. Natural  Language  Processing   •  Dbpedia  Spotlight   •  EnFty  recogniFon  (idenFfy  names  and  places  in  a  text)   •  DisambiguaFon  (apple  can  be  a  company,  a  fruit  or  a  river)  
  21. 21. Image  processing   APIs  for  mobile  apps   Open  Source  Libraries    
  22. 22. how  image  processing  can  be  used?   •  Similar  to  how  Google  Goggles  works  but  trained  for  your  data  
  23. 23. Machine  Learning  
  24. 24. how  machine  learning  can  be  used?   Provide  a  recommendaFon  service   Mine  and  Cluster  Data  
  25. 25. JavaScript   •  can  do  almost  everything   •  even  image  processing  e.g.  PixasFc  for  image  processing   •  powerful  libraries  e.g.  D3  for  real  Fme  data  visualizaFon  
  26. 26. AnalyFcs   •  Track  the  users  and  events  of  your  applicaFon   – ElasFc  Search,  MongoDB   – Google  AnalyFcs  
  27. 27. Follow  a  well  accepted  framework   •  Model  View  Controller  (MVC)   •  There  are  many  nice  MVC  based  frameworks   – Rails   – Spring   – Zend   – Yii   – Laravel   – and  more  …  
  28. 28. SHARING  DATA  
  29. 29.                   Publisher Date Catalog Subject ID Author Title we actually share metadata
  30. 30. the  value  is  sFll  in  metadata   •  Metadata  connect  us  to  the  real  data     – Search  metadata  records  not  raw  data   •  Metadata  gives  the  context   – themaFc  classificaFon   – target  users   •  Well  defined  standards  (e.g.  DC,  IEEE  LOM,   Darwin  Core,  ABCD)   •  RDF,  XML  and  JSON  binding  
  31. 31. AGGREGATING  METADATA  
  32. 32. …  the  power  is  at  the  back  end   CulQvaQon   HarvesQng   Blossom   Unorganized  content  in   local  and  remote  sites   Organized  and   structured  content  in   local  and  remote   databases   EducaQonal   Bibliographic   EducaQonal   Geographical   Bibliographic   Agricultural   Data   PlaVorm   Aggregate   data  from   diverse   sources   Work  with   different   type  of  data   Prepare   data  for   meaningful   services   IngesQon  Enrichment  TranslaQon  Publishing   Data  discovery   services   Widgets   Authoring   services   AnalyQcs     services  
  33. 33. Why  to  aggregate?   •  Enrich  metadata   •  Provide  developer  friendly  APIs  that  can  be   used  to  build  good  services   •  Data  providers  rarely  have  APIs    
  34. 34. LINKING  DATA  
  35. 35. Why  we  sFll  have  data  silos?   •  CompeFng  metadata  standards  (e.g.  DC,  IEEE  LOM)   •  Diversity  of  web  interfaces  (e.g.  REST,  OAI-­‐PMH,  SOAP,  SPI,  SQI)   •  Different  exchange  format  (e.g.  XML,  RDF,  JSON)   •  Fragmented  use  of  texonomies   LD for educational data/resource sharing Overview Approaches for LD in educational data sharing  On the-fly/automated integration of heterogeneous APIs and data (http://www.meducator.net)  Dataset (transformation and) cataloging (http://linkedup-project.eu) ?  We  are  sFll  here  …   …  and  not  here  …  
  36. 36. What  is  Linked  Data?   •  A  set  of  principles  and  technologies  for  the  Web   of  Data   •  Principles   –  Put  the  data  online  with  permanent  address  (URIs)   –  Describe  the  data  with  a  standard  representaFon  (RDF)   –  Link  to  other  data  through  published  taxonomies   •  Technologies   –  Triple  stores   –  SPARQL   –  RDF,  OWL     –  SWRL  
  37. 37. Seman&c  Web  /  Linked  Data  Technologies  by  Mathieu  d'Aquin  on  Sep  11,  2013   ApplicaFons  
  38. 38. Mash  up  and  Linked  Data:  FAO  case  
  39. 39. How  it  works?   •  Based  on  AGROVOC   •  Using  AGROVOC  mappings  to  other  ontologies  e.g.  GeopoliFcal  and  DBpedia   <hLp://aims.fao.org/aos/agrovoc/c_690>  <hLp://www.w3.org/2004/02/skos/core#closeMatch>  <hLp://dbpedia.org/resource/Salmon>     <hLp://dbpedia.org/page/Salmon>     replace  resource  with  page  and  get  rdf  
  40. 40. Why  to  rely  on  linked  data   •  Easy  to  be  processed  e.g.  microdata,  rdf   •  Get  more  out  of  data  through  linking  e.g.  get  the  staFsFcs  for  a  country   through  GeoPoliFcal  Ontology   Web  resource  about  Greece  annotated  with  Agrovoc  term  “Greece”   hLp://www.fao.org/countryprofiles/geoinfo/geopoliFcal/data/Greece   Country  GDP  
  41. 41. and  even  more  …   •  Data  that  you  can  get  from  World  Bank  based  on  ISO3   country  code   –  Agricultural  irrigated  land  (%  of  total  agricultural  land)   –  Cereal  yield  (kg  per  hectare)   –  Rural  populaFon  (%  of  total  populaFon)   –  Poverty  headcount  raFo  at  rural  poverty  line  (%  of  rural   populaFon)   –  Agricultural  machinery,  tractors  per  100  sq.  km  of  arable   land   –  Arable  land  (%  of  land  area)   –  CO2  emissions  (kt)   –  Fish  species,  threatened  
  42. 42. DECOUPLING  FRONT  END  FROM   BACK  END  
  43. 43. how  it  works   Template  customizaQon   html,  css,  Ajax,  JS   Data  mash  up   Search  API   Cloud   Edu  resources   Data  processing  and  enrichment   e.g.  semanQc  enrichment  or  clustering   Search  API   Cultural  resources   from  Museums  
  44. 44. how  it  works   Template  customizaQon   html,  css,  Ajax,  JS   widget  in  Facebook  page   Data  mash  up   Search  API   Cloud   Edu  resources   Data  processing  and  enrichment   e.g.  semanQc  enrichment  or  clustering   Search  API   Cultural  resources   from  Museums  
  45. 45. AUTHENTICATION  
  46. 46. Use  authenFcaFon  APIs  
  47. 47. NATIVE  OR  WEB  APP?  
  48. 48. NaFve  vs  Web  app   •  Depends  on  the  business  case   •  Responsive  web  app  are  cross  plaworm   •  Responsive  design  is  great  but  it  is  not  so   friendly  as  naFve  app   •  NaFve  app  has  nice  workflow   •  NaFve  app  can  use  device  sensors   •  In  many  data-­‐powered  cases  you  need  both  
  49. 49. BUILDING  A  BASIC  APP  IN  HOURS  
  50. 50. THE  IDEA  
  51. 51. A  mobile  app  for  discovering  green   learning  resources   •  Similar  apps   – OER  commons   – iMarine   •  Similar  web  app   – hLp://www.greenlearningnetwork.org/ organicedunet/  
  52. 52. Which  data   •  Learning  Resources  about  environment,   agriculture  and  biodiversity   •  API  end  point  hLp://83.212.96.219:8080/ glnRepo/api/ariadne/restp?json=Χ&callback=Y  
  53. 53. Recipe   •  Mobile  app  SDK  e.g.  XCODE   •  Mobile  simulator   •  Some  reading   •  Similar  examples  e.g.  Searching  in  twiLer   •  A  workflow   •  JSON  parser   •  No  beauFes  at  this  stage  
  54. 54. Development  is  friendly  
  55. 55. Defining  the  workflow  is  a  game  
  56. 56. ConnecFng  to  the  data   Use  NSJSONSerializaFon  Class  to  get  the  data  
  57. 57. and  finally  
  58. 58. SuggesFons   •  focus  on  the  idea  and  business  value   •  re-­‐use  the  workflow  of  successful  apps   •  processing  gives  the  value   •  do  not  create  unnecessary  procedures  e.g.  RegistraFon  page   •  re-­‐use  exisFng  data   •  put  emphasis  on  the  value  of  your  app  and  not  on  the   complexity  of  the  technology   •  always  check  first  if  there  is  open  source  code   •  design  maLers   •  go  beyond  trivial  and  benefit  from  innovaFon  
  59. 59.     thank  you!   stoitsis@agroknow.gr    
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×