CONNECT. TRANSFORM. AUTOMATE.
Big Data Meets FME
Agenda
 What is Big Data
 Big Data Challenges
 FME and Big Data
 Big Data Technologies
 DynamoDB Workflow
 MarkLogic...
Big Data and Cloud
Big Data needs big resources
 Big data stores
 Big processing power
 Big bandwidth
Cloud technology ...
Big Data and FME
 Big Data is a new data
“classification” for FME.
 Big Data is no different than
other data to FME
 FM...
Big Data and FME Support
Amazon S3
 Limitless internet based
storage
Amazon RDS
 See blog article on Amazon RDS (PostGIS...
Big Data Challenges
 Loading Data
 Lacks Spatial Support
 Big Data Analysis
 Querying and Exporting Data
Why Demo FME with
MarkLogic and DynamoDB?
Different from other
databases supported by
FME
Demo #1 – Limitless Spatial Database
DynamoDB
 NoSQL SSD-based database service
 No limit on size of database
 Specify the needed performance
 Autoscale th...
Dynamodb Big Data Demo
Spatially locate and store
anything in DynamoDB!
Dynamodb Demo – Index Strategy
Generate GeoHash Index
for each feature and
Write to
GeoHashSpatialIndex
DynamoDB Demo –
Storing Vector, Raster, Lidar
Write small features
to DynamoDB
Write large features
to Amazon S3, link
to ...
DynamoDB Demo–
Storing Geocoded Images
Generate Geohash record
of picture location
Write Image to S3, link
to DynamoDB
DynamoDB Demo –
Spatially Locate and Store Any document or Web Resource
Generate Geohash
index
Write Document to
S3 and Li...
DynamoDB Demo –
Retrieve any stored document
Write URI Link to
DynamoDB
Generate Geohash
index
location
What is ?
 NoSQL database – XML optimized
 Powerful search and analysis
 Native spatial support
 XML based data model ...
FME and MarkLogic – A Natural Fit
 Convert data to XML/GML*
 Easily load XML into MarkLogic with FME
 Process and conve...
Demo #1a - Loading MarkLogic
Convert GIS / CAD
data to GML (XML)
Compose REST request
to PUT to MarkLogic
database
1.Convert GIS / CAD data into Valid GML
2.Generate Key Fields
3.Build insert message
4.Execute PUT REST call
MarkLogic acc...
Loading GIS to MarkLogic with FME
Demo #1b Exporting from MarkLogic
GET Query to find
URI’s for features
of interest
GET Query using URI’s to
get feature
XM...
Exporting XML from MarkLogic
1. Query database via GET request
2. Parse search result and compose GET feature request
3. E...
Exporting XML from MarkLogic
Search GET request to find URI based on query:
http://localhost:8003/v1/keyvalue?element=comm...
AIXM from MarkLogic via FMEServer
http://UHURA/fmedatastreaming/Demos/QueryMarkLogicDB.
fmw?Element=airportCode&Value=CYVR...
AIXM from MarkLogic via FMEServer
Summary
Big Data = big new opportunities
FME great for working with Big Data
Cloud model is a natural fit for Big Data
Thi...
Thank You!
 Questions?
 For more information:
 info@safe.com
 www.safe.com
Big Data Meets FME
Big Data Meets FME
Upcoming SlideShare
Loading in …5
×

Big Data Meets FME

1,806 views

Published on

See more FME World Tour 2014 presentations at www.safe.com/recap2014

Published in: Technology
  • Be the first to comment

Big Data Meets FME

  1. 1. CONNECT. TRANSFORM. AUTOMATE. Big Data Meets FME
  2. 2. Agenda  What is Big Data  Big Data Challenges  FME and Big Data  Big Data Technologies  DynamoDB Workflow  MarkLogic Workflow
  3. 3. Big Data and Cloud Big Data needs big resources  Big data stores  Big processing power  Big bandwidth Cloud technology gives you this for fraction of traditional cost!
  4. 4. Big Data and FME  Big Data is a new data “classification” for FME.  Big Data is no different than other data to FME  FME Cloud is a natural fit for data in the Cloud FME makes it easy to leverage the power of Big Data
  5. 5. Big Data and FME Support Amazon S3  Limitless internet based storage Amazon RDS  See blog article on Amazon RDS (PostGIS/SQLServer/Oracle) Amazon DynamoDB  NoSQL limitless database service Amazon RedShift  Petabyte scale database warehouse service. Google BigQuery  Superfast append only tables MarkLogic  Large XML based NoSQL database
  6. 6. Big Data Challenges  Loading Data  Lacks Spatial Support  Big Data Analysis  Querying and Exporting Data
  7. 7. Why Demo FME with MarkLogic and DynamoDB? Different from other databases supported by FME
  8. 8. Demo #1 – Limitless Spatial Database
  9. 9. DynamoDB  NoSQL SSD-based database service  No limit on size of database  Specify the needed performance  Autoscale thru Dynamic DynamoDB  Amazon EMR (Hadoop) integration
  10. 10. Dynamodb Big Data Demo Spatially locate and store anything in DynamoDB!
  11. 11. Dynamodb Demo – Index Strategy Generate GeoHash Index for each feature and Write to GeoHashSpatialIndex
  12. 12. DynamoDB Demo – Storing Vector, Raster, Lidar Write small features to DynamoDB Write large features to Amazon S3, link to DynamoDB
  13. 13. DynamoDB Demo– Storing Geocoded Images Generate Geohash record of picture location Write Image to S3, link to DynamoDB
  14. 14. DynamoDB Demo – Spatially Locate and Store Any document or Web Resource Generate Geohash index Write Document to S3 and Link to DynamoDB location
  15. 15. DynamoDB Demo – Retrieve any stored document Write URI Link to DynamoDB Generate Geohash index location
  16. 16. What is ?  NoSQL database – XML optimized  Powerful search and analysis  Native spatial support  XML based data model (GML, XML, etc.)  Deploy on Hadoop HDFS
  17. 17. FME and MarkLogic – A Natural Fit  Convert data to XML/GML*  Easily load XML into MarkLogic with FME  Process and convert XML results  FME 2014: New schema based GML Writer
  18. 18. Demo #1a - Loading MarkLogic Convert GIS / CAD data to GML (XML) Compose REST request to PUT to MarkLogic database
  19. 19. 1.Convert GIS / CAD data into Valid GML 2.Generate Key Fields 3.Build insert message 4.Execute PUT REST call MarkLogic accepts any valid XML – just PUT it! Loading GIS to MarkLogic
  20. 20. Loading GIS to MarkLogic with FME
  21. 21. Demo #1b Exporting from MarkLogic GET Query to find URI’s for features of interest GET Query using URI’s to get feature XML/GML, then Conversion to format of choice (CAD, GIS …) /WFS
  22. 22. Exporting XML from MarkLogic 1. Query database via GET request 2. Parse search result and compose GET feature request 3. Extract attributes and geometry from result 4. Validate and write XML Result
  23. 23. Exporting XML from MarkLogic Search GET request to find URI based on query: http://localhost:8003/v1/keyvalue?element=comment&value=AIXM.Chicago Document Retrieval GET request based on URI: http://localhost:8003/v1/documents?uri=/docs/myXML_653c46c3-fdfb-4837-ae1c- 49735dd29356.xml
  24. 24. AIXM from MarkLogic via FMEServer http://UHURA/fmedatastreaming/Demos/QueryMarkLogicDB. fmw?Element=airportCode&Value=CYVR /AIXM
  25. 25. AIXM from MarkLogic via FMEServer
  26. 26. Summary Big Data = big new opportunities FME great for working with Big Data Cloud model is a natural fit for Big Data This is just the beginning - more to come!
  27. 27. Thank You!  Questions?  For more information:  info@safe.com  www.safe.com

×