The document provides an overview of a webinar about leveraging FME technology for big data management within cloud infrastructure. It discusses big data challenges, technologies like Amazon DynamoDB and MarkLogic, and includes demonstrations on loading and exporting data. The session emphasizes the benefits of cloud solutions for big data and highlights upcoming webinars and events.

1. Big Data
Tap into Cloud Infrastructure with FME
March 18, 2014

2. Meet the presenters.
Don Murray
 President and Co-Founder
@DonAtSafe
Dean Hintz
 Senior Product Specialist
@DeanAtSafe

3. Ask us. And join the discussion.
Please submit using the
GoToWebinar panel.
We will follow-up with
unanswered questions.

4. Agenda.
 What is Big Data
 Big Data Challenges
 FME and Big Data
 FME Demos:
 Loading and Extracting from
MarkLogic
 Spatial Indexing and Loading to
DynamoDB

6. What we do.
www.safe.com

7. Poll: Which version of FME
are you using?

8. New to FME?
 Get your bearings from our Getting
Started Page:
www.safe.com/fme/getting-started
 Learn from our crew in one of the
weekly FME Overview webinars:
safe.com/WeeklyIntro

9. What is Big Data?

10. Big Data and Cloud
Big Data needs big resources
 Big datastores
 Big processing power
 Big bandwidth
Cloud technology gives you this for fraction of
traditional cost!

11. Big Data and FME
 Big Data is a new data
“classification” for FME.
 Big Data is no different than
other data to FME
 FME Cloud is a natural fit for
data in the Cloud
FME makes it easy to leverage the power of Big Data.

12. Big Data and FME Support
Amazon S3
 Limitless internet based
storage
Amazon RDS
 See blog article on Amazon RDS (PostGIS)
Amazon DynamoDB
 NoSQL limitless database service
Amazon RedShift
 Petabyte scale database warehouse service.
Google BigQuery
 Superfast append only tables
MarkLogic
 Large XML based database

13. Poll: How are you currently
working with Big Data?

14. Big Data Challenges
 Loading Data
 Lacks spatial support
 Big Data Analysis
 Querying and Exporting
Data

15. Demo #1
 MarkLogic
Demo #2
 Limitless Spatial
Database

16. Why Demo FME with
MarkLogic and DynamoDB?
Different from other
databases supported by
FME.

17. What is ?
 NoSQL database – XML optimized
 Powerful search and analysis
 Native Spatial Support
 XML based data model (GML, XML, etc.)
 Deploy on Hadoop HDFS

18. FME and MarkLogic – A Natural Fit
 Convert data to XML/GML*
 Easily Load XML into MarkLogic with FME
 Process and convert XML results
 FME 2014: New schema based GML Writer

19. Demo #1a Loading MarkLogic
Convert GIS / CAD
data to GML (XML)
Compose REST request
to PUT to MarkLogic
database

20. 1. Convert GIS / CAD data into Valid GML
2.Generate Key Fields
3. Build insert message
4. Execute PUT REST call
MarkLogic accepts any valid XML – just PUT it!
Loading GIS to MarkLogic

21. Loading GIS to MarkLogic with FME

22. What ​Big Data technology
are you most interested in?

23. Demo #1b Exporting from MarkLogic
GET Query to find
URI’s for features
of interest
GET Query using URI’s to
get feature XML/GML,
then
Conversion to format of
choice (CAD, GIS …)
/WFS

24. Exporting XML from MarkLogic
1. Query database via GET request
2. Parse search result and compose GET feature request
3. Extract attributes and geometry from result
4. Validate and Write XML Result

25. Exporting XML from MarkLogic
Search GET request:
http://localhost:8003/v1/keyvalue?element=comment&value=AIXM.Chicago
Retrieval GET request:
http://localhost:8003/v1/documents?uri=/docs/myXML_653c46c3-fdfb-4837-ae1c-
49735dd29356.xml

26. AIXM from MarkLogic via FMEServer
http://UHURA/fmedatastreaming/Demos/QueryMarkLogicDB.fmw
?Element=airportCode&Value=CYVR
/AIXM

27. AIXM from MarkLogic via FMEServer

28. MarkLogic -> Anything
(JSON, KML, GML …)

29. MarkLogic to ArcGIS via FME Server:
1. Submit search to MarkLogic as described earlier
2. Extract attributes and geometry from result
3. Generate update ESRIJSON message from feature
4. Post update ESRIJSON to ArcGIS Server
MarkLogic / ArcGIS Integration

30. ArcGIS Server to MarkLogic
via FME Server
1. Retrive JSON data from ArcGIS Server
2. Generate output GML
3. Write data to MarkLogic via PUT REST call

31. ArcGIS Server to MarkLogic

32. Demo #2 – Limitless Spatial Database

33. DynamoDB
 NoSQL SSD-based database service
 No limit on size of Database
 Specify the needed performance
 Autoscale thru Dynamic DynamoDB
 Amazon EMR (Hadoop) integration

34. Demo # 2 – Index Strategy
Generate GeoHash Index
for each feature and
Write to
GeoHashSpatialIndex

35. Demo #2a – Vector, Raster, Lidar
Write small features
to DynamoDB
Write large features
to Amazon S3, link
to DynamoDB

36. Demo #2b – Geocoded Images
Generate Geohash record
of picture location
Write Image to S3, link
to DynamoDB

37. Demo #2c – Spatially Store Anything
Generate Geohash
index
Write Document to
S3 and Link to
DynamoDB
location

38. Demo #2d – Spatially Locate any
internet resource
Write URI Link to
DynamoDB
Generate Geohash
index
location

39. What data types are you
planning to store in Big Data?

40. Save the date.
Webinar: How to Automate Practically
Anything with FME Server (March 25th)
Webinar: How to Load Data into Google
Maps Engine (April 16th)
FME World Tour 2014 (April – June 2014)
FME International User Conference 2014
(20th Anniversary Celebration)
• June 10 – 13, 2014 in Vancouver, Canada

41. Free and fun to learn.
Online Courses - Live & Hands-On
 Feb 18-19: FME Desktop
Tutorials & Recorded Courses

42. Stay informed.
fmepedia.com/community
fmepedia.com/knowledge
@SafeSoftware
youtube.com/FMEChannel
blog.safe.com

43. Summary
Big Data = big new opportunities
FME great for working with Big Data
Cloud model is a natural fit for Big Data
This is just the beginning - more to come!

44. Hand raising has now
been enabled.
 If you’d like to ask a
question over the
air, please click the
hand icon and
ensure your audio
input is set up.

45. Thank you!
Sales
 info@safe.com
Support
 www.safe.com/support
 (604) 501-9985 ext. 278
Don Murray
 Don.murray@safe.com
Dean Hintz
 dean@safe.com