Fme 2019
•Download as PPTX, PDF•
0 likes•121 views
À travers plusieurs scénarios, découvrez l’univers FME 2019 et créez vos propres flux de travail. Guidés par nos experts, vous débloquerez de nouvelles fonctionnalités dans FME Desktop et FME Server.
Recommended
Recommended
More Related Content
Similar to Fme 2019
Similar to Fme 2019 (20)
More from Consortech
More from Consortech (20)
Recently uploaded
Recently uploaded (20)
Fme 2019
- 1. Level 2: FME 2019 Scenarios FME Desktop
- 2. 1 2 Explore FME 2019 and help the FME Lizard FME 2019 Highlights AGENDA Help the FME Lizard find a new office! ● Mission 1: ○ Find how many FME Users are in each city ● Mission 2: ○ Find which city is warm and dry ● Mission 3: ○ Find which city has the most vegetation ● Mission 4: ○ Find Zipster ● Choose the office Demo ● FME Highlights
- 3. The FME Lizard is on the Move! But where should its next office be? FME to the Rescue! Let’s build a workspace to help the FME Lizard find a new office. (Caveat: the FME Lizard is picky and has very specific needs.)
- 4. FME Lizard’s New Office: Location Selection Criteria 1. FME Lizard likes FME users and wants them nearby. 2. Warm and dry. 3. Fresh supply of vegetation. 4. FME Zipster sightings.
- 5. May the best city win ● Generate index ranking all FME World Tour cities. ● You can game the system by assigning a weight to the indexes. ● Vote at fme.ly/wtvote now!
- 6. Warm and dry Zipster sightings FME users Vegetation Weight* Weight* Weight* Weight* Index 0-1 for each World Tour city Weight each index 0-100%, must add up to 100% Weighted= Weighted= Weighted= Weighted= + + + = Final index WT City Ranking Change Vancouver 1 +3 Bonn 2 +1 ... ... ... You vote on these! Try to improve your city’s ranking by increasing the weight for criteria that fit your city.
- 7. Mission 1 How many FME Users are in each city?
- 10. Mission 2 Which city is warm and dry? Fick, S. E., & Hijmans, R. J. (2017). WorldClim 2: new 1‐km spatial resolution climate surfaces for global land areas. International Journal of Climatology, 37(12), 4302-4315.
- 12. Precipitation raster Temperature raster Warm and dry index
- 13. Mission 3 What city has lots of vegetation?
- 16. Mission 4 Where is Zipster?
- 20. ● Integrated multiple sources ● Transformation tips and tricks: ○ Feature Caching and Partial Runs ○ Visual Preview ○ Bookmarks and Annotations ○ Database and Web Connections ○ FeatureReader and FeatureWriter ● Displayed data in a friendly way Demo Recap
- 22. Performance “EVERY FME RELEASE WILL BE FASTER THAN THE PREVIOUS OR WE WON’T SHIP IT!” Dale Lutz / Don Murray 2018+
- 24. Dark Mode
- 25. Visual Preview
- 26. Animate Mode
- 29. Auto-sized Transformers & Feature Types
- 31. ChangeDetector
- 32. Decompression
- 33. DGN Mesh Reading
- 34. New Formats • TopoJSON • Socrata (New API) • Digital Weather Markup Language (DWML) Reader • OGC WCS Reader • Garmin POI Binary • FME AR Reader
- 35. New Connectors • AzureBlobStorageConnector • AzureFileStorageConnector • CesiumIonConnector • CityworksConnector • GoogleCloudStorageConnector • S3Connector • AzureQueueStorageConnector (coming soon) • KafkaConnector (coming soon) • TrelloConnector (coming soon)
- 36. FME Packages Delivering transformers, reader/writers, and connections to you faster. ● Zipped .fpkg files available via FME Hub. ● Get fixes and new features into your hands within hours, not months. ● Developers can contribute their own content to FME. ● Currently released as FME Packages: ○ S3Connector ○ CesiumIonConnector ○ RasterObjectDetector series ○ NLP series
- 38. Questions? Now let’s take a look at FME Server...
- 39. Niveau 2 : FME 2019 Scénarios FME Server
- 40. To advance we must solve 5 challenges
- 41. PRESENTATION AGENDA 1 2 3 4 Challenge 2: Data Watching Challenge 3: Nightly Updates Challenge 4: Workspace Orchestration Challenge 5: New User Notification5 Bonus Challenge: Easy User Creation6 Challenge 1: FME Server Sharing 7 FME Server 2019.0 New Features
- 43. Challenge 1 “We want to share FME workflows with people who don’t have an FME Server account.”
- 44. FME Server Apps
- 45. Challenge 1 - Demo FME Server Apps
- 46. Challenge 2 “We want to watch for data and run an FME task whenever new data arrives.”
- 47. AutomationsA simple concept consisting of Triggers and Actions
- 50. When Trigger Occurs then Perform Action Every Automation has the same pattern:
- 52. Watching for Data Trigger when data arrives. Amazon S3 Watch Triggers when data arrives in location in S3 Bucket Directory Watch Triggers when data arrives in File Directory. FTP Watch Triggers when data arrives at location on ftp site Dropbox Watch Triggers when data arrives in Dropbox Directory. Azure Event Grid Can Trigger when data arrives in Azure Storage
- 53. When data arrives then Run workspace. Trigger Occurs Data Arrival Workflow Perform Action
- 54. Demo: Directory Watch Automation
- 55. Challenge 3 “We want to automatically update a database overnight. ”
- 56. Scheduling The most popular automation
- 57. When Scheduled time arrives then Run workspace. Trigger Occurs Scheduled Automation Perform Action
- 58. Schedule the workspace with Automations Add a Schedule as a Trigger Save and start your Automation Add a Workspace as an Action
- 59. Demo: Update Database on Schedule
- 60. Challenge 4 “We want to orchestrate complex workflows consisting of many FME workspaces.”
- 61. When scheduled time arrives then perform complex set of actions Trigger Occurs Workflow Orchestration Perform Action
- 66. Automations: Workflow Orchestration Wait Until All Paths Complete
- 69. Challenge 5 “As an Administrator, we want to know when a new user is created on our system.”
- 70. System Events Generated by FME Server activities
- 71. System Events FME Server events made available. Security Account Activity: User create, update, delete System Info Warnings and errors License expiry Server Status Server startup Backup, restore Server Activity Repositories and Items: Create, update, delete Project export/import
- 72. When new account is created then send email to Administrator. Trigger Occurs User Creation Notification Perform Action
- 73. System Events & Automations Bring power to System Administration
- 74. Demo: Administrator Account Notification
- 75. Bonus Challenge “We want anyone to be able create an account on our FME Server so that: 1. They can use FME Server. 2. We see new accounts created.”
- 76. Run FME Server from other applications Add a Topic as a Trigger Save and enable your Automation Add a Workspace Action
- 77. Run FME Server from other applications Use Direct URL Application runs workspace directly Add correct parameter values
- 78. FME Server 2019.0 New features we showed FME Server Apps Share FME workflows with anyone – no FME Server account required. Automations Build multi-step workflows to connect applications with triggers and actions. System Events Monitor FME Server events generated based on system activities.
- 79. Job Log Filtering Filter out warnings, errors, and info. Token Management Share exactly what you want with support for multiple tokens. FME Server Projects on FME Hub Load and share projects on FME Hub. Password Security Rules Specify password complexity. User Preferences Experience the same user preferences from browser to browser. FME Server 2019.0 More new and improved
- 80. Missions Complete 1 2 3 4 Challenge 2: Data Watching Challenge 3: Nightly Updates Challenge 4: New User Creation Notification Challenge 5: Workspace Orchestration5 Bonus Challenge: User Creation6 Challenge 1: FME Server Sharing 7 FME Server 2019.0 New Features
- 81. Questions?
Editor's Notes
- Predemo should take ~10 mins Demo should take ~10 mins (probably easiest to lengthen) New feature highlights should take ~10 mins (probably easiest to shorten)
- Amazon HQ search FME McDonald’s China Site selection
- Amazon HQ search FME McDonald’s China Site selection (Be clear that Safe Software is not opening any new offices :P Just an example )
- Redo as “challenges” Frame as challenge and solution. Levels, checkpoints? Quests and rewards Where are our customers?
- Make sure you explain the game here: we will use four datasets to make four indexes. Then we’ll make a final index to rank the cities. You can game the system to try to get your city higher in the rankings by adjusting how each index is weighted. Think your city will score well on an index? Give it more weight. Think it will do poorly? Give it less weight. At the end we’ll show you if you improved your city’s ranking or not! Good luck!
- The workspace was generated by grabbing our user information from salesforce by address and then geocoding it. . We used the ArcGIS Online Geocoder and produced points for the majority of our users. Then we buffered each city for the WT and spatially filtered the ones within our target cities. A bit of Stats-calculating and normalizing later and we have our users/city! FYI to normalize the data to a 0-1 scale you do the following: PercentageOfFMErs-_min/_max-_min
- Show in DI: actual % of users, use Filter in DI to show it Number shown here is 0.17% of Vancouverites are Safe Software customers
- http://docs.safe.com/fme/2017.1/html/FME_Desktop_Documentation/FME_Workbench/Workbench/Usage-Statistics-Tracking.htm The FME lizard loves a warm and dry environment. For this data, we used WorldClim Version 2 dataset to get average annual temperature and precipitation for all the world tour cities.
- We read in raster files of average precipitation and temperature from the WorldClim 2 project at a 1 km resolution. However, the WorldClim data has a separate raster for each monthly average. So, we used some raster transformers, including RasterBandCombiner, RasterStatisticsCalculator (new for 2018!), and RasterExpression evaluator, to calculate the average annual value. Next, we used a PointOnRasterValueExtractor to get the values for each World Tour city, which we read in as points. Then we normalized the data to create two indexes: for temperature, the hottest cities received a 1; for precipitation, the driest cities received a 1. Then we averaged the two indexes to arrive at a single (rather arbitrary) measurement of lizard habitat suitability!
- We started with 24 global 1km resolution rasters, and ended up with annual mean precipitation and temperature values for all World Tour cities, normalized to a scale of 0-1. We wrote this out to a simple CSV file.
- Image is Vancouver FYI Satellite imagery - landsat. What is landsat? An ongoing collection of satellite imagery of all land on Earth produced by the Landsat 8 satellite. Project was join effort from USGS + NASA, now all imagery hosted on AWS - and FME has a dedicated reader to access the API.
- Workspace details: Inputs: City Boundaries taken from Open Street Map (GeoJSON) Flow: Split into northern and southern hemispheres Query landsat data for high quality cloud free red and near infrared bands. Reproject and clip to input boundaries. If more than one image exists for a boundary, mosaick into one. Perform NDVI (divide NIR - Red by NIR + Red). Custom transformer on the FME Hub made by Dmitri. Mask out water features. Calculate statistics on each image and on all the images for normalization. Join to Cities dataset. Output: Cities dataset with green index (CSV)
- NDVI raster image for Vancouver. We created one of these for each WT city and analyzed them statistically to create our green index. Doesn’t look very green but trust us. This is just a greyscale where light pixels mean lots of vegetation and dark pixels mean less vegetation.
- Two workspaces: First workspace: really simple. Really easy to get the data using the HTML table reader (source is the URL to the crowdsourced Alien sightings webiste: http://www.nuforc.org/webreports.html. (top image) Second workspace: Data was all user-collected, so super unclean and disorganized. Workspace needed 4 StringReplacers and 5 StringSearchers (among other attr. mgmnt. xformers) to whip the data into shape. (bottom image for a laugh?) Stay tuned for more on data validation later in the day. Can plug Data Validation talk later in the day when discussing StringSearchers.
- Output: Green Index ranking the amount of UFO sightings (reported to our source) since the year 2000. (Icon is person taking a shower - supposed to represent someone being “beamed up”.)
- Now it’s time to integrate our data! Before demo: Have Run with Prompt and Feature Caching on Demo: Open workspace Show turning on Dark Mode Run readers and show them in Visual Preview while you review what they measure: Geopackage, CSV, and PostGres formats, reading in data from four sources we just spoke about. One is missing (red question mark)! Sam must have forgotten to add it! Add an Excel reader, choosing Data\Lizard_Habitat.xlsx as the data. Mission 1 Add a FeatureJoiner. Connect the Geopackage and Excel readers. Set City as the key for both datasets. Duplicate the FeatureJoiner twice, using the same key to join the other two readers. Use partial runs and Visual Preview to check FeatureJoiner progress after each one to show join working. Mention that InlineQuerier could be used to do this in one if you want to write SQL Add AttributeCreator: creating the weights for each index. Here is where we make it interesting! We are going to let you try to game the system to get your city higher in the rankings! Here you can manually enter the weights for each dataset index. You can do manual audience participation to get the 4 numbers, or you can use the FME Server App hosted at fme.ly/wtvote-temp (should update to fme.ly/wtvote soon), and view the results on fme.ly/wtweights. These weights each need to be a value from 0-1 and all must add up to 1. -user_weight -habitat_weight -green_weight -alien_weight Add ExpressionEvaluator: use objects to build formula to create final index (source indexes * weights). Explain this final number will fall between 0-1 and will determine the ranking of the cities. @ceil(@Value(alien_weight)*@Value(Alien_Index)+@Value(green_weight)*@Value(Green Index)+@Value(user_weight)*@Value(PercentageOfFMErs)+@Value(habitat_weight)*@Value(Habitat Index),4) -Good time to demo presets, if you want to save this as a “Cheat Code” or “up, up, down, down, left, right, left, right, B, A” (the Konami code :-D) preset and load it instead of using the Cheat Code transformer. Add Sorter: sort data by ranking. Index, Numeric, descending Add Counter: create an attribute of the ranking, _count. Set Count Start to 1. Add AttributeRounder: round off attributes for displaying in a table: Green Index, Alien_Index, PercentageOfFMErs, Habitat Index, index Add HTMLReportGenerator: now let’s make an HTML document with the ranking table -Add Header, Ideal FME Lizard Habitat, H1 -Add Header, FME World Tour 2019, H2 (could combine if wanted) -Add Table: -_count = Ranking -City = City -Country = Country -index = Final Index -PercentageOfFMErs = FME Users Index -Habitat Index = Lizard Habitat Index -Green Index = Vegetation Index -Alien Index = Zipster Sightings Index -Add HTML writer, ranking.html -Workspace should run and produce a ranked table of World Tour cities -Optional enhancements to improve HTML output: -connect AttributeRounder to Generate HTML map (optional) bookmark to show Google Map of ranking (requires API key) -connect HTMLReportGenerator to Highlight city row in output table (optional) to highlight chosen World Tour city row in final table -connect Order HTML document (optional) bookmark to HTML writer and ensure nothing else is connected Random notes: The Background Map was disappearing in Visual Preview sometimes. You can get it back by turning it off and on again.
- Add presets
- This is a graph of the CRAZILY improved Dissolver performance. Many other transformers have performance improvements too. (Underlying ‘feature tables’ improvements among other things.) -Kevin: up to 32x faster The performance claim applies to 1993-2016, 2018+ ...not 2017 :(
- Font from https://onextrapixel.com/25-free-pixel-perfect-fonts-for-8-bit-designs/
- Visual Preview: Inspect data in a dockable pane instead of having to open the FME Data Inspector.
- Animate Mode: Inspect changing data by scrolling along the timeline at the bottom. To do: Format? Netcdf? Add why for this mode here.
- Save preset values in transformer parameter dialogs, so it’s easy to fill in next time.
- Major interface improvements to our most popular transformer. Less clickalicious
- There’s a setting in case you don’t like. Reader feature types will “right justify” to keep the ports lined up. Transformers are just as big as they need to be.
- Reader/Writers and Connectors included in this list.
- https://blog.safe.com/2019/01/change-detection-2019-fmeevangelist184/
- Below are some specific observations. Point #2 might be a tricky one to get across to an audience. (As a more general comment, I wonder if it would be wise to remove the list of file extensions from the slide, as there isn't a 1:1 mapping to actual suppoorted coompression formats.) 1. We also support bzip2 compression. (gzip and bzip2 are both file-level, so tarfiles compressed with either will be supported, although our UI doesn't recognise any single extension for bzip2-encoded tarballs like it does for gzip.) 2. .zipx is really just a zip container with a proprietary compression inside. We might want to put an asterisk on it, as we can't support all zip files. As per [FMEENGINE-56659], we support ZIPX, LZMA, XZ, PPMD and BZIP2 compression, but do NOT support Deflate64, JPEG or WavPack. 3. Only single-volume RAR files are currently supported. Support for multi-volume (.r00, .r01, etc.) will come sometime, but has not been requested. 4. Password-encrypted .zip files are not yet supported.
- NOte -- if doing Level 5 later, this can be brushed over ‘cause we will talk more then. Else... Worth mentioning that the old FME Exporter for Revit no longer needed. If you still have workflows with that reader, use Revit IFC export and that can be used as input.
- Left screenshot shows LOD changing as the view zooms in or out. Source data is ~250000 buildings in CityGML. Right screenshot is a textured terrain layer showing drainages in BC interior. Source data was DEM + Shape files.
- If doing level 5 - mostly whip by this ‘cause we’ll talk more then.
- If doing level 5, suggest you skip this slide
- If doing level 5, don’t say much...
- https://github.com/topojson/topojson/wiki -- http://www.somebits.com/~nelson/SotM-2013-TopoJSON.pdf worth a flip through FME AR is just a reader in case people wanted to see the writer. DWML is Digital Weather Markup Language -- a reader only. All the (R) ones are reading only
- Because these are packages, they will just show up in FME 2019 once they are available in FME HUb. BLog article coming
- Don as packages
- Done in collaboration with our good friends at Esri. Coming next year -- a Proj6 reprojector as well -- open source reference reprojection engine
- When it comes the FME Platform it is all about Connect, Transform and Automate. FME Server is about “AUTOMATE” and in 2019 this is taken to a whole new level! These challenges will show you how it is easier than ever to share the power of FME Server!!!
- On https://fmewt2019-safe-software.fmecloud.com create a shareable URL for the Ranking workspace, located in the Site Selection Scenario repository. Maybe a side trip to show folks the power of “favorites” in FME Server. The workspace we are going to share is the “Ranking” workspace from the Level 1. It should just work. Database connection and data files are all shared with the FME Author. There may be an updated workspace at a later date.
- On https://fmewt2019-safe-software.fmecloud.com create a shareable URL for the Ranking workspace, located in the Site Selection Scenario repository. It should just work. Database connection and data files are all shared with the FME Author. There may be an updated workspace at a later date.
- On https://fmewt2019-safe-software.fmecloud.com create a shareable URL for the Ranking workspace, located in the Site Selection Scenario repository. It should just work. Database connection and data files are all shared with the FME Author. There may be an updated workspace at a later date.
- On https://fmewt2019-safe-software.fmecloud.com create a shareable URL for the Ranking workspace, located in the Site Selection Scenario repository. It should just work. Database connection and data files are all shared with the FME Author. There may be an updated workspace at a later date.
- This is the biggest thing in FME Server 2019. This is where we take “Automate” to the next level!!!!
- These are many triggers available in automations. When a trigger fires an automation is invoked. For previous FME Server users anyone is familiar with the notification service, all of the publications are listed here and some extra ones.
- Two types of actions. Internal and External. Internal actions are all those things that are done that are within the server (run a worspace) or actions that are part of the automation process. An automation can have many actions all chained together.
- Create an automation that runs the Contours/ContoursReader.fmw workspace. You will need: 1x Directory Watch Trigger (Watch ContoursInput resource folder) 2x Filters (one filter .shp, one .zip) 1x Workspace (Contours/ContoursReader.fmw). Instructions: Add the Directory Watch and set it to watch the ContoursInput resource folder every 1 minute. Optional step: View the workspace in WorkspaceViewer to see what’s up. (We’re looking to read in shapefiles) Add a Filter tool, so the workspace only runs with shapefiles or zips (you don’t want .prj, .shx etc triggering the workspace or you’ll get job failures) You will need 2 Filters, one for .shp and one for .zip to filter on the file path key that comes from the Directory Watch. Make sure the two filters are connected to the DirectoryWatch before configuring and that they’re in parallel. Connect the success port from both filters to a Workspace action. Set it to run the ContoursReader workspace. Set the source dataset to the file path from the DirectoryWatch. Then add some contours files from the FMEData2018 > Elevation > Contours You can also check the automation log or triggered jobs page to show that it’s ran. Talk about how easy this is now. How simple it is to avoid errors.
- Open automations and create a Schedule (timing optional) that links to a Workspace (DatabaseUpdates/spatialite2postgis) That’s it. The workspace (you can inspect in workspace viewer) adds a timestamp to the data so you can prove it’s ran in PGAdmin (it’s our demo postgis on RDS) or you can check the logfile, all features are logged. You can also check the automation log or triggered jobs page to show that it’s ran.
- Open automations and create a Schedule (timing optional) that links to a Workspace (DatabaseUpdates/spatialite2postgis) That’s it. The workspace (you can inspect in workspace viewer) adds a timestamp to the data so you can prove it’s ran in PGAdmin (it’s our demo postgis on RDS) or you can check the logfile, all features are logged. You can also check the automation log or triggered jobs page to show that it’s ran.
- Don’t forget to show workspace viewer. The point of this demo is to show how easy it is to build an automation that controls the workflow of a number of workspaces. In the workspace repository called “Orchestration” are a few workspaces. They have names that specify their lenght of execution. What they do is not important. Simply connect workspaces and show how merge and sequencing of workspaces can easily be done. Use a schedule and then show how you can trigger a schedule at any time. Also show this on the schedule page.
- Don’t forget to show workspace viewer
- Don’t forget to show workspace viewer
- Don’t forget to show workspace viewer
- Don’t forget to show workspace viewer
- Don’t forget to show workspace viewer
- Does this mean everyone who wants to upload data needs access to FME Server? Go back to the Run Workspace page Open Advanced Create Shareable URL Give it permissions to only run that workspace - for now turn custom permissions off - enable data upload service (click on user icon > token management) (bug fixed in 194) When creating a token, it’s definitely best practice to give it a good name and description so you know what the tokens are doing. Paste the URL into a new browser tab/incognito window. Upload some more data and run
- Spend some time on the ‘Manage Automations’ page I saved the app password for email account as an annotation in the automation Does this mean everyone who wants to upload data needs access to FME Server? Go back to the Run Workspace page Open Advanced Create Shareable URL Give it permissions to only run that workspace - for now turn custom permissions off - enable data upload service (click on user icon > token management) (bug fixed in 194) When creating a token, it’s definitely best practice to give it a good name and description so you know what the tokens are doing. Paste the URL into a new browser tab/incognito window. Upload some more data and run
- http://fme.ly/fmeserveraccess. If you click on the lizard you go there. Magic.
- http://fme.ly/fmeserveraccess As the administrator you should get an email when the new users is created. The user that signed up for FME Server will get an email.