11. Building a Team Photo credit: americanistadechiapas @ flickr
12. Building a Team Customer (Charlie Frye) Lead Developer (Alex Yule) Project Manager/Data Guru (Jim Herries) Cartographer/Designer (MamataAkella) Sysadmin (Kenny Ling) Friends for testing, UI advice, moral support (Mapping Center Team & Co.)
13. Conceptual Planning Audience: General Public (sports fans) Functionality: Vote submission Vote display Vote reporting Geolocation Look and feel: Dark, minimalist slate basemap with slick UI
14. SuperbowlFanMap: Conceptual Planning Audience: General Public (sports fans) Functionality: Vote submission Vote display Vote reporting Geolocation Look and feel: Dark, minimalist slate basemap with slick UI Platform: Flex User load: Low with slight potential for virality Deployment target: Single rack server
37. SuperbowlFanMap: Architectural Issues Client High Server Load Slow response time for app, maps High Client Load High number of features result in slow drawing, panning UI lockup during vote summarization Web/Map Server SDE GDB Basemap App Feature Service Voting Vote Symbolization Summarization
40. Conceptual Planning Audience: General Public (sports fans) Functionality: Same as Superbowl app Configurability Look and feel: Dark, minimalist slate basemap with slick UI Existing codebase: SuperbowlFanMap Platform: Javascript User load: Moderate with potential for large viral spikes 10 concurrent users Deployment target: 4-tier Amazon cloud
42. Basketball FanMap: Architecture Web/Map Server Client SDE GDB Basemap App Feature Service SDE GDB Voting Vote Symbolization Summarization Web/Map Servers GP Server Maps App Summarization Voting Load Balancer
43.
44. Basketball FanMap: Production – Server-side! Vote Symbology: Dynamic map services – Must create a separate map service for each game – Increased server load + Can handle unlimited numbers of features Voting: Geoprocessing service – Harder to setup (custom python GP tool) + Simpler to deploy (ArcSDE not required) + Handles collisions automatically Summarization: Geoprocessing service – Harder to setup (custom python GP tool) + Handles unlimited features
49. Basketball FanMap Prototyping: Voting Start with the highest risk functionality (voting) Get that working, then move on Mock out steps for vote service
50. Basketball FanMap Prototyping: Voting Service SDE GDB Inputs User ID Insert new row with all vote data: User ID, DateTime, Location, Votes Insert Cursor (Votes) Location Votes where locationfield == Location for vote in Votes: votefield += 1 Update Cursor (Places) Places Votes
51. Basketball FanMap Prototyping: Voting Start with the highest risk functionality (voting) Get that working, then move on Mock out steps for vote service Test each step in ArcMap Put steps together into script Test in ArcMap as script tool Publish and test as GP service Integrate into App
52. Basketball FanMap Prototyping: Summarization Output Table Inputs Extent Select all features in Extent with at least some votes for these teams Select Features Fields (Teams) Stats Construct summary_stats String so we get SUM for each field Construct Stats String Summary Stats Get SUM of each field In Selected Features
54. Basketball FanMap: “Map Sandwich” How it works The ‘basemap’ is split into two map services: Reference (top) Base (bottom) Combined with thematic content Votes (middle)
69. FanMap – Deployment/Architecture Why use it? Amazon Machine Images (AMIs) No need for new hardware requests or software installation Start up new machines or backup Data portability+reliability Volumes Snapshots Deployment Higher configuration machines Machine replicas
70. FanMap – Deployment/Architecture Replicating your servers Updating services Load balancing Failsafe in case Amazon servers in one region goes down Monitoring your services Set up Performance Monitor logs Track performance in Task Manager
71. FanMap – Deployment/Architecture Things to be aware of: Instances are inaccessible whilst creating images Transferring your cache to the cloud may take a long time Option: Cache in the cloud Plan security settings/groups in advance Security groups cannot be changed once instance is started
73. FanMap – Deployment/Architecture GP Server Security Group 2 SDE GDB Web/Map Servers Summarization Voting Maps App Load Balancer Security Group 3 Security Group 1
75. FanMap PollMap: Birth of a Template What’s Included Application source code Geoprocessing source code Map service MXDs Data for postal codes in the U.S. and Canada URLs to basemaps you can use during development Simple documentation http://bit.ly/pollmaptemplate
78. Download the PollMap Template Download .ZIP file Unzip to a convenient folder Read the “Getting Started” document
79. Copy Application folder contents Open the Application folder Copy PollMap folder over to c:/inetpub/wwwroot (or equivalent)
80. Set up you Question First in VoteWindow.html Getting the question right takes iteration This template is best for multiple A vs. B types of questions
86. Thank You! Alex Yule - ayule@esri.com - twitter: @yuletide Jim Herries: jherries@esri.com - twitter: @jherries MamataAkella: makella@esri.com Kenny Ling: kling@esri.com Mapping Center Team http://mappingcenter.esri.com twitter: @mappingcenter
Editor's Notes
We want to point out that you don’t need to take lots of notes because we’ll be posting this presentation on our Mapping Center Web site on the Other Resources page. The bottom notes will have everything we are going to say.So, in this session, you can sit back and focus on the concepts and demos about mapping with ArcGIS. And maybe you will think of a few questions you want to ask us.We’ll show these URLs again at the end of this talk. But now, let’s get started.
Intro (5 minutes) { Jim }IntroCrowd surveyBackgroundHow the project(s) came aboutLong-term: Build a voting template for local govShort-term: Try to make something viral and crowd-sourced for the superbowlEssence of user experience: plant a flag in your backyard, put yourself and your allegiance on the mapWhat, why, how[Endgame was to create a template for building voting apps]
This project began as a hallway conversation about various maps we’d seen which represent public opinion. In many cases, the aggregation unit (e.g. states or counties) obscures the actual voting patterns that exist at larger scales, such as at the neighborhood level or even city level.
How the project(s) came aboutLong-term: Build a voting template for local governmentShort-term: Try to make something viral and crowd-sourced for the superbowlEssence of user experience: plant a flag in your backyard, put yourself and your allegiance on the mapHow can the cartography enhance the user experience (move beyond “dots on maps” approach)?
Jim passes things along to Alex
Key to a successful web project is a diverse team with distinct roles and responsibilities. http://www.flickr.com/photos/americanistadechiapas/5504689399/
Our team consisted of: a customer (Charlie) who went to the Project Manager (Jim) and asked for a web map that could take votes and show them on a map. Jim then found a developer (Alex). Jim knew he could handle all the data work, but needed someone to build the application itself. We then assembled a team of other people to handle the remaining pieces: maps and design (Mamata) and server deployments and maintenance (Kenny).
The whiteboard was instrumental during the requirements phase, helping us sort out what was possible vs. what was essential.
Here is the basic spec we came up with after our preliminary planning meeting. Most important here are the first two items: our audience and the core functionality.
NowMamata and I will continue the FanMap story with some more juicy details about the production process…
First up is Mamata, who will delve into the aesthetic side of the application
Mamata passes off to Alex
Demo
Everything on one serverVote Symbology: Client-side+ Great for dynamic, algorithmic cartography+ Easy to add interactivity- Slow to draw once you get a few thousand featuresSummarization: Client-sideSlow once you get a few thousand features, locks up UISymbology – Feature ServiceShow symbol renderer codeExplain advantages of using client-side renderingClient-side color ramps, manual classing, proportional symbolsSymbol scaling can depend on the current number of votesExample: if votes > 50 && votes < 75: symbol size = maxTotalVotes/4Or if maxTotalVotes > 1000: use renderer A otherwise user renderer BLimitations: 3-4k features for Flex APITODO: research or ask and see when the JS api maxes out?Working on this now with JS people…Reporting – Client-side via Feature ServiceAdvantages (simplicity, speed at low feature counts)Already have all the features in the client for symbolization, just loop through and calculate your statsDrawbacks (slowness at large feature counts, UI lockups)Looping through large feature sets locks the UI in most web clients (Flex in this case)
{Alex takes over here?}Now that we’ve talked a little bit about the basemap, let’s talk a little bit about the vote symbology. On a basic level, the colors represent each team that was in a specific match-up and the size of the symbol represents number of votes for a given team. BUT there is more to the story.
Symbology – Feature ServiceShow symbol renderer codeExplain advantages of using client-side renderingClient-side color ramps, manual classing, proportional symbolsSymbol scaling can depend on the current number of votesExample: if votes > 50 && votes < 75: symbol size = maxTotalVotes/4Or if maxTotalVotes > 1000: use renderer A otherwise user renderer BLimitations: 3-4k features for Flex APITODO: research or ask and see when the JS api maxes out?Working on this now with JS people…Reporting – Client-side via Feature ServiceAdvantages (simplicity, speed at low feature counts)Already have all the features in the client for symbolization, just loop through and calculate your statsDrawbacks (slowness at large feature counts, UI lockups)Looping through large feature sets locks the UI in most web clients (Flex in this case)
Everything on one serverVote Symbology: Client-side+ Great for dynamic, algorithmic cartography+ Easy to add interactivity- Slow to draw once you get a few thousand featuresSummarization: Client-sideSlow once you get a few thousand features, locks up UISymbology – Feature ServiceShow symbol renderer codeExplain advantages of using client-side renderingClient-side color ramps, manual classing, proportional symbolsSymbol scaling can depend on the current number of votesExample: if votes > 50 && votes < 75: symbol size = maxTotalVotes/4Or if maxTotalVotes > 1000: use renderer A otherwise user renderer BLimitations: 3-4k features for Flex APITODO: research or ask and see when the JS api maxes out?Working on this now with JS people…Reporting – Client-side via Feature ServiceAdvantages (simplicity, speed at low feature counts)Already have all the features in the client for symbolization, just loop through and calculate your statsDrawbacks (slowness at large feature counts, UI lockups)Looping through large feature sets locks the UI in most web clients (Flex in this case)
Incremental Voting: Client-side+ Uses the new FeatureLayer for easy-peazy feature editing- Vote collisions when using incremental voting- Requires ArcSDEWhy it didn’t work hereClient A loads the map, pulls current vote count: xClient B loads the map, pulls current vote count: xClient A and B both send votes for zip code z:votesLayer.applyEdits(votes=x+1)Vote count gets updated twice, but both times with the same value, and ends up at x+1 even though two votes were submitted!http://www.flickr.com/photos/viriyincy/2535350990/
Demo the app Summarization extent saving voting social media capabilities panes
The spec for our second FanMap is much the same as the Superbowl version with a few key differences:ConfigurabilityDifferent platform (JS)Higher expected user load and multi-tier architecture
Importance of architecture before deployment
As you’ll recall, theSuperbowlFanMap was based on a very simple, Client/Server architecture with lots of functionality delegated to the web app itself, for the NCAA FanMap we moved to a multi-tier architecture to isolate each major functional area
Starting at the top, we have one server dedicated entirely to serving and storing our data, which is stored in an ArcSDE-enabled MS SQL database.Next, we have two client-facing tiers:-one made up of two web/map servers covered by a load-balancer to serve up all the maps and the application itself, -and one GP server to handle the core application logic of voting and summarization, which are served as geoprocessing services
I am going to take a little time to talk about some of the specific decisions that went into designing the base map and symbology that is in the FanMap.
Now that we’ve talked a little bit about the basemap, let’s talk a little bit about the vote symbology. On a basic level, the colors represent each team that was in a specific match-up and the size of the symbol represents number of votes for a given team. BUT there is more to the story.
To symbolize each vote, we used Quantities>Graduated Symbols. There were six custom classes created ranging from 1 vote to 800 votes. Anyone wondering why the symbol size ranges from 3.56 to 15? Well, let’s take a look and get into some more specifics about why we see that odd minimum size symbol and what we found out about what symbol types are best to use for this dynamic map service.
As alex mentioned, for the SuperbowlFanMap, all of the vote symbology was drawn on the client side. Even though the votes were slow to draw, we got a really good graphic representation and that’s what we were aiming for with this app as well.So first, let’s talk about the symbol type that we are using.
Typically, when you opt to symbolize your data using Quantities>Graduated Symbols, ArcMap defaults to using Simple Marker Symbols. What we found is if we used Character Marker Symbols is that we got better rendering of symbols. This also gave us better control over the look of the outline. With Simple Marker Symbols, the outline is a fixed part of the symbol. On the other hand, with Character Marker Symbols you are using fonts and have better control over sizing each part of the multi-layer symbol. Another reason we are using Character Marker symbols is because these symbols when displayed render more like circles compared to Simple Marker Symbols.But we also had to manipulate one other property of the symbol…
Now let’s talk about these interesting numbers you see for symbol size… especially the minimum sized one. It took a bit of trial and error, but what we found is that by setting the minimum size of the symbol to a non-integer value, that all of the symbols looked more like symbols.All of these pieces combined gave us very similar results to the client side rendering achieved with the Superbowl application.
Also, the vote symbology is a multi-scale operational overlay. As you can see here, each team’s mxd had three group layers: very small scales, small scales, and large scales. For each one of the group layers, the symbol sizes representing a specific vote class was increased depending on the viewing scale in the application.We also had two sets of symbology we were using one for many votes, and one for fewer votes (when the application was first releasedWhen the application was first released, we had one set of classifications and symbol sizes as you can see, the TOC shot on the right, uses larger symbol sizes at all of the scale ranges while the second shot of the TOC uses smaller symbol sizes.You’ll also notice that our largest number was 100, and then we increased that to 800 once more people started voting for team preferences.
The cloud offers us convenient access and control over any number of pre-configured servers Eliminates the need to have to request for new hardware or software You can get a machine up and running with ArcGIS software in as little as 10 minutesYou can save images of your machines that you can use to start up new machines or use as backups in case you mess something upAll the data in the cloud can be replicated or backed up using volumes or snapshotsVolumes are virtual harddrives which you can attach or detach at anytimeSnapshots are images of these harddrives which you can use as backupsSo even if you have problems with your machine, you can start up a new machine and attach your volume to it very quicklyWhen we realized how popular our app was getting, we switched to using higher configuration machines that had more memory and processing powerWe made sure to regularly create images of our machines so that we could revert to previous versions when we encounter problemsThis also gives us the ability to easily add or remove machines to deal with spikes or drops in traffic
The biggest advantage of using the cloud for a large scale deployment is the ability to replicate your servers.For the FanMap, we had daily updates made to the servers whether it was to update the code or the map services. We had two map servers which we updated one by one so that there was always one running and service would not be interrupted. We simply made an image of the main instance, launched it, remove the previous version from the load balancer, and added the new version.And then we have the load balancer. Even with peak traffic, our map servers were never overwhelmed because the load balancer did a good job distributing the workload between them.Finally, we made sure our two map servers were located in different physical regions so that the app would still be running even if there was an outage in one region.To make sure we could handle the amount of traffic that our servers had to deal with, we also made sure to regularly monitor our server performances. We set up Performance Monitor logs that kept track of how much resources was being used for specific functions on each serverWe constantly checked to make sure that our CPU usage wasn’t going off the charts in the Task Manager
Deploying our app in the cloud allowed us to learn a few things, unfortunately, the hard way.While creating images of machines, the machines are inaccessible for about 10-15 minutes (depending on size of machine)The main problem with using virtual machines is transferring files over to them. We transferred our cache over to our machines via FTP and it took a long time. One option for avoiding this is to copy your data over and cache it on the cloud. You can cache using a more powerful caching machine there and then transfer the volume over to a less powerful and cheaper machine.Finally, the most important thing that we realized is the need to plan out our security settings in advance because security groups cannot be changed once an instance is started. Security groups determine access permissions for the machines and play a big role in safeguarding your servers from getting hacked.
When we first developed this app, we had all the servers in just one security group but we realized that we wouldn’t be able to modify the settings for individual machines and if one machine was compromised, the others would be compromised as well.
So we switched to a multi-tiered security system, with each server type getting assigned its own security group.This way we would be able to set the security settings for each individual machine. For example, we allowed HTTP access to the web servers, but not to the other two.Having multiple security groups also makes it much harder for the servers to get hacked since there are three different levels of protection.
One version of the PollMap Template is available on ArcGIS.com right now. Enter “pollmap” in the search box (may need to switch “Show:” option at the top of the screen to “All Content”)
Unzip the file to your wwwroot directory, under /WaterPoll (or any other name desired – this documentation assumes /WaterPoll. You can use anything you want, but fair warning: you won’t want to change this later unless you absolutely have to.
The cloud offers us convenient access and control over any number of pre-configured servers Eliminates the need to have to request for new hardware or software You can get a machine up and running with ArcGIS software in as little as 10 minutesYou can save images of your machines that you can use to start up new machines or use as backups in case you mess something upAll the data in the cloud can be replicated or backed up using volumes or snapshotsVolumes are virtual harddrives which you can attach or detach at anytimeSnapshots are images of these harddrives which you can use as backupsSo even if you have problems with your machine, you can start up a new machine and attach your volume to it very quicklyWhen we realized how popular our app was getting, we switched to using higher configuration machines that had more memory and processing powerWe made sure to regularly create images of our machines so that we could revert to previous versions when we encounter problemsThis also gives us the ability to easily add or remove machines to deal with spikes or drops in traffic
Label and Title appear to the user on theVoteWindow page and on the chartrespectivelyValue is the field name this choice is linked to in the map serviceColor is the chart color, fill first, then outline.
The cloud offers us convenient access and control over any number of pre-configured servers Eliminates the need to have to request for new hardware or software You can get a machine up and running with ArcGIS software in as little as 10 minutesYou can save images of your machines that you can use to start up new machines or use as backups in case you mess something upAll the data in the cloud can be replicated or backed up using volumes or snapshotsVolumes are virtual harddrives which you can attach or detach at anytimeSnapshots are images of these harddrives which you can use as backupsSo even if you have problems with your machine, you can start up a new machine and attach your volume to it very quicklyWhen we realized how popular our app was getting, we switched to using higher configuration machines that had more memory and processing powerWe made sure to regularly create images of our machines so that we could revert to previous versions when we encounter problemsThis also gives us the ability to easily add or remove machines to deal with spikes or drops in traffic
This sample from Ireland shows a “Happiness Index” discussed on a radio broadcast from time to time.North American postal codes are replaced by a mixture of Irish cities and townlandsOpted for a basemap which showed more small towns at scales 144k and below, for local patterns
This sample from Ireland shows a “Happiness Index” discussed on a radio broadcast from time to time.North American postal codes are replaced by a mixture of Irish cities and townlandsOpted for a basemap which showed more small towns at scales 144k and below, for local patterns
The biggest advantage of using the cloud for a large scale deployment is the ability to replicate your servers.For the FanMap, we had daily updates made to the servers whether it was to update the code or the map services. We had two map servers which we updated one by one so that there was always one running and service would not be interrupted. We simply made an image of the main instance, launched it, remove the previous version from the load balancer, and added the new version.And then we have the load balancer. Even with peak traffic, our map servers were never overwhelmed because the load balancer did a good job distributing the workload between them.Finally, we made sure our two map servers were located in different physical regions so that the app would still be running even if there was an outage in one region.To make sure we could handle the amount of traffic that our servers had to deal with, we also made sure to regularly monitor our server performances. We set up Performance Monitor logs that kept track of how much resources was being used for specific functions on each serverWe constantly checked to make sure that our CPU usage wasn’t going off the charts in the Task Manager
The biggest advantage of using the cloud for a large scale deployment is the ability to replicate your servers.For the FanMap, we had daily updates made to the servers whether it was to update the code or the map services. We had two map servers which we updated one by one so that there was always one running and service would not be interrupted. We simply made an image of the main instance, launched it, remove the previous version from the load balancer, and added the new version.And then we have the load balancer. Even with peak traffic, our map servers were never overwhelmed because the load balancer did a good job distributing the workload between them.Finally, we made sure our two map servers were located in different physical regions so that the app would still be running even if there was an outage in one region.To make sure we could handle the amount of traffic that our servers had to deal with, we also made sure to regularly monitor our server performances. We set up Performance Monitor logs that kept track of how much resources was being used for specific functions on each serverWe constantly checked to make sure that our CPU usage wasn’t going off the charts in the Task Manager
