Recommended
More Related Content
Viewers also liked
Similar to Jesse Siegel Capstone Project
Similar to Jesse Siegel Capstone Project (20)
Recently uploaded
Recently uploaded (20)
Jesse Siegel Capstone Project
- 1. CREATING ARCGIS ONLINE MAPS USING TWEEPY AND GSPREAD, AND DISPLAYING THEM IN AN ANDROID APP BY: JESSE SIEGEL MAY 26TH, 2016
- 2. TWEEPY • Python library for accessing the Twitter API. • Requires you to create an “application” at https://apps.twitter.com/app/new in order to obtain credentials. • Also requires the oauth2client library, and OAuth credentials from https://console.developers.google.com/project • Can be used to track tweets as they’re created, or look at tweets that were made in the last few weeks. • Can also be used to write tweets.
- 3. A SINGLE TWEET CONTAINS A LOT OF INFORMATION {"created_at":"Thu Apr 07 00:46:59 +0000 2016","id":717876289495937024,"id_str":"717876289495937024","text":"@derpuskhan #PS4sharennhttps://t.co/iWJXhyT89U https://t.co/yODrYcDU4H","source":"u003ca href="http://www.playstation.com" rel="nofollow"u003ePlayStation(R)4u003c/au003e","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":634239598,"in_reply_to_user_id_str":"634 239598","in_reply_to_screen_name":"DerpusKhan","user":{"id":545261627,"id_str":"545261627","name":"Adri","screen_name":"DovahBap","location":"The Salt Fields, California","url":"http://WeAreTheHomeworldGems.Com","description":"Failed Kamikaze pilot. #BornstellarFamily #BapSquad #JusticeForTay","protected":false,"verified":false,"followers_count":1136,"friends_count":630,"listed_count":35,"favourites_count":40326,"statuses_count":171940,"created_at":"Wed Apr 04 16:38:28 +0000 2012","utc_offset":-25200,"time_zone":"Pacific Time (US & Canada)","geo_enabled":true,"lang":"en","contributors_enabled":false,"is_translator":false,"profile_background_color":"131516","profile_background_image_url":"http://abs.twimg.com/images/the mes/theme14/bg.gif","profile_background_image_url_https":"https://abs.twimg.com/images/themes/theme14/bg.gif","profile_background_tile":true,"profile_link_color":"009999","profile_sid ebar_border_color":"EEEEEE","profile_sidebar_fill_color":"EFEFEF","profile_text_color":"333333","profile_use_background_image":true,"profile_image_url":"http://pbs.twimg.com/profile_images/71 7149939915038720/DMuspQjY_normal.jpg","profile_image_url_https":"https://pbs.twimg.com/profile_images/717149939915038720/DMuspQjY_normal.jpg","profile_banner_url":"https:/ /pbs.twimg.com/profile_banners/545261627/1378251626","default_profile":false,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates" :null,"place":null,"contributors":null,"is_quote_status":false,"retweet_count":0,"favorite_count":0,"entities":{"hashtags":[{"text":"PS4share","indices":[12,21]}],"urls":[{"url":"https://t.co/iWJXhyT89U","expan ded_url":"https://store.playstation.com/#!/en-us/tid=CUSA00419_00","display_url":"store.playstation.com/#!/en- us/tid=Cu2026","indices":[23,46]}],"user_mentions":[{"screen_name":"DerpusKhan","name":"Kris Manuelos","id":634239598,"id_str":"634239598","indices":[0,11]}],"symbols":[],"media":[{"id":717875773831426048,"id_str":"717875773831426048","indices":[47,70],"media_url":"http://pbs.twim g.com/ext_tw_video_thumb/717875773831426048/pu/img/i- ZH66Yz2S7G7826.jpg","media_url_https":"https://pbs.twimg.com/ext_tw_video_thumb/717875773831426048/pu/img/i- ZH66Yz2S7G7826.jpg","url":"https://t.co/yODrYcDU4H","display_url":"pic.twitter.com/yODrYcDU4H","expanded_url":"http://twitter.com/DovahBap/status/717876289495937024/video /1","type":"photo","sizes":{"thumb":{"w":150,"h":150,"resize":"crop"},"medium":{"w":600,"h":338,"resize":"fit"},"small":{"w":340,"h":191,"resize":"fit"},"large":{"w":1024,"h":576,"resize":"fit"}}}]},"extended_ entities":{"media":[{"id":717875773831426048,"id_str":"717875773831426048","indices":[47,70],"media_url":"http://pbs.twimg.com/ext_tw_video_thumb/717875773831426048/pu/img/i- ZH66Yz2S7G7826.jpg","media_url_https":"https://pbs.twimg.com/ext_tw_video_thumb/717875773831426048/pu/img/i- ZH66Yz2S7G7826.jpg","url":"https://t.co/yODrYcDU4H","display_url":"pic.twitter.com/yODrYcDU4H","expanded_url":"http://twitter.com/DovahBap/status/717876289495937024/video /1","type":"video","sizes":{"thumb":{"w":150,"h":150,"resize":"crop"},"medium":{"w":600,"h":338,"resize":"fit"},"small":{"w":340,"h":191,"resize":"fit"},"large":{"w":1024,"h":576,"resize":"fit"}},"video_info":{" aspect_ratio":[16,9],"duration_millis":7007,"variants":[{"bitrate":320000,"content_type":"video/mp4","url":"https://video.twimg.com/ext_tw_video/717875773831426048/pu/vid/320x180 /cBtmMgWPzfQWb7c-.mp4"},{"content_type":"application/x- mpegURL","url":"https://video.twimg.com/ext_tw_video/717875773831426048/pu/pl/u8zsifq1hZVdvcmZ.m3u8"},{"content_type":"application/dash+xml","url":"https://video.twimg.com/ ext_tw_video/717875773831426048/pu/pl/u8zsifq1hZVdvcmZ.mpd"},{"bitrate":832000,"content_type":"video/mp4","url":"https://video.twimg.com/ext_tw_video/717875773831426048 /pu/vid/640x360/YqUKfLiGVATfqnN_.mp4"},{"bitrate":2176000,"content_type":"video/mp4","url":"https://video.twimg.com/ext_tw_video/717875773831426048/pu/vid/1280x720 /6rVetAqFOUiDJqxl.mp4"}]}}]},"favorited":false,"retweeted":false,"possibly_sensitive":false,"filter_level":"low","lang":"und","timestamp_ms":"1459990019569"}
- 4. FINDING KEY FIELDS https://dev.twitter.com/overview/api Links to each section, with fields listed
- 5. EXTRACTING INFO FROM TWEETS
- 6. TWEETING WITH PRECISE LOCATION
- 8. GSPREAD • Python library for accessing the Google Spreadsheets API. • Credentials are stored in a json file, and read by the script. • Can be used to create and write to worksheets (tabs) in a spreadsheet, and to read from them.
- 10. Publishing produces this type of link: https://docs.google.com/spreadsheets/d/1_aLWgvAdCGqaeJR6Vqs0aOJu2J _6K2-tIurvHfg8_C4/pub?gid=1266833031&single=true&output=csv Change to: https://docs.google.com/spreadsheets/d/1_aLWgvAdCGqaeJR6Vqs0aOJu2J _6K2-tIurvHfg8_C4/export?gid=1266833031&format=csv&id GETTING INSTANT UPDATES FROM CSV
- 11. ADD TO ARCGIS ONLINE
- 12. TESTJESSEAPP
- 13. CUSTOM POPUP WITH HTML
- 14. ARCGIS RUNTIME SDK FOR ANDROID
- 15. RUNNING SCRIPTS FROM EMAIL • Uses win32com Python library to read emails received by Outlook • Email’s subject determines script to run, body of email is passed as command line arguments to determine terms to track on Twitter.
- 16. EMAIL SHORTCUT APP CREATED WITH ANDROID STUDIO
- 17. • Website used to run python scripts from cloud. • Select from multiple versions of Python. • Many preconfigured libraries, can import missing ones. • Free and paid plans.
Editor's Notes
- Last 200 tweets by user I used writing tweets to automatically respond to users who successfully had their tweets added to the map
- Shows field name and how information is stored, I used samples like this frequently to find out how to grab specific fields
- The twitter API site also has the fields listed, with descriptions of how they can be used.
- Full_name is the city/rough location Followers was added for fun so I could symbolize based on number of followers
- In order for tweets to be added to my map, they have to be sent with precise location from a mobile device. Explain 3 steps. Conveying instructions for how to do this proved a little difficult, and I learned you have to try to make things as clear as possible. I made this image, and version with some text instructions above it. Many people would select from the list of locations above, not seeing the precise location switch at the bottom. Along with creating this instructions I also added a part in my script that would reply to the any tweet that was successfully added to the map, thanking the user and providing a link to the map.
- I used gspread to write the information to Google spreadsheets so it could be shared on agol. With my script, when I pass a command line argument to the script, it checks to see if I have a worksheet with that name, if not it will create one set up the column headers
- In order to publish the worksheet to be able to use it on agol you go to file>Publish to the web,select the sheet to share, and select csv. This provides a link to use add to agol. Now this is the method that ESRI explains in their blogs, and that other resources suggest, but even though it says it automatically republishes when changes are made, this only happens every 5 minutes.
- So this was fine for awhile, but eventually I went back and looked for a way to get faster updates. After some sleuthing I found this little trick to change the url that is created into one that displays updates instantly.
- Then in agol you can select add layer from web. Add that new link as a csv, and every refresh of the map will have the latest version of the spreadsheet.
- Here’s an image of the map I produced after running the script for a few days following the word testjesseapp. Dave helped get the word out to people to try to get their tweets added to the map. During the this phase some issues were discovered with my script. The first, and most frustrating problem to troubleshoot, was my script losing access to the google spreadsheet login after an hour. This meant when I made a change to the script to see if I’d fixed it I had to wait an hour to see if worked. It probably took around 10 tries to get it working… Another issue that came up was my script not being able to read tweets with emoticons, so I had to write in code to convert it to a form that it could handle, and that would also display the emoticons properly in agol. Dave also discovered that sending a tweet with an image would use the location that the image was taken from, regardless of where the tweet was sent.
- I wrote out the HTML for the popups to make sure they displayed properly in the browser and on my app, and so every time I made a new map I just paste in the code instead of configuring fields each time.
- Finally, here are some screenshots of the app I made in Android Studio using the ArcGIS Runtime SDK for Android. It has a simple menu screen to bring you to the different maps. Clicking on a point will bring up the popup, and touching an image in popup will expand the image, with another tap on the image taking you the to tweet in the twitter app.
- So I also thought it would be fun to try to figure out how to get my scripts to launch when I receive an email. I created a completely separate script that uses win32com to read emails as they’re received by Outlook. Here’s a chunk of the script where it reads the emails and decides what script to run. This step is totally unnecessary , but was fun to set up
- I took the email launching a step further and created an app would start emails to launch the scripts. If I select the “run single term tracker” button it will pass the recipient, and subject required to launch the app to my phone’s email client. Whatever word (or 2 space separated words for two term) I put as the body of the email will be passed to the script and tracked on twitter.
- A few days after setting up a computer to run my script non-stop, I looked into using pythonanywhere to run it. Let’s you run scripts from the cloud, so once you run a script in a console from your browser, you can close the browser and it will still be running. Multiple versions of Python, my scripts all work on version 2.7. Comes with a lot of libraries, but was missing the gspread library so I had to import it. Free plans limit the number of consoles you can run at a time, the amount of CPU you can use daily, and the storage space for your account. Once you hit the CPU limit they will slow down your processes until the daily counter resets. Successfully ran for a few days, but console reset once.