Using Data Science & Serverless Python to find apartment in Toronto
•
0 likes•222 views
See how Ian Whitestone (Data Scientist at Shopify) created Domi – #Toronto Apartment Finder app, using #Serverless Framework #Zappa for #Python on #AWS, #PostGIS, #Slack, and some #Regression Techniques: https://www.youtube.com/watch?v=JE_zEqe7M_8 http://ServerlessToronto.org thanks https://www.linkedin.com/company/trend-micro for catering, https://www.linkedin.com/company/myplanethq for hosting, and https://www.linkedin.com/company/manning-publications-co for book giveaways!
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Using Data Science & Serverless Python to find apartment in Toronto
Similar to Using Data Science & Serverless Python to find apartment in Toronto (20)
More from Daniel Zivkovic
More from Daniel Zivkovic (20)
Recently uploaded
Recently uploaded (20)
Using Data Science & Serverless Python to find apartment in Toronto
- 1. Thursday, Feb 27, 2020 1. Intro & Activity Update 2. Community Open Mic 3. Ian Whitestone: “Bootstrapping a data-driven application with Zappa (Serverless Python) to find an apartment in Toronto” 4. Networking 1 ServerlessToronto.org Meetup Agenda
- 2. Serverless is not just about the Tech: 2 Serverless is New Agile & Mindset Serverless Dev (gluing other people’s APIs and managed services) We're obsessed by creating business value (meaningful MVPs, products) and helping Startups We build bridges between Serverless Community (“Dev leg”), and Front-end & Voice- First folks (“UX leg”), and empower UX developers Achieve agility NOT by “sprinting” faster (like in Scrum), but by working smarter (by using bigger building blocks and less Ops)
- 4. Venue Sponsor 4 As Certified B Corporation, Myplanet is purpose-driven and creates benefit for all stakeholders, not just shareholders!
- 5. Knowledge Sponsor 5 Get your raffle tickets… and GOOD LUCK! 1. Go to www.manning.com 2. Select eBook or Video title you like 3. Add it to the shopping cart 4. Raffle winner will send me email address used 5. For Manning staff to move to your dashboard
- 6. Bonus Raffle from our friends 6 Get your raffle tickets… and GOOD LUCK! 2 tickets for “Full-Day on JAMstack Web Technology” paid event.
- 7. © 2020 Trend Micro Inc.1 Trend Micro Cloud One™ Cloud Security Simplified Albert Kramer Technical Director Trend Micro
- 8. © 2020 Trend Micro Inc.2 Cloud Na)ve Applica)ons Strategic Priori,es for Cloud Builders Cloud Migra)on Cloud Opera)onal Excellence Risk & Compliance GovernanceAssurance Containers Serverless DevOps Google Kubernetes Engine Azure Kubernetes Service AWS Lambda Azure Func;ons Google Serverless Amazon S3 Azure Blob Google Storage GitHub Jenkins How do you secure such a complex & fast-paced environment? TransiCon, not a cut-over Hybrid cloud is the norm • Deliver fast, iterate oFen Infrastructure-as-codeCode leverage: code re-use, open- source and public code repositories Repeatable & consistent Infrastructure and cost opCmizaCon MulC-cloud Cloud Center of Excellence (CCoE) Physical Virtual Cloud Cloud Storage Cloud Workloads
- 9. © 2020 Trend Micro Inc.3 Trend Micro Cloud One™ Security Services Pla/orm for Cloud Builders Cloud Na)ve Applica)ons Containers Serverless DevOps Cloud StorageCloud Workloads Cloud Opera)onal Excellence Risk & Compliance GovernanceAssurance Cloud Migra)on Physical Virtual Cloud
- 10. © 2020 Trend Micro Inc.4 Trend Micro Cloud One™ Security Services Pla/orm for Cloud Builders Workload & container host security Security for container images File scanning for cloud storage services Network layer IPS to secure entire VPCs Security for serverless funcCons, APIs, and applicaCons Assurance cloud infrastructure is configured securely Cloud-na)ve, SaaS-based pla2orm with the most extensive set of cloud security services Pla$orm: • Single-sign-on • Common user and cloud account enrollment • Common procurement & billing • Common support & documenta6on • Expandable pla9orm
- 11. © 2020 Trend Micro Inc.5 Trend Micro Cloud One™ Security Services Pla/orm for Cloud Builders FREE TRIAL https://www.cloudconformity.com DEMO Serverless and API protection https://www.trendmicro.com/en_us/business/products/hybrid-cloud/cloud-one-application-security.html Automation center https://automation.deepsecurity.trendmicro.com
- 13. Upcoming 2020 #ServerlessTO Meetups 2 1. Intro to PySpark – Python Data Analysis at scale in the Cloud – Jonathan Rioux, Lead Data Scientist at EPAM Systems & author of PySpark in Action book ** MARCH 19 ** 2. Introduction to Google BigQuery – Matt Welke, Software Developer at GroupBy Inc 3. Solving your Business Problems with Serverless Architectures – Panel discussion ** BACK BY POPULAR DEMAND ** 4. Serverless with Pivotal Cloud Foundry – Adib Saikali, Principal Platform Architect at VMware 5. Fivetran – Data Pipelines, Reinvented – Replicate your data into the Cloud Warehouse of your choice 6. Your Own Presentation – PLEASE VOLUNTEER ☺
- 14. Community Open Mic 3 Your 10 sec. pitch ☺ - Looking for work? - Offering work? About You – because without you, there would be no meetups!
- 15. Feature Talk Ian Whitestone, Data Scientist at Shopify 4
- 16. Bootstrapping a data-driven application with ZappaBootstrapping a data-driven application with Zappa ServerlessToronto-February2020 IanWhitestone
- 19. >>> df[df.bedrooms == 1].price.median()>>> df[df.bedrooms == 1].price.median() $2,200
- 20. >>> df[df.bedrooms == 0].price.median()>>> df[df.bedrooms == 0].price.median() $1,800
- 21. >>> df[df.housing_type == 'basement'].price.median()>>> df[df.housing_type == 'basement'].price.median() $1,500
- 22. Not only is it expensive..Not only is it expensive.. constantlylookingonmultiplesites listingsgoreallyfast competitiveapplications biddingwars ...
- 23. Maybe there's a better way?Maybe there's a better way? InspiredbyasimpleSanFrancisco madebyVik Paruchuri... apartmentpostingslackbot
- 25. 0:00
- 34. Serverless Python from ScratchServerless Python from Scratch
- 35. UseCase:Periodicallydownloadsomedata,savetocloudstorage(S3) # Create virtualenv and install packages → pipenv install requests
- 36. handler.py import requests import yaml import main def my_handler(event=None, context=None): """Kick off the desired function Parameters ---------- event : dict, optional AWS Lambda uses this parameter to pass in event data to the handler context : LambdaContext, optional AWS Lambda uses this parameter to provide runtime information to your handler """ main.do_stuff() # and things
- 37. → tree ├── Pipfile ├── Pipfile.lock ├── app │ ├── main.py │ └── handler.py
- 38. Step 1: Build Deployment PackageStep 1: Build Deployment Package AWSlambdajustprovidesalinuxenvironmentwithpythoninstalled Youneedtouploadanyadditionalpackagesyouneedforyourfunctions
- 39. → pipenv run pip show requests Name: requests Version: 2.22.0 Summary: Python HTTP for Humans. Home-page: http://python-requests.org Author: Kenneth Reitz Author-email: me@kennethreitz.org License: Apache 2.0 Location: /Users/ianwhitestone/.../virtualenvs/.../lib/python3.7/site-packages 👈 Requires: idna, urllib3, certifi, chardet Required-by: zappa
- 41. → cd $PACKAGES_DIR → zip -r ${PROJECT_DIR}/deployment-package.zip . ... → cd ${PROJECT_DIR}/app → zip -r ${PROJECT_DIR}/deployment-package.zip .
- 42. Step 2: Create Identity & Access Management (IAM)Step 2: Create Identity & Access Management (IAM) RoleRole InAWSyouuseIAMrolestodictatewhatthingsaserviceoruserhasaccessto Forourusecase,theIAMrolewilldescribealltheotherAmazonservicesour lambdafunctioncaninteractwith
- 43. → aws iam create-role --role-name lambda_basic_role --assume-role-policy-document file://lambda_trust_policy.json { "Role": { "Path": "/", "RoleName": "lambda_basic_role", "RoleId": "AROA......", "Arn": "arn:aws:iam::<account_num>:role/lambda_basic_role", "CreateDate": "2019-09-22T16:48:43Z", "AssumeRolePolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "lambda.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } } }
- 44. # Give it full access to S3 → aws iam attach-role-policy --role-name lambda_basic_role --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess # And cloudwatch (logs) → aws iam attach-role-policy --role-name lambda_basic_role --policy-arn arn:aws:iam::aws:policy/CloudWatchFullAccess
- 45. Step 3: Create Lambda FunctionStep 3: Create Lambda Function
- 46. → aws lambda create-function --function-name download_stuff --runtime python3.7 😎 --role arn:aws:iam::<account_num>:role/lambda_basic_role --handler handler.my_handler --zip-file fileb://../deployment-package.zip --memory-size 128 --timeout 900 # max timeout (15 minutes)
- 47. Step 4: Create Cloudwatch Events to Trigger LambdaStep 4: Create Cloudwatch Events to Trigger Lambda Think"cronjob"
- 48. # Run it every hour aws events put-rule --name "RunLambdaFunction" --schedule-expression "rate(1 hour)" --state "ENABLED" # Add lambda function as target aws events put-targets --rule "RunLambdaFunction" --targets "Id"="1","Arn"="arn:aws:lambda:us-east-1:<account_num>:function:download_stuff"
- 49. Testitout...
- 52. → chmod -R 755 $PACKAGES_DIR → chmod -R 755 $PROJECT_DIR
- 54. → aws lambda update-function-code --function-name download_stuff --zip-file fileb://../deployment-package.zip
- 55. Tryagain...
- 56. UseCase:Periodicallydownloadsomedata,savetocloudstorage(S3)database # Create virtualenv and install packages → pipenv install requests → pipenv install psycopg2 # new dependency!
- 62. Now let's talk about ZappaNow let's talk about Zappa
- 64. UseCase:Periodicallydownloadsomedata,savetodatabase # Create virtualenv and install packages → pipenv install requests → pipenv install psycopg2 → pipenv install zappa # new dependency!
- 65. zappa_settings.json (canbecreatedstepbystepwithzappa init) { "dev": { "apigateway_enabled": false, "aws_region": "us-east-1", "profile_name": "default", "project_name": "download_stuff", "runtime": "python3.7", "s3_bucket": "download_stuff", "keep_warm": false, "events": [{ "function": "main.do_stuff", "expression": "rate(1 hour)" }] }, "prod": { // config for production } }
- 66. → zappa deploy dev Calling deploy for stage dev.. Downloading and installing dependencies.. - psycopg2-binary==2.8.3: Using locally cached manylinux wheel - sqlite==python3: Using precompiled lambda package 'python3.7' Packaging project as zip. Uploading zappa-cron-test-dev-1569183776.zip (9.5MiB).. 100%|█████████████████████████████████████████| 9.97M/9.97M [00:21<00:00, 528KB/s] Scheduling.. Scheduled zappa-cron-test-dev-test.run with expression rate(1 minute)! Deployment complete!
- 69. Easilyviewlogs # Show all logs → zappa tail dev Calling tail for stage dev.. [1569183806942] Instancing.. [1569183806943] [DEBUG] 2019-09-22T20:23:26.942Z 97e8-d0b23aaf17a0 Zappa Event: {'time': '2019-09-22T20:23:24Z', 'detail-type': 'Scheduled Event', 'source': 'aws.events', 'region': 'us-east-1', 'detail': {}, 'version': '0', 'resources': ['arn:aws:events:us-east-1:<>:rule/zappa-cron-test-dev-test.run'], 'id': '75265076-af20-30ca-fd1e-b3fcbe478843', 'kwargs': {}} [1569183806988] hello world!! [1569183865861] [DEBUG] 2019-09-22T20:24:25.861Z 8064-931e09d761e6 Zappa Event: {'time': '2019-09-22T20:24:24Z', 'detail-type': 'Scheduled Event', 'source': 'aws.events', 'region': 'us-east-1', 'detail': {}, 'version': '0', 'resources': ['arn:aws:events:us-east-1:<>:rule/zappa-cron-test-dev-test.run'], 'id': '823d2b37-6a85-c162-5084-1906492f4b93', 'kwargs': {}} [1569183865861] hello world!!
- 70. Easilyviewlogs # Show logs from specific timeframe → zappa tail dev --since 1m # Show logs from specific timeframe and filter → zappa tail batch_secondary_us_east_1 --since 1d --filter "ERROR"
- 71. Invokerawcommandsonlambdafortesting(avoidre-deploying) → zappa invoke dev "import psycopg2; print('hello')" --raw Calling invoke for stage dev.. [START] RequestId: e35516da-b71d-4452-9896-e622fe263d1f Version: $LATEST Instancing.. [DEBUG] 2019-09-22T20:20:09.25Z e622fe263d1f Zappa Event: {'raw_command': "import psycopg2; print('hello')"} hello [END] RequestId: e35516da-b71d-4452-9896-e622fe263d1f [REPORT] RequestId: e35516da-b71d-4452-9896-e622fe263d1f Duration: 198.44 ms Billed Duration: 200 ms Memory Size: 512 MB Max Memory Used: 84 MB Init Duration: 525.29 ms
- 74. Many more features..Many more features.. ExecuteinresponsetootherAWSevents Easyrollbackswithzappa rollback prod -n 1 Easyinfrateardownwithzappa undeploy prod Extensibilitythroughcustomcallbacks Seeexample andmore... here
- 75. Other serverless frameworksOther serverless frameworks ⚠WritteninNode.js CreatedbyAmazon Serverless Chalice
- 79. Overview of domiOverview of domi
- 81. "app": { "app_function": "domi.app.app", "aws_region": "us-east-1", "slim_handler": false, "runtime": "python3.7", "certificate_arn": "arn:aws:acm:us-east-1:XXXXXX:certificate/XXXXXX", "domain": "domi.cloud", "keep_warm": true, "keep_warm_expression": "cron(0/3 12-4 ? * * *)", "timeout_seconds": 3, },
- 83. "batch_primary_us_east_1": { "slim_handler": false, "keep_warm": false, "aws_region": "us-east-1", "runtime": "python3.7", "events": [ { "function": "domi.apartments.handlers.get_all_listings", "expression": "cron(0 */2 * * ? *)" }, { "function": "domi.apartments.handlers.process_new_listings", "expression": "cron(15 */2 * * ? *)" }, { "function": "domi.apartments.handlers.check_listing_statuses", "expression": "cron(15 */2 * * ? *)" }, ], "timeout_seconds": 900, }
- 86. Run fast, powerful spatial queriesRun fast, powerful spatial queries SELECT listings.* FROM listings, user_regions WHERE ST_Contains(user_regions.geom, listings.geom) AND bedrooms >= 1 AND bathrooms >= 1 AND ...
- 87. from geoalchemy2 import Geometry from sqlalchemy import Column, Integer class Listing(BASE): __tablename__ = "listings" id = Column(Integer, primary_key=True) geom = Column(Geometry(geometry_type="POINT", srid=4326)) bedrooms = Column(Integer) class UserRegion(BASE): __tablename__ = "user_regions" id = Column(Integer, primary_key=True) user_id = Column(Integer, ForeignKey("users.id")) geom = Column(Geometry(geometry_type="POLYGON", srid=4326))
- 88. from models import Listing, UserRegion, SESSION from sqlalchemy import func listings = ( SESSION.query( Listing.id, Listing.source, Listing.price, ... ) .join( UserRegion, and_( UserRegion.user_id == 123, func.ST_Contains(UserRegion.geom, Listing.geom), ), )
- 93. Option 1: ClusteringOption 1: Clustering Theory:Clustersimilarlistingsanduseactualpricedistributionofcluster Source
- 94. Key problem with this approach:Key problem with this approach: Eachvariableistreatedashavingthesameimpactonprice(afterscaling)
- 95. Option 2: Linear RegressionOption 2: Linear Regression Theory:Linearregressiontopredictmean,calculatepredictionintervaltoget rangeofexpectedvalues
- 98. Option 3: Quantile RegressionOption 3: Quantile Regression Theory:Quantileregressiontopredictp25&p75
- 102. Westartwithsomestandardfeatures: numberofbedrooms,bathrooms size(sqft) isitfurnished? unittype(apartmentbuilding,house,condo,basement,etc..) ... price ~ bedrooms + bathrooms + size + is_furnished + ...
- 103. But how do we account for location?But how do we account for location?
- 104. Area Encoding?Area Encoding? Automaticallyclustereachpointintoan"area" from sklearn.cluster import KMeans X = df[['lat', 'long']].values km = KMeans(20, init='k-means++') km.fit(X) clusters = km.predict(X) # classify points into 1 of 20 clusters
- 106. price ~ bedrooms + bathrooms + size + is_furnished + ... + cluster_0 + cluster_1 + ...
- 111. Nearest NeighborsNearest Neighbors 👪 🏠 ...?... 🏠 👪👪 🏠 ...?... 🏠 👪 Source:ErikBernhardsson'sFantasticBlog
- 113. >>> from annoy import AnnoyIndex # build the tree >>> featurees = ["lat_scaled", "long_scaled", "bedrooms_scaled"] >>> tree = AnnoyIndex(len(features), "euclidean") >>> for index, row in df[features].iterrows(): tree.add_item(index, row.values) >>> tree.build(10) ... # search da tree >>> apartment_index = 1 # index of apartment to search >>> tree.get_nns_by_item(apartment_index, 51) # get 50 closest points [1, 23412, 424, 794, 12, 939, 58, 3, ...]
- 114. price ~ bedrooms + bathrooms + size + is_furnished + ... + nn_50_avg_price + ...
- 116. Better handling for remote apartments (outskirts)Better handling for remote apartments (outskirts) Addexponentiallydecayedweightingtoeachpointbasedondistance
- 117. "price_rank_primary": { "project_name": "domi", "slim_handler": true, "memory_size": 3000, "apigateway_enabled": false, "keep_warm": false, "aws_region": "us-east-1", "runtime": "python3.7", "events": [ { "function": "domi.apartments.price_rank.price_rank", "expression": "cron(0 */2 * * ? *)" } ], "timeout_seconds": 900, },
- 118. Displaying to usersDisplaying to users
- 120. User design considerationsUser design considerations User'sdon'twantablackbox,otherwisetheywon'ttrustit.Givethemcontext! "$3,250isnormal" versus "$3,250istypicalforthistypeoflisting.Listingswiththesamenumberofbedrooms, bathroomsandsimilarsquarefootageandlocationtypicallyhavepricerangesbetween $3,175and$4,200" Notetherounding..."pricerangesbetween$3,183.23and$4,177.69"justseems sketchy
- 122. All ya need is a little...All ya need is a little...
- 124. Monitoring with Great ExpectationsMonitoring with Great Expectations
- 127. Enter Great ExpectationsEnter Great Expectations Frameworkforwritingtestsfordata validatedatasetagainstour"expectations" Runchecksondatasets: atbeginning/endofpipelines beforefeedingintomachinelearningmodel periodically,on-schedule github.com/great-expectations
- 128. Types of ExpectationsTypes of Expectations expect_column_values_to_not_be_null expect_column_values_to_match_regex expect_column_values_to_be_unique expect_column_values_to_match_strftime_format expect_table_row_count_to_be_between expect_column_median_to_be_between ...andmanymore
- 129. Example: Validating Row CountsExample: Validating Row Counts
- 130. Example: Validating Row CountsExample: Validating Row Counts expectations.json { "data_asset_name": "yesterdays_craigslist_listings", "expectation_suite_name": "default", "expectations": [ { "expectation_type": "expect_table_row_count_to_be_between", "kwargs": { "min_value": 300 } } ], }
- 131. run_data_checks.py from domi.db import DB_ENGINE from great_expectations.dataset import SqlAlchemyDataset sql_query = """ SELECT id FROM {tablename} WHERE TRUE AND DATE_TRUNC('day', created_at) = CURRENT_DATE - INTERVAL '1' DAY """ new_sql_dataset = SqlAlchemyDataset(custom_sql=sql_query, engine=db_engine) validation_results = new_sql_dataset.validate(expectation_suite="expectations.json") if validation_results["success"]: ...
- 133. Example: Model Monitoring with DistributionalExample: Model Monitoring with Distributional ExpectationsExpectations
- 134. Serverless Gotchas and WorkaroundsServerless Gotchas and Workarounds
- 135. Gotcha 1: Shared SESSION objectGotcha 1: Shared SESSION object Objectsinstantiatedonimportaresharedacrossfunctioninvocations theyonlygetresetduringacoldstart Ifonetransactionfails,allsubsequenttransactionsinotherinvocationswill startfailling from domi.handlers import process_new_listings from domi.db import SESSION # 👆 everything instantiated above here is shared across future function invocations def lambda_handler(event, context): process_new_listings()
- 136. WorkaroundWorkaround # Automatically ensure all transactions are succesfully committed, # or rolled back if not def commit_session(_raise=True): if not SESSION: return try: SESSION.commit() except Exception as e: SESSION.rollback() if _raise: raise def session_committer(func): def wrapper(*args, **kwargs): try: return func(*args, **kwargs) finally: commit_session() return wrapper # Use decorator on any function doing database transactions @session_committer def process_new_listings(): ...
- 137. Gotcha 2: Slim handler kills performance on coldGotcha 2: Slim handler kills performance on cold startsstarts {"slim_handler": true}savesdeploymentpackagetoS3 Ineedthisbecausesomeofmyprocessesrelyonpandas,numpy,scipy pullsdownpackageandcopiesto/tmpdirectoryatruntime Forawarmlambdafunction,thisonlyhappensonce,andweareokay Whentrafficspikes,AWSautoscalesandstartsspinningupmorelambdas allofthesegothroughcoldstart mustpulldownandunzip>100MBfilefromS3
- 138. WorkaroundWorkaround ZappaallowscustomPythonfunctionstobeinvokedatdifferentstagesofthe deploymentprocess Buildacustomzipcallbacktoremovelargepackagesfromdeployment packages "callbacks": { // Call custom functions during the local Zappa deployment/update process "settings": "my_app.settings_callback", // After loading the settings "zip": "my_app.zip_callback", // After creating the package "post": "my_app.post_callback", // After command has executed }
- 139. WorkaroundWorkaround See formoredetails. "app": { "app_function": "domi.app.app", "aws_region": "us-east-1", "runtime": "python3.7", "certificate_arn": "arn:aws:acm:us-east-1:XXXXXX:certificate/XXXXXX", "domain": "domi.cloud", "keep_warm": true, "keep_warm_expression": "cron(0/3 12-4 ? * * *)", "timeout_seconds": 3, // updated settings "slim_handler": false, "regex_excludes": [ "pandas", "scipy", "numpy", "PIL", "statsmodels", "matplotlib" ], "callbacks": { "zip": "zappa_package_cleaner.main" }, }, blogpost
- 140. Gotcha 3: Adding new, pre-compiled packagesGotcha 3: Adding new, pre-compiled packages hasprecompiledC-extensions Notallpackagesareavailablein Nonewonesarebeingadded annoy Miserlou/lambda-packages
- 141. WorkaroundWorkaround (⚠hackiness follows)(⚠hackiness follows) SpinupEC2withAmazonLinuxAMI Installannoyandbuildcompileddependencies Savecompiledpackagetorepo try: # when running locally this will import succesfully # when running on lambda, this will fail and fallback to pre-compiled version from annoy import AnnoyIndex except: from lambda_annoy import AnnoyIndex
- 142. Workaround (long term)Workaround (long term) Inthe(hopefullynottoodistant)future,zappawillsupportdeploymentpackage creationwithdocker. Workaroundsareavailable,seehere
- 144. AppendixAppendix
- 145. Deploying with Github ActionsDeploying with Github Actions ianwhitestone.work/AWS-Serverless-Deployments-With-Github-Actions
- 146. import statsmodels.formula.api as smf mod = smf.quantreg('foodexp ~ income', data) # uses patsy model formulas res = mod.fit(q=.5) print(res.summary())