![Job (in Jobs DB)
[
{
“job”: “Installation Technician”,
“job_id”: “http://foo.com/jobs/1234”,
“page”: “Personal Information”,
“label”: “First Name”,
“widget_type”: “text_box”,
“locator_type”: “xpath”,
“locator”: “./first-name”
},
...
]
Web application
Now understands how to represent this
question on UI.](https://image.slidesharecdn.com/seleniumforjobseekersseleniumconferencebangalore2016-160629040111/75/Selenium-for-Jobseekers-30-2048.jpg)
![Application (in applications DB)
[
{
“job”: “Installation Technician”,
“job_id”: “http://foo.com/jobs/1234”,
“page”: “Personal Information”,
“label”: “First Name”,
“widget_type”: “text_box”,
“locator_type”: “xpath”,
“locator”: “./first-name”,
“answer”: “Mitchell Johnson”
},
...
]
Web application
appends answer given by Job seeker](https://image.slidesharecdn.com/seleniumforjobseekersseleniumconferencebangalore2016-160629040111/75/Selenium-for-Jobseekers-34-2048.jpg)
![How Apply Service replays the application ?
[
{
“job”: “Installation Technician”,
“job_id”: “http://foo.com/jobs/1234”,
“page”: “Personal Information”,
“label”: “First Name”,
“widget_type”: “text_box”,
“locator_type”: “xpath”,
“locator”: “./first-name”,
“answer”: “Mitchell Johnson”
},
...
]
Uses Selenium to load Job URL
Locates WebElements (locator_type & locator)
Types the answer](https://image.slidesharecdn.com/seleniumforjobseekersseleniumconferencebangalore2016-160629040111/75/Selenium-for-Jobseekers-37-2048.jpg)
More Related Content
Similar to Selenium for Jobseekers
Selenium for Jobseekers
- 1. Selenium for Jobseekers HowIndeed uses Selenium to submit job applications
- 2. Seshu Madhav Chaturvedula SoftwareEngineer Indeed Inc.
- 4. Selenium ~= TestAutomation
- 5. How Indeed usedSelenium to help people get Jobs more specifically, on Smart Phones
- 6. Smart Phones Evolution becomesrevolution
- 7. There is anapp for everything
- 8. Jun July AugSept Oct Nov Dec Jan Feb Mar Apr May 0 100 20 40 60 80 Desktop Traffic Mobile Traffic Searching for jobs, on Mobile Percent of total q1 vs time, 2015-2016
- 9. Applying for jobs,on Mobile 51% Of all applications submitted on Indeed are mobile 200K Mobile applications completed each day 4X Increase of mobile Indeed Applies in the last year Employers who accept mobile applications receive twice as many quality applicants
- 10. How people applyfor jobs ? Employer website
- 11. How people wantto apply for jobs ? Employer website Employer website
- 12. Typical problems withEmployer careers portals slownon-mobile friendly
- 13.
- 14.
- 15.
- 16. Indeed’s idea -MoBolt Employer website Employer website Mobile friendly website
- 17.
- 18.
- 19. Mobolt Mobile friendly website APIScraping Employer website
- 20.
- 21. Capabilities of Employerwebsites to emulate Jobs 1 Authentication 2 Applications 3
- 22. Capabilities of Employerwebsites to emulate Job Description List of Pages that in that job application flow List of Questions asked in each page Jobs 1
- 23. Capabilities of Employerwebsites to emulate Don’t want to add another authentication Privacy concerns (PII) Authentication 2
- 24. Capabilities of Employerwebsites to emulate Capture all info employer Website expects Privacy concerns (PII) Provide a fallback mechanism if we fail Applications 3
- 25.
- 26.
- 27.
- 28. Web AppJobs Service Employerwebsite Jobs
- 29. Web App Selenium Code JobsService Employer website Jobs
- 30. Job (in JobsDB) [ { “job”: “Installation Technician”, “job_id”: “http://foo.com/jobs/1234”, “page”: “Personal Information”, “label”: “First Name”, “widget_type”: “text_box”, “locator_type”: “xpath”, “locator”: “./first-name” }, ... ] Web application Now understands how to represent this question on UI.
- 31. Web App Selenium Code JobsService Employer website Jobs Auth. Service
- 32. Web App Selenium CodeSeleniumCode Jobs Service Employer website Jobs Auth. Service
- 33. Web App Selenium CodeSeleniumCode Jobs Service Employer website ApplicationsJobs Auth. Service
- 34. Application (in applicationsDB) [ { “job”: “Installation Technician”, “job_id”: “http://foo.com/jobs/1234”, “page”: “Personal Information”, “label”: “First Name”, “widget_type”: “text_box”, “locator_type”: “xpath”, “locator”: “./first-name”, “answer”: “Mitchell Johnson” }, ... ] Web application appends answer given by Job seeker
- 35. Web App Selenium CodeSeleniumCode Jobs Service Apply Service Employer website ApplicationsJobs Auth. Service
- 36. Web App Selenium CodeSeleniumCode Jobs Service Apply Service Employer website Selenium Code ApplicationsJobs Auth. Service
- 37. How Apply Servicereplays the application ? [ { “job”: “Installation Technician”, “job_id”: “http://foo.com/jobs/1234”, “page”: “Personal Information”, “label”: “First Name”, “widget_type”: “text_box”, “locator_type”: “xpath”, “locator”: “./first-name”, “answer”: “Mitchell Johnson” }, ... ] Uses Selenium to load Job URL Locates WebElements (locator_type & locator) Types the answer
- 38. If we doall of what we talked so far perfectly, (at least near perfectly!)
- 39.
- 40.
- 41. How do weextend this for many employer sites?
- 42. Web App Selenium CodeSeleniumCode Jobs Service Apply Service Employer website ApplicationsJobs Auth. Service Selenium Code
- 43. Web App Selenium Code JobsService Apply ServiceApplicationsJobs Auth. Service Selenium CodeS S S S S S Employer Websites
- 44. Web AppJobs ServiceApply ServiceApplicationsJobs Selenium Code C C C C C C backend framework Auth. Service Selenium Code Employer Websites
- 45. Web AppJobs ServiceApply ServiceApplicationsJobs Selenium Code C C C C C C backend framework Auth. Service C C C C C C backend framework Employer Websites
- 46. Web AppJobs ServiceApply ServiceApplicationsJobs Auth. Service C C C C C C backend framework C C C C C C backend framework C C C C C C backend framework Employer Websites
- 47. Web AppJobs ServiceApply ServiceApplicationsJobs C C C C C C backend framework C C C C C C backend framework Auth. Service C C C C C C backend framework Employer Websites
- 48. Auth. ServiceThis ishow we extended Mobolt for many employer sites.
- 49. Some Challenges Stay insync with Employer Website Captcha 1 2 Dependent Questions & Knockout Questions3 Testing our Implementations4
- 50. Current Work Real-time applications Reduceimplementation time 3 1 Reducing drop-off rate, Perform better than Desktop 2
- 52. Sasi Kumar Baratam SoftwareEngineer
- 54. Mobile friendly website JobsService Apply ServiceWeb App Browser Service Mobolt
- 55. Http Browser Service ReturningSessions 1 2 Purge Expired Sessions3 No Single Point of Failure4 Requirements
- 56. State and HealthAPIs Logging and real-time Monitoring 5 6 Resilient to Browser Memory Leaks7 Horizontally Scalable8 Requirements
- 57.
- 58. Native Selenium Grid NodeNode Node Hub
- 59. Native Selenium Grid NodeNode Node HubClient Code
- 60. Native Selenium Grid- Limitations 1 Hub is the single point of failure. If it fails, we lose access to all nodes 2 Difficult to add logging and monitoring 3 Can’t restart node gracefully
- 61. Native Selenium Grid- Limitations 1 Hub is the single point of failure. If it fails, we lose access to all nodes 2 Difficult to add logging and monitoring 3 Can’t restart node gracefully
- 62. Native Selenium Grid- Limitations 1 Hub is the single point of failure. If it fails, we lose access to all nodes 2 Difficult to add logging and monitoring 3 Can’t restart node gracefully
- 63. Mobolt Selenium Grid ClientCode Grid Servers AWS ELB
- 64. Mobolt Selenium Grid ClientCode M Grid Servers Grid Node 1 Maintenance scripts Selenium Server Grid Node n Maintenance scripts Selenium Server AWS ELB
- 65. Mobolt Selenium Grid ClientCode M Grid SchedulerGrid Servers Grid Node 1 Maintenance scripts Selenium Server Grid Node n Maintenance scripts Selenium Server AWS ELB
- 66. Grid Servers Web Server bundledwith JDK
- 67. Grid Servers Web Server bundledwith JDK State, Node, Session, Health Http endpoints
- 68. Grid Servers Web Server bundledwith JDK State, Node, Session, Health Http endpoints Maintains inventory of Nodes and Sessions in Mongodb
- 69. Selenium Server createsbrowser based on Grid Server request Grid Node Maintenance scripts Selenium server
- 70. Selenium Server createsbrowser based on Grid Server request Set of shell scripts which monitor node health Grid Node Maintenance scripts Selenium server
- 71. Selenium Server createsbrowser based on Grid Server request Set of shell scripts which monitor node health Restart node if available memory or available sessions is too low Grid Node Maintenance scripts Selenium server
- 72. Selenium Server createsbrowser based on Grid Server request Set of shell scripts which monitor node health Restart node if available memory or available sessions is too low Kill unresponsive browsers Grid Node Maintenance scripts Selenium server
- 73. Grid Scheduler HeartBeat - Pingsnodes to check if they are running or not
- 74. HeartBeat - Pingsnodes to check if they are running or not Purging Expired Browsers Grid Scheduler
- 75. Machine Configuration (AWS) 5Grid Servers (2vCPU/3.75G memory, ec2-classic) 25 Grid Nodes (2vCPU/15.75G memory, ubuntu, xvfb-enabled)
- 76. Comparative Analysis Architecture HeartBeat Returning Sessions PurgeExpired Sessions State, Health APIs No Single point of failure Logging and Monitoring Resilient to memory leaks Graceful Node restart Horizontally Scalable Native Selenium Grid Tree of Grid Nodes Mobolt Selenium Grid Bipartite graph
- 77. Challenges Rogue Sessions Sometimes browsersdon’t respond as expected Gateway timeouts If session creation takes more than 1 minute, then we respond with timeout error.
- 78. Challenges Rogue Sessions Sometimes browsersdon’t respond as expected Gateway timeouts If session creation takes more than 1 minute, then we respond with timeout error.
- 79. Grid Traffic Statistics ~625 Max req per min ~ 89K Max req per day ~ 2.0M Total req in last 30 days ~ 8.7M Total req in last 1 year
- 80.
- 81. Future Work Browser Pool Maintaina pool of browsers and allocate browsers from pool Autoscale Autoscale grid nodes and servers
- 82. Future Work Browser Pool Maintaina pool of browsers and allocate browsers from pool Autoscale Autoscale grid nodes and servers
- 83. Engineering blog &talks - indeed.tech Open Source - opensource.indeedeng.io Careers - indeed.jobs Twitter - @IndeedEng Learn More
- 84.
- 85.
- 86. How is CaptchaHandled?
- 87. Employer Page Sign Up Submit MoboltPage Username Password
- 88. Employer Page Captcha Image SignUp Submit Selenium Code Mobolt Page Username Password Other questions
- 89. Employer Page Captcha Image SignUp Submit Selenium Code Mobolt Page Sign Up Username Password Submit Username Password Other questions
- 90. Employer Page Captcha Image SignUp Submit Selenium Code Mobolt Page Sign Up Username Password blah blah V4XBG Submit Username Password Other questions
- 91. Employer Page Captcha Image SignUp Submit Selenium Code Selenium Code Mobolt Page Sign Up Username Password blah blah V4XBG Submit Username Password
- 92. Challenges in scrapingmodel?
- 93. Stay in syncwith Employer website Jobs get added and deleted Page flow for a job application CSS Selectors / Xpaths change New questions get added OR questions get updated Job Seekers use a password with us Then change password on the ATS
- 94. Why not curlbased crawling?
- 95. Dynamic things happen onemployer websites Dependent questions Job dependent flows It is difficult and in some cases impossible to scrape using curl. We want to write content to the web pages Then change password on the ATS
Editor's Notes
- #5 Generally Selenium is used more in the context of Test Automation
- #6 In this talk, We are going to see ….
- #8 Monica will design
- #9 At Indeed we are hiring all the time and are growing quickly. Over the past two years, we’ve gone from adding a few new developers per month to having up to 50 start on a single day. This rapid growth has added even more pressure onto the setup process.
- #10 At Indeed we also gathered some stats about number of people applying for jobs on Mobile.
- #45 When you do a similar thing many number of times, there is an opportunity to abstract out the core repetitive part. We called that backend-framework.
- #50 Build in as much automatic recovery as possible.
- #54 by ensuring that our job data is complete and current
- #62 Test
- #67 com.sun.net.httpserver.HttpServer
- #77 Move this to after slide 60
- #84 If that sounds interesting to you, you should follow our engineering blog, github, and social media to learn more.
- #86 If that sounds interesting to you, you should follow our engineering blog, github, and social media to learn more.