4. •User requirement
• A desktop application for scraping Odds and OnlineList from rental site
• Total 1, 652 links include 16,5200 records
• Data format arrange and export to a excel file
•Non-function requirement
• Avoid to lock account by action as similar as DDOS
Background
6. Behavior Analysis
•Website need Login first
•Login session will keep alive if you idle in
MainPage(timely sync request post from client)
•After login to MainPage, each click open Pop-
up window to display
•Each data page will display 100 records by
filter you give
7. System Analysis
•Security
• Website need login to get SessionId and StickyId for request
• Website have security mechanism to redirect invalid request
• Using one time token to avoid user request page data without permission
when login to Main page
• All sub-page(pop-up window) only allow open from Main page
• Website using RESTful-like routing include UserSession token
•Routing and Request Post
• URL routing included Login Token
• MainPage routing included BuildVersion
• Request need add Query key(it pass from Main window) for Odds and
OnlineList service
8. Chanllege
•Issue 1
• Too many links to scraping if using Selenium or other similar solution.
•Issue 2
• Some data need using JavaScript to decrypt and re-generate(RSA token, one
time token and etc…).
•Issue 3
• Need capture response header(Session and StickyId) to mock the request to
query the Odds an OnlineList service.
10. Use of Technology
•C# and .Net framework
•AutoIt(Download)
• AutoIt v3 is a freeware BASIC-like scripting language designed for
automating the Windows GUI and general scripting.
• Have script Editor to build up the script
• Can execute ShellScript
• Can compile script to .exe file
•AutoItX aka NAutoIt(Download)
• Methods available to AutoIt BASIC, but not provided via AutoItX, are
replaced by .NET counterparts.
• AutoItX with PowerShell, .NET, C, COM, COM interop and reg free COM
interfaces.