Automate Data Scraping and Extraction for Web
•Download as PPTX, PDF•
0 likes•494 views
The data you use every day comes from so many places: websites, Excel files, PDFs, CSV reports, databases, emails, and more. If you add up all your data-related tasks, like extracting information for reporting and analysis or manual data entry, you’re probably using up a lot of valuable time. Automate’s data scraping automation capabilities allow you to read, write, and update a wide variety of data sources automatically. In this webinar you'll learn how you can save time and increase the accuracy of your data-driven processes, allowing your employees to focus on more important things like meeting business goals and providing great service.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Automate Data Scraping and Extraction for Web
Similar to Automate Data Scraping and Extraction for Web (20)
More from HelpSystems
More from HelpSystems (20)
Recently uploaded
Recently uploaded (20)
Automate Data Scraping and Extraction for Web
- 1. Back to Basics Webinar Series: Automate Data Scraping and Extraction for Web and More
- 2. Today’s Presenters Director of Document Management Richard Schoen HelpSystems Pat Cameron Director of Automation Technology HelpSystems
- 3. HelpSystems. All rights reserved. Today’s Agenda 1. HelpSystems overview 2. Painful data entry and extraction processes 3. Automation use cases 4. Introduction to Automate capabilities 5. Demo 6. Q&A
- 4. Broad Solutions in Growing Markets Secure • Risk Assessment • Anti-virus • Security Event Monitoring • Identity & Access Management • Compliance Reporting • Managed Security Services • Professional Security Services • Managed File Transfer • Encryption Inform • Enterprise Data Access • Mobile Data Access • Operations Analytics • Executive Dashboards & Reporting • Data Warehousing Automate • Workload Automation • Business Process Automation • Network Monitoring • Message & Event Monitoring • Performance Monitoring • Data Backup Management • Remote Monitoring & Management • Capacity Planning • Document Management
- 5. HelpSystems. All rights reserved. Painful Data Entry and Extraction Processes
- 6. • Daily repetitive data entry tasks • Reformatting data from CSV to Excel and other formats • Entering same data to more than one system • Copying and pasting information • Re-saving to new CSV or Excel files • Edge tasks not easily trainable for new employees • Wasting time Common Daily Data Entry and Automation Problems
- 7. The Received Report Processing Pattern Report, CSV, or Excel file received Lookup, review, or copy/paste performed Data entered or updated into one or more systems Report, CSV, or Excel file is saved or archived
- 8. The Received Report Processing Pattern Report, CSV, or Excel file received Lookup, review, or copy/paste performed Data entered or updated into one or more systems Report, CSV, or Excel file is saved or archivedX X X
- 9. IT Departments • Running scheduled jobs • File transfers • Data extractions • DevOps automation Business Departments • Automate repetitive daily tasks • Eliminate copy/paste of data • Robotic Process Automation Departments With Automation Needs
- 10. • Do any of your users spend 30-60 minutes or more per day on a manual process? • Do workers spend time entering data from CSV or spreadsheets today? • Can process information be queued for scalable processing? • Is data entered to more than one system? • Can the process access a database? • Can the process be done without a human being making a decision or reviewing data? How to Identify High-Value Repeatable Processes
- 11. HelpSystems. All rights reserved. Data Extraction Use Cases
- 12. Extract Ticker Data from Google Finance Website to Excel
- 13. Transfer Files, Read and Process Automatically • Upload • Download • Copy • Move • Process • Read • Write • Archive • Delete • Replicate • Convert • FTP • SFTP • FTPS • Network • Local Automate
- 14. Query, Summarize and Export or Update Another Database Source Database Table Export or Update Destination Data Warehouse Table(s) Perform ETL Logic Automate
- 15. Read Database Table and Write Report to Excel or CSV • Customers • Orders • Inventory • Vendors • Parts Export to Excel, CSV, XML or Text Automate
- 16. Reading Excel and CSV and Importing to Database • Customers • Orders • Inventory • Vendors • Parts Import Excel and CSV Files Process, Validate and Write to Database Automate
- 17. Capture And Process Inbound Email Documents Read Inbox, Validate, Process Email Info and Attachments Automate
- 18. Automate Capturing Documents for Document Management • Use file name • Extract file metadata • Extract ERP index values • Store in doc management WebDocs
- 20. • Process requirements gathering • Create an outline of manual steps • What does the human do? • How much can we automate? • Process is consistent • Follows a general predictable pattern • Can be automated Approach to Defining Automation Tasks
- 21. HelpSystems. All rights reserved. Introduction to Automate
- 22. Our Differentiation Start anywhere, realize value today, and expand anytime. RAPID TIME TO VALUE LOWER TOTAL COST OF OWNERSHIP COMPLETE SOLUTION BRIDGING IT AND BUSINESS 600+ pre-built actions Drag-and-drop builder Visual workflow designer Priced as software—not digital workers Low training burden Low scaling costs Automation across the full spectrum of enterprise needs Enabling platform that is powerful enough for IT but intuitive for business users Customer Value Support & Maintenance Model Build NetValue Scaling Costs (Hardware & Software) Reliability Purchase Price Time to Value Functionality Usability & Training
- 23. Complete Solution Offering BUSINESS PRODUCTIVITY IT PRODUCTIVITY Individuals & Teams Business Applications Systems & Infrastructure Front-of-Glass Automation Back-of-Glass Automation Document Management Enterprise Job Scheduling Secure FTP Monitoring Start anywhere. Realize value today. Expand anytime.
- 24. Bridging IT and Business • Low-code Environment • Reusable Components • Extensive Online User Support • Simple Deployment • Web Scraping & Interactivity • Back of Glass: APIs, Web Services, SOAP/REST • MS Integrations: Outlook, Excel, SharePoint • Cloud Capable: AWS, Azure • PDF Reader/OCR • Rules, Variables, Calculations, Exception Handling, and Triggers • Scale from POC to Enterprise Deployment • 3-Tier Architecture • Agent Groups • Centralized and Secure Repository • Role-Based Security • Full Audit Trail • Secure Deployment • Supports DevOps Approach • Operations Console • Active Directory (Role-based Access Control)
- 25. Rapid Time to Value No Code 600+ drag-and-drop actions with business designed easy-to-use interface Visual Design Workflow designer allows for process visualization allowing optimizations to happen on the fly Extensibility Universal connector provides organization extensibility to build connections today and tomorrow Reusable Components Easily build a library of templates that can enable automation build speed
- 26. HelpSystems. All rights reserved. Demo Time
- 27. HelpSystems. All rights reserved. Questions and Answers
- 28. Thank you for attending! Next steps: Don’t forget to attend next week’s webinar – the last in the series: Automate Best Practices Thursday, Sept. 27 Website: http://www.helpsystems.com/automate Telephone: US Sales: 800-328-1000 Outside US: +44 (0) 870.120.3148 Technical Experts: richard.schoen@helpsystems.com pat.cameron@helpsystems.com
Editor's Notes
- Good Morning Everyone and welcome to our live webinar. Today is February 15th I’m Richard Schoen coming to you from our offices in Eden Prairie MN I’ll be the moderator today for our webinar titled: Automate Data Scraping and Extraction for Web and More Today’s 30 minute session will provide an introduction to some of the ways in which customers are using our AutoMate software to streamline their daily operations by automating web site scraping and data extraction and entry to prevent double-keying or re-keying of data. Automating the user interface is just one method of streamlining. We will explore several of the different methods available in the Automate automation platform that can be used to create automated and robotic processes for data entry and extraction. Hopefully this session will get you thinking about ways your team can save time and money in 2017 by implementing process automation.
- As mentioned I am Richard Schoen, Director of Document Management Technologies at HelpSystems. I am part of the technical solutions group at HelpSystems bringing topics like this to our customers and prospective customers. I have over 28 years of background with IBMi, Windows and Linux platform software development, system integration, managing and delivering forms and documents helping customers automate key processes. My co-host is Pat Cameron who is our director of automation technology. Pat why don’t you say hello and tell us a little bit about your role here at HelpSystems.
- Our session today will hopefully provide a good introduction to HelpSystems and some of the ways we can streamline your users daily manual date entry and extraction workloads. We’ll provide a brief overview of HelpSystems and its history. Then we’ll talk about the common business reasons companies want to automate data extraction automation tasks. Then we’ll provide an overview of the AutoMate software and related database actions. We will end with a technology demo and a few minutes of Q&A and a couple of polling questions. Feel free to enter your questions in the chat window as we go and we will address them towards the end of the webinar. Select “All Presenters” so the questions are directed to both me and Pat. We’ll also plan to complete our session in 30-40 minutes so you have plenty of time to make your next important meeting. Also todays event is being recorded and you will receive a link after the webinar to share with anyone in your organization who couldn’t attend todays session. Polling: What type of database interactions do you plan to do with AutoMate ? (Data extraction to Excel and other formats, data extraction for FTP purposes, data extraction for analysis, import data from Excel files, import data from CSV, XML or text files, I don’t use AutoMate yet so I’m not sure.) What type of automation user are you ? (End User, IT Admin User, IT Developer, Business Management, IT Management) Will you or your team build your own automation tasks ? (Yes/No) Are you already a HelpSystems customer ? (Y/N)
- Before we get things rolling, a little introduction to HelpSystems and what we do. HelpSystems has been in business for over 35 years providing solutions for every day business automation needs. Our solutions help customers Automate their daily operations including: system management, network and infrastructure monitoring, business and desktop process automation and report and document management. We also help customers Secure their networks and business systems from cyber attacks and help companies maintain compliance. And we help keep users and management Informed all day long through the use of our business reporting, data warehousing and executive dashboarding software products. Our solutions can be deployed across multiple platforms including IBMi, Windows, Linux and AIX.
- So let’s talk about some of the painful data entry and extraction processes your teams are dealing with.
- TODO – Complete. Maybe talk about following points. Might be in use cases. Types of data extraction: Excel, CSV, Text Database SharePoint Web Sites Within our organizations users are performing many daily repetitive tasks. Some tasks make sense such as taking and entering orders via phone or performing customer service requests where human interaction is required. Many of our manual daily tasks have been inherited because of shortcomings in our ERP or other business systems. Edge tasks that involve custom processing of data using CSV, Excel and other utilities are often good candidates to be automated. These tasks are often tedious and involve repetitive actions such as re-keying or copying and pasting of data and are prone to mistakes. These types of tasks can also be difficult to train employees to do because they involve business rules that might seem odd or not easily documentable for the un-initiated. By automating these tasks we provide consistent, repeatable and easily documentable processes so it becomes easier to focus on our actual jobs of keeping customers, vendors and employees happy.
- I call this work pattern the Received Report Processing Pattern. Or you could call at it: the review, manipulate, repeat and get very bored pattern. Either way this is a mind-numbing way to process repetitive data. This is a scenario where a report or work is received in a CSV or text file, Excel worksheet or a PDF document and all the items in the report need to be reviewed and processed. The user opens the report and prints it or starts working through the data. Hopefully they have at least two monitors to work with their data. For each entry they perform some data entry or other work (possibly in another system) or maybe data gets copied and pasted to a new report or spreadsheet. When the work has been completed for the list, the original report and any new documents may get saved for archival and audit purposes. And maybe the final results get uploaded to another system for back end processing. The mouse was invented to be helpful for interacting with the desktop, but this is not the way to be doing important daily work unless there’s no other way. Fortunately there is.
- Since this report processing task is driven by a consistent, repeatable list of information it’s a good candidate for robotic process automation and eliminating all the manual steps. Now when the report gets received, the user can save the incoming report file to an input folder for automated processing. Or if the report comes in an email, an email box process monitor can grab the report and save it to the input folder to get automatically processed which means even the first step could be automated even though we’re showing it here because many people like to still manually receive the files until the process has been fully tested and trusted. Assuming the review and data steps we eliminated were repeatable, the new process simply runs in the background via an AutoMate robotic process agent and then the results are automatically entered or updated into the back end systems and the files are archived and audit trails created during the process to document what work the robotic process completed. And if process notifications are required the process can send out notification and completion emails, update database information or other systems to audit the work that was completed so all business systems stay informed of the work being done automatically. This is a great way to improve the daily redundant work processes done across various departments within the business.
- There are two main areas where the Automate software can be used to implement automation IT departments In IT departments automate can be used to schedule and run back office business processes and workflows. These are IT processes normally done during the day or run after hours to keep the business operational. This might include report generation, email automation, reading and writing database, network and service monitoring and more. The software can also be used to interact with and automate many development operation processes. This is commonly called DevOps. End user departments In end user business departments Automate is often used to automate the time consuming data entry process and data extraction processes normally done by users utilizing data entry into Windows or web applications or by copying and pasting data between applications. Eliminating or reducing data entry or entering the same data into multiple systems is a key benefit to Automate that can quickly justify the software implementation. Who should build automation processes ? In an end user department this may include someone with business analysis or macro building skills. These people are usually familiar with analyzing and implementing new processes. In the IT areas the creators of automation processes can include operations and network administrators as well as developers only if needed.
- There is one main question to ask that can quickly help identify areas for robotic process automation in any department within your business. “Do any of your users spend 30-60 minutes or more today directly interacting with any applications or web sites to download or process information or are they doing lots of copy and paste data manipulation ?” If the answer is yes then you’ve taken your first steps to start identifying key processes ready for automation. Now the trick is to articulate the process on paper and start thinking about how to automate the process.
- This is an example that uses website interactivity to capture information from the google finance site and places the current stock price along with timings for service level tracking into the final spreadsheet based on the original template. After completing the data extraction, the Excel information can be emailed, uploaded to Sharepoint or FTP site or automatically entered into another interactive business or database system line by line. Web interactivity can be used with almost any web site to automate data extraction or data entry processes.
- This scenario is all too common for ETL and Data Warehousing scenarios. Imaging being able to quickly write a workflow to read a table form one database, summarize the contents and write the data to a second database table. You can also join in additional information along the way because you can use standard SQL syntax to read, summarize and write information between databases. Workflows can also be triggered by information changing in a database such as Oracle or SQL Server as well.
- Exporting lists of daily information to Excel, CSV and other formats for daily reporting, daily processing or dashboarding is another common need our users have. Imagine setting up a simple workflow to query the open orders database each morning at 7am and having that information waiting in your email box, SharePoint directory, network folder or document management system.
- Erp, Inboud AP Invoices,
- One way to keep teams apprised of what’s going on throughout the day is to use an auto-refreshing dashboard. This example shows a nice looking simple HTML dashboard template that can be automatically updated and displayed all day long on a small or large monitor to keep track of what’s happening. One area this could be used is on the shop floor or some other production area where 70-80” large screen monitors might be used as a board to track daily production process. Every few minutes the dashboard can self-update with information so the large screen progress monitors are kept up to date automatically all day long without anyone doing anything. I recently talked to someone who was going to be placing monitors at ten or more locations throughout their plant and all the large screens would be driven be a simple auto-updating HTML template and data extracted using AutoMate every few minutes. Another use might be to provide a daily browser based management snapshot of information with key metrics or to possibly create a self-updating information kiosk in the front lobby of your building to welcome visitors as they arrive. Either way creating self-updating dashboards is a great way to provide key visual metrics information on-the-fly.
- Let’s take a quick look at the requirements gathering process when starting to automate a new task. This process can be used for any task you want to automate. I always ask people to create a simple bullet list or outline of the steps for a desired automation task. In other words: what does the human doing the work do today ? Based on assumptions and talking it through, you have to determine can a chosen task by systemized for automation ? Ex: Can task inputs be driven from an email message, file or database ? What kind of decisions have to be made in the task and does the task need manual input most of the time or not ? Is the process consistent, predictable and repeatable every time ? If so then chances are the process can be automated whether it’s an email task, a browser interaction task, file transfer or some other automation process. Gathering task business requirements not only helps your team determine the manual steps that need to be automated, but you can decide whether your internal team members have the skills to build automation tasks or whether it’s more efficient to utilize an outside service team to build out automation tasks for you.
- Now we’re going to take a quick look at some AutoMate samples. Sample 1- Read a Database With Sql and Extract to Excel Sample 2 – Read Access Database Table1 Write To Table2 Sample 3 – Read Database With Sql And Extract To CSV Sample 4 – Read CSV File And Write To Access Table2 Sample 5 - IBMi-Run Stored Procedure-TESTCL1
- OK, now we’re going to take time for a little Q&A. When entering questions, please remember to Select “All Presenters” so your questions are directed to me. I will also now display a short poll that we can share with the group, so if you don’t mind answering that would be great. Which database drivers are supported with Automate ? Pretty much any driver that you can use to build a connection string. This would include MS Access, SQL Server, Oracle, MySQL, IBMi, etc. As long as you have a valid database driver available then Automate should be able to work with it. You will need to be careful and use 64-bit drivers if using the x64 version of automate and use 32-bit drivers if using the x86 version of automate. In the case of IBMi I generally recommend users utilize the x86 version because the client access database drivers are 32-bit. When I create a new automation task, what’s the best way to trigger it ? It all depends. If an end user needs to run a task from their desktop they may need an Automate agent installed and they can trigger a task interactively. If a task will use a database or other file as an input source the task can be scheduled to run every few minutes or once per day as required or can be triggered with the database trigger we discussed earlier. The nice thing about building tasks in Automate is that you don’t have to worry about building polling or scheduling into your task. You design tasks and workflows as single pass and the triggering mechanisms in Automate handle polling and other task kickoff methods for you . We receive daily order data via email in Excel and CSV format and need to import to our ERP system. How would you design a task to do that. In this case I would create a task that polls an email box regularly and processes any inbound emails that contain attachments. As an email comes in the automation task would grab any file attachments and then read the contents of the file and place data in a directory for import by the ERP system or possibly import the data directly to an import database table before processing. An example of this would be JD Edwards has file that they call Z-Files which are mean to receive data and then JDE processes and validates the files before importing data. We have to regularly log into a trading partner web site to download files for processing. How might you design this process ? Many trading partners are using services such as Ariba or they have their own home grown sites where files are downloaded from rather than being directly emailed or set up on an FTP site for download. While Automate can handle pulling or pushing files from FTP or email, web site interaction is becoming more common. The process might look like this: your team receives a daily notification of available file downloads via email. Then someone logs into the web site, navigates to the appropriate page and then downloads a file and the processes it. This entire sequence can be automated using a combination of our email actions to monitor a mailbox, GUI interaction actions to log in to a web site and finally the file system, Excel and database actions to import and process the data. This pattern can be applicable to processing inbound order information, accounts payable invoices and more.
- Thank you for attending our webinar today. We hope you learned some helpful information about how you can use AutoMate to streamline many if not all of your tedious Excel automation and processing work to keep your employees from going insane. If you have an Excel task you’re looking to automate feel free to schedule a free personalized demo and our team can show you the best way to automate your most painful Excel based processes. Simply visit helpsystem.com/automate and fill out a request form today. Also if you have any additional questions on Automate or any of our other software products, please reach out to the sales team or feel free to email me or pat and we can connect you with the appropriate sales or support team members. You will also receive a link to this recording so you can share this webinar with those in your company who could not attend today’s session. Again thank you for attending todays webinar. Have a great day and enjoy the rest of your week. Pause for a moment. Stop screen sharing Save polling answers.