The document describes the evolution of a web scraping project using Ruby from a simple initial implementation to a robust production system. It started as a quick script but grew to handle logging in, extracting data from multiple pages, error handling, caching, and performance improvements like using a proxy. Testing and refactoring helped increase confidence and maintainability. The system was eventually able to replicate most of the target website's features, handling high volumes of traffic through caching and other optimizations.
With more and more sites falling victim to data theft, you've probably read the list of things (not) to do to write secure code. But what else should you do to make sure your code and the rest of your web stack is secure ? In this tutorial we'll go through the basic and more advanced techniques of securing your web and database servers, securing your backend PHP code and your frontend javascript code. We'll also look at how you can build code that detects and blocks intrusion attempts and a bunch of other tips and tricks to make sure your customer data stays secure.
I put on my mink and wizard behat (tutorial)xsist10
An indepth dive into using Behat/Mink/Selenium for BDD testing.
* http://behat.org
* http://mink.behat.org/
* http://docs.seleniumhq.org/
In this talk I'll cover:
* why and when to use Behat (and when not)
* Installation and configuration of Behat and Mink
* Building Behat Contexts
* Avoiding data deadlocks and "test user account" syndrome
* Introduction to Selenium and testing JavaScript
* Best practises for writing tests (what to avoid, what to aspire for, writing stories like you mean it, how to get your product owners to write them)
* Common gotchas
With more and more sites falling victim to data theft, you've probably read the list of things (not) to do to write secure code. But what else should you do to make sure your code and the rest of your web stack is secure ? In this tutorial we'll go through the basic and more advanced techniques of securing your web and database servers, securing your backend PHP code and your frontend javascript code. We'll also look at how you can build code that detects and blocks intrusion attempts and a bunch of other tips and tricks to make sure your customer data stays secure.
I put on my mink and wizard behat (tutorial)xsist10
An indepth dive into using Behat/Mink/Selenium for BDD testing.
* http://behat.org
* http://mink.behat.org/
* http://docs.seleniumhq.org/
In this talk I'll cover:
* why and when to use Behat (and when not)
* Installation and configuration of Behat and Mink
* Building Behat Contexts
* Avoiding data deadlocks and "test user account" syndrome
* Introduction to Selenium and testing JavaScript
* Best practises for writing tests (what to avoid, what to aspire for, writing stories like you mean it, how to get your product owners to write them)
* Common gotchas
This is my speech at PHPDAY2011
WordPress for developers:
WordPress features overview
Custom type and Taxonomy
Routing and rewrite rules
Custom query and manipulation
Cache tips
Debugging tools
Rails, Postgres, Angular, and Bootstrap: The Power StackDavid Copeland
Rails, Postgres, Angular, and Bootstrap are all powerful technologies in and of themselves. When used together, however, you get a powerful web application development stack that reduces friction between what you want for your users and what you can deliver.
With more and more sites falling victim to data theft, you've probably read the list of things (not) to do to write secure code. But what else should you do to make sure your code and the rest of your web stack is secure ? In this tutorial we'll go through the basic and more advanced techniques of securing your web and database servers, securing your backend PHP code and your frontend javascript code. We'll also look at how you can build code that detects and blocks intrusion attempts and a bunch of other tips and tricks to make sure your customer data stays secure.
https://speakerdeck.com/willroth/50-laravel-tricks-in-50-minutes - origin
Laravel 5.1 raised the bar for framework documentation, but there's much, much more lurking beneath the surface. In this 50-minute session, we'll explore 50 (yes, 50!) high-leverage implementation tips & tricks that you just won't find in the docs: the IoC Container, Blade, Eloquent, Middleware, Routing, Commands, Queues, Events, Caching — we'll cover them all! Join us as we drink from the fire hose & learn to take advantage of everything that Laravel has to offer to build better software faster!
Come to this talk prepared to learn about the Doctrine PHP open source project. The Doctrine project has been around for over a decade and has evolved from database abstraction software that dates back to the PEAR days. The packages provided by the Doctrine project have been downloaded almost 500 million times from packagist. In this talk we will take you through how to get started with Doctrine and how to take advantage of some of the more advanced features.
This is my speech at PHPDAY2011
WordPress for developers:
WordPress features overview
Custom type and Taxonomy
Routing and rewrite rules
Custom query and manipulation
Cache tips
Debugging tools
Rails, Postgres, Angular, and Bootstrap: The Power StackDavid Copeland
Rails, Postgres, Angular, and Bootstrap are all powerful technologies in and of themselves. When used together, however, you get a powerful web application development stack that reduces friction between what you want for your users and what you can deliver.
With more and more sites falling victim to data theft, you've probably read the list of things (not) to do to write secure code. But what else should you do to make sure your code and the rest of your web stack is secure ? In this tutorial we'll go through the basic and more advanced techniques of securing your web and database servers, securing your backend PHP code and your frontend javascript code. We'll also look at how you can build code that detects and blocks intrusion attempts and a bunch of other tips and tricks to make sure your customer data stays secure.
https://speakerdeck.com/willroth/50-laravel-tricks-in-50-minutes - origin
Laravel 5.1 raised the bar for framework documentation, but there's much, much more lurking beneath the surface. In this 50-minute session, we'll explore 50 (yes, 50!) high-leverage implementation tips & tricks that you just won't find in the docs: the IoC Container, Blade, Eloquent, Middleware, Routing, Commands, Queues, Events, Caching — we'll cover them all! Join us as we drink from the fire hose & learn to take advantage of everything that Laravel has to offer to build better software faster!
Come to this talk prepared to learn about the Doctrine PHP open source project. The Doctrine project has been around for over a decade and has evolved from database abstraction software that dates back to the PEAR days. The packages provided by the Doctrine project have been downloaded almost 500 million times from packagist. In this talk we will take you through how to get started with Doctrine and how to take advantage of some of the more advanced features.
URLs are not meant to be given to the clients but they should be discovered through the interaction with the server inside the representation of the resources. But thinking about URLs it's like thinking about names and relations between your domain resources. Creating a good URL grammar helps a good resource design. A good resource design is more stable and more extensible.
Node.js is one of those technologies that should not exist. Definitely, theoretically, is not supposed to have this kind of success. But like the bumblebee he don't know he can't and so it goes :-)
This talk I gave at the 2017 madewithlove retreat (https://2017.mwl.be/) and also at the Maceió DEV Meetup #21 (https://www.meetup.com/pt-BR/maceio-dev-meetup/events/245530346/)
The idea is to compare building the same application in two web frameworks in different languages. One I know and work daily, which is Laravel, and the other one is new to me and I had no experience in Elixir, the language the framework was built in.
The idea is not to compare both languages or frameworks. My goal was mainly learn something new building the same thing twice and see how different languages and frameworks can be used to build similar things.
---
## Slides links
Slide 13:
- https://www.youtube.com/watch?v=7Zlp9rKHGD4
Slide 18:
- https://www.youtube.com/watch?v=aA-XHI-EYcM
Slide 20:
- https://github.com/tonysm/slackish-laravel
- https://github.com/tonysm/slackish-phoenix
Slide 32:
- http://phoenixframework.org/blog/the-road-to-2-million-websocket-connections
Slide 38:
- https://www.youtube.com/watch?v=H686MDn4Lo8
---
## References for this talk
Functional Programming; What? Why? When? (Robert C Martin) https://www.youtube.com/watch?v=7Zlp9rKHGD4
Real World Elixir Deployment (Pete Gamache) https://www.youtube.com/watch?v=H686MDn4Lo8
Erlang: The Movie https://www.youtube.com/watch?v=xrIjfIjssLE
Lonestar ElixirConf 2017- KEYNOTE: Phoenix 1.3 by Chris McCord https://www.youtube.com/watch?v=tMO28ar0lW8
GOTO 2016 • Phoenix a Web Framework for the New Web • José Valim https://www.youtube.com/watch?v=bk3icU8iIto
ElixirConf 2016 - Giving up the Object-Oriented Ghost by Morgan Lanco https://www.youtube.com/watch?v=_VpZ6gQsyDY
GOTO 2017 • Elixir: The only Sane Choice in an Insane World • Brian Cardarella https://www.youtube.com/watch?v=gom6nEvtl3U
Elixir, quem é esse pokemon? - Bruno Volcov https://www.youtube.com/watch?v=aA-XHI-EYcM
Ecto, você sabe o que é? - Amanda Sposito https://www.youtube.com/watch?v=hQM4VdEpz6g
Programming Phoenix (book) by Chris McCord, Bruce Tate, and José Valim https://pragprog.com/book/phoenix/programming-phoenix
Learn the ins and outs of running background tasks with the popular python module Celery. We'll hit the ground running. With everything you need to know to run your first task, to scaling your stack to run millions each day.
A few techniques for everyday Ruby hacking
Touching on the following topics:
DRY Assignment
Ternary operator
Bang bang
Conditional assignment
Parallel assignment
Multiple return
Implied begin
Exception lists
Symbol to Proc
MapReduce
Regex captures
tap
sprintf
case equality
Splat Array
Splat args
blank?
present?
presence
truncate
try
in?
Delegation
delegate
Memoization
memoize
alias_method_chain
class_attribute
HashWithIndifferentAccess
Lithium: The Framework for People Who Hate FrameworksNate Abele
This is the presentation was given at ConFoo on March 11th by Nate Abele and Joël Perras, and is an introduction to the architectural problems with other frameworks that Lithium was designed to address, and how it addresses them. It also introduces programming paradigms like functional and aspect-oriented programming which address issues that OOP doesn't account for.
Finally, the talk provides a quick overview of the innovative and unparalleled features that Lithium provides, including the data layer, which supports both relational and non-relational databases.
MongoDB is the trusted document store we turn to when we have tough data store problems to solve. For this talk we are going to go a little bit off the path and explore what other roles we can fit MongoDB into. Others have discussed how to turn MongoDB’s capped collections into a publish/subscribe server. We stretch that a little further and turn MongoDB into a full fledged broker with both publish/subscribe and queue semantics, and a the ability to mix them. We will provide code and a running demo of the queue producers and consumers. Next we will turn to coordination services: We will explore the fundamental features and show how to implement them using MongoDB as the storage engine. Again we will show the code and demo the coordination of multiple applications.
Callbacks, Promises, and Coroutines (oh my!): Asynchronous Programming Patter...Domenic Denicola
This talk takes a deep dive into asynchronous programming patterns and practices, with an emphasis on the promise pattern.
We go through the basics of the event loop, highlighting the drawbacks of asynchronous programming in a naive callback style. Fortunately, we can use the magic of promises to escape from callback hell with a powerful and unified interface for async APIs. Finally, we take a quick look at the possibilities for using coroutines both in current and future (ECMAScript Harmony) JavaScript.
What went wrong for my clients in the past 6 years trying to implement Microservice Architectures? This is a retrospective, a list of things we must to avoid to gainable with this kind of software architecture.
Not so many years have passed since we started programming computers and even less since programming computers has been recognised as a profession. Even still, so many things depend on the quality of our work. What does it mean to be professional? What are we expected to do? Are we up for the task? I will talk about my journey of becoming the programmer of my dreams, the obstacles I've faced and the strategies that I've applied to overcome them
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Generating a custom Ruby SDK for your web service or Rails API using Smithyg2nightmarescribd
Have you ever wanted a Ruby client API to communicate with your web service? Smithy is a protocol-agnostic language for defining services and SDKs. Smithy Ruby is an implementation of Smithy that generates a Ruby SDK using a Smithy model. In this talk, we will explore Smithy and Smithy Ruby to learn how to generate custom feature-rich SDKs that can communicate with any web service, such as a Rails JSON API.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
7. enter mechanize + nokogiri
require "mechanize"
agent = Mechanize.new do | agent |
agent.user_agent_alias = "Linux Mozilla"
end
agent.get("http://example.com/login") do | login_page |
result_page = login_page.form_with(:name => "login") do | login_form |
login["username"] = username
login["password"] = password
end.submit
result_page.search("//table[starts-with(@class,'boundaries')]").map do | option_table |
{ "name" => option_table.search("./caption/child::text()")
"credits" => option_table.search("./descendant::td[position()=3]/child::text()")
}
end
end
8. and then...
good, but
we’d like to
extract more
informations
from a few
different
pages
9. enter commander
describe arguments
command :is_registered do | command |
command.syntax = "is_registered --username TELEPHONE_NUMBER [ --without-cache ]"
command.description = "Check if user is registered"
command.option "-u", "--username TELEPHONE_NUMBER", String, "user's telephone number"
command.option "-n", "--without-cache", "bypass user's profile informations cache"
command.when_called do | arguments, options |
options.default :username => "", :without_cache => false
ok(is_registered(options.username, options.without_cache))
end
end
extract code into functions
10. use page object pattern
def is_registered(username)
browse do | agent, configuration |
LoginPage.new(
agent.get(configuration["login_page_url"])
).is_registered?(username)
end
end
11. use page object pattern
class LoginPage < PageToScrub
def is_registered?(username)
begin
login(username, "fake password")
rescue WrongPassword
true
rescue NotRegistered, WrongUsername, WrongArea
false
end
end
def login(username, password)
check_page(
use_element(:login_form) do | login |
login["username"] = username
login["password"] = password
end.submit
)
end
def login_form
@page.form_with(:name => "login")
end
12. use page object pattern
class LoginPage < PageToScrub
def is_registered?(username)
begin
login(username, "fake password")
rescue WrongPassword
true
rescue NotRegistered, WrongUsername, WrongArea
false
end
end useful abstractions
def login(username, password)
check_page(
use_element(:login_form) do | login |
login["username"] = username
login["password"] = password
end.submit
)
end
def login_form
@page.form_with(:name => "login")
end
13. use page object pattern
class PageToScrub
...
def use_element(element_name)
element = self.send(element_name)
raise MalformedPage.new(@page, "unable to locate #{element_name}") if (
element.nil? || (element.empty? rescue true)
)
return yield(element) if block_given?
element
end
...
end
14. after a while...
...few pages
my A@@
45 pages and
93 different
pieces of
data
16. rspec is your friend :-)
describe "is_registered" do
context "XXX3760593" do
it "should be a consumer registered" do
result = command(:is_registered, :username => "XXX3760593")
result.should_not be_an_error
result["area"].should == "consumer"
result["registered"].should == true
end
end
end
17. and then...
obviously
not all the
requests
can be live
on our
systems
18. enter the cache
def browse
begin
cache = CommandCache.new(database_path)
configuration = YAML::load(File.open(configuration_path))
agent = Mechanize.new do | agent |
agent.user_agent_alias = "Linux Mozilla"
end
yield(agent, configuration, cache)
rescue Mechanize::ResponseCodeError => error
failure(LoadPageError.new(error))
rescue Timeout::Error
failure(TimeoutPageError.new)
rescue ScrubError => error
failure(error)
rescue => error
failure(UnknownError.new(error.to_s))
ensure
cache.close!
end
end
19. enter the cache
single line change
def is_registered(username, without_cache)
browse do | agent, configuration, cache |
cache.command([ username, "is_registered" ]) do
LoginPage.new(
agent.get(configuration["login_page_url"])
).is_registered?(username)
end
end
end
20. enter the cache
class CommandCache
def initialize(database_path)
@database = create_database(database_path)
end
better ask
def command(keys) forgiveness than
begin permission
from_cache(keys)
rescue NotInCache => e
raise e if not block_given?
to_cache(keys, yield)
end
end
end
21. and then...
our systems
cannot take
more than 25
concurrent
requests...
make sure of
it!!!
26. god bless ruby
class Mechanize
alias real_fetch_page fetch_page
def fetch_page(params)
...
attempts = 0
begin
attempts += 1
real_fetch_page(params)
look at this line!!!
rescue Net::HTTPServerException => error
if is_overloaded?(error)
sleep wait_for_seconds and retry if attempts < retry_for_times
raise SystemError.new("SystemOverloaded")
end
raise error
end
end
def is_overloaded?(error)
error.response.code == "403"
end
end
27. we can also test it :-)
class WEBrick::HTTPResponse
def serve(content)
self.body = content
self["Content-Length"] = content.length
end
def overloaded
serve("<html><body>squid</body></html>")
self.status = 403
end
end
proxy = WEBrick::HTTPProxyServer.new(
:Port => 2200,
:ProxyContentHandler => Proc.new do | request, response |
response.overloaded
end
)
trap("INT") { proxy.shutdown }
proxy.start
28. finally ;-)
well... i
guess we can
release it...
29. the unexpected
but... wait...
our i.t.
department
said that
sometimes it
crashes
32. If you want something done, do it yourself
how to transform a command line program
into a web application
class ScrubsHandler < Mongrel::HttpHandler
def process(request, response)
command = request.params["PATH_INFO"].tr("/", "")
elements = Mongrel::HttpRequest.query_parse(request.params["QUERY_STRING"])
parameters = elements.inject([]) do | parameters, parameter |
name, value = parameter
parameters << if value.nil?
"--#{name}"
else
"--#{name}='#{value}'"
end
end.join(" ")
almost a single line change
response.start(200) do | head, out |
head["Content-Type"] = "application/json"
out.write(scrubs.execute(command, parameters))
end
end
...
end
33. can this be true ?!?!?
well... i
guess we can
release it...
34. after a while...
all the
requests
are live!!!
our systems
are melting
down!!! fix
it!!! now!!!
38. maintenance page detection
class PageToScrub
def initialize(page)
@page = page
check_page_errors
check_for_maintenance
end
def check_for_maintenance
@page.search("//td[@class='txtbig']").each do | node |
if extract_text_from(node.search("./descendant::text()")) =~
/^.+?area.+?clienti.+?non.+?disponibile.+?stiamo.+?lavorando/im
raise OnMaintenancePage.new(@page, "??? is on maintenance")
end
end
end
...
end
39. after few days
good job
gabriele, it’s
working
beyond our
expectations
40. after few days
tell me,
these
“robots” of
yours can be
used to
check our
systems
42. in the end...
• almost all self care’s
features are replicated
• ~500.000 unique users/day
• ~12.000.000 requests/day
• ~4gb of cached data
• specs are used to
monitoring the entire
system
43. but in the beginning was...
it’s very
easy, we
need
something
quick and
dirty...