This document provides an overview of using the Mechanize gem to programmatically interact with and parse web pages in Ruby. It demonstrates how to navigate pages, submit forms, select and manipulate page elements, and parse content from the DOM. Key capabilities covered include making HTTP requests, submitting forms, interacting with forms and fields, selecting elements, and parsing returned pages.

1. The Agent Accessing page elements Forms
# The current page # Submitting a form without a button:
require 'rubygems' form.submit
agent.page
require 'mechanize'
# The HTML page content # Submitting a form with the default button
agent = WWW::Mechanize.new page.body form.click_button()
# disable keep_alive, when running into # forms, links, frames # Submitting a form with a specific button
# problems with session timeouts or getting an page.forms, page.links, frames form.click_button(form.button_with(:name => 'OK')
# EOFError
# Selecting by criteria follows the pattern: # Form elements
agent.keep_alive = false form.fields, form.buttons, form.file_uploads, form.radio_buttons,
# page.element(s)_with(:criteria => value)
# The plural form (.elements) returns an form.checkboxes
# Setting user agent: # array, the singular form (.element) the
agent.user_agent = 'Friendly Mechanize Script" # first matching element or nil. Criteria # Form elements can be selected just like page elements
# is an attribute symbol and value may be # form.element(s)_with(:criteria => value)
# Using one of the predefined user agents: # a string or a regular expression. If no # e.g.:
# 'Mechanize', 'Mac Mozilla', 'Linux Mozilla'. # criteria attributr is given, :name will form.field_with(:name => 'password')
# be used. e.g.: form.field_with('password')
# 'Windows IE 6', 'iPhone', 'Linux Konqueror',
page.form_with(:name => 'formName') form.checkboxes(:value => /view_.*/)
# 'Windows IE 7', 'Mac FireFox', 'Mac Safari',
page.form_with('formName')
# 'Windows Mozilla' # Field values can also be selected directly by their name
page.links_with(:text => /[0-9]*/
agent.user_agent_alias = 'Mac Safari' form.password = 'secret'
# To verify server certificates: # Setting field values
# (A collection of certificates is available # field : .value = 'something'
# here: http://curl.haxx.se/ca/ ) # checkbox : .(un)check / .checked = true|false
# radio_button: .(un)check / .checked = true|false
agent.ca_file = 'cacert.pem'
# file_upload : .file_name = '/tmp/upload.dat'
Ruby / Mechanize
# e.g.:
# Don't follow HTTP redirects form.field_with('foo').value = 'something'
agent.redirect_ok = false form.checkbox_with(:value => 'blue').uncheck
http://mechanize.rubyforge.org/mechanize/ form.radio_buttons[3].check
# Follow refresh in meta tags
agent.follow_meta_refresh = true Nokogiri # Select lists / drop down fields:
form.field_with('color').option[2].select
http://nokogiri.org/ form.field_with('color').options.find{|o| o.value == 'red'}.select
# Enable logging
form.field_with('color').select_none
require 'logger' form.field_with('color').select_all
agent.log = Logger.new('mechanize.log')
Parsing the page content
# Selecting elements from the documents DOM
Hello Mechanize! nodes = agent.page.search('expression')
nodes = agent.page / 'expression'
require 'rubygems'
require 'mechanize'
Navigation/History # Selecting the first matching element or nil
agent = WWW::Mechanize.new node = agent.page.at('expression')
# load a page agent.get('http://rubyforge.org/')
agent.get('http://the.internet.net') # 'expression' might be an XPath or CSS selector
agent.page.forms.first.words = 'mechanize' nodes = agent.page.search('//h2/a[@class="title"]')
# Go back to the last page: agent.page.forms.first.click_button nodes = agent.page.search('.h2 a.title')
agent.back
agent.page.link_with(:text => /WWW::Mechanize/).click
agent.page.link_with(:text => 'Files').click
# navigating the document tree:
# Follow a link by its text
agent.link_with(:text => 'click me').click node.parent
links = agent.page / 'strong/a' node.children
# Backup history, execute block and version = links.find do |link|
# restore history link['href'] =~ /shownotes.*release_id/ # node content and attributes
agent.transact do end.text node.text
... node.inner_html
end puts "Hello Mechanize #{version}!"
node.attributes['width']
# found nodes, can be searched the same way
rows = agent.page / 'table/tr'
Version 2010-01-30 Creative Commons License value = rows[0].at('td[@class="value"]').text
(c) 2010 Tobias Grimm http://creativecommons.org/licenses/by/3.0