The document discusses scraping the Tower Bridge website for lift schedule data and converting it to an iCalendar format. It describes various challenges encountered like timezones and validation issues. The author ultimately creates a simple website hosting the daily rebuilt iCalendar feed of Tower Bridge lift times.

1. Freeing
Tower
Bridge

2. It’s 2017!
Why the f*ck are
we still scraping
web sites?

3. Bad
Londoner
Confession

4. I have lived in
London for
over 35 years

5. I have
never seen
Tower Bridge
lift

7. * Who are hiring

8. * Who are hiring (obviously)

10. Geographical
Convenience

11. Opportunity!

12. Notification

13. Data feed

14. Data feed
Android→

15. Data feed
Android→
Pebble→

16. Profit!

17. Data

19. No
Data

23. No
Data
(Machine readable)

24. DIY

25. It’s 2017!
Why the f*ck are
we still scraping
web sites?

26. Web::Query

27. wq('http://www.towerbridge.org.uk/lift-times/')
->find('table.lined tbody tr')
->each(sub {
push @lifts, [
map { $_->text } $_[1]->contents
]
});

28. wq('http://www.towerbridge.org.uk/lift-times/')
->find('table.lined tbody tr')
->each(sub {
push @lifts, [
map { $_->text } $_[1]->contents
]
});

29. wq('http://www.towerbridge.org.uk/lift-times/')
->find('table.lined tbody tr')
->each(sub {
push @lifts, [
map { $_->text } $_[1]->contents
]
});

30. wq('http://www.towerbridge.org.uk/lift-times/')
->find('table.lined tbody tr')
->each(sub {
push @lifts, [
map { $_->text } $_[1]->contents
]
});

31. wq('http://www.towerbridge.org.uk/lift-times/')
->find('table.lined tbody tr')
->each(sub {
push @lifts, [
map { $_->text } $_[1]->contents
]
});

32. $VAR1 = [
[
'Sat',
'11 Mar',
'07:30',
'Maintenance Lift ',
'Up river'
],
[
'Sat',
'11 Mar',
'08:00',
'Maintenance Lift ',
'Down river'
],
...
];

33. Munge

34. iCal

35. Profit!

36. Well...

37. Data::ICal

38. Data::ICal::Entry::Event

39. my $ical = Data::ICal->new();
for (@lifts) {
my $date = ...;
my $event = Data::Ical::Entry::Event->new();
$event->add_properties(...);
$ical->add_entry($event);
}
print $fh, '>', $ical->as_string;

42. No Year

43. my $date = $dt_parser->parse_datetime(
"$_->[2] $_->[1] $curr_year"
);
# If the month number of this event is less
# than the current month number then we've
# gone to the next year. Increment the year
# number and re-calculate.
if ($date->mon < $curr_mon) {
++$curr_year;
$date = $dt_parser→parse_datetime(
"$_->[2] $_->[1] $curr_year"
);
}

44. # Tower Bridge web site occasionally
# has duplicates
next if $seen{$date->epoch}++;

45. my $event =
Data::ICal::Entry::Event->new();
$event->add_properties(
summary => 'Tower Bridge Lift',
description => "$_->[3] ($_->[4])",
dtstart => dt2ical($date),
duration => 'PT30M',
dtstamp => $now_ical,
uid => $date->epoch .
'@towerbridge.dave.org.uk',
);

46. A detour

49. Different
Timezones

50. (I assume)

53. Timezones
are easy

54. my $dt_parser =
DateTime::Format::Strptime->new(
pattern => '%H:%M %d %b %Y',
time_zone => 'Europe/London',
);

55. sub dt2ical {
my ($dt) = @_;
return $dt->ymd('') . 'T' .
$dt->hms('') .
# Or something like this.
# Check iCal specs.
$dt->time_zone_short_name;
}

56. Failed
validation

57. https://icalendar.org/validator.html

58. DateTime::Format::ICal

59. TZID=Europe/London:20170311T105600

60. Looked
OK

61. Failed
validation

62. “Invalid
TZID”

63. TZID=Europe/London:20170311T105600

64. TZID=Europe/London:20170311T105600

65. To the
standard
definition!

66. This property specifies the
text value that uniquely
identifies the
"VTIMEZONE" calendar
component in the scope of
an iCalendar object.

67. If present, the
"VTIMEZONE" calendar
component defines the set of
Standard Time and Daylight
Saving Time observances (or
rules) for a particular time
zone for a given interval of
time.

68. Add VTIMEZONE
section to the iCal
file

69. Data::ICal::Entry::TimeZone

70. This module is not yet useful,
because every time zone
declaration needs to contain
at least one STANDARD or
DAYLIGHT component, and
these have not yet been
implemented.

72. Plan C

73. Back to the
iCal standard
definition

74. DATE WITH LOCAL TIME
The date with local time form is simply
a DATE-TIME value that does not
contain the UTC designator nor does it
reference a time zone. For example, the
following represents January 18, 1998,
at 11 PM:
19980118T230000

75. sub dt2ical {
my ($dt) = @_;
return $dt->ymd('') . 'T' .
$dt->hms('');
}

76. And we have a
valid iCal feed

77. Throw
together
a web site

79. http://towerbridge.dave.org.uk

80. Rebuild the
data daily

81. Stick the code
on Github

82. https://github.com/davorg/towerbridge

83. Subscribe to
the calendar

84. Profit!

86. 13th
Feb 2017 13:30

87. Prior
Art

93. Sun 2nd
April:
20:30 & 21:15

94. Dave Cross
@davorg
@perlhacks
https://perlhacks.com/