Site story wadl2013

•Download as PPTX, PDF•

0 likes•2,862 views

This document summarizes a presentation about SiteStory, an alternative approach to web archiving. SiteStory allows transactional archiving by capturing HTTP transactions between browsers and servers. This addresses issues with traditional crawler-based archiving like timing problems and inability to archive certain content. The presentation describes how SiteStory works, current status including benchmark testing, and how to access a SiteStory testbed installation.

WADL 2013
July 25-26th Indianapolis, IN
Martin Klein
@mart1nkle1n
martinklein0815@gmail.com
SiteStory
Archiving Done Differently
http://mementoweb.github.io/SiteStory/
Justin F. Brunelle
jbrunelle@cs.odu.edu

WADL 2013
July 25-26th Indianapolis, IN
LANL SiteStory Teamlead developer

WADL 2013
July 25-26th Indianapolis, IN
Archiving - the traditional way
• Actively crawl the web
• For example, using Heritrix

WADL 2013
July 25-26th Indianapolis, IN
• Issues with crawler based archiving:
• Request can be rejected (robots.txt, user-agent, IP)
• Can be deceived (geo-location, user-agent)
• Can be trapped (crawl my calendar!)
• Requires constant and massive bandwidth
• Implied timing problem, when to crawl?
Archiving - the traditional way

WADL 2013
July 25-26th Indianapolis, IN
Timing problem:
• Update 1 viewed but not archived
t1
R
created
t2
browser
visit1
t3
crawler
visit1
t4
R
update1
t5
browser
visit2
t6
R
update2
Archiving - the traditional way

WADL 2013
July 25-26th Indianapolis, IN
Archiving - the SiteStory way
• Transactional Web archiving
• Archive accepts HTTP transaction between browser
and server

WADL 2013
July 25-26th Indianapolis, IN
Timing problem:
• Update 1 viewed and archived
t1
R
created
t2
browser
visit1
t3
crawler
visit1
t4
R
update1
t5
browser
visit2
t6
R
update2
Archiving - the traditional way

WADL 2013
July 25-26th Indianapolis, IN
• Challenges with transactional archiving:
• To be archived server has to cooperate
• Transfer data to archive, batch mode or real-time
• Archive must trust transmission to be authentic
• Resources from external servers have to be archived
out-of-band
• Deduplication challenges
• Alias: different URI, same response
• Conneg: same URI, different response
• Determine “significant” content change
Archiving - the SiteStory way

WADL 2013
July 25-26th Indianapolis, IN
SiteStory Status Quo
• mod_sitestory sends HTTP PUT to SiteStory Web
Archive upon client’s GET request
• not for POST, DELETE, etc
• for HTTP response codes 200, 302, 303
• Client IP can be included in stored headers, configurable
• Header info stored in BerkeleyDB, response body in FS
• Dedup via hash(body)
• Offloading content as WARC files possible
(read: recommended)

WADL 2013
July 25-26th Indianapolis, IN
To Appear: TPDL 2013
• SiteStory benchmark with ab&wget
o ApacheBench (ab): server stress test tool
o wget: Web page download
- All content: -p
• Local network
• Negligible difference between
SiteStory and No SiteStory

WADL 2013
July 25-26th Indianapolis, IN
Re-executed on testbed
ws-dl-03.cs.odu.edu
x99
,…
,
,
megalodon.lanl.gov
@AWS

WADL 2013
July 25-26th Indianapolis, IN
Testing with ab

WADL 2013
July 25-26th Indianapolis, IN
Testing with wget

WADL 2013
July 25-26th Indianapolis, IN
Round Trip Time -- Distributed

WADL 2013
July 25-26th Indianapolis, IN
Results
• Distributed: Higher variance
• Increased delay due to network
• On vs. Off Comparison still comparable
• Viable solution without crippling service

WADL 2013
July 25-26th Indianapolis, IN
SiteStory Installation
• Apache module mod_sitestory
• Option to exclude a list of directories
• SiteStory Web Archive
• Trivial for existing Tomcat environments
• Tanuki Java wrapper (stand-alone) available
• Configure, open ports, go!
Or…

WADL 2013
July 25-26th Indianapolis, IN
SiteStoryTestbed
We have a SiteStory Web Archive installed for you!
1. Install and configure mod_sitestory
2. Send an email containing:
1. Your contact info
2. Web server IP address
3. Server domain name used
3. Happy Sitestory’ing!
mailto: SiteStory-Testbed@googlegroups.com
http://mementoweb.github.io/SiteStory/

Build a simple open data lake on AWS using a combination of open-source software (OSS), including Red Hat’s Debezium, Apache Kafka, and Kafka Connect for change data capture (CDC), and Apache Hive, Apache Spark, Apache Hudi, and Hudi’s DeltaStreamer for managing our data lake. We will use fully-managed AWS services to host the open data lake components, including Amazon RDS, Amazon MKS, Amazon EKS, and EMR. Link to the blog post and video: https://garystafford.medium.com/building-open-data-lakes-with-debezium-and-apache-hudi-c3370d3f86fb

Introduction to Apache Airflow - Data Day Seattle 2016

Sid Anand

Apache Airflow is a platform for authoring, scheduling, and monitoring workflows or directed acyclic graphs (DAGs) of tasks. It includes a DAG scheduler, web UI, and CLI. Airflow allows users to author DAGs in Python without needing to bundle many XML files. The UI provides tree and Gantt chart views to monitor DAG runs over time. Airflow was accepted into the Apache Incubator in 2016 and has over 300 users from 40+ companies. Agari uses Airflow to orchestrate message scoring pipelines across AWS services like S3, Spark, SQS, and databases to enforce SLAs on correctness and timeliness. Areas for further improvement include security, APIs, execution scaling, and on

Workflow Engines + Luigi

Vladislav Supalov

Vladislav Supalov introduces data pipeline architecture and workflow engines like Luigi. He discusses how custom scripts are problematic for maintaining data pipelines and recommends using workflow engines instead. Luigi is presented as a Python-based workflow engine that was created at Spotify to manage thousands of daily Hadoop jobs. It provides features like parameterization, email alerts, dependency resolution, and task scheduling through a central scheduler. Luigi aims to minimize boilerplate code and make pipelines testable, versioning-friendly, and collaborative.

RIPE Atlas

RIPE NCC

Christian Teuschel gave a presentation about RIPE Atlas at the Peering Day event in Bratislava in March 2015. RIPE Atlas is a network for Internet measurements that uses over 8000 probes to collect data on latency, routing, and outages. It allows users to evaluate the benefits of peering by measuring latency before and after a peering connection. Attendees were encouraged to host their own probes or anchors to contribute data and receive credits to run custom measurements.

Care and Feeding of Large Scale Graphite Installations - DevOpsDays Austin 2013

Nick Galbreath

Stream Processing with Apache Spark, Kafka, Avro, and Apicurio Registry on AW...

Gary Stafford

We will read and write messages to and from Amazon MSK in Apache Avro format. We will store the Avro-format Kafka message’s key and value schemas in Apicurio Registry and retrieve the schemas instead of hard-coding the schemas in the PySpark scripts. We will also use the registry to store schemas for CSV-format data files. Link to the blog post and video: https://itnext.io/stream-processing-with-apache-spark-kafka-avro-and-apicurio-registry-on-amazon-emr-and-amazon-13080defa3be

This document discusses how the author learned to use Airflow for data pipelining and scheduling tasks. It describes some early tools like Cron and Luigi that were used for scheduling. It then evaluates options like Drake, Pydoit, Pinball, Luigi, and AWS Data Pipeline before settling on Airflow due to its sophistication in handling complex dependencies, built-in scheduling and monitoring, and flexibility. The author also develops a plugin called smart-airflow to add file-based checkpointing capabilities to Airflow to track intermediate data transformations.

What is Spark

Bruno Faria

Processing genetic data at scale

Mark Schroering

This document discusses processing large genetic data sets at scale in the cloud. It describes ingesting variant call format (VCF) files into a data lake in Amazon S3 using Apache Parquet format and annotating variants using Docker containers on AWS Batch. It also covers indexing variants and annotations for analytics using PostgreSQL and DynamoDB and providing queries through a Lambda function. Lessons learned include using DynamoDB on-demand capacity, monitoring compute resource limits, utilizing spot instances, partitioning large tables, and using reader endpoints for Aurora databases. Current stats show over 101 million variants stored across 289GB in S3 and 94GB in DynamoDB with query response times averaging 1 second.

AWS_Data_Pipeline

Ahasan Habib

AWS Data Pipeline is a web service that allows users to design data driven workflows to move and transform data between different AWS services reliably and in a cost effective manner. It allows users to schedule, run, and manage recurring data processing workloads. Data Pipeline includes components like pipeline definitions, schedules, task runners, and objects like shell command activities and S3 data nodes to design extract, transform, load (ETL) processes. It works with services like DynamoDB, RDS, Redshift, S3, and EC2. Pipelines are created by composing definition objects in a file and can be accessed through the AWS Management Console, CLI, SDKs, and APIs.

Acid ORC, Iceberg and Delta Lake

Michal Gancarski

The document compares three table formats for large scale data storage and analytics: ACID ORC, Apache Iceberg, and Delta Lake. ACID ORC provides ACID transactions for Hive tables stored as ORC files but has slow metadata operations. Iceberg supports multiple file formats, robust schema changes, and time travel capabilities but lacks commercial support. Delta Lake has great Spark integration, SQL merge syntax, and time travel with optimized compaction, but only supports Parquet files and multicluster writes on HDFS.

Running Airflow Workflows as ETL Processes on Hadoop

clairvoyantllc

While working with Hadoop, you'll eventually encounter the need to schedule and run workflows to perform various operations like ingesting data or performing ETL. There are a number of tools available to assist you with this type of requirement and one such tool that we at Clairvoyant have been looking to use is Apache Airflow. Apache Airflow is an Apache Incubator project that allows you to programmatically create workflows through a python script. This provides a flexible and effective way to design your workflows with little code and setup. In this talk, we will discuss Apache Airflow and how we at Clairvoyant have utilized it for ETL pipelines on Hadoop.

Intro to Airflow: Goodbye Cron, Welcome scheduled workflow management

Burasakorn Sabyeying

This document discusses Apache Airflow, an open-source workflow management platform for authoring, scheduling, and monitoring workflows or pipelines. It provides an overview of Airflow's key features and components, including Directed Acyclic Graphs (DAGs) for defining workflows as Python code, various operators for building tasks, and its rich web UI. The document compares Airflow to traditional cron jobs, noting Airflow can handle task dependencies and failures better than cron. It also outlines how to set up an Airflow cluster on multiple nodes for scaling workflows.

Scaling Graphite At Yelp

Paul O'Connor

This document discusses metrics and the Graphite monitoring system. It describes the main components of Graphite including Carbon, which persists metrics to disk and supports replication and sharding. It also describes Whisper for data storage and aggregation, and the web interface for rendering graphs. The document provides an overview of how these components work together and tips for optimizing performance such as aggregating metrics before ingestion and controlling the metrics that can be sent. It also briefly mentions alternative time-series databases like InfluxDB and Cassandra that could be used in the future.

Airflow for Beginners

Varya Karpenko

Lighthouse - an open-source library to build data lakes - Kris Peeters

Data Science Leuven

presto-at-netflix-hadoop-summit-15

Zhenxiao Luo

This document summarizes Presto, an open source distributed SQL query engine, and its use at Netflix for interactive queries against large datasets. Key points: - Presto allows Netflix to run interactive SQL queries across petabytes of data stored in S3 and Cassandra. It is fast, scalable, supports ANSI SQL, and works well on AWS. - Netflix's Presto deployment includes over 220 worker nodes and handles thousands of queries per day across 15+ petabytes of data. - Netflix has contributed several optimizations to Presto's query planning, including pushing down aggregations, handling complex types, and improved support for the Parquet file format. - Future work includes further Parquet optimizations

Analysing GitHub commits with R

Barbara Fusinska

This document outlines a presentation on analyzing GitHub commits with R. The presentation covers obtaining GitHub data from sources like the GitHub Archive and API, exploring and processing the data with R, and visualizing results. It includes demos of basic R functions, reading and filtering GitHub event data, handling missing data, plotting languages over time, and techniques for working with large datasets in R using libraries, Hadoop, and other tools. The overall goals are to learn methods for analyzing GitHub data and exploring it visually using the R programming language.

Real Time Big Data

InfoFarm

Spark: The Good, the Bad, and the Ugly

Sarah Guido

Given at Data Day Texas 2016. Apache Spark has been hailed as a trail-blazing new tool for doing distributed data science. However, since it's so new, it can be difficult to set up and hard to use. In this talk, I'll discuss the journey I've had using Spark for data science at Bitly over the past year. I'll talk about the benefits of using Spark, the challenges I've had to overcome, the caveats for using a cutting-edge technology such as this, and my hopes for the Spark project as a whole.

Dataset Descriptions in Open PHACTS and HCLS

Alasdair Gray

This presentation gives an overview of the dataset description specification developed in the Open PHACTS project (http://www.openphacts.org/). The creation of the specification was driven by a real need within the project to track the datasets used. Details of the dataset metadata captured and the vocabularies used to model this metadata are given together with the tools developed to enable the specification's uptake. Over the course of the last 12 months, the W3C Healthcare and Life Science Interest Group have been developing a community profile for dataset descriptions. This has drawn on the ideas developed in the Open PHACTS specification. A brief overview of the forthcoming community profile is given in the presentation. This presentation was given to the Network Data Exchange project http://www.ndexbio.org/ on 2 April 2014.

Apache Airflow

Sumit Maheshwari

Contributing to Apache Airflow | Journey to becoming Airflow's leading contri...

Kaxil Naik

From not knowing Python (let alone Airflow), and from submitting the first PR that fixes typo to becoming Airflow Committer, PMC Member, Release Manager, and #1 Committer this year, this talk walks through Kaxil’s journey in the Airflow World. The second part of this talk explains: how you can also start your OSS journey by contributing to Airflow Expanding familiarity with a different part of the Airflow codebase Continue committing regularly & steadily to become Airflow Committer. (including talking about current Guidelines of becoming a Committer) Different mediums of communication (Dev list, users list, Slack channel, Github Discussions etc)

Automatic Query-Centric API for Routine Access to Linked Data

Albert Meroño-Peñuela

ISWC 2017 In-Use paper. Despite the advantages of Linked Data as a data integration paradigm, accessing and consuming Linked Data is still a cumbersome task. Linked Data applications need to use technologies such as RDF and SPARQL that, despite their expressive power, belong to the data integration stack. As a result, applications and data cannot be cleanly separated: SPARQL queries, endpoint addresses, namespaces, and URIs end up as part of the application code. Many publishers address these problems by building RESTful APIs around their Linked Data. However, this solution has two pitfalls: these APIs are costly to maintain; and they blackbox functionality by hiding the queries they use. In this paper we describe grlc, a gateway between Linked Data applications and the LOD cloud that offers a RESTful, reusable and uniform means to routinely access any Linked Data. It generates an OpenAPI compatible API by using parametrized queries shared on the Web. The resulting APIs require no coding, rely on low-cost external query storage and versioning services, contain abundant provenance information, and integrate access to different publishing paradigms into a single API. We evaluate grlc qualitatively, by describing its reported value by current users; and quantitatively, by measuring the added overhead at generating API specifications and answering to calls.

Semantic web and Drupal: an introduction

Kristof Van Tomme

Who and What Links to the Internet Archive

Yasmin AlNoamany, PhD

This document analyzes access patterns for robots and humans on web archives. It finds that English pages are the most requested, followed by European languages. Most human sessions come to the Wayback Machine via referrals, led by Wikipedia, the Internet Archive homepage, Reddit, and Google. The analysis also shows that most links from outside archives go to past versions ("mementos") of pages, and 83% of linked mementos no longer exist on the live web. The study provides insights into what content languages users look for and how people discover and link to archived web pages.

Archiving the Mobile Web

Frank McCown

The document discusses archiving mobile web pages using Heritrix, an open-source web crawler. It describes modifying Heritrix to support crawling both desktop and mobile versions of sites. The Mobile Finder web service identifies if a given URL has a mobile version. Heritrix was configured with two crawl options - using desktop or mobile user-agents. URLs were crawled using both options and differences analyzed. While Heritrix cannot execute JavaScript, Mobile Finder finds mobile pages so crawling continues correctly. Analysis found distribution of content like HTML, images and CSS between desktop and mobile versions.

What's hot

Apache Airflow (incubating) NL HUG Meetup 2016-07-19

Bolke de Bruin

How I learned to time travel, or, data pipelining and scheduling with Airflow

PyData

What is Spark

Bruno Faria

Processing genetic data at scale

Mark Schroering

AWS_Data_Pipeline

Ahasan Habib

Acid ORC, Iceberg and Delta Lake

Michal Gancarski

Running Airflow Workflows as ETL Processes on Hadoop

clairvoyantllc

Intro to Airflow: Goodbye Cron, Welcome scheduled workflow management

Burasakorn Sabyeying

Scaling Graphite At Yelp

Paul O'Connor

Airflow for Beginners

Varya Karpenko

Lighthouse - an open-source library to build data lakes - Kris Peeters

Data Science Leuven

presto-at-netflix-hadoop-summit-15

Zhenxiao Luo

Analysing GitHub commits with R

Barbara Fusinska

Real Time Big Data

InfoFarm

Spark: The Good, the Bad, and the Ugly

Sarah Guido

Dataset Descriptions in Open PHACTS and HCLS

Alasdair Gray

Apache Airflow

Sumit Maheshwari

Contributing to Apache Airflow | Journey to becoming Airflow's leading contri...

Kaxil Naik

Automatic Query-Centric API for Routine Access to Linked Data

Albert Meroño-Peñuela

Semantic web and Drupal: an introduction

Kristof Van Tomme

What's hot (20)

Apache Airflow (incubating) NL HUG Meetup 2016-07-19

How I learned to time travel, or, data pipelining and scheduling with Airflow

What is Spark

Processing genetic data at scale

AWS_Data_Pipeline

Acid ORC, Iceberg and Delta Lake

Running Airflow Workflows as ETL Processes on Hadoop

Intro to Airflow: Goodbye Cron, Welcome scheduled workflow management

Scaling Graphite At Yelp

Airflow for Beginners

Lighthouse - an open-source library to build data lakes - Kris Peeters

presto-at-netflix-hadoop-summit-15

Analysing GitHub commits with R

Real Time Big Data

Spark: The Good, the Bad, and the Ugly

Dataset Descriptions in Open PHACTS and HCLS

Apache Airflow

Contributing to Apache Airflow | Journey to becoming Airflow's leading contri...

Automatic Query-Centric API for Routine Access to Linked Data

Semantic web and Drupal: an introduction

Viewers also liked

Who and What Links to the Internet Archive

Yasmin AlNoamany, PhD

Archiving the Mobile Web

Frank McCown

Old Dominion University Computer Science IIPC New Member

Michael Nelson

Generating stories from Archive-It collections

Yasmin AlNoamany, PhD

The document outlines a framework for generating stories from archived web collections. It discusses possible story types using the same page over time, different pages at the same time, or different pages over time. Criteria for choosing pages include being in English and on-topic. Collections are suggested to generate initial stories from, with an expectation of three stories per collection using the different story types.

Access Patterns for Robots and Humans in Web Archives

Yasmin AlNoamany, PhD

The document discusses a study of access patterns for robots and humans on web archive sites. It aims to analyze web server logs from the Internet Archive to understand how users access archived web pages. The study seeks to determine differences in how humans and robots/crawlers interact with web archives. The methodology section indicates the researchers analyzed sample logs from the Wayback Machine to classify requests and discover patterns.

Using Web Archives to Enrich the Live Web Experience Through Storytelling

Yasmin AlNoamany, PhD

The document discusses using web archives to automatically construct stories about past events by identifying relevant web pages from the event timeframe. It proposes a 6-step process: 1) Calculate the datetime range of the story, 2) Get seed URIs related to the story, 3) Determine datetimes of web pages, 4) Choose high-quality candidate pages for each event, 5) Visualize the story using interactive timelines or slideshows, and 6) Collect feedback on the automatically constructed stories. The goal is to use web archives to automatically "replay" the story of past events through curated web pages from that period.

Viewers also liked (6)

Who and What Links to the Internet Archive

Archiving the Mobile Web

Old Dominion University Computer Science IIPC New Member

Generating stories from Archive-It collections

Access Patterns for Robots and Humans in Web Archives

Using Web Archives to Enrich the Live Web Experience Through Storytelling

Similar to Site story wadl2013

H-Hypermap - Heatmap Analytics at Scale: Presented by David Smiley, D W Smile...

Lucidworks

This document provides an agenda and overview for a presentation on H-Hypermap, a project to build a search platform called the Billion Object Platform (BOP) to index and search over billions of geo-tagged tweets in near real-time. The presentation will cover the architecture using Apache Kafka, Solr sharding, and techniques for fast geo-spatial queries and heatmaps. It will also discuss experiences using technologies like Kotlin, Dropwizard, Docker and Kontena.

Jcdl2013 mklein

Site story wadl2013

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (6)

Similar to Site story wadl2013

Similar to Site story wadl2013 (20)

More from Martin Klein

More from Martin Klein (20)

Recently uploaded

Recently uploaded (20)

Site story wadl2013