Indexing Big Data on Amazon AWS

•

1 like•640 views

Presented by Scott Stults | OpenSource Connections - See conference video - http://www.lucidimagination.com/devzone/events/conferences/lucene-revolution-2012 Amazon Web Services offers a quick and easy way to build a scalable search platform, a flexibility is especially useful when an initial data load is required but the hardware is no longer needed for day-to-day searching and adding new documents. This presentation will cover one such approach capable of enlisting hundreds of worker nodes to ingest data, track their progress, and relinquish them back to the cloud when the job is done. The data set that will be discussed is the collection of published patent grants available through Google Patents. A single Solr instance can easily handle searching the roughly 1 million patents issued between 2010 and 2005, but up to 50 worker nodes were necessary to load that data in a reasonable amount of time. Also, the same basic approach was used to make three sizes of PNG thumbnails of the patent grant TIFF images. In that case 150 worker nodes were used to generate 1.6 Tb of data over the course of three days. In this session, attendees will learn how to leverage EC2 as a scalable indexer and tricks for using XSLT on very large XML documents.

Technology

Me
Scott Stults
Co-Founder of OpenSource Connections

Solr / Lucene

Bash / Python / Java

Indexing Big Data in the Cloud 2

Big Data

Indexing Big Data in the Cloud 4

Big Data Wrangler

Indexing Big Data in the Cloud 5

How?
Address a Real Project
Be Agile
Make Small Mistaeks Fast
Succeed BIG

Indexing Big Data in the Cloud 6

USPTO Goals
Prototype Search UX

Prove Solr:
Scales
Integrates
Excels

Indexing Big Data in the Cloud 7

Scale?

Indexing Big Data in the Cloud 8

Our Approach

KISS
YAGNI

(This space intentionally left blank)

Indexing Big Data in the Cloud 9

Minimal Flair

Indexing Big Data in the Cloud 10

Record Everything!

Indexing Big Data in the Cloud 11

Some Numbers

Doc Count 1.1 Million
Zip Files 313
Docs per Zip File 4,000
Zip File Size 75M
File Size 300M

Indexing Big Data in the Cloud 12

Testing
Start some servers
Process a batch
Check the clock

Indexing Big Data in the Cloud 13

$start_nodes start_nodes() { ec2-run-instances ami-1b814f72 --block-device-mapping '/dev/sdb=snap-48adde35::true' --block-device-mapping '/dev/sdi1=:10:false' --block-device-mapping '/dev/sdi2=:10:false' --block-device-mapping '/dev/sdi3=:20:false' --instance-type m1.large --key uspto-proto --instance-count $MAX_NODES --group default > ~/run-output } Indexing Big Data in the Cloud 14$

Gut Check

How fast can we do this?

What can we do in parallel?

Indexing Big Data in the Cloud 15

Scaling

Raise our instance limit

xargs -P GNU parallel

Indexing Big Data in the Cloud 16

Shortcomings

SSH?
Error recovery
One Solr

Indexing Big Data in the Cloud 17

Alternatives
CloudFormation
Puppet / Chef
Multiple Cores / Shards
Hadoop

Indexing Big Data in the Cloud 18

Success

Indexing Big Data in the Cloud 19

Victory Lap

Indexing Big Data in the Cloud 20

Instances / Time

Indexing Big Data in the Cloud 21

Thank You

https://github.com/sstults/patent-indexing

@scottstults
#o19s

Indexing Big Data in the Cloud 22

This document outlines a presentation on analyzing large raster data in a Jupyter notebook with GeoPySpark on AWS. The presentation covers introductory material, exercises on working with land cover and Landsat imagery data, combining data layers to detect crop cycles, and combining different data types to create maps. It discusses where the notebooks are running, data sources, and GeoPySpark capabilities like working with space-time raster data. Attendees are encouraged to tweet maps created during the exercises.

Deep Learning on Aerial Imagery: What does it look like on a map?

Rob Emanuele

This document discusses using deep learning models to segment aerial imagery from a mapping contest. It provides an overview of deep learning concepts like artificial neurons, neural networks, convolutional neural networks and fully convolutional networks. It then describes experiments using a U-Net and FCN models with cross validation and ensemble learning to segment the aerial imagery. Results of the deep learning predictions are visualized on an interactive map. Future work discussed includes applying these techniques to other computer vision tasks and improving the infrastructure for distributed training.

The Directions Pipeline at Mapbox - AWS Meetup Berlin June 2015

Johan

The Mapbox Directions Pipeline aims to always have the freshest map data available for routing. It involves getting the latest OpenStreetMap data, pre-processing it for directions, loading the new data into API servers, and then repeating the process. Each step uses its own CloudFormation stack. The pipeline downloads planet files from OpenStreetMap, pre-processes them for different transport profiles, uploads the results to S3, and updates the API CloudFormation stacks to fetch the new data.

Taking Your Database Global with Kubernetes

Christopher Bradford

Deploying and running applications globally, at scale, is becoming easier and faster with Kubernetes. From a single container Express app to microservice-powered service meshes, cloud-native applications are the norm. In this session, Christopher Bradford explores the aspects you’ll want to consider when taking the database of your application global including: Deployment topologies like hybrid and multi-cloud Infrastructure and configuration planning Resiliency and planning for failure Security and privacy Application connectivity After this session, you’ll be equipped with the bigger picture of what it takes to move your data operations onto Kubernetes in a globally distributed space, as part of a cloud-native stack.

Building a Data Plane with K8ssandra, Apache Cassandra on Kubernetes

Christopher Bradford

K8ssandra has made it effortless to deploy Apache Cassandra on Kubernetes. Long a simple means of deploying stateless applications, modern tooling and APIs has facilitated the move of databases to this pervasive platform. Join Chris Bradford in deploying the K8ssandra stack to Kubernetes. Learn how it packages a production Cassandra deployment with supporting tooling alongside Stargate, a next generation data gateway. We will explore everything from the management interfaces leveraged by DevOps teams to performant, highly available, REST, Graph, and Document APIs for developers.

Migrating a 130TB Cluster from Elasticsearch 2 to 5 in 20 Hours Without Downtime

Fred de Villamil

NRP Engagement webinar - Running a 51k GPU multi-cloud burst for MMA with Ic...

Igor Sfiligoi

Burst data retrieval after 50k GPU Cloud run

Igor Sfiligoi

This document discusses a large-scale GPU-based cloud burst simulation run by the IceCube collaboration to calibrate simulations of natural ice. The simulation was data-intensive, producing over 130 TB of data and exceeding 10 Gbps of egress bandwidth. Internet2 Cloud Connect service was used to provision over 20 dedicated network links between collaborators' institutions and cloud providers to enable high-throughput data transfer at a lower cost than commercial routes. Careful planning was required to smoothly ramp up the burst and avoid overloading individual network links.

Climate data in r with the raster package

Alberto Labarga

Advanced git

satya sudheer

Git stores information in a key-value store, with blobs (data) as the value and SHA1 hashes as the keys. Git stores its data in objects, trees, and commits within the .git directory. Git optimizes storage by compressing objects and storing them in compressed packfiles. A commit represents a snapshot of the code. Code moves between the working area, staging area, and repository. Common Git commands include git add to stage files, git commit to commit snapshots, and git checkout to switch between commits or files.

I See NoSQL Document Stores in Geospatial Applications

IBM Cloud Data Services

IBM Cloudant describe the geospatial tools used in their database-as-a-service offering (DBaaS). Based upon Apache CouchDB, the geospatial extensions used by IBM Cloudant rely on a number of well known open source libraries to provide geospatial indexing, query and projection support to Apache CouchDB. Discussion topics include: - Overview of the architecture & tools - Best practices for building geospatial apps with NoSQL doc stores - Use cases for leveraging geospatial capabilities of a NoSQL doc store

Openshift GeoSpatial Capabilities

Steven Pousty

This document discusses using the OpenShift Platform as a Service (PaaS) for geospatial applications. It provides an overview of OpenShift and demonstrates how to deploy PostGIS and MongoDB for geospatial data storage and GeoServer for serving maps on OpenShift. The presentation assumes basic command line and geospatial knowledge and shows how OpenShift allows developers to write code and apps without managing servers.

Big Data Analytics with Google BigQuery, by Javier Ramirez, datawaki, at Span...

javier ramirez

DSD-INT 2017 The use of big data for dredging - De Boer

Deltares

Scaling Elasticsearch at Synthesio

Fred de Villamil

- Fred de Villamil is the director of infrastructure at Synthesio and has been working with Linux/BSD and open source since 1996. - Synthesio uses Elasticsearch to power over 13,000 dashboards, indexing over 75 billion documents and 200TB of data across 5 clusters with 163 servers and 400TB of storage. - They initially had performance issues with cross-cluster queries in MySQL but migrated to Elasticsearch in 2015 and saw significant performance improvements with their "Clipping Revolution" implementation. - Over time they encountered issues at scale including too many shards, slow restarts, and garbage collection problems. They optimized their implementation with changes like rack awareness, G1GC tuning, and field data cache configuration.

Drinking from the firehose: Logging at scale with ELK

Cteodorski

20171012 found IT #9 PySparkの勘所

Ryuji Tamagawa

20170210 sapporotechbar7

Ryuji Tamagawa

1. PyData is a community for users and developers of open-source data tools in Python including NumPy, Pandas, SciPy, scikit-learn, IPython, and Jupyter. 2. Pandas is a software library written for data manipulation and analysis in Python, built on top of NumPy and SciPy. It provides data structures and operations for working with relational or labeled data and time series. 3. Jupyter Notebook is an open-source web application that allows users to create and share documents that contain live code, equations, visualizations and explanatory text. It supports over 40 programming languages including Python, R and Julia.

Building maps for apps in the cloud - a Softlayer Use Case

Timan Rebel

Geospatial Data Visualization: WorldMap Integration by Raman Prasad

datascienceiqss

The Boston Area Research Initiative (BARI) has started work to integrate the Dataverse and Harvard WorldMap systems. This will give Dataverse users the opportunity to map shapefiles as well as tabular files. The maps will be saved in the WorldMap system, allow users to take advantage of the WorldMap’s rich feature set as well as share the maps through Dataverse. This effort involved building APIs on both systems and connecting them via a small web application.

20170927 pydata tokyo データサイエンスな皆様に送る分散処理の基礎の基礎、そしてPySparkの勘所

Ryuji Tamagawa

This document discusses PySpark and how it relates to Python, Spark, and big data frameworks. Some key points discussed include: - PySpark allows users to write Spark applications in Python, enabling Python users to leverage Spark's capabilities for large-scale data processing. - PySpark supports both the RDD API and DataFrame API for working with distributed datasets. It also integrates with Spark libraries like MLlib, GraphX, and Spark SQL. - The document discusses how PySpark fits into the broader Spark and Hadoop ecosystems. It also covers topics like Parquet and Apache Arrow for efficient data serialization between Python and Spark.

2021 Dask Summit - Using STAC to catalog SpatioTemporal datasets

Rob Emanuele

The document discusses using the SpatioTemporal Asset Catalog (STAC) to catalog geospatial datasets. STAC defines JSON schemas to encode metadata about spatiotemporal data like remote sensing imagery. This allows datasets like the European Space Agency's Sentinel-2 satellite data, containing petabytes of images, to be more easily searched. The STAC API also defines standards for searching and discovering STAC metadata. Tools like PySTAC and pystac-client make it easier to work with STAC catalogs and APIs in Python. Open questions remain around best representing multi-dimensional datasets like Zarr in STAC.

Pachyderm: Building a Big Data Beast On Kubernetes

KubeAcademy

Pachyderm is a containerized data analytics solution that's completely deployed using Kubernetes. We take all the amazing tools and potential in the container ecosystem and unlock that power for massive-scale data processing. In this talk we'll show you how to leverage Docker, Kubernetes, and Pachyderm, to build incredibly robust and scalable data infrastructure. We'll start by discussing the key components of a modern data-drive company and how your infrastructure choices can have a massive impact on your product and scalability roadmap. We'll then dive into some architecture details to show how Kubernetes, Docker, and Pachyderm all work in tandem to create a cohesive data infrastructure stack. Finally, we will demonstrate some high-level use cases and powerful benefits you get from the architecture we've outlined. KubeCon schedule link: http://sched.co/4WWA

Green material, encryption and gate in Ark Load

Brij Consulting, LLC

OpenShift with Eclipse Tooling - EclipseCon 2012

Steven Pousty

This document provides an overview of the Eclipse tooling for OpenShift. It begins with an agenda and assumptions. It then defines Infrastructure as a Service, Platform as a Service, and Software as a Service. It highlights benefits of using a PaaS like OpenShift. Supported technologies are listed, including Java, JBoss Tools, Maven, and Jenkins. Steps are provided to get started, including signing up, installing plugins, creating a domain and applications. Demo steps are outlined. Command line tools are discussed. Creating an application in Eclipse is demonstrated.

Spatial mongo for PHP and Zend

Steven Pousty

This document provides an overview of building geospatial applications with Zend, MongoDB, and OpenShift. It includes an agenda that covers loading spatial data into MongoDB, performing queries, and showing PHP code to access spatial data. The document also discusses assumptions, what OpenShift is, supported technologies, and concludes by stating spatial is easy and fun with MongoDB and PHP, and applications can now be built and deployed quickly on OpenShift without infrastructure management.

Indexing big data in the cloud

OpenSource Connections

Amazon Web Services offers a quick and easy way to build a scalable search platform, a flexibility is especially useful when an initial data load is required but the hardware is no longer needed for day-to-day searching and adding new documents. This presentation will cover one such approach capable of enlisting hundreds of worker nodes to ingest data, track their progress, and relinquish them back to the cloud when the job is done. The data set that will be discussed is the collection of published patent grants available through Google Patents. A single Solr instance can easily handle searching the roughly 1 million patents issued between 2010 and 2005, but up to 50 worker nodes were necessary to load that data in a reasonable amount of time. Also, the same basic approach was used to make three sizes of PNG thumbnails of the patent grant TIFF images. In that case 150 worker nodes were used to generate 1.6 Tb of data over the course of three days. In this session, attendees will learn how to leverage EC2 as a scalable indexer and tricks for using XSLT on very large XML documents.

Indexing big data in the cloud

Indexing Big Data on Amazon AWS

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Indexing Big Data on Amazon AWS

Similar to Indexing Big Data on Amazon AWS (20)

More from lucenerevolution

More from lucenerevolution (20)

Recently uploaded

Recently uploaded (20)

Indexing Big Data on Amazon AWS