Loading 350M documents into a large Solr cluster: Presented by Dion Olsthoorn, Wolters Kluwer

•

1 like•625 views

This document summarizes a presentation about loading 350 million documents into a Solr cluster in 8 hours or less. It describes using an external cloud platform for preprocessing content before indexing into Solr. It also details using a queueing system to post content to Solr in batches to avoid overloading the cluster. The presentation recommends indexing large content sets on an isolated Solr environment before restoring indexes to the production cluster.

Technology

O C T O B E R
1 1 -‐ 1 4 ,
2 0 1 6

•

B O S T O N ,
M A

Loading 350M documents
into a large Solr cluster
…in
8
hours
or
less

About me
•  Dion
Olsthoorn

•  Senior
SoAware
Engineer
at
Wolters
Kluwer

•  Wolters
Kluwer

•  InformaJon,
SoAware
and
Services

•  19,000
employees,
revenue
€3.7
billion
(2014)

•  Ovid

•  World’s
leading
online
Search/Delivery
Pla[orm
for
Medical
Research

•  >
350M
documents
in
our
index,
from
100+
content
sources

•  >
600K
searches
a
day

About this talk
•  “Loading
350M
documents
into
a
large
Solr
cluster
in
8
hours
or
less”

•  Why
within
8
hours?

1.  Indexing
of
yearly
content
reloads
from
content
suppliers

2.  Full
reindex
(diﬀerent
analyzers,
etc.)
between
content
updates

•  350M
docs
/
8
hrs
≈
12K
docs
/
second

•  Yes,
we
did
it!
Using
an
cloud
pla[orm
for
preprocessing.

•  Can
you
do
it?
It
depends…

•  What
you
will
get
out
of
this:

Speed
up
the
loading
of
content
into
your
Solr
cloud

Indexing Solr XML
•  We
!
xml
ﬁles

•  post.jar
–
the
obvious
“out-‐of-‐the-‐box”
choice

•  We
found
out
we
were
overloading
our
Solr
cluster!

…&update.distrib=TOLEADER

…

post.jar (SimplePostTool)
post.jar

shard
1

(leader)

shard
1

(replica)

shard
2

(replica)

shard
2

(leader)

Solr
XML
ﬁles

cloudpost.jar – “the cloud-‐aware post.jar”®
•  Using
SolrJ
(CloudSolrClient)
to
post
directly
to
shard
leaders

•  Fast:
uses
XmlStreamReader
and
generates
SolrInputDocuments

•  Other
features:
reading
from
zipped
ﬁles,
post
ﬁles
in
parallel

• 

Shared
on
github
soon…

cloudpost.jar

shard
1

(leader)

shard
1

(replica)

shard
2

(replica)

shard
2

(leader)

zipped

Solr
XML
ﬁles

We built a funnel to make it more robust
•  Reducing
risk
of
overloading
our
Solr
cluster

•  Modiﬁed
ETL
process,
so
when
E
&
T
are
done:

•  Store
results
in
Blob
storage
(batches
of
5k
documents)

•  Post
a
job
message
to
a
queue

•  Set
of
cloudpost
daemons
(“workers”)
watching
the
queue
and

picking
up
“content
posJng”
jobs

Blob

storage

queue

1

2

worker

worker

cloudpost

workers

3

4

?

What ﬁnally worked best for us…
Two
soluJons:

1.  For
iniJal
loads
of
a
content
set
(>
1M
records)

•  Index
on
(isolated)
environment
using
the
same
#shards
(no
replicas!)

•  Backup
&
Restore
shard-‐indexes
to
producJon
Solr
cluster

2.  For
daily
“small”
updates

•  Index
on
producJon

•  Single
commit
(to
minimize
reopening
of
IndexSearcher)

Conclusion & Takeaways
•  Do
all
content
pre-‐processing
outside
Solr.

Use
the
power
of
a
cloud
pla[orm!

•  Use
a
queuing
system
to
prevent
overfeeding
Solr

•  Use
SolrJ
when
posJng
content
to
SolrCloud

à
Got
Solr
XML
ﬁles?
Use
our
cloudpost.jar
J

•  Got
a
lot
of
content
to
index
on
a
live
Solr
cluster?

à
Index
on
a
isolated
environment
(idenJcal
#shards,
no
replica’s)

à
Backup
each
shard
and
restore
(Solr
6.1
has
SolrCloud
backup!)

Q & A
•  More
quesJons?

à
dion.olsthoorn@wolterskluwer.com

•  CloudPost.jar

à

…soon

•  ZooKeeper
Master-‐Worker
example

à

github.com/fpj/zookeeper-‐book-‐example

•  Wolters
Kluwer

à
www.wolterskluwer.com

Loading 350M documents into a large Solr cluster: Presented by Dion Olsthoorn, Wolters Kluwer

Spark can be used to improve the performance of importing and searching large datasets in Solr. Data can be imported from HDFS files into Solr in parallel using Spark, speeding up the import process. Spark can also be used to stream data from Solr into RDDs for further processing, such as aggregation, filtering, and joining with other data. Techniques like column-based denormalization and compressed storage of event data in Solr documents can reduce data volume and improve import and query speeds by orders of magnitude.

Building and Running Solr-as-a-Service: Presented by Shai Erera, IBM

Lucidworks

This document discusses building and running Solr as a service in the cloud. It covers: - The challenges of deploying Solr in cloud environments and the need for a managed service. - The architecture of the Solr-as-a-Service, which uses Docker, Mesos, and other tools to provide multi-tenant Solr clusters. - Key aspects of managing Solr clusters in the cloud service, including software upgrades, resizing clusters, handling replicas, and balancing clusters.

Tuning Solr and its Pipeline for Logs: Presented by Rafał Kuć & Radu Gheorghe...

Lucidworks

The document summarizes key points from a presentation on optimizing Solr and log pipelines for time-series data. The presentation covered using time-based Solr collections that rotate based on size, tiering hot and cold clusters, tuning OS and Solr settings, parsing logs, buffering pipelines, and shipping logs using protocols like UDP, TCP, and Kafka. The overall conclusions were that tuning segments per tier and max merged segment size improved indexing throughput, and that simple, reliable pipelines like Filebeat to Kafka or rsyslog over UNIX sockets generally work best.

H-Hypermap - Heatmap Analytics at Scale: Presented by David Smiley, D W Smile...

Lucidworks

This document provides an agenda and overview for a presentation on H-Hypermap, a project to build a search platform called the Billion Object Platform (BOP) to index and search over billions of geo-tagged tweets in near real-time. The presentation will cover the architecture using Apache Kafka, Solr sharding, and techniques for fast geo-spatial queries and heatmaps. It will also discuss experiences using technologies like Kotlin, Dropwizard, Docker and Kontena.

Lessons From Sharding Solr At Etsy: Presented by Gregg Donovan, Etsy

Lucidworks

Gregg Donovan presented on lessons learned from sharding Solr at Etsy over three versions: 1) Initially, Etsy did not shard to avoid problems, but the single node approach did not scale. 2) The first sharding version used local sharding across multiple JVMs per host for better latency and manageability. 3) The current version uses distributed sharding across data centers for further latency gains, but this introduced challenges of partial failures, synchronization, and distributed queries.

Solr4 nosql search_server_2013

Lucidworks (Archived)

The document discusses Solr 4, an open source search platform built on Apache Lucene. Some key points: - Solr 4 is a NoSQL search server that provides distributed indexing, fault tolerance, and real-time search capabilities. - Solr Cloud is Solr's distributed architecture which uses Zookeeper for coordination to provide features like automatic sharding and replication of indexes across multiple servers. - The document outlines Solr 4's capabilities including schema-less options, atomic updates, optimistic concurrency, and a REST API for managing the schema dynamically.

Building a Large Scale SEO/SEM Application with Apache Solr: Presented by Rah...

Lucidworks

The document discusses building a large scale SEO/SEM application using Apache Solr. It describes some of the key challenges faced in indexing and searching over 40 billion records in the application's database each month. It discusses techniques used to optimize the data import process, create a distributed index across multiple tables, address out of memory errors, and improve search performance through partitioning, index optimization, and external caching.

What's New on AWS and What it Means to You

Amazon Web Services

Timothy Potter presented at a Big Data conference in Boston from October 11-14, 2016. He discussed how Lucidworks Fusion provides an alternative to traditional big data stacks that emphasizes fast access, agility and automation over integration. Fusion allows for common access patterns like fast lookups, ranked retrieval and distributed scans while integrating technologies like Solr, Spark, HDFS and more. It provides tools for data ingestion, time-based partitioning, analytics, machine learning and more to solve business problems rather than focus on infrastructure.

SolrCloud on Hadoop

Alex Moundalexis

This document provides an overview of SolrCloud on Hadoop. It discusses how SolrCloud allows for distributed, highly scalable search capabilities on Hadoop clusters. Key components that work with SolrCloud are also summarized, including HDFS for storage, MapReduce for processing, and ZooKeeper for coordination services. The document demonstrates how SolrCloud can index and query large datasets stored in Hadoop.

SearchHub - How to Spend Your Summer Keeping it Real: Presented by Grant Inge...

Lucidworks

This document provides an overview and agenda for a SearchHub presentation on using Apache Fusion to build a search application called SearchHub. The presentation will cover configuring Fusion and the SearchHub UI, acquiring data from sources like Apache mailing lists and GitHub, deploying SearchHub on AWS, using signals and machine learning in Fusion, and next steps for the project. It includes demos of features like recommendations, tokenization, topic detection, and experiment management in Fusion.

Mail Search As A Sercive: Presented by Rishi Easwaran, Aol

Lucidworks

The document discusses Aol's migration from a multicore Solr architecture to a hybrid cloud architecture for its mail search service. Some key points: - The multicore architecture led to non-availability, high response times, and required manual intervention for load balancing. - Aol migrated to a hybrid cloud architecture using Solr Cloud, with SSDs for recent data and HDDs in the cloud for older data. This reduced costs by 30% and improved performance. - The hybrid cloud architecture introduced new issues around disk space management and Solr's overseer node. Aol is working on solutions like automatic split/merge to address these issues. - Future work includes cross-data center deployment,

Rebalance API for SolrCloud: Presented by Nitin Sharma, Netflix & Suruchi Sha...

Lucidworks

The document describes Netflix's Rebalance API for SolrCloud, which provides strategies and capabilities for scaling Solr clusters while maintaining service level agreements (SLAs). The Rebalance API includes scaling strategies for operations like auto-sharding, redistributing data, and replacing nodes with zero downtime. It also includes allocation strategies that determine core placement. The goal is to allow for fine-grained SLA management when indexing and querying large datasets across multiple data centers. The Rebalance API has been open sourced and is aimed at forming the basis for auto-scaling in Solr.

Deploying and managing Solr at scale

Anshum Gupta

This document discusses deploying and managing Apache Solr at scale. It introduces the Solr Scale Toolkit, an open source tool for deploying and managing SolrCloud clusters in cloud environments like AWS. The toolkit uses Python tools like Fabric to provision machines, deploy ZooKeeper ensembles, configure and start SolrCloud clusters. It also supports benchmark testing and system monitoring. The document demonstrates using the toolkit and discusses lessons learned around indexing and query performance at scale.

Introduction to SolrCloud

Varun Thacker

Data Engineering with Solr and Spark

Lucidworks

Faceting Optimizations for Solr: Presented by Toke Eskildsen, State & Univers...

Lucidworks

This document discusses optimizations for field faceting in Solr. It begins with an overview of faceting at web scale for the Danish Net Archive. Various techniques for optimizing faceting performance are then presented, including reusing counters, tracking updated counters, caching counters across shards, and using alternative counter structures like PackedInts and n-plane counters. The optimizations are shown to significantly speed up faceting for fields with high cardinality.

Scaling SolrCloud to a large number of Collections

Anshum Gupta

Anshum Gupta presented on scaling SolrCloud to support thousands of collections. Some challenges included limitations on the cluster state size, overseer performance issues under high load, and difficulties moving or exporting large amounts of data. Solutions involved splitting the cluster state, improving overseer performance through optimizations and dedicated nodes, enabling finer-grained shard splitting and data migration between collections, and implementing distributed deep paging for large result sets. Testing was performed on an AWS infrastructure to validate scaling to billions of documents and thousands of queries/updates per second. Ongoing work continues to optimize and benchmark SolrCloud performance at large scales.

Efficient Scalable Search in a Multi-Tenant Environment: Presented by Harry H...

Lucidworks

This document discusses efficient scalable search in a multi-tenant environment. It describes Bloomberg Vault, which hosts large volumes of enterprise communications and documents for compliance. The system uses a distributed architecture with shards that are loaded on demand to serve search queries. Security is ensured by dynamically generating field values that encapsulate access permissions for each user's view of a document.

Solrcloud Leader Election

ravikgiitk

SolrCloud uses Zookeeper to elect a leader node for each shard. The leader coordinates write requests to ensure consistency. When the leader dies, Zookeeper detects this and elects a new leader based on the nodes' sequence numbers registered with Zookeeper. The new leader syncs updates with replicas and can replay logs if any replicas are too far behind. This allows write requests to continue being served with high availability despite leader failures.

Lucene Revolution 2013 - Scaling Solr Cloud for Large-scale Social Media Anal...

thelabdude

My presentation focuses on how we implemented Solr 4 to be the cornerstone of our social marketing analytics platform. Our platform analyzes relationships, behaviors, and conversations between 30,000 brands and 100M social accounts every 15 minutes. Combined with our Hadoop cluster, we have achieved throughput rates greater than 8,000 documents per second. Our index currently contains more than 620M documents and is growing by 3 to 4 million documents per day. My presentation will include details about: 1) Designing a Solr Cloud cluster for scalability and high-availability using sharding and replication with Zookeeper, 2) Operations concerns like how to handle a failed node and monitoring, 3) How we deal with indexing big data from Pig/Hadoop as an example of using the CloudSolrServer in SolrJ and managing searchers for high indexing throughput, 4) Example uses of key features like real-time gets, atomic updates, custom hashing, and distributed facets. Attendees will come away from this presentation with a real-world use case that proves Solr 4 is scalable, stable, and is production ready.

Streaming Aggregation in Solr - New Horizons for Search: Presented by Erick E...

Lucidworks

Erick Erickson presented on Streaming Aggregation in Solr. SA allows processing very large result sets across Solr nodes in parallel. It enables SQL-like queries and arbitrary operations on result sets. SA uses docValues fields and tuples to export data for processing. Parallel SQL and Streaming Expressions provide additional interfaces to SA beyond Java code. SA complements Solr's search capabilities by enabling analytics on large result sets.

High Performance Solr

Shalin Shekhar Mangar

This document provides tips for tuning Solr for high performance. It discusses optimizing queries and facets for CPU usage, tuning memory usage such as using docValues, optimizing disk usage through merge policies and commit settings, reducing network overhead through batching and caching, and techniques like deep paging to improve performance for large result sets. The document emphasizes only indexing and retrieving necessary fields to reduce resource usage and tuning garbage collection to avoid pauses.

Benchmarking Solr Performance at Scale

thelabdude

Organizations continue to adopt Solr because of its ability to scale to meet even the most demanding workflows. Recently, LucidWorks has been leading the effort to identify, measure, and expand the limits of Solr. As part of this effort, we've learned a few things along the way that should prove useful for any organization wanting to scale Solr. Attendees will come away with a better understanding of how sharding and replication impact performance. Also, no benchmark is useful without being repeatable; Tim will also cover how to perform similar tests using the Solr-Scale-Toolkit in Amazon EC2.

Webinar: What's New in Solr 6

Lucidworks

Parallel Computing with SolrCloud: Presented by Joel Bernstein, Alfresco

Lucidworks

This document summarizes Joel Bernstein's presentation on parallel SQL in Solr 6.0. The key points are: 1. SQL provides an optimizer to choose the best query plan for complex queries in Solr, avoiding the need for users to determine optimal faceting APIs or parameters. 2. SQL queries in Solr 6.0 can perform distributed joins, aggregations, sorting, and filtering using Solr search predicates. Aggregations can be performed using either map-reduce or facets. 3. Under the hood, SQL queries are compiled to TupleStreams which are serialized to Streaming Expressions and executed in parallel across worker collections using Solr's streaming API framework.

Solr + Hadoop = Big Data Search

Mark Miller

This document discusses integrating Apache Solr with Apache Hadoop for big data search capabilities. It provides background on Mark Miller and the history of search on Hadoop. It outlines how Solr, Lucene, Hadoop, and related projects can be integrated to allow full-text search across large datasets in HDFS. Specific integration points discussed include allowing Solr to read and write directly to HDFS, custom directory support in Solr, replication support, and using Morphlines for extraction, transformation, and loading of data into Solr.

Near Real Time Indexing: Presented by Umesh Prasad & Thejus V M, Flipkart

Lucidworks

This document summarizes a presentation given by Umesh Prasad and Thejus V M of Flipkart on building a real-time search index for e-commerce. It discusses the need for real-time indexing to support high update rates and microservices architecture at Flipkart. It evaluates using SolrCloud but finds that update-by-delete-and-add hinders performance. The presentation then describes Flipkart's approach using a near real-time Lucene store with optimized data structures and filtering to enable low-latency search across updated documents.

Solr Highlighting at Full Speed: Presented by Timothy Rodriguez, Bloomberg & ...

Lucidworks

The document discusses highlighting in legal search applications. It provides an overview of existing highlighters in Solr and their tradeoffs between accuracy, speed and memory usage. It then describes improvements made to the standard highlighter and the development of a new Unified Highlighter that aims to improve performance while maintaining accuracy. Benchmark results show the Unified Highlighter performs similarly or better than other highlighters. Future work is discussed to further improve accuracy and relevancy.

Queue Based Solr Indexing with Collection Management: Presented by Devansh Dh...

Lucidworks

This document discusses Gannett's transition from synchronous to asynchronous indexing using queues. It summarizes Gannett's current indexing architecture, the problems with synchronous operations like outages during schema changes, and how moving to a queue-based system with RabbitMQ allows for faster schema changes, auto-scaling of workers, and more resilient indexing with minimal authoring impact. The queue-based approach includes real-time, batch, and prep queues to handle different types of document changes and updates in a decoupled, eventually consistent manner.

What's hot

Your Big Data Stack is Too Big!: Presented by Timothy Potter, Lucidworks

Lucidworks

SolrCloud on Hadoop

Alex Moundalexis

SearchHub - How to Spend Your Summer Keeping it Real: Presented by Grant Inge...

Lucidworks

Mail Search As A Sercive: Presented by Rishi Easwaran, Aol

Lucidworks

Rebalance API for SolrCloud: Presented by Nitin Sharma, Netflix & Suruchi Sha...

Lucidworks

Deploying and managing Solr at scale

Anshum Gupta

Introduction to SolrCloud

Varun Thacker

Data Engineering with Solr and Spark

Lucidworks

Faceting Optimizations for Solr: Presented by Toke Eskildsen, State & Univers...

Lucidworks

Scaling SolrCloud to a large number of Collections

Anshum Gupta

Efficient Scalable Search in a Multi-Tenant Environment: Presented by Harry H...

Lucidworks

Solrcloud Leader Election

ravikgiitk

Lucene Revolution 2013 - Scaling Solr Cloud for Large-scale Social Media Anal...

thelabdude

Streaming Aggregation in Solr - New Horizons for Search: Presented by Erick E...

Lucidworks

High Performance Solr

Shalin Shekhar Mangar

Benchmarking Solr Performance at Scale

thelabdude

Webinar: What's New in Solr 6

Lucidworks

Parallel Computing with SolrCloud: Presented by Joel Bernstein, Alfresco

Lucidworks

Solr + Hadoop = Big Data Search

Mark Miller

Near Real Time Indexing: Presented by Umesh Prasad & Thejus V M, Flipkart

Lucidworks

What's hot (20)

Your Big Data Stack is Too Big!: Presented by Timothy Potter, Lucidworks

SolrCloud on Hadoop

SearchHub - How to Spend Your Summer Keeping it Real: Presented by Grant Inge...

Mail Search As A Sercive: Presented by Rishi Easwaran, Aol

Rebalance API for SolrCloud: Presented by Nitin Sharma, Netflix & Suruchi Sha...

Deploying and managing Solr at scale

Introduction to SolrCloud

Data Engineering with Solr and Spark

Faceting Optimizations for Solr: Presented by Toke Eskildsen, State & Univers...

Scaling SolrCloud to a large number of Collections

Efficient Scalable Search in a Multi-Tenant Environment: Presented by Harry H...

Solrcloud Leader Election

Lucene Revolution 2013 - Scaling Solr Cloud for Large-scale Social Media Anal...

Streaming Aggregation in Solr - New Horizons for Search: Presented by Erick E...

High Performance Solr

Benchmarking Solr Performance at Scale

Webinar: What's New in Solr 6

Parallel Computing with SolrCloud: Presented by Joel Bernstein, Alfresco

Solr + Hadoop = Big Data Search

Near Real Time Indexing: Presented by Umesh Prasad & Thejus V M, Flipkart

Viewers also liked

Solr Highlighting at Full Speed: Presented by Timothy Rodriguez, Bloomberg & ...

Lucidworks

Queue Based Solr Indexing with Collection Management: Presented by Devansh Dh...

Lucidworks

Events Processing and Data Analysis with Lucidworks Fusion: Presented by Kira...

Lucidworks

The document discusses using signals and events collected from user interactions to power recommendations and analytics. It describes how signals are collected using Snowplow and stored in Solr. Signals can then be aggregated using Spark to generate recommendations by boosting related search results or constructing a co-occurrence graph. The demo shows how a recommendation API uses aggregated signals to modify search behavior based on a user's environment.

PlayStation and Lucene - Indexing 1M documents per second: Presented by Alexa...

Lucidworks

Large Scale Log Analytics with Solr: Presented by Rafał Kuć & Radu Gheorghe, ...

Lucidworks

Understanding the Solr Security Framekwork: Presented by Anshum Gupta, IBM

Lucidworks

Anshum Gupta presented on the Apache Solr security framework. He began with an introduction of himself and overview of Apache Lucene and Solr. The presentation then covered the need for security in Solr, available security options which include SSL, ZooKeeper ACLs, and authentication and authorization frameworks. Gupta discussed the authentication and authorization plugin architectures, available plugins including BasicAuth and Kerberos, and benefits of the security frameworks such as enabling multi-tenant and access controlled systems.

Build a Great Application in Minutes!: Presented by Stefan Olafsson, Twigkit

Lucidworks

This document discusses building great applications quickly using Twigkit's tools. It focuses on using connectors, algorithms, and tools like Zookeeper to handle the complex parts so developers can focus on the user experience. It recommends prototyping early and designing for your audience to bridge the gap between design and development and ensure success. The overall goal is to connect people with information quickly using Twigkit's prototype driven development approach.

State of Solr Security 2016: Presented by Ishan Chattopadhyaya, Lucidworks

Lucidworks

Where Search Meets Machine Learning: Presented by Diana Hu & Joaquin Delgado,...

Lucidworks

This document discusses how machine learning problems can be framed as search-based systems and how search technologies can be leveraged to build and serve machine learning models at scale. It begins with an introduction to search and information retrieval systems. It then discusses how recommender systems and other machine learning problems can be viewed as search problems involving relevance, ranking, and retrieval. The document explores options for integrating machine learning models into search systems like Solr and Lucene using techniques like custom scoring plugins and the Predictive Model Markup Language (PMML). It provides examples of training models and exporting them to PMML for use in search systems.

Building a Vibrant Search Ecosystem @ Bloomberg: Presented by Steven Bower & ...

Lucidworks

This document summarizes a presentation given by Steven Bower and Ken LaPorte of Bloomberg about building their search ecosystem. They started by reviewing Bloomberg's existing fragmented search solutions and selected Apache Solr as their new platform. They created a specialized search team and designed Solr as a middleware service. This supported migrating over 1000 applications and indexing over 10 billion documents. They discussed challenges around monitoring, configuration management, and infrastructure scaling. Their solutions involved improved monitoring tools, adopting DevOps practices like Git and continuous integration, and optimizing hardware resources. Future plans include containerization, failure prediction, and expanding Solr's capabilities.

Scaling your time series data with Newts

jwhite08

This document discusses Newts, an open-source time series database. It describes Newts' data model for storing time-indexed metrics and attributes in tables. It also covers Newts' indexing of metrics to enable fast searching and filtering. Finally, it discusses how Newts integrates with data collection systems and supports APIs for retrieving measurements with options for aggregation, filtering and expressions.

Webinar: Site Search in an Hour with Fusion

Lucidworks

Aplicación de aprendizaje automático en minería de datos en la evaluación del...

mariasusanaSD

A How-to Guide to OAuth & API Security

CA API Management

This document provides an overview of OAuth and how it can be implemented securely. It defines OAuth as an emerging web standard for authorizing limited access to applications and data. A simple example is given showing how OAuth allows users to grant access to resources like photos on Flickr to third party applications like a photo printing site. The document also discusses how OAuth formalizes delegation of identity mapping to users while promoting a model of least privilege. It notes that while OAuth is important, it only one component of a full API access control and security solution, and the benefits Layer 7 provides in implementing OAuth securely for enterprises.

Goraft and InfluxDB

nobu_k

Rebuilding Solr 6 Examples - Layer by Layer: Presented by Alexandre Rafalovit...

Lucidworks

Best practices-for-production-environments

Artem Kovardin

This document provides best practices for production environments in Go projects, including recommendations for development environments, repository structure, formatting and style, configuration, logging and telemetry, testing and validation, dependency management, and building and deploying. It recommends using a single GOPATH, structuring repositories with README, Makefile, and package folders, formatting code with go fmt and following Google style guidelines. It also discusses setting configuration via flags, implementing logging and metrics, writing tests including integration tests, managing dependencies with the VENDOR folder, and building and deploying through Makefiles and scaling stateless services.

Building a Real-Time News Search Engine: Presented by Ramkumar Aiyengar, Bloo...

Lucidworks

The document discusses the challenges of building a news search engine at Bloomberg L.P. It describes how Bloomberg uses Apache Solr/Lucene to index millions of news stories and handle complex search queries from customers. Some key challenges discussed include optimizing searches over huge numbers of documents and metadata fields, handling arbitrarily complex queries, and developing an alerting system to notify users of new matching results. The system has been scaled up to include thousands of Solr cores distributed across data centers to efficiently search and retrieve news content.

Near Real Time Indexing Kafka Messages into Apache Blur: Presented by Dibyend...

Lucidworks

This document discusses Pearson's use of Apache Blur for distributed search and indexing of data from Kafka streams into Blur. It provides an overview of Pearson's learning platform and data architecture, describes the benefits of using Blur including its scalability, fault tolerance and query support. It also outlines the challenges of integrating Kafka streams with Blur using Spark and the solution developed to provide a reliable, low-level Kafka consumer within Spark that indexes messages from Kafka into Blur in near real-time.

Chronix Poster for the Poster Session FAST 2017

Florian Lautenschlager

Chronix is a domain specific time series database designed for anomaly detection in operational data. It is optimized for the needs of anomaly detection by supporting domain specific data types, analysis algorithms, data models, and query languages. It aims to address limitations of general purpose time series databases by exploiting characteristics of operational data through features like optional pre-computation of extras, timestamp compression, domain specific records and compression techniques, and multi-dimensional storage. An evaluation using data from five industry projects found that Chronix has significantly smaller memory and storage footprints and faster data retrieval and analysis times compared to other time series databases.

Viewers also liked (20)