Event Pipe - Lambda Architecture
•Download as PPTX, PDF•
3 likes•1,183 views
Travelbird Lambda Architecture with not so big data Bringing realtime business intelligence into reality.
Report
Share
Report
Share
Recommended
Onyx data processing the clojure way
Onyx is a data processing framework for Clojure that allows users to define workflows, functions, and windows to process streaming and batch data across distributed clusters. It uses concepts like peers, virtual peers, and Zookeeper for scheduling and Aeron for messaging. Users can write Onyx jobs in Clojure to perform ETL, analytics, and other data processing tasks in a declarative way.
Hadoop Pig: MapReduce the easy way!
Hadoop Pig provides a high-level language called Pig Latin for analyzing large datasets in Hadoop. Pig Latin allows users to express data analysis jobs as sequences of operations like filtering, grouping, joining and ordering data. This simplifies programming with Hadoop by avoiding the need to write Java MapReduce code directly. Pig jobs are compiled into sequences of MapReduce jobs that operate in parallel on large datasets distributed across a Hadoop cluster.
Fluentd and Docker - running fluentd within a docker container
Walks through the basics of Fluentd, as run on Docker. Introduces a demo where fluentd is run in a docker container.
ELK: a log management framework
This document summarizes an overview of the ELK stack presented at LinuxCon Europe 2016. It discusses the components of ELK including Beats, Logstash, Elasticsearch, and Kibana. It provides examples of using these components to collect, parse, store, search, and visualize log data. Specific topics covered include collecting log files using Filebeat and Logstash, parsing logs with Logstash filters, visualizing data in Kibana, programming Elasticsearch with REST APIs and client libraries, and alerting using the open source ESWatcher tool.
Bigdata : Big picture
Big data hadoop flume spark cloudera Oracle big data appliance apache , oracle loader for hadoop, Big data copy. Exadata to Big data appliance. bilginc It academy.
HBaseCon 2013: OpenTSDB at Box
This document discusses Box's use of OpenTSDB to store and query time series metrics data. It describes how OpenTSDB provides a scalable and easy way to collect, store, and query large amounts of metrics data compared to previous solutions. It includes examples of using OpenTSDB, such as a script to collect MySQL metrics and adding it as a cron job, and examples of querying the data through the OpenTSDB API and web interface. It also provides some statistics about Box's OpenTSDB deployment and next steps.
Scalable real-time processing techniques
This document discusses techniques for scalable real-time processing and counting of streaming data. It outlines several approaches for counting distinct items and top items in a stream in real-time, including using hashes, bitmaps, Bloom filters, HyperLogLog counters, and Count-Min sketches. It also discusses using these techniques to power features like recommendations by analyzing item co-occurrence matrices from user activity streams.
ClickHouse Data Warehouse 101: The First Billion Rows, by Alexander Zaitsev a...
Columnar stores like ClickHouse enable users to pull insights from big data in seconds, but only if you set things up correctly. This talk will walk through how to implement a data warehouse that contains 1.3 billion rows using the famous NY Yellow Cab ride data. We'll start with basic data implementation including clustering and table definitions, then show how to load efficiently. Next, we'll discuss important features like dictionaries and materialized views, and how they improve query efficiency. We'll end by demonstrating typical queries to illustrate the kind of inferences you can draw rapidly from a well-designed data warehouse. It should be enough to get you started--the next billion rows is up to you!
Recommended
Onyx data processing the clojure way
Onyx is a data processing framework for Clojure that allows users to define workflows, functions, and windows to process streaming and batch data across distributed clusters. It uses concepts like peers, virtual peers, and Zookeeper for scheduling and Aeron for messaging. Users can write Onyx jobs in Clojure to perform ETL, analytics, and other data processing tasks in a declarative way.
Hadoop Pig: MapReduce the easy way!
Hadoop Pig provides a high-level language called Pig Latin for analyzing large datasets in Hadoop. Pig Latin allows users to express data analysis jobs as sequences of operations like filtering, grouping, joining and ordering data. This simplifies programming with Hadoop by avoiding the need to write Java MapReduce code directly. Pig jobs are compiled into sequences of MapReduce jobs that operate in parallel on large datasets distributed across a Hadoop cluster.
Fluentd and Docker - running fluentd within a docker container
Walks through the basics of Fluentd, as run on Docker. Introduces a demo where fluentd is run in a docker container.
ELK: a log management framework
This document summarizes an overview of the ELK stack presented at LinuxCon Europe 2016. It discusses the components of ELK including Beats, Logstash, Elasticsearch, and Kibana. It provides examples of using these components to collect, parse, store, search, and visualize log data. Specific topics covered include collecting log files using Filebeat and Logstash, parsing logs with Logstash filters, visualizing data in Kibana, programming Elasticsearch with REST APIs and client libraries, and alerting using the open source ESWatcher tool.
Bigdata : Big picture
Big data hadoop flume spark cloudera Oracle big data appliance apache , oracle loader for hadoop, Big data copy. Exadata to Big data appliance. bilginc It academy.
HBaseCon 2013: OpenTSDB at Box
This document discusses Box's use of OpenTSDB to store and query time series metrics data. It describes how OpenTSDB provides a scalable and easy way to collect, store, and query large amounts of metrics data compared to previous solutions. It includes examples of using OpenTSDB, such as a script to collect MySQL metrics and adding it as a cron job, and examples of querying the data through the OpenTSDB API and web interface. It also provides some statistics about Box's OpenTSDB deployment and next steps.
Scalable real-time processing techniques
This document discusses techniques for scalable real-time processing and counting of streaming data. It outlines several approaches for counting distinct items and top items in a stream in real-time, including using hashes, bitmaps, Bloom filters, HyperLogLog counters, and Count-Min sketches. It also discusses using these techniques to power features like recommendations by analyzing item co-occurrence matrices from user activity streams.
ClickHouse Data Warehouse 101: The First Billion Rows, by Alexander Zaitsev a...
Columnar stores like ClickHouse enable users to pull insights from big data in seconds, but only if you set things up correctly. This talk will walk through how to implement a data warehouse that contains 1.3 billion rows using the famous NY Yellow Cab ride data. We'll start with basic data implementation including clustering and table definitions, then show how to load efficiently. Next, we'll discuss important features like dictionaries and materialized views, and how they improve query efficiency. We'll end by demonstrating typical queries to illustrate the kind of inferences you can draw rapidly from a well-designed data warehouse. It should be enough to get you started--the next billion rows is up to you!
Presto Bangalore Meetup1 Presto Raptor@ola
Presto Raptor is a columnar storage system designed to work natively with Presto that provides real-time analytics capabilities. Raptor is optimized for high performance on flash storage and scales to handle large volumes of data and high query throughput. Key features of Raptor include bucketed tables to co-locate related data and enable fast joins, temporal columns to optimize queries on time-series data, and physical data awareness to skip unnecessary data during queries. Raptor can be used for real-time dashboards, funnels, and event analytics on large datasets stored in its distributed database.
Solving Low Latency Query Over Big Data with Spark SQL-(Julien Pierre, Micros...
The document discusses a data analytics platform that provides capabilities for ingesting, processing, storing, and analyzing data at scale. It includes instrumentation and ingestion of data from various sources, processing and storage using technologies like Spark and Cosmos, and reporting and analytics through tools like Zeppelin and Avocado. The platform is designed for a mobile-first analytics experience and experimentation.
Paul Dix (Founder InfluxDB) - Organising Metrics at #DOXLON
The document discusses organizing time series metrics data and compares hierarchical and tagged approaches. It argues that a tagged approach is superior as it allows for more flexible querying of metrics by different dimensions and tags. A tagged approach stores tags as metadata alongside metric names and values, allowing filtering and aggregation by any tag combination. This enables more powerful queries and computations across diverse sets of metrics than is possible with a hierarchical organization.
IOT with PostgreSQL
This document provides an overview of using PostgreSQL for IoT applications. Chris Ellis discusses why PostgreSQL is a good fit for IoT due to its flexibility and extensibility. He describes various ways of storing, loading, and processing IoT time series and sensor data in PostgreSQL, including partitioning, batch loading, and window functions. The document also briefly mentions the TimescaleDB extension for additional time series functionality.
Yahoo! Hadoop User Group - May 2010 Meetup - Apache Hadoop Release Plans for ...
Tom White discusses release plans for Apache Hadoop 0.21, which includes fixing blockers, creating build artifacts, testing, voting, and later bug fix releases. The 0.21.0 release is aimed at improving quality and should not be used for production. It features a new MapReduce API, file context symlinks, and hundreds of bug fixes and other improvements.
Device Synchronization with Javascript and PouchDB
This document provides code examples for using PouchDB, an open-source JavaScript database, to set up a local database, synchronize it with a remote CouchDB database, handle conflicts, and implement messaging through document publishing and subscriptions. It includes snippets for installing PouchDB, initializing a database, syncing with options to handle changes live and errors, resolving conflicts by selecting a revision, and handling message documents with a specific channel through putting and logging documents.
Dangerous on ClickHouse in 30 minutes, by Robert Hodges, Altinity CEO
- The document summarizes a presentation about ClickHouse, an open source column-oriented database management system.
- It discusses how ClickHouse stores and indexes data to enable fast queries, how it scales horizontally across servers, and how different engines like MergeTree and ReplicatedMergeTree allow for high performance and fault tolerance.
- Examples are provided showing how ClickHouse can quickly analyze large datasets with SQL and optimize queries using its features like distributed processing, partitioning, and specialized functions.
Redis: REmote DIctionary Server
Redis is an in-memory data structure store that can be used as a database, cache, and message broker. It supports string, list, set and sorted set data types and provides operations on each type. Redis is fast, open source, and can be used for tasks like caching, leaderboards, and workload distribution between processes.
Buildingsocialanalyticstoolwithmongodb
This document discusses building a social analytics tool using MongoDB from a developer's perspective. It covers using MongoDB for its schema-less data and ability to handle fast read-write operations. Key topics include using aggregation queries to gain insights from data by chaining queries together and filtering/manipulating results at each stage. JavaScript capabilities in MongoDB allow applying business logic directly to data. Examples demonstrate removing garbage data and stopwords. Indexes, current progress, and tips/tricks learned around cloning collections and removing vs dropping are also covered, with a demo planned.
Elasticsearch, Logstash, Kibana. Cool search, analytics, data mining and more...
In the age of information and big data, ability to quickly and easily find a needle in a haystack is extremely important. Elasticsearch is a distributed and scalable search engine which provides rich and flexible search capabilities. Social networks (Facebook, LinkedIn), media services (Netflix, SoundCloud), Q&A sites (StackOverflow, Quora, StackExchange) and even GitHub - they all find data for you using Elasticsearch. In conjunction with Logstash and Kibana, Elasticsearch becomes a powerful log engine which allows to process, store, analyze, search through and visualize your logs.
Video: https://www.youtube.com/watch?v=GL7xC5kpb-c
Scripts for the Demo: https://github.com/opanchenko/morning-at-lohika-ELK
If you give a mouse a clickhouse, by Alex Hofsteede, Sentry
Presented at ClickHouse Meetup in Cloudflare, February 19, 2019
If you give a mouse a clickhouse, by Alex Hofsteede, Sentry
RESTful API – How to Consume, Extract, Store and Visualize Data with InfluxDB...
Nowadays, every single modern application, system or solution does expose a RESTful API. On one hand, this is absolutely great and it has led to where we are today, having hundreds of other solutions or applications that can leverage these APIs, extend them, or even build on top of them.
On the other hand, we have difficulty monitoring these new and modern systems, applications or solutions.
In this session, we will learn how to query the data first using Swagger, when available, extract and parse the data that’s useful for us, store it in InfluxDB, and finally how to create beautiful and meaningful dashboards to have everything on a single pane of glass.
Hdfs high availability
This document discusses high availability solutions for HDFS and proposes AvatarNode as an improvement. AvatarNode uses an active-standby pair of NameNodes coordinated by Zookeeper. During normal operation, the active NameNode writes transaction logs to persistent storage. The standby NameNode reads the logs to keep its metadata up-to-date. Failover occurs within seconds by switching the roles and updating Zookeeper. This allows clients to retrieve the new primary and resume operations with minimal downtime.
Elk stack
Jilles van Gurp presents on the ELK stack and how it is used at Linko to analyze logs from applications servers, Nginx, and Collectd. The ELK stack consists of Elasticsearch for storage and search, Logstash for processing and transporting logs, and Kibana for visualization. At Linko, Logstash collects logs and sends them to Elasticsearch for storage and search. Logs are filtered and parsed by Logstash using grok patterns before being sent to Elasticsearch. Kibana dashboards then allow users to explore and analyze logs in real-time from Elasticsearch. While the ELK stack is powerful, there are some operational gotchas to watch out for like node restarts impacting availability and field data caching
Cascalog internal dsl_preso
Cascalog is an internal DSL for Clojure that allows defining MapReduce workflows for Hadoop. It provides helper functions, a way to define custom functions analogous to UDFs, and functions to programmatically generate all possible data aggregations from an input based on business requirements. The workflows can be unit tested and executed on Hadoop. Cascalog abstracts away lower-level MapReduce details and allows defining the entire workflow within a single language.
Rupy2012 ArangoDB Workshop Part1
The document outlines an agenda for a workshop on ArangoDB and Ashikawa. The agenda includes introducing ArangoDB, installing it, performing CRUD operations, using the query language, and building a small example with the Ruby driver Ashikawa. It also provides information on importing data and performing queries on ArangoDB.
Luigi presentation OA Summit
Foursquare uses Luigi to manage their complex data workflows. Luigi allows them to define tasks with dependencies in Python code rather than XML, making the workflows easier to write, test, visualize, and reuse components of. It also avoids wasted time from Cron jobs waiting and helps ensure tasks are only run once through its centralized scheduler. This provides a more robust replacement for both Cron jobs and Oozie workflows at Foursquare.
Building a Scalable Distributed Stats Infrastructure with Storm and KairosDB
Building a Scalable Distributed Stats Infrastructure with Storm and KairosDB
Many startups collect and display stats and other time-series data for their users. A supposedly-simple NoSQL option such as MongoDB is often chosen to get started... which soon becomes 50 distributed replica sets as volume increases. This talk describes how we designed a scalable distributed stats infrastructure from the ground up. KairosDB, a rewrite of OpenTSDB built on top of Cassandra, provides a solid foundation for storing time-series data. Unfortunately, though, it has some limitations: millisecond time granularity and lack of atomic upsert operations which make counting (critical to any stats infrastructure) a challenge. Additionally, running KairosDB atop Cassandra inside AWS brings its own set of challenges, such as managing Cassandra seeds and AWS security groups as you grow or shrink your Cassandra ring. In this deep-dive talk, we explore how we've used a mix of open-source and in-house tools to tackle these challenges and build a robust, scalable, distributed stats infrastructure.
From Zero to Production Hero: Log Analysis with Elasticsearch (from Velocity ...
From Zero to Production Hero: Log Analysis with Elasticsearch (from Velocity ...Sematext Group, Inc.
This talk covers the basics of centralizing logs in Elasticsearch and all the strategies that make it scale with billions of documents in production. Topics include:
- Time-based indices and index templates to efficiently slice your data
- Different node tiers to de-couple reading from writing, heavy traffic from low traffic
- Tuning various Elasticsearch and OS settings to maximize throughput and search performance
- Configuring tools such as logstash and rsyslog to maximize throughput and minimize overheadPractical Hadoop using Pig
So you want to get started with Hadoop, but how. This session will show you how to get started with Hadoop development using Pig. Prior Hadoop experience is not needed.
Thursday, May 8th, 02:00pm-02:50pm
WebWorkersCamp 2010
Presentation from my talk at WebWorkersCamp 2010, probably a bit hard to follow with the slides only.
Non-Relational Databases: This hurts. I like it.
The document discusses non-relational databases, providing an overview of their characteristics and comparing them to relational databases. It outlines some popular non-relational database platforms, and uses the example of an open government project to demonstrate how CouchDB could be used to store and query schema-less data in a scalable way.
More Related Content
What's hot
Presto Bangalore Meetup1 Presto Raptor@ola
Presto Raptor is a columnar storage system designed to work natively with Presto that provides real-time analytics capabilities. Raptor is optimized for high performance on flash storage and scales to handle large volumes of data and high query throughput. Key features of Raptor include bucketed tables to co-locate related data and enable fast joins, temporal columns to optimize queries on time-series data, and physical data awareness to skip unnecessary data during queries. Raptor can be used for real-time dashboards, funnels, and event analytics on large datasets stored in its distributed database.
Solving Low Latency Query Over Big Data with Spark SQL-(Julien Pierre, Micros...
The document discusses a data analytics platform that provides capabilities for ingesting, processing, storing, and analyzing data at scale. It includes instrumentation and ingestion of data from various sources, processing and storage using technologies like Spark and Cosmos, and reporting and analytics through tools like Zeppelin and Avocado. The platform is designed for a mobile-first analytics experience and experimentation.
Paul Dix (Founder InfluxDB) - Organising Metrics at #DOXLON
The document discusses organizing time series metrics data and compares hierarchical and tagged approaches. It argues that a tagged approach is superior as it allows for more flexible querying of metrics by different dimensions and tags. A tagged approach stores tags as metadata alongside metric names and values, allowing filtering and aggregation by any tag combination. This enables more powerful queries and computations across diverse sets of metrics than is possible with a hierarchical organization.
IOT with PostgreSQL
This document provides an overview of using PostgreSQL for IoT applications. Chris Ellis discusses why PostgreSQL is a good fit for IoT due to its flexibility and extensibility. He describes various ways of storing, loading, and processing IoT time series and sensor data in PostgreSQL, including partitioning, batch loading, and window functions. The document also briefly mentions the TimescaleDB extension for additional time series functionality.
Yahoo! Hadoop User Group - May 2010 Meetup - Apache Hadoop Release Plans for ...
Tom White discusses release plans for Apache Hadoop 0.21, which includes fixing blockers, creating build artifacts, testing, voting, and later bug fix releases. The 0.21.0 release is aimed at improving quality and should not be used for production. It features a new MapReduce API, file context symlinks, and hundreds of bug fixes and other improvements.
Device Synchronization with Javascript and PouchDB
This document provides code examples for using PouchDB, an open-source JavaScript database, to set up a local database, synchronize it with a remote CouchDB database, handle conflicts, and implement messaging through document publishing and subscriptions. It includes snippets for installing PouchDB, initializing a database, syncing with options to handle changes live and errors, resolving conflicts by selecting a revision, and handling message documents with a specific channel through putting and logging documents.
Dangerous on ClickHouse in 30 minutes, by Robert Hodges, Altinity CEO
- The document summarizes a presentation about ClickHouse, an open source column-oriented database management system.
- It discusses how ClickHouse stores and indexes data to enable fast queries, how it scales horizontally across servers, and how different engines like MergeTree and ReplicatedMergeTree allow for high performance and fault tolerance.
- Examples are provided showing how ClickHouse can quickly analyze large datasets with SQL and optimize queries using its features like distributed processing, partitioning, and specialized functions.
Redis: REmote DIctionary Server
Redis is an in-memory data structure store that can be used as a database, cache, and message broker. It supports string, list, set and sorted set data types and provides operations on each type. Redis is fast, open source, and can be used for tasks like caching, leaderboards, and workload distribution between processes.
Buildingsocialanalyticstoolwithmongodb
This document discusses building a social analytics tool using MongoDB from a developer's perspective. It covers using MongoDB for its schema-less data and ability to handle fast read-write operations. Key topics include using aggregation queries to gain insights from data by chaining queries together and filtering/manipulating results at each stage. JavaScript capabilities in MongoDB allow applying business logic directly to data. Examples demonstrate removing garbage data and stopwords. Indexes, current progress, and tips/tricks learned around cloning collections and removing vs dropping are also covered, with a demo planned.
Elasticsearch, Logstash, Kibana. Cool search, analytics, data mining and more...
In the age of information and big data, ability to quickly and easily find a needle in a haystack is extremely important. Elasticsearch is a distributed and scalable search engine which provides rich and flexible search capabilities. Social networks (Facebook, LinkedIn), media services (Netflix, SoundCloud), Q&A sites (StackOverflow, Quora, StackExchange) and even GitHub - they all find data for you using Elasticsearch. In conjunction with Logstash and Kibana, Elasticsearch becomes a powerful log engine which allows to process, store, analyze, search through and visualize your logs.
Video: https://www.youtube.com/watch?v=GL7xC5kpb-c
Scripts for the Demo: https://github.com/opanchenko/morning-at-lohika-ELK
If you give a mouse a clickhouse, by Alex Hofsteede, Sentry
Presented at ClickHouse Meetup in Cloudflare, February 19, 2019
If you give a mouse a clickhouse, by Alex Hofsteede, Sentry
RESTful API – How to Consume, Extract, Store and Visualize Data with InfluxDB...
Nowadays, every single modern application, system or solution does expose a RESTful API. On one hand, this is absolutely great and it has led to where we are today, having hundreds of other solutions or applications that can leverage these APIs, extend them, or even build on top of them.
On the other hand, we have difficulty monitoring these new and modern systems, applications or solutions.
In this session, we will learn how to query the data first using Swagger, when available, extract and parse the data that’s useful for us, store it in InfluxDB, and finally how to create beautiful and meaningful dashboards to have everything on a single pane of glass.
Hdfs high availability
This document discusses high availability solutions for HDFS and proposes AvatarNode as an improvement. AvatarNode uses an active-standby pair of NameNodes coordinated by Zookeeper. During normal operation, the active NameNode writes transaction logs to persistent storage. The standby NameNode reads the logs to keep its metadata up-to-date. Failover occurs within seconds by switching the roles and updating Zookeeper. This allows clients to retrieve the new primary and resume operations with minimal downtime.
Elk stack
Jilles van Gurp presents on the ELK stack and how it is used at Linko to analyze logs from applications servers, Nginx, and Collectd. The ELK stack consists of Elasticsearch for storage and search, Logstash for processing and transporting logs, and Kibana for visualization. At Linko, Logstash collects logs and sends them to Elasticsearch for storage and search. Logs are filtered and parsed by Logstash using grok patterns before being sent to Elasticsearch. Kibana dashboards then allow users to explore and analyze logs in real-time from Elasticsearch. While the ELK stack is powerful, there are some operational gotchas to watch out for like node restarts impacting availability and field data caching
Cascalog internal dsl_preso
Cascalog is an internal DSL for Clojure that allows defining MapReduce workflows for Hadoop. It provides helper functions, a way to define custom functions analogous to UDFs, and functions to programmatically generate all possible data aggregations from an input based on business requirements. The workflows can be unit tested and executed on Hadoop. Cascalog abstracts away lower-level MapReduce details and allows defining the entire workflow within a single language.
Rupy2012 ArangoDB Workshop Part1
The document outlines an agenda for a workshop on ArangoDB and Ashikawa. The agenda includes introducing ArangoDB, installing it, performing CRUD operations, using the query language, and building a small example with the Ruby driver Ashikawa. It also provides information on importing data and performing queries on ArangoDB.
Luigi presentation OA Summit
Foursquare uses Luigi to manage their complex data workflows. Luigi allows them to define tasks with dependencies in Python code rather than XML, making the workflows easier to write, test, visualize, and reuse components of. It also avoids wasted time from Cron jobs waiting and helps ensure tasks are only run once through its centralized scheduler. This provides a more robust replacement for both Cron jobs and Oozie workflows at Foursquare.
Building a Scalable Distributed Stats Infrastructure with Storm and KairosDB
Building a Scalable Distributed Stats Infrastructure with Storm and KairosDB
Many startups collect and display stats and other time-series data for their users. A supposedly-simple NoSQL option such as MongoDB is often chosen to get started... which soon becomes 50 distributed replica sets as volume increases. This talk describes how we designed a scalable distributed stats infrastructure from the ground up. KairosDB, a rewrite of OpenTSDB built on top of Cassandra, provides a solid foundation for storing time-series data. Unfortunately, though, it has some limitations: millisecond time granularity and lack of atomic upsert operations which make counting (critical to any stats infrastructure) a challenge. Additionally, running KairosDB atop Cassandra inside AWS brings its own set of challenges, such as managing Cassandra seeds and AWS security groups as you grow or shrink your Cassandra ring. In this deep-dive talk, we explore how we've used a mix of open-source and in-house tools to tackle these challenges and build a robust, scalable, distributed stats infrastructure.
From Zero to Production Hero: Log Analysis with Elasticsearch (from Velocity ...
From Zero to Production Hero: Log Analysis with Elasticsearch (from Velocity ...Sematext Group, Inc.
This talk covers the basics of centralizing logs in Elasticsearch and all the strategies that make it scale with billions of documents in production. Topics include:
- Time-based indices and index templates to efficiently slice your data
- Different node tiers to de-couple reading from writing, heavy traffic from low traffic
- Tuning various Elasticsearch and OS settings to maximize throughput and search performance
- Configuring tools such as logstash and rsyslog to maximize throughput and minimize overheadPractical Hadoop using Pig
So you want to get started with Hadoop, but how. This session will show you how to get started with Hadoop development using Pig. Prior Hadoop experience is not needed.
Thursday, May 8th, 02:00pm-02:50pm
What's hot (20)
Solving Low Latency Query Over Big Data with Spark SQL-(Julien Pierre, Micros...
Solving Low Latency Query Over Big Data with Spark SQL-(Julien Pierre, Micros...
Paul Dix (Founder InfluxDB) - Organising Metrics at #DOXLON
Paul Dix (Founder InfluxDB) - Organising Metrics at #DOXLON
Yahoo! Hadoop User Group - May 2010 Meetup - Apache Hadoop Release Plans for ...
Yahoo! Hadoop User Group - May 2010 Meetup - Apache Hadoop Release Plans for ...
Device Synchronization with Javascript and PouchDB
Device Synchronization with Javascript and PouchDB
Dangerous on ClickHouse in 30 minutes, by Robert Hodges, Altinity CEO
Dangerous on ClickHouse in 30 minutes, by Robert Hodges, Altinity CEO
Elasticsearch, Logstash, Kibana. Cool search, analytics, data mining and more...
Elasticsearch, Logstash, Kibana. Cool search, analytics, data mining and more...
If you give a mouse a clickhouse, by Alex Hofsteede, Sentry
If you give a mouse a clickhouse, by Alex Hofsteede, Sentry
RESTful API – How to Consume, Extract, Store and Visualize Data with InfluxDB...
RESTful API – How to Consume, Extract, Store and Visualize Data with InfluxDB...
Building a Scalable Distributed Stats Infrastructure with Storm and KairosDB
Building a Scalable Distributed Stats Infrastructure with Storm and KairosDB
From Zero to Production Hero: Log Analysis with Elasticsearch (from Velocity ...
From Zero to Production Hero: Log Analysis with Elasticsearch (from Velocity ...
Similar to Event Pipe - Lambda Architecture
WebWorkersCamp 2010
Presentation from my talk at WebWorkersCamp 2010, probably a bit hard to follow with the slides only.
Non-Relational Databases: This hurts. I like it.
The document discusses non-relational databases, providing an overview of their characteristics and comparing them to relational databases. It outlines some popular non-relational database platforms, and uses the example of an open government project to demonstrate how CouchDB could be used to store and query schema-less data in a scalable way.
VoltDB 소개
- In-Memory 기반 초고성능 관계형 데이터베이스
- Scale-Out, Shared Nothing 구조의 Cluster 기능
- 초당 수백만건의 Streaming 데이터 분석 처리
- Global 고객사 다수 - 삼성반도체, LG, Huawei, Intel, Nokia, Mitsubishi 등 고객사 보유
- 국내 중견 IT 기업 성능/기능 검증 진행 중
SD, a P2P bug tracking system
The document introduces SD, a peer-to-peer bug tracking tool developed by Best Practical to allow tracking bugs offline and syncing work across devices. SD uses a decentralized model where each installation can pull changes from any other replica. It supports syncing with other bug trackers like RT, Trac and Google Code. The author argues that cloud services make users dependent while SD empowers fully offline and distributed work by syncing like users naturally share files.
GalvanizeU Seattle: Eleven Almost-Truisms About Data
http://www.meetup.com/Seattle-Data-Science/events/223445403/
Almost a dozen almost-truisms about Data that almost everyone should consider carefully as they embark on a journey into Data Science. There are a number of preconceptions about working with data at scale where the realities beg to differ. This talk estimates that number to be at least eleven, through probably much larger. At least that number has a great line from a movie. Let's consider some of the less-intuitive directions in which this field is heading, along with likely consequences and corollaries -- especially for those who are just now beginning to study about the technologies, the processes, and the people involved.
Big data and APIs for PHP developers - SXSW 2011
Slides from the Big Data presentation by Bradley Holt, Laura Thomson, Julie Steele, Eli White, Dennis Yang, and David Zülke
Addressing dm-cloud
The document discusses addressing data management challenges in the cloud. It begins by introducing the scale of digital data using common size prefixes like kilobyte and petabyte. It then discusses sources of massive data from sensors, social media, and scientific experiments. The challenges of big data are defined through the 3Vs model of increasing volume, velocity and variety of data types. Cloud computing architectures and delivery models like IaaS, PaaS and SaaS are introduced as ways to provide elastic resources for data management. The concept of polyglot persistence using the appropriate data store for the job is discussed over relying solely on relational databases.
Spring Cloud Gateway - Nate Schutta
Spring Cloud Gateway is a gateway that provides routing, filtering, and monitoring capabilities for microservices. It is non-blocking and built on Spring Framework and uses reactive streams. Spring Cloud Gateway offers a simpler and more developer-friendly alternative to other gateway options that are often heavy-weight and difficult to integrate. It provides a Java-based configuration that gives developers control over routing, filtering, and other gateway features without vendor lock-in.
Teaching Elephants to Dance (Federal Audience): A Developer's Journey to Digi...
We can be brilliant developers, but we won’t succeed—and won’t lead our organizations to succeed—without a new perspective (if you will) and new assumptions about the components of the “technology ecosystem” that are fundamentally critical to our success. This includes the operators, QA team, DBAs, security folks, and even the pure business contingent—in most cases, each of these individuals and groups plays a critical role in the success of what we create and give birth to as developers. What we do in isolation might be genius, but if we insulate ourselves—especially with arrogance—from these colleagues, neither our code nor our organizations will realize their full potential, and most will fail. The bottom line is that our old ways are no longer viable, and as the elite within our industry, we will be the leaders and heroes who discard old assumptions and adopt a new perspective in this exciting journey to digital transformation—where the impossible can become reality.
Dataiku - hadoop ecosystem - @Epitech Paris - janvier 2014
Overview of Hadoop and its ecosystem: Pig, Hive, GraphLab, Mahout, Impala, Spark, Storm, ...
Presented at Epitech Paris on January 2014
UnConference for Georgia Southern Computer Science March 31, 2015
I presented to the Georgia Southern Computer Science ACM group. Rather than one topic for 90 minutes, I decided to do an UnConference. I presented them a list of 8-9 topics, let them vote on what to talk about, then repeated.
Each presentation was ~8 minutes, (Except Career) and was by no means an attempt to explain the full concept or technology. Only to wake up their interest.
Cristiano Rastelli - Atomic Design, Design Systems and React. Cool, but... - ...
Cristiano Rastelli gave a presentation about design systems and component-based design. He discussed Atomic Design, React, Cosmos, and examples from Badoo's design system. He emphasized that design systems provide consistency, efficiency, and collaboration benefits. Complex systems work best when evolved from simple, working systems rather than designed entirely from scratch.
Cristiano Rastelli - Atomic Design, Design Systems and React. Cool, but... - ...
The principles of Atomic Design have transformed (probably forever) the way we look at UI components and code modularization. Pattern Libraries and Design Systems – predominantly built in React – have become widespread across many companies. No doubts, these are cool tools and approaches, and we have all fallen in love with them. But... In this talk, I'll share not only the learnings but also all the "buts" that we have found in our exciting journey developing (in React, of course) a Design System for Badoo.
Tech
This document discusses trends in cutting edge technologies, including the evolution of platforms and programming languages. It covers the shift from mainframes and client-server architectures to modern mobile and cloud platforms driven by technologies like JavaScript, HTML5, and cloud computing. Key areas like big data, IoT, and DevOps are also summarized.
ERS downscale2016
The document describes an Entity Registry System (ERS) that allows for decentralized, linked data storage in a document store. It was designed to work in environments with poor network connectivity. The ERS uses contributors to write data, bridges to connect isolated parts of the system, and an optional aggregator for high-performance read-only data retrieval. Testing showed the ERS could tolerate disconnects and poor networks as long as connections lasted at least half a second. It was tested with up to 40 nodes and was able to reliably synchronize data in real-world simulation scenarios like a conference social network and remote merchants updating prices between villages via a mobile bridge.
Improve your Tech Quotient
This document discusses various topics related to website development and optimization. It covers front-end performance techniques like using content delivery networks and gzipping components. It also discusses tools for front-end performance analysis. Other topics covered include tag management systems, version control systems like Git and SVN, responsive vs adaptive design, and content management systems. The document provides information on technologies and best practices for building high performing websites.
Yahoo compares Storm and Spark
Bobby Evans and Tom Graves, the engineering leads for Spark and Storm development at Yahoo will talk about how these technologies are used on Yahoo's grids and reasons why to use one or the other.
Bobby Evans is the low latency data processing architect at Yahoo. He is a PMC member on many Apache projects including Storm, Hadoop, Spark, and Tez. His team is responsible for delivering Storm as a service to all of Yahoo and maintaining Spark on Yarn for Yahoo (Although Tom really does most of that work).
Tom Graves a Senior Software Engineer on the Platform team at Yahoo. He is an Apache PMC member on Hadoop, Spark, and Tez. His team is responsible for delivering and maintaining Spark on Yarn for Yahoo.
Hadoop and the Relational Database: The Best of Both Worlds
This document summarizes a presentation about the Splice Machine database product. Splice Machine is described as a SQL-on-Hadoop database that is ACID-compliant and can handle both OLTP and OLAP workloads. It provides typical relational database functionality like transactions and SQL on top of Apache Hadoop. Customers reportedly see a 10x improvement in price/performance compared to traditional databases. The presentation provides details on Splice Machine's architecture, performance benchmarks, customer use cases, and support for analytics and business intelligence tools.
Spark meets Spring
This document provides an overview of Spring XD, which allows for ingesting, processing, and exporting streaming and batch data. Some key points:
- Spring XD provides modules for sources, processors, and sinks to build streams for ingesting data from various sources and exporting to various systems. It also supports batch jobs.
- Core concepts include modules, streams, taps, and jobs. Streams are composed of sources, processors, and sinks. Taps dynamically add listeners. Jobs provide ETL and workflow capabilities.
- Spring XD supports ingesting from sources like Kafka, files, databases. It can process data in real-time or using batch and export to systems like HDFS, databases.
Scalability
The document discusses architecting systems to support millions of transactions per second (TPS). It covers several key topics:
1) Scaling out by adding more computation nodes is better than scaling up single nodes due to hardware limitations.
2) Distributed systems must be designed to handle failures, as they are inevitable as systems scale out.
3) Real-time processing should be minimized in favor of batch processing to improve scalability.
Similar to Event Pipe - Lambda Architecture (20)
GalvanizeU Seattle: Eleven Almost-Truisms About Data
GalvanizeU Seattle: Eleven Almost-Truisms About Data
Teaching Elephants to Dance (Federal Audience): A Developer's Journey to Digi...
Teaching Elephants to Dance (Federal Audience): A Developer's Journey to Digi...
Dataiku - hadoop ecosystem - @Epitech Paris - janvier 2014
Dataiku - hadoop ecosystem - @Epitech Paris - janvier 2014
UnConference for Georgia Southern Computer Science March 31, 2015
UnConference for Georgia Southern Computer Science March 31, 2015
Cristiano Rastelli - Atomic Design, Design Systems and React. Cool, but... - ...
Cristiano Rastelli - Atomic Design, Design Systems and React. Cool, but... - ...
Cristiano Rastelli - Atomic Design, Design Systems and React. Cool, but... - ...
Cristiano Rastelli - Atomic Design, Design Systems and React. Cool, but... - ...
Hadoop and the Relational Database: The Best of Both Worlds
Hadoop and the Relational Database: The Best of Both Worlds
Recently uploaded
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays, June 2024 with Maria Copot 20
Fueling AI with Great Data with Airbyte Webinar
This talk will focus on how to collect data from a variety of sources, leveraging this data for RAG and other GenAI use cases, and finally charting your course to productionalization.
Leveraging the Graph for Clinical Trials and Standards
Katja Glaß
OpenStudyBuilder Community Manager - Katja Glaß Consulting
Marius Conjeaud
Principal Consultant - Neo4j
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
How we managed to grow and scale a RAG application from zero to thousands of users in 7 months. Lessons from technical challenges around managing high load for LLMs, RAGs and Vector databases.
Northern Engraving | Nameplate Manufacturing Process - 2024
Manufacturing custom quality metal nameplates and badges involves several standard operations. Processes include sheet prep, lithography, screening, coating, punch press and inspection. All decoration is completed in the flat sheet with adhesive and tooling operations following. The possibilities for creating unique durable nameplates are endless. How will you create your brand identity? We can help!
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
HERE IS YOUR WEBINAR CONTENT! 'Mastering Customer Journey Management with Dr. Graham Hill'. We hope you find the webinar recording both insightful and enjoyable.
In this webinar, we explored essential aspects of Customer Journey Management and personalization. Here’s a summary of the key insights and topics discussed:
Key Takeaways:
Understanding the Customer Journey: Dr. Hill emphasized the importance of mapping and understanding the complete customer journey to identify touchpoints and opportunities for improvement.
Personalization Strategies: We discussed how to leverage data and insights to create personalized experiences that resonate with customers.
Technology Integration: Insights were shared on how inQuba’s advanced technology can streamline customer interactions and drive operational efficiency.
Apps Break Data
How information systems are built or acquired puts information, which is what they should be about, in a secondary place. Our language adapted accordingly, and we no longer talk about information systems but applications. Applications evolved in a way to break data into diverse fragments, tightly coupled with applications and expensive to integrate. The result is technical debt, which is re-paid by taking even bigger "loans", resulting in an ever-increasing technical debt. Software engineering and procurement practices work in sync with market forces to maintain this trend. This talk demonstrates how natural this situation is. The question is: can something be done to reverse the trend?
The Microsoft 365 Migration Tutorial For Beginner.pptx
This presentation will help you understand the power of Microsoft 365. However, we have mentioned every productivity app included in Office 365. Additionally, we have suggested the migration situation related to Office 365 and how we can help you.
You can also read: https://www.systoolsgroup.com/updates/office-365-tenant-to-tenant-migration-step-by-step-complete-guide/
Dandelion Hashtable: beyond billion requests per second on a commodity server
This slide deck presents DLHT, a concurrent in-memory hashtable. Despite efforts to optimize hashtables, that go as far as sacrificing core functionality, state-of-the-art designs still incur multiple memory accesses per request and block request processing in three cases. First, most hashtables block while waiting for data to be retrieved from memory. Second, open-addressing designs, which represent the current state-of-the-art, either cannot free index slots on deletes or must block all requests to do so. Third, index resizes block every request until all objects are copied to the new index. Defying folklore wisdom, DLHT forgoes open-addressing and adopts a fully-featured and memory-aware closed-addressing design based on bounded cache-line-chaining. This design offers lock-free index operations and deletes that free slots instantly, (2) completes most requests with a single memory access, (3) utilizes software prefetching to hide memory latencies, and (4) employs a novel non-blocking and parallel resizing. In a commodity server and a memory-resident workload, DLHT surpasses 1.6B requests per second and provides 3.5x (12x) the throughput of the state-of-the-art closed-addressing (open-addressing) resizable hashtable on Gets (Deletes).
Session 1 - Intro to Robotic Process Automation.pdf
👉 Check out our full 'Africa Series - Automation Student Developers (EN)' page to register for the full program:
https://bit.ly/Automation_Student_Kickstart
In this session, we shall introduce you to the world of automation, the UiPath Platform, and guide you on how to install and setup UiPath Studio on your Windows PC.
📕 Detailed agenda:
What is RPA? Benefits of RPA?
RPA Applications
The UiPath End-to-End Automation Platform
UiPath Studio CE Installation and Setup
💻 Extra training through UiPath Academy:
Introduction to Automation
UiPath Business Automation Platform
Explore automation development with UiPath Studio
👉 Register here for our upcoming Session 2 on June 20: Introduction to UiPath Studio Fundamentals: https://community.uipath.com/events/details/uipath-lagos-presents-session-2-introduction-to-uipath-studio-fundamentals/
What is an RPA CoE? Session 1 – CoE Vision
In the first session, we will review the organization's vision and how this has an impact on the COE Structure.
Topics covered:
• The role of a steering committee
• How do the organization’s priorities determine CoE Structure?
Speaker:
Chris Bolin, Senior Intelligent Automation Architect Anika Systems
AppSec PNW: Android and iOS Application Security with MobSF
Mobile Security Framework - MobSF is a free and open source automated mobile application security testing environment designed to help security engineers, researchers, developers, and penetration testers to identify security vulnerabilities, malicious behaviours and privacy concerns in mobile applications using static and dynamic analysis. It supports all the popular mobile application binaries and source code formats built for Android and iOS devices. In addition to automated security assessment, it also offers an interactive testing environment to build and execute scenario based test/fuzz cases against the application.
This talk covers:
Using MobSF for static analysis of mobile applications.
Interactive dynamic security assessment of Android and iOS applications.
Solving Mobile app CTF challenges.
Reverse engineering and runtime analysis of Mobile malware.
How to shift left and integrate MobSF/mobsfscan SAST and DAST in your build pipeline.
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
This LF Energy webinar took place June 20, 2024. It featured:
-Alex Thornton, LF Energy
-Hallie Cramer, Google
-Daniel Roesler, UtilityAPI
-Henry Richardson, WattTime
In response to the urgency and scale required to effectively address climate change, open source solutions offer significant potential for driving innovation and progress. Currently, there is a growing demand for standardization and interoperability in energy data and modeling. Open source standards and specifications within the energy sector can also alleviate challenges associated with data fragmentation, transparency, and accessibility. At the same time, it is crucial to consider privacy and security concerns throughout the development of open source platforms.
This webinar will delve into the motivations behind establishing LF Energy’s Carbon Data Specification Consortium. It will provide an overview of the draft specifications and the ongoing progress made by the respective working groups.
Three primary specifications will be discussed:
-Discovery and client registration, emphasizing transparent processes and secure and private access
-Customer data, centering around customer tariffs, bills, energy usage, and full consumption disclosure
-Power systems data, focusing on grid data, inclusive of transmission and distribution networks, generation, intergrid power flows, and market settlement data
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
Tomaz Bratanic
Graph ML and GenAI Expert - Neo4j
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
5th LF Energy Power Grid Model Meet-up Slides
5th Power Grid Model Meet-up
It is with great pleasure that we extend to you an invitation to the 5th Power Grid Model Meet-up, scheduled for 6th June 2024. This event will adopt a hybrid format, allowing participants to join us either through an online Mircosoft Teams session or in person at TU/e located at Den Dolech 2, Eindhoven, Netherlands. The meet-up will be hosted by Eindhoven University of Technology (TU/e), a research university specializing in engineering science & technology.
Power Grid Model
The global energy transition is placing new and unprecedented demands on Distribution System Operators (DSOs). Alongside upgrades to grid capacity, processes such as digitization, capacity optimization, and congestion management are becoming vital for delivering reliable services.
Power Grid Model is an open source project from Linux Foundation Energy and provides a calculation engine that is increasingly essential for DSOs. It offers a standards-based foundation enabling real-time power systems analysis, simulations of electrical power grids, and sophisticated what-if analysis. In addition, it enables in-depth studies and analysis of the electrical power grid’s behavior and performance. This comprehensive model incorporates essential factors such as power generation capacity, electrical losses, voltage levels, power flows, and system stability.
Power Grid Model is currently being applied in a wide variety of use cases, including grid planning, expansion, reliability, and congestion studies. It can also help in analyzing the impact of renewable energy integration, assessing the effects of disturbances or faults, and developing strategies for grid control and optimization.
What to expect
For the upcoming meetup we are organizing, we have an exciting lineup of activities planned:
-Insightful presentations covering two practical applications of the Power Grid Model.
-An update on the latest advancements in Power Grid -Model technology during the first and second quarters of 2024.
-An interactive brainstorming session to discuss and propose new feature requests.
-An opportunity to connect with fellow Power Grid Model enthusiasts and users.
Recently uploaded (20)
Leveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and Standards
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
Session 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdf
AppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSF
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...
9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Event Pipe - Lambda Architecture
- 1. EventPipe-λArchitecture w/NotsoBigData Bahadir Cambel - Senior Data Engineer @bahadircambel bahadir.io TravelBird
- 2. There are 10^11 stars in the galaxy. That used to be a huge number. But it's only a hundred billion. It's less than the national deficit! We used to call them astronomical numbers. Now we should call them economical numbers. -RF We only have half a billion events, yet.
- 3. What’sanEvent? a thing that happens (takes place), especially one of importance - Page view - Link Click - Order Save - Sales Flow - Mobile App Launch.. - A/B Visit - Newsletter View - Scroll - Ad View - and many more...
- 4. What’saPipe?theBADBADONE - Events flowing in through. Once upon a time in LinkedIn
- 6. logslogslogs->Kafka-Kinesis- Jay Kreps ( Ex-LinkedIn, co-creator Apache Kafka, Confluent.io founder) http://bit.ly/realtime-data-logs ( MUST READ - LONG READ - READ TWICE I’m aiming to read 3rd time) http://bit.ly/i-love-logs-video
- 7. What’s λ Nathan Marz(Storm,Cascalog) come up with the term.
- 11. Howtocount? - Counter +1 - Aggregate in batch - What happens if you need to reprocess data ? - Page Views, Users, Sessions - How about Counter’s for Segments - per page/device/domain/ - How to handle Big DATA? Is it going to perform with - Million
- 12. Hyperloglog-HLL - DVc ( almost accurate ) - Redis has it since 2.8.9 ( PFCOUNT - PFADD ) - 12 KBytes per key up to 2^64 elements per key ( 0.81% standard-error)
- 13. CLOJURE - Lazy - Concise - Fast - Simple - Perfect for Data processing, distributed computing - the power of JVM
- 14. stats
- 15. OPensource - https://github.com/travelbird/star-track - https://github.com/travelbird/kinesis3 - https://github.com/travelbird/modular
- 16. A/BTests-ExperimentsExperimentsExperiments It doesn't matter how beautiful your theory is, it doesn't matter how smart you are. If it doesn't agree with experiment, it's wrong. - RF
- 17. We’reallfoolsofourown The first principle is that you must not fool yourself and you are the easiest person to fool. - RF
- 18. ORDERinCHaos
- 19. Thanks - Andrii Mishkovski (Intro to Clojure) - Philipp Wassibauer (CTO of TravelBird) - Egle, Brandon, Niels, Brandon, Rob (Teammates) - Rich Hickey for Clojure - Nathan Marz for λArchitecture - Opensourcedevelopers-You’reamazing