Geo-Trending Example
•
0 likes•129 views
This document summarizes a data engineering project for analyzing trending topics by geo-location in 3 sentences or less: The project involves building a pipeline to ingest real-time social media data from Kafka into HDFS for batch processing with Spark and storing results in Cassandra, with the goal of exposing trending hashtag data via a web API. Some initial components including a simple Flask API are complete, while work remains on real-time streaming, a NoSQL database interface, and fully configuring the cluster. The presenter has a computer science degree and experience as a software engineer at Citrix and a university research center.
Report
Share
Report
Share
Download to read offline
Recommended
Cloud in your Cloud
Cloud in your Cloud. How we use Microservices to build Digital Ocean's cloud platform. (GOphercon India conference 2016)
Making Wallstreet talk with GO (GO India Conference 2015)
A talk about how we built one of the worlds largest Instant Messenger servers for stock traders. This talk was given at GO India Conference in Bangalore in 2015
The journey of Moving from AWS ELK to GCP Data Pipeline
This is a real case from VMfive to shifting ELK architecture from AWS. Currently GCP Data Pipeline provide us more efficiency and stable environment for running our service.
DigitalOcean Microservices Talk Rocket Internet Conf '16
This document discusses building cloud applications using a microservices architecture. It describes how microservices can help with organizational scaling and that most large companies started as monolithic applications. It then provides examples of different types of applications and services that may be needed for a cloud platform. The document outlines best practices for developing, deploying, monitoring, and maintaining microservices including using Docker, feature flags, service discovery, distributed tracing, and logging. It concludes by inviting questions and noting the company is hiring.
Spotify's journey to GCP
Spotify is moving their entire backend to GCP. You will get to hear their challenges, problems and also success stories and what to think about when choosing and moving services into the cloud.
Stream Processing Live Traffic Data with Kafka Streams
In this workshop we will set up a streaming framework which will process realtime data of traffic sensors installed within the Belgian road system.
Starting with the intake of the data, you will learn best practices and the recommended approach to split the information into events in a way that won’t come back to haunt you.
With some basic stream operations (count, filter, … ) you will get to know the data and experience how easy it is to get things done with Spring Boot & Spring Cloud Stream. But since simple data processing is not enough to fulfill all your streaming needs, we will also let you experience the power of windows.
After this workshop, tumbling, sliding and session windows hold no more mysteries and you will be a true streaming wizard.
RedisConf17- How Redis Saved Us a Boatload of Money and Boosted Efficiency
Redis helped a delivery startup save money and boost efficiency by solving their "pick-up location" problem. They used Redis for messaging, updating content quickly without full deploys, and caching user preferences and profiles. Implementing Redis's geospatial functionality allowed them to suggest optimal pick-up locations to customers based on their locations. This led to measurable results, with the company now exploring additional uses of Redis for marketing and dynamic pricing tools.
Building real-time data analytics on Google Cloud
Presented at the 25th Data Science Leuven meetup on 2020/03/11
Jonny Daenen explains the steps they took at Selligent to create a multi-tenant real-time data pipeline. He discusses all challenges the team encountered as well as the tools they used. The benefits of using Google Cloud Platform to remove operational hurdles when moving data pipelines to production are demonstrated.
Recommended
Cloud in your Cloud
Cloud in your Cloud. How we use Microservices to build Digital Ocean's cloud platform. (GOphercon India conference 2016)
Making Wallstreet talk with GO (GO India Conference 2015)
A talk about how we built one of the worlds largest Instant Messenger servers for stock traders. This talk was given at GO India Conference in Bangalore in 2015
The journey of Moving from AWS ELK to GCP Data Pipeline
This is a real case from VMfive to shifting ELK architecture from AWS. Currently GCP Data Pipeline provide us more efficiency and stable environment for running our service.
DigitalOcean Microservices Talk Rocket Internet Conf '16
This document discusses building cloud applications using a microservices architecture. It describes how microservices can help with organizational scaling and that most large companies started as monolithic applications. It then provides examples of different types of applications and services that may be needed for a cloud platform. The document outlines best practices for developing, deploying, monitoring, and maintaining microservices including using Docker, feature flags, service discovery, distributed tracing, and logging. It concludes by inviting questions and noting the company is hiring.
Spotify's journey to GCP
Spotify is moving their entire backend to GCP. You will get to hear their challenges, problems and also success stories and what to think about when choosing and moving services into the cloud.
Stream Processing Live Traffic Data with Kafka Streams
In this workshop we will set up a streaming framework which will process realtime data of traffic sensors installed within the Belgian road system.
Starting with the intake of the data, you will learn best practices and the recommended approach to split the information into events in a way that won’t come back to haunt you.
With some basic stream operations (count, filter, … ) you will get to know the data and experience how easy it is to get things done with Spring Boot & Spring Cloud Stream. But since simple data processing is not enough to fulfill all your streaming needs, we will also let you experience the power of windows.
After this workshop, tumbling, sliding and session windows hold no more mysteries and you will be a true streaming wizard.
RedisConf17- How Redis Saved Us a Boatload of Money and Boosted Efficiency
Redis helped a delivery startup save money and boost efficiency by solving their "pick-up location" problem. They used Redis for messaging, updating content quickly without full deploys, and caching user preferences and profiles. Implementing Redis's geospatial functionality allowed them to suggest optimal pick-up locations to customers based on their locations. This led to measurable results, with the company now exploring additional uses of Redis for marketing and dynamic pricing tools.
Building real-time data analytics on Google Cloud
Presented at the 25th Data Science Leuven meetup on 2020/03/11
Jonny Daenen explains the steps they took at Selligent to create a multi-tenant real-time data pipeline. He discusses all challenges the team encountered as well as the tools they used. The benefits of using Google Cloud Platform to remove operational hurdles when moving data pipelines to production are demonstrated.
Analyzing and processing FInancial Market Data on AWS with Kinesis - AWS Pop ...
Processing real-time market data is an essential part of financial services companies operating on the stock market. This data comes in “big and fast” - thousands of ticks per second - requiring transformation for fast and efficient access by our apps and services as well as storage for compliance and research purposes. In this session, you will learn about Scalable Capital’s evaluation journey. How we ended up with a solution around Amazon Kinesis, and how we are processing and analyzing market data now. A combination of AWS services allowed us to build a solution that is scalable, efficient and tailored to our needs. A Fargate service receives market data and puts it into Kinesis. Lambda functions transform and store data in ElastiCache. From there, the data can either be accessed in an aggregated form or streamed to other services for further processing. Amazon Kinesis Data Firehose is used to store all data on Amazon S3.
Presented at the AWS Pop Up Loft Munich on November 14, 2019
How Mapbox Scales over 9 AWS Regions
Mapbox runs its map tile services across 9 data centers globally to provide high availability and low latency for its customers worldwide. A map request first hits the nearest content delivery network, then the local load balancer which routes it to an application server in that region. The server authenticates the request and retrieves the tile data from a distributed database and object storage, checking a local cache first. This global infrastructure allows Mapbox to meet its service level agreement of 99.9% uptime while minimizing latency for users around the world.
Hadoop summit - Scaling Uber’s Real-Time Infra for Trillion Events per Day
Building data pipelines is pretty hard! Building a multi-datacenter active-active real time data pipeline for multiple classes of data with different durability, latency and availability guarantees is much harder.
Real time infrastructure powers critical pieces of Uber (think Surge) and in this talk we will discuss our architecture, technical challenges, learnings and how a blend of open source infrastructure (Apache Kafka and Samza) and in-house technologies have helped Uber scale.
Balkan - data eng meetup - data fusion
Google Cloud Data Fusion is a managed service that allows users to visually build and run data pipelines without coding. It converts the graphical pipeline design into a directed acyclic graph (DAG) to run batch or streaming jobs on an ephemeral Dataproc cluster. The presenter demoed building a sample pipeline using the drag and drop interface and explored the tool's monitoring and logging capabilities. While the tool aims to save labor hours, its beta limitations include slow instance creation, lack of input validation, and Java stacktraces in errors. Pricing is based on hourly development costs and Dataproc execution costs.
Understanding and Improving Code Generation
Code generation is integral to Spark’s physical execution engine. When implemented, the Spark engine creates optimized bytecode at runtime improving performance when compared to interpreted execution. Spark has taken the next step with whole-stage codegen which collapses an entire query into a single function.
Михаил Максимов ( Software engineer, DataArt. AWS certified Solution Architect)
Serverless architectures allow developers to run code without provisioning or managing servers. With serverless, code runs in ephemeral containers that are managed by third-party cloud providers. Key benefits include automatic scaling, high availability, and paying only for the resources consumed. Some limitations are vendor lock-in, memory and computing limits per function, and potential cold start delays. Serverless functions can be triggered synchronously, asynchronously via events, or via streaming data sources. Common event sources that trigger serverless functions include S3, API Gateway, IoT, and SQS. Serverless is well suited for building event-driven architectures and processing streaming data at scale.
uReplicator: Uber Engineering’s Scalable, Robust Kafka Replicator
Presented at Stream Processing Meetup (7/19/2018)(https://www.meetup.com/Stream-Processing-Meetup-LinkedIn/events/251481797/).
At Uber, we operate 20+ Kafka clusters to collect system and application logs as well as event data from rider and driver apps. We need a Kafka replication solution to replicate data between Kafka clusters across multiple data centers for different purposes. This talk will introduce the history behind uReplicator and the high level architecture. As the original uReplicator ran into scalability challenges and operational overhead as the scale of Kafka clusters increased, we built the Federated uReplicator which addressed above issues and provide an extensible architecture for further scaling.
Serverless Architecture GCP In Production
The document defines serverless architecture and discusses various Google Cloud Platform products that can be used to build serverless applications. It examines compute options like Cloud Functions, App Engine, and CloudRun. It also looks at services for persistence, processing, triggering, messaging, machine learning, containers, CI/CD pipelines, and more. The document provides considerations for serverless projects, architecture patterns, and use case examples.
Netflix's Could Migration
Netflix migrated its infrastructure from on-premise data centers to the cloud starting in 2008 due to issues with its monolithic architecture. It moved services gradually to AWS using microservices. This provided agility through continuous delivery, elasticity to dynamically scale, and resiliency through redundancy. Netflix also moved corporate systems like email to SaaS. The cloud transformation required changes to its operations model and culture to become more decentralized and cloud-native. While Netflix's size enabled this transformation, the document argues that any organization can make this shift to gain competitive advantages.
2016/09/15 - Everest
This document describes an incremental backup solution with cloud storage. It provides an architecture that allows for backup and restore plans from file systems to Amazon S3 and Glacier storage. The solution includes scheduled backups, synchronization, email reports, and purging options. The roadmap details current and planned features such as restore functionality, scheduling, versioning, performance improvements, notifications, custom actions, and open sourcing.
Stream Processing with Kafka and Samza
This document discusses stream processing using Kafka and Samza. It notes that Samza provides low-latency, high-throughput stream processing and tightly integrates with Kafka. Samza handles stream processing, local state storage, and has a simple API. It also notes that at LinkedIn, Kafka is used extensively to handle hundreds of brokers and terabytes of messaging data daily.
Intro to the Google Cloud for Developers
This document provides an introduction to developing applications on Google Cloud. It discusses Google's cloud infrastructure and services like Compute Engine and App Engine. It demonstrates how to use the Google Cloud SDK and APIs to manage resources and build applications using various languages and tools. Specifically, it shows how to create instances in Compute Engine and Big Query, deploy applications to App Engine from Eclipse, and use command line tools to manage storage, databases and other services.
19th February 2013, AWS User Group UK, Meetup #3, The CentraStage Experience,...
Agenda entry: The CentraStage experience, Ian van Reenen, CTO CentraStage: With 200,000 distributed agents with persistent SSL connections, keeping them all alive in the cloud can be a challenge. In this talk Ian will share the learnings of CentraStage regarding ELB vs HAProxy, and give insight into their strategies to cater for connection avalanches after outages.
Organiser's comment: Our first double-act as Ian from CentraStage brought up his DevOps lead, Jason, for a strong discussion-led session about how CentraStage have worked with ELB and HAProxy. This was a great talk with some really engaging Q&A, thanks guys for bringing us an interesting topic.
Lieven Vandegaer from MEDIAGENIX - Orchestrating a video-on-demand pipeline w...
Lieven Vandegaer from MEDIAGENIX - Orchestrating a video-on-demand pipeline w...camunda services GmbH
WHATS'ON is a software solution of MEDIAGENIX that manages the lifecycle of content within media companies.
For a new project with a telecom operator we needed to handle incoming metadata, video files and images and make this content available on several VOD platforms. These workflows can become very complex and require intense communication with various systems (transcoders, encryption software, a VOD backend system...).
In this presentation, we will show how we are using our in-house product WHATS'ON (written in Smalltalk) in combination with Camunda BPM and Apache NiFi, using external tasks and the Camunda BPM REST API.XStream: stream processing platform at facebook
XStream is Facebook's unified stream processing platform that provides a fully managed stream processing service. It was built using the Stylus C++ stream processing framework and uses a common SQL dialect called CoreSQL. XStream employs an interpretive execution model using the new Velox vectorized SQL evaluation engine for high performance. This provides a consistent and high efficiency stream processing platform to support diverse real-time use cases at planetary scale for Facebook.
Operating Kafka on AutoPilot mode @ DBS Bank (Arpit Dubey, DBS Bank) Kafka Su...
Kafka is among the few cool technologies that play a pivotal role in the digital journey of DBS to become one of the best bank in the world.
Kafka is the foundation for our vision of application modernisation by:
- making our systems event-driven
- moving batch workloads to real-time
- serving various financial industry use-cases that warrants proactive actions i.e Fraud detection, Anomaly and unusual behaviour detections.
To realize our Digital vision and to keep up with the pace of this transformation, we decided to run Kafka ""as-a-self-managed-service"" at large scale. This requires minimising reliance on human operators/ engineers and have intelligent apps make all important operation related decisions.
In this session, I would like to deep-dive into these intelligent apps that not only helped us maintain 100% availability but also provides self-service experience and a truly democratised Kafka ecosystem for our community.
Some of these apps include :
1. Self-service on-boarding and stream discovery built by esp. leveraging Kafka admin API
2. AI Ops - using machine learning for predicting resource utilisation and early anomaly detection
3. Automated Cluster scaling, deployment and maintenance
4. Self-healing services
5. Intelligent Chaos engineering - to make apps ready for production
6. Black box Kafka monitoring
7. Load simulator for performance testing - to bring production like experience in development environments
8. Automatic tiering and streams archival - to meet several regulatory compliance and to make brokers lightweight
The Benefits of Publicly-Accessible Data - SNODAS
The document discusses how publicly accessible web services can make government data more usable. It describes how the author automated the retrieval, conversion and distribution of snowpack data from NOAA's SNODAS program as web services. This allows developers to easily add the latest SNODAS data to webmaps and applications. The author demonstrates this through an open source Leaflet map called "Slippy-SNODAS" that consumes the SNODAS web services. Making this data more accessible through modern web standards empowers others to use and improve upon the data.
Serverless with Google Cloud Functions
This document discusses Google Cloud Functions, a serverless platform for running code in response to events. It provides an overview of Google Cloud Functions' features such as triggers from Cloud Pub/Sub and Storage, integration with other Google Cloud services, and use cases including building mobile backends, APIs, data processing, and IoT. The document also discusses using Google Cloud Functions with Firebase and pricing.
Feedback on AWS re:invent 2016
The Reinvent 2016 conference hosted by Amazon Web Services included keynotes, over 400 sessions across 4 locations over 5 days. New services and updates were announced across compute, analytics, database, developer tools, artificial intelligence, monitoring, migration, mobile, containers, and lambda. Significant announcements included new instance types, elastic GPUs, IPv6 support for EC2, Athena for querying S3 data with SQL, Glue for data integration and transformations, and expanded capabilities for many existing services like Lambda, CloudFront, and Snowmobile for large data transfers.
The Many Faces of Apache Kafka: Leveraging Real-time Data at Scale
The document discusses Confluent, an open-source platform for building real-time streaming data applications using Apache Kafka. It outlines Confluent's mission to make Kafka-based streaming platforms practical everywhere. Confluent's product includes tools for stream processing integration with Kafka, connectors to move data between systems, monitoring of end-to-end data flows, and schema and metadata management. An upcoming release, Confluent Platform 2.0, is planned for December.
Trending Topics
This document discusses trending topics by geo-location using data from Twitter. It describes the data flow and pipeline, including streaming tweets from the Twitter API to Kafka and processing them with Spark on HDFS for hourly and daily trends. The cluster setup is outlined showing the various components. Challenges around scaling to millions of tweets per day are discussed, requiring upgrades to memory and server sizes. Storm topologies are used to consume from Kafka, write tweets to storage, and aggregate minute-based trends for a live page. Time discrepancies between servers are also noted.
Real-Time Web; Trending Social Data
These slides were created for the course:
Comm 350R Social Media
Dr. Matthew J. Kushin
Department of Communication
Utah Valley University
For more on the course see:
http://profkushinsocial.wordpress.com
For more about the professor, see:
http://profkushin.wordpress.com
or @mjkushin on Twitter
More Related Content
What's hot
Analyzing and processing FInancial Market Data on AWS with Kinesis - AWS Pop ...
Processing real-time market data is an essential part of financial services companies operating on the stock market. This data comes in “big and fast” - thousands of ticks per second - requiring transformation for fast and efficient access by our apps and services as well as storage for compliance and research purposes. In this session, you will learn about Scalable Capital’s evaluation journey. How we ended up with a solution around Amazon Kinesis, and how we are processing and analyzing market data now. A combination of AWS services allowed us to build a solution that is scalable, efficient and tailored to our needs. A Fargate service receives market data and puts it into Kinesis. Lambda functions transform and store data in ElastiCache. From there, the data can either be accessed in an aggregated form or streamed to other services for further processing. Amazon Kinesis Data Firehose is used to store all data on Amazon S3.
Presented at the AWS Pop Up Loft Munich on November 14, 2019
How Mapbox Scales over 9 AWS Regions
Mapbox runs its map tile services across 9 data centers globally to provide high availability and low latency for its customers worldwide. A map request first hits the nearest content delivery network, then the local load balancer which routes it to an application server in that region. The server authenticates the request and retrieves the tile data from a distributed database and object storage, checking a local cache first. This global infrastructure allows Mapbox to meet its service level agreement of 99.9% uptime while minimizing latency for users around the world.
Hadoop summit - Scaling Uber’s Real-Time Infra for Trillion Events per Day
Building data pipelines is pretty hard! Building a multi-datacenter active-active real time data pipeline for multiple classes of data with different durability, latency and availability guarantees is much harder.
Real time infrastructure powers critical pieces of Uber (think Surge) and in this talk we will discuss our architecture, technical challenges, learnings and how a blend of open source infrastructure (Apache Kafka and Samza) and in-house technologies have helped Uber scale.
Balkan - data eng meetup - data fusion
Google Cloud Data Fusion is a managed service that allows users to visually build and run data pipelines without coding. It converts the graphical pipeline design into a directed acyclic graph (DAG) to run batch or streaming jobs on an ephemeral Dataproc cluster. The presenter demoed building a sample pipeline using the drag and drop interface and explored the tool's monitoring and logging capabilities. While the tool aims to save labor hours, its beta limitations include slow instance creation, lack of input validation, and Java stacktraces in errors. Pricing is based on hourly development costs and Dataproc execution costs.
Understanding and Improving Code Generation
Code generation is integral to Spark’s physical execution engine. When implemented, the Spark engine creates optimized bytecode at runtime improving performance when compared to interpreted execution. Spark has taken the next step with whole-stage codegen which collapses an entire query into a single function.
Михаил Максимов ( Software engineer, DataArt. AWS certified Solution Architect)
Serverless architectures allow developers to run code without provisioning or managing servers. With serverless, code runs in ephemeral containers that are managed by third-party cloud providers. Key benefits include automatic scaling, high availability, and paying only for the resources consumed. Some limitations are vendor lock-in, memory and computing limits per function, and potential cold start delays. Serverless functions can be triggered synchronously, asynchronously via events, or via streaming data sources. Common event sources that trigger serverless functions include S3, API Gateway, IoT, and SQS. Serverless is well suited for building event-driven architectures and processing streaming data at scale.
uReplicator: Uber Engineering’s Scalable, Robust Kafka Replicator
Presented at Stream Processing Meetup (7/19/2018)(https://www.meetup.com/Stream-Processing-Meetup-LinkedIn/events/251481797/).
At Uber, we operate 20+ Kafka clusters to collect system and application logs as well as event data from rider and driver apps. We need a Kafka replication solution to replicate data between Kafka clusters across multiple data centers for different purposes. This talk will introduce the history behind uReplicator and the high level architecture. As the original uReplicator ran into scalability challenges and operational overhead as the scale of Kafka clusters increased, we built the Federated uReplicator which addressed above issues and provide an extensible architecture for further scaling.
Serverless Architecture GCP In Production
The document defines serverless architecture and discusses various Google Cloud Platform products that can be used to build serverless applications. It examines compute options like Cloud Functions, App Engine, and CloudRun. It also looks at services for persistence, processing, triggering, messaging, machine learning, containers, CI/CD pipelines, and more. The document provides considerations for serverless projects, architecture patterns, and use case examples.
Netflix's Could Migration
Netflix migrated its infrastructure from on-premise data centers to the cloud starting in 2008 due to issues with its monolithic architecture. It moved services gradually to AWS using microservices. This provided agility through continuous delivery, elasticity to dynamically scale, and resiliency through redundancy. Netflix also moved corporate systems like email to SaaS. The cloud transformation required changes to its operations model and culture to become more decentralized and cloud-native. While Netflix's size enabled this transformation, the document argues that any organization can make this shift to gain competitive advantages.
2016/09/15 - Everest
This document describes an incremental backup solution with cloud storage. It provides an architecture that allows for backup and restore plans from file systems to Amazon S3 and Glacier storage. The solution includes scheduled backups, synchronization, email reports, and purging options. The roadmap details current and planned features such as restore functionality, scheduling, versioning, performance improvements, notifications, custom actions, and open sourcing.
Stream Processing with Kafka and Samza
This document discusses stream processing using Kafka and Samza. It notes that Samza provides low-latency, high-throughput stream processing and tightly integrates with Kafka. Samza handles stream processing, local state storage, and has a simple API. It also notes that at LinkedIn, Kafka is used extensively to handle hundreds of brokers and terabytes of messaging data daily.
Intro to the Google Cloud for Developers
This document provides an introduction to developing applications on Google Cloud. It discusses Google's cloud infrastructure and services like Compute Engine and App Engine. It demonstrates how to use the Google Cloud SDK and APIs to manage resources and build applications using various languages and tools. Specifically, it shows how to create instances in Compute Engine and Big Query, deploy applications to App Engine from Eclipse, and use command line tools to manage storage, databases and other services.
19th February 2013, AWS User Group UK, Meetup #3, The CentraStage Experience,...
Agenda entry: The CentraStage experience, Ian van Reenen, CTO CentraStage: With 200,000 distributed agents with persistent SSL connections, keeping them all alive in the cloud can be a challenge. In this talk Ian will share the learnings of CentraStage regarding ELB vs HAProxy, and give insight into their strategies to cater for connection avalanches after outages.
Organiser's comment: Our first double-act as Ian from CentraStage brought up his DevOps lead, Jason, for a strong discussion-led session about how CentraStage have worked with ELB and HAProxy. This was a great talk with some really engaging Q&A, thanks guys for bringing us an interesting topic.
Lieven Vandegaer from MEDIAGENIX - Orchestrating a video-on-demand pipeline w...
Lieven Vandegaer from MEDIAGENIX - Orchestrating a video-on-demand pipeline w...camunda services GmbH
WHATS'ON is a software solution of MEDIAGENIX that manages the lifecycle of content within media companies.
For a new project with a telecom operator we needed to handle incoming metadata, video files and images and make this content available on several VOD platforms. These workflows can become very complex and require intense communication with various systems (transcoders, encryption software, a VOD backend system...).
In this presentation, we will show how we are using our in-house product WHATS'ON (written in Smalltalk) in combination with Camunda BPM and Apache NiFi, using external tasks and the Camunda BPM REST API.XStream: stream processing platform at facebook
XStream is Facebook's unified stream processing platform that provides a fully managed stream processing service. It was built using the Stylus C++ stream processing framework and uses a common SQL dialect called CoreSQL. XStream employs an interpretive execution model using the new Velox vectorized SQL evaluation engine for high performance. This provides a consistent and high efficiency stream processing platform to support diverse real-time use cases at planetary scale for Facebook.
Operating Kafka on AutoPilot mode @ DBS Bank (Arpit Dubey, DBS Bank) Kafka Su...
Kafka is among the few cool technologies that play a pivotal role in the digital journey of DBS to become one of the best bank in the world.
Kafka is the foundation for our vision of application modernisation by:
- making our systems event-driven
- moving batch workloads to real-time
- serving various financial industry use-cases that warrants proactive actions i.e Fraud detection, Anomaly and unusual behaviour detections.
To realize our Digital vision and to keep up with the pace of this transformation, we decided to run Kafka ""as-a-self-managed-service"" at large scale. This requires minimising reliance on human operators/ engineers and have intelligent apps make all important operation related decisions.
In this session, I would like to deep-dive into these intelligent apps that not only helped us maintain 100% availability but also provides self-service experience and a truly democratised Kafka ecosystem for our community.
Some of these apps include :
1. Self-service on-boarding and stream discovery built by esp. leveraging Kafka admin API
2. AI Ops - using machine learning for predicting resource utilisation and early anomaly detection
3. Automated Cluster scaling, deployment and maintenance
4. Self-healing services
5. Intelligent Chaos engineering - to make apps ready for production
6. Black box Kafka monitoring
7. Load simulator for performance testing - to bring production like experience in development environments
8. Automatic tiering and streams archival - to meet several regulatory compliance and to make brokers lightweight
The Benefits of Publicly-Accessible Data - SNODAS
The document discusses how publicly accessible web services can make government data more usable. It describes how the author automated the retrieval, conversion and distribution of snowpack data from NOAA's SNODAS program as web services. This allows developers to easily add the latest SNODAS data to webmaps and applications. The author demonstrates this through an open source Leaflet map called "Slippy-SNODAS" that consumes the SNODAS web services. Making this data more accessible through modern web standards empowers others to use and improve upon the data.
Serverless with Google Cloud Functions
This document discusses Google Cloud Functions, a serverless platform for running code in response to events. It provides an overview of Google Cloud Functions' features such as triggers from Cloud Pub/Sub and Storage, integration with other Google Cloud services, and use cases including building mobile backends, APIs, data processing, and IoT. The document also discusses using Google Cloud Functions with Firebase and pricing.
Feedback on AWS re:invent 2016
The Reinvent 2016 conference hosted by Amazon Web Services included keynotes, over 400 sessions across 4 locations over 5 days. New services and updates were announced across compute, analytics, database, developer tools, artificial intelligence, monitoring, migration, mobile, containers, and lambda. Significant announcements included new instance types, elastic GPUs, IPv6 support for EC2, Athena for querying S3 data with SQL, Glue for data integration and transformations, and expanded capabilities for many existing services like Lambda, CloudFront, and Snowmobile for large data transfers.
The Many Faces of Apache Kafka: Leveraging Real-time Data at Scale
The document discusses Confluent, an open-source platform for building real-time streaming data applications using Apache Kafka. It outlines Confluent's mission to make Kafka-based streaming platforms practical everywhere. Confluent's product includes tools for stream processing integration with Kafka, connectors to move data between systems, monitoring of end-to-end data flows, and schema and metadata management. An upcoming release, Confluent Platform 2.0, is planned for December.
What's hot (20)
Analyzing and processing FInancial Market Data on AWS with Kinesis - AWS Pop ...
Analyzing and processing FInancial Market Data on AWS with Kinesis - AWS Pop ...
Hadoop summit - Scaling Uber’s Real-Time Infra for Trillion Events per Day
Hadoop summit - Scaling Uber’s Real-Time Infra for Trillion Events per Day
Михаил Максимов ( Software engineer, DataArt. AWS certified Solution Architect)
Михаил Максимов ( Software engineer, DataArt. AWS certified Solution Architect)
uReplicator: Uber Engineering’s Scalable, Robust Kafka Replicator
uReplicator: Uber Engineering’s Scalable, Robust Kafka Replicator
19th February 2013, AWS User Group UK, Meetup #3, The CentraStage Experience,...
19th February 2013, AWS User Group UK, Meetup #3, The CentraStage Experience,...
Lieven Vandegaer from MEDIAGENIX - Orchestrating a video-on-demand pipeline w...
Lieven Vandegaer from MEDIAGENIX - Orchestrating a video-on-demand pipeline w...
Operating Kafka on AutoPilot mode @ DBS Bank (Arpit Dubey, DBS Bank) Kafka Su...
Operating Kafka on AutoPilot mode @ DBS Bank (Arpit Dubey, DBS Bank) Kafka Su...
The Many Faces of Apache Kafka: Leveraging Real-time Data at Scale
The Many Faces of Apache Kafka: Leveraging Real-time Data at Scale
Viewers also liked
Trending Topics
This document discusses trending topics by geo-location using data from Twitter. It describes the data flow and pipeline, including streaming tweets from the Twitter API to Kafka and processing them with Spark on HDFS for hourly and daily trends. The cluster setup is outlined showing the various components. Challenges around scaling to millions of tweets per day are discussed, requiring upgrades to memory and server sizes. Storm topologies are used to consume from Kafka, write tweets to storage, and aggregate minute-based trends for a live page. Time discrepancies between servers are also noted.
Real-Time Web; Trending Social Data
These slides were created for the course:
Comm 350R Social Media
Dr. Matthew J. Kushin
Department of Communication
Utah Valley University
For more on the course see:
http://profkushinsocial.wordpress.com
For more about the professor, see:
http://profkushin.wordpress.com
or @mjkushin on Twitter
Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
The document appears to be notes from a MongoDB training session that discusses various MongoDB features like MapReduce, geospatial indexes, and GridFS. It also covers topics like database commands, indexing, and querying documents with embedded documents and arrays. Examples are provided for how to implement many of these MongoDB features and functions.
What’s Trending in Social Media
What’s Trending in Social Media - Facebook, Twitter, Youtube, Engagement, zettabytes
Google, Google+. Authority is the the power to influence others or one's recognized knowledge about something. Expert is a person or organization who has a comprehensive and authoritative knowledge of or skill in a particular area.
Python Development (MongoSF)
While some parts of Django like its URL routing, templates, and caching are not dependent on Django's ORM, integrating MongoDB would require replacing Django's default SQLite database and models with MongoDB-specific database and ODM libraries to support MongoDB's document-oriented data structure and queries. Several third-party libraries provide MongoDB support by replacing Django's ORM with a MongoDB ODM to define schemas and queries.
Introduction to Cassandra (June 2010)
This document provides an overview of Apache Cassandra, including its origins from Amazon Dynamo and Google BigTable, its data model using a ring topology and column families, and how it provides horizontal scalability and eventual consistency through replication. It discusses Cassandra's write path using commit logs and memtables as well as its read path involving caching. It also covers client access, practical considerations, and Cassandra's future direction.
Big data 5Vs 2014 - View from World to Vietnam by Dinh Le Dat
Big Data Analytics - Volume,Velocity,Variety,Veracity, #Value! view from expert
Big Data Analytics - Volume,Velocity,Variety,Veracity, #Value! view from expert
http://dinhledat.com/data-driven-marketing/big-data-goc-nhin-nguoi-trong-cuoc-itlc-offline-7/
#Big Data: Góc nhìn người trong cuộc -- ITLC Offline 7
AdTechVietnam -- Với trách nhiệm vận hành cộng đồng CNTT Việt Nam, ITLC phải làm cách này cách kia để câu hỏi trên có lời giải. Vì vậy sau sự kiện "Ngày công nghệ FPT", ITLC đã nhanh tay "đặt hàng" anh Đinh Lê Đạt, một chuyên gia về Data-Driven của ngành giúp trả lời cụ thể hơn, thực tế hơn về tồn tại của "huyền thoại" dữ liệu lớn!
http://dinhledat.com
MongoDB for Time Series Data Part 1: Setting the Stage for Sensor Management
The document discusses schema design considerations for time series traffic monitoring data stored in MongoDB. It evaluates alternatives like storing a document per event, per minute, or per hour. Storing aggregated data improves write performance by changing inserts to updates, improves analytic queries by reducing document reads, and reduces memory requirements by decreasing index size. The best schema depends on the read and write workload of the specific application.
MongoDB for Time Series Data Part 2: Analyzing Time Series Data Using the Agg...
The United States will be deploying 16,000 traffic speed monitoring sensors - 1 on every mile of US interstate in urban centers. These sensors update the speed, weather, and pavement conditions once per minute. MongoDB will collect and aggregate live sensor data feeds from roadways around the country, support real-time queries from cars on traffic conditions on their route as well as be the platform for real-time dashboards displaying traffic conditions and more complex analytical queries used to identify traffic trends. In this session, we’ll implement a few different data aggregation techniques to query and dashboard the metrics gathered from the US interstate.
Facebook Ranking Factors - Toàn bộ mọi bí mật về Facebook
Facebook Ranking Factors - Toàn bộ mọi bí mật về Facebook do Vinalink nghiên cứu và công bố
Back to Basics Webinar 1: Introduction to NoSQL
This document provides an overview of an introduction to NoSQL webinar. It discusses why NoSQL databases were created, the different types of NoSQL databases including key-value stores, column stores, graph stores, multi-model databases and document stores. It provides details on MongoDB, describing how MongoDB stores data as JSON-like documents with dynamic schemas and supports features like indexing, aggregation and geospatial queries. The webinar agenda is also outlined.
Viewers also liked (11)
Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
Big data 5Vs 2014 - View from World to Vietnam by Dinh Le Dat
Big data 5Vs 2014 - View from World to Vietnam by Dinh Le Dat
MongoDB for Time Series Data Part 1: Setting the Stage for Sensor Management
MongoDB for Time Series Data Part 1: Setting the Stage for Sensor Management
MongoDB for Time Series Data Part 2: Analyzing Time Series Data Using the Agg...
MongoDB for Time Series Data Part 2: Analyzing Time Series Data Using the Agg...
Facebook Ranking Factors - Toàn bộ mọi bí mật về Facebook
Facebook Ranking Factors - Toàn bộ mọi bí mật về Facebook
Recently uploaded
➒➌➎➏➑➐➋➑➐➐ Dpboss Satta Matka Matka Guessing Kalyan Chart Indian Matka Satta ...
➒➌➎➏➑➐➋➑➐➐ Dpboss Satta Matka Matka Guessing Kalyan Chart Indian Matka Satta ...➒➌➎➏➑➐➋➑➐➐Dpboss Matka Guessing Satta Matka Kalyan Chart Indian Matka
➒➌➎➏➑➐➋➑➐➐ Dpboss Satta Matta Matka Kalyan Chart Indian Matka Dpboss Matka Kalyan panel Chart Matka Guessing Satta Matka ➒➌➎➏➑➐➋➑➐➐ Kalyan Matka Satta Matka Dpboss Matka Guessing Indian Matka
➒➌➎➏➑➐➋➑➐➐ Kalyan Matka Satta Matka Dpboss Matka Guessing Indian Matka➒➌➎➏➑➐➋➑➐➐Dpboss Matka Guessing Satta Matka Kalyan Chart Indian Matka
➒➌➎➏➑➐➋➑➐➐ Kalyan Matka Satta Matka Dpboss Matka Guessing Indian MatkaColour Theory for Painting - Fine Artist.pdf
This document is all about Colour Theory for Fine Artist / Painter.
原版制作(UNITO毕业证书)都灵大学毕业证Offer一模一样
学校原件一模一样【微信：741003700 】《(UNITO毕业证书)都灵大学毕业证》【微信：741003700 】学位证，留信认证（真实可查，永久存档）原件一模一样纸张工艺/offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。
本公司拥有海外各大学样板无数，能完美还原。
1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700
【主营项目】
一.毕业证【q微741003700】成绩单、使馆认证、教育部认证、雅思托福成绩单、学生卡等！
二.真实使馆公证(即留学回国人员证明,不成功不收费)
三.真实教育部学历学位认证（教育部存档！教育部留服网站永久可查）
四.办理各国各大学文凭(一对一专业服务,可全程监控跟踪进度)
如果您处于以下几种情况：
◇在校期间，因各种原因未能顺利毕业……拿不到官方毕业证【q/微741003700】
◇面对父母的压力，希望尽快拿到；
◇不清楚认证流程以及材料该如何准备；
◇回国时间很长，忘记办理；
◇回国马上就要找工作，办给用人单位看；
◇企事业单位必须要求办理的
◇需要报考公务员、购买免税车、落转户口
◇申请留学生创业基金
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
一比一原版美国亚利桑那大学毕业证（ua毕业证书）如何办理
一模一样【微信：A575476】【美国亚利桑那大学毕业证（ua毕业证书）成绩单Offer】【微信：A575476】（留信学历认证永久存档查询）采用学校原版纸张、特殊工艺完全按照原版一比一制作（包括：隐形水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠，文字图案浮雕，激光镭射，紫外荧光，温感，复印防伪）行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备，十五年致力于帮助留学生解决难题，业务范围有加拿大、英国、澳洲、韩国、美国、新加坡，新西兰等学历材料，包您满意。
【业务选择办理准则】
一、工作未确定，回国需先给父母、亲戚朋友看下文凭的情况，办理一份就读学校的毕业证【微信：A575476】文凭即可
二、回国进私企、外企、自己做生意的情况，这些单位是不查询毕业证真伪的，而且国内没有渠道去查询国外文凭的真假，也不需要提供真实教育部认证。鉴于此，办理一份毕业证【微信：A575476】即可
三、进国企，银行，事业单位，考公务员等等，这些单位是必需要提供真实教育部认证的，办理教育部认证所需资料众多且烦琐，所有材料您都必须提供原件，我们凭借丰富的经验，快捷的绿色通道帮您快速整合材料，让您少走弯路。
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
→ 【关于价格问题（保证一手价格）
我们所定的价格是非常合理的，而且我们现在做得单子大多数都是代理和回头客户介绍的所以一般现在有新的单子 我给客户的都是第一手的代理价格，因为我想坦诚对待大家 不想跟大家在价格方面浪费时间
对于老客户或者被老客户介绍过来的朋友，我们都会适当给一些优惠。
选择实体注册公司办理，更放心，更安全！我们的承诺：可来公司面谈，可签订合同，会陪同客户一起到教育部认证窗口递交认证材料，客户在教育部官方认证查询网站查询到认证通过结果后付款，不成功不收费！
❼❷⓿❺❻❷❽❷❼❽ Dpboss Matka ! Fix Satta Matka ! Matka Result ! Matka Guessing ! ...
❼❷⓿❺❻❷❽❷❼❽ Dpboss Matka ! Fix Satta Matka ! Matka Result ! Matka Guessing ! ...❼❷⓿❺❻❷❽❷❼❽ Dpboss Kalyan Satta Matka Guessing Matka Result Main Bazar chart
❼❷⓿❺❻❷❽❷❼❽ Dpboss Matka ! Fix Satta Matka ! Matka Result ! Matka Guessing ! Final Matka ! Matka Result ! Dpboss Matka ! Matka Guessing ! Satta Matta Matka 143 ! Kalyan Matka ! Satta Matka Fast Result ! Kalyan Matka Guessing ! Dpboss Matka Guessing ! Satta 143 ! Kalyan Chart ! Kalyan final ! Satta guessing ! Matka tips ! Matka 143 ! India Matka ! Matka 420 ! matka Mumbai ! Satta chart ! Indian Satta ! Satta King ! Satta 143 ! Satta batta ! Satta मटका ! Satta chart ! Matka 143 ! Matka Satta ! India Matka ! Indian Satta Matka ! Final ank哪里购买美国乔治城大学毕业证硕士学位证书原版一模一样
原版一模一样【微信：741003700 】【美国乔治城大学毕业证硕士学位证书】【微信：741003700 】学位证，留信认证（真实可查，永久存档）offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。
本公司拥有海外各大学样板无数，能完美还原海外各大学 Bachelor Diploma degree, Master Degree Diploma
1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
❼❷⓿❺❻❷❽❷❼❽ Dpboss Kalyan Satta Matka Guessing Matka Result Main Bazar chart
❼❷⓿❺❻❷❽❷❼❽ Dpboss Kalyan Satta Matka Guessing Matka Result Main Bazar chart❼❷⓿❺❻❷❽❷❼❽ Dpboss Kalyan Satta Matka Guessing Matka Result Main Bazar chart
❼❷⓿❺❻❷❽❷❼❽ Dpboss Kalyan Satta Matka Guessing Matka Result Main Bazar chart Final Matka Satta Matta Matka 143 Kalyan Chart Satta fix Jodi Kalyan Final ank Matka Boss Satta 143 Matka 420 Golden Matka Final Satta Kalyan Penal Chart Dpboss 143 Guessing Kalyan Night Chart ARNAUVALERY RECORD STORE SCAVENGER HUNT.
This is my presentation for the Record Store Scavenger Hunt.
My storyboard for a sword fight scene with lightsabers
My storyboard for a sword fight scene with lightsabers
Heart Touching Romantic Love Shayari In English with Images
Explore our beautiful collection of Romantic Love Shayari in English to express your love. These heartfelt shayaris are perfect for sharing with your loved one. Get the best words to show your love and care.
Recently uploaded (20)
➒➌➎➏➑➐➋➑➐➐ Dpboss Satta Matka Matka Guessing Kalyan Chart Indian Matka Satta ...
➒➌➎➏➑➐➋➑➐➐ Dpboss Satta Matka Matka Guessing Kalyan Chart Indian Matka Satta ...
➒➌➎➏➑➐➋➑➐➐ Kalyan Matka Satta Matka Dpboss Matka Guessing Indian Matka
➒➌➎➏➑➐➋➑➐➐ Kalyan Matka Satta Matka Dpboss Matka Guessing Indian Matka
storyboard: Victor and Verlin discussing about top hat
storyboard: Victor and Verlin discussing about top hat
❼❷⓿❺❻❷❽❷❼❽ Dpboss Matka ! Fix Satta Matka ! Matka Result ! Matka Guessing ! ...
❼❷⓿❺❻❷❽❷❼❽ Dpboss Matka ! Fix Satta Matka ! Matka Result ! Matka Guessing ! ...
❼❷⓿❺❻❷❽❷❼❽ Dpboss Kalyan Satta Matka Guessing Matka Result Main Bazar chart
❼❷⓿❺❻❷❽❷❼❽ Dpboss Kalyan Satta Matka Guessing Matka Result Main Bazar chart
My storyboard for a sword fight scene with lightsabers
My storyboard for a sword fight scene with lightsabers
Heart Touching Romantic Love Shayari In English with Images
Heart Touching Romantic Love Shayari In English with Images
Geo-Trending Example
- 1. Trending Topics by Geo-Location Insight Data Engineering Tigran Antonyan New York
- 2. Demo!!! API/Web Form Works! • Only the a simple API/Form works so far ‣ via Flask • http://trendinghashtags.net:5000
- 3. Data Transformation Map ReduceFilter Map Map Reduce
- 4. Data Flow / Pipeline Message Broker Real-Time Streaming DB No-SQL UI Web-based Camus Batch Processing Some work remains ❌ ❌ ❌
- 5. Cluster Setup HDFS Namenode Camus (cron) / Kafka Consumer Spark Master Kafka Broker HDFS Datanode #1 Stream Reader / Kafka Producer Kafka Broker Spark HDFS Datanode #2 Kafka Broker Spark HDFS Datanode #2 Kafka Broker Spark Cassandra Seed Cassandra Cassandra m4.large nodes m3.medium nodes
- 6. About Me Camping • BS in CS at FAU • Software Engineer ‣ Citrix ‣ UCONN VoTeR Center