TSS - Beyond Data Grid
•
1 like•534 views
In this session, we will discuss DataGrid usage and integration patterns that go beyond the typical Cache#put and Cache#get. Attendees will share their own experiences and we will discuss: • How DataGrid can be used with ESB to provide a reliable and fault tolerant SEDA implementation (based on Mule); • How a DataGrid can be used to execute dynamic jobs ala MapReduce using scripting languages; • How a DataGrid can be used to scale Lucene index storage, as well as use Compass to index the DataGrid.
Report
Share
Report
Share
![About Me ,[object Object],[object Object],[object Object],[object Object],[object Object]](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
Apache Arrow: Leveling Up the Analytics Stack
This document discusses the development of Apache Arrow, an open source in-memory data format designed for efficient analytical data processing on modern hardware. It provides a brief history of big data and analytics technologies leading to the need for Arrow. Key points about Arrow include that it aims to eliminate data serialization, enable code sharing across languages, and has over 400 contributors representing 11 programming languages. Notable subcomponents include DataFusion, Gandiva, and Plasma; and development is supported by organizations like Ursa Labs.
DIAL Update
The document discusses the DIAL system, which was created to allow small data producers to easily distribute Earth science data. DIAL uses publicly available components like Apache and NCSA browsers. It provides searchable metadata, basic data visualization, and immediate download capabilities. DIAL has been successfully integrated with EOSDIS standards and the Version 0 protocol. Future work includes full support for CIP and additional data access methods.
Cosmosdb graph
SQL databases use a relational model and are vertically scalable, while NoSQL databases are non-relational, horizontally scalable, and support semi-structured or unstructured data. NoSQL databases provide eventual consistency and are easier for programmers to use compared to SQL databases, which support atomic transactions but can be more difficult for programmers. Common types of NoSQL databases include document, graph, key-value pair, and column-oriented databases.
NoSQL for SQL Users
SQL-based databases have been around for decades and they power a wide range of applications. So what exactly do NoSQL databases bring to the table? In this webcast, you'll find out how NoSQL can liberate your development cycle, allow your application to scale and improve your system's uptime.
Ten Commandants For Picking NoSQL Database
The slides will walk thru the requirements and will enable to pick appropriate No SQL database as a best match for the requirements i.e. fitness for the purpose
Apache Arrow Flight: A New Gold Standard for Data Transport
This document discusses how structured data is often moved inefficiently between systems, causing waste. It introduces Apache Arrow, an open standard for in-memory data, and how Arrow can help make data movement more efficient. Systems like Snowflake and BigQuery are now using Arrow to help speed up query result fetching by enabling zero-copy data transfers and sharing file formats between query processing and storage.
Build 2017 - P4010 - A lap around Azure HDInsight and Cosmos DB Open Source A...
Recently, we released the Spark Connector for our distributed NoSQL service – Azure Cosmos DB (formerly known as Azure DocumentDB). By connecting Apache Spark running on top Azure HDInsight to Azure Cosmos DB, you can accelerate your ability to solve fast-moving data science problems and machine learning. The Spark to Azure Cosmos DB connector efficiently exploits the native Cosmos DB managed indexes and enables updateable columns when performing analytics, push-down predicate filtering against fast-changing globally-distributed data, ranging from IoT, data science, and analytics scenarios. Come learn how you can perform blazing fast planet-scale data processing with Azure Cosmos DB and HDInsight.
Aesop change data propagation
1) Aesop is an open source change data capture and propagation tool that reliably captures changes from data sources and propagates them to other data stores and systems to enable eventual consistency across polyglot data platforms.
2) It uses log mining to capture changes from data sources like MySQL and propagates the change events to consumers like ElasticSearch and HBase through an enhanced relay component.
3) It provides utilities for bootstrapping consumers, monitoring and administering the system, and has been used in production at Flipkart for applications including payments, ETL, and data serving.
Recommended
Apache Arrow: Leveling Up the Analytics Stack
This document discusses the development of Apache Arrow, an open source in-memory data format designed for efficient analytical data processing on modern hardware. It provides a brief history of big data and analytics technologies leading to the need for Arrow. Key points about Arrow include that it aims to eliminate data serialization, enable code sharing across languages, and has over 400 contributors representing 11 programming languages. Notable subcomponents include DataFusion, Gandiva, and Plasma; and development is supported by organizations like Ursa Labs.
DIAL Update
The document discusses the DIAL system, which was created to allow small data producers to easily distribute Earth science data. DIAL uses publicly available components like Apache and NCSA browsers. It provides searchable metadata, basic data visualization, and immediate download capabilities. DIAL has been successfully integrated with EOSDIS standards and the Version 0 protocol. Future work includes full support for CIP and additional data access methods.
Cosmosdb graph
SQL databases use a relational model and are vertically scalable, while NoSQL databases are non-relational, horizontally scalable, and support semi-structured or unstructured data. NoSQL databases provide eventual consistency and are easier for programmers to use compared to SQL databases, which support atomic transactions but can be more difficult for programmers. Common types of NoSQL databases include document, graph, key-value pair, and column-oriented databases.
NoSQL for SQL Users
SQL-based databases have been around for decades and they power a wide range of applications. So what exactly do NoSQL databases bring to the table? In this webcast, you'll find out how NoSQL can liberate your development cycle, allow your application to scale and improve your system's uptime.
Ten Commandants For Picking NoSQL Database
The slides will walk thru the requirements and will enable to pick appropriate No SQL database as a best match for the requirements i.e. fitness for the purpose
Apache Arrow Flight: A New Gold Standard for Data Transport
This document discusses how structured data is often moved inefficiently between systems, causing waste. It introduces Apache Arrow, an open standard for in-memory data, and how Arrow can help make data movement more efficient. Systems like Snowflake and BigQuery are now using Arrow to help speed up query result fetching by enabling zero-copy data transfers and sharing file formats between query processing and storage.
Build 2017 - P4010 - A lap around Azure HDInsight and Cosmos DB Open Source A...
Recently, we released the Spark Connector for our distributed NoSQL service – Azure Cosmos DB (formerly known as Azure DocumentDB). By connecting Apache Spark running on top Azure HDInsight to Azure Cosmos DB, you can accelerate your ability to solve fast-moving data science problems and machine learning. The Spark to Azure Cosmos DB connector efficiently exploits the native Cosmos DB managed indexes and enables updateable columns when performing analytics, push-down predicate filtering against fast-changing globally-distributed data, ranging from IoT, data science, and analytics scenarios. Come learn how you can perform blazing fast planet-scale data processing with Azure Cosmos DB and HDInsight.
Aesop change data propagation
1) Aesop is an open source change data capture and propagation tool that reliably captures changes from data sources and propagates them to other data stores and systems to enable eventual consistency across polyglot data platforms.
2) It uses log mining to capture changes from data sources like MySQL and propagates the change events to consumers like ElasticSearch and HBase through an enhanced relay component.
3) It provides utilities for bootstrapping consumers, monitoring and administering the system, and has been used in production at Flipkart for applications including payments, ETL, and data serving.
Mindtalk Tech - Behind the scenes
This document summarizes Mindtalk's approach to scalability. It discusses how Mindtalk uses databases like MongoDB and Redis with sharding and replication to handle high demand. It also covers Mindtalk's use of services like Nginx, HAProxy, Elastic Search, and message queues. The document provides tips on optimization, discusses Mindtalk's development processes involving tools like Git, Buildbot, and PandoraFMS, and lists some open job positions.
Hidden gems in SXA that you might not be aware of
Discover content tokens, variant rules, data source advanced syntax and scaffolding. This deck was presented during SUGPL meeting at Cognifide office in Poznań on October 6th, 2017.
Superset druid realtime
This document discusses using real-time data with Superset by ingesting data streams using Apache Flink and storing the data in Apache Druid. It outlines use cases like operational insights and service debugging. It proposes using Flink for its scalable streaming capabilities and SQL interface and integrating with Druid for its columnar storage and geospatial support. It also describes transforming the data through UDFs before querying and visualizing it in Superset.
Oss as a competitive advantage
Slides from my talk at SiliconIndia OSS conference in Bangalore Nov 2012. Talks about use of OSS as a competitive advantage, especially in areas like eCommerce citing Flipkart as an example.
Modern Data architecture Design
The document discusses modern data architectures. It presents conceptual models for data ingestion, storage, processing, and insights/actions. It compares traditional vs modern architectures. The modern architecture uses a data lake for storage and allows for on-demand analysis. It provides an example of how this could be implemented on Microsoft Azure using services like Azure Data Lake Storage, Azure Data Bricks, and Azure Data Warehouse. It also outlines common data management functions such as data governance, architecture, development, operations, and security.
NoSQL - what's that
The document discusses NoSQL databases as an alternative to SQL databases for big data. It provides an overview of why NoSQL databases were created due to limitations of SQL for large, distributed datasets. It then categorizes and describes some popular NoSQL databases, including key-value stores like Dynamo and Redis, document databases like MongoDB and CouchDB, graph databases like Neo4J and FlockDB, and column-oriented databases like BigTable and HBase. The document also contrasts ACID transactions with the BASE model and eventual consistency used by many NoSQL databases.
CosmosDb for beginners
CosmosDB is Microsoft's multi-model database that can be accessed using multiple APIs and provides options for consistency and geographic distribution of data across regions. It supports document, key-value, graph and table-based data models and can be accessed via SQL, MongoDB, Cassandra, Azure table and Gremlin APIs. Data can be distributed globally across regions while maintaining various levels of consistency including strong, bounded staleness, session, or eventual.
5min Lightning talk at Paris.rb
Algolia provides a hosted search engine as a service that allows for full-text search across multiple attributes with features like typo-tolerance, faceting, numerical search, and geo search. It has a JavaScript API client and can perform searches in under 10 ms to enable Google Instant-search like experiences, and integrates with Rails applications through ActiveRecord and MongoID compliance as well as automatic indexing through callbacks and separate indexes for development, test, and production environments.
HBaseConAsia2018 Track2-2: Apache Kylin on HBase: Extreme OLAP for big data
This document discusses Apache Kylin, an OLAP engine for big data. It provides 3 key points:
1. Apache Kylin is designed to provide fast, interactive queries on large datasets stored in Hadoop. It uses pre-calculated cube structures stored in HBase to enable sub-second query performance on trillion row datasets.
2. HBase was selected as the storage engine because it is natively integrated with Hadoop, supports high throughput and low-latency queries, and can store very large datasets. Kylin leverages HBase's capabilities for cube storage, metadata storage, online calculation pushing, and caching.
3. Apache Kylin is used by over 1000 global companies across
Dropbox Talk at Netflix ML Platform Meetup Spe 2019
In this talk at the Netflix Machine Learning Platform Meetup on 12 Sep 2019, Tsahi Glik gave an overview of the ML Platform at Dropbox
Useful data presentation from DataShaka
How can you identify the useful data? How to harness customer data? What is the best data to use? How can you optimise customer data?
ML Infrastracture @ Dropbox
The document describes Dropbox's machine learning infrastructure and platform. It discusses how the platform provides scalable access to Dropbox's large data sources for offline and online ML use cases. The platform aims to accelerate ML development at Dropbox by standardizing workflows, automating processes, and making ML deployment and experimentation easy. It utilizes various services like Antenna for activity data and dbxlearn for distributed training across Dropbox and AWS resources. The platform supports all stages of the ML lifecycle from data preparation to model deployment and monitoring.
Bleeding Edge Databases
This document provides information on and demonstrations of several bleeding edge database technologies: Aerospike, Algebraix Data, and Google BigQuery. It includes benchmark results, architecture diagrams, pricing and deployment details for each one. Example use cases and instructions for getting started with the technologies are also provided.
Azure CosmosDB
This document provides an overview of Azure Cosmos DB, a fully managed NoSQL database service. It describes key features like global distribution, high availability, and ease of development. It also outlines the supported programming languages and frameworks. Additionally, it explains the resource hierarchy including databases, collections, documents, and more. It provides details on querying data using SQL and modeling approaches. Finally, it discusses integrating with Azure Search for indexing and exploring data using the online query playground.
Distributed Tracing: New DevOps Foundation
This talk will serve as a practical introduction to Distributed Tracing. We will see how we can make best use of open source distributed tracing platforms like Hypertrace with Azure and find the root cause of problems and predict issues in our critical business applications beforehand.
Cloudera – One Platform to Rule Them All
Presentation about the Cloudera platform – a bit of history and the use cases.
By Nuno Barreto – Partner & Big Data Lead @XpandIT
Not only SQL - Database Choices
This document discusses database choices and provides an overview of different database technologies including relational databases, NoSQL databases, and Hadoop. It highlights key-value, columnar, document, and graph NoSQL databases and provides demos of technologies like DynamoDB, MongoDB, Neo4j, and Hadoop. The document also discusses using these database options on premises or in the cloud with providers like AWS, Google, and Microsoft and how to query data from NoSQL databases.
Community day ppt_kinesisv1.0
Kinesis is an AWS streaming data service that allows users to ingest, process, and analyze real-time streaming data. It offers three main services: Kinesis Data Streams to collect and process streaming data in real-time; Kinesis Data Firehose to load streaming data into AWS data stores; and Kinesis Data Analytics to process and analyze streaming data using SQL or Java for real-time analytics and alerts. Kinesis provides scalability, is fully managed, and enables users to build real-time applications on streaming data.
The Holy Grail of Data Analytics
My talk from Database Camp 2016 at the United Nations. I focus on how we can bridge the gap between OLTP and OLAP workloads and discuss a very promising new technology called Apache Kudu.
H2O World - Survey of Available Machine Learning Frameworks - Brendan Herger
H2O World 2015 - Brendan Herger of Capital One
- Powered by the open source machine learning software H2O.ai. Contributors welcome at: https://github.com/h2oai
- To view videos on H2O open source machine learning software, go to: https://www.youtube.com/user/0xdata
Web 2.0 i forvaltningen
En presentasjon av ulike web2.0 verktøy og relasjon til det offentlige og skolen
yGen Club Slideshow
The yGen club is an online community and network for young, successful professionals. It provides inspiration, direction, and connections with other members. Resources include exclusive networking, monthly teleseminars with experts, podcasts, e-books, e-courses, whitepapers, forums to ask career advice and find mentors. Membership includes these resources and discounts on special events. The goal is to help young professionals build their careers through networking, advice, and learning secrets to personal, leadership, and professional success.
More Related Content
What's hot
Mindtalk Tech - Behind the scenes
This document summarizes Mindtalk's approach to scalability. It discusses how Mindtalk uses databases like MongoDB and Redis with sharding and replication to handle high demand. It also covers Mindtalk's use of services like Nginx, HAProxy, Elastic Search, and message queues. The document provides tips on optimization, discusses Mindtalk's development processes involving tools like Git, Buildbot, and PandoraFMS, and lists some open job positions.
Hidden gems in SXA that you might not be aware of
Discover content tokens, variant rules, data source advanced syntax and scaffolding. This deck was presented during SUGPL meeting at Cognifide office in Poznań on October 6th, 2017.
Superset druid realtime
This document discusses using real-time data with Superset by ingesting data streams using Apache Flink and storing the data in Apache Druid. It outlines use cases like operational insights and service debugging. It proposes using Flink for its scalable streaming capabilities and SQL interface and integrating with Druid for its columnar storage and geospatial support. It also describes transforming the data through UDFs before querying and visualizing it in Superset.
Oss as a competitive advantage
Slides from my talk at SiliconIndia OSS conference in Bangalore Nov 2012. Talks about use of OSS as a competitive advantage, especially in areas like eCommerce citing Flipkart as an example.
Modern Data architecture Design
The document discusses modern data architectures. It presents conceptual models for data ingestion, storage, processing, and insights/actions. It compares traditional vs modern architectures. The modern architecture uses a data lake for storage and allows for on-demand analysis. It provides an example of how this could be implemented on Microsoft Azure using services like Azure Data Lake Storage, Azure Data Bricks, and Azure Data Warehouse. It also outlines common data management functions such as data governance, architecture, development, operations, and security.
NoSQL - what's that
The document discusses NoSQL databases as an alternative to SQL databases for big data. It provides an overview of why NoSQL databases were created due to limitations of SQL for large, distributed datasets. It then categorizes and describes some popular NoSQL databases, including key-value stores like Dynamo and Redis, document databases like MongoDB and CouchDB, graph databases like Neo4J and FlockDB, and column-oriented databases like BigTable and HBase. The document also contrasts ACID transactions with the BASE model and eventual consistency used by many NoSQL databases.
CosmosDb for beginners
CosmosDB is Microsoft's multi-model database that can be accessed using multiple APIs and provides options for consistency and geographic distribution of data across regions. It supports document, key-value, graph and table-based data models and can be accessed via SQL, MongoDB, Cassandra, Azure table and Gremlin APIs. Data can be distributed globally across regions while maintaining various levels of consistency including strong, bounded staleness, session, or eventual.
5min Lightning talk at Paris.rb
Algolia provides a hosted search engine as a service that allows for full-text search across multiple attributes with features like typo-tolerance, faceting, numerical search, and geo search. It has a JavaScript API client and can perform searches in under 10 ms to enable Google Instant-search like experiences, and integrates with Rails applications through ActiveRecord and MongoID compliance as well as automatic indexing through callbacks and separate indexes for development, test, and production environments.
HBaseConAsia2018 Track2-2: Apache Kylin on HBase: Extreme OLAP for big data
This document discusses Apache Kylin, an OLAP engine for big data. It provides 3 key points:
1. Apache Kylin is designed to provide fast, interactive queries on large datasets stored in Hadoop. It uses pre-calculated cube structures stored in HBase to enable sub-second query performance on trillion row datasets.
2. HBase was selected as the storage engine because it is natively integrated with Hadoop, supports high throughput and low-latency queries, and can store very large datasets. Kylin leverages HBase's capabilities for cube storage, metadata storage, online calculation pushing, and caching.
3. Apache Kylin is used by over 1000 global companies across
Dropbox Talk at Netflix ML Platform Meetup Spe 2019
In this talk at the Netflix Machine Learning Platform Meetup on 12 Sep 2019, Tsahi Glik gave an overview of the ML Platform at Dropbox
Useful data presentation from DataShaka
How can you identify the useful data? How to harness customer data? What is the best data to use? How can you optimise customer data?
ML Infrastracture @ Dropbox
The document describes Dropbox's machine learning infrastructure and platform. It discusses how the platform provides scalable access to Dropbox's large data sources for offline and online ML use cases. The platform aims to accelerate ML development at Dropbox by standardizing workflows, automating processes, and making ML deployment and experimentation easy. It utilizes various services like Antenna for activity data and dbxlearn for distributed training across Dropbox and AWS resources. The platform supports all stages of the ML lifecycle from data preparation to model deployment and monitoring.
Bleeding Edge Databases
This document provides information on and demonstrations of several bleeding edge database technologies: Aerospike, Algebraix Data, and Google BigQuery. It includes benchmark results, architecture diagrams, pricing and deployment details for each one. Example use cases and instructions for getting started with the technologies are also provided.
Azure CosmosDB
This document provides an overview of Azure Cosmos DB, a fully managed NoSQL database service. It describes key features like global distribution, high availability, and ease of development. It also outlines the supported programming languages and frameworks. Additionally, it explains the resource hierarchy including databases, collections, documents, and more. It provides details on querying data using SQL and modeling approaches. Finally, it discusses integrating with Azure Search for indexing and exploring data using the online query playground.
Distributed Tracing: New DevOps Foundation
This talk will serve as a practical introduction to Distributed Tracing. We will see how we can make best use of open source distributed tracing platforms like Hypertrace with Azure and find the root cause of problems and predict issues in our critical business applications beforehand.
Cloudera – One Platform to Rule Them All
Presentation about the Cloudera platform – a bit of history and the use cases.
By Nuno Barreto – Partner & Big Data Lead @XpandIT
Not only SQL - Database Choices
This document discusses database choices and provides an overview of different database technologies including relational databases, NoSQL databases, and Hadoop. It highlights key-value, columnar, document, and graph NoSQL databases and provides demos of technologies like DynamoDB, MongoDB, Neo4j, and Hadoop. The document also discusses using these database options on premises or in the cloud with providers like AWS, Google, and Microsoft and how to query data from NoSQL databases.
Community day ppt_kinesisv1.0
Kinesis is an AWS streaming data service that allows users to ingest, process, and analyze real-time streaming data. It offers three main services: Kinesis Data Streams to collect and process streaming data in real-time; Kinesis Data Firehose to load streaming data into AWS data stores; and Kinesis Data Analytics to process and analyze streaming data using SQL or Java for real-time analytics and alerts. Kinesis provides scalability, is fully managed, and enables users to build real-time applications on streaming data.
The Holy Grail of Data Analytics
My talk from Database Camp 2016 at the United Nations. I focus on how we can bridge the gap between OLTP and OLAP workloads and discuss a very promising new technology called Apache Kudu.
H2O World - Survey of Available Machine Learning Frameworks - Brendan Herger
H2O World 2015 - Brendan Herger of Capital One
- Powered by the open source machine learning software H2O.ai. Contributors welcome at: https://github.com/h2oai
- To view videos on H2O open source machine learning software, go to: https://www.youtube.com/user/0xdata
What's hot (20)
HBaseConAsia2018 Track2-2: Apache Kylin on HBase: Extreme OLAP for big data
HBaseConAsia2018 Track2-2: Apache Kylin on HBase: Extreme OLAP for big data
Dropbox Talk at Netflix ML Platform Meetup Spe 2019
Dropbox Talk at Netflix ML Platform Meetup Spe 2019
H2O World - Survey of Available Machine Learning Frameworks - Brendan Herger
H2O World - Survey of Available Machine Learning Frameworks - Brendan Herger
Viewers also liked
Web 2.0 i forvaltningen
En presentasjon av ulike web2.0 verktøy og relasjon til det offentlige og skolen
yGen Club Slideshow
The yGen club is an online community and network for young, successful professionals. It provides inspiration, direction, and connections with other members. Resources include exclusive networking, monthly teleseminars with experts, podcasts, e-books, e-courses, whitepapers, forums to ask career advice and find mentors. Membership includes these resources and discounts on special events. The goal is to help young professionals build their careers through networking, advice, and learning secrets to personal, leadership, and professional success.
Evaluacion Desempeno Docente
Este documento presenta la evaluación del desempeño docente de un profesor de informática en una escuela secundaria de Guanacaste. El profesor demostró un alto nivel de compromiso con sus estudiantes y el programa, planeando lecciones completas, creando material didáctico, y explotando recursos tecnológicos. También participó activamente en capacitaciones y aplicó comentarios de asesores para mejorar continuamente. Los estudiantes produjeron aplicaciones y representaciones digitales que respondieron a la propuesta pedagógica y abordaron problem
Document Management
The document discusses different aspects of document management including formal structures like office attributes and domain ontologies. It also discusses metadata for embedding information in documents, interfaces for usage, and infrastructures for creation. Different levels of interpretation are mentioned like linguistic, district/cluster, and community interpretations from both author and reader perspectives.
Compass - JBUG presentation
This document summarizes a presentation given by Shay Banon on the Compass search framework. Compass provides simplified search capabilities for applications by mapping data to a search index using various mapping types. It includes core components for indexing and searching, as well as integrations with ORM tools and data grids through Compass Gps and Needle. The presentation covered the capabilities and architecture of Compass.
Ant 227 Final Presentation
The document summarizes research conducted on the Westside Farmers Market (WFM) in Rochester, NY. The research included surveys of customers, interviews with vendors and community members, and discussions with market planners. Key findings were that most customers are regulars who value local and organic products. Vendors see the market as important for business but want more variety and customers. University of Rochester students and faculty were generally unaware of the market due to issues around location, cost, safety perceptions, and lack of promotion on campus. Recommendations focused on expanding the market's customer base and fostering a stronger sense of community ownership.
Clark Family
Family is a gift from God that cannot be chosen; while families have faults, they should accept each other despite imperfections.
Viewers also liked (8)
Similar to TSS - Beyond Data Grid
Engineering practices in big data storage and processing
Data
Preparation
Offline
Data
(CRM)
Data
Warehouse
Data
Mining
Data
Visualization
离线批量分析为主
19
大数据服务平台
用户
应用
Web应用
移动应用
数据服务平台
数据源
数据仓库
数据处理
数据分析
数据可视化
原则:
1. 提供统一的数据访问接口
2. 支持多种数据源与多种应用
3. 以离线批量分析为主,兼顾实时
Introduction to MongoDB
This document contains information about Justin Smestad and MongoDB. It includes Justin's contact information and background working as a software engineer with skills in Ruby, JavaScript, Clojure, and passion for DevOps. It also provides an overview of MongoDB, describing it as a scalable, high-performance, open source, schema-free, document-oriented database. Key features of MongoDB like indexing, master-slave replication, and horizontal scaling with replica sets and sharding are summarized.
Large scale, interactive ad-hoc queries over different datastores with Apache...
Presented at JAX London 2013
Apache Drill is a distributed system for interactive ad-hoc query and analysis of large-scale datasets. It is the Open Source version of Google’s Dremel technology. Apache Drill is designed to scale to thousands of servers and able to process Petabytes of data in seconds, enabling SQL-on-Hadoop and supporting a variety of data sources.
Strata NY 2018: The deconstructed database
The document discusses the evolution of big data systems from flat files in Hadoop to deconstructed databases. Originally, Hadoop used flat files with no schema and tied data storage to execution. This was flexible but inefficient. SQL databases provided schemas and separated storage from execution but were inflexible. Modern systems decompose databases into independent components like storage, processing, and machine learning that can be mixed and matched. Standards are emerging for columnar storage, querying, schemas, and data exchange. Future improvements include better interoperability, data abstractions, and governance.
From flat files to deconstructed database
From flat files to deconstructed databases:
- Originally, Hadoop used flat files and MapReduce which was flexible but inefficient for queries.
- The database world used SQL and relational models with optimizations but were inflexible.
- Now components like storage, processing, and machine learning can be mixed and matched more efficiently with standards like Apache Calcite, Parquet, Avro and Arrow.
Azure Synapse Analytics Overview (r2)
Azure Synapse Analytics is Azure SQL Data Warehouse evolved: a limitless analytics service, that brings together enterprise data warehousing and Big Data analytics into a single service. It gives you the freedom to query data on your terms, using either serverless on-demand or provisioned resources, at scale. Azure Synapse brings these two worlds together with a unified experience to ingest, prepare, manage, and serve data for immediate business intelligence and machine learning needs. This is a huge deck with lots of screenshots so you can see exactly how it works.
Hopsworks in the cloud Berlin Buzzwords 2019
This talk, given at Berlin Buzzwords 2019, describes the recent progress in making Hopsworks a cloud-native platform, with HA data-center support added for HopsFS.
Power BI with Essbase in the Oracle Cloud
This document discusses connecting Oracle Analytics Cloud (OAC) Essbase data to Microsoft Power BI. It provides an overview of Power BI and OAC, describes various methods for connecting the two including using a REST API and exporting data to Excel or CSV files, and demonstrates some visualization capabilities in Power BI including trends over time. Key lessons learned are that data can be accessed across tools through various connections, analytics concepts are often similar between tools, and while partnerships exist between Microsoft and Oracle, integration between specific products like Power BI and OAC is still limited.
Spark and Couchbase: Augmenting the Operational Database with Spark
The document discusses integrating Couchbase NoSQL with Apache Spark for augmenting operational databases with analytics. It outlines architectural alignment between Couchbase and Spark, including automatic data sharding and locality, data streaming replication from Couchbase to Spark, predicate pushdown to Couchbase global indexes from Spark, and flexible schemas. Integration points discussed include using the Couchbase data locality hints in Spark, limitations on predicate pushdown for Couchbase views and N1QL, and using the Couchbase change data capture protocol for low-latency data streaming into Spark Streaming.
Azure Data platform
Azure data platform services including: SQL Server, DocumentDB, Azure Search, HDInsight, Security features and Data lake.
Michael stack -the state of apache h base
The document provides an overview of Apache HBase, an open source, distributed, scalable, big data non-relational database. It discusses that HBase is modeled after Google's Bigtable and built on Hadoop for storage. It also summarizes that HBase is used by many large companies for applications such as messaging, real-time analytics, and search indexing. The project is led by an active community of committers and sees steady improvements and new features with each monthly release.
Keeping Analytics Data Fresh in a Streaming Architecture | John Neal, Qlik
Qlik is an industry leader across its solution stack, both on the Data Integration side of things with Qlik Replicate (real-time CDC) and Qlik Compose (data warehouse and data lake automation), and on the Analytics side with Qlik Sense. These two “sides” of Qlik are coming together more frequently these days as the need for “always fresh” data increases across organizations.
When real-time streaming applications are the topic du jour, those companies are looking to Apache Kafka to provide the architectural backbone those applications require. Those same companies turn to Qlik Replicate to put the data from their enterprise database systems into motion at scale, whether that data resides in “legacy” mainframe databases; traditional relational databases such as Oracle, MySQL, or SQL Server; or applications such as SAP and SalesForce.
In this session we will look in depth at how Qlik Replicate can be used to continuously stream changes from a source database into Apache Kafka. From there, we will explore how a purpose-built consumer can be used to provide the bridge between Apache Kafka and an analytics application such as Qlik Sense.
Data Analytics Meetup: Introduction to Azure Data Lake Storage
Microsoft Azure Data Lake Storage is designed to enable operational and exploratory analytics through a hyper-scale repository. Journey through Azure Data Lake Storage Gen1 with Microsoft Data Platform Specialist, Audrey Hammonds. In this video she explains the fundamentals to Gen 1 and Gen 2, walks us through how to provision a Data Lake, and gives tips to avoid turning your Data Lake into a swamp.
Learn more about Data Lakes with our blog - Data Lakes: Data Agility is Here Now https://bit.ly/2NUX1H6
Enterprise guide to building a Data Mesh
Making Data Mesh simple, Open Source and available to all; without vendor lock-in, without complex tooling and to use an approach centered around ‘specifications’, existing tools and baking in a ‘domain’ model.
Using Data Lakes: Data Analytics Week SF
Data Analytics Week at the San Francisco Loft
Using Data Lakes
A data lake can be used as a source for both structured and unstructured data - but how? We'll look at using open standards including Spark and Presto with Amazon EMR, Amazon Redshift Spectrum and Amazon Athena to process and understand data.
Speakers:
John Mallory - Principal Business Development Manager Storage (Object), AWS
Hemant Borole - Sr. Big Data Consultant, AWS
Modern Analytics Academy - Data Modeling (1).pptx
This document provides an overview of Modern Analytics Academy and Azure Synapse Analytics. It introduces the Modern Analytics Academy team and their agenda to discuss modeling, data lakes, Synapse, and a demo. It then covers key concepts like the data lake, logical data warehouse, and data warehouse. It describes the role of data in modern analytics between data lakes and data warehouses. Finally, it introduces Azure Synapse Analytics and its capabilities for dedicated SQL pools, serverless SQL pools, and Apache Spark pools for unified analytics.
Alfresco WCM For High Scalability
The document discusses strategies for scaling Alfresco web content management deployments. It covers types of scalability including horizontal and vertical scaling. Horizontal scaling involves adding more servers while vertical scaling means adding more resources to individual servers. The document provides blueprints for scaling static and dynamic sites using techniques like load balancing, replication to multiple file system receivers and dynamic site servers, and caching. It also addresses how to determine whether replication or clustering is better suited for a given deployment.
Real Time Streaming with Flink & Couchbase
Content from a recent meetup in Madrid about Real Time Streaming with Flink & Couchbase.
Two demos included
Microsoft Data Platform - What's included
This document provides an overview of a speaker and their upcoming presentation on Microsoft's data platform. The speaker is a 30-year IT veteran who has worked in various roles including BI architect, developer, and consultant. Their presentation will cover collecting and managing data, transforming and analyzing data, and visualizing and making decisions from data. It will also discuss Microsoft's various product offerings for data warehousing and big data solutions.
H2O at BelgradeR Meetup
The document summarizes a presentation given by Joe Chow on H2O at the BelgradeR Meetup. The agenda includes an introduction to H2O, the company, why H2O is useful, the H2O machine learning platform, Deep Water for deep learning, latest H2O developments, and demos. Joe will discuss H2O's introduction to machine learning, distributed algorithms, interfaces for R, Python and Flow, and Deep Water for distributed deep learning on GPUs with TensorFlow, MXNet or Caffe.
Similar to TSS - Beyond Data Grid (20)
Engineering practices in big data storage and processing
Engineering practices in big data storage and processing
Large scale, interactive ad-hoc queries over different datastores with Apache...
Large scale, interactive ad-hoc queries over different datastores with Apache...
Spark and Couchbase: Augmenting the Operational Database with Spark
Spark and Couchbase: Augmenting the Operational Database with Spark
Keeping Analytics Data Fresh in a Streaming Architecture | John Neal, Qlik
Keeping Analytics Data Fresh in a Streaming Architecture | John Neal, Qlik
Data Analytics Meetup: Introduction to Azure Data Lake Storage
Data Analytics Meetup: Introduction to Azure Data Lake Storage
Recently uploaded
Communications Mining Series - Zero to Hero - Session 1
This session provides introduction to UiPath Communication Mining, importance and platform overview. You will acquire a good understand of the phases in Communication Mining as we go over the platform with you. Topics covered:
• Communication Mining Overview
• Why is it important?
• How can it help today’s business and the benefits
• Phases in Communication Mining
• Demo on Platform overview
• Q/A
20240609 QFM020 Irresponsible AI Reading List May 2024
Everything I found interesting about the irresponsible use of machine intelligence in May 2024
Climate Impact of Software Testing at Nordic Testing Days
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
Pushing the limits of ePRTC: 100ns holdover for 100 days
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
Full-RAG: A modern architecture for hyper-personalization
Mike Del Balso, CEO & Co-Founder at Tecton, presents "Full RAG," a novel approach to AI recommendation systems, aiming to push beyond the limitations of traditional models through a deep integration of contextual insights and real-time data, leveraging the Retrieval-Augmented Generation architecture. This talk will outline Full RAG's potential to significantly enhance personalization, address engineering challenges such as data management and model training, and introduce data enrichment with reranking as a key solution. Attendees will gain crucial insights into the importance of hyperpersonalization in AI, the capabilities of Full RAG for advanced personalization, and strategies for managing complex data integrations for deploying cutting-edge AI solutions.
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank
Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.
Securing your Kubernetes cluster_ a step-by-step guide to success !
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
20240605 QFM017 Machine Intelligence Reading List May 2024
Everything I found interesting about machines behaving intelligently during May 2024
20 Comprehensive Checklist of Designing and Developing a Website
Dive into the world of Website Designing and Developing with Pixlogix! Looking to create a stunning online presence? Look no further! Our comprehensive checklist covers everything you need to know to craft a website that stands out. From user-friendly design to seamless functionality, we've got you covered. Don't miss out on this invaluable resource! Check out our checklist now at Pixlogix and start your journey towards a captivating online presence today.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
20240607 QFM018 Elixir Reading List May 2024
Everything I found interesting about the Elixir programming ecosystem in May 2024
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Presentation of the OECD Artificial Intelligence Review of Germany
Consult the full report at https://www.oecd.org/digital/oecd-artificial-intelligence-review-of-germany-609808d6-en.htm
TrustArc Webinar - 2024 Global Privacy Survey
How does your privacy program stack up against your peers? What challenges are privacy teams tackling and prioritizing in 2024?
In the fifth annual Global Privacy Benchmarks Survey, we asked over 1,800 global privacy professionals and business executives to share their perspectives on the current state of privacy inside and outside of their organizations. This year’s report focused on emerging areas of importance for privacy and compliance professionals, including considerations and implications of Artificial Intelligence (AI) technologies, building brand trust, and different approaches for achieving higher privacy competence scores.
See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe.
This webinar will review:
- The top 10 privacy insights from the fifth annual Global Privacy Benchmarks Survey
- The top challenges for privacy leaders, practitioners, and organizations in 2024
- Key themes to consider in developing and maintaining your privacy program
Recently uploaded (20)
Communications Mining Series - Zero to Hero - Session 1
Communications Mining Series - Zero to Hero - Session 1
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
Pushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 days
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Full-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalization
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website
Monitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR Events
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
TSS - Beyond Data Grid
- 1. Beyond a Data Grid Shay Banon System Architect GigaSpaces Technologies http://www.kimchy.org
- 4. Typical Tier Based Architecture
- 5. Collocated and Partitioned Architecture
- 9. The Data Grid As a Search Index Storage
- 10. Write Behind POJO Indexing
- 11. Distributed POJO Indexing & Search
- 13. IMDG As A Remoting Backbone (Direct)
- 14. IMDG As A Remoting Backbone (Parallel)
- 15. Q & A