Big data uservices
•
0 likes•62 views
This document discusses trends in computing and data architectures. It notes that hardware is becoming more virtualized through containers and functions, bringing code closer to storage. Meanwhile, data structures are becoming more distributed through databases, distributed databases, and distributed ledgers. Computing scopes are decreasing while storage scopes are increasing. This is because data is inert but code can be more tightly controlled at smaller scopes. The document also discusses challenges of integrating microservices with data, including consistency, and notes the importance of people and processes ("metadata") alongside technology.
Report
Share
Report
Share
Download to read offline
Recommended
Exploring MongoDB & Elasticsearch: Better Together
An Open Talk at DeveloperWeek Austin 2017 by Kimberly Wilkins (@dba_denizen), Principal Engineer - Databases at ObjectRocket. Featuring new use cases like Bitcoin, AI, IoT, and all the cool things.
Accelerating Delivery of Data Products - The EBSCO Way
EBSCO Information Services (EBSCO) is the leading provider of electronic journals, magazines, eBooks, audioBooks, and online research content for libraries, including hundreds of research databases, historical archives, point-of-care medical reference, and corporate learning tools serving millions of end users at tens of thousands of institutions worldwide. The EBSCO platform is a widely used platform serving the needs of researchers at all levels in academic institutions, schools, public libraries, hospitals, medical institutions, corporations and government institutions. Data is our business, and delivering new products quickly is our competitive advantage. We build hundreds of data products and accelerating the analysis, transformation of new datasets translates to revenue and competitiveness. And since our data is so varied, using MognoDB to store data flexibly and JSON Studio to analyze this data allows us to deliver products to market faster. In this session we will describe this process that helped us expedite delivery of new datasets, and give real examples of how data is used, analyzed and processed.
Big data processing system
The document discusses big data processing systems. It begins with an overview of big data and its evolution due to technologies like IoT, social media, and smart cars. This has led to an exponential increase in data volume and variety, including structured, semi-structured and unstructured data. Traditional databases cannot handle this type and size of data. The document then introduces Hadoop as an open source framework to process large, diverse datasets across clusters. It uses HDFS for storage and MapReduce for parallel processing of data stored in HDFS. Hadoop provides scalable solutions to the problems of storing huge, growing datasets and processing complex, diverse data faster.
Migration strategies for a mission critical cluster
The document outlines a migration plan to improve the performance and scalability of an Elasticsearch cluster. The current cluster has performance issues due to a large inverted index, outdated software version, and lack of document purge policies. The plan involves defining requirements, measuring the new infrastructure needs, installing an updated version, defining index structures, performing a remote reindex to migrate data, and adding logic to avoid downtime during migration. The new cluster will have dedicated roles, monthly indices of optimal size, and policies to retain only one year of data.
Towards Data Operations
The document discusses integrating data science workflows with continuous integration and delivery (CICD) practices, known as Data Operations or DataOps. It outlines challenges in traditional data science workflows around data versioning, reproducibility, and delivering value incrementally. Key aspects of CICD for data and models are described, including continuous data quality assessment, model tuning, and deployment. The Data-Mill project is introduced as an open-source platform for enforcing DataOps principles on Kubernetes clusters through modular "flavors" of software components and built-in exploration environments.
Mastro
This document discusses Mastro, an open-source metadata management system developed in Go. It provides connectors to various data sources, a catalog service to index and retrieve assets, a feature store to manage processed data, and crawlers to automatically index data. The document outlines Mastro's components, including Data Assets, Connectors, the Catalogue Service, Feature Store, Crawlers, and UI. It also describes Mastro Version Control (MVC), which provides versioning capabilities for datasets.
Mastro
This document discusses Mastro, an open-source metadata management system developed in Go. It provides connectors to various data sources, a catalog service to index and retrieve assets, a feature store to manage processed data, and crawlers to automatically index data. The document outlines Mastro's components, including Data Assets, Connectors, the Catalogue Service, Feature Store, Crawlers, and UI. It also describes Mastro Version Control (MVC), which provides versioning capabilities for datasets.
Patient list problem solved!
This document summarizes a patient list project created at NHS Hack Day 2012. It estimates that enabling 10,000 junior doctors to save just 5 minutes per day through the project could save over £3.6 million per year. It describes some of the technical components used including Mirth HL7, Linux, Apache, MySQL, Ruby and Git. It also lists some of the contributors to the project.
Recommended
Exploring MongoDB & Elasticsearch: Better Together
An Open Talk at DeveloperWeek Austin 2017 by Kimberly Wilkins (@dba_denizen), Principal Engineer - Databases at ObjectRocket. Featuring new use cases like Bitcoin, AI, IoT, and all the cool things.
Accelerating Delivery of Data Products - The EBSCO Way
EBSCO Information Services (EBSCO) is the leading provider of electronic journals, magazines, eBooks, audioBooks, and online research content for libraries, including hundreds of research databases, historical archives, point-of-care medical reference, and corporate learning tools serving millions of end users at tens of thousands of institutions worldwide. The EBSCO platform is a widely used platform serving the needs of researchers at all levels in academic institutions, schools, public libraries, hospitals, medical institutions, corporations and government institutions. Data is our business, and delivering new products quickly is our competitive advantage. We build hundreds of data products and accelerating the analysis, transformation of new datasets translates to revenue and competitiveness. And since our data is so varied, using MognoDB to store data flexibly and JSON Studio to analyze this data allows us to deliver products to market faster. In this session we will describe this process that helped us expedite delivery of new datasets, and give real examples of how data is used, analyzed and processed.
Big data processing system
The document discusses big data processing systems. It begins with an overview of big data and its evolution due to technologies like IoT, social media, and smart cars. This has led to an exponential increase in data volume and variety, including structured, semi-structured and unstructured data. Traditional databases cannot handle this type and size of data. The document then introduces Hadoop as an open source framework to process large, diverse datasets across clusters. It uses HDFS for storage and MapReduce for parallel processing of data stored in HDFS. Hadoop provides scalable solutions to the problems of storing huge, growing datasets and processing complex, diverse data faster.
Migration strategies for a mission critical cluster
The document outlines a migration plan to improve the performance and scalability of an Elasticsearch cluster. The current cluster has performance issues due to a large inverted index, outdated software version, and lack of document purge policies. The plan involves defining requirements, measuring the new infrastructure needs, installing an updated version, defining index structures, performing a remote reindex to migrate data, and adding logic to avoid downtime during migration. The new cluster will have dedicated roles, monthly indices of optimal size, and policies to retain only one year of data.
Towards Data Operations
The document discusses integrating data science workflows with continuous integration and delivery (CICD) practices, known as Data Operations or DataOps. It outlines challenges in traditional data science workflows around data versioning, reproducibility, and delivering value incrementally. Key aspects of CICD for data and models are described, including continuous data quality assessment, model tuning, and deployment. The Data-Mill project is introduced as an open-source platform for enforcing DataOps principles on Kubernetes clusters through modular "flavors" of software components and built-in exploration environments.
Mastro
This document discusses Mastro, an open-source metadata management system developed in Go. It provides connectors to various data sources, a catalog service to index and retrieve assets, a feature store to manage processed data, and crawlers to automatically index data. The document outlines Mastro's components, including Data Assets, Connectors, the Catalogue Service, Feature Store, Crawlers, and UI. It also describes Mastro Version Control (MVC), which provides versioning capabilities for datasets.
Mastro
This document discusses Mastro, an open-source metadata management system developed in Go. It provides connectors to various data sources, a catalog service to index and retrieve assets, a feature store to manage processed data, and crawlers to automatically index data. The document outlines Mastro's components, including Data Assets, Connectors, the Catalogue Service, Feature Store, Crawlers, and UI. It also describes Mastro Version Control (MVC), which provides versioning capabilities for datasets.
Patient list problem solved!
This document summarizes a patient list project created at NHS Hack Day 2012. It estimates that enabling 10,000 junior doctors to save just 5 minutes per day through the project could save over £3.6 million per year. It describes some of the technical components used including Mirth HL7, Linux, Apache, MySQL, Ruby and Git. It also lists some of the contributors to the project.
NoSQL document oriented data access for .net systems with postgresql and marten
Relational-only databases are great fit for storing relational models, but are fairly problematic and limited when it comes to storing aggregates in DDD (domain-driven design) systems. On the other hand, NoSQL document-oriented databases are natural fit for storing and loading [DDD] aggregates.
In this session you are going to learn:
- Why relational-only databases are problematic and limited for storing aggregates and how document-oriented databases come into play.
- Why PostgreSQL is the perfect database for storing, loading and querying DDD aggregates.
- How Marten .NET library makes developers productive working with PostgreSQL's document features, leveraging the powerful JSONB data type, without losing any of your existing LINQ and SQL knowledge.
- And finally, how to use the features of both worlds combined (relational and document oriented) in the same database server, in order to get maximum out of this data access approach.
Overall, this approach helps dev teams be more productive and focus on designing good domain models, instead of fighting the impediment mismatch between the object model and the relational model, which comes from relational-only databases, with or without using an object-relational mapper. Ultimately, this approach opens the doors to polyglot persistence.
Relational is the new Big Data by Miguel Ángel Fajardo and Daniel Dominguez a...
Relational databases were the persistence system of choice for decades, until the Web 2.0 in the 2000s required to process volumes of data so big it needed distributed systems running in parallel. A new type of databases (NoSQL) was adopted to solve this problem in different ways.
https://www.bigdataspain.org/2017/talk/relational-is-the-new-big-data
Big Data Spain 2017
16th - 17th November Kinépolis Madrid
A Gentle Introduction to Big Data
This document provides an introduction to big data, including definitions and key concepts. Big data refers to large datasets that cannot be processed using traditional methods due to issues of volume, velocity, variety, veracity, and value. It discusses characteristics of big data like volume (scale), velocity (speed of data production), and variety (different data formats). The document also outlines different data types, processing methods like batch and stream, common big data architectures, and popular tools used for big data like Hadoop, Spark, and Kafka. In closing, it emphasizes that big data deals with large-scale data processing and highlights some takeaways.
How to get started in Big Data for master's students
For interested students, or to some extent, for general curious persons, here's an experience-based guidelines about the topic of Big Data.
Ipc clipboard and data copy
This document describes two techniques for inter-process communication (IPC) in Windows: using the clipboard and using data copy. It explains how to share data between processes using the Windows clipboard API functions like OpenClipboard(), SetClipboardData(), and CloseClipboard(). It also explains how to send data between processes using the WM_COPYDATA message, a COPYDATASTRUCT structure, and the SendMessage() API function. Both techniques allow sharing data between processes on the same computer.
Hadoop @ eBuddy
The document discusses eBuddy's use of Hadoop for data warehousing and analytics needs. It describes eBuddy's logging history and current large volume of login records. It outlines the needs of different teams for analysis. It then reviews eBuddy's timeline of using traditional RDBMS, custom MapReduce code, Pig scripts and migrating to Hive for SQL queries. The document discusses their current use of Hive, ETL processes, future plans to use Cassandra and streaming ETL to Flume and questions about Hive partition pruning and left outer joins in Pig.
Big Data Streams Architectures. Why? What? How?
With a current zoo of technologies and different ways of their interaction it's a big challenge to architect a system (or adopt existed one) that will conform to low-latency BigData analysis requirements. Apache Kafka and Kappa Architecture in particular take more and more attention over classic Hadoop-centric technologies stack. New Consumer API put significant boost in this direction. Microservices-based streaming processing and new Kafka Streams tend to be a synergy in BigData world.
Exploring Large Chemical Data Sets
This document describes ChemData, an open-source tool for interactive analysis and visualization of large chemical datasets. ChemData stores chemical data in a MongoDB database and uses VTK for visualization. It allows users to perform queries, similarity searching, and multidimensional analysis through features like scatter plots, histograms, parallel coordinates, and k-means clustering. ChemData also integrates with other open chemistry tools like Avogadro and represents molecular structures using the ChemicalJSON format. Future directions include integration with additional databases and computational job results.
Introduction to Big Data
The document discusses big data, defining it as very large data sets with high volumes, velocities, and varieties that are complex to process. It notes trends like Moore's Law, Kryder's Law, and Butter's Law that support big data growth by increasing processing and storage capabilities while decreasing costs. Examples of big data include NASA climate simulations, Large Hadron Collider data, and Walmart customer data. Tools for working with big data are discussed, including Hadoop, NoSQL databases, and MPP databases. The future is predicted to include continued data growth, lower storage and processing costs, higher network capacities, and potential for data growth to exceed processing abilities.
Mobicents Summit 2012 - Alexandre Mendonca - Mobicents jDiameter
This document summarizes a presentation on Mobicents Diameter. It provides an overview of Diameter basics and architecture, including support for high availability and fault tolerance. It discusses past achievements from 2011-2012, including supported applications and stability improvements. Future goals for 2012-2013 are outlined, such as improving documentation, examples, and out-of-box experience. Specific upcoming releases through 1.7.0 are also summarized.
Data Management in Cloud Platforms
This document discusses data management in cloud platforms. It begins with an introduction to cloud computing, noting its benefits like reduced costs and ability to scale resources. It then covers cloud characteristics like elasticity and security risks. The document discusses data analysis challenges in clouds like fault tolerance and heterogeneous environments. It focuses on data replication techniques between master and slave nodes and different replication types. It also covers master-slave election processes to select a primary node and considers factors like priority, network partitions, and quick detection of a failed primary.
Geo data analytics
This document summarizes a presentation on geo data analytics. It discusses why geo data matters, common data formats and libraries for working with spatial data, challenges of working with spatial data at scale, and solutions including dimension reduction techniques and spatial databases. It also provides tips for working with spatial data in tools like Spark, R, and Javascript libraries.
Data Wrangling
Challenges faced during extracting data from websites and tools to do the extraction. Presentation by Manish Dangol
Big Data Lakes Benchmarking 2018
This document summarizes benchmarks for accessing data stored in different formats in a data lake. Scanning 1.2GB of text data stored in JSON, CSV, or Parquet formats was fastest when stored as Parquet, at 8.6 seconds. Retrieving a single row was also fastest from Parquet, at 1.75 seconds, because the Parquet format allows limiting the data read through predicate pushdown. The document concludes that Parquet is the best format even for row-oriented data when accessed through a distributed system like a data lake.
TechEvent Time Seriesd Databases
Time series databases are optimized for storing time-stamped sensor or event data in regular intervals, with little change to stored values. They differ from relational databases by using column-oriented storage, indexing time as a key, and supporting high write volumes, automatic data compaction, and continuous queries over aggregated time series data. Examples include InfluxDB, which uses a time-structured merge tree to optimize storage of billions of data points within a low disk footprint. While time series databases excel at certain use cases, relational databases may still be better for other types of complex, non-time-based data.
polystore_NYC_inrae_sysinfo2021-1.pdf
- The document discusses query optimization in polystore systems using New York City open data as a proof of concept.
- It outlines the NYC polystore implementation including workload characteristics, ETL workflows, examples of ad-hoc queries, and materialized views across different data stores.
- Future work discussed includes performance evaluation of ad-hoc queries versus queries on materialized data, and polystore data warehouse maintenance.
- Related work covers topics like polystore architectures, OLAP on different data models, and benchmarking of polystore systems.
Pass Elk: CAP Theorem since 90s and Beyond
This talk is about CAP theorem since its inception in 90s, and the PACELC theorem which followed it, and the present.
Ruby,no sql and tokyocabinet
This document discusses the transition from relational databases to NoSQL databases. It provides an overview of key-value databases like Tokyo Cabinet and how they provide high performance and scalability alternatives to SQL databases. It also describes how Tokyo Cabinet can be used with an ActiveRecord adapter to integrate NoSQL style databases into Rails applications, bridging the gap between SQL and NoSQL. Finally, it outlines a vision for further developing Tokyo Cabinet into a cloud database that can provide a unified interface to both SQL and NoSQL databases.
Week4
This document provides an overview of week 4 topics for a Code Club on wrangling data with Python. The key points covered are:
- Merging data from different data frames and working with "tidy data" formats.
- Taking first steps in reshaping datasets, including long to wide data transforms and pivots.
- Various strategies for cleaning datasets, such as handling empty/duplicate rows and columns, cleaning text strings, and mapping values.
- The melt and pivot functions for reshaping data between wide and long formats.
Quick dive into the big data pool without drowning - Demi Ben-Ari @ Panorays
Everybody wants to go on the “Big Data” hype cycle, “To do Scale”, to use the coolest tools in the market like Hadoop, Apache Spark, Apache Cassandra, etc.
But do they ask themselves is there really a reason for that?
In the talk we’ll make a brief overview to all of the technologies in the Big Data world nowadays and we’ll talk about the problems that really emerge when you’d like to enter the great world of Big Data handling.
Showing you the Hadoop ecosystem and Apache Spark and all of the distributed tools leading the market today, will give you all a notion of what will be the real costs entering that world.
Promise that I’ll share some stories from the trenches :)
(And about the “pool” thing...I don’t really know how to swim)
Data platform architecture principles - ieee infrastructure 2020
This document discusses principles for building a healthy data platform, including:
1. Establishing explicit contracts between teams to define dependencies and service level agreements.
2. Abstracting the data platform into services for ingesting, storing, and processing data in motion and at rest.
3. Enabling observability of data pipelines through metadata collection and integration with tools like Marquez to provide lineage, availability, and change management visibility.
AWS Big Data Demystified #1.2 | Big Data architecture lessons learned
A while ago I entered the challenging world of Big Data. As an engineer, at first, I was not so impressed with this field. As time went by, I realised more and more, The technological challenges in this area are too great to master by one person. Just look at the picture in this articles, it only covers a small fraction of the technologies in the Big Data industry…
Consequently, I created a meetup detailing all the challenges of Big Data, especially in the world of cloud. I am using AWS & GCP and Data Center infrastructure to answer the basic questions of anyone starting their way in the big data world.
how to transform data (TXT, CSV, TSV, JSON) into Parquet, ORC,AVRO which technology should we use to model the data ? EMR? Athena? Redshift? Spectrum? Glue? Spark? SparkSQL? GCS? Big Query? Data flow? Data Lab? tensor flow? how to handle streaming? how to manage costs? Performance tips? Security tip? Cloud best practices tips?
In this meetup we shall present lecturers working on several cloud vendors, various big data platforms such hadoop, Data warehourses , startups working on big data products. basically - if it is related to big data - this is THE meetup.
Some of our online materials (mixed content from several cloud vendor):
Website:
https://big-data-demystified.ninja (under construction)
Meetups:
https://www.meetup.com/Big-Data-Demystified
https://www.meetup.com/AWS-Big-Data-Demystified/
You tube channels:
https://www.youtube.com/channel/UCMSdNB0fGmX5dXI7S7Y_LFA?view_as=subscriber
https://www.youtube.com/channel/UCzeGqhZIWU-hIDczWa8GtgQ?view_as=subscriber
Audience:
Data Engineers
Data Science
DevOps Engineers
Big Data Architects
Solution Architects
CTO
VP R&D
More Related Content
What's hot
NoSQL document oriented data access for .net systems with postgresql and marten
Relational-only databases are great fit for storing relational models, but are fairly problematic and limited when it comes to storing aggregates in DDD (domain-driven design) systems. On the other hand, NoSQL document-oriented databases are natural fit for storing and loading [DDD] aggregates.
In this session you are going to learn:
- Why relational-only databases are problematic and limited for storing aggregates and how document-oriented databases come into play.
- Why PostgreSQL is the perfect database for storing, loading and querying DDD aggregates.
- How Marten .NET library makes developers productive working with PostgreSQL's document features, leveraging the powerful JSONB data type, without losing any of your existing LINQ and SQL knowledge.
- And finally, how to use the features of both worlds combined (relational and document oriented) in the same database server, in order to get maximum out of this data access approach.
Overall, this approach helps dev teams be more productive and focus on designing good domain models, instead of fighting the impediment mismatch between the object model and the relational model, which comes from relational-only databases, with or without using an object-relational mapper. Ultimately, this approach opens the doors to polyglot persistence.
Relational is the new Big Data by Miguel Ángel Fajardo and Daniel Dominguez a...
Relational databases were the persistence system of choice for decades, until the Web 2.0 in the 2000s required to process volumes of data so big it needed distributed systems running in parallel. A new type of databases (NoSQL) was adopted to solve this problem in different ways.
https://www.bigdataspain.org/2017/talk/relational-is-the-new-big-data
Big Data Spain 2017
16th - 17th November Kinépolis Madrid
A Gentle Introduction to Big Data
This document provides an introduction to big data, including definitions and key concepts. Big data refers to large datasets that cannot be processed using traditional methods due to issues of volume, velocity, variety, veracity, and value. It discusses characteristics of big data like volume (scale), velocity (speed of data production), and variety (different data formats). The document also outlines different data types, processing methods like batch and stream, common big data architectures, and popular tools used for big data like Hadoop, Spark, and Kafka. In closing, it emphasizes that big data deals with large-scale data processing and highlights some takeaways.
How to get started in Big Data for master's students
For interested students, or to some extent, for general curious persons, here's an experience-based guidelines about the topic of Big Data.
Ipc clipboard and data copy
This document describes two techniques for inter-process communication (IPC) in Windows: using the clipboard and using data copy. It explains how to share data between processes using the Windows clipboard API functions like OpenClipboard(), SetClipboardData(), and CloseClipboard(). It also explains how to send data between processes using the WM_COPYDATA message, a COPYDATASTRUCT structure, and the SendMessage() API function. Both techniques allow sharing data between processes on the same computer.
Hadoop @ eBuddy
The document discusses eBuddy's use of Hadoop for data warehousing and analytics needs. It describes eBuddy's logging history and current large volume of login records. It outlines the needs of different teams for analysis. It then reviews eBuddy's timeline of using traditional RDBMS, custom MapReduce code, Pig scripts and migrating to Hive for SQL queries. The document discusses their current use of Hive, ETL processes, future plans to use Cassandra and streaming ETL to Flume and questions about Hive partition pruning and left outer joins in Pig.
Big Data Streams Architectures. Why? What? How?
With a current zoo of technologies and different ways of their interaction it's a big challenge to architect a system (or adopt existed one) that will conform to low-latency BigData analysis requirements. Apache Kafka and Kappa Architecture in particular take more and more attention over classic Hadoop-centric technologies stack. New Consumer API put significant boost in this direction. Microservices-based streaming processing and new Kafka Streams tend to be a synergy in BigData world.
Exploring Large Chemical Data Sets
This document describes ChemData, an open-source tool for interactive analysis and visualization of large chemical datasets. ChemData stores chemical data in a MongoDB database and uses VTK for visualization. It allows users to perform queries, similarity searching, and multidimensional analysis through features like scatter plots, histograms, parallel coordinates, and k-means clustering. ChemData also integrates with other open chemistry tools like Avogadro and represents molecular structures using the ChemicalJSON format. Future directions include integration with additional databases and computational job results.
Introduction to Big Data
The document discusses big data, defining it as very large data sets with high volumes, velocities, and varieties that are complex to process. It notes trends like Moore's Law, Kryder's Law, and Butter's Law that support big data growth by increasing processing and storage capabilities while decreasing costs. Examples of big data include NASA climate simulations, Large Hadron Collider data, and Walmart customer data. Tools for working with big data are discussed, including Hadoop, NoSQL databases, and MPP databases. The future is predicted to include continued data growth, lower storage and processing costs, higher network capacities, and potential for data growth to exceed processing abilities.
Mobicents Summit 2012 - Alexandre Mendonca - Mobicents jDiameter
This document summarizes a presentation on Mobicents Diameter. It provides an overview of Diameter basics and architecture, including support for high availability and fault tolerance. It discusses past achievements from 2011-2012, including supported applications and stability improvements. Future goals for 2012-2013 are outlined, such as improving documentation, examples, and out-of-box experience. Specific upcoming releases through 1.7.0 are also summarized.
Data Management in Cloud Platforms
This document discusses data management in cloud platforms. It begins with an introduction to cloud computing, noting its benefits like reduced costs and ability to scale resources. It then covers cloud characteristics like elasticity and security risks. The document discusses data analysis challenges in clouds like fault tolerance and heterogeneous environments. It focuses on data replication techniques between master and slave nodes and different replication types. It also covers master-slave election processes to select a primary node and considers factors like priority, network partitions, and quick detection of a failed primary.
Geo data analytics
This document summarizes a presentation on geo data analytics. It discusses why geo data matters, common data formats and libraries for working with spatial data, challenges of working with spatial data at scale, and solutions including dimension reduction techniques and spatial databases. It also provides tips for working with spatial data in tools like Spark, R, and Javascript libraries.
Data Wrangling
Challenges faced during extracting data from websites and tools to do the extraction. Presentation by Manish Dangol
Big Data Lakes Benchmarking 2018
This document summarizes benchmarks for accessing data stored in different formats in a data lake. Scanning 1.2GB of text data stored in JSON, CSV, or Parquet formats was fastest when stored as Parquet, at 8.6 seconds. Retrieving a single row was also fastest from Parquet, at 1.75 seconds, because the Parquet format allows limiting the data read through predicate pushdown. The document concludes that Parquet is the best format even for row-oriented data when accessed through a distributed system like a data lake.
TechEvent Time Seriesd Databases
Time series databases are optimized for storing time-stamped sensor or event data in regular intervals, with little change to stored values. They differ from relational databases by using column-oriented storage, indexing time as a key, and supporting high write volumes, automatic data compaction, and continuous queries over aggregated time series data. Examples include InfluxDB, which uses a time-structured merge tree to optimize storage of billions of data points within a low disk footprint. While time series databases excel at certain use cases, relational databases may still be better for other types of complex, non-time-based data.
polystore_NYC_inrae_sysinfo2021-1.pdf
- The document discusses query optimization in polystore systems using New York City open data as a proof of concept.
- It outlines the NYC polystore implementation including workload characteristics, ETL workflows, examples of ad-hoc queries, and materialized views across different data stores.
- Future work discussed includes performance evaluation of ad-hoc queries versus queries on materialized data, and polystore data warehouse maintenance.
- Related work covers topics like polystore architectures, OLAP on different data models, and benchmarking of polystore systems.
Pass Elk: CAP Theorem since 90s and Beyond
This talk is about CAP theorem since its inception in 90s, and the PACELC theorem which followed it, and the present.
Ruby,no sql and tokyocabinet
This document discusses the transition from relational databases to NoSQL databases. It provides an overview of key-value databases like Tokyo Cabinet and how they provide high performance and scalability alternatives to SQL databases. It also describes how Tokyo Cabinet can be used with an ActiveRecord adapter to integrate NoSQL style databases into Rails applications, bridging the gap between SQL and NoSQL. Finally, it outlines a vision for further developing Tokyo Cabinet into a cloud database that can provide a unified interface to both SQL and NoSQL databases.
Week4
This document provides an overview of week 4 topics for a Code Club on wrangling data with Python. The key points covered are:
- Merging data from different data frames and working with "tidy data" formats.
- Taking first steps in reshaping datasets, including long to wide data transforms and pivots.
- Various strategies for cleaning datasets, such as handling empty/duplicate rows and columns, cleaning text strings, and mapping values.
- The melt and pivot functions for reshaping data between wide and long formats.
What's hot (19)
NoSQL document oriented data access for .net systems with postgresql and marten
NoSQL document oriented data access for .net systems with postgresql and marten
Relational is the new Big Data by Miguel Ángel Fajardo and Daniel Dominguez a...
Relational is the new Big Data by Miguel Ángel Fajardo and Daniel Dominguez a...
How to get started in Big Data for master's students
How to get started in Big Data for master's students
Mobicents Summit 2012 - Alexandre Mendonca - Mobicents jDiameter
Mobicents Summit 2012 - Alexandre Mendonca - Mobicents jDiameter
Similar to Big data uservices
Quick dive into the big data pool without drowning - Demi Ben-Ari @ Panorays
Everybody wants to go on the “Big Data” hype cycle, “To do Scale”, to use the coolest tools in the market like Hadoop, Apache Spark, Apache Cassandra, etc.
But do they ask themselves is there really a reason for that?
In the talk we’ll make a brief overview to all of the technologies in the Big Data world nowadays and we’ll talk about the problems that really emerge when you’d like to enter the great world of Big Data handling.
Showing you the Hadoop ecosystem and Apache Spark and all of the distributed tools leading the market today, will give you all a notion of what will be the real costs entering that world.
Promise that I’ll share some stories from the trenches :)
(And about the “pool” thing...I don’t really know how to swim)
Data platform architecture principles - ieee infrastructure 2020
This document discusses principles for building a healthy data platform, including:
1. Establishing explicit contracts between teams to define dependencies and service level agreements.
2. Abstracting the data platform into services for ingesting, storing, and processing data in motion and at rest.
3. Enabling observability of data pipelines through metadata collection and integration with tools like Marquez to provide lineage, availability, and change management visibility.
AWS Big Data Demystified #1.2 | Big Data architecture lessons learned
A while ago I entered the challenging world of Big Data. As an engineer, at first, I was not so impressed with this field. As time went by, I realised more and more, The technological challenges in this area are too great to master by one person. Just look at the picture in this articles, it only covers a small fraction of the technologies in the Big Data industry…
Consequently, I created a meetup detailing all the challenges of Big Data, especially in the world of cloud. I am using AWS & GCP and Data Center infrastructure to answer the basic questions of anyone starting their way in the big data world.
how to transform data (TXT, CSV, TSV, JSON) into Parquet, ORC,AVRO which technology should we use to model the data ? EMR? Athena? Redshift? Spectrum? Glue? Spark? SparkSQL? GCS? Big Query? Data flow? Data Lab? tensor flow? how to handle streaming? how to manage costs? Performance tips? Security tip? Cloud best practices tips?
In this meetup we shall present lecturers working on several cloud vendors, various big data platforms such hadoop, Data warehourses , startups working on big data products. basically - if it is related to big data - this is THE meetup.
Some of our online materials (mixed content from several cloud vendor):
Website:
https://big-data-demystified.ninja (under construction)
Meetups:
https://www.meetup.com/Big-Data-Demystified
https://www.meetup.com/AWS-Big-Data-Demystified/
You tube channels:
https://www.youtube.com/channel/UCMSdNB0fGmX5dXI7S7Y_LFA?view_as=subscriber
https://www.youtube.com/channel/UCzeGqhZIWU-hIDczWa8GtgQ?view_as=subscriber
Audience:
Data Engineers
Data Science
DevOps Engineers
Big Data Architects
Solution Architects
CTO
VP R&D
AWS big-data-demystified #1.1 | Big Data Architecture Lessons Learned | English
Aws big-data-demystified #1.1 | Big Data Architecture Lessons Learned | spoken Language: English. AWS. Big Data. Architecture. Hadoop. Open Source. EMR.
Distributed Databases - Concepts & Architectures
Concepts, architectures and uses of distributed databases. A gentle introduction to get you up to speed and understand the value and potential of distributed databases.
Hadoop Training Tutorial for Freshers
TIB Academy is one of the best Hadoop Training Institute in bangalore. We Offers Hands-On Training with Live Projects.
Intro to dbms
This document discusses database management systems (DBMS). A DBMS is software that allows for the storage, management, and retrieval of large amounts of data. It provides benefits like data independence, concurrency control, crash recovery, and security. A DBMS typically uses a multi-layer architecture and implements concepts like transactions, locking, and logging to ensure data integrity and consistency when multiple users access the database concurrently. Database administrators are responsible for designing database schemas and tuning the system to meet evolving needs.
Data Science as Scale
The document provides an introduction to data science at scale and distributed thinking. It discusses the motivation for data science at scale due to increasing data volumes, varieties, and velocities. It distinguishes between data science, which focuses on accuracy, and data engineering, which focuses on scale, performance, and reliability. The document then provides a crash course on data engineering concepts like distributed computation and the SMACK stack. It introduces Spark as a framework that can scale data processing. Finally, it discusses probabilistic algorithms as an approach for processing large datasets that may be inexact but use less resources than exact algorithms.
MongoDB World 2019: Packing Up Your Data and Moving to MongoDB Atlas
Moving to a new home is daunting. Packing up all your things, getting a vehicle to move it all, unpacking it, updating your mailing address, and making sure you did not leave anything behind. Well, the move to MongoDB Atlas is similar, but all the logistics are already figured out for you by MongoDB.
Introduction to Big Data Technologies & Applications
Big Data Myths, Current Mainstream Technologies related to Collecting, Storing, Computing & Stream Processing Data. Real-life experience with E-commerce businesses.
Big Data in 200 km/h | AWS Big Data Demystified #1.3
What we're about
A while ago I entered the challenging world of Big Data. As an engineer, at first, I was not so impressed with this field. As time went by, I realised more and more, The technological challenges in this area are too great to master by one person. Just look at the picture in this articles, it only covers a small fraction of the technologies in the Big Data industry…
Consequently, I created a meetup detailing all the challenges of Big Data, especially in the world of cloud. I am using AWS infrastructure to answer the basic questions of anyone starting their way in the big data world.
how to transform data (TXT, CSV, TSV, JSON) into Parquet, ORCwhich technology should we use to model the data ? EMR? Athena? Redshift? Spectrum? Glue? Spark? SparkSQL?how to handle streaming?how to manage costs?Performance tips?Security tip?Cloud best practices tips?
Some of our online materials:
Website:
https://big-data-demystified.ninja/
Youtube channels:
https://www.youtube.com/channel/UCzeGqhZIWU-hIDczWa8GtgQ?view_as=subscriber
https://www.youtube.com/channel/UCMSdNB0fGmX5dXI7S7Y_LFA?view_as=subscriber
Meetup:
https://www.meetup.com/AWS-Big-Data-Demystified/
https://www.meetup.com/Big-Data-Demystified
Facebook Group :
https://www.facebook.com/groups/amazon.aws.big.data.demystified/
Facebook page (https://www.facebook.com/Amazon-AWS-Big-Data-Demystified-1832900280345700/)
Audience:
Data Engineers
Data Science
DevOps Engineers
Big Data Architects
Solution Architects
CTO
VP R&D
Apache Storm Concepts
Storm is an open source distributed real-time computation system. It allows processing large amounts of streaming data reliably and fault-tolerantly. Storm concepts include tuples (data records), streams, spouts (data sources), bolts (processing units), topologies (processing graphs), and stream groupings (data routing). Storm provides reliable processing through its master-slave architecture with a Nimbus master node and supervisor worker nodes. It integrates with other technologies like Hadoop and is used in real cases like collecting sensor data, processing user feeds, and generating recommendations.
Big Data Streaming processing using Apache Storm - FOSSCOMM 2016
Our presentation on FOSSCOMM conference (17 April 2016):
Agenda:
* Big Data concepts
* Batch & Streaming processing
* NoSQL persistence
* Apache Storm and Apache Kafka
* Streaming application demo
* Considerations for Big Data applications
Event: http://fosscomm.cs.unipi.gr/index.php/event/adrianos-dadis/?lang=en
AWS Big Data Demystified #1: Big data architecture lessons learned
AWS Big Data Demystified #1: Big data architecture lessons learned . a quick overview of a big data techonoligies, which were selected and disregard in our company
The video: https://youtu.be/l5KmaZNQxaU
dont forget to subcribe to the youtube channel
The website: https://amazon-aws-big-data-demystified.ninja/
The meetup : https://www.meetup.com/AWS-Big-Data-Demystified/
The facebook group : https://www.facebook.com/Amazon-AWS-Big-Data-Demystified-1832900280345700/
Py tables
This document discusses using PyTables to analyze large datasets. PyTables is built on HDF5 and uses NumPy to provide an object-oriented interface for efficiently browsing, processing, and querying very large amounts of data. It addresses the problem of CPU starvation by utilizing techniques like caching, compression, and high performance libraries like Numexpr and Blosc to minimize data transfer times. PyTables allows fast querying of data through flexible iterators and indexing to facilitate extracting important information from large datasets.
Large Data Analyze With PyTables
This document discusses PyTables, a Python library for managing hierarchical datasets and efficiently analyzing large amounts of data. It begins by introducing PyTables and its use of HDF5 for portability and extensibility. Key features of PyTables discussed include its object-oriented interface, optimization of memory and disk usage, and fast querying capabilities. The document then covers techniques for maximizing performance like Numexpr for complex expressions, NumPy for powerful data containers, compression algorithms, and caching. Blosc compression is highlighted for its ability to compress faster than memory speed.
PyTables
This document discusses using PyTables to analyze large datasets. PyTables is built on HDF5 and uses NumPy to provide an object-oriented interface for efficiently browsing, processing, and querying very large amounts of data. It addresses the problem of CPU starvation by utilizing techniques like caching, compression, and high performance libraries like Numexpr and Blosc to minimize data transfer times. PyTables allows fast querying of data through flexible iterators and indexing to facilitate extracting important information from large datasets.
A primer on building real time data-driven products
This document provides an overview of building real-time data products using stream processing. It discusses why stream processing is useful for providing low-latency reactions to data from 1 second to 1 hour. Key aspects covered include using a unified log to decouple producers and consumers, common stream processing building blocks like filtering and joining, and technologies like Spark Streaming, Kafka Streams, and Flink. The document also addresses challenges like out-of-order events and software bugs, and architectural patterns for handling imperfections in streams.
BigData Hadoop
This document summarizes a summer training seminar on BigData Hadoop that was attended. The training was provided by LinuxWorld Informatics Pvt Ltd, which offers open source and commercial training programs. The attendee learned about Hadoop, MapReduce, single and multi-node clusters, Docker, and Ansible. Big data challenges related to volume, variety, velocity, and veracity of data were also covered. Hadoop and its core components HDFS and MapReduce were explained as solutions for storing and processing large datasets in a distributed manner across commodity hardware. Docker containers were introduced as a lightweight alternative to virtual machines.
The “obsession” with checksums by Helen Hockx-Yu
Checksums are used to verify the integrity of digital files and objects by determining if they have remained fixed or unchanged over time. Checksum-based fixity checking workflows are commonly used by libraries and archives to preserve digital assets. While bit flips are not a major cause of data loss, risks like human error, viruses, bugs in software, and malicious actors threaten the long-term preservation of digital content. There is a debate around whether checksumming all files is necessary for preservation or if storage services that provide hashes can be trusted instead.
Similar to Big data uservices (20)
Quick dive into the big data pool without drowning - Demi Ben-Ari @ Panorays
Quick dive into the big data pool without drowning - Demi Ben-Ari @ Panorays
Data platform architecture principles - ieee infrastructure 2020
Data platform architecture principles - ieee infrastructure 2020
AWS Big Data Demystified #1.2 | Big Data architecture lessons learned
AWS Big Data Demystified #1.2 | Big Data architecture lessons learned
AWS big-data-demystified #1.1 | Big Data Architecture Lessons Learned | English
AWS big-data-demystified #1.1 | Big Data Architecture Lessons Learned | English
MongoDB World 2019: Packing Up Your Data and Moving to MongoDB Atlas
MongoDB World 2019: Packing Up Your Data and Moving to MongoDB Atlas
Introduction to Big Data Technologies & Applications
Introduction to Big Data Technologies & Applications
Big Data in 200 km/h | AWS Big Data Demystified #1.3
Big Data in 200 km/h | AWS Big Data Demystified #1.3
Big Data Streaming processing using Apache Storm - FOSSCOMM 2016
Big Data Streaming processing using Apache Storm - FOSSCOMM 2016
AWS Big Data Demystified #1: Big data architecture lessons learned
AWS Big Data Demystified #1: Big data architecture lessons learned
A primer on building real time data-driven products
A primer on building real time data-driven products
More from Felix Crisan
Bitcoin:Next
What are the next changes that we might see in the near future coming to Bitcoin? Improvement in the signature algorithms and scripts leading to better privacy and size optimizations
BigData in BlockChains
Blockchains provide a fertile ground for storing and managing big data in a decentralized manner. Some examples of using blockchains for big data include Numer.ai for storing machine learning data shapes, IPFS and FileCoin for distributed storage, and BigchainDB as a distributed database with blockchain characteristics. Blockchains allow users to store and control their own data while granting view rights to interested and trusted parties, making it even GDPR compliant by keeping users as the data controllers.
Lightning Network
The document discusses the Lightning Network, which aims to scale bitcoin transactions by enabling instant, low-cost payments through off-chain payment channels. It describes key concepts like payment channels, routing payments across multiple nodes, and implementations. The Lightning Network allows for near-instant micropayments, smart contracts, and cross-chain atomic swaps by using timelocks and hashed timelock contracts to securely transfer bitcoin off the main blockchain.
Smart contracts using web3.js
Smart contracts can be deployed and executed on the Ethereum blockchain using web3.js. Web3.js is a JavaScript API that allows interaction with Ethereum clients like Geth to deploy contracts, call contract functions, and get information from the blockchain. Contracts are written in languages like Solidity and deployed through transactions that store the compiled code at an address. Once deployed, the contract code and data resides on the blockchain and can be interacted with through web3.js by calling functions or accessing public variables.
Smart contracts in Solidity
Smart contracts are computer protocols that facilitate, verify, or enforce the negotiation or performance of a contract. On Ethereum, a smart contract is code and data that resides at a specific address on the blockchain. The Ethereum Virtual Machine is the runtime for smart contracts, which can call other contracts, have storage to store data, and can remove themselves from the blockchain. Smart contracts are defined through transactions and deployed to the blockchain, then users can interact with them through their ABI interface. Developers use languages like Solidity to code smart contracts and tools like Truffle to test contracts locally before deployment.
Mashing the data
A proposal for replicating relational databases to non-relational ones. We choose Google Cloud Datastore, but could as well use MongoDB
Big(data) in block(chains)
This document provides an overview of Bitcoin and blockchain technology. It describes how Bitcoin was created in 2008, functioning as a transmission protocol, distributed ledger, and stored value account without the need for settlement. Transactions are recorded on a blockchain, with a new block added every 10 minutes. The blockchain records transaction hashes in blocks linked by cryptographic hashes. Analyzing Bitcoin transaction data can provide useful statistics about usage trends, price movements, and detect spam attacks.
Enablers for o commerce
oCommerce is omniCommerce. If we talk about omni channel why do we keep talking about mCommerce and eCommerce?
mcommad
MCOMMAD discusses blending advertising and mobile commerce (m-commerce). It provides an overview of NETOPIA, the largest SMS aggregator and payment gateway in Romania, and their mobilPay mobile wallet. The document notes that while mobile advertising currently makes up around 5% of global ad spend, mobile purchases account for 35-40% of all purchases. It introduces the mobilPay wallet as not just a payment tool but a marketing platform, allowing clear tracking of transactions resulting from marketing actions. Features mentioned include Push2Pay notifications, offline in-store functionality, and QR/BLE/NFC codes to enable online and offline interactions.
NoSQL solutions
MongoDB and Redis are popular NoSQL alternatives to SQL databases. MongoDB is a document-oriented database that does not require a predefined schema and allows embedding documents. It supports features like sharding, replication, and indexing. Redis is an in-memory key-value store that persists data to disk. It supports data structures like strings, hashes, lists and sets. Both databases are commonly used for caching, queues, and other use cases where flexibility and performance are important.
Deconstructing Lambda architectures
The document discusses the lambda architecture, which is designed to handle large volumes of data and queries over that data. It describes the three main layers - batch, speed, and serving layers - that make up the lambda architecture. The batch layer handles batch processing and generates a master dataset. The speed layer handles real-time queries and compensates for latency in the batch layer. The serving layer indexes and exposes views of the data to support ad-hoc queries.
402 @ Mobile next
This document discusses the history and current state of Bitcoin and blockchain technology. It provides an overview of Bitcoin's origins with Satoshi Nakamoto's 2008 whitepaper, the genesis block from 2009 referencing the financial crisis, and prior related work. Benefits of Bitcoin include anonymity, transparency, security, and speed of value transfer. Use cases mentioned include escrow, smart contracts, issuing securities, and digital notaries. Challenges integrating with traditional finance are noted. The document concludes with background on the author's payments company in Romania.
Presentation for the first Bucharest Big data meetup
RT Analytics provides real-time analytics and insights for their mobile payment and SMS aggregation businesses in Romania. They use a data streaming platform with Storm for scalable stream processing, RabbitMQ for queuing transaction data, and Redis for fast in-memory storage. This allows them to aggregate transaction details in real-time by merchant, status, error, currency and time interval. The setup provides operational insights, transaction monitoring, and business dashboards to gain better understanding of queries, insights, and predictions to improve their services.
Data analysis with Pandas and Spark
This document discusses using Jupyter notebooks, Pandas, and Spark for analytics pipelines on both small and large datasets. It summarizes the challenges of working with different data volumes and timeframes. For small mobile transaction data, notebooks with Pandas and R are used, while larger retail data is analyzed with Spark ML and scikit-learn in notebooks running in Docker containers. Future work includes applying Spark to additional domains and building forecasting and streaming capabilities.
TCP/IP of money
This document discusses the history and potential of blockchain technology. It traces the origins of TCP/IP and Bitcoin, comparing Bitcoin to TCP/IP for the internet. Blockchain is described as a transmission protocol, distributed ledger, and stored value account. The document outlines increasing corporate investment and experimentation in blockchain applications across banking, exchanges, infrastructure, and other areas. It argues that blockchain will fundamentally change the financial industry in the same way TCP/IP transformed communication, and that "blockchain is the TCP/IP of money."
More from Felix Crisan (15)
Presentation for the first Bucharest Big data meetup
Presentation for the first Bucharest Big data meetup
Recently uploaded
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Discover the seamless integration of RPA (Robotic Process Automation), COMPOSER, and APM with AWS IDP enhanced with Slack notifications. Explore how these technologies converge to streamline workflows, optimize performance, and ensure secure access, all while leveraging the power of AWS IDP and real-time communication via Slack notifications.
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
20 Comprehensive Checklist of Designing and Developing a Website
Dive into the world of Website Designing and Developing with Pixlogix! Looking to create a stunning online presence? Look no further! Our comprehensive checklist covers everything you need to know to craft a website that stands out. From user-friendly design to seamless functionality, we've got you covered. Don't miss out on this invaluable resource! Check out our checklist now at Pixlogix and start your journey towards a captivating online presence today.
Artificial Intelligence for XMLDevelopment
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
Microsoft - Power Platform_G.Aspiotis.pdf
Revolutionizing Application Development
with AI-powered low-code, presentation by George Aspiotis, Sr. Partner Development Manager, Microsoft
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank
Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIVladimir Iglovikov, Ph.D.
Presented by Vladimir Iglovikov:
- https://www.linkedin.com/in/iglovikov/
- https://x.com/viglovikov
- https://www.instagram.com/ternaus/
This presentation delves into the journey of Albumentations.ai, a highly successful open-source library for data augmentation.
Created out of a necessity for superior performance in Kaggle competitions, Albumentations has grown to become a widely used tool among data scientists and machine learning practitioners.
This case study covers various aspects, including:
People: The contributors and community that have supported Albumentations.
Metrics: The success indicators such as downloads, daily active users, GitHub stars, and financial contributions.
Challenges: The hurdles in monetizing open-source projects and measuring user engagement.
Development Practices: Best practices for creating, maintaining, and scaling open-source libraries, including code hygiene, CI/CD, and fast iteration.
Community Building: Strategies for making adoption easy, iterating quickly, and fostering a vibrant, engaged community.
Marketing: Both online and offline marketing tactics, focusing on real, impactful interactions and collaborations.
Mental Health: Maintaining balance and not feeling pressured by user demands.
Key insights include the importance of automation, making the adoption process seamless, and leveraging offline interactions for marketing. The presentation also emphasizes the need for continuous small improvements and building a friendly, inclusive community that contributes to the project's growth.
Vladimir Iglovikov brings his extensive experience as a Kaggle Grandmaster, ex-Staff ML Engineer at Lyft, sharing valuable lessons and practical advice for anyone looking to enhance the adoption of their open-source projects.
Explore more about Albumentations and join the community at:
GitHub: https://github.com/albumentations-team/albumentations
Website: https://albumentations.ai/
LinkedIn: https://www.linkedin.com/company/100504475
Twitter: https://x.com/albumentations“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...Edge AI and Vision Alliance
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/building-and-scaling-ai-applications-with-the-nx-ai-manager-a-presentation-from-network-optix/
Robin van Emden, Senior Director of Data Science at Network Optix, presents the “Building and Scaling AI Applications with the Nx AI Manager,” tutorial at the May 2024 Embedded Vision Summit.
In this presentation, van Emden covers the basics of scaling edge AI solutions using the Nx tool kit. He emphasizes the process of developing AI models and deploying them globally. He also showcases the conversion of AI models and the creation of effective edge AI pipelines, with a focus on pre-processing, model conversion, selecting the appropriate inference engine for the target hardware and post-processing.
van Emden shows how Nx can simplify the developer’s life and facilitate a rapid transition from concept to production-ready applications.He provides valuable insights into developing scalable and efficient edge AI solutions, with a strong focus on practical implementation.GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
Neha Bajwa, Vice President of Product Marketing, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
Removing Uninteresting Bytes in Software Fuzzing
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
Climate Impact of Software Testing at Nordic Testing Days
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
UiPath Test Automation using UiPath Test Suite series, part 6
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
Maruthi Prithivirajan, Head of ASEAN & IN Solution Architecture, Neo4j
Get an inside look at the latest Neo4j innovations that enable relationship-driven intelligence at scale. Learn more about the newest cloud integrations and product enhancements that make Neo4j an essential choice for developers building apps with interconnected data and generative AI.
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
Mind map of terminologies used in context of Generative AI
Mind map of common terms used in context of Generative AI.
How to Get CNIC Information System with Paksim Ga.pptx
Pakdata Cf is a groundbreaking system designed to streamline and facilitate access to CNIC information. This innovative platform leverages advanced technology to provide users with efficient and secure access to their CNIC details.
20240607 QFM018 Elixir Reading List May 2024
Everything I found interesting about the Elixir programming ecosystem in May 2024
A tale of scale & speed: How the US Navy is enabling software delivery from l...
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
Recently uploaded (20)
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...
Big data uservices
- 2. whoami ● NETOPIA Payments ○ Surprisingly (or not) we do other things than payments ● mobilPay Wallet ● Elrond ● Blockchain Romania
- 3. The (physical) universe consists of a 3 x Space + 1 x Time manifold
- 4. The computational universe consists of 1 x Computing + 1 x Storage + 1 x Time manifold
- 5. In the computational universe Computing and Storage are bound together by Code
- 6. These days Code moves/runs further and further away* from Computing
- 7. Id est ● Hardware has morphed into virtual machines ● VMs have morphed into containers ● Containers are morphing into functions
- 8. At the same time ● When talking about “pure” data - the closer to the hardware the faster it can be accessed - and with smaller latency
- 9. Id est ● Files have morphed into databases ● Databases have morphed into distributed databases ● Distributed DBs have morphed into ○ distributed Ledgers and Blockchains ○ also into Data Lakes and Deltas (not sure yet about Seas and Oceans)
- 10. Aside: Big(ger) Data Structures OLAP OLTP Homogenous Data Lake, Delta, Sea Distributed Ledger* Heterogeneous Blockchain
- 11. Empirical observation Storage Increases in scope Computing Decreases in scope
- 12. Guesstimation ● This happens because data is “inert” but ● The smaller the unit of code that needs to run, the more controllable the output/behavior (we can even formally verify some - if not all - of it)
- 13. Microservices unify interaction (even if data is also involved)
- 15. Destructuring Monoliths ● 3 TA with distributed application layer and/or data layer ● Request driven (pull) vs Event driven (push) ● Messaging - Event Sourcing ● Sagas w/ or w/o orchestrators
- 16. Microservice Moving away from Monoliths Monolithic Application Computing Storage Storage Computing Microservice Storage Computing
- 17. Option 1 With this approach ● Big Data = Σ Small(ish) data ● Data consistency (say, transactions) is hard
- 18. Microservice Another option MicroserviceMonolithic Application Computing Storage Storage Computing Microservice Computing
- 19. Option 2 ● Storage = House of Big Data ● Does not work all the time. It might even be an anti-pattern.
- 20. Modus Operandi ● API Gateway / Service Mesh ● Service Registry These tend to handle only the computing side
- 21. Data side ● Additional concerns: “at rest” behavior vs “in flight” behavior ● Schemas => APIs ● Stateful serverless?
- 22. DataOps ● It’s not just DevOps for Data ● Intersection of Computing and Storage w/ Organization ○ Human Resource 4-dimensions ■ Wants ■ Knows ■ Time / Availability ■ Privileges
- 23. Storage and Computing are just as important as metadata (including processes, people ...)
- 24. Conway’s law Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure Melvin Conway, 1967
- 25. But even if we only talk about technology things are complicated
- 26. CNCF Landscape Img Src https://landscape.cncf.io/
- 30. YMMV: just because you can do it, it doesn’t mean that you should do it