Big Data Landscape 2019
•
6 likes•1,955 views
Die QAware Big Data Landscape gibt einen detaillierten Überblick zu den relevantesten Big Data Technologien, vornehmlich aus dem Open-Source-Ökosystem. (Dokument bitte herunterladen für bessere Lesbarkeit / komplette Ansicht) ----------------- The QAware Big Data Landscape provides a detailed overview over the most relevant Big Data technologies, most of them open source. (Please download file for a better readability / complete view)
Report
Share
Report
Share
Download to read offline
Recommended
Big Data Landscape 2019
Die QAware Big Data Landscape gibt einen detaillierten Überblick zu den relevantesten Big Data Technologien, vornehmlich aus dem Open-Source-Ökosystem.
(Dokument bitte herunterladen für bessere Lesbarkeit / rotierte Ansicht)
-----------------
The QAware Big Data Landscape provides a detailed overview over the most relevant Big Data technologies, most of them open source.
(Please download file for a better readability / rotated view)
Introduction To Spark - Durham LUG 20150916
Apache Spark is a fast and general engine for large-scale data processing. It is a framework for massive parallel computing that harnesses the power of cheap memory. Spark provides high performance through its in-memory computing engine and supports Scala, Java, Python and R. It includes modules for streaming, machine learning, graph processing and SQL. Resilient Distributed Datasets (RDDs) are Spark's basic abstraction, which are immutable and fault-tolerant collections that can be operated on in parallel.
Hadoop Big data Solution Provider
The document discusses data lakes, which are repositories for large amounts of structured and unstructured data from various sources. Data in lakes can be accessed for real-time analytics or discovery and ideation by data scientists. Data lakes take advantage of cheap storage techniques to store files in any format at low cost. The document also describes how "cold" or infrequently used data can be offloaded from a data warehouse to a Hadoop-based data lake for lower storage costs and improved warehouse performance while still allowing the cold data to be queried.
Building Data Lakes with Apache Airflow
Build a simple Data Lake on AWS using a combination of services, including Amazon Managed Workflows for Apache Airflow (Amazon MWAA), AWS Glue, AWS Glue Studio, Amazon Athena, and Amazon S3.
Blog post and link to the video: https://garystafford.medium.com/building-a-data-lake-with-apache-airflow-b48bd953c2b
Data science big data and analytics
This document discusses new analysis skills required for working with big data technologies like NoSQL databases. It provides examples of popular open source NoSQL databases like Cassandra, HBase, MongoDB and Couchbase. It also classifies NoSQL databases into categories like column-oriented, document, key-value and graph databases. The document then discusses how big data solves problems related to volume, velocity, variety and value of data. It provides examples of sources of big data and trends shaping interest in big data. Finally, it discusses use cases of big data in retail and finance industries.
Next Generation Data Platforms - Deon Thomas
A new generation of technologies and architectures designed to economically extract value from very large volumes of a wide variety of data, by enabling high velocity capture, discovery and/or analysis.
The IoT and big data
In the next five years, 15 to 40 billion additional connected devices are expected to hit the market. How can we handle such volumes and velocity of data?
Introduction to Dynamo storage systems, Riak, Cassandra, time series databases and edge analytics.
Piyali Kamra - Analytics and Data Visualization pipeline backed by AWS Glue &...
AWS Community Day Midwest 2020
Piyali Kamra - Analytics and Data Visualization pipeline backed by AWS Glue & Athena
Recommended
Big Data Landscape 2019
Die QAware Big Data Landscape gibt einen detaillierten Überblick zu den relevantesten Big Data Technologien, vornehmlich aus dem Open-Source-Ökosystem.
(Dokument bitte herunterladen für bessere Lesbarkeit / rotierte Ansicht)
-----------------
The QAware Big Data Landscape provides a detailed overview over the most relevant Big Data technologies, most of them open source.
(Please download file for a better readability / rotated view)
Introduction To Spark - Durham LUG 20150916
Apache Spark is a fast and general engine for large-scale data processing. It is a framework for massive parallel computing that harnesses the power of cheap memory. Spark provides high performance through its in-memory computing engine and supports Scala, Java, Python and R. It includes modules for streaming, machine learning, graph processing and SQL. Resilient Distributed Datasets (RDDs) are Spark's basic abstraction, which are immutable and fault-tolerant collections that can be operated on in parallel.
Hadoop Big data Solution Provider
The document discusses data lakes, which are repositories for large amounts of structured and unstructured data from various sources. Data in lakes can be accessed for real-time analytics or discovery and ideation by data scientists. Data lakes take advantage of cheap storage techniques to store files in any format at low cost. The document also describes how "cold" or infrequently used data can be offloaded from a data warehouse to a Hadoop-based data lake for lower storage costs and improved warehouse performance while still allowing the cold data to be queried.
Building Data Lakes with Apache Airflow
Build a simple Data Lake on AWS using a combination of services, including Amazon Managed Workflows for Apache Airflow (Amazon MWAA), AWS Glue, AWS Glue Studio, Amazon Athena, and Amazon S3.
Blog post and link to the video: https://garystafford.medium.com/building-a-data-lake-with-apache-airflow-b48bd953c2b
Data science big data and analytics
This document discusses new analysis skills required for working with big data technologies like NoSQL databases. It provides examples of popular open source NoSQL databases like Cassandra, HBase, MongoDB and Couchbase. It also classifies NoSQL databases into categories like column-oriented, document, key-value and graph databases. The document then discusses how big data solves problems related to volume, velocity, variety and value of data. It provides examples of sources of big data and trends shaping interest in big data. Finally, it discusses use cases of big data in retail and finance industries.
Next Generation Data Platforms - Deon Thomas
A new generation of technologies and architectures designed to economically extract value from very large volumes of a wide variety of data, by enabling high velocity capture, discovery and/or analysis.
The IoT and big data
In the next five years, 15 to 40 billion additional connected devices are expected to hit the market. How can we handle such volumes and velocity of data?
Introduction to Dynamo storage systems, Riak, Cassandra, time series databases and edge analytics.
Piyali Kamra - Analytics and Data Visualization pipeline backed by AWS Glue &...
AWS Community Day Midwest 2020
Piyali Kamra - Analytics and Data Visualization pipeline backed by AWS Glue & Athena
Philly Code Camp 2013 Mark Kromer Big Data with SQL Server
These are my slides from May 2013 Philly Code Camp at Penn State Abington. I will post the samples, code and scripts on my blog here following the event this Saturday: http://www.kromerbigdata.com
Datalab 101 (Hadoop, Spark, ElasticSearch) par Jonathan Winandy - Paris Spark...
Datalab 101 (Hadoop, Spark, ElasticSearch) par Jonathan Winandy - Paris Spark...Modern Data Stack France
Datalab 101 (Hadoop, Spark, ElasticSearch) par Jonathan Winandy
Retour d'expérience sur la mise en place d'un Datalab avec Hadoop, Spark et ElasticSearch dans un environnement contraint. Nous allons exposer les méthodes qui nous ont permis d'améliorer la conception, le développement, les performances et la recette d'une application complexe en Spark.
Jonathan Winandy est MOE, développeur Java/Scala spécialisé dans les pipelines de données.Hugfr SPARK & RIAK -20160114_hug_france
Presentation faite lors du Hadoop User Group France du 14 janvier 2016.
L’analytique temps réel avec Riak et Spark par Michael Carney (Basho) et Olivier Girardot de Lateral Thoughts
Selon un rapport de Salesforce, le nombre de sources de données analysées par les entreprises progressera de 83% au cours des cinq prochaines années, ainsi les organisations veulent désormais fournir des connaissances en temps réel même sur les appareils mobiles. Le traitement temps réel est donc, le futur de l’analyse big data.
Ce talk présentera des nouveautés en matière de l’analyse temps réel autour de la famille SGBD Riak et Spark.
Michael Carney est le Directeur Commercial de Basho pour le Sud d’Europe. Fondateur de MySQL France et de MariaDB, Michael a rejoint Basho en janvier 2015 pour explorer le monde de données sans tables !
Olivier Girardot est le CTO de Lateral Thoughts, il est développeur et formateur au sujet de Spark et également spécialiste de Java/Python dans le domaine de la finance de marché.
Dealing with Drift: Building an Enterprise Data Lake
This document discusses Cox Automotive's use of StreamSets to build an enterprise data lake. It describes the challenges of acquiring and ingesting data from 25 companies with different source types and schemas at scale. Cox Automotive addressed this by using StreamSets to separate the acquisition of data from its various source systems from the ingestion into the data lake. StreamSets pipelines automatically apply data standards and load balancing during ingestion. This architecture has allowed Cox Automotive to significantly increase its monthly ingestion volumes.
Minería de Datos en Sql Server 2008
Inteligencia de Negocios en SQL Server 2008 y Minería de Datos.
Ing. Eduardo Castro Martinez, PhD
Microsoft SQL Server MVP
http://ecastrom.blogspot.com
http://comunidadwindows.org
Dealing With Drift - Building an Enterprise Data Lake
Data drift, the gradual morphing of data structure and semantics, is a fact of life in enterprise IT. New requirements force schema changes, the meaning of database columns changes over time, and infrastructure upgrades add new fields to log files. Left unchecked, drift in data sources can cause applications and dataflows to fail, with costly downtime and, in the worst case, corruption in downstream data stores.
Cox Automotive comprises more than 25 companies dealing with different aspects of the car ownership lifecycle, with data as the common language they all share. The challenge for Cox was to create an efficient engine for the timely and trustworthy ingest of data capability for an unknown but large number of data assets from practically any source. Discover how their big data engineering team overcame data drift and are now populating a data lake, allowing analysts easy access to data from their subsidiary companies and producing new data assets unique to the industry.
Dirty Data? Clean it up! - Rocky Mountain DataCon 2016
Dan Lynn and Patrick Russell discuss data cleaning, data engineering, and how to operationalize data science at Rocky Mountain DataCon 2016.
Big Data LDN 2016: Out of the Data Warehouses, and into the Data Lakes and St...
This document discusses using data streams and lakes for big data analytics in utilities. It begins by explaining how smart meter rollouts have increased meter reads from 80 million to 350 billion per year. Traditional data warehousing is challenged by the time criticality, resource demands, and scale of this unbounded data. The document then introduces data streams for real-time analytics and data lakes for flexible storage of huge amounts of structured, semi-structured, and unstructured data. It describes Valo, an open lambda architecture that uses multiple repositories and stream processing to enable both real-time and historical analysis across disparate data sources.
Open source big data landscape and possible ITS applications
What is big data, and how open-source big data projects, such as Apache Spark, Kafka and Cassandra can be used in ITS (Intelligent Transport Systems) related projects.
PhD Projects in Cloudsim Research Assistance
Innovative Ideas in Cloudsim Projects
Exciting Fields on Cloudsim Projects
Renewed Subjects in Cloudsim Projects
Database Week San Francisco: Database Services at AWS
AWS provides a broad portfolio of databases and analytics services including data lakes, data movement services, non-relational databases like DynamoDB and ElastiCache, relational databases like RDS and Aurora, analytics services like Redshift and EMR, and machine learning services like SageMaker and Comprehend. These services are purpose-built to help customers build applications and analyze data.
AWS and Analytics Services
Short overview of AWS Database and Analytics offerings and an introduction of the day’s topics.
Speaker: Bill Baldwin - Database Technical Evangelist, AWS
HUG France - Paris - Data Engineer's Toolkit
1. The document discusses data integration challenges and how different technologies like Apache Beam, Spark, Flink, and Google Dataflow can be used to solve them.
2. It also covers data processing engines like Hadoop 2.0 including YARN, MapReduce, Pig, and Hive as well as streaming technologies like Apache Kafka.
3. The role of the data engineer is highlighted as being responsible for choosing the correct infrastructure to address data integration needs.
IaaS, PaaS, and DevOps for Data Scientist
This document discusses Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and DevOps for data science. It provides an overview of Azure services for IaaS including virtual machines and GPU instances. It also discusses PaaS options like Azure Machine Learning for deploying models as web services. The document advocates for using Azure services like HDInsight, Data Factory and Machine Learning to build distributed and scalable data science systems following architectures like lambda architecture. It highlights pros and cons of different approaches for flexibility, scalability and using open source tools for data science workloads on Azure.
GDBinSV_Meetup_DBMS_Trends_10062016
The DBMS market trends focused on the Graph DBMS. The benefit of the Graph Database and its forecasted the growth rate. The Advice from the renowned market research institute.
Building a data pipeline to ingest data into Hadoop in minutes using Streamse...
Slides from my talk at the Hadoop User Group Ireland meetup on June 13th 2016: building a data pipeline to ingest data from sources of different nature into Hadoop in minutes (and no coding at all) using the Open Source Streamsets Data Collector tool.
Solution architecture for big data projects
Solution architecture for big data projects
solution architecture,big data,hadoop,hive,hbase,impala,spark,apache,cassandra,SAP HANA,Cognos big insights
Big Data - Linked In_DEEPU
The document discusses big data technologies including Apache HBase, Hive, Hadoop, MapReduce, Spring XD, Sqoop, Spark, Kafka, Solr, Zookeeper, and Oozie. It also outlines several proof of concept projects involving importing structured and unstructured data using these technologies, real-time ingestion of data from databases into HDFS, Hive and HBase integration, indexing and searching large datasets with Solr, monitoring and charting big data growth, data compression and encryption, job scheduling with Oozie, and implementing machine learning algorithms with Spark MLib.
Alexander Pavlenko, Java Software Engineer, DataArt.
This document discusses building a Spark connector for Ryft, a hardware appliance that performs high-speed compute on big data. It provides examples of how to query data stored on Ryft using its query language. The Spark connector allows querying and processing Ryft data using Spark and RDDs in a distributed manner. It maps structured Ryft data to DataFrames for SQL queries and shows performance benefits of using Ryft compared to running Spark on EC2 servers.
Big Data: Improving capacity utilization of transport companies
Asparuh Koev is a successful serial entrepreneur. Currently, he is a CEO of Transmetrics, a solution for cargo transport companies that uses Big Data and predictive analytics. Asparuh holds a Bachelor’s Degree in Computer Science from the American University in Bulgaria and an MBA degree from the Vlerick Business School in Belgium.
“Big Data: Improving capacity utilization of transport companies” will explore the practical benefits, IT tools and challenges of implementing a big data solution in a traditional industry (cargo transport), using as a showcase a predictive analytics project done for Speedy.
AWS Big Data Landscape
This document provides an overview of big data concepts and architectures, as well as AWS big data services. It begins with introducing big data challenges around variety, volume, and velocity of data. It then covers the Hadoop ecosystem including HDFS, MapReduce, Hive, Pig and Spark. The document also discusses data lake architectures and how AWS services like S3, Glue, Athena, EMR, Redshift, QuickSight can be used to build them. Specific services covered in more detail include Kinesis, MSK, Glue, EMR and Redshift. Real-world examples of big data usage are also presented.
Databricks Platform.pptx
The document provides an overview of the Databricks platform, which offers a unified environment for data engineering, analytics, and AI. It describes how Databricks addresses the complexity of managing data across siloed systems by providing a single "data lakehouse" platform where all data and analytics workloads can be run. Key features highlighted include Delta Lake for ACID transactions on data lakes, auto loader for streaming data ingestion, notebooks for interactive coding, and governance tools to securely share and catalog data and models.
More Related Content
What's hot
Philly Code Camp 2013 Mark Kromer Big Data with SQL Server
These are my slides from May 2013 Philly Code Camp at Penn State Abington. I will post the samples, code and scripts on my blog here following the event this Saturday: http://www.kromerbigdata.com
Datalab 101 (Hadoop, Spark, ElasticSearch) par Jonathan Winandy - Paris Spark...
Datalab 101 (Hadoop, Spark, ElasticSearch) par Jonathan Winandy - Paris Spark...Modern Data Stack France
Datalab 101 (Hadoop, Spark, ElasticSearch) par Jonathan Winandy
Retour d'expérience sur la mise en place d'un Datalab avec Hadoop, Spark et ElasticSearch dans un environnement contraint. Nous allons exposer les méthodes qui nous ont permis d'améliorer la conception, le développement, les performances et la recette d'une application complexe en Spark.
Jonathan Winandy est MOE, développeur Java/Scala spécialisé dans les pipelines de données.Hugfr SPARK & RIAK -20160114_hug_france
Presentation faite lors du Hadoop User Group France du 14 janvier 2016.
L’analytique temps réel avec Riak et Spark par Michael Carney (Basho) et Olivier Girardot de Lateral Thoughts
Selon un rapport de Salesforce, le nombre de sources de données analysées par les entreprises progressera de 83% au cours des cinq prochaines années, ainsi les organisations veulent désormais fournir des connaissances en temps réel même sur les appareils mobiles. Le traitement temps réel est donc, le futur de l’analyse big data.
Ce talk présentera des nouveautés en matière de l’analyse temps réel autour de la famille SGBD Riak et Spark.
Michael Carney est le Directeur Commercial de Basho pour le Sud d’Europe. Fondateur de MySQL France et de MariaDB, Michael a rejoint Basho en janvier 2015 pour explorer le monde de données sans tables !
Olivier Girardot est le CTO de Lateral Thoughts, il est développeur et formateur au sujet de Spark et également spécialiste de Java/Python dans le domaine de la finance de marché.
Dealing with Drift: Building an Enterprise Data Lake
This document discusses Cox Automotive's use of StreamSets to build an enterprise data lake. It describes the challenges of acquiring and ingesting data from 25 companies with different source types and schemas at scale. Cox Automotive addressed this by using StreamSets to separate the acquisition of data from its various source systems from the ingestion into the data lake. StreamSets pipelines automatically apply data standards and load balancing during ingestion. This architecture has allowed Cox Automotive to significantly increase its monthly ingestion volumes.
Minería de Datos en Sql Server 2008
Inteligencia de Negocios en SQL Server 2008 y Minería de Datos.
Ing. Eduardo Castro Martinez, PhD
Microsoft SQL Server MVP
http://ecastrom.blogspot.com
http://comunidadwindows.org
Dealing With Drift - Building an Enterprise Data Lake
Data drift, the gradual morphing of data structure and semantics, is a fact of life in enterprise IT. New requirements force schema changes, the meaning of database columns changes over time, and infrastructure upgrades add new fields to log files. Left unchecked, drift in data sources can cause applications and dataflows to fail, with costly downtime and, in the worst case, corruption in downstream data stores.
Cox Automotive comprises more than 25 companies dealing with different aspects of the car ownership lifecycle, with data as the common language they all share. The challenge for Cox was to create an efficient engine for the timely and trustworthy ingest of data capability for an unknown but large number of data assets from practically any source. Discover how their big data engineering team overcame data drift and are now populating a data lake, allowing analysts easy access to data from their subsidiary companies and producing new data assets unique to the industry.
Dirty Data? Clean it up! - Rocky Mountain DataCon 2016
Dan Lynn and Patrick Russell discuss data cleaning, data engineering, and how to operationalize data science at Rocky Mountain DataCon 2016.
Big Data LDN 2016: Out of the Data Warehouses, and into the Data Lakes and St...
This document discusses using data streams and lakes for big data analytics in utilities. It begins by explaining how smart meter rollouts have increased meter reads from 80 million to 350 billion per year. Traditional data warehousing is challenged by the time criticality, resource demands, and scale of this unbounded data. The document then introduces data streams for real-time analytics and data lakes for flexible storage of huge amounts of structured, semi-structured, and unstructured data. It describes Valo, an open lambda architecture that uses multiple repositories and stream processing to enable both real-time and historical analysis across disparate data sources.
Open source big data landscape and possible ITS applications
What is big data, and how open-source big data projects, such as Apache Spark, Kafka and Cassandra can be used in ITS (Intelligent Transport Systems) related projects.
PhD Projects in Cloudsim Research Assistance
Innovative Ideas in Cloudsim Projects
Exciting Fields on Cloudsim Projects
Renewed Subjects in Cloudsim Projects
Database Week San Francisco: Database Services at AWS
AWS provides a broad portfolio of databases and analytics services including data lakes, data movement services, non-relational databases like DynamoDB and ElastiCache, relational databases like RDS and Aurora, analytics services like Redshift and EMR, and machine learning services like SageMaker and Comprehend. These services are purpose-built to help customers build applications and analyze data.
AWS and Analytics Services
Short overview of AWS Database and Analytics offerings and an introduction of the day’s topics.
Speaker: Bill Baldwin - Database Technical Evangelist, AWS
HUG France - Paris - Data Engineer's Toolkit
1. The document discusses data integration challenges and how different technologies like Apache Beam, Spark, Flink, and Google Dataflow can be used to solve them.
2. It also covers data processing engines like Hadoop 2.0 including YARN, MapReduce, Pig, and Hive as well as streaming technologies like Apache Kafka.
3. The role of the data engineer is highlighted as being responsible for choosing the correct infrastructure to address data integration needs.
IaaS, PaaS, and DevOps for Data Scientist
This document discusses Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and DevOps for data science. It provides an overview of Azure services for IaaS including virtual machines and GPU instances. It also discusses PaaS options like Azure Machine Learning for deploying models as web services. The document advocates for using Azure services like HDInsight, Data Factory and Machine Learning to build distributed and scalable data science systems following architectures like lambda architecture. It highlights pros and cons of different approaches for flexibility, scalability and using open source tools for data science workloads on Azure.
GDBinSV_Meetup_DBMS_Trends_10062016
The DBMS market trends focused on the Graph DBMS. The benefit of the Graph Database and its forecasted the growth rate. The Advice from the renowned market research institute.
Building a data pipeline to ingest data into Hadoop in minutes using Streamse...
Slides from my talk at the Hadoop User Group Ireland meetup on June 13th 2016: building a data pipeline to ingest data from sources of different nature into Hadoop in minutes (and no coding at all) using the Open Source Streamsets Data Collector tool.
Solution architecture for big data projects
Solution architecture for big data projects
solution architecture,big data,hadoop,hive,hbase,impala,spark,apache,cassandra,SAP HANA,Cognos big insights
Big Data - Linked In_DEEPU
The document discusses big data technologies including Apache HBase, Hive, Hadoop, MapReduce, Spring XD, Sqoop, Spark, Kafka, Solr, Zookeeper, and Oozie. It also outlines several proof of concept projects involving importing structured and unstructured data using these technologies, real-time ingestion of data from databases into HDFS, Hive and HBase integration, indexing and searching large datasets with Solr, monitoring and charting big data growth, data compression and encryption, job scheduling with Oozie, and implementing machine learning algorithms with Spark MLib.
Alexander Pavlenko, Java Software Engineer, DataArt.
This document discusses building a Spark connector for Ryft, a hardware appliance that performs high-speed compute on big data. It provides examples of how to query data stored on Ryft using its query language. The Spark connector allows querying and processing Ryft data using Spark and RDDs in a distributed manner. It maps structured Ryft data to DataFrames for SQL queries and shows performance benefits of using Ryft compared to running Spark on EC2 servers.
Big Data: Improving capacity utilization of transport companies
Asparuh Koev is a successful serial entrepreneur. Currently, he is a CEO of Transmetrics, a solution for cargo transport companies that uses Big Data and predictive analytics. Asparuh holds a Bachelor’s Degree in Computer Science from the American University in Bulgaria and an MBA degree from the Vlerick Business School in Belgium.
“Big Data: Improving capacity utilization of transport companies” will explore the practical benefits, IT tools and challenges of implementing a big data solution in a traditional industry (cargo transport), using as a showcase a predictive analytics project done for Speedy.
What's hot (20)
Philly Code Camp 2013 Mark Kromer Big Data with SQL Server
Philly Code Camp 2013 Mark Kromer Big Data with SQL Server
Datalab 101 (Hadoop, Spark, ElasticSearch) par Jonathan Winandy - Paris Spark...
Datalab 101 (Hadoop, Spark, ElasticSearch) par Jonathan Winandy - Paris Spark...
Dealing with Drift: Building an Enterprise Data Lake
Dealing with Drift: Building an Enterprise Data Lake
Dealing With Drift - Building an Enterprise Data Lake
Dealing With Drift - Building an Enterprise Data Lake
Dirty Data? Clean it up! - Rocky Mountain DataCon 2016
Dirty Data? Clean it up! - Rocky Mountain DataCon 2016
Big Data LDN 2016: Out of the Data Warehouses, and into the Data Lakes and St...
Big Data LDN 2016: Out of the Data Warehouses, and into the Data Lakes and St...
Open source big data landscape and possible ITS applications
Open source big data landscape and possible ITS applications
Database Week San Francisco: Database Services at AWS
Database Week San Francisco: Database Services at AWS
Building a data pipeline to ingest data into Hadoop in minutes using Streamse...
Building a data pipeline to ingest data into Hadoop in minutes using Streamse...
Alexander Pavlenko, Java Software Engineer, DataArt.
Alexander Pavlenko, Java Software Engineer, DataArt.
Big Data: Improving capacity utilization of transport companies
Big Data: Improving capacity utilization of transport companies
Similar to Big Data Landscape 2019
AWS Big Data Landscape
This document provides an overview of big data concepts and architectures, as well as AWS big data services. It begins with introducing big data challenges around variety, volume, and velocity of data. It then covers the Hadoop ecosystem including HDFS, MapReduce, Hive, Pig and Spark. The document also discusses data lake architectures and how AWS services like S3, Glue, Athena, EMR, Redshift, QuickSight can be used to build them. Specific services covered in more detail include Kinesis, MSK, Glue, EMR and Redshift. Real-world examples of big data usage are also presented.
Databricks Platform.pptx
The document provides an overview of the Databricks platform, which offers a unified environment for data engineering, analytics, and AI. It describes how Databricks addresses the complexity of managing data across siloed systems by providing a single "data lakehouse" platform where all data and analytics workloads can be run. Key features highlighted include Delta Lake for ACID transactions on data lakes, auto loader for streaming data ingestion, notebooks for interactive coding, and governance tools to securely share and catalog data and models.
Introduction to apache kafka, confluent and why they matter
This is a short and introductory presentation on Apache Kafka (including Kafka Connect APIs, Kafka Streams APIs, both part of Apache Kafka) and other open source components part of the Confluent platform (such as KSQL).
This was the first Kafka Meetup in South Africa.
Microsoft Ignite AU 2017 - Orchestrating Big Data Pipelines with Azure Data F...
Data orchestration is the lifeblood of any successful data analytics solution. Take a deep dive into Azure Data Factory's data movement and transformation activities, particularly its integration with Azure's Big Data PaaS offerings such as HDInsight, SQL Data warehouse, Data Lake, and AzureML. Participants will learn how to design, build and manage big data orchestration pipelines using Azure Data Factory and how it stacks up against similar Big Data orchestration tools such as Apache Oozie.
Video of presentation:
https://channel9.msdn.com/Events/Ignite/Australia-2017/DA332
Building Data Warehouses and Data Lakes in the Cloud - DevDay Austin 2017 Day 2
This document discusses building data warehouses and data lakes in the cloud using AWS services. It provides an overview of AWS databases, analytics, and machine learning services that can be used to store and analyze data at scale. These services allow customers to migrate existing data warehouses to the cloud, build new data warehouses and data lakes more cost effectively, and gain insights from their data more easily.
Astroinformatics 2014: Scientific Computing on the Cloud with Amazon Web Serv...
An overview of Amazon Web Services (AWS) and a survey of scientific computing applications of cloud computing. Examples come from the fields of Astronomy, High Energy Physics and include examples from CERN, NASA and others.
Building a Data Lake on AWS
Build a simple data lake on AWS using a combination of services, including AWS Glue Data Catalog, AWS Glue Crawlers, AWS Glue Jobs, AWS Glue Studio, Amazon Athena, Amazon Relational Database Service (Amazon RDS), and Amazon S3.
Link to the blog post and video: https://garystafford.medium.com/building-a-simple-data-lake-on-aws-df21ca092e32
Ai tour 2019 Mejores Practicas en Entornos de Produccion Big Data Open Source...
En esta presentacion mostramos las buenas practicas en la implementacion de Big Data Open Source usando HDInsight
Slides: Proven Strategies for Hybrid Cloud Computing with Mainframes — From A...
Mainframes continue to perform mission-critical transaction processing and contain massive amounts of core business data. But digital transformation initiatives and cloud computing have created both opportunities and challenges for unlocking and utilizing this data. Qlik and AWS will share some of the proven strategies from successful customer deployments across a range of different mainframe to cloud use cases, including legacy application modernization, data analytics, and data migrations.
In this presentation, you will learn how to:
• Replicate very large volumes of mainframe data in real-time to the cloud
• Automate the creation of analytics-ready data lakes and data warehouses
• Achieve a 30% reduction in cost of compute
Building Serverless Data Infrastructure in the AWS Cloud
Presentation given at the Utah Big Mountain Data & Developer Conference in November 2017. Describes the new data economy, a reference architecture for Big Data infrastructure, and its application to Amazon Web Services serverless services.
When NOT to use Apache Kafka?
Apache Kafka is the de facto standard for data streaming to process data in motion. With its significant adoption growth across all industries, I get a very valid question every week: When NOT to use Apache Kafka? What limitations does the event streaming platform have? When does Kafka simply not provide the needed capabilities? How to qualify Kafka out as it is not the right tool for the job?
This session explores the DOs and DONTs. Separate sections explain when to use Kafka, when NOT to use Kafka, and when to MAYBE use Kafka.
No matter if you think about open source Apache Kafka, a cloud service like Confluent Cloud, or another technology using the Kafka protocol like Redpanda or Pulsar, check out this slide deck.
A detailed article about this topic:
https://www.kai-waehner.de/blog/2022/01/04/when-not-to-use-apache-kafka/
When NOT to Use Apache Kafka? With Kai Waehner | Current 2022
When NOT to Use Apache Kafka? With Kai Waehner | Current 2022
Apache Kafka is the de facto standard for data streaming to process data in motion. With its significant adoption growth across all industries, I get a very valid question every week: When NOT to use Apache Kafka? What limitations does the event streaming platform have? When does Kafka simply not provide the needed capabilities? How to qualify Kafka out as it is not the right tool for the job? This session explores the DOs and DONTs. Separate sections explain when to use Kafka, when NOT to use Kafka, and when to MAYBE use Kafka.
Semplificare l'analisi dei dati con architetture "Serverless": architetture e...
This document discusses serverless big data architectures using AWS services. It describes how serverless computing eliminates the need to provision and manage servers. AWS services like S3, DynamoDB, Kinesis, Lambda, Glue and Athena allow ingesting, storing, processing and analyzing data without managing infrastructure. These services scale automatically based on usage and provide built-in availability and fault tolerance.
Sharing Experiences in Cloud Adoption: Burlington, MA
A presentation by Matt Aslett, 451 Research and Seth Proctor, NuoDB CTO about the state of the database landscape.
(BDT302) Big Data Beyond Hadoop: Running Mahout, Giraph, and R on Amazon EMR ...
We will explore the strengths and limitations of Hadoop for analyzing large data sets and review the growing ecosystem of tools for augmenting, extending, or replacing Hadoop MapReduce. We will introduce the Amazon Elastic MapReduce (EMR) platform as the big data foundation for Hadoop and beyond by providing specific examples of running Machine Learning (Mahout), Graph Analytics (Giraph), and Statistical Analysis (R) on EMR. We will discuss also big data analytics and visualization of results with Amazon Redshift + third party business intelligence tools, as well as typical end-to-end Big Data workflow on AWS.
We will conclude with real-world examples from ICAO of Big Data analytics for aviation safety data on AWS. The integrated Safety Trend Analysis and Reporting System (iSTARS) is a web based system linking a collection of safety datasets and related web application to perform online safety and risk analysis. It uses AWS EC2, S3, EMR and related partner tools for continuous data aggregation and filtering.
AWS Webcast - Tableau Big Data Solution Showcase
2 years ago if someone had claimed they could stand up a petabyte scale data warehouse in under an hour and then have a non-technical business user querying it live 30 minutes later without knowing any SQL or coding language, they would have been laughed out of the room. These days, that’s called taking advantage of disruptive technology. Amazon Web Services and Tableau Software have shifted the entire paradigm by which organizations not only store and access their data, but ultimately how they innovate with it. The fast, scalable, and inexpensive services that AWS provides for housing data combined with Tableau’s unbelievably flexible and user friendly visual analytic solution means that within hours an organization can securely put the power of their massive data assets into the hands of their domain experts without expensive overhead or lengthy ramp-up time. Attend this webinar to learn how Amazon Web Services and Tableau Software are leveraged together everyday to: • Empower visual ad-hoc data discovery against big data • Revolutionize corporate reporting and dashboards • Promote data driven decision making at every level The presentation will include: • A live demonstration of AWS and Tableau working together • A real customer case study focused on fraud detection and online video metrics • Live Q&A and an opportunity to trial both solutions
Keynote sp summit 2014 final
This document summarizes an AWS summit that took place in 2014. It had over 3,000 attendees and 25 breakout sessions. The summit celebrated AWS's 8th birthday since launching in 2006. It highlighted how startups, enterprises, the public sector, and systems integrators are using AWS. It also discussed how AWS provides infrastructure as a service with the largest market share according to Gartner, and how its services allow for agility, breadth of platform, and continual innovation.
Industry experts webinar slides (final v1.0)
Slides from the live Webiner: Industry Experts Examine the State of Databases
451 Research analyst, Matt Aslett, discusses the state of the next-generation database market, the various categories of DBMS being offered on his database landscape map, and how to get a leg up on competitors with the best use cases for each of these categories.
Aslett and NuoDB CTO, Seth Proctor, visit a growing trend of “back to SQL” and what each of them believes lies ahead for in the data management marketplace.
Artur Fejklowicz - “Data Lake architecture” AI&BigDataDay 2017
AI&BigDataDay 2017
Artur Fejklowicz - “Data Lake architecture”
Сайт: www.aiconf.com.ua
FB group: www.fb.com/groups/974150252615820/
Designing big data analytics solutions on azure
This document discusses designing big data analytics solutions on Azure. It provides an overview of Azure's data landscape and common architectural patterns and scenarios for building analytics solutions using various Azure data and analytics services. These include Azure SQL Data Warehouse, Azure Data Lake Store, Azure Data Factory, Azure Machine Learning, and Power BI for reporting and visualization. The document also discusses using these services to build solutions for scenarios like data warehousing, data lakes, ETL/ELT, machine learning, streaming analytics and more.
Similar to Big Data Landscape 2019 (20)
Introduction to apache kafka, confluent and why they matter
Introduction to apache kafka, confluent and why they matter
Microsoft Ignite AU 2017 - Orchestrating Big Data Pipelines with Azure Data F...
Microsoft Ignite AU 2017 - Orchestrating Big Data Pipelines with Azure Data F...
Building Data Warehouses and Data Lakes in the Cloud - DevDay Austin 2017 Day 2
Building Data Warehouses and Data Lakes in the Cloud - DevDay Austin 2017 Day 2
Astroinformatics 2014: Scientific Computing on the Cloud with Amazon Web Serv...
Astroinformatics 2014: Scientific Computing on the Cloud with Amazon Web Serv...
Ai tour 2019 Mejores Practicas en Entornos de Produccion Big Data Open Source...
Ai tour 2019 Mejores Practicas en Entornos de Produccion Big Data Open Source...
Slides: Proven Strategies for Hybrid Cloud Computing with Mainframes — From A...
Slides: Proven Strategies for Hybrid Cloud Computing with Mainframes — From A...
Building Serverless Data Infrastructure in the AWS Cloud
Building Serverless Data Infrastructure in the AWS Cloud
When NOT to Use Apache Kafka? With Kai Waehner | Current 2022
When NOT to Use Apache Kafka? With Kai Waehner | Current 2022
Semplificare l'analisi dei dati con architetture "Serverless": architetture e...
Semplificare l'analisi dei dati con architetture "Serverless": architetture e...
Sharing Experiences in Cloud Adoption: Burlington, MA
Sharing Experiences in Cloud Adoption: Burlington, MA
(BDT302) Big Data Beyond Hadoop: Running Mahout, Giraph, and R on Amazon EMR ...
(BDT302) Big Data Beyond Hadoop: Running Mahout, Giraph, and R on Amazon EMR ...
Artur Fejklowicz - “Data Lake architecture” AI&BigDataDay 2017
Artur Fejklowicz - “Data Lake architecture” AI&BigDataDay 2017
More from QAware GmbH
Mit ChatGPT Dinosaurier besiegen - Möglichkeiten und Grenzen von LLM für die ...
Generative AI for Developers, 27.05.2024, Bonn (Martin Binder, Lead IT Consultant bei QAware)
Die kritische Infrastruktur wird nach wie vor von Dinosauriersystemen aus dem letzten Jahrtausend getragen. Die Ablösung dieser Altsysteme ist ein drängendes Problem der Digitalisierung in Deutschland. In meinem Projekt musste ich in wenigen Wochen ein 30 Jahre altes System auf Basis von IBM iSeries (AS/400) mit RPG-Code analysieren. Überraschenderweise kannte ChatGPT die RGP-Referenz von IBM. Mithilfe von ChatGPT konnte ich mich schnell einarbeiten und so ein brauchbares Arbeitsverständnis des Systems erreichen. Aber wo Licht ist, ist auch Schatten: ChatGPT hat kein Kontextverständnis, ist unvollständig und neigt zu Fehlschlüssen. Als Vorgeschmack hier der Powermove: ChatGPT nach jeder Antwort grundsätzlich danach fragen, was es falsch gemacht hat. Du wirst überrascht sein!
50 Shades of K8s Autoscaling #JavaLand24.pdf
JavaLand 2024, April 2024, Mario-Leander Reimer
== Dokument bitte herunterladen, falls unscharf! Please download slides if blurred! ==
Abstract:
Elasticity and unlimited scalability are the holy grail for any cloud-native application. Now you might think: “That’s easy!!! I just run my containers on a Kubernetes cluster and I am done.” But is it really that simple?
Turns out it’s not! Your application as well as your cloud infrastructure and K8s cluster need to address and support these non-functional requirements.
This session will have a detailed and also practical look at the different ways of autoscaling in Kubernetes. We will give an overview of the technical foundations and prerequisites, and then showcase several frameworks and technologies that can be used to flexibly autoscale your cluster and your cloud-native workloads.
Make Agile Great - PM-Erfahrungen aus zwei virtuellen internationalen SAFe-Pr...
GPM Regionalgruppe Chemnitz (Patrick Albert)
Wegen ihres Umfangs und Komplexität sind größere SAFe-Programme bereits in Präsenz hinsichtlich ihres Managements und ihrer Steuerung anspruchsvoll. Aufgrund von COVID19 jedoch war eine Verlegung in den virtuellen Raum im beschriebenen Praxisfall unausweichlich. Das Management hatte hierbei sicherzustellen, dass die Programmziele trotz des verminderten Kontaktes allen beteiligten Teams dauerhaft klar und präsent sind und dass die in den Teams umgesetzten Funktionen außerdem den genannten Programmzielen dienen.
Besonders wichtig ist dieses Alignment im Rahmen der regelmäßigen PI-Plannings, in welchen alle Teams gleichzeitig die jeweils kommenden Iterationen planen und dabei auch teamübergreifende Abhängigkeiten zuverlässig berücksichtigen müssen.
Es werden Erfolgsfaktoren für den virtuellen Einsatz von SAFe herausgearbeitet und beleuchtet.
Fully-managed Cloud-native Databases: The path to indefinite scale @ CNN Mainz
When it comes to the question: "Where do we actually store our application data?", we are spoilt for choice, especially when it comes to the major cloud providers.
The simple and often completely valid answer is still the classic relational database! It is very suitable for many areas of application, as the technology is tried and tested and can cover a very broad spectrum. It is therefore not surprising that all major cloud providers offer this as a "managed service".
For some years now, however, there have also been so-called cloud-native databases that have been specially developed for the requirements of the cloud. The big promise: "Infinite scalability"
In a large customer project, we have been using such a database productively for over 4 years with Azure CosmosDB. The presentation will deal with the following questions, among others
What does "upscalability" mean in practice ?
What do you have to pay attention to when designing?
What are the actual limits?
What other special features do I get?
When do I need a cloud-native database?
But that's not all! We also look beyond Azure to the other two major cloud providers: AWS and Google Cloud. With DynamoDB and Datastore/Firestore, they have similar products on offer.
Down the Ivory Tower towards Agile Architecture
iSAQB Software Architecture Gathering – Digital 2023, November 2023
Architecture for agile projects must be defined and described differently, as well as continuously developed and evolved. Not all decisions are made at once, nor are they all known right from the start of the project. This sessions presents various useful and lightweight methods, tools and team topologies that can be applied in (large) agile projects to avoid uncontrolled growth and architectural erosion but without acting from the ivory tower and thus suffocating team autonomy.
"Mixed" Scrum-Teams – Die richtige Mischung macht's!
IT-Tage 2023, Frankfurt am Main (Patrick Albert)
Die Rollen und insbesondere deren Zuständigkeiten sind in Scrum recht klar geregelt: Der Product Owner sorgt für das "Was", der Scrum Master für das "Wie" und die Developer für die eigentliche Umsetzung. Solange es sich dabei um ein internes Projekt mit einem komplett internen Team handelt, sind damit bereits die zentralen Fragen beantwortet. Ein wenig differenzierter zu betrachten sind allerdings Teams mit mehreren Parteien wie etwa beim Einsatz von Dienstleistern. Muss etwa zwingend der Kunde den Product Owner stellen oder kann dieser auch auf der Seite des Dienstleisters stehen? Falls alle Developer vom gleichen Dienstleister bereitgestellt werden, würde diese Konstellation sicherlich einige Kommunikationswege verkürzen. Allerdings hat ein Product Owner auf der Seite eines Dienstleisters sicher nicht die gleichen Verbindungen zu den Stakeholdern (Nutzer, Geldgeber, ...) des Produkts wie ein interner Product Owner – wie also könnte er ihre Anforderungen dann gut vor dem Team vertreten? Ähnliche Fragen stellen sich auch für den Scrum Master und das Development-Team. In den meisten Fällen gibt es für verschiedene Konstellationen jeweils Vor- und Nachteile – und zwar sowohl für den Kunden als auch für den Dienstleister. Wichtig bei der Entscheidung für eine dieser Konstellationen ist außerdem die Art des Projekts, der Kreis der Stakeholder, das zur Verfügung stehende Budget, der Zeitrahmen und noch einiges mehr.
Dieser Vortrag beleuchtet verschiedene dieser Varianten und zeigt Vorteile, Nachteile und Risiken auf.
Make Developers Fly: Principles for Platform Engineering
Make Developers Fly – Helping developers to build better applications
Cloud Native Night, Mainz, November 2023, Alex Krause
Platform Engineering is the next stage of DevOps and accelerates software developers even more to build applications faster and bring products rapidly to the customers. In this meetup, we show you the key principles of platform engineering, as we experienced them in our projects, and additionally show you a better way to manage your internal software platforms.
PRINCIPLES FOR PLATFORM ENGINEERING, Alex Krause
How do we help our developers to fly instead of crashing miserablely? The answer is Platform Engineering, a discipline for building internal developer platforms (IDPs) to simplify software delivery for product teams. In this talk, you'll learn how Platform Engineering evolved from the DevOps movement and what principles and best practices make for a good implementation. Finally, we'll take a look at reference architectures that can support your platform.
Der Tod der Testpyramide? – Frontend-Testing mit Playwright
Codineers Rosenheim Meetup, 2. November 2023 (Dominik Haas, QAware)
Wer kennt es nicht: Langsame, instabile und wartungsintensive Tests von Frontends und die Schmerzen damit.
Aber in den letzten Jahren und Monaten hat sich hier viel getan und Playwright ist der vielversprechende neue Stern der Web-Automatisierung.
Ich werde in meinem Talk über das Testen von Frontends sprechen und speziell auf Playwright als spannendes Werkzeug eingehen.
Neben der praktischen Einführung möchte ich auch mit Euch über Best Practices sprechen und einen Versuch wagen die Testpyramide zu töten (oder sie zumindest etwas zu beschädigen).
Was kommt nach den SPAs
Sie kennen doch bestimmt das "Gesetz des Instruments": Wer als Werkzeug nur ein modernes Frontend-Framework hat, löst jedes Problem mit einer Single Page-Applikation. So oder so ähnlich, nur halt mit Hammer und Nagel lautet es, beschreibt jedoch ganz gut die aktuelle Situation der JavaScript-Welt. Auf nahezu jede Anforderung wird mit einer aufgeblähten, clientseitig gerenderten SPA geantwortet. Doch ist es schön langsam an der Zeit, dass wir uns fragen sollten: Ist das wirklich alles? Und die Antwort lautet ziemlich sicher "Nein". Doch genau diesem Thema widmen wir uns und werfen einen Blick auf die Alternativen und da gibt es viele.
Im React-Ökosystem findet aktuell ein kleiner Umbruch statt. Mit Server Side Rendering, Static Site Generation, Server Components und Frameworks wie Next verschiebt sich ein Teil der Arbeit in Richtung Server. Dieser Trend ist auch bei Vue, Svelte und Angular zu beobachten. Und genau das ist es, was die sogenannten Meta-Frameworks ausmacht. Uns als EntwicklerInnen gibt das mehr Flexibilität, um auf Anforderungen reagieren zu können. Sie müssen nicht mehr den kompletten Quellcode zum Client übertragen, haben bessere Caching-Möglichkeiten und auch die Suchmaschinen sind Ihnen dankbar.
Dieser Vortrag gibt Ihnen einen Überblick über die wichtigsten Features von Meta-Frameworks und wo und vor allem wie sie gewinnbringend eingesetzt werden können.
Cloud Migration mit KI: der Turbo
Digital Future Congress (DFC) 2023, September 2023, München, Andreas Zitzelsberger (Business Unit Director bei QAware)
== Dokument bitte herunterladen, falls unscharf! Please download slides if blurred! ==
Alt-Systeme in die Cloud migrieren – mit sechs Leitplanken geht das effizient und sicher. Neu: Künstliche Intelligenz kann jetzt zum Turbo werden! Wir nutzen KI bei der Cloud-Migration und zeigen in diesem Vortrag, was geht und was nicht.
Migration von stark regulierten Anwendungen in die Cloud: Dem Teufel die See...
MedTech Stars, Webkonferenz, September 2023, Josef Adersberger (CEO bei QAware)
== Dokument bitte herunterladen, falls unscharf! Please download slides if blurred! ==
Der Vortrag „Migration von stark regulierten Anwendungen in die Cloud: Dem Teufel die Seele verkaufen oder Himmel auf Erden?“ erkundet das wachsende Bedürfnis und die Herausforderungen, medizinische Anwendungen in die Cloud zu migrieren.
Bei den entsprechenden Anwendungen stellt laut einer Umfrage der Datenschutz für 82% der Anwendungen die größte Herausforderung dar, um den Schritt in die Cloud zu machen. Trotzdem wächst der Druck, diesen Schritt zu vollziehen, aufgrund des Innovations- und Digitalisierungsbedarfs, der durch gesetzliche Initiativen wie das Krankenhaus-Zukunftsgesetz, gestiegene Patientenerwartungen und den immer klareren medizinischen Impact der Digitalisierung angetrieben wird.
Die zentrale Frage, die in diesem Vortrag erörtert wird, ist, wie man hochregulierte Anwendungen schrittweise in die Cloud migrieren kann. Hierzu werden wir einen Ansatz vorstellen und anhand von exemplarischen Anwendungen demonstrieren, wie eine solche Migration erfolgreich durchgeführt werden kann. Abschließend fassen wir den Vortrag mit fünf leitenden Prinzipien zusammen, die bei der Migration von stark regulierten Anwendungen in die Cloud beachtet werden sollten. Diese Prinzipien bieten einen nützlichen Leitfaden für Unternehmen und Institutionen, die den Umzug ihrer Anwendungen in die Cloud erwägen, und sollen dazu beitragen, die Herausforderungen der Cloud-Migration in diesem hochregulierten Bereich erfolgreich zu bewältigen.
Aus blau wird grün! Ansätze und Technologien für nachhaltige Kubernetes-Cluster
SAA 2023 | Software Architecture Alliance, September 2023, München, Mario-Leander Reimer (@LeanderReimer, CTO bei QAware)
== Dokument bitte herunterladen, falls unscharf! Please download slides if blurred! ==
Die Cloud hat bereits heute einen größeren CO2-Fußabdruck als die Luftfahrtindustrie, mit steigender Digitalisierung und Cloudifizierung wird sich dieser Trend fortsetzen, wenn wir nichts dagegen unternehmen. Viele Kubernetes-basierte Installationen sind gemessen am eigentlich benötigten Ressourcen Bedarf stark überdimensioniert und tragen so unnötig zur globalen Erwärmung bei. Wie sieht die Energiebilanz Ihres Clusters und Workloads aus?
In diesem Vortrag zeigen wir Ansätze und Technologien, die dabei helfen K8s-Cluster grün(er) zu machen. Zunächst braucht es Transparenz: Wie sieht die Energiebilanz des Clusters und seiner Workloads aus? Erst danach lassen sich diese gezielt auf ihre Energiesparsamkeit hin optimieren. Und das ist gar nicht so schwer, also packen wir es an!
Endlich gute API Tests. Boldly Testing APIs Where No One Has Tested Before.
SAA 2023 | Software Architecture Alliance, September 2023, München, Ildikó Tárkányi (Senior Software Engineer bei QAware)
== Dokument bitte herunterladen, falls unscharf! Please download slides if blurred! ==
Der Traum von den unendlichen Weiten der Daten ist schnell ausgeträumt, wenn wir unsere APIs nicht im Griff haben. Dabei ist es egal, ob wir uns APIs zwischen unseren Services, zwischen Frontend und Backend oder zu Fremdsystemen anschauen - APIs sind ein fundamentaler Bestandteil unserer Softwarearchitektur und ohne Testautomatisierung geht gar nichts.
Schlecht geschriebene Integrationstests mit selbst erfundenen Testfällen für unsere APIs können richtig nerven: Wir rufen die APIs auf und hoffen, dass genau das zurückkommt, was wir erwarten.
Das geht auch anders: Wir werfen einen Blick auf andere Testansätze, bewerten, welche Aspekte diese jeweils testen und schauen zum Beispiel auf:
• Contract Based Testing
• Black Box Testing der OpenAPI Specs
• Trace-basiertes Testing
Kubernetes with Cilium in AWS - Experience Report!
Cloud Native Night, Munich, September 2023, Bernhard Schaidhammer
=== Please download slides if blurred! ===
Cilium is a powerful tool for network policies and also encryption between the Kubernetes nodes. Cilium hooks deep into Kubernetes in the network stack as an plugin and can even replace the AWS CNI Plugin. This talk will share our project experiences.
Topics involve:
- Network Policies
- Encryption
- Hubble (Observability)
- Installation
- CLI Usage (Hubble / Cilium)
50 Shades of K8s Autoscaling
Container Days 2023, September 2023, Hamburg, Mario-Leander Reimer (@LeanderReimer, CTO @QAware).
== Dokument bitte herunterladen, falls unscharf! Please download slides if blurred! ==
Elasticity and unlimited scalability are the holy grail for any cloud-native application. Now you might think: “That’s easy!!! I just run my containers on a Kubernetes cluster and I am done.” But is it really that simple?
Turns out it’s not! Your application as well as your cloud infrastructure and K8s cluster need to address and support these non-functional requirements.
This session will have a detailed and also practical look at the different ways of autoscaling in Kubernetes. We will give an overview of the technical foundations and prerequisites, and then showcase several frameworks and technologies that can be used to flexibly autoscale your cluster and your cloud-native workloads.
Kontinuierliche Sicherheitstests für APIs mit Testkube und OWASP ZAP
This document discusses continuous API security testing using OWASP ZAP on Kubernetes. It proposes running ZAP scans against microservices continuously or on-demand directly in the Kubernetes cluster. This would find vulnerabilities earlier in the development process compared to only running security tests at the end. The document outlines how Testkube, a Kubernetes-native testing framework, could be used to orchestrate and run ZAP scans as tests against microservices to achieve continuous security testing. Demo projects are referenced to show how this approach could work.
Service Mesh Pain & Gain. Experiences from a client project.
Cloud Native Night, Mainz, September 2023, Markus Zimmermann
=== Please download slides if blurred! ===
The topic of service mesh is still present at every major DevOps conference and is the subject of controversial discussions.
A service mesh comes with the promise of implementing cross-functional requirements of microservices such as observability or secure communication without changes within the services. The operational effort and the integration of the services within the service mesh should also be problem-free.
But do the common service mesh implementations keep these promises? In a client project we have gained experience with Linkerd and can say: we did not achieve the benefits without pain. We want to share these experiences and what you can take away for your next project!
50 Shades of K8s Autoscaling
WeAreDevelopers World Congress 2023, July 2023, Mario-Leander Reimer
=== Please download slides if blurred! ===
Elasticity and unlimited scalability are the holy grail for any cloud-native application. Now you might think: “That’s easy!!! I just run my containers on a Kubernetes cluster and I am done.” But is it really that simple?
Turns out it’s not! Your application as well as your cloud infrastructure and K8s cluster need to address and support these non-functional requirements.
This session will have a detailed and also practical look at the different ways of autoscaling in Kubernetes.
We will give an overview of the technical foundations and prerequisites, and then showcase several frameworks and technologies that can be used to flexibly autoscale your cluster and your cloud-native workloads.
Blue turns green! Approaches and technologies for sustainable K8s clusters.
Kubernetes Community Days Munich 2023, Juli 2023, Mario-Leander Reimer
The cloud already has a larger CO2 footprint than the worldwide aviation industry. With increasing digitalization and cloudification this trend will continue if we don't do anything about it. Many Kubernetes installations are greatly oversized in terms of the resources actually required and thus contribute unnecessarily to global warming. What is the energy balance of your cluster and its workloads?
In this session we will discuss approaches and technologies that help to make K8s clusters green(er). First of all, transparency is needed: what is the energy balance of the cluster and its workloads? Only then we can start to optimize for better energy efficiency. And it's not that difficult, so let's get started!
Per Anhalter zu Cloud Nativen API Gateways
Mastering Kubernetes 2023, Juli 2023, Sonja Wegner
Gute APIs sind das Herzstück erfolgreicher digitaler Produkte und Cloud-nativer Anwendungen. Doch schlecht verwaltete APIs werden schnell zum Albtraum. Damit es kein böses Erwachen gibt, setzen wir auf API Gateways: Diese sind etabliert und bekannt und helfen uns bei der Verwaltung der APIs. Sie regeln unter anderem Traffic Management, Rollout-Szenarien, Versionierung, Zugriffskontrolle und Diagnostizierbarkeit.
In diesem Vortrag werden wir das Cloud-native API-Gateway-Ökosystem näher betrachten: Gloo, KrakenD, Kong, Envoy et al. Aber welches davon ist das Richtige für den Einsatz im nächsten Projekt? Lasst es uns herausfinden!
More from QAware GmbH (20)
Mit ChatGPT Dinosaurier besiegen - Möglichkeiten und Grenzen von LLM für die ...
Mit ChatGPT Dinosaurier besiegen - Möglichkeiten und Grenzen von LLM für die ...
Make Agile Great - PM-Erfahrungen aus zwei virtuellen internationalen SAFe-Pr...
Make Agile Great - PM-Erfahrungen aus zwei virtuellen internationalen SAFe-Pr...
Fully-managed Cloud-native Databases: The path to indefinite scale @ CNN Mainz
Fully-managed Cloud-native Databases: The path to indefinite scale @ CNN Mainz
"Mixed" Scrum-Teams – Die richtige Mischung macht's!
"Mixed" Scrum-Teams – Die richtige Mischung macht's!
Make Developers Fly: Principles for Platform Engineering
Make Developers Fly: Principles for Platform Engineering
Der Tod der Testpyramide? – Frontend-Testing mit Playwright
Der Tod der Testpyramide? – Frontend-Testing mit Playwright
Migration von stark regulierten Anwendungen in die Cloud: Dem Teufel die See...
Migration von stark regulierten Anwendungen in die Cloud: Dem Teufel die See...
Aus blau wird grün! Ansätze und Technologien für nachhaltige Kubernetes-Cluster
Aus blau wird grün! Ansätze und Technologien für nachhaltige Kubernetes-Cluster
Endlich gute API Tests. Boldly Testing APIs Where No One Has Tested Before.
Endlich gute API Tests. Boldly Testing APIs Where No One Has Tested Before.
Kubernetes with Cilium in AWS - Experience Report!
Kubernetes with Cilium in AWS - Experience Report!
Kontinuierliche Sicherheitstests für APIs mit Testkube und OWASP ZAP
Kontinuierliche Sicherheitstests für APIs mit Testkube und OWASP ZAP
Service Mesh Pain & Gain. Experiences from a client project.
Service Mesh Pain & Gain. Experiences from a client project.
Blue turns green! Approaches and technologies for sustainable K8s clusters.
Blue turns green! Approaches and technologies for sustainable K8s clusters.
Recently uploaded
一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理
UMN硕士毕业证成绩单【微信95270640】购买（明尼苏达大学毕业证成绩单硕士学历）Q微信95270640代办UMN学历认证留信网伪造明尼苏达大学学位证书精仿明尼苏达大学本科/硕士文凭证书补办明尼苏达大学 diplomaoffer,Transcript购买明尼苏达大学毕业证成绩单购买UMN假毕业证学位证书购买伪造明尼苏达大学文凭证书学位证书,专业办理雅思、托福成绩单，学生ID卡，在读证明，海外各大学offer录取通知书，毕业证书，成绩单，文凭等材料:1:1完美还原毕业证、offer录取通知书、学生卡等各种在读或毕业材料的防伪工艺（包括 烫金、烫银、钢印、底纹、凹凸版、水印、防伪光标、热敏防伪、文字图案浮雕，激光镭射，紫外荧光，温感光标）学校原版上有的工艺我们一样不会少，不论是老版本还是最新版本，都能保证最高程度还原，力争完美以求让所有同学都能享受到完美的品质服务。
#毕业证成绩单 #毕业証 #成绩单 #學生卡 #OFFER录取通知书 #雅思#托福等……
国外大学明尼苏达大学明尼苏达大学毕业证offer制作方法（一对一专业服务）
1客户提供办理信息：姓名生日专业学位毕业时间等（如信息不确定可以咨询顾问：我们有专业老师帮你查询）；
2开始安排制作毕业证成绩单电子图；
3毕业证成绩单电子版做好以后发送给您确认；
4毕业证成绩单电子版您确认信息无误之后安排制作成品；
5成品做好拍照或者视频给您确认；
6快递给客户（国内顺丰国外DHLUPS等快读邮寄）
— — 制作工艺 【高仿真】— —
凭借多年的制作经验本公司制作明尼苏达大学明尼苏达大学毕业证offer《激光》《水印》《钢印》《烫金》《紫外线》凹凸版uv版等防伪技术一流高精仿度几乎跟学校100%相同！让您绝对满意。
— — -公司理念 【诚信为主】— — —
我們以質量求生存.以服务求发展有雄厚的实力专业的团队咨询顾问为您细心解答可详谈是真是假眼见为实让您真正放心平凡人生,尽我所能助您一臂之力让我們携手圆您梦想!
此贴长年有效【贴心专线/微-信: 95270640】敬请保留此联系方式以备用！如有不在线请给我们留言！我们将在第一时间给您回复!上散发着一抹抹的光晕而这每处自然形成的细节融合在一起浑然天成的美实在令人心生愉悦小道的周边无秩序的生长着几株艳丽的野花红的粉的紫的虽混乱无章却给这幅美景更增添一份性感夹杂着一份纯洁的妖娆毫无违和感实在给人带来一份悠然幸福的心情如果说现在的审美已经断然拒绝了无声的话那么在树林间飞掠而过的小鸟叽叽咋咋的叫声是否就是这最后的点睛之笔悠然走在林间的小路上宁静与清香一丝丝的盛夏气息吸入身体昔日生活里的繁忙多
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
Regarding mobile operating systems, two major players dominate our thoughts: Android and iPhone. With Android leading the market, software development companies are focused on delivering apps compatible with this OS. Ensuring an app's functionality across various Android devices, OS versions, and hardware specifications is critical, making Android app testing essential.
All you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVM 🐰
Webinar On-Demand: Using Flutter for Embedded
Flutter is a popular open source, cross-platform framework developed by Google. In this webinar we'll explore Flutter and its architecture, delve into the Flutter Embedder and Flutter’s Dart language, discover how to leverage Flutter for embedded device development, learn about Automotive Grade Linux (AGL) and its consortium and understand the rationale behind AGL's choice of Flutter for next-gen IVI systems. Don’t miss this opportunity to discover whether Flutter is right for your project.
在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样
原版一模一样【微信：741003700 】【加拿大英属哥伦比亚大学毕业证本科学位证书】【微信：741003700 】学位证，留信认证（真实可查，永久存档）offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。
本公司拥有海外各大学样板无数，能完美还原海外各大学 Bachelor Diploma degree, Master Degree Diploma
1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
Unveiling the Advantages of Agile Software Development.pdf
Learn about Agile Software Development's advantages. Simplify your workflow to spur quicker innovation. Jump right in! We have also discussed the advantages.
Measures in SQL (SIGMOD 2024, Santiago, Chile)
SQL has attained widespread adoption, but Business Intelligence tools still use their own higher level languages based upon a multidimensional paradigm. Composable calculations are what is missing from SQL, and we propose a new kind of column, called a measure, that attaches a calculation to a table. Like regular tables, tables with measures are composable and closed when used in queries.
SQL-with-measures has the power, conciseness and reusability of multidimensional languages but retains SQL semantics. Measure invocations can be expanded in place to simple, clear SQL.
To define the evaluation semantics for measures, we introduce context-sensitive expressions (a way to evaluate multidimensional expressions that is consistent with existing SQL semantics), a concept called evaluation context, and several operations for setting and modifying the evaluation context.
A talk at SIGMOD, June 9–15, 2024, Santiago, Chile
Authors: Julian Hyde (Google) and John Fremlin (Google)
https://doi.org/10.1145/3626246.3653374
E-commerce Development Services- Hornet Dynamics
For any business hoping to succeed in the digital age, having a strong online presence is crucial. We offer Ecommerce Development Services that are customized according to your business requirements and client preferences, enabling you to create a dynamic, safe, and user-friendly online store.
Modelling Up - DDDEurope 2024 - Amsterdam
What to do when you have a perfect model for your software but you are constrained by an imperfect business model?
This talk explores the challenges of bringing modelling rigour to the business and strategy levels, and talking to your non-technical counterparts in the process.
Enums On Steroids - let's look at sealed classes !
These are slides for my session"Enums On Steroids - let's look at sealed classes !" - delivered among others, on Devoxx UK 2024 conference
Fundamentals of Programming and Language Processors
Fundamentals of Programming and Language Processors
zOS Mainframe JES2-JES3 JCL-JECL Differences
Document which explains the difference between Jes2 and Jes3
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
原版一模一样【微信：741003700 】【美国纽约州立大学奥尔巴尼分校毕业证学位证书】【微信：741003700 】学位证，留信认证（真实可查，永久存档）offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。
本公司拥有海外各大学样板无数，能完美还原海外各大学 Bachelor Diploma degree, Master Degree Diploma
1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
Salesforce Healthcare CRM, implemented by VALiNTRY360, revolutionizes patient management by enhancing patient engagement, streamlining administrative processes, and improving care coordination. Its advanced analytics, robust security, and seamless integration with telehealth services ensure that healthcare providers can deliver personalized, efficient, and secure patient care. By automating routine tasks and providing actionable insights, Salesforce Healthcare CRM enables healthcare providers to focus on delivering high-quality care, leading to better patient outcomes and higher satisfaction. VALiNTRY360's expertise ensures a tailored solution that meets the unique needs of any healthcare practice, from small clinics to large hospital systems.
For more info visit us https://valintry360.com/solutions/health-life-sciences
WWDC 2024 Keynote Review: For CocoaCoders Austin
Overview of WWDC 2024 Keynote Address.
Covers: Apple Intelligence, iOS18, macOS Sequoia, iPadOS, watchOS, visionOS, and Apple TV+.
Understandable dialogue on Apple TV+
On-device app controlling AI.
Access to ChatGPT with a guest appearance by Chief Data Thief Sam Altman!
App Locking! iPhone Mirroring! And a Calculator!!
一比一原版(USF毕业证)旧金山大学毕业证如何办理
USF硕士毕业证成绩单【微信95270640】一比一伪造旧金山大学文凭@假冒USF毕业证成绩单+Q微信95270640办理USF学位证书@仿造USF毕业文凭证书@购买旧金山大学毕业证成绩单USF真实使馆认证/真实留信认证回国人员证明
#一整套旧金山大学文凭证件办理#—包含旧金山大学旧金山大学本科毕业证成绩单学历认证|使馆认证|归国人员证明|教育部认证|留信网认证永远存档教育部学历学位认证查询办理国外文凭国外学历学位认证#我们提供全套办理服务。
一整套留学文凭证件服务：
一：旧金山大学旧金山大学本科毕业证成绩单毕业证 #成绩单等全套材料从防伪到印刷水印底纹到钢印烫金
二：真实使馆认证（留学人员回国证明）使馆存档
三：真实教育部认证教育部存档教育部留服网站永久可查
四：留信认证留学生信息网站永久可查
国外毕业证学位证成绩单办理方法：
1客户提供办理旧金山大学旧金山大学本科毕业证成绩单信息：姓名生日专业学位毕业时间等（如信息不确定可以咨询顾问：我们有专业老师帮你查询）；
2开始安排制作毕业证成绩单电子图；
3毕业证成绩单电子版做好以后发送给您确认；
4毕业证成绩单电子版您确认信息无误之后安排制作成品；
5成品做好拍照或者视频给您确认；
6快递给客户（国内顺丰国外DHLUPS等快读邮寄）。
教育部文凭学历认证认证的用途：
如果您计划在国内发展那么办理国内教育部认证是必不可少的。事业性用人单位如银行国企公务员在您应聘时都会需要您提供这个认证。其他私营 #外企企业无需提供！办理教育部认证所需资料众多且烦琐所有材料您都必须提供原件我们凭借丰富的经验帮您快速整合材料让您少走弯路。
实体公司专业为您服务如有需要请联系我: 微信95270640奈一次次令他失望山娃今年岁上五年级识得很多字从走出小屋开始山娃就知道父亲的家和工地共有一个很动听的名字——天河工地的底层空空荡荡很宽阔很凉爽在地上铺上报纸和水泥袋父亲和工人们中午全睡在地上地面坑坑洼洼山娃曾多次绊倒过也曾有长铁钉穿透凉鞋刺在脚板上但山娃不怕工地上也常有五六个从乡下来的小学生他们的父母亲也是高楼上的建筑工人小伙伴来自不同省份都操着带有浓重口音的普通话可不知为啥山娃不仅很快与他们熟识了
Liberarsi dai framework con i Web Component.pptx
In Italian
Presentazione sulle feature e l'utilizzo dei Web Component nell sviluppo di pagine e applicazioni web. Racconto delle ragioni storiche dell'avvento dei Web Component. Evidenziazione dei vantaggi e delle sfide poste, indicazione delle best practices, con particolare accento sulla possibilità di usare web component per facilitare la migrazione delle proprie applicazioni verso nuovi stack tecnologici.
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
Presented at NLJUG's J-Spring 2024.
Preparing Non - Technical Founders for Engaging a Tech Agency
Preparing non-technical founders before engaging a tech agency is crucial for the success of their projects. It starts with clearly defining their vision and goals, conducting thorough market research, and gaining a basic understanding of relevant technologies. Setting realistic expectations and preparing a detailed project brief are essential steps. Founders should select a tech agency with a proven track record and establish clear communication channels. Additionally, addressing legal and contractual considerations and planning for post-launch support are vital to ensure a smooth and successful collaboration. This preparation empowers non-technical founders to effectively communicate their needs and work seamlessly with their chosen tech agency.Visit our site to get more details about this. Contact us today www.ishtechnologies.com.au
Recently uploaded (20)
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
All you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVM
Unveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdf
Enums On Steroids - let's look at sealed classes !
Enums On Steroids - let's look at sealed classes !
Fundamentals of Programming and Language Processors
Fundamentals of Programming and Language Processors
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
Preparing Non - Technical Founders for Engaging a Tech Agency
Preparing Non - Technical Founders for Engaging a Tech Agency
Big Data Landscape 2019
- 1. qaware.de/news/big-data-landscape For more big data know-how see: Ingestion Data Sources Data Lake Processing Data Warehouse Analytics User Interface Dashboards Distributed Filesystem NoSQL DB Efficient data serialization formats: Integrated compression Column-oriented storage Predicate pushdown Data Workflows ETL Jobs Massive Parallelization Data Logistics Stream Processing Charting Libraries pandas shinydashboard Thrift Index Machines: Fast aggregation and search. In-Memory Databases: Fast access. Memcached Graph Databases NewSQL: SQL meets NoSQL. Time Series Databases ImpalaSQL Pig Streams Rickshaw NVD3 InfoVis Vaadin Charts dc.js Highcharts ChartJS RazorFlow AtlasPrometheus STRUCTURED DATA LAKE: The eternal memory. POLYGLOTT PERSISTENCE & ANALYTICS ANALYTICS FRONTENDSALGORITHMS MICRO ANALYTICS SERVICES: Subsitutes for reporting servers. Tarantool Heroic Kylin Flink twitter.com/qaware slideshare.net/qaware youtube.com/qawaregmbh xing.com/companies/qawaregmbh github.com/qaware linkedin.com/company/qaware-gmbh QAware GmbH +49 89 232315-0 info@qaware.de qaware.de GraphX Streaming Flink Flink qaware.de/news/big-data-landscape For more big data know-how see:Big Data Landscape 2019 Interactive Analytics Reporting & DashboardsData Science