Slides of IMC Essentials workshop.
The workshop covers fundamental capabilities of in-memory computing platforms that boost high-load applications and services, and bring existing IT architecture to the next level by storing and processing a massive amount of data both in RAM and, optionally, on disk.
The capabilities and benefits of such platforms will be demonstrated with the usage of Apache Ignite, which is the in-memory computing platform that is durable, strongly consistent, and highly available with powerful SQL, key-value and processing APIs.
OCCIware, an extensible, standard-based XaaS consumer platform to manage ever...OCCIware
The OCCIware project aims at managing in a unified manner all layers and domains of the Cloud (XaaS), by building on the Open Cloud Computing (OCCI) standard. OCCIware Metamodel formally specifies the main OCCI concepts. Today a first EMF metamodel is defined that adds to OCCI new concepts such as Extension, Configuration, and EDataType, addressing some limitations of OCCI.
This session highlights OCCIware platform two main components:
– The OCCIware Studio Factory, allowing to produce visually customizable diagram editors for any Cloud configuration business domain modeled in OCCI using the OCCI Extension Studio, such as the flagship Docker Studio ;
– The OCCIware Runtime, based on OW2 erocci project, including the tools for deployment, supervision and administration, and allowing to federate multiple XaaS Cloud runtimes, such as the Roboconf PaaS server and the ActiveEon Cloud Automation multi-IaaS connector.
This talk includes a demonstration of the Docker connector and of how to use the OCCIware Cloud Designer to configure a real life Cloud application (a Java API server on top of a MongoDB cluster)’s business, platform and infrastructure layers seamlessly on both VirtualBox and OpenStack infrastructure.
Distributed Database DevOps Dilemmas? Kubernetes to the RescueDenis Magda
Distributed databases can make so many things easier for a developer... but not always for DevOps. OK, almost never for DevOps. Kubernetes has come to the rescue with an easy application orchestration!
It’s straightforward to do the orchestration leaning on relational databases as a data layer. However, it’s becoming a bit trickier to do the same when a distributed SQL database or other kind of distributed storage is used instead.
In this talk you will learn how Kubernetes can orchestrate distributed database like Apache Ignite, in particular:
- Cluster Assembling - database nodes auto-discovery in Kubernetes.
- Database Resilience - automated horizontal scalability.
- Database Availability - what’s the role of Kubernetes and the database.
- Utilizing both RAM and disk - set up Apache Ignite in a way to get in-memory performance with the durability of disk.
Apache Ignite comes with ANSI-99 compliant, horizontally scalable and fault-tolerant distributed SQL database. The distribution is provided either by partitioning the data across cluster nodes or by full replication, depending on the use case.
Unlike many distributed SQL databases, Ignite durable memory treats both memory and disk as active storage tiers. The disk tier, a.k.a. native persistence, is disabled by default, in which case Ignite becomes a pure in-memory database (IMDB).
You can interact with Ignite as you would with any other SQL storage, using standard JDBC or ODBC connectivity. Ignite also provides native SQL APIs for Java, .NET and C++ developers for better performance.
Microservices Architectures With Apache IgniteDenis Magda
This webinar discusses how in-memory computing using Apache® Ignite™ can overcome the performance limitations common to microservices architectures built using traditional database architectures.
Apache Spark and Apache Ignite: Where Fast Data Meets the IoTDenis Magda
It is not enough to build a mesh of sensors or embedded devices to obtain more insights about the surrounding environment and optimize your production systems. Usually, your IoT solution needs to be capable of transferring enormous amounts of data to storage or the cloud where the data have to be processed further. Quite often, the processing of the endless streams of data has to be done in real-time so that you can react on the IoT subsystem's state accordingly.
This session will show attendees how to build a Fast Data solution that will receive endless streams from the IoT side and will be capable of processing the streams in real-time using Apache Ignite's cluster resources.
Apache Ignite: In-Memory Hammer for Your Data Science ToolkitDenis Magda
Machine learning is a method of data analysis that automates the building of analytical models. By using algorithms that iteratively learn from data, computers are able to find hidden insights without the help of explicit programming. These insights bring tremendous benefits into many different domains. For business users, in particular, these insights help organizations improve customer experience, become more competitive, and respond much faster to opportunities or threats. The availability of very powerful in-memory computing platforms, such as the open-source Apache Ignite (https://ignite.apache.org/), means that more organizations can benefit from machine learning today.
In this presentation, Denis will look at some of the main components of Apache Ignite, such as a distributed database, distributed computations, and machine learning toolkit. Through examples, attendees will learn how Apache Ignite can be used for data analysis.
In-Memory Computing Essentials for Software EngineersDenis Magda
Attendees will be introduced to the fundamental capabilities of in-memory computing platforms that are proven to boost application performance and solve scalability problems by storing and processing massive data sets in RAM and on disk.
The session is tailored for software engineers (with code samples in Java) who thirst for practical experience with in-memory computing technologies. You will be given an overview of in-memory concepts such as caches, databases and data grids combined with a technical deep-dive of the following topics:
• Distributed in-memory cluster deployment strategies
• How data partitioning and replication works in a nutshell
• APIs for data access - key-value, SQL and compute APIs
• Affinity collocation tips and tricks
• Making your cluster durable - persistence and other forms of reliability
Implementation examples presented will utilize Apache Ignite, open-source in-memory computing platform.
Apache Spark and Apache Ignite: Where Fast Data Meets IoTDenis Magda
It is not enough to build a mesh of sensors or embedded devices to obtain more insights about the surrounding environment and optimize your production systems. Usually, your IoT solution needs to be capable of transferring enormous amounts of data to storage or the cloud where the data have to be processed further. Quite often, the processing of the endless streams of data has to be done in real-time so that you can react on the IoT subsystem's state accordingly. This session will show attendees how to build a Fast Data solution that will receive endless streams from the IoT side and will be capable of processing the streams in real-time using Apache Ignite's cluster resources.
In particular, attendees will learn about data streaming to an Apache Ignite cluster from embedded devices and real-time data processing with Apache Spark.
OCCIware, an extensible, standard-based XaaS consumer platform to manage ever...OCCIware
The OCCIware project aims at managing in a unified manner all layers and domains of the Cloud (XaaS), by building on the Open Cloud Computing (OCCI) standard. OCCIware Metamodel formally specifies the main OCCI concepts. Today a first EMF metamodel is defined that adds to OCCI new concepts such as Extension, Configuration, and EDataType, addressing some limitations of OCCI.
This session highlights OCCIware platform two main components:
– The OCCIware Studio Factory, allowing to produce visually customizable diagram editors for any Cloud configuration business domain modeled in OCCI using the OCCI Extension Studio, such as the flagship Docker Studio ;
– The OCCIware Runtime, based on OW2 erocci project, including the tools for deployment, supervision and administration, and allowing to federate multiple XaaS Cloud runtimes, such as the Roboconf PaaS server and the ActiveEon Cloud Automation multi-IaaS connector.
This talk includes a demonstration of the Docker connector and of how to use the OCCIware Cloud Designer to configure a real life Cloud application (a Java API server on top of a MongoDB cluster)’s business, platform and infrastructure layers seamlessly on both VirtualBox and OpenStack infrastructure.
Distributed Database DevOps Dilemmas? Kubernetes to the RescueDenis Magda
Distributed databases can make so many things easier for a developer... but not always for DevOps. OK, almost never for DevOps. Kubernetes has come to the rescue with an easy application orchestration!
It’s straightforward to do the orchestration leaning on relational databases as a data layer. However, it’s becoming a bit trickier to do the same when a distributed SQL database or other kind of distributed storage is used instead.
In this talk you will learn how Kubernetes can orchestrate distributed database like Apache Ignite, in particular:
- Cluster Assembling - database nodes auto-discovery in Kubernetes.
- Database Resilience - automated horizontal scalability.
- Database Availability - what’s the role of Kubernetes and the database.
- Utilizing both RAM and disk - set up Apache Ignite in a way to get in-memory performance with the durability of disk.
Apache Ignite comes with ANSI-99 compliant, horizontally scalable and fault-tolerant distributed SQL database. The distribution is provided either by partitioning the data across cluster nodes or by full replication, depending on the use case.
Unlike many distributed SQL databases, Ignite durable memory treats both memory and disk as active storage tiers. The disk tier, a.k.a. native persistence, is disabled by default, in which case Ignite becomes a pure in-memory database (IMDB).
You can interact with Ignite as you would with any other SQL storage, using standard JDBC or ODBC connectivity. Ignite also provides native SQL APIs for Java, .NET and C++ developers for better performance.
Microservices Architectures With Apache IgniteDenis Magda
This webinar discusses how in-memory computing using Apache® Ignite™ can overcome the performance limitations common to microservices architectures built using traditional database architectures.
Apache Spark and Apache Ignite: Where Fast Data Meets the IoTDenis Magda
It is not enough to build a mesh of sensors or embedded devices to obtain more insights about the surrounding environment and optimize your production systems. Usually, your IoT solution needs to be capable of transferring enormous amounts of data to storage or the cloud where the data have to be processed further. Quite often, the processing of the endless streams of data has to be done in real-time so that you can react on the IoT subsystem's state accordingly.
This session will show attendees how to build a Fast Data solution that will receive endless streams from the IoT side and will be capable of processing the streams in real-time using Apache Ignite's cluster resources.
Apache Ignite: In-Memory Hammer for Your Data Science ToolkitDenis Magda
Machine learning is a method of data analysis that automates the building of analytical models. By using algorithms that iteratively learn from data, computers are able to find hidden insights without the help of explicit programming. These insights bring tremendous benefits into many different domains. For business users, in particular, these insights help organizations improve customer experience, become more competitive, and respond much faster to opportunities or threats. The availability of very powerful in-memory computing platforms, such as the open-source Apache Ignite (https://ignite.apache.org/), means that more organizations can benefit from machine learning today.
In this presentation, Denis will look at some of the main components of Apache Ignite, such as a distributed database, distributed computations, and machine learning toolkit. Through examples, attendees will learn how Apache Ignite can be used for data analysis.
In-Memory Computing Essentials for Software EngineersDenis Magda
Attendees will be introduced to the fundamental capabilities of in-memory computing platforms that are proven to boost application performance and solve scalability problems by storing and processing massive data sets in RAM and on disk.
The session is tailored for software engineers (with code samples in Java) who thirst for practical experience with in-memory computing technologies. You will be given an overview of in-memory concepts such as caches, databases and data grids combined with a technical deep-dive of the following topics:
• Distributed in-memory cluster deployment strategies
• How data partitioning and replication works in a nutshell
• APIs for data access - key-value, SQL and compute APIs
• Affinity collocation tips and tricks
• Making your cluster durable - persistence and other forms of reliability
Implementation examples presented will utilize Apache Ignite, open-source in-memory computing platform.
Apache Spark and Apache Ignite: Where Fast Data Meets IoTDenis Magda
It is not enough to build a mesh of sensors or embedded devices to obtain more insights about the surrounding environment and optimize your production systems. Usually, your IoT solution needs to be capable of transferring enormous amounts of data to storage or the cloud where the data have to be processed further. Quite often, the processing of the endless streams of data has to be done in real-time so that you can react on the IoT subsystem's state accordingly. This session will show attendees how to build a Fast Data solution that will receive endless streams from the IoT side and will be capable of processing the streams in real-time using Apache Ignite's cluster resources.
In particular, attendees will learn about data streaming to an Apache Ignite cluster from embedded devices and real-time data processing with Apache Spark.
Apache Ignite: In-Memory Hammer for Your Data Science ToolkitDenis Magda
Machine learning is a method of data analysis that automates the building of analytical models. By using algorithms that iteratively learn from data, computers are able to find hidden insights without the help of explicit programming. These insights bring tremendous benefits into many different domains. For business users, in particular, these insights help organizations improve customer experience, become more competitive, and respond much faster to opportunities or threats.
The availability of very powerful in-memory computing platforms, such as Apache Ignite, means that more organizations can benefit from machine learning today. In this presentation, we will discuss how the Compute Grid, Data Grid, and Machine Learning Grid components of Apache Ignite work together to enable your business to start reaping the benefits of machine learning. Through examples, attendees will learn how Apache Ignite can be used for data analysis and be the in-memory hammer in your machine learning toolkit.
On Cloud Nine: How to be happy migrating your in-memory computing platform to...Stephen Darlington
You’ve heard a lot about The Cloud and its benefits, but how do you migrate your application there? Should you migrate your application there. What are the trade-offs? Are there special considerations if you’re using an in-memory computing platform like GridGain or Apache Ignite?
In this talk, Stephen investigates the challenges and suggests some best practices. He also investigates the differences between an on-premise deployment versus using some of the major public cloud vendors, some of the special “cloud native” tools you might come across and suggests a neat method that you can use to seamlessly move your data over once you’ve decided that you should move to the cloud.
"Troubleshooting Apache Ignite (and best practices)" with Stan Lukyanov, [software engineer at GridGain Systems].
ummary: Whether you are getting started with Apache Ignite or have already deployed, this session is for you. Stan will explain how to set up deployments to make them easier to monitor, manage and keep up and running properly. He'll also hare best practice examples on how to:
* Configure Ignite and GridGain for deployment, management and monitoring
* Leverage log files during troubleshooting
* Use monitoring interfaces and tools such as JMX, Visor and Web Console
* Identify and fix top errors for newly installed and existing deployments
Continuous Machine and Deep Learning with Apache IgniteDenis Magda
With most machine learning (ML) and deep learning (DL) frameworks, it can take hours to move data, and hours to train models. It's also hard to scale, with data sets increasingly being larger than the capacity of any single server. The size of the data also makes it hard to incrementally test and retrain models in near real-time to improve results. Learn how Apache Ignite and GridGain help to address these limitations with model training and execution, and help achieve near-real-time, continuous learning. It will be explained how ML/DL work with Apache Ignite, and how to get started.
Topics include:
— Overview of distributed ML/DL including design, implementation, usage patterns, pros and consn
— Overview of Apache Ignite ML/DL, including prebuilt ML/DL, and how to add your own ML/DL algorithms
— Model execution with Apache Ignite, including how to build models with Apache Spark and deploy them in Ignite
— How Apache Ignite and TensorFlow can be used together to build distributed DL model training and execution
This document provides an overview and agenda for a presentation on Apache Ignite. The presentation covers an introduction to Apache Ignite as an in-memory computing platform, use cases, distributed database orchestration using Kubernetes, deploying an Ignite cluster on Kubernetes, and scaling the cluster. It also includes steps to deploy a cloud environment, access the Kubernetes dashboard, create an Ignite service, and check logs.
In this presentation we look at the roadmap for Apache Ignite 2.0 towards becoming one of the first convergent data platform that would combine cross-channel tiered storage model (DRAM, Flash, HDD) and multi-paradigm access pattern (K/V, SQL, MapReduce, MPP) into one highly integrated and easy to use data platform.
Bridging Your Business Across the Enterprise and Cloud with MongoDB and NetAppMongoDB
This document discusses how NetApp solutions can help businesses bridge their MongoDB databases across on-premises and cloud environments. It provides an introduction to NetApp and describes how their storage solutions and data fabric can enable hybrid cloud for MongoDB. Specific solutions and technologies discussed include NetApp ONTAP for storage management and provisioning, FlexClone for development/testing, and SolidFire for high performance MongoDB deployments. Customer examples and performance benefits are also summarized.
This document discusses remote monitoring of scientific instruments. It describes connecting instruments to a cloud platform for remote monitoring. Key aspects covered include collecting instrument data streams, storing the data in databases like Redis and Redshift, and building applications to allow remote monitoring and control. The document discusses different architecture designs, performance tests, and how Redis provided better performance than other approaches for real-time visualization of instrument data streams.
Apache Ignite is a high-performance, integrated and distributed in-memory platform for computing and transacting on large-scale data sets in real-time. But, did you know it provides streaming and complex event processing (CEP)? In this hands-on demonstration we will take Apache Ignite’s Streaming and CEP features for a test drive. We will start with an example streaming use case then demonstrate how to implement each component in Apache Ignite. Finally we will show how to connect a dashboard application to Apache Ignite to display the results.
Apache Ignite™ is a rapidly changing platform: if you were to look at 3 years ago, you would see a completely different product. In this talk, we will follow the path that led Apache Ignite™ from a compute grid and data grid product to a distributed database and In-memory computing platform. We will examine technical tasks and decisions that were driving the transformations (as an example - how we added native persistence to Apache Ignite™) and will wrap up the talk with the outstanding problems that are going to be solved for Apache Ignite™.
KubeCon 2017 - Kubernetes SIG Scheduling and Resource Management Working Grou...Jeremy Eder
This document outlines the agenda and discussion topics for the SIG Schedule + Resource Management Working Group Deep Dive meeting on December 6th. The meeting will cover graduating features to general availability, the resource API design proposal, device plugins, and dynamic resource binding. It will also include a group discussion on areas of overlap between SIGs, two-level scheduling, common pain points in resource management, and testing priority and preemption features.
PostgreSQL continuous backup and PITR with BarmanEDB
How can I achieve an RPO of 5 minutes for the backups of my PostgreSQL databases? And what about RPO=0 for zero data loss backups? This talk will give you answers to those questions, by guiding you through an overview of Disaster Recovery of PostgreSQL databases with Barman, covering its key concepts and providing useful patterns and tips.
The document discusses Pure Storage and its all-flash storage solutions. It provides 10 reasons to choose Pure, including that Pure storage is built for NVMe, has a disruptively simple architecture, proven high availability of 99.9999%, AI-driven management and predictive support, application support, self-protecting storage, an open full stack, Evergreen upgrades, and industry recognition as a leader in Gartner reports. The document then discusses Pure's business model, customer experience, technology advantages, and culture that emphasize customer satisfaction.
Kanister is an open source framework for application-level data management in Kubernetes. It allows developers to define data operations for applications through Blueprint definitions. These Blueprints capture tasks like backup and restore. Kanister then executes those tasks across Kubernetes using CustomResourceDefinitions and tools. It provides a way to perform complex data operations for stateful applications from the Kubernetes layer regardless of the underlying application architecture.
RedisConf17 - Turbo-charge your apps with Amazon Elasticache for RedisRedis Labs
This document provides an overview and summary of Amazon ElastiCache for Redis. It discusses the key features of ElastiCache including easy deployment and monitoring, enhanced Redis engine capabilities, high availability, cost effectiveness, and integration with other AWS services. It also covers usage patterns such as database caching, streaming data processing, and building real-time apps. Finally, it discusses best practices for building resilient architectures on ElastiCache including reference architectures, failure scenarios, and open source contributions from AWS.
No Time to Waste: Migrate from Oracle to EDB Postgres in MinutesEDB
Discover how to migrate from Oracle to Postgres quickly and without risk. Attend this webinar to learn how to leverage tools and technologies to convert your Oracle database to EDB Postgres with ease.
Plus, watch a live demo on a sample Oracle database migration, including the schema and the data, in just minutes using the fully-automated EDB Postgres Migration Portal.
These slides cover:
- Why Migrate?
- Different components of database migration
- Choosing the right ground to start with
- EDB's offering
Next Generation Scheduling for YARN and K8s: For Hybrid Cloud/On-prem Environ...DataWorks Summit
Scheduler of a container orchestration system, such as YARN and K8s, is a critical component that users rely on to plan resources and manage applications.
And if we assess where we are today, in YARN effectively it had two power schedulers (Fair and Capacity scheduler) and both serve many strong use cases in big data ecosystem. It can scale up to 50k nodes per cluster, and schedule 20k containers per second, and extremely efficient to manage batch workloads.
K8s default scheduler is an industry-proven solution to efficiently manage long-running services. As more big data apps are moving to K8s and cloud world, but many features like hierarchical queues to support multi-tenancy better, fairness resource sharing, and preemption, etc. are either missing or not mature enough at this point of time to support big data apps running on K8s.
At this point, there is no solution that exists to address the needs of having a unified resource scheduling experiences across platforms. That makes it extremely difficult to manage workloads running on different environments, from on-premise to cloud.
Hence evolving a common scheduler powered from YARN and K8s’s legacy capabilities and improving towards cloud use cases will focus more on use cases like:
Better bin-packing scheduling (and gang scheduling)
Autoscale up and shrink policy management
Effectively run batch workloads and services with clear SLA’s
In summary, we are improving core scheduling capabilities to manage both K8s and YARN cluster which is cloud aware as a separate initiative and above-mentioned cases will be the core focus of this initiative. More details of our works will be presented in this talk.
PostgreSQL 12: What is coming up?, Enterprise Postgres DayEDB
This document provides an agenda for discussing new features in PostgreSQL 12. It begins with introductions and thanks, then outlines the following topics: JIT compilation improvements, psql client improvements, new DBA features like SKIP_LOCKED for VACUUM and ANALYZE, partitioning improvements including partition pruning and faster COPY, indexing changes, SQL features, and an advanced server postscript.
Hear how Cisco runs ECE in production on Cisco UCS Integrated Infrastructure and learn about the configurations that help them index 3 billion documents each day at a rate of ~400K docs per second using Apache Spark.
Dive deep into an actual enterprise Linux migration by walking through the planning and execution of the process as seen by our customers. Our enterprise architects will break down the key migration steps to explain the available options, decisions made, and demonstrate actions on a live system. This episode gives you a representative migration experience before you actually migrate, illustrating: Side-by-side comparisons between Red Hat Enterprise Linux and CentOS; steps to consider for the operating system; and
steps to consider for common application stacks and packages.
Linux Security APIs and the Chromium Sandbox (SwedenCpp Meetup 2017)Patricia Aas
The Linux Security and Isolation APIs have become the basis of some of the most useful features server-side, providing the isolation required for efficient containers. However, these APIs also form the basis of the Chromium Sandbox on Linux, and we will study them in that context.
This presentation goes more in depth on some key points from the NDC (2017) presentation.
Apache Ignite: In-Memory Hammer for Your Data Science ToolkitDenis Magda
Machine learning is a method of data analysis that automates the building of analytical models. By using algorithms that iteratively learn from data, computers are able to find hidden insights without the help of explicit programming. These insights bring tremendous benefits into many different domains. For business users, in particular, these insights help organizations improve customer experience, become more competitive, and respond much faster to opportunities or threats.
The availability of very powerful in-memory computing platforms, such as Apache Ignite, means that more organizations can benefit from machine learning today. In this presentation, we will discuss how the Compute Grid, Data Grid, and Machine Learning Grid components of Apache Ignite work together to enable your business to start reaping the benefits of machine learning. Through examples, attendees will learn how Apache Ignite can be used for data analysis and be the in-memory hammer in your machine learning toolkit.
On Cloud Nine: How to be happy migrating your in-memory computing platform to...Stephen Darlington
You’ve heard a lot about The Cloud and its benefits, but how do you migrate your application there? Should you migrate your application there. What are the trade-offs? Are there special considerations if you’re using an in-memory computing platform like GridGain or Apache Ignite?
In this talk, Stephen investigates the challenges and suggests some best practices. He also investigates the differences between an on-premise deployment versus using some of the major public cloud vendors, some of the special “cloud native” tools you might come across and suggests a neat method that you can use to seamlessly move your data over once you’ve decided that you should move to the cloud.
"Troubleshooting Apache Ignite (and best practices)" with Stan Lukyanov, [software engineer at GridGain Systems].
ummary: Whether you are getting started with Apache Ignite or have already deployed, this session is for you. Stan will explain how to set up deployments to make them easier to monitor, manage and keep up and running properly. He'll also hare best practice examples on how to:
* Configure Ignite and GridGain for deployment, management and monitoring
* Leverage log files during troubleshooting
* Use monitoring interfaces and tools such as JMX, Visor and Web Console
* Identify and fix top errors for newly installed and existing deployments
Continuous Machine and Deep Learning with Apache IgniteDenis Magda
With most machine learning (ML) and deep learning (DL) frameworks, it can take hours to move data, and hours to train models. It's also hard to scale, with data sets increasingly being larger than the capacity of any single server. The size of the data also makes it hard to incrementally test and retrain models in near real-time to improve results. Learn how Apache Ignite and GridGain help to address these limitations with model training and execution, and help achieve near-real-time, continuous learning. It will be explained how ML/DL work with Apache Ignite, and how to get started.
Topics include:
— Overview of distributed ML/DL including design, implementation, usage patterns, pros and consn
— Overview of Apache Ignite ML/DL, including prebuilt ML/DL, and how to add your own ML/DL algorithms
— Model execution with Apache Ignite, including how to build models with Apache Spark and deploy them in Ignite
— How Apache Ignite and TensorFlow can be used together to build distributed DL model training and execution
This document provides an overview and agenda for a presentation on Apache Ignite. The presentation covers an introduction to Apache Ignite as an in-memory computing platform, use cases, distributed database orchestration using Kubernetes, deploying an Ignite cluster on Kubernetes, and scaling the cluster. It also includes steps to deploy a cloud environment, access the Kubernetes dashboard, create an Ignite service, and check logs.
In this presentation we look at the roadmap for Apache Ignite 2.0 towards becoming one of the first convergent data platform that would combine cross-channel tiered storage model (DRAM, Flash, HDD) and multi-paradigm access pattern (K/V, SQL, MapReduce, MPP) into one highly integrated and easy to use data platform.
Bridging Your Business Across the Enterprise and Cloud with MongoDB and NetAppMongoDB
This document discusses how NetApp solutions can help businesses bridge their MongoDB databases across on-premises and cloud environments. It provides an introduction to NetApp and describes how their storage solutions and data fabric can enable hybrid cloud for MongoDB. Specific solutions and technologies discussed include NetApp ONTAP for storage management and provisioning, FlexClone for development/testing, and SolidFire for high performance MongoDB deployments. Customer examples and performance benefits are also summarized.
This document discusses remote monitoring of scientific instruments. It describes connecting instruments to a cloud platform for remote monitoring. Key aspects covered include collecting instrument data streams, storing the data in databases like Redis and Redshift, and building applications to allow remote monitoring and control. The document discusses different architecture designs, performance tests, and how Redis provided better performance than other approaches for real-time visualization of instrument data streams.
Apache Ignite is a high-performance, integrated and distributed in-memory platform for computing and transacting on large-scale data sets in real-time. But, did you know it provides streaming and complex event processing (CEP)? In this hands-on demonstration we will take Apache Ignite’s Streaming and CEP features for a test drive. We will start with an example streaming use case then demonstrate how to implement each component in Apache Ignite. Finally we will show how to connect a dashboard application to Apache Ignite to display the results.
Apache Ignite™ is a rapidly changing platform: if you were to look at 3 years ago, you would see a completely different product. In this talk, we will follow the path that led Apache Ignite™ from a compute grid and data grid product to a distributed database and In-memory computing platform. We will examine technical tasks and decisions that were driving the transformations (as an example - how we added native persistence to Apache Ignite™) and will wrap up the talk with the outstanding problems that are going to be solved for Apache Ignite™.
KubeCon 2017 - Kubernetes SIG Scheduling and Resource Management Working Grou...Jeremy Eder
This document outlines the agenda and discussion topics for the SIG Schedule + Resource Management Working Group Deep Dive meeting on December 6th. The meeting will cover graduating features to general availability, the resource API design proposal, device plugins, and dynamic resource binding. It will also include a group discussion on areas of overlap between SIGs, two-level scheduling, common pain points in resource management, and testing priority and preemption features.
PostgreSQL continuous backup and PITR with BarmanEDB
How can I achieve an RPO of 5 minutes for the backups of my PostgreSQL databases? And what about RPO=0 for zero data loss backups? This talk will give you answers to those questions, by guiding you through an overview of Disaster Recovery of PostgreSQL databases with Barman, covering its key concepts and providing useful patterns and tips.
The document discusses Pure Storage and its all-flash storage solutions. It provides 10 reasons to choose Pure, including that Pure storage is built for NVMe, has a disruptively simple architecture, proven high availability of 99.9999%, AI-driven management and predictive support, application support, self-protecting storage, an open full stack, Evergreen upgrades, and industry recognition as a leader in Gartner reports. The document then discusses Pure's business model, customer experience, technology advantages, and culture that emphasize customer satisfaction.
Kanister is an open source framework for application-level data management in Kubernetes. It allows developers to define data operations for applications through Blueprint definitions. These Blueprints capture tasks like backup and restore. Kanister then executes those tasks across Kubernetes using CustomResourceDefinitions and tools. It provides a way to perform complex data operations for stateful applications from the Kubernetes layer regardless of the underlying application architecture.
RedisConf17 - Turbo-charge your apps with Amazon Elasticache for RedisRedis Labs
This document provides an overview and summary of Amazon ElastiCache for Redis. It discusses the key features of ElastiCache including easy deployment and monitoring, enhanced Redis engine capabilities, high availability, cost effectiveness, and integration with other AWS services. It also covers usage patterns such as database caching, streaming data processing, and building real-time apps. Finally, it discusses best practices for building resilient architectures on ElastiCache including reference architectures, failure scenarios, and open source contributions from AWS.
No Time to Waste: Migrate from Oracle to EDB Postgres in MinutesEDB
Discover how to migrate from Oracle to Postgres quickly and without risk. Attend this webinar to learn how to leverage tools and technologies to convert your Oracle database to EDB Postgres with ease.
Plus, watch a live demo on a sample Oracle database migration, including the schema and the data, in just minutes using the fully-automated EDB Postgres Migration Portal.
These slides cover:
- Why Migrate?
- Different components of database migration
- Choosing the right ground to start with
- EDB's offering
Next Generation Scheduling for YARN and K8s: For Hybrid Cloud/On-prem Environ...DataWorks Summit
Scheduler of a container orchestration system, such as YARN and K8s, is a critical component that users rely on to plan resources and manage applications.
And if we assess where we are today, in YARN effectively it had two power schedulers (Fair and Capacity scheduler) and both serve many strong use cases in big data ecosystem. It can scale up to 50k nodes per cluster, and schedule 20k containers per second, and extremely efficient to manage batch workloads.
K8s default scheduler is an industry-proven solution to efficiently manage long-running services. As more big data apps are moving to K8s and cloud world, but many features like hierarchical queues to support multi-tenancy better, fairness resource sharing, and preemption, etc. are either missing or not mature enough at this point of time to support big data apps running on K8s.
At this point, there is no solution that exists to address the needs of having a unified resource scheduling experiences across platforms. That makes it extremely difficult to manage workloads running on different environments, from on-premise to cloud.
Hence evolving a common scheduler powered from YARN and K8s’s legacy capabilities and improving towards cloud use cases will focus more on use cases like:
Better bin-packing scheduling (and gang scheduling)
Autoscale up and shrink policy management
Effectively run batch workloads and services with clear SLA’s
In summary, we are improving core scheduling capabilities to manage both K8s and YARN cluster which is cloud aware as a separate initiative and above-mentioned cases will be the core focus of this initiative. More details of our works will be presented in this talk.
PostgreSQL 12: What is coming up?, Enterprise Postgres DayEDB
This document provides an agenda for discussing new features in PostgreSQL 12. It begins with introductions and thanks, then outlines the following topics: JIT compilation improvements, psql client improvements, new DBA features like SKIP_LOCKED for VACUUM and ANALYZE, partitioning improvements including partition pruning and faster COPY, indexing changes, SQL features, and an advanced server postscript.
Hear how Cisco runs ECE in production on Cisco UCS Integrated Infrastructure and learn about the configurations that help them index 3 billion documents each day at a rate of ~400K docs per second using Apache Spark.
Dive deep into an actual enterprise Linux migration by walking through the planning and execution of the process as seen by our customers. Our enterprise architects will break down the key migration steps to explain the available options, decisions made, and demonstrate actions on a live system. This episode gives you a representative migration experience before you actually migrate, illustrating: Side-by-side comparisons between Red Hat Enterprise Linux and CentOS; steps to consider for the operating system; and
steps to consider for common application stacks and packages.
Linux Security APIs and the Chromium Sandbox (SwedenCpp Meetup 2017)Patricia Aas
The Linux Security and Isolation APIs have become the basis of some of the most useful features server-side, providing the isolation required for efficient containers. However, these APIs also form the basis of the Chromium Sandbox on Linux, and we will study them in that context.
This presentation goes more in depth on some key points from the NDC (2017) presentation.
[若渴計畫] Challenges and Solutions of Window Remote ShellcodeAj MaChInE
This document discusses challenges and solutions related to window remote shellcode. It outlines challenges posed by antivirus software, EMET, firewalls, and IDS/IPS systems. It then describes various techniques for bypassing these protections, such as encryption, obfuscation, non-standard programming languages, and the use of tools like Meterpreter and Veil Framework payloads. Specific bypass techniques covered include DLL injection, process hollowing, reflective loading, and the use of techniques like one-way shells and HTTP stagers.
Graduating To Go - A Jumpstart into the Go Programming LanguageKaylyn Gibilterra
This workshop jumps through a lot of what is covered in the Go Tour. The exercises are new and match more along with the class content, and some pieces (like testing and APIs) are not covered in the Go Tour.
Docker networking allows containers to communicate in several ways. Containers can communicate using Docker's default bridge (Docker0), by binding container ports to the host's ports, or using the host's network stack directly. More advanced options include linking containers to share information, using overlay networks with technologies like Open vSwitch, or running containers across multiple hosts with tunnels. The document provides examples of setting up different Docker networking configurations and discusses which methods suit different communication requirements between containers, hosts, and external networks.
Scale Up with Lock-Free Algorithms @ JavaOneRoman Elizarov
This document provides a summary of a presentation on using lock-free algorithms to scale shared mutable state on the JVM. It begins with an introduction to the speaker and discusses why shared mutable state is needed for big data and real-time processing. It then uses a toy problem of implementing a concurrent stack to demonstrate the challenges of synchronization and contention. The presentation introduces the use of atomic references and compare-and-set operations to implement lock-free push and pop operations on the concurrent stack in a non-blocking manner, improving scalability.
In queste slide potete trovare una raccolta di informazioni generiche e qualche link a risorse utili per iniziare o per approfondire la conoscenza di questo nuovo linguaggio di programmazione.
Communication hardware refers to electric devices and systems for transferring data or information from one place to another. Examples include modems, cables, fax modems, routers, and wireless technologies like infrared, Bluetooth, and Wi-Fi. The document provides details on each type of communication hardware, including what they are and how they function. It also includes multiple choice questions to test understanding of the different hardware.
numPYNQ is a hardware library that offers an accelerated version of NumPy core functions to be used transparently from data science applications. It implements these functions on an FPGA to provide better performance, energy efficiency, and flexibility compared to GPUs. Experimental results show speedups for tasks like matrix multiplication and cross-correlation. The library uses runtime input analysis and adaptation to optimize implementations. It has potential in the growing big data market, and the team plans partnerships and a freemium business model to commercialize numPYNQ.
The Linux Foundation has over 500 corporate members involved in over 70 member-sponsored projects. In 2016, the Linux Foundation convened over 20,000 people from 85 countries and over 4000 companies at 150 events around the world. Over 800,000 students from 215 countries have enrolled in Linux Foundation training programs. Who is driving this growth? Why do companies invest valuable resources in collaborative development? What have we learned along the way?
Virtualization Concepts
This document discusses various types of virtualization including server, storage, network, and application virtualization. It begins with defining virtualization as creating virtual versions of hardware platforms, operating systems, storage devices, and network resources. Server virtualization partitions physical servers into multiple virtual servers. Storage virtualization pools physical storage to appear as a single device. Network virtualization combines network resources into software-defined logical networks. Application virtualization encapsulates programs from the underlying OS. The document then covers the history of virtualization in mainframes and personal computers and dives deeper into specific virtualization types.
This document discusses the Go execution tracer tool. It provides an overview of the tool, how it can be used to collect trace data, and how the trace data can be analyzed. Specifically, it explains that the tracer instruments the Go runtime to capture events like goroutine creation and scheduling with nanosecond precision. It provides examples of using the tracer to optimize concurrency in applications and debug strange behaviors. The document concludes by noting that while the tracer is useful for understanding concurrency, it lacks documentation, providing an opportunity for community contributions.
Server virtualization concepts allow partitioning of physical servers into multiple virtual servers using virtualization software and hardware techniques. This improves resource utilization by running multiple virtual machines on a single physical server. Server virtualization provides benefits like reduced costs, higher efficiency, lower power consumption, and improved availability compared to running each application on its own physical server. Key components of server virtualization include virtual machines, hypervisors, CPU virtualization using techniques like Intel VT-x or AMD-V, memory virtualization, and I/O virtualization through methods like emulated, paravirtualized or direct I/O. KVM and QEMU are popular open source virtualization solutions, with KVM providing kernel-level virtualization support and Q
In-depth forensic analysis of Windows registry filesMaxim Suhanov
Uncovering the details of how a registry file is organized, how to locate & recover deleted data, and why third-party offline registry editors & viewers are failing to do their job well.
Errata.
- Page 8: "Zero-based", should be: "Zero-based, unset bits not counted".
- Page 12: "multiple delete records (entities)", should be: "multiple deleted records (entities)".
This document provides an overview of OpenFlow including:
- The need for OpenFlow to facilitate network innovation and programmability.
- How OpenFlow separates the control and data planes through an open interface.
- The basic components of an OpenFlow switch including flow tables, action sets, and packet processing.
- How OpenFlow controllers communicate with switches through secure channels to program flow entries.
- A demonstration of the packet flow through an OpenFlow network from switch to controller.
- Details on OpenDayLight and Mininet which are commonly used for OpenFlow demonstrations.
Network virtualization logically separates network resources and allows multiple virtual networks to operate over a shared physical infrastructure. It provides benefits like efficient usage of network resources, logical isolation of traffic between users, and accommodating dynamic server virtualization. Key enablers of network virtualization are cloud computing, server virtualization, software-defined networking (SDN), and network functions virtualization (NFV). A virtual tenant network (VTN) uses an underlay physical network and an overlay virtual network to logically isolate traffic for different users or groups. Common uses of network virtualization are in data centers and telecommunication networks.
This document provides an introduction to OpenFlow, SDN, and NFV. It describes the need for new networking paradigms and outlines some of the key problems with traditional networking approaches. OpenFlow is presented as providing open interfaces and programmability to network nodes. SDN is defined as separating the control logic from the forwarding plane and enabling programmable automation through open APIs. NFV aims to virtualize network functions to improve flexibility, reduce costs, and accelerate service deployment using standard IT virtualization technologies.
A Gentle Introduction to GPU Computing by Armen DonigianData Con LA
Abstract:- As data science continues to mature and evolve, the demand for more computationally extensive machines is rising. GPU Computing provides the core capabilities that data scientists today are looking for, and when implemented effectively, it accelerates deep learning, analytics and other sophisticated engineering applications. During this talk, Armen Donigian, Data Science Engineer at ZestFinance, will introduce the GPU programming model and parallel computing patterns, as well as practical implications of GPU computing, such as how to accelerate applications on a GPU with CUDA (C++/Python), GPU memory optimizations and multi GPU programming with MPI and OpenACC. As an example of how GPU programming can be implemented in real-life business models, Armen will present how ZestFinance has successfully tapped into the power of GPU Computing for the deep learning algorithm behind its new platform, Zest Automated Machine Learning platform (ZAML). Currently, ZAML is used by major tech, credit and auto companies to successfully apply cutting-edge machine learning models to their toughest credit decisioning problems. ZAML leverages GPU Computing for data parallelism, model parallelism and training parallelism.
From Mainframe to Microservices: Vanguard’s Move to the Cloud - ENT331 - re:I...Amazon Web Services
The document discusses Vanguard's move from a mainframe-based architecture to microservices in the cloud. It describes Vanguard's initial complex IT environment with monolithic applications and a mainframe. Vanguard's approach was to replicate data from the mainframe to the cloud, refactor applications to make API calls to microservices, and migrate batch processes. This "strangulation strategy" allowed the monolith to be gradually replaced. The document outlines Vanguard's cloud data architecture and how it leveraged AWS services like RDS, DynamoDB, Lambda and Kinesis while addressing compliance and operational requirements. Lessons learned included preparing for regulatory needs and pushback to cloud migration.
Apache Ignite is an integrated and distributed In-Memory Data Fabric for computing and transacting on large-scale data sets in real-time, orders of magnitude faster than possible with traditional disk-based or flash technologies. It is designed to easily power both existing and new applications in a distributed, massively parallel architecture on affordable, industry-standard hardware. Apache Ignite addresses today's Fast Data and Big Data needs by providing a comprehensive in-memory data fabric, which includes a data grid with SQL and transactional capabilities, in-memory streaming, an in-memory file system, and more.
This is a slide dump of a talk I gave at the 2017 Chicago Coder Conference (CCC) on June 26th, 2017.
http://www.chicagocoderconference.com/sessions/serverless-scheduled-job-processing/
Case Study: Sprinklr Uses Amazon EBS to Maximize Its NoSQL Deployment - DAT33...Amazon Web Services
Sprinklr delivers a complete social media management system for the enterprise. It also helps the world’s largest brands do marketing, advertising, care, sales, research, and commerce on Facebook, Twitter, LinkedIn, and 21 other channels on a global level. This is all done on a single integrated platform. In this session, you learn about Sprinklr’s journey to the cloud and discover how to optimize your NoSQL database on AWS for cost, efficiency, and scale. We also do dive deep into best practices and architectural considerations for designing and managing NoSQL databases, such as Apache Cassandra, MongoDB, Apache CouchDB, and Aerospike on Amazon EC2 and Amazon EBS. We share best practices for instance and volume selection, provide performance tuning hints, and describe cost optimization techniques throughout.
How we broke Apache Ignite by adding persistence, by Stephen Darlington (Grid...Altinity Ltd
Apache Ignite is an in-memory computing platform that combines fast in-memory performance with disk durability. The developers wanted to add persistence to Ignite to store more data than memory and enable crash recovery. They started with an ARIES architecture using page-based write-ahead logging to store everything off-heap. This worked initially but performance degraded with disk I/O. To maintain predictable speeds, they throttled load based on dirty page production and disk write rates. They also avoided doubling memory usage with the OS page cache by using direct I/O.
How to build a highly available PostgreSQL Cluster using EDB Failover Manager?
Database downtime can cost your business significant revenue. And while PostgreSQL is incredibly stable, it does not identify a database failure or automatically switches traffic to a standby database. In such a scenario, EDB Failover Manager (EFM) can be at the forefront of automatically detecting master database failures and promoting the most up-to-date standby database to take over.
Register for this free webinar and learn how EDB Failover Manager helps you build a highly available PostgreSQL Cluster for overall business continuity.
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark with Ma...Databricks
Building accurate machine learning models has been an art of data scientists, i.e., algorithm selection, hyper parameter tuning, feature selection and so on. Recently, challenges to breakthrough this “black-arts” have got started. We have developed a Spark-based automatic predictive modeling system. The system automatically searches the best algorithm, the best parameters and the best features without any manual work. In this talk, we will share how the automation system is designed to exploit attractive advantages of Spark. Our evaluation with real open data demonstrates that our system could explore hundreds of predictive models and discovers the highly-accurate predictive model in minutes on a Ultra High Density Server, which employs 272 CPU cores, 2TB memory and 17TB SSD in 3U chassis. We will also share open challenges to learn such a massive amount of models on Spark, particularly from reliability and stability standpoints.
Demystifying Data Warehousing as a Service - DFWKent Graziano
This document provides an overview and introduction to Snowflake's cloud data warehousing capabilities. It begins with the speaker's background and credentials. It then discusses common data challenges organizations face today around data silos, inflexibility, and complexity. The document defines what a cloud data warehouse as a service (DWaaS) is and explains how it can help address these challenges. It provides an agenda for the topics to be covered, including features of Snowflake's cloud DWaaS and how it enables use cases like data mart consolidation and integrated data analytics. The document highlights key aspects of Snowflake's architecture and technology.
Data Summer Conf 2018, “Apache Ignite + Apache Spark RDDs and DataFrames inte...Provectus
This topic will explain how Apache Spark and Ignite are integrated, and how they are used to together for analytics, stream processing and machine learning. And you will understand: – How Apache Ignite’s native RDD and new native DataFrame APIs work – How to use Ignite as an in-memory database and massively parallel processing (MPP) style collocated processing for preparing and managing data for Spark – How to leverage Ignite to easily share state across Spark jobs using mutable RDDs and DataFrames – How to leverage Ignite distributed SQL and advanced indexing in memory to improve SQL performance.
Advanced Patterns in Microservices Implementation with Amazon ECS - CON402 - ...Amazon Web Services
This document discusses advanced patterns for implementing microservices architectures using Amazon ECS. It covers microservices concepts, characteristics, and advantages of ECS. It also discusses how to deploy containers on ECS using tasks and services, implement the twelve-factor app model, perform continuous deployment, service discovery, and task placement. Finally, it shares BuzzFeed's experience building a microservices platform on ECS, including challenges overcome and lessons learned.
DAT332_How Verizon is Adopting Amazon Aurora PostgreSQL for Enterprise WorkloadsAmazon Web Services
Learn how Verizon is adopting the Amazon Aurora PostgreSQL-compatible edition for their mission-critical applications. Verizon has a history of adopting best of breed database technologies as they continue to serve their 140M+ customers. As Verizon moves its enterprise applications to the cloud, database performance and reliability are the key considerations. With heavy dependence on commercial databases, learn how a large enterprise like Verizon evaluated performance, reliability and operational characteristics of Amazon Aurora, and was able to create internal momentum behind adoption of open source technologies by showcasing early wins. This session also highlights best practices for using Amazon Aurora and the newly-announced RDS Performance Insights.
An Introduction to Apache Ignite - Mandhir Gidda - Codemotion Rome 2017Codemotion
Apache Ignite is a high-performance, integrated and distributed in-memory platform for computing and transacting on large-scale data sets in real-time, orders of magnitude faster than possible with traditional disk-based or flash technologies.
Prevention is better than cure. Learn 3 stages of AWS optimization.
1) Arrest Cloud Leakage
2) Implement Continuous Optimization
3) Explore cost-effective cloud options
ActOnMagic empowers cloud-first and cloud-only companies to Utilise any cloud services efficiently and securely without fear and losing freedom. Visit www.actonmagic.com
Cloudureka: Cloud IaaS Discovery (CID) Platform
Essential Tool kit for every cloud engineer
Search and Compare Any Cloud or Multi- Cloud to measure ROI
ActOnCloud: Intelligent Cloud Essentials (ICE) Platform *
Manage, Optimize and Provision Any Cloud or Multi-Cloud
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark with Ma...Databricks
Building accurate machine learning models has been an art of data scientists, i.e., algorithm selection, hyper parameter tuning, feature selection and so on. Recently, challenges to breakthrough this “black-arts” have got started. We have developed a Spark-based automatic predictive modeling system. The system automatically searches the best algorithm, the best parameters and the best features without any manual work. In this talk, we will share how the automation system is designed to exploit attractive advantages of Spark. Our evaluation with real open data demonstrates that our system could explore hundreds of predictive models and discovers the highly-accurate predictive model in minutes on a Ultra High Density Server, which employs 272 CPU cores, 2TB memory and 17TB SSD in 3U chassis. We will also share open challenges to learn such a massive amount of models on Spark, particularly from reliability and stability standpoints.
DDD&Scalaで作られたプロダクトはその後どうなったか?(Current state of products made with DDD & Scala)MicroAd, Inc.(Engineer)
ScalaMatsuri2023登壇スライド
https://scalamatsuri.org/ja/program/J1681614000
DDD&Scalaで作られたプロダクトはその後どうなったか?
Current state of products made with DDD & Scala
#ScalaMatsuri
Deploying Distributed Databases and In-Memory Computing Platforms with Kubern...Stephen Darlington
Presented at OpenStack Summit, Berlin 2018.
In this presentation, attendees will learn how Kubernetes can orchestrate a distributed database like Apache Ignite, in particular:
* Cluster Assembling - database nodes auto-discovery in Kubernetes.
* Database Resilience - automated horizontal scalability.
* Database Availability - what’s the role of Kubernetes and the database.
* Utilizing both RAM and disk - set up Apache Ignite in a way to get in-memory performance with the durability of disk.
Improving Apache Spark™ In-Memory Computing with Apache Ignite™Tom Diederich
GridGain Systems Lead Architect Valentin (Val) Kulichenko presented the following talk at the May 17 Bay Area In-Memory Computing Meetup: Improving Apache Spark™ In-Memory Computing with Apache Ignite™
Val explained how Apache Ignite™ simplifies development and improves performance for Apache Spark™. He'll demonstrate how Apache Spark and Ignite are integrated, and how they are used to together for analytics, stream processing and machine learning.
The following was covered:
* How Apache Ignite’s native RDD and new native DataFrame APIs work
* How to use Ignite as an in-memory database and massively parallel processing (MPP) style collocated processing for preparing and managing data for Spark
* How to leverage Ignite to easily share state across Spark jobs using mutable RDDs and DataFrames
* How to leverage Ignite distributed SQL and advanced indexing in memory to improve SQL performance
Azure + DataStax Enterprise Powers Office 365 Per User StoreDataStax Academy
We will present our O365 use case scenarios, why we chose Cassandra + Spark, and walk through the architecture we chose for running DataStax Enterprise on azure.
Similar to In-Memory Computing Essentials for Architects and Engineers (20)
Null Bangalore | Pentesters Approach to AWS IAMDivyanshu
#Abstract:
- Learn more about the real-world methods for auditing AWS IAM (Identity and Access Management) as a pentester. So let us proceed with a brief discussion of IAM as well as some typical misconfigurations and their potential exploits in order to reinforce the understanding of IAM security best practices.
- Gain actionable insights into AWS IAM policies and roles, using hands on approach.
#Prerequisites:
- Basic understanding of AWS services and architecture
- Familiarity with cloud security concepts
- Experience using the AWS Management Console or AWS CLI.
- For hands on lab create account on [killercoda.com](https://killercoda.com/cloudsecurity-scenario/)
# Scenario Covered:
- Basics of IAM in AWS
- Implementing IAM Policies with Least Privilege to Manage S3 Bucket
- Objective: Create an S3 bucket with least privilege IAM policy and validate access.
- Steps:
- Create S3 bucket.
- Attach least privilege policy to IAM user.
- Validate access.
- Exploiting IAM PassRole Misconfiguration
-Allows a user to pass a specific IAM role to an AWS service (ec2), typically used for service access delegation. Then exploit PassRole Misconfiguration granting unauthorized access to sensitive resources.
- Objective: Demonstrate how a PassRole misconfiguration can grant unauthorized access.
- Steps:
- Allow user to pass IAM role to EC2.
- Exploit misconfiguration for unauthorized access.
- Access sensitive resources.
- Exploiting IAM AssumeRole Misconfiguration with Overly Permissive Role
- An overly permissive IAM role configuration can lead to privilege escalation by creating a role with administrative privileges and allow a user to assume this role.
- Objective: Show how overly permissive IAM roles can lead to privilege escalation.
- Steps:
- Create role with administrative privileges.
- Allow user to assume the role.
- Perform administrative actions.
- Differentiation between PassRole vs AssumeRole
Try at [killercoda.com](https://killercoda.com/cloudsecurity-scenario/)
Batteries -Introduction – Types of Batteries – discharging and charging of battery - characteristics of battery –battery rating- various tests on battery- – Primary battery: silver button cell- Secondary battery :Ni-Cd battery-modern battery: lithium ion battery-maintenance of batteries-choices of batteries for electric vehicle applications.
Fuel Cells: Introduction- importance and classification of fuel cells - description, principle, components, applications of fuel cells: H2-O2 fuel cell, alkaline fuel cell, molten carbonate fuel cell and direct methanol fuel cells.
Redefining brain tumor segmentation: a cutting-edge convolutional neural netw...IJECEIAES
Medical image analysis has witnessed significant advancements with deep learning techniques. In the domain of brain tumor segmentation, the ability to
precisely delineate tumor boundaries from magnetic resonance imaging (MRI)
scans holds profound implications for diagnosis. This study presents an ensemble convolutional neural network (CNN) with transfer learning, integrating
the state-of-the-art Deeplabv3+ architecture with the ResNet18 backbone. The
model is rigorously trained and evaluated, exhibiting remarkable performance
metrics, including an impressive global accuracy of 99.286%, a high-class accuracy of 82.191%, a mean intersection over union (IoU) of 79.900%, a weighted
IoU of 98.620%, and a Boundary F1 (BF) score of 83.303%. Notably, a detailed comparative analysis with existing methods showcases the superiority of
our proposed model. These findings underscore the model’s competence in precise brain tumor localization, underscoring its potential to revolutionize medical
image analysis and enhance healthcare outcomes. This research paves the way
for future exploration and optimization of advanced CNN models in medical
imaging, emphasizing addressing false positives and resource efficiency.
Advanced control scheme of doubly fed induction generator for wind turbine us...IJECEIAES
This paper describes a speed control device for generating electrical energy on an electricity network based on the doubly fed induction generator (DFIG) used for wind power conversion systems. At first, a double-fed induction generator model was constructed. A control law is formulated to govern the flow of energy between the stator of a DFIG and the energy network using three types of controllers: proportional integral (PI), sliding mode controller (SMC) and second order sliding mode controller (SOSMC). Their different results in terms of power reference tracking, reaction to unexpected speed fluctuations, sensitivity to perturbations, and resilience against machine parameter alterations are compared. MATLAB/Simulink was used to conduct the simulations for the preceding study. Multiple simulations have shown very satisfying results, and the investigations demonstrate the efficacy and power-enhancing capabilities of the suggested control system.
Discover the latest insights on Data Driven Maintenance with our comprehensive webinar presentation. Learn about traditional maintenance challenges, the right approach to utilizing data, and the benefits of adopting a Data Driven Maintenance strategy. Explore real-world examples, industry best practices, and innovative solutions like FMECA and the D3M model. This presentation, led by expert Jules Oudmans, is essential for asset owners looking to optimize their maintenance processes and leverage digital technologies for improved efficiency and performance. Download now to stay ahead in the evolving maintenance landscape.
An improved modulation technique suitable for a three level flying capacitor ...IJECEIAES
This research paper introduces an innovative modulation technique for controlling a 3-level flying capacitor multilevel inverter (FCMLI), aiming to streamline the modulation process in contrast to conventional methods. The proposed
simplified modulation technique paves the way for more straightforward and
efficient control of multilevel inverters, enabling their widespread adoption and
integration into modern power electronic systems. Through the amalgamation of
sinusoidal pulse width modulation (SPWM) with a high-frequency square wave
pulse, this controlling technique attains energy equilibrium across the coupling
capacitor. The modulation scheme incorporates a simplified switching pattern
and a decreased count of voltage references, thereby simplifying the control
algorithm.
Electric vehicle and photovoltaic advanced roles in enhancing the financial p...IJECEIAES
Climate change's impact on the planet forced the United Nations and governments to promote green energies and electric transportation. The deployments of photovoltaic (PV) and electric vehicle (EV) systems gained stronger momentum due to their numerous advantages over fossil fuel types. The advantages go beyond sustainability to reach financial support and stability. The work in this paper introduces the hybrid system between PV and EV to support industrial and commercial plants. This paper covers the theoretical framework of the proposed hybrid system including the required equation to complete the cost analysis when PV and EV are present. In addition, the proposed design diagram which sets the priorities and requirements of the system is presented. The proposed approach allows setup to advance their power stability, especially during power outages. The presented information supports researchers and plant owners to complete the necessary analysis while promoting the deployment of clean energy. The result of a case study that represents a dairy milk farmer supports the theoretical works and highlights its advanced benefits to existing plants. The short return on investment of the proposed approach supports the paper's novelty approach for the sustainable electrical system. In addition, the proposed system allows for an isolated power setup without the need for a transmission line which enhances the safety of the electrical network
The Apache Ignite Platform
Apache Ignite is a memory-centric data platform that is used to build fast, scalable & resilient solutions.
At the heart of the Apache Ignite platform lies a distributed memory-centric data storage platform with ACID semantics, and powerful processing APIs including SQL, Compute, Key/Value and transactions. Built with a memory-centric approach, this enables Apache Ignite to leverage memory for high throughput and low latency whilst utilising local disk or SSD to provide durability and fast recovery.
The main difference between the memory-centric approach and the traditional disk-centric approach is that the memory is treated as a fully functional storage, not just as a caching layer, like most databases do. For example, Apache Ignite can function in a pure in-memory mode, in which case it can be treated as an In-Memory Database (IMDB) and In-Memory Data Grid (IMDG) in one.
On the other hand, when persistence is turned on, Ignite begins to function as a memory-centric system where most of the processing happens in memory, but the data and indexes get persisted to disk. The main difference here from the traditional disk-centric RDBMS or NoSQL system is that Ignite is strongly consistent, horizontally scalable, and supports both SQL and key-value processing APIs.
Apache Ignite platform can be integrated with third-party databases and external storage mediums and can be deployed on any infrastructure. It provides linear scalability, built-in fault tolerance, comprehensive security and auditing alongside advanced monitoring & management.
The Apache Ignite platform caters for a range of use cases including: Core banking services, Real-time product pricing, reconciliation and risk calculation engines, analytics and machine learning.
Ignite Data Grid is a distributed key-value store that enables storing data both in memory and on disk within distributed clusters and provides extensive APIs. Ignite Data Grid can be viewed as a distributed partitioned hash map with every cluster node owning a portion of the overall data. This way the more cluster nodes we add, the more data we can store.
Apache Ignite incorporates distributed SQL database capabilities as a part of its platform. The database is horizontally scalable, fault tolerant and SQL ANSI-99 compliant. It supports all SQL, DDL, and DML commands including SELECT, UPDATE, INSERT, MERGE, and DELETE queries. It also provides support for a subset of DDL commands relevant for distributed databases.
Data sets as well as indexes can be stored both in RAM and on disk thanks to the durable memory architecture. This allows executing distributed SQL operations across different memory layers achieving in-memory performance with durability of disk.
You can interact with Apache Ignite using SQL language via natively developed APIs for Java, .NET and C++, or via the Ignite JDBC or ODBC drivers. This provides a true cross-platform connectivity from languages such as PHP, Ruby and more.
Ignite In-Memory Compute Grid allows executing distributed computations in a parallel fashion to gain high performance, low latency, and linear scalability. Ignite compute grid provides a set of simple APIs that allow users distribute computations and data processing across multiple computers in the cluster.
The disk-centric systems, like RDBMS or NoSQL, generally utilize the classic client-server approach, where the data is brought from the server to the client side where it gets processed and then is usually discarded. This approach does not scale well as moving the data over the network is the most expensive operation in a distributed system.
A much more scalable approach is collocated processing that reverses the flow by bringing the computations to the servers where the data actually resides. This approach allows you to execute advanced logic or distributed SQL with JOINs exactly where the data is stored avoiding expensive serialization and network trips.
https://ignite.apache.org/collocatedprocessing.html
Collocation of computations with data allow for minimizing data serialization within network and can significantly improve performance and scalability of your application. Whenever possible, you should always make best effort to colocate your computations with the cluster nodes caching the data that needs to be processed.
Let's assume that a blizzard is approaching New York City. You, as a telecommunication company has to warn all the people sending a message to everyone with precise instructions on how to behave during such weather conditions. There are around 8 million New Yorkers in your database that have to receive the text message.
With the client-server approach the company has to connect to the database, move all 8 million (!) records from there to a client application that will text to everyone. This is highly inefficient that wastes network and computational resources of company's IT infrastructure.
However, if the company initially collocates all the cities it covers with the people who live there then it can send a single computation (!) to the cluster node that stores information about all New Yorkers and send the text message from there. This approach avoids 8 million records movement over the network and helps utilizing cluster resources for computation needs. That's the collocated processing in action!
https://github.com/techbysample/gagrid
GA Grid (Beta) is an in memory Genetic Algorithm (GA) component for Apache Ignite. A GA is a method of solving optimization problems by simulating the process of biological evolution. GA Grid provides a distributive GA library built on top of a mature and scalable Apache Ignite platform. GAs are excellent for searching through large and complex data sets for an optimal solution. Real world applications of GAs include: automotive design, computer gaming, robotics, investments, traffic/shipment routing and more.
Glossary
Chromosome is a sequence of Genes. A Chromosome represents a potential solution.
Crossover is the process in which the genes within chromosomes are combined to derive new chromosomes.
Fitness Score is a numerical score that measures the value of a particular Chromosome (ie: solution) relative to other Chromosome in the population.
Gene is the discrete building blocks that make up the Chromosome.
Genetic Algorithm (GA) is a method of solving optimization problems by simulating the process of biological evolution. A GA continuously enhances a population of potential solutions. With each iteration, a GA selects the 'best fit' individuals from the current population to create offspring for the next generation. After subsequent generations, a GA will "evolve" the population toward an optimal solution.
Mutation is the process where genes within a chromosomes are randomly updated to produce new characteristics.
Population is the collection of potential solutions or Chromosomes.
Selection is the process of choosing candidate solutions (Chromosomes) for the next generation.
DEMO: run several ML samples from the standard distribution.
Main benefits:
No ETL – online “in place” ML
In-memory speed & scale
Large scale parallelization
Optimized ML/DL algorithms
Last-mile GPU optimization
The rationale for building ML Grid is quite simple. Many users employ Ignite as the central high-performance storage and processing systems for various data sets. If they wanted to perform ML or Deep Learning (DL) on these data sets (i.e training sets or model inference) they had to ETL them first into some other systems like Apache Mahout or Apache Spark.
The roadmap for ML Grid is to start with core algebra implementation based on Ignite co-located distributed processing. The initial version was released with Ignite 2.0. Future releases will introduce custom DSLs for Python, R and Scala, growing collection of optimized ML algorithms such as Linear and Logistic Regression, Decision Tree/Random Forest, SVM, Naive Bayes, as well support for Ignite-optimized Neural Networks and integration with TensorFlow.
Current beta version of Apache Ignite Machine Learning Grid (ML Grid) supports a distributed machine learning library built on top of highly optimized and scalable Apache Ignite platform and implements local and distributed vector and matrix algebra operations as well as distributed versions of widely used algorithms.
Apache Ignite memory-centric platform is based on the Durable Memory architecture that allows storing and processing data and indexes both in memory and on disk when the Ignite Persistent Store feature is enabled. The memory architecture helps achieve in-memory performance with durability of disk using all the available resources of the cluster.Ignite's durable memory is built and operates in a way similar to the Virtual Memory of operating systems such as Linux. However, one significant difference between these two types of architectures is that Durable Memory always keeps the whole data set and indexes on disk if the Ignite Persistent Store is used, while Virtual Memory uses the disk for swapping purposes only.
In-Memory
• Off-Heap memory
• Removes noticeable GC pauses
• Automatic Defragmentation
• Predictable memory consumption
• Boosts SQL performance
On Disk
• Optional Persistence
• Support of flash, SSD, Intel 3D Xpoint
• Stores superset of data
• Fully Transactional
◦ Write-Ahead-Log (WAL)
• Instantaneous Cluster Restarts
Ignite Native Persistence is a distributed ACID and SQL-compliant disk store that transparently integrates with Ignite's Durable Memory as an optional disk layer storing data and indexes on SSD, Flash, 3D XPoint, and other types of non-volatile storages.
With the Ignite Persistence enabled, you no longer need to keep all the data and indexes in memory or warm it up after a node or cluster restart because the Durable Memory is tightly coupled with persistence and treats it as a secondary memory tier. This implies that if a subset of data or an index is missing in RAM, the Durable Memory will take it from the disk.
B-tree is a self-balancing tree data structure that keeps data sorted and allows searches, sequential access, insertions, and deletions in logarithmic time.
B+Tree is a central part of the whole Ignite Virtual memory architecture because even basic key-value operations work via it (cache.get and cache.put)! Move to the next slide.
On the previous slide we explained how to look up a value inside of the virtual memory. However, how does the virtual memory know where to put a new value?
In fact, Ignite uses a special data structure called Free List to support this. Basically, a free list is a doubly linked list that stores references to pages of approximately equal free space. For instance, there is a free list that stores all the data pages that have up to 75% free space and a list that keeps track of the index pages with 25% capacity left.
Data and index pages are tracked in separate free lists.
Ignite Native Persistence is a distributed ACID and SQL-compliant disk store that transparently integrates with Ignite's Durable Memory as an optional disk layer storing data and indexes on SSD, Flash, 3D XPoint, and other types of non-volatile storages.
With the Ignite Persistence enabled, you no longer need to keep all the data and indexes in memory or warm it up after a node or cluster restart because the Durable Memory is tightly coupled with persistence and treats it as a secondary memory tier. This implies that if a subset of data or an index is missing in RAM, the Durable Memory will take it from the disk.