The CloudStack European User group met on Thursday 11th for our quarterly meeting.
Stuart Mcall from Basho talked about their RiakCS technology & community
The CloudStack European User group met on Thursday 11th for our quarterly meeting.
Bob Defoe from Cloudian gave a good overview of the use of Cloudian for object based storage within CloudStack
How to Migrate from Cassandra to Amazon DynamoDB - AWS Online Tech TalksAmazon Web Services
Learning Objectives:
- Learn how to migrate from Cassandra to DynamoDB
- Learn about the considerations and pre-requisites for migrating to DynamoDB
- Learn the benefits of a fully managed nosql database - DynamoDB
AWS re:Invent 2016: From Resilience to Ubiquity - #NetflixEverywhere Global A...Amazon Web Services
Building and evolving a pervasive, global service requires a multi-disciplined approach that balances requirements with service availability, latency, data replication, compute capacity, and efficiency. In this session, we’ll follow the Netflix journey of failure, innovation, and ubiquity. We'll review the many facets of globalization and then delve deep into the architectural patterns that enable seamless, multi-region traffic management; reliable, fast data propagation; and efficient service infrastructure. The patterns presented will be broadly applicable to internet services with global aspirations.
Apache Spark on Kubernetes Anirudh Ramanathan and Tim ChenDatabricks
Kubernetes is a fast growing open-source platform which provides container-centric infrastructure. Conceived by Google in 2014, and leveraging over a decade of experience running containers at scale internally, it is one of the fastest moving projects on GitHub with 1000+ contributors and 40,000+ commits. Kubernetes has first class support on Google Cloud Platform, Amazon Web Services, and Microsoft Azure.
Unlike YARN, Kubernetes started as a general purpose orchestration framework with a focus on serving jobs. Support for long-running, data intensive batch workloads required some careful design decisions. Engineers across several organizations have been working on Kubernetes support as a cluster scheduler backend within Spark. During this process, we encountered several challenges in translating Spark considerations into idiomatic Kubernetes constructs. In this talk, we describe the challenges and the ways in which we solved them. This talk will be technical and is aimed at people who are looking to run Spark effectively on their clusters. The talk assumes basic familiarity with cluster orchestration and containers.
The CloudStack European User group met on Thursday 11th for our quarterly meeting.
Bob Defoe from Cloudian gave a good overview of the use of Cloudian for object based storage within CloudStack
How to Migrate from Cassandra to Amazon DynamoDB - AWS Online Tech TalksAmazon Web Services
Learning Objectives:
- Learn how to migrate from Cassandra to DynamoDB
- Learn about the considerations and pre-requisites for migrating to DynamoDB
- Learn the benefits of a fully managed nosql database - DynamoDB
AWS re:Invent 2016: From Resilience to Ubiquity - #NetflixEverywhere Global A...Amazon Web Services
Building and evolving a pervasive, global service requires a multi-disciplined approach that balances requirements with service availability, latency, data replication, compute capacity, and efficiency. In this session, we’ll follow the Netflix journey of failure, innovation, and ubiquity. We'll review the many facets of globalization and then delve deep into the architectural patterns that enable seamless, multi-region traffic management; reliable, fast data propagation; and efficient service infrastructure. The patterns presented will be broadly applicable to internet services with global aspirations.
Apache Spark on Kubernetes Anirudh Ramanathan and Tim ChenDatabricks
Kubernetes is a fast growing open-source platform which provides container-centric infrastructure. Conceived by Google in 2014, and leveraging over a decade of experience running containers at scale internally, it is one of the fastest moving projects on GitHub with 1000+ contributors and 40,000+ commits. Kubernetes has first class support on Google Cloud Platform, Amazon Web Services, and Microsoft Azure.
Unlike YARN, Kubernetes started as a general purpose orchestration framework with a focus on serving jobs. Support for long-running, data intensive batch workloads required some careful design decisions. Engineers across several organizations have been working on Kubernetes support as a cluster scheduler backend within Spark. During this process, we encountered several challenges in translating Spark considerations into idiomatic Kubernetes constructs. In this talk, we describe the challenges and the ways in which we solved them. This talk will be technical and is aimed at people who are looking to run Spark effectively on their clusters. The talk assumes basic familiarity with cluster orchestration and containers.
Scalable Media Processing in the Cloud (MED302) | AWS re:Invent 2013Amazon Web Services
The cloud empowers you to process media at scale in ways that were previously not possible, enabling you to make business decisions that are no longer constrained by infrastructure availability. Hear about best practices to architect scalable, highly available, high-performance workflows for digital media processing. In addition, this session covers AWS and partner solutions for transcoding, content encryption (watermarking and DRM), QC, and other processing topics.
Slides presented during the Strata SF 2019 conference. Explaining how Lyft is building a multi-cluster solution for running Apache Spark on kubernetes at scale to support diverse workloads and overcome challenges.
In this presentation we will look at how customers in the broadcast and media space are using AWS to innovate and serve their content to end users in new ways. We will cover how AWS is being used for media creation, storage, processing, delivery, and monetization of content.
After the session you will understand how to apply these technologies and techniques to your own business.
For more training on AWS, visit: https://www.qa.com/amazon
AWS Loft | London - Deep Dive: Amazon RDS by Toby Knight, Manager Solutions Architecture, 18 April 2016
AWS re:Invent 2016: Get Technically Inspired by Container-Powered Migrations ...Amazon Web Services
This session is a technical journey through application migration and refactoring using containerized technologies. Flux 7 recently worked with Rent-a-Center to perform a Hybris migration from their datacenter to AWS and you can hear how they used Amazon ECS, the new Application Load Balancer, and Auto Scaling to meet the customers' business objectives.
AWS re:Invent 2016: Optimizing Network Performance for Amazon EC2 Instances (...Amazon Web Services
Many customers are using Amazon EC2 instances to run applications with high performance networking requirements. In this session, we provide an overview of Amazon EC2 network performance features (enhanced networking, ENA, placement groups, etc.), and discuss how we are innovating on behalf of our customers to improve networking performance in a scalable and cost-efficient manner. We share best practices and performance tips for getting the best networking performance out of your Amazon EC2 instances.
Dev ops for big data cluster management toolsRan Silberman
What are the tools that we can find to day to manage Hadoop cluster and its ecosystem?
There are two tools ready today:
Cloudera Manager and Ambari from Hortonworks.
In this presentation I explain what they do and why to use them, as well as Pros. and Cons.
Responding to Digital Transformation With RDS Database TechnologyAlibaba Cloud
See Webinar Recording at https://resource.alibabacloud.com/webinar/detail.htm?webinarId=30
Learn how your business can gain an advantage and utilize ApsaraDB for RDS to enhance data flexibility, security, performance, and stability amid the backdrop of digital transformation.
Chen Zhaoshang of the Alibaba Cloud Database Product team will share how traditional industries can meet database challenges and leverage distributed architecture transformation using MySQL and other open technologies to reduce the overall cost of ownership and realize cross-data-center disaster tolerance deployment while ensuring data consistency. He will also share how to ensure database security databases and alleviate concerns about data migration to the cloud.
ApsaraDB for RDS: www.alibabacloud.com/product/apsaradb-for-rds
More Webinars: https://resource.alibabacloud.com/webinar/index.htm
AWS re:Invent 2016: Moving Mountains: Netflix's Migration into VPC (NET304)Amazon Web Services
Netflix was one of the earliest very large AWS customers. By 2014, we were running hundreds of applications in Amazon EC2. That was great, until we needed to move to VPC. Given our scale, uptime requirements, and the decentralized nature of how we manage our production environment, the VPC migration (still ongoing) presented particular challenges for us and for AWS as it sought to support our move. In this talk, we discuss the starting state, our requirements and the operating principles we developed for how we wanted to drive the migration, some of the issues we ran into, and how the tight partnership with AWS helped us migrate from an EC2-Classic platform to an EC2-VPC platform.
Day 3 - AWS MySQL Relational Database Service Best Practices for Performance ...Amazon Web Services
Amazon RDS makes it easy to set up, operate, and scale, relational databases in the cloud. Amazon RDS for MySQL supports applications that require up to tens of thousands of IOPS, and allows you to scale on demand without administrative complexity. In this webinar, we will discuss best practices for getting the most out of Amazon RDS for MySQL, as well as techniques for migrating data to and from the service.
Reasons to attend:
- Learn the details of Master Slave dual AZ configuration.
- Learn about cross region replication.
- Learn about Provisioned IOPS and tips on getting the most from your Amazon RDS MySQL Service.
Build clouds the way some of the world’s biggest public and private clouds are built—using CloudStack. This 60-minute webinar with the Cloudstack team will help you gain a better understanding of the CloudStack architecture and feature set.
An overview of the Netflix Security Monkey Open Source tool. The presentation provides some background information, architectural overview, and screenshots showing the tool in action.
Cloudstack European user group 11 april 2013ShapeBlue
The CloudStack European User group met on Thursday 11th for our quarterly meeting.
Giles Sirett (CEO of ShapeBlue) opened the meeting with his usuaal roundup of all things Cloudstack
The CloudStack European User group met on Thursday 11th for our quarterly meeting.
Oliver Leech (Platform Architect at Tata Communications) demonstrated using puppet to deploy applications within a CloudStack environment.
Scalable Media Processing in the Cloud (MED302) | AWS re:Invent 2013Amazon Web Services
The cloud empowers you to process media at scale in ways that were previously not possible, enabling you to make business decisions that are no longer constrained by infrastructure availability. Hear about best practices to architect scalable, highly available, high-performance workflows for digital media processing. In addition, this session covers AWS and partner solutions for transcoding, content encryption (watermarking and DRM), QC, and other processing topics.
Slides presented during the Strata SF 2019 conference. Explaining how Lyft is building a multi-cluster solution for running Apache Spark on kubernetes at scale to support diverse workloads and overcome challenges.
In this presentation we will look at how customers in the broadcast and media space are using AWS to innovate and serve their content to end users in new ways. We will cover how AWS is being used for media creation, storage, processing, delivery, and monetization of content.
After the session you will understand how to apply these technologies and techniques to your own business.
For more training on AWS, visit: https://www.qa.com/amazon
AWS Loft | London - Deep Dive: Amazon RDS by Toby Knight, Manager Solutions Architecture, 18 April 2016
AWS re:Invent 2016: Get Technically Inspired by Container-Powered Migrations ...Amazon Web Services
This session is a technical journey through application migration and refactoring using containerized technologies. Flux 7 recently worked with Rent-a-Center to perform a Hybris migration from their datacenter to AWS and you can hear how they used Amazon ECS, the new Application Load Balancer, and Auto Scaling to meet the customers' business objectives.
AWS re:Invent 2016: Optimizing Network Performance for Amazon EC2 Instances (...Amazon Web Services
Many customers are using Amazon EC2 instances to run applications with high performance networking requirements. In this session, we provide an overview of Amazon EC2 network performance features (enhanced networking, ENA, placement groups, etc.), and discuss how we are innovating on behalf of our customers to improve networking performance in a scalable and cost-efficient manner. We share best practices and performance tips for getting the best networking performance out of your Amazon EC2 instances.
Dev ops for big data cluster management toolsRan Silberman
What are the tools that we can find to day to manage Hadoop cluster and its ecosystem?
There are two tools ready today:
Cloudera Manager and Ambari from Hortonworks.
In this presentation I explain what they do and why to use them, as well as Pros. and Cons.
Responding to Digital Transformation With RDS Database TechnologyAlibaba Cloud
See Webinar Recording at https://resource.alibabacloud.com/webinar/detail.htm?webinarId=30
Learn how your business can gain an advantage and utilize ApsaraDB for RDS to enhance data flexibility, security, performance, and stability amid the backdrop of digital transformation.
Chen Zhaoshang of the Alibaba Cloud Database Product team will share how traditional industries can meet database challenges and leverage distributed architecture transformation using MySQL and other open technologies to reduce the overall cost of ownership and realize cross-data-center disaster tolerance deployment while ensuring data consistency. He will also share how to ensure database security databases and alleviate concerns about data migration to the cloud.
ApsaraDB for RDS: www.alibabacloud.com/product/apsaradb-for-rds
More Webinars: https://resource.alibabacloud.com/webinar/index.htm
AWS re:Invent 2016: Moving Mountains: Netflix's Migration into VPC (NET304)Amazon Web Services
Netflix was one of the earliest very large AWS customers. By 2014, we were running hundreds of applications in Amazon EC2. That was great, until we needed to move to VPC. Given our scale, uptime requirements, and the decentralized nature of how we manage our production environment, the VPC migration (still ongoing) presented particular challenges for us and for AWS as it sought to support our move. In this talk, we discuss the starting state, our requirements and the operating principles we developed for how we wanted to drive the migration, some of the issues we ran into, and how the tight partnership with AWS helped us migrate from an EC2-Classic platform to an EC2-VPC platform.
Day 3 - AWS MySQL Relational Database Service Best Practices for Performance ...Amazon Web Services
Amazon RDS makes it easy to set up, operate, and scale, relational databases in the cloud. Amazon RDS for MySQL supports applications that require up to tens of thousands of IOPS, and allows you to scale on demand without administrative complexity. In this webinar, we will discuss best practices for getting the most out of Amazon RDS for MySQL, as well as techniques for migrating data to and from the service.
Reasons to attend:
- Learn the details of Master Slave dual AZ configuration.
- Learn about cross region replication.
- Learn about Provisioned IOPS and tips on getting the most from your Amazon RDS MySQL Service.
Build clouds the way some of the world’s biggest public and private clouds are built—using CloudStack. This 60-minute webinar with the Cloudstack team will help you gain a better understanding of the CloudStack architecture and feature set.
An overview of the Netflix Security Monkey Open Source tool. The presentation provides some background information, architectural overview, and screenshots showing the tool in action.
Cloudstack European user group 11 april 2013ShapeBlue
The CloudStack European User group met on Thursday 11th for our quarterly meeting.
Giles Sirett (CEO of ShapeBlue) opened the meeting with his usuaal roundup of all things Cloudstack
The CloudStack European User group met on Thursday 11th for our quarterly meeting.
Oliver Leech (Platform Architect at Tata Communications) demonstrated using puppet to deploy applications within a CloudStack environment.
Spark Summit EU 2015: SparkUI visualization: a lens into your applicationDatabricks
Your application is slow but you don’t know why. It could be doing unnecessary shuffles, or evicting heavily-used cached data from memory, or suffering from data skew, or… With the recent visualization additions to the SparkUI, users can now quickly pinpoint bottlenecks in their applications and derive compelling insights about their usages of Spark. In this talk, we will walk through how to leverage these visuals to illuminate the design decisions of several example Spark applications. The applications showcased will include those that use SparkSQL, Spark Streaming, MLlib, and dynamic allocation.
Presentation faite lors du Hadoop User Group France du 14 janvier 2016.
L’analytique temps réel avec Riak et Spark par Michael Carney (Basho) et Olivier Girardot de Lateral Thoughts
Selon un rapport de Salesforce, le nombre de sources de données analysées par les entreprises progressera de 83% au cours des cinq prochaines années, ainsi les organisations veulent désormais fournir des connaissances en temps réel même sur les appareils mobiles. Le traitement temps réel est donc, le futur de l’analyse big data.
Ce talk présentera des nouveautés en matière de l’analyse temps réel autour de la famille SGBD Riak et Spark.
Michael Carney est le Directeur Commercial de Basho pour le Sud d’Europe. Fondateur de MySQL France et de MariaDB, Michael a rejoint Basho en janvier 2015 pour explorer le monde de données sans tables !
Olivier Girardot est le CTO de Lateral Thoughts, il est développeur et formateur au sujet de Spark et également spécialiste de Java/Python dans le domaine de la finance de marché.
Better, faster, cheaper infrastructure with apache cloud stack and riak cs reduxJohn Burwell
Software is eating infrastructure. Migrating reliability and
scalability responsibilities up the stack from specialized hardware to software, cloud orchestration platforms such as Apache CloudStack (ACS) and object stores such as Riak CS increase the utilization and density of compute and storage resources by dynamically shifting workloads based on demand. Together, these platform can saturate compute and storage of 1000s of commodity hosts with strong operational visibility and end-user self-service.
This presentation explores cloud design strategies to achieve high availability and reliability using commodity components. It then applies these strategies using Apache CloudStack and Riak CS.
Riak ( http://wiki.basho.com ), a Dynamo-inspired, open-source key/value datastore, was built to scale from a single machine to a 100+ server cluster without driving you or your operations team crazy. This presentation discusses the characteristics of Riak that become important in small, medium, and large clusters.
Riak ( http://wiki.basho.com ), a Dynamo-inspired, open-source key/value datastore, was built to scale from a single machine to a 100+ server cluster without driving you or your operations team crazy. This presentation discusses the characteristics of Riak that become important in small, medium, and large clusters.
AWS re:Invent 2016 - Scality's Open Source AWS S3 ServerScality
Presented by Giorgio Regni, CTO
Try Scality S3 Server Today!
https://s3.scality.com/
http://www.scality.com/scality-s3-server/
https://hub.docker.com/r/scality/s3server/
Leveraging Scala and Akka to build NSDb, a distributed open source time-series database.
The talk had been given by Saverio Veltri and Paolo Mascetti during ScalaItaly 2018 on the 14th September.
The code repository is hosted at https://github.com/radicalbit/NSDb
Apache Spark 2.0 set the architectural foundations of structure in Spark, unified high-level APIs, structured streaming, and the underlying performant components like Catalyst Optimizer and Tungsten Engine. Since then the Spark community has continued to build new features and fix numerous issues in releases Spark 2.1 and 2.2.
Continuing forward in that spirit, the upcoming release of Apache Spark 2.3 has made similar strides too, introducing new features and resolving over 1300 JIRA issues. In this talk, we want to share with the community some salient aspects of soon-to-be-released Spark 2.3 features:
• New deployment mode: Kubernetes scheduler backend
• PySpark performance and enhancements
• New structured streaming execution engine: continuous processing
• Data source v2 APIs for both structured streaming and Spark SQL
• ML on structured streaming
• Image reader
• Stable codegen engine
• Spark History Server V2
• Native ORC support
• Vectorized ORC and SQL cache readers
• Stream-stream Join
• UDF enhancements
• Various SQL enhancements
Speakers
Xiao Li, Software Engineer, Databricks
Wenchen Fan, Software Engineer, Databricks
Modernise Your Data Warehouse with Amazon Redshift and Amazon Redshift Spectrum
In this session you will learn how to migrate and modernise your legacy data warehouse, moving from an on-premises server or application, to the cloud. You will learn how to easily migrate your data by leveraging serverless ETL, data cataloging as well as the techniques needed to successfully modernise your data warehouse, reduce costs, and increase performance and scalability.
David McAmis, Big Data Consultant, AWS Professional Services, Amazon Web Services
Akka Streams And Kafka Streams: Where Microservices Meet Fast DataLightbend
In a recent survey, 90% of over 2400 developers reported having at least some real-time functionality in their systems. Enterprises are realizing that the ability to extract value from streaming data in near real-time is the new competitive advantage.
Two technologies–Akka Streams and Kafka Streams–have emerged as popular tools to use with Apache Kafka for addressing the shared requirements of availability, scalability, and resilience for both streaming microservices and Fast Data. So which one should you use for specific use cases?
Fully-managed Cloud-native Databases: The path to indefinite scale @ CNN MainzQAware GmbH
When it comes to the question: "Where do we actually store our application data?", we are spoilt for choice, especially when it comes to the major cloud providers.
The simple and often completely valid answer is still the classic relational database! It is very suitable for many areas of application, as the technology is tried and tested and can cover a very broad spectrum. It is therefore not surprising that all major cloud providers offer this as a "managed service".
For some years now, however, there have also been so-called cloud-native databases that have been specially developed for the requirements of the cloud. The big promise: "Infinite scalability"
In a large customer project, we have been using such a database productively for over 4 years with Azure CosmosDB. The presentation will deal with the following questions, among others
What does "upscalability" mean in practice ?
What do you have to pay attention to when designing?
What are the actual limits?
What other special features do I get?
When do I need a cloud-native database?
But that's not all! We also look beyond Azure to the other two major cloud providers: AWS and Google Cloud. With DynamoDB and Datastore/Firestore, they have similar products on offer.
CloudStack provides versatile authentication methods to ensure secure access and identity management. This talk explores key authentication mechanisms within CloudStack, including LDAP, SAML, OAuth2, API keys, etc. LDAP integration enables centralized user authentication, while SAML facilitates single sign-on (SSO) across various services. OAuth2 ensures secure authorization for third-party applications, and API keys offer programmatic access to resources. Additionally, CloudStack supports Two-Factor Authentication for an extra layer of security, enhancing user verification through multiple verification steps.
-----------------------------------------
The CloudStack India User Group 2024 took place in Hyderabad on 23rd February. The conference, arranged by a group of volunteers from the Apache CloudStack Community, saw multiple sessions held about the cloud orchestration platform and its latest advancements.
In this session, Kiran gives a talk about the rich ecosystem of tools (cmk, CAPC, Terraform, Ansible, Packer, csbench, mbx), that support Cloudstack.
Find out how the various tools work and how easy it is to integrate with Apache CloudStack.
This session provides a great way to speed up CloudStack adoption and improve performance by saving valuable time.
-----------------------------------------
The CloudStack India User Group 2024 took place in Hyderabad on 23rd February. The conference, arranged by a group of volunteers from the Apache CloudStack Community, saw multiple sessions held about the cloud orchestration platform and its latest advancements.
Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...ShapeBlue
In this session, Vishesh Jindal and Jithin Raju give a demonstration on Apache CloudStack's 4.19 marquee features - Object Storage, DRS, VM schedule & DRaaS.
-----------------------------------------
The CloudStack India User Group 2024 took place in Hyderabad on 23rd February. The conference, arranged by a group of volunteers from the Apache CloudStack Community, saw multiple sessions held about the cloud orchestration platform and its latest advancements.
VM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlueShapeBlue
The support for migrating VMware instances, and importing KVM instances to a CloudStack-managed KVM environment has been added to CloudStack 4.19.
In this talk, Suresh provides the details about the import/migration process in CloudStack along with a demo, and discusses the future improvements.
-----------------------------------------
The CloudStack India User Group 2024 took place in Hyderabad on 23rd February. The conference, arranged by a group of volunteers from the Apache CloudStack Community, saw multiple sessions held about the cloud orchestration platform and its latest advancements.
How We Grew Up with CloudStack and its Journey – Dilip Singh, DataHubShapeBlue
In this session, Senior IT Manager at DataHub Nepal, Dilip Singh, shares how DataHub grew up with CloudStack and details the journey the company had with the cloud orchestration platform.
-----------------------------------------
The CloudStack India User Group 2024 took place in Hyderabad on 23rd February. The conference, arranged by a group of volunteers from the Apache CloudStack Community, saw multiple sessions held about the cloud orchestration platform and its latest advancements.
What’s New in CloudStack 4.19, Abhishek Kumar, Release Manager Apache CloudSt...ShapeBlue
This session gives a brief introduction to the new and exciting feature in the latest CloudStack LTS release, ie, 4.19.0. The discussion includes the details on the timeline of the CloudStack 4.19.0 release, overview of some of the marquee, new feature of the release – Object storage framework, KVM ingestion, Hypervisor agnostic simple DRS, CAPC aware CKS, OAuth2, DRaaS with Multi zone disaster recovery, etc and a summary of improvements added since the previous major LTS release of the CloudStack, ie, 4.18.0.
-----------------------------------------
The CloudStack India User Group 2024 took place in Hyderabad on 23rd February. The conference, arranged by a group of volunteers from the Apache CloudStack Community, saw multiple sessions held about the cloud orchestration platform and its latest advancements.
CloudStack 101: The Best Way to Build Your Private Cloud – Rohit Yadav, VP Ap...ShapeBlue
Apache CloudStack is an open-source software designed to deploy and manage large networks of virtual machines, as a highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform. This talk gives an introduction to the technology, its architecture, its history and community.
-----------------------------------------
The CloudStack India User Group 2024 took place in Hyderabad on 23rd February. The conference, arranged by a group of volunteers from the Apache CloudStack Community, saw multiple sessions held about the cloud orchestration platform and its latest advancements.
How We Use CloudStack to Provide Managed Hosting - Swen Brüseke - proIOShapeBlue
Swen shows how proIO utilize Cloudstack to provide customers with managed hosting solutions and versatile public and private cloud solutions, mainly based on open-source software.
-----------------------------------------
The CloudStack Collaboration Conference 2023 took place on 23-24th November. The conference, arranged by a group of volunteers from the Apache CloudStack Community, took place in the voco hotel, in Porte de Clichy, Paris. It hosted over 350 attendees, with 47 speakers holding technical talks, user stories, new features and integrations presentations and more.
In cloud computing environments, VMs require fast access to resources like storage and networking. The hardware that the VMs access is implemented in software and/or by passing through a dedicated hardware device. Software-based solutions consume extra CPU cycles, thus resulting in poor performance. Also, these require to expose a device-model to the guest, thus increasing the attack surface. Conversely, hardware passthrough provides better performance and security but can be expensive in terms of the number of physical resources, since each device is dedicated to a single VM. This talk focuses on how Vates is working on sharing hardware resources among VMs by relying on dedicated processors named Data Processing Units (DPU). More precisely, Vates work on offloading Xen hypervisor of storage emulation by relying on Kalray K200 DPU PCIe controllers, a hardware accelerator based on MPPA architecture.
-----------------------------------------
The CloudStack Collaboration Conference 2023 took place on 23-24th November. The conference, arranged by a group of volunteers from the Apache CloudStack Community, took place in the voco hotel, in Porte de Clichy, Paris. It hosted over 350 attendees, with 47 speakers holding technical talks, user stories, new features and integrations presentations and more.
Zero to Cloud Hero: Crafting a Private Cloud from Scratch with XCP-ng, Xen Or...ShapeBlue
Dive into the seamless integration of the Vates stack as the foundation for your CloudStack deployment. In this workshop, you’ll witness the power and simplicity of XCP-ng and Xen Orchestra. From a blank slate to a fully operational private cloud, Olivier guides you through each pivotal step. Learn how to streamline your cloud setup process and unlock the potential of a private cloud infrastructure that’s both efficient and easy to manage. Watch to discover how to transform bare metal into a cloud powerhouse in mere minutes.
-----------------------------------------
The CloudStack Collaboration Conference 2023 took place on 23-24th November. The conference, arranged by a group of volunteers from the Apache CloudStack Community, took place in the voco hotel, in Porte de Clichy, Paris. It hosted over 350 attendees, with 47 speakers holding technical talks, user stories, new features and integrations presentations and more.
KVM Security Groups Under the Hood - Wido den Hollander - Your.OnlineShapeBlue
They are just a few clicks in the UI or a single API call, but how do security groups work at KVM hypervisor level? How do they filter traffic and what else do they do in addition to firewalling? What Anti-Spoofing policies are implemented by the security groups?
In this talk, Wido dives into the specifics of the security groups on the KVM hypervisor for both IPv4 and IPv6.
-----------------------------------------
The CloudStack Collaboration Conference 2023 took place on 23-24th November. The conference, arranged by a group of volunteers from the Apache CloudStack Community, took place in the voco hotel, in Porte de Clichy, Paris. It hosted over 350 attendees, with 47 speakers holding technical talks, user stories, new features and integrations presentations and more.
How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...ShapeBlue
CloudStack allows you to use older hardware for a longer time in your cloud environment. By using older hardware for a longer time you can save money and the environment by not producing new hardware.
-----------------------------------------
The CloudStack Collaboration Conference 2023 took place on 23-24th November. The conference, arranged by a group of volunteers from the Apache CloudStack Community, took place in the voco hotel, in Porte de Clichy, Paris. It hosted over 350 attendees, with 47 speakers holding technical talks, user stories, new features and integrations presentations and more.
Use Existing Assets to Build a Powerful In-house Cloud Solution - Magali Perv...ShapeBlue
How to minimize the impact when it’s time to implement a cloud solution for automating internal workloads and delivering efficient solutions? Magali, Joffrey, and Grégoire present a case study of a successful hardware reuse project, including key metrics: Business objectives, Performance objectives and Financial objectives.
-----------------------------------------
The CloudStack Collaboration Conference 2023 took place on 23-24th November. The conference, arranged by a group of volunteers from the Apache CloudStack Community, took place in the voco hotel, in Porte de Clichy, Paris. It hosted over 350 attendees, with 47 speakers holding technical talks, user stories, new features and integrations presentations and more.
Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...ShapeBlue
Ayush talks about his contribution as a GSoC Contributor, for implementing the Import-Export Instances feature for the KVM Hypervisor.
-----------------------------------------
The CloudStack Collaboration Conference 2023 took place on 23-24th November. The conference, arranged by a group of volunteers from the Apache CloudStack Community, took place in the voco hotel, in Porte de Clichy, Paris. It hosted over 350 attendees, with 47 speakers holding technical talks, user stories, new features and integrations presentations and more.
DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti...ShapeBlue
Apache CloudStack 4.19 introduces the capability for end-users to copy their root disk or volume snapshots to one (or more) ACS Zones without operator intervention. In this talk, Alex shows how this simple yet powerful new feature enables for end-users to control where their data resides and for operators to provide low-cost and robust DRaaS to their customers.
-----------------------------------------
The CloudStack Collaboration Conference 2023 took place on 23-24th November. The conference, arranged by a group of volunteers from the Apache CloudStack Community, took place in the voco hotel, in Porte de Clichy, Paris. It hosted over 350 attendees, with 47 speakers holding technical talks, user stories, new features and integrations presentations and more.
A discussion on the common failures when using CloudStack taking instance deployment as an example. The session includes 15 specific failure scenarios, their causes, and possible mitigation steps.
-----------------------------------------
The CloudStack Collaboration Conference 2023 took place on 23-24th November. The conference, arranged by a group of volunteers from the Apache CloudStack Community, took place in the voco hotel, in Porte de Clichy, Paris. It hosted over 350 attendees, with 47 speakers holding technical talks, user stories, new features and integrations presentations and more.
Elevating Privacy and Security in CloudStack - Boris Stoyanov - ShapeBlueShapeBlue
In an increasingly interconnected digital landscape, safeguarding data privacy and ensuring robust security measures are paramount. CloudStack offers a dynamic ecosystem for deploying and managing cloud resources. However, to fully harness its potential, it is crucial to address privacy and security concerns effectively.
This presentation explores the realm of possibilities and demonstrates how CloudStack can enhance the privacy and security of your cloud deployments. Boris examines practical approaches to protect sensitive data, fortify communications, and secure your infra against emerging threats. Join us on a journey to discover how CloudStack can be your trusted ally in the quest for a more secure and private cloud environment.
-----------------------------------------
The CloudStack Collaboration Conference 2023 took place on 23-24th November. The conference, arranged by a group of volunteers from the Apache CloudStack Community, took place in the voco hotel, in Porte de Clichy, Paris. It hosted over 350 attendees, with 47 speakers holding technical talks, user stories, new features and integrations presentations and more.
Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit...ShapeBlue
In this session, Marco explores the potential of migrating from VMware vCloud to Apache CloudStack with KVM. VMware vCloud Suite is a robust cloud infrastructure and management solution that combines vSphere and vRealize Suite, providing automation and operations capabilities for traditional and modern infrastructure and apps. However, the transition to Apache CloudStack can offer enhanced profitability and competitiveness.
Marco delves into the benefits of Apache CloudStack, including its cost-effectiveness and open-source nature, and discusses how a gradual migration from VMware vCloud can reduce ownership costs, increase profitability, and enhance competitiveness. He also covers the practical steps and considerations in planning and executing this transition effectively.
-----------------------------------------
The CloudStack Collaboration Conference 2023 took place on 23-24th November. The conference, arranged by a group of volunteers from the Apache CloudStack Community, took place in the voco hotel, in Porte de Clichy, Paris. It hosted over 350 attendees, with 47 speakers holding technical talks, user stories, new features and integrations presentations and more.
Vishesh has been working on the feature hypervisor-agnostic DRS in Cloudstack. He briefly overviews the implementation and discusses the algorithms currently available and how they can improve resource allocation and workload balancing in virtualized environments. Additionally, Vishesh showcases a live demo of hypervisor agnostic DRS in action, highlighting its capabilities and effectiveness.
-----------------------------------------
The CloudStack Collaboration Conference 2023 took place on 23-24th November. The conference, arranged by a group of volunteers from the Apache CloudStack Community, took place in the voco hotel, in Porte de Clichy, Paris. It hosted over 350 attendees, with 47 speakers holding technical talks, user stories, new features and integrations presentations and more.
What’s New in CloudStack 4.19 - Abhishek Kumar - ShapeBlueShapeBlue
This session gives a brief introduction of the new and exciting feature in the latest (upcoming) CloudStack LTS release, ie, 4.19.0. The discussion includes the details on the timeline of the CloudStack 4.19.0 release, overview of some of the marquee, new feature of the release – Object storage framework, KVM ingestion, Hypervisor agnostic simple DRS, CAPC aware CKS, OAuth2, DRaaS with Multi zone disaster recovery, etc and a summary of improvements added since the previous major LTS release of the CloudStack, ie, 4.18.0.
-----------------------------------------
The CloudStack Collaboration Conference 2023 took place on 23-24th November. The conference, arranged by a group of volunteers from the Apache CloudStack Community, took place in the voco hotel, in Porte de Clichy, Paris. It hosted over 350 attendees, with 47 speakers holding technical talks, user stories, new features and integrations presentations and more.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Essentials of Automations: The Art of Triggers and Actions in FMESafe Software
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
Removing Uninteresting Bytes in Software FuzzingAftab Hussain
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
Enhancing Performance with Globus and the Science DMZGlobus
ESnet has led the way in helping national facilities—and many other institutions in the research community—configure Science DMZs and troubleshoot network issues to maximize data transfer performance. In this talk we will present a summary of approaches and tips for getting the most out of your network infrastructure using Globus Connect Server.
6. Riak CS
is... storage
enterprise cloud
built g
in S3-compatibility
on top fe r
of o f
multi-tenancy
Riak per user reporting
large object storage
7. Enabling you to host your own
PUBLIC &
PRIVATE CLOUDS
or….
Reliable Storage Behind Apps
8. Basho's Commits
@john_burwell 's contribution:
S3-backed secondary storage feature in 4.1.0
Uses S3 to sync secondary storage across zones
Long term: (shhhhhh!)
Native S3 Support
Federated authentication and authorization
9. DataPipe
blog.datapipe.com/datapipe-cloudstack
“Riak CS provides the high-performance,
distributed datastore
we need to deliver a sound foundation for
our cloud storage needs now
and for many years into the future”
- Ed Laczynski, VP Cloud Strategy, Datapipe.
10. Yahoo!
“Today, Yahoo! leverages Riak CS Enterprise to offer an
S3-compatible public cloud storage service,
as well as dedicated hosting options ...
Yahoo! is highly supportive of open source software
and we view Basho’s (OSS) announcement as
a positive move that will work
to accelerate its ability to innovate
and ultimately strengthen our cloud platform.”
- Shingo Saito, cloud product manager, Yahoo!
12. Riak
Dynamo-inspired key/value store
Written in Erlang with C/C++
Open source under Apache 2 license
Thousands of production deployments
13. Riak
High availability
Low-latency
Horizontal scalability
Fault-tolerance
Ops friendliness
14. Riak
Masterless
• No master/slave or different roles
• All nodes are equal
• Write availability and scalability
• All nodes can accept/route requests
15. Riak
No Sharding
• Consistent hashing
• Prevents “hot spots”
• Lowers operational burden of scale
• Data rebalanced automatically
16. Riak
Availability and Fault-Tolerance
• Automatically replicates data
• Read and write data during hardware
failure and network partition
• Hinted handoff
20. Large Object 1. User uploads an
object
S3 Reporting S3 Reporting S3 Reporting S3 Reporting S3 Reporting
API API API API API API API API API API
Riak CS Riak CS Riak CS Riak CS Riak CS
1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB 1 MB
2. Riak CS
3. Riak CS Riak breaks object
streams chunks
Node
into 1 MB chunks
to Riak nodes Riak Riak
Node Node
Riak Riak
4. Riak replicates
Node Node and stores chunks
21. IC S
S T
BA EP
C
CON
USERS
multi-tenancy:
Riak CS will track
individual usage/stats
users identified by users authenticated by
access_key secret_key
22. IC S
S T
BA EP
C
CON
BUCKETS
users create buckets.
buckets are like folders.
store objects in buckets.
names are globally unique.
23. IC S
S T
BA EP
C
CON
OBJECTS
stored in buckets.
objects are opaque.
store any file type.
25. Riak CS
Large Object Support
• Started with 5GB / object
• Now have multipart upload
• Content agnostic
26. Riak CS
S3-Compatible API
• Use existing S3 libraries and tools
• RESTful operations
• Multipart upload
• S3-style ACLs for object/bucket
permissions
• S3 authentication scheme
27. Riak CS
Administration and Users
• Interface for user creation, deletion,
and credentials
• Configure so only admins can create
users
28. Riak CS
New Stuff in Riak 1.3
• Multipart upload: parts between 5MB
and 5GB
• Support for GET range queries
• Restrict access to buckets based on
source IP
34. THE
“USAGE”
BUCKET
TRACK INDIVIDUAL USER’S
ACCESS STORAGE
35. QUERY USAGE STATS
Storage and access statistics tracked on
per-user basis, as rollups for slices of time
•Operations, Count, BytesIn,
BytesOut, + system and user
error
•Objects, Bytes
37. Multi-Datacenter Replication
• For active backups, availability zones,
disaster recovery, global traffic
• Real-time or full-sync
• 24/7 support
• Per-node or storage-based pricing
38. SIGN UP FOR AN
ENTERPRISE DEVELOPER
TRIAL
basho.com
http://docs.basho.com/
Very high level discussion, segue into brief discussion of Riak
What you get is a platform on which you can host your own public and private clouds.
You can think of Riak in many ways as a distributed filesystem. Riak is awesome because all nodes are equal. Has distribution protocols that allows incredibly straightforward scaling and even balance of load by tokenizing a huge keyspace and using consistent hashing, etc. This however is not conducive to large object storage because of latency and other network limitations when moving files around during topology changes. Also, developing on top of Riak is non-trivial, so interactions with the database can be a pain. Riak CS abstracts away both the complexity of interactions via a simple, S3-compatible API as well as uses Riak ’ s inherent functionality to provide a solution for large object storage.
You can think of Riak in many ways as a distributed filesystem. Riak is awesome because all nodes are equal. Has distribution protocols that allows incredibly straightforward scaling and even balance of load by tokenizing a huge keyspace and using consistent hashing, etc. This however is not conducive to large object storage because of latency and other network limitations when moving files around during topology changes. Also, developing on top of Riak is non-trivial, so interactions with the database can be a pain. Riak CS abstracts away both the complexity of interactions via a simple, S3-compatible API as well as uses Riak ’ s inherent functionality to provide a solution for large object storage.
You can think of Riak in many ways as a distributed filesystem. Riak is awesome because all nodes are equal. Has distribution protocols that allows incredibly straightforward scaling and even balance of load by tokenizing a huge keyspace and using consistent hashing, etc. This however is not conducive to large object storage because of latency and other network limitations when moving files around during topology changes. Also, developing on top of Riak is non-trivial, so interactions with the database can be a pain. Riak CS abstracts away both the complexity of interactions via a simple, S3-compatible API as well as uses Riak ’ s inherent functionality to provide a solution for large object storage.
You can think of Riak in many ways as a distributed filesystem. Riak is awesome because all nodes are equal. Has distribution protocols that allows incredibly straightforward scaling and even balance of load by tokenizing a huge keyspace and using consistent hashing, etc. This however is not conducive to large object storage because of latency and other network limitations when moving files around during topology changes. Also, developing on top of Riak is non-trivial, so interactions with the database can be a pain. Riak CS abstracts away both the complexity of interactions via a simple, S3-compatible API as well as uses Riak ’ s inherent functionality to provide a solution for large object storage.
You can think of Riak in many ways as a distributed filesystem. Riak is awesome because all nodes are equal. Has distribution protocols that allows incredibly straightforward scaling and even balance of load by tokenizing a huge keyspace and using consistent hashing, etc. This however is not conducive to large object storage because of latency and other network limitations when moving files around during topology changes. Also, developing on top of Riak is non-trivial, so interactions with the database can be a pain. Riak CS abstracts away both the complexity of interactions via a simple, S3-compatible API as well as uses Riak ’ s inherent functionality to provide a solution for large object storage.
You can think of Riak in many ways as a distributed filesystem. Riak is awesome because all nodes are equal. Has distribution protocols that allows incredibly straightforward scaling and even balance of load by tokenizing a huge keyspace and using consistent hashing, etc. This however is not conducive to large object storage because of latency and other network limitations when moving files around during topology changes. Also, developing on top of Riak is non-trivial, so interactions with the database can be a pain. Riak CS abstracts away both the complexity of interactions via a simple, S3-compatible API as well as uses Riak ’ s inherent functionality to provide a solution for large object storage.
You can think of Riak in many ways as a distributed filesystem. Riak is awesome because all nodes are equal. Has distribution protocols that allows incredibly straightforward scaling and even balance of load by tokenizing a huge keyspace and using consistent hashing, etc. This however is not conducive to large object storage because of latency and other network limitations when moving files around during topology changes. Also, developing on top of Riak is non-trivial, so interactions with the database can be a pain. Riak CS abstracts away both the complexity of interactions via a simple, S3-compatible API as well as uses Riak ’ s inherent functionality to provide a solution for large object storage.
You can think of Riak in many ways as a distributed filesystem. Riak is awesome because all nodes are equal. Has distribution protocols that allows incredibly straightforward scaling and even balance of load by tokenizing a huge keyspace and using consistent hashing, etc. This however is not conducive to large object storage because of latency and other network limitations when moving files around during topology changes. Also, developing on top of Riak is non-trivial, so interactions with the database can be a pain. Riak CS abstracts away both the complexity of interactions via a simple, S3-compatible API as well as uses Riak ’ s inherent functionality to provide a solution for large object storage.
a Riak CS stack is composed of 3 critical components. Riak CS exposes an API to the users and is responsible for logging/tracking stats. All the data is stored in and retrieved from Riak. Run multiple instances of Riak and Riak CS for scale. Theres a third component, a single instance of a piece of software called stanchion that is responsible for tying it all together. Stanchion in essence provides the S3-like behavior at an architectural level, ensures user and bucket uniqueness globally, etc....
1-to-1 pairing, and why.
1. user PUTs object into Riak CS. The request will be via an S3 API and signed by their credentials. 2. once authenticated, object is chunked (remind why this is important) 3. as object is chunked, chunks sent to Riak. (you can use haproxy in the middle here) 4. Riak stores the chunks, yay!
Riak CS is multi-tenant. Each user is assigned an access_key and a secret_key. Users are authenticated by the system by signing requests using a combination of both keys. If the keys are valid, the requests will be allowed; else, denied. User details stored in “ user ” bucket, identified by access_key. Furthermore, every user ’ s activity will be tracked by Riak CS and stored for billing/metering purposes(more later)
Objects are stored in buckets. Users ’ s can create and remove buckets as well as list their contents. Buckets are essentially a namespace, and are very much like folders. Bucket names must be globally unique, so if you have two users both try to create a bucket named “ kittens ” , whoever creates that bucket first will own it, etc.
Put objects in buckets. Objects are chunked and replicated, but that all happens behind the scenes and not exposed to the user.
You can think of Riak in many ways as a distributed filesystem. Riak is awesome because all nodes are equal. Has distribution protocols that allows incredibly straightforward scaling and even balance of load by tokenizing a huge keyspace and using consistent hashing, etc. This however is not conducive to large object storage because of latency and other network limitations when moving files around during topology changes. Also, developing on top of Riak is non-trivial, so interactions with the database can be a pain. Riak CS abstracts away both the complexity of interactions via a simple, S3-compatible API as well as uses Riak ’ s inherent functionality to provide a solution for large object storage.
You can think of Riak in many ways as a distributed filesystem. Riak is awesome because all nodes are equal. Has distribution protocols that allows incredibly straightforward scaling and even balance of load by tokenizing a huge keyspace and using consistent hashing, etc. This however is not conducive to large object storage because of latency and other network limitations when moving files around during topology changes. Also, developing on top of Riak is non-trivial, so interactions with the database can be a pain. Riak CS abstracts away both the complexity of interactions via a simple, S3-compatible API as well as uses Riak ’ s inherent functionality to provide a solution for large object storage.
You can think of Riak in many ways as a distributed filesystem. Riak is awesome because all nodes are equal. Has distribution protocols that allows incredibly straightforward scaling and even balance of load by tokenizing a huge keyspace and using consistent hashing, etc. This however is not conducive to large object storage because of latency and other network limitations when moving files around during topology changes. Also, developing on top of Riak is non-trivial, so interactions with the database can be a pain. Riak CS abstracts away both the complexity of interactions via a simple, S3-compatible API as well as uses Riak ’ s inherent functionality to provide a solution for large object storage.
You can think of Riak in many ways as a distributed filesystem. Riak is awesome because all nodes are equal. Has distribution protocols that allows incredibly straightforward scaling and even balance of load by tokenizing a huge keyspace and using consistent hashing, etc. This however is not conducive to large object storage because of latency and other network limitations when moving files around during topology changes. Also, developing on top of Riak is non-trivial, so interactions with the database can be a pain. Riak CS abstracts away both the complexity of interactions via a simple, S3-compatible API as well as uses Riak ’ s inherent functionality to provide a solution for large object storage.
You can think of Riak in many ways as a distributed filesystem. Riak is awesome because all nodes are equal. Has distribution protocols that allows incredibly straightforward scaling and even balance of load by tokenizing a huge keyspace and using consistent hashing, etc. This however is not conducive to large object storage because of latency and other network limitations when moving files around during topology changes. Also, developing on top of Riak is non-trivial, so interactions with the database can be a pain. Riak CS abstracts away both the complexity of interactions via a simple, S3-compatible API as well as uses Riak ’ s inherent functionality to provide a solution for large object storage.
You can think of Riak in many ways as a distributed filesystem. Riak is awesome because all nodes are equal. Has distribution protocols that allows incredibly straightforward scaling and even balance of load by tokenizing a huge keyspace and using consistent hashing, etc. This however is not conducive to large object storage because of latency and other network limitations when moving files around during topology changes. Also, developing on top of Riak is non-trivial, so interactions with the database can be a pain. Riak CS abstracts away both the complexity of interactions via a simple, S3-compatible API as well as uses Riak ’ s inherent functionality to provide a solution for large object storage.
Riak CS provides stats on user activity and total cluster operations, as well as ships with DTrace probes you can use to inspect/debug a live system. So at any given time you can monitor a Riak CS cluster for both expected behavior and anomalies. From an administrative perspective, (as mentioned earlier) Riak CS will track each individual user ’ s activity, so that you can define usage limits and billing policies if necessary.
Riak CS, just like Riak, uses Boundary ’ s Folsom stats library for monitoring cluster operations. These start when Riak CS starts, are not persisted to disk. Get stats with an HTTP request to /riak-cs/stats. You ’ ll get back counters and histograms that track the total number of operations performed on blocks, buckets and objects. For instance, see the total number of GET or PUT operations on objects in the Riak CS cluster. These stats are going to be most useful if you ’ re trying to diagnose unexpected behavior. Hopefully that ’ s never the case, but shit happens.
RiakCS has a reserved namespace for tracking user activity. This is the “ usage ” bucket and is the foundation for metering and building custom billing policies in Riak CS. Every time a user performs an operation, RiakCS will store this data in an object in the usage bucket identified by that users ’ s access_key. You can configure the frequency with which these reports are persisted as well as the ability for user ’ s to request their own usage statistics.