In a brisk presentation, we introduce Zenko, the multi-cloud data controller. Zenko is a stack of microservices (Docker containers) deployed via either Kubernetes or Docker Swarm.
The presentation comprised a demo deployment, so make sure to check out the video.
There is a lot of extra material at the end of the presentation going more in depth in topics that were only touched on briefly during the presentation.
talked by CI/CD Conference 2021 by CloudNative Days https://event.cloudnativedays.jp/cicd2021
re-upload: https://speakerdeck.com/whywaita/cyberagent-oss-cicd-myshoes-cicd2021
talked by CI/CD Conference 2021 by CloudNative Days https://event.cloudnativedays.jp/cicd2021
re-upload: https://speakerdeck.com/whywaita/cyberagent-oss-cicd-myshoes-cicd2021
Building clouds with apache cloudstack apache roadshow 2018ShapeBlue
Talk given at Apache Roadshow, FOSS Backstage, Berlin, June 2018
Apache CloudStack is open source software designed to deploy and manage large networks of virtual machines, as a highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform. This talk will give an introduction to the technology, its history and its architecture. It will look common use-cases (and some real production deployments) that are seen across both public and private cloud infrastructures and where CloudStack can be completed by other open source technologies.
The talk will also compare and contrast Apache Cloudstack with other IaaS platforms and why he thinks that the technology, combined with the Apache governance model will see CloudStack become the de-facto open source cloud platform. He will run a live demo of the software and talk about ways that people can get involved in the Apache CloudStack project.
Who carries your container? Zun or Magnum?Madhuri Kumari
There are multiple solution in OpenStack to enable containers. These slides talk about two projects i.e. Magnum and Zun in OpenStack and their use cases.
Guaranteeing Storage Performance by Mike Tutkowskibuildacloud
This session will introduce the basics of primary storage in CloudStack. Additionally, I discuss the challenges of guaranteeing storage performance in a cloud and how by leveraging the latest enhancements to CloudStack, storage administrators can deliver consistent, repeatable performance to 10s, 100s or 1,000s of application workloads in parallel. I'll review the CloudStack enhancements in detail, outline the management benefits they provide and discuss common go-to-market approaches.
About Mike Tutkowski
Mike Tutkowski, a member of the CloudStack PMC, develops software for the Apache Software Foundation's CloudStack project to help drive improvements in its storage component and to integrate SolidFire more deeply into the product.
Zenko & MetalK8s @ Dublin Docker Meetup, June 2018Laure Vergeron
A presentation of the Zenko project, that gives you a single namespace and a single API (the S3 API) across all of your clouds (AWS S3, MS Azure, GCP, Wasabi, Digital Ocean, and private clouds), and that lets you perform metadata seach across all these backends.
We also cover MetalK8s, an opinionated release of Kubernetes committed to bare metal environments, and growing on Kuberspray.
Building clouds with apache cloudstack apache roadshow 2018ShapeBlue
Talk given at Apache Roadshow, FOSS Backstage, Berlin, June 2018
Apache CloudStack is open source software designed to deploy and manage large networks of virtual machines, as a highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform. This talk will give an introduction to the technology, its history and its architecture. It will look common use-cases (and some real production deployments) that are seen across both public and private cloud infrastructures and where CloudStack can be completed by other open source technologies.
The talk will also compare and contrast Apache Cloudstack with other IaaS platforms and why he thinks that the technology, combined with the Apache governance model will see CloudStack become the de-facto open source cloud platform. He will run a live demo of the software and talk about ways that people can get involved in the Apache CloudStack project.
Who carries your container? Zun or Magnum?Madhuri Kumari
There are multiple solution in OpenStack to enable containers. These slides talk about two projects i.e. Magnum and Zun in OpenStack and their use cases.
Guaranteeing Storage Performance by Mike Tutkowskibuildacloud
This session will introduce the basics of primary storage in CloudStack. Additionally, I discuss the challenges of guaranteeing storage performance in a cloud and how by leveraging the latest enhancements to CloudStack, storage administrators can deliver consistent, repeatable performance to 10s, 100s or 1,000s of application workloads in parallel. I'll review the CloudStack enhancements in detail, outline the management benefits they provide and discuss common go-to-market approaches.
About Mike Tutkowski
Mike Tutkowski, a member of the CloudStack PMC, develops software for the Apache Software Foundation's CloudStack project to help drive improvements in its storage component and to integrate SolidFire more deeply into the product.
Zenko & MetalK8s @ Dublin Docker Meetup, June 2018Laure Vergeron
A presentation of the Zenko project, that gives you a single namespace and a single API (the S3 API) across all of your clouds (AWS S3, MS Azure, GCP, Wasabi, Digital Ocean, and private clouds), and that lets you perform metadata seach across all these backends.
We also cover MetalK8s, an opinionated release of Kubernetes committed to bare metal environments, and growing on Kuberspray.
AWS re:Invent 2016 - Scality's Open Source AWS S3 ServerScality
Presented by Giorgio Regni, CTO
Try Scality S3 Server Today!
https://s3.scality.com/
http://www.scality.com/scality-s3-server/
https://hub.docker.com/r/scality/s3server/
Zenko: Enabling Data Control in a Multi-cloud WorldScality
Watch the webinar replay here:
http://www.zenko.io/webinar
Announcing New Scality Open Source Data Controller, Zenko
How to simplify your data management and get a global view across clouds?
Zenko, the new multi-cloud data controller by Scality, provides a unified interface across clouds. This allows any cloud to be accessed with the same API and access layer. It can run anywhere in physical, virtualized or cloud environments.
Zenko builds on the success of Scality Cloud Server, the open-source implementation of the Amazon S3 API, which has enjoyed more than half a million DockerHub downloads since it was introduced in June 2016. Scality is releasing this new code to the open source community, under an Apache 2.0 license, so that any developer can use and extend Zenko in their development.
Stackato presentation done at the Nordic Perl Workshop 2012 in Stockholm, Sweden
More information available at: https://logiclab.jira.com/wiki/display/OPEN/Stackato
Confluent On Azure: Why you should add Confluent to your Azure toolkit | Alic...HostedbyConfluent
As a data professional, you are the glue that makes cross-platform integrations possible. With the increase in adoption of hybrid cloud architectures, Kafka is an increasingly relevant tool for building data pipelines between platforms and accelerating delivery on cloud projects. Early exposure to Kafka on Azure capabilities gives you an edge to build better mousetraps at the design phase.
Customers already running Kafka on premises and are looking to extend Kafka systems to Azure can get started quickly with Confluent Cloud. Additionally, DevOps for self-managed options can be easily scalable with Ansible for Virtual Machines or containers via Azure Kubernetes Services or Azure Container Instances.
This session is presented from the Microsoft Solution Architect perspective by Israel Ekpo, Microsoft Cloud Solution Architect and Alicia Moniz, Microsoft MVP. They will cover use cases and scenarios, along with key Azure integration points and architecture patterns.
My Stackato presentation given to the CopenhagenJS user group. Basic examples were implemented in Node.
More information available at: https://logiclab.jira.com/wiki/display/OPEN/Stackato
Presentation of ActiveStates micro-cloud solution Stackato at Open Source Days 2012.
Stackato is a cloud solution from renowned ActiveState. It is based on the Open Source CloudFoundry and offers a serious cloud solution for Perl programmers, but also supports Python, Ruby, Node.js, PHP, Clojure and Java.
Stackato is very strong in the private PaaS area, but do also support as public PaaS and deployment onto Amazon's EC2.
The presentation will cover basic use of Stackato and the reason for using a PaaS, public as private. Stackato can also be used as a micro-cloud for developers supporting vSphere, VMware Fusion, Parallels and VirtualBox.
Stackato is currently in public beta, but it is already quite impressive in both features and tools. Stackato is not Open Source, but CloudFoundry is and Stackato offers a magnificent platform for deployment of Open Source projects, sites and services.
ActiveState has committed to keeping the micro-cloud solution free so it offers an exciting capability and extension to the developers toolbox and toolchain.
More information available at: https://logiclab.jira.com/wiki/display/OPEN/Stackato
OCCIware presentation at EclipseDay in Lyon, November 2017, by Marc Dutoo, SmileOCCIware
Presentation title: Model and pilot all cloud layers with OCCIware, from IoT to Big Data
Abstract: Who uses multi cloud today ? Everybody. Alas, this leads to a lot of "technical glue". Enter OCCIware's Studio and Runtime : manage all layers and domains of the Cloud (XaaS) in a uniform, standard, extensible way - the Cloud consumer platform.presentation.
This talk presents how the OCCIware Studio - currently being contributed to the Eclipse Foundation by Inria and Obeo - takes advantage of Eclipse Modeling and SIrius in order to support a metamodel for the generic Open Cloud Computing Interface (OCCI) REST API and build a "studio factory", while providing feedback and lessons learned on various other Eclipse components.
It concludes on a live demonstration of using it to model and pilot an IoT (nodeMCU/ESP8266), Linked & Big Data (JSON-LD, Spark), containerized Cloud solution to let electricity consumption be monitored across territories by all actors - individuals, utility providers, up to regional public bodies.
Model and pilot all cloud layers with OCCIware - Eclipse Day Lyon 2017Marc Dutoo
Who uses multi cloud today ? Everybody. Alas, this leads to a lot of "technical glue". Enter OCCIware's Studio and Runtime : manage all layers and domains of the Cloud (XaaS) in a uniform, standard, extensible way - the Cloud consumer platform.
This talk presents how the OCCIware Studio - currently being contributed to the Eclipse Foundation by Inria and Obeo - takes advantage of Eclipse Modeling and SIrius in order to support a metamodel for the generic Open Cloud Computing Interface (OCCI) REST API and build a "studio factory", while providing feedback and lessons learned on various other Eclipse components.
It concludes on a live demonstration of using it to model and pilot an IoT (nodeMCU/ESP8266), Linked & Big Data (JSON-LD, Spark), containerized Cloud solution to let electricity consumption be monitored across territories by all actors - individuals, utility providers, up to regional public bodies.
Convergence of Containers and Serverless by Mency WooTriNimbus
Presentation slides from the Vancouver AWS User Group Meetup on Jan. 29, 2019 with Mency Woo. Title is "The Convergence of Containers and Severless on AWS"
Eager to learn more about OpenStack? This presentation provides an overview of OpenStack basics and an introduction to the types of storage in OpenStack. Choosing the right storage for your cloud can be the hardest part of building out your environment – this is a great primer to picking the right storage for your OpenStack deployment.
Similar to Docker Meetup Tokyo #23 - Zenko Open Source Multi-Cloud Data Controller - Laure Vergeron (20)
Large Language Models and the End of ProgrammingMatt Welsh
Talk by Matt Welsh at Craft Conference 2024 on the impact that Large Language Models will have on the future of software development. In this talk, I discuss the ways in which LLMs will impact the software industry, from replacing human software developers with AI, to replacing conventional software with models that perform reasoning, computation, and problem-solving.
Cyaniclab : Software Development Agency Portfolio.pdfCyanic lab
CyanicLab, an offshore custom software development company based in Sweden,India, Finland, is your go-to partner for startup development and innovative web design solutions. Our expert team specializes in crafting cutting-edge software tailored to meet the unique needs of startups and established enterprises alike. From conceptualization to execution, we offer comprehensive services including web and mobile app development, UI/UX design, and ongoing software maintenance. Ready to elevate your business? Contact CyanicLab today and let us propel your vision to success with our top-notch IT solutions.
Experience our free, in-depth three-part Tendenci Platform Corporate Membership Management workshop series! In Session 1 on May 14th, 2024, we began with an Introduction and Setup, mastering the configuration of your Corporate Membership Module settings to establish membership types, applications, and more. Then, on May 16th, 2024, in Session 2, we focused on binding individual members to a Corporate Membership and Corporate Reps, teaching you how to add individual members and assign Corporate Representatives to manage dues, renewals, and associated members. Finally, on May 28th, 2024, in Session 3, we covered questions and concerns, addressing any queries or issues you may have.
For more Tendenci AMS events, check out www.tendenci.com/events
In software engineering, the right architecture is essential for robust, scalable platforms. Wix has undergone a pivotal shift from event sourcing to a CRUD-based model for its microservices. This talk will chart the course of this pivotal journey.
Event sourcing, which records state changes as immutable events, provided robust auditing and "time travel" debugging for Wix Stores' microservices. Despite its benefits, the complexity it introduced in state management slowed development. Wix responded by adopting a simpler, unified CRUD model. This talk will explore the challenges of event sourcing and the advantages of Wix's new "CRUD on steroids" approach, which streamlines API integration and domain event management while preserving data integrity and system resilience.
Participants will gain valuable insights into Wix's strategies for ensuring atomicity in database updates and event production, as well as caching, materialization, and performance optimization techniques within a distributed system.
Join us to discover how Wix has mastered the art of balancing simplicity and extensibility, and learn how the re-adoption of the modest CRUD has turbocharged their development velocity, resilience, and scalability in a high-growth environment.
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERRORTier1 app
Even though at surface level ‘java.lang.OutOfMemoryError’ appears as one single error; underlyingly there are 9 types of OutOfMemoryError. Each type of OutOfMemoryError has different causes, diagnosis approaches and solutions. This session equips you with the knowledge, tools, and techniques needed to troubleshoot and conquer OutOfMemoryError in all its forms, ensuring smoother, more efficient Java applications.
Custom Healthcare Software for Managing Chronic Conditions and Remote Patient...Mind IT Systems
Healthcare providers often struggle with the complexities of chronic conditions and remote patient monitoring, as each patient requires personalized care and ongoing monitoring. Off-the-shelf solutions may not meet these diverse needs, leading to inefficiencies and gaps in care. It’s here, custom healthcare software offers a tailored solution, ensuring improved care and effectiveness.
Enhancing Research Orchestration Capabilities at ORNL.pdfGlobus
Cross-facility research orchestration comes with ever-changing constraints regarding the availability and suitability of various compute and data resources. In short, a flexible data and processing fabric is needed to enable the dynamic redirection of data and compute tasks throughout the lifecycle of an experiment. In this talk, we illustrate how we easily leveraged Globus services to instrument the ACE research testbed at the Oak Ridge Leadership Computing Facility with flexible data and task orchestration capabilities.
Code reviews are vital for ensuring good code quality. They serve as one of our last lines of defense against bugs and subpar code reaching production.
Yet, they often turn into annoying tasks riddled with frustration, hostility, unclear feedback and lack of standards. How can we improve this crucial process?
In this session we will cover:
- The Art of Effective Code Reviews
- Streamlining the Review Process
- Elevating Reviews with Automated Tools
By the end of this presentation, you'll have the knowledge on how to organize and improve your code review proces
May Marketo Masterclass, London MUG May 22 2024.pdfAdele Miller
Can't make Adobe Summit in Vegas? No sweat because the EMEA Marketo Engage Champions are coming to London to share their Summit sessions, insights and more!
This is a MUG with a twist you don't want to miss.
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptxrickgrimesss22
Discover the essential features to incorporate in your Winzo clone app to boost business growth, enhance user engagement, and drive revenue. Learn how to create a compelling gaming experience that stands out in the competitive market.
How to Position Your Globus Data Portal for Success Ten Good PracticesGlobus
Science gateways allow science and engineering communities to access shared data, software, computing services, and instruments. Science gateways have gained a lot of traction in the last twenty years, as evidenced by projects such as the Science Gateways Community Institute (SGCI) and the Center of Excellence on Science Gateways (SGX3) in the US, The Australian Research Data Commons (ARDC) and its platforms in Australia, and the projects around Virtual Research Environments in Europe. A few mature frameworks have evolved with their different strengths and foci and have been taken up by a larger community such as the Globus Data Portal, Hubzero, Tapis, and Galaxy. However, even when gateways are built on successful frameworks, they continue to face the challenges of ongoing maintenance costs and how to meet the ever-expanding needs of the community they serve with enhanced features. It is not uncommon that gateways with compelling use cases are nonetheless unable to get past the prototype phase and become a full production service, or if they do, they don't survive more than a couple of years. While there is no guaranteed pathway to success, it seems likely that for any gateway there is a need for a strong community and/or solid funding streams to create and sustain its success. With over twenty years of examples to draw from, this presentation goes into detail for ten factors common to successful and enduring gateways that effectively serve as best practices for any new or developing gateway.
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...Juraj Vysvader
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I didn't get rich from it but it did have 63K downloads (powered possible tens of thousands of websites).
top nidhi software solution freedownloadvrstrong314
This presentation emphasizes the importance of data security and legal compliance for Nidhi companies in India. It highlights how online Nidhi software solutions, like Vector Nidhi Software, offer advanced features tailored to these needs. Key aspects include encryption, access controls, and audit trails to ensure data security. The software complies with regulatory guidelines from the MCA and RBI and adheres to Nidhi Rules, 2014. With customizable, user-friendly interfaces and real-time features, these Nidhi software solutions enhance efficiency, support growth, and provide exceptional member services. The presentation concludes with contact information for further inquiries.
How Recreation Management Software Can Streamline Your Operations.pptxwottaspaceseo
Recreation management software streamlines operations by automating key tasks such as scheduling, registration, and payment processing, reducing manual workload and errors. It provides centralized management of facilities, classes, and events, ensuring efficient resource allocation and facility usage. The software offers user-friendly online portals for easy access to bookings and program information, enhancing customer experience. Real-time reporting and data analytics deliver insights into attendance and preferences, aiding in strategic decision-making. Additionally, effective communication tools keep participants and staff informed with timely updates. Overall, recreation management software enhances efficiency, improves service delivery, and boosts customer satisfaction.
Navigating the Metaverse: A Journey into Virtual Evolution"Donna Lenk
Join us for an exploration of the Metaverse's evolution, where innovation meets imagination. Discover new dimensions of virtual events, engage with thought-provoking discussions, and witness the transformative power of digital realms."
Check out the webinar slides to learn more about how XfilesPro transforms Salesforce document management by leveraging its world-class applications. For more details, please connect with sales@xfilespro.com
If you want to watch the on-demand webinar, please click here: https://www.xfilespro.com/webinars/salesforce-document-management-2-0-smarter-faster-better/
Docker Meetup Tokyo #23 - Zenko Open Source Multi-Cloud Data Controller - Laure Vergeron
1. DOCKER TOKYO - 2018-05-15
Zenko: the
Multi-Cloud Data
ControllerWe enable people who create value with data
by making AWS storage cheaper
Laure
Vergeron
Technology Evangelist
R&D Engineer
3. Agenda
1 - What is multi-cloud?
2 - Zenko: introduction to the multi-cloud data controller
3 - Zenko Orbit: introduction to the multi-cloud management UI
4 - CloudServer standalone: a simple local S3 endpoint
5 - Zenko S3 API: basically just AWS S3 API
6 - Zenko Enterprise Edition: coming later in 2018
7 - Zenkommunity
8 - Demos!
6. ▪Content Distribution
▪ Media companies have tens of thousands of movies, which they store on Private Cloud for
control. When it is time to publish a movie, it makes sense to copy it to a public cloud to use
its transcoding and CDN services.
▪Compute Bursting
▪ Banks have to do risk analysis leveraging thousands of CPU every night. These intense
computation only run for a few hours. Rather than having idle servers for the rest of the day, it
makes sense to use Public Cloud services for the computation
▪Analytics
▪ E-commerce company do more and more machine learning on their very large data lake.
Rather than setting up Hadoop infrastructures in-house, the company can copy just a data set
to an Hadoop cloud, compute the appropriate algorithm, and get back the result and destroy
the cloud copy of the data to save on storage cost.
▪Long-term Archival / cold storage
▪ While storing data which is regularly accessed is cheaper in a private cloud, long term archive
of never accessed data is cheaper in long term archive cloud offering. Automatic archival of
never accessed data would save a lot of money.
Examples of Use-Cases for Multi-Cloud
12. • Single Interface to any Cloud
▪ S3 API as a single API set to any cloud
• Allow reuse in the Cloud
▪ Maintain the native cloud format
• Always know your data and where it is
▪ Metadata search
• Trigger actions based on data
▪ Data Workflow to manage replication, location
The Zenko Multi-Cloud Data Controller
12
14. Zenko: one endpoint, one API, one namespace, any cloud in native format
● One API: the S3 API
● One endpoint: your Zenko
endpoint
● Any cloud
currently supported: Google Cloud
Platform, Microsoft Azure,
Amazon S3, Wasabi, Digital
Ocean Spaces, Scality RING,
local storage
● Data stored in native format:
use services native to your
cloud; Zenko does not lock
you in!
15. METADATA
DATA STORAGE
DMD REST/Sproxyd AWS S3 API CLOUD API
Shared Local
Storage
S3 API
APP
METADATA
APP
S3 CALLS
Zenko Open Source
S3 API—Single API set and 360° access to any cloud
Native format—Data written through Zenko is stored
in the native format of the target cloud storage and
can be read directly, without going through Zenko.
Backbeat, your data workflow manager —
Policy-based data management engine
Leveraging MongoDB for metadata search—
Agregated metadata across all clouds in a replicated
MongoDB gives a search tool for optimal data insight
HA/Failover – Deployed as multiple containers for
resilience via metal-k8s
Simple Security – single-tenant credentials
managed locally
S3 API
S3 CALLS
METADATA DATA
MONGO DB
Metadata Search
Bucket LOCATION
BACKBEAT
Data Policy Engine
Bucket LOCATION
CRR/DATADATA
Zenko: a stack of microservices
17. Zenko: Orbit management UI
● https://admin.zenko.io
● Free to use until 1TB
● Can either
○ get a sandbox
○ register own
instance
● Available as a
pay-as-you-go service for
large capacities
● No support with free
edition
20. 20
CloudServer: What?
• S3 API served in a Docker Container
• Written in Node.js
• 100% open source (Apache 2.0)
• S3 stands for Simple Storage Service. The S3
API provides a simple interface used to store
objects in buckets.
• Single AWS S3 API interface to access
multiple backend data storage both
on-premise or public in the cloud.
21. Open Source Scality CloudServer Adoption
▪Launched June 2016
▪Open-source
implementation of AWS S3
API
▪Code available on Github
under Apache 2.0 license
▪Packaged in Docker
container for easy
deployment
▪Seamless upgrade to S3
Connector for the RING
Now Over
1,000,000
25. • Zenko EE is not yet available
– We are doing our first beta deployments
at large American customers
– We are looking at Q1 2019 for GA
• Zenko CE is readily available, with
community support
– https://forum.zenko.io
– Plenty of documentation on readthedocs
– Send an email to your SE or to
zenko@scality.com
• Zenko Orbit is available pay-as-you-go
– First TB of data managed is free
25
Zenko EE : COMING SOON! Just not yet...
27. Community Meetups
• Initiated prior to our S3 Server launch
• At Docker Tokyo on May 15th
• At Scality Tokyo Open Source Night on May 16th
• Participating at open source events
for Docker, Nodejs, etc...
Developer Hackathons
• Paris and San Francisco in 2015-2018 ; maybe Tokyo next?
• Co-sponsoring with partners – focused on a specific project goal
(e.g., IP Drives, Backblaze integrations)
• Great for building visibility & community participation
27
Building a Developer community
28. How can I get involved with Zenko?
• Let us know what you do with Zenko stack!
▪ zenko@scality.com
▪ Get your project/company featured on the website in a quote
• Contribute tutorials
▪ Get a blogpost featuring your introduction of your tutorial
▪ Become part of our readTheDocs hosted documentation
• Contribute code
▪ It’s an opportunity to drive the roadmap with us !
▪ Join the team and be part of the Zenko craze !
▪ We have Contributing Guidelines on the GitHub repos, and we’ll answer your
questions via GitHub issues or our forum forum.zenko.io
• Meet us at AWS Re:invent, DockerCon, Meetups...
▪ All info is on www.zenko.io
28
30. 30
Ready to join us?
• Create an account on our Forum
• http://forum.zenko.io/
• Clone Zenko, and its microservices
• https://github.com/scality/Zenko/
• https://github.com/scality/S3/
• https://github.com/scality/backbeat
• Install s3cmd and AWS CLI
• Read the docs ;)
• Start with a minikube deployment
• https://github.com/scality/Zenko/blob/master/charts/minikube.md
• Reach out on the Forum as you have questions
• Try a full bare-metal-kubespray deployment
• https://github.com/scality/Zenko/tree/master/charts
31. DIY and Demo
Get your own Zenko sandbox
Deploy Zenko on Minikube and register it on Orbit
36. Zenko: one namespace, any cloud
Zenko local Zenko to Public Cloud AWS
Region / Location Region Public Cloud “bucket” Region
Bucket Bucket Prefix in public cloud bucket Bucket
$ aws --profile zenko s3 mb s3://remote-bucket --region aws-zenkobucket
make_bucket: s3://mybucket
$ aws --profile zenko s3 cp /etc/hosts s3://remote-bucket/test
upload: /etc/hosts to s3://remote-bucket/test
$ aws --profile zenko s3 ls
2018-05-14 17:08:50 remote-bucket
$ aws --profile zenko s3 ls s3://remote-bucket
2018-05-14 17:09:18 235 test
$ aws --profile aws s3 ls
2018-05-14 16:00:53 zenkobucket
$ aws --profile aws s3 ls s3://zenkobucket
PRE remote-bucket/
2018-05-14 17:09:18 235 test
37. Zenko: a stack of microservices
$ docker stack services ls
ID NAME MODE REPLICAS IMAGE PORTS
1j8jb41llhtm zenko-prod_s3-data replicated 1/1 zenko/cloudserver:pensieve-3 *:30010->9991/tcp
3y7vayna97bt zenko-prod_s3-front replicated 1/1 zenko/cloudserver:pensieve-3 *:30009->8000/tcp
957xksl0cbge zenko-prod_mongodb-init replicated 0/1 mongo:3.6.3-jessie
cn0v7cf2jxkb zenko-prod_queue replicated 1/1 wurstmeister/kafka:1.0.0 *:30008->9092/tcp
jjx9oabeugx1 zenko-prod_mongodb replicated 1/1 mongo:3.6.3-jessie *:30007->27017/tcp
o530bkuognu5 zenko-prod_lb global 1/1 zenko/loadbalancer:latest *:80->80/tcp
r69lgbue0o3o zenko-prod_backbeat-api replicated 1/1 zenko/backbeat:pensieve-4
ut0ssvmi10tx zenko-prod_backbeat-consumer replicated 1/1 zenko/backbeat:pensieve-4
vj2fr90qviho zenko-prod_cache replicated 1/1 redis:alpine *:30011->6379/tcp
vqmkxu7yo859 zenko-prod_quorum replicated 1/1 zookeeper:3.4.11 *:30006->2181/tcp
y7tt98x7jdl9 zenko-prod_backbeat-producer replicated 1/1 zenko/backbeat:pensieve-4
[...]
Zenko: Multi-Cloud Data Controller
Cloudserver Backbeat Utapi
S3 API
Multi Cloud API translation
Event-driven data manager
Replication engine
Usage Stats
Custom node.js Kafka- & Zookeeper- -based Redis-based
Bare-Metal Kubespray: custom deployment of Kubespray
39. CloudServer implements the AWS S3 Bucket Versioning API
• Create a versioned Bucket (PUT Bucket Versioning) – enables Bucket to maintain object versions
• If an object with the same key is PUT or DELETED, it becomes the current version
– DELETE marker used for indicating current version deleted as per AWS semantics
– Assigns version IDs for older versions
Enables Data Restores
• Access to previous states/versions of an object (GET object with specified version ID)
Required for both CRR & Lifecycle Management APIs
• As specified in S3 API
• When writing to AWS S3, the target bucket must have
versioning enabled!
39
Zenko S3 API: Bucket Versioning
40. ● When versioning is enabled on a bucket:
● CREATE NEW VERSIONS:
○ Put Object, Complete Multipart Upload and Object Copy (to a versioning-enabled bucket) will
return a version id in the ‘x-amz-version-id’ response header.
○ No special syntax necessary.
● When versioning is enabled or suspended:
● TARGETING SPECIFIC VERSIONS:
○ Include the version id in the request query for GET/HEAD Object or PUT/GET Object ACL
■ Example: `GET [bucket]/[object]?versionId=[versionId]`
○ For Object Copy or Upload Copy Part, to copy a specific version from a version-enabled
bucket, add the version id to the ‘x-amz-copy-source’ header:
■ Example value: `[sourcebucket]/[sourceobject]?versionId=[versionId]`
○ Omitting a specific version will get the result for the latest / current version.
Zenko S3 API: Bucket Versioning
41. ● When versioning is enabled or suspended (cont.):
● NULL VERSIONS:
○ Null versions are created when putting an object before versioning is configured or when
versioning is suspended.
■ Only one null version is maintained in version history.
New null versions will overwrite previous null versions.
○ Target the null version in version-specific actions by specifying the version ID ‘null’.
● DELETING OBJECTS:
○ Regular deletion of objects will create delete markers and return ‘x-amz-delete-marker’: ‘true’
and the version ID of the delete marker in ‘x-amz-version-id’ response headers.
○ Objects with delete markers as the latest version will behave as if they have been deleted when
performing non-version specific actions.
○ Permanently remove delete markers or specific versions by specifying the version ID in the
request query. Example: `DELETE [bucket]/[object]?versionId=[versionId]`
Zenko S3 API: Bucket Versioning
42. ● When versioning is enabled or suspended (cont.):
● MULTI-OBJECT DELETE:
○ Specify the specific version of an object to delete in a multi-object delete request in the XML
body. Example: http://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html
● At any time:
● LISTING OBJECTS:
○ A regular listing will list the most recent versions of an object and ignore objects with delete
markers as their latest version.
○ To list all object versions and delete markers in a bucket, specify ‘versions’ in request query:
■ Example: `GET [bucket]?versions`
○ FMI about output: consult S3 Connector documentation
● GET BUCKET VERSIONING STATUS: use Get Bucket Versioning API.
Zenko S3 API: Bucket Versioning
43. ● Utapi can be accessed through a REST API with service available on a dedicated port
● API routes use AWS Signature Version V4 for authentication
● API calls for listing metrics can use account credentials or create an IAM user with policy allowing
to list metrics
● Metrics are collected in 15 minute intervals (not configurable)
● Requests for listing metrics use POST routes and require at least start time and a list of resources
(accounts/buckets/users)
● Refer to wiki here for listing metrics https://github.com/scality/utapi#listing-metrics-with-utapi
Extended S3 API: UTAPI: UTilization API
44. S3 Buckets with associated Location
•Assigned as an optional request parameter
“LocationConstraint”
•In PUT Bucket API command, application can
specify a location for each Bucket
•Enables S3 Connector to manage Buckets across
multiple RINGs for scaling or access to multiple DC’s
•Enables Zenko to access multiple Public Clouds
Location Mapping
- Configuration file to manage multiple location to
multiple backends mappings
- Defines the default location for object PUTs
- Object GET Access is transparent
44
Zenko S3 API: Bucket Location Control
46. Zenko Multi-Cloud Async Replication - CRR
Remote disaster recovery (DR) for WAN environments
- Follows the AWS S3 API “cross-region replication” (CRR) API
- Async bucket replication: source Bucket -> target Bucket
- Versioning must be enabled on source & target
Target bucket in S3/RING
- CRR to remote S3/RING – in current release
CRR Features:
- Full site sync
- Bucket to bucket sync
- Monitoring statistics (throughput, backlog, RTO/RPO)
- Failback
- CRR from 1 region to many others (one public cloud to several others)
46
50. 50
File or object?
Why we do file:
- We know it
- Easy hierarchy
- fopen() and fclose()
- Lots of best practices
- Perf of NAS / over LAN
Why we do object:
- Billions of entries
- Storage accessed over
WAN
- For modern apps (REST)
- Listing large volumes
52. 52
CloudServer tree structure CheatSheet
S3/locationConfig.json and S3/config.json - setup your own endpoints
S3/lib/server.js - your entrypoint into the service
Arsenal/lib/s3routes/routes/*.js - S3 route calls
S3/lib/api/{{yourAPIcommand}} - S3 API calls
S3/lib/data/wrapper.js & multipleBackendGateway.js - gateway to external
clients
S3/lib/data/external/*Client.js - current clients
S3/conf/authData.json - setup your own credentials
Arsenal/lib/storage/metadata/* - check how metadata works
53. These commands assume you have S3 cloned locally, s3cmd configured for your S3 server, AWS
cli configured for a real AWS bucket, and your locationConfig set up
- START SERVER:
S3BACKEND=mem S3DATA=multiple npm start
- MAKE BUCKET:
s3cmd mb s3://[bucket-name]
- PUT OBJECT TO SPECIFIC LOCATION:
s3cmd put [/path/to/file] s3://[bucket-name]/[object-name] --add-header
x-amz-meta-scal-location-constraint:‘[location-name]’
- LIST OBJECTS IN BUCKET:
s3cmd ls s3://[bucket-name]/[object-name]
- GET S3 OBJECT METADATA:
s3cmd info s3://[bucket-name]/[object-name]
- IF PUT TO AWS, LIST OBJECTS ON AWS:
aws s3api list-objects --bucket [bucket-name]
Start S3 Server & Put Object Commands