In this presentation, we are going to take a look at the mechanism about how ceph storage is provided as persistence storage to docker container.
First of all, we are going to show the process how to deploy ceph storage using openstack kola and how we developed the docker volume plugin.
Scaling Development Environments with DockerDocker, Inc.
We set out to solve the problems of quickly building high quality games for a fragmented mobile market. Taking advantage of HTML5 allowed a fast, familiar and highly iterative local development process, and a hybrid build process for native apps meant high performance games on mobile. Our product is designed to comprehensively handle complex UI flows, related server tasks as well as deep integrations with any social platform. This is necessarily complex piece of engineering, with dozens of large dependencies, and 5 local web servers powering a single user’s experience. When we set out to make this easily available to 3rd parties, we used Docker to solve to major challenges: 1) Fitting many users, each with a unique development environment, on to one machine; 2) Managing all of these development environments in a scalable way.
In addition to authorization policies that control what a user can do, OpenShift Container Platform gives its administrators the ability to manage a set of security context constraints (SCCs) for limiting pods and securing their cluster.
Default security context may be too restrictive for containers pulled down from DockerHub, thorugh this talk we'll explore the various steps to execute for enabling required permissions on selected OpenShift's pods.
In this presentation, we are going to take a look at the mechanism about how ceph storage is provided as persistence storage to docker container.
First of all, we are going to show the process how to deploy ceph storage using openstack kola and how we developed the docker volume plugin.
Scaling Development Environments with DockerDocker, Inc.
We set out to solve the problems of quickly building high quality games for a fragmented mobile market. Taking advantage of HTML5 allowed a fast, familiar and highly iterative local development process, and a hybrid build process for native apps meant high performance games on mobile. Our product is designed to comprehensively handle complex UI flows, related server tasks as well as deep integrations with any social platform. This is necessarily complex piece of engineering, with dozens of large dependencies, and 5 local web servers powering a single user’s experience. When we set out to make this easily available to 3rd parties, we used Docker to solve to major challenges: 1) Fitting many users, each with a unique development environment, on to one machine; 2) Managing all of these development environments in a scalable way.
In addition to authorization policies that control what a user can do, OpenShift Container Platform gives its administrators the ability to manage a set of security context constraints (SCCs) for limiting pods and securing their cluster.
Default security context may be too restrictive for containers pulled down from DockerHub, thorugh this talk we'll explore the various steps to execute for enabling required permissions on selected OpenShift's pods.
In this talk, Damien describes the infrastructure Nuxeo has built around Docker containers, which is mainly based on CoreOS and Docker, and how it provides a way to generically run applications not only on a single host, but across a whole cluster of hosts. The resulting architecture can be used to implement a PaaS approach for any application.
Nowadays, the best strategy to put your containerized applications in production is to rely on a container orchestrator. But which one should you use? This talk will demystify the notion of container orchestrator and will compare some of them among the well known: Mesos, Kubernetes and Swarm. The main topics we will deal with are: what are the benefits of orchestrators, what are the potential constraints regarding the way you will need to develop your applications in an orchestrator context.
Docker at Shopify: From This-Looks-Fun to Production by Simon Eskildsen (Shop...Docker, Inc.
Since July 2014 Shopify's been serving thousands of requests per second of production web traffic from Docker containers. This was an 8 month effort, with multiple pivots of direction from the team—and we're only getting started. This talk covers the lessons learned through the trial and error of an in-flight architecture redesign, spanning hundreds of hosts, as well as the technical vision of the future of our platform.
Disaggregated Container Attached Storage - Yet Another Topology with What Pur...DoKC
The storage topology in vogue seems to cycle every few years. Internal storage is followed by centralized Storage Area Networks only to be superseded by one-size-fits-all Hyperconverged models - until scalability constraints led to distributed storage. Then comes NVMe, offering blistering speeds that all of these storage stacks struggle with. Kubernetes inspires Container Attached Storage aspiring to be the perfect model, so why is disaggregated storage now making an appearance?
This talk considers the motivations behind yet another storage topology and examines a modern, flexible architecture for delivering high-performance storage under Kubernetes.
This talk was given by Nick Connolly for DoK Day Europe @ KubeCon 2022.
Disaggregated Container Attached Storage - Yet Another Topology with What Pur...DoKC
Link: https://youtu.be/YhktX1W0geM
https://go.dok.community/slack
https://dok.community/
From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)
The storage topology in vogue seems to cycle every few years. Internal storage is followed by centralized Storage Area Networks only to be superseded by one-size-fits-all Hyperconverged models - until scalability constraints led to distributed storage. Then comes NVMe, offering blistering speeds that all of these storage stacks struggle with. Kubernetes inspires Container Attached Storage aspiring to be the perfect model, so why is disaggregated storage now making an appearance?
This talk considers the motivations behind yet another storage topology and examines a modern, flexible architecture for delivering high-performance storage under Kubernetes.
-----
Nick Connolly is a pioneer of storage virtualisation and the Chief Scientist at DataCore, where his background in real-time computing and multiprocessing led to the creation of a world-class high-performance storage stack on Windows. He holds patents ranging from highly scalable algorithms through to data protection techniques. Recently he has been working with OpenEBS to bring the power and performance of NVMe to Kubernetes.
In this talk, Damien describes the infrastructure Nuxeo has built around Docker containers, which is mainly based on CoreOS and Docker, and how it provides a way to generically run applications not only on a single host, but across a whole cluster of hosts. The resulting architecture can be used to implement a PaaS approach for any application.
Nowadays, the best strategy to put your containerized applications in production is to rely on a container orchestrator. But which one should you use? This talk will demystify the notion of container orchestrator and will compare some of them among the well known: Mesos, Kubernetes and Swarm. The main topics we will deal with are: what are the benefits of orchestrators, what are the potential constraints regarding the way you will need to develop your applications in an orchestrator context.
Docker at Shopify: From This-Looks-Fun to Production by Simon Eskildsen (Shop...Docker, Inc.
Since July 2014 Shopify's been serving thousands of requests per second of production web traffic from Docker containers. This was an 8 month effort, with multiple pivots of direction from the team—and we're only getting started. This talk covers the lessons learned through the trial and error of an in-flight architecture redesign, spanning hundreds of hosts, as well as the technical vision of the future of our platform.
Disaggregated Container Attached Storage - Yet Another Topology with What Pur...DoKC
The storage topology in vogue seems to cycle every few years. Internal storage is followed by centralized Storage Area Networks only to be superseded by one-size-fits-all Hyperconverged models - until scalability constraints led to distributed storage. Then comes NVMe, offering blistering speeds that all of these storage stacks struggle with. Kubernetes inspires Container Attached Storage aspiring to be the perfect model, so why is disaggregated storage now making an appearance?
This talk considers the motivations behind yet another storage topology and examines a modern, flexible architecture for delivering high-performance storage under Kubernetes.
This talk was given by Nick Connolly for DoK Day Europe @ KubeCon 2022.
Disaggregated Container Attached Storage - Yet Another Topology with What Pur...DoKC
Link: https://youtu.be/YhktX1W0geM
https://go.dok.community/slack
https://dok.community/
From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)
The storage topology in vogue seems to cycle every few years. Internal storage is followed by centralized Storage Area Networks only to be superseded by one-size-fits-all Hyperconverged models - until scalability constraints led to distributed storage. Then comes NVMe, offering blistering speeds that all of these storage stacks struggle with. Kubernetes inspires Container Attached Storage aspiring to be the perfect model, so why is disaggregated storage now making an appearance?
This talk considers the motivations behind yet another storage topology and examines a modern, flexible architecture for delivering high-performance storage under Kubernetes.
-----
Nick Connolly is a pioneer of storage virtualisation and the Chief Scientist at DataCore, where his background in real-time computing and multiprocessing led to the creation of a world-class high-performance storage stack on Windows. He holds patents ranging from highly scalable algorithms through to data protection techniques. Recently he has been working with OpenEBS to bring the power and performance of NVMe to Kubernetes.
Percona Live 4/14/15: Leveraging open stack cinder for peak application perfo...Tesora
In this session, speakers Amrith Kumar (Tesora), Steven Walchek (SolidFire), and Chris Merz (SolidFire) discuss Cinder, the OpenStack block storage service, and OpenStack Trove.
My talk at ScaleConf 2017 in Cape Town on some tips and tactics for scaling WordPress, with reference to WordPress.com and the container-based VIP Go platform.
Video of my talk is here: https://www.youtube.com/watch?v=cs0DcY80spw
Leveraging OpenStack Cinder for Peak Application PerformanceNetApp
Deploying performance sensitive, database-driven applications in OpenStack can be tenuous if you are unsure how to utilize the Cinder API to get the most out of your OpenStack block storage.
This presentation:
Introduces Cinder, the OpenStack block storage service
Talks about the unique attributes of performance-sensitive applications and what this means in OpenStack
Walks you through how to use Cinder volume types and extra specs to guarantee performance to your various cloud workloads
Discusses OpenStack Trove and what it means for running database as a service in your OpenStack cloud
Today, most any application can be “Dockerized.” However, there are special challenges when deploying a distributed application such as Spark on containers. This session will describe how to overcome these challenges in deploying Spark on Docker containers, with many practical tips and techniques for running Spark in a container environment.
Containers are typically used to run stateless applications on a single host. There are significant real-world enterprise requirements that need to be addressed when running a stateful, distributed application in a secure multi-host container environment.
There are decisions that need to be made concerning which tools and infrastructure to use. There are many choices with respect to container managers, orchestration frameworks, and resource schedulers that are readily available today and some that may be available tomorrow including:]
• Mesos
• Kubernetes
• Docker Swarm
Each has its own strengths and weaknesses; each has unique characteristics that may make it suitable, or unsuitable, for Spark. Understanding these differences is critical to the successful deployment of Spark on Docker containers.
This session will describe the work done by the BlueData engineering team to run Spark inside containers, on a distributed platform, including the evaluation of various orchestration frameworks and lessons learned. You will learn how to apply practical networking and storage techniques to achieve high performance and agility in a distributed, container environment.
Speaker
Thomas Phelan, Chief Architect, Blue Data, Inc
Latest (storage IO) patterns for cloud-native applications OpenEBS
Applying micro service patterns to storage giving each workload its own Container Attached Storage (CAS) system. This puts the DevOps persona within full control of the storage requirements and brings data agility to k8s persistent workloads. We will go over the concept and the implementation of CAS, as well as its orchestration.
What is coming for VMware vSphere?
Delivered at VMUG DK/UK/BE in November 2014. Session is all about vSphere futures, what can be expected in the near future.
Whats new in Microsoft Windows Server 2016 Clustering and StorageJohn Moran
In this webinar we will learn what the High Availability & Storage team in Microsoft has cooked up for us in Windows Server 2016, which is being launched at Microsoft Ignite at the end of September.
There’s lots of new stuff in this release, including better high availability for Hyper-V, greater control over resource utilization, improved fault tolerance of transient events, newer design options for stretch or multi-site clusters, a whole new way of doing software defined storage with SATA and NVMe drives, built-in block-level storage replication, and hyper-convergence without having to break the bank.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
4. What is Container?
• VM vs Container
• OS Level Virtualization
• Performance
• Run Everywhere – Portable
• Scalable – lightweight
• Environment Consistency
• Easy to manage Image
• Easy to deploy
Docker Container Virtual Machine
Reference : https://www.docker.com/what-container
5. What is Docker Container?
• VM vs Container
• OS Level Virtualization
• Performance
• Run Everywhere – Portable
• Scalable – lightweight
• Environment Consistency
• Easy to manage Image
• Easy to deploy
namespace
cgroup
isolation
resource
limiting
CPU MEM DISK NWT
Reference : https://www.slideshare.net/PhilEstes/docker-london-container-security
6. What is Docker Container?
• VM vs Container
• OS Level Virtualization
• Performance
• Run Everywhere – Portable
• Scalable – lightweight
• Environment Consistency
• Easy to manage Image
• Easy to deploy
Reference : https://www.theregister.co.uk/2014/08/18/docker_kicks_kvms_butt_in_ibm_tests/
7. What is Docker Container?
• VM vs Container
• OS Level Virtualization
• Performance
• Run Everywhere – Portable
• Scalable – lightweight
• Environment Consistency
• Easy to manage Image
• Easy to deploy
Reference : https://www.ibm.com/blogs/bluemix/2015/08/c-ports-docker-containers-across-multiple-clouds-datacenters/
PM / VM / Cloud
8. What is Docker Container?
• VM vs Container
• OS Level Virtualization
• Performance
• Run Everywhere – Portable
• Scalable – lightweight
• Environment Consistency
• Easy to manage Image
• Easy to deploy
App A
Libs
Original Image
A’
Changed Image Deployment
A’App A
Libs
change update
9. What is Docker Container?
• VM vs Container
• OS Level Virtualization
• Performance
• Run Everywhere – Portable
• Scalable – lightweight
• Environment Consistency
• Easy to manage Image
• Easy to deploy
Reference : http://www.devopsschool.com/slides/docker/docker-web-development/index.html#/17
10. What is Docker Container?
• VM vs Container
• OS Level Virtualization
• Performance
• Run Everywhere – Portable
• Scalable – lightweight
• Environment Consistency
• Easy to manage Image
• Easy to deploy
Reference : https://www.slideshare.net/insideHPC/docker-for-hpc-in-a-nutshell
12. Stateless vs Stateful Container
• Nothing to write on disk
• Web (Front-end)
• Easy to scale in/out
• Container is ephemeral
• If delete, will be lost data
Stateless
…
Easy to scale out
• Needs storage to write
• Database
• Logs
• CI config / repo data
• Secret Keys
Stateful
Hard to scale out
…
13. Ephemeral vs Persistent
Ephemeral Storage
• Data Lost
• Local Storage
• Stateless Container
Persistent Storage
• Data Save
• Network Storage
• Stateful Container
VS
Reference : https://www.infoworld.com/article/3106416/cloud-computing/containerizing-stateful-applications.html
14. Needs for Persistent Storage
Our mission is to provide “Persistence Storage Service”
while maintaining “Agility” & “Automation” of Docker Container
15. macvlan
IPVS IPVS
vrrp
Docker swarm cluster
ECMP enabled
BGP/ECMP
internet
consul
consul consul
Dynamic DNS
AA
MON MON MON OSD
OSD OSD OSD OSD
OSD OSD OSD OSD
OSD OSD OSD OSD
OSD OSD OSD OSD
KEYSTONE
CINDER
Manage Volume
PLUGIN
Get Auth-Token
Request
Attach Volume
/dev/rbd0
PLUGIN
DEVIEW 2016 : https://www.slideshare.net/deview/221-docker-orchestration
OpenStack 2017 : https://www.slideshare.net/JangseonRyu/on-demandblockstoragefordocker-77895159
OSD
Container in NAVER
27. ◼ Strong Consistency
◼ CAP Theorem : CP system
(Consistency / network Partition)
Client
Primary OSD
Secondary OSD Tertiary OSD
①
② ②
③ ③
④
Write Flow
② Primary OSD sends data to replica OSDs,
write data to local disk.
④ Primary OSD signals completion to client.
③ Replica OSDs write data to local disk,
signal completion to primary.
① Client writes data to primary osd.
28. ◼ All of Data Disks are SATA.
◼ 7.2k rpm SATA : ~ 75 ~ 100 IOPS
① Client writes data to primary osd.
② Primary OSD sends data to replica OSDs,
write data to local disk.
③’ Write data to local disk,
Send ack to primary.
③’’ Slow write data to local disk,
Send ack to primary.
④ Slow send ack to client.
Client
Primary OSD
Secondary OSD Tertiary OSD
①
② ②
③’ ③’’
④
SATA
High Disk Usage
Slow Requests
29. Client
Primary OSD
①
② ②
③ ③
④
Journal FileStore
Secondary OSD
Journal FileStore
Tertiary OSD
Journal FileStore
Primary OSD
Journal
SSD
O_DIRECT
O_DSYNC
FileStore
SATA
Buffered IOs
XFS
Journal on SSD
31. Network
Switch (10G)
Clients
Ceph
Public Networks Cluster Networks
KRBD
- /dev/rbd0
- mkfs / mount
- fio
Ceph
- FileStore
- Luminous (12.2.0)
Server
- Intel® Xeon CPU L5640 2.27 GHz
- 16 GB Memory
- 480GB SAS 10K x 5
- 480GB SAS SSD x 1
- Centos 7
Test Environment
32. Expect Result : 4K Rand Write
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS
SAS : 10,000 rpm / 600 IOPS
Per Node : SAS * 3 = 600 * 3 = 1,800 IOPS
Total : Node * 3 = 1,800 * 3 = 5,400 IOPS
Replicas 3 = 5,400 / 3 = 1,800 IOPS
600
1800
2094
34.36
0
500
1000
1500
2000
2500
IOPS Latency
No SSD for Journal
2094 IOPS/s
Case #1. Only Use SAS DISK
33. Expect
SAS
SAS
SAS
SSD : 15,000 IOPS
Total : Node * 3 = 15,000* 3 = 45,000 IOPS
Replicas 3 = 45,000 / 3 = 15,000 IOPS
SSD
600
15k
SAS
SAS
SAS
SSD
SAS
SAS
SAS
SSD
2094 à 2273
2273
31.65
0
500
1000
1500
2000
2500
IOPS Latency
SSD for Journal
Case #2. Use SSD for Journal
Result : 4K Rand Write
41. ◼ High performance improvements with SSD : 2,094 à 14,264 (x7)
◼ Must need Throttling for stable storage operation : Dynamic Throttle
◼ Need to tune OS(page cache, io scheduler), Ceph config
Conculsion