SlideShare a Scribd company logo
1 of 67
Download to read offline
Zenko x Tanker Hackathon @ 42
Technical Deep Dive
The Prizes
Parrot Mambo Mission Drone
(+ accessories, flypad)
Arduino Genuino Kit
(15 tutorials, 214 pieces)
OTHA Pico Projector
(32GB, Miracast, HD, accessories)
1st Prize 2nd Prize 3rd Prize
Logistics
Where?
> At 42 Paris
When?
> October 20th: tech deep dive
> October 21st-November 5th: Slack support
> November 6th-10th (Scality presence 09:42->17:42)
How?
> PHP Piscine
> Scality expertise, lunch, your energy, your creativity
> Tanker expertise
Project Ideas
Name Description # of people
MongoDB backend Provide a MongoDB backend for S3 server/Zenko 3
Basic Vault (Accounts
only) Write an auth module for accounts (create/delete); use vault-cli 2
Encrypted PouchDB Use Tanker to store encrypted stuff in PouchDB 2-3
B2 as a backend Backblaze publishes its API : https://www.backblaze.com/b2/docs/ 3
Tanker encrypted git Write a git filter to encrypt git objects 3-4
Encrypted storage Suggested usecase : photo gallery 3
IPFS as a backend
Follow up on 42 Fremont project:
https://github.com/ssalaues/zenko-ipfs-module
3
Add your own ideas by commenting at:
Zenko hackathon ideas
Check your hackathon page (and soon, your team’s profile):
Zenko Hackathon Webpage
Agenda for today
Introduction to Zenko and CloudServer
Introduction to Tanker
Get started as a developer - Zenko
Get started as a developer - Tanker
Live deployment of Zenko
Live demo of Tanker
Introduction to
the MultiCloud world
and the Zenko stack
7
File or object?
8
File or object?
9
File or object?
Why we do file:
- We know it
- Easy hierarchy
- fopen() and fclose()
- Lots of best practices
- Perf of NAS / over LAN
Why we do object:
- Billions of entries
- Storage accessed over
WAN
- For modern apps (REST)
- Listing large volumes
▪
▪
▪
▪
▪
▪
▪
▪
▪
▪
▪
▪
12
13
CloudServer (formerly S3Server)
•
What is CloudServer?
• Open source object storage server
• Written in Node.js
• Single instance running in a Docker
container
• Uses Docker volumes for persistent
storage
• Same code as Scality’s RING S3
interface
14
●
○
○
●
○
○
○
○
●
○
○
○
○
15
UTAPI (Utilization API)
16
Backbeat
➢
➢
17
Zenko
Zenko stack:
Getting started as a dev
19
Ready, steady?
20
CloudServer tree structure CheatSheet
These commands assume you have S3 cloned locally, s3cmd configured for your S3 server, AWS
cli configured for a real AWS bucket, and your locationConfig set up
-
Start CloudServer & Put Object Commands CheatSheet
-
-
Deploy Zenko CheatSheet
CloudServer Highlights: Bucket Location Control
aws --endpoint-url http://127.0.0.1:8000 s3api create-bucket --bucket test-bucket --create-bucket-configuration
LocationConstraint=scality-1
aws --endpoint-url http://127.0.0.1:8000 s3 cp /etc/hosts s3://test-bucket/hosts
• Specify bucket location at bucket creation
• Object creation remains as usual
PUT / HTTP/1.1
Host: {{BucketName}}.{{StorageService}}.com
Content-Length: {{length}}
Date: {{date}}
Authorization: {{authenticationInformation}}
<CreateBucketConfiguration xmlns="http://s3.scality.com/doc/2006-03-01/">
<LocationConstraint>scality-1</LocationConstraint>
</CreateBucketConfiguration>
• Request syntax
● When versioning is enabled on a bucket:
● CREATE NEW VERSIONS:
○ Put Object, Complete Multipart Upload and Object Copy (to a versioning-enabled bucket) will
return a version id in the ‘x-amz-version-id’ response header.
○ No special syntax necessary.
● When versioning is enabled or suspended:
● TARGETING SPECIFIC VERSIONS:
○ Include the version id in the request query for GET/HEAD Object or PUT/GET Object ACL
■ Example: `GET [bucket]/[object]?versionId=[versionId]`
○ For Object Copy or Upload Copy Part, to copy a specific version from a version-enabled
bucket, add the version id to the ‘x-amz-copy-source’ header:
■ Example value: `[sourcebucket]/[sourceobject]?versionId=[versionId]`
○ Omitting a specific version will get the result for the latest / current version.
CloudServer Highlights: object versioning
● When versioning is enabled or suspended (cont.):
● NULL VERSIONS:
○ Null versions are created when putting an object before versioning is configured or when
versioning is suspended.
■ Only one null version is maintained in version history.
New null versions will overwrite previous null versions.
○ Target the null version in version-specific actions by specifying the version ID ‘null’.
● DELETING OBJECTS:
○ Regular deletion of objects will create delete markers and return ‘x-amz-delete-marker’: ‘true’
and the version ID of the delete marker in ‘x-amz-version-id’ response headers.
○ Objects with delete markers as the latest version will behave as if they have been deleted when
performing non-version specific actions.
○ Permanently remove delete markers or specific versions by specifying the version ID in the
request query. Example: `DELETE [bucket]/[object]?versionId=[versionId]`
CloudServer Highlights: object versioning
CloudServer Highlights: object versioning
● When versioning is enabled or suspended (cont.):
● MULTI-OBJECT DELETE:
○ Specify the specific version of an object to delete in a multi-object delete request in the XML
body. Example: http://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html
● At any time:
● LISTING OBJECTS:
○ A regular listing will list the most recent versions of an object and ignore objects with delete
markers as their latest version.
○ To list all object versions and delete markers in a bucket, specify ‘versions’ in request query:
■ Example: `GET [bucket]?versions`
○ FMI about output: consult S3 Connector documentation
● GET BUCKET VERSIONING STATUS: use Get Bucket Versioning API.
CloudServer Highlights: UTAPI
Utapi was meant to be used programmatically. For debugging, we have included a nifty tool in CloudServer’s
docker container. It can be accessed as follows
• docker exec -it <container id> bash
• the cmd is available as node bin/list_metrics
• Sample command
node bin/list_metrics -a myAccessKey -k mySecretKey 
--buckets demo -s 1490230800000 -h 127.0.0.1 -p 8100
This would list bucket level metrics for bucket demo starting from 2017-03-23T01:00:00.000Z
• Note: Since metrics are stored in 15 minute increment intervals, Utapi server requires start time to be the
start of nearest 15 minute interval. For example, valid start times would look like 09:00:00:000,
09:15:00:000, 09:30:00:000 and 09:45:00:000
End time needs to be the end of the nearest 15 minute interval. For example, valid end times would look
like 09:14:59:999, 09:29:59:999, 09:44:59:999 and 09:59:59:999
CloudServer Highlights: AWS S3 as a backend
...
"aws-test-2": {
"type": "aws_s3",
"legacyAwsBehavior": true,
"details": {
"awsEndpoint": "s3.amazonaws.com",
"bucketName": "multitester222",
"bucketMatch": true,
"credentialsProfile": "default_2"
}
}
...
CloudServer Highlights: Microsoft Azure as a backend
...
"azure-test": {
"type": "azure",
"legacyAwsBehavior": false,
"details": {
"azureBlobEndpoint": "https://zenkomeetups.blob.core.windows.net/",
"bucketMatch": true,
"azureBlobSAS": "{{YOUR AZURE SAS}}",
"azureContainerName": "meetupscontainer"
}
}
...
Deploying Zenko
Prerequisites
- 2 machines (or more)
- a copy of the Zenko git repository
How to do it (if you’ve not read the doc…):
- bundle these machines together in a Docker Swarm cluster;
- apply a specific tag to the machine which will physically host the data and metadata
- add your credentials to the secret.txt file
- edit the docker-stack.yml file to set the number of replicas (global vs replicated)
- export your endpoint as an environment variable
- docker stack deploy -c docker-stack.yml {{stack-name}}
Example: 3 managers, 2 workers
Lb (Nginx)
Redis
Lb (Nginx) Lb (Nginx)
Lb (Nginx)
Data
Metadata
Lb (Nginx)
CloudServer
Leader
$> docker node inspect {{id}} -f '{{ .Spec.Labels }}'
map[io.zenko.type:storage]
Failure: 3 managers, 2 workers
Lb (Nginx)
Redis
Lb (Nginx) Lb (Nginx)
Lb (Nginx)
Data
Metadata
Lb (Nginx)
CloudServer
Failure: 2 managers, 2 workers, lost leader
Lb (Nginx)
Redis
Lb (Nginx)
Lb (Nginx)
Data
Metadata
Lb (Nginx)
CloudServer
Failure: 3 managers, 2 workers, election
Lb (Nginx)
Redis
Lb (Nginx)
Lb (Nginx)
Data
Metadata
Lb (Nginx)
CloudServer
Failure: 2 managers, 2 workers, new leader
Lb (Nginx)
Redis
Lb (Nginx)
Lb (Nginx)
Data
Metadata
Lb (Nginx)
CloudServer
Lb (Nginx)
Nominal: 3 managers, 2 workers, new leader
Lb (Nginx)
Redis
Lb (Nginx)
Lb (Nginx)
Data
Metadata
Lb (Nginx)
CloudServer
Lb (Nginx) Lb (Nginx)
Tanker SDK
Core concepts
Async Javascript
Getting started
Tanker ensures confidentiality and
authenticity of resources among a defined
set of parties
4
0
Confidentiality
Parties can be users or servers
4
1
Parties
Parties can have multiple devices
Each device
has its own key pair
4
2
Multi-device
A resource represents
a piece of data shared by
a set of parties
Resources can be of any type:
● A document
● A record in a database
● A data structure ...
43Resource
Resources can be shared
44Sharing
45Symmetric and asymmetric keys
●Collection of linked blocks
●Each block contains a payload
○ (custom binary format)
●Each block has an author
●Each block is signed
●Verified by both client and server
46Trustchain
User ID
Tanker user IDs should match those of your existing
app or service.
Note: they are obfuscated for security reasons
Core concepts
Async Javascript
Getting started
Javascript SDK
● Tanker SDK is written in Javascript
● Two flavours:
○ Node (“server” side)
○ Browser (client side)
Asynchronous
In both cases, it is not recommended to wait for I/O.
Prehistory: callbacks
Callback hell
Renaissance: Promises (1)
Renaissance: Promises (2)
Modern times: async / await
Modern times: async / await
Just don’t forget the await ...
Core concepts
Async Javascript
Getting started
Read the fine documentation
https://beta.tanker.io/docs
Requirements
● Node 8.0 or better
● yarn
Create a trustchain
● Send a trustchain creation request on Slack
● Someone from Tanker staff will create a Trustchain
for you
● Store the private key and the ID in a safe place
Creating users - the safe way
You should use a delegation token generated by your
service, so that user additions blocks are signed
properly.
Create your first user - the easy way
Here the delegation token is generated for you by the
SDK.
But the private key of the Trustchain is readable by the
client ...
Encrypt and share
On Alice’s device:
Then send encryptedData to Bob somehow
Decrypt
On Bob’s device:
Demo !
Get the sources on github:
github.com/TankerApp/hello-42
We also have a blog :)
https://dev.to/tanker
Thank you!

More Related Content

What's hot

Scala, docker and testing, oh my! mario camou
Scala, docker and testing, oh my! mario camouScala, docker and testing, oh my! mario camou
Scala, docker and testing, oh my! mario camouJ On The Beach
 
Who is afraid of privileged containers ?
Who is afraid of privileged containers ?Who is afraid of privileged containers ?
Who is afraid of privileged containers ?Marko Bevc
 
What Have Syscalls Done for you Lately?
What Have Syscalls Done for you Lately?What Have Syscalls Done for you Lately?
What Have Syscalls Done for you Lately?Docker, Inc.
 
Nomad, l'orchestration made in Hashicorp - Bastien Cadiot
Nomad, l'orchestration made in Hashicorp - Bastien CadiotNomad, l'orchestration made in Hashicorp - Bastien Cadiot
Nomad, l'orchestration made in Hashicorp - Bastien CadiotParis Container Day
 
Third Party Auth in WebObjects
Third Party Auth in WebObjectsThird Party Auth in WebObjects
Third Party Auth in WebObjectsWO Community
 
Docker orchestration voxxed days berlin 2016
Docker orchestration   voxxed days berlin 2016Docker orchestration   voxxed days berlin 2016
Docker orchestration voxxed days berlin 2016Grzegorz Duda
 
Ansible fest Presentation slides
Ansible fest Presentation slidesAnsible fest Presentation slides
Ansible fest Presentation slidesAaron Carey
 
Service discovery in mesos miguel, Angel Guillen
Service discovery in mesos miguel, Angel GuillenService discovery in mesos miguel, Angel Guillen
Service discovery in mesos miguel, Angel GuillenJ On The Beach
 
How to deploy spark instance using ansible 2.0 in fiware lab v2
How to deploy spark instance using ansible 2.0 in fiware lab v2How to deploy spark instance using ansible 2.0 in fiware lab v2
How to deploy spark instance using ansible 2.0 in fiware lab v2Fernando Lopez Aguilar
 
Kubernetes internals (Kubernetes 해부하기)
Kubernetes internals (Kubernetes 해부하기)Kubernetes internals (Kubernetes 해부하기)
Kubernetes internals (Kubernetes 해부하기)DongHyeon Kim
 
Living the Nomadic life - Nic Jackson
Living the Nomadic life - Nic JacksonLiving the Nomadic life - Nic Jackson
Living the Nomadic life - Nic JacksonParis Container Day
 
Declare your infrastructure: InfraKit, LinuxKit and Moby
Declare your infrastructure: InfraKit, LinuxKit and MobyDeclare your infrastructure: InfraKit, LinuxKit and Moby
Declare your infrastructure: InfraKit, LinuxKit and MobyMoby Project
 
2017-03-11 02 Денис Нелюбин. Docker & Ansible - лучшие друзья DevOps
2017-03-11 02 Денис Нелюбин. Docker & Ansible - лучшие друзья DevOps2017-03-11 02 Денис Нелюбин. Docker & Ansible - лучшие друзья DevOps
2017-03-11 02 Денис Нелюбин. Docker & Ansible - лучшие друзья DevOpsОмские ИТ-субботники
 
Nomad + Flatcar: a harmonious marriage of lightweights
Nomad + Flatcar: a harmonious marriage of lightweightsNomad + Flatcar: a harmonious marriage of lightweights
Nomad + Flatcar: a harmonious marriage of lightweightsIago López Galeiras
 
Using MongoDB with Kafka - Use Cases and Best Practices
Using MongoDB with Kafka -  Use Cases and Best PracticesUsing MongoDB with Kafka -  Use Cases and Best Practices
Using MongoDB with Kafka - Use Cases and Best PracticesAntonios Giannopoulos
 
Memulai Data Processing dengan Spark dan Python
Memulai Data Processing dengan Spark dan PythonMemulai Data Processing dengan Spark dan Python
Memulai Data Processing dengan Spark dan PythonRidwan Fadjar
 
2020-02-20 - HashiTalks 2020 - HashiCorp Vault configuration as code via Hash...
2020-02-20 - HashiTalks 2020 - HashiCorp Vault configuration as code via Hash...2020-02-20 - HashiTalks 2020 - HashiCorp Vault configuration as code via Hash...
2020-02-20 - HashiTalks 2020 - HashiCorp Vault configuration as code via Hash...Andrey Devyatkin
 
Inside Sqale's Backend at Sapporo Ruby Kaigi 2012
Inside Sqale's Backend at Sapporo Ruby Kaigi 2012Inside Sqale's Backend at Sapporo Ruby Kaigi 2012
Inside Sqale's Backend at Sapporo Ruby Kaigi 2012Gosuke Miyashita
 

What's hot (20)

Scala, docker and testing, oh my! mario camou
Scala, docker and testing, oh my! mario camouScala, docker and testing, oh my! mario camou
Scala, docker and testing, oh my! mario camou
 
Who is afraid of privileged containers ?
Who is afraid of privileged containers ?Who is afraid of privileged containers ?
Who is afraid of privileged containers ?
 
What Have Syscalls Done for you Lately?
What Have Syscalls Done for you Lately?What Have Syscalls Done for you Lately?
What Have Syscalls Done for you Lately?
 
Nomad, l'orchestration made in Hashicorp - Bastien Cadiot
Nomad, l'orchestration made in Hashicorp - Bastien CadiotNomad, l'orchestration made in Hashicorp - Bastien Cadiot
Nomad, l'orchestration made in Hashicorp - Bastien Cadiot
 
Third Party Auth in WebObjects
Third Party Auth in WebObjectsThird Party Auth in WebObjects
Third Party Auth in WebObjects
 
Docker orchestration voxxed days berlin 2016
Docker orchestration   voxxed days berlin 2016Docker orchestration   voxxed days berlin 2016
Docker orchestration voxxed days berlin 2016
 
Ansible fest Presentation slides
Ansible fest Presentation slidesAnsible fest Presentation slides
Ansible fest Presentation slides
 
Docker n co
Docker n coDocker n co
Docker n co
 
Service discovery in mesos miguel, Angel Guillen
Service discovery in mesos miguel, Angel GuillenService discovery in mesos miguel, Angel Guillen
Service discovery in mesos miguel, Angel Guillen
 
How to deploy spark instance using ansible 2.0 in fiware lab v2
How to deploy spark instance using ansible 2.0 in fiware lab v2How to deploy spark instance using ansible 2.0 in fiware lab v2
How to deploy spark instance using ansible 2.0 in fiware lab v2
 
Kubernetes internals (Kubernetes 해부하기)
Kubernetes internals (Kubernetes 해부하기)Kubernetes internals (Kubernetes 해부하기)
Kubernetes internals (Kubernetes 해부하기)
 
Living the Nomadic life - Nic Jackson
Living the Nomadic life - Nic JacksonLiving the Nomadic life - Nic Jackson
Living the Nomadic life - Nic Jackson
 
Pharos
PharosPharos
Pharos
 
Declare your infrastructure: InfraKit, LinuxKit and Moby
Declare your infrastructure: InfraKit, LinuxKit and MobyDeclare your infrastructure: InfraKit, LinuxKit and Moby
Declare your infrastructure: InfraKit, LinuxKit and Moby
 
2017-03-11 02 Денис Нелюбин. Docker & Ansible - лучшие друзья DevOps
2017-03-11 02 Денис Нелюбин. Docker & Ansible - лучшие друзья DevOps2017-03-11 02 Денис Нелюбин. Docker & Ansible - лучшие друзья DevOps
2017-03-11 02 Денис Нелюбин. Docker & Ansible - лучшие друзья DevOps
 
Nomad + Flatcar: a harmonious marriage of lightweights
Nomad + Flatcar: a harmonious marriage of lightweightsNomad + Flatcar: a harmonious marriage of lightweights
Nomad + Flatcar: a harmonious marriage of lightweights
 
Using MongoDB with Kafka - Use Cases and Best Practices
Using MongoDB with Kafka -  Use Cases and Best PracticesUsing MongoDB with Kafka -  Use Cases and Best Practices
Using MongoDB with Kafka - Use Cases and Best Practices
 
Memulai Data Processing dengan Spark dan Python
Memulai Data Processing dengan Spark dan PythonMemulai Data Processing dengan Spark dan Python
Memulai Data Processing dengan Spark dan Python
 
2020-02-20 - HashiTalks 2020 - HashiCorp Vault configuration as code via Hash...
2020-02-20 - HashiTalks 2020 - HashiCorp Vault configuration as code via Hash...2020-02-20 - HashiTalks 2020 - HashiCorp Vault configuration as code via Hash...
2020-02-20 - HashiTalks 2020 - HashiCorp Vault configuration as code via Hash...
 
Inside Sqale's Backend at Sapporo Ruby Kaigi 2012
Inside Sqale's Backend at Sapporo Ruby Kaigi 2012Inside Sqale's Backend at Sapporo Ruby Kaigi 2012
Inside Sqale's Backend at Sapporo Ruby Kaigi 2012
 

Similar to Zenko x Tanker Hackathon Technical Deep Dive

Kubernetes #1 intro
Kubernetes #1   introKubernetes #1   intro
Kubernetes #1 introTerry Cho
 
Kubernetes @ Squarespace: Kubernetes in the Datacenter
Kubernetes @ Squarespace: Kubernetes in the DatacenterKubernetes @ Squarespace: Kubernetes in the Datacenter
Kubernetes @ Squarespace: Kubernetes in the DatacenterKevin Lynch
 
Running your dockerized application(s) on AWS Elastic Container Service
Running your dockerized application(s) on AWS Elastic Container ServiceRunning your dockerized application(s) on AWS Elastic Container Service
Running your dockerized application(s) on AWS Elastic Container ServiceMarco Pas
 
Kubernetes Basis: Pods, Deployments, and Services
Kubernetes Basis: Pods, Deployments, and ServicesKubernetes Basis: Pods, Deployments, and Services
Kubernetes Basis: Pods, Deployments, and ServicesJian-Kai Wang
 
Scaling docker with kubernetes
Scaling docker with kubernetesScaling docker with kubernetes
Scaling docker with kubernetesLiran Cohen
 
containerD
containerDcontainerD
containerDstrikr .
 
Docker 1.11 Presentation
Docker 1.11 PresentationDocker 1.11 Presentation
Docker 1.11 PresentationSreenivas Makam
 
MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...
MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...
MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...MongoDB
 
Dessi docker kubernetes paas cloud
Dessi docker kubernetes paas cloudDessi docker kubernetes paas cloud
Dessi docker kubernetes paas cloudMassimiliano Dessì
 
Come costruire una Platform As A Service con Docker, Kubernetes Go e Java
Come costruire una Platform As A Service con Docker, Kubernetes Go e JavaCome costruire una Platform As A Service con Docker, Kubernetes Go e Java
Come costruire una Platform As A Service con Docker, Kubernetes Go e JavaCodemotion
 
Introduction to kubernetes
Introduction to kubernetesIntroduction to kubernetes
Introduction to kubernetesRishabh Indoria
 
K8s in 3h - Kubernetes Fundamentals Training
K8s in 3h - Kubernetes Fundamentals TrainingK8s in 3h - Kubernetes Fundamentals Training
K8s in 3h - Kubernetes Fundamentals TrainingPiotr Perzyna
 
[NYC Meetup] Docker at Nuxeo
[NYC Meetup] Docker at Nuxeo[NYC Meetup] Docker at Nuxeo
[NYC Meetup] Docker at NuxeoNuxeo
 
Making Service Deployments to AWS a breeze with Nova
Making Service Deployments to AWS a breeze with NovaMaking Service Deployments to AWS a breeze with Nova
Making Service Deployments to AWS a breeze with NovaGregor Heine
 
Docker Athens: Docker Engine Evolution & Containerd Use Cases
Docker Athens: Docker Engine Evolution & Containerd Use CasesDocker Athens: Docker Engine Evolution & Containerd Use Cases
Docker Athens: Docker Engine Evolution & Containerd Use CasesPhil Estes
 
Managing Container Clusters in OpenStack Native Way
Managing Container Clusters in OpenStack Native WayManaging Container Clusters in OpenStack Native Way
Managing Container Clusters in OpenStack Native WayQiming Teng
 
An intro to Docker, Terraform, and Amazon ECS
An intro to Docker, Terraform, and Amazon ECSAn intro to Docker, Terraform, and Amazon ECS
An intro to Docker, Terraform, and Amazon ECSYevgeniy Brikman
 

Similar to Zenko x Tanker Hackathon Technical Deep Dive (20)

Kubernetes #1 intro
Kubernetes #1   introKubernetes #1   intro
Kubernetes #1 intro
 
Kubernetes @ Squarespace: Kubernetes in the Datacenter
Kubernetes @ Squarespace: Kubernetes in the DatacenterKubernetes @ Squarespace: Kubernetes in the Datacenter
Kubernetes @ Squarespace: Kubernetes in the Datacenter
 
Running your dockerized application(s) on AWS Elastic Container Service
Running your dockerized application(s) on AWS Elastic Container ServiceRunning your dockerized application(s) on AWS Elastic Container Service
Running your dockerized application(s) on AWS Elastic Container Service
 
Kubernetes Basis: Pods, Deployments, and Services
Kubernetes Basis: Pods, Deployments, and ServicesKubernetes Basis: Pods, Deployments, and Services
Kubernetes Basis: Pods, Deployments, and Services
 
Scaling docker with kubernetes
Scaling docker with kubernetesScaling docker with kubernetes
Scaling docker with kubernetes
 
containerD
containerDcontainerD
containerD
 
Docker 1.11 Presentation
Docker 1.11 PresentationDocker 1.11 Presentation
Docker 1.11 Presentation
 
MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...
MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...
MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...
 
Dessi docker kubernetes paas cloud
Dessi docker kubernetes paas cloudDessi docker kubernetes paas cloud
Dessi docker kubernetes paas cloud
 
Come costruire una Platform As A Service con Docker, Kubernetes Go e Java
Come costruire una Platform As A Service con Docker, Kubernetes Go e JavaCome costruire una Platform As A Service con Docker, Kubernetes Go e Java
Come costruire una Platform As A Service con Docker, Kubernetes Go e Java
 
Introduction to kubernetes
Introduction to kubernetesIntroduction to kubernetes
Introduction to kubernetes
 
K8s in 3h - Kubernetes Fundamentals Training
K8s in 3h - Kubernetes Fundamentals TrainingK8s in 3h - Kubernetes Fundamentals Training
K8s in 3h - Kubernetes Fundamentals Training
 
Kubernetes 101 for_penetration_testers_-_null_mumbai
Kubernetes 101 for_penetration_testers_-_null_mumbaiKubernetes 101 for_penetration_testers_-_null_mumbai
Kubernetes 101 for_penetration_testers_-_null_mumbai
 
[NYC Meetup] Docker at Nuxeo
[NYC Meetup] Docker at Nuxeo[NYC Meetup] Docker at Nuxeo
[NYC Meetup] Docker at Nuxeo
 
Kubernetes 1001
Kubernetes 1001Kubernetes 1001
Kubernetes 1001
 
Making Service Deployments to AWS a breeze with Nova
Making Service Deployments to AWS a breeze with NovaMaking Service Deployments to AWS a breeze with Nova
Making Service Deployments to AWS a breeze with Nova
 
Docker Athens: Docker Engine Evolution & Containerd Use Cases
Docker Athens: Docker Engine Evolution & Containerd Use CasesDocker Athens: Docker Engine Evolution & Containerd Use Cases
Docker Athens: Docker Engine Evolution & Containerd Use Cases
 
Managing Container Clusters in OpenStack Native Way
Managing Container Clusters in OpenStack Native WayManaging Container Clusters in OpenStack Native Way
Managing Container Clusters in OpenStack Native Way
 
Kubernetes Intro
Kubernetes IntroKubernetes Intro
Kubernetes Intro
 
An intro to Docker, Terraform, and Amazon ECS
An intro to Docker, Terraform, and Amazon ECSAn intro to Docker, Terraform, and Amazon ECS
An intro to Docker, Terraform, and Amazon ECS
 

Recently uploaded

A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxLoriGlavin3
 
Commit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyCommit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyAlfredo García Lavilla
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brandgvaughan
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfMounikaPolabathina
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
Streamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project SetupStreamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project SetupFlorian Wilhelm
 
Unleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubUnleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubKalema Edgar
 
Developer Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLDeveloper Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLScyllaDB
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningLars Bell
 
Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Mattias Andersson
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebUiPathCommunity
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsSergiu Bodiu
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek SchlawackFwdays
 
unit 4 immunoblotting technique complete.pptx
unit 4 immunoblotting technique complete.pptxunit 4 immunoblotting technique complete.pptx
unit 4 immunoblotting technique complete.pptxBkGupta21
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfAddepto
 
DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenHervé Boutemy
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii SoldatenkoFwdays
 
From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .Alan Dix
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024Lonnie McRorey
 

Recently uploaded (20)

A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
 
Commit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyCommit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easy
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brand
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdf
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
Streamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project SetupStreamlining Python Development: A Guide to a Modern Project Setup
Streamlining Python Development: A Guide to a Modern Project Setup
 
Unleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubUnleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding Club
 
Developer Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLDeveloper Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQL
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine Tuning
 
Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio Web
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platforms
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
 
unit 4 immunoblotting technique complete.pptx
unit 4 immunoblotting technique complete.pptxunit 4 immunoblotting technique complete.pptx
unit 4 immunoblotting technique complete.pptx
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdf
 
DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache Maven
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko
 
From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024
 

Zenko x Tanker Hackathon Technical Deep Dive

  • 1. Zenko x Tanker Hackathon @ 42 Technical Deep Dive
  • 2. The Prizes Parrot Mambo Mission Drone (+ accessories, flypad) Arduino Genuino Kit (15 tutorials, 214 pieces) OTHA Pico Projector (32GB, Miracast, HD, accessories) 1st Prize 2nd Prize 3rd Prize
  • 3. Logistics Where? > At 42 Paris When? > October 20th: tech deep dive > October 21st-November 5th: Slack support > November 6th-10th (Scality presence 09:42->17:42) How? > PHP Piscine > Scality expertise, lunch, your energy, your creativity > Tanker expertise
  • 4. Project Ideas Name Description # of people MongoDB backend Provide a MongoDB backend for S3 server/Zenko 3 Basic Vault (Accounts only) Write an auth module for accounts (create/delete); use vault-cli 2 Encrypted PouchDB Use Tanker to store encrypted stuff in PouchDB 2-3 B2 as a backend Backblaze publishes its API : https://www.backblaze.com/b2/docs/ 3 Tanker encrypted git Write a git filter to encrypt git objects 3-4 Encrypted storage Suggested usecase : photo gallery 3 IPFS as a backend Follow up on 42 Fremont project: https://github.com/ssalaues/zenko-ipfs-module 3 Add your own ideas by commenting at: Zenko hackathon ideas Check your hackathon page (and soon, your team’s profile): Zenko Hackathon Webpage
  • 5. Agenda for today Introduction to Zenko and CloudServer Introduction to Tanker Get started as a developer - Zenko Get started as a developer - Tanker Live deployment of Zenko Live demo of Tanker
  • 6. Introduction to the MultiCloud world and the Zenko stack
  • 9. 9 File or object? Why we do file: - We know it - Easy hierarchy - fopen() and fclose() - Lots of best practices - Perf of NAS / over LAN Why we do object: - Billions of entries - Storage accessed over WAN - For modern apps (REST) - Listing large volumes
  • 11.
  • 14. What is CloudServer? • Open source object storage server • Written in Node.js • Single instance running in a Docker container • Uses Docker volumes for persistent storage • Same code as Scality’s RING S3 interface 14
  • 21. These commands assume you have S3 cloned locally, s3cmd configured for your S3 server, AWS cli configured for a real AWS bucket, and your locationConfig set up - Start CloudServer & Put Object Commands CheatSheet
  • 23. CloudServer Highlights: Bucket Location Control aws --endpoint-url http://127.0.0.1:8000 s3api create-bucket --bucket test-bucket --create-bucket-configuration LocationConstraint=scality-1 aws --endpoint-url http://127.0.0.1:8000 s3 cp /etc/hosts s3://test-bucket/hosts • Specify bucket location at bucket creation • Object creation remains as usual PUT / HTTP/1.1 Host: {{BucketName}}.{{StorageService}}.com Content-Length: {{length}} Date: {{date}} Authorization: {{authenticationInformation}} <CreateBucketConfiguration xmlns="http://s3.scality.com/doc/2006-03-01/"> <LocationConstraint>scality-1</LocationConstraint> </CreateBucketConfiguration> • Request syntax
  • 24. ● When versioning is enabled on a bucket: ● CREATE NEW VERSIONS: ○ Put Object, Complete Multipart Upload and Object Copy (to a versioning-enabled bucket) will return a version id in the ‘x-amz-version-id’ response header. ○ No special syntax necessary. ● When versioning is enabled or suspended: ● TARGETING SPECIFIC VERSIONS: ○ Include the version id in the request query for GET/HEAD Object or PUT/GET Object ACL ■ Example: `GET [bucket]/[object]?versionId=[versionId]` ○ For Object Copy or Upload Copy Part, to copy a specific version from a version-enabled bucket, add the version id to the ‘x-amz-copy-source’ header: ■ Example value: `[sourcebucket]/[sourceobject]?versionId=[versionId]` ○ Omitting a specific version will get the result for the latest / current version. CloudServer Highlights: object versioning
  • 25. ● When versioning is enabled or suspended (cont.): ● NULL VERSIONS: ○ Null versions are created when putting an object before versioning is configured or when versioning is suspended. ■ Only one null version is maintained in version history. New null versions will overwrite previous null versions. ○ Target the null version in version-specific actions by specifying the version ID ‘null’. ● DELETING OBJECTS: ○ Regular deletion of objects will create delete markers and return ‘x-amz-delete-marker’: ‘true’ and the version ID of the delete marker in ‘x-amz-version-id’ response headers. ○ Objects with delete markers as the latest version will behave as if they have been deleted when performing non-version specific actions. ○ Permanently remove delete markers or specific versions by specifying the version ID in the request query. Example: `DELETE [bucket]/[object]?versionId=[versionId]` CloudServer Highlights: object versioning
  • 26. CloudServer Highlights: object versioning ● When versioning is enabled or suspended (cont.): ● MULTI-OBJECT DELETE: ○ Specify the specific version of an object to delete in a multi-object delete request in the XML body. Example: http://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html ● At any time: ● LISTING OBJECTS: ○ A regular listing will list the most recent versions of an object and ignore objects with delete markers as their latest version. ○ To list all object versions and delete markers in a bucket, specify ‘versions’ in request query: ■ Example: `GET [bucket]?versions` ○ FMI about output: consult S3 Connector documentation ● GET BUCKET VERSIONING STATUS: use Get Bucket Versioning API.
  • 27. CloudServer Highlights: UTAPI Utapi was meant to be used programmatically. For debugging, we have included a nifty tool in CloudServer’s docker container. It can be accessed as follows • docker exec -it <container id> bash • the cmd is available as node bin/list_metrics • Sample command node bin/list_metrics -a myAccessKey -k mySecretKey --buckets demo -s 1490230800000 -h 127.0.0.1 -p 8100 This would list bucket level metrics for bucket demo starting from 2017-03-23T01:00:00.000Z • Note: Since metrics are stored in 15 minute increment intervals, Utapi server requires start time to be the start of nearest 15 minute interval. For example, valid start times would look like 09:00:00:000, 09:15:00:000, 09:30:00:000 and 09:45:00:000 End time needs to be the end of the nearest 15 minute interval. For example, valid end times would look like 09:14:59:999, 09:29:59:999, 09:44:59:999 and 09:59:59:999
  • 28. CloudServer Highlights: AWS S3 as a backend ... "aws-test-2": { "type": "aws_s3", "legacyAwsBehavior": true, "details": { "awsEndpoint": "s3.amazonaws.com", "bucketName": "multitester222", "bucketMatch": true, "credentialsProfile": "default_2" } } ...
  • 29. CloudServer Highlights: Microsoft Azure as a backend ... "azure-test": { "type": "azure", "legacyAwsBehavior": false, "details": { "azureBlobEndpoint": "https://zenkomeetups.blob.core.windows.net/", "bucketMatch": true, "azureBlobSAS": "{{YOUR AZURE SAS}}", "azureContainerName": "meetupscontainer" } } ...
  • 30. Deploying Zenko Prerequisites - 2 machines (or more) - a copy of the Zenko git repository How to do it (if you’ve not read the doc…): - bundle these machines together in a Docker Swarm cluster; - apply a specific tag to the machine which will physically host the data and metadata - add your credentials to the secret.txt file - edit the docker-stack.yml file to set the number of replicas (global vs replicated) - export your endpoint as an environment variable - docker stack deploy -c docker-stack.yml {{stack-name}}
  • 31. Example: 3 managers, 2 workers Lb (Nginx) Redis Lb (Nginx) Lb (Nginx) Lb (Nginx) Data Metadata Lb (Nginx) CloudServer Leader $> docker node inspect {{id}} -f '{{ .Spec.Labels }}' map[io.zenko.type:storage]
  • 32. Failure: 3 managers, 2 workers Lb (Nginx) Redis Lb (Nginx) Lb (Nginx) Lb (Nginx) Data Metadata Lb (Nginx) CloudServer
  • 33. Failure: 2 managers, 2 workers, lost leader Lb (Nginx) Redis Lb (Nginx) Lb (Nginx) Data Metadata Lb (Nginx) CloudServer
  • 34. Failure: 3 managers, 2 workers, election Lb (Nginx) Redis Lb (Nginx) Lb (Nginx) Data Metadata Lb (Nginx) CloudServer
  • 35. Failure: 2 managers, 2 workers, new leader Lb (Nginx) Redis Lb (Nginx) Lb (Nginx) Data Metadata Lb (Nginx) CloudServer Lb (Nginx)
  • 36. Nominal: 3 managers, 2 workers, new leader Lb (Nginx) Redis Lb (Nginx) Lb (Nginx) Data Metadata Lb (Nginx) CloudServer Lb (Nginx) Lb (Nginx)
  • 37.
  • 40. Tanker ensures confidentiality and authenticity of resources among a defined set of parties 4 0 Confidentiality
  • 41. Parties can be users or servers 4 1 Parties
  • 42. Parties can have multiple devices Each device has its own key pair 4 2 Multi-device
  • 43. A resource represents a piece of data shared by a set of parties Resources can be of any type: ● A document ● A record in a database ● A data structure ... 43Resource
  • 44. Resources can be shared 44Sharing
  • 46. ●Collection of linked blocks ●Each block contains a payload ○ (custom binary format) ●Each block has an author ●Each block is signed ●Verified by both client and server 46Trustchain
  • 47. User ID Tanker user IDs should match those of your existing app or service. Note: they are obfuscated for security reasons
  • 49. Javascript SDK ● Tanker SDK is written in Javascript ● Two flavours: ○ Node (“server” side) ○ Browser (client side)
  • 50. Asynchronous In both cases, it is not recommended to wait for I/O.
  • 56. Modern times: async / await Just don’t forget the await ...
  • 58. Read the fine documentation https://beta.tanker.io/docs
  • 59. Requirements ● Node 8.0 or better ● yarn
  • 60. Create a trustchain ● Send a trustchain creation request on Slack ● Someone from Tanker staff will create a Trustchain for you ● Store the private key and the ID in a safe place
  • 61. Creating users - the safe way You should use a delegation token generated by your service, so that user additions blocks are signed properly.
  • 62. Create your first user - the easy way Here the delegation token is generated for you by the SDK. But the private key of the Trustchain is readable by the client ...
  • 63. Encrypt and share On Alice’s device: Then send encryptedData to Bob somehow
  • 65. Demo ! Get the sources on github: github.com/TankerApp/hello-42
  • 66. We also have a blog :) https://dev.to/tanker