Managing terabytes on aws

•

2 likes•994 views

JFrog is handling huge amount of binaries files for all our customers. Since each customer has its own space and domain, using a global Object Store can be tricky. Also our application cannot work with an “eventually consistent” storage, and cannot deliver customer requirements with current S3 performance. Learn in this session how we managed fast upload, critical replication and backups, and global download availability of the terabytes of JFrog customer binaries files.

Technology Sports

Request
Storage File Listing Checksum
Storage

Request
Storage File Listing Checksum
Storage
Metadata
requests
Cache check, FS
request
DB request

Request
Storage File Listing Checksum
Storage
Metadata
requests
Cache check, FS
request
DB request
HEAD requests Cache check, FS
request
DB request

/data
250GB 250GB 250GB 250GB 250GB 250GB

Backup files
Backup database
Keep them in sync!

Backup only deltas
Manage threads to
ec2 efficiently

Backup only deltas
Manage threads to
ec2 efficiently
Keep the data
isolated

De-
duped
EBS
HA Storage
Customer
EBS
Customer
EBS
Customer
EBS

De-
duped
Glacier
De-
duped
EBS
Backup
HA Storage
Customer
EBS
Customer
EBS
Customer
EBS

Create empty Artifactory instance
Populate from back
storage on demand

Create empty Artifactory instance
Populate from back
storage on demand
Drink
Cosmopolitan

Conclusions time
1. Be lazy.
2. Be smart.

Conclusions time
1. Be lazy.
2. Be smart.
3. Be agile.

The document provides instructions for installing a font file called diattusa.ttf on Windows and contact information for a person named Avery C. It also lists potential future improvements to a chess program or database including better database organization and handling, separate opening and endgame databases, an improved playing engine, ability to load PGN files and database games, and possible interface improvements.

Hostingultraso com dedicated_server_washington_d_c

SwatiDhaka2

Database File System

Anas R.

The document proposes a database file system (DBFS) as an alternative to traditional hierarchical file systems. A DBFS stores files and their metadata in a database rather than a folder structure. It allows multiple files to have the same metadata and uses keywords instead of folders for organization. The DBFS has been implemented as a daemon that integrates with an SQL database and can be used via KDE file managers. It stores only references to files rather than the files themselves and can provide faster access to files compared to hierarchical file systems.

TopDB data transfer

Chonpin HSU

This document discusses transferring data between Redis and MongoDB using TopDB. Redis is an open source, advanced key-value store that can store different data structures like strings, hashes, lists, and sets under keys. MongoDB is an open source document database and leading NoSQL database written in C++. TopDB implements a Jedis-like interface to support Redis key types and mappings between Redis databases and MongoDB databases to enable hot/cold data transfer between the two databases.

Using software modules welcome to hell!

Baruch Sadogursky

Using software modules today is the default way of working for most systems and frameworks. With the advent of many software languages and OSS frameworks, new module systems are constantly created and new module ecosystems start to prevail. This trend is horizontal and covers operating system packages, language libraries and application modules (plugins). But while some module systems are nicer to use, others are repeating past mistakes and are a daily source for developer agony and pain. In this short talk I will present the \"lessons learned\" at JFrog, where we make software for managing software libraries and deal with many types of module systems. This talk will show what works and what doesn\'t work in a module system; what features can make a module ecosystem thrive or fail; and why, despite all downsides, modules are here to stay and conquer more space as the Cloud continues to grow.

FaithRadioEvents

irvinwill

This document appears to be a collection of photos from various events hosted by Faith Radio between January and February 2010. The photos show Faith Radio team photos, family gatherings, events hosted by churches and organizations, guest speakers, and coverage areas of Faith Radio stations. The document promotes Faith Radio as a listener supported ministry that can be found on the radio in various Alabama cities and online.

Point2 Presentation For Real Estate Agents

PRETEC, Professional Real Estate Training, Education & Consulting

Point2 provides online marketing tools and services to over 183,000 real estate professionals in 100 countries. Their platform offers 12 main features including unlimited rich listings, syndication to over 30 partners, websites, virtual tours, neighborhood targeting, prospecting tools, predictive marketing, drip emails, blogs, and statistics. Members have control over their listings and tools to maximize online exposure and marketing effectiveness. Point2 offers various subscription levels from free standard to premium options with additional features and support.

VREBC Presentation - Mike Bowler Sr.

PRETEC, Professional Real Estate Training, Education & Consulting

The document discusses how real estate agents can differentiate themselves and raise their professional standards in a changing industry. It emphasizes getting educated on new tools and technologies, using social media effectively for branding and community involvement, and providing high quality services through professionalism and a strong online presence. Real estate agents are encouraged to adopt these strategies to adapt to new paradigms and ensure their relevance and success going forward.

- Remote BLOB Storage (RBS) allows storing large binary objects (BLOBs) outside of a SQL database to improve performance and reduce storage costs. - RBS supports storing BLOBs on a file system using the FILESTREAM provider. It provides benefits like potential performance improvements and easier management of storage, but has limitations around features like database snapshots and encryption. - Configuring RBS involves enabling FILESTREAM, provisioning a BLOB store within a file group, and using PowerShell to activate the RBS provider to migrate BLOBs from the database to the external store. Best practices include regular file system maintenance and tuning.

Armazenamento em Amazon Web Service para Setor Publico

Amazon Web Services LATAM

This document provides an overview of AWS storage services including EBS, EFS, S3, Glacier and data transfer options. It discusses block storage with EBS volumes, file storage with EFS, and object storage with S3 and Glacier. It also covers services for moving large amounts of data like Snowball, Direct Connect, and Storage Gateway. Use cases and architectures for databases, applications, analytics and backup/recovery are reviewed.

Database Storage Engine Internals

AdewumiSunkanmi

Database storage engine internals.pptx

AdewumiSunkanmi

DMS

Awais Majeed

This document discusses various methods for managing documents in an application, including uploading, downloading, scanning, copying, and moving files. It describes saving large files as BLOBs in the database or in a temporary table. Alternative methods like saving files in an external directory are also covered. The document provides details on configuring a directory to store files, relevant file management APIs, integrating document scanners, extracting text from PDFs, and an upload file procedure.

Breaking the 1 Million OPS/SEC Barrier in HOPS Hadoop

DataWorks Summit/Hadoop Summit

In this talk, we will present a new distribution of Hadoop, Hops, that can scale the Hadoop Filesystem (HDFS) by 16X, from 70K ops/s to 1.2 million ops/s on Spotiy's industrial Hadoop workload. Hops is an open-source distribution of Apache Hadoop that supports distributed metadata for HSFS (HopsFS) and the ResourceManager in Apache YARN. HopsFS is the first production-grade distributed hierarchical filesystem to store its metadata normalized in an in-memory, shared nothing database. For YARN, we will discuss optimizations that enable 2X throughput increases for the Capacity scheduler, enabling scalability to clusters with >20K nodes. We will discuss the journey of how we reached this milestone, discussing some of the challenges involved in efficiently and safely mapping hierarchical filesystem metadata state and operations onto a shared-nothing, in-memory database. We will also discuss the key database features needed for extreme scaling, such as multi-partition transactions, partition-pruned index scans, distribution-aware transactions, and the streaming changelog API. Hops (www.hops.io) is Apache-licensed open-source and supports a pluggable database backend for distributed metadata, although it currently only support MySQL Cluster as a backend. Hops opens up the potential for new directions for Hadoop when metadata is available for tinkering in a mature relational database.

DevOps Patterns & Antipatterns for Continuous Software Updates @ NADOG April ...

Baruch Sadogursky

The document discusses patterns and antipatterns for continuous software updates. It describes challenges with continuous updates including things that can go wrong with updates and solutions like local rollbacks, over-the-air updates, continuous rather than batch updates, automated deployments, frequent updates, state awareness, progressive delivery, observability, rollbacks, and feature flags. The goal is to transition to extremely tiny and frequent updates, called Liquid Software, to provide an illusion of software flowing continuously from development to the target.

DevOps Patterns & Antipatterns for Continuous Software Updates @ DevOps.com A...

Baruch Sadogursky

So, you want to update the software for your user, be it the nodes in your K8s cluster, a browser on user’s desktop, an app in user’s smartphone or even a user’s car. What can possibly go wrong? In this talk, we’ll analyze real-world software update fails and how multiple DevOps patterns, that fit a variety of scenarios, could have saved the developers. Manually making sure that everything works before sending an update and expecting the user to do acceptance tests before they update is most definitely not on the list of such patterns. Join us for some awesome and scary continuous update horror stories and some obvious (and some not so obvious) proven ideas for improvement and best practices you can start following tomorrow.

DevOps @Scale (Greek Tragedy in 3 Acts) as it was presented at Oracle Code NY...

Baruch Sadogursky

In this talk, we’ll take you to a scaling journey, from 3 developers to a 100. We’ll talk about the challenges each milestone in this growth brings, both technological and methodological, and how to solve those challenges using the right mix of people, the right selection of tools and the correctly crafted process. The speakers excel in the different aspects of this triangle and went through this journey (more than once) themselves. And the fun and entertaining presentation as a Greek tragedy can’t hurt, can it?

Data driven devops as presented at QCon London 2018

Baruch Sadogursky

Devops is usually viewed from a traditional perspective of a collaboration of Dev, Ops, and QA, driven by the change in Culture, People, and Process. But how do you know where you stand and where to move? As in almost any field, data and metrics give you the gauges and instruments. In this talk, we’ll talk about the key measurements for the DevOps transformation process and provide you with 3 metrics you can start measuring tomorrow.

A Research Study Into DevOps Bottlenecks as presented at Oracle Code LA 2018

Baruch Sadogursky

Java Puzzlers NG S03 a DevNexus 2018

Baruch Sadogursky

Where the Helm are your binaries? as presented at Canada Kubernetes Meetups

Baruch Sadogursky

Do you always know what’s going on with your product artifacts since the moment they are built by the CI server from Git sources all the way to being deployed by Helm into Kuberenetes? In this talk, we will show how to build a reliable and transparent pipeline from code to cluster using Git, Artifactory, Docker, Kubernetes, and Helm. We’ll show how you such a pipeline can help you answer the big questions: What to deploy, What is deployed, and what is this artifact that I am looking for. This kind of transparency is critical for today’s environments, and Kubernetes with Helm shouldn’t be an exception.

Data driven devops as presented at Codemash 2018

Baruch Sadogursky

By Baruch Sadogursky Devops is usually viewed from a traditional perspective of a collaboration of Dev, Ops and QA, driven by the change in Culture, People and Process. But how do you know where you stand and were to move? As in almost any field, data and metrics give you the gauges and instruments. In this talk we’ll talk about the key measurements for the DevOps transformation process and provide you with 3 metrics you can start measuring tomorrow.

A Research Study into DevOps Bottlenecks as presented at Codemash 2018

Baruch Sadogursky

Best Practices for Managing Docker Versions as presented at JavaOne 2017

Baruch Sadogursky

By Baruch Sadogursky There are three hard things in computer science: cache invalidation, naming things, and off-by-one errors. This session tackles naming, especially Docker version naming. Labels, tags, checksums...how should you use them to keep track of Docker versions? What about dev versus prod images—how best to distinguish those? What about the “latest” tag? What about cleanup? Could we do more? Versioning often seems like a simple problem, but when you have a tool that gives you as much power and flexibility as Docker does, it often helps to develop guidelines. The presentation examines the tools available for managing Docker images and some simple patterns you can employ in various use cases for CI/CD to keep track of your containers.

Troubleshooting & Debugging Production Microservices in Kubernetes as present...

Baruch Sadogursky

Debugging applications in production is like being the detective in a crime movie. Especially with microservices. Especially with containers. Especially in the cloud. Trying to see what’s going on in a production deployment at scale is impossible without proper tools! Google has spent over a decade deploying containerized Java applications at unprecedented scale and the infrastructure and tools developed by Google have made it uniquely possible to manage, troubleshoot, and debug, at scale. Join this session to see how you can diagnose and troubleshoot production issues w/ out of the box Kubernetes tools, as well as getting insight from the ecosystem with Weave Scope, JFrog Artifactory & Stackdriver tools.

DevOps @Scale (Greek Tragedy in 3 Acts) as it was presented at Devoxx 2017

Baruch Sadogursky

Amazon Alexa Skills vs Google Home Actions, the Big Java VUI Faceoff as prese...

Baruch Sadogursky

DevOps @Scale (Greek Tragedy in 3 Acts) as it was presented at DevOps Days Be...

Baruch Sadogursky

Java Puzzlers NG S02: Down the Rabbit Hole as it was presented at The Pittsbu...

Baruch Sadogursky

Moar puzzlers! The more we work with Java 8, the more we go into the rabbit hole. Did they add all those streams, lambdas, monads, Optionals and CompletableFutures only to confuse us? It surely looks so! And Java 9 that heads our way brings even more of what we like the most, more puzzlers, of course! In this season we as usual have a great batch of the best Java WTF, great jokes to present them and great prizes for the winners!

DevOps @Scale (Greek Tragedy in 3 Acts) as it was presented at The Pittsburgh...

Baruch Sadogursky

Similar to Managing terabytes on aws

Share point rbs in depth englisch

Samuel Zürcher

Armazenamento em Amazon Web Service para Setor Publico

Amazon Web Services LATAM

Database Storage Engine Internals

AdewumiSunkanmi

Database storage engine internals.pptx

AdewumiSunkanmi

DMS

Awais Majeed

Breaking the 1 Million OPS/SEC Barrier in HOPS Hadoop

DataWorks Summit/Hadoop Summit

Similar to Managing terabytes on aws (6)

Share point rbs in depth englisch

Armazenamento em Amazon Web Service para Setor Publico

Database Storage Engine Internals

Database storage engine internals.pptx

DMS

Breaking the 1 Million OPS/SEC Barrier in HOPS Hadoop

More from Baruch Sadogursky

DevOps Patterns & Antipatterns for Continuous Software Updates @ NADOG April ...

Baruch Sadogursky

DevOps Patterns & Antipatterns for Continuous Software Updates @ DevOps.com A...

Baruch Sadogursky

DevOps @Scale (Greek Tragedy in 3 Acts) as it was presented at Oracle Code NY...

Baruch Sadogursky

Data driven devops as presented at QCon London 2018

Baruch Sadogursky

A Research Study Into DevOps Bottlenecks as presented at Oracle Code LA 2018

Baruch Sadogursky

Java Puzzlers NG S03 a DevNexus 2018

Baruch Sadogursky

Where the Helm are your binaries? as presented at Canada Kubernetes Meetups

Baruch Sadogursky

Data driven devops as presented at Codemash 2018

Baruch Sadogursky

A Research Study into DevOps Bottlenecks as presented at Codemash 2018

Baruch Sadogursky

Best Practices for Managing Docker Versions as presented at JavaOne 2017

Baruch Sadogursky

Troubleshooting & Debugging Production Microservices in Kubernetes as present...

Baruch Sadogursky

DevOps @Scale (Greek Tragedy in 3 Acts) as it was presented at Devoxx 2017

Baruch Sadogursky

Amazon Alexa Skills vs Google Home Actions, the Big Java VUI Faceoff as prese...

Baruch Sadogursky

DevOps @Scale (Greek Tragedy in 3 Acts) as it was presented at DevOps Days Be...

Baruch Sadogursky

Java Puzzlers NG S02: Down the Rabbit Hole as it was presented at The Pittsbu...

Baruch Sadogursky

DevOps @Scale (Greek Tragedy in 3 Acts) as it was presented at The Pittsburgh...

Baruch Sadogursky

Let’s Wing It: A Study in DevRel Strategy

Baruch Sadogursky

Log Driven First Class Customer Support at Scale

Baruch Sadogursky

In this talk, Baruch Sadogursky presents the challenges of a high demand SaaS product incident triage at scale, as well as discuss the sources of log items, including the platform, tenants and other types of log sources. He will show practical examples of collector and filters configuration and will take you through a number of real world examples of problems investigations using Artifactory and Sumo Logic.

[Webinar] The Frog And The Butler: CI Pipelines For Modern DevOps

Baruch Sadogursky

No relationship in DevOps is more important than that between your CI/CD server and your Binary Repository. Jenkins has long been the go-to server for CI/CD, and JFrog Artifactory has long been one of the most popular integrations with it. This webinar focuses on the new features of the integration, leveraging the Jenkins Pipeline DSL for infrastructure-as-code of your favorite artifactory features whether it be generic, maven, gradle or Docker, and will show an end-to-end example of pipelines across multiple technologies and how powerful these new capabilities are.

Patterns and antipatterns in Docker image lifecycle as was presented at DC Do...

Baruch Sadogursky

While Docker has enabled an unprecedented velocity of software production, it is all too easy to spin out of control. A promotion-based model is required to control and track the flow of Docker images as much as it is required for a traditional software development lifecycle. New tools often introduce new paradigms. We will examine the patterns and the antipatterns for Docker image management, and what impact the new tools have on the battle-proven paradigms of the software development lifecycle.

More from Baruch Sadogursky (20)