РИТ++ 2017, Root Conf
Зал Пекин + Шанхай, 5 июня, 11:00
Тезисы:
http://rootconf.ru/2017/abstracts/2627.html
Процесс разработки не начинается и не заканчивается на написании кода программного продукта. Мы пишем документацию, придумываем, как это всё оттестировать, и заботимся о том, чтобы доступность приложения была на высоком уровне.
Мы все делаем привычные вещи привычным для нас способом. Порой выполняя много ручной и неэффективной работы. Но что, если есть другой, радикальный подход. Можно ли формализовать свою деятельность и переложить её в код? Какие практики и инструменты для этого использовать?
В докладе будет представлен личный опыт автора по автоматизации различных элементов разработки ПО.
Zero Downtime Deployment with Ansible - learn how to provision Linux servers with a web-proxy, a database and automate zero downtime deployment of a Java application to a load balanced environment.
These are the slides from a tutorial held at the Velocity Conference in Barcelona November 19th, 2014.
Git repo: https://github.com/steinim/zero-downtime-ansible
Continuous Integration: SaaS vs Jenkins in CloudIdeato
Dopo la diffusione del Cloud Computing e di Docker, è ancora preferibile
adottare i classici SaaS di Continuous Integration rispetto ad un
sistema Jenkins in cloud?
L'intervento ha l’obiettivo di mostrare un caso d'uso applicato in
Ideato di migrazione da un SaaS quale Travis ad un sistema Jenkins in
cloud, sfruttando funzionalità di on demand tramite il cloud di Amazon
Web Services e di containerizzazione tramite Docker.
Tenendo in considerazione gli aspetti tecnici legati all’implementazione
e quelli che potrebbero impattare sul fronte economico come la mancanza
di automatizzazione e i tempi di setup, verranno mostrati pregi e
difetti di questo sistema e come può essere applicato ad una serie di
progetti. Infine verranno elencati una serie di prodotti recentemente
rilasciati e in grado di far evolvere ulteriormente l'attuale sistema.
Even basic AEM deployment involves some network communication. All services need to be aware of each other to make the entire AEM stack usable for both content editors and end users.
The truth is, basic AEM deployments are not that common these days. In many cases it's much more complex - there's plenty of services around you (search engines, caching servers, data feeds, etc) and you need to talk to them in this way or another. Even though that's not the case in your project, most probably you have more than one environment to deal with (unless you're Facebook, as they run just production). All in all it makes perfect sense to run service discovery tool in your AEM infrastructure, as in a long term it gets really painful to manage all these communication channels by hand.
During my talk I'd present how Cognifide combined Consul and Chef to:
- make sure AEM always talk to correct endpoint, no matter how many instances of given service we run
- no longer worry about hardcoded IP addresses in AEM configs or Chef cookbooks
- automatically pick up new services as they go online
- enable even faster, zero-downtime deployments
- orchestrate the entire AEM infrastructure
An interesting fact is that we were able to achieve all of these without a single change in our AEM app!
It's crucial to make sure everyone's local environment is identical if you're a part of geographically distributed team and your project is composed of many services. It's also quite unlikely that all cogs in your app will spin self-sufficiently. Search engines, various APIs, authentication endpoints and HTTP accelerators - all of these services are standard these days. Your app is a part of a much wider picture.
This complexity forces you not only to broaden your mind during design phase, but it also affects the way your software is deployed. All the moving parts should be available on every single environment - including the one that works on your machine. Having in mind aforementioned complexity double click on a ZIP file is a poor man's solution. Duct taping things will end badly. And if anything can go wrong, it will, so better be prepared.
Our goal was clear - portable, versionable and predictable environment everyone can run. Throughout our journey with Vagrant, Packer and Chef we learned a lot of things:
- how box over-optimization can lead to poor user experience
- why it's important to make first "vagrant up" as quick as possible
- not everything should be dockerized
- Vagrant box is a 'virtual' release artifact and needs the same care your app receives
- successful Packer build is often half of the story
- why Consul deployment in a Vagrant box is not as trivial as it may sound
- how to optimize long Chef runs and provide more granular approach to your dev team
- it's surprisingly easy to get from "I don't want to run any VMs on my laptop!" to "Hope you have Vagrant for that"
Preparation study for Docker Event
Mulodo Open Study Group (MOSG) @Ho chi minh, Vietnam
http://www.meetup.com/Open-Study-Group-Saigon/events/229781420/
What Have Syscalls Done for you Lately?Docker, Inc.
If you've ever written any code - even just Hello World - you've used some syscalls. In this talk we'll explore what syscalls are, how they are used to set up containers, and how to make your deployment more secure at runtime by limiting the syscalls your containers can make thanks to seccomp and Linux security modules like AppArmor.
We'll also discuss how, if your architecture is broken into containerized microservices, this gives you a great opportunity to improve security by limiting what each container can do. This is where containerized microservices really shine over traditional monoliths from a security perspective - so it's helpful to know about if you're trying to convince your security team that containers are a good idea.
There will be lots of live demos!
Paweł Skotnicki: Masz zacząć pracę z legacy code? Nie wiesz, gdzie najlepiej zrobić refactor? A może zaczynasz nowy projekt, który, tym razem, ma utrzymywać najwyższą jakość kodu? Istnieje sporo narzędzi, które mają w tym pomóc, a my często o nich zapominamy. Przedstawię lub przypomnę część z nich.
How and Why Prometheus' New Storage Engine Pushes the Limits of Time Series D...Docker, Inc.
The Prometheus monitoring system collects and stores time series data to give valuable insights over hosts, containers, and applications. Its storage engine was designed to be multiple orders of magnitude faster and more space efficient than, say, RRD or SQL storage. However, with the rise of orchestration systems such as Docker Swarm and Kubernetes, and their extensive use of techniques like rolling updates and auto-scaling, environments are becoming increasingly dynamic. This increases the strain on metrics collection systems. To deal with the challenges, a new storage engine has been developed from scratch, bringing a sharp increase in performance and enabling new features.
This talk will describe this new storage engine, its architecture, its data structures, and explain why and how it is well suited to gracefully handle high turnover rates of monitoring targets and provide consistent query performance.
kubernetes install and practice
* Environment (bare metal installation, not using cloud service)
- VM 1 : Mater node, 30GB, 2 vCPU, 4GB Mem
- VM 2 : Worker node, 30GB, 2 vCPU, 4GB Mem
* Practice
- deploying pod, make a deployment and service
- expose service using ingress(nginx-ingress)
Zero Downtime Deployment with Ansible - learn how to provision Linux servers with a web-proxy, a database and automate zero downtime deployment of a Java application to a load balanced environment.
These are the slides from a tutorial held at the Velocity Conference in Barcelona November 19th, 2014.
Git repo: https://github.com/steinim/zero-downtime-ansible
Continuous Integration: SaaS vs Jenkins in CloudIdeato
Dopo la diffusione del Cloud Computing e di Docker, è ancora preferibile
adottare i classici SaaS di Continuous Integration rispetto ad un
sistema Jenkins in cloud?
L'intervento ha l’obiettivo di mostrare un caso d'uso applicato in
Ideato di migrazione da un SaaS quale Travis ad un sistema Jenkins in
cloud, sfruttando funzionalità di on demand tramite il cloud di Amazon
Web Services e di containerizzazione tramite Docker.
Tenendo in considerazione gli aspetti tecnici legati all’implementazione
e quelli che potrebbero impattare sul fronte economico come la mancanza
di automatizzazione e i tempi di setup, verranno mostrati pregi e
difetti di questo sistema e come può essere applicato ad una serie di
progetti. Infine verranno elencati una serie di prodotti recentemente
rilasciati e in grado di far evolvere ulteriormente l'attuale sistema.
Even basic AEM deployment involves some network communication. All services need to be aware of each other to make the entire AEM stack usable for both content editors and end users.
The truth is, basic AEM deployments are not that common these days. In many cases it's much more complex - there's plenty of services around you (search engines, caching servers, data feeds, etc) and you need to talk to them in this way or another. Even though that's not the case in your project, most probably you have more than one environment to deal with (unless you're Facebook, as they run just production). All in all it makes perfect sense to run service discovery tool in your AEM infrastructure, as in a long term it gets really painful to manage all these communication channels by hand.
During my talk I'd present how Cognifide combined Consul and Chef to:
- make sure AEM always talk to correct endpoint, no matter how many instances of given service we run
- no longer worry about hardcoded IP addresses in AEM configs or Chef cookbooks
- automatically pick up new services as they go online
- enable even faster, zero-downtime deployments
- orchestrate the entire AEM infrastructure
An interesting fact is that we were able to achieve all of these without a single change in our AEM app!
It's crucial to make sure everyone's local environment is identical if you're a part of geographically distributed team and your project is composed of many services. It's also quite unlikely that all cogs in your app will spin self-sufficiently. Search engines, various APIs, authentication endpoints and HTTP accelerators - all of these services are standard these days. Your app is a part of a much wider picture.
This complexity forces you not only to broaden your mind during design phase, but it also affects the way your software is deployed. All the moving parts should be available on every single environment - including the one that works on your machine. Having in mind aforementioned complexity double click on a ZIP file is a poor man's solution. Duct taping things will end badly. And if anything can go wrong, it will, so better be prepared.
Our goal was clear - portable, versionable and predictable environment everyone can run. Throughout our journey with Vagrant, Packer and Chef we learned a lot of things:
- how box over-optimization can lead to poor user experience
- why it's important to make first "vagrant up" as quick as possible
- not everything should be dockerized
- Vagrant box is a 'virtual' release artifact and needs the same care your app receives
- successful Packer build is often half of the story
- why Consul deployment in a Vagrant box is not as trivial as it may sound
- how to optimize long Chef runs and provide more granular approach to your dev team
- it's surprisingly easy to get from "I don't want to run any VMs on my laptop!" to "Hope you have Vagrant for that"
Preparation study for Docker Event
Mulodo Open Study Group (MOSG) @Ho chi minh, Vietnam
http://www.meetup.com/Open-Study-Group-Saigon/events/229781420/
What Have Syscalls Done for you Lately?Docker, Inc.
If you've ever written any code - even just Hello World - you've used some syscalls. In this talk we'll explore what syscalls are, how they are used to set up containers, and how to make your deployment more secure at runtime by limiting the syscalls your containers can make thanks to seccomp and Linux security modules like AppArmor.
We'll also discuss how, if your architecture is broken into containerized microservices, this gives you a great opportunity to improve security by limiting what each container can do. This is where containerized microservices really shine over traditional monoliths from a security perspective - so it's helpful to know about if you're trying to convince your security team that containers are a good idea.
There will be lots of live demos!
Paweł Skotnicki: Masz zacząć pracę z legacy code? Nie wiesz, gdzie najlepiej zrobić refactor? A może zaczynasz nowy projekt, który, tym razem, ma utrzymywać najwyższą jakość kodu? Istnieje sporo narzędzi, które mają w tym pomóc, a my często o nich zapominamy. Przedstawię lub przypomnę część z nich.
How and Why Prometheus' New Storage Engine Pushes the Limits of Time Series D...Docker, Inc.
The Prometheus monitoring system collects and stores time series data to give valuable insights over hosts, containers, and applications. Its storage engine was designed to be multiple orders of magnitude faster and more space efficient than, say, RRD or SQL storage. However, with the rise of orchestration systems such as Docker Swarm and Kubernetes, and their extensive use of techniques like rolling updates and auto-scaling, environments are becoming increasingly dynamic. This increases the strain on metrics collection systems. To deal with the challenges, a new storage engine has been developed from scratch, bringing a sharp increase in performance and enabling new features.
This talk will describe this new storage engine, its architecture, its data structures, and explain why and how it is well suited to gracefully handle high turnover rates of monitoring targets and provide consistent query performance.
kubernetes install and practice
* Environment (bare metal installation, not using cloud service)
- VM 1 : Mater node, 30GB, 2 vCPU, 4GB Mem
- VM 2 : Worker node, 30GB, 2 vCPU, 4GB Mem
* Practice
- deploying pod, make a deployment and service
- expose service using ingress(nginx-ingress)
During our most recent workshops, we presented the most popular libraries of Node.js, and their installation on Docker. Thanks to our presentation, you’ll be able to program applications which use Node.js and ES6.
A step-by-step guide to deploying your first Hello World chaincode onto Hyperledger Fabric.
These slides were created by James Bowkett, Principal Consultant at Excelian.
Inria Tech Talk : Comment améliorer la qualité de vos logiciels avec STAMPStéphanie Roger
Que vous soyez développeur ou entrepreneur, découvrez le projet STAMP piloté par Inria, l'institut national de recherche dédié aux sciences du numérique.
JavaFX 8 est disponible depuis mars 2014 et apporte son lot de nouveautés. Gradle est en version 2 depuis juillet 2014. Deux technologies plus que prometteuses: JavaFX donne un coup de jeune au développement d’applications desktop en Java en apportant un navigateur web intégré, le support des WebSockets, de la 3D, et bien d’autres. Gradle est l’outil de d’automatisation de build à la mode, apportant de superbes possibilités par rapport rapport à maven, outil vieillissant, grâce à l’engouement de la communauté vis à vis de cet outil mais aussi par le fait de la technologie utilisée en son sein: groovy. Venez découvrir comment il est possible de réaliser rapidement une application à la mode en JavaFX avec un outil à la mode également. Bref venez à une session trendy.
Integration testing is hard, and often teams are tempted to do it in production. Testcontainers allows writing meaningful integration tests spawning Docker containers for databases, queue systems, kv-store, other services. The talk, a blend of slides and live code, will show how we are able to deploy without fear while integrating with a dozen of different datastores. Don't mock your database with fake data anymore, work with real data
PuppetDB: A Single Source for Storing Your Puppet Data - PUG NYPuppet
James Sweeney presents on "PuppetDB: A Single Source for Storing Your Puppet Data" at Puppet User Group NYC.
Video: http://www.youtube.com/watch?v=HTr4b02aU7A
Puppet NYC: http://www.meetup.com/puppetnyc-meetings/
GeeCON 2017 - TestContainers. Integration testing without the hassleAnton Arhipov
TestContainers is a Java library that supports JUnit tests, providing lightweight, throwaway instances of common databases, Selenium web browsers, or anything else that can run in a Docker container.
In this deck from the Stanford HPC Conference, Christian Kniep from Docker, Inc. gives a tutorial on linux containers.
"This tutorial provides a detailed overview of the components needed to run containerized applications and explores how distributed HPC applications can be tackled. We’ll explain the concept of Linux Containers and describe the bits and pieces participants will explore following step-by-step examples.
The workshop will introduce the predominant forms of orchestration in the industry; what problems they solve and how to approach the problem.
Attendees will explore the benefits and drawbacks of orchestrators first hand with their own small exemplary stack deployments.
Finally the workshop will introduce how HPC and Big Data workloads can be tackled on-top of these service-oriented clusters."
Watch the video: https://youtu.be/LJinZpCTyk0
Learn more: http://www.docker.com/
and
http://hpcadvisorycouncil.com
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Presented by Gregg Donovan, Senior Software Engineer, Etsy.com, Inc.
Understanding the impact of garbage collection, both at a single node and a cluster level, is key to developing high-performance, high-availability Solr and Lucene applications. After a brief overview of garbage collection theory, we will review the design and use of the various collectors in the JVM.
At a single-node level, we will explore GC monitoring -- how to understand GC logs, how to monitor what % of your Solr request time is spend on GC, how to use VisualGC, YourKit, and other tools, and what to log and monitor. We will review GC tuning and how to measure success.
At a cluster-level, we will review how to design for partial availability -- how to avoid sending requests to a GCing node and how to be resilient to mid-request GC pauses.For application development, we will review common memory leak scenarios in custom Solr and Lucene application code and how to detect them.
[CB20] Vulnerabilities of Machine Learning Infrastructure by Sergey GordeychikCODE BLUE
The boom of AI brought to the market a set of impressive solutions both on the hardware and software side. On the other hand, massive implementation of AI in various areas brings about problems, and security is one of the greatest concerns.
In this talk we will present results of hands-on vulnerability research of different components of AI infrastructure including NVIDIA DGX GPU servers, ML frameworks such as Pytorch, Keras and Tensorflow, data processing pipelines and specific applications, including Medical Imaging and face recognition powered CCTV. Updated Internet Census toolkit based on the Grinder framework will be introduced.
안드로이드 앱의 빌드 툴인 Gradle은 빌드 스크립트로 Groovy DSL을 사용합니다. DSL은 큰 학습 없이 사용할 수 있다는 장점이 있지만, 각 구문의 역할이나 의미를 정확히 알기는 어렵습니다. 이에 대한 기본적인 이해가 생긴다면, 직접 코드를 추가하거나 기존 기능을 개선시키는 등 적극적으로 빌드 과정을 조절하는데 도움이 될 것입니다. 이 세션에서는 Groovy 문법을 간단하게 소개하고, Gradle 파일들을 구문별로 의미하는 바를 짚어보며 gradle 파일을 읽는 법을 알아봅니다. 그리고 원하는 기능을 찾기 위해서 어떤 레퍼런스를 찾아야 하는지에 대한 팁을 공유합니다.
Similar to Everything as a Code / Александр Тарасов (Одноклассники) (20)
One-cloud — система управления дата-центром в Одноклассниках / Олег Анастасье...Ontico
HighLoad++ 2017
Зал «Калининград», 8 ноября, 15:00
Тезисы:
http://www.highload.ru/2017/abstracts/2964.html
Одноклассники состоят из более чем восьми тысяч железных серверов, расположенных в нескольких дата-центрах. Каждая из этих машин была специализированной под конкретную задачу - как для обеспечения изоляции отказов, так и для обеспечения автоматизированного управления инфраструктурой.
...
Масштабируя DNS / Артем Гавриченков (Qrator Labs)Ontico
HighLoad++ 2017
Зал «Калининград», 8 ноября, 16:00
Тезисы:
http://www.highload.ru/2017/abstracts/3032.html
Протокол DNS на семь лет старше, чем Всемирная паутина. Стандарты RFC 882 и 883, определяющие основную функциональность системы доменных имён, появились в конце 1983 года, а первая реализация последовала уже годом позже. Естественно, что у технологии столь старой и при этом по сей день активнейшим образом используемой просто не могли не накопиться особенности, неочевидные обыкновенным пользователям.
...
Создание BigData-платформы для ФГУП Почта России / Андрей Бащенко (Luxoft)Ontico
HighLoad++ 2017
Зал «Калининград», 8 ноября, 13:00
Тезисы:
http://www.highload.ru/2017/abstracts/3010.html
В этом докладе я расскажу, как BigData-платформа помогает трансформировать Почту России, как мы управляем построением и развитием платформы. Расскажу про найденные удачные решения, например, как разбиение на продукты с понятными SLA и интерфейсами между ними помогло нам сохранять управляемость с ростом масштабов проекта.
...
Готовим тестовое окружение, или сколько тестовых инстансов вам нужно / Алекса...Ontico
HighLoad++ 2017
Зал «Кейптаун», 8 ноября, 10:00
Тезисы:
http://www.highload.ru/2017/abstracts/2914.html
Казалось бы, что нужно для организации тестового окружения? Тестовая железка и копия боевого окружения - и тестовый сервер готов. Но как быть, когда проект сложный? А когда большой? А если нужно тестировать одновременно много версий? А если все это вместе?
Организация тестирования большого развивающегося проекта, где одновременно в разработке и тестировании около полусотни фич - достаточно непростая задача. Ситуация обычно осложняется тем, что иногда есть желание потрогать еще не полностью готовый функционал. В таких ситуациях часто возникает вопрос: "А куда это можно накатить и где покликать?"
...
Новые технологии репликации данных в PostgreSQL / Александр Алексеев (Postgre...Ontico
HighLoad++ 2017
Зал «Кейптаун», 8 ноября, 18:00
Тезисы:
http://www.highload.ru/2017/abstracts/2854.html
Из этого доклада вы узнаете о возможностях репликации и автофейловера PostgreSQL, в том числе о возможностях, ставших доступных в PostgreSQL 10.
Среди прочих, будет затронуты следующие темы:
* Виды репликации и решаемые с ее помощью проблемы.
* Настройка потоковой репликации.
* Настройка логической репликации.
* Настройка автофейловера / HA средствами Stolon и Consul.
После прослушивания доклада вы сможете самостоятельно настраивать репликацию и автофейловер PostgreSQL.
PostgreSQL Configuration for Humans / Alvaro Hernandez (OnGres)Ontico
HighLoad++ 2017
Зал «Кейптаун», 8 ноября, 17:00
Тезисы:
http://www.highload.ru/2017/abstracts/3096.html
PostgreSQL is the world’s most advanced open source database. Indeed! With around 270 configuration parameters in postgresql.conf, plus all the knobs in pg_hba.conf, it is definitely ADVANCED!
How many parameters do you tune? 1? 8? 32? Anyone ever tuned more than 64?
No tuning means below par performance. But how to start? Which parameters to tune? What are the appropriate values? Is there a tool --not just an editor like vim or emacs-- to help users manage the 700-line postgresql.conf file?
Join this talk to understand the performance advantages of appropriately tuning your postgresql.conf file, showcase a new free tool to make PostgreSQL configuration possible for HUMANS, and learn the best practices for tuning several relevant postgresql.conf parameters.
Inexpensive Datamasking for MySQL with ProxySQL — Data Anonymization for Deve...Ontico
HighLoad++ 2017
Зал «Кейптаун», 8 ноября, 16:00
Тезисы:
http://www.highload.ru/2017/abstracts/3115.html
During this session we will cover the last development in ProxySQL to support regular expressions (RE2 and PCRE) and how we can use this strong technique in correlation with ProxySQL's query rules to anonymize live data quickly and transparently. We will explain the mechanism and how to generate these rules quickly. We show live demo with all challenges we got from the Community and we finish the session by an interactive brainstorm testing queries from the audience.
Опыт разработки модуля межсетевого экранирования для MySQL / Олег Брославский...Ontico
HighLoad++ 2017
Зал «Кейптаун», 8 ноября, 15:00
Тезисы:
http://www.highload.ru/2017/abstracts/2957.html
Расскажем о нашем опыте разработки модуля межсетевого экрана для MySQL с использованием генератора парсеров ANTLR и языка Kotlin.
Подробно рассмотрим следующие вопросы:
— когда и почему целесообразно использовать ANTLR;
— особенности разработки ANTLR-грамматики для MySQL;
— сравнение производительности рантаймов для ANTLR в рамках задачи синтаксического анализа MySQL (C#, Java, Kotlin, Go, Python, PyPy, C++);
— вспомогательные DSL;
— микросервисная архитектура модуля экранирования SQL;
— полученные результаты.
ProxySQL Use Case Scenarios / Alkin Tezuysal (Percona)Ontico
HighLoad++ 2017
Зал «Кейптаун», 8 ноября, 14:00
Тезисы:
http://www.highload.ru/2017/abstracts/3114.html
ProxySQL aims to be the most powerful proxy in the MySQL ecosystem. It is protocol-aware and able to provide high availability (HA) and high performance with no changes in the application, using several built-in features and integration with clustering software. During this session we will quickly introduce its main features, so to better understand how it works. We will then describe multiple use case scenarios in which ProxySQL empowers large MySQL installations to provide HA with zero downtime, read/write split, query rewrite, sharding, query caching, and multiplexing using SSL across data centers.
MySQL Replication — Advanced Features / Петр Зайцев (Percona)Ontico
HighLoad++ 2017
Зал «Кейптаун», 8 ноября, 13:00
Тезисы:
http://www.highload.ru/2017/abstracts/2954.html
MySQL Replication is powerful and has added a lot of advanced features through the years. In this presentation we will look into replication technology in MySQL 5.7 and variants focusing on advanced features, what do they mean, when to use them and when not, Including.
When should you use STATEMENT, ROW or MIXED binary log format?
What is GTID in MySQL and MariaDB and why do you want to use them?
What is semi-sync replication and how is it different from lossless semi-sync?
...
Внутренний open-source. Как разрабатывать мобильное приложение большим количе...Ontico
HighLoad++ 2017
Зал «Кейптаун», 8 ноября, 12:00
Тезисы:
http://www.highload.ru/2017/abstracts/3120.html
Количество разработчиков мобильных приложений Сбербанк Онлайн с начала 2016 года выросло на порядок. Для того чтобы продолжать выпускать качественный продукт, мы кардинально перестраиваем процесс разработки.
Количество внутренних заказчиков тех или иных доработок в какой-то момент выросло настолько, что разработчики стали узким местом. Мы внедрили культуру разработки, которую можно условно назвать "внутренним open-source", сохранив за собой контроль над архитектурой и качеством проекта, но позволив разрабатывать новые фичи всем желающим.
...
Подробно о том, как Causal Consistency реализовано в MongoDB / Михаил Тюленев...Ontico
HighLoad++ 2017
Зал «Мумбай», 8 ноября, 18:00
Тезисы:
http://www.highload.ru/2017/abstracts/2836.html
При использовании Eventually Consistent распределенных баз данных нет гарантий, что чтение возвращает результаты последних изменений данных, если чтение и запись производятся на разных узлах. Это ограничивает пропускную способность системы. Поддержка свойства Causal Consistency снимает это ограничение, что позволяет улучшить масштабируемость, не требуя изменений в коде приложения.
...
Балансировка на скорости проводов. Без ASIC, без ограничений. Решения NFWare ...Ontico
HighLoad++ 2017
Зал «Мумбай», 8 ноября, 16:00
Тезисы:
http://www.highload.ru/2017/abstracts/2858.html
Аудитория Одноклассников превышает 73 миллиона человек в России, СНГ и странах дальнего зарубежья. При этом ОК.ru - первая социальная сеть по просмотрам видео в рунете и крупнейшая сервисная платформа.
Качественный и количественный рост DDoS-атак за последние годы превращает их в одну из первоочередных проблем для крупнейших интернет-ресурсов. В зависимости от вектора атаки “узким” местом становится та или иная часть инфраструктуры. В частности, при SYN-flood первый удар приходится на систему балансировки трафика. От ее производительности зависит успех в противостоянии атаке.
...
Перехват трафика — мифы и реальность / Евгений Усков (Qrator Labs)Ontico
HighLoad++ 2017
Зал «Мумбай», 8 ноября, 15:00
Тезисы:
http://www.highload.ru/2017/abstracts/3008.html
Никогда не было и вот снова случилось! Компания Google в результате перенаправления трафика сделала недостпуными в Японии несколько тысяч различных сервисов, большинство из которых никак не связано с самой компанией Google. Однако, подобные инциденты происходят с завидной регулярностью, вот только не всегда попадают в большие СМИ. У таких инцидентов могут быть разные причины, начиная от ошибок сетевых инженеров и заканчивая государственным регулированием.
...
И тогда наверняка вдруг запляшут облака! / Алексей Сушков (ПЕТЕР-СЕРВИС)Ontico
HighLoad++ 2017
Зал «Мумбай», 8 ноября, 14:00
Тезисы:
http://www.highload.ru/2017/abstracts/2925.html
Облака и виртуализация – современные тренды развития IT-технологий. Операторы связи строят свои TelcoClouds на стандартах NFV (Network Functions Virtualization) и SDN (Software-Defined Networking). В докладе начнем с основ виртуализации, далее разберемся, для чего используются NFV и SDN, потом полетим к облакам и вернемся на землю для решения практических задач!
...
Как мы заставили Druid работать в Одноклассниках / Юрий Невиницин (OK.RU)Ontico
HighLoad++ 2017
Зал «Мумбай», 8 ноября, 10:00
Тезисы:
http://www.highload.ru/2017/abstracts/3045.html
Как мы заставили Druid работать в Одноклассниках.
«Druid is a high-performance, column-oriented, distributed data store» http://druid.io.
Мы расскажем о том, как, внедрив Druid, мы справились с ситуацией, когда MSSQL-based система статистики на 50 терабайт стала:
- медленной: средняя скорость ответа была в разы меньше требуемой (и увеличилась в 20 раз);
- нестабильной: в час пик статистика отставала до получаса (теперь ничего не отстает);
- дорогой: изменилась политика лицензирования Microsoft, расходы на лицензии могли составить миллионы долларов.
...
Разгоняем ASP.NET Core / Илья Вербицкий (WebStoating s.r.o.)Ontico
HighLoad++ 2017
Зал «Рио-де-Жанейро», 8 ноября, 18:00
Тезисы:
http://www.highload.ru/2017/abstracts/2905.html
Прошло более года с того момента, как Microsoft выпустила первую версию своего нового фреймворка для разработки web-приложений ASP.NET Core, и с каждым днем он находит все больше поклонников. ASP.NET Core базируется на платформе .NET Core, кроссплатформенной версии платформы .NET c открытым исходным кодом. Теперь у С#-разработчиков появилась возможность использовать Mac в качестве среды разработки, и запускать приложения на Linux или внутри Docker-контейнеров.
...
100500 способов кэширования в Oracle Database или как достичь максимальной ск...Ontico
HighLoad++ 2017
Зал «Рио-де-Жанейро», 8 ноября, 14:00
Тезисы:
http://www.highload.ru/2017/abstracts/2913.html
Изначально будут раскрыты базовые причины, которые заставили появиться такой части механизма СУБД, как кэш результатов, и почему в ряде СУБД он есть или отсутствует.
Будут рассмотрены различные варианты кэширования результатов как sql-запросов, так и результатов хранимой в БД бизнес-логики. Произведено сравнение способов кэширования (программируемые вручную кэши, стандартный функционал) и даны рекомендации, когда и в каких случаях данные способы оптимальны, а порой опасны.
...
Apache Ignite Persistence: зачем Persistence для In-Memory, и как он работает...Ontico
HighLoad++ 2017
Зал «Рио-де-Жанейро», 8 ноября, 13:00
Тезисы:
http://www.highload.ru/2017/abstracts/2947.html
Apache Ignite — Open Source платформа для высокопроизводительной распределенной работы с большими данными с применением SQL или Java/.NET/C++ API. Ignite используют в самых разных отраслях. Сбербанк, ING, RingCentral, Microsoft, e-Therapeutics — все эти компании применяют решения на основе Ignite. Размеры кластеров разнятся от всего одного узла до нескольких сотен, узлы могут быть расположены в одном ЦОД-е или в нескольких геораспределенных.
...
HighLoad++ 2017
Зал «Рио-де-Жанейро», 8 ноября, 12:00
Тезисы:
http://www.highload.ru/2017/abstracts/3005.html
Когда мы говорим о нагруженных системах и базах данных с большим числом параллельных коннектов, особый интерес представляет практика эксплуатации и сопровождения таких проектов. В том числе инструменты и механизмы СУБД, которые могут быть использованы DBA и DevOps-инженерами для решения задач мониторинга жизнедеятельности базы данных и ранней диагностики возможных проблем.
...
Hierarchical Digital Twin of a Naval Power SystemKerry Sado
A hierarchical digital twin of a Naval DC power system has been developed and experimentally verified. Similar to other state-of-the-art digital twins, this technology creates a digital replica of the physical system executed in real-time or faster, which can modify hardware controls. However, its advantage stems from distributing computational efforts by utilizing a hierarchical structure composed of lower-level digital twin blocks and a higher-level system digital twin. Each digital twin block is associated with a physical subsystem of the hardware and communicates with a singular system digital twin, which creates a system-level response. By extracting information from each level of the hierarchy, power system controls of the hardware were reconfigured autonomously. This hierarchical digital twin development offers several advantages over other digital twins, particularly in the field of naval power systems. The hierarchical structure allows for greater computational efficiency and scalability while the ability to autonomously reconfigure hardware controls offers increased flexibility and responsiveness. The hierarchical decomposition and models utilized were well aligned with the physical twin, as indicated by the maximum deviations between the developed digital twin hierarchy and the hardware.
Hybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdffxintegritypublishin
Advancements in technology unveil a myriad of electrical and electronic breakthroughs geared towards efficiently harnessing limited resources to meet human energy demands. The optimization of hybrid solar PV panels and pumped hydro energy supply systems plays a pivotal role in utilizing natural resources effectively. This initiative not only benefits humanity but also fosters environmental sustainability. The study investigated the design optimization of these hybrid systems, focusing on understanding solar radiation patterns, identifying geographical influences on solar radiation, formulating a mathematical model for system optimization, and determining the optimal configuration of PV panels and pumped hydro storage. Through a comparative analysis approach and eight weeks of data collection, the study addressed key research questions related to solar radiation patterns and optimal system design. The findings highlighted regions with heightened solar radiation levels, showcasing substantial potential for power generation and emphasizing the system's efficiency. Optimizing system design significantly boosted power generation, promoted renewable energy utilization, and enhanced energy storage capacity. The study underscored the benefits of optimizing hybrid solar PV panels and pumped hydro energy supply systems for sustainable energy usage. Optimizing the design of solar PV panels and pumped hydro energy supply systems as examined across diverse climatic conditions in a developing country, not only enhances power generation but also improves the integration of renewable energy sources and boosts energy storage capacities, particularly beneficial for less economically prosperous regions. Additionally, the study provides valuable insights for advancing energy research in economically viable areas. Recommendations included conducting site-specific assessments, utilizing advanced modeling tools, implementing regular maintenance protocols, and enhancing communication among system components.
Overview of the fundamental roles in Hydropower generation and the components involved in wider Electrical Engineering.
This paper presents the design and construction of hydroelectric dams from the hydrologist’s survey of the valley before construction, all aspects and involved disciplines, fluid dynamics, structural engineering, generation and mains frequency regulation to the very transmission of power through the network in the United Kingdom.
Author: Robbie Edward Sayers
Collaborators and co editors: Charlie Sims and Connor Healey.
(C) 2024 Robbie E. Sayers
Explore the innovative world of trenchless pipe repair with our comprehensive guide, "The Benefits and Techniques of Trenchless Pipe Repair." This document delves into the modern methods of repairing underground pipes without the need for extensive excavation, highlighting the numerous advantages and the latest techniques used in the industry.
Learn about the cost savings, reduced environmental impact, and minimal disruption associated with trenchless technology. Discover detailed explanations of popular techniques such as pipe bursting, cured-in-place pipe (CIPP) lining, and directional drilling. Understand how these methods can be applied to various types of infrastructure, from residential plumbing to large-scale municipal systems.
Ideal for homeowners, contractors, engineers, and anyone interested in modern plumbing solutions, this guide provides valuable insights into why trenchless pipe repair is becoming the preferred choice for pipe rehabilitation. Stay informed about the latest advancements and best practices in the field.
Welcome to WIPAC Monthly the magazine brought to you by the LinkedIn Group Water Industry Process Automation & Control.
In this month's edition, along with this month's industry news to celebrate the 13 years since the group was created we have articles including
A case study of the used of Advanced Process Control at the Wastewater Treatment works at Lleida in Spain
A look back on an article on smart wastewater networks in order to see how the industry has measured up in the interim around the adoption of Digital Transformation in the Water Industry.
Cosmetic shop management system project report.pdfKamal Acharya
Buying new cosmetic products is difficult. It can even be scary for those who have sensitive skin and are prone to skin trouble. The information needed to alleviate this problem is on the back of each product, but it's thought to interpret those ingredient lists unless you have a background in chemistry.
Instead of buying and hoping for the best, we can use data science to help us predict which products may be good fits for us. It includes various function programs to do the above mentioned tasks.
Data file handling has been effectively used in the program.
The automated cosmetic shop management system should deal with the automation of general workflow and administration process of the shop. The main processes of the system focus on customer's request where the system is able to search the most appropriate products and deliver it to the customers. It should help the employees to quickly identify the list of cosmetic product that have reached the minimum quantity and also keep a track of expired date for each cosmetic product. It should help the employees to find the rack number in which the product is placed.It is also Faster and more efficient way.
About
Indigenized remote control interface card suitable for MAFI system CCR equipment. Compatible for IDM8000 CCR. Backplane mounted serial and TCP/Ethernet communication module for CCR remote access. IDM 8000 CCR remote control on serial and TCP protocol.
• Remote control: Parallel or serial interface.
• Compatible with MAFI CCR system.
• Compatible with IDM8000 CCR.
• Compatible with Backplane mount serial communication.
• Compatible with commercial and Defence aviation CCR system.
• Remote control system for accessing CCR and allied system over serial or TCP.
• Indigenized local Support/presence in India.
• Easy in configuration using DIP switches.
Technical Specifications
Indigenized remote control interface card suitable for MAFI system CCR equipment. Compatible for IDM8000 CCR. Backplane mounted serial and TCP/Ethernet communication module for CCR remote access. IDM 8000 CCR remote control on serial and TCP protocol.
Key Features
Indigenized remote control interface card suitable for MAFI system CCR equipment. Compatible for IDM8000 CCR. Backplane mounted serial and TCP/Ethernet communication module for CCR remote access. IDM 8000 CCR remote control on serial and TCP protocol.
• Remote control: Parallel or serial interface
• Compatible with MAFI CCR system
• Copatiable with IDM8000 CCR
• Compatible with Backplane mount serial communication.
• Compatible with commercial and Defence aviation CCR system.
• Remote control system for accessing CCR and allied system over serial or TCP.
• Indigenized local Support/presence in India.
Application
• Remote control: Parallel or serial interface.
• Compatible with MAFI CCR system.
• Compatible with IDM8000 CCR.
• Compatible with Backplane mount serial communication.
• Compatible with commercial and Defence aviation CCR system.
• Remote control system for accessing CCR and allied system over serial or TCP.
• Indigenized local Support/presence in India.
• Easy in configuration using DIP switches.
Final project report on grocery store management system..pdfKamal Acharya
In today’s fast-changing business environment, it’s extremely important to be able to respond to client needs in the most effective and timely manner. If your customers wish to see your business online and have instant access to your products or services.
Online Grocery Store is an e-commerce website, which retails various grocery products. This project allows viewing various products available enables registered users to purchase desired products instantly using Paytm, UPI payment processor (Instant Pay) and also can place order by using Cash on Delivery (Pay Later) option. This project provides an easy access to Administrators and Managers to view orders placed using Pay Later and Instant Pay options.
In order to develop an e-commerce website, a number of Technologies must be studied and understood. These include multi-tiered architecture, server and client-side scripting techniques, implementation technologies, programming language (such as PHP, HTML, CSS, JavaScript) and MySQL relational databases. This is a project with the objective to develop a basic website where a consumer is provided with a shopping cart website and also to know about the technologies used to develop such a website.
This document will discuss each of the underlying technologies to create and implement an e- commerce website.
Sachpazis:Terzaghi Bearing Capacity Estimation in simple terms with Calculati...Dr.Costas Sachpazis
Terzaghi's soil bearing capacity theory, developed by Karl Terzaghi, is a fundamental principle in geotechnical engineering used to determine the bearing capacity of shallow foundations. This theory provides a method to calculate the ultimate bearing capacity of soil, which is the maximum load per unit area that the soil can support without undergoing shear failure. The Calculation HTML Code included.
Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)MdTanvirMahtab2
This presentation is about the working procedure of Shahjalal Fertilizer Company Limited (SFCL). A Govt. owned Company of Bangladesh Chemical Industries Corporation under Ministry of Industries.
13. Что такое код?
● Коллекция инструкций
● Человекочитаемый формат
○ plain text
● Может быть понят и “проигран”
○ после компиляции
○ интерпретатором
13
29. # checkout repositories from git
- include: checkout.yml
# configure your local environment
- include: configure.yml
# add useful mappings to your hosts file
- include: hosts.yml
# add gradle support
- include: gradle.yml
# clean and build all projects
- include: build.yml
# delpoy all services to dev
- include: deploy.yml
29
Алгоритм
36. {% if services is not none %}{% for service in services %}
if (project.name == '{{ service.name }}') {
{% if 'sourceDir' in service %}
main.java.srcDir '{{ service.sourceDir }}'
{% endif %}{% if 'testDir' in service %}
test.java.srcDir '{{ service.testDir }}'
{% endif %}
}
Условные операторы
36
37. {% if services is not none %}{% for service in services %}
if (project.name == '{{ service.name }}') {
{% if 'sourceDir' in service %}
main.java.srcDir '{{ service.sourceDir }}'
{% endif %}{% if 'testDir' in service %}
test.java.srcDir '{{ service.testDir }}'
{% endif %}
}
Опять циклы
37
38. {% if services is not none %}{% for service in services %}
if (project.name == '{{ service.name }}') {
{% if 'sourceDir' in service %}
main.java.srcDir '{{ service.sourceDir }}'
{% endif %}{% if 'testDir' in service %}
test.java.srcDir '{{ service.testDir }}'
{% endif %}
}
Переменные
38
40. Install apps Code Checkout
Configure
workspace
Multiplie Use
40
Петля обратной связи
git
41. Что получили
● Автоконфигурация рабочего
пространства
○ повторяемая
○ немутабельная
● Можно дать скрипт новичку
● Можно и нужно пользоваться этим
каждый день
41
85. attacker := vegeta.NewAttacker()
var metrics vegeta.Metrics
for res := range attacker.Attack(targeter, rate, duration) {
metrics.Add(res)
}
metrics.Close()
if metrics.Success*100 < test.SLA.SuccessRate {
os.Exit(1)
}
if metrics.Latencies.P99() > SLA.Latency*time.Millisecond.Nanoseconds() {
os.Exit(1)
}
Немного go-кода
85
86. attacker := vegeta.NewAttacker()
var metrics vegeta.Metrics
for res := range attacker.Attack( targeter, rate, duration) {
metrics.Add(res)
}
metrics.Close()
if metrics.Success*100 < test.SLA.SuccessRate {
os.Exit(1)
}
if metrics.Latencies.P99() > SLA.Latency*time.Millisecond.Nanoseconds() {
os.Exit(1)
}
Майним метрики
86
87. attacker := vegeta.NewAttacker()
var metrics vegeta.Metrics
for res := range attacker.Attack(targeter, rate, duration) {
metrics.Add(res)
}
metrics.Close()
if metrics.Success*100 < test.SLA.SuccessRate {
os.Exit(1)
}
if metrics.Latencies.P99() > SLA.Latency*time.Millisecond.Nanoseconds() {
os.Exit(1)
}
Не слишком ли много ошибок?
87
88. attacker := vegeta.NewAttacker()
var metrics vegeta.Metrics
for res := range attacker.Attack(targeter, rate, duration) {
metrics.Add(res)
}
metrics.Close()
if metrics.Success*100 < test.SLA.SuccessRate {
os.Exit(1)
}
if metrics.Latencies.P99() > SLA.Latency*time.Millisecond.Nanoseconds() {
os.Exit(1)
}
Уложились ли по времени?
88
89. > cat config.json | docker run -i apistress -config=stdin
> echo $?
0
89
Код возврата
90. Requests [total, rate] 200, 10.05
Duration [total, attack, wait] 23.04784s, 19.899754743s, 3.148093811s
Latencies [mean, 50, 95, 99, max] 3.023677499s, ..., 11.832287083s
Bytes In [total, mean] 6874, 34.37
Bytes Out [total, mean] 1349, 6.75
Success [ratio] 35.50%
Status Codes [code:count] 0:129 200:71
Error Set:
Get http://host:9000/rent-service/rent: EOF
Get http://host:9000/rent-service/rent: http: server closed idle connection
...
Test#2
90
91. > cat config.json | docker run -i apistress -config=stdin
> echo $?
1
91
Код возврата
92. Unit tests API tests Stress tests UI tests
Selenium,
Selenide
92
93. Что получили
● Все тестовые сценарии в коде
● Можно встроить в процесс сборки
или доставки ПО
● Если хотите, то можно
генерировать отчёты для разбора
полётов
93
98. ISupplier<String, Ctx> srcsetSupplier = (ctx) -> {
if (configuration. isDoubleDensityAvatarsEnabled(user.getModel())) {
String link = imageSrc.getModel();
return linkBuilder.createSourceSetLink(link);
}
return "";
};
Очень простой эксперимент
98
99. Code Branches
One server or
partition
99
//by partition
app.photos.doubleDensityAvatarsEnabled: 0
Step#1
100. Code Branches
One server or
partition
Part of servers
or partitions
100
//by partition
app.photos.doubleDensityAvatarsEnabled: 0,4,7-9
Step#1 Step#2
101. Code Branches
One server or
partition
Part of servers
or partitions
All servers or
partitions
101
//by partition
app.photos.doubleDensityAvatarsEnabled: ALL
Step#1 Step#2 Step#3
115. Что получили
● Эксперименты хранятся в git-
репозитории
● Можно применить любой шаг в
любой момент времени на любой
среде
● Можно встроить в пайплайн
доставки
115
122. = Hippo Rent Service
This is documentation for Open API of our hippo renting service
== Methods
=== Rent
==== Request specification
===== Headers
//тут опишем http-заголовки
===== Example
//а здесь будут примеры вызова
==== Response specification
===== Response fields
//здесь описание полей ответа
===== Example
//ну и пример того, что вообще ожидать в ответе
Вот такой документ
122
130. = Hippo Rent Service
This is documentation for Open API of our hippo renting service
== Methods
=== Rent
==== Request specification
===== Headers
include::{snippets}/rent-hippo/request-headers.adoc[]
===== Example
include::{snippets}/rent-hippo/http-request.adoc[]
==== Response specification
===== Response fields
include::{snippets}/rent-hippo/response-fields.adoc[]
===== Example
include::{snippets}/rent-hippo/http-response.adoc[]
Вставляем их в документ
130
131. = Hippo Rent Service
This is documentation for Open API of our hippo renting
service
== Methods
=== Rent
==== Request specification
===== Headers
include::{snippets}/rent-hippo/request-headers.adoc[]
===== Example
include::{snippets}/rent-hippo/http-request.adoc[]
==== Response specification
===== Response fields
include::{snippets}/rent-hippo/response-fields.adoc[]
===== Example
include::{snippets}/rent-hippo/http-response.adoc[]
131
133. Что получили?
● Документация как код
○ лежит в репозитории с кодом
○ встроена в цикл сборки
○ рендерится в html, pdf и т.д.
○ почти всегда актуальна
● Синергия с тестами
133
194. Позитивные выводы
● Почти любой процесс можно
формализовать, представить в
виде кода и автоматизировать
● Мы все пишем код, хотя можем
думать, что это не так
● Рано или поздно всё превращается
в код
194
195. Trade-offs
● Необходимы как разовые
“капиталовложения”, так и
постоянные затраты ресурсов
● Могут потребоваться изменения в
архитектуре
● Требует дисциплины и более
высокой квалификации
специалистов
195