This document provides an overview and instructions for installing and configuring ProxySQL. It discusses:
1. What ProxySQL is and its functions like load balancing and query caching
2. How to install ProxySQL on CentOS and configure the /etc/proxysql.cnf file
3. How to set up the ProxySQL schema to define servers, users, variables and other settings needed for operation
4. How to test ProxySQL functions like server status changes and benchmark performance
【de:code 2020】 Azure Red hat OpenShift (ARO) によるシステムアーキテクチャ構築の実践日本マイクロソフト株式会社
コンテナをベースとしたプラットフォーム上でのシステム構築において、システムアーキテクチャの設計、構築、運用を効率的に行うために、Kubernetes をラップしてデプロイや運用機能の付加機能をもつ OpenShift を利用することにしました。インフラ運用負荷を軽減する観点から、マイクロソフトのマネージドサービスである Azure Red Hat OpenShift (ARO) を使ってみました。本プラットフォームにおいて、エンタープライズレベルのシステムを稼働させるのに必要になる開発・運用を含めた全体アーキテクチャの概要、選定したソリューションや実現案を紹介します。
InfluxDB IOx Tech Talks: Query Engine Design and the Rust-Based DataFusion in...InfluxData
The document discusses updates to InfluxDB IOx, a new columnar time series database. It covers changes and improvements to the API, CLI, query capabilities, and path to open sourcing builds. Key points include moving to gRPC for management, adding PostgreSQL string functions to queries, optimizing functions for scalar values and columns, and monitoring internal systems as the first step to releasing open source builds.
At a blistering pace and for a variety of reasons, companies are migrating their on-premise database infrastructures to cloud-based solutions - to save costs on hardware, tame the impact of disaster recovery, or even to improve security. Zalando is not an exception: more than two years ago we migrated our first production services to AWS.
In addition to the fully managed database services like RDS and Aurora, Amazon offers a wide spectra of EC2 Instances with different types of performance and price. Without a lot of experience in running cloud databases it’s not easy to make the right choice, and as a result you will either have pure database performance or will overpay for over-provisioned resources.
In this talk I will compare different ways of running PostgreSQL on AWS, explain why we decided to run most of our databases on EC2 Instances instead of RDS, how we chose EC2 Instance types and EBS Volumes, which AWS CloudWatch metrics MUST be monitored (and why), and what problems we hit plus how to avoid them.
This document contains 3 links related to Azure managed disks. The first two links are to Microsoft documentation pages about Azure managed disks overview and FAQs for disks. The third link goes to a GitHub repository called MDPP.
Streaming Data Analytics with Amazon Redshift and Kinesis FirehoseAmazon Web Services
by Joyjeet Banerjee, Enterprise Solutions Architect, AWS
Evolving your analytics from batch processing to real-time processing can have a major business impact, but ingesting streaming data into your data warehouse requires building complex streaming data pipelines. Amazon Kinesis Firehose solves this problem by making it easy to transform and load streaming data into Amazon Redshift so that you can use existing analytics and business intelligence tools to extract information in near real-time and respond promptly. In this session, we will dive deep using Amazon Kinesis Firehose to load streaming data into Amazon Redshift reliably, scalably, and cost-effectively. Level: 200
This document provides an overview and instructions for installing and configuring ProxySQL. It discusses:
1. What ProxySQL is and its functions like load balancing and query caching
2. How to install ProxySQL on CentOS and configure the /etc/proxysql.cnf file
3. How to set up the ProxySQL schema to define servers, users, variables and other settings needed for operation
4. How to test ProxySQL functions like server status changes and benchmark performance
【de:code 2020】 Azure Red hat OpenShift (ARO) によるシステムアーキテクチャ構築の実践日本マイクロソフト株式会社
コンテナをベースとしたプラットフォーム上でのシステム構築において、システムアーキテクチャの設計、構築、運用を効率的に行うために、Kubernetes をラップしてデプロイや運用機能の付加機能をもつ OpenShift を利用することにしました。インフラ運用負荷を軽減する観点から、マイクロソフトのマネージドサービスである Azure Red Hat OpenShift (ARO) を使ってみました。本プラットフォームにおいて、エンタープライズレベルのシステムを稼働させるのに必要になる開発・運用を含めた全体アーキテクチャの概要、選定したソリューションや実現案を紹介します。
InfluxDB IOx Tech Talks: Query Engine Design and the Rust-Based DataFusion in...InfluxData
The document discusses updates to InfluxDB IOx, a new columnar time series database. It covers changes and improvements to the API, CLI, query capabilities, and path to open sourcing builds. Key points include moving to gRPC for management, adding PostgreSQL string functions to queries, optimizing functions for scalar values and columns, and monitoring internal systems as the first step to releasing open source builds.
At a blistering pace and for a variety of reasons, companies are migrating their on-premise database infrastructures to cloud-based solutions - to save costs on hardware, tame the impact of disaster recovery, or even to improve security. Zalando is not an exception: more than two years ago we migrated our first production services to AWS.
In addition to the fully managed database services like RDS and Aurora, Amazon offers a wide spectra of EC2 Instances with different types of performance and price. Without a lot of experience in running cloud databases it’s not easy to make the right choice, and as a result you will either have pure database performance or will overpay for over-provisioned resources.
In this talk I will compare different ways of running PostgreSQL on AWS, explain why we decided to run most of our databases on EC2 Instances instead of RDS, how we chose EC2 Instance types and EBS Volumes, which AWS CloudWatch metrics MUST be monitored (and why), and what problems we hit plus how to avoid them.
This document contains 3 links related to Azure managed disks. The first two links are to Microsoft documentation pages about Azure managed disks overview and FAQs for disks. The third link goes to a GitHub repository called MDPP.
Streaming Data Analytics with Amazon Redshift and Kinesis FirehoseAmazon Web Services
by Joyjeet Banerjee, Enterprise Solutions Architect, AWS
Evolving your analytics from batch processing to real-time processing can have a major business impact, but ingesting streaming data into your data warehouse requires building complex streaming data pipelines. Amazon Kinesis Firehose solves this problem by making it easy to transform and load streaming data into Amazon Redshift so that you can use existing analytics and business intelligence tools to extract information in near real-time and respond promptly. In this session, we will dive deep using Amazon Kinesis Firehose to load streaming data into Amazon Redshift reliably, scalably, and cost-effectively. Level: 200
An introduction and future of Ruby coverage librarymametter
Ruby's current test coverage feature, coverage.so, only measures line coverage. The speaker proposes expanding it to support function and branch coverage in Ruby 2.5. This would involve updating the coverage.so API to return additional coverage data types and structure the output data in a more extensible way. A preliminary demo applying the new coverage.so to Ruby code showed it can integrate with C code coverage from GCOV and display results in LCOV format. The speaker seeks feedback on the proposed API design to finalize it for Ruby 2.5.
The presentation at DevFest Tokyo 2017 / @__timakin__
An introduction of blockchain and why go is nice to implement blockchain.
Additionally described about the blockchain projects that are based on Go.
Operations: Production Readiness Review – How to stop bad things from HappeningAmazon Web Services
The document provides an overview of key areas to review for production readiness including architecture design, monitoring, logging, documentation, alerting, service level agreements, expected throughput, testing, and deployment strategy. It summarizes best practices and considerations for each area such as using circuit breakers in monitoring, consistent logging formats, storing documentation near code, automating level 1 operations, and strategies for testing, deployments, and managing error budgets.
Apache Spark Streaming + Kafka 0.10 with Joan ViladrosarieraSpark Summit
Spark Streaming has supported Kafka since it’s inception, but a lot has changed since those times, both in Spark and Kafka sides, to make this integration more fault-tolerant and reliable.Apache Kafka 0.10 (actually since 0.9) introduced the new Consumer API, built on top of a new group coordination protocol provided by Kafka itself. So a new Spark Streaming integration comes to the playground, with a similar design to the 0.8 Direct DStream approach. However, there are notable differences in usage, and many exciting new features. In this talk, we will cover what are the main differences between this new integration and the previous one (for Kafka 0.8), and why Direct DStreams have replaced Receivers for good. We will also see how to achieve different semantics (at least one, at most one, exactly once) with code examples. Finally, we will briefly introduce the usage of this integration in Billy Mobile to ingest and process the continuous stream of events from our AdNetwork.
神に近づくx/net/context (Finding God with x/net/context)guregu
This document discusses different approaches to building an authentication middleware in Go web applications. It begins with using the standard library, then explores Goji and its request context. It settles on using the x/net/context package and kami router, which allow sharing database connections and authentication objects across requests and tests through the request context. Middleware is defined hierarchically in kami. This approach avoids global variables and simplifies testing.
by Joyjeet Banerjee, Enterprise Solutions Architect, AWS
Amazon Aurora is a MySQL- and PostgreSQL-compatible database engine that combines the speed and availability of high-end commercial databases with the simplicity and cost-effectiveness of open source databases. In this deep dive session, we’ll discuss best practices and explore new features in areas like high availability, security, performance management and database cloning. Level 300
1) Mercari has transitioned some services to microservices architecture running on Kubernetes in the US region to improve development velocity.
2) Key challenges in operating microservices include deployment automation using Spinnaker, and observability of distributed systems through request tracing, logging, and metrics.
3) The architecture is still evolving with discussions on service mesh and chaos engineering to improve reliability in the face of failures. Microservices adoption is just beginning in the JP region.
The document discusses using gRPC and Protocol Buffers to build fast and reliable APIs, describing how gRPC uses Protocol Buffers to define service interfaces and handle serialization, and allows building clients and servers in various languages that can communicate over the network through language-independent services. It provides examples of using gRPC to define and call both unary and streaming RPC services from Swift clients and servers.
Andrew Betts Web Developer, The Financial Times at Fastly Altitude 2016
Running custom code at the Edge using a standard language is one of the biggest advantages of working with Fastly’s CDN. Andrew gives you a tour of all the problems the Financial Times and Nikkei solve in VCL and how their solutions work.
This document contains the transcript from a presentation titled "So You Wanna Go Fast?" by Tyler Treat. Some of the key topics discussed include measuring performance using tools like pprof, how different language features in Go like channels, interfaces, and memory management can impact performance, and techniques for writing concurrent and multi-core friendly code in Go like using read-write mutexes. The overall message is that performance depends greatly on the specific situation and trade-offs must be considered between concurrency, memory usage, and execution speed. Measuring first is emphasized to guide any optimizations.
The document discusses building apps for the Google Assistant using Google Cloud Functions and Actions on Google. It provides an overview of the architecture, development workflow, and ways for users to discover apps. Key points include using Cloud Functions as a serverless environment to handle requests, the Actions Console for configuration, and in-dialogue discovery or the Assistant Directory as ways for users to find actions.
MySQL 5.7 and MySQL 8.0 have an issue that all slave's replications are stopped.
Current status of fixing
MySQL 5.7 fixed at 5.7.25
MySQL 8.0 fixed at 5.8.14
This document discusses different ways to migrate an existing database table to a sharded structure using the Spider storage engine in MariaDB. It covers using replication, triggers, Spider functions, and vertical partitioning. The replication method involves copying data to new tables, starting replication, and switching to the new structure. The trigger method uses triggers to copy data in real-time. Spider functions allow copying data without locks. Vertical partitioning splits the table across multiple servers based on column values.
When your database is growing, you definitely need to think about other techniques like database sharding. SPIDER is a MariaDB Server / MySQL storage engine for database sharding. Using SPIDER, you can access your data efficiently across multiple database backends.
In this time we will introduce the following things.
1. why SPIDER? what SPIDER can do for you?
2. when SPIDER is right for you? what cases should you use SPIDER?
3. how long is SPIDER used in the big environment?
4. SPIDER sharding architecture
5. how to get SPIDER working?
6. multi dimenstional sharding technique with VP storage engine
7. roadmap of SPIDER
8. where to get SPIDER (with VP)
This document discusses Spider, a storage engine plugin for MariaDB/MySQL that allows sharding and partitioning of tables across multiple remote databases. Key points:
- Spider provides database sharding by using table partitioning to divide huge datasets across multiple servers for high traffic processing and parallel processing.
- An application can use multiple backend databases as one database through Spider by connecting only to the Spider database.
- Spider's features include redundancy, fault tolerance, fulltext/geo search, and connecting to Oracle databases. Its roadmap includes improving startup performance, reducing memory usage, and direct joining of data on backend nodes.
Newest topic of spider 20131016 in Buenos Aires ArgentinaKentoku
Spider Storage Engine is a plugin for MySQL/MariaDB that allows tables to be sharded across multiple database servers for high traffic processing and parallel querying. It provides a single interface to applications while data is stored across multiple databases. Spider tables can reference tables in MySQL, MariaDB, and OracleDB. This allows huge amounts of data to be divided across servers transparently to users. Spider also includes features for fault tolerance, fulltext/geo search, and integration with other plugins like Handlersocket and Mroonga for additional functionality.
Spider's HA structure includes data nodes, spider nodes, and monitoring nodes. Data nodes store data, spider nodes provide load balancing and failover, and monitoring nodes monitor data nodes. To add a new data node without stopping service: 1) Create a new table on the node, 2) Alter tables on monitoring nodes to include new node, 3) Alter clustered table connection to include new node, 4) Copy data to new node. This maintains redundancy when a node fails without service interruption.
22. When SPIDER is right for you? What cases should you use SPIDER?
Spiderは以下のような要件がある場合に、
利用をご検討ください。
3.任意のルールでシャーディングを行いたい
場合
4.シャーディングと一貫性が同時に必要である
場合
32. Spiderのセットアップ(2/5)
1対1Spiderテーブルの作成
CREATE TABLE t1(
c1 int,
c2 varchar(100),
PRIMARY KEY(c1)
)ENGINE=spider DEFAULT CHARSET=utf8
COMMENT '
table "rt1", database "test", port "3306",
host "host name of data node",
user "user name for data node",
password "password for data node"
';
Engine名に“Spider”を指定し、接続情報とパラメータを
Commentに記載する。
34. Spiderのセットアップ(4/5)
1対多(シャーディング)Spiderテーブルの作成
CREATE TABLE t1(
c1 int,
c2 varchar(100),
PRIMARY KEY(c1)
)ENGINE=spider DEFAULT CHARSET=utf8
COMMENT 'table "rt1", database "test", port "3306",
user "user name for data node", password "password for data node"'
PARTITION BY RANGE(c1) (
PARTITION p0 VALUES LESS THAN (100000) COMMENT 'host "h1"',
PARTITION p1 VALUES LESS THAN (200000) COMMENT 'host "h2"',
PARTITION p2 VALUES LESS THAN (300000) COMMENT 'host "h3"',
PARTITION p3 VALUES LESS THAN MAXVALUE COMMENT 'host "h4"'
);
共通の接続情報をテーブルのCommentに記載する。
シャード毎に異なる接続情報をパーティションのCommentに記載する。
35. Spiderのセットアップ(5/5)
“CREATE SERVER”コマンドで接続情報を事前に定義することも可能です。
CREATE SERVER srv1
FOREIGN DATA WRAPPER mysql
HOST 'host name of data node',
DATABASE 'test',
USER 'user name for data node',
PASSWORD 'password for data node',
PORT 3306
;
上記で定義したサーバ定義は、SpiderではCommentに“server”パラメータ
として記述することができます。
CREATE TABLE t1(
c1 int,
c2 varchar(100),
PRIMARY KEY(c1)
)ENGINE=spider DEFAULT CHARSET=utf8
COMMENT 'table "rt1", server "srv1"';