This document describes techniques for scaling out a web application across multiple database shards. It introduces Incline, which handles denormalization and data replication between shards transparently using triggers and queue tables. It also discusses Pacific utilities like mysqld_jumpstart for provisioning MySQL instances and pacific_divide for splitting shards without downtime. Incline addresses issues with complex queries and consistency across shards under an eventual consistency model.
projekt, który wysłaliśmy na konkurs pt. Urban legends. publikujemy go w takiej formie dnia 26.03.2009 ze względu na często zdarzające się kradzieże pomysłów i projektów.
Design Patterns for Tablets and SmartphonesMichael Galpin
This is a talk I gave at AnDevCon. It talks about ways to take advantage of features introduced in Android 3.0 to create more modular and better looking apps.
Projet d'accès aux résultats des étudiant via client mobile Patrick Bashizi
J'ai réalisé ce petit projet en 2009 (je crois) dans le cadre d'un cours à la faculté. Je l'ai retrouvé par hasard et vu sa simplicité, il pourait aider quelqu'un..
Il s'agit d'une application mobile J2ME cliente à un web service, developpé dans le même cadre, permettant aux étudiants de s'authentifier et accéder aux résultants de leurs examens.
projekt, który wysłaliśmy na konkurs pt. Urban legends. publikujemy go w takiej formie dnia 26.03.2009 ze względu na często zdarzające się kradzieże pomysłów i projektów.
Design Patterns for Tablets and SmartphonesMichael Galpin
This is a talk I gave at AnDevCon. It talks about ways to take advantage of features introduced in Android 3.0 to create more modular and better looking apps.
Projet d'accès aux résultats des étudiant via client mobile Patrick Bashizi
J'ai réalisé ce petit projet en 2009 (je crois) dans le cadre d'un cours à la faculté. Je l'ai retrouvé par hasard et vu sa simplicité, il pourait aider quelqu'un..
Il s'agit d'une application mobile J2ME cliente à un web service, developpé dans le même cadre, permettant aux étudiants de s'authentifier et accéder aux résultants de leurs examens.
A pragmatic approach to different SQL Injection techniques such as Stacked statements, Tautology based, Union based, Error based, Second Order and Blind SQL Injection coherently explaining the path behind these attacks including tips and tricks to make them more likely to work in real life.
Also I will show you ways to avoid weak defenses as black listing and quote filtering as well as how privilege escalation may take place from this sort of vulnerabilities.
There will be a live demonstration where you can catch on some handy tools and actually see blind sql injection working efficiently with the latest techniques showing you why this type of SQL injection shouldn't be taken any less seriously than any other.
Finally, a word on countermeasures and real solutions to prevent these attacks, what you should do and what you should not.
http://videos.sapo.pt/ZvwITnTBMzD8HYvEZrov (video)
Over-the-Air: How we Remotely Compromised the Gateway, BCM, and Autopilot ECU...Priyanka Aash
We, Keen Security Lab of Tencent, have successfully implemented two remote attacks on the Tesla Model S/X in year 2016 and 2017. Last year, at Black Hat USA, we presented the details of our first attack chain. At that time, we showed a demonstration video of our second attack chain, but without technical aspects. This year, we are willing to share our full, in-depth details on this research.
In this presentation, we will explain the inner workings of this technology and showcase the new capability that was developed in the Tesla hacking 2017. Multiple 0-days of different in-vehicle components are included in the new attack chain.
We will also present an in-depth analysis of the critical components in the Tesla car, including the Gateway, BCM(Body Control Modules), and the Autopilot ECUs. For instance, we utilized a code-signing bypass vulnerability to compromise the Gateway ECU; we also reversed and then customized the BCM to play the Model X "Holiday Show" Easter Egg for entertainment.
Finally, we will talk about a remote attack we carried out to successfully gain an unauthorized user access to the Autopilot ECU on the Tesla car by exploiting one more fascinating vulnerability. To the best of our knowledge, this presentation will be the first to demonstrate hacking into an Autopilot module.
A slidedeck from the Manchester Serverless Meetup about our journey of adopting a serverless architecture. From the more traditional approach of provisioning servers, and the handoffs required resulting in lengthy delays to the promised land of serverless with the Serverless Framework.
Cross the Streams! Creating Streaming Data Pipelines with Apache Flink + Apac...StreamNative
Despite what the Ghostbusters said, we’re going to go ahead and cross (or, join) the streams. This session covers getting started with streaming data pipelines, maximizing Pulsar’s messaging system alongside one of the most flexible streaming frameworks available, Apache Flink. Specifically, we’ll demonstrate the use of Flink SQL, which provides various abstractions and allows your pipeline to be language-agnostic. So, if you want to leverage the power of a high-speed, highly customizable stream processing engine without the usual overhead and learning curves of the technologies involved (and their interconnected relationships), then this talk is for you. Watch the step-by-step demo to build a unified batch and streaming pipeline from scratch with Pulsar, via the Flink SQL client. This means you don’t need to be familiar with Flink, (or even a specific programming language). The examples provided are built for highly complex systems, but the talk itself will be accessible to any experience level.
1
Database Security
Lab 2 – Virtual Private Databases
2019
Part I: Implementing VPD by Views
This lab shows an example of implementing access control through views and triggers. Notice:
In this lab and the following ones, only the SYS user is the SYSDBA, the DBSEC user is a
normal user.
What to submit: Your answers to the questions in steps 5(a), 6, and 7.
1. Create DBSEC account. (using Database Control Tool)
(a) For the sake of simplicity, set its password as dbsec.
(b) Assign the following systems privileges to DBSEC.
CREATE ANY CONTEXT
CREATE PROCEDURE
CREATE SESSION
CREATE SEQUENCE
CREATE TABLE
CREATE TRIGGER
CREATE USER
CREATE VIEW
GRANT ANY OBJECT PRIVILEGE
GRANT ANY PRIVILEGE
GRANT ANY ROLE
(c) Assign Quotas to DBEC. Click Quotas Select Unlimited for USERS.
2. Create a user SCOTT. For the sake of simplicity, set its password as dbsec. Assign the
following system privileges to SCOTT:
CREATE SESSION
3. Logon as DBSEC and execute the following commands
drop table employee;
create table employee (
empID number(3) primary key,
Fname varchar2(25),
Lname varchar2(25),
Email varchar2(50),
Tel char(11),
Hdate Date,
JobID varchar2(10),
Salary number(6),
ManagerID number(3),
DeptID number(3),
ctl_upd_user varchar2(25));
2
drop view employee_view1;
create view employee_view1 as
select empID, Fname, Lname, Email, Tel, Hdate, JobID, Salary, managerID,
deptID, ctl_upd_user user_name
from employee
where ctl_upd_user = user;
grant select, insert, delete, update on employee_view1 to scott;
insert into employee_view1 values (100,'Sam','Doe', 'sdoe', '501-1112222',
sysdate, 'job1','60000', 199, 1, user);
commit;
4. Logon as SYS, and execute the following commands:
create or replace trigger TRG_EMPLOYEE_VIEW1_BF_INS
instead of insert on dbsec.EMPLOYEE_VIEW1
for each row
begin
insert into dbsec.EMPLOYEE values
(:new.empID, :new.Fname, :new.Lname, :new.Email, :new.Tel, :new.Hdate, :new.J
obID, :new.Salary, :new.ManagerID, :new.deptID, user);
end;
5. Logon as SCOTT and execute the following commands:
insert into dbsec.employee_view1 values (101,'Julia','Rice', 'sdoe', '501-
1013333', sysdate, 'job_julia','50000', 299, 1, user);
commit;
select *
from dbsec.employee_view1;
(a) Did the above query output the only rows that Scott is allowed to see?
6. Read the code of create or replace trigger TRG_EMPLOYEE_VIEW1_BF_INS. (in step 4)
(a) It differs from the create trigger syntax that we discussed in PL/SQL review. Google the
internet for the key words “instead of” and “trigger”, and explain what is instead of
trigger mainly used for.
(b) Explain what does Oracle do when Scott performs
insert into dbsec.employee_view1 values (101,'Julia','Rice', 'sdoe', '501-
1013333', sysdate, 'job_julia','50000', 299, 1, user);
commit;
3
(c) Create a trigge ...
We all know that load testing is important, but it's all too common that it's left to the very end of a project and it's invariably the first thing that gets dropped when budgets and timeframes get cut. Furthermore, most of us don't know where or how to start implementing effective load tests, let alone how to analyse the results.
Lindsay Holmwood, Software Manager at Bulletproof Networks, will be talking about integrating performance testing into your application development + deploy cycle from the very beginning, using inexpensive and easy to use SaaS tools.
There will be a hands on demonstration of the Blitz load + performance testing tool, coupled with a brief dive into the Blitz API internals to retrieve and analyse advanced reporting information.
[CB16] Esoteric Web Application Vulnerabilities by Andrés RianchoCODE BLUE
This talk will show esoteric web application vulnerabilities in detail, these vulnerabilities would be missed in a quick review by most security consultants, but could lead to remote code execution, authentication bypass and purchasing items in merchants using Paypal as their payment gateway without actually paying. SQL injections are dead, and I don’t care: let's explore the world of null, nil and NULL; noSQL injections; host header injections that lead to phone call audio interception; paypal’s double spent and Rails’ MessageVerifier remote code execution.
--- Andres Riancho
Andrés Riancho is an application security expert that currently leads the community driven, Open Source, w3af project and provides in-depth Web Application Penetration Testing services to companies around the world.
In the research field, he discovered critical vulnerabilities in IPS appliances from 3com and ISS, contributed with SAP research performed at one of his former employers and reported vulnerabilities in hundreds of web applications.
His main focus has always been the Web Application Security field, in which he developed w3af, a Web Application Attack and Audit Framework used extensively by penetration testers and security consultants.
Andrés has spoken and hold trainings at many security conferences around the globe, like BlackHat (USA and Europe), SEC-T (Sweden),DeepSec (Austria), PHDays (Moscow), SecTor (Toronto), OWASP (Poland),CONFidence (Poland), OWASP World C0n (USA), CanSecWest (Canada),PacSecWest (Japan), T2 (Finland) and Ekoparty (Buenos Aires).
Andrés founded Bonsai Information Security, a web security focused consultancy firm, in 2009 in order to further research into automated Web Application Vulnerability detection and exploitation.
This presentation will provide set of simple, yet powerful, JavaScript tips, tricks and best practices that should be known by all hybrid mobile app developers. These are techniques that all JavaScript programmers can use now, you didn't need to be an advanced JavaScript developer to benefit from these tips. After detailed explanations of how each technique works and when to use it, you will have become a more enlightened JavaScript developer, if you aren't already one.
5 minute presentation
20 slides
15 seconds per slide
This talk is a very quick intro to Docker, Terraform, and Amazon's EC2 Container Service (ECS). In just 15 minutes, you'll see how to take two apps (a Rails frontend and a Sinatra backend), package them as Docker containers, run them using Amazon ECS, and to define all of the infrastructure-as-code using Terraform.
Similar to A Clever Way to Scale-out a Web Application (20)
Presentation material for TokyoRubyKaigi11.
Describes techniques used by H2O, including: techniques to optimize TCP for responsiveness, server-push and cache digests.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Mission to Decommission: Importance of Decommissioning Products to Increase E...
A Clever Way to Scale-out a Web Application
1. A Clever Way to Scale-out
a Web Application
Cybozu Labs, Inc.
Kazuho Oku
2. RDB sharding
denormalization is inevitable
uid:1-2000
uid:2001-4000
uid:4001-6000
tweet
tweet
tweet
following
following
following
...
followed_by
followed_by
followed_by
timeline
timeline
timeline
when uid:123 tweets, write his tweet, read uids of his followers, and
update the timeline table of his followers
Sep 11 2009 A Clever Way to Scale-out a Web Application 2
3. Two methods to update the shards
eventual consistency
asynchonous updates using worker processes
pros: fast response, high scalability
cons: hard to maintain
2-phase commit
synchronous updates
pros: synchronous, doesn't require external
daemon
cons: slow response
Sep 11 2009 A Clever Way to Scale-out a Web Application 3
4. The problems
complex queries
reading from / writing to multiple DB nodes
cannot use secondary indexes
need to maintain per-user views (denormalized tables)
maintain consistency between the nodes
when using eventual consistency model
dynamic scaling
adding new nodes without stopping the service
Sep 11 2009 A Clever Way to Scale-out a Web Application 4
6. Incline
solution for the two problems of
eventual consistency:
complex update queries
maintenance of the denormalized tables
basic idea
do not let app. developers write denormalization
logic
handle denormalization below the SQL layer
by using triggers and queue tables
Sep 11 2009 A Clever Way to Scale-out a Web Application 6
7. Incline – illustrated
insert / update / delete rows of related
tables automatically
uid:1-2000
uid:2001-4000
uid:4001-6000
tweet
tweet
tweet
following
following
following
followed_by
followed_by
followed_by
...
timeline
timeline
timeline
queue
queue
queue
when uid:123 tweets, write only to his tweet table. Incline updates
other tables automatically
Sep 11 2009 A Clever Way to Scale-out a Web Application 7
8. Incline – illustrated (cont'd)
insert / update / delete rows of related
tables automatically
uid:1-2000
uid:2001-4000
uid:4001-6000
tweet
tweet
tweet
following
following
following
followed_by
followed_by
followed_by
...
timeline
timeline
timeline
queue
queue
queue
when uid:2431 starts following uid:940 only write to his following table
Sep 11 2009 A Clever Way to Scale-out a Web Application 8
9. Incline – details
triggers generated from def. files
sync. updates within each node
async. updates between the nodes
each DB node has a queue table
helper program (C++) applies the queued events
to other nodes
uses a fault tolerant algorithm
application only needs to write to the
user's shard
Sep 11 2009 A Clever Way to Scale-out a Web Application 9
10. Incline – the commands
# create queue tables
% incline --mode=shard --rdbms=mysql --database=microblog
--host=10.0.200.10 --source=microblog.json --shard-source=shard.json
create-queue
# create triggers
% incline --mode=shard --rdbms=mysql --database=microblog
--host=10.0.200.10 --source=microblog.json --shard-source=shard.json
create-trigger
# run forwarder (transfers data from specified host to other shards)
% incline --mode=shard --rdbms=mysql --database=microblog
--host=10.0.200.10 --source=microblog.json --shard-source=shard.json
forward
Sep 11 2009 A Clever Way to Scale-out a Web Application 10
12. Incline – FYI the generated triggers
CREATE TRIGGER _INCLINE_followed_by_INSERT AFTER INSERT ON followed_by FOR EACH NEW.following_id,NEW.user_id,'I';
ROW BEGIN
END IF;
IF (((1<=NEW.follower_id AND NEW.follower_id<2001))) THEN
ENDCREATE TRIGGER _INCLINE_following_DELETE AFTER DELETE ON following FOR EACH
INSERT INTO timeline (user_id,ctime,tweet_id,tweet_user_id) SELECT ROW BEGIN
NEW.follower_id,tweet.ctime,tweet.tweet_id,tweet.user_id FROM tweet WHERE IF (((1<=OLD.following_id AND OLD.following_id<2001))) THEN
tweet.user_id=NEW.user_id;
DELETE FROM followed_by WHERE followed_by.user_id=OLD.following_id AND
ELSE
followed_by.follower_id=OLD.user_id;
INSERT INTO _iq_timeline (user_id,ctime,tweet_id,tweet_user_id,_iq_action) ELSE
SELECT NEW.follower_id,tweet.ctime,tweet.tweet_id,tweet.user_id,'I' FROM
INSERT INTO _iq_followed_by (user_id,follower_id,_iq_action) SELECT
tweet WHERE tweet.user_id=NEW.user_id;
OLD.following_id,OLD.user_id,'D';
END IF;
END IF;
END
END
CREATE TRIGGER _INCLINE_followed_by_UPDATE AFTER UPDATE ON followed_by FOR EACH CREATE TRIGGER _INCLINE_tweet_INSERT AFTER INSERT ON tweet FOR EACH ROW BEGIN
ROW BEGIN
INSERT INTO timeline (user_id,ctime,tweet_id,tweet_user_id) SELECT
IF (((1<=NEW.follower_id AND NEW.follower_id<2001))) THEN
followed_by.follower_id,NEW.ctime,NEW.tweet_id,NEW.user_id FROM
REPLACE INTO timeline (user_id,ctime,tweet_id,tweet_user_id) SELECT followed_by WHERE ((1<=followed_by.follower_id AND
NEW.follower_id,tweet.ctime,tweet.tweet_id,tweet.user_id FROM tweet WHERE followed_by.follower_id<2001)) AND NEW.user_id=followed_by.user_id;
tweet.user_id=NEW.user_id;
INSERT INTO _iq_timeline (user_id,ctime,tweet_id,tweet_user_id,_iq_action)
ELSE
SELECT followed_by.follower_id,NEW.ctime,NEW.tweet_id,NEW.user_id,'I' FROM
INSERT INTO _iq_timeline (user_id,ctime,tweet_id,tweet_user_id,_iq_action) followed_by WHERE NOT (((1<=followed_by.follower_id AND
SELECT NEW.follower_id,tweet.ctime,tweet.tweet_id,tweet.user_id,'U' FROM followed_by.follower_id<2001))) AND NEW.user_id=followed_by.user_id;
tweet WHERE tweet.user_id=NEW.user_id;
END
END IF;
CREATE TRIGGER _INCLINE_tweet_UPDATE AFTER UPDATE ON tweet FOR EACH ROW BEGIN
END
REPLACE INTO timeline (user_id,ctime,tweet_id,tweet_user_id) SELECT
CREATE TRIGGER _INCLINE_followed_by_DELETE AFTER DELETE ON followed_by FOR EACH followed_by.follower_id,NEW.ctime,NEW.tweet_id,NEW.user_id FROM
ROW BEGIN
followed_by WHERE ((1<=followed_by.follower_id AND
IF (((1<=OLD.follower_id AND OLD.follower_id<2001))) THEN
followed_by.follower_id<2001)) AND NEW.user_id=followed_by.user_id;
DELETE FROM timeline WHERE timeline.user_id=OLD.follower_id AND INSERT INTO _iq_timeline (user_id,ctime,tweet_id,tweet_user_id,_iq_action)
tweet_user_id=OLD.user_id;
SELECT followed_by.follower_id,NEW.ctime,NEW.tweet_id,NEW.user_id,'U' FROM
ELSE
followed_by WHERE NOT (((1<=followed_by.follower_id AND
followed_by.follower_id<2001))) AND NEW.user_id=followed_by.user_id;
INSERT INTO _iq_timeline (user_id,tweet_id,tweet_user_id,_iq_action) SELECT
OLD.follower_id,tweet.tweet_id,tweet.user_id,'D' FROM tweet WHERE END
tweet.user_id=OLD.user_id;
CREATE TRIGGER _INCLINE_tweet_DELETE AFTER DELETE ON tweet FOR EACH ROW BEGIN
END IF;
DELETE FROM timeline WHERE timeline.tweet_id=OLD.tweet_id AND
timeline.tweet_user_id=OLD.user_id;
END
INSERT INTO _iq_timeline (tweet_id,tweet_user_id,user_id,_iq_action) SELECT
CREATE TRIGGER _INCLINE_following_INSERT AFTER INSERT ON following FOR EACH ROW
OLD.tweet_id,OLD.user_id,followed_by.follower_id,'D' FROM followed_by
BEGIN
WHERE OLD.user_id=followed_by.user_id AND NOT
IF (((1<=NEW.following_id AND NEW.following_id<2001))) THEN
(((1<=followed_by.follower_id AND followed_by.follower_id<2001)));
INSERT INTO followed_by (user_id,follower_id) SELECT
END
NEW.following_id,NEW.user_id;
ELSE
INSERT INTO _iq_followed_by (user_id,follower_id,_iq_action) SELECT
Sep 11 2009 A Clever Way to Scale-out a Web Application 12
14. Range-based sharding vs. hash-based
Range-based sharding is better
range queries are sometimes necessary
manual tuning is easy
number of nodes increase continuously
with hash-based sharding, you have to add
1,2,4,8,16,32,64,... servers at once
Sep 11 2009 A Clever Way to Scale-out a Web Application 14
15. Pacific
utility programs for dynamic scaling
mysqld_jumpstart
pacific_divide
Sep 11 2009 A Clever Way to Scale-out a Web Application 15
16. mysqld_jumpstart – summary
create a mysqld instance in a single
command
service automatically started by daemontools
setup of primary nodes and slaves
auto-generated backup script: install_dir/etc/
backup.sh
uses XtraBackup for hot-backup
Sep 11 2009 A Clever Way to Scale-out a Web Application 16
17. mysql_jumpstart – the commands
# create and start a master database
% mysqld_jumpstart --mysql-install-db=/usr/local/mysql/bin/
mysql_install_db --mysqld=/usr/local/mysql/libexec/mysqld --base-
dir=/var/servicedb --server-id=1252619462 --socket=/tmp/mysql-
servicedb.sock --service-dir=/service/mysql-servicedb --replication-
network='10.0.200.0/255.255.255.0'
# backup
% /var/servicedb/etc/backup.sh /var/backup/servicedb.backup.20090911
# create and start a slave database
% mysqld_jumpstart --mysql-install-db=/usr/local/mysql/bin/
mysql_install_db --mysqld=/usr/local/mysql/libexec/mysqld --base-
dir=/var/servicedb --server-id=1252619493 --socket=/tmp/mysql-
servicedb.sock --service-dir=/service/mysql-servicedb --replication-
network='10.0.200.0/255.255.255.0' --master-host=10.0.200.1 --from-
innobackupex
Sep 11 2009 A Clever Way to Scale-out a Web Application 17
18. Splitting a MySQL shard
use replication to prepare, then upgrade
a slave to master
Before:
1 2,000
2,001 4,000
4,001 6,000
replication
slave
After:
1 2,000
2,001 3,000
3,001 4,000
4,001 6,000
Sep 11 2009 A Clever Way to Scale-out a Web Application 18
19. Problems in splitting a shard
speed vs. safety
downtime should be minimum
guarantee that all the application servers write to
the new node
reads may switch to the new node eventually
Sep 11 2009 A Clever Way to Scale-out a Web Application 19
20. Pacific_divide – the blurbs
fail-safe
application servers using the old sharding
definition cannot access the split nodes
app. servers reload the definition upon such case
minimum impact on users
no read-locks during division
in eventual-consistency mode
acquires write lock only against the dividing node
write lock time < 10 seconds
if no delay in replication
Sep 11 2009 A Clever Way to Scale-out a Web Application 20
21. Pacific_divide – the split algorithm
1. create a new slave node
2. drop write privileges of existing username on the dividing
node
3. wait until the new node becomes in sync.
4. update incline triggers
5. create new user and give read / write privileges
6. update shard def.
7. drop read privileges granted to the old username
Sep 11 2009 A Clever Way to Scale-out a Web Application 21
22. Pacific_divide – the comand
# upgrade 10.0.200.18 to a master with range uid:3,000-
#
# when instructed by pacific_divide, transmit shard.json to all
# application servers and mysql shards (or you may use nfs, etc.)
% pacific_divide --shard-def=shard.json --database=microblog --new-
host=10.0.200.18 --from-id=3000 --incline-source=microblog.json
Before:
1 2,000
2,001 4,000
4,001 6,000
replication
slave
After:
1 2,000
2,001 3,000
3,001 4,000
4,001 6,000
Sep 11 2009 A Clever Way to Scale-out a Web Application 22
25. DBIx::ShardManager – the code
# create manager object
my $mgr = DBIx::ShardManager->new(
definition => DBIx::ShardManager::Definition::JSON->new(
file => 'etc/user_shard_def.json',
auto_reload => 1,
),
connector => DBIx::ShardManager::Connector::DBI->new(
driver => 'mysql',
dbname => 'microblog',
attr => {
mysql_enable_utf8 => 1,
RaiseError => 1,
},
),
);
Sep 11 2009 A Clever Way to Scale-out a Web Application 25
26. DBIx::ShardManager – the code (cont'd)
# read user's timeline
# first, read my timeline table
my $timeline = $mgr->rw_handle($user_id)->selectall_arrayref(
'SELECT * FROM timeline WHERE user_id=? ORDER BY ctime DESC LIMIT
20',
{ Slice => {} },
$user_id,
);
# fetch the tweets using (tweet_user_id,tweet_id) from other shards
$mgr->shard_inner_join(
$timeline,
tweet_user_id => {
'tweet.tweet_id' => 'tweet_id',
},
}
Sep 11 2009 A Clever Way to Scale-out a Web Application 26
27. DBIx::ShardManager – blurbs
access to raw DBI handles
easy to use ORM above DBIx::ShardManager
detects changes and reloads shard def.
but may throw exceptions on writes during node
divisions by pacific_divide
display maintenance error, and let the user retry
shard_join to be optimized
with Net::Drizzle, or mycached
Sep 11 2009 A Clever Way to Scale-out a Web Application 27
29. Conclusion
RDB sharding is not difficult when using
Incline, Pacific, DBIx::ShardManager
IMO it is as easy as writing code for a standalone
database system
app. developers can use 2-phase commit
if necessary
or rely on Incline for async. updates
Sep 11 2009 A Clever Way to Scale-out a Web Application 29
30. Current Status & ToDo
Incline - early beta
ToDo: add support for multiple shard keys, add
recovery support on data-loss
Pacific - early beta
ToDo: make it a distribution
DBIx::ShardManager - still alpha
ToDo: write more join functions, concurrent
access, etc.
Sep 11 2009 A Clever Way to Scale-out a Web Application 30
31. Miscellaneous
Mycached
currently in alpha status
access MySQL tables using memcached protocol
higher concurrency (thousands of connections)
higher throughput (2x SQL)
Sep 11 2009 A Clever Way to Scale-out a Web Application 31
32. For more information
see my blog
http://developer.cybozu.co.jp/kazuho/
DBIx::ShardManager is in coderepos.org/share/
lang/perl
come to BPStudy #25 on 9/25
2h30m talk on Incline, Pacific,
DBIx::ShardManager (hopefully including demos)
Sep 11 2009 A Clever Way to Scale-out a Web Application 32