SlideShare a Scribd company logo
can you download
the whole Instagram?
A bold tale of struggle and friendship
Agenda
➔ start a demo
➔ understand the challenge
➔ integrate people
➔ HOW WE DID IT
➔ see demo results
CEO COO Prod. owner Developer CTO
BusinessAwareness
THE COMPANY GAP
Technical Awareness
making
instagram profiles
SEARCHABLE
THE NEW GUY
CEO COO Prod. owner Developer CTO
BusinessAwareness
THE COMPANY GAP
Technical Awareness
Getting into engineering
LevelofBlackMagic
Technical skills
Level zero in dev.
Study & Google
copy/paste

npm install
NEED TO REDUCE POINTS
Getting into engineering
Technical skills
New guy level
LevelofBlackMagic
Gained more knowledge
Introduced to “THE BACKEND”
First full-stack feature
A NEW HOPE
• Look at competitors data
• Find public API

(json bases)
What is a queue?
REQUIREMENTS ● PARALLEL JOBS
● FAULTS TOLERANCE
● UNIQUE DOCUMENTS
● RESCHEDULE DOCUMENTS
● AVOID RE-INSERT
● COST EFFICIENT
● JAVASCRIPT
CREATE TABLE tasks (
subject CHARACTER … PRIMARY KEY,
next_iteration TIMESTAMP,
attempts INTEGER,
payload JSONB
);
DISCOVERY
PROFILE

TRACKER
PROFILE

BUILDER
LEARN
Dataprocessing
Knowledge
300k
100M+
1 WORKERS ~ 10 profiles / minute
CHALLENGE
2 WORKERS ~ 13 profiles / minute
3 WORKERS ~ 14 profiles / minute
SELECT subject FROM tasks
WHERE next_iteration <= NOW()
LIMIT 1;
UPDATE tasks
SET next_iteration = NOW() + interval ‘5m’
WHERE subject = ‘xxx’;
SELECT subject FROM tasks
WHERE next_iteration <= NOW()
LIMIT 1;
UPDATE tasks
SET next_iteration = NOW() + interval ‘5m’
WHERE subject = ‘xxx’;
WORKERS GAP
UPDATE tasks SET
next_iteration = NOW() + interval ‘5m’
WHERE subject in (
SELECT subject FROM tasks
WHERE next_iteration <= NOW()
LIMIT 1
FOR UPDATE SKIP LOCKED
)
RETURNING subject;
Dataprocessing
Knowledge
300k
1M
100% CPU CREDITS LIMIT
CHALLENGE
100% DISK I/O LIMIT
~200 $/month “m4.large” machine
INSERT INTO tasks
(subject, next_iteration)
VALUES
(`xxx`, NOW() ),
(`yyy`, NOW() + INTERVAL ‘5m’ ),
(`zzz`, NOW() - INTERVAL ‘1y’ );
for (const task in stuff_to_insert) {
const exists = await query(`
SELECT subject FROM tasks
WHERE subject = '`${task}`' LIMIT 1
`);
if (!exists) {
await query("INSERT INTO ...")
}
}
INSERT INTO tasks
(subject, next_iteration)
VALUES
(`xxx`, NOW() ),
(`yyy`, NOW() + INTERVAL ‘5m’ ),
(`zzz`, NOW() - INTERVAL ‘1y’ )
ON CONFLICT
DO NOTHING;
Dataprocessing
Knowledge
300k
1M
5M
40s+ to pick a document!
CHALLENGE
SELECT FROM tasks
WHERE next_iteration < NOW()
AND attempts < 5
ORDER BY
next_iteration ASC,
attempts ASC
LIMIT 1;
CREATE TABLE tasks (
subject CHARACTER … PRIMARY KEY,
next_iteration TIMESTAMP,
status INTEGER,
attempts INTEGER,
payload JSONB
);
SELECT FROM tasks
WHERE status = 1
AND attempts < 5
ORDER BY
next_iteration ASC,
attempts ASC
LIMIT 1;
CREATE INDEX pending ON tasks(
next_iteration ASC,
attempts ASC
)
WHERE (
status = 1
AND
attempts < 5
);
Need to update STATUS
CHALLENGE
UPDATE tasks SET status = 1
WHERE subject IN (
SELECT subject FROM tasks
WHERE status = 0
AND next_iteration < NOW()
FOR UPDATE SKIP LOCKED
);
Dataprocessing
Knowledge
300k
1M
20M
5M
SCALE
Scaling in AWS
➔ EC2 Clicking
➔ Auto Scaling Groups
➔ CloudFormation
Speed differences among queues
Errors (thousands of them!)
CHALLENGE
WITH BIG DATA
COMES BIG TROUBLES
true story
MEET GRAFANA
What to measure?
# How much cumulated work?
SELECT COUNT (*) FROM tasks
WHERE status = 1;
# How much planned word?
SELECT COUNT (*) FROM tasks
WHERE status = 0;
# Other metrics…
CREATE TABLE metrics (
metric CHARACTER … NOT NULL,
value INTEGER
);
CREATE TABLE metrics_logs (
ctime TIMESTAMP,
metric CHARACTER … NOT NULL,
increment INTEGER
);
INSERT INTO metrics_logs (
ctime, metric, increment
) VALUES (
NOW(), `pending`, 1
);
FUNCTIONS
FOR VAR_r IN
SELECT
DISTINCT ON (metric),
metric, increment
FROM metrics_logs
ORDER BY ctime ASC
LOOP
…
END LOOP;
SELECT SUM(increment) INTO VAR_sum
FROM metrics_logs
WHERE metric = VAR_r.metric;
UPDATE metrics AS t
SET value = t.value + VAR_sum
WHERE metric = VAR_r.metric;
Dataprocessing
Knowledge
300k
1M
20M
100M+
5M
TAKEAWAY
➔ With good communication comes
good results
➔ Keep it simple
➔ No limits for new knowledge
➔ Mindset, curiosity, improvements
CEO COO Prod. owner Developer CTO
BusinessAwareness
THE COMPANY GAP
Technical Awareness

More Related Content

Similar to FetchQ - the origins

jQuery & 10,000 Global Functions: Working with Legacy JavaScript
jQuery & 10,000 Global Functions: Working with Legacy JavaScriptjQuery & 10,000 Global Functions: Working with Legacy JavaScript
jQuery & 10,000 Global Functions: Working with Legacy JavaScript
Guy Royse
 
In search of JavaScript code quality: unit testing
In search of JavaScript code quality: unit testingIn search of JavaScript code quality: unit testing
In search of JavaScript code quality: unit testing
Anna Khabibullina
 
Grokking TechTalk #24: Thiết kế hệ thống Background Job Queue bằng Ruby & Pos...
Grokking TechTalk #24: Thiết kế hệ thống Background Job Queue bằng Ruby & Pos...Grokking TechTalk #24: Thiết kế hệ thống Background Job Queue bằng Ruby & Pos...
Grokking TechTalk #24: Thiết kế hệ thống Background Job Queue bằng Ruby & Pos...
Grokking VN
 
PostgreSQL Open SV 2018
PostgreSQL Open SV 2018PostgreSQL Open SV 2018
PostgreSQL Open SV 2018
artgillespie
 
Triggers and Stored Procedures
Triggers and Stored ProceduresTriggers and Stored Procedures
Triggers and Stored Procedures
Tharindu Weerasinghe
 
Second Level Cache in JPA Explained
Second Level Cache in JPA ExplainedSecond Level Cache in JPA Explained
Second Level Cache in JPA Explained
Patrycja Wegrzynowicz
 
Tddbdd workshop
Tddbdd workshopTddbdd workshop
Tddbdd workshop
BestBrains
 
Javascript unit testing, yes we can e big
Javascript unit testing, yes we can   e bigJavascript unit testing, yes we can   e big
Javascript unit testing, yes we can e big
Andy Peterson
 
NetBeans Plugin Development: JRebel Experience Report
NetBeans Plugin Development: JRebel Experience ReportNetBeans Plugin Development: JRebel Experience Report
NetBeans Plugin Development: JRebel Experience Report
Anton Arhipov
 
When to NoSQL and when to know SQL
When to NoSQL and when to know SQLWhen to NoSQL and when to know SQL
When to NoSQL and when to know SQL
Simon Elliston Ball
 
Apache Spark in your likeness - low and high level customization
Apache Spark in your likeness - low and high level customizationApache Spark in your likeness - low and high level customization
Apache Spark in your likeness - low and high level customization
Bartosz Konieczny
 
Reliable Javascript
Reliable Javascript Reliable Javascript
Reliable Javascript
Glenn Stovall
 
How to Vim - for beginners
How to Vim - for beginnersHow to Vim - for beginners
How to Vim - for beginners
Marcin Rogacki
 
Good Practices On Test Automation
Good Practices On Test AutomationGood Practices On Test Automation
Good Practices On Test Automation
Gustavo Labbate Godoy
 
Intro to GraphQL on Android with Apollo DroidconNYC 2017
Intro to GraphQL on Android with Apollo DroidconNYC 2017Intro to GraphQL on Android with Apollo DroidconNYC 2017
Intro to GraphQL on Android with Apollo DroidconNYC 2017
Mike Nakhimovich
 
Fast Web Applications Development with Ruby on Rails on Oracle
Fast Web Applications Development with Ruby on Rails on OracleFast Web Applications Development with Ruby on Rails on Oracle
Fast Web Applications Development with Ruby on Rails on Oracle
Raimonds Simanovskis
 
Ml pipelines with Apache spark and Apache beam - Ottawa Reactive meetup Augus...
Ml pipelines with Apache spark and Apache beam - Ottawa Reactive meetup Augus...Ml pipelines with Apache spark and Apache beam - Ottawa Reactive meetup Augus...
Ml pipelines with Apache spark and Apache beam - Ottawa Reactive meetup Augus...
Holden Karau
 
On SQL Managment studioThis lab is all about database normalizatio.pdf
On SQL Managment studioThis lab is all about database normalizatio.pdfOn SQL Managment studioThis lab is all about database normalizatio.pdf
On SQL Managment studioThis lab is all about database normalizatio.pdf
infomalad
 
Shibuya.js Lightning Talks
Shibuya.js Lightning TalksShibuya.js Lightning Talks
Shibuya.js Lightning Talks
jeresig
 
Developer Test - Things to Know
Developer Test - Things to KnowDeveloper Test - Things to Know
Developer Test - Things to Know
vilniusjug
 

Similar to FetchQ - the origins (20)

jQuery & 10,000 Global Functions: Working with Legacy JavaScript
jQuery & 10,000 Global Functions: Working with Legacy JavaScriptjQuery & 10,000 Global Functions: Working with Legacy JavaScript
jQuery & 10,000 Global Functions: Working with Legacy JavaScript
 
In search of JavaScript code quality: unit testing
In search of JavaScript code quality: unit testingIn search of JavaScript code quality: unit testing
In search of JavaScript code quality: unit testing
 
Grokking TechTalk #24: Thiết kế hệ thống Background Job Queue bằng Ruby & Pos...
Grokking TechTalk #24: Thiết kế hệ thống Background Job Queue bằng Ruby & Pos...Grokking TechTalk #24: Thiết kế hệ thống Background Job Queue bằng Ruby & Pos...
Grokking TechTalk #24: Thiết kế hệ thống Background Job Queue bằng Ruby & Pos...
 
PostgreSQL Open SV 2018
PostgreSQL Open SV 2018PostgreSQL Open SV 2018
PostgreSQL Open SV 2018
 
Triggers and Stored Procedures
Triggers and Stored ProceduresTriggers and Stored Procedures
Triggers and Stored Procedures
 
Second Level Cache in JPA Explained
Second Level Cache in JPA ExplainedSecond Level Cache in JPA Explained
Second Level Cache in JPA Explained
 
Tddbdd workshop
Tddbdd workshopTddbdd workshop
Tddbdd workshop
 
Javascript unit testing, yes we can e big
Javascript unit testing, yes we can   e bigJavascript unit testing, yes we can   e big
Javascript unit testing, yes we can e big
 
NetBeans Plugin Development: JRebel Experience Report
NetBeans Plugin Development: JRebel Experience ReportNetBeans Plugin Development: JRebel Experience Report
NetBeans Plugin Development: JRebel Experience Report
 
When to NoSQL and when to know SQL
When to NoSQL and when to know SQLWhen to NoSQL and when to know SQL
When to NoSQL and when to know SQL
 
Apache Spark in your likeness - low and high level customization
Apache Spark in your likeness - low and high level customizationApache Spark in your likeness - low and high level customization
Apache Spark in your likeness - low and high level customization
 
Reliable Javascript
Reliable Javascript Reliable Javascript
Reliable Javascript
 
How to Vim - for beginners
How to Vim - for beginnersHow to Vim - for beginners
How to Vim - for beginners
 
Good Practices On Test Automation
Good Practices On Test AutomationGood Practices On Test Automation
Good Practices On Test Automation
 
Intro to GraphQL on Android with Apollo DroidconNYC 2017
Intro to GraphQL on Android with Apollo DroidconNYC 2017Intro to GraphQL on Android with Apollo DroidconNYC 2017
Intro to GraphQL on Android with Apollo DroidconNYC 2017
 
Fast Web Applications Development with Ruby on Rails on Oracle
Fast Web Applications Development with Ruby on Rails on OracleFast Web Applications Development with Ruby on Rails on Oracle
Fast Web Applications Development with Ruby on Rails on Oracle
 
Ml pipelines with Apache spark and Apache beam - Ottawa Reactive meetup Augus...
Ml pipelines with Apache spark and Apache beam - Ottawa Reactive meetup Augus...Ml pipelines with Apache spark and Apache beam - Ottawa Reactive meetup Augus...
Ml pipelines with Apache spark and Apache beam - Ottawa Reactive meetup Augus...
 
On SQL Managment studioThis lab is all about database normalizatio.pdf
On SQL Managment studioThis lab is all about database normalizatio.pdfOn SQL Managment studioThis lab is all about database normalizatio.pdf
On SQL Managment studioThis lab is all about database normalizatio.pdf
 
Shibuya.js Lightning Talks
Shibuya.js Lightning TalksShibuya.js Lightning Talks
Shibuya.js Lightning Talks
 
Developer Test - Things to Know
Developer Test - Things to KnowDeveloper Test - Things to Know
Developer Test - Things to Know
 

Recently uploaded

如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
gapen1
 
Preparing Non - Technical Founders for Engaging a Tech Agency
Preparing Non - Technical Founders for Engaging  a  Tech AgencyPreparing Non - Technical Founders for Engaging  a  Tech Agency
Preparing Non - Technical Founders for Engaging a Tech Agency
ISH Technologies
 
UI5con 2024 - Bring Your Own Design System
UI5con 2024 - Bring Your Own Design SystemUI5con 2024 - Bring Your Own Design System
UI5con 2024 - Bring Your Own Design System
Peter Muessig
 
Using Query Store in Azure PostgreSQL to Understand Query Performance
Using Query Store in Azure PostgreSQL to Understand Query PerformanceUsing Query Store in Azure PostgreSQL to Understand Query Performance
Using Query Store in Azure PostgreSQL to Understand Query Performance
Grant Fritchey
 
Enums On Steroids - let's look at sealed classes !
Enums On Steroids - let's look at sealed classes !Enums On Steroids - let's look at sealed classes !
Enums On Steroids - let's look at sealed classes !
Marcin Chrost
 
How to write a program in any programming language
How to write a program in any programming languageHow to write a program in any programming language
How to write a program in any programming language
Rakesh Kumar R
 
All you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVMAll you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVM
Alina Yurenko
 
Mobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona InfotechMobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona Infotech
Drona Infotech
 
How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?
ToXSL Technologies
 
GreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-JurisicGreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-Jurisic
Green Software Development
 
Fundamentals of Programming and Language Processors
Fundamentals of Programming and Language ProcessorsFundamentals of Programming and Language Processors
Fundamentals of Programming and Language Processors
Rakesh Kumar R
 
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
Bert Jan Schrijver
 
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling ExtensionsUI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
Peter Muessig
 
Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)
Julian Hyde
 
Energy consumption of Database Management - Florina Jonuzi
Energy consumption of Database Management - Florina JonuziEnergy consumption of Database Management - Florina Jonuzi
Energy consumption of Database Management - Florina Jonuzi
Green Software Development
 
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian CompaniesE-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
Quickdice ERP
 
What next after learning python programming basics
What next after learning python programming basicsWhat next after learning python programming basics
What next after learning python programming basics
Rakesh Kumar R
 
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdfTop Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
VALiNTRY360
 
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
kalichargn70th171
 
fiscal year variant fiscal year variant.
fiscal year variant fiscal year variant.fiscal year variant fiscal year variant.
fiscal year variant fiscal year variant.
AnkitaPandya11
 

Recently uploaded (20)

如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
 
Preparing Non - Technical Founders for Engaging a Tech Agency
Preparing Non - Technical Founders for Engaging  a  Tech AgencyPreparing Non - Technical Founders for Engaging  a  Tech Agency
Preparing Non - Technical Founders for Engaging a Tech Agency
 
UI5con 2024 - Bring Your Own Design System
UI5con 2024 - Bring Your Own Design SystemUI5con 2024 - Bring Your Own Design System
UI5con 2024 - Bring Your Own Design System
 
Using Query Store in Azure PostgreSQL to Understand Query Performance
Using Query Store in Azure PostgreSQL to Understand Query PerformanceUsing Query Store in Azure PostgreSQL to Understand Query Performance
Using Query Store in Azure PostgreSQL to Understand Query Performance
 
Enums On Steroids - let's look at sealed classes !
Enums On Steroids - let's look at sealed classes !Enums On Steroids - let's look at sealed classes !
Enums On Steroids - let's look at sealed classes !
 
How to write a program in any programming language
How to write a program in any programming languageHow to write a program in any programming language
How to write a program in any programming language
 
All you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVMAll you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVM
 
Mobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona InfotechMobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona Infotech
 
How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?
 
GreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-JurisicGreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-Jurisic
 
Fundamentals of Programming and Language Processors
Fundamentals of Programming and Language ProcessorsFundamentals of Programming and Language Processors
Fundamentals of Programming and Language Processors
 
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
 
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling ExtensionsUI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
 
Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)
 
Energy consumption of Database Management - Florina Jonuzi
Energy consumption of Database Management - Florina JonuziEnergy consumption of Database Management - Florina Jonuzi
Energy consumption of Database Management - Florina Jonuzi
 
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian CompaniesE-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
 
What next after learning python programming basics
What next after learning python programming basicsWhat next after learning python programming basics
What next after learning python programming basics
 
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdfTop Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
 
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...
 
fiscal year variant fiscal year variant.
fiscal year variant fiscal year variant.fiscal year variant fiscal year variant.
fiscal year variant fiscal year variant.
 

FetchQ - the origins