A collection of problems and solutions worked through Wanelo team as they were scaling the site with the rapid demand of users. By Konstantin Gredeskoul and Eric Saxby.
"Closing the Gap Between Developers & Customers," Wanelo >> Deena Varshavskay...500 Startups
STRATEGIC ALLIANCES: CLOSING THE GAP BETWEEN DEVELOPERS & CUSTOMERS, Deena Varshavskaya, Founder and CEO, Wanelo
Check out Deena's presentation that goes with this slide deck here: http://www.youtube.com/watch?v=E2Ey3e3dqhE
From Obvious to Ingenius: Incrementally Scaling Web Apps on PostgreSQLKonstantin Gredeskoul
In this exciting and informative talk, presented at PgConf Sillicon Valley 2015, Konstantin cut through the theory to deliver a clear set of practical solutions for scaling applications atop PostgreSQL, eventually supporting millions of active users, tens of thousands concurrently, and with the application stack that responds to requests with a 100ms average. He will share how his team solved one of the biggest challenges they faced: effectively storing and retrieving over 3B rows of "saves" (a Wanelo equivalent of Instagram's "like" or Pinterest's "pin"), all in PostgreSQL, with highly concurrent random access.
Over the last three years, the team at Wanelo optimized the hell out of their application and database stacks. Using PostgreSQL version 9 as their primary data store, Joyent Public Cloud as a hosting environment, the team re-architected their backend for rapid expansion several times over, as the unrelenting traffic kept climbing up. This ultimately resulted in a highly efficient, horizontally scalable, fault tolerant application infrastructure. Unimpressed? Now try getting there without the OPS or DBA teams, all while deploying seven times per day to production, with an application measuring 99.999% uptime over the last 6 months.
HackConf2016 - Ruby on Rails: Unexpected journeyDimitar Danailov
Презентацията ми за HackConf2016 за Ruby on Rails
Appplication: https://github.com/dimitardanailov/hobbit-hackconf-2015
Gist: https://gist.github.com/dimitardanailov/980b1dd45bee7fac4b44d9b91885a8c8
Rails security: above and beyond the defaultsMatias Korhonen
In a world with increasingly sophisticated adversaries employing both targeted and automated attacks, what can we do to keep our users and our web apps safe?
While Rails provides pretty decent security options straight out of the box, we can go further and make attacks more difficult to accomplish.
For example, why and how to implement a Content Security Policy. Should you use HTTP Public Key Pinning? How do you know if you've configured HTTPS correctly?
"Closing the Gap Between Developers & Customers," Wanelo >> Deena Varshavskay...500 Startups
STRATEGIC ALLIANCES: CLOSING THE GAP BETWEEN DEVELOPERS & CUSTOMERS, Deena Varshavskaya, Founder and CEO, Wanelo
Check out Deena's presentation that goes with this slide deck here: http://www.youtube.com/watch?v=E2Ey3e3dqhE
From Obvious to Ingenius: Incrementally Scaling Web Apps on PostgreSQLKonstantin Gredeskoul
In this exciting and informative talk, presented at PgConf Sillicon Valley 2015, Konstantin cut through the theory to deliver a clear set of practical solutions for scaling applications atop PostgreSQL, eventually supporting millions of active users, tens of thousands concurrently, and with the application stack that responds to requests with a 100ms average. He will share how his team solved one of the biggest challenges they faced: effectively storing and retrieving over 3B rows of "saves" (a Wanelo equivalent of Instagram's "like" or Pinterest's "pin"), all in PostgreSQL, with highly concurrent random access.
Over the last three years, the team at Wanelo optimized the hell out of their application and database stacks. Using PostgreSQL version 9 as their primary data store, Joyent Public Cloud as a hosting environment, the team re-architected their backend for rapid expansion several times over, as the unrelenting traffic kept climbing up. This ultimately resulted in a highly efficient, horizontally scalable, fault tolerant application infrastructure. Unimpressed? Now try getting there without the OPS or DBA teams, all while deploying seven times per day to production, with an application measuring 99.999% uptime over the last 6 months.
HackConf2016 - Ruby on Rails: Unexpected journeyDimitar Danailov
Презентацията ми за HackConf2016 за Ruby on Rails
Appplication: https://github.com/dimitardanailov/hobbit-hackconf-2015
Gist: https://gist.github.com/dimitardanailov/980b1dd45bee7fac4b44d9b91885a8c8
Rails security: above and beyond the defaultsMatias Korhonen
In a world with increasingly sophisticated adversaries employing both targeted and automated attacks, what can we do to keep our users and our web apps safe?
While Rails provides pretty decent security options straight out of the box, we can go further and make attacks more difficult to accomplish.
For example, why and how to implement a Content Security Policy. Should you use HTTP Public Key Pinning? How do you know if you've configured HTTPS correctly?
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
2024 State of Marketing Report – by HubspotMarius Sescu
https://www.hubspot.com/state-of-marketing
· Scaling relationships and proving ROI
· Social media is the place for search, sales, and service
· Authentic influencer partnerships fuel brand growth
· The strongest connections happen via call, click, chat, and camera.
· Time saved with AI leads to more creative work
· Seeking: A single source of truth
· TLDR; Get on social, try AI, and align your systems.
· More human marketing, powered by robots
ChatGPT is a revolutionary addition to the world since its introduction in 2022. A big shift in the sector of information gathering and processing happened because of this chatbot. What is the story of ChatGPT? How is the bot responding to prompts and generating contents? Swipe through these slides prepared by Expeed Software, a web development company regarding the development and technical intricacies of ChatGPT!
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
2024 State of Marketing Report – by HubspotMarius Sescu
https://www.hubspot.com/state-of-marketing
· Scaling relationships and proving ROI
· Social media is the place for search, sales, and service
· Authentic influencer partnerships fuel brand growth
· The strongest connections happen via call, click, chat, and camera.
· Time saved with AI leads to more creative work
· Seeking: A single source of truth
· TLDR; Get on social, try AI, and align your systems.
· More human marketing, powered by robots
ChatGPT is a revolutionary addition to the world since its introduction in 2022. A big shift in the sector of information gathering and processing happened because of this chatbot. What is the story of ChatGPT? How is the bot responding to prompts and generating contents? Swipe through these slides prepared by Expeed Software, a web development company regarding the development and technical intricacies of ChatGPT!
Product Design Trends in 2024 | Teenage EngineeringsPixeldarts
The realm of product design is a constantly changing environment where technology and style intersect. Every year introduces fresh challenges and exciting trends that mold the future of this captivating art form. In this piece, we delve into the significant trends set to influence the look and functionality of product design in the year 2024.
How Race, Age and Gender Shape Attitudes Towards Mental HealthThinkNow
Mental health has been in the news quite a bit lately. Dozens of U.S. states are currently suing Meta for contributing to the youth mental health crisis by inserting addictive features into their products, while the U.S. Surgeon General is touring the nation to bring awareness to the growing epidemic of loneliness and isolation. The country has endured periods of low national morale, such as in the 1970s when high inflation and the energy crisis worsened public sentiment following the Vietnam War. The current mood, however, feels different. Gallup recently reported that national mental health is at an all-time low, with few bright spots to lift spirits.
To better understand how Americans are feeling and their attitudes towards mental health in general, ThinkNow conducted a nationally representative quantitative survey of 1,500 respondents and found some interesting differences among ethnic, age and gender groups.
Technology
For example, 52% agree that technology and social media have a negative impact on mental health, but when broken out by race, 61% of Whites felt technology had a negative effect, and only 48% of Hispanics thought it did.
While technology has helped us keep in touch with friends and family in faraway places, it appears to have degraded our ability to connect in person. Staying connected online is a double-edged sword since the same news feed that brings us pictures of the grandkids and fluffy kittens also feeds us news about the wars in Israel and Ukraine, the dysfunction in Washington, the latest mass shooting and the climate crisis.
Hispanics may have a built-in defense against the isolation technology breeds, owing to their large, multigenerational households, strong social support systems, and tendency to use social media to stay connected with relatives abroad.
Age and Gender
When asked how individuals rate their mental health, men rate it higher than women by 11 percentage points, and Baby Boomers rank it highest at 83%, saying it’s good or excellent vs. 57% of Gen Z saying the same.
Gen Z spends the most amount of time on social media, so the notion that social media negatively affects mental health appears to be correlated. Unfortunately, Gen Z is also the generation that’s least comfortable discussing mental health concerns with healthcare professionals. Only 40% of them state they’re comfortable discussing their issues with a professional compared to 60% of Millennials and 65% of Boomers.
Race Affects Attitudes
As seen in previous research conducted by ThinkNow, Asian Americans lag other groups when it comes to awareness of mental health issues. Twenty-four percent of Asian Americans believe that having a mental health issue is a sign of weakness compared to the 16% average for all groups. Asians are also considerably less likely to be aware of mental health services in their communities (42% vs. 55%) and most likely to seek out information on social media (51% vs. 35%).
AI Trends in Creative Operations 2024 by Artwork Flow.pdfmarketingartwork
This article is all about what AI trends will emerge in the field of creative operations in 2024. All the marketers and brand builders should be aware of these trends for their further use and save themselves some time!
A report by thenetworkone and Kurio.
The contributing experts and agencies are (in an alphabetical order): Sylwia Rytel, Social Media Supervisor, 180heartbeats + JUNG v MATT (PL), Sharlene Jenner, Vice President - Director of Engagement Strategy, Abelson Taylor (USA), Alex Casanovas, Digital Director, Atrevia (ES), Dora Beilin, Senior Social Strategist, Barrett Hoffher (USA), Min Seo, Campaign Director, Brand New Agency (KR), Deshé M. Gully, Associate Strategist, Day One Agency (USA), Francesca Trevisan, Strategist, Different (IT), Trevor Crossman, CX and Digital Transformation Director; Olivia Hussey, Strategic Planner; Simi Srinarula, Social Media Manager, The Hallway (AUS), James Hebbert, Managing Director, Hylink (CN / UK), Mundy Álvarez, Planning Director; Pedro Rojas, Social Media Manager; Pancho González, CCO, Inbrax (CH), Oana Oprea, Head of Digital Planning, Jam Session Agency (RO), Amy Bottrill, Social Account Director, Launch (UK), Gaby Arriaga, Founder, Leonardo1452 (MX), Shantesh S Row, Creative Director, Liwa (UAE), Rajesh Mehta, Chief Strategy Officer; Dhruv Gaur, Digital Planning Lead; Leonie Mergulhao, Account Supervisor - Social Media & PR, Medulla (IN), Aurelija Plioplytė, Head of Digital & Social, Not Perfect (LI), Daiana Khaidargaliyeva, Account Manager, Osaka Labs (UK / USA), Stefanie Söhnchen, Vice President Digital, PIABO Communications (DE), Elisabeth Winiartati, Managing Consultant, Head of Global Integrated Communications; Lydia Aprina, Account Manager, Integrated Marketing and Communications; Nita Prabowo, Account Manager, Integrated Marketing and Communications; Okhi, Web Developer, PNTR Group (ID), Kei Obusan, Insights Director; Daffi Ranandi, Insights Manager, Radarr (SG), Gautam Reghunath, Co-founder & CEO, Talented (IN), Donagh Humphreys, Head of Social and Digital Innovation, THINKHOUSE (IRE), Sarah Yim, Strategy Director, Zulu Alpha Kilo (CA).
Trends In Paid Search: Navigating The Digital Landscape In 2024Search Engine Journal
The search marketing landscape is evolving rapidly with new technologies, and professionals, like you, rely on innovative paid search strategies to meet changing demands.
It’s important that you’re ready to implement new strategies in 2024.
Check this out and learn the top trends in paid search advertising that are expected to gain traction, so you can drive higher ROI more efficiently in 2024.
You’ll learn:
- The latest trends in AI and automation, and what this means for an evolving paid search ecosystem.
- New developments in privacy and data regulation.
- Emerging ad formats that are expected to make an impact next year.
Watch Sreekant Lanka from iQuanti and Irina Klein from OneMain Financial as they dive into the future of paid search and explore the trends, strategies, and technologies that will shape the search marketing landscape.
If you’re looking to assess your paid search strategy and design an industry-aligned plan for 2024, then this webinar is for you.
5 Public speaking tips from TED - Visualized summarySpeakerHub
From their humble beginnings in 1984, TED has grown into the world’s most powerful amplifier for speakers and thought-leaders to share their ideas. They have over 2,400 filmed talks (not including the 30,000+ TEDx videos) freely available online, and have hosted over 17,500 events around the world.
With over one billion views in a year, it’s no wonder that so many speakers are looking to TED for ideas on how to share their message more effectively.
The article “5 Public-Speaking Tips TED Gives Its Speakers”, by Carmine Gallo for Forbes, gives speakers five practical ways to connect with their audience, and effectively share their ideas on stage.
Whether you are gearing up to get on a TED stage yourself, or just want to master the skills that so many of their speakers possess, these tips and quotes from Chris Anderson, the TED Talks Curator, will encourage you to make the most impactful impression on your audience.
See the full article and more summaries like this on SpeakerHub here: https://speakerhub.com/blog/5-presentation-tips-ted-gives-its-speakers
See the original article on Forbes here:
http://www.forbes.com/forbes/welcome/?toURL=http://www.forbes.com/sites/carminegallo/2016/05/06/5-public-speaking-tips-ted-gives-its-speakers/&refURL=&referrer=#5c07a8221d9b
ChatGPT and the Future of Work - Clark Boyd Clark Boyd
Everyone is in agreement that ChatGPT (and other generative AI tools) will shape the future of work. Yet there is little consensus on exactly how, when, and to what extent this technology will change our world.
Businesses that extract maximum value from ChatGPT will use it as a collaborative tool for everything from brainstorming to technical maintenance.
For individuals, now is the time to pinpoint the skills the future professional will need to thrive in the AI age.
Check out this presentation to understand what ChatGPT is, how it will shape the future of work, and how you can prepare to take advantage.
A brief introduction to DataScience with explaining of the concepts, algorithms, machine learning, supervised and unsupervised learning, clustering, statistics, data preprocessing, real-world applications etc.
It's part of a Data Science Corner Campaign where I will be discussing the fundamentals of DataScience, AIML, Statistics etc.
Time Management & Productivity - Best PracticesVit Horky
Here's my presentation on by proven best practices how to manage your work time effectively and how to improve your productivity. It includes practical tips and how to use tools such as Slack, Google Apps, Hubspot, Google Calendar, Gmail and others.
The six step guide to practical project managementMindGenius
The six step guide to practical project management
If you think managing projects is too difficult, think again.
We’ve stripped back project management processes to the
basics – to make it quicker and easier, without sacrificing
the vital ingredients for success.
“If you’re looking for some real-world guidance, then The Six Step Guide to Practical Project Management will help.”
Dr Andrew Makar, Tactical Project Management
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Scaling Wanelo.com 100x in Six Months
1. Scaling
100x
in six months
by Eric Saxby & Konstantin Gredeskoul
April 2013
Proprietary and
Thursday, April 18, 13 Confidential 1
2. What is Wanelo?
■ Wanelo (“Wah-nee-lo” from Want, Need
Love) is a global platform for shopping.
Proprietary and
Thursday, April 18, 13 Confidential 2
3. What is Wanelo?
■ Wanelo (“Wah-nee-lo” from Want, Need
Love) is a global platform for shopping.
Proprietary and
Thursday, April 18, 13 Confidential 2
4. ■ It’s marketing-free shopping across
100s of thousands of unique stores
Proprietary and
Thursday, April 18, 13 Confidential 3
7. iOS + Android
Proprietary and
Thursday, April 18, 13 Confidential 5
8. iOS + Android
Proprietary and
Thursday, April 18, 13 Confidential 5
9. Early Decisions
Proprietary and
Thursday, April 18, 13 Confidential 6
10. Early Decisions
■ Optimize for iteration speed, not
performance
Proprietary and
Thursday, April 18, 13 Confidential 6
11. Early Decisions
■ Optimize for iteration speed, not
performance
■ Keep scalability in mind, track metrics,
and fix as needed
Proprietary and
Thursday, April 18, 13 Confidential 6
12. Early Decisions
■ Optimize for iteration speed, not
performance
■ Keep scalability in mind, track metrics,
and fix as needed
■ Introduce many levels of caching early
Proprietary and
Thursday, April 18, 13 Confidential 6
14. Technology Timeline
■ 2010 - 2011
Wanelo v1 stack is Java, JSP, MySQL, Hibernate
90K lines of code, 53+ DB tables, no tests
Proprietary and
Thursday, April 18, 13 Confidential 7
15. Technology Timeline
■ 2010 - 2011
Wanelo v1 stack is Java, JSP, MySQL, Hibernate
90K lines of code, 53+ DB tables, no tests
■ May 2012 - June 2012
Rewrite from scratch to RoR on PostgreSQL (v2)
Proprietary and
Thursday, April 18, 13 Confidential 7
16. Technology Timeline
■ 2010 - 2011
Wanelo v1 stack is Java, JSP, MySQL, Hibernate
90K lines of code, 53+ DB tables, no tests
■ May 2012 - June 2012
Rewrite from scratch to RoR on PostgreSQL (v2)
■ Ruby app is 10K LOC, full test coverage, 8
database tables, less features
Proprietary and
Thursday, April 18, 13 Confidential 7
17. The “Big” Rewrite
Proprietary and
Thursday, April 18, 13 Confidential 8
18. The “Big” Rewrite
More info here....
Proprietary and
Thursday, April 18, 13 Confidential 8
19. The “Big” Rewrite
More info here....
building.wanelo.com/
http://
Proprietary and
Thursday, April 18, 13 Confidential 8
20. The “Big” Rewrite
More info here....
building.wanelo.com/
http://
Proprietary and
Thursday, April 18, 13 Confidential 8
21. Growth Timeline
Proprietary and
Thursday, April 18, 13 Confidential 9
22. Growth Timeline
■ 06/2012 - RoR App Relaunches
Proprietary and
Thursday, April 18, 13 Confidential 9
23. Growth Timeline
■ 06/2012 - RoR App Relaunches
■ 2-3K requests per minute (RPM) peak
Proprietary and
Thursday, April 18, 13 Confidential 9
24. Growth Timeline
■ 06/2012 - RoR App Relaunches
■ 2-3K requests per minute (RPM) peak
■ 08/2012 - iOS App is launched
Proprietary and
Thursday, April 18, 13 Confidential 9
25. Growth Timeline
■ 06/2012 - RoR App Relaunches
■ 2-3K requests per minute (RPM) peak
■ 08/2012 - iOS App is launched
■ 10-40K RPM peak
Proprietary and
Thursday, April 18, 13 Confidential 9
30. Requests Per Minute (RPM)
Proprietary and
Thursday, April 18, 13 Confidential 10
31. Current Numbers...
■ 4M active monthly users
■ 5M products saved 700M times
■ 8M products saved per day
■ 200k stores
Proprietary and
Thursday, April 18, 13 Confidential 11
33. Wanelo Web Architecture
nginx
6 x 2GB
haproxy
unicorn x 14 sidekiq
20 x 8GB
4 x 8GB
haproxy pgbouncer twemproxy haproxy pgbouncer twemproxy
Solr PostgreSQL Redis MemCached
Proprietary and
Thursday, April 18, 13 Confidential 13
34. This talk is about:
Proprietary and
Thursday, April 18, 13 Confidential 14
35. This talk is about:
1. How much traffic can your database handle?
Proprietary and
Thursday, April 18, 13 Confidential 14
36. This talk is about:
1. How much traffic can your database handle?
2. Special report on counters
Proprietary and
Thursday, April 18, 13 Confidential 14
37. This talk is about:
1. How much traffic can your database handle?
2. Special report on counters
3. Scaling database reads
Proprietary and
Thursday, April 18, 13 Confidential 14
38. This talk is about:
1. How much traffic can your database handle?
2. Special report on counters
3. Scaling database reads
4. Scaling database writes
Proprietary and
Thursday, April 18, 13 Confidential 14
39. 1.
How much traffic can your
database handle?
Thursday, April 18, 13 15
41. PostgreSQL is Awesome!
■ Does a fantastic job of not corrupting
your data
Proprietary and
Thursday, April 18, 13 Confidential 16
42. PostgreSQL is Awesome!
■ Does a fantastic job of not corrupting
your data
■ Streaming replication in 9.2 is
extremely reliable
Proprietary and
Thursday, April 18, 13 Confidential 16
43. PostgreSQL is Awesome!
■ Does a fantastic job of not corrupting
your data
■ Streaming replication in 9.2 is
extremely reliable
■ Won’t write to a read-only replica
Proprietary and
Thursday, April 18, 13 Confidential 16
44. PostgreSQL is Awesome!
■ Does a fantastic job of not corrupting
your data
■ Streaming replication in 9.2 is
extremely reliable
■ Won’t write to a read-only replica
■ But... No master/master replication
Proprietary and
Thursday, April 18, 13 Confidential 16
45. PostgreSQL is Awesome!
■ Does a fantastic job of not corrupting
your data
■ Streaming replication in 9.2 is
extremely reliable
■ Won’t write to a read-only replica
■ But... No master/master replication
(good!)
Proprietary and
Thursday, April 18, 13 Confidential 16
46. Is the database healthy?
Proprietary and
Thursday, April 18, 13 Confidential 17
47. What’s healthy?
Proprietary and
Thursday, April 18, 13 Confidential 18
48. What’s healthy?
■ Able to respond quickly to queries from
application (< 4ms disk seek time)
Proprietary and
Thursday, April 18, 13 Confidential 18
49. What’s healthy?
■ Able to respond quickly to queries from
application (< 4ms disk seek time)
■ Has enough room to grow
Proprietary and
Thursday, April 18, 13 Confidential 18
50. What’s healthy?
■ Able to respond quickly to queries from
application (< 4ms disk seek time)
■ Has enough room to grow
■ How do we know when we’re
approaching a dangerous threshold?
Proprietary and
Thursday, April 18, 13 Confidential 18
51. Oops!
NewRelic Latency (yellow = database)
Proprietary and
Thursday, April 18, 13 Confidential 19
52. Oops!
NewRelic Latency (yellow = database)
Proprietary and
Thursday, April 18, 13 Confidential 19
53. pg_stat_statements
■ Maybe your app is to blame for
performance...
select
query,
calls,
total_time
from
pg_stat_statements
order
by
total_time
desc
limit
12;
Proprietary and
Thursday, April 18, 13 Confidential 20
54. pg_stat_statements
■ Maybe your app is to blame for
performance...
select
query,
calls,
total_time
from
pg_stat_statements
order
by
total_time
desc
limit
12;
Similar to Percona Toolkit, but runs all the
time collecting stats.
Proprietary and
Thursday, April 18, 13 Confidential 20
55. pg_stat_statements
Proprietary and
Thursday, April 18, 13 Confidential 21
56. pg_stat_user_indexes
■ Using indexes as much as you think
you are?
■ Using indexes at all?
Proprietary and
Thursday, April 18, 13 Confidential 22
57. pg_stat_user_indexes
■ Using indexes as much as you think
you are?
■ Using indexes at all?
Proprietary and
Thursday, April 18, 13 Confidential 22
58. pg_stat_user_tables
■ Full table scans? (seq_scan)
Proprietary and
Thursday, April 18, 13 Confidential 23
59. pg_stat_user_tables
■ Full table scans? (seq_scan)
Proprietary and
Thursday, April 18, 13 Confidential 23
60. Throw that in a graph
Reads/second for one large table, daily
Proprietary and
Thursday, April 18, 13 Confidential 24
61. Non-linear changes
Suspicious spike!
Proprietary and
Thursday, April 18, 13 Confidential 25
62. Correlate different data
Deployments! Aha!
Proprietary and
Thursday, April 18, 13 Confidential 26
63. Utilization vs Saturation
# of Active PostgreSQL connections
Proprietary and
Thursday, April 18, 13 Confidential 27
64. Utilization vs Saturation
Red line: % of max connections established
Purple: % of connections in query
Proprietary and
Thursday, April 18, 13 Confidential 28
65. Disk reads/writes
green: reads, red: writes
Proprietary and
Thursday, April 18, 13 Confidential 29
66. Disk reads/writes
green: reads, red: writes
Usage increases, but are the disks saturated?
Proprietary and
Thursday, April 18, 13 Confidential 29
69. Utilization vs Saturation
[
How much are you waiting on disk?
Proprietary and
Thursday, April 18, 13 Confidential 31
70. File system cache (ARC)
Proprietary and
Thursday, April 18, 13 Confidential 32
71. File system cache (ARC)
Proprietary and
Thursday, April 18, 13 Confidential 32
72. File system cache (ARC)
Proprietary and
Thursday, April 18, 13 Confidential 32
73. Watch the right things
Hit ratio of the file system cache (ARC)
Proprietary and
Thursday, April 18, 13 Confidential 33
74. Watch the right things
Hit ratio of the file system cache (ARC)
Proprietary and
Thursday, April 18, 13 Confidential 33
75. Room to grow...
Size (including indexes) of a key table
Proprietary and
Thursday, April 18, 13 Confidential 34
76. Working set in RAM?
Adding index increases the size
Proprietary and
Thursday, April 18, 13 Confidential 35
77. Working set in RAM?
Adding index increases the size
Proprietary and
Thursday, April 18, 13 Confidential 35
78. Collect all the data you can
Once we knew where to look, graphs added
later could explain behavior we could
only guess at earlier
Proprietary and
Thursday, April 18, 13 Confidential 36
79. Collect all the data you can
Once we knew where to look, graphs added
later could explain behavior we could
only guess at earlier
Proprietary and
Thursday, April 18, 13 Confidential 36
80. 2.
Special report on
Counters and Pagination
Thursday, April 18, 13 37
81. Problem #1: DB Latency Up...
Proprietary and
Thursday, April 18, 13 Confidential 38
82. Problem #1: DB Latency Up...
■ iostat shows 100% disk busy
Proprietary and
Thursday, April 18, 13 Confidential 38
87. Problem #1: Diagnostics
■ Database is running very very hot.
Initial investigation shows large number of counts.
Proprietary and
Thursday, April 18, 13 Confidential 39
88. Problem #1: Diagnostics
■ Database is running very very hot.
Initial investigation shows large number of counts.
■ Turns out anytime you page with Kaminari, it
always does a count(*)!
Proprietary and
Thursday, April 18, 13 Confidential 39
89. Problem #1: Diagnostics
■ Database is running very very hot.
Initial investigation shows large number of counts.
■ Turns out anytime you page with Kaminari, it
always does a count(*)!
SELECT
"stores".*
FROM
"stores"
WHERE
(state
=
'approved')
LIMIT
20
OFFSET
0
SELECT
COUNT(*)
FROM
"stores"
WHERE
(state
=
'approved')
Proprietary and
Thursday, April 18, 13 Confidential 39
91. Problem #1: Pagination
■ Doing count(*) is pretty expensive, as DB
must scan many rows (either the actual table
or an index)
Proprietary and
Thursday, April 18, 13 Confidential 40
93. Problem #1: Pagination
■ We are paginating everything! Even infinite
scroll is a paged view behind the scenes.
Proprietary and
Thursday, April 18, 13 Confidential 41
94. Problem #1: Pagination
■ We are paginating everything! Even infinite
scroll is a paged view behind the scenes.
■ But we really DON’T want to run count(*) for
every paged view.
Proprietary and
Thursday, April 18, 13 Confidential 41
95. Problem #1: Pagination
■ We are showing most popular stores
■ Maybe it’s OK to hard-code the total number to,
say, 1000?
Proprietary and
Thursday, April 18, 13 Confidential 42
96. Problem #1: Pagination
■ We are showing most popular stores
■ Maybe it’s OK to hard-code the total number to,
say, 1000?
■ How do we tell Kaminari NOT to issue a
count query in this case?
Proprietary and
Thursday, April 18, 13 Confidential 42
98. Solution #1: Monkey Patch!!
Proprietary and
Thursday, April 18, 13 Confidential 44
99. Solution #1: Monkey Patch!!
Proprietary and
Thursday, April 18, 13 Confidential 44
100. Solution #1: Pass in the
counter
Proprietary and
Thursday, April 18, 13 Confidential 45
101. Solution #1: Pass in the
counter
SELECT
"stores".*
FROM
"stores"
WHERE
(state
=
'approved')
LIMIT
20
OFFSET
0
Proprietary and
Thursday, April 18, 13 Confidential 45
102. Problem #2: Count Draculas
■ AKA: We still are doing too many counts!
Proprietary and
Thursday, April 18, 13 Confidential 46
103. Problem #2: Count Draculas
■ AKA: We still are doing too many counts!
Proprietary and
Thursday, April 18, 13 Confidential 46
104. Problem #2: Count Draculas
■ AKA: We still are doing too many counts!
■ Rails makes it so easy to do it the lazy way.
Proprietary and
Thursday, April 18, 13 Confidential 46
105. Problem #2: Too Many Counts!
■ But it just doesn’t scale well
Proprietary and
Thursday, April 18, 13 Confidential 47
106. Problem #2: Too Many Counts!
■ But it just doesn’t scale well
■ Fortunately, Rails has just a feature for this...
Proprietary and
Thursday, April 18, 13 Confidential 47
107. Problem #2: Too Many Counts!
■ But it just doesn’t scale well
■ Fortunately, Rails has just a feature for this...
Proprietary and
Thursday, April 18, 13 Confidential 47
108. Counter Caches
■ Unfortunately, it has one massive issue:
Proprietary and
Thursday, April 18, 13 Confidential 48
109. Counter Caches
■ Unfortunately, it has one massive issue:
■ It causes database deadlocks at high volume
Proprietary and
Thursday, April 18, 13 Confidential 48
110. Counter Caches
■ Unfortunately, it has one massive issue:
■ It causes database deadlocks at high volume
■ Because many ruby processes are creating child
records concurrently
Proprietary and
Thursday, April 18, 13 Confidential 48
111. Counter Caches
■ Unfortunately, it has one massive issue:
■ It causes database deadlocks at high volume
■ Because many ruby processes are creating child
records concurrently
■ Each is executing a callback, trying to update
counter_cache column on the parent, requiring
row-level lock
Proprietary and
Thursday, April 18, 13 Confidential 48
112. Counter Caches
■ Unfortunately, it has one massive issue:
■ It causes database deadlocks at high volume
■ Because many ruby processes are creating child
records concurrently
■ Each is executing a callback, trying to update
counter_cache column on the parent, requiring
row-level lock
■ Deadlocks ensue
Proprietary and
Thursday, April 18, 13 Confidential 48
113. Possible Solution:
Use Background Jobs
Proprietary and
Thursday, April 18, 13 Confidential 49
114. Possible Solution:
Use Background Jobs
■ It works like this:
Proprietary and
Thursday, April 18, 13 Confidential 49
115. Possible Solution:
Use Background Jobs
■ It works like this:
■ As the record is created, we enqueue a request
to recalculate counter_cache on the parent
Proprietary and
Thursday, April 18, 13 Confidential 49
116. Possible Solution:
Use Background Jobs
■ It works like this:
■ As the record is created, we enqueue a request
to recalculate counter_cache on the parent
■ The job performs a complete recalculation of
the counter cache and is idempotent
Proprietary and
Thursday, April 18, 13 Confidential 49
118. Solution #2: Explained
■ Sidekiq with UniqueJob extension
Proprietary and
Thursday, April 18, 13 Confidential 50
119. Solution #2: Explained
■ Sidekiq with UniqueJob extension
■ Short wait for “buffering”
Proprietary and
Thursday, April 18, 13 Confidential 50
120. Solution #2: Explained
■ Sidekiq with UniqueJob extension
■ Short wait for “buffering”
■ Serialize updates via small number of workers
Proprietary and
Thursday, April 18, 13 Confidential 50
121. Solution #2: Explained
■ Sidekiq with UniqueJob extension
■ Short wait for “buffering”
■ Serialize updates via small number of workers
■ Can temporarily stop workers (in an
emergency) to alleviate DB load
Proprietary and
Thursday, April 18, 13 Confidential 50
122. Solution #2: Code
Proprietary and
Thursday, April 18, 13 Confidential 51
123. Things are better. BUT...
Proprietary and
Thursday, April 18, 13 Confidential 52
124. Things are better. BUT...
Still too many fucking counts!
Proprietary and
Thursday, April 18, 13 Confidential 52
125. Things are better. BUT...
Still too many fucking counts!
■ Even doing count(*) from workers is too
much on the databases
Proprietary and
Thursday, April 18, 13 Confidential 52
126. Things are better. BUT...
Still too many fucking counts!
■ Even doing count(*) from workers is too
much on the databases
■ We need to stop doing count(*) in DB. But
keep counter_caches. How?
Proprietary and
Thursday, April 18, 13 Confidential 52
127. Things are better. BUT...
Still too many fucking counts!
■ Even doing count(*) from workers is too
much on the databases
■ We need to stop doing count(*) in DB. But
keep counter_caches. How?
■ We could use Redis for this.
Proprietary and
Thursday, April 18, 13 Confidential 52
128. save product product_id
Solution #3:
Counts Deltas unicorn
counter_cache column
1. INCR product_id 2. ProductCountWorker.enqueue product_id
Redis Redis
PostgreSQL
Counters Sidekiq
4. GET
3. Dequeue
5. RESET
5. SQL Update INCR by N
sidekiq
Proprietary and
Thursday, April 18, 13 Confidential 53
129. save product product_id
Solution #3:
Counts Deltas unicorn
■ Web request increments
counter value in Redis
counter_cache column
■ Enqueues request to
update counter_cache 1. INCR product_id 2. ProductCountWorker.enqueue product_id
■ Background Job picks up
a few minutes later, reads Redis Redis
PostgreSQL
Redis delta value, and Counters Sidekiq
removes it.
■ Updates counter_cache 4. GET
5. RESET
3. Dequeue
column by incrementing it
by delta. 5. SQL Update INCR by N
sidekiq
Proprietary and
Thursday, April 18, 13 Confidential 53
130. Define counter_cache_on...
■ Internal GEM, will open source soon!
Proprietary and
Thursday, April 18, 13 Confidential 54
131. Can now use counter caches
in pagination!
Proprietary and
Thursday, April 18, 13 Confidential 55
133. Multiple optimization cycles
■ Caching
action caching, fragment, CDN
■ Personalization via AJAX
Cache the entire page, then add
personalized details
■ 25ms/req memcached time is cheaper than
12ms/req of database time
Proprietary and
Thursday, April 18, 13 Confidential 57
134. Cache optimization
40% hit ratio! Woo!
Wait... is that even good?
Proprietary and
Thursday, April 18, 13 Confidential 58
135. Cache optimization
Increasing your hit ratio means less
queries against your database
Proprietary and
Thursday, April 18, 13 Confidential 59
136. Cache optimization
Caveat: even low hit ratio caches
can save your ass. You’re removing
load from the DB, remember?
Proprietary and
Thursday, April 18, 13 Confidential 60
137. Cache saturation
Blue: cache writes How long before your caches
Red: automatic evictions start evicting data?
Proprietary and
Thursday, April 18, 13 Confidential 61
138. Cache saturation
Blue: cache writes How long before your caches
Red: automatic evictions start evicting data?
Proprietary and
Thursday, April 18, 13 Confidential 61
139. Cache saturation
Blue: cache writes How long before your caches
Red: automatic evictions start evicting data?
Proprietary and
Thursday, April 18, 13 Confidential 61
143. Nice!
■ Rails Action Caching
Runs before_filters, so A/B experiments can still run
■ Extremely fast pages
4ms application time for some of our
computationally heaviest pages
■ Could be served via CDN in the future
Proprietary and
Thursday, April 18, 13 Confidential 63
144. Sad trombone...
■ Are you actually logged in?
Pages don’t know until Ajax successfully runs
■ Selenium AND Jasmine tests!
Proprietary and
Thursday, April 18, 13 Confidential 64
145. Read/write splitting
■ Sometime in December 2012...
Proprietary and
Thursday, April 18, 13 Confidential 65
146. Read/write splitting
■ Sometime in December 2012...
■ Database reaching 100% saturation
Proprietary and
Thursday, April 18, 13 Confidential 65
147. Read/write splitting
■ Sometime in December 2012...
■ Database reaching 100% saturation
■ Latency starting to increase non-linearly
Proprietary and
Thursday, April 18, 13 Confidential 65
148. Read/write splitting
■ Sometime in December 2012...
■ Database reaching 100% saturation
■ Latency starting to increase non-linearly
■ We need to distribute database load
Proprietary and
Thursday, April 18, 13 Confidential 65
149. Read/write splitting
■ Sometime in December 2012...
■ Database reaching 100% saturation
■ Latency starting to increase non-linearly
■ We need to distribute database load
■ We need to use read replicas!
Proprietary and
Thursday, April 18, 13 Confidential 65
150. DB adapters for read/write
■ Looked at several, including DbCharmer
Proprietary and
Thursday, April 18, 13 Confidential 66
151. DB adapters for read/write
■ Looked at several, including DbCharmer
■ Features / Configurability / Stability
■ Thread safety? This may be Ruby, but some
people do actually use threads.
■ If I tell you it’s a read-only replica, DON’T
ISSUE WRITES
■ Failover on errors?
Proprietary and
Thursday, April 18, 13 Confidential 66
152. Chose Makara, by TaskRabbit
■ Used in production
■ We extended it to work with PostgreSQL
■ Works with Sidekiqs (thread-safe!)
■ Failover code is very simple. Simple is
sometimes better.
https://github.com/taskrabbit/makara
Proprietary and
Thursday, April 18, 13 Confidential 67
153. We rolled out Makara and...
■ 1 master, 3 read-only async replicas
Proprietary and
Thursday, April 18, 13 Confidential 68
154. We rolled out Makara and...
■ 1 master, 3 read-only async replicas
Wait, what?
Proprietary and
Thursday, April 18, 13 Confidential 68
155. A note about graphs
■ NewRelic is great!
■ Not easy to predict when your
systems are about to fall over
■ Use something else to visualize
Database and disk saturation
Proprietary and
Thursday, April 18, 13 Confidential 69
156. 3 days later, in production
■ 3 read replicas distributing load from master
■ app servers and sidekiqs create lots of
connections to DB backends
Proprietary and
Thursday, April 18, 13 Confidential 70
157. 3 days later, in production
■ 3 read replicas distributing load from master
■ app servers and sidekiqs create lots of
connections to DB backends
■ Mysterious spikes in errors at high traffic
Proprietary and
Thursday, April 18, 13 Confidential 70
158. 3 days later, in production
■ 3 read replicas distributing load from master
■ app servers and sidekiqs create lots of
connections to DB backends
■ Mysterious spikes in errors at high traffic
Proprietary and
Thursday, April 18, 13 Confidential 70
159. Replication! Doh!
Replication lag (yellow)
correlates with application errors (red)
Proprietary and
Thursday, April 18, 13 Confidential 71
160. Replication lag! Doh!
■ Track latency sending xlog to slaves
select client_addr,
pg_xlog_location_diff(sent_location, write_location)
from pg_stat_replication;
■ Track latency applying xlogs on slaves
select pg_xlog_location_diff(
pg_last_xlog_receive_location(),
pg_last_xlog_replay_location()),
extract(epoch from now()) -
extract(epoch from pg_last_xact_replay_timestamp());
Proprietary and
Thursday, April 18, 13 Confidential 72
162. Eventual Consistency
■ Some code paths should always go to
master for reads (ie, after signup)
Proprietary and
Thursday, April 18, 13 Confidential 73
163. Eventual Consistency
■ Some code paths should always go to
master for reads (ie, after signup)
■ Application should be resilient to
getting RecordNotFound to tolerate
replication delays
Proprietary and
Thursday, April 18, 13 Confidential 73
164. Eventual Consistency
■ Some code paths should always go to
master for reads (ie, after signup)
■ Application should be resilient to
getting RecordNotFound to tolerate
replication delays
■ Not enough to scale reads.
Writes become the bottleneck.
Proprietary and
Thursday, April 18, 13 Confidential 73
165. Write load delays replication
Replicas are busy trying to apply XLOGs
and serve heavy read traffic
Proprietary and
Thursday, April 18, 13 Confidential 74
166. 4.
Scaling database writes
Thursday, April 18, 13 75
167. First, No-Brainers:
■ Move stuff out of the DB. Easiest first.
Proprietary and
Thursday, April 18, 13 Confidential 76
168. First, No-Brainers:
■ Move stuff out of the DB. Easiest first.
■ Tracking user activity is very easy to do
with a database table. But slow.
Proprietary and
Thursday, April 18, 13 Confidential 76
169. First, No-Brainers:
■ Move stuff out of the DB. Easiest first.
■ Tracking user activity is very easy to do
with a database table. But slow.
■ 2000 inserts/sec while also handling site
critical data? Not a good idea.
Proprietary and
Thursday, April 18, 13 Confidential 76
170. First, No-Brainers:
■ Move stuff out of the DB. Easiest first.
■ Tracking user activity is very easy to do
with a database table. But slow.
■ 2000 inserts/sec while also handling site
critical data? Not a good idea.
■ Solution:
UDP packets to rsyslog, ASCII delimited files, log-
rotate, analyze them later
Proprietary and
Thursday, April 18, 13 Confidential 76
172. Next: Async Commits
■ PostgreSQL supports delayed
(batched) commits
Proprietary and
Thursday, April 18, 13 Confidential 77
173. Next: Async Commits
■ PostgreSQL supports delayed
(batched) commits
■ Delays fsync for some # of
microseconds
Proprietary and
Thursday, April 18, 13 Confidential 77
174. Next: Async Commits
■ PostgreSQL supports delayed
(batched) commits
■ Delays fsync for some # of
microseconds
■ At high volume helps disk IO
Proprietary and
Thursday, April 18, 13 Confidential 77
186. Next: Vertical Sharding
■ Move out largest table into its own
master database (150 inserts/sec)
Proprietary and
Thursday, April 18, 13 Confidential 81
187. Next: Vertical Sharding
■ Move out largest table into its own
master database (150 inserts/sec)
■ Remove any SQL joins, do them in
application, drop foreign keys
Proprietary and
Thursday, April 18, 13 Confidential 81
188. Next: Vertical Sharding
■ Move out largest table into its own
master database (150 inserts/sec)
■ Remove any SQL joins, do them in
application, drop foreign keys
■ Switch model to establish_connection
to another DB. Fix many broken tests.
Proprietary and
Thursday, April 18, 13 Confidential 81
189. Vertical Sharding
unicorns
haproxy pgbouncer twemproxy
PostgreSQL
saves master
PostgreSQL PostgreSQL PostgreSQL
main replica main replica main master
streaming replication
Proprietary and
Thursday, April 18, 13 Confidential 82
192. Future: Services Approach
unicorns
haproxy pgbouncer twemproxy
http / json
PostgreSQL PostgreSQL PostgreSQL sinatra services app
main replica main replica main master
streaming replication
Shard1 Shard2 Shard3
Proprietary and
Thursday, April 18, 13 Confidential 84
193. In Conclusion. Tasty gems :)
https://github.com/wanelo/pause
https://github.com/wanelo/spanx
https://github.com/wanelo/redis_with_failover
https://github.com/kigster/ventable
Proprietary and
Thursday, April 18, 13 Confidential 85
194. In Conclusion. Tasty gems :)
https://github.com/wanelo/pause
■ distributed rate limiting using redis
https://github.com/wanelo/spanx
https://github.com/wanelo/redis_with_failover
https://github.com/kigster/ventable
Proprietary and
Thursday, April 18, 13 Confidential 85
195. In Conclusion. Tasty gems :)
https://github.com/wanelo/pause
■ distributed rate limiting using redis
https://github.com/wanelo/spanx
■ rate-limit-based IP blocker for nginx
https://github.com/wanelo/redis_with_failover
https://github.com/kigster/ventable
Proprietary and
Thursday, April 18, 13 Confidential 85
196. In Conclusion. Tasty gems :)
https://github.com/wanelo/pause
■ distributed rate limiting using redis
https://github.com/wanelo/spanx
■ rate-limit-based IP blocker for nginx
https://github.com/wanelo/redis_with_failover
■ attempt another redis server if available
https://github.com/kigster/ventable
Proprietary and
Thursday, April 18, 13 Confidential 85
197. In Conclusion. Tasty gems :)
https://github.com/wanelo/pause
■ distributed rate limiting using redis
https://github.com/wanelo/spanx
■ rate-limit-based IP blocker for nginx
https://github.com/wanelo/redis_with_failover
■ attempt another redis server if available
https://github.com/kigster/ventable
■ observable pattern with a twist
Proprietary and
Thursday, April 18, 13 Confidential 85
Our mission is to democratize and transform the world's commerce by reorganizing shopping around people.
Some of the stores have close to half a million followers. Some are big and known, and some aren’t at all, outside of Wanelo.
Near real time updates to your feed, as people post products to stores you follow, or collections. Following a hashtag is very powerful.
Rails backend API, simple JSON in/out, using RABL for rendering JSON back (slow!). JSON.generate() is so much faster than to_json
included in /contrib in the Postgres source. Very easy to install. If a package does not come with pg_stat_statements, this is a reason to compile it yourself.
This is why we like Postgres: visibility tools
Sometimes you throw everything in a single graph, not knowing if it’s useful Sometimes that graph saves your ass when you happen to see it out of the corner of your eye
Extremely useful to correlate different data points visually
Why are you even waiting on disks? Postgres relies heavily on the file cache
Adaptive Replacement Cache This is why we like SmartOS/Illumos/Solaris: visibility tools
Great thing about ARC: even when your query misses in-RAM db cache, you hit an in-RAM file cache
Slowed down the site to the point where errors started happening
purple is hit ratio of cache servers
purple is hit ratio of cache servers
purple is hit ratio of cache servers
blue: writes red: automatic eviction
Hard to do this after the fact
This is why you want to already be on Postgres. You can take risks knowing that PG will throw errors, not corrupt data.
When you pull aside the curtain of a Ruby DB adapter, you can get a sense of... betrayal. Why is it written like this? Why method_missing? Why????? ActiveRecord is a finely crafted pile of code defined after the fact. Unfortunately, the DB adapters that don’t use crazy metaprogramming do things even worse to avoid it. Error handling is set of regexs. Easy to extend. Requests after a write read from master.
Putting everything into a class namespace per thread is not thread safety. Threaded code often spawns new threads.
New Relic application graph for month of December
Graphite / Circonus
Postgres 9.2 specific. 9.1 you basically have to connect to both master and replica, do binary math