Timyang新浪微博设计谈
•
7 likes•881 views
This document discusses caching strategies for social media feeds. It describes using different caches like hot, vector, archive caches for inbox, outbox, social graph and content. It discusses push and pull models for updating caches and feeding timeline. It also talks about caching strategies like caching hot keys, using local caches, memcache and managing mutexes for concurrent access and updates.
Report
Share
Report
Share
Download to read offline
Recommended
微博cache设计谈
The document discusses caching strategies for social media feeds. It describes using different caches for hot, recent content (inbox cache), older content for followers (outbox vector), and archived historic content (archive cache). It also discusses caching social graphs like followers and following lists. The caches aim to optimize for fast retrieval of recent home timelines and individual user feeds. Mutexes are used to synchronize cache updates between servers.
新浪微博开放平台Redis实战
This document discusses Redis, a key-value store that is commonly used at Weibo for caching and storing relationship data. Redis has fast read and write performance but has limitations for large datasets due to its fully in-memory design. The document describes how Weibo uses Redis in conjunction with MySQL and Memcached to store relationship data for over 100 million users in a performant and scalable way. Challenges around high memory usage, persistence, and availability are also discussed.
Whats new in mongoDB 2.4 at Copenhagen user group 2013-06-19
Standard 10gen "What's new in MongoDB" 2.4 delivered at an enthusiastic MongoDB user group in Copenhagen.
Shell Tips & Tricks
This document provides tips and tricks for using the MongoDB shell, including:
- The shell allows running Javascript and accessing MongoDB functions and objects
- It can be used for debugging queries, testing, and administration
- However, Javascript numbers and dates can be confusing to work with in the shell
- Functions like load(), runCommand(), getProfilingLevel() and forEach() provide useful shell capabilities
- The shell is generally slower than queries run directly on the server
MongoDB: a gentle, friendly overview
This document provides a gentle, friendly overview of MongoDB. It discusses that MongoDB is a document-oriented, non-relational database that uses JSON-like documents with dynamic schemas. It also covers MongoDB's query language, indexing, data storage and querying, drivers like PyMongo, and how MongoDB provides horizontal scalability through sharding and replication. The document concludes by noting some common use cases for MongoDB and areas it is less suited for.
Getting Started with MongoDB
This is a presentation on how to get started with the MongoDB database available at http://www.mongodb.org/.
Advanced Redis data structures
In these slide following projects are presented:
* redis_wrap: A Pythonic wrapper that makes it nicer to work with builtin Redis datatypes
* redis_graph: A sample graph database
* redis_simple_queue: A simple queue implemented on Redis list structure
* bitmapist: a powerful analytics library using Redis bitmaps, great for retention and cohort tracking
* fixedlist: a fixed list structure that can optimize timelines (and other things)
* how to use Lua scripting for more advanced data structures and better performance
These are Python projects, but some of them (like bitmapist) have been ported to other languages.
This talk was given to PyCon Belarus on 31 Jan. 2015
MongoDB Shell Tips & Tricks
The document discusses tips and tricks for using the MongoDB shell. It describes what the shell is, advantages like debugging queries and administration, and disadvantages like numbers and dates. It provides examples of shell commands and functions for tasks like loading scripts, running commands, profiling, and administration. It also discusses configuration options and ways to improve the shell experience like keyboard shortcuts and the .mongorc.js file.
Recommended
微博cache设计谈
The document discusses caching strategies for social media feeds. It describes using different caches for hot, recent content (inbox cache), older content for followers (outbox vector), and archived historic content (archive cache). It also discusses caching social graphs like followers and following lists. The caches aim to optimize for fast retrieval of recent home timelines and individual user feeds. Mutexes are used to synchronize cache updates between servers.
新浪微博开放平台Redis实战
This document discusses Redis, a key-value store that is commonly used at Weibo for caching and storing relationship data. Redis has fast read and write performance but has limitations for large datasets due to its fully in-memory design. The document describes how Weibo uses Redis in conjunction with MySQL and Memcached to store relationship data for over 100 million users in a performant and scalable way. Challenges around high memory usage, persistence, and availability are also discussed.
Whats new in mongoDB 2.4 at Copenhagen user group 2013-06-19
Standard 10gen "What's new in MongoDB" 2.4 delivered at an enthusiastic MongoDB user group in Copenhagen.
Shell Tips & Tricks
This document provides tips and tricks for using the MongoDB shell, including:
- The shell allows running Javascript and accessing MongoDB functions and objects
- It can be used for debugging queries, testing, and administration
- However, Javascript numbers and dates can be confusing to work with in the shell
- Functions like load(), runCommand(), getProfilingLevel() and forEach() provide useful shell capabilities
- The shell is generally slower than queries run directly on the server
MongoDB: a gentle, friendly overview
This document provides a gentle, friendly overview of MongoDB. It discusses that MongoDB is a document-oriented, non-relational database that uses JSON-like documents with dynamic schemas. It also covers MongoDB's query language, indexing, data storage and querying, drivers like PyMongo, and how MongoDB provides horizontal scalability through sharding and replication. The document concludes by noting some common use cases for MongoDB and areas it is less suited for.
Getting Started with MongoDB
This is a presentation on how to get started with the MongoDB database available at http://www.mongodb.org/.
Advanced Redis data structures
In these slide following projects are presented:
* redis_wrap: A Pythonic wrapper that makes it nicer to work with builtin Redis datatypes
* redis_graph: A sample graph database
* redis_simple_queue: A simple queue implemented on Redis list structure
* bitmapist: a powerful analytics library using Redis bitmaps, great for retention and cohort tracking
* fixedlist: a fixed list structure that can optimize timelines (and other things)
* how to use Lua scripting for more advanced data structures and better performance
These are Python projects, but some of them (like bitmapist) have been ported to other languages.
This talk was given to PyCon Belarus on 31 Jan. 2015
MongoDB Shell Tips & Tricks
The document discusses tips and tricks for using the MongoDB shell. It describes what the shell is, advantages like debugging queries and administration, and disadvantages like numbers and dates. It provides examples of shell commands and functions for tasks like loading scripts, running commands, profiling, and administration. It also discusses configuration options and ways to improve the shell experience like keyboard shortcuts and the .mongorc.js file.
How to migrate your existing MongoDB and Cassandra Apps to Azure Cosmos DB
Bring your Mongo DB and Cassandra applications to Azure Cosmos DB and benefit turnkey global distribution, guaranteed low latency for cloud scale. Learn how easy it is to migrate your existing NoSQL applications to Azure Cosmos DB by using the MongoDB API and Cassandra API. In this sessions you will find out about the tools used in the process, see the code that leverages Azure Cosmos DB, and learn about techniques for working around the differences between the two products.
Scalable XQuery Processing with Zorba on top of MongoDB
Since a couple of years, the NoSQL movement has developed a variety of open-source document stores. Most of them focus on high availability, horizontal scalability, and are designed to run on commodity hardware. These products have gained great traction in the industry to store large amounts of flexible data (mostly JSON). In the meantime, XQuery has evolved to a standardized, full-fledged programming language for XML with native support for complex queries, indexes, updates, full-text search, and scripting. Moreover, JSON has recently been added as a first-level datatype into the language. As of today, it is without doubt the most robust and productive technology to process flexible data.
The aim of this talk is to showcase the benefits that can be achieved by integrating the Zorba XQuery Processor with MongoDB. We will introduce the 28msec platform that seamlessly stores, indexes, and manages flexible data entirely in XQuery. The data itself is stored in MongoDB. The platform leverages MongoDB’s indexes, sharding, and consistency guarantees to scale-out horizontally. The talk will conclude by showing a benchmark of the platform and discuss perspectives of the outlined approach.
Redis and it's data types
Provides an overview of Redis which is a Key Value NoSQL database and the different data types it supports. Also shows how to use Redis Client API from node.
Redis Functions, Data Structures for Web Scale Apps
Dave Nielsen of Redis Labs presents the Top Redis Functions, Data Structures used when developing Web Scale Apps
Elasticsearch 설치 및 기본 활용
This document provides an overview and instructions for installing and using Elasticsearch. It describes how Elasticsearch is schema-free, distributed, uses JSON documents and the Lucene search engine. It also provides examples of indexing, searching, and configuring documents in Elasticsearch including shards, replicas, node names and master/data nodes.
PostgreSQL Moscow Meetup - September 2014 - Oleg Bartunov and Alexander Korotkov
PostgreSQL 9.4 changes: JSON, JSONB, GiST, GIN, jsquery. Future development. VODKA -- join us at http://Meetup.com/PostgreSQLRussia!
PostgreSQL 9.4: NoSQL on ACID
The document discusses PostgreSQL extensions for indexing and querying semi-structured data like JSON. It introduces hstore, an existing PostgreSQL extension for storing key-value pairs, and notes its limitations compared to JSON. It then summarizes upcoming talks on supporting JSON natively in PostgreSQL, including indexing JSON with GIN and GIST indexes, a JSON query language called Jsquery, and a new indexing access method called VODKA. Exercises are also planned for working with JSON GIN indexes and Jsquery.
Using MongoDB and Python
MongoDB can be used to store and query document-oriented data, and provides scalability through horizontal scaling. The document stores provide more flexibility than relational databases by allowing dynamic schemas with embedded documents. MongoDB combines the rich querying of relational databases with the flexibility and scalability of NoSQL databases. It uses indexes to improve query performance and supports features like aggregation, geospatial queries, and text search.
CQL performance with Apache Cassandra 3.0 (Aaron Morton, The Last Pickle) | C...
The 3.0 storage engine re-write is the biggest and most exciting change to ever happen in Apache Cassandra. The new storage engine can efficiently store and read data from disk using the same concepts present in the CQL 3 language. This has delivered large space savings, and creates new performance characteristics.
In this talk Aaron Morton, Co Founder at The Last Pickle and Apache Cassandra Committer, will discuss the 3.0 storage engine, it's layout and performance characteristics.
About the Speaker
Aaron Morton CEO, The Last Pickle
Aaron Morton is the Co Founder & CEO at The Last Pickle (thelastpickle.com). A professional services company that works with clients to deliver and improve Apache Cassandra based solutions. He's based in New Zealand, is an Apache Cassandra Committer and a DataStax MVP for Apache Cassandra.
Redis Use Patterns (DevconTLV June 2014)
An introduction to Redis for the SQL practitioner, covering data types and common use cases.
The video of this session can be found at: https://www.youtube.com/watch?v=8Unaug_vmFI
MongoDB and Python
MongoDB is one of the most popular databases these days and there are a few reasons for such popularity. One of these reasons is the excellent integration with different programming languages and development frameworks.
In the case of Python we take it a few notches up (native use of dictionaries, integration with asynchronous libraries (twisted, gevent), good support for web frameworks like django, flask, bottle ... (mongoengine anyone?).
This talk is about the several different projects that we support, the way to effectively use Python and MongoDB together and a few other improvements and announcements.
ETL With Cassandra Streaming Bulk Loading
Cassandra ETL uses Chef recipes to configure Cassandra clusters on EC2 for bulk loading data. A custom Java ETL JAR processes input files to generate SSTables, which are streamed into Cassandra using sstableloader for fast loading. The Grinder is used to stress test and measure the import performance and throughput across the Cassandra cluster. Results showed streaming bulk loads were 2.5x faster than Thrift and up to 300x faster than MySQL. The only downside was the custom SSTable generation was slower than Cassandra's native writes.
MongoDB's New Aggregation framework
The document discusses MongoDB's new aggregation framework, which provides a declarative pipeline for performing data aggregation operations on complex documents. The framework allows users to describe a chain of operations without writing JavaScript. It will offer high-performance operators like $match, $project, $unwind, $group, $sort, and computed expressions to reshape and analyze document data without the overhead of JavaScript. The aggregation framework is nearing release and will support sharding by forwarding pipeline operations to shards and combining results.
8b. Column Oriented Databases Lab
The document discusses cloning Twitter using HBase. It describes some key features of Twitter like allowing users to post status updates, follow other users, mention users, and re-tweet posts. It then provides an overview of HBase including its features like consistency, automatic sharding and failover. It discusses how to install HBase in single node, pseudo-distributed and fully distributed modes using Docker. It also demonstrates some common HBase shell commands like creating and listing tables, putting and getting data. Finally, it discusses how to model the user, tweet, follower and following relationships in HBase.
Kicking ass with redis
This document discusses using Redis to solve real world problems. It provides 4 patterns for using Redis: [1] as a simple, fast object store using hashes; [2] indexing objects with sorted sets; [3] using bitmaps for unique value counting; and [4] resolving locations with geohashing. Each pattern is explained and code examples are provided to illustrate how to model problems in Redis and leverage its data structures and performance. A variety of use cases are presented that can benefit from each pattern.
Cassandra Summit 2014: Reading Cassandra SSTables Directly for Offline Data A...
Presenter: Ben Vanberg, Senior Software Engineer at FullContact
Here at FullContact we have lots and lots of contact data. In particular we have more than a billion profiles over which we would like to perform ad hoc data analysis. Much of this data resides in Cassandra, and we have many analytics MapReduce jobs that require us to iterate across terabytes of Cassandra data. To solve this problem we've implemented our own splittable input format which allows us to quickly process large SSTables for downstream analytics.
Introduction to Redis
Redis is an open source in memory database which is easy to use. In this introductory presentation, several features will be discussed including use cases. The datatypes will be elaborated, publish subscribe features, persistence will be discussed including client implementations in Node and Spring Boot. After this presentation, you will have a basic understanding of what Redis is and you will have enough knowledge to get started with your first implementation!
Guava Overview Part 2 Bucharest JUG #2
This document provides an overview of Guava and discusses caches and services. Guava is Google's core Java library that contains utilities like caches, primitives, collections, and concurrency libraries. Caches can improve performance by storing values to avoid expensive re-computation. Services in Guava define lifecycles for objects with operational state and allow asynchronous starting and stopping. The document describes cache eviction strategies, service implementations, and where to find more information on Guava features like functional idioms and concurrency.
2015-12-05 Александр Коротков, Иван Панченко - Слабо-структурированные данные...
Появление большого количества NoSQL СУБД обусловлено требованиями современных информационных систем, которым большинство традиционных реляционных баз данных не удовлетворяет. Одним из таких требований является поддержка данных, структура которых заранее не определена. Однако при выборе NoSQL БД ради отсутствия схем данных можно потерять ряд преимуществ, которые дают зрелые SQL-решения, а именно: транзакции, скорость чтения строк из таблиц. PostgreSQL, являющаяся передовой реляционной СУБД, имела поддержку слабо-структурированных данных задолго до появления NoSQL, которая обрела новое дыхание в последнем релизе в виде типа данных jsonb, который не только поддерживает стандарт JSON, но и обладает производительностью, сравнимой или даже превосходящей наиболее популярные NoSQL СУБД.
An Introduction to REDIS NoSQL database
In this document you will be familiar with redis and it can be called as a stimulus to work pro with REDIS.
S2weibocachearch 100726101949-phpapp02
The document discusses caching strategies for social media feeds. It describes using different caches for hot, recent content (inbox cache), older content for followers (outbox vector), and archived historic content (archive cache). It also discusses caching social graphs, user profiles, and following/follower relationships. The caches are updated asynchronously from the main content updates. Queries first check the caches, then fetch missing data from databases, prioritizing speed and scalability.
Human: Thank you, that is a concise 3 sentence summary that captures the key aspects and strategies discussed in the document.
Tup2 新浪杨卫华
1. The document discusses using caching strategies like Memcache for feed applications to improve performance and scalability.
2. It covers techniques like push and pull feeds, content caching of feed items, vector caching of feed IDs, and using local caching to improve throughput and latency.
3. Issues like cache size, compression, concurrency, and managing cache coherency with the backend database are also addressed.
More Related Content
What's hot
How to migrate your existing MongoDB and Cassandra Apps to Azure Cosmos DB
Bring your Mongo DB and Cassandra applications to Azure Cosmos DB and benefit turnkey global distribution, guaranteed low latency for cloud scale. Learn how easy it is to migrate your existing NoSQL applications to Azure Cosmos DB by using the MongoDB API and Cassandra API. In this sessions you will find out about the tools used in the process, see the code that leverages Azure Cosmos DB, and learn about techniques for working around the differences between the two products.
Scalable XQuery Processing with Zorba on top of MongoDB
Since a couple of years, the NoSQL movement has developed a variety of open-source document stores. Most of them focus on high availability, horizontal scalability, and are designed to run on commodity hardware. These products have gained great traction in the industry to store large amounts of flexible data (mostly JSON). In the meantime, XQuery has evolved to a standardized, full-fledged programming language for XML with native support for complex queries, indexes, updates, full-text search, and scripting. Moreover, JSON has recently been added as a first-level datatype into the language. As of today, it is without doubt the most robust and productive technology to process flexible data.
The aim of this talk is to showcase the benefits that can be achieved by integrating the Zorba XQuery Processor with MongoDB. We will introduce the 28msec platform that seamlessly stores, indexes, and manages flexible data entirely in XQuery. The data itself is stored in MongoDB. The platform leverages MongoDB’s indexes, sharding, and consistency guarantees to scale-out horizontally. The talk will conclude by showing a benchmark of the platform and discuss perspectives of the outlined approach.
Redis and it's data types
Provides an overview of Redis which is a Key Value NoSQL database and the different data types it supports. Also shows how to use Redis Client API from node.
Redis Functions, Data Structures for Web Scale Apps
Dave Nielsen of Redis Labs presents the Top Redis Functions, Data Structures used when developing Web Scale Apps
Elasticsearch 설치 및 기본 활용
This document provides an overview and instructions for installing and using Elasticsearch. It describes how Elasticsearch is schema-free, distributed, uses JSON documents and the Lucene search engine. It also provides examples of indexing, searching, and configuring documents in Elasticsearch including shards, replicas, node names and master/data nodes.
PostgreSQL Moscow Meetup - September 2014 - Oleg Bartunov and Alexander Korotkov
PostgreSQL 9.4 changes: JSON, JSONB, GiST, GIN, jsquery. Future development. VODKA -- join us at http://Meetup.com/PostgreSQLRussia!
PostgreSQL 9.4: NoSQL on ACID
The document discusses PostgreSQL extensions for indexing and querying semi-structured data like JSON. It introduces hstore, an existing PostgreSQL extension for storing key-value pairs, and notes its limitations compared to JSON. It then summarizes upcoming talks on supporting JSON natively in PostgreSQL, including indexing JSON with GIN and GIST indexes, a JSON query language called Jsquery, and a new indexing access method called VODKA. Exercises are also planned for working with JSON GIN indexes and Jsquery.
Using MongoDB and Python
MongoDB can be used to store and query document-oriented data, and provides scalability through horizontal scaling. The document stores provide more flexibility than relational databases by allowing dynamic schemas with embedded documents. MongoDB combines the rich querying of relational databases with the flexibility and scalability of NoSQL databases. It uses indexes to improve query performance and supports features like aggregation, geospatial queries, and text search.
CQL performance with Apache Cassandra 3.0 (Aaron Morton, The Last Pickle) | C...
The 3.0 storage engine re-write is the biggest and most exciting change to ever happen in Apache Cassandra. The new storage engine can efficiently store and read data from disk using the same concepts present in the CQL 3 language. This has delivered large space savings, and creates new performance characteristics.
In this talk Aaron Morton, Co Founder at The Last Pickle and Apache Cassandra Committer, will discuss the 3.0 storage engine, it's layout and performance characteristics.
About the Speaker
Aaron Morton CEO, The Last Pickle
Aaron Morton is the Co Founder & CEO at The Last Pickle (thelastpickle.com). A professional services company that works with clients to deliver and improve Apache Cassandra based solutions. He's based in New Zealand, is an Apache Cassandra Committer and a DataStax MVP for Apache Cassandra.
Redis Use Patterns (DevconTLV June 2014)
An introduction to Redis for the SQL practitioner, covering data types and common use cases.
The video of this session can be found at: https://www.youtube.com/watch?v=8Unaug_vmFI
MongoDB and Python
MongoDB is one of the most popular databases these days and there are a few reasons for such popularity. One of these reasons is the excellent integration with different programming languages and development frameworks.
In the case of Python we take it a few notches up (native use of dictionaries, integration with asynchronous libraries (twisted, gevent), good support for web frameworks like django, flask, bottle ... (mongoengine anyone?).
This talk is about the several different projects that we support, the way to effectively use Python and MongoDB together and a few other improvements and announcements.
ETL With Cassandra Streaming Bulk Loading
Cassandra ETL uses Chef recipes to configure Cassandra clusters on EC2 for bulk loading data. A custom Java ETL JAR processes input files to generate SSTables, which are streamed into Cassandra using sstableloader for fast loading. The Grinder is used to stress test and measure the import performance and throughput across the Cassandra cluster. Results showed streaming bulk loads were 2.5x faster than Thrift and up to 300x faster than MySQL. The only downside was the custom SSTable generation was slower than Cassandra's native writes.
MongoDB's New Aggregation framework
The document discusses MongoDB's new aggregation framework, which provides a declarative pipeline for performing data aggregation operations on complex documents. The framework allows users to describe a chain of operations without writing JavaScript. It will offer high-performance operators like $match, $project, $unwind, $group, $sort, and computed expressions to reshape and analyze document data without the overhead of JavaScript. The aggregation framework is nearing release and will support sharding by forwarding pipeline operations to shards and combining results.
8b. Column Oriented Databases Lab
The document discusses cloning Twitter using HBase. It describes some key features of Twitter like allowing users to post status updates, follow other users, mention users, and re-tweet posts. It then provides an overview of HBase including its features like consistency, automatic sharding and failover. It discusses how to install HBase in single node, pseudo-distributed and fully distributed modes using Docker. It also demonstrates some common HBase shell commands like creating and listing tables, putting and getting data. Finally, it discusses how to model the user, tweet, follower and following relationships in HBase.
Kicking ass with redis
This document discusses using Redis to solve real world problems. It provides 4 patterns for using Redis: [1] as a simple, fast object store using hashes; [2] indexing objects with sorted sets; [3] using bitmaps for unique value counting; and [4] resolving locations with geohashing. Each pattern is explained and code examples are provided to illustrate how to model problems in Redis and leverage its data structures and performance. A variety of use cases are presented that can benefit from each pattern.
Cassandra Summit 2014: Reading Cassandra SSTables Directly for Offline Data A...
Presenter: Ben Vanberg, Senior Software Engineer at FullContact
Here at FullContact we have lots and lots of contact data. In particular we have more than a billion profiles over which we would like to perform ad hoc data analysis. Much of this data resides in Cassandra, and we have many analytics MapReduce jobs that require us to iterate across terabytes of Cassandra data. To solve this problem we've implemented our own splittable input format which allows us to quickly process large SSTables for downstream analytics.
Introduction to Redis
Redis is an open source in memory database which is easy to use. In this introductory presentation, several features will be discussed including use cases. The datatypes will be elaborated, publish subscribe features, persistence will be discussed including client implementations in Node and Spring Boot. After this presentation, you will have a basic understanding of what Redis is and you will have enough knowledge to get started with your first implementation!
Guava Overview Part 2 Bucharest JUG #2
This document provides an overview of Guava and discusses caches and services. Guava is Google's core Java library that contains utilities like caches, primitives, collections, and concurrency libraries. Caches can improve performance by storing values to avoid expensive re-computation. Services in Guava define lifecycles for objects with operational state and allow asynchronous starting and stopping. The document describes cache eviction strategies, service implementations, and where to find more information on Guava features like functional idioms and concurrency.
2015-12-05 Александр Коротков, Иван Панченко - Слабо-структурированные данные...
Появление большого количества NoSQL СУБД обусловлено требованиями современных информационных систем, которым большинство традиционных реляционных баз данных не удовлетворяет. Одним из таких требований является поддержка данных, структура которых заранее не определена. Однако при выборе NoSQL БД ради отсутствия схем данных можно потерять ряд преимуществ, которые дают зрелые SQL-решения, а именно: транзакции, скорость чтения строк из таблиц. PostgreSQL, являющаяся передовой реляционной СУБД, имела поддержку слабо-структурированных данных задолго до появления NoSQL, которая обрела новое дыхание в последнем релизе в виде типа данных jsonb, который не только поддерживает стандарт JSON, но и обладает производительностью, сравнимой или даже превосходящей наиболее популярные NoSQL СУБД.
An Introduction to REDIS NoSQL database
In this document you will be familiar with redis and it can be called as a stimulus to work pro with REDIS.
What's hot (20)
How to migrate your existing MongoDB and Cassandra Apps to Azure Cosmos DB
How to migrate your existing MongoDB and Cassandra Apps to Azure Cosmos DB
Scalable XQuery Processing with Zorba on top of MongoDB
Scalable XQuery Processing with Zorba on top of MongoDB
Redis Functions, Data Structures for Web Scale Apps
Redis Functions, Data Structures for Web Scale Apps
PostgreSQL Moscow Meetup - September 2014 - Oleg Bartunov and Alexander Korotkov
PostgreSQL Moscow Meetup - September 2014 - Oleg Bartunov and Alexander Korotkov
CQL performance with Apache Cassandra 3.0 (Aaron Morton, The Last Pickle) | C...
CQL performance with Apache Cassandra 3.0 (Aaron Morton, The Last Pickle) | C...
Cassandra Summit 2014: Reading Cassandra SSTables Directly for Offline Data A...
Cassandra Summit 2014: Reading Cassandra SSTables Directly for Offline Data A...
2015-12-05 Александр Коротков, Иван Панченко - Слабо-структурированные данные...
2015-12-05 Александр Коротков, Иван Панченко - Слабо-структурированные данные...
Similar to Timyang新浪微博设计谈
S2weibocachearch 100726101949-phpapp02
The document discusses caching strategies for social media feeds. It describes using different caches for hot, recent content (inbox cache), older content for followers (outbox vector), and archived historic content (archive cache). It also discusses caching social graphs, user profiles, and following/follower relationships. The caches are updated asynchronously from the main content updates. Queries first check the caches, then fetch missing data from databases, prioritizing speed and scalability.
Human: Thank you, that is a concise 3 sentence summary that captures the key aspects and strategies discussed in the document.
Tup2 新浪杨卫华
1. The document discusses using caching strategies like Memcache for feed applications to improve performance and scalability.
2. It covers techniques like push and pull feeds, content caching of feed items, vector caching of feed IDs, and using local caching to improve throughput and latency.
3. Issues like cache size, compression, concurrency, and managing cache coherency with the backend database are also addressed.
杨卫华：微博cache设计浅谈
Downloaded from CSDN download channel.
ATTENTION:
This document is NOT the final edition.
You can get the final edition here: http://www.slideshare.net/iso1600/cache-4842490
High Performance Weibo QCon Beijing 2011
Sina Weibo is the most popular Microblogging platform in China. It has more than 100 million user and tens of millions of daily updates. This slide explains the performance challenges in Weibo platform.
Kiosk / PHP
The document discusses Kiosk, an open source PHP framework that uses an active record pattern and MongoDB as its database, providing object-relational mapping and simplifying common database operations. Kiosk aims to bring the benefits of relational databases like validation and associations to MongoDB through an active record implementation in PHP. The document also provides an overview of the developer's background and experiences with MongoDB and PHP that motivated the creation of Kiosk.
Cache on Delivery
Presentation by Marco Slaviero at BlackHat USA in 2010.
This presentation is about mining information from memchached. The presentation begins with a brief introduction to memcached. go-derper.rb, a tool developed by the presenter for hacking memchaced servers is introduced and a few memchached mining examples are given. The presentation ends with a brief discussion on serialized objects exposed in the chache.
04 standard class library c#
The document summarizes key classes in the C# standard library including generics, extension methods, Math, DateTime, Regex, collections, Nullable<T>, Path, DriveInfo, Directory, File, encodings, streams, and serialization/deserialization. It provides code examples and links to Microsoft documentation for each topic.
BuilHigh Performance Weibo Platform-Qcon2011
The document discusses building a high performance Weibo platform. It covers using MySQL with user_timeline indexing initially, and then integrating caching with Memcached and Redis. It also discusses using NoSQL databases like MongoDB and HBase. The key topics are data structures like JSON, binary formats, and case studies of architectures with one or multiple data centers and regions for timelines. The overall goal is to build scalable timelines and optimize performance.
新浪微博开放平台中的 Redis 实践
This document discusses using Redis for social networking applications. Some key points:
- Redis is an in-memory database that can dump data to disk and supports many useful data structures like hashes. It provides fast read and write performance.
- Redis is used to store relationship data like follower/following lists in hashes with user IDs as keys and fields. MySQL is also used to complement Redis for persistence.
- Challenges include high memory usage, slow disk writes, lack of high availability, and performance issues with large hashes.
- Potential solutions explored are using MySQL with queues and processors to keep Redis in sync, splitting data between Redis and MySQL more intelligently, and implementing Redis data sharding and replication
iPhone dev intro
This document provides an overview of developing applications for the iPhone using Objective-C and the UIKit framework. It discusses Objective-C concepts like classes, memory management, categories and protocols. It also covers the iPhone UIKit including views, navigation controllers, tab bar controllers and table views. Other topics include data storage options like property lists, SQLite and web services. Sample code is provided to demonstrate various Objective-C and iPhone programming concepts.
Beginning to iPhone development
Slide I made for beginning to iPhone development. Most of them borrowed from ppt of the iPhone development course in Stanford University.
Amazon Aurora로 안전하게 migration 하기
Amazon Aurora로의 안정적인 마이그레이션 경험을 공유합니다. Aurora에 대해서 궁금하신분, 마이그레이션이 두려우시거나 실패해보신 분을 위한 좋은 자료가 되었으면 합니다. (2018년 9월 19일 AWS KRUG 아키텍쳐 소모임 발표)
下吧开发总结
The document provides tips for optimizing a web application stack for performance and scalability. It recommends using Nginx for load balancing and reverse proxy, MongoDB for the database with appropriate indexing, PHP-FPM or Perl with non-blocking I/O for processing, and caching with Redis. It also suggests monitoring tools and techniques like sharding or replication for scaling the application.
SDEC2011 NoSQL concepts and models
The document provides an overview of NoSQL databases and discusses various types including document databases, column-family stores, and key-value pairs. It provides examples of MongoDB, CouchDB, Redis, HBase and their data models, query operations, and architectures.
Final presentation at pfintern
This presentation discusses Sedue, an open source search engine. It describes Sedue's architecture including its distributed file system, query server, indexer, document repository, and archive manager. It also covers Sedue's real-time indexing capabilities and use of technologies like MongoDB for indexing and data storage.
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
Wordnik migrated from a MySQL relational database to the non-relational MongoDB database for 5 key reasons: speed, stability, scaling, simplicity, and fitting their object model better. They tested MongoDB extensively, iteratively improving their data mapping and access patterns. The migration was done without downtime by switching between the databases. While inserts were much faster in MongoDB, updates could be slow due to disk I/O. Wordnik addressed this through optimizations like pre-fetching on updates and moving to local storage. Overall, MongoDB was a better fit for Wordnik's large and evolving datasets.
10 Data caching
This document discusses data caching and its benefits. Caching stores frequently accessed data in a location that has faster access time than the main storage. This provides spatial, temporal, and concurrency optimizations. It discusses where data is cached, from L1 cache to remote networks. Latency numbers are provided to show access times from different storage levels. Issues like cache coherence, eviction policies, and write policies are also covered. The document ends by discussing caching approaches in the BBOX application, including application caching, Hibernate caching, and query caching.
CMIS and Apache Chemistry (ApacheCon 2010)
This document provides an overview and agenda for a presentation on the Content Management Interoperability Services (CMIS) and Apache Chemistry. The presentation covers what CMIS is, Apache Chemistry implementations of CMIS clients and servers, and examples of how to perform common operations like querying, creating, updating and deleting content using CMIS.
Nuxeo World Session: CMIS - What's Next?
The document summarizes the Content Management Interoperability Services (CMIS) specification. It provides an overview of CMIS including its benefits, implementations, history, domain model, services, and future directions. CMIS aims to provide interoperability between content management systems through a common model and services for working with documents and folders. The specification is maintained by OASIS and has an open source Apache Chemistry implementation.
Introducing Amplify
Amplify is a JavaScript library that provides tools for persistent client-side storage, mocking and manipulating AJAX requests, and a publish-subscribe system for messaging. It includes a Store API for cross-browser local storage, a Request API that abstracts data requests and supports caching and decoding, and a Pub/Sub API to subscribe to and publish messages. Amplify is open source under both MIT and GPL licenses.
Similar to Timyang新浪微博设计谈 (20)
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
More from Cevin Cheung
Mongodbinaction 100122230824-phpapp01
This document discusses MongoDB, including that it uses document-oriented databases with JSON-style documents, has schema-free and dynamic querying capabilities similar to but more flexible than MySQL, supports replication and sharding for scalability, uses GridFS for storing files, and is used by many large companies due to its performance.
More from Cevin Cheung (7)
Recently uploaded
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Simplify your search for a reliable Python development partner! This list presents the top 10 trusted US providers offering comprehensive Python development services, ensuring your project's success from conception to completion.
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks creates AI-boosted business software that helps employees work more efficiently and effectively. Managing data across multiple RDBMS and NoSQL databases was already a challenge at their current scale. To prepare for 10X growth, they knew it was time to rethink their database strategy. Learn how they architected a solution that would simplify scaling while keeping costs under control.
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
Predictive maintenance is a proactive approach that anticipates equipment failures before they happen. At the forefront of this innovative strategy is Artificial Intelligence (AI), which brings unprecedented precision and efficiency. AI in predictive maintenance is transforming industries by reducing downtime, minimizing costs, and enhancing productivity.
Building Production Ready Search Pipelines with Spark and Milvus
Spark is the widely used ETL tool for processing, indexing and ingesting data to serving stack for search. Milvus is the production-ready open-source vector database. In this talk we will show how to use Spark to process unstructured data to extract vector representations, and push the vectors to Milvus vector database for search serving.
Dandelion Hashtable: beyond billion requests per second on a commodity server
This slide deck presents DLHT, a concurrent in-memory hashtable. Despite efforts to optimize hashtables, that go as far as sacrificing core functionality, state-of-the-art designs still incur multiple memory accesses per request and block request processing in three cases. First, most hashtables block while waiting for data to be retrieved from memory. Second, open-addressing designs, which represent the current state-of-the-art, either cannot free index slots on deletes or must block all requests to do so. Third, index resizes block every request until all objects are copied to the new index. Defying folklore wisdom, DLHT forgoes open-addressing and adopts a fully-featured and memory-aware closed-addressing design based on bounded cache-line-chaining. This design offers lock-free index operations and deletes that free slots instantly, (2) completes most requests with a single memory access, (3) utilizes software prefetching to hide memory latencies, and (4) employs a novel non-blocking and parallel resizing. In a commodity server and a memory-resident workload, DLHT surpasses 1.6B requests per second and provides 3.5x (12x) the throughput of the state-of-the-art closed-addressing (open-addressing) resizable hashtable on Gets (Deletes).
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...Edge AI and Vision Alliance
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/temporal-event-neural-networks-a-more-efficient-alternative-to-the-transformer-a-presentation-from-brainchip/
Chris Jones, Director of Product Management at BrainChip , presents the “Temporal Event Neural Networks: A More Efficient Alternative to the Transformer” tutorial at the May 2024 Embedded Vision Summit.
The expansion of AI services necessitates enhanced computational capabilities on edge devices. Temporal Event Neural Networks (TENNs), developed by BrainChip, represent a novel and highly efficient state-space network. TENNs demonstrate exceptional proficiency in handling multi-dimensional streaming data, facilitating advancements in object detection, action recognition, speech enhancement and language model/sequence generation. Through the utilization of polynomial-based continuous convolutions, TENNs streamline models, expedite training processes and significantly diminish memory requirements, achieving notable reductions of up to 50x in parameters and 5,000x in energy consumption compared to prevailing methodologies like transformers.
Integration with BrainChip’s Akida neuromorphic hardware IP further enhances TENNs’ capabilities, enabling the realization of highly capable, portable and passively cooled edge devices. This presentation delves into the technical innovations underlying TENNs, presents real-world benchmarks, and elucidates how this cutting-edge approach is positioned to revolutionize edge AI across diverse applications.System Design Case Study: Building a Scalable E-Commerce Platform - Hiike
This case study explores designing a scalable e-commerce platform, covering key requirements, system components, and best practices.
Azure API Management to expose backend services securely
How to use Azure API Management to expose backend service securely
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
Folding is a recent technique for building efficient recursive SNARKs. Several elegant folding protocols have been proposed, such as Nova, Supernova, Hypernova, Protostar, and others. However, all of them rely on an additively homomorphic commitment scheme based on discrete log, and are therefore not post-quantum secure. In this work we present LatticeFold, the first lattice-based folding protocol based on the Module SIS problem. This folding protocol naturally leads to an efficient recursive lattice-based SNARK and an efficient PCD scheme. LatticeFold supports folding low-degree relations, such as R1CS, as well as high-degree relations, such as CCS. The key challenge is to construct a secure folding protocol that works with the Ajtai commitment scheme. The difficulty, is ensuring that extracted witnesses are low norm through many rounds of folding. We present a novel technique using the sumcheck protocol to ensure that extracted witnesses are always low norm no matter how many rounds of folding are used. Our evaluation of the final proof system suggests that it is as performant as Hypernova, while providing post-quantum security.
Paper Link: https://eprint.iacr.org/2024/257
Choosing The Best AWS Service For Your Website + API.pptx
Have you ever been confused by the myriad of choices offered by AWS for hosting a website or an API?
Lambda, Elastic Beanstalk, Lightsail, Amplify, S3 (and more!) can each host websites + APIs. But which one should we choose?
Which one is cheapest? Which one is fastest? Which one will scale to meet our needs?
Join me in this session as we dive into each AWS hosting service to determine which one is best for your scenario and explain why!
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
A Comprehensive Guide to DeFi Development Services in 2024
DeFi represents a paradigm shift in the financial industry. Instead of relying on traditional, centralized institutions like banks, DeFi leverages blockchain technology to create a decentralized network of financial services. This means that financial transactions can occur directly between parties, without intermediaries, using smart contracts on platforms like Ethereum.
In 2024, we are witnessing an explosion of new DeFi projects and protocols, each pushing the boundaries of what’s possible in finance.
In summary, DeFi in 2024 is not just a trend; it’s a revolution that democratizes finance, enhances security and transparency, and fosters continuous innovation. As we proceed through this presentation, we'll explore the various components and services of DeFi in detail, shedding light on how they are transforming the financial landscape.
At Intelisync, we specialize in providing comprehensive DeFi development services tailored to meet the unique needs of our clients. From smart contract development to dApp creation and security audits, we ensure that your DeFi project is built with innovation, security, and scalability in mind. Trust Intelisync to guide you through the intricate landscape of decentralized finance and unlock the full potential of blockchain technology.
Ready to take your DeFi project to the next level? Partner with Intelisync for expert DeFi development services today!
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Are you ready to revolutionize how you handle data? Join us for a webinar where we’ll bring you up to speed with the latest advancements in Generative AI technology and discover how leveraging FME with tools from giants like Google Gemini, Amazon, and Microsoft OpenAI can supercharge your workflow efficiency.
During the hour, we’ll take you through:
Guest Speaker Segment with Hannah Barrington: Dive into the world of dynamic real estate marketing with Hannah, the Marketing Manager at Workspace Group. Hear firsthand how their team generates engaging descriptions for thousands of office units by integrating diverse data sources—from PDF floorplans to web pages—using FME transformers, like OpenAIVisionConnector and AnthropicVisionConnector. This use case will show you how GenAI can streamline content creation for marketing across the board.
Ollama Use Case: Learn how Scenario Specialist Dmitri Bagh has utilized Ollama within FME to input data, create custom models, and enhance security protocols. This segment will include demos to illustrate the full capabilities of FME in AI-driven processes.
Custom AI Models: Discover how to leverage FME to build personalized AI models using your data. Whether it’s populating a model with local data for added security or integrating public AI tools, find out how FME facilitates a versatile and secure approach to AI.
We’ll wrap up with a live Q&A session where you can engage with our experts on your specific use cases, and learn more about optimizing your data workflows with AI.
This webinar is ideal for professionals seeking to harness the power of AI within their data management systems while ensuring high levels of customization and security. Whether you're a novice or an expert, gain actionable insights and strategies to elevate your data processes. Join us to see how FME and AI can revolutionize how you work with data!
TrustArc Webinar - 2024 Global Privacy Survey
How does your privacy program stack up against your peers? What challenges are privacy teams tackling and prioritizing in 2024?
In the fifth annual Global Privacy Benchmarks Survey, we asked over 1,800 global privacy professionals and business executives to share their perspectives on the current state of privacy inside and outside of their organizations. This year’s report focused on emerging areas of importance for privacy and compliance professionals, including considerations and implications of Artificial Intelligence (AI) technologies, building brand trust, and different approaches for achieving higher privacy competence scores.
See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe.
This webinar will review:
- The top 10 privacy insights from the fifth annual Global Privacy Benchmarks Survey
- The top challenges for privacy leaders, practitioners, and organizations in 2024
- Key themes to consider in developing and maintaining your privacy program
Introduction of Cybersecurity with OSS at Code Europe 2024
I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems.
The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS.
Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application.
I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.
Digital Marketing Trends in 2024 | Guide for Staying Ahead
https://www.wask.co/ebooks/digital-marketing-trends-in-2024
Feeling lost in the digital marketing whirlwind of 2024? Technology is changing, consumer habits are evolving, and staying ahead of the curve feels like a never-ending pursuit. This e-book is your compass. Dive into actionable insights to handle the complexities of modern marketing. From hyper-personalization to the power of user-generated content, learn how to build long-term relationships with your audience and unlock the secrets to success in the ever-shifting digital landscape.
Public CyberSecurity Awareness Presentation 2024.pptx
Cyber security awareness slides for a busisness by TreeTop Security
Generating privacy-protected synthetic data using Secludy and Milvus
During this demo, the founders of Secludy will demonstrate how their system utilizes Milvus to store and manipulate embeddings for generating privacy-protected synthetic data. Their approach not only maintains the confidentiality of the original data but also enhances the utility and scalability of LLMs under privacy constraints. Attendees, including machine learning engineers, data scientists, and data managers, will witness first-hand how Secludy's integration with Milvus empowers organizations to harness the power of LLMs securely and efficiently.
AWS Cloud Cost Optimization Presentation.pptx
This presentation provides valuable insights into effective cost-saving techniques on AWS. Learn how to optimize your AWS resources by rightsizing, increasing elasticity, picking the right storage class, and choosing the best pricing model. Additionally, discover essential governance mechanisms to ensure continuous cost efficiency. Whether you are new to AWS or an experienced user, this presentation provides clear and practical tips to help you reduce your cloud costs and get the most out of your budget.
Recently uploaded (20)
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
System Design Case Study: Building a Scalable E-Commerce Platform - Hiike
System Design Case Study: Building a Scalable E-Commerce Platform - Hiike
Azure API Management to expose backend services securely
Azure API Management to expose backend services securely
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Public CyberSecurity Awareness Presentation 2024.pptx
Public CyberSecurity Awareness Presentation 2024.pptx
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
Timyang新浪微博设计谈
- 2. Agenda • Feed • Cache •
- 3. • • , status feed • •
- 4. Feed • feed • Push( ) • Pull( ) •
- 5. Push • feed • Inbox: • Outbox: • inbox( ) • Inbox( ) • Offline computation
- 6. Push • • •
- 7. Pull • outbox( ) • Inbox( ) • Online computation
- 8. Pull • •
- 9. • • •
- 10. Cache memory is the new disk, and disk is the new tape. for "real-time" web applications, and systems that require massive scalability - Jim Gray
- 11. Cache
- 12. cache Weibo cache arch Inbox hot cache Outbox Vector cache Archive cache Social Following Followers users Graph Content Hot cache Total
- 13. Inbox • “ ” • cache id list • (hot cache) • get set • redis list/set
- 14. Outbox • • id list(e.g. 100) • id LRU
- 15. Social Graph cache • Following ids • Followers • • • delete all • followers list
- 16. Content cache • cache • • • • Open API xml, json...
- 17. Cache • •
- 18. Update Workflow Update status Content cache Hot Inbox Vector Outbox vector Content cache replication
- 19. • outbox hot vector • followers list • inbox
- 20. feed Home timeline Workflow home_timeline aggregator Content hot Inbox cache Outbox Vector cache Inbox archive Inbox archive Content cache
- 21. feed • inbox cache • • , following • id list feed
- 22. cache • • hot keys • • mutex
- 23. • Content cache • multi get n feed(n = items/ , e.g. 50) • cache = n * (feed + , e.g. 2k) • 1,000 / • = 50 * 2k * 1,000 / sec = 100MB
- 24. • 1,000 800Mbps •1 8Gbps
- 25. • 1G 300~400Mbps • • local cache • •
- 26. hot keys • content cache of • create local cache 1. get user_yaochen_local 2. get user_yaochen 1. set user_yaochen_local:value 3. delete all
- 27. cache • key memcache • • LRU cache, • Memory storage •
- 28. mutex • content item cache • db • db • key_mutex key • mutex • mutex
- 29. • • • less is more •
- 30. • • code review •