案例分析：系统设计fb的newsfeed

上文发了  热门公司系统设计题实战参考 （36家）

https://www.1point3acres.com/bbs/thread-889046-1-1.html

那完整的系统设计应该如何做，举个例子，设计fb的newsfeed

1. Requirement clarification
App/web/cross platform

Features/Purpose

1. Generate newsfeed
2. Follow other
3. Get/store image/video/text
4. Notification/publish feed

Sorting,search

How many/ traffic


2. Non-functional req
- Fast, in 5s
- Scale of the system
- Traffic: 30M DAU, 10 newsfeed/user, 300M newsfeed request /day = 3500req/s
- Storage estimate, cache 500 post, each one is text 1kb, one user 500kb
- 30M * 500KB = 15TB memcache, server can store 100 GB, 150 machines in total
- A modern server can handle 50K concurrent connection at any time,


4. System api

Feed publishing API
Post /v1/me/feed
Params: content, user_id, auth_token
News feed retrieval API
Get /v1/me/feed
Params: auth_token, user_id

Database design

User, entity (page, group), feedItem, media
userfollow

5. High level design

Feed Generation (building)

Notification, when user online,

Fanout service, write vs read

1. Use group db to get follow ids
2. Get user info from cache,
3. Post, fried list send to message queue
4. Fan out worker saved data to newsfeed cache, <post id, user_id> append when new comes

Attention to rate limit and auth on server

6. Detailed design
Pull model or fan out on load, recent feed kept in memory on server
Pro: dead user not need occupy, no hot key for celebrity
Cons: old till next pull request
       Refresh post w/o new content, wast resource

Push model or fan out on write, when post is updated , user need to long pulling /websocket
Pro: idol send feed, fans get it, newsfeed push to user, no need to fetch id and read files
Con: idol is too hot, push consume a lot, hot key

Hybrid: common user: push, celebrities : pull model

7 bottleneck (single post of failure, replica of data if lost, monitoring, alert, auto fix .ticket)

Do we send newsfeed for everyone? Zombie no need
1, LRU cache
2. Predict login time based on ML model, and warm up your feeds

Follow up on notification and ranking

1. Optional on user
2. If device is limit traffic, pull munully

Ranking system:
1. Likes, stay time, likes, audience category, hashtag, target ads

Scaling database
Vertical/ horizontal
Sql vs nodal
Master slvae replicas
Consistency models
Database sharing
Keep web tier stateless
Hotkey festival special case, downgrade some uncreital features
------------------

The Kappa Architecture is considered a simpler alternative to the Lambda Architecture as it uses the same technology stack to handle both real-time stream processing and historical batch processing. Both architectures entail the storage of historical data to enable large-scale analytics. Both architectures are also useful for addressing “human fault tolerance,” in which problems with the processing code (either bugs or just known limitations) can be overcome by updating the code and running it again on the historical data. The main difference with the Kappa Architecture is that all data is treated as if it were a stream, so the stream processing engine acts as the sole data transformation engine.