中级农民
- 积分
- 111
- 大米
- 颗
- 鳄梨
- 个
- 水井
- 尺
- 蓝莓
- 颗
- 萝卜
- 根
- 小米
- 粒
- 学分
- 个
- 注册时间
- 2016-6-26
- 最后登录
- 1970-1-1
|
原帖 sd：camerlcamerlcamerl, 请注意api设计
https://www.1point3acres.com/bbs/thread-1100761-1-1.html
https://leetcode.com/discuss/interview-question/system-design/1835136/design-price-tracker-like-camelcamelcamelcom-looking-for-answers
参考：
https://sumit-ghosh.com/posts/serverless-price-tracker-alert-aws/
https://medium.com/contentarea/a-guide-to-building-your-own-price-tracking-engine-65b2db4b47c7
ChatGPT解读
Goals of the System
1. Track product prices across multiple platforms or stores.
2. Notify users about price changes or specific price thresholds.
3. Provide a user interface for browsing price histories and setting alerts.
4. Ensure scalability for handling a large number of products and users.
需求是创建一个监控类似亚马逊网站的产品价格波动的系统。非功能性需求，考虑大量的产品和用户。
Amazon API Gateway提供RESTFul 和 Websocket两种API。Websocket通常用于双向工业应用，比如控制路由器，升级路由器的时候需要双向通讯发送命令。这个例子应该是用REST。API Gateway还可以提供Rate Limit and Load Balance.
https://aws.amazon.com/api-gateway/
Amazon DynamoDB是Cassandra的早期版本。Cassandra 用Column Family->SuperColumn->Column->Value, 多层K-V结构；DynamoDB 基于 KV+Document数据结构。Cassandra自带Region， DynamoDB用Amazon Region。 https://www.influxdata.com/comparison/cassandra-vs-dynamodb/
本例我倾向下文的MongoDB，这样Sharding好最点。本身数据比较小。
SQS用于存Request Message。一个Lambda抓产品信息，一个抓价格信息，两者通过SQS传递。然后价格信息，CRON定期SNS通知用户。
High-Level Architecture
1. Frontend:
• User-facing application for viewing price histories, setting alerts, and managing subscriptions.
• Built with modern frontend frameworks like React or Angular for a responsive UI.
2. Backend API:
• Handles user requests, manages alerts, and interfaces with the price-tracking engine.
• Implemented as a RESTful or GraphQL API.
3. Price Scraper:
• Scrapes product prices from e-commerce websites using tools like Puppeteer, Selenium, or APIs provided by platforms.
• Runs periodically (using cron jobs or serverless triggers) to update the price database.
4. Database:
• Stores product information, price history, user data, and alerts.
• Relational databases like PostgreSQL for structured data or MongoDB for flexibility.
• Time-series databases (e.g., InfluxDB) for efficient storage and querying of price history.
5. Notification Service:
• Sends alerts to users via email, SMS, or push notifications when price changes occur.
• Integrate with services like Amazon SES, Twilio, or Firebase Cloud Messaging.
6. Scheduler:
• Automates periodic scraping and alert-checking tasks.
• Can be implemented using tools like AWS Lambda + EventBridge or Celery with workers.
我记得今年比较过几个Time- Series DB，InfluxDB Performance好一点，Prometheus开发容易一点。考虑Scalability得话，倾向InfluxDB。 https://www.influxdata.com/comparison/influxdb-vs-prometheus/
Detailed Design
1. Frontend
• Users log in, search for products, and view price histories.
• Users can set alerts for desired price points or percentage drops.
• Real-time updates using WebSockets or Server-Sent Events (SSE).
2. Backend API
• Exposes endpoints for:
• User management (POST /users, GET /users/{id}).
• Product and price history queries (GET /products/{id}/history).
• Setting alerts (POST /alerts).
• Implements rate limiting and caching for high-traffic endpoints.
• Authentication and authorization using OAuth 2.0 or JWT.
3. Price Scraper
• Crawls target websites using:
• Headless browsers (e.g., Puppeteer, Playwright) for dynamic pages.
• API integrations where available for efficiency.
• Avoids getting blocked by using:
• Proxies or IP rotation.
• User-agent randomization.
• Stores scraped data in a temporary queue (e.g., AWS SQS or RabbitMQ) for processing.
4. Data Storage
• Relational Database (PostgreSQL):
• Tables for users, products, price history, and alerts.
• Efficient indexing for fast query performance.
• Time-Series Database (InfluxDB):
• For storing and analyzing price trends over time.
5. Notification Service
• Evaluates price changes against user-defined thresholds.
• Sends notifications through:
• Email (Amazon SES, SendGrid).
• SMS (Twilio).
• Push notifications (Firebase Cloud Messaging).
6. Scalability
• Use serverless architectures (e.g., AWS Lambda, Azure Functions) to handle fluctuating workloads.
• Auto-scaling with containers (e.g., Kubernetes or AWS ECS) for high scraper demand.
上面的Backend API可以结合帖子自己细化一下。Meta我面的是Troubleshooting。Amazon正在准备电话面试，就不深入。
https://systemdesignfightclub.com/price-drop-tracker/ 这个设计要更细一些，思路差不多。
Workflow Example
1. Price Scraping:
• A scheduled job triggers the scraper to fetch product prices.
• Scraped data is validated and stored in the database.
• Price history is updated, and alerts are evaluated.
2. User Interaction:
• User logs in and searches for a product.
• Backend retrieves price history and sends it to the frontend.
• User sets a price alert for a specific product.
3. Notification:
• When a price drop is detected, the system sends a notification to the user.
Technologies
• Frontend: React, Angular, or Vue.js.
• Backend: Node.js (Express), Python (Flask/Django), or Java (Spring Boot).
• Scraping: Puppeteer, Playwright, BeautifulSoup.
• Database: PostgreSQL, MongoDB, InfluxDB.
• Message Queue: RabbitMQ, AWS SQS.
• Notifications: Amazon SES, Twilio, Firebase.
• Hosting: AWS (Lambda, EC2, S3), Google Cloud, Azure.
前两年给亚马逊做白老鼠，搭建一个卖奢侈品的网站，搭建很方便。就是环境很烂，用的应该是LAMDA这种技术。运行一段时间，服务器就挂了，重启之后，Domain/IP要重新设置。如果面Meta可以把他贬🉐一无是处（Sucks）。当然面Amazon就不行了，可以说很方便，就是有时候不稳定。 https://aws.amazon.com/lightsail/projects/wordpress/
语言的选用，Golang从工业的测试性能最优。相比NodeJS，差不多翻倍。我的理解，Golang充分运用了服务器的多CPU，做Parallel Computing。 其他语言不具备这个能力。
Key Features
1. Scalable Scraping:
• Use distributed systems to scrape multiple websites concurrently.
2. Efficient Storage:
• Optimize database queries for large-scale data.
3. User Personalization:
• Allow users to create custom price alerts.
4. Real-Time Updates:
• Notify users immediately when price changes occur.
This system design ensures a robust, scalable, and user-friendly price-tracking engine. The architecture leverages serverless and modern cloud technologies to handle large-scale operations effectively.
Scraping抓价格，我想提的是，要注意时间间隔，不让很容易被网站的Rate LIMITER禁IP。
|
本帖子中包含更多资源
您需要 登录 才可以下载或查看附件。没有帐号？注册账号 
x
|
狗仔卡
收藏
淘帖
好苗
杂草
提升卡
置顶卡
关贴卡
开贴卡
变色卡
千斤顶
楼主
