ML System Design - Design Perception System to Yield to Construction Zone

🙏求加米点评！最近和朋友在Mock ML System Design，想贴出来和大家讨论，这是我设计的一个题目。明天把4-6贴出来

Problem: Design an ML-based system to detect, understand, and yield to workers at construction zones. Construction zones present unique challenges for autonomous vehicles due to their temporary nature, non-standard road layouts, the presence of workers, equipment, and unique signage.
key points

• Data and Sensors
• Model Architecture
• Training Strategy
• Evaluation and Metrics
• Model Deployment
• Failure Models and Iterations
  

1. Data and Sensors

• Sensors: front wide camera + front narrow camera + side fisheye cameras + lidar + radar
• Retrieve construction sites: curate routes of construction sites based on the map;
• Targeted collection during challenging conditions: Dawn/dusk (glare conditions), Night construction, Different weather (rain, light snow, fog), High-traffic construction areas
• Annotation method: manual annotation and auto annotation via vision language model (Qwen 2.5 VL)
  

2. Model Architecture

• BEVFusion for 3D object detection and BEV segmentation
• multiview image input data -> camera encoder (SwinT feature extraction) -> camera to bev transformation (bev pooling)
• lidar input data -> lidar encoder (Voxelnet feature extraction) -> lidar to bev transformation (flatten along z-axis)
• both branches merge: camera bev features + lidar bev features (concatenation) -> bev encoder (fully connected with residual blocks) -> task head (3dod / bev map seg)
• 3dod head: CenterPoint to predict center location of objects, object size, rotation, veolcity
• bev map seg head: multiple binary semantic segmentation heads with focal loss
  

3. Training Strategy

• Loss function for 3dod: Gaussian Focal Loss(predict object center), Regression loss (estimate object size, rotation, velocity)
• Loss function for bev segmentation: Focal Loss, with one loss for each of the 6 map classes (drivable space, pedestrian crossing, walkway, stop line, car-parking area, and lane divider)
• Curriculum learning: start with clear daytime construction, progress to challenging lighting and weather conditions
• Knowledge distillation from larger teacher models, e.g. distill depth information from lidar model to camera model
• Self-supervised learning on unlabeled construction data