Autonomous Navigation on a Physical Car

Autonomous driving from raw sensor data comes down to two questions: which sensors to trust, and how to combine them. For our Autonomous Navigation course we tackled both, on a real physical robot. We built a full self-driving pipeline (data collection, sensor fusion, training, deployment) around CIL v2, a vision-based end-to-end driving architecture with multi-view attention, and fed it multimodal input: an RGB camera at the front of the car, and a 2D LiDAR on top. We prototyped the model and evaluation loop in the CARLA simulator to iterate quickly, then moved the whole pipeline to the physical vehicle. There was no autopilot to bootstrap from and no synthetic sensors to fall back on, so every single training frame on the real car came from one of us physically driving it with a joystick, across several circuits, drivers, and lighting conditions. The data was the project.

The hardest thing to get right was obstacle avoidance. With only the front camera, we could not get the car to reliably come back to its lane after passing an obstacle. The reason turned out to be structural: from a forward-facing camera, the three phases of an overtake (see the obstacle, drive past it in the other lane, come back to your lane) look almost identical once the obstacle has left the field of view. From the model's perspective there is no visible difference between "still overtaking" and "already back", so it collapses to a shortcut and learns that driving in the middle of the road is fine, which then breaks normal lane-following whenever obstacle-avoidance data is mixed in. Adding LiDAR resolved the ambiguity: an obstacle right next to the car shows up as a persistent, distinctive circle in the 2D point cloud, giving the model a signal that survives the whole manoeuvre. The LiDAR-conditioned model overtook obstacles correctly and came back to its lane only once the circle disappeared.