Self-driving vehicles are receiving increasing attention in the development of smart cities, as the (partial) automation of transportation contributes to creating a more efficient and sustainable urban environment. Detecting road defects and surface irregularities plays a significant role in achieving safe transportation. The Machine Perception Research Laboratory (HUN-REN SZTAKI), in collaboration with the D3 Seeron startup company, has developed an artificial intelligence-based system capable of pre-identifying potholes, speed bumps, and manhole covers, among other obstacles. The algorithm aids in the control of autonomous vehicles by not only identifying obstacles but also determining their positions, enabling adjustments to the vehicle’s speed and trajectory. The system presented in the article combines the low computational demands of vision-based object detection with the precise distance measurement capabilities of LIDAR technology.