This paper presents a velocity planning method for autonomous vehicles (AVs) to guarantee safe interactions with pedestrians at unsignalized crosswalks and with surrounding vehicles on the AV’s route. The method is structured within a hierarchical framework that includes robust control, a learning-based component, and a supervisory element. The learning-based component is trained using reinforcement learning techniques to reduce traveling time, minimize control interventions, and set the priority ratio between the AV and pedestrians. The supervisory element employs scenario optimization, using statistical data on pedestrian motions to ensure collision avoidance. A complex game-theory-based pedestrian model is formulated and analyzed in order to evaluate the effectiveness of the proposed velocity planning method. Extensive simulations are performed using the high-precision traffic simulator software SUMO. These simulations evaluate various aspects of the velocity planner, including computation time, traveling time, control interventions, and parameter settings. The results demonstrate the method’s ability to achieve real-time implementation while maintaining safety and performance objectives.