The paper presents a combined observer and lateral control design approach for autonomous vehicles. The goal of the observer design is to estimate the front and rear slip angles of the vehicle together with the cornering stiffness. The observer is based on the combination of the polytopic LPV approach and the ultra-local model. The ultra-local model is used to update the cornering stiffness and, in this way, improve the performance level of the LPV observer. Then, the resulted observer is exploited during the LPV-based lateral control design. The improved lateral control can adapt to different circumstances such as low adhesion coefficient. The proposed observer and LPV controller is implemented and tested in MATLAB/Simulink environment and using the high-fidelity simulation software, CarMaker.