Learning to drive requires obtaining a skill that can be transferred to vehicles with different dynamics. After a short time of adaptation, humans are able to maneuver completely different vehicles. General Reinforcement Learning-based (RL) agents fell short in such scenarios since they encompass the environment dynamics in their control. If this environment changes, e.g., the expected next state differs, their performance decreases. In this work, we address this phenomenon and extend a conventional RL algorithm with modules that support adaptation to diverse environmental dynamics. We show how our method compares against baselines and what benefits it holds over them. With our results, the paper opens a line of work on making RL agents more general by preserving the behavior and decomposing it from the realizing control while keeping the method in the model-free realm.