Sample efficiency is a crucial problem in Reinforcement Learning, especially when tackling environments with sparse reward signals that make convergence and learning cumbersome. In this work, a novel method is developed that combines Rapidly Exploring Random Trees with Reinforcement Learning to mitigate the inefficiency of the trial-and-error-based experience-gathering concept through the systematic exploration of the state space. The combined approach eliminates the redundancy in irrelevant training samples. Consequently, the pivotal training signals, despite their sparsity, can be further exposed to support the learning process. Experiments are made on several OpenAI gym environments to demonstrate that the proposed method does not have any context-dependent components, and the results show that it can outperform the classic trial-and-error-based training approach.