Computational and sample efficiency remains a major challenge for Diffusion-Based Reinforcement Learning (RL) algorithms. With this project, we want to investigate a novel Model-Based algorithm employing guided diffusion methods to encourage efficient exploration. The first step will be to implement a naive solution based on the value function learned by the RL agent, to steer the diffusion generated samples towards trajectories with a high expected return. While this is approach is appealing for its simplicity, its impact is limited to the test time performance: during the learning phase, the diffusion model does not take into account the underlying structure of decision problem and the RL algorithm that intends to solve it. We would like to follow the work of Value-Aware Model-Based RL to implement a new loss for the diffusion model that takes into account the agent's value function.