De novo drug design is the design of novel chemical entities that fit certain constraints. De novo drug design is a major challenge in pharmacology and a new focus in AI for science research. There has been recent success in using reinforcement learning and generative models for de novo drug design. This work aims to investigate further and improve the use of reinforcement learning for de novo drug design to better generate novel chemical entities that fit certain constraints. This could ultimately increase the productivity of de novo drug design to search the chemical space for new drugs more effectively.