In recent years, closed-loop adaptive deep brain stimulation (aDBS) for Parkinson's disease (PD) has gained focus in the research community, due to promising proof-of-concept studies showing its suitability for improving DBS therapy and ameliorating related side effects.The main challenges faced in the aDBS control problem is the presence of non-stationary/non-linear dynamics and the heterogeneity of PD's phenotype, making the exploration of data-driven dynamics-aware control algorithms a promising research direction. However, due to the severe safety constraints related to working with patients, aDBS is a sensitive research field that requires surrogate development platforms with growing complexity, as novel control algorithms are validated.With our current contribution, we propose the characterization and categorization of non-stationary dynamics found in the aDBS problem. We show how knowledge about these dynamics can be embedded in a surrogate simulation environment, which has been designed to support early development stages of aDBS control strategies, specifically those based on reinforcement learning (RL) algorithms. Finally, we present a comparison of representative RL methods designed to cope with the type of non-stationary dynamics found in aDBS.To allow reproducibility and encourage adoption of our approach, the source code of the developed methods and simulation environment are made available online.