We propose RMASBench, a new benchmarking tool based on the RoboCup Rescue Agent simulation system, to easily compare coordination approaches in a dynamic rescue scenario. In particular, we offer simple interfaces to plug-in coordination algorithms without the need for implementing and tuning low-level agents behaviors. Moreover, we add to the realism of the simulation by providing a large scale crowd simulator, which exploits GPUs parallel architecture, to simulate the behavior of thousands of agents in real time. Finally, we focus on a specific coordination problem where fire fighters must combat fires and prevent them from spreading across the city. We formalize this problem as a Distributed Constraint Optimization Problem and we compare two state-of-the art solution techniques: DSA and MaxSum. We perform an extensive empirical evaluation of such techniques considering several standard measures for performance (e.g. damages to buildings) and coordination overhead (e.g., message exchanged and non concurrent constraint checks). Our results provide interesting insights on limitations and benefits of DSA and MaxSum in our rescue scenario and demonstrate that RMASBench offers powerful tools to compare coordination algorithms in a dynamic environment.