Pump-probe spectroscopy is a versatile technique to explore ultrafast dynamics on the femtosecond timescale. Here the authors report a pump-probe experiment and quantum modeling combined study revealing dynamics of collective polaritonic states that are formed between a molecular photoswitch and plasmonic nanoantennas.