Recent methods applying deep learning to model neural activity often rely on “black-box” approaches that lack an interpretable connection between neural activity and network parameters. Here, building on advances in interpretable AI, Tolooshams et al. propose a new method, deconvolutional unrolled neural learning (DUNL), and apply it to several datasets.