Tosefta for Megillah 56:5

[With respect to] synagogues, one may not behave frivolously (lit. "light-headed") in them. One may not bring an animal inside them in summer due to the heat, in winter due to the cold, or in the rainy season due to the rains, nor may one eat in them, nor drink in them, nor sleep in them, nor wander around in them, nor adorn oneself in them. Rather, one may read [the Torah], or study the oral law, or give sermons in them. Public eulogies may be eulogized in them. Said Rabbi Yehuda, in what case does this apply? When they are occupied [by congregants]. But when they are in a state of ruin, we leave them alone and let grass grow up in them, due to the anguish of the soul [that they cause to those who see them in disrepair].