פירוש על קידושין 67:1

Sukkah and lulav can be performed only on Sukkot, and the shofar only on Rosh Hashanah.
Tzitzit is not worn at night, and tefillin are not worn on Shabbat or holidays. Thus these too have a set time.