הלכה על סוכה 52:17

One forgot, and "used his bed" in tefillin. He should not grab them by the straps, much less by the tefillin themselves, until he has washed his hands. For the hands are "busybodies" and he certainly touched a place that requires hand washing. And when he washes his hands, he should take them off until he has washed off the emission, and then he is permitted to wear them again, for someone who is impure because of a seminal emission is permitted to wear tefillin. And if he sleeps in tefillin and has an emission, he should not grab them by the boxes. Rather, he should grab them by the straps and take them off until he has washed the emission off himself and washed his hands. Our master, the Beit Yosef wrote in subsections 8: "One who goes in to an established meal should remove them and set them on the table until the time for the blessing (grace after meals), and then put them on again. However one who is eating a snack need not remove them." For it is permitted to eat a snack in tefillin. This is according to their custom, for they would go the whole day in their tefillin, but not for us. It seems to me that by us, it is not correct even to eat a casual meal in tefillin.