Reference for Chullin 261:37

(ויקרא יט, י) וכרמך לא תעולל ופרט כרמך לא תלקט וכתיב

With regard to the Poorman's Tithe which is distributed in the house, the owner has the benefit of disposal, and it is exacted even from the poorest in Israel.