Midrash for Chullin 268:52

הזרוע כנגד היד

The only reason [for returning it] is that they were in the habit [of doing so], but ha they not been of this habit it certainly would have belonged to them.

Sifrei Devarim