Commentary for Berakhot 62:19

Hence it is deduced that it is forbidden to sit within four cubits of one engaged in prayer – whether in front of them, behind them or on the side of them. The Teshuvat Geonim explain that that this is only if one is sitting idly, but if they are engaged in the recitation of Shema or something similar, even if they are sitting it is okay.

The ground for this inference is not clear. The explanation offered by the commentators (see Tosafot) is as follows: The word "by thee" עמכה (page:209) is spelt with a redundant letter ה (page:209) which has the numerical value of five. Hence it is taken to mean "I was within five [cubits] of thee," because one is not allowed to sit within four cubits of one who prays.