Responsa על הוריות 6:17

They responded: "They do not issue a decree on the people that the majority are unable to withstand," (Horayot 3b:6) "and one witness is credible in matters of prohibitions" (Gittin 2b:5) - as long as they are knowledgeable. And all who abstain from something when it is not appropriate to abstain, their actions are considered foolish.