Staying silent was the least-worst option for those who weren't interested in WW2 refugees, which isn't noble.
However, it is understandable.

It's akin to the nonracist white population in the South who didn't vocally oppose racism. It's not so easy to condemn your own community.
Let's not also forget that there was plenty of overt racist support in the North, as well...

I'm not attempting to make excuses for any group.
However, it's more difficult to openly discuss social problems with someone who doesn't openly state that they understand your social predicament. It's generally the first step to progress. Neutral engagement.