Isolation is surely tough on social creatures like us and some people are more susceptible to ill-effects of it. I'm only one episode in, but I think those who left families with small kids at home are the ones who are going to suffer from it most; I really wouldn't do it in such circumstances. One of the participants said that having a small kid and a pregnant wife at home motivates him, I think he's wrong - on the contrary, it holds him back and makes him homesick.
However, I'd expect that someone who literally has to fight the elements daily to survive, forage food, start and keep fire burning, build shelters, fight wild animals etc. should not be suffering from too many effects of isolation a few days or weeks in...