It's the fact he travelled 250+ miles to do it.
If you're so ill, how have you managed such a long trip with a kid and an ill wife in the car too, presumably without stopping?
Guidelines at the time were not to visit family if you have symptoms. He had symptoms. His wife actually had the virus.
So not only did they travel with the virus (safeguarding the kid is no excuse considering, by their own account, they stayed in a separate part of the house), they also put their own family members at risk, and with his parents potentially being in vulnerable categories too.
Now, I know this bit is speculation, but I find it hard to believe and very much doubt he's travelled that far only to go straight in to another part of the house and not seen his parents or sisters/nieces etc.
Also, his own credibility has gone out the window if this 2nd trip story is true. Where he's gone back up to Durham and ended up going sightseeing or walking in the woods. Especially given we were still under strict lockdown rules at the time (mid-April). So he's gone Durham, come back to London and the gone back up to Durham. Again it smacks of hypocrisy and double standards. Calderwood, the health minister in Scotland, was sacked for travelling in lockdown when rules were explicitly stay at home. Why did Cummings go back up to Durham for second time?
The other point to raise is how is it that a man in his position has his closest care providers 250 odd miles away? It later came out in the day that his sister-in-law lives in London too and his own aide lives 2 streets away. Yes there may be other circumstances which prevent them from looking after his kid but having said that, in an emergency surely they'd be easier to accommodate with as oppose to risking the long journey up north?!
I think if you don't see what he's done wrong it's a bit worrying...