There is a payoff in Littlefinger dying at the hands of Sansa (and Bran). His interest in Sansa was his only irrational weakness in the show and his pursuit of her brought him down. Just like Neds honour, Robb marrying for love, Stannis being obsessed with becoming king at all costs, Tywins hatred for Tyrion etc. Just because LF was smarter than these characters in his own way, doesn't mean he should he impervious to his weaknesses.
As for Varys, his role in the story is as a pro-Targaryen conspirator. He was forced to leave Kings Landing and surrender his influence because he freed Tyrion so all that's left is to join Daenerys. It's a shame he becomes a bit part character late on but the same has happened to many other characters. At least they didn't just kill him off because they don't need him anymore like they've done with other characters (e.g. Barristan).