In the BOOKS, Harry didn't destroy ANY of them. In the movies Harry destroyed ONE of them. As to the Elder Wand it's all by 'defeating' the owner.
Grindelwald defeated whoever. Dumbledore defeated Grindewald. Draco defeated Dumbledore by disarming him. Harry defeated Draco by disarming him.
Voldemort could use the wand because it's just that, a wand. But the wand didn't CHOOSE Voldemort, therefore it wasn't at it's full power. Voldemort thought Snape outright killed Dumbledore, so he killed Snape.