Yeah, that previous reply nails it. I'd just add - don't sleep on Paralysis even though it's technically Alteration. Seriously, cast that once and suddenly a dragon or a tough bandit is just... frozen. On Legendary that spell does more work than any pure damage spell will.
Also, if you're not already doing this, stack your debuffs. Frost slows them down, shock interrupts their attacks, then you layer in a summon or a follower to actually finish them while they're vulnerable. That's how destruction actually wins fights on higher difficulties - not by out-damaging enemies, but by controlling the battlefield. A frozen, staggered enemy is way less threatening than a full-health one tanking your Infernos.
One thing I'd push back on though - you don't necessarily need mods or a full hybrid build if you don't want to. I did a pure destruction run on Legendary (granted, not my hardest difficulty) and the key was just accepting that you're not the damage dealer. You're crowd control. Cast Mayhem or Fury to make enemies fight each other, apply crowd control effects with your elemental spells, and let time do the work. It's slower and less flashy, but it works.
That said, if you want destruction to feel powerful and not just functional, yeah, a summon or two takes a lot of pressure off. Conjuration scales way better on Legendary.