Vanilla Destruction does scale pretty badly, yeah. The issue is that spell damage doesn't scale with your Destruction skill past a certain point the way weapon damage scales with your weapon skill, so you're basically stuck on a damage curve that flattens out hard once enemies get real HP pools.
That said, there's actually a legit build path that works: stack the perks that aren't just raw damage. The one that staggers enemies when you land crit hits, the one that makes your spells cost less, the Quiet Casting line if you're sneaking - those actually matter way more than you'd think because they let you control fights instead of just hoping your numbers are big enough. And grab the perk that boosts damage when you dual-cast; dual-casting is genuinely underrated because the damage multiplier is solid and it triggers impact effects more reliably.
The real issue is that your Dremora Lord is just... better designed for scaling than Destruction magic is. If you want to keep spells competitive without mods, you basically have to commit to making them a control tool (stagger, knockdown, slowing) rather than your primary damage source. It's not ideal, but it works.
If you were on PC I'd just tell you to grab Apocalypse or Ordinator and call it a day - both add perks that actually make Destruction viable at endgame - but yeah, vanilla Adept you're fighting an uphill battle. Not impossible, just tedious. Your summons carrying is kind of the intended balance, even if it feels lame.
