Legendary difficulty is definitely part of it, but you're not wrong that vanilla Destruction feels gutted until you grab Augmented. That said, there's a few things that make it actually playable early on.
First - and this is the big one - you need to abuse the hell out of crowd control. Frost isn't about damage, it's about slowing enemies down so they can't close the gap. Pair it with a perk that staggers on hit (there's one in the Destruction tree) and suddenly you're locking down melee fighters instead of just tickling them. On Legendary this matters way more than raw numbers.
Second, gear matters. Get yourself some magicka regen gear early - even basic enchanted robes from Dragonsreach or the College will help you cast more frequently. You're not meant to one-shot anything, you're meant to cast constantly. More casts = more crowd control = you stay alive.
Third, don't play pure Destruction. Grab a dagger or sword as backup and use Destruction to control, then swap for the kill. Or go full mage and grab Conjuration for a meat shield - yeah it feels like cheating but it's literally how the game balances mage difficulty. One conjured dremora lord trivializes early game, and honestly if that's what it takes to not hate yourself, do it.
Honestly the real answer is that vanilla Destruction is just... not great until mid-levels. If you want it to feel good from level 1, Ordinator (if you're on PC) completely overhauls the tree and makes early Destruction actually fun. Otherwise you're gonna have a rough 15-20 levels no matter what you do.