Your frost stagger perk is good, but you're relying on it too hard. The problem is you need to stop enemies before they close distance, not just slow them down once they're already swinging at you.
A few things: First, make sure you're actually casting while moving backward. Sounds obvious, but a lot of new mages stand still and panic-cast. Second, Mage Armor is helping your survivability, but it's not a replacement for actual crowd control. You need something that actually stops enemies—not stagger, stop. That means either a frost spell with the perk that freezes them solid (there's one in the frost tree), or honestly, a single Mayhem/Fury spell cast early so enemies fight each other while you kite.
Third, and this might sound weird, but you're probably underleveled for Adept at 32 if you're pure destruction. Your spell damage scales slower than weapon damage in vanilla Skyrim. If you're on PC, grab Ordinator—it completely fixes mage scaling and makes pure destruction actually viable without feeling like you're cheesing. The perks actually reward positioning and spell choice instead of just "cast the same fireball 50 times."
If you're on console, you're a bit stuck with vanilla balance, so honestly dual-casting your frost spell with the stagger perk + backing away is your best bet. And maybe grab the Atronach Stone temporarily if you're getting oneshot—it's not a permanent fix but it buys you breathing room while you figure out your rotation.
What difficulty are you actually on, and are you on PC or console?