Yeah, you're hitting on something real - Alteration feels bad early because you're comparing it to just... wearing armor. But the issue is you're thinking about it backwards. Mage Armor isn't a replacement for robes, it's a replacement for heavy armor in a mage build.
The actual play pattern is: grab the base Mage Armor perk, level Alteration to like 40-50 doing the transmute ore spell (or just casting Oakflesh in dungeons), then grab the second rank. By that point you're sitting at maybe +50% armor from the spell itself, and suddenly you're tanky enough to not die instantly while casting. You're not unkillable - you still need positioning - but you're not glass cannon anymore.
The thing people don't mention: Alteration gets way better if you're also using Paralysis or other control spells. You're not trying to tank damage like a warrior; you're trying to buy time to cast the spell that makes enemies stop moving. Paralysis especially trivializes a lot of mid-game content if you can land it, and Alteration's perks support that playstyle better than you'd think.
On Switch though... yeah, you're stuck with vanilla perks, which are honestly underwhelming compared to what mods like Ordinator do for the tree. Vanilla Alteration is genuinely a bit of a trap. If you want to make it work, commit to a transmutation + control spell build and accept that it peaks around mid-game rather than scaling into late-game like Destruction does. It's doable, just not as flashy.