Yeah, that's the honest take. I'd add though - on Switch especially, you're stuck with vanilla perks, so you're dealing with the baseline Illusion tree which is... rough early. No mods to patch the scaling or add better low-level spells.
The thing people miss is that Illusion isn't a damage school, it's a survival school. Fury is bad because you want enemies confused, not just angry at each other - but Calm actually stops combat cold if it lands. Problem is it doesn't land reliably until you've sunk perks into it. So yeah, you're right to feel like it sucks at level 12. It does, relatively speaking.
If you want to make it work sooner without waiting until 20+, honestly just... don't rely on it solo. Pair it with Destruction. Calm an enemy, then nuke their buddy. Or use Muffle to actually sneak instead of treating it like a combat spell - which is what it's for anyway. The spell isn't failing; you're using it in a role it wasn't designed for.
That said, if pure efficiency matters more than variety, stick with Destruction. There's no shame in that. Illusion becomes legitimately powerful later, but "later" is the operative word. Your instinct that it's a trap early game is basically correct.