Illusion does hit a wall, yeah - you're not crazy. The issue is that enemy magic resist scales way faster than spell potency does, and most illusion spells don't get the damage scaling that destruction spells do.
The perks that actually matter: grab the one that makes spells work on higher-level enemies (there's one in the illusion tree that extends effectiveness), and honestly the bigger fix is Quiet Casting so you can actually land spells without breaking invisibility. That's where illusion shines at higher levels - not as your main damage, but as setup for backstabs while stuff is confused or fighting each other.
If you want to keep the sneaky angle without pivoting entirely, consider mixing in some elemental destruction - frost especially synergizes with sneaking since it slows targets. You don't need to go full mage, just enough to handle what illusion can't. Alternatively, conjuration isn't a bad pivot since it scales better, but I get the appeal of pure illusion.
On Adept at level 31 though, you might just need to accept that enemies are tougher now and illusion is better as a utility tool than your primary offense. The build still works, just differently than it did at 15.
