Unarmed actually scales better than people think, but you're hitting on something real - it does have a soft cap that makes it trickier on higher difficulties. The good news is your Restoration combo is probably doing more work than you realize.
So the perks thing: yeah, they're in One-Handed, which is weird and nobody talks about it. The one that boosts your fist damage (I think it's called Fists of Steel?) is the cornerstone, and then you've got the basic damage multipliers further down the tree. Enchanted gloves help a ton since they count as weapons for scaling purposes. If you're not already doing it, stacking those with smithing (tempering your gear) makes a real difference.
Where unarmed gets weird on Legendary is that your damage doesn't keep pace with enemy scaling the way a proper weapon does. Like, a sword at level 50 feels way more viable than unarmed at 50, even with perks. But here's the thing - you're not actually playing pure unarmed if Restoration is your real damage mitigation. You're playing a tank-healer, which is a totally different (and honestly more viable) thing. Restoration scaling is actually solid, especially if you're taking the perks that reduce spell cost.
The real question is whether you're comfortable with longer fights. Unarmed will work, but you'll be punching things longer than a melee character would. That's not a bug, it's just how it is. If you don't mind that pacing, you're probably fine. If you get bored waiting for health bars to tick down, Legendary might feel grindy.
What difficulty are you actually planning to run?
