Yeah, that's the reality of it. I've done a couple Switch playthroughs and hit the same wall around the 50-60 hour mark myself. The previous reply nailed the core issue - it's just how the engine handles save bloat on weaker hardware.
One thing I'd add: if you're doing a ton of crafting (smithing, enchanting, alchemy), that can accelerate the problem. Every item you create, every spell you cast, every NPC you kill - it all gets logged in your save file. Not saying don't play the game, just be aware that a character who's crafted 200 iron daggers is gonna tank performance worse than one who mostly just loots and sells. Doesn't mean you have to stop, but it's worth knowing.
Also, if you haven't already, make sure you're not carrying around a bunch of quest items or books you don't need. I know inventory management sounds tedious, but the Switch version is genuinely sensitive to it. Drop stuff at your player home if you're a hoarder.
Real talk though - if combat's getting genuinely sluggish and it's killing your fun, a fresh start with a different build might actually be worth it. You've got 40 hours of experience now so you'll blaze through the early game anyway. But if it's just occasional stuttering in Whiterun and you're still enjoying yourself, you can probably finish this playthrough fine.