Conjuration's scaling issue is legit—vanilla just doesn't give your summons the tools to keep up past mid-game. You're not crazy for hitting a wall.
The perk tree itself is the main culprit. There's that one perk that lets you summon two Dremora at once, which helps, but it doesn't actually boost their stats. They're still running on the same base damage and health as when you summoned them at level 10. Enemy scaling just leaves them behind.
If you want to stay lightweight, honestly your best bet is just respeccing into a hybrid. I know that's not what you want to hear, but pure summoner in vanilla is basically a hard cap at mid-game unless you're also running damage buffs through other schools. Fortify Conjuration gear helps a bit, but it's not enough to carry you solo.
If you're set on staying summoner-focused, grab Apocalypse or something in that vein - EnaiSiaion's mods are solid and don't require a full overhaul like Ordinator does. There's also Wildcat that tweaks enemy AI to be less aggressive toward summons, which indirectly makes them feel less fragile. Not a direct stat fix, but it helps.
Alternatively, Necromage exploit still works if you want to cheese it - that perk that boosts damage against undead applies to raised corpses too, so you can stack some nasty bonuses that way. Bit jank but it does solve the scaling problem.
What difficulty are you running?
