Yeah, that's the right call. The other thing that bites you on Legendary specifically: even a dismissed follower sitting outside can sometimes aggro stuff if a patrol wanders too close, but at least they're not actively swinging at your pickpocket target mid-lockpick. With a summon, you've got zero control - it just exists and does summon things.
If you're dead set on keeping conjuration in your build though, ngl there's a middle ground nobody mentions. Grab the perk that lets you summon two atronachs at once (or dremoras, whatever), but use them as a planned distraction for specific fights you know are coming - like, you scout ahead, see guards clustered in a room, summon before you enter, let them whale on each other while you slip past to the objective. It's clunky but it works. Just don't expect it to replace a real follower for general dungeon crawling.
Honestly though, if you're on PC and have MO2 set up, there are mods that let you toggle follower aggression or make summons respect certain zones, but I'm guessing you're on console based on the setup. Vanilla Skyrim just wasn't designed for summons to play nice with stealth, so the previous reply nailed it - use them after you commit, not before.