Honestly, the performance hit is real but it's not because Apocalypse itself is bloated—it's the particle effects and spell interactions stacking on top of your ENB and Community Shaders. Those two together are already taxing VRAM pretty hard, and Apocalypse just adds more stuff for your GPU to render.
That said, vanilla + Ordinator isn't really the same toolkit. Ordinator gives you way more perk flexibility and power scaling, but the actual spell pool stays vanilla—you're still throwing the same Fireball and Frostbolt you've always had. Apocalypse adds genuinely different spell types: the utility stuff, the weird crowd control options, the spells that do things vanilla destruction just doesn't touch. If you're into experimenting with builds, losing that variety does feel restrictive.
If performance is the real problem though, before you nuke Apocalypse entirely, try turning off the optional particle effects patch (if you're using one) or lowering your ENB's particle quality. Sometimes it's not the mod, it's the combo. I'd also check if you're actually CPU-bottlenecked instead of VRAM-bottlenecked—destruction spam can hammer the CPU pretty hard, especially with 180 mods.
Real talk: Ordinator alone is worth keeping. Apocalypse is the luxury item. If you're dropping frames during actual combat, cut Apocalypse and don't look back. You won't regret it.