Yeah, that's the honest take. I'd just add that Oblivion's fall damage is way more punishing than Skyrim's, so the previous reply nails it - it's not about speed, it's about not dying to your own mistakes in a dungeon.
That said, there's a middle ground nobody mentions: you don't need to dump points into Acrobatics like it's a major skill. Just letting it level passively while you jump around (which you'll do anyway exploring) gets you to like 40-50 without trying, and that's usually enough to shrug off most falls. You're not sacrificing anything by letting it tag along.
Where I'd disagree slightly is that it matters more if you're playing on a higher difficulty. On Normal? Yeah, skip it, chug a potion. On Oblivion difficulty or with mods that make enemies hit harder? That extra survivability from not eating fall damage frees up your potion slots for actual combat healing. Small thing, but it adds up.
The real trap skill in Oblivion is Athletics, imo - that one actually does feel mandatory because running speed is so slow, but at least it levels while you're just moving around the world. Acrobatics is optional in a way Athletics isn't.