Speech is genuinely weak in vanilla Skyrim, so you're not crazy for skipping it. But you're also missing some stuff that actually does matter.
The perks that move the needle: the ones that let you sell stolen goods to any merchant (can't remember the exact name, but it's mid-tree) and the perk that boosts prices you get when selling / reduces prices when buying. Those two alone save you from having to fence everything through the Thieves Guild or Delvin/Vex, which is quality of life. Not a game-changer, but real.
The persuasion / intimidation stuff is honestly underwhelming. Yeah, those perks make dialogue checks easier, but you can usually just reload a save or pickpocket a solution instead. On Switch you're stuck with reloads though, so it might feel slightly less pointless to you than it does to me on PC where I can just... not care and move on.
Where Speech gets interesting is if you're doing a no-combat or roleplay build - merchant, diplomat, con artist type. Then those perks feel purposeful. Otherwise, you're right that the gold perks are overkill by mid-game. By level 24 you're probably already swimming in it.
If you want Speech to feel less invisible, the modding community (PC only, sorry) has overhauls that make the tree actually do things - better dialogue outcomes, real mechanical bonuses, stuff like that. But vanilla? Dump it if you want. Destruction will serve you way better on Switch.