Your buddy's sword is probably just hitting harder because smithing scales way better in the early-to-mid game, but that doesn't mean you're doing something wrong—you're just playing a slower build. Frost mage legitimately takes longer to ramp up.
Here's the thing: enchanting gear for Destruction gives you a flat damage boost (like +15% frost damage or whatever), but it doesn't stack the way smithing does. Your friend's daedric sword gets better from both the base weapon being inherently strong AND the smithing perks that improve it further. Destruction spells don't have a "perks that make your base spell stronger" system in the same way. You're relying on spell perks (like the ones that boost frost damage and reduce casting cost) plus gear enchants, and that combo just doesn't match a well-crafted melee weapon until you get deeper into the tree.
The real fix is making sure you've grabbed the right perks. The perk that reduces frost spell cost is huge because it lets you cast more, and there's one that staggers enemies with frost spells—that's a game-changer for survivability. Also, are you using the Mage standing stone? That helps early on. And ngl, if you're level 34 and still struggling, you might just be underleveled in Destruction itself—the skill level matters a lot for spell effectiveness.
Conjuration is easier to make work early, yeah, but frost mage absolutely comes online eventually. It just takes patience and the right perk path. Don't bail yet.