At first I thought I might join the stormcloaks, given the imperials tried to execute me at the start of the game, but when I explored Windhelm, and noticed Ulfric's racist tendencies against the Dunmer and Argonians (and apparently any other non-Nord race), not to mention walking in on Galmar suggesting Jarl Balgruuf be killed, I just up and walked back out. I still visit Windhelm now and then, but I won't be joining the Stormcloaks anytime soon.
I also looked into Solitude and the Empire and talked with them. Elisif seemed friendly enough, and General Tullius does seem to at least try to tolerate Nord customs and culture, even having a Nord for a right-hand woman. However, they too have their flaws. For one thing, the Empire these days is just a puppet for the Thalmor, and I refuse to serve anyone willing to side with them, or to be a puppet to them.
But here's the real kicker: It doesn't matter what side you chose. Choosing a side at all is playing right into the Thalmor's hands. It is revealed in the Thalmor Embassy questline (assuming you look into the details of the Thalmor records) that the Thalmor want this civil war to keep going, and are willing to do anything to extend it's duration. While the Empire and Stormcloaks waste resources and kill each other, the real villains just sit there and laugh, with Skyrim under their thumb.