I'm 6' and have driven a fair amount with the top up. It's a different experience.....but not traumatic by any stretch. It takes some adjustment. I put the top up when it's below 45-degrees. Will the car heat up like a modern car?....no. But at least you won't feel the 45-degree wind hitting your face at 70mph.
Visibility is unique. It's odd being taller than the side windows. You need to bend down a bit to see out....which I actually like. It definitely provides a bit of anonymity, which I actually like. I don't have a passenger side mirror, and really haven't had an issue.....I just look around before changing lanes.
I also haven't found street lights being difficult to see when the top is up.
I like to put the top up when it is extremely hot and I don't feel like being burnt to a crisp. With the side windows off, there's enough air blowing through the cabin. I wish I had a zip out window in the back, that would be killer.
My biggest issue with the top up? I can't get out of the car in my tight garage. With it off, I stand up and step out. With the top up, I'm sure folks around get a good laugh at watching my "roll" out of the car.
Anyway, it's not that terrible as long as you adapt.
-Kevin