Well I haven't heard that cold-like symptoms are associated with hypervitaminosis D, but I will say that 20,000 IU a day is a lot.
The upper limit recommended is 10,000.

I've never gone above 10,000 IU in supplements. However, there are times when I'm either sick or trying to not get sick that I've gone up to 10,000 IU supplements, and had positive effects...a cold kept more subclinical or shortened time of illness.
I don't take that daily when I'm feeling fine, though, because frankly I'm too lazy.

As for your story about getting a sinus infection and the flu, I'm sure that had more to do with partying than being outside...or just a coincidence. I don't think it's possible to get vitamin D toxicity from sun exposure, the body has feedback mechanisms that regulate endogenous vitamin D production.