Just solved my problem, just as you guys said, it was a bad ground to the switch.
I have a standard gauge set-up so I was able to pull out that part and put a separate ground wire from the top screw on the wiper switch to the dash.
I'm glad it was an easy fix, and didn't need to take 1/2 the dash apart.
Thanks for everyone's help.