First off, what grit were you using before you primed it? Did you sand between coats? Typically I'm up to 220 before I primer, paint will tend to shrink over time, so sand scratches that were "gone" when you primered can re-appear down the road, so I try to have it fairly smooth even before primer.
As for now, I'd just wash it off with soap and water (not car wash soap, a lot of these contain waxes), you're going to be wet sanding once you get above 180 anyway right? If its really greasy, I might use a wax and grease remover (from the paint shop) on it. I usually start with 120/180 after my first prime for a quick knock down, then move up to 220 before the second coat of primer. I might start with 220 after the second coat of primer, but after another quick knock down I go to 320/360. Depending on how things go I usually finish after 2-3 coats of primer, and will finish the last coat with 400-600 before going to topcoat.