Primer is the sealer for your paint, that's what its for, and it does a better job of sealing than the topcoat does. So, for protecting the metal it'll be fine as long as you put down a good, solid coat of primer (or two, or more depending on your primer).
But, it doesn't hold up to UV light, so it will deteriorate if it's outside on its own too long. And then there's the contamination issue. Topcoat is more susceptible to contaminants than primer, and if you've been driving your primered car around for awhile it will be covered in contaminants, which kinda defeats the purpose of the primer.
Not saying you can't do it, but you'll have some extra work. Short term, it'll need a serious bath (think dish soap, no waxes), then a wipe down with wax and grease remover. Long term, you'll need to do that, then consider another coat of primer. If the primer has started to break down it won't do it's job, and the longer it's on there the harder it'll be to clean, so a fresh coat of primer will help ensure your topcoat actually sticks and comes out contaminant free.