My initial concern is the statement, "No filler or any of that": If the shop told you that either they don't know what they are talking about or are trying to impress you, both of which are warning signs to me. When body panels are replaced, or patch panels installed, a certain amount of glazing with either some sort of putty, or leading (rare now these days) in the least will be required. Thats how body and patch weld seams are smoothed out so they don't show when painted.
Frankly, I think you should shop around for at least three different quotes from reputable body shops: I believe the quote you got was not only high (especially with not being painted/finished off), but I have also found that over the last few years allot of the so called "Hot Rod/Specialty resto" shops. tend to really up their quotes only because of the recent resto/reality car repair shows.
Body work is body work, period. Take a quality body shop, and a quality "Hot Rod/Resto shop", in a high end aerea (Like LA as mentioned) and their work techniques and materials will most likely be the same, but so called "Speciality shops" will most likely charge more
It is never advisable to drive around with either just a primer or sealer. I don't think any reliable shop would tell you it is. While its understandable when a budget is at stake, road grime and contaniments will impregnate the surface over the time, and depending on what "primer" was sprayed on it, I have seen light surface rust appear in patches. And what about the inside of patch welds. Also think of chips that might come up over time. If you store it and don't drive it, that might over come these potential issues
Last, when you are ready to finally paint the panel or patches, some body shops will not paint over someone else's body work, especially if you come in with a primer or sealer in place thats been there for awhile. So you might have to come back to the same exact place that did the original work. Then consider trying to match the panel color with the rest of the bodies existing color
All that said, replacing what you just mentioned will not be cheap, but at least shop the market with quality body repair shops will give you an idea of what is fair in price
Also, ask to see past like work that the shop has done before. Most keep photo albums of past projects. If no photos, have them let you talk to past customers