but because its all numbers matching would I be stupid to rip out the 318 and hide it in the corner?
I don't think so. A crate 360 would go right in, and you could always put the #'s 318 back in if you decided to show/sell it. Best of both worlds really. Or build a 408 stroker. Plenty of power, and no major changes to the car to go back to #'s matching.
On the other hand, it really depends on what you're going to do with the car. A streetable 350 hp 318 isn't all that hard to accomplish, and if your plan is just to cruise/drive it they're plenty fun. Hell I think the stock-ish rebuilt 318 in my Challenger is fun, and I still only have 3.23 rear gears. It can't put out more than 250 hp. Obviously not nearly the same as have a high hp big block, but most of the high hp big block cars I see get driven to shows like grandma going to the grocery store. Why bother? So you can say "yeah its got a big block in it" without mentioning that its never been over the posted speed limit by more than 5 mph and never seen the track? A 350 hp 318 with a set of 3.55's or 3.73's would be plenty of fun.
But maybe I'm just biased. I like small blocks, I think the 318 is horribly underrated (they can easily do battle with the cookie-cutter 350 Chevy crowd), and with the advent of the small block stroker kits (390, 408, 416 etc) I think that small blocks are a better choice for a well balanced, good handling, frequently driven car. But you know what they say about opinions...