First your car looks very nice, it will only go up in value.
Since originality is not a concern I would yank the small block and sell it local or E-bay and drop in a 440.
Frankly, the Mopar crate engines, including the Hemi have a very bad reputation for major build quality issues.
I have heard your exact complaints from many people using the 380/360 crate engine.
Most people who buy the crate Hemi pull it completely apart and mic every piece as there have been a lot of major issues found in them also.
This is not unique to Mopar only, as I've seen first hand major quality issues with GM's 502 big block engines.
A fresh, mildly built 440 can easily put out 500 plus horsepower on pump gas for VERY little money.
I have a 440 in my 73 Challenger that has a completely bone stock bottom end, it was taken from a 43k original mile 1966 New Yorker.
I checked compression, pulled the heads and checked the bores, dropped the pan and checked the rod and main bearings... everything checked out perfect so I didn't touch them.
I very slightly worked the original heads, standard MP .509 purple shaft hydraulic cam, Torker II intake, Holley 750 duoble pumper, Hemi 4 speed and 4.10 gear.
The car runs low 12's in the 1/4 in full street trim and exhaust, runs on high test pump gas and could easily get into the 11's.
I spank my buddie's LS7 powered 70 SS Chevelle on a monthly basis
The standard response to a small block person complaining about lack of performance is step up to a big block.
A big block Mopar engine will drop right into your car with very little changes needed and since your car's original number matching engine is not in the picture, a big block will actually add to your car's value.
ps. are you really looking for those big 73 Challenger bumper guards?