I don't know of too many Mopar guys that wouldn't appreciate a nice Hemi, but if you can keep your car in it's near original state, I think that it gives it a lot more character and is uniquely yours. As you said above, you were already the only Challenger.
When I was a kid in the early 80's and my dad decided on finding a Challenger, I think a lot of the decision was based on the fact that we didn't see too many Challengers out there at the time. Around us and at the many mopar shows we went to back then, it seemed like the Cuda/Challenger ratio about 6 to 1 and I know he wanted something different and today it still is.