You certainly don't "need" power brakes. Remember, power brakes do nothing to decrease your stopping distance. They just make the pedal easier to push. If you can stop your car with manual brakes and don't have a problem with the brake effort, you don't need power brakes.
I HATE the power brakes on my Challenger. I didn't realize how awful they are until I started driving my Duster, which has manual brakes. Even when I was running the stock manual master cylinder and 11.75" disks it was a much better braking feel than the power brakes on my Challenger, running the same size disks on both. I upgraded my Duster to a set of Dr. Diff's 13" cobra brakes and a 15/16" bore modern master cylinder, and its by far a better experience. I really should just buy another master cylinder and conversion plate from Cass and ditch the power brakes on my Challenger too.
Oh, and A LOT more room to work without that big old booster in the way too. Probably even more helpful on a Hemi.