I lived on a dirt road up until I joined the military. My parents still live there and it's about a 1/2 mile to the highway. The good thing is there is a car wash right at the hardball so it got alot of quarters from us. My neighbors had old Mercs and some others were car enthusiast. When another neighbor was coming down the road they would slow down so as not to dust you out or throw rocks.
If I ever saw someone barreling down the road I would just pull into the middle and they would slow down and be forced to be considerate, hey it was MY road. Most people were considerate though.
After going through all that though, a dirt road is a deal breaker for me when we finally purchase a house in the future, as is a garage.
Must haves in a prospective house are in this order:
a) safe neighborhood for my daughter
b) good scool district
c) necessary infrastructure for a car guy (pavement and garage
Without those it's a no deal!