Access (and the weather) are the things that kill me. Where I live is 1 hour's drive from the closest available a/c for hire (and then only for brief periods booked well in advance - see note about weather!), 1 1/2 hours to the next closest, 2 hours to the next and 2 1/2 hours to the next. It's a bummer to drive a 3-4 hour round trip only to find that circumstances at the strip are not suitable, regardless of how good the weather forecasters predicted it to be.
Of course, when I finally get in the air, I really enjoy the challenge of being a clean, consistent, in-control pilot. Unfortunately, those things remain a challenge...