The next time you are asked for directions you could, rather than give a detailed itinerary, simply intone "not all who wander are lost". Of course, people may think you are insane but proceeding thus does have the net advantage of circumventing the gender spatial abilities issue:)