Ok there is a place called Deckers that we could meet. It's basically a fishing shop in the middle of nowhere. See Deckers Colorado in a google search. There is no fuel there, so be topped before you leave any main roads. We could also meet at Crossroads on 285. That's where you take the road south to head down into that area.
My only concern is that it's been years since I've been there. I'm confident in getting us there - just not sure I'll remember the trails.