I personally love the Crewmax tundras.
I've talked to a number of folks who have them, and they all said roughly the same thing: they do just fine on dirt roads and mild trails, but they wouldn't take them up anything really rough.
To me, it comes down to what you are willing to do. I have a feeling the Tundra's are plenty capable of doing mild to moderate trails, but the question is, how much damage are you willing to risk in a 40-50k truck?
I have thought about one as a camping truck and hunting truck, with the ability to tow a crawler around for actual wheeling. Alas, like you, its a daydream, as I have no room for all that.
I hope you do get one, then I can live vicariously through you! LOL