There's quite a few options for primitive camping along Animas Forks Road (dirt track) not long after you leave Silverton heading NE toward Animas Forks. I was there last year for Fathers Day weekend, and we found a spot a few miles out of Silverton on the right side of the road (there are several spurs and side roads).
Those pass roads around Ouray and Silverton are usually pretty busy with traffic that time a year, so I wouldn't worry too much about traveling solo. Black Bear is probably the worst of them, only because of the steep section and falls on the Telluride side.
1976 FJ40 - green and mostly stock
1996 FZJ80 - not so stock