The only way the birf will come out or go in is with the flat spot on the birf aligned with the knuckle . So if you managed to get it out, then you figured it out. It will only go back in one way, with the flat spot on the birf facing up. Getting the axle all the way in and seated is the bigger challenge, which just takes some finesse and patience.
I suppose that as long as you can get the pre-load on the steering within spec using the ultra-precise fish scale, then the lack of a shim isn't a problem. Most rebuild kits include a few shims in case you need them.
1976 FJ40 - green and mostly stock
1996 FZJ80 - not so stock