What Jeff said. My guess is your booster too. I will only add that I noticed a very slight change in idle speed when my cruiser was a two or three pumper and my booster does not leak AFAIK. After I replaced the master cylinder, adjusted, and bleed, I no longer notice the change in idle. Buy one of those vacuum pump dealies and test your booster. It should hold a vacuum. Also, I dont think the line from the booster to the intake manifold could be clogged since there should be nothing in your intake to clog it but you might as well check.
78 FJ40, 2F, HEI, 4" lift, 33x10.5, rear aussie locker, rust.