I'd be surprized if a slightly off center aluminum wheel spacer would create enough of an imbalance that it'd cause vibration. I think it's more likely that the wheel spacer is allowing the wheel to mount up off center, or out of square. Is the wheel located by the studs, or by the hub in the center? I wouldn't run a plastic experiment at high speed as I bet the lug nuts won't stay tight.
How about adding some rings (sleeves) to the existing spacers to tighten up the bolt holes?
'03 KTM 450EXC
"If you know what you're doing, then you aren't learning anything." - A. Lincoln