This is purely on the basis that I had a Bailey single-piston valve in mine, and it would often give high idle... and then sometimes normal. - That doesn't mean it's your problem; it's just one more thing to eliminate.
I mean the smaller vacuum/boost line going to the dump valve (not the big one it's probably connected to the plenum by). Just squish it with some pliers then rev the car a bit (as if you've been drving). - Because the vacuum on mine was pulling the valve open at idle, it was letting air in (thus the idle speed went up); as soon as I pinched the line, the revs dropped.
After that, I replaced it with an HKS SQV (the old one that looks like a spaceship) and haven't had any more problems with the idle going up and down like that.
Taking the idle screw out and spraying throttle body cleaner inside also helped bring the base idle speed down.