I ended up purchasing some bushing installation/removal tools, they are pretty much short stepped sockets, you insert one and a handle and drive the bushing in or out. I kept looking at the bushing thinking there was some trick to removing it, like the hole for the retaining ring. I'd recommend against removing it unless yours is badly damaged...it's a lot of work.
Reassembly went a little smoother but said retaining ring was a pain to get back in. I really had to put a lot of force on the cap to compress the spring to the point that the ring would snap in. The bushing was a little tight too so I cleaned up the inside and now the shaft turns smooth. I'm not sure I have the right o ring on the pressure union either, the one I put on there had a lot smaller ID, but it stretched to fit.
I also managed to damage the pulley removing it, so while I'm waiting for a replacement, I'll probably hook the hoses back up, fill it, and spin it with a drill for a bit to make sure I've got all the leaks fixed. I'd hate to put it back on the car and then just have to pull it off again