sounds like a real pain in the arse,,,,,, from your pics, it looks as if the one side is out of the column a little bit,,,, Not sure but,,,, you may want to aply a small amout of heat to it and using your drift pin gently tap the already pushed out side of your pin back in to hellp free and cut the rust/debris away, then rotate your column and from the drilled and mashed side use your drift pin to push the spring pin back through the column,,,,,,Make sure you cleaned all your drill shavings out first so you can get a good seat on top of the pin to use your punch. Or if it was drivable you could bring it over here and i could help you remove it,, Sometimes a new set of hands and hammers work wonders
Doug