This is intended behavior. Removing a GPO doesn't magically revert the settings made in the GPO.
The way I'd handle this to create a GPO with the settings you want to the machine to have once it's unjoined, apply that GPO to the machine, force a group policy update and then unjoin it from the domain.
Or just image the machine to a state before it's domain joined; both appraoches are pretty easy and low on effort.
The other, much suckier way of handling this is to hunt down the registry keys set by the GPOs and manually change them. This is time-consuming and dangerous, however, and I wouldn't do it unless I had a gun to my head.
Now, having said that, I'd just like to point out that I can't imagine a sane use-case for joining a machine to a domain just to get Windows Updates, and then unjoining it, so I'd say there's something very not right with that whole process, and you might be better suited fixing that mess than dealing with the GPOs, which is just a symptom of this silly procedure.