I have installed php from apt -- How can I remove that version and install an older one?
You can install an older version of any package with Apt - you just have to look up the specific version name that you want. Assuming you're using Debian, you can look up old versions of packages at their site http://www.debian.org/distrib/packages
The format for installing said specific package is like this:
Such that if you want to install the version named 5.3.3-7+squeeze3, you use this:
But this also kind of brings me to the question "Why would you want to do that?" While you might have PHP scripts that require an older version of PHP, there hasn't been an upgrade in a very long time that has broken PHP scripts, in my experience (I run a web server that hosts a several hundred websites). As such, you'd be running a dangerously insecure version of PHP on your server. Are you sure you want to do that?
Get the deb file of the version you need.