I'm not quite sure I understand your question but I believe you are saying that you have two web servers that can both be accessed from the outside world. Your IIS (Windows) server is currently serving www.example.com on port 80. You want to set up your second, Debian server to serve redmine.example.com on port 80. When you say, "They are both in the same network and have single public ip address." does this mean that each server has its own, separate IP, or do you have one IP address that routs to your LAN's router?
If each server has its own IP address, you simply need to update your DNS A-record to reflect your Debian server's IP address at port 80 (e.i. 1.1.1.1:80) (port 80 is default but some DNS servers can get confused it you don't clarify this.)
If you only have one IP address, you will need to forward some outside port to port 80 at your Debian server's IP address.
We do this by first coming up with some outside port other than port 80 (because port 80 is already forwarded to your IIS server.). You can choose any port that you wish so long as it does not conflict with anything (i.e. choosing port 8080 would not be a good idea). When forced to do this, I usually take the last block of the LAN IP address and combine it with the real-world service port. So lets say your Debian's LAN IP is 10.27.1.102 and I wanted to use SSH (port 22) I would create the outside-world port refrence: 10222. You don't have to do this, just something I always do.
Secondly we need to configure the NAT setting on your router (sometimes refereed to as "Port Forwarding" or "Port Triggering" on some networking gear.) Create a new entry. The "Out Side Port" is the port that you just came up with (I created 10222). The "Internal Address" is the LAN IP address of your Debian server. The "Internal Port" is port 80 (per your original post) Make sure to save this.
Now you just need to update the DNS A-record for redmine.example.com. You should set it to your external/public IP address at the port that you had to come up with (Again, I chose port 10222 in order to explain this) So if your public address is: 1.1.1.1, then your record should be 1.1.1.1:10222.
Give the DNS some time to update and you should be good!
Sorry if I wrote a book, I am very enthusiastic about network administration, it's what I do best :)
If you have any questions or you would like me to explain why this works, feel free to ask. Or just wiki "port forwarding" haha