My office is planning to have an intra-office software-based intercom system. As USB phones are costly, is it possible to do them with analog phones (connecting to RJ-11) instead? The phones won't have any connection to the PSTN network. Also, what software would you recommend?
If it's a single physical location, the cheapest and simplest solution would be to use a traditional analogue PBX. They all support internal calling and don't need to be connected to PSTN.
If you must use a software solution then consider Asterisk - but you'll need to buy some FXO hardware to terminate your handset connections.
I'd recommend looking at a Hosted SIP provider. These services typically support internal calling at no cost, so the only cost would be the monthlies; which could be as low as $40 a month. Best part is you have no hardware to support. A USB headset and softphone client is all that you'd need. You may even decide it's a better option for whatever your existing phone/PBX situation is.
I moved our company to a Hosted SIP option about 2 years ago and haven't looked back.
Alternatively, you could even use Gmail to make voice calls, but it would mean other users would have to be logged into their Gmail account for it to work. This may or may not be feasible for you.