Debit / credit pinpad terminals disconnect from network after 15 mins. Reconnect after one error

Question

We have an HP procurve network and about 20 of the standard debit/credit pinpad terminals everyone is used to seeing at just about every store these days. They connect directly to the LAN and only communicate out to a payment site on SSL/443. No software or server in the middle.

The problem is that the devices usually give a TCP connection failure on the first attempted use. They'll then work fine for an hour straight. But, if allowed to sit idle for 10 - 15 mins (approx), they'll throw the initial error once.

Initially, they were all from a single company and we figured it had something to do with their setup, or the make/model. But recently, we've installed some new devices from a completely different vendor, using different types of pinpads ... and they have the same error.

We've tried static vs. DHCP IP addressing. We've added the outside payment site to a special firewall rule that allows them to exit without the normal threat checks. We've tried them on various Vlans. We've tried conencting them to various types of areas switches. Have even tried a scheduled batch file that pings them (home made stay-alive), every 3 mins. Nothing makes any difference. In terms of network problems, the devices are all conencted to the exact same vlans and areas switches as their nearby cash computers/printers - and we have no problems with anything else. The cash systems run full client/server/database apps and if the same disconenct issue was present for them because the networking in the area was bad, we'd hear about it fast.

Latest theory I'm about to tackle is related to arp cache timeouts, but I'm just getting started.

Would appreciate some assistance ... crazy ideas also welcome.

W.

Answer 1

I have seen a similar issue in the past. My issue was related to my device establishing a connection through a NAT device, and then that connection stayed idle for too long (nothing sent, nothing received). Both ends of the connection had no idea, but the NAT device in the middle decided to close the connection because of the inactivity. Then when traffic was trying to traverse the NAT, the packets were being dropped because the NAT rule no longer existed.

Your devices might be doing something similar. My solution was to use a keep-alive packet between the two devices. It would send a packet every 60 seconds, and this solved my issue (the system has been running for several years now without needing to be touched). Simply pinging either device from the same LAN was not enough to keep the NAT rule in-place. The devices MUST talk to each other regularly.

However, without knowing more about your particular systems it's difficult to say if this applies to you.

Hope this helps.

Answer 2

The problem is that the devices usually give a TCP connection failure on the first attempted use. They'll then work fine for an hour straight. But, if allowed to sit idle for 10 - 15 mins (approx), they'll throw the initial error once.

The first thing I would recommend is to get ahold of a copy of the manual or talk to the vendor to get an explanation of exactly what the error the devices generate means. I've wasted time looking for Layer-3/4 issues when the error actually meant something else. Not all vendors use terminology correctly or consistently.

It sounds like the devices are not sending handling or doing keep-alive correctly. If there is no data traversing over your TCP connection it will be closed eventually. To prevent this one endpoint (or both) can send keep-alive packets to prevent the connection from being terminated. I know this can be done with TCP (Layer-4) and presumably it could also be done with SSL/TLS (Layer-7).

Put a packet sniffer of choice between one of these devices and your infrastructure and record all of its traffic from the time it works until the time it doesn't. Then look through it and find where the device or the server it's connecting to is starting the termination sequence, then see what immediately precedes that. Also take a look at the point in time where the device throws the "TCP connection failure" error. Is it trying to use a connection that it thinks is established but that the server thinks is terminated? Something strange is happening here as well - if the connection isn't established, instead of throwing an error, your credit card device should attempt to create a new one (which apparently happens successfully the second time).

And finally if you are using NAT, consider giving one of these devices a direct, non-NAT connection for testing purposes (again, take a packet capture). NAT can do very strange things to applications or protocols that depend on the End-to-End Princple and do not take the widespread use NAT or other stateful devices interfering with the connection into account.

If you're using a proxy, ensure that it is not involved or that it is correctly configured to handle these devices. We have lots of devices or processes that are smart enough to use their host operating system's WPAD settings but not submit the active directory credentials of the user account that is running them with their HTTP/HTTPS requests and the proxy expects all connections to be authenticated and so the process will quietly fail client-side.