Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am trying to send commands to a SOAP service, and getting nothing in reply. The SOAP service is at a completely separate site from either server I am testing with.

I have written a dummy script with the SOAP XML embedded. When I run it at my local site, on any of three machines -- OSX, Ubuntu, or CentOS 5.3 -- it completes successfully with a good response.

I then sent the script to our public host at Slicehost, where I fail to get the response back from the SOAP service. It accepts the TCP socket and proceeds with the SSL handshake. I do not however receive any valid HTTP response.

This is the case whether I use my script or curl on the command line. I have rewritten the script using SOAP4R, Net::HTTP and Curb. All of which work at my local site, none of which work at the Slicehost site.

I have tried to assemble the CentOS box as closely to match my Slicehost server as possible. I rebuilt the Slice to be a stock CentOS 5.3 and stock CentOS 5.4 with the same results.

When I look at a tcpdump of the bad sessions on Slicehost, I see my script or curl send the XML to the remote server, and nothing comes back. When I look at the tcpdump at my local site, I see the response just fine. I have entirely disabled iptables on the Slice.

Does anyone have any ideas what could be causing these results? Please let me know what additional information I can furnish.

Thank you!

Below is a wire trace of a sample session. The IP that starts with 173 is my server while the IP that starts with 12 is the SOAP server's.

No.     Time        Source                Destination           Protocol Info
      1 0.000000    173.45.x.x        12.36.x.x         TCP      36872 > https [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=137633469 TSER=0 WS=6

Frame 1 (74 bytes on wire, 74 bytes captured)
Ethernet II, Src: 40:40:17:3a:f4:e6 (40:40:17:3a:f4:e6), Dst: Dell_fb:49:a1 (00:21:9b:fb:49:a1)
Internet Protocol, Src: 173.45.x.x (173.45.x.x), Dst: 12.36.x.x (12.36.x.x)
Transmission Control Protocol, Src Port: 36872 (36872), Dst Port: https (443), Seq: 0, Len: 0

No.     Time        Source                Destination           Protocol Info
      2 0.040000    12.36.x.x         173.45.x.x        TCP      https > 36872 [SYN, ACK] Seq=0 Ack=1 Win=8760 Len=0 MSS=1460

Frame 2 (62 bytes on wire, 62 bytes captured)
Ethernet II, Src: Dell_fb:49:a1 (00:21:9b:fb:49:a1), Dst: 40:40:17:3a:f4:e6 (40:40:17:3a:f4:e6)
Internet Protocol, Src: 12.36.x.x (12.36.x.x), Dst: 173.45.x.x (173.45.x.x)
Transmission Control Protocol, Src Port: https (443), Dst Port: 36872 (36872), Seq: 0, Ack: 1, Len: 0

No.     Time        Source                Destination           Protocol Info
      3 0.040000    173.45.x.x        12.36.x.x         TCP      36872 > https [ACK] Seq=1 Ack=1 Win=5840 Len=0

Frame 3 (54 bytes on wire, 54 bytes captured)
Ethernet II, Src: 40:40:17:3a:f4:e6 (40:40:17:3a:f4:e6), Dst: Dell_fb:49:a1 (00:21:9b:fb:49:a1)
Internet Protocol, Src: 173.45.x.x (173.45.x.x), Dst: 12.36.x.x (12.36.x.x)
Transmission Control Protocol, Src Port: 36872 (36872), Dst Port: https (443), Seq: 1, Ack: 1, Len: 0

No.     Time        Source                Destination           Protocol Info
      4 0.050000    173.45.x.x        12.36.x.x         SSLv2    Client Hello

Frame 4 (156 bytes on wire, 156 bytes captured)
Ethernet II, Src: 40:40:17:3a:f4:e6 (40:40:17:3a:f4:e6), Dst: Dell_fb:49:a1 (00:21:9b:fb:49:a1)
Internet Protocol, Src: 173.45.x.x (173.45.x.x), Dst: 12.36.x.x (12.36.x.x)
Transmission Control Protocol, Src Port: 36872 (36872), Dst Port: https (443), Seq: 1, Ack: 1, Len: 102
Secure Socket Layer

No.     Time        Source                Destination           Protocol Info
      5 0.130000    12.36.x.x         173.45.x.x        TCP      [TCP segment of a reassembled PDU]

Frame 5 (1434 bytes on wire, 1434 bytes captured)
Ethernet II, Src: Dell_fb:49:a1 (00:21:9b:fb:49:a1), Dst: 40:40:17:3a:f4:e6 (40:40:17:3a:f4:e6)
Internet Protocol, Src: 12.36.x.x (12.36.x.x), Dst: 173.45.x.x (173.45.x.x)
Transmission Control Protocol, Src Port: https (443), Dst Port: 36872 (36872), Seq: 1, Ack: 103, Len: 1380
Secure Socket Layer

No.     Time        Source                Destination           Protocol Info
      6 0.130000    173.45.x.x        12.36.x.x         TCP      36872 > https [ACK] Seq=103 Ack=1381 Win=8280 Len=0

Frame 6 (54 bytes on wire, 54 bytes captured)
Ethernet II, Src: 40:40:17:3a:f4:e6 (40:40:17:3a:f4:e6), Dst: Dell_fb:49:a1 (00:21:9b:fb:49:a1)
Internet Protocol, Src: 173.45.x.x (173.45.x.x), Dst: 12.36.x.x (12.36.x.x)
Transmission Control Protocol, Src Port: 36872 (36872), Dst Port: https (443), Seq: 103, Ack: 1381, Len: 0

No.     Time        Source                Destination           Protocol Info
      7 0.130000    12.36.x.x         173.45.x.x        TLSv1    Server Hello, Certificate, Server Hello Done

Frame 7 (1280 bytes on wire, 1280 bytes captured)
Ethernet II, Src: Dell_fb:49:a1 (00:21:9b:fb:49:a1), Dst: 40:40:17:3a:f4:e6 (40:40:17:3a:f4:e6)
Internet Protocol, Src: 12.36.x.x (12.36.x.x), Dst: 173.45.x.x (173.45.x.x)
Transmission Control Protocol, Src Port: https (443), Dst Port: 36872 (36872), Seq: 1381, Ack: 103, Len: 1226
[Reassembled TCP Segments (2606 bytes): #5(1380), #7(1226)]
Secure Socket Layer

No.     Time        Source                Destination           Protocol Info
      8 0.130000    173.45.x.x        12.36.x.x         TCP      36872 > https [ACK] Seq=103 Ack=2607 Win=11040 Len=0

Frame 8 (54 bytes on wire, 54 bytes captured)
Ethernet II, Src: 40:40:17:3a:f4:e6 (40:40:17:3a:f4:e6), Dst: Dell_fb:49:a1 (00:21:9b:fb:49:a1)
Internet Protocol, Src: 173.45.x.x (173.45.x.x), Dst: 12.36.x.x (12.36.x.x)
Transmission Control Protocol, Src Port: 36872 (36872), Dst Port: https (443), Seq: 103, Ack: 2607, Len: 0

No.     Time        Source                Destination           Protocol Info
      9 0.130000    173.45.x.x        12.36.x.x         TLSv1    Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message

Frame 9 (236 bytes on wire, 236 bytes captured)
Ethernet II, Src: 40:40:17:3a:f4:e6 (40:40:17:3a:f4:e6), Dst: Dell_fb:49:a1 (00:21:9b:fb:49:a1)
Internet Protocol, Src: 173.45.x.x (173.45.x.x), Dst: 12.36.x.x (12.36.x.x)
Transmission Control Protocol, Src Port: 36872 (36872), Dst Port: https (443), Seq: 103, Ack: 2607, Len: 182
Secure Socket Layer

No.     Time        Source                Destination           Protocol Info
     10 0.190000    12.36.x.x         173.45.x.x        TLSv1    Change Cipher Spec, Encrypted Handshake Message

Frame 10 (97 bytes on wire, 97 bytes captured)
Ethernet II, Src: Dell_fb:49:a1 (00:21:9b:fb:49:a1), Dst: 40:40:17:3a:f4:e6 (40:40:17:3a:f4:e6)
Internet Protocol, Src: 12.36.x.x (12.36.x.x), Dst: 173.45.x.x (173.45.x.x)
Transmission Control Protocol, Src Port: https (443), Dst Port: 36872 (36872), Seq: 2607, Ack: 285, Len: 43
Secure Socket Layer

No.     Time        Source                Destination           Protocol Info
     11 0.190000    173.45.x.x        12.36.x.x         TLSv1    Application Data

Frame 11 (347 bytes on wire, 347 bytes captured)
Ethernet II, Src: 40:40:17:3a:f4:e6 (40:40:17:3a:f4:e6), Dst: Dell_fb:49:a1 (00:21:9b:fb:49:a1)
Internet Protocol, Src: 173.45.x.x (173.45.x.x), Dst: 12.36.x.x (12.36.x.x)
Transmission Control Protocol, Src Port: 36872 (36872), Dst Port: https (443), Seq: 285, Ack: 2650, Len: 293
Secure Socket Layer

No.     Time        Source                Destination           Protocol Info
     12 0.190000    173.45.x.x        12.36.x.x         TCP      [TCP segment of a reassembled PDU]

Frame 12 (1514 bytes on wire, 1514 bytes captured)
Ethernet II, Src: 40:40:17:3a:f4:e6 (40:40:17:3a:f4:e6), Dst: Dell_fb:49:a1 (00:21:9b:fb:49:a1)
Internet Protocol, Src: 173.45.x.x (173.45.x.x), Dst: 12.36.x.x (12.36.x.x)
Transmission Control Protocol, Src Port: 36872 (36872), Dst Port: https (443), Seq: 578, Ack: 2650, Len: 1460
Secure Socket Layer

No.     Time        Source                Destination           Protocol Info
     13 0.450000    12.36.x.x         173.45.x.x        TCP      https > 36872 [ACK] Seq=2650 Ack=578 Win=64958 Len=0

Frame 13 (54 bytes on wire, 54 bytes captured)
Ethernet II, Src: Dell_fb:49:a1 (00:21:9b:fb:49:a1), Dst: 40:40:17:3a:f4:e6 (40:40:17:3a:f4:e6)
Internet Protocol, Src: 12.36.x.x (12.36.x.x), Dst: 173.45.x.x (173.45.x.x)
Transmission Control Protocol, Src Port: https (443), Dst Port: 36872 (36872), Seq: 2650, Ack: 578, Len: 0

No.     Time        Source                Destination           Protocol Info
     14 0.450000    173.45.x.x        12.36.x.x         TCP      [TCP segment of a reassembled PDU]

Frame 14 (206 bytes on wire, 206 bytes captured)
Ethernet II, Src: 40:40:17:3a:f4:e6 (40:40:17:3a:f4:e6), Dst: Dell_fb:49:a1 (00:21:9b:fb:49:a1)
Internet Protocol, Src: 173.45.x.x (173.45.x.x), Dst: 12.36.x.x (12.36.x.x)
Transmission Control Protocol, Src Port: 36872 (36872), Dst Port: https (443), Seq: 2038, Ack: 2650, Len: 152

No.     Time        Source                Destination           Protocol Info
     15 0.510000    12.36.x.x         173.45.x.x        TCP      [TCP Dup ACK 13#1] https > 36872 [ACK] Seq=2650 Ack=578 Win=64958 Len=0

Frame 15 (54 bytes on wire, 54 bytes captured)
Ethernet II, Src: Dell_fb:49:a1 (00:21:9b:fb:49:a1), Dst: 40:40:17:3a:f4:e6 (40:40:17:3a:f4:e6)
Internet Protocol, Src: 12.36.x.x (12.36.x.x), Dst: 173.45.x.x (173.45.x.x)
Transmission Control Protocol, Src Port: https (443), Dst Port: 36872 (36872), Seq: 2650, Ack: 578, Len: 0

No.     Time        Source                Destination           Protocol Info
     16 0.850000    173.45.x.x        12.36.x.x         TCP      [TCP Retransmission] [TCP segment of a reassembled PDU]

Frame 16 (1514 bytes on wire, 1514 bytes captured)
Ethernet II, Src: 40:40:17:3a:f4:e6 (40:40:17:3a:f4:e6), Dst: Dell_fb:49:a1 (00:21:9b:fb:49:a1)
Internet Protocol, Src: 173.45.x.x (173.45.x.x), Dst: 12.36.x.x (12.36.x.x)
Transmission Control Protocol, Src Port: 36872 (36872), Dst Port: https (443), Seq: 578, Ack: 2650, Len: 1460
Secure Socket Layer

No.     Time        Source                Destination           Protocol Info
     17 1.650000    173.45.x.x        12.36.x.x         TCP      [TCP Retransmission] [TCP segment of a reassembled PDU]

Frame 17 (1514 bytes on wire, 1514 bytes captured)
Ethernet II, Src: 40:40:17:3a:f4:e6 (40:40:17:3a:f4:e6), Dst: Dell_fb:49:a1 (00:21:9b:fb:49:a1)
Internet Protocol, Src: 173.45.x.x (173.45.x.x), Dst: 12.36.x.x (12.36.x.x)
Transmission Control Protocol, Src Port: 36872 (36872), Dst Port: https (443), Seq: 578, Ack: 2650, Len: 1460
Secure Socket Layer

No.     Time        Source                Destination           Protocol Info
     18 3.250000    173.45.x.x        12.36.x.x         TCP      [TCP Retransmission] [TCP segment of a reassembled PDU]

Frame 18 (1514 bytes on wire, 1514 bytes captured)
Ethernet II, Src: 40:40:17:3a:f4:e6 (40:40:17:3a:f4:e6), Dst: Dell_fb:49:a1 (00:21:9b:fb:49:a1)
Internet Protocol, Src: 173.45.x.x (173.45.x.x), Dst: 12.36.x.x (12.36.x.x)
Transmission Control Protocol, Src Port: 36872 (36872), Dst Port: https (443), Seq: 578, Ack: 2650, Len: 1460
Secure Socket Layer

No.     Time        Source                Destination           Protocol Info
     19 6.450000    173.45.x.x        12.36.x.x         TCP      [TCP Retransmission] [TCP segment of a reassembled PDU]

Frame 19 (1514 bytes on wire, 1514 bytes captured)
Ethernet II, Src: 40:40:17:3a:f4:e6 (40:40:17:3a:f4:e6), Dst: Dell_fb:49:a1 (00:21:9b:fb:49:a1)
Internet Protocol, Src: 173.45.x.x (173.45.x.x), Dst: 12.36.x.x (12.36.x.x)
Transmission Control Protocol, Src Port: 36872 (36872), Dst Port: https (443), Seq: 578, Ack: 2650, Len: 1460
Secure Socket Layer
share|improve this question

2 Answers 2

up vote 1 down vote accepted

I think you have some kind of MTU problem. It happens when you send packet larger then minimum MTU on the network path (PMTU) with Don't fragment bit set, and ICMP error message Fragmentation needed but Don't Fragment bit set is somewhere blocked.

You should check local and remote firewalls to allow ICMP first.

Then trace path to see what the PMTU is and where package drop can occure. Turn on Don't fragment bit! You should do this on the same port you use for client-server communications. Use hping2, for example.

If nothing helped, turn off PMTU discovery on both machines.

sudo sysctl net.ipv4.ip_no_pmtu_disc = 0

It is not good idea to use this, but if it works, you can be sure you have MTU problem.

Good luck!

share|improve this answer

Could you provide the tcpdup from the slice and localhost (IP's removed if needbe)?

This may help work out where the communication is failing/differs.

Ideally run tcpdump with a -v for better output.

share|improve this answer
    
I have added the trace to the question. Thanks! :-) –  Alex McHale Dec 4 '09 at 15:09

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.