0

I have been doing the most complicated IP tricks in my life before, so I am by no means a newbie. Now I have an extremely weird situation.

I have a system that I replicated dozens if not hundreds of time, with a jetty servlet server, and I am using an old trusted version of org.mortbay.jetty 6.11. Very light weight. Ran on JRE-1.5, -1.6, -1.7, -1.8, ... on Windows, Linux, FreeBSD, Solaris, what have you. No problem.

Now, the system is running fine on Amazon Linux this particular version.

       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/
104 package(s) needed for security, out of 190 available
Run "sudo yum update" to apply all updates.
[ec2-user@ws ~]$ uname -a
Linux ws.pill.guru 4.14.88-88.73.amzn2.x86_64 #1 SMP Thu Dec 13 18:04:55 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

If I only clone the system, by making an AMI and then launch an instance, in the same hardware category (t2.nano, t2.micro). As the instance comes up, the server is immediately started. No errors in the log. But just something simple as:

curl -v http://localhost/

will simply get stuck.

*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> Host: localhost
> User-Agent: curl/7.55.1
> Accept: */*
>

and it's stuck here. No response. Just stuck.

Netstat shows the connection as ESTABLISHED.

The port is 80, and I use the setcap method to allow that port to be opened, cat /etc/rc.local:

touch /var/lock/subsys/local
setcap cap_net_bind_service=+ep $(readlink -f $(which java))

and /etc/ld.so.conf.d/java.conf is

/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.amzn2.x86_64/jre/lib/amd64/jli
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.amzn2.x86_64/lib/amd64/jli

created with find /usr/lib/jvm/ -name jli >/etc/ld.so.conf.d/java.conf

The strange thing is, if I am creating a very simple Java netcat type server like this:

import java.net.Socket;
import java.net.ServerSocket;
import java.io.InputStream;
import java.io.OutputStream;

public class SimpleServer {
  public static void main(String[] args) throws Exception {
    try(ServerSocket serverSocket = new ServerSocket(Integer.parseInt(args[0]))) {
      while(true) {
        Socket conn = serverSocket.accept();
        InputStream in = conn.getInputStream();
        byte buffer[] = new byte[1024];
        while(true) {
          int n = in.read(buffer);
          if(n < 0)
            break;
          System.out.write(buffer, 0, n);
        }
      }
    }
  }
}

then it works just fine.

Now you might say: there is something wrong with your jetty server. Show me this configuration. Try that upgrade of eclipe jetty, whatever, no, I cannot afford that type of run-around. I rather ditch that entire Linux system and go to FreeBSD. But I am reporting this detail in the hopes that someone has seen a problem like that, and might give some free association. Comment or answer. I vote you up, I promise.

And I come back here and report myself if I ever find out what's wrong.

PS: yes, it is weird that on EC2 these connections are showing as tcp6 on netstat -d -a -t, even though they show IPv4 addresses. But that is not the issue. It still works on the old server, and fails on the exact clone.

PS: now I just moved everything over to FreeBSD and I get the exact same problem!

0

I found the issue. Found it with the debugger inside that Jetty code.

This was a configuration error.

It had to do with the configuration parameter "spawnOrShrinkAt", in fact, I think that is still a parameter in newer org.eclipe Jetty code, so it might help somebody who builds "embedded" web servers.

I started out with two threads in the thread pool, both were used by the SocketConnectors, one for HTTP and one for HTTPS. Now I made one connection. The spawnOrShrinkAt configuration parameter of the ThreadPool was set to 5. What happened is that a request handler thread was not spawned because 3 < spawnOrShrinkAt, so the request was put on the queue. I kept seeing this because I always disconnected the connection before I tried again. Had I just left the connection open and tried two more times, the threshold of 5 would finally have been exceeded and the requests would have been handled.

Why it happened when I cloned the server was because there were no clients making connection attempts (that's what I think it was).

The Jetty thread pool configuration parameters are rather obscure IMO, and the detrimental effect of that spawnOrShrink value was unclear to me. I just leave it at the default now and it works.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.