Can't launch glassfish on ec2 - can't open port

Question

I'm trying to start glassfish on an EBS-based AMI of Ubuntu 10.04 64-bit. I have used glassfish on non-ec2 servers with no problems, but on ec2 I get this message:

$ sudo -u glassfish bin/asadmin start-domain domain1
There is a process already using the admin port 4848 -- it probably is another instance of a GlassFish server.
Command start-domain failed.

I know that ec2 has requires that firewall rules be modified using ec2-authorize to let outside traffic thru the firewall, as I had to do to make ssh work. This still doesn't explain the port error when all I'm trying to do is start glassfish so I can try

$ wget localhost:8080

and make sure it's working.

This is very frustrating and I'd really appreciate any help.

Thanks.

FINAL UPDATE:
Sorry if you came here looking for answers. I never figured out what was causing the problem. I created another fresh instance, installed the same stuff, and Glassfish worked perfectly. Something obviously got boned during installation, but I have no idea what. I guess it will remain a mystery.

UPDATE: Here's what I get from netstat:

# netstat -nuptl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      462/sshd        
tcp6       0      0 :::22                   :::*                    LISTEN      462/sshd        
udp        0      0 0.0.0.0:5353            0.0.0.0:*                           483/avahi-daemon: r
udp        0      0 0.0.0.0:1194            0.0.0.0:*                           589/openvpn     
udp        0      0 0.0.0.0:37940           0.0.0.0:*                           483/avahi-daemon: r
udp        0      0 0.0.0.0:68              0.0.0.0:*                           377/dhclient3   

UPDATE: One more thing... I know that the "net.ipv6.bindv6only" kernel option can cause problems with java networking, so I did set this:

# sysctl -w net.ipv6.bindv6only=0

UPDATE: I also verified that it has nothing at all to do with the port number (4848). As you can see here, when I changed the admin-listener port in domain.xml to 4949, I get a similar message:

# sudo -u glassfish bin/asadmin start-domain domain1
There is a process already using the admin port 4949 -- it probably is another instance of a GlassFish server.
Command start-domain failed.

UPDATE:
Here are the contents of /etc/hosts:

127.0.0.1 localhost

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

I should mention that I have another Ubuntu Lucid 10.04 64-bit slice that is NOT hosted on ec2, and set it up the exact same way with no problems whatsoever.

Also server.log doesn't offer much insight either:

# cat ./server.log 
Nov 20, 2010 8:46:49 AM com.sun.enterprise.admin.launcher.GFLauncherLogger info
INFO: JVM invocation command line:
/usr/lib/jvm/java-6-sun-1.6.0.22/bin/java
-cp
/opt/glassfishv3/glassfish/modules/glassfish.jar
-XX:+UnlockDiagnosticVMOptions
-XX:MaxPermSize=192m
-XX:NewRatio=2
-XX:+LogVMOutput
-XX:LogFile=/opt/glassfishv3/glassfish/domains/domain1/logs/jvm.log
-Xmx512m
-client
-javaagent:/opt/glassfishv3/glassfish/lib/monitor/btrace-agent.jar=unsafe=true,noServer=true
-Dosgi.shell.telnet.maxconn=1
-Djdbc.drivers=org.apache.derby.jdbc.ClientDriver
-Dfelix.fileinstall.dir=/opt/glassfishv3/glassfish/modules/autostart/
-Djavax.net.ssl.keyStore=/opt/glassfishv3/glassfish/domains/domain1/config/keystore.jks
-Dosgi.shell.telnet.port=6666
-Djava.security.policy=/opt/glassfishv3/glassfish/domains/domain1/config/server.policy
-Dfelix.fileinstall.poll=5000
-Dcom.sun.aas.instanceRoot=/opt/glassfishv3/glassfish/domains/domain1
-Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory
-Dosgi.shell.telnet.ip=127.0.0.1
-Djava.endorsed.dirs=/opt/glassfishv3/glassfish/modules/endorsed:/opt/glassfishv3/glassfish/lib/endorsed
-Dcom.sun.aas.installRoot=/opt/glassfishv3/glassfish
-Djava.ext.dirs=/usr/lib/jvm/java-6-sun-1.6.0.22/lib/ext:/usr/lib/jvm/java-6-sun-1.6.0.22/jre/lib/ext:/opt/glassfishv3/glassfish/domains/domain1/lib/ext
-Dfelix.fileinstall.bundles.new.start=true
-Djavax.net.ssl.trustStore=/opt/glassfishv3/glassfish/domains/domain1/config/cacerts.jks
-Dcom.sun.enterprise.security.httpsOutboundKeyAlias=s1as
-Djava.security.auth.login.config=/opt/glassfishv3/glassfish/domains/domain1/config/login.conf
-DANTLR_USE_DIRECT_CLASS_LOADING=true
-Dfelix.fileinstall.debug=1
-Dorg.glassfish.web.rfc2109_cookie_names_enforced=false
-Djava.library.path=/opt/glassfishv3/glassfish/lib:/usr/lib/jvm/java-6-sun-1.6.0.22/jre/lib/amd64/server:/usr/lib/jvm/java-6-sun-1.6.0.22/jre/lib/amd64:/usr/lib/jvm/java-6-sun-1.6.0.22/lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
com.sun.enterprise.glassfish.bootstrap.ASMain
-domainname
domain1
-asadmin-args
start-domain,,,domain1
-instancename
server
-verbose
false
-debug
false
-asadmin-classpath
/opt/glassfishv3/glassfish/modules/admin-cli.jar
-asadmin-classname
com.sun.enterprise.admin.cli.AsadminMain
-upgrade
false
-domaindir
/opt/glassfishv3/glassfish/domains/domain1
-read-stdin
true

Answer 1

You're right - firewalling has nothing to do with the open (presumed TCP?) port.

netstat is your friend. Something similar to netstat -nupt -l will give you an indication of what processes are listening on what ports.

Let us know the output of that command, and whether it helps point you in the right direction.

Answer 2

I don't know why your hosts file was brought up, as these have nothing to do with each other.

This sounds to me (based on your revisions) that its not that something is currently listening on the port--but that you may have configured TWO things to listen on the same port in the same Glassfish instance. So if you start Glassfish and it tries to launch listeners on each of the ports, it will fail, and consequently shutdown the previously started services. So when you use netstat before and after, it reports correctly that nothing is currently listening on the ports.

Check that each of your listeners is on a different port using asadmin, and I'll bet you find that the port number is used twice. Or it could simply be that the initialization is causing two instances to start up simultaneously, which would give you the same behavior. Can you try configuring a brand new domain on different ports to see if you experience a similar issue?

Best of luck!

UPDATE: Try using the following to get all the listeners:

asadmin get server | grep -i listener

Answer 3

same problem here. It is related to /etc/hosts i did a cat /etc/hostname and added this for 127.0.0.1

Answer 4

This error is thrown not only when the port is taken, but also erroneously when there is an error with the /etc/hosts file.

Make sure you have proper definitions for your hostname (have you changed it recently?) in the file, and that it is not missing. For example, if you just changed your hostname (either via hostname or in /etc/hostname) to server.example.com, then make sure that your /etc/hosts file has:

127.0.0.1 server.example.com localhost

Answer 5

Recently, I found that launching a server into VPC requires quite a bit of extra configuration (once off) to get started. A key clue:

 nslookup `hostname`

should work. If it doesn't work, that's likely to be the reason.

You can either: 1) launch a server into EC2-classic 2) configure your VPC / host to be able to run the command above successfully.

Hope that helps.