I have a system (the "host") that runs several containers, using LXC (i.e. the "guests"). I've installed Jenkins inside the guests and they appear to be working as intended, except that they don't respond to requests. (I've made several successful Jenkins installations before, including LXC.) In this case, the observed problem is that the built-in Jenkins web server (Jetty) is not responding to HTTP requests, even if those request are made from within the very LXC guest it's running in, i.e. pointing at the
I've been working to resolve this issue for several days, without success.
This is what you get when trying to contact the Jenkins web server from the
root@base:~# curl -vI http://localhost:8080/jenkins/ * Trying 127.0.0.1... * TCP_NODELAY set * Connected to localhost (127.0.0.1) port 8080 (#0) > HEAD /jenkins/ HTTP/1.1 > Host: localhost:8080 > User-Agent: curl/7.58.0 > Accept: */* >
On a working setup, you should get an HTTP-403 because you have not been authenticated and it shouldn't take more than a second or two to reply, but even after a few hours, there's no response. The Jenkins log file doesn't report any errors, either.
I need help to root-cause and resolve this problem, so that the Jenkins install works as intended and becomes accessible.
Any pointers on what/where to look for to figure out and fix this problem?
Here're some things I've already looked into:
- Jenkins Configuration: The configuration file at
/etc/default/jenkinsis similar to my other working setups and has had minimal changes (e.g. binding to localhost only, and the prefix).
- Apache Configuration: I reviewed the Apache reverse-proxy config and compared to other working systems, but that was not a problem. Also, Apache was always accessible (e.g. the "It works!" page), even from outside the LXC container, so traffic was not getting blocked by firewall rules. Apache would fail with HTTP-502 Proxy Error because Jenkins wouldn't reply to it. (That said, I've uninstalled Apache to simplify the environment.)
- Log Files: The Jenkins log file at
/var/log/jenkins/jenkins.logdoes not report any problems, which would usually show up as Java stack traces from exceptions.
- Firewall Rules (
iptables -S): All the chains/rules (
OUTPUT) are set to
ACCEPT. Still, since communication here is within the
localhost, I wouldn't expect issues even if there had been other firewall rules in place.
- Network Packets and Ports (
netstat -tapon): Shows Jenkins (java process) listening on the expected port (default 8080, but I've tried others); it also shows the connection as
ESTABLISHED(on both ends) after the
curlclient sends a request like the one shown above. This shows a successful TCP handshake.
- Network Traffic (
tcpdump -i lo): Shows the 3-way handshake being made; it explains why
netstatshows connections as
- Comparing Against Working Setups: The other Jenkins installations I've made have similar environments and configurations (e.g. Ubuntu 18.04 hosts, same changes to Jenkins config file, installation procedures, etc).
- Reproducing the Problem: I've tried (and failed) to reproduce the problem in other systems; I've used the exact same environment, installation process, configs, etc (e.g. my laptop, separate server at work, separate server at home, same LXC versions, matching guest OS image fingerprints, etc); everything works as expected outside the production server in question (Dell PowerEdge R640 Server).
- Nuking the System from Orbit1: I've destroyed/rebuilt all the containers from scratch several times (including destroying the ZFS pool where all the data is stored); it made no difference.
- Installing in Host Directly: I have confirmed that installing Jenkins directly on the host, i.e. outside any LXC containers/guests, also shows the problem.
- Rule out Java/JVM: I can confirm that other Java-based applications work correctly, so it does not appear to be a problem that affects any/all Java-based programs. (I tested this by setting up an Apache Tomcat server, which worked as expected.)
- Relocate Host: To rule out potential data center environment issues, I moved the server onto my desk area, where I have another test server with a working setup. This made no difference.
- Run Stand-alone Jetty: I got the closest-matching Jetty server version I could find to the one bundled with Jenkins. Could not reproduce the problem. The stand-alone Jetty server replied to requests as expected, even though the one bundled with Jenkins still doesn't. (Jenkins' Jetty version is reported as
jetty-9.4.z-SNAPSHOT; built: 2018-06-05T18:24:03.829Zin the log. There's no version with this
.z-SNAPSHOTname on the Jetty releases page, so I used the closest match based on build date for this test:
- Switch from OpenJRE to Oracle JRE: Installed/Set Oracle's JRE to be used (i.e.
update-alternatives --config java). The same non-responsive behavior is observed.
Some of the questions I've already looked at, but weren't related or helpful:
- Unable to access Jenkins server
- Can't access Jenkins using nginx
- Correct Apache reverse proxy config with SSL for Jenkins and Sonar
- Proxy error when accessing jenkins
- Reverse Proxy Linux Containers
I've read way more than these; they're only a sample.
1 It's the only way to be sure... mostly...