telnet cannot connect to port without firewall

Question

If I try to connect to MySQL with

telnet myhost.com 3306

I get a network not reachable error, even when I flush all iptables rules. I can reach other ports like ssh (with the same iptables rules applying as for ssh).

Below is what I see in netstat. It gives me the impression that there is more to accessing ports than iptables: ports like 10025 are closed in my iptables.

And I see MySQL open for tcp but not for tcp6 (ssh is open for both).

This issue started after upgrading from Ubuntu server 12.04 to 14.04. Anyone any suggestion as to the connection time out?

$ netstat -tlp 3306
(No info could be read for "-p": geteuid()=1000 but you should be root.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 *:pop3s                 *:*                     LISTEN      -               
tcp        0      0 localhost:10023         *:*                     LISTEN      -               
tcp        0      0 localhost:10024         *:*                     LISTEN      -               
tcp        0      0 localhost:10025         *:*                     LISTEN      -               
tcp        0      0 *:mysql                 *:*                     LISTEN      -               
tcp        0      0 *:pop3                  *:*                     LISTEN      -               
tcp        0      0 localhost:spamd         *:*                     LISTEN      -               
tcp        0      0 *:imap2                 *:*                     LISTEN      -               
tcp        0      0 *:urd                   *:*                     LISTEN      -               
tcp        0      0 *:smtp                  *:*                     LISTEN      -               
tcp        0      0 *:sieve                 *:*                     LISTEN      -               
tcp        0      0 *:ssh                  *:*                     LISTEN      -               
tcp        0      0 *:imaps                 *:*                     LISTEN      -               
tcp6       0      0 [::]:pop3s              [::]:*                  LISTEN      -               
tcp6       0      0 localhost:10023         [::]:*                  LISTEN      -               
tcp6       0      0 [::]:pop3               [::]:*                  LISTEN      -               
tcp6       0      0 localhost:spamd         [::]:*                  LISTEN      -               
tcp6       0      0 [::]:imap2              [::]:*                  LISTEN      -               
tcp6       0      0 [::]:http               [::]:*                  LISTEN      -               
tcp6       0      0 [::]:urd                [::]:*                  LISTEN      -               
tcp6       0      0 [::]:smtp               [::]:*                  LISTEN      -               
tcp6       0      0 [::]:https              [::]:*                  LISTEN      -               
tcp6       0      0 [::]:sieve              [::]:*                  LISTEN      -               
tcp6       0      0 [::]:ssh               [::]:*                  LISTEN      -               
tcp6       0      0 [::]:imaps              [::]:*                  LISTEN      -   

150407 12:31:07 [Note] /usr/sbin/mysqld: Normal shutdown

150407 12:31:07 [Note] Event Scheduler: Purging the queue. 0 events
150407 12:31:07  InnoDB: Starting shutdown...
150407 12:31:10  InnoDB: Shutdown completed; log sequence number 574674933
150407 12:31:10 [Note] /usr/sbin/mysqld: Shutdown complete

150407 12:31:11 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150407 12:31:11 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.
150407 12:31:12 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
150407 12:31:12 [Note] Plugin 'FEDERATED' is disabled.
150407 12:31:12 InnoDB: The InnoDB memory heap is disabled
150407 12:31:12 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150407 12:31:12 InnoDB: Compressed tables use zlib 1.2.8
150407 12:31:12 InnoDB: Using Linux native AIO
150407 12:31:12 InnoDB: Initializing buffer pool, size = 128.0M
150407 12:31:12 InnoDB: Completed initialization of buffer pool
150407 12:31:12 InnoDB: highest supported file format is Barracuda.
150407 12:31:12 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
150407 12:31:12 [Note] Plugin 'FEDERATED' is disabled.
150407 12:31:12 InnoDB: The InnoDB memory heap is disabled
150407 12:31:12 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150407 12:31:12 InnoDB: Compressed tables use zlib 1.2.8
150407 12:31:12 InnoDB: Using Linux native AIO
150407 12:31:12 InnoDB: Initializing buffer pool, size = 128.0M
150407 12:31:12 InnoDB: Completed initialization of buffer pool
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
150407 12:31:12  InnoDB: Retrying to lock the first data file
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
150407 12:31:13  InnoDB: Waiting for the background threads to start
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
150407 12:31:14 InnoDB: 5.5.41 started; log sequence number 574674933
150407 12:31:14 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
150407 12:31:14 [Note]   - '0.0.0.0' resolves to '0.0.0.0';
150407 12:31:14 [Note] Server socket created on IP: '0.0.0.0'.
150407 12:31:15 [Note] Event Scheduler: Loaded 0 events
150407 12:31:15 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.41-0ubuntu0.14.04.1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu)
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
...
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
150407 12:32:52  InnoDB: Unable to open the first data file
InnoDB: Error in opening ./ibdata1
150407 12:32:52  InnoDB: Operating system error number 11 in a file operation.
InnoDB: Error number 11 means 'Resource temporarily unavailable'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/operating-system-error-codes.html
150407 12:32:52 InnoDB: Could not open or create data files.
150407 12:32:52 InnoDB: If you tried to add new data files, and it failed here,
150407 12:32:52 InnoDB: you should now edit innodb_data_file_path in my.cnf back
150407 12:32:52 InnoDB: to what it was, and remove the new ibdata files InnoDB created
150407 12:32:52 InnoDB: in this failed attempt. InnoDB only wrote those files full of
150407 12:32:52 InnoDB: zeros, but did not yet use them in any way. But be careful: do not
150407 12:32:52 InnoDB: remove old data files which contain your precious data!
150407 12:32:52 [ERROR] Plugin 'InnoDB' init function returned error.
150407 12:32:52 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
150407 12:32:52 [ERROR] Unknown/unsupported storage engine: InnoDB
150407 12:32:52 [ERROR] Aborting

150407 12:32:52 [Note] /usr/sbin/mysqld: Shutdown complete

150407 12:32:52 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
150407 12:32:52 [Note] Plugin 'FEDERATED' is disabled.
......  

output of netstat after changing bind-address to 0.0.0.0:

tcp        0      0 0.0.0.0:3306            0.0.0.0:*               LISTEN      18890/mysqld

Output dig:

; <<>> DiG 9.9.5-3ubuntu0.2-Ubuntu <<>> myhost.com ip r get 123.45.67.890 telnet 123.45.67.890 3306
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55636
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;myhost.com.        IN  A

;; ANSWER SECTION:
myhost.com. 3600    IN  A   123.45.67.890

;; Query time: 856 msec
;; SERVER: 127.0.1.1#53(127.0.1.1)
;; WHEN: Tue Apr 07 22:55:03 CEST 2015
;; MSG SIZE  rcvd: 60

;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 35733
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;ip.                IN  A

;; AUTHORITY SECTION:
.           528 IN  SOA a.root-servers.net. nstld.verisign-grs.com. 2015040701 1800 900 604800 86400

;; Query time: 159 msec
;; SERVER: 127.0.1.1#53(127.0.1.1)
;; WHEN: Tue Apr 07 22:55:03 CEST 2015
;; MSG SIZE  rcvd: 106

;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 17760
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;r.             IN  A

;; AUTHORITY SECTION:
.           528 IN  SOA a.root-servers.net. nstld.verisign-grs.com. 2015040701 1800 900 604800 86400

;; Query time: 55 msec
;; SERVER: 127.0.1.1#53(127.0.1.1)
;; WHEN: Tue Apr 07 22:55:03 CEST 2015
;; MSG SIZE  rcvd: 105

;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 20236
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;get.               IN  A

;; AUTHORITY SECTION:
.           527 IN  SOA a.root-servers.net. nstld.verisign-grs.com. 2015040701 1800 900 604800 86400

;; Query time: 62 msec
;; SERVER: 127.0.1.1#53(127.0.1.1)
;; WHEN: Tue Apr 07 22:55:04 CEST 2015
;; MSG SIZE  rcvd: 107

;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29568
;; flags: qr aa rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;123.45.67.890.         IN  A

;; ANSWER SECTION:
123.45.67.890.      0   IN  A   123.45.67.890

;; Query time: 0 msec
;; SERVER: 127.0.1.1#53(127.0.1.1)

/etc/mysql/my.cnf:

#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
# 
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html

# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.
[client]
port        = 3306
socket      = /var/run/mysqld/mysqld.sock

# Here is entries for some specific programs
# The following values assume you have at least 32M ram

# This was formally known as [safe_mysqld]. Both versions are currently parsed.
[mysqld_safe]
socket      = /var/run/mysqld/mysqld.sock
nice        = 0

[mysqld]
#
# * Basic Settings
#
user        = mysql
pid-file    = /var/run/mysqld/mysqld.pid
socket      = /var/run/mysqld/mysqld.sock
port        = 3306
basedir     = /usr
datadir     = /var/lib/mysql
tmpdir      = /tmp
lc-messages-dir = /usr/share/mysql
#skip-external-locking
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
#bind-address       = 127.0.0.1
bind-address        = 0.0.0.0

#
# * Fine Tuning
#
key_buffer      = 16M
max_allowed_packet  = 16M
thread_stack        = 192K
thread_cache_size       = 8
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover         = BACKUP
#max_connections        = 100
#table_cache            = 64
#thread_concurrency     = 10
#
# * Query Cache Configuration
#
query_cache_limit   = 1M
query_cache_size        = 16M
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file        = /var/log/mysql/mysql.log
#general_log             = 1
#
# Error log - should be very few entries.
#
log_error = /var/log/mysql/error.log
#
# Here you can see queries with especially long duration
#log_slow_queries   = /var/log/mysql/mysql-slow.log
#long_query_time = 2
#log-queries-not-using-indexes
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
#       other settings you may need to change.
#server-id      = 1
#log_bin            = /var/log/mysql/mysql-bin.log
expire_logs_days    = 10
max_binlog_size         = 100M
#binlog_do_db       = include_database_name
#binlog_ignore_db   = include_database_name
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem



[mysqldump]
quick
quote-names
max_allowed_packet  = 16M

[mysql]
#no-auto-rehash # faster start of mysql but no tab completition

[isamchk]
key_buffer      = 16M

#
# * IMPORTANT: Additional settings that can override those from this file!
#   The files must end with '.cnf', otherwise they'll be ignored.
#
!includedir /etc/mysql/conf.d/

traceroute:

 1  192.168.1.1 (192.168.1.1)  4.728 ms  4.720 ms  4.707 ms
 2  1.16.15.37.dynamic.jazztel.es (37.15.16.1)  26.522 ms  26.529 ms  28.352 ms
 3  10.255.160.254 (10.255.160.254)  30.024 ms  30.017 ms  29.987 ms
 4  41.217.106.212.static.jazztel.es (212.106.217.41)  44.086 ms 45.217.106.212.static.jazztel.es (212.106.217.45)  52.257 ms 41.217.106.212.static.jazztel.es (212.106.217.41)  42.428 ms
 5  * 42.217.106.212.static.jazztel.es (212.106.217.42)  47.672 ms  52.229 ms
 6  129.216.106.212.static.jazztel.es (212.106.216.129)  57.838 ms  61.308 ms *
 7  142.216.106.212.static.jazztel.es (212.106.216.142)  89.549 ms  106.063 ms *
 8  142.216.106.212.static.jazztel.es (212.106.216.142)  76.570 ms 195.66.225.53 (195.66.225.53)  87.575 ms 142.216.106.212.static.jazztel.es (212.106.216.142)  84.337 ms
 9  195.66.225.53 (195.66.225.53)  106.011 ms  76.555 ms  105.993 ms
10  openpeering.pcextreme.nl (82.150.154.35)  84.274 ms telecity2.openpeering.nl (82.150.154.26)  87.533 ms nikhef.openpeering.nl (82.150.154.25)  105.973 ms
11  openpeering.pcextreme.nl (82.150.154.35)  87.506 ms  87.474 ms 185.27.173.130 (185.27.173.130)  79.570 ms
12  185.27.173.150 (185.27.173.150)  95.558 ms  95.510 ms 185.27.173.130 (185.27.173.130)  81.846 ms
13  185.27.173.150 (185.27.173.150)  68.465 ms *  84.567 ms
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

Answer 1

How would I go about network unreachable using plain telnet to some port?

On the client

1. Name resolution (what are you actually connecting to?)

   nslookup myhost.com
   

What is the result? What IP? IPv4 or IPv6? (It would help not obfuscating the output.)

2. Network path (Is something blocking the way to the server?)

Oh and BTW what is the OS you are trying from? If Linux:

traceroute myhost.com

On the server

1. Is the service listening (If it's not listening, no way to connect to it)

   netstat -tulpn | grep mysql
   

2. On the server try a local connection

   telnet localhost 3306
   

What is the result?

3. During telnet from client start

   tshark -ta -n port 3306
   

Do you see packets coming from the client? (make sure tshark is running on the correct interface, if many)

If no packets: something on the net is blocking them (NOT a potential iptables firewall on the server, we will get to that next.)

If packets are indeed seen: no network problem, check iptables

iptables -vnL

What does it say? What about iptables -t raw -vnL, iptables -t mangle -vnL, iptables -t nat -vnL?

Is Selinux active? Or some other host protection?

I see locking errors during shutdown. Are you maybe having multiple mysqlds running by mistake? What does ps -ef | grep mysql say? Is startup as messy as shutdown?

Please reply with detailed results.

Answer 2

This has been an issue with me before whereby I cannot connect to a remote MySQL server via the internet. I did a few quick troubleshooting tests, such as:

• Making sure mysqld is active/had started.
• Made sure SELinux was disabled so the port can be spawned

By the information you have provided, you seem to have done the above as I can see it has bound to tcp with the line:

tcp        0      0 *:mysql                 *:*                     LISTEN      -     

Then I recalled an article I read on Stackoverflow from apesa, which consisted of:

To expose MySQL to anything other than localhost you will have to have the following line uncommented in /etc/mysql/my.cnf and assigned to your computers IP address and not loopback

#Replace xxx with your IP Address 
bind-address        = xxx.xxx.xxx.xxx

Or add a bind-address = 0.0.0.0 if you don't want to specify the IP

Then stop and restart MySQL with the new my.cnf entry. Once running go to the terminal and enter the following command.

lsof -i -P | grep :3306

That should come back something like this with your actual IP in the xxx's

mysqld  1046  mysql  10u  IPv4  5203  0t0  TCP  xxx.xxx.xxx.xxx:3306 (LISTEN)

If the above statement returns correctly you will then be able to accept remote users. However for a remote user to connect with the correct priveleges you need to have that user created in both the localhost and '%' as in.

CREATE USER 'myuser'@'localhost' IDENTIFIED BY 'mypass';
CREATE USER 'myuser'@'%' IDENTIFIED BY 'mypass';

Then

GRANT ALL ON *.* TO 'myuser'@'localhost';
GRANT ALL ON *.* TO 'myuser'@'%';

If you don't have the same user created as above, when you logon locally you may inherit base localhost privileges and have access issues. If you want to restrict the access myuser has then you would need to read up on the GRANT statement syntax HERE If you get through all this and still have issues post some additional error output and the my.cnf appropriate lines.

NOTE: If lsof does not return or is not found you can install it HERE based on your Linux distribution. You do not need lsof to make things work, but it is extremely handy when things are not working as expected.

-> answered by Apesa: Remote connections MySQL Ubuntu

Generally this fixed my issue as I did not configure my.cnf

Answer 3

If bind-address is set to 127.0.0.1 in /etc/mysql/my.cnf try changing it to the following, which will allow MySQL to bind to all available interfaces:

bind-address = 0.0.0.0

Also, try granting yourself network access in MySQL itself:

mysql> GRANT ALL ON . to your-user-name@'your-hostname' IDENTIFIED BY 'your-password';

Answer 4

what about tcpwrapers: /etc/hosts.{allow,deny} files? Looking at your netstat output it looks like MySQL is listening on all IPv4 addresses and your iptables are open too. So it boils down to either IPv6 issue or tcpwrappers as they sit in the middle of things and are not easy to debug.