Setup machines network in Windows Azure

Question

I'm building a web application and I've decided to host it on Windows Azure.

My application has three components:

• A front-end created with Play Framework (machine A)
• A back-end core written in Java that executes my business logic (machine B)
• A database (machine C)

Since I have a free Bizspark account I created 3 free Azure users: for each one I created a medium Ubuntu 12.04 LTS machine and I installed one component on each machine.

Each of the component is running on its own machine to boost performance. The connections I'm interested in is the connection from machines A and B to machine C.

I successfully connect to the database machines, I created all of the needed endpoints and the corrects access rights have been given

BUT

the problems I'm getting are the following:

SEVERE: FATAL: DataSourcePool [mysql] is down!!!
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 1,197,486 milliseconds ago.
Caused by: java.net.SocketException: Connection timed out

Or:

WARNING: DataSourcePool closing leaked connection?
Nov 27, 2013 8:55:46 PM com.avaje.ebeaninternal.server.lib.sql.DataSourcePool validateConnection
WARNING: heartbeatsql test failed on connection[mysql.31]
Nov 27, 2013 8:55:46 PM com.avaje.ebeaninternal.server.lib.sql.PooledConnection  closeConnectionFully
INFO: Closing Connection[mysql.31] psReuse[19] psCreate[19] psSize[19]

Or:

javax.persistence.PersistenceException: java.sql.SQLException: Unsuccessfully waited
[1000] millis for a connection to be returned. No connections are free. You need to
Increase the max connections of [100] or look for a connection pool leak using
datasource.xxx.capturestacktrace=true

Since my application on another web server was running fine I think there's something wrong in the configuration of my network. I think that the 1000ms run out before eBean can get a connection from the pool. I tried to execute some basic connectivity tests like Ping or Traceroute from my core machine to the db machine but Azure blocks ICMP messages. I then tried TCPPING and the output is the following:

tcpping xxx.cloudapp.net 3306
TCP Ping 1:0,808ms
TCP Ping 2:0,934ms
TCP Ping 3:1,036ms
TCP Ping 4:0,860ms
TCP Ping 5:0,927ms
TCP Ping 6:0,905ms
TCP Ping 7:0,949ms
TCP Ping 8:1,079ms
TCP Ping 9:1,274ms
TCP Ping 10:0,983ms

TCPTraceroute is not working, it only shows asterisks...

I don't know if these results are believable or not but I don't know how Azure works. I think that 1 second on average is an enormity.

I then tried to connect remotely to the database machine: I installed a MySQL 5.5 client on the core machine and connected like this:

mysql -u [username] -h [db machine hostname].cloudapp.net --port=3306 -p[password]

All's fine and it seems also pretty fast, I can't see any sort of lag from the console view.

Some details about the machines:

• all are in the same region (West Europe)

• they are created from different accounts (like [email protected] owns db machine, [email protected] owns core machine, etc.)

• they all run the same Ubuntu version

Can you please tell me what can I do to decrease this delay?

Answer 1

Well, apart from the fact that you are exploiting the platform benefits, here are my thoughts.

I would say the best way to interconnect VMs is to utilize the Azure Virtual Network. However, as of today (Dec. 5, 2013) you cannot span a Virtual Network across multiple accounts. Your only option is to access the machines via the VIP (public IP Address), which you do now. I would say that what you observe is about 1 ms (millisecond) and not 1 s (second) as you second! And I would say that 1 ms is somehow neglectable. Also, you have to rewind your Match classes. The error message you quote:

last packet successfully received from the server was 1,197,486 milliseconds ago

Refers to 1,197,486 milliseconds. There are just 1000 milliseconds in 1 second. This means that 1,197,486 milliseconds are approx. 1,198 seconds, or roughly 20 minutes. Well, 20 minutes of timeout is definitely not related to network latency.

If you want the things right, or correct - just put all within same account, single Virtual Network and use internal IP Addresses for communication - this way you will skip the Azure Load Balancer, which you cannot avoid when doing cross-deployment communication over public IP Addresses.

More or less, again as of today, every Azure Deployment (including your VMs) runs under the picture described in this blog post. Just consider that you have 3 such diagrams, every single one of them with just one VM, instead of 5 as shown on the diagram.