How do you update your Java EE app in production?

Question

I’ve been trying to figure out what is happening in the world of Java EE production deployment and frankly it seems pretty scary. After speaking to over a 100 people these are my hypotheses:

1. Nobody uses redeployment in production (as in the actual button that does in-server update). It just isn’t reliable enough due to OutOfMemoryError-s and other failures.
2. The common way to update an application is to:
   1. Take all servers down at 2am and hope no one is using it.
   2. Take servers down one at a time, upgrade them and either drop or migrate the user sessions.
   3. Use weird hacks like copying one file at a time.

I’m also trying to find out how the update process happens, how hard it is and what does it cost in human measure (hours) and in soulless business measure (dollars).

I ask you to help me out and provide me with some semi-solid data I can use to better understand what’s going on in reality. Hopefully you’ll prove me wrong. I'm looking forward to you answers, but would also ask you to answer this survey to get some data: http://0t.ee/appupdq11

I know this is not the typical ServerFault question, as there isn't a right answer here, but by filling in the survey you'll help me a great deal and if you provide a longer answer here it will be just awesome.

Answer 1

Strongly depends on the application.

Most J2EE applications run on server cluster farms where a basic application upgrade can be done with a rolling upgrade. I.e. take one server out of the load balancer farm, wait for its work to finish, update the EAR deployment, test it, put it back in the lb farm, take the next out and repeat. You may require an outage to do database schema upgrades, just to make sure consistency of data. Database software upgrades can usually be done without a major outage, Oracle is one good example.

Answer 2

in weblogic i use wlst (python on java - jython) to do deployments. you can make any logic you want with it. the complicated part of all the deployments is when you do a database schema update ... which is not really a java problem but it affects the java systems.

on glassfish and tomcat is simpler ... tar up the directory install a fresh instance with the old configs and then deploy the war,ear file.

of course you can couple all this with a configuration management tool like cfengine, puppet or even fabric will do.