I noticed this while running a performance test with JMeter. For first half an hour everything was fine and the /server/all/tmp directory size was around 36M.
Then suddenly the tmp directory grew up to 6.1G. The space was taken by jar files inside vfs-nested.tmp. If I let the server sit idle there is no such behaviour.
I found https://jira.jboss.org/jira/browse/JBAS-7126 but adding that config made no difference.
Edit: Starting up with -Djboss.vfs.forceCopy=false does not create such directory at all. But does that affect clustered setup somehow?