I can't imagine that the GNOME desktop would slow things down in any appreciable way, except perhaps by using memory that you might want to allocate to other processes. But memory is very cheap these days, so this is a small issue.
Applications that you run while logged in are far more likely to have an impact than GNOME itself. At the moment that you log in, a number of applications will load, and that will have a momentary impact on CPU performance of other applications due to the disk being heavily used for a few tens of seconds, max.
I also cannot imagine Webadmin or any other web app causing a significant performance loss on a server. Not unless you are aggressively using it or unless it is somehow under web attack, causing a large and unusual number of requests going to that web app -- far more than a human being could enter in the same time period. A firewall can prevent that from occurring.
I see no reason today to run any server without a GUI.