It depends on the installer of your software. Many installers wont run because they rely on Windows components that are not present in Server Core.
Many installers come with a -quiet or -silent switch to perform the installation without starting the GUI part of the installer. In many cases they have an answer file in which you can specify the components to install.
As Chris S. said, Server Core is not GUI-less, it has the normal Windows GUI with windows, what it is missing is Windows Explorer, IE, management console, WPF etc.
It is hard to tell whether a GUI program will run on Server Core, many small tools do. I even run the Netscape 3 browser sometimes for local troubleshooting
Some of my dot.NET Windows forms applications are running as well, while others don't because certain components are not present.
If the installer of your software does not run, you may still be able to get the software to run. In some cases I just installed the software on a GUI server, then copied the files and potentially registry entries over to the core server.
Over time more and more vendors will offer installers that work on Server Core.