It's interesting really when there are a lot of files/names in a directory, particularly if they have a lot of overlapping characters at the start of a bunch of long file names. File creation times will go up in these cases (because the algorithm to create a short name for a long file name has the requirement that the short name cannot already be in the index, naturally). Enumeration of the directory can also become more expensive.
Generally, if there is a lot of metadata activity in a very large directory, and no apps that depend upon short names are present, it can help a good deal. It's hard to quantify, since YMMV.