It's just the difference of whether you consider memory that contains discardable data as used or not. The memory is used in the sense that it contains information that may be useful. But it's free in the sense that the information can simply be discarded if the memory is needed.
For example, say you run a program. The executable file that holds the program itself is still in memory. But that data is not needed at the time. If the program runs again, however, the information can be used from memory so it doesn't have to be loaded from disk again. But if that memory is needed for some other purpose, the file data can be thrown away since it's on the disk. Should this be considered free memory or not?
The reason so much memory is used is because free memory is wasted memory. Only memory that is being used can improve performance. The OS tries to keep as little memory free as possible, only enough to handle special situations such as bursts of data received from the network. It's not like if you use only 3GB today you can use 5GB tomorrow. If you only use 3GB today, you just wasted 1GB.