I always feel anxious when installing new libraries to my server. Are are any other implications then little less free hard drive space? Theoretically,
apt-get install ... should not effect the performance anyhow, right?
Performance? Not unless the drive were filled or there's really really bad fragmentation (and if it's hitting the drive that much, you probably need more memory or have another problem)
The only way more libraries would "hurt" anything are if those libraries hose other system libraries or if they're trojaned. If you have backups you shouldn't need to worry.
Performance-wise having things sitting on the hard drive shouldn't affect anything. Only running applications using resources (memory, processor, network I/O, etc.) would affect performance...except when doing backups. Using more disk space means it can take longer to do backups.