how can I discover if my server is performing some illegal actions, like flooding?
It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, visit the help center. If this question can be reworded to fit the rules in the help center, please edit the question.
Monitoring network I/O.
You don't mention platform, internet connection, etc...
Run malware scans/virus scans. Again, don't know your platform so can't recommend anything.
Check your routing/ARP tables for unusual activities. You didn't mention what kind of server it is, so can't recommend anything there...