how can I discover if my server is performing some illegal actions, like flooding?
It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, see the FAQ.
Monitoring network I/O.
You don't mention platform, internet connection, etc...
Run malware scans/virus scans. Again, don't know your platform so can't recommend anything.
Check your routing/ARP tables for unusual activities. You didn't mention what kind of server it is, so can't recommend anything there...