Okay, you've not given some of the basic information (like particular OS distribution or kernel version). That matters because the sysctl/kernel setting defaults differ across distros and certain tunables aren't exposed in some Linux systems. You're working with a server from 2008, so how do we know that your OS and kernel aren't from the same era?
Looking at your network parameters, though, I'd increase the default buffer sizes. A recent system setup for high-frequency trading I deployed had much higher
rmem_default settings. Try "8388608" to start and see if that helps. It's a basic change, but usually the first step...
I would also look at changing the realtime priorities of your (presumably custom) application. Are you using any form of CPU affinity (taskset, cgroups) in your app or wrapper script? How about the realtime priority of your app? Look into the chrt command and its options to see what would be appropriate for your situation. Is your application multithreaded?
Luckily, the 5400-series CPU doesn't have hyperthreading to deal with, but how are your other BIOS settings? Did you disable power management and C-states? Are there any unnecessary daemons running on the system? Is
Now, as to the hardware you're using, if this if for HFT use, you're behind; literally THREE jumps in CPU and architectural changes... The Nehalem (5500-series) brought a big jump in tech over the 5400-series you're using. Westmere (5600) was even better. Sandy Bridge was a big enough change over the 5500/5600 to spur another hardware refresh in my environments.
It also sounds like you're using the onboard NICs. There were some hoops we needed to jump through when dealing with Broadcom... But you're not at that point yet. How does CPU load look when you encounter dropped-packets? What type of data flow rate are you experiencing during your captures? This may just be a case of your system not keeping up.
There are a lot of knobs to tweak/tune here. A better understanding of what you're working with will help us narrow things down, though.
Edit: you mentioned Red Hat. The options for EL5 and EL6 differ, but the suggestions above do apply in theory.
Edit: It's good that you're on RHEL 6. There's a lot you can do. Try setting the priority of your app and test. Another useful guide is the RHEL MRG tuning guide. Not all of the features will be available to your kernel, but this will give you some ideas and explanations for some of the things you can modify for more deterministic performance.