Are you monitoring server load?
There's performance tools for RoR that can measure your server's performance when running RoR scripts, but I can't remember the name offhand. Shouldn't be too hard to find it, though. The one I am thinking about was discussed on the Ruby on Rails podcast.
Are you checking the logs for errors?
You're saying server latency, but it may not be latency at all. It could very well be a performance issue on the server. Does it run low on memory due to a bug? Processes getting killed after running errant?
Can you monitor the server to see what top says is happening at the time of one of these lags?
Sounds almost like something has to be transferred or moved into working memory and after that is being cached so it's running quickly when asked for again, unless something is being proxied or hung up somewhere. If the server itself isn't having any issues show up, you would need to diagnose if it's you or everyone having the occasional lag.
There are test tools out there for measuring performance. Maybe you should look into one of those services to test your server and profile the RoR/database code.
You might also want to consider installing some kind of server monitoring software like Nagios to alert you when/if a resource is running low or if there's an alert condition to check on.