4

I use a distributed user-space filesystem (GlusterFS) and I would like to be sure GlusterFS processes will always have the computing power they need.

Each execution node of my grid have 2 CPU, with 4 cores per CPU and 2 threads per core (16 "processors" are seen by Linux).

My goal is to guarantee that GlusterFS processes have enough processing power to be reliable, responsive and fast. (There is no marketing here, just the dreams of a sysadmin ;-)

I consider two main points :

  • GlusterFS processes
  • I/O for data access (on local disks, or remote disks)

I thought about binding GlusterFS instances on a specific "processor".

I would like to be sure that :

  • No grid job will impact the kernel and the GlusterFS instances
  • Researchers jobs won't be affected by system processes (I'd like to reserve a pool of cores to job execution and be sure that no system process will use these CPUs)

But what about I/O ? As we handle a huge amount of data (several terabytes), we'll have a lot of interuptions.

How can I distribute these operations on my processors ? What are the "best practices" ?

Thanks for your comments!

1

Pinning a userspace to a specific processor sometimes can make it perform more consistently; but not the kernel. (check taskset)

if it were possible, you would be turning the userspace/kernel switchtme (already a measurable factor in an OS performance) into an inter-processor communication / synchronization issue. Many orders of magnitude worse.

edit: now that you've removed the 'pin the kernel' idea, it's a lot more reasonable. Yes, you can use taskset to start all grid processes and leave one or two CPUs free for GlusterFS. For an analogous example, in Xen systems is considered a 'best practice' to reserve one CPU for the Dom0, which handles all I/O.

  • Thanks for your input. What I'm afraid of is having the 16 "processors" used by grid jobs, leaving no cpu for GlusterFS and the I/O. Right now, I think I'll set a limit of 15 jobs per exec nodes, hoping that 1 processor will be fine for GlusterFS and the I/O. – Philippe Mar 31 '10 at 14:33
1

That's why I don't want to use FUSE filesystems in production... I'm using PVFS2 instead.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.