Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Every now and then on this site there are people asking what are some sys admin interview questions. Mostly when reading them they are all junior to mid-level questions.

I'm wondering what are your best senior level Linux admin interview questions.

Two of mine are

1) How do you stop a fork bomb if you are already logged into a system

2) You delete a log file that apache is using and did not restart apache yet, how can you recover that log file?

share|improve this question
    
Another variant of the first Q is: What does :(){ :|:& };: do on your system and why you would care about that? Cheers! –  MacUsers Apr 20 '12 at 14:17
add comment

11 Answers

up vote 10 down vote accepted
  • How look shared library dependencies?
  • How trace system call and signal?
  • How profile app?
  • How print the strings of printable characters in files?
  • What fields are stored in an inode?
  • What is nscd?
  • What is Automake and Autoconf?
  • What steps to add a user to a system without using useradd/adduser?
  • How see look information about ELF files?
  • What is MAJOR and MINOR numbers of special files?
  • How link layer filtering?
share|improve this answer
add comment
  • Tell me about the last major Linux project you finished. What were some of the obstacles and how did you overcome them. Sometimes asking these open questions will reveal much more then small questions that are easily answered by Google. A great senior admin doesn't need to know everything, but they should be able to come up with amazing solutions to impossible projects.
  • Do you contribute to any Open Source projects? Doesn't matter if it's just documentation, it shows a certain dedication to the craft.

To figure out how good they are, ask the open ended questions...

share|improve this answer
    
+1 for open ended questions. Closed questions boil down to knowing trivia. –  Sirex Jan 24 '11 at 8:26
2  
Absolutely. To me a senior is not just someone who can memorise stuff you can Google, but someone who can solve complex problems, design, take a leadership role in projects, and so on. "Can you remember how to use ldd" is not going to tell me if I'm hiring someone who can help pinpoint the flaw in a complex multi-tier production app, or do meaningful capacity planning. –  Rodger Jan 26 '11 at 10:24
add comment

Question: Describe a scenario when you get a "filesystem is full" error, but 'df' shows there is free space

Answer: The filesystem can run out of inodes, 'df -i' will show that.

Open ended questions:

  • tell me about how do you manage your knowledge database (wikis, outlines, spreadsheets, plain text files) and why did you choose particular options
  • do you use version control? which one and why? are your commit messages good looking?
share|improve this answer
3  
Related might be, "Why would du and df disagree?" - Because du checks usage of directories, but df checks free'd inodes, and files can be held open and take space after they're deleted –  Matt Simmons Jan 24 '11 at 4:06
    
+1 for this & Matt's comment (Actually I ask this on ALL unix interviews, regardless of level. It shows how well someone understands filesystem concepts) –  voretaq7 Jan 24 '11 at 5:40
    
Ohh.. the difference between du / and df is always a good one –  Mike Jan 24 '11 at 11:03
    
makes a note Aha! Now I know this one! :) –  Mister IT Guru Jan 24 '11 at 11:47
add comment

I'm always a fan of

  • Describe the linux boot process with as much detail as possible, starting from when the system is powered on and ending when you get a prompt.

I like to ask questions that demonstrate how much awareness someone has of the differences between different unixes. I also like to see how much a person understands that linux does things one way which is not necessarily how everyone else does it. Many of these questions revolve around default linux tools, for example:

  • Tell me two ways to redirect both stderr and stdin at once
    • &> and >/dev/null 2>&1
    • demonstrates knowledge of bash vs. bourne shell

Another key point: for senior positions, I expect to see evidence of public writing and presentation skills. If you are a senior person you should have been published multiple times on the web and in magazines. Heck, just having your own tech blog is sufficient. Having presented at least one paper at a conference is a big plus. I think this in particular separates junior from senior linux admins.

share|improve this answer
    
My variant of your question is: From the end of the POST until you see a login prompt, please describe the boot process of a PC server running Linux in as much detail as you feel is appropriate, with an emphasis on troubleshooting? (Thus things about how the boot device is found by the BIOS and how the bootloader is fetched by code in the MBR, how the kernel starts the init process and so on, are all relevant; because each of these gives feedback and has associated troubleshooting steps). –  Jim Dennis Jan 24 '11 at 5:03
add comment

I just ran 'chmod -x /bin/chmod'. What did I do? How do I recover?

Describe TCP's handshake process.

How does traceroute work?

When might you need to use CTRL-Z or CTRL-D?

What does the sticky bit do?

What kernel options might you need to tune?

How do you tell what distribution you're running?

How do you tell what hardware you're running on?

What is the difference between a SAN, a NAS and local disk?

I have 30 servers and I'm not sure if each has the same apache config. How do I find out how many copies there are and what the differences are?

What's a chroot jail?

How do you tell if you've been hacked?

Name all the two letter unix commands you can think of and what they do? How you could look up all the two letter unix commands on your system.

I also ask folks pre-onsite to do some simple homework. Especially if they profess some experience writing scripts. I ask them to parse two different data files (one csv, one |sv) and effectively do a join of the data. I put in a few gotchas (poorly formatted data), missing fields needed for joins, strangely out of sequence data, etc.. I ask them to send me the source code and output. (Machine Problem style) I usually give a 3-4 hour time window to tackle it. This task has helped filter out lots of candidates who list 'scripting' as a skill, but can't pull off basics when pressured.

share|improve this answer
    
I'm not convinced these are technical enough for a senior role (based on that i can answer them, and i'm a junior). –  Sirex Jan 24 '11 at 8:30
    
With some time, I could answer all of these, but I want to be able to do it in my sleep! Me Thinks I need more learning :) –  Mister IT Guru Jan 24 '11 at 11:45
add comment

My favorite closing question:

If you were stuck on a desert island with only 5 command-line utilities, which would you choose?

My personal list:

  • ping
  • tcpdump
  • ps
  • lsof
  • strace

It's a simple question to see what commands an admin is most comfortable with, and it also helps to demonstrate their comfort with unorthodox (playful) questions. Some interviewees read way too much into the question and get flustered. If you can't answer something this basic it tells me you have little confidence in your own skills. On the other hand, if you try too hard to come up with the perfect list, it might hint that you're afraid of failure and unable to think on your feet.

Admittedly, this is more useful for junior/mid-level applicants, but it doesn't hurt to ask either. :)

share|improve this answer
4  
I guess I'd go with busybox to cover the basics, plus mplayer to help deal with the boredom. That leaves me three leftovers.... –  mattdm Jan 24 '11 at 4:28
2  
I'd just want echo and mail - echo 'please come and rescue me!' | mail -s rescue admin@mcga.gov.uk –  symcbean Jan 24 '11 at 9:54
add comment

One I like to ask, and I'm comfortable being asked, is this:

Tell me about the biggest mistake you've made in [some recent time period] and how you would do it differently today. What did you learn from this experience?

There are a lot of sysadmins with 10 years experience. Many seem to have the same year 10 times in a row. I want one on my team with 10 different, progressively better years. And if you've never made a big mistake, one that made you wonder if you should just pack your desk up, you haven't truly lived. :)

I don't care for the trivia questions, as interviewer or interviewee. I do like the questions that have stumped my vendors before, or the ones that gave me "Eureka!" moments.

share|improve this answer
    
i support this question –  ZaphodB Feb 6 '11 at 9:21
add comment

I use this one :

  • What's happening when the Linux kernel is starting the OOM killer, how does it choose which process to kill first.

and others which I don't have from the top of my head..

share|improve this answer
    
That is a great question.. I don't think a lot of senior level people would know that. I certainly did not and it got me reading on it. thanks! –  Mike Jan 24 '11 at 0:39
    
My answer to this would be: the kernel maintains various statistics related to virtual memory allocation and paging activity. When certain thresholds are exceeded then the OOM killer is invoked to select processes for termination. The details have been tweaked and changed with just about every kernel release and vendor kernel build for the last several years. –  Jim Dennis Jan 24 '11 at 5:07
1  
(My overall point is that the normal sysadmin should consider OOM killer process selection to be almost random. A couple years ago I spent an embarrassing amount of time tracking down the mysterious sporadic deaths on my portmapper processes (a few per week across clusters of a few thousand machine) only discover that it was being killed by OOM killer --- despite the fact that it uses very little memory, almost never allocates additional memory, never does any forking and is, in most respects, an extremely unlikely candidate for OOM selection). (No sysctl tweaks had been applied to these). –  Jim Dennis Jan 24 '11 at 5:13
    
My brain just exploded - These are VERY very good questions! –  Mister IT Guru Jan 24 '11 at 11:48
    
Jim, you are right, it does change all the time but I am more interested about how the interviewee would start thinking about it and what would make more sense to kill first and bonus if he can mention how to protect certain process to being killed. –  Chmouel Boudjnah Jan 24 '11 at 12:36
add comment

I have a strong favorite here. This question has excellent predictive value for how candidates will fare in more advanced questions:

A user/colleague comes to you to complain that a zero-length file named "-fr" has appeared in the root directory of some system.

What is the easiest, safest and most portable way to remove that directory entry, and why is the question worth asking?

People get half credit for saying things like: rm -- -fr or perl -le 'unlink("-fr"); (They're effective but not optimally portable).

People who go on about shell command line quoting and character escaping are almost as dangerous as those who simply don't even recognize why a file name like that poses any problem at all.

Those who chuckle ... say anything about using "dot slash" and point out that this is one of the oldest entries in the UNIX shell FAQs from USENIX get extra credit.

share|improve this answer
    
What you mean by "most portable"? My answer is, anyway: "delete by inode number". find . -inum <number> -exec rm -i {} \; –  MacUsers Apr 20 '12 at 14:00
    
@MacUsers: In my book that's about the same as using Perl for the task (half credit ... it will work, but some versions of find might not implement the -inum option and it's unnecessarily complicated compared to rm ./-fr (not "easiest"). In particular we can measure easier in terms of how much effort and how reliably we can walk a user through the process over the phone. –  Jim Dennis Apr 20 '12 at 21:30
    
Like to know your answer to get your "full credit". Cheers!! –  MacUsers Apr 23 '12 at 9:43
    
The simplest and most portable method is: rm ./-fr ... it's always worked from the oldest versions of UNIX and it only requires the insertion of two, un-shifted keystrokes. More importantly one can prefix any glob pattern with ./ and be assured that no shell expansion of the glob will be misinterpreted as a switch by some external command. (A program would have to concatenate its args together and re-parse the string even to be confused by ./foo\ -bar ... a file with a space-dash sequence in it). –  Jim Dennis Apr 24 '12 at 7:35
add comment

If you were stuck on a desert island with only 5 command-line utilities, which would you choose?

date whoami echo sleep kill

share|improve this answer
    
fsck fsck fsck fsck and fsck. –  Sirex Jan 24 '11 at 8:34
    
Is this a dupe, an attempt to turn the answer at serverfault.com/questions/225946/… into a discussion, or what? –  quux Feb 6 '11 at 9:54
    
'whoami' - LOL! –  symcbean Aug 4 '11 at 9:51
add comment

"What unixes that aren't linux have you used? Could you tell me some of the differences between them?"

Because, after all, not the whole world is a linux (I've used a handful of non-linux commercial unices and a few non-linux open ones, if you rely on skills from one in the environment of another, you WILL shoot yourself in the foot).

share|improve this answer
3  
Solaris killall != Linux killall –  Iain Jan 24 '11 at 13:29
    
It at least used to be the case that editing config files in AIX did not result in daemons changing behaviour, as they picked up the config from a binary database, instead of the plain-text config file. –  Vatine Jan 25 '11 at 10:07
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.