25
votes

Every now and then on this site there are people asking what are some sys admin interview questions. Mostly when reading them they are all junior to mid-level questions.

I'm wondering what are your best senior level Linux admin interview questions.

Two of mine are

1) How do you stop a fork bomb if you are already logged into a system

2) You delete a log file that apache is using and did not restart apache yet, how can you recover that log file?

1
  • Another variant of the first Q is: What does :(){ :|:& };: do on your system and why you would care about that? Cheers!
    – MacUsers
    Apr 20, 2012 at 14:17

12 Answers 12

13
votes
  • How look shared library dependencies?
  • How trace system call and signal?
  • How profile app?
  • How print the strings of printable characters in files?
  • What fields are stored in an inode?
  • What is nscd?
  • What is Automake and Autoconf?
  • What steps to add a user to a system without using useradd/adduser?
  • How see look information about ELF files?
  • What is MAJOR and MINOR numbers of special files?
  • How link layer filtering?
16
votes
  • Tell me about the last major Linux project you finished. What were some of the obstacles and how did you overcome them. Sometimes asking these open questions will reveal much more then small questions that are easily answered by Google. A great senior admin doesn't need to know everything, but they should be able to come up with amazing solutions to impossible projects.
  • Do you contribute to any Open Source projects? Doesn't matter if it's just documentation, it shows a certain dedication to the craft.

To figure out how good they are, ask the open ended questions...

2
  • 2
    +1 for open ended questions. Closed questions boil down to knowing trivia.
    – Sirex
    Jan 24, 2011 at 8:26
  • 2
    Absolutely. To me a senior is not just someone who can memorise stuff you can Google, but someone who can solve complex problems, design, take a leadership role in projects, and so on. "Can you remember how to use ldd" is not going to tell me if I'm hiring someone who can help pinpoint the flaw in a complex multi-tier production app, or do meaningful capacity planning.
    – Rodger
    Jan 26, 2011 at 10:24
9
votes

Question: Describe a scenario when you get a "filesystem is full" error, but 'df' shows there is free space

Answer: The filesystem can run out of inodes, 'df -i' will show that.

Open ended questions:

  • tell me about how do you manage your knowledge database (wikis, outlines, spreadsheets, plain text files) and why did you choose particular options
  • do you use version control? which one and why? are your commit messages good looking?
4
  • 3
    Related might be, "Why would du and df disagree?" - Because du checks usage of directories, but df checks free'd inodes, and files can be held open and take space after they're deleted Jan 24, 2011 at 4:06
  • +1 for this & Matt's comment (Actually I ask this on ALL unix interviews, regardless of level. It shows how well someone understands filesystem concepts)
    – voretaq7
    Jan 24, 2011 at 5:40
  • Ohh.. the difference between du / and df is always a good one
    – Mike
    Jan 24, 2011 at 11:03
  • makes a note Aha! Now I know this one! :) Jan 24, 2011 at 11:47
8
votes

One I like to ask, and I'm comfortable being asked, is this:

Tell me about the biggest mistake you've made in [some recent time period] and how you would do it differently today. What did you learn from this experience?

There are a lot of sysadmins with 10 years experience. Many seem to have the same year 10 times in a row. I want one on my team with 10 different, progressively better years. And if you've never made a big mistake, one that made you wonder if you should just pack your desk up, you haven't truly lived. :)

I don't care for the trivia questions, as interviewer or interviewee. I do like the questions that have stumped my vendors before, or the ones that gave me "Eureka!" moments.

1
  • i support this question
    – ZaphodB
    Feb 6, 2011 at 9:21
7
votes

I'm always a fan of

  • Describe the linux boot process with as much detail as possible, starting from when the system is powered on and ending when you get a prompt.

I like to ask questions that demonstrate how much awareness someone has of the differences between different unixes. I also like to see how much a person understands that linux does things one way which is not necessarily how everyone else does it. Many of these questions revolve around default linux tools, for example:

  • Tell me two ways to redirect both stderr and stdin at once
    • &> and >/dev/null 2>&1
    • demonstrates knowledge of bash vs. bourne shell

Another key point: for senior positions, I expect to see evidence of public writing and presentation skills. If you are a senior person you should have been published multiple times on the web and in magazines. Heck, just having your own tech blog is sufficient. Having presented at least one paper at a conference is a big plus. I think this in particular separates junior from senior linux admins.

1
  • My variant of your question is: From the end of the POST until you see a login prompt, please describe the boot process of a PC server running Linux in as much detail as you feel is appropriate, with an emphasis on troubleshooting? (Thus things about how the boot device is found by the BIOS and how the bootloader is fetched by code in the MBR, how the kernel starts the init process and so on, are all relevant; because each of these gives feedback and has associated troubleshooting steps).
    – Jim Dennis
    Jan 24, 2011 at 5:03
6
votes

I just ran 'chmod -x /bin/chmod'. What did I do? How do I recover?

Describe TCP's handshake process.

How does traceroute work?

When might you need to use CTRL-Z or CTRL-D?

What does the sticky bit do?

What kernel options might you need to tune?

How do you tell what distribution you're running?

How do you tell what hardware you're running on?

What is the difference between a SAN, a NAS and local disk?

I have 30 servers and I'm not sure if each has the same apache config. How do I find out how many copies there are and what the differences are?

What's a chroot jail?

How do you tell if you've been hacked?

Name all the two letter unix commands you can think of and what they do? How you could look up all the two letter unix commands on your system.

I also ask folks pre-onsite to do some simple homework. Especially if they profess some experience writing scripts. I ask them to parse two different data files (one csv, one |sv) and effectively do a join of the data. I put in a few gotchas (poorly formatted data), missing fields needed for joins, strangely out of sequence data, etc.. I ask them to send me the source code and output. (Machine Problem style) I usually give a 3-4 hour time window to tackle it. This task has helped filter out lots of candidates who list 'scripting' as a skill, but can't pull off basics when pressured.

2
  • I'm not convinced these are technical enough for a senior role (based on that i can answer them, and i'm a junior).
    – Sirex
    Jan 24, 2011 at 8:30
  • With some time, I could answer all of these, but I want to be able to do it in my sleep! Me Thinks I need more learning :) Jan 24, 2011 at 11:45
6
votes

My favorite closing question:

If you were stuck on a desert island with only 5 command-line utilities, which would you choose?

My personal list:

  • ping
  • tcpdump
  • ps
  • lsof
  • strace

It's a simple question to see what commands an admin is most comfortable with, and it also helps to demonstrate their comfort with unorthodox (playful) questions. Some interviewees read way too much into the question and get flustered. If you can't answer something this basic it tells me you have little confidence in your own skills. On the other hand, if you try too hard to come up with the perfect list, it might hint that you're afraid of failure and unable to think on your feet.

Admittedly, this is more useful for junior/mid-level applicants, but it doesn't hurt to ask either. :)

3
  • 5
    I guess I'd go with busybox to cover the basics, plus mplayer to help deal with the boredom. That leaves me three leftovers....
    – mattdm
    Jan 24, 2011 at 4:28
  • 2
    I'd just want echo and mail - echo 'please come and rescue me!' | mail -s rescue [email protected]
    – symcbean
    Jan 24, 2011 at 9:54
  • "Mail" and "ntpdate". I could use Mail to call for help, and use NTP to determine my exact lattitude and longitude. :)
    – Ernie
    Jul 8, 2015 at 20:46
5
votes

I use this one :

  • What's happening when the Linux kernel is starting the OOM killer, how does it choose which process to kill first.

and others which I don't have from the top of my head..

5
  • That is a great question.. I don't think a lot of senior level people would know that. I certainly did not and it got me reading on it. thanks!
    – Mike
    Jan 24, 2011 at 0:39
  • My answer to this would be: the kernel maintains various statistics related to virtual memory allocation and paging activity. When certain thresholds are exceeded then the OOM killer is invoked to select processes for termination. The details have been tweaked and changed with just about every kernel release and vendor kernel build for the last several years.
    – Jim Dennis
    Jan 24, 2011 at 5:07
  • 1
    (My overall point is that the normal sysadmin should consider OOM killer process selection to be almost random. A couple years ago I spent an embarrassing amount of time tracking down the mysterious sporadic deaths on my portmapper processes (a few per week across clusters of a few thousand machine) only discover that it was being killed by OOM killer --- despite the fact that it uses very little memory, almost never allocates additional memory, never does any forking and is, in most respects, an extremely unlikely candidate for OOM selection). (No sysctl tweaks had been applied to these).
    – Jim Dennis
    Jan 24, 2011 at 5:13
  • My brain just exploded - These are VERY very good questions! Jan 24, 2011 at 11:48
  • Jim, you are right, it does change all the time but I am more interested about how the interviewee would start thinking about it and what would make more sense to kill first and bonus if he can mention how to protect certain process to being killed. Jan 24, 2011 at 12:36
4
votes

I have a strong favorite here. This question has excellent predictive value for how candidates will fare in more advanced questions:

A user/colleague comes to you to complain that a zero-length file named "-fr" has appeared in the root directory of some system.

What is the easiest, safest and most portable way to remove that directory entry, and why is the question worth asking?

People get half credit for saying things like: rm -- -fr or perl -le 'unlink("-fr");' (They're effective but not optimally portable).

People who go on about shell command line quoting and character escaping are almost as dangerous as those who simply don't even recognize why a file name like that poses any problem at all.

Those who chuckle ... say anything about using "dot slash" and point out that this is one of the oldest entries in the UNIX shell FAQs from USENIX get extra credit.

5
  • What you mean by "most portable"? My answer is, anyway: "delete by inode number". find . -inum <number> -exec rm -i {} \;
    – MacUsers
    Apr 20, 2012 at 14:00
  • @MacUsers: In my book that's about the same as using Perl for the task (half credit ... it will work, but some versions of find might not implement the -inum option and it's unnecessarily complicated compared to rm ./-fr (not "easiest"). In particular we can measure easier in terms of how much effort and how reliably we can walk a user through the process over the phone.
    – Jim Dennis
    Apr 20, 2012 at 21:30
  • Like to know your answer to get your "full credit". Cheers!!
    – MacUsers
    Apr 23, 2012 at 9:43
  • The simplest and most portable method is: rm ./-fr ... it's always worked from the oldest versions of UNIX and it only requires the insertion of two, un-shifted keystrokes. More importantly one can prefix any glob pattern with ./ and be assured that no shell expansion of the glob will be misinterpreted as a switch by some external command. (A program would have to concatenate its args together and re-parse the string even to be confused by ./foo\ -bar ... a file with a space-dash sequence in it).
    – Jim Dennis
    Apr 24, 2012 at 7:35
  • There seems to be a ' missing from your perl unlink example.
    – GnP
    Jul 7, 2015 at 2:16
3
votes

"What unixes that aren't linux have you used? Could you tell me some of the differences between them?"

Because, after all, not the whole world is a linux (I've used a handful of non-linux commercial unices and a few non-linux open ones, if you rely on skills from one in the environment of another, you WILL shoot yourself in the foot).

2
  • 4
    Solaris killall != Linux killall
    – user9517
    Jan 24, 2011 at 13:29
  • It at least used to be the case that editing config files in AIX did not result in daemons changing behaviour, as they picked up the config from a binary database, instead of the plain-text config file.
    – Vatine
    Jan 25, 2011 at 10:07
2
votes

If you were stuck on a desert island with only 5 command-line utilities, which would you choose?

date whoami echo sleep kill

3
1
vote

I'll say that I feel like many of the "where is __, what is __, how do you __" types of questions feel like junior/intermediate questions to me. When I'm interviewing for an intermediate level Linux position, I ask the typical base knowledge stuff (what's a kernel for, what's the difference between a symlink and a hard link, what do I do if I remove the executable bits from chmod?). Some of the stuff I would think of as being more senior level questions:

  1. What are some of the benefits of using systemd over SysV init? Drawbacks? Debian just moved to systemd as opposed to Upstart. What was the reasoning behind that?
    • One of the things this shows is interest and investment in the Linux community and how this impacts a sysadmin's everyday work. It shows that this person is informed about how the field is evolving and is (potentially) willing to change with it.
  2. If I wanted to install Apache on a remote server, what's the best way to do that? What if I wanted to do the same thing on 100 servers? What if half of them are RHEL and half are Debian?
    • Shows a familiarity with using third party software to solve practical problems; familiarity with leveraging configuration management utilities.
  3. What considerations come into play when designing a highly available application, both at the architecture level and the application level?
    • Shows this person can consider a big picture problem. Shows that they understand how load balancing works, stateless applications, maybe displays some knowledge about common Apache configuration options and modules used for multithreaded applications and performance optimization at the level of the individual server.
  4. What are some differences between a microkernel and a monolithic kernel? What is advantageous about Linux being a monolithic kernel?
    • Shows familiarity with the Linux system at a deeper level than user space.
  5. If I type "www.google.com" into by browser and hit enter, what's everything that happens from the time my key bottoms out to the time the page renders in my browser?
    • Even though the position isn't for a network engineer, I expect people to be able to think outside the microcosm. This gets into a lot of networking fundamentals and their significance for individual systems and small networks in terms of how they fit into larger systems. This can address local browser/OS DNS caches, how ARP works, how route tables work, what a recursive DNS query is, the significance of BGP/OSPF and the role they play in larger networks, the SSL handshake and cryptography in general, TCP connections, and gobs of other useful little tidbits to delve into.

These are just some offhand examples, but I think in general a senior admin should be familiar with big picture concepts and rather than things you can just Google for.

Not the answer you're looking for? Browse other questions tagged .