I have been a Linux user on and off for many years. Recently, I have been using it daily. I am looking for suggestions for media; commercial books, free information, online videos etc that will help me to understand the OS at a much deeper level. Your suggestions please.
You will find several free guides at The Linux Documentation Project, guides page. This is a short list i pick from there, but you should scroll through the page to find what you need.
- Introduction to Linux - A Hands on Guide; Jun 2008
- Linux on the Road; Nov 2005
- The Linux System Administrators' Guide; July 2005
- Advanced Bash-Scripting Guide; Mar 2009 (yes, this is good for understanding linux too)
There are some starters at the Linux reviews beginners page.
Finally, this is a small book available online -- Linux Kernel in a Nutshell. Its in PDF form of the 2007 edition.
You should use these references only as a feeler to start you own search for things that you need.
I'm not sure, when you say a "deeper level", if you're talking about getting more into the command line (some people never leave the GUI that comes with their distro, if you can imagine that), or with really gritty low-level stuff like writing kernel drivers. I'm going to assume more of the former than the latter.
I'd advise you to set reasonable goals for things you'd like to do using Linux and applications running on Linux (run a web server, serve files to Windows hosts, serve DHCP, run a graphical desktop, etc). It's my firm belief that you only learn when you're "doing". Once you've got a goal in mind, pursue it. That means reading man pages (lots and lots of man pages), "HOWTO" documentation, mailing lists archies, random blog posts, and, of course, the documentation that comes with the various programs you're installing or compiling to run on your boxes.
Having taught classroom-based IT certification courses for several years, I believe I can say with some degree of authority that the students who I saw make the most progress were the ones that were doing crazy projects of their own design, and learning by the seats of their pants.
As I said before, man pages, "HOWTO" documentation (http://tldp.org/docs.html and in many, many other places on the 'net) and mailing list archives are your friends. I'd steer clear of any books that talk about kernel internals, at least at this stage. You don't need that kind of deep knowledge to get started.
Talk to other people. Server Fault looks like it's turning out to be a great place to get good advice. Ask questions here, and don't think that you're going to look stupid doing so. If you can get some "face time" with people who are familiar with Linux, go for it. (Though I'd recommend you try and separate opinion from fact. There are as many "holy wars" in the Linux community over differing opinions of how to do things as in any other community-- perhaps more, given the nature of the community.)
To get really "deep" knowledge of the Unix heritage of Linux, you might go for some older Unix-specific administration or reference manuals. I'd steer clear of these early on (at least until you can appreciate the historical nature of the "paths not taken" with Linux as compared to some of the Unix-derived operating sysems).
Not knowing what your overall skill-level is, I will throw a shout-out to a book that my or may not be heplful. I highly recommend "TCP/IP Illustrated" (http://www.amazon.com/Illustrated-Volumes-Addison-Wesley-Professional-Computing/dp/0201776316). It's not Linux-specific at all, but you will be doing so many things that deal with TCP/IP that knowing it "cold" is a no-brainer.
I guess, in short, I'm saying that there's no magic book or books, no super-secret videos that the "masters" learned from, but absolutely no limit to what you can learn if you stick to it and aren't afraid to get your hands dirty.
Only years of experience playing with individual components will give you a deep understanding. Having said that, the vast majority of Oreilly Press books are really well written and perfect if you're not a dummy and have more than 24 hours. :)
There is a project called Linux From Scratch, which shows you how to build a linux distribution from nothing. You may find it educational to do once. It'll be time consuming as you need to compile everything from source and you'll throw it all away at the end.
I find LWN.net's kernel page invaluable for keeping up with how things work in the kernel at a high level.
Other than using it everyday (Which I think is a great thing!), I recommend you start thinking about services that could be implemented for your network on a linux box. Start designing it based on Linux services, research and implement once your ready. There will be hopefully mistakes in the process, and that experience will help you understand the OS at a deeper level.
For general Unix philosophy and an excellent introduction to the command line, there is Brian Kernighan and Rob Pike's classic The Unix Programming Environment.
Also, IBM's DeveloperWorks web site has over 900 articles in its Linux section. You can browse the article list for topics you find interesting.
Finally, once you're ready to get into the nitty-gritty, go to the source. Grab the source code for the kernel version you're using at kernel.org (your distribution will also have kernel source packages you can install) and check out the Documentation directory. You'll find lots of reference material on kernel internals and configuration settings.
School. To really understand the Linux system you need a wide array of Computer Science systems backgrounds. Compilers, Computer Architecture and Operating Systems. And you need a guide who can show you what's important and correct any misunderstandings you have or form.
Once you understand how modern processors work, and how C compilers exploit them, you can dive into books like Minix 3:
This is the latest version of the book that Torvalds, author of the Linux kernel, (loosely) based his work on. You'll learn about the fundamental components and algorithms of an operating system, and how exactly to implement one: the text comes with a nearly complete printing of the source code to Minix for reference and instructional purposes. Check out the interrupt handler to get a complete understanding of how the system works and where control flows.
And before you scoff at schooling and Computer Science, it's important to note that the jobs Operating Systems are asked to do are generally NP-complete. So understanding a wide variety of available algorithms is critical to performance tuning, since there won't be a provably optimal algorithm.
In addition to the kernel, there's a number of other areas. Because Linux is open source, this is an academic gold mine. Systems like Debian and Ubuntu make their source available, and it's dead simple:
apt-get source package-name
Many upstreams also host their code in revision control, so you can read the most up to date version of a program's source, or even see how it was built years ago.
That, of course, if you mean the kernel.
If you are trying to learn the system above the kernel and the API, I would start at learning old school
Unix. Maybe try some FreeBSD/OpenBSD/NetBSD/DragonflyBSD. A little Minix could help too. Then move to Slackware, Arch, Debian and Gentoo. They are all different (and come with great documentation), and in those diferencies, your curiousity will make you search why it is so, you will gain a lot of technical, comercial, political and historical knowledge.
Another way would be go diggin' in
/etc, old Unix manuals, and of course, Google.
Linux from scratch is a great way of
Learning in any of the *nix environments is a holistic approach. It involves a series of epiphanies and experiences. This is not by accident. It is the deliberate byproduct of its design, as the original environment was designed around and for computer programmers. It is also its greatest shortcoming, as the focus is on technical aspects and not user experience. It is "a house with a sturdy frame that will last 100 years without service, yet its siding is haphazard and the paint clearly neglected".
Contrast this with Windows, which is the mirror image of this philosphy - do something that gives the user a tangible experience, but shield them from the inner workings of things. It is "a house with beautiful trim and paint, but the foundation that has been built-over several times with multiple work-arounds".
To really start learning, I would suggest building custom kernel images that have options specific to your hardware, and installing them with the options you want. You should fully expect going into this that something, somewhere will break, and you may or may not be able to back things out to "normal". This is a normal part of that learning process and you should approach this as if things will break (i.e. don't make this a primary install, use a separate drive or virtual machine or something...)
I've done an answer on a similar question, so to reduce typing I'll crosslink it here. You'll also want to read the first few paragraphs on this page, which will give you a better feel for what you're in for.
Start reading/cat'ing files in
/etc - you won't do any damage by simply reading the files (save maybe screwing up your terminal/ssh-connection if you cat a file that produces binary info), and much of it is human readable. '
sysctl -a' output is also a gold-mine. When you find something interesting, google or
man for more information about it.
You can find some real gems about how the kernel and OS work this way.
A few quick pointers:
/etc/inittab /etc/rcS.d /etc/rc2.d /etc/crontab /proc/1/environ /proc/filesystems /proc/meminfo /proc/cpuinfo sysctl -a | grep vm.swappiness /etc/default # debian-based /etc/sysconfig # redhat-based
Quick tip - some output in /proc is NUL separated, and hence is hard to read. Use 'tr' to convert the NUL's to newlines, eg:
sudo cat /proc/1/environ | tr '\0', '\n'
There is a lot to learn about Linux, or any operating system. One kind of learning is what I call "in depth" learning, which is finding out how the kernel operates, what assumptions it makes, how the various bits talk to each other, and how it deals with hardware. That's kernel stuff. Stuff like that is very useful in figuring out why the operating system broke in just that way. In fact, I learned it the same way Karl Katzke did. I broke stuff, and made it better.
For this kind of knowledge, Linux is pretty easy. It's all documented in many places. The same can't be quite said for Windows, though there is still a lot of doc out there for it.
Then there is the 'getting around in the OS' learning, which is where knowledge of bash/sed/awk/regex and all that other stuff comes in handy. Because the fact is, an operating system is much more than its kernel these days. You have vendor supplied driver blobs. You have how this particular distribution packages its startup scripts. You have various patch/update mechanisms. You have system daemons that everyone needs, but aren't kernel.
This is a much broader spectrum of knowledge than simple kernel-fu. It varies by distributer so how things work on Red Hat may not work the same on Slackware. Where files are kept can vary. As can what they picked to replaced 'vi'.
"Learning Linux" is more about learning an entire ecosystem than it is a simple operating system.
I had a book called "Linux Programming" that taught me more about the OS than any "Learn Linux" book. I wish I had the ISBN number, but it was amazing..partly because it went into the programming "why"s, not not just the "how"s.
Overall, most of my learning was honestly through breaking it and fixing it. It happened slowly, and I'm still learning new things after 12 years.
It's definitely a process. The key is to be very, very curious. When you encounter something that you're unfamiliar with, become familiar with it. Seek out knowledge about how and why it ticks, and learn the underlying technologies, too.
When it comes to books, I like the O'Reilly's, so "Essential System Administration Pocket Reference" and "Linux System Administration" are possible starting points.
My copy of TCP/IP Network Administration is an older one and I haven't used it in a while, but I found it very useful to get a more in-depth understanding of what's going on.
You're already making the first step by using Linux as your regular desktop.
If you want to understand how all the parts hook together and are configured, I would suggest running Gentoo. From a regular stage3 install, you have to compile your own kernel, install all your services (except SSH which is there out of the box), build and configure X yourself, etc. When you're doing that, you end up with a much greater understanding of all the bits that go into a binary linux installation. It also doesn't hurt that Gentoo has the best documentation of any Distro I've found, or that portage is a fantastically powerful and flexible package manager.
I would strongly recommend this as "required reading":
The Unix Programming Environment - Brian W. Kernighan / Rob Pike
Someone else mentioned this, and I think its the thing that helped me most with using various linux flavors. If you feel you have a reasonable handle on linux, start using open/net/freebsd.
As for why, most linux folks I've worked with may use one or two linux distros that aren't all that different from the other. Sure, most of them knew those distros like the back of their hand, but they could still get surprised on an alarming basis with certain things. I think (not to start a flame war) that the way the BSDs are assembled its a little easier to discover the "why" of the way things are done (plus, lurking on the bsd mailing lists they explain that "why" to death). It comes down to changing your point of view, it may never occur to you to ask why certain things are the way they are until someone does it in a completely different manner.
This posting on Stackoverflow has a large listing of Unix/Linux resources including most of the canonical works on the subject.
Another way to learn what makes Linux tick behind the scenes is to work through Linux From Scratch (http://www.linuxfromscratch.org/). It is essentially a set of exercises in manually installing and configuring the various components of a linux system and is a very good intro to learn the nuts and bolts from.