Not the worst at all, but probably one I found most amusing. One of my jobs included some sys admin tasks (this wasn't the position, but we all did dev ops), among my other responsibilities. I spent half a day going through everything with the person responsible for most of the admin tasks at the time. She was an extremely dilligent and competent admin, did absolutely everything through configuration management and kept very thorough personal logs and documentation on the entire network. One of my first tasks was to change backup frequency (or other singular change) and going by how I usually did things at the time, just sudid a vi session, changed the frequency and restarted the service.
She found out about it pretty quickly due to having syslog be a constant presence in one of her gnu screen windows and gave me a look. She quickly reverted what I did, updated our config management tool, tested it, then deployed it, while explaining why this was the right way to do things. I slowly came around to doing things the right way and haven't thought much about the initial incident until we found her personal logs that she archived and left on our public network share for future reference.
In the entries for the day that I started, we saw the following two lines:
[*] 2007/09/09 09:58 - yan started. gave sudo privs and initial hire forms.
[*] 2007/09/09 10:45 - revoked yan's sudo privs.
She found out about it pretty quickly due to having syslog be a constant presence in one of her gnu screen windows
I'm amazed that this is possible. How would I set something like that up? A realtime log of only the most significant events of a remote system?
In fact, I'd like to take this opportunity of ignorance-admitting to ask the community for general linux/bsd sysadmin resources. What books should I read, or what topics should I study? I want to become an expert at modern sysops. Modern deployment, hardening, backup, managing dozens of boxen, etc.
I've been thinking of going through any MIT OCW on the subject, but it seems like hard-earned experience might not necessarily translate well to an academic setting. What would you recommend I do?
What position are you starting from? My old workplace was a university group where we (admins) were recruted from the available pool of PhD students. So I'm used to guiding people from "no knowledge" to "enough knowledge to be dangerous". The first step was to force the prospective admins to run a specific system on their "productive machine" and keep it in such conditions that _everything_ works.
This way, a complete admin newbie would learn about digging through the systems by working out the kinks of practical everyday problems. Remember, this is only the most basic instruction, nowhere near enterprise-grade.
If there was a "prospective admin" who had never before run Linux, I'd tell them to install and use Ubuntu/Mint. (Those guys whould usually only be trained to be a helping hand for a "senior" admin.)
If he'd already used Ubuntu at home, I'd tell them to start using Debian and work out how to set up an SSH server and set up their home machine so they could access it remotely.
If they had dabbled with Debian, Fedora, SuSe or something similar, I'd tell them to install Arch and set up some "interesting things", like a mail server or a nis server.
If they were using Arch or Gentoo at home, I'd just personally show them the important things about our system and have them wingman with me for a few days.
If you are already an advanced Linux or BSD user, my approach is of course not appropriate. Instead I'd recommend to pick skills that you want to learn (iptables? Exim?) and set that up. Read Manuals! Read RFCs!
Accurately assessing one's own competence is difficult and makes for boring reading, but since it's probably necessary here, I'll give some background.
If he'd already used Ubuntu at home, I'd tell them to start using Debian and work out how to set up an SSH server and set up their home machine so they could access it remotely.
If they had dabbled with Debian, Fedora, SuSe or something similar, I'd tell them to install Arch and set up some "interesting things", like a mail server or a nis server.
If they were using Arch or Gentoo at home, I'd just personally show them the important things about our system and have them wingman with me for a few days.
I'd say my current skill level is a mixture of those three. For example, I don't know how to deploy a web service which can send out email for users to e.g. reset passwords. So I don't know anything about email. On the other hand, I've been trying to hone my skills by hardening a Debian server using iptables. On my third hand, while I could set up a box at home that can be SSH'd remotely, I'm not yet confident I know all the best practices. I think the best SSH practices are: change the default SSH port, disable root login, and disable password-based login (use a password-protected keyfile instead).
Beyond that, what is interesting to me is being able to set up dozens or hundreds of systems. Doing this by hand is fraught with error, so it seems like I should learn about virtualization + deployment systems. I've heard good things about Ansible and Salt, but I've also heard Salt considered security an afterthought, which didn't sound good.
It's sounding like my best bet is just to try things, but I want to set things up correctly from a security perspective.
I should also enhance my knowledge of networking... perhaps by spending a few weeks on OCW material regarding the networking stack. How packets are routed, the details of TCP, that sort of thing.
You're welcome. If you want to deploy and maintain many machines, then maybe FAI[1] might be worth a look. It allows you to maintain a consistent state over an arbitrary number of machines running a Debian-based distribution, with _and without_ virtualization. We used it to run about 40 user-facing desktop machines and about the same number of cluster nodes. You basically have a central server that contains configuration, configuration-modifying scripts and package configurations. It is possible to define classes of machines, and one machine can belong to multiple classes, so you can have a part of the configuration identical on all machines and then other parts only on some of them.
If they are using gentoo, you should be finding someone else. Gentoo users are typically the most dangerous combination of profoundly ignorant, yet absurdly overconfident in their abilities. Seeing a bunch of autotools and gcc output scroll by does not teach you anything. But the mistaken reputation as an "advanced" distro makes people think that by using gentoo, they are therefore "advanced".
There's something to be said for the installer being a random liveCD and documentation for manually installing & configuring a system.
If you go though the handbook properly (and potentially enough times until you don't need it to install), then the amount of inherent linux usage and admin knowledge you can pick up is just phenomenal -for example I love the xkcd[1] even if it stopped applying when I started using gentoo.
I would expect a gentoo user to be comfortable on the command line, which doesn't hold true for a lot of other desktop users. That said, isn't it immense desktop linux has gotten to the point where the barriers to entry are grandma level low :)
It's also probably fair to say that every userbase has it's vocal idiots...
[1] http://www.xkcd.com/1168/
I have never seen a gentoo users with any more knowledge or experience than ubuntu, mint, mandrake, etc users. They are in fact almost exclusively people who used a "noob" distro, then switched to gentoo to feel "advanced" even though nobody with any unix knowledge would waste their time with gentoo.
No, specifically gentoo users can. The distro literally serves no real purpose, nobody with any unix experience would consider using it. It is quite literally the distro for people who don't know what they are doing, but want to feel "advanced" by watching stuff they don't understand scroll by.
Papertrail is great for this...you can of course setup syslog to route to a central server and just be logged in tmux / screen on that particular machine to read off the stream of logs (I prefer papertrail though + saved searches and hipchat "pings" when saved searches are matched on incoming events).
General devops / sysops/ sysadmin knowledge can be had through a variety of means - I got most of my knowledge from simply reading the FreeBSD manual and making a lot of mistakes with my own servers.
She found out about it pretty quickly due to having syslog be a constant presence in one of her gnu screen windows and gave me a look. She quickly reverted what I did, updated our config management tool, tested it, then deployed it, while explaining why this was the right way to do things. I slowly came around to doing things the right way and haven't thought much about the initial incident until we found her personal logs that she archived and left on our public network share for future reference.
In the entries for the day that I started, we saw the following two lines: