Speaker Joh Clarke
Time 2008-01-29 10:30
Conference LCA2008

Started playing around with computers with siblings. Never really stopped.

Debian based distribution preferred.


Operates and maintains computers and/or network.

Wide range of duties, roles, and technologies.

See Wikipedia entry for System administrator:

  • audits of systems and software
  • backups
  • answering technical queries
  • security
  • documentation - normally poorly done
  • trouble shooting, problem solving

Not what you know, but being able to solve problems as you encounter them.

Being able to plan for your stupidity, other peoples stupidity, and being able to solve the problems that occur.

Usually with horror stories is how the problem was fixed.

All tools are open source.

Experiment. Preferably on virtual machine, so if something goes wrong, it is easy to rebuild it.


  • stuff goes wrong, normally human error. Sometimes hardware failure.
  • dirvish - is a wrapper for rsync, can do snapshots of disk
  • Amanda - mostly used for backup to tape, but can backup to disk as virtual tape.
  • Bacula - better protocol
  • dar - complete rewrite of tar for disk archiving.
  • test restores.
  • can’t backup a live database; backup database dumps.


  • When stuff goes wrong, somebody has to know about it.
  • Chronic vs acute problems. Is problem gradually increasing with age, or has it occurred suddenly?
  • cacti - web front end to rrdtool, pretty graphs, with colour.
  • nagios - can write plugins, bash or perl, 0 for OK, 2 not OK. Notify by jabber, IRC, pager, SMS, etc. SMS not reliable, pager more reliable. Pager annoying. For reliability, pager every 15 minutes, until somebody acknowledges or fixes it. Not very intuitive. Does not control the Internet. Use different settings for remote hosts, to prevent false alarms. Generally just works.


Make everything as simple as possible, don’t give any more rights then they need.

Use sudo. /etc/sudoers is highly configurable.

Use ssh-keys, two factor authentication.

Don’t allow remote logins as root.

  • Use sudo instead, create an audit trail.

Don’t use telnet or FTP.


fwbuilder, with change control

  • Closed by default
  • Open to allow legitimate traffic.
  • Explicitly allow traffic from management network, don’t look yourself out of remote computer. Rusty Roulette. If screw up, pay with chocolate.

Use version control, with notes.

Package management

Official repositories only

Use distribution name (etch) instead of (stable).

Use default packages if possible. Otherwise, no security updates.


bash is very powerful.

  • The more you script, the better you learn your shell.
  • “looks like your keyboard just vomited”
  • “Escape - Shift 3” - put command in history without executing it.
  • database backups


  • any bash script longer then 6 lines should be Perl or python.
  • Perl is not self documenting, and can become a mess.

Network diagnostics

ping and telnet for network discovery

tcpdump, mtr, netcat, hping3, socat


  • previously known as Ethereal.
  • can read output of tcpdump.
  • “I felt absolutely smug until I didn’t something stupid.”


Useful things

Live CDs

  • Ubuntu, Knoppix
  • Used for hacking grub or chrooting into semi-broken systems.
  • Requires a working CDROM drive.
  • Try to match OS on CD and computer.


  • CluserSSH


  • text based browser with mouse support
  • Wikipedia looks a lot prettier



  • create system from scratch
  • can then chroot into bootstrapped system

Because I said so!

Label cables at both ends.

No such thing as a temporary fix.

Documentation - do it - even if I don’t.

  • You were doing this last, now it is broken, fix it.

backup private keys, on USB somewhere else.

  • Use a pass phrase

Test your configuration

  • Apache is evil and breaks easily

Don’t split /usr

Run “John the Ripper” and “nmap” against own network.

Do no send passwords in plain text.

Bash timeouts, so ex-employees don’t remain logged n.

Disaster recovery, doesn’t work in case of a disaster.

Female administrators

Not many female administrators.

  • Female toilets used for storage area.
  • Boss didn’t think female capable of lifting large heavy computers.