So you want to be a sysadmin?
Speaker | Joh Clarke |
---|---|
Time | 2008-01-29 10:30 |
Conference | LCA2008 |
Started playing around with computers with siblings. Never really stopped.
Debian based distribution preferred.
Sysadmin
Operates and maintains computers and/or network.
Wide range of duties, roles, and technologies.
See Wikipedia entry for System administrator:
- audits of systems and software
- backups
- answering technical queries
- security
- documentation - normally poorly done
- trouble shooting, problem solving
Not what you know, but being able to solve problems as you encounter them.
Being able to plan for your stupidity, other peoples stupidity, and being able to solve the problems that occur.
Usually with horror stories is how the problem was fixed.
All tools are open source.
Experiment. Preferably on virtual machine, so if something goes wrong, it is easy to rebuild it.
Backups
- stuff goes wrong, normally human error. Sometimes hardware failure.
- dirvish - is a wrapper for rsync, can do snapshots of disk
- Amanda - mostly used for backup to tape, but can backup to disk as virtual tape.
- Bacula - better protocol
- dar - complete rewrite of tar for disk archiving.
- test restores.
- can’t backup a live database; backup database dumps.
Monitoring
- When stuff goes wrong, somebody has to know about it.
- Chronic vs acute problems. Is problem gradually increasing with age, or has it occurred suddenly?
- cacti - web front end to rrdtool, pretty graphs, with colour.
- nagios - can write plugins, bash or perl, 0 for OK, 2 not OK. Notify by jabber, IRC, pager, SMS, etc. SMS not reliable, pager more reliable. Pager annoying. For reliability, pager every 15 minutes, until somebody acknowledges or fixes it. Not very intuitive. Does not control the Internet. Use different settings for remote hosts, to prevent false alarms. Generally just works.
Security
Make everything as simple as possible, don’t give any more rights then they need.
Use sudo. /etc/sudoers is highly configurable.
Use ssh-keys, two factor authentication.
Don’t allow remote logins as root.
- Use sudo instead, create an audit trail.
Don’t use telnet or FTP.
Firewalls
fwbuilder, with change control
- Closed by default
- Open to allow legitimate traffic.
- Explicitly allow traffic from management network, don’t look yourself out of remote computer. Rusty Roulette. If screw up, pay with chocolate.
Use version control, with notes.
Package management
Official repositories only
Use distribution name (etch) instead of (stable).
Use default packages if possible. Otherwise, no security updates.
Scripting
bash is very powerful.
- The more you script, the better you learn your shell.
- “looks like your keyboard just vomited”
- “Escape - Shift 3” - put command in history without executing it.
- database backups
perl
- any bash script longer then 6 lines should be Perl or python.
- Perl is not self documenting, and can become a mess.
Network diagnostics
ping and telnet for network discovery
tcpdump, mtr, netcat, hping3, socat
Wireshark
- previously known as Ethereal.
- can read output of tcpdump.
- “I felt absolutely smug until I didn’t something stupid.”
mtrtiny
Useful things
Live CDs
- Ubuntu, Knoppix
- Used for hacking grub or chrooting into semi-broken systems.
- Requires a working CDROM drive.
- Try to match OS on CD and computer.
CSSH
- CluserSSH
elinks
- text based browser with mouse support
- Wikipedia looks a lot prettier
minicom
debootstrap
- create system from scratch
- can then chroot into bootstrapped system
Because I said so!
Label cables at both ends.
No such thing as a temporary fix.
Documentation - do it - even if I don’t.
- You were doing this last, now it is broken, fix it.
backup private keys, on USB somewhere else.
- Use a pass phrase
Test your configuration
- Apache is evil and breaks easily
Don’t split /usr
Run “John the Ripper” and “nmap” against own network.
Do no send passwords in plain text.
Bash timeouts, so ex-employees don’t remain logged n.
Disaster recovery, doesn’t work in case of a disaster.
Female administrators
Not many female administrators.
- Female toilets used for storage area.
- Boss didn’t think female capable of lifting large heavy computers.