The document presents an overview of sysadmin and DevOps practices, emphasizing the importance of command-line skills, configuration management, and server security. It discusses essential directory structures, software installation methods, user authentication, and tools for monitoring and managing servers. Additionally, it covers scripting with languages such as Bash and Python, as well as configuration management with tools like Ansible and Puppet.

1. Your Inner Sysadmin
Chris Tankersley
@dragonmantank
SunshinePHP 2015
SunshinePHP 2015 1

2. SunshinePHP 2015 2

3. Who Am I
• PHP Programmer for over 10 years
• Sysadmin/DevOps for around 8 years
• Using Linux for more than 15 years
• https://github.com/dragonmantank
SunshinePHP 2015 3

4. Here Be Dragons
SunshinePHP 2015 4

5. Traditional Lamp Stack
SunshinePHP 2015 5

6. Our Players
SunshinePHP 2015 6

7. And of course…
SunshinePHP 2015 7

8. The Server
• /bin - Essential user executable files
• /boot - Stuff that makes the OS boot up!
• /dev - Special device stuff you probably won't touch
• /etc - Configuration files
• /home - User home directories
• /sbin - System binaries
• /usr - Multi-user apps and utilities
• /var - Data usually lives here
SunshinePHP 2015 8

9. Installing Software
• Compile software from scratch
• Use the package manager (yum/apt)
SunshinePHP 2015 9

10. Learn to love the Command Line
SunshinePHP 2015 10

11. Learn a CLI text editor
• vi/vim
• emacs
• nano
SunshinePHP 2015 11

12. Authentication and Authorization
SunshinePHP 2015 12

13. sudo
You can give admin access to users (or groups of users) without giving
them root.
SunshinePHP 2015
13
# Add sudo access to a single user to run as root
dragonmantank ALL=(ALL) ALL
# Add sudo access to a full group
%admin ALL=(ALL) ALL
You can even restrict what commands the users can run
# Restrict web developers to only restart Apache and MySQL
%webdevs 192.168.1.0/255.255.225.0=(root) NOPASSWD:/usr/sbin/service apache2
restart, /usr/sbin/service mysql restart

14. Jailing Users
Keeps people from getting to things they shouldn't. Protects the users
from themselves.
SunshinePHP 2015 14

15. Jailed Shells
Gives users a full shell but not the entire file system. You can pick and
choose what programs the user can have access too. Jailkit makes this
incredibly easy to set up.
SunshinePHP 2015 15

16. Jailed SFTP
Locks the user to a specific base path, but doesn’t give them a shell,
much like FTP. You get the security of SSH though! It does require a
system user however.
SunshinePHP 2015 16

17. Jailing SFTP
# In /etc/ssh/sshd_config
Subsystem ftp sftp-internal
# At the bottom of the file
Match User jailedsftp
ChrootDirectory /some/path
AllowTCPForwarding no
X11Forwarding no
ForceCommand internal-sftp
SunshinePHP 2015 17

18. Docker
SunshinePHP 2015 18
If you do it the non-Docker way

19. Scripting Languages
SunshinePHP 2015 19

20. Bash
Most servers use bash as the default shell. Most shells understand
bash's syntax. If you find yourself running the same commands over
and over, throw it in a bash script.
SunshinePHP 2015 20

21. Python
Ships with most distros. Great for
when you need more power than
what bash has.
SunshinePHP 2015 21

22. PHP!
Leverage your PHP skills to write shell scripts.
• Symfony Console Component
• Aura CLI
SunshinePHP 2015 22

23. Locking Down your Code
SunshinePHP 2015 23

24. Running Apache as a different user
MPM-ITK
SunshinePHP 2015 24
MOD_RUID2
<IfModule mpm_itk_module>
AssignUserId [user] [user]
</IfModule>
RMode config
RUidGid myuser mygroup
RDocumentChRoot /var/www/vhosts/domain.com/
www/public

25. PHP-FPM
user = myuser
group = mygroup
chroot = /path/to/my/chroot
SunshinePHP 2015 25

26. Logs
SunshinePHP 2015 26

27. Logrotate
Rotates logs out for organization (or other purposes)
SunshinePHP 2015 27
weekly
rotate 4
create
include /etc/logrotate.d
/var/log/wtmp {
monthly
minsize 1M
create 0664 root utmp
rotate 1
}

28. Logwatch
Script that runs every so often and scans a bunch of logs so you get a
pretty e-mail with a summary of events
SunshinePHP 2015 28
--------------------- httpd Begin ------------------------
0.17 MB transferred in 792 responses (1xx 0, 2xx 786, 3xx 0, 4xx 6, 5xx 0)
199 Content pages (0.09 MB),
593 Other (0.09 MB)
Requests with error response codes
400 Bad Request
/w00tw00t.at.ISC.SANS.DFind:): 1 Time(s)
404 Not Found
/MyAdmin/scripts/setup.php: 1 Time(s)
/phpmyadmin/scripts/setup.php: 1 Time(s)
/w00tw00t.at.blackhats.romanian.anti-sec:): 1 Time(s)
/webdav/: 2 Time(s)
---------------------- httpd End -------------------------

29. OSSEC
Actually a Host Intrusion Detection system, but it does this by watching
logs. Will alert you immediately to problems, and even shut down the
attacks.
SunshinePHP 2015 29
OSSEC HIDS Notification.
2012 Oct 24 11:38:10
Received From: maple->/var/log/auth.log
Rule: 5712 fired (level 10) -> "SSHD brute force trying to get access to the system."
Portion of the log(s):
Oct 24 11:38:09 maple sshd[1062]: Failed password for invalid user alias from
199.167.138.44 port 59988 ssh2
Oct 24 11:38:07 maple sshd[1062]: Invalid user alias from 199.167.138.44
Oct 24 11:38:06 maple sshd[1059]: Failed password for invalid user recruit from
199.167.138.44 port 59884 ssh2

30. Preventing Intruders
SunshinePHP 2015 30

31. hosts.deny and hosts.allow
Set of files to allow or deny access to the machine or certain apps/ports
on the machine
SunshinePHP 2015 31

32. IPTables
A firewall that is generally available on Linux machines that can be
configured many different ways to allow or block or mangle traffic
SunshinePHP 2015 32

33. OSSEC
IDS that was logs and will use hosts.deny and iptables to block stuff
automatically for you!
SunshinePHP 2015 33

34. Configuration Management
SunshinePHP 2015 34

35. What is Configuration Management?
Process by which you figure out what goes on your servers, how you
want them set up, and keeping track of that information. Files are
usually stored in source control on one server and pushed to clients.
SunshinePHP 2015 35

36. Why do you need it?
• Ever needed to keep track of when files get changed?
• Ever needed to roll back a change?
• Ever needed to push the same change to a bunch of servers
• Ever needed to set up a server exactly the same way as another
server?
SunshinePHP 2015 36

37. General CM Workflow
SunshinePHP 2015 37
Write a Manifest file
Client checks and compiles
the manifests
Client makes changes
based on manifests

38. Ansible
• https://serversforhackers.com/getting-started-with-ansible/
SunshinePHP 2015 38

39. Puppet
• http://www.erikaheidi.com/page/vagrant
SunshinePHP 2015 39

40. Server Monitoring
SunshinePHP 2015 40

41. Quick Poll
• Who here knows that their server is up right now?
• Are all of the required services running?
• Are there enough resources currently available?
SunshinePHP 2015 41

42. Service Monitoring with Monit
SunshinePHP 2015 42

43. Host Monitoring with Icinga
SunshinePHP 2015 43

44. Software Tools
SunshinePHP 2015 44

45. tmux/screen
Command line multiplexer
SunshinePHP 2015 45

46. tail
Look at the newest entries in a log, or even watch log files as they are
generated
SunshinePHP 2015 46

47. curl
Command line program for transferring data via a URL
SunshinePHP 2015 47

48. iftop
Displays a breakdown of bandwidth usage by host
SunshinePHP 2015 48

49. htop
Slightly better interface for checking memory and CPU usage
SunshinePHP 2015 49

50. tcpdump
Allows you to view and record data transmitted over the network.
Couple this with wireshark and you can inspect the packets!
SunshinePHP 2015 50

51. Servers for Hackers
Chris Fidao
@fideloper
http://serversforhackers.com
SunshinePHP 2015 51

52. Questions?
SunshinePHP 2015 52

53. Thank You!
http://ctankersley.com
chris@ctankersley.com
@dragonmantank
https://joind.in/13421
SunshinePHP 2015 53