VPS Handbook Vol.2 – VPS Optimization , Security and Hardening
Index
- Introduction
- Terms used ( Node , container , VPS, etc,. )
- Requirement
- Node Optimization & Security
- Container Optimization & Security
- Before Installing VPS Platform
- Optimization
- Security & Hardening
- Secure tmp folders
- SHM Security
- Loop Backup Resolver Security
- Kernel Firmware Security
- Sysctl Security
- Disable Proxy ARP
- Enable Magic Sys Request Key
- Enable the default interface to send redirects
- Enable IP spoofing Protection
- Disable ICMP Redirect acceptance
- Enable Log Spoofed Packets, Source Routed Packets, Redirect Packets
- Disable TCP Time stamps
- Enable TCP synchronize cookie protection
- Enable Ignoring Broadcast requests
- Enable bad error message Protection
- Secure Openssh Servers
- Find all open ports and services
- Compiler Security
- SHELL Fork Bomb Protection
- History Time STAMP
- Identify Local Openports
- Chrootkit Configuration
- Rkhunder
- LogWatch
- Analyze System Users With Empty Passwords
- Locate zero ‘0’ uid owners
- Remove X window Group Ware System Software
- Find all SUID & SGID Binaries
- Find world writable Files
- Find Noowner Files
- Install and configure CLAMD
- Conclusion
Introduction
It has been a long time since my first vps handbook published on Linux Today . So I am writing this new version VPS Hadnbook ,V2 ,for VPS/ Cloud users and System engineers. It will be a good checklist for optimizing and securing a vps node.
The aim of this book is to optimize and harden a VPS node.
Terms used :
The Following technical terms are used in this article,
Node – Represent the Physical / Hardware Node
VPS , Container , Guest – Represent the virtual servers hosted inside the node
Requirement :
This article is based on the Openvz virtualization Platform . You can easily adopt this procedure for most of the Linux VPS/Cloud technologies and Nodes. I used the following environment for doing this task
OS – Centos 6 64 bit
Vps technology – Openvz
The VPS node optimization and Security have two parts as follows,
a) Node Optimization & Security
b) container Optimization & Security
In this article I am referring only to the node optimization and security part. If you have any type of custom application and services running on your vps node ,then you have to think twice before applying the patches and steps mentioned below.
Before Installing VPS Platform:
You need to install and OS along with proper disk partition for building a proper production vps node. These are some tips to you.
a) Slow File Systems : Don’t use software RAID in Nodes. Software is running over the host kernel , so which have to use the CPU and RAM . It is slower than hardware RAID. So I recommend to use Hardware RAID instead of software RAID . Also don’t use LVM partitions. LVM is slower than generic raid ext3/ext4 partitions . But still some control panels prefer this. LVM have the ability to resize , but if your are planning for ultimate performance, then don’t use it. Using the ext4 filesystem is a better choice.
b) Use Faster HDD’s : It is better to use 15K SAS HDD’s or SSDs . The disk I/O problem is a biggest enemy for VPS hosters . So if you chose faster HDD’s then you can add three times the number of containers in a node. These HDD’s are little costly than normal SATA HDD’s , but you can save it from selling more vps from a single node.
c) Chose CPU Wisely : You may need to chose a CPU for VPS node based on the performance and your budget. You need to plan the total cost that you are spending on a vps node and you are going to get i from selling the containers. There is a best place , that you can compare the CPU model. It is from http://www.cpubenchmark.net/
d) Choosing RAM : Well you may need to remember only one thing . Don’t allow the node to use massive amount of SWAP. So if you see using large SWAP , it is a time to do the RAM upgrade. RAM is not so costly . You can add 32 GB or 64 GB RAM according to your need.
Optimization
In this sections we will do some sort of optimization. It is a generic optimization technique that can be usefull for your node.
HDD Optimization :
As I suggested you may need to chose a good Physical HDD initially with a proper disk partition. It is better to use a disk partition as follows for an openvz node.
/ = 20 GB /tmp = 5 GB /vz = Big space for VPS /backup (optional ) for backups
After installing the OS and Openvz . You may need to adjust the above disk partitions mount options in /etc/fstab as follows ,
/dev/sdb3 /vz ext4 defaults,noatime 0 0
The noatime flag will stop disk timestamps and increase the I/O speed
Service Optimization:
You can remove all useless services from the containers. It is better to use the hostnode only for hosting the containers. If you are using the node for apache / mysql / email services ,that will result in performance degrade , because the host node have to share its resources for all these services. It is also better to remove services like FTP , Rlogin ,etc,. from the nodes too.
Removal Of Useless Software:
If you minimize the softwares installed in the node, then you minimize the security vulnerability and increase the performance too. You can list the following commands for finding the installed packages.
# yum list installed # yum list PACKAGE-NAME # yum remove PACKAGE_NAME # rpm -qa # rpm -e PACKAGE-Name
Removal of useless Services:
You can disable all useless services that running in the main node. The services like xfs, cups , bluetooth must be disabled. You can check it as follows,
# chkconfig --list | grep '3:on' crond 0:off 1:off 2:on 3:on 4:on 5:on 6:off ip6tables 0:off 1:off 2:on 3:on 4:on 5:on 6:off iptables 0:off 1:off 2:on 3:on 4:on 5:on 6:off lighttpd 0:off 1:off 2:on 3:on 4:on 5:on 6:off messagebus 0:off 1:off 2:on 3:on 4:on 5:on 6:off network 0:off 1:off 2:on 3:on 4:on 5:on 6:off rsyslog 0:off 1:off 2:on 3:on 4:on 5:on 6:off solusvm-ipsecure 0:off 1:off 2:on 3:on 4:on 5:on 6:off solusvm-netspeed 0:off 1:off 2:on 3:on 4:on 5:on 6:off sshd 0:off 1:off 2:on 3:on 4:on 5:on 6:off sysstat 0:off 1:on 2:on 3:on 4:on 5:on 6:off udev-post 0:off 1:on 2:on 3:on 4:on 5:on 6:off vpsmon-agent 0:off 1:off 2:off 3:on 4:on 5:on 6:off vpsmon-enforce 0:off 1:off 2:off 3:on 4:on 5:on 6:off vpsmon-io 0:off 1:off 2:off 3:on 4:on 5:on 6:off vz 0:off 1:off 2:on 3:on 4:on 5:on 6:off vzeventd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
You can disable a service as follows,
# service stop servicename # chkconfig servvicename off
Kernel Optimization:
You can use an uptodate kernel with a good sysctl.conf parameter for tuning it. It will be discussed deeply in the comming sections.
Uptodate System Software:
It is better to keep the system software always uptodate . You can update this node as follows,
# yum update
Fix System Clock:
It is better to keep the server clock correctly. You can update it using rdate and set a daily cron for it as follows,
# yum install rdate # rdate -s rdate.cpanel.net
Set cron as follows under root,
0 1 * * * /usr/bin/rdate -s rdate.cpanel.net && hwclock --systohc
Optimize the System Iptable:
In an openvps vps , there is always issues with iptables and loaded firewall moduels. Well you can add a set of generic iptable modules in the main node configuration file /etc/vz/vz.conf . You can update it as follows,
## IPv4 iptables kernel modules to be enabled in CTs by default IPTABLES="ipt_REJECT ipt_tos ipt_limit ipt_multiport iptable_filter iptable_mangle ipt_TCPMSS ipt_tcpmss ipt_ttl ipt_length ip_tables iptable_filter ipt_LOG ipt_state ipt_owner iptable_nat ipt_REDIRECT ip_conntrack iptable_nat ip_nat_ftp ip_conntrack_ftp" ## IPv4 iptables kernel modules to be loaded by init.d/vz script IPTABLES_MODULES="$IPTABLES"
The above iptables loaded modules will help your container to avoid issues related with CSF firewall installations and other iptable firewall scripts installations inside the containers.
VE0 CPUUnits:
It is better to increase the cpu cycle from default value 1000 to 10000 at least in /etc/vz/vz.conf .
VE0CPUUNITS=1000
You may need to reboot the vps service . But please note it will result in reboot of all containers. So better schedule a single reboot at the end of all the optimization task.
CPUUNITS Optimizations:
You can see some of the containers will be slow and you are not properly using the cpu cycle from the node. Please run the following command,
# vzcpucheck -v VEID CPUUNITS ------------------------- 0 1000 1002 166666 1003 166666 1004 166666 1007 10000 1001 166666 1006 10000 Current CPU utilization: 687664 Power of the node: 1360012
In the above result , you can see a total utilization and the power of the node. You can allocate more cpu . If you see a 1000 Units in the result for each container , then that is the default value of the containers and your vps containers are not tunned properly as per the node cpu cycle utilization. Adjusting the CPU units will increase stability of the containers and make your clients Happy. But improper adjustment will also result in performance abuse of a single container too. So if yo are planing to optimize the container based on the openvz configuration , it is better to find an expert in this task for doing this job. Syslint Support Team have a lot of hands on experience in this area too.
Change Memory Setting To VSWAP:
The option VSWAP only available for RHEL6 containers. It takes openvz to a next level memory management. Now you can set two parameters with this value RAM and SWAP. You can use vzctl command to set the VSWAP in openvz as follows,
# vzctl set 1000 --swappages 0:1024M --save
See the swap result below,
[root@node ~]# vzctl exec 1000 free -m total used free shared buffers cached Mem: 4096 1814 2281 0 0 1814 -/+ buffers/cache: 0 4096 Swap: 1024 2 1021
Please see more about VSWAP from https://syslint.com//enable-swap-in-openvz-vps/ and http://openvz.org/VSwap
CPULIMIT Tuning:
You can setup proper CPULIMIT for each containers . It is the limite of the CPU usage for the container. The default value will be 0. That means there is no limit in CPU . Suppose if you have 4 CPU cores in the node , then then total cpu limit will be 400%. If you only need to use two core inside a container , you can set the value 200 using the command vzctl
# vzctl set 1000 --cpulimit 200 --save
Limit the number of CPU cores:
You can also limit the number of CPU cores that using inside a vps node. Suppose if you have 64 cores inside a node and you only need to allow 4 cores for a cpu. You can set it as follows,
# vzctl set 1000 --cpu 4 --save
Security & Hardening
Now we can add some basic security and analyze tests in the following sections.
Secure tmp folders:
This is one of the basic security that need to implement in all productions servers. First of all you need to make the following soft links
# rm -fr /var/tmp # ln -s /tmp /var/tmp # rm -rf /usr/tmp # ln -s /tmp /usr/tmp
Now need to secure /tmp . If you a separate HDD partition , then please add the following mount flag options to /tmp
/dev/sdb3 /tmp ext4 defaults,noatime,noexec,nosuid 0 0
Now remount /tmp as follows,
# mount -o remount /tmp
If you don’t have separate HDD partition . Then you can secure it as follows,
Create a raw file of size 5 GB as follows
dd if=/dev/zero of=/usr/tmpfile bs=1M count=5000
Format it using ext4 file system
# mkfs.ext4 /usr/tmpfile
Add the mount entry in /etc/fstab as follows
/usr/tmpfile /tmp ext3 loop,nosuid,noexec,rw 0 0 mount -a
Now set the proper permission for /tmp
chmod 1777 /tmp
SHM Security:
You may need to secure /dev/shm mount options in /etc/fstab as follows,
tmpfs /dev/shm tmpfs defaults,noatime,noexec,nosuid 0 0
Loop Backup Resolver Security :
Most of those vps nodes will be configured as localhost resolver. That is not good. You may need to check the resolver configuration file , /etc/resolv.conf and remove the following lines
nameserver localhost nameserver 127.0.0.1 search localhost
You may need to ask to your server provider or ISP for getting the proper resolver IPs , otherwise you can use the following google resolvers,
nameserver 8.8.8.8 nameserver 8.8.4.4
Kernel Firmware Security:
I have seen a lot of Openvz nodes using the wrong kernel firmware package. You must chose the package “vzkernel-firmware”. You can install it as follows,
# yum install vzkernel-firmware
You can test it as follows too:
# rpm -qa | grep -i vzkernel-firmware
Sysctl Security:
You can secure the /etc/sysctl.conf by adjusting some of the following parameters below,
1) Disabled Proxy ARP:
# Disable proxy arp net.ipv4.conf.default.proxy_arp = 0
Enabled Magic Sys Request Key:
# Enables the magic-sysrq key kernel.sysrq = 1
Enabled the default interface to send redirects:
# We do not want all our interfaces to send redirects net.ipv4.conf.default.send_redirects = 1 net.ipv4.conf.all.send_redirects = 0
Enabled IP spoofing Protection:
# Enable IP spoofing protection, turn on source route verification net.ipv4.conf.all.rp_filter = 1 net.ipv4.conf.lo.rp_filter = 1 net.ipv4.conf.default.rp_filter = 1
Disabled ICMP Redirect acceptance:
# Disable ICMP Redirect Acceptance net.ipv4.conf.all.accept_redirects = 0 net.ipv4.conf.lo.accept_redirects = 0 net.ipv4.conf.default.accept_redirects = 0
Enabled Log Spoofed Packets, Source Routed Packets, Redirect Packets:
# Enable Log Spoofed Packets, Source Routed Packets, Redirect Packets net.ipv4.conf.all.log_martians = 0 net.ipv4.conf.lo.log_martians = 0
7) Disabled TCP Time stamps:
# Turn off the tcp_timestamps net.ipv4.tcp_timestamps = 0
8) Enabled TCP synchronize cookie protection:
# Enable TCP SYN Cookie Protection net.ipv4.tcp_syncookies = 1
Enabled Ignoring Broadcast requests:
# Enable ignoring broadcasts request net.ipv4.icmp_echo_ignore_broadcasts = 1
Enabled bad error message Protection:
# Enable bad error message Protection net.ipv4.icmp_ignore_bogus_error_responses = 1
A sample sysctl.conf file will be as follows after applying all the above settings and some basic security tools,
# Enable packet forwarding enabled net.ipv4.ip_forward = 1 net.ipv6.conf.default.forwarding = 1 net.ipv6.conf.all.forwarding = 1 # Disable proxy arp net.ipv4.conf.default.proxy_arp = 0 # Enables source route verification net.ipv4.conf.all.rp_filter = 1 # Enables the magic-sysrq key kernel.sysrq = 1 # We do not want all our interfaces to send redirects net.ipv4.conf.default.send_redirects = 1 net.ipv4.conf.all.send_redirects = 0 # Enable IP spoofing protection, turn on source route verification net.ipv4.conf.all.rp_filter = 1 net.ipv4.conf.lo.rp_filter = 1 net.ipv4.conf.default.rp_filter = 1 # Disable ICMP Redirect Acceptance net.ipv4.conf.all.accept_redirects = 0 net.ipv4.conf.lo.accept_redirects = 0 net.ipv4.conf.default.accept_redirects = 0 # Enable Log Spoofed Packets, Source Routed Packets, Redirect Packets net.ipv4.conf.all.log_martians = 0 net.ipv4.conf.lo.log_martians = 0 # Decrease the time default value for tcp_fin_timeout connection net.ipv4.tcp_fin_timeout = 15 # Decrease the time default value for tcp_keepalive_time connection net.ipv4.tcp_keepalive_time = 1800 # Turn off the tcp_timestamps net.ipv4.tcp_timestamps = 0 # Enable TCP SYN Cookie Protection net.ipv4.tcp_syncookies = 1 # Enable ignoring broadcasts request net.ipv4.icmp_echo_ignore_broadcasts = 1 # Enable bad error message Protection net.ipv4.icmp_ignore_bogus_error_responses = 1 # Log Spoofed Packets, Source Routed Packets, Redirect Packets net.ipv4.conf.all.log_martians = 1 # Increases the size of the socket queue (effectively, q0). net.ipv4.tcp_max_syn_backlog = 1024 # Increase the tcp-time-wait buckets pool size net.ipv4.tcp_max_tw_buckets = 1440000 #tcp time stamp disable net.ipv4.tcp_timestamps = 0
Secure Openssh Servers :
This is one of the important security changes. You need to disable direct ssh login of root user and the default ssh port too. You may need to edit /etc/ssh/sshd_config and make the following changes,
a) Enable only protocol 2 in sshd
# Protocol 2
b) Limit users to ssh . Only allow some users as follows,
# AllowUsers root syslinttech supportadmin
c) Configure Idle Log Out Timeout Interval
User can login to server via ssh and you can set an idle timeout interval to avoid unattended ssh session. Open sshd_config and make sure following values are configured:
# ClientAliveInterval 300 # ClientAliveCountMax 0
d) Disable Hostbase authentication
# HostbasedAuthentication no
e) Disable rhost
# IgnoreRhosts yes
f) Disable Empty Passwords
# PermitEmptyPasswords no
g) Turn on privilege separation
# UsePrivilegeSeparation yes
h) Prevent the use of insecure home directory and key file permissions
# StrictModes yes
compiler Security:
You can disable all compilers that are using in node.
eg : chmod 700 /usr/bin/gcc
SHELL Fork Bomb Protection:
You can add the following to bashrc file for protecting the local shell users from fork bomb attack.
#unlimit so we can run the whoami ulimit -n 4096 -u 14335 -m unlimited -d unlimited -s 8192 -c 1000000 -v unlimited 2>/dev/null LIMITUSER=$USER if [ -e "/usr/bin/whoami" ]; then LIMITUSER=`/usr/bin/whoami` fi if [ "$LIMITUSER" != "root" ]; then ulimit -n 100 -u 35 -m 200000 -d 200000 -s 8192 -c 200000 -v unlimited 2>/dev/null else ulimit -n 4096 -u 14335 -m unlimited -d unlimited -s 8192 -c 1000000 -v unlimited 2>/dev/null fi
History Time STAMP :
This will help the server administrator to track the VPS node shell operations . Add the following line to bashrc
HISTTIMEFORMAT='%F %T '
Identify Local Openports:
You can find all open ports as follows,
# netstat -tulpn Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.0.2:25 0.0.0.0:* LISTEN 142797/master tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 142797/master tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 101481/sshd tcp 0 0 127.0.0.1:587 0.0.0.0:* LISTEN 238905/sendmail: MT tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN 142409/mysqld tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 340045/httpd tcp 0 0 0.0.0.0:6767 0.0.0.0:* LISTEN 106680/lighttpd tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 649832/named tcp 0 0 127.0.0.1:5557 0.0.0.0:* LISTEN 8824/vpsmon-agent l tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN 649832/named tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN 858317/vsftpd tcp 0 0 ::1:25 :::* LISTEN 142797/master tcp 0 0 :::22 :::* LISTEN 12070/sshd tcp 0 0 :::4925 :::* LISTEN 567946/(squid) tcp 0 0 ::1:53 :::* LISTEN 649832/named tcp 0 0 ::1:953 :::* LISTEN 649832/named tcp 0 0 :::80 :::* LISTEN 12125/httpd udp 0 0 127.0.0.1:53 0.0.0.0:* 649832/named udp 0 0 162.216.155.101:1194 0.0.0.0:* 238790/openvpn udp 0 0 0.0.0.0:35346 0.0.0.0:* 567946/(squid) udp 0 0 199.48.164.93:123 0.0.0.0:* 6698/ntpd udp 0 0 127.0.0.1:123 0.0.0.0:* 6698/ntpd udp 0 0 0.0.0.0:123 0.0.0.0:* 6698/ntpd udp 0 0 :::36810 :::* 567946/(squid) udp 0 0 ::1:53 :::* 649832/named udp 0 0 fe80::2a01ff:feed:e878:123 :::* 6698/ntpd udp 0 0 ::1:123 :::* 6698/ntpd udp 0 0 fe80::1:123 :::* 6698/ntpd udp 0 0 :::123 :::* 6698/ntpd
Chrootkit Configuration:
You may need to install and configure the chrootkit tool in the server. You can setup a daily cron too for scanning it for rootkits.
cd /usr/local/src rm -rf chkrootkit* wget http://files.syslint.com/src/chkrootkit.tar.gz tar -xzf chkrootkit.tar.gz cd chkrootkit-0.49/ make
Now setup a daily cron scripts, /etc/cron.daily/chkrootkit_cron.sh , as follows,
#!/bin/bash EMAIL="youremail@yourdomain.tld" cd /usr/local/bin/; ./chkrootkit 2>&1 | mail -s "Syslint : CHKROOTKIT Scan Details" $EMAIL
Rkhunder:
This is a rootkit hunter tool. You can install it and configure as a daily cron. This will also help to do a daily scanning for rootkits.
cd /usr/local/src rm -rf rkhunter* wget http://files.syslint.com/src/rkhunter-1.4.0.tar.gz tar -xzf rkhunter-1.4.0.tar.gz cd rkhunter-1.4.0/ sh installer.sh --install
You can setup a cron as follows,
#!/bin/bash EMAIL="youremail@yourdomain.tld" /usr/local/bin/rkhunter --update /usr/local/bin/rkhunter -sk -c --nocolors --versioncheck | mail -s "Syslint: RKhunter Scan Details" $EMAIL
LogWatch:
Logwatch will watch the logs. You can install and configure it for sending daily log reports to your email address too. For installing logwatch , do it as follows,
# yum install logwatch
Now edit the file /etc/cron.daily/00logwatch and modify the line begin with “logwatch” as follows,
/usr/sbin/logwatch --mailto alertemail@yourdomain.tld
Please also change the email address in /usr/share/logwatch/default.conf/logwatch.conf . After that you will receive the daily alerts from this.
Analyze System Users With Empty Passwords:
It is better to find out which users are using empty passwords in your server and remove them. To do so you can try the following command,
# awk -F: '($2 == "") {print}' /etc/shadow
Locate zeron ‘0’ uid owners:
Only root account have UID 0 with full permissions to access the system.It is better to find out some one already have this. Type the following command to display all:
# awk -F: '($3 == "0") {print}' /etc/passwd
Remove X window Group Ware System Software :
It is better to remove all X-window software from your Nodes. Normally no one install and use desk tops in Node. Desktop applications will have a lot of bugs and security holes. So please remove it as follows,
# yum groupremove "X Window System"
Find all SUID & SGID Binaries :
Find out all SUID and SGID binaries . And remove all the useless binaries or change the permissions of these binaries. You can run the following command and check it,
# find /{bin,usr} \( -perm -4000 -o -perm -2000 \) -print /bin/ping /bin/cgexec /bin/ping6 /bin/mount /bin/su /bin/umount /usr/libexec/openssh/ssh-keysign /usr/libexec/pt_chown /usr/libexec/polkit-1/polkit-agent-helper-1 /usr/libexec/abrt-action-install-debuginfo-to-abrt-cache /usr/libexec/utempter/utempter /usr/bin/chsh /usr/bin/sudo /usr/bin/chage /usr/bin/gpasswd /usr/bin/ssh-agent /usr/bin/pkexec /usr/bin/wall /usr/bin/passwd /usr/bin/crontab /usr/bin/solusvmconsolevz /usr/bin/solusvmconsolexen /usr/bin/locate /usr/bin/chfn /usr/bin/write /usr/bin/newgrp /usr/bin/at /usr/bin/staprun /usr/local/solusvm/core/solusvmc-tc /usr/local/solusvm/core/solusvmc-xen /usr/local/solusvm/core/solusvmc-h /usr/local/solusvm/core/solusvmc-vz /usr/local/solusvm/core/solusvmc-node /usr/sbin/postqueue /usr/sbin/userhelper /usr/sbin/postdrop /usr/sbin/usernetctl /usr/sbin/suexec #
Find world writable Files:
It is important to find all world writable files and remove it or change the permissions . You can run the following command and check it,
# find /{bin,usr,lib,scripts,var,home} -xdev -type d \( -perm -0002 -a ! -perm -1000 \) -print
Find No-owner Files:
Files not owned by any user or group can pose a security problem. It is better to remove those those files or change the ownerships . You can find it as follows,
# find /{bin,usr,lib,scripts,var,home} \( -nouser -o -nogroup \) -print
Install and configure CLAMD:
Clamd is an antivirus software. It will help to scan your node and find out all windows virus files uploaded from your clients desktops.
Conclusion:
This document only provide a minimal and standard set of procedures that your can do it your self for securing and optimizing a vps nodes. It is always better to consult with and experienced admin too.
A. Author
Mr. Sherin A
Syslint Technologies ( syslint.com )