Useful *NIX Shell Commands for Web Developers

So, why *NIX and why do you need to go to the console?

According to the statistics of W3Techs, Unix is
used by 68% of all the websites whose operating system they know.
This means that if you are web developer, your code is most probably running on a Linux server. And at least you need to know how to
configure and debug your code on Unix and Linux systems. Let's find
out what you need to know to feel comfortable in the command line.

The Basics

The basic *NIX command consists of three components:

• 
  command or program to run


• 
  options to alter or specify the behavior of the command


• 
  arguments or input data that is
  needed to run the command   

For example, if you need to get a list of
files in the directory /var/www, you need to run the command ls with the argument /var/www. To add the size of files to the output, you need to add the -s option, and the final command will look like this:

ls -s /var/www

I/O Redirections and Pipelines

Many *NIX commands use text input and
output which you can operate with, and the great feature of this is
that you can send the output results of the command to a file using redirect,
or even pass the output of one command to the input of another command
using the pipelines. For example, we can output the command from the previous
example to a file:

ls -s /var/www > /var/www/files.txt

This command will create or erase file
/var/www/files.txt and output a list of files in the /var/www directory.
Here is a list of standard I/O redirections and pipelines:

• 
  > Redirect output from a
  command to a file on disk. The file will be erased and overwritten.


• 
  >>
  The same redirect, but appending the output file.


• 
  < Get
  input to command from a file.


• 
  | Pass the
  output of one command to the input of another command.


• 
  tee Both
  redirect output to a file and pass it to the next command in the
  pipeline.

The Main Commands

To get manual pages for a command, run
man. Manual pages follow a common layout and may include name, synopsis,
description, and usage examples. This will display the documentation
for the chmod command:

man chmod

To execute some commands like saving
configurations or rebooting processes, you need to run them as
the super user. To do this, you need to prepend sudo to your command:

user@server:/var/www$ chmod 777 log
chmod: changing permissions of ‘log’: Operation not permitted
user@server:/var/www$ sudo chmod 777 log
sudo chmod 777 log

If you need to execute a bunch of
commands as a super user, you can use su, or switch user command.

user@server:/var/www$ su
Password:
root@server:/var/www#

Note: To save the security layer and
avoid accidental execution of objectionable commands, do not use
sudo and su without any purpose.

Into the Real World

Basic Navigation

There are three main commands to
navigate in the file tree:

• 
  pwd to print the name of the current
  working directory


• 
  cd to change directory


• 
  ls to list directory contents

Here is an example of using these commands with the output of terminal:

user@server:~$ pwd
/home/user
user@server:~$ cd /var/www
user@server:/var/www$ ls -alF
total 16
drwxr-xr-x  5 root    root    4096 Jan 22 09:45 ./
drwxr-xr-x 14 root    root    4096 Jan 22 09:38 ../
drwxr-xr-x  2 root    root    4096 Jan 22 09:45 html/
drwxr-xr-x  3 root    root    4096 Jan 22 09:45 log/
drwxrwxrwx  1 user    user    442  Mar 24 12:22 testing/

Searching Files

There is the find command to search for
files in a directory hierarchy. This command is very powerful and can
search for files and directories by name, access permissions, date, and size.

Find all directories with the “logs”
name in the /var/www directory using the -type option:

find /var/www -type d -name logs

To search PHP files in the current
directory, add the -name option:

find . -type f -name "*.php"

Find files with defined permissions
using the -perm option:

find . -type f -perm 0777 -print

Find all files which are greater than
500MB:

find / -size +500M

Of course, you can combine all of those
options in one command, and this is only the basics of the find command, which
is a very powerful tool for searching files. Use manual pages to get more
information.

Manipulating Files and Folders

There are five
main commands for manipulating files and folders in *NIX system:

• 
  touch is used to change timestamps on existing files and
  directories, but also this is the easiest way to create new file


• 
  mkdir to make
  directories


• 
  cp to copy
  files and directories


• 
  mv to move or
  rename files and directories


• 
  rm to delete
  files and folders

The next example will
create a file index.html, copy this file to the new directory in
/var/www, and remove the source file.

root@localserver:~# touch index.html
root@localserver:~# mkdir /var/www/newdir
root@localserver:~# cp index.html /var/www/newdir/
root@localserver:~# rm index.html

Another
great command is ln, which is designed to make links between files.
The command ln is often used to create a symbolic link for
enabling a virtual host:

sudo ln -s /etc/apache2/sites-available/newvirtualhost.com.conf /etc/apache/sites-enabled/ newvirtualhost.com.conf

Change Access Permissions

To change the file owner and group, use
chown. Don't forget to grant ownership to the apache user when you are
creating a new virtual host of your web application:

sudo chown -R www-data:www-data /var/www/newvirtualhost.com

Sometimes cache or log directories of
your application must be writable for all users so you need to change
access modes to 777 with the chmod command. Add the -R option to add permission to all nested
files and folders.

sudo chmod -R 777 /var/www/private/cache

If you just want to make a file
executable, use chmod with the +x option.

sudo chmod +x /var/www/private/backup.sh

File Reading

To view files in the console, you can use the cat command. With cat, you can concatenate files' contents using extra
parameters, and you can also use mask in filenames.

cat /etc/apache2/apache2.conf
cat /etc/apache2/apache2.conf /etc/apache2/ports.conf 
cat /etc/apache2/mods-enabled/*

But the cat command will get you
confused very fast, because it shows output in raw format without
any paging—so it is inconvenient to use with log output. To get
a filter for paging through text one screenful at a time, you should
use the more or less commands, which are much of a muchness.

less /etc/apache2/apache2.conf
cat /etc/apache2/mods-enabled/* | less

Another useful command is tail, which is
created to output the last part of files. This command is perfect to look through log histories. By default, this tail command prints the last 10
lines, and you can change this number using the -n parameter.

tail /var/log/apache2/error.log
tail -n 25 /var/log/apache2/error.log

But if you have, for example, a bunch
of log files, you need something more powerful to make a proper
search. Something like grep—a program that reads from standard
input, tests each line against a pattern, and writes to standard
output the lines that match this pattern. By using it in combination with cat
and pipelines, you will get what you want.

If you want to filter text-lines of
output, you can use the grep command:

grep notice /var/log/apache2/error.log
cat /var/log/apache2/*.log | grep "shutting down"

As you can see, grep is good for using
in pipelines. In this example, the last command will output all lines
containing the “shutting down” string from log-files.

File Editing

If you want to edit text files in
console mode, you can use one of the three most popular text editors:

• 
  GNU nano, a small and friendly
  default text editor, which is a perfect choice for basic tasks


• 
  Vim, an improved programmers' text
  editor, which is most powerful, but complex for beginners


• 
  mcedit, a full-featured windowed
  editor from Midnight Commander, which is easy to use but not
  installed by default on *NIX systems

Compare them and make your choice:

nano /var/log/apache2/error.log
vim /var/log/apache2/error.log
mcedit /var/log/apache2/error.log

Archiving

Sometimes you need to back up or
compress some data on your server.

The most common archiving utilities are tar
and zip. Notice that the zip command may not be installed on
your server by default.

You can create an archive with the following commands:

tar -zcvf archive-name.tar.gz directory-or-file-name
tar -jcvf archive-name.tbz2 directory-or-file-name
zip archive-name.zip directory-or-file-name

If you want just to see a list of files
in the archive, you can use the -l option for both tar and unzip:

tar -ztvf archive-name.tar.gz
tar -jtvf archive-name.tbz2
unzip -l archive-name.zip

Or extract some source files:

tar -zxvf archive-name.tar.gz
tar -jxvf archive-name.tbz2
zip archive-name.zip

Schedule Tasks

If you want to schedule your scripts to
run periodically, you need to use the Cron utility, which is driven by a
cron table—a configuration file that specifies the shell commands to
run periodically on a given schedule. And the command to maintain
cron tables is crontab.

Calling crontab with option -l will
show your cron table.

Also, the -u option is provided to specify
the name of the user whose crontab is being used. If you are going
to run tasks of your web application, it is better to edit crontab
for user www-data.

user@server:~$ sudo crontab -lu www-data
# m h  dom mon dow   command
*/5 * * * * php5 /var/www/yii do/tasks >> /var/www/tasks.log
00 15 * * 1-5 /var/www/backuper.sh

In this output, you can take a look at an example of a cron table. As you can see, every line is scheduled by minute, hour,
day of month, month, and day of week. Every field may be an asterisk, which means every value of the field. Also you can use sets and ranges using
commas and hyphens. Following a range with a slash specifies skips of
the number's value through the range. In this example, the first command
will run every five minutes, and the second command will run from
Monday to Friday at 15:00.

To edit this list, run crontab with the -e
key instead of -l. The cron list will be opened in your default editor.
Use the -r option to clear the cron list.

Performance Monitoring

Command top shows system summary
information and provides a dynamic real-time view of running
system processes. Press Shift-M to sort processes by memory usage, or Shift-P to sort by CPU usage.

top - 21:33:02 up 308 days, 21:24,  1 user,  load average: 0.00, 0.01, 0.05
Tasks:  87 total,   1 running,  86 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.3 us,  0.0 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:    501800 total,   471348 used,    30452 free,    49672 buffers
KiB Swap:  4194300 total,    56192 used,  4138108 free.   149488 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND     
16269 www-data  20   0  348592  38884  12044 S  0.0  7.7   0:02.42 php5        
26533 www-data  20   0  409516  38488  24312 S  0.0  7.7   1:00.04 php5-fpm    
 1076 mysql     20   0  887824  32748   1616 S  0.0  6.5 276:46.59 mysqld      
  862 syslog    20   0  256612  31428    368 S  0.0  6.3  32:45.88 rsyslogd    
18901 root      20   0  105632   4316   3248 S  0.0  0.9   0:00.04 sshd        
25393 www-data  20   0   87356   4312   1564 S  0.0  0.9   4:46.92 nginx       
27846 memcache  20   0  328464   3828    252 S  0.0  0.8   1:04.30 memcached 

To display the amount of free and used memory
in the system, use the free command. Add the -h option to display output fields
in human-readable format.

user@server:~$ free -h
             total       used       free     shared    buffers     cached
Mem:          490M       453M        36M        23M        46M       140M
-/+ buffers/cache:       265M       224M
Swap:         4.0G        54M       3.9G

Another useful command is df, which is a
command to report file system disk space usage. You can call it with the -a option to show all the file systems of your server. Also, don't
forget to add the -h option for human-readable format.

user@server:~$ df -ah
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        20G  6.3G   13G  34% /
udev            235M  4.0K  235M   1% /dev
tmpfs            50M  344K   49M   1% /run

Command Line History

You can use the !! command to repeat the previous command, or use sudo !! if you forgot to run a command with
sudo.

user@server:/var/www$ chmod 777 log
chmod: changing permissions of ‘log’: Operation not permitted
user@server:/var/www$ sudo !!
sudo chmod 777 log

If you forgot the syntax of commands or are feeling lazy about typing a large command query, you can use history to
display your command history. It is good to combine this command with
strings filter commands like grep, tail and others to find exactly what
you want.

history | tail
history | grep crontab
history | egrep -i 'ssh|ftp'

Conclusion

Using the console is not rocket science.
Unix and Linux systems are easy to understand and use because of their simple design and good documentation. I hope this article will make
you pretty comfortable with the command line and bring you to the
next level of managing your web applications with the command line.

If you have any questions or you want
to share your favourite console commands, don't hesitate to leave a
comment below the article.

Further Reading

• Directory of Linux Commands


• Bash Guide for Beginners


• Using the Terminal


• How Can I Quickly Learn Terminal
  Commands?


• The Command Line Is Your Best
  Friend