Book Cover

Chapter 3

Essential Knowledge: Operating Systems

Note: The examples below are abridged; the book contains more details.

  1. Bash - Introduction
  2. Bash - Variables
  3. Bash - Pipes
  4. Bash - Loops
  5. Bash - Conditional Logic
  6. Bash - Aliases
  7. Command Prompt - Introduction
  8. Command Prompt - Variables
  9. Command Prompt - Pipes
  10. Command Prompt - Loop
  11. Command Prompt - Conditional Logic
  12. PowerShell - Introduction
  13. PowerShell - Pipes
  14. PowerShell - Variables
  15. PowerShell - Loops
  16. PowerShell - Conditional Logic
  17. Processes in Linux - Part I
  18. Processes in Linux - Part II
  19. Processes in Windows
  20. Files in Linux
  21. Displaying Files
  22. Moving, Copying, and Removing Files and Directories
  23. Wildcard Characters
  24. Soft Links
  25. Listing Directory Contents
  26. The PATH Environment Variable
  27. Compression in Linux
  28. Bash Initialization File
  29. Script Files
  30. Files in Windows
  31. Hierarchy of Directories
  32. Compression in Windows
  33. Users and Permissions in Linux
  34. Users and Permissions in Windows
  35. Redirecting Input and Output in Linux
  36. Redirecting Input and Output in Windows
  37. Working on Remote Linux Computers

Bash - Introduction

bash-introduction.sh

# this is a comment
# the prefix ## indicates an output line
# the echo command displays its arguments
# three commands separated by a semicolon
echo hello world; echo hello; echo world

# Example: the -n flag
echo hello; echo world; echo -n hello; echo -n world

# Example: combining flags
# cat concatenates and prints files
# -e displays $ at the end of each line
# -n displays the line number
# "cat -en" and "cat -e -n" are equivalent
cat -en hello.txt

# Example: named arguments
# this launched another bash program that runs
# the command specified by the -c option then quits
bash -c "ls"

# Example: the man command
man echo

# Example: the version command
man --version
ls --version

hello.txt

hello
world

Bash - Variables

# Example: setting a variable
a=3; echo a; echo $a

# Example: environment variables
# type the active shell program
echo $SHELL

# Example: the PS1 environment variable
export PS1="% "
export PS1="$ "
export PS1="\\[email protected]\h \\W>"  # set prompt to [email protected] dir>

Bash - Pipes

The following example shows a piped combination of the echo command, and the wc command, which prints the number of lines, words, and characters in its input:

echo a sentence with 8 words and 42 characters
# count lines, words, characters in the following sentence
echo a sentence with 8 words and 42 characters | wc
# feed the output of the piped combination above to another wc command
echo a sentence with 8 words and 42 characters | wc | wc
# and so on
echo a sentence with 8 words and 42 characters | wc | wc | wc

Bash - Loops

# Example: the seq command
seq 1 5
echo seq 1 5
echo `seq 1 5`

# Example: for-loops
for i in `seq 1 5`;  do echo the current number is $i; done
# parentheses start a new sub-shell
for i in `seq 1 5`;  do (echo the current number is $i| wc); done

# Example: while-loops
i=0; while [ $i -lt 5 ]; do  echo $i; let i=i+1; done

# Example: until-loops
i=0; until [ $i -ge 5 ]; do echo $i; let i=i+1; done

Bash - Conditional Logic

# Example: if-else statements
a=4 ; b=3 ; if [ $a -eq $b ] ; then echo 1 ; else echo 0 ; fi

# Example: logical operators
a=4 ; b=3 ; [ $a -eq $b ] && echo 1 || echo 0

# Example: brace expansion
echo {b,c}; echo a{b,c}d; echo a{a..m}d

Bash - Aliases

date
alias date="date -u"  # modify format with optional flag -u
date     # alias in effect
\date    # escape alias, original command in effect
unalias date  # remove alias
date

Command Prompt - Introduction

To start Command Prompt, open the Start menu and search for cmd; the search results should contain an entry for Command Prompt.

rem This is a comment
rem The prefix ## indicates an output line
rem The echo command displays its arguments
echo hello world

rem Example: run multiple commands on the same line
echo hello & echo world

rem Example: the dir command
rem The current directory is C:\test
rem The "." entry is a reference to the current directory
rem The ".." entry is a reference to the parent directory
dir

rem Example: specifying options
rem Options are specified using a slash
rem The /b option instrcuts dir to use a bare format
dir /b

rem Example: combining options
rem The /a option instrcuts dir to show all contents
rem The /b option instrcuts dir to use a bare format
dir /a /b

rem Example: sorting the output of dir
rem The /a option instrcuts dir to show all contents
rem The /b option instrcuts dir to use a bare format
rem The /o:s option instrcuts dir to sort by file size
dir /a /b /o:s

rem Example: getting help
rem The /? option instrcuts tree to show its help message
tree /?
help help

Command Prompt - Variables

rem To read a variable, use %VARIABLE_NAME%
echo %TMP%

rem Example: the set command
rem Use the set command to set, read, and remove a variable
set X=13
set X
set X=
set X
set PROMPT=%USERNAME%@%COMPUTERNAME% $P$G

Command Prompt - Pipes

dir /b | sort /r

Command Prompt - Loop

rem Example: a for-loop using a counter
for /L %i in (0, 2, 10) do @echo %i

rem Example: a for-loop using a list
FOR %a IN (eggs milk bread) DO @echo %a

rem Example: a for-loop using a text file
(echo eggs & echo milk & echo bread) > lines.txt
for /f %i in (lines.txt) do @echo %i

rem Example: a for-loop used to load a CSV file
echo eggs,milk,bread > data.csv
for /f "delims=, tokens=1-3" %i in (data.csv) do ^
@echo %i & @echo %j & @echo %k

rem Example: a for-loop using the output of a command
FOR /F "delims==" %i IN ('set') DO @echo %i

rem Example: a for-loop using a back-quoted string
FOR /F "usebackq delims==" %i IN (`set`) DO @echo %i

Command Prompt - Conditional Logic

rem Example: various comparators
set X=10
IF %X% EQU 10 echo X is 10
set Y=bar
IF NOT %Y%==foo echo Y is not foo
IF NOT a==A echo case-sensitive
IF /i a==A echo now equal

rem Example: other checks
IF NOT DEFINED SOME_VAR echo SOME_VAR is not set
IF NOT EXIST some_file.txt echo.>some_file.txt
IF NOT EXIST some_file.txt (echo.>some_file.txt) ^
ELSE echo file already exists
IF %ERRORLEVEL% EQU 0 echo last command succeeded

PowerShell - Introduction

PowerShell is a command-line shell that provides a richer feature set for power users. Thanks to built-in support for the .NET Framework, PowerShell can be a more appealing shell to .NET developers when compared to Command Prompt.

Note: PowerShell examples in the book ran on Windows; the code on this website runs on Linux — the output may vary.

# This is a single-line comment
<#
  This is a block comment...
  It can span multiple lines
#>
# The prefix ## indicates an output line
# The Write-Host cmdlet writes output to a PowerShell host
# Host here refers to the process that's hosing PowerShell
Write-Host hello world
## hello world

Write-Host hello; Write-Host world
## hello
## world

# The first cmdlet below has the NoNewLine switch turned on
Write-Host -NoNewLine hello; Write-Host world
## helloworld

# The separator below is added between printed objects
Write-Host -Separator ", " hello world
## hello, world

# 0..9 creates an array of integers from 0 to 9
# Parentheses are required to evaluate the expression correctly
Write-Host -Separator ";" (0..9)
## 0;1;2;3;4;5;6;7;8;9
# The + operator below concatenates ranges
Write-Host -Separator ";" (0..9 + 8..0)
## 0;1;2;3;4;5;6;7;8;9;8;7;6;5;4;3;2;1;0

# The namespace prefix System is optional in PowerShell
Write-Host -Separator ([IO.Path]::PathSeparator) (0..9)
## 0;1;2;3;4;5;6;7;8;9

[String]::Join([IO.Path]::PathSeparator, 0..9)
## 0;1;2;3;4;5;6;7;8;9

# This is also an example of a multi-line entry
# When an entry is incomplete, you'll see this prompt: >>
# To complete the entry, press Enter after the last input
# Get-Date returns a DateTime object
(Get-Date).
AddDays(1).
ToUniversalTime().
ToLongDateString().
ToUpper()
## FRIDAY, MAY 13, 2016

# Example: comparing data extraction with Command Prompt
## rem ^ allows us to escape special characters
## for /f "tokens=4,5" %i in ('dir c:\test') ^
## do @if exist %j if %i neq ^<DIR^> echo %j %i
## bar.txt 22
## foo.txt 3
## 

# dir is an alias for the Get-ChildItem cmdlet
# The output of dir is piped as input to Select-Object
dir | Select-Object Name, Length
## 
## Name           Length
## ----           ------
## bar.txt            42
## foo.txt             3

# Example: getting help
# The output below is truncated for brevity
Get-Help Out-Null
## 
## NAME
##     Out-Null
## 
## SYNOPSIS
##     Deletes output instead of sending it down the pipeline.
## 
## 
## SYNTAX
##     Out-Null [-InputObject [<PSObject>]] [<CommonParameters>]
## (truncated for brevity)

PowerShell - Pipes

Piping in PowerShell has the same syntax like Command Prompt and the Linux terminal; however, one major difference is that what gets piped is an object. The pipeline is at the heart of PowerShell and its design philosophy.

# Many aliases are used below for brevity
dir -s * | group extension | Sort-Object count | where count -gt 1 |
select count, name | ConvertTo-Json

PowerShell - Variables

Sometimes piping isn’t enough; we need variables to store, manipulate, and read data. Variables in PowerShell are objects; variable names have to start with a $ as the examples below demonestrate:

# Assign the value 13 to $var then read it
$var = 13
$var
## 13

# Assign the value 42 to ${a$b} then read it
${a$b} = 42
${a$b}
## 42

$x = 3
$x.GetType().Name
$x = "hello world"
$x.GetType().Name
## Int32
## String

# Define $t to be of type [DateTime]
[DateTime]$t = (Get-Date) # Use () to evaluate the cmdlet first
$t = "5/23/2016" # Valid conversion from String
$t
$t = "hello world" # Error
## 
## Monday, May 23, 2016 12:00:00 AM
## Cannot convert value "hello world" to type "System.DateTime".
## (truncated for brevity)

# Example: data validation
[ValidateRange(1, 118)][int]$atomicNumber = 1
$atomicNumber = 119 # Error
## The variable cannot be validated because the value 119
## is not a valid value for the atomicNumber variable.
## (truncated for brevity)

# A variable in PowerShell can have a description
Set-Variable Mg -option ReadOnly `
	-description "Magnesium" -value 12
$Mg
$Mg = 13 # Error
## 12
## Cannot overwrite variable Mg because it is read-only
## or constant.
## (truncated for brevity)

# Example: string concatenation
$x = 1
$y = 2
Write-Host ("x = " + $x + ", y = " + $y)
## x = 1, y = 2

# Example: formatting a string
Write-Host "x = $x, y = $y"
## x = 1, y = 2

# Example: escaping
Write-Host "`$x = $x, `$y = $y"
## $x = 1, $y = 2

# Example: literal strings
Write-Host '$x is a variable'
## $x is a variable

# Example: list all variables
$a = 1
# Get the content of the virtual drive (variable:)
dir variable:
## 
## Name                           Value
## ----                           -----
## $                              1
## ?                              True
## ^                              $a
## a                              1
## args                           {}
## ConfirmPreference              High
## (truncated for brevity)

# Example: test for a variable's existence
Test-Path variable:nonexistent
## False

# Example: deleting a variable
Test-Path variable:x
del variable:x
Test-Path variable:x
## True
## False

PowerShell - Loops

# Example: a for-loop using a counter
for ($i = 0; $i -lt 5; $i++) { Write-Host $i }

# Example: a while-loop
$i = 0; while ($i -lt 5) { Write-Host $i; $i++ }

# Example: using foreach
foreach ($i in 0..4) { Write-Host $i }

# Example: using .ForEach
(0..4).ForEach({ Write-Host $_ })

# Example: piping to ForEach-Object
(0..4) | ForEach-Object { Write-Host $_ }

# Example: another example of piping to ForEach-Object
Get-Item *.* | Group extension | ForEach-Object { Write-Host $_.Name }

PowerShell - Conditional Logic

"time flies" -like "an arrow"
"fruit flies" -notlike "*lies"
-not ("anything" -match ".*")
Test-Path nonexistent.txt
$?
## False
## False
## False
## False
## True
## 

Processes in Linux - Part I

touch a.txt; touch b.txt  # create two empty files
tail -f a.txt &  # launch a never-ending background job
jobs  # display current jobs

## [1]+  Running                 tail -f a.txt &

ps  # display current processes launched by current user

##   PID TTY           TIME CMD
## 11185 ttys005    0:00.00 tail -f a.txt

ps -A | head -n 7 # display all running processes (first 7 lines)

##   PID TTY           TIME CMD
##     1 ??         0:10.84 /sbin/launchd
##    10 ??         0:00.79 /usr/libexec/kextd
## (truncated for brevity)

tail -f b.txt &  # launch another never-ending background job
jobs  # displays current jobs

## [1]-  Running                 tail -f a.txt &
## [2]+  Running                 tail -f b.txt &

kill %1  # kill job 1
jobs

## [1]-  Terminated: 15          tail -f a.txt
## [2]+  Running                 tail -f b.txt &

jobs

## [2]+  Running                 tail -f b.txt &

fg  # bring the single current job into foreground

## [CTRL+z keystroke suspends foreground job
## and creates a new prompt]

jobs  # job was suspended by CTRL+z keystroke

## [2]+  Stopped                 tail -f b.txt

bg  # resume single current job in the background
jobs

## [2]+  Running                 tail -f b.txt &

Processes in Linux - Part II

# run five jobs sequentially in the foreground
for i in `seq 1 5`; do  (sleep 100) ; done ;

[ hitting CTRL-Z to suspend the job ]

jobs

## [1]+  Stopped                 ( sleep 100 )

bg  # resume process in the background
jobs

## [1]+  Running                 ( sleep 100 ) &

ps  # shows single active process

##   PID TTY           TIME CMD
## 57140 ttys004    0:00.00 sleep 100

# run processes concurrently in the background
# run 5 processes in the background concurrently
for i in `seq 1 5` ; do  (sleep 100 &) ; done ;
ps

##   PID TTY           TIME CMD
## 57148 ttys004    0:00.00 sleep 100
## 57150 ttys004    0:00.00 sleep 100
## 57152 ttys004    0:00.00 sleep 100
## 57154 ttys004    0:00.00 sleep 100
## 57156 ttys004    0:00.00 sleep 100

kill 57148  # kill the first process

ps

##   PID TTY           TIME CMD
## 57150 ttys004    0:00.00 sleep 100
## 57152 ttys004    0:00.00 sleep 100
## 57154 ttys004    0:00.00 sleep 100
## 57156 ttys004    0:00.00 sleep 100

top

## Tasks: 195 total,   2 running, 193 sleeping,   0 stopped,  
## Cpu(s): 69.4%us,  0.3%sy,  0.0%ni, 30.3%id,  0.0%wa,  0.0%hi
## Mem:  99197580k total, 92876600k used,  6320980k free,  
## Swap: 100652028k total,    19548k used, 100632480k free
##
##   PID USER PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+ COMMAND
## 26323 joe  20   0 24.2g 1.2g 9396 S  100  1.3   3546:44 java
## 31696 joe  20   0 24.2g 1.3g 9392 S  100  1.4 874:38.32 java
## 29854 joe  20   0 24.2g 971m 9408 S  100  1.0   1962:39 java
## 29419 joe  20   0 24.2g 992m 9396 S  100  1.0   2204:30 java

Processes in Windows

rem Example: start a new window to run a command
start cmd /c "echo poof"

rem Example: start a new window to run a command and pause
start cmd /c "echo poof && pause"

rem Example: start a command without creating a new window
start /b powershell "sleep 1; echo done"

rem Example: query the currently running processes
tasklist /FI "PID eq 4"

Files in Linux

cd /  # change current directory to top-most directory
pwd  # display current path

cd /home  # change directory using absolute path
pwd  # display current path

mkdir /home/joe
cd joe  # change directory using relative path
pwd  # display current path

mkdir /home/jane
cd ../jane  # change directory using relative path
pwd  # display current path

# display the size of the disk drive and amount of used and 
# available space using abbreviations such as Gi for Gigabyte
df -h ~/
du -sh ~/

Displaying Files

echo 'Feb  5 08:01:24 chance rsyslogd: [origin software="rsyslogd"' > /var/log/syslog.1
echo 'Feb  6 07:35:01 chance anacron[7614]: Updated timestamp for' >> /var/log/syslog.1

head -1 /var/log/syslog.1  # first line
tail -1 /var/log/syslog.1  # last line
# count number of messages (first column is number of lines)
wc /var/log/syslog.1
# count number of messages on Feb 6
grep "Feb  6" /var/log/syslog.1 | wc
cat /var/log/syslog.1 | grep "Feb  6" | wc # same as above

Moving, Copying, and Removing Files and Directories

touch file1 /tmp/file1
mkdir ~/tmpFiles

# copy file1 in current dir to file2 in current dir
cp file1 file2  
# copy file1 in /tmp to file2 in ~
cp /tmp/file1 ~/file2  
# rename file1 as file2 in current dir
mv file1 file2  
# move file1 in /tmp to file2 in ~
mv /tmp/file1 ~/file2  
# remove file2 in home dir
rm ~/file2  
# removes directory tmpFiles and all its contents
rm -R ~/tmpFiles 
# same as above but avoids calling an aliased version of rm
\rm -R ~/tmpFiles

Wildcard Characters

# removes all non-hidden files in directory ~/tmpFiles
rm ~/tmpFiles/*
# removes all hidden files in ~/tmpFiles
rm ~/tmpFiles/.*
# copies all files in ~/tmpFiles whose names end with .html
# to directory /tmp
cp ~/tmpFiles/*.html /tmp/
# remove all files ending with .c or .o
rm *.[co]
# remove files whose extension is a lower-case character
rm *.[a-z]

touch /tmp/file1 /tmp/file2
ln -s /tmp tmpLink # create a link tmpLink to the directory /tmp
ls /tmp  # display files in /tmp directory
# enter the directory /tmp by referencing the softlink tmpLink
cd tmpLink
ls

Listing Directory Contents

chmod +x main.sh # change file mode to display with the -F flag

ls
ls -F # ppends a character denoting file type
ls -l # displays more detailed format

The PATH Environment Variable

# show path to file matching command ls
which ls

# display PATH variable (note the current directory is the third
# directory in the list below, denoted by the period notation)
echo $PATH

# add the directory /home/joe/bin to PATH
export PATH=$PATH:/home/joe/bin
echo $PATH

Compression in Linux

echo "Call me Ishmael.
Some years ago—never mind how long precisely—
having little or no money in my purse," > mobyDick.txt

cat mobyDick.txt | head -n 1 # first line

bzip2 -v mobyDick.txt # compress .txt to .txt.bz2

# first line of compressed file
cat mobyDick.txt.bz2 | head -n 1

# using bzcat to display compressed file
bzcat mobyDick.txt.bz2 | head -n 1

mkdir tmp tmp2
echo "hello" > tmp/hello.txt
# pack all files in tmp/ into a file archive.tar
tar cvf archive.tar tmp/*
# compress the file archive.tar (creating archive.tar.bz2)
bzip2 archive.tar
# uncompress archive.tar.vz2
bunzip2 archive.tar.bz2
# moves archive.tar.bz2 to subdirectory tmp2
mv archive.tar tmp2 
# change current directory to tmp2
cd tmp2
# unpack the tar file archive.tar in current directory
tar xvf archive.tar

Bash Initialization File

# Example: a sample ~/.bash_profile file
# customize the prompt appearance
export PS1="\\[email protected]\h \\W]\\$"
# add current directory to PATH
export PATH=.:$PATH
# avoid overwriting files with output redirection
set -o noclobber
# prompt user for removing or overwriting files with rm, cp, mv
alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'
# store most recent terminal commands in the file .bash_history
export HISTFILE=".bash_history"
# alias for printing last 10 commands
alias h='history | tail'
# alias ls and ll to use my favorite flags by default
alias ls='ls -Ft'
alias ll="ls -Fthalr";

Script Files

echo '#!/usr/bin/env bash
echo this is a bash script file that accepts two arguments
echo $1 $2' > printTwoArgs

chmod a+x printTwoArgs  # add executable permission to scriptFile
./printTwoArgs one two  # executing script by listing its path

Files in Windows

To view the logical volumes, the following query uses WMIC (Windows Management Instrumentation Command-line) to list the names of logical volumes:

wmic logicaldisk get name
## Name
## C:
## D:
## 

Hierarchy of Directories

Note: PowerShell examples in the book ran on Windows; the code on this website runs on Linux — the output may vary.

get-psdrive –psprovider filesystem | select name

pwd

# Example: change the current directory properly in Command Prompt
# rem switch to d:
# d:
# cd \
# cd c:\windows
# echo %CD%
# rem to change directory to c:\windows, switch to c:
# c:
# echo %CD%

# Example: display directory hierarchy
# tree /A
## Folder PATH listing
## Volume serial number is 0123-4567
## D:.
## +---Data
## |   \---Speech
## \---Code
## 
## 

Compression in Windows

compact /c foo.txt
##  Compressing files in C:\Users\Geish\Desktop\
## foo.txt                  5967 :      4096 = 1.5 to 1 [OK]
## 1 files within 1 directories were compressed.
## 5,967 total bytes of data are stored in 4,096 bytes.
## The compression ratio is 1.5 to 1.
## 

Users and Permissions in Linux

touch file.txt

ls -l file.txt

chmod g+w file.txt # add write permission to group
ls -l file.txt

chmod a+w file.txt # add write permission to all users
ls -l file.txt

chmod a-w file.txt # remove write permission from all users
ls -l file.txt

chown guest file.txt
ls -l file.txt

Users and Permissions in Windows

runas /user:geish "cmd /k echo hello"

whoami
## my-desktop\geish
## 

Redirecting Input and Output in Linux

echo hello >> output.txt
echo world >> output.txt
cat output.txt

# create a text file and then print it in uppercase
echo this is a text file > a.txt
tr "a-z" "A-Z" < a.txt

# convert text file to a sorted list of distinct words
# annotated with their count
echo this file is a text file > b.txt
tr < b.txt -cs "[:alpha:]" "\n" | sort | uniq -c

tr < b.txt -d ' ' # remove all white spaces

Redirecting Input and Output in Windows

echo hello >> output.txt
echo world >> output.txt
type output.txt
## hello
## world

Working on Remote Linux Computers

ssh [email protected]

# copy file /home/joe/file1 from server1.domain.com 
# (authenticating as user joe) to local home directory
scp [email protected]/home/joe/file1 ~/
# copy entire home directory (including sub-directories)
# on server1.domain.com to local ~/tmp
scp -R [email protected]/home/joe/* ~/tmp/
# copy local files ~/file2.* to home directory on remote computer
scp ~/file2.* [email protected]/home/joe/