In Chap. 2, we discussed the CPU and how it executes a sequence of assembly language instructions. This suggests the following model: a programmer writes a computer program as a sequence of assembly language instructions, loads them into memory, and instructs the CPU to execute them one by one by pointing the program counter at the relevant memory address. Unfortunately, there are many problems with this scheme: only one program can run at any particular time, one programmer may overwrite information important for another programmer, and it is hard to reuse instructions implementing common tasks. The operating system mediates between the computer hardware and programmers, and resolves difficulties such as the ones mentioned above. This chapter describes the concepts of an operating system, then delves into the Linux and Windows operating systems in some detail as concrete examples; in addition, it explores command-line interfaces like bash, Command Prompt, and PowerShell, which are essential for developers to know intimately.
Note: The examples below are abridged; the book contains more details.
# this is a comment
# the prefix ## indicates an output line
# the echo command displays its arguments
# three commands separated by a semicolon
echo hello world; echo hello; echo world
# Example: the -n flag
echo hello; echo world; echo -n hello; echo -n world
# Example: combining flags
# cat concatenates and prints files
# -e displays $ at the end of each line
# -n displays the line number
# "cat -en" and "cat -e -n" are equivalent
cat -en hello.txt
# Example: named arguments
# this launched another bash program that runs
# the command specified by the -c option then quits
bash -c "ls"
# Example: the man command
man echo
# Example: the version command
man --version
ls --version
hello
world
# Example: setting a variable
a=3; echo a; echo $a
# Example: environment variables
# type the active shell program
echo $SHELL
# Example: the PS1 environment variable
export PS1="% "
export PS1="$ "
export PS1="\\u@\h \\W>" # set prompt to user@host dir>
The following example shows a piped combination of the echo command,
and the wc command, which prints the number of lines, words, and characters
in its input:
echo a sentence with 8 words and 42 characters
# count lines, words, characters in the following sentence
echo a sentence with 8 words and 42 characters | wc
# feed the output of the piped combination above to another wc command
echo a sentence with 8 words and 42 characters | wc | wc
# and so on
echo a sentence with 8 words and 42 characters | wc | wc | wc
# Example: the seq command
seq 1 5
echo seq 1 5
echo `seq 1 5`
# Example: for-loops
for i in `seq 1 5`; do echo the current number is $i; done
# parentheses start a new sub-shell
for i in `seq 1 5`; do (echo the current number is $i| wc); done
# Example: while-loops
i=0; while [ $i -lt 5 ]; do echo $i; let i=i+1; done
# Example: until-loops
i=0; until [ $i -ge 5 ]; do echo $i; let i=i+1; done
# Example: if-else statements
a=4 ; b=3 ; if [ $a -eq $b ] ; then echo 1 ; else echo 0 ; fi
# Example: logical operators
a=4 ; b=3 ; [ $a -eq $b ] && echo 1 || echo 0
# Example: brace expansion
echo {b,c}; echo a{b,c}d; echo a{a..m}d
date
alias date="date -u" # modify format with optional flag -u
date # alias in effect
\date # escape alias, original command in effect
unalias date # remove alias
date
To start Command Prompt, open the Start menu and search for cmd; the search
results should contain an entry for Command Prompt.
rem This is a comment
rem The prefix ## indicates an output line
rem The echo command displays its arguments
echo hello world
rem Example: run multiple commands on the same line
echo hello & echo world
rem Example: the dir command
rem The current directory is C:\test
rem The "." entry is a reference to the current directory
rem The ".." entry is a reference to the parent directory
dir
rem Example: specifying options
rem Options are specified using a slash
rem The /b option instrcuts dir to use a bare format
dir /b
rem Example: combining options
rem The /a option instrcuts dir to show all contents
rem The /b option instrcuts dir to use a bare format
dir /a /b
rem Example: sorting the output of dir
rem The /a option instrcuts dir to show all contents
rem The /b option instrcuts dir to use a bare format
rem The /o:s option instrcuts dir to sort by file size
dir /a /b /o:s
rem Example: getting help
rem The /? option instrcuts tree to show its help message
tree /?
help help
rem To read a variable, use %VARIABLE_NAME%
echo %TMP%
rem Example: the set command
rem Use the set command to set, read, and remove a variable
set X=13
set X
set X=
set X
set PROMPT=%USERNAME%@%COMPUTERNAME% $P$G
dir /b | sort /r
rem Example: a for-loop using a counter
for /L %i in (0, 2, 10) do @echo %i
rem Example: a for-loop using a list
FOR %a IN (eggs milk bread) DO @echo %a
rem Example: a for-loop using a text file
(echo eggs & echo milk & echo bread) > lines.txt
for /f %i in (lines.txt) do @echo %i
rem Example: a for-loop used to load a CSV file
echo eggs,milk,bread > data.csv
for /f "delims=, tokens=1-3" %i in (data.csv) do ^
@echo %i & @echo %j & @echo %k
rem Example: a for-loop using the output of a command
FOR /F "delims==" %i IN ('set') DO @echo %i
rem Example: a for-loop using a back-quoted string
FOR /F "usebackq delims==" %i IN (`set`) DO @echo %i
rem Example: various comparators
set X=10
IF %X% EQU 10 echo X is 10
set Y=bar
IF NOT %Y%==foo echo Y is not foo
IF NOT a==A echo case-sensitive
IF /i a==A echo now equal
rem Example: other checks
IF NOT DEFINED SOME_VAR echo SOME_VAR is not set
IF NOT EXIST some_file.txt echo.>some_file.txt
IF NOT EXIST some_file.txt (echo.>some_file.txt) ^
ELSE echo file already exists
IF %ERRORLEVEL% EQU 0 echo last command succeeded
PowerShell is a command-line shell that provides a richer feature set for power users. Thanks to built-in support for the .NET Framework, PowerShell can be a more appealing shell to .NET developers when compared to Command Prompt.
Note: PowerShell examples in the book ran on Windows; the code on this website runs on Linux — the output may vary.
# This is a single-line comment
<#
This is a block comment...
It can span multiple lines
#>
# The prefix ## indicates an output line
# The Write-Host cmdlet writes output to a PowerShell host
# Host here refers to the process that's hosing PowerShell
Write-Host hello world
## hello world
Write-Host hello; Write-Host world
## hello
## world
# The first cmdlet below has the NoNewLine switch turned on
Write-Host -NoNewLine hello; Write-Host world
## helloworld
# The separator below is added between printed objects
Write-Host -Separator ", " hello world
## hello, world
# 0..9 creates an array of integers from 0 to 9
# Parentheses are required to evaluate the expression correctly
Write-Host -Separator ";" (0..9)
## 0;1;2;3;4;5;6;7;8;9
# The + operator below concatenates ranges
Write-Host -Separator ";" (0..9 + 8..0)
## 0;1;2;3;4;5;6;7;8;9;8;7;6;5;4;3;2;1;0
# The namespace prefix System is optional in PowerShell
Write-Host -Separator ([IO.Path]::PathSeparator) (0..9)
## 0;1;2;3;4;5;6;7;8;9
[String]::Join([IO.Path]::PathSeparator, 0..9)
## 0;1;2;3;4;5;6;7;8;9
# This is also an example of a multi-line entry
# When an entry is incomplete, you'll see this prompt: >>
# To complete the entry, press Enter after the last input
# Get-Date returns a DateTime object
(Get-Date).
AddDays(1).
ToUniversalTime().
ToLongDateString().
ToUpper()
## FRIDAY, MAY 13, 2016
# Example: comparing data extraction with Command Prompt
## rem ^ allows us to escape special characters
## for /f "tokens=4,5" %i in ('dir c:\test') ^
## do @if exist %j if %i neq ^<DIR^> echo %j %i
## bar.txt 22
## foo.txt 3
##
# dir is an alias for the Get-ChildItem cmdlet
# The output of dir is piped as input to Select-Object
dir | Select-Object Name, Length
##
## Name Length
## ---- ------
## bar.txt 42
## foo.txt 3
# Example: getting help
# The output below is truncated for brevity
Get-Help Out-Null
##
## NAME
## Out-Null
##
## SYNOPSIS
## Deletes output instead of sending it down the pipeline.
##
##
## SYNTAX
## Out-Null [-InputObject [<PSObject>]] [<CommonParameters>]
## (truncated for brevity)
Piping in PowerShell has the same syntax like Command Prompt and the Linux terminal; however, one major difference is that what gets piped is an object. The pipeline is at the heart of PowerShell and its design philosophy.
# Many aliases are used below for brevity
dir -s * | group extension | Sort-Object count | where count -gt 1 |
select count, name | ConvertTo-Json
Sometimes piping isn’t enough; we need variables to store, manipulate, and read
data. Variables in PowerShell are objects; variable names have to start with
a $ as the examples below demonestrate:
# Assign the value 13 to $var then read it
$var = 13
$var
## 13
# Assign the value 42 to ${a$b} then read it
${a$b} = 42
${a$b}
## 42
$x = 3
$x.GetType().Name
$x = "hello world"
$x.GetType().Name
## Int32
## String
# Define $t to be of type [DateTime]
[DateTime]$t = (Get-Date) # Use () to evaluate the cmdlet first
$t = "5/23/2016" # Valid conversion from String
$t
$t = "hello world" # Error
##
## Monday, May 23, 2016 12:00:00 AM
## Cannot convert value "hello world" to type "System.DateTime".
## (truncated for brevity)
# Example: data validation
[ValidateRange(1, 118)][int]$atomicNumber = 1
$atomicNumber = 119 # Error
## The variable cannot be validated because the value 119
## is not a valid value for the atomicNumber variable.
## (truncated for brevity)
# A variable in PowerShell can have a description
Set-Variable Mg -option ReadOnly `
-description "Magnesium" -value 12
$Mg
$Mg = 13 # Error
## 12
## Cannot overwrite variable Mg because it is read-only
## or constant.
## (truncated for brevity)
# Example: string concatenation
$x = 1
$y = 2
Write-Host ("x = " + $x + ", y = " + $y)
## x = 1, y = 2
# Example: formatting a string
Write-Host "x = $x, y = $y"
## x = 1, y = 2
# Example: escaping
Write-Host "`$x = $x, `$y = $y"
## $x = 1, $y = 2
# Example: literal strings
Write-Host '$x is a variable'
## $x is a variable
# Example: list all variables
$a = 1
# Get the content of the virtual drive (variable:)
dir variable:
##
## Name Value
## ---- -----
## $ 1
## ? True
## ^ $a
## a 1
## args {}
## ConfirmPreference High
## (truncated for brevity)
# Example: test for a variable's existence
Test-Path variable:nonexistent
## False
# Example: deleting a variable
Test-Path variable:x
del variable:x
Test-Path variable:x
## True
## False
# Example: a for-loop using a counter
for ($i = 0; $i -lt 5; $i++) { Write-Host $i }
# Example: a while-loop
$i = 0; while ($i -lt 5) { Write-Host $i; $i++ }
# Example: using foreach
foreach ($i in 0..4) { Write-Host $i }
# Example: using .ForEach
(0..4).ForEach({ Write-Host $_ })
# Example: piping to ForEach-Object
(0..4) | ForEach-Object { Write-Host $_ }
# Example: another example of piping to ForEach-Object
Get-Item *.* | Group extension | ForEach-Object { Write-Host $_.Name }
"time flies" -like "an arrow"
"fruit flies" -notlike "*lies"
-not ("anything" -match ".*")
Test-Path nonexistent.txt
$?
## False
## False
## False
## False
## True
##
touch a.txt; touch b.txt # create two empty files
tail -f a.txt & # launch a never-ending background job
jobs # display current jobs
## [1]+ Running tail -f a.txt &
ps # display current processes launched by current user
## PID TTY TIME CMD
## 11185 ttys005 0:00.00 tail -f a.txt
ps -A | head -n 7 # display all running processes (first 7 lines)
## PID TTY TIME CMD
## 1 ?? 0:10.84 /sbin/launchd
## 10 ?? 0:00.79 /usr/libexec/kextd
## (truncated for brevity)
tail -f b.txt & # launch another never-ending background job
jobs # displays current jobs
## [1]- Running tail -f a.txt &
## [2]+ Running tail -f b.txt &
kill %1 # kill job 1
jobs
## [1]- Terminated: 15 tail -f a.txt
## [2]+ Running tail -f b.txt &
jobs
## [2]+ Running tail -f b.txt &
fg # bring the single current job into foreground
## [CTRL+z keystroke suspends foreground job
## and creates a new prompt]
jobs # job was suspended by CTRL+z keystroke
## [2]+ Stopped tail -f b.txt
bg # resume single current job in the background
jobs
## [2]+ Running tail -f b.txt &
# run five jobs sequentially in the foreground
for i in `seq 1 5`; do (sleep 100) ; done ;
[ hitting CTRL-Z to suspend the job ]
jobs
## [1]+ Stopped ( sleep 100 )
bg # resume process in the background
jobs
## [1]+ Running ( sleep 100 ) &
ps # shows single active process
## PID TTY TIME CMD
## 57140 ttys004 0:00.00 sleep 100
# run processes concurrently in the background
# run 5 processes in the background concurrently
for i in `seq 1 5` ; do (sleep 100 &) ; done ;
ps
## PID TTY TIME CMD
## 57148 ttys004 0:00.00 sleep 100
## 57150 ttys004 0:00.00 sleep 100
## 57152 ttys004 0:00.00 sleep 100
## 57154 ttys004 0:00.00 sleep 100
## 57156 ttys004 0:00.00 sleep 100
kill 57148 # kill the first process
ps
## PID TTY TIME CMD
## 57150 ttys004 0:00.00 sleep 100
## 57152 ttys004 0:00.00 sleep 100
## 57154 ttys004 0:00.00 sleep 100
## 57156 ttys004 0:00.00 sleep 100
top
## Tasks: 195 total, 2 running, 193 sleeping, 0 stopped,
## Cpu(s): 69.4%us, 0.3%sy, 0.0%ni, 30.3%id, 0.0%wa, 0.0%hi
## Mem: 99197580k total, 92876600k used, 6320980k free,
## Swap: 100652028k total, 19548k used, 100632480k free
##
## PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
## 26323 joe 20 0 24.2g 1.2g 9396 S 100 1.3 3546:44 java
## 31696 joe 20 0 24.2g 1.3g 9392 S 100 1.4 874:38.32 java
## 29854 joe 20 0 24.2g 971m 9408 S 100 1.0 1962:39 java
## 29419 joe 20 0 24.2g 992m 9396 S 100 1.0 2204:30 java
rem Example: start a new window to run a command
start cmd /c "echo poof"
rem Example: start a new window to run a command and pause
start cmd /c "echo poof && pause"
rem Example: start a command without creating a new window
start /b powershell "sleep 1; echo done"
rem Example: query the currently running processes
tasklist /FI "PID eq 4"
cd / # change current directory to top-most directory
pwd # display current path
cd /home # change directory using absolute path
pwd # display current path
mkdir /home/joe
cd joe # change directory using relative path
pwd # display current path
mkdir /home/jane
cd ../jane # change directory using relative path
pwd # display current path
# display the size of the disk drive and amount of used and
# available space using abbreviations such as Gi for Gigabyte
df -h ~/
du -sh ~/
echo 'Feb 5 08:01:24 chance rsyslogd: [origin software="rsyslogd"' > /var/log/syslog.1
echo 'Feb 6 07:35:01 chance anacron[7614]: Updated timestamp for' >> /var/log/syslog.1
head -1 /var/log/syslog.1 # first line
tail -1 /var/log/syslog.1 # last line
# count number of messages (first column is number of lines)
wc /var/log/syslog.1
# count number of messages on Feb 6
grep "Feb 6" /var/log/syslog.1 | wc
cat /var/log/syslog.1 | grep "Feb 6" | wc # same as above
touch file1 /tmp/file1
mkdir ~/tmpFiles
# copy file1 in current dir to file2 in current dir
cp file1 file2
# copy file1 in /tmp to file2 in ~
cp /tmp/file1 ~/file2
# rename file1 as file2 in current dir
mv file1 file2
# move file1 in /tmp to file2 in ~
mv /tmp/file1 ~/file2
# remove file2 in home dir
rm ~/file2
# removes directory tmpFiles and all its contents
rm -R ~/tmpFiles
# same as above but avoids calling an aliased version of rm
\rm -R ~/tmpFiles
# removes all non-hidden files in directory ~/tmpFiles
rm ~/tmpFiles/*
# removes all hidden files in ~/tmpFiles
rm ~/tmpFiles/.*
# copies all files in ~/tmpFiles whose names end with .html
# to directory /tmp
cp ~/tmpFiles/*.html /tmp/
# remove all files ending with .c or .o
rm *.[co]
# remove files whose extension is a lower-case character
rm *.[a-z]
touch /tmp/file1 /tmp/file2
ln -s /tmp tmpLink # create a link tmpLink to the directory /tmp
ls /tmp # display files in /tmp directory
# enter the directory /tmp by referencing the softlink tmpLink
cd tmpLink
ls
chmod +x main.sh # change file mode to display with the -F flag
ls
ls -F # ppends a character denoting file type
ls -l # displays more detailed format
# show path to file matching command ls
which ls
# display PATH variable (note the current directory is the third
# directory in the list below, denoted by the period notation)
echo $PATH
# add the directory /home/joe/bin to PATH
export PATH=$PATH:/home/joe/bin
echo $PATH
echo "Call me Ishmael.
Some years ago—never mind how long precisely—
having little or no money in my purse," > mobyDick.txt
cat mobyDick.txt | head -n 1 # first line
bzip2 -v mobyDick.txt # compress .txt to .txt.bz2
# first line of compressed file
cat mobyDick.txt.bz2 | head -n 1
# using bzcat to display compressed file
bzcat mobyDick.txt.bz2 | head -n 1
mkdir tmp tmp2
echo "hello" > tmp/hello.txt
# pack all files in tmp/ into a file archive.tar
tar cvf archive.tar tmp/*
# compress the file archive.tar (creating archive.tar.bz2)
bzip2 archive.tar
# uncompress archive.tar.vz2
bunzip2 archive.tar.bz2
# moves archive.tar.bz2 to subdirectory tmp2
mv archive.tar tmp2
# change current directory to tmp2
cd tmp2
# unpack the tar file archive.tar in current directory
tar xvf archive.tar
# Example: a sample ~/.bash_profile file
# customize the prompt appearance
export PS1="\\u@\h \\W]\\$"
# add current directory to PATH
export PATH=.:$PATH
# avoid overwriting files with output redirection
set -o noclobber
# prompt user for removing or overwriting files with rm, cp, mv
alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'
# store most recent terminal commands in the file .bash_history
export HISTFILE=".bash_history"
# alias for printing last 10 commands
alias h='history | tail'
# alias ls and ll to use my favorite flags by default
alias ls='ls -Ft'
alias ll="ls -Fthalr";
echo '#!/usr/bin/env bash
echo this is a bash script file that accepts two arguments
echo $1 $2' > printTwoArgs
chmod a+x printTwoArgs # add executable permission to scriptFile
./printTwoArgs one two # executing script by listing its path
To view the logical volumes, the following query uses WMIC (Windows Management Instrumentation Command-line) to list the names of logical volumes:
wmic logicaldisk get name
## Name
## C:
## D:
##
Note: PowerShell examples in the book ran on Windows; the code on this website runs on Linux — the output may vary.
get-psdrive –psprovider filesystem | select name
pwd
# Example: change the current directory properly in Command Prompt
# rem switch to d:
# d:
# cd \
# cd c:\windows
# echo %CD%
# rem to change directory to c:\windows, switch to c:
# c:
# echo %CD%
# Example: display directory hierarchy
# tree /A
## Folder PATH listing
## Volume serial number is 0123-4567
## D:.
## +---Data
## | \---Speech
## \---Code
##
##
compact /c foo.txt
## Compressing files in C:\Users\Geish\Desktop\
## foo.txt 5967 : 4096 = 1.5 to 1 [OK]
## 1 files within 1 directories were compressed.
## 5,967 total bytes of data are stored in 4,096 bytes.
## The compression ratio is 1.5 to 1.
##
touch file.txt
ls -l file.txt
chmod g+w file.txt # add write permission to group
ls -l file.txt
chmod a+w file.txt # add write permission to all users
ls -l file.txt
chmod a-w file.txt # remove write permission from all users
ls -l file.txt
chown guest file.txt
ls -l file.txt
runas /user:geish "cmd /k echo hello"
whoami
## my-desktop\geish
##
echo hello >> output.txt
echo world >> output.txt
cat output.txt
# create a text file and then print it in uppercase
echo this is a text file > a.txt
tr "a-z" "A-Z" < a.txt
# convert text file to a sorted list of distinct words
# annotated with their count
echo this file is a text file > b.txt
tr < b.txt -cs "[:alpha:]" "\n" | sort | uniq -c
tr < b.txt -d ' ' # remove all white spaces
echo hello >> output.txt
echo world >> output.txt
type output.txt
## hello
## world
ssh [email protected]
# copy file /home/joe/file1 from server1.domain.com
# (authenticating as user joe) to local home directory
scp [email protected]/home/joe/file1 ~/
# copy entire home directory (including sub-directories)
# on server1.domain.com to local ~/tmp
scp -R [email protected]/home/joe/* ~/tmp/
# copy local files ~/file2.* to home directory on remote computer
scp ~/file2.* [email protected]/home/joe/