Unix

POSIX

POSIX is a subset of Unix, standardized by the Institute of Electrical and Electronics Engineers (IEEE). POSIX is an acronym for "Portable Operating System Interface for Unix."

Basic User Management

• Enabling the root account:

  sudo passwd
  

• Disabling the root account:

  # `--lock`
  sudo passwd -l root
  

date

The date command lets you get the date in your terminal. It contains the -I flag which allows you to get the date in ISO8601 format. -Is to print the ISO8601 date as well as the current time up to seconds.

Examples

• Print the date in ISO8601 format

  date -I
  

• Print the local date & time in ISO8601 format

  date -Is
  

• Print the UTC date & time in ISO8601 format

  date -uIs
  

disown

If you have a process running in the background of your terminal, you can release it. The process will continue, retaining the original PID, but it will no longer be part of your terminal session.

# disown the process with job id %2
disown %2

stat

stat() is a Unix system call that returns file attributes about an inode. ¹ The semantics of stat() vary between operating systems. As an example, Unix command ls uses this system call to retrieve information on files that includes:

• atime: time of last access (ls -lu)
• mtime: time of last modification (ls -l)
• ctime: time of last status change (ls -lc)

You can use stat to get information about a file.

• Get the octal permission code

  stat -c '%a' <file>
  

• Get the owner's user name

  stat -c '%U' <file>
  

• Get the owner's user ID

  stat -c '%u' <file>
  

• Get the owner's group name

  stat -c '%G' <file>
  

• Get the owner's group ID

  stat -c '%g' <file>
  

du

The du command stands for disk usage. It can tell you information about how much space a given file on your computer is taking up.

Viewing the size of a directory

• In human-readable format

  du -sh ~/Desktop
  

• In kilobytes

  du -sBK ~/Desktop
  du -sk ~/Desktop
  

• In megabytes

  du -sBM ~/Desktop
  du -sm ~/Desktop
  

• In gigabytes

  du -sBG ~/Desktop
  

• In terabytes

  du -sBT ~/Desktop
  

You can also set the environment variable DU_BLOCK_SIZE to have a default setting for the size of blocks reported by du

• Configure du to use 1GB blocks

  export DU_BLOCK_SIZE=1G
  

lsof

The lsof command stands for list open files. On the surface, this doesn't seem useful, you'd know what files are open on your computer. However, Unix thinks of everything as a file. This includes sockets, ports, etc.

• Check which program is running on a given port with the command below:

  # Check for processes running on port 4000
  lsof -i :4000
  # Check for processes running on a range of ports, from 400 to 500
  lsof -i :400-500
  
  # Returns only the PID
  lsof -ti :4000
  

• Check for connections to a specific host

  # Check for connections to a specific host
  lsof -i@172.16.12.5
  # Check for connections to a specific host, but only on port 22
  lsof -i@172.16.12.5:22
  # Check for connections to a specific host on a range of ports, from 1249 to 65535
  lsof -i@172.16.12.5:1249-65535
  

• Check which files, sockets, ports are opened by a given PID

  lsof -p 4125
  

• Get all information about files opened by processes run by command ruby

  lsof -c ruby
  

• Get just the process id corresponding with processes run by command ruby

  lsof -tc ruby
  

• Get the process id of just processes run by user tommy

  lsof -i -u tommy
  

mkdir

The -m flag allows you to specify the octal permissions code for the files inside the directory to be created.

• Make a folder whose contents can only be written by the user

  mkdir -m 755 folder
  

head

By default, head will return the first 10 lines of the specified file. You can use the -n flag to specify the number of lines that should be returned.

• Print the first 5 lines of the file

  head -n 5 file.txt
  

• Print the first 5 lines of output

  ls | head -n 5 file.txt
  

• Print everything except the last 5 lines of the file

  head -n -5 file.txt
  

tail

By default, tail will return the last 10 lines of the specified file. You can use the -n flag to specify the number of lines that should be returned.

• Print the last 5 lines of the file

  tail -n 5 file.txt
  

• Print everything except the first 5 lines of the file

  tail -n +5 file.txt
  

• Print the last 5 lines of the output

  ls | tail -n 5 file.txt
  

gzip

Although more people are familiar with the .zip extention written by microsoft, there also exists the GNU zip .gz extension, which has a slightly higher performance benchmark in comarison to .zip.

• Compress file.txt into file.txt.gz

  gzip file.txt
  # => Removes file.txt and creates file.txt.gz
  

• Decompress file.txt.gz into file.txt

  gzip -d file.txt.gz
  

• Use the -k flag to --keep the original file.txt when creating the compressed file.txt.gz

  gzip -k file.txt
  # => Creates file.txt.gz without deleting file.txt
  

• Use the -l flag to --list some statistics about the compression, such as the percentage smaller the compressed file is compared to the original.

  gzip -l file.txt.gz
  

• Use the -f flag to --force the archive to be created, removing the pre-existing output file if it already exists.

  gzip -cf file.txt
  # => Deletes file.txt.gz if it already exists
  

dig

The Unix dig command, which stands for "Domain Information Groper" is a program used to query DNS servers for the records held by domain names.

dig @1.1.1.1 helpful.wiki A +short
# => 185.199.108.153

• The @1.1.1.1 specifies the Domain Name Service provider that we want to use. In this case, it's the IP address owned by Cloudflare. If you'd like to specify a nameserver to use by default, you can edit the file /etc/resolv.conf

• The helpful.wiki is the domain that we want the records for.

• The +short is the flag that removes all information except the answer provided by the domain name service. Other useful flags include +trace which will show the path taken.

• The A is the argument that specifies we'd like the IP address returned to us. Other valid options include NS for the nameservers used by this domain, MX for mail exchanges, and TXT for text annotations.

tar

tar is a tool used for archiving files onto tape drives, which is why the program stands for tape archive. Today it is still used to create a compressed copy of a directory. tar is used to create tarballs, a term used to denote an archive .tar file.

• The .tar filetype denotes a tarball
• The .tgz filetype denotes a compressed tarball
• gzip will understand how to decompress a file with the .tgz extension, and will return the .tar version of the file after it is uncompressed.

Useful flags

Creating an archive

When creating an archive, the -C or --dir option is very useful. It allows you to specify which directory to be in for the subsequent operations. In the case of archiving a file, by default tar will create the archive including every directory specified in the path. If you tried to type tar -cf archive.tar ~/example then it would actually create an archive of /Users/tommy/example where the root directory is /Users, containing a single directory /tommy, which itself contained a single directory /example. To avoid this, you must specify which directory tar should operate from. In this case, it should be the parent directory of example which is /Users/tommy.

Examples

• Create an archive of ~/example with root directory example

  # Short form
  tar -C ~ -c example -f archive.tar
  # Long form
  tar --dir /Users/tommy --create example --file archive.tar
  

• Create a compressed archive of example

  # Short form
  tar -C ~ -c example -f archive.tgz -a
  # Long form
  tar --dir /Users/tommy --create example --file archive.tgz -a
  

• Append a file to an archive

  # Short form
  tar -f archive.tar -r extra.txt
  # Long form
  tar --file archive.tar --append extra.txt
  

• Listing the contents of an archive

tar -f archive.tar -t
tar --file archive.tar --list
# -> example/spaced out.txt
tar --file archive.tar --list --quoting shell

• By default, tar will use escapes to denote file paths, but can optionally specify file paths using literal strings, the same way sh does, by adding the --quoting shell argument to the command.

• Unpack a tarball

  tar --extract --verbose --file archive.tar
  tar -xvf archive.tar
  

• Unpack a compressed tarball (exactly the same)

  tar --extract --verbose --file archive.tgz
  tar -xvf archive.tgz
  

• Unpack the tarball archive.tgz inside directory ~/Documents

  tar -xvf archive.tgz -C ~/Documents
  tar --extract --verbose --file archive.tgz --dir ~/Documents
  

tree

Getting Started

brew install tree

• View the directory structure up to 4 layers deep

  tree -L 4
  

• View only directories, not files

  tree -d
  

• View only files/folders matching a wildcard pattern

  tree -P '.md'
  

• View only files/folders that don't match a wildcard pattern

  # 1 directory
  tree -I 'node_modules'
  # >1 directory
  tree -I 'lib|bin|node_*'
  

wc

• Count the number of characters in file.txt

  wc -m file.txt # the number of characters (supports utf-8)
  wc --chars file.txt
  

• Count the number of words in file.txt

  wc -w file.txt # the number of words
  wc --words file.txt
  

• Count the number of lines in file.txt

  wc -l file.txt # number of lines
  wc --lines file.txt
  

• Count the number of words using heredoc style notation

  wc -w << EOF
  one
  two
  three
  EOF
  # => 3
  

• Count the number of chars using herestring style notation

  wc -m <<< "你好"
  # => 3
  

• Count the occurences of a single char / from stdin

  echo '/Users/tommy/local/temp' | tr -cd '/' | wc -c
  

rs and datamash

• Given a spreadsheet of rows and columns, transpose the rows and columns

  # Using BSD `rs`
  rs -T < input.txt > output.txt
  
  # Using GNU `datamash`
  datamash transpose --whitespace < input.txt > output.txt
  

awk

FS

• By default, awk delimits entries by runs of whitespace. If FS contains more than one character, it will delimit the line based on every occurance of the pattern.

  # "./pattern.awk"
  
  BEGIN {
  # Delimited once per comma
  FS = ","
  # Delimited once per tab
  FS = "\t"
  # Delimited once per consecutive run of tabs
  FS = "\t+"
  # Delimited on every comma, tab, and underline
  FS = "[,\t_]"
  }
  

• Matching a pattern to a specific column in each row

  # "./pattern.awk"
  
  # If the first entry is "me"
  $1 ~ /^me$/ {
  print "person: " $1 " blog: fire"
  }
  

man

Typically, a manpage for a given command is located at /usr/share/man/man1/command.1

• View the location of the man page associated with printf

  # Short form
  man -w printf
  
  # Long form
  man --path printf
  

• Configuring DNS nameserver resolutions

  sudo vi /etc/resolv.conf
  nameserver 1.1.1.1
  nameserver 1.0.0.1
  nameserver 2606:4700:4700::1111
  nameserver 2606:4700:4700::1001
  

• Adding a symlink for an executable file

```shell
sudo update-alternatives /usr/bin/yarn yarn /usr/bin/yarnpkg 1
```

You can also do the same for python3

```shell
sudo apt install python3
sudo apt install python3-pip
sudo update-alternatives --install /usr/bin/python python /usr/bin/python3 1
sudo update-alternatives --install /usr/bin/pip pip /usr/bin/pip3 1
```

File Systems

File System Hierarchy

• View the manual on file system hierarchies

  man hier
  

Volumes & Drives

The collection of non-volatile memory stored on a computer is kept in one or more Volumes. The collection of data is the Volume, and the device this volume exists on is what's known as the Drive. Back in the day, these drives were implemented as discs, hence the term Floppy Disk Drive and Hard Disk Drive. These days, we're moving on from disc drives, instead transitioning to Solid State Drives, which use transistors and circuits to store the volume (which contains the file system).

The commands for performing various operations are different on macOS and Linux. I'll try to include an example of how to do each in the relevant section. (I ran out of time while writing this, so this is to be continued for now...)

curl

PAC Files

A Proxy Auto-Config PAC file contains a JavaScript function that decides which proxy a given network connection (URL) should use. curl does not support PAC files, but I'm putting this note here for now.

• the -x or --proxy specifies to curl to use the following argument as the HTTP proxy. Unless specified otherwise, it will default to using port 3128.

  # Using an HTTP proxy
  curl -x http://remote.net:80 http://example.com
  
  # Using password protected HTTP proxy
  curl -U username:password -x http://remote.net:80 http://example.com
  
  # Using a SOCKS5 proxy
  curl -x socks5://remote.net http://example.com
  

Authentication

Article: Authentication: Everything curl

curl --anyauth \
    --user USERNAME:PASSWORD \
    URL

cron

Example crontab using zsh globbing

SHELL=/bin/zsh

# When the computer reboots,
# delete any files in ~/tmp that haven't been accessed in over a day
@reboot rm ~/tmp/**/*(.ad+1) 2> ~/tmp/cron.log

# When the computer reboots,
# delete any directories in ~/tmp that are empty
@reboot rm -rf ~/tmp/**/*(/^F) 2> ~/tmp/cron.log

# Month, Hour, Day, Month, Weekday
#   *     *     *     *       *

# @yearly
0 0 1 1 * echo 'Once a year' &> ~/log.txt

# @monthly
0 0 1 * * echo 'Once a month' &> ~/log.txt

# @weekly
0 0 * * 0 echo 'Once a week' &> ~/log.txt

# @daily
0 0 * * * echo 'Once a day' &> ~/log.txt

# @hourly
0 * * * * echo 'Once an hour' &> ~/log.txt

• Edit crontab file for the user tommy

  crontab -u tommy -e
  

• on Linux, each user's crontab file can be found at /var/spool/cron/crontabs

• on macOS, each user's crontab file can be found at /usr/lib/cron/tabs

zip

• Create an archive containing the contents of inside folder/

  # Short form
  zip -r sandbox.zip sandbox
  
  # Long form
  zip --recurse-paths sandbox.zip sandbox
  

less

• On Unix systems, the system-wide binary lesskey file is stored in /usr/local/etc/sysless

Useful Flags

• -S: Truncate any lines that extend beyond the length of the terminal
• -R: Render any ANSI escape sequences that specify SGR formatting
• -s: Squeeze consecutive blank lines into a single blank line

ln

Useful Flags

• -s: Create a symbolic link
• -v: Write output to stdout when links are made
• -n: If the destination file exists, and is a symbolic link, don't follow it
• -i: Interactively prompt the user for a response before removing a file
• -f: Forcefully remove existing destination files

coreutils exclusive flags

• -b: If there is an existing file at the destination, make a backup of it
• -T: Treat the destination supplied as a file, never a target directory

sudo

Useful Flags

• -v: Escalate permissions during a script, so that sudo commands can be run inside a script

• -n: Only run the command if you don't have to prompt for a password

• -S: Read the password from stdin and write the prompt to stderr

• -s: Run the shell specified by the SHELL environment variable

• Ask for permission to run sudo commands in a script

  #!/bin/zsh
  
  # Ask for elevated privileges with -v
  sudo -v
  
  sudo -n echo "Success, I have root access!"
  sudo -n echo "I am $(whoami)"
  
  # Discard elevated privileges with -k
  sudo -k
  

• Logging into the root user

  # Using the user `root`'s password
  sudo -i
  
  # Using the root user's password
  su
  

• Logging into the user tommy's account

  sudo su -l tommy
  

• Execute a sudo command using a plaintext password

  sudo -S <<< "letmein" 2> /dev/null echo "No password"
  

• Allow the user tommy to type sudo commands without having to input his password

  # As the root user
  <<-EOF > /etc/sudoers.d/tommy
  tommy ALL = NOPASSWD: ALL
  EOF
  sudo chmod 644 /etc/sudoers.d/tommy
  sudo chown root:wheel /etc/sudoers.d/tommy
  

who

• See which users are logged on, and from where

  who
  # tommy ttys001      2019-07-31 15:05
  # tommy ttys002      2019-07-31 15:21 (127.0.0.1)
  

diff

• Show the differences between two files in side-by-side format

  # [ Short Form ]
  diff -y one.txt two.txt
  
  # [ Long Form ]
  diff --side-by-side one.txt two.txt
  

chsh

• Change the default shell for user tommy to /bin/sh

  chsh -s /bin/sh tommy
  

uname

• Print the name of the current operating system

  uname
  # Darwin
  

sysctl

• Print the number of CPUs on the current machine's processor:

  sysctl -n hw.ncpu
  

  Output:

  4
  

arch

• Print the current architecture, either i386 for Intel architectures capable of running Intel's 32-bit.

arch

i386

tr

• Translate the character \n to , for the contents of /dev/stdin

  grep -o 'regex' | tr '\n' ','
  

visudo & vipw

These are two special commands to manage the super user and password files on your computer

/etc/localtime

The file that determines the timezone of the computer is located at /etc/localtime

To change what time it is, make a symbolic link from another timezone, and store it at this location.

• Set the timezone to America/Los_Angeles

  sudo ln -sf /usr/share/zoneinfo/America/Los_Angeles /etc/localtime
  

• Set the timezone to Etc/UTC

  sudo ln -sf /usr/share/zoneinfo/Etc/UTC /etc/localtime
  

motd

• Disable standard message banners that appear when logging into a UNIX-like system (tested on raspbian and ubuntu)

  sudo chmod -x /etc/update-motd.d/*
  

• You can add a custom script to be run when an interactive shell is starting up by adding an executable file to /etc/update-motd.d/*

dns-sd

The dns-sd diagnostic tool is useful for network scanning. If you are trying to find users to connect to via ssh, this is a great way to find out what their multicast DNS (mDNS) name is, such as tommy.local

• Browse the local network for .local domains to connect to via ssh

  dns-sd -Z _ssh._tcp
  

• Lookup detailed information about Tommy's MacBook Pro

  dns-sd -L "Tommy's MacBook Pro" _ssh._tcp
  

• Look up all advertised bonjour services

  dns-sd -B _services._dns-sd._udp .
  

Filesystem Hierarchy Standard

• /bin Binary files

• /etc: Host specific system configurations (config files)

• /lib: Shared libraries

• /sbin: System binaries

• /tmp: Temporary files

• /include: C/C++ header files

• /lib: Object files and libraries

• /libexec: Executable binaries that are only run by other commands, not by the user

• /man: Manual pages

• /src: Source code

• /share: Architecture independent hierarchy

• /mnt: Temporary mount point for a filesystem

Disk Utility Functions

• Get a report of the last recorded amount of memory available in storage

  df -kh
  

• Begin recording a new entry for the last recorded amount of memory available in storage

  du -ch
  

Encoding / Decoding Text

You can use the uuencode and uudecode commands to encode and decode text files.

Archiving Tools

These are a couple of archiving tools I am interested in exploring later. Both of these tools boast high compression rates, and seem useful.

• zpaq: https://github.com/zpaq/zpaq

  ZPAQ is an open source command line archiver for Windows and Linux. It uses a journaling or append-only format which can be rolled back to an earlier state to retrieve older versions of files and directories. It supports fast incremental update by adding only files whose last-modified date has changed since the previous update.²

• dar (Disk Archive): https://github.com/edrusb/dar

  dar is a shell command, that makes backup of a directory tree and files.

Docs just now