Skip to content

Linux macOS Administration Notes

Jump to bottom
Chris Jones edited this page Sep 7, 2019 · 292 revisions
Contents for Linux & macOS Administration Notes
*nix, BSD, & macOS Universal Tooling
macOS
Linux
Useful Links
TODOs

*nix, BSD, & macOS Universal Tooling 🔝

The concepts outlined in this section should be applicable to most if not all modern OS's derived from a UNIX platform.

Backing up with rsync 🔝

To display a progress meter when transferring a file to a remote box, use rsync and not scp

rsync -e "ssh -p4242" --info=progress2 /path/to/mr-fancy.txt remote:/path/to/

One of the many benefits of backing up with rsync is that it performs incremental backups by default, so it should not recopy files if they have already been backed up.

Backing up a root partition using rsync 🔝

The below write up will demonstrate how to backup an SD card that contains Raspbian on it.

To perform a full system backup of a root file system using rsync

rsync -aAXh --exclude=/path/to/rsync-exclude-file --info=progress2 /path/to/source/directory/or/file /path/to/destination/directory/or/file

To restore from an rsync backup reverse the source and destination paths.

A typical rsync excludes file, ie. rsync-excludes will contain

/.fseventsd/*
/boot/*
/dev/*
/lib/modules/*
/media/*
/mnt/*
/proc/*
/run/*
/sys/*
/tmp/*
/var/log/*
/lost+found
/etc/fstab
/etc/mtab
/etc/modules
/etc/network/interfaces

rsync > Backing up across a network, ie. LAN

rsync is useful for backing up files and folders across a network attached disk, ie. a large mass storage device attached to a Raspberry Pi on a LAN. However, there are some gotchas to be aware of.

  1. I did not get any decent results using NFS for whatever reason, I was averaging ~ 30KB transfer speeds from my MBP to ext USB device attached to a rpi.
  2. NFS is kind of a PITA to get setup and going.
  3. Installing sshfs on a local box, ie. my MBP is far less painless then setting up NFS.
  • make sure that the local $USER and GROUP exist on both the local and remote systems along with having the same UID and GID for best results.
  • even though there is a pi user on the stock Raspbian OS, I'd suggest creating a new user that has the same name and GID as on the local box (my MBP).

After all the above criteria has been met, then one can mount the mass storage device attached to the pi using the newly created credentials. If all goes well the rsync files should match the local UID and GID (my MBP) and the files contained on the USB ext drive (mass storage device) should have the same UID and GID

❗️ As far as I can tell, any non root user can mount the USB device / filesystem / partition on the pi and as long as the local user sshfs's the remote filesystem using the same remote / local credentials all should be good.

❗️ If the remote filesystem is mounted using sshfs such as,

sshfs pi@hostname.local:/remote/mnt/point /local/mnt/point

then issues could arise where when rsyncing file to the remote filesystem will the UID of the pi user, ie. NOT HOT DOG 🌭

Useful rsync flags 🔝

  • -a, --archive archive mode; files should be archived, meaning most characteristics are preserved

the -a flag implies --recursive and also implies --links.

  • -A, --acls preserve ACLs and update the destination ACLs with the source ACLs.

  • --exclude exclude the directories contained within curly braces.

  • --info=[FLAGS] control over the output rsync displays to STDOUT.

    • --info=help to display a list of flags that can be used with --info.
      • Ex --info=progress2

  • -h, --human-readable print output in friendly human-readable format.

  • -l, --links symlinks are recreated on the destination.

  • --no-i-r, --no-inc-recurssive disable incremental recursion

  • -P, --partial --progress keeps partially transferred files

  • -r, --recursive to recurse into directories

  • -S, --sparse handle sparse files efficiently, ie. this is useful when backing up files such as Docker images or virtual hard disks for emulators such as QEMU.

  • --stats record a verbose set of statistics of the file transfer.

  • -X update the destination with extended attributes to be the same as the source.

  • -x, --one-file-system prohibits rsync from going beyond the local file system.

  • -u, --update update destination file with source files if the destination file already exists.

  • --delete-excluded removes files from destination that no longer exist on the source. This is flag is useful when running into the below error message when running rsync

cannot delete non-empty directory:

Useful rsync commands 🔝

dat new new new rsync command

rsync \
...
--filter 'protect /path/to/dir/to/not/delete/but/add/new/files'
/path/to/src
/path/to/destination/

To backup a file or dir without preserving UID and GID

rsync -a --no-o --no-g /path/to/local/dir /path/to/backup/

To backup a directory of files from a local disk to an external volume on macOS

rsync \
--archive \
--acls \
--info=progress2 \
--human-readable \
--stats  \
--update \
-X \
--one-file-system \
--sparse \
--delete \
--delete-excluded \
--exclude-from=/path/to/rsync-excludes-file \
/path/to/src \
/Volumes/external-disk/path/to/bkp/dir/

The above command should create a 1:1 backup the source directory to destination volume, and also delete files on the destination that are no longer present on the source, and should also update file attributes and files in modifications have occurred since last backup.

A useful command that can compliment the above rsync command when backing up files is using the watch command.

There may be a yarn / NPM package that installs a watch binary that can conflict with the watch command installed by brew that is located in /usr/local/bin, so the watch package may need to be relinked, or specify the absolute path to watch

/usr/local/Cellar/watch/3.3.15/bin/watch
Complimentary watch commands
watch -n 1 -d 'du -hs ~/path/of/dir/to/watch
echo "the below two cmds not hotdog recently"
watch -d -n1 -c -x du -hs /path/to/bkup/dir/to/watch/
watch -d -n1 -c -x du -s /path/to/bkup/dir/to/watch/

To copy files using rsync and ssh across a network

rsync -avz \
-e "ssh -o StrictHostKeyChecking=no \
-o UserKnowHostsFile=/devnull" \
--progress [user]@[hostname.local]:/path/to/remote/file_or_directory /path/to/local/file_or_directory

The above command will copy a remote file or directory using ssh authorization and using rsync for the copying / transfer process, and copy a remote file from a remote box on the local network to the computer that the rsync command was issued from.

To do a simple file / directory transfer using rsync and have it display a progress bar for each file transferred

rsync -ah --progress /path/to/src/file_or_dir[/] /path/to/dest/file_or_dir

The above command is useful for displaying a progress bar when copying large files or directories.

rsync > Copying large files over a network or between filesystems

There are times were extremely large files need to be copied across filesystems and if the rsync command does not complete or fails for whatever reason it's useful to how to resume a failed rsync operation.

To start a rsync operation that can be resumed

rsync -ah --partial --info=progress2 /path/to/lrg/src/file /path/to/lrg/dest/file

If the above rsync operation fails / stops for whatever reason run the below command to resume

rsync -ah --append-verify --info=progress2 /path/to/lrg/src/file /path/to/lrg/dest/file

If screen or tmux is installed on the box running the rsync commands run the rsync command within tmux so one can logout or exit the current terminal / shell session and bring the job back to the foreground at a later date.

❗️ If a rsync job is started in the foreground and not within a terminal multiplexer the job can be suspended and moved to the background, however if the job is disowned then there is not an easy way to associate the disowned job back with the $USER who started the job and the rsync command will more than like fail for reasons I do not yet know. All that said run rsync within a terminal multiplexer to avoid this nonsense.

Useful Links rsync 🔝

Security 🔝

To calculate the SHA1 of a file

sha1sum /path/to/file.tar.bz

Working with find 🔝

To find / search for particular information on a topic through the system man pages, ie. editor

apropos editor

apropos searches man pages reading the description for a particular program and then printing it to STDOUT within the terminal.

To find all regular files related to a particular search query

find / -iname "*silverlight*" -type f 2>/dev/null

The above command will perform a case insensitive search of all files on the local system with the text silverlight contained anywhere within a file name, and suppress any errors to STDOUT, ie. the dreaded permission denied errors and what not.

To delete all files within a particular folder / directory and preserve the folder / directory itself.

find . -path ['*/mr_fancy_pants_dir/*'] -delete

A practical example

find . -path '*/node_modules/*' -delete

will delete all files and directories within the node_modules directory.

To find all directories on a file system with a particular name

find / -type d -name "mr-fancy-dir" -ls

To find all directories on a local file system with the word .kext in the directory name

find / -type d -name "*.kext"

To find all binary / executable files along with shell scripts within a specified directory

find /opt/code -executable -type f

To set the executable bit for all binary files within a specified directory

find /opt/code -executable -type f -exec chmod a+x {} \;

VirtualBox stores its custom kernel extensions within,

/Library/Application Support/VirtualBox/

To find all regular files with a specific criteria on a system using find

find / -name '[mr-fancy-pants]' -type f

To replace all spaces ␣ with a hyphen - for files and directories

find -name "* *" -print0 | sort -rz | \
while read -d $'\0' f; do mv -v "$f" "$(dirname "$f")/$(basename "${f// /_}")"; done

The above command needs be run with a POSIX shell, ie. bash or zsh

To change permissions of files and not folders recursively

find . -name '*.jpg' -type f | xargs chmod -x

To count the total number of files and directories within a path

find /path/to/directory -type f | wc -l

The above command will output the total count of the files and directories within the path specified. An alternative, but much more verbose solution is to use the tree command with the a flag. Personally, 🙋‍♂️ I would not recommend this solution.

Useful Examples using find command 🔝

To find all .DS_Store files on a file sytem without printing errors to STDERR

find / 2>/dev/null -type f -name ".DS_Store"

The above command will redirect all error output ie. Permission denied messages to /dev/null and NOT to the screen or console, thus making STDOUT more terse.

Useful Links find 🔝

Working with grep 🔝

To search for expression from STDOUT, ie. searching for nerd and plex from the output of fc-list on macOS

grep -iP '^(?=.*nerd)(?=.*plex)'

To invert the matching using grep

grep -v

To search for a pattern within all text files through a directory and sub directories.

grep -r [mr-fancy-search-pattern] .

To recursively search for a pattern in all text files, ie. source code files within the current directory

grep -rnw '/path/to/mr/fancy/dir' -e 'mr-fancy-pattern'
grep -rnw . -e 'mrs-fancy-pattern'

-r search recursively through sub directories -n print the line number for returned results -w match whole word

To search and print all markdown documents recursively for the word linux

find . -name "*.md" | xargs grep -i "linux"

grep Gotchas 🔝

When searching for the phrase set showmode grep will not return any results because the search string will explicitly search for set showmode even though the config line within the .vimrc is set noshowmode

Working with GNU sed 🔝

To iterate over all files in a directory and search for a pattern and remove all occurrences of the pattern within the files.

for i in *.rb
  sed -i -e 's/<<<<<<< HEAD//g' $i
end

The above command will remove all occurances of <<<<<<< HEAD within all the files defined in the for loop.

To verify the pattern has been erased

pt "<<<<<<< HEAD"

GNU sed > Useful Links

  • To see how I used GNU sed to process entries in my $PATH environment variable for fish shell, see

Working with GNU Core Utilities 🔝

Working with GNU Core Utilties dd 🔝

dd provided by GNU Core Utitlies can be used to backup an entire partition to a file if needed. For my particular use case I used dd provided by GNU Core Utilities on macOS installed via brew to install GNU Core Utilities which provides its variant of the dd command.

To backup an entire partition on macOS using dd

dd if=/dev/disk[NUM]s[PARTITION] of=./path/backedup.img status=progress bs=512 conv=noerror

The above command can take quite a bit of time if the partition is a large size, ie. > 100GB or more. Also, if using dd on macOS, Apple provides diskutil which is a CLI program for working with disks, partitions, and file systems.

To find the block size of a file system, which is useful for running the above dd command to backup a partition for a disk on macOS diskutil can be used.

diskutil info /dev/disk[NUM]

To create zero'd out file using dd which can be formatted using a file system such as ext2, ext3, ext4, or even HFS.

dd if=/dev/zero of=./path/to/zero.file bs=1024 count=1024000

The above command will create ~ 1GB file filled with zeros.

Working with GNU Core Utitilites split 🔝

To split a large file on a system that has split installed which is like 💁 99.9% all *nix systems.

split -b [SIZE_OF_SPLIT_FILE] "/path/to/large_file.ext" "/path/to/split_files.ext."

To join the split files back together

cat /path/to/split_files.ext.\* > /path/to/large_file.join.ext

To flash broken symlinks on macOS, GNU Core Utilities will need be installed, and setup for the shell environment.

Flash broken symlink with GNU Core Utilities

Flash broken symlink

brew install coreutils

Out of the box macOS supports $LSCOLORS

To configure BSD based $LSCOLORS online, see

Core Utilities uses a user directive for $LS_COLORS that is generated by running the dircolors command provided by Core Utilities.

Working with tree 🔝

The tree command can be used to list a hierarchical structure of a directory(ies) and it's files.

The command I use to generate file structure for this git project / repo and the accompanying submodules, I run the below,

tree -a -I "tmux_resurrect_*|.git|undo|swap|target|tmp|*.pyc|*.vader|tests|.vscode|.netrwhist|*.weechatlog|*.gpg"

Working with du 🔝

To display the to 10 largest files and directories within a given directory

\du -a $HOME | sort -n -r | head -n 10

Applying the -h flag jacks 💪 shit up

To get a recursive size of a directory and all files and directories within that directory

du -sh /path/to/mr-fancy-dir

The above command should output a single line and not the typical du output shit.

Working with df

Working with df display free disk space on macOS

macOS can perform some heavy caching of files, ie. Spotlight.app indexes and what not, ie. local library caches for applications, ~/Library/Caches That said, rebuilding the spotlight index from time to time can free up a substantial amount of disk space. Also, note that df uses a different blocksize than does Finder.app and Disk Utility.app

Example

Finder.app

Finder.app-free-space

df -h

df

To compare the df -k to the free space calculated from Finder.app multiply the Available free space, ie. in my case 572163364 by 1024 the side of a 1K block which is what Finder.app Disk Utility.app appear to be using.

$$572163364 * 1024 = 585895284736$$

Convert 585895284736 to Gigabyte notation

$$585895284736 ÷ 1,073,741,824 = Gigabytes ~ 545.65GB$$
Understanding purgeable space on macOS

When Apple released macOS 10.13 and the APFS file system for High Sierra, APFS file systems take local snapshots of the local file system, ie. the internal disk that is running macOS. That said if a large file or directory is deleted df will not be able to reflect the new changes to file system because macOS can take a snapshot of the large deletion and supposedly recover the deleted files in a Recovery Mode. These local snapshots features are coupled with Time Machine.app ie. tmutil which allows several sub commands for working with localsnapshots. However, presently I can not figure out how to permenently disable local snapshots on a APFS volume.

For a visual understanding of what I am describing, click here

  • figure out where the 40GB of discrepency space is coming from

Working with the watch command 🔝

To continously monitor a directory for file changes, ie. display the newest files at the top of the listing while continously listing results in real time.

watch -n 1 -d 'ls -lt /dev'

The above command is useful for monitoring when devices are connected / added to a system.

Legacy communication 🔝

To write a message to another user on the system

write <user>
My friendly message 8)
EOF

Working with different search utilities 🔝

Searching for files in a directory with a specific file extension 🔝

Using mdfind

To search for all files in /opt/Code/dotfiles with a fish extension using one of the below search tools, and print an integer of all files contained within the search.

mdfind is a CLI utility that interfaces with Spotlight, ie. macOS search

mdfind -onlyin /opt/Code/dotfiles -name .fish -count

The above command can report false positives, ie. a file with the name of mr-fancy.fish=

Using GNU find 🔝 
find -name "*.fish" | wc -l
Useful find commands

To change permissions for a type of file or directory recursively throughout a directory, ie. useful when working with rsync to backup directories.

find /Users/mr-fancy-42 -type d -exec chmod g+wrx {} +
find /Users/mr-fancy-42 -type f -exec chmod g+wr {} +

The above command will modify files and dirs for the user mr-fancy-42 to allow users in the same group to read, write, and execute files and dirs within mr-fancy-42's $HOME dir.

Using BSD locate 🔝 
sudo /usr/libexec/locate.updatedb
locate '/opt/Code/dotfiles/**.fish' | wc -l

To update the native locate database on macOS

sudo /usr/libexec/locate.updatedb

Using GNU locate 🔝 
/usr/local/bin/locate --version

GNU locate allows specifying a path to a database file that can be queried from the CLI, also GNU locate supports using the $LOCATE_PATH env var. However, the macOS version of BSD locate DB, ie. /var/db/locate.database is NOT compatible with the version of GNU locate installed by brew.

To create or update the locate database using updatedb provided by GNU locate

sudo /usr/local/bin/updatedb --localpaths='/' --output='/path/to/gnu-locate-database-file'

The database file will be created by the super user on the system,

To search for all files with a fish extension using GNU locate

/usr/local/bin/locate -c '/opt/Code/dotfiles/*ME.md' --database=/path/to/gnu-locate-database-file

Using the_platinum_searcher pt 🔝 
TODO

Using ripgrep rg 🔝 
TODO

Working with X11

Working with X11 on macOS

To print a list of available resolutions supported by X11

xrandr

To print the dots per inch, ie. DPI

xdpyinfo | grep -i resolution

To print the current keymap table

xmodmap -pke

To print the current keyboard layout configuration

setxkbmap -query

Working with Networking

To print a process ID related to a networking port

lsof -n -i :[PORT_NUM] | grep LISTEN

HOSTS files

Most if not all UNIX derivatives, ie. Linux, BSD's macOS support editing a hosts file, even Microsoft Windows supports a hosts file.

The hosts file on macOS is

/etc/hosts

and entries can be added to quickly navigate to a particular host on a intranet / LAN for experimenting with different services, ie. a web server and what not. A sample entry in /etc/hosts would look like the following.

10.0.1.42 the-meaning-of-life.exp # Local IP address domain name

Microsft Windows based OSes have hosts file at

C:\WINDOWS\SYSTEM32\DRIVERS\ETC\HOSTS

Working with OpenSSH

To suppress verbose output when logging into a OpenSSH server that is running on a GNU+Linux box, add a .hushlogin file in the $USER $HOME directory.

Troubleshooting Networking related issues

A fix for the below error message

sudo: unable to resolve host [NAME_OF_HOST]

There is no entry in the /etc/hosts file that corresponds to the name in /etc/hostname. see for more info.

Working with man and man pages

To print a list of paths that will be searched for man pages

manpath

Working with man pages on macOS

To print a list of directories that can be searched for man pages

man -d man

To print the directory where man would look for the MANPATH for a binary

man -d

My man page breakthrough

On macOS if binaries, ie. rvm, who, and mv etc etc, are contained within a bin directory, then if a sibling level man directory exists macOS will be able to find the accompanying man page that corresponds to the command.

./
bin/
man/

The parent paths aren't important, and editing of system level files contained with the /etc is not required.

Working with GNU Coreutils on macOS

If GNU Coreutils are installed with default names on macOS the binaries are located within

$brew_prefix/Cellar/coreutils/[MAJOR.MINOR]/libexec/gnubin
$brew_prefix/Cellar/coreutils/[MAJOR.MINOR]/libexec/gnuman

Copy the gnubin and gnuman to bin and man within the same libexec directory.

cd $brew_prefix/Cellar/coreutils/[MAJOR.MINOR]/libexec
ln -sf ./gnubin ./bin
ln -sf ./gnuman ./man

After the above directories have been copied the man command should be able to read the accomponying man pages for the commands.

macOS 🔝

Working with security related features on macOS

macOS provides a feature know as SIP System Integrity Protection which basically prevents certain files and settings from being modified.

To disable SIP on a modern version of macOS

  1. Boot into recovery mode ⌘ + r
  2. Enter the below command to disable SIP 
csrutil disable; reboot

To check the status of SIP

csrutil status

Useful Links macOS 🔝

For a comprehensive list of changes of macOS between releases see

To download macOS High Sierra officially using App Store.app via macOS

https://itunes.apple.com/us/app/macos-high-sierra/id1246284741?mt=12

To download old versions of Apple Software, ie. iMovie

support.apple.com/downloads

Networking in macOS 🔝

Setting up a NFS Server / Share in macOS High Sierra 🔝

By default macOS has the NFS daemon disabled by default, but it can be enabled.

  1. Make certain that SIP, ie. System Integrity Protection has been disabled.
  2. Edit /System/Library/LaunchDaemons/com.apple.nfsd.plist to include the -N flag, so that non root users can mount nfs shares hosted on macOS.
  3. Edit /etc/exports and include a path to a directory to be shared.

Ex

/Users/mr-fany/Movies/42 -ro -mapall=mr-fancy:mr-fancy -alldirs

The above example presently works on macOS 10.13.6 🤞

  • With super user privileges load the above mentioned plist using launchctl 
sudo -E launchctl -load /System/Library/LaunchDaemons/com.apple.nfsd.plist

To verify if the service was successfully loaded into launchd service manager with launchctl

sudo -E launchctl list | grep -i "nfsd"

The above command should list the process ID of nfsd

Troubleshooting nfsd, ie. a NFS Server on macOS 🔝

To verify if the /etc/exports has properly been configured

nfsd checkexports

If there is an error the above mentioned file, nfsd will print the errors to STDOUT.

If nfsd needs to restarted, it can be restarted directly ie. launchctl is not required. 👍

nfsd restart

To get the status of nfsd

nfsd status

To display locally served file shares

showmount -e

To verify there are no syntax errors in the above mentioned .plist file

plutil -lint /System/Library/LaunchDaemons/com.apple.nfsd.plist

Useful Resources for working with nfsd 🔝 
man nfs
man nfs.conf
man nfsd
man exports
ifconfig en0

macOS > nfs > Useful Links

ncures 🔝

To print / show the current version of ncures installed on a system, ie. macOS

cat /usr/include/ncurses.h | grep -i "version"

Working with OpenSSH 🔝

When troubleshooting OpenSSH, and making changes to ~/.ssh/config for particular host entries, neither the OpenSSH server or the current shell need to be restarted, ie. reinvoked as changes made to ~/.ssh/config are reflected in realtime when using ssh.

For an incompetent guide comparing LibreSSL to OpenSSL, see this

To set up password less login, ie. public key authentication

cat ~/.ssh/id_rsa.pub | ssh -vvv -p [PORT_NUM] [user@[hostname] 'cat >> $HOME.ssh/authorized_keys && echo "Key copied"'

If the id_rsa.pub file doesn't exists one can be generated using ssh-keygen

An alternative way to copy a public key from one host to another is using ssh-copy-id

ssh-copy-id [USER]@[HOSTNAME/IP].local -p[PORT]

[USER] is the name of the $USER that will contain the public key to allow loggin in of the current $USER

Working with ssh-agent 🔝

ssh-agent is a utility app that allows the OS / user session to store unencrypted SSH keys in memory.

The safest place to store unencrypted keys for SSH sessions would be in memory and NOT on disk.

Working with brewed OpenSSH 🔝

First off, grab a cup of joe or tea if it is that time of the day for you because this isn't something that your gunna why to skim through, that said, if you have your cup of covfefe 🙄 and you're ready to roll 🚗

  1. Download and install OpenSSH using homebrew
brew install openssh

If you prefer to use LibreSSL over OpenSSL 🙋 then you're gunna wanna check out no pun intended there the version of OpenSSH I rolled 🚬 for homebrew.

  1. 1.1 optional 
brew install openssh --with-libressl

OpenSSH does not need to be recompiled when upgrading LibreSSL, ie. I tested this when upgrading LibreSSL from 2.7.2 to 2.7.3 and running, ssh -V

The output from the above command is

OpenSSH_7.7p1, LibreSSL 2.7.3

However, the sshd service will need to be stopped and started because launchd doesn't have a "restart" feature per se on macOS in order for OpenSSH to load the new version of LibreSSL.

> launchctl unload homebrew.mxcl.sshd
> launchctl load homebrew.mxcl.sshd

If using the Apple provided service of OpenSSH which happens to be compiled with LibreSSL as well ...hmmm 🤔 wonder why that is then your first gunna wanna stop that launchd service, and unload it from launchd.

Recent version of macOS provide a nice little feature known as SIP that supposedly prevents third parties 🙋 from modfying certain files and services. thanks Obama

Personally, I disabled SIP on my local box a while ago because I like to pretend that I know what I'm doing when using "apps" such as DTrace.

  • To stop the Apple provided OpenSSH daemon if it is currently running, and prevent it from launching in the future 
launchctl stop com.openssh.sshd
  • To unload the service from launchd
sudo -E launchctl unload /System/Library/LaunchDaemons/ssh.plist
  • To verify the service is stopped
lsof -i | grep "sshd"

...and you shouldn't 🙈 see anything printed to STDOUT if sshd isn't running.

  • Copy or download my provided sshd_config along with my com.chrisrjones.sshd.plist from this repo

Personally I keep those two files checked into my dotfiles repo, and symlink them to their respected locations on macOS, so I can preserve a copy of the files, and not have to modify / keep track of two separate files. 👍

curl https://raw.githubusercontent.com/ipatch/dotfiles/master/config/brew/macOS/Sierra/etc/ssh/sshd_config --output /usr/local/etc/ssh/sshd_config
curl https://raw.githubusercontent.com/ipatch/dotfiles/master/config/brew/macOS/Sierra/Library/LaunchDaemons/homebrew.mxcl.sshd.plist --output /Library/LaunchDaemons/com.chrisrjones.sshd.plist
  • Create the respected log files and the PID for the OpenSSH service.
touch /usr/local/var/log/{sshd-error.log,sshd-out.log,sshd.log}
touch /usr/local/var/run/ssh/sshd.pid
  • Next load the newly created com.chrisrjones.sshd.plist into launchd
sudo -E launchctl load -w /Library/LaunchDaemons/com.chrisrjones.sshd.plist
  • Cross fingers 🤞 and if everything goes well you should be able to see your newly built and configured sshd server running with
ps aux | grep -i "sshd"

Pat yourself on the back if you made it this far and your server is up and running, and make sure to tip your DevOps engineer.

because developers need heros too.

Gotchas Working with brewed OpenSSH 🔝

Presently the OpenSSH provided by homebrew compiles with OpenSSL by default when installing 🙄 boo

The access list for allowing users who can login remotely using SSH is still maintained via System Preferences... > Sharing > Remote Login and toggling the users by adding their name to the Only theses users: list.

In order to get password auth working with OpenSSH Server UsePAM yes must be set within the respected sshd_config file. Most recent build of OpenSSH 7.x > do not allow running sshd as a standard user, in short sshd needs to be run as the super user on the system, ie. root if on a UNIX based system. To aviod permission issues make sure the respected `ssh_host_{dsa_key,ecdsa_key,ed25519_key,rsa_key} files have a root/wheel ownership to avoid

debug1: setgroups() failed: Operation not permitted

launchctl list | grep "chrisrjones"

presently does not display the sshd service 🤷‍♂️

Troubleshooting Working with brewed OpenSSH 🔝

I am avaible for children's birthday parties and bar mitzvahs.

If launchd runs into any errors with the provided plist file, when starting sshd it should dump the error messages into /usr/local/var/log/sshd-error.log.

If there is an issue with the OpenSSH server itself, ie. a bad configuration setting with the sshd_config the server should put the error message into /usr/local/var/log/sshd.log

Uncomment the -d flag in the above mentioned .plist file to start sshd in debug mode for more versobe troubleshooting.

-d support the use of -d -d -d, ie. 3 levels of d's. When sshd is run in debug mode the server will stop running after the first disconnection.

TODOs Working with brewed OpenSSH 🔝

With your newly aquired knowledge you should be able to setup ssh-agent.

  • Setup ssh-agent to work with newly built OpenSSH server
  • See if it's possible if the homebrewed version of OpenSSH server can play nice 👨 👧 👦 with macOS System Preferences. 🤔

Setting Up X11 Forwarding through SSH 🔝

Native macOS apps won't run on *nix because their not X11 apps, but rather their cocoa apps.

To allow for X11 forwarding through SSH, edit sshd_config

X11Forwarding yes

Connecting to a X11 Server with X11 forwarding enabled 🔝 
ssh -Y [user]@[host]

Useful Links OpenSSH 🔝

Working with Locales on macOS 🔝

To print the set locales on macOS

locale

macOS > Working with Users and Groups 🔝

To list all users on a macOS

dscl . list /Users

To list all groups on a macOS

dscl . list /groups

To create, edit, update, or delete a group on macOS from a CLI

⚠️ When managing groups from a CLI System Preferences.app will not update the Users & Groups pane with the updated data until the System Preferences.app is closed and then reopened.

echo "macOS group editing syntax"
dseditgroup -p -o VERB "GROUP_NAME"
echo "macOS create group example"
dseditgroup -p -o create "mr-fancies"
echo "macOS delete group example"
dseditgroup -v -p -o delete -n /Local/Default "mr-fancies"

An optional -v flag can be passed to print more verbose output to STDOUT

To add a user to a group on macOS, ie,

dseditgroup -v -p -o edit -a ipatch -t user wheel

To verify if a user is a member of a group on macOS

dseditgroup -o checkmember ntfsusers

To list all users of a particular group

dscacheutil -q group -a name [mr-fancy]

The above command should print all local users who belong to the group mr-fancy

To add a user to the system

sysadminctl -addUser [ipatch]

To print the GID UID $HOME directory, and SHELL for a specific user

dscacheutil -q user -a name mr-fancy

❗️ If only the UID of the user is known but not the name of the user then try the below example

dscacheutil -q user -a uid 42

To change the UID for mr-fancy on macOS using a CLI

dscl . -change /Users/mr-fancy UniqueID 42 4242
echo "❗️ chage the Users $HOME dir so they have access to it"
chown -R 4242 /Users/pi

To rename a user from the System Preferences using the Cocoa GUI see

Useful Links working with users and groups 🔝

Image Processing 🔝

To take a screen capture using a GUI on macOS use Grab

Grab is able to take a screen capture including the mouse cursor.

Useful Links Image Processing 🔝

Working with X11 on macOS 🔝

To install X11 windowing manager on macOS using brew

brew cask install xquartz

To prevent X11, ie. XQuartz.app from polluting the $HOME dir with .serverauth.* files

  1. Edit /opt/X11/bin/startx
  2. Change
xserverauthfile=$HOME/.serverauth.$$

to

xserverauthfile=$XAUTHORITY

Credit

Xcode 🔝

To install Xcode Command Line Tools

xcode-select --install

Install Command Line Developer Tools.app can be located within /System/Library/CoreServices

The command line developer tools, ie. clang are located in as of macOS 10.13.x with Xcode 9.4 installed.

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/

To display all the Xcode SDKs installed on a system

xcodebuild -showsdks

To purge old symbol files for unused versions iOS used by Xcode iOS version [X.X.X] can be deleted from

$USER/Developer/Xcode/iOS DeviceSupport/[MAJOR.MINOR.PATCH]

Working with Xcode from the command line 🔝

To compile an Objective-C file from the CLI on macOS, use xcrun

xcrun clang /path/to/mr-fancy-42.m -c

The above will only create an object file ie. .o of the hello world source file wrtiten in ojbc

To compile a Swift source file using the Xcode toolchain from the CLI on macOS, use xcrun

xcrun swiftc -c /path/to/mr-fancy-42.swift

To compile a binary that can be run from the CLI using ./path/to/hello-world-objc

clang -x objective-c -framework Foundation hello-world.m

The above command will create an intermediate object file within the directory that the source file, ie. hello-world.m resides. The -x flag explicity tells clang the source file is an Objective-C source file.

To syntax check a source file that can be compiled with clang use the -fsyntax-only flag

-fsyntax-only

Working with files in Finder 🔝

#Finder #HideAllWindows #HideAllApplications #macOS #OnlyShowDesktop #DisplayDesktop

To toggle the visibility 🙈 of hidden files in macOS >= Sierra

⌘+shift+.

A workflow for for hiding all windows including Finder ie. to only show the desktop is to

  1. Make sure the Finder.app is in focus ie. the Finder menu bar is presented
  2. Hide Others ⌘ + ⌥ + h a. One all other application windows have been hidden but the finder window.
  3. Close the active finder window, or even Quit the Finder.app a. Then there shouldn't be any application windows visible.

See notes contained within this document for allowing to Quit Finder

To hide all icons on the desktop

defaults write com.apple.finder CreateDesktop -bool false
killall Finder

To show all files & folders including hidden files and folders in macOS Finder

defualts wirte com.apple.finder AppleShowAllFiles TRUE
killall Finder

⚠️ on MacOS >= Sierra .DS_Store files are still hidden from Finder when setting AppleShowAllFiles TRUE

To show Finder command variables

defaults read com.apple.Finder

To prevent writing .DS_Store files to network shares

defaults write com.apple.desktopservices DSDontWriteNetworkStores TRUE

To show a specific setting for a Finder variable

defaults read com.apple.Finder AppleShowAllFiles

To show full path in Finder windows

defaults write com.apple.finder _FXShowPosixPathInTitle -bool true; killall Finder

To allow text selection in Quick Look

defaults write com.apple.finder QLEnableTextSelection -bool true

Working with PreferencePanes 🔝

Custom installed Preference Panes on macOS 10.13.x are located within /Library/PrefrencePanes/

Try and avoid putting custom Preference Panes within /System/Library/PreferencePanes because macOS does not like 3rd party files in that location. The above mentioned directory is 755 with root owner and wheel group perms by default.

IMHO I put my daily driver user echo (whoami) in the wheel group and set the permissions to 775, so my $USER could create custom PreferencePanes if needed.

Working with files via CLI 🔝

To delete all .DS_Store files in the current working directory & and all sub directories recursively

find . -name '.DS_Store' -delete

Tweaking macOS settings 🔝

❗️ tags macOS osx os x settings setting customize

To disable app icons from bouncing in the dock

defaults write com.apple.dock no-bouncing -bool TRUE
killall Finder
echo "inverse `TRUE` to reenable dock bounce"

macOS menu bar icon entries located in the following path

/System/Library/CoreServices/Menu Extras

Ex

/System/Library/CoreServices/Menu Extras/Volume.menu

To automatically reopen windows for an application after it has been closed on macOS

 → System Preferences... → General

restore-app-windows-macOS

Tweaking macOS settings from the CLI 🔝

To check and verify if SIP is enabled

csrutil status

To disable the macOS boot chime

nvram SystemAudioVolume=" "

To always boot macOS in verbose mode, ie. typically how a GNU/Linux displays boot messages when booting

sudo nvram boot-args="-v"

To disable boot-args ie. change macOS from booting verbosely

sudo nvram boot-args=

To check and see if verbose booting is enabled on macOS

nvram -p | grep -i "boot-args"

To always have the save dialog / panel expanded

defaults write NSGlobalDomain NSNavPanelExpandedStateForSaveMode -bool true
defaults write NSGlobalDomain NSNavPanelExpandedStateForSaveMode2 -bool true

To automatically illuminate built-in MacBook keyboard in low light

defaults write com.apple.BezelServices kDim -bool true

To turn off keyboard illumination when computer is not used for 5 minutes

defaults write com.apple.BezelServices kDimTime -int 300

To disable shadow in screenshots

defaults write com.apple.screencapture disable-shadow -bool true

When editing system settings using defaults write settings for $USER are stored in /Users/$USER/Library/Preferences as a Apple binary Plist file.

macOS Automating CLI tasks 🔝

The below directories run commands automatically

/etc/periodic/daily
/etc/periodic/weekly
/etc/periodic/monthly

To run the periodic scripts in macOS

periodic daily weekly monthly

macOS Automating tasks on macOS >= Sierra 🔝

General launchd commands 🔝

The preferred way to load a launchd services

launchctl bootstrap gui/[USER_ID] ~/Library/LaunchAgents/com.domain.service-name.plist

The preferred way to unload a launchd service

launchctl bootout gui/[USER_ID] ~/Library/LaunchAgents/com.domain.service-name.plist

To find the user id for a the current $USER operating the $SHELL

id -u

Setting up a launchd service 🔝

macOS does not support automating tasking using cron anymore. The preferred way is to use launchd to load a .plist file.

  • Create a com.chrisrjones.mr-fancy-task.plist file within,
$HOME/Library/LaunchAgents/

  • Generate a .plist file via this

  • Load the .plist file via

launchctl load -w ~/Library/LaunchAgents/com.chrisrjones.mr-fancy-task.plist

Useful Links macOS Automating tasks 🔝

Working with Audio on macOS 🔝

To determine the value of SystemAudioVolume

nvram SystemAudioVolume

Working with Power Management on macOS 🔝

To check the remaining time the system can run on battery power

pmset -g batt

To get an approximate time of how long a macOS system has been running on battery 🔋 power see Activity Monitor.app

To display all settings related to power management on macOS

pmset -g

Experimenting power management settings on macOS 🔝

To change the standbydelay time on macOS using pmset

sudo pmset -a standbydelay [INTEGER]

The above integer is in milliseconds

Ex

To set the standbydelay time on macOS for 12 hours

sudo pmset -a standbydelay 43200000

To disable standby ie. deep sleep on macOS

sudo pmset -a standby 0

Working with Disk Images 🔝

To convert an ISO file system to a macOS image file

hdiutil convert /path/to/ISO -format UDRW -o /path/to/destination

To create an ISO to be burnt as a data DVD for PC and MAC

mkisofs -J -r -o myimage.iso data_dir

mkisofs is not native on macOS 10.12.x

To copy the contents of an ISO to a USB drive

dd if=/path/to/myISO.iso of=/dev/myUSB bs=1m

Creating a Bootable Windows 10 USB Install Drive for Intel Macs 🔝

The quickest and most painless way of creating a bootable Windows 10 USB drive on macOS is to use Boot Camp Assistant provided by Apple.

🚨 A bootable Windows 10 USB Install drive created with Boot Camp Assistant will only work with a Intel Mac, ie. modern MacBook Pro, and will not work with other UEFI laptops! Etcher.app will NOT be able to create a bootable Windows 10 USB drive. 😩

Troubleshooting modern Intel Mac hardware 🔝

SMC a Intel Mac controls the the power management features of a Mac.

To reset SMC on an Intel Mac without a touch bar

  1. Press and hold shift + control + option
  2. Keep the keys pressed for 10 seconds while holding the power button.
  3. Release the power button along with the three keys
  4. Then press the power button again to turn on the computer.

To check and see if the EFI on a Intel Mac is 32bit or 64 bit

ioreg -l -p IODeviceTree | grep firmware-abi

The output from the above command should either ouput "EFI64" or "EFI32"

To boot an Intel Mac, ie. a modern MacBook Pro, ie. 8,2 version into single-user mode.

⌘+s after booting up the computer, and shortly after hearing the Mac boot chime.

To reset the NVRAM on a modern MacBook

⌘+option+p+r

To check and see if an Intel Mac is using a 32 or 64 bit UEFI system

ioreg -l -p IODeviceTree | grep firmware-abi

The above command 👆 should either print EFI64 or EFI32

To boot an Intel Mac into Internet Recovery Mode

option+⌘+R or shift+option+⌘+R

To boot an Intel Mac into Apple Diagnostic/Hardware Test mode

  1. Turn the computer completely off
  2. Hold option+D after the startup chime, select the appropiate WIFI network, and then the Mac should download the appropiate Apple Diagnosit/Hardware Test for the Mac.

Apple changed the Apple Diagnostic/Hardware Test program some time around 2013, and Macs prior to 2013 user an older blue GUI version of the test.

Working with file systems and disks, ie. storage devices 🔝

To securely wipe a partition on a disk, ie. storage medium

/path/to/shred --verbose --random-source=/dev/urandom -n1 /dev/disk[Number]s[Partition]

A quick way to list the file system, ie. HFS+ for a particular mount point

diskutil info -all

or

diskutil list

To display / show all file systems accessible to a macOS system

mount

No flags or arguments are required to display the currently mounted file systems on macOS, and the above command should the mounted disks, ie. a removable USB drive and their respected file systems accessible to macOS.

To unmount a file system on macOS

diskutil unmountDisk /path/to/disk

The above command will unmount all volumes associated with a disk.

Ex

diskutil unmountDisk /dev/disk2

To quick and dirty way to see the type of partition table a disk using on macOS

diskutil list

The above command should 99% of the time list volume 0 as either GUID_partition_scheme or FDISK_partition_scheme

Ex

/dev/disk3 (external, physical):
0: GUID_partition_shceme

To format a file system on macOS with a specific format

newfs_* -F /path/to/device

To format a USB disk with a MSDOS FAT 16 file system on macOS

Ex

newfs_msdos -v "MR_FANCY_VOLUME" -F 16 /dev/disk2

BSD derivatives of UNIX use newfs_[FS_TYPE] to where as GNU derivatives use mkfs.[FS_TYPE]

On macOS an alternative way to format a disk is to use diskutil

Ex

diskutil eraseDisk "MS-DOS FAT16" MR_FANCY_DISK_NAME /dev/disk2

To copy / burn the contents of an ISO image to a USB drive or other removable storage

sudo dd if=/path/to/mr-fancy-42.iso of=/dev/disk[NUMBER]

sudo will be required to write to block level device

Protip to print a status of the transfer on macOS

control+shift+T

To display an active progress bar when using a command such as dd use pv

pv /path/to/mr-fancy-42.iso | sudo dd of=/dev/disk[NUMBER]

Working with FAT-32 file systems 🔝

To the best of my knowledge the largest file a FAT-32 file system can store is 4GB.

To mount a FAT[16-32] file system on macOS using diskutility

diskutil mount /path/to/blockDevice

Ex

diskutil mount /dev/disk3s2

The above command will mount the file system within the /Volumes directory on the local macOS system with the respected volume label as the mount point.

To unmount a volume / partition on macOS with FAT[16-32] file system.

diskutil umount /path/to/mount/point

The path to the block device can also be specified.

Working with large files on a FAT-32 file system 🔝

To split a really large file into chunks 🤮 that can be copied to a FAT-32 file system

rar a -v4G /path/to/mr-fancy-42-universe.rar /path/to/mr-fancy-42-universe.qcow2

An alternative to using rar is using GNU Core Utilites to split the file using the aptly named split command.

split --bytes=3500M /path/to/mr-fancy-42-universe.qcow2 /path/to/output/splits/mr-fancy-42-universe.qcow2

When using the split command, specifying -bytes=4G will create files too large for a FAT32 file system.

Transferring large files to a FAT-32 file system 🔝

cp is great for copying files, however it lacks any sort of progress bar, but there are various bolt-on 🔩 solutions, ie. piping through pv or using some form of python, but from my personal expierence rsync is the best solution for transferring large files.

rsync -ah --progress /path/to/mr-fancy-42-universe.qcow2 /path/to/external/media/fat32/mr-fancy-42-universe.qcow2

-a will preserve file permissions 👍 FAT32 doesn't support permissions though -h will make the progres bar in human readable output --progress will transfer the file will outputting the progress to STDOUT

Building GNU GRUB v2 on macOS from source 🔝

  • Clone the grub source
git clone git://git.savannah.gnu.org/grub.git
  • Clone objconv from GitHub
git clone git@github.com:vertis/objconv.git

GRUB will not compile using Xcode tooling out of the box, thus the dependency for objconv

  • Build objconv 
g++ -o objconv -O2 src/*.cpp
  • Put objconv in the $PATH
  • Build GRUB to support x86_64 and efi 
cd /path/to/src/grub
mkdir grub-build
../configure --prefix=/path/to/put/the/grub/binary --target=x86_64 --with-platform=efi
make
make install

make should be run the grub-build dir and NOT the grub git source project root.

  • Optional Specify a font to use with GRUB, on macOS most system fonts can be found in /Library/Fonts, that said, choose a .ttf font for the below command.
/path/to/grub-mkfont -o share/grub/unicode.pf2 /Library/Fonts/Times\ New\ Roman.ttf
  • Create a build dir preferably in the git source root for building a self contained GRUB binary to work with a UEFI system, ie. a MacBook
cd /path/to/grub/git/src
mkdir build
  • Create a grub.cfg file, and put the following lines within the file
insmod part_msdos
set root=(hd0,msdos1)
  • Build a standalone GRUB binary to bootstrap an ISO, ie. an install media from a USB drive.
/path/to/bin/grub-mkstandalone -O x86_64-efi -o BOOTX64.EFI "boot/grub/grub.cfg=build/grub.cfg"

If all goes well there should be a BOOTX64.EFI file within the build/ directory.

Install GRUB on a removable USB drive 🔝

To install the GRUB bootloader on a removable USB drive using macOS

grub-install -v --target=x86_64-efi --efi-directory=/mnt/[USB-DRIVE-LABEL] --boot-directory=/Volumes/[USB-DRIVE-LABEL]/boot --removable --bootloader-id=grub

The above command will not succeed on a removable USB drive if --removable is omitted from the command. If the above command succeeds the following message will output to STDOUT

Installation finished. No error reported
grub-install alternative 🔝

To install grub on a GPT formatted removable USB disk using macOS a couple of steps will need to be performed

  • Verify the disk has been formatted with a GPT parititon table, and not an MBR partition table.
diskutil list
  • set a path to where the EFI partition can be mounted, and you have read/write access to the path.

Ex

mkdir /mnt/usb-gpt-efi
sudo chown -R $USER:wheel /mnt/usb-gpt-efi

If the above directory is created within the /Volumes directory then the mount point, ie. directory that holds the mounted file system will be deleted.

  • mount the EFI partition of the USB disk
sudo mount -t msdos /dev/disk3s1 /Volumes/usb-gpt-efi
  • Verify the partition has properly been mounted.
mount
  • Install GRUB boot loader to the USB drive to work with a UEFI system / boot rom.
grub-install -v --target=x86_64-efi --efi-directory=/Volumes/usb-gpt-efi --boot-directory=/Volumes/grub-for-mac/boot --removable --bootloader-id=grub

macOS and GRUB Useful Links 🔝

refind is the successor to refit

Working with the bless command on macOS 🔝

In order to boot from a removable media, ie. a USB flash drive on macOS a drive or Volume needs to blessed, using the native bless command from macOS or using the hfs-bless provided by mactel linux.

To see / print the currently blessed disk that will boot macOS

bless -getBoot

Working with automountd 🔝

automountd is a service provided by macOS that can automount file systems, disks, local & remote, ie. across the network.

The configuration files for automountd

/etc/autofs.conf
/etc/auto_master
/etc/auto_resources

To show the file systems / partitions that have been automatically mounted

automount -v

If a change is made to one of the above files, the cache needs to be flushed

automount -cv

Some useful man pages include

man automount
man automountd
man auto_master
man autofs.conf

Benchmarking HDD performance 🔝

To test HDD performance on a macOS / Linux system, see

A general universal command to test read / write performance to a storage media

time sh -c "dd if=/dev/zero of=/tmp/testfile bs=100k count=1k && sync"

Results running macOS 10.12.6 with a HFS+ filesystem on a 1TB Samsung SSD

╭─capin at rogue in /opt/Code/dotfiles (master ✔)
╰─λ time sh -c "dd if=/dev/zero of=/tmp/testfile bs=100k count=1k && sync"
1024+0 records in
1024+0 records out
104857600 bytes transferred in 0.212054 secs (494485334 bytes/sec)
        0.34 real         0.00 user         0.16 sys

╭─capin at rogue in /opt/Code/dotfiles (master ✔)
╰─λ ls -lah /tmp/testfile
-rw-r--r--  1 capin  wheel   100M May 19 20:24 /tmp/testfile
╭─capin at rogue in /opt/Code/dotfiles (master ✔)
╰─λ time sh -c "dd if=/dev/zero of=/tmp/testfile2 bs=100k count=1k && sync"
1024+0 records in
1024+0 records out
104857600 bytes transferred in 0.086153 secs (1217108667 bytes/sec)
        0.20 real         0.00 user         0.12 sys

╭─capin at rogue in /opt/Code/dotfiles (master ✔)
╰─λ time sh -c "dd if=/dev/zero of=/tmp/testfile3 bs=100k count=1k && sync"
1024+0 records in
1024+0 records out
104857600 bytes transferred in 0.299482 secs (350129766 bytes/sec)
        0.49 real         0.00 user         0.18 sys

Building macOS install / boot media 🔝

To create a bootable macOS High Sierra install media on a removable USB disk drive.

Make sure Install macOS High Sierra.app has been downloaded from the App Store.

sudo /Applications/Install\ macOS\ High\ Sierra.app/Contents/Resources/createinstall \
--volume /Volumes/Install-macOS-High-Sierra \
--applicationpath /Application/Install\ macOS\ High\ Sierra.app

To create a bootable macOS Mavericks install media USB

sudo /Applications/Install\ OS\ X\ Mavericks.app/Contents/Resources/createinstallmedia \
--volume /Volumes/<MEDIA> \
--applicationpath /Applications/Install\ OS\ X\ Mavericks.app --nointeraction

Building a USB drive / install media for an Intel Mac with GRUB 🔝

directory layout

EFI
  debian
    grubx64.ef
mach_kernel
System
  Library
    CoreServices
      boot.efi
      SystemVersion.plist
.VolumeIcon.icns

The contents of the SystemVersion.plist file will roughly look like the following

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>ProductBuildVersion</key>
  <string></string>
  <key>ProductName</key>
  <string>GNU+Linux</string>
  <key>ProductVersion</key>
  <string>GRUB V2</string>
</dict>
</plist>

Disk Management 🔝

To enable Debug mode for Disk Utility.app

defaults write com.apple.DiskUtility DUDebugMenuEnabled 1

The above command may not be applicable for macOS High Sierra as Disk Utility by default can show all partitions.

To list the supported file systems a disk can formatted with

diskutil listFilesystems

To list the available disks attached to the system

diskutil list

To erase a Volume using diskutil and not allocate the free space with a partition.

diskutil eraseVolume free %noformat% [IDENTIFIER]

[IDENTIFIER] can be printed using diskutil list

To erase an entire disk and not just a partition / volume on a disk and allocate with free space

diskutil eraseDisk free %noformat% [IDENTIFIER]

The [IDENTIFIER] can be obtained with diskutil list, and verified with the /dev/disk[NUMBER], also using the above command will format the disk with a GUID partition map, ie. known as a Scheme in the macOS world.

Ex

diskutil list

Output

/dev/disk3 (external, physical):
...

To resize a partition for a particular disk, ie. a USB flash drive using diskutil on macOS, use the eraseDisk in conjunction with the resizeVolume command.

From my empirical evidence (could be wrong on this assessment) using diskUtil on macOS it's not possible to create a partition with a specific size from a free space drive using diskUtil on macOS. The reason for this is because there are no logical volumes, ie. partitions on the disk, that said, that's why the simplest solution I found was to create a JHFS+ partition that can be resized at later date and time.

  1. Create a JHFS+ partition that occupies all free / empty space for a disk.
diskutil eraseDisk JHFS+ grub-for-mac /dev/disk3

About the above command, eraseDisk will nuke 🤯 the entire disk ie. removing all data from it, grub-for-mac will be the volume label for the disk, and /dev/disk3 specifies the path to the disk to put the JHFS file system and partition on.

  • Validate the disk is properly partitioned and formatted.
diskutil list

Should see ~ 3 partitions indexed at 0

  1. GUID_partition_scheme
  2. EFI
  3. Apple_HFS
  • Next resize the JHFS+ partition to a smaller size
diskutil resizeVolume /dev/disk3s2 512M

To get an idea of what size a file system on a partition / logical volume can be

diskutil resizeVolume /dev/disk3s2 limits

There now should be a partition on the removable USB disk that is ~ 512MB in size and formatted with a JHFS+ file system.

To create an additional partition on the above mentioned disk with a 512MB JHFS+ partition use an external program named gdisk.

  1. gdisk can be easily obtained if Homebrew is installed.
brew cask install gdisk

⚠ To modify any attributes associated with a disk gdisk needs be launched with super user privileges.

  • Launch gdisk 
sudo gdisk
  • Specify path to disk
/dev/disk[NUMBER]
  • Create a partition with they type hex code of 0700 so a FAT file system can be formatted to the partition.

To create a partition with a specific size using gdisk ie. create a 5GB on a 32GB disk use +5G

  • Format the newly created partition with a FAT-32 file system
diskutil eraseVolume FAT32 [VOLUME_NAME] /dev/disk[ID_PARTITION_NUMBER]

To erase a disk and format with a particular file system

diskutil eraseDisk [fileSystem] [label / name] [/path/to/disk]

Ex

diskutil eraseDisk fuse-ext2 samsung-1tb /dev/disk2

To create Linux compatible file system for a particular partition on a disk

/opt/gnu/sbin/mkfs.ext2 /dev/disk2s2

🚨 The above mentioned command will create a ext2 formatted filesystem. If you'd prefer to use an ext4 filesystem, use mkfs.ext4 instead. Also mkfs.ext2 requires tools to be properly installed and setup on macOS in order to format a disk with a ext{2,3,4} file system.

To disable / turn off journaling for a hfs+ disk

sudo diskutil disableJournal /Volumes/mr-fancy-disk-label

Working with fuse-ext2 🔝

To mount an external disk with an ext{2,3,4} file system using fuse-ext2

- sudo -E mount -t fuse-ext2 /path/to/external/disk/partition /path/to/fancy/mnount/point
+ sudo fuse-ext2 /path/to/block/device/partition /path/to/mount/point -o rw+,allow_other,uid=[INT],gid=[INT]

Ex

sudo fuse-ext2 /dev/disk2s2 ~/mnt/sdcard -o rw+,allow_other,uid=501,gid=20

sudo must be used to mount the file system.

To verify if the file system successfully mounted using FUSE for macOS

mount

The above command will display all locally mounted file systems.

To unmount a fuse-ext2 file system

sudo -E unmount /full/path/to/mount/point

fuse-ext2 Gotchas 🔝

When working with fuse-ext2 to mount ext{2,3,4} file systems, presently sudo is required to mount a file system boo. In order to mount a file system and to allow access for a particular user on the system

sudo -E fuse-ext2 /path/to/disk/s[num] /path/to/mount/point -o rw+,allow_other,uid=[num],gid=[num]

Ex

sudo -E fuse-ext2 /dev/disk2s2 ~/mnt/ext-m2-1tb -o rw+,allow_other,uid=501,gid=20

Useful Links Disk Management 🔝

Working with Java 🔝

To completely remove / uninstall Java JRE and JDK

rm -rf /Library/Java/JavaVirtualMachines/jdk<version>.jdk

rm -rf /Library/PreferencePanes/JavaControlPanel.prefPane
rm -rf /Library/Internet\ Plug-Ins/JavaAppletPlugin.plugin
rm -rf /Library/LaunchAgents/com.oracle.java.Java-Updater.plist
rm -rf /Library/LaunchDaemons/com.oracle.java.Helper-Tool.plist
rm -rf /Library/Preferences/com.oracle.java.Helper-Tool.plist

To completely remove Android Studio, see this

Microsoft Silverlight on macOS

To remove / uninstall Silverlight on macOS, perform a search using GNU find or locate and manually remove all files.

Ex

find / -iname "*silverlight*" -type f 2>/dev/null

Working with launchd 🔝

TL;DR

  • working with launchd / launchctl is a total pain the ass, that said, tip 💵 your dev ops engineers next time you see them.
  • when loading or unloading a launchd service, ie. a .plist file within $HOME/Library/LaunchAgents refer to the .plist file, ie.
launchctl [load/unload] $HOME/Library/LaunchAgents/mr-fancy.plist

...and NOT the Label in the .plist file.

  • when testing a launchd service refer to the value in defined below the Label within the .plist, ie.
launchctl list com.example.mr-fancy-42-service
  • if StandardErrorPath & StandardOutPath or defined in the .plist file / service, launchd should create the files if not created yet, NOTE double check permissions of directory where files will be written.
  • launchd should append to the files defined by StandardErrorPath & StandardOutPath and not overwrite them each time the service is started. 👌

To get a frame of reference on how launchd service files, ie. .plist files can be written, see /System/Library/Launch{Angents,Daemons}

From all my empirical evidence, environment variables can not be set within the StandardOutPath or the StandardErrorPath

launchd General Notes 🔝

A daemon is a program that runs in the background as part of the overall system (that is, it is not tied to a particular user). A daemon cannot display any GUI; more specifically, it is not allowed to connect to the window server.

An agent is a process that runs in the background on behalf of a particular user. Agents are useful because they can do things that daemons can't, like reliably access the user's home directory or connect to the window server.

If a launchd service / .plist file calls / runs a shell script one can verify the the shell script contains no errors by passing the -n flag as an arugument to the shell script, ie.

sh -n mr_fancy_shell_script.sh

If the above command does not print any output in the terminal then their aren't any syntax errors using that particular shell interpreter.

To test / lint a .plist file for syntax errors

plutil -lint $HOME/Library/LaunchAgents/com.mr-fancy.plist

To manually / arbitrarily start a launchd service / .plist

- launchd start $HOME/Library/LaunchAgents/com.mr-fancy.plist

To start a launchd service as a specific user via a .plist file

<key>UserName</key>
<string>mr-fancy-user</string>

Working with Environment Variables in Launchd 🔝

To create an environment variable in launchd add the below to the respected .plist file.

-<array>
-  <string>/bin/launchctl</string>
-  <string>setenv</string>
-  <string>MR_FANCY</string>
-  <string>42</string>
-</array>
+<key>EnvironmentVariables</key>
<dict>
  <key>PATH</key>
  <string>/usr/local/bin:/usr/bin:/bin</string>
</dict>

To create a launchd variable from the command line

launchctl setenv FOO "BAR"

echo $FOO will output BAR

To create a environment variable in a .plist

<string>launchctl setenv FOO "BAR"</string>
<string>launchctl setenv USER "${USER}";</string>

launchd Gotchas 🔝

launchd .plist files can contain spaces in <string> values, ie.

<string>/Mr Super/fancy/path/to/special.plist</string>

When working with launchd services, MAKE sure to remove the service if it is no longer required to use it, ie.

launchctl remove com.github.asdf-vm.plist

...otherwise the service will remain loaded even if the env var is manually removed, and the service is stopped or unloaded 😡

To get a human readable explanation of a launchd error code, ie. 127

launchctl error 127

When creating a launchd service at a system level, ie. putting a service file in /Library/LaunchDaemons/, running

lc list | grep `chrisrjones`

...will not list the service, because it is running at system level.

Useful Links launchd 🔝

TODOs launchd

  • see if environment variables can be used in .sh associated with launchd.
  • see if environment variables can be used in .plist files associated with launchd.

Working with devices on macOS 🔝

To status devices, ie. devices connected via Bluetooth or USB, macOS provides ioreg and system_profiler

Either one of the tools can be used from the CLI to status devices internal or external to a running version of macOS.

To get a crude list of devices attached to a USB bus on macOS

ioreg -p IOUSB

The -p flag will scan the ioregistry over a given plane, and in the above example the plane specified is IOUSB.

To get more verbose / detailed information about USB devices connected to macOS

ioreg -p IOUSB -w0 -l

-w0 will not constrain the output to the width of the window ,and the -l will add to the verbosity of the ioreg command.

⚠️ The -l flag is useful for listing the idProduct and idVendor

An alternative to ioreg is system_profiler

To status USB devices connected to macOS using system_profiler

system_profiler SPUSBDataType

Working with Kernel, ie. Darwin boot arguments 🔝

To manually kernel boot arguments for the Darwin kernel on macOS, edit the below file.

/Library/Preferences/SystemConfiguration/com.apple.Boot.plist

Working with Graphics Programming on macOS 🔝

Working with OpenGL on macOS 🔝

macOS does not ship with OpenGL header files for building programs against OpenGL, however Xcode can be installed to build against some of the OpenGL header files.

To the best of my knowledge Apple stopped pulling in upstream features of OpenGL as of v3.2.

macos-opengl

That said, certain apps that use Vulkan, ie. the successor to OpenGL run on macOS.

Working with Postgres 🔝

To start the homebrew implementation of PostgreSQL

launchctl load ~/Library/LaunchAgents/homebrew.mxcl.postgresql.plist

The above command will load the homebrew implementation PostgreSQL as a service using launchd on macOS.

To verify if the service is running

launchctl list | grep -i "homebrew"

To stop the newly added launchd service

launchctl stop [com.mr-fancy-service]

or

launchctl unload [com.mr-fancy-service]

Troubleshooting Postgres on macOS

Make sure the Postgres data directories on macOS have a 700 permission

/usr/local/var/postgres

Working with Screen Savers on macOS 🔝

The default location for system Screen Savers on macOS

/System/Library/Screen Savers/

The default $USER location for screen savers on macOS

~/Library/Screen Savers/

Useful Links 🔝

TODO 🔝

  • Mess around with Screen Saver project template in Xcode

Working with AppleScript 🔝

Useful Links AppleScript 🔝

Working with Messages 🔝

To instert a newline or line break using Messages

option + return

Linux 🔝

  • TODO Add a graphic to symbolize GNU+Linux to quickly see when browsing document in a web browser!

Working with dkms

To print a list of added dkms modules in the various versions of linux kernels

sudo dkms status

❗️ there is a difference between a addedd and installed kernel module.

To remove a dkms kernel module for all kernels present on a system

dkms remove <module>/<module-version> --all

The --all flag should remove all versions of the specified module for all kernels on the system.

General Commands

To print the UUID for a hard disk or a partition of a hard disk

blkid

To find the /dev node asosciated with a removable storage device, ie. a USB flash drive.

dmesg

Run the above mentioned command after attaching a USB disk to see what the /dev node, ie. /dev/sdd is asosciated with the device.

Partitions for a particular USB drive are listed as /dev/sdd1, /dev/sdd2, etc. etc.

To print the major & minor release version of a Debian system

lsb_release -a

To mount a partition with an ext2 filesytem

mount -t ext2 /dev/sda1 /opt

To change the hostname of the system

hostnamectl set-hostname mr-fancy-pants-hostname-here

To extract a directory of rar files

find . -name "*.rar" -exec unrar x -o+ {} \;

Resizing a partition for a QEMU virtual disk

If a virtual hard disk for a QEMU VM needs to be increased then fdisk can be used to increase the partition after qemu-img has been used to increase the size of the virtual disk.

To increase the size a virtual qcow disk

qemu-img resize /path/to/virtual-disk.qcow2 10G # resize disk to 10G

To add 10 Gigabytes to a virtual disk

qemu-img resize /path/to/virtual-disk.qcow2 +10G # append 10G to virtual disk

Working with Users and Groups 🔝

To change the login password for a user on a GNU+Linux system

sudo passwd $USER
sudo passwd mr-fancy

To change the UID for a user on a GNU+Linux system

usermod -u [NEW_UID] mr-fancy

❗️mr-fancy can not change his own UID and no processes can be associated with mr-fancy. If mr-fancy has associated processes run the below commands

pkill -u mr-fancy pid
pkill -9 -u mr-fancy

The same procedure is applicable for changing the GID of mr-fancy as well

groupmod -g [NEW_GID] mr-fancy

❗️ Be certain to update old files and dirs that were associated with the GID see

To print the GID for a particular group name ie. staff

getent group staff

To print the GID of a group name of macOS

dscl . -read /Group/staff | awk '($1 == "PrimaryGroupID:") { print $2 }'

Raspbian > Working with sudo

To set an empty / blank password for the user pi on Raspbian

  1. Make certain the pi user has sudo permissions.

❗️ The pi may require setting the default editor to run visudo otherwise the default editor will be set to nano

To set the default editor for visudo

sudo update-alternatives --config editor
  1. Add the below line to /etc/sudoers 
pi ALL=(ALL) NOPASSWD:ALL
  1. Delete the current password for the pi user
sudo passwd -d `whoami`

After the above settings have been made the pi user should be able to login without being prompted for a password.

Working with tty / pts from a CLI

A quick way to list tty's / pts's on a GNU+Linux distro

w

  • ❗️ On macOS listing all active terminal sessions is a little different, and the w command only shows the initial console session when running w. However, a crude count can be obtained by listing all /dev/ttysXXX* devices within the dev directory.

  • [To print print the active pseudo terminal / tty for a shell

tty
  • To kill a different tty from an existing tty session
pkill -9 -t pts/[NUMBER]

TODO

ps -ft pts/[NUMBER] pts/42 pts/24
kill [PID] [PID]

❗️ BSD and macOS are different beasts from GNU+Linux

To list the default options that will be applied when creating a new user on the system

useradd -D

To delete a user from the system, for more info see

userdel mr_fancy_user

To create a new user with a specified shell, home directory, and user name

see

useradd -s /bin/bash -m -d /home/vanilla vanilla

To print a users home directory, ie. /home/mr-fancy-42-user

grep mr-fancy-42-user /etc/passwd | cut -d ":" -f6

Setting new files in a directory to belong to a particular group 🔝

  1. Set the directory to have a umask of 007
umask 007

The above command will make all newly created files to

u=rwx,g=rwx,o=

To verify the above setting

umask -S

To remove the setuid ie. sticky bits from a directory

chmod ug-s /path/to/dir
echo "or try the below"
chmod 0755 /path/to/dir

To print all users on a Linux box

cut -d: -f1 /etc/passwd

To print all groups on a Linux system

cut -d: -f1 /etc/group

To print user and groups values of a particular user

id [user_name]

To print the group memberships for a particular user

groups [mr_fancy_user_name]

To create a new group on a Linux system

groupadd [mr-fancy-42-group]

To add an existing Linux system user to an existing group

usermod -a -G [name_of_group] [name_of_user]

If adding a $USER to a group on a remote system through ssh more than likely the $USER will have to logout and then log back in after adding the $USER to a group.

Ex

usermod -a -G prettyladies diana

Working with disks block devices and filesystems on Linux 🔝

To list all the partitions for block devices, ie. disks on a Linux system

lsblk
echo "for more a descriptive listing"
lsblk -f

To create, modify, delete a partition on Linux system, gparted is a popular command

gparted only works with devices and volumes, ie disks and partitions.

To display the free space on a disk using gparted

gparted /dev/disk
(parted) print free

To display the partition labels for a particular device, ie. disk

e2label /dev/disk

To display a partition label for a particular volume / partition

e2label /dev/disk[Partiton#]

To set a label for a partition which can referenced ie in a /etc/fstab file for auto mounting a file system when present.

e2label /dev/disk[Parttion#] [mr-fancy-partition-label]

To remove a partition label

e2label /dev/disk[Parttion#] ""

To create FAT-32 file system on a removable USB device on the 3rd partition / volume of the device

mkdosfs -F 32 -I /dev/sda3

/dev/sda is dependent on the disk path!

To remove a removable USB disk from a Linux system, make sure all partitions have been unmounted, then use udisksctl

  • keywords power, off, turn, poweroff, turnoff, USB, powerdown, down
udiskctl power-off -b /dev/sd[A,B,C]

[A,B,C] refers to the disk that one would like to unmount.

To safely remove a USB mass storage, 1) unmount the disks, 2) then power down the drive

sudo udisks --unmount /dev/sd[???]
echo "the below command will power down the drive
sudo udisks --detach /dev/[sd[???]

Working with autofs

To unmount all filesystems that have been mounted with autofs

umount -a -t autofs

Working with udev 🔝

For an exhaustive guide to working with udev, see

To print the currently installed version of udev

udevadm version

To setup a custom network interface name using udev for a standard interface name, ie. eth0 or wlan0 create a 70-persistent-net.rules within /etc/udev/rules.d

SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="01:23:45:ab:cd:ef", NAME="eth-pi"

SELinux 🔝

Stretch enabled support for SELinux on Debian, but is disabled on a default install.

To check whether SELinux is enabled / disabled on Stretch

grep FSC /etc/default/rcS

If the output from the above command returns

FSCKFIX=no

then SELinux has not been enabled on the box.

Another way to check if SELinux is enabled / disabled on a Debian 9.x box.

sestatus

Debian 🔝

Setting up backports on Debian Stretch 🔝

To setup Debian Stretch 9.x to work with a backports-repo

  1. Edit /etc/apt/sources.list file or add a .list file, ie. a debian-strech-backports.list file within the /etc/apt/sources.list.d/ directory and add the below entry.
deb http://ftp.debian.org/debian stretch-backports main

To install a package from a backports repo

apt-get update
apt-get -t stretch-backports install [mr-fancy-backport-package]

For more info on working with backports see

Debian > Working with apt & apt-get

To generate a list of all packages installed with apt / apt-get

dpkg --get-selection

Debian 9.x and above provides apt-mark

To generate a list of all manually installed packages by a user, and not their dependencies

apt-mark showmanual > [mr-fancy-42-package.list.txt]

To reinstall all the manually installed packages provided by apt / apt-get

xargs < [mr-fancy-42-package.list.txt] apt-get install

To print a list of files associated with a package

dpkg -L [mr-fancy-42-package]

To show package records for a particular package

apt-cache show postgresql

To remove a specific version from the list

apt-get remove postgresql=9.6+181+deb9u1

To list all packages related to particular string, ie. [postgres]

dpkg --get-selections | grep postgres

To display the package version number of all packages currently installed on the system

dpkg-query -l

To find the package that provides a particular ".h" file

dpkg -S <name_of_h_file.h>

Example,

dpkg -S stdio.h

Output

libc6-dev: /usr/include/stdio.h
libc6-dev: /usr/include/bits/stdio.h
perl: /usr/lib/perl/5.6.0/CORE/nostdio.h

Adding a key signature to apt-get, see

To remove artifacts from apt-get installed packages

rm -rf /var/lib/apt/lists/*

then rerun the below command.

apt-get update

To see what packages can be upgraded

apt-get upgrade --dry-run

To search for a particular package from the apt-get sources

apt-cache search [mr-fancy-apt-package]

To show or list the dependencies a package requires using apt

apt-cache depends [mr-fancy-package]

ie.

apt-cache depends xsel

or

apt-cache depends xclip

Running apt-get update on Debian 9.x was giving an expired an key error.

See the following issue about getting the updated key in order to resolve the expired key issue for OpenSuse server, ie. it is related to fish shell.

To reinstall an app on Debian using apt-get

apt-get install --reinstall postgresql

To download the source for a particular package that would installed using apt-get command

apt-get source [packagename]

To pin / hold back a package from being upgraded via apt or apt-get

sudo apt-mark hold [PACKAGE_NAME]

To unpin / unhold a package

sudo apt-mark unhold [PACKAGE_NAME]

To add a path for every user on the system edit /etc/login.defs add the respected path entries for normal users, and the root user.

Troubleshooting apt on Debian
Debian > apt > troubleshooting partial install

Error message

apt 2 not fully installed or removed.

To list the partially installed or removed packages

dpkg -C

Then remove or reinstall the offending apt packages

apt install [OFFENDING_PKG]
apt install --force-yes [OFFENDING_PKG]
apt remove [OFFENDING_PKG]
apt remove --force-yes [OFFENDING_PKG]

From my empirical evidence apt performs without errors if there is only a single version of python in the PATH, ie. do not have pyenv to use both a 2.7.x and a 3.{6,7}.x variant of python with the below command

pyenv global system 3.6.5 2.7.15

Instead use

pyenv global system 3.6.5

Debian date, time, timezone 🔝

The quickest way to setup the proper time + date, and correct timezone is to install ntp if a network connection is present.

apt-get install ntp

To configure the Debian system for a specific timezone, ie. US Centeral TZ

sudo dpkg-reconfigure tzdata

Follow the curses based menus for setting up tz data.

Debian Working with KVM 🔝

Debian Stretch, ie. 9.x 🔝

Setting up SELinux on Debian 🔝 

apt-get install selinux-basics selinux-policy-default auditd

  • Download the _load_selinux_policy script.

  • Move the script into /usr/share/initramfs-tools/scripts/init-bottom

  • Run update-initramfs -u

  • Run selinux-activate to configure GRUB and PAM to work with SELinux

    1. Create /.autorelabel

  • Reboot the system.

    1. Run check-selinux-installation to see the errors / warnings about SELinux

  • Finally add enforcing=1 to /etc/default/grub

  • Reboot

Debian packaged kernels provide support for SELinux out the box.

SELinux Useful Links 🔝

Working with apt 🔝

To remove / purge old packages .deb files on a Debian system

sudo aptitude clean

Research missing packages with apt 🔝

To get a list of files a package can provide install apt-file, generate a DB for apt-file and then query apt-file with search strings. Also, apt-file can take a list argument to see what files a package provides, ie.

apt-file list libglib2.0-dev | grep '.m4$'
sudo apt-get update; \
sudo apt-get sudo apt-get install apt-file \
sudo apt-file update
apt-file search [string]

Credit

Working with source code using apt-get 🔝

Source code for Debian packages installed via apt-get is currently being maintained on salsa.debian.org

An account will need me made on the GitLab clone thingy, ie. server specifically designed for Debian packages, and then the apropriate package can be cloned to a local box.

To clone a package from salsa.debian.org

  1. sudo apt-get install git-buildpackage
  2. gbp clone git@salsa.debian.org:[mr-fancy-guest]/[mr-fancy-42-package].git

To build a debian package from source using a git clone

cd [mr-fancy-42-package]/
gbp buildpackage --git-pbuilder

Pbuilder Howto 🔝

Working with source using apt-get Useful Links 🔝

Troubleshooting apt 🔝

If apt-get is having difficulty finding scripts to run, ie. /usr/sbin/update-info-dir make certain that /usr/sbin is in the current $USER's $PATH

Upgrading from Jessie to Stretch 🔝

Make sure all Jessie updates have been applied to the system first. To make sure no updates need to be applied

apt-get update upgrade --dry-run

Then run the two below commands to make sure there are no issues pending

dpkg --audit
dpkg --get-selection | grep hold

Services such as postgres will need to be halted when performing a system upgrade because tools / libs ie, lib-c require them to be halted during the upgrade process.

To stop the postgres service on Debian jessie

/etc/init.d/postgresql-[version] stop

If postgres has been setup to work with systemd then the postgres server can be halted with the below command.

systemctl stop postgres

Then edit sources.list

nvim /etc/apt/sources.list

Remove or comment out source entries related to jessie and add the below entries related for stretch

deb http://mirrors.digitalocean.com/debian stretch main
deb-src http://mirrors.digitalocean.com/debian stretch main

deb http://security.debian.org/ stretch/updates main
deb-src http://security.debian.org/ stretch/updates main

deb http://mirrors.digitalocean.com/debian stretch-updates main
deb-src http://mirrors.digitalocean.com/debian stretch-updates main

Then run the below commands again

apt-get update
apt-get upgrade

Then run the below command

apt-get dist-upgrade

If the upgrade process halts for whatever reason it can be resumed using the below command

apt-get -f dist-upgrade

After everything has been installed, and the system has been rebooted, the old kernels can be purged.

dpkg --list | grep linux-image
aptitude purge ~ilinux-image-\[0-9\]\(\!`uname -r`\)

To purge / remove unneeded packages from the system

apt-get remove --purge

Working with cron 🔝

To test the various cron tasks on a Debian system

run-parts /etc/cron.daily
run-parts /etc/cron.weekly
run-parts /etc/cron.monthly

To arbitrarily run a cron job with the bare metal cron setup see this

To display the particular cron jobs for a user on a Debian box.

crontab -l

The file printed from the above command is stored in the following in the following location on the system.

/var/spool/cron/cron/crontabs/

When running a cron job / shell script make sure the script does not contain any commands being run with sudo. Long story short, a password will have to be stored in plain text somewhere on the system. 😬 When using env vars in a shell script run by cron make sure the env vars are set within the script itself.

See this for more details.

Working with pam.d 🔝

To restart the pam.d service

/usr/sbin/pam-auth-update

Working with locales 🔝

Update February 17 2019

The below command resolved locale issues I was having after upgrading locales on Debian Stretch see for more info.

sudo sh -c 'echo en_US.UTF-8 UTF-8 >>/etc/locale.gen' && sudo /usr/sbin/locale-gen

To list the locales on a system

locale -a

To configure locales on a Debian based distro, ie. Raspbian

dpkg-reconfigure locales

Follow the instructions in the curses based menu program. ⚠ dpkg-reconfigure locales must be run as root and not used via sudo

🚨 When generating locales using locale-gen run as root do not use sudo if you value your sanity 👩‍⚕️

To remove erronous locales from a system edit /etc/locale.gen and comment out the unnecessary locales.

The following link appears to have resolved the locale / perl warnings

To fix a broken locale system on a Debian box, see

To remove unused / extra locales on a Debian box

sudo apt-get install localepurge

Unselect all the locales that aren't being used.

When changing locale for a particular user, sometimes reloading the shell, ie. exec bash will not pick up on the locale change, so it's important to "logout" and "log" back in.

Useful Commands for Working with locales 🔝 

apt-get install localepurge
bass . /etc/default/locale
dpkg-reconfigure localepurge

Postgresql 🔝

General Notes about working with PostgreSQL 🔝

To check the server version of PostgreSQL if it has been installed

postgres -V

To check the installed version the PostgreSQL client

psql -V

TL;DR

To restart PostgreSQL on Debian 8.x & 9.x

/usr/lib/postgresql/10/bin/pg_ctl -D /path/to/where/DBs/are/located -l /path/to/server/log/file.log start

pg_ctl can be located in different locations depending on how PostgreSQL was installed.

For my particular Digtal Ocean droplet the below command works 👍

/usr/lib/postgresql/10/bin/pg_ctl -D /var/lib/postgresql/10_3_1 -l /usr/local/var/postgres/server.log start

The above command should be run as the postgres user on the system, DO NOT USE sudo or root to start the daemon unless you like headaches 🤕 If the postgresql.service is being used in conjuction with PostgreSQL then the postgres can be controlled via Systemd

systemctl start postgresql.service

To check and see if postgresql server is running on the system

ps -ef | grep postgre

To reload postgresql

/etc/init.d/postgresql reload

To create a user for Postgres

createdb [mr-fancy-42-user]

The above command helps alleviate the common below error message

psql: FATAL: database [user] does not exist

Postgres Backing up databases 🔝

To implement a backup solution for postgres, see

To setup a cron job to run the backup scripts daily copy or symlink pg_back* in the Linux dir within the jobs dir.

To test a daily cron job on Debain Linux

run-parts -v /etc/cron.daily -v

To back a Postgresql database, see for more details.

pg_dump [name_of_database] > name_of_backup_file.sql

Postgres Restoring a database 🔝

As of June 30 2018 I have a couple of shell scripts running to automate the process of taking daily snapshots of my Postgres databases. That said the scripts produce two back up files, ie. a .gz and a .custom

To restore a Postgres database from a snapshot using the .custom file, use the following,

pg_restore --dbname=mr-fancy-42-db --verbose --host=localhost --username=mr-fancy-username --password ~/path/to/mr-fancy-42-db.custom

When restoring a DB that works with a rails web application, make sure to restore the DB, before running the migrations, and then run the migrations after restoring the database. 👌

If the schema.rb file is affected from importing data, and migrations are rerun, more than likely puma will need to be restarted after intitial deploy with new data in the database.

Working with Postgres on Debian 🔝

pg_ctlcluster supersedes pg_ctl and in practice pg_ctlcluster should be used in favor of using pg_ctl when possible. That said, pg_ctl can be found at the below location

/usr/lib/postgresql/[MAJOR.MINOR]/bin/pg_ctl

The PostgreSQL configuration files, ie. pg_hba.con are located in the below location

/var/lib/postgresql/10_3_1/

The postgres binaries, ie. commands are located in the below location

/usr/lib/postgresql/10/bin

psql is located in both /usr/bin/psql and /usr/lib/postgresql/10/bin

Getting postgres to work with systemd

postgresql@.service acts a template for setting up postgres to work with systemd.

The units file that interacts with Postgres server is,

/lib/systemd/system/postgresql.service

As of March 21, 2018, all database records / data are stored in the below location

/var/lib/postgresql/10_3_1

Upgrading Postgres from 10.3.x to 10.4.x 🔝

  1. Installed Postgresql 10.4.x using apt-get
  2. Dumped all existing data from 10.3.x
pg_dumpall > backup-may-30-2018
  1. Stop the current database server, ie. Postgresql 10.3.x
pg_ctl stop

if using systemd

systemctl stop postgresql
  1. Create a new database cluster 
initdb -D /var/lib/postgresql/10_4_1
  1. Start the new database server
postgres -D /var/lib/postgresql/10_4_1

systemd

systemctl start postgresql
  1. Restore the data from 10_3_1
psql -d postgres -f m backup-may-30-2018

For more detailed instructions see

Working with psql 🔝

To connect to a particular database using psql

postgres=# \connect database_name

To list all the databases

postgres=# \list

To list all tables for a particular database

postgres=# \dt

To change the owner of a particular database

postgres=# ALTER DATABASE [mr_fancy_database] OWNER TO [mr_fancy_owner_name];

PostgreSQL Useful Links 🔝

Working with PHP 🔝

To restart the PHP service / process

/etc/init.d/php5-fpm restart

To find the the local user on the system associated with the PHP service / process

Create a php file with the following snippet

<?php
phpinfo();
?>

Then look for a heading of Environment in the rendered output from the snippet.

Nginx 🔝

To restart nginx web server

service nginx restart

The above command won't work on Debian 9.4 out of the box because it does not contain the service binary, instead use systemd, ie. systemctl

Working with MySQL 🔝

To install MySQL on a Debian based distro

apt-get install mysql-server

To complete the MySQL installation

mysql_secure_installation

To access the MySQL shell as the root user

mysql -u root -p

To list local MySQL databases on the system

show databases;

To show local permissions a particular user has on a system 📦

show grants for 'database_user'@'localhost';

Sometimes the below command will need to be run after making permission changes

flush privileges;

Useful Links MySQL 🔝

MediaWiki 🔝

Make sure the proper user ie, the php user associated with the system has write permissions to the images directory

To see if Google Analytics is installed within mediawiki

Special:Version

Useful Links MediaWiki

Working with SSH 🔝

To properly set permissions of SSH directories and files, see 🙈

Some useful settings for setting up SSH permissions

  1. $HOME/.ssh should be set to 700
  2. $HOME/.ssh/[encryption_type].pub should be set to 644
  3. $HOME/.ssh/[encryption_type] ie. id_rsa the private key should be set to 600
  4. The users $HOME directory should not have a group write permission, and should be set to 755

A useful command for debugging an SSH session, ie. debugging X11 Forwarding

ssh -v -X -l [mr_fancy_user_name] -p [mr_fancy_port] [hostname]

To suppress the last login from X.X.X.X IP address

  1. Edit /path/to/sshd_config

Add the below line to the file

PrintLastLog no

To suppress the connection from IP closed

  1. Edit $HOME/.ssh/config

Add the below setting

LogLevel QUIET

Working with systemd 🔝

To restart a Systemd service

sudo systemctl restart [NAME_OF_SERVICE]

Ex

sudo systemctl restart sshd

systemd History 🔝

Systemd was introduced by RedHat to replace / compliment SYSV for Linux distributions. Traditionally SYSV used syslogd or rsyslogd to log errors and output to log files for services and daemons running on the system. However systemd provides own logging facility which interacts for special journal files.

Systemd Working with journaling 🔝

To control the journaling settings for systemd edit the below

/etc/systemd/journald.conf

To get journaling to work properly on Debian 9.x I had to explicity set compression for the journal files to off

To allow a standard user to access and manipulate log files controlled by the journaling capabilities provided by systemd, perform the below

  1. Add the standard user, ie. whoami to the systemd-journal group.

See notes above, about working with users and groups, if needed.

  1. Make sure the path to the journal directory and all sub folders and files are writeable by the systemd-journal group, ie. 775 for /var/log/journal

To rotate logs generated by systemd, one can use the --vacuum-time=2weeks flag / option to the journalctl command to trim / shrink the size of log files.

On Debian 9.x systemd stores system services in the below location

/lib/systemd/system/[mr-fancy-system.service]

systemd also stores $USER services in the below location

/usr/lib/systemd/user/[mr-fancy-user.service]

To start a systemd service for a particular $USER

systemctl --user start [service-name]

The full path and the .service extension can be omitted if the unit file, (ie. that is what systemd calls the .service files.) is either of the above mentioned paths.

To add a additionaly systemd unit file the preferred location to add a unit file is within the /etc/systemd/system directory on Debian.

The unit files located in the /etc/systemd/system will take precedence over the ones that are located in /lib/systemd/system. Also, if there is a already a Debian provided systemd unit file in /lib/systemd/system the user provided will be chosen if located in the /etc/systemd/system directory. 👍

To install libpam-systemd

apt-get install libpam-systemd

See, 🙈 for more details

To list all enabled services from systemctl

systemctl list-unit-files | grep enabled

To analyze system boot time, ie. measure startup time of GNU+Linux using systemd performance

systemd-analyze

When modifying a systemd .service file, the below command will need to be executed in order for systemd to pick up on the changes.

systemctl daemon-reload

Troubleshooting systemd

To remove / delete old systemd log files

sudo journalctl --flush --rotate
sudo journalctl --vacuum-time=2d

The above command will remove all non active logs, and delete all active logs that 2 days or older.

Working with man pages 🔝

man pages are divided into various groups, ie. man pages about using a specific command, ie. uname will be located within Section 1 where as getting a understanding of how the programming works for the uname command will be located within Section 2

man page directory layout 🔝

Directory Section Name
man1 (1) User Commands
man2 (2) System Calls
man3 (3) Subroutines
man4 (4) Devices
man5 (5) File Formats
man6 (6) Games
man7 (7) Miscellaneous
man8 (8) Sys. Administration
manl (l) Local
mann (n) New
mano (o) Old

When working with man pages on macOS user added CLI commands can store their man page in /usr/local/share/man/man1. The man1 directory is reserved for user added CLI commands, but make certain the man page has a .1 or a .1.gz unless you don't want to read 📖 the man page ಠ_ಠ

Useful Links man pages 🔝

Working with ld 🔝

To print the ld search path, see 🙈

All Useful Links 🔝

TODOs 🔝

  • ~~~Add Table of Contents to this document.~~~

If you find any of this info helpful on your journey 🏍 click that 👆 ⭐️ star button. It sure makes me feel warm and fuzzy 🐻 on the inside.

Frequent Flyers ✈️

❤️ ~/.🛠🐈

Clone this wiki locally