My wiki: Editing Command line tips

## Detachable terminal sessions
You can use ''screen'' or ''dtach'' to make sure that if your connection is dropped from a linux server, that your session is preserved and you can link back to it. This is especially useful for keeping processes running and being able to come back and check on them later.

## Detachable sessions with SCREEN
Screen is very powerful, it's like a text-terminal window manager. You can create a session with multiple terminal windows all detachable and reattachable in one screen session.

Steps:
[[[
- Make sure you have ''screen'' installed on the target server. From a session on that server, run {{sudo yum install screen}}
- Start a ''screen'' session by entering {{>screen -S ''mysession''}} at the prompt (replacing ''mysession'' with a name you'll remember when you re-attach)
- screen is mostly run with ctl-a &lt;something&gt; key combos, for example, ''ctl-a h'' will bring up a help screen, and ctl-a ctl-c creates a new shell.
(((
* A trick to remember is that in screen your normal vertical scrolling won't work. See the help to see how to activate scroll-mod
* Another trick is that {{ctl-a}} is co-opted by ''screen'' while you are in a screen session. To get an actual ctl-a, type {{ctl-a a}}\\
To detach a screen session so you can logout, type {{ctl-a d}} inside of screen, then you can log off (if you are disconnect, the detach happens automatically)
)))
- To re-attach a session, log back in to the machine, and then you can re-attach with {{screen -r ''mysession''}}
(((
* If you log in to a machine, you can check that there's a screen session with {{screen -list}}, and connect to a detached session with {{screen -r}}
)))
]]]

## Detachable sessions with dtach
Another, less powerful but lower-overhead tool to use that lets you detach like in screen but doesn't do the other screen stuff is {{dtach}}. It is not as powerful as screen, but has the advantage that you can see with the {{ps}} command what is running what socket.

Steps:
[[[
- Make sure you have [i]dtach[/i] installed on the target server. From a session on that server, run {{sudo yum install dtach}}
- [i]dtach[/i] uses named sockets, so you will be creating a socket in {{/usr/tmp}}, and running a bash session (type {{dtach --help}} or check the ''man'' page for details):
(((
* {{dtach -c /tmp/checkbotrun bash}} creates a process run by dtach with the socket <tt>/tmp/checkbotrun</tt>
)))
- in the bash session, run whatever program you want
- Detach your session with [b]ctl-\[/b] key if you want
- Re-attach later (even after logging off) with {{dtach -a /temp/checkbotrun}}
(((
* when you log back into the machine after detaching and logging, off, the job will appear as the initial dtach command, e.g. {{dtach -c /tmp/checkbotrun bash}}
)))
]]]

## How to Set Up a User-level Perl Library
If you can't figure out how to get elevated perms to install Perl modules and apps for everyone to use, you can set them to your local user account. Below where it says ".profile" you may need to change depending on your linux/unix environment to {{.bash_profile}}. This example also grabs and sets up[i]cpanm[/i] on the first line, which you can skip if you already have it:
  curl http://cpanmin.us | perl - -l ~/perl5 App::cpanminus local::lib
  eval `perl -I ~/perl5/lib/perl5 -Mlocal::lib`
  echo 'eval `perl -I ~/perl5/lib/perl5 -Mlocal::lib`' >> ~/.profile
  echo 'export MANPATH=$HOME/perl5/man:$MANPATH' >> ~/.profile

You will now have a local perl library, <tt>~/perl5</tt> where all subsequent perl module installs you do from the command line will go.

## How to read the system logs on RHEL Machines

system logs are not in /var/log anymore. App and service logs are, ex: Apache, cron. But if you want to see the system log, you need to run [b]{{jourtnalctl}}[/b], probably with {{sudo}}, a la {{sudo jourtnalctl}}. There are details on how to use this command here (also a man page, of course):

(((
* https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/System_Administrators_Guide/s1-Using_the_Journal.html
* https://www.certdepot.net/rhel7-interpret-system-log-files/
)))

## How to see what services are running on an RHEL linux server
Two steps:
(((
* systemctl services listed with shell command <tt>systemctl list-unit-files</tt>
* SysV services listed with shell command {{chkconfig --list}}
)))

## How to add a user to a linux server
 To add a new user:

ex: <tt>sudo useradd &lt;username&gt; --groups webpub -p &lt;password&gt;</tt>

(the '''-e''' expires the account, if you have to put in an expire date, use YYYY-MM-DD format for expire date. You can expire the account *after* you create it, with <tt>usermod</tt>: ex: <tt>usermod &lt;username&gt; -e </tt> )

[b]The password has certain rules[/b] (can't be a dictionary word or close), and this is not enforced with {{useradd}} and if it *is* too simple, the user won't be able to log in even after you create the account. So '''make it complicated OR you can set the password manually to make sure it passes after you create it as above''':

ex: <tt>sudo passwd &lt;username&gt;</tt>
 
 New password: 
 BAD PASSWORD: it is too short
 BAD PASSWORD: is too simple
 Retype new password: 
 etc

Also, if you want to add them to the [i]sudoers[/i] group, an admin has to use {{sudo visudo}} to edit the sudo file in vi, add the new user like the other people in the list:
 ## Read drop-in files from /etc/sudoers.d (the # here does not mean a comment)
 #includedir /etc/sudoers.d
 trusteduser         ALL=(ALL)       ALL
 ...
 &lt;username&gt;          ALL=(ALL)       ALL

## How to preview your Markdown document
{{.md}} files in git are written in Markdown. Even if you find a tutorial and figure it out, you may not be using a tool with preview to write or edit your markdown document. Here is one: https://daringfireball.net/projects/markdown/dingus

## How to pretty-print JSON at the command line

Check out this page: http://www.skorks.com/2013/04/the-best-way-to-pretty-print-json-on-the-command-line/

For quick-and-dirty, if python is already installed, you can use a python tool <tt>mjson.tool</tt>, you just pipe the output to it, ex:
 curl http://stp-trt-otpapi.soundtransit.org/otp/routers/default/index/agencies | python -mjson.tool |less

There is a C tool called [[http://lloyd.github.io/yajl/ YAJL]] mentioned there that works great that can be grabbed to your machine with  [i]yum[/i] to a linux virtual:

sudo yum install yajl

You can cat the file and pipe it into the YAJL tool like so, for example, if you wanted to save the file (first example) or don't care to (2nd example):
 wget --output-document=test.json http://oba-otp-prod.soundtransit.org/otp/routers/default/index/agencies/1
 cat test.json | json_reformat | less

OR
   curl http://oba-otp-prod.soundtransit.org/otp/routers/default/index/agencies/3 | json_reformat | less

The other advantage of YAJL is that its a library and it has bindings that allow it to be used in other languages like [i]perl[/i] and  [i]ruby[/i]

## How to add up a column of numbers in a text file in Unix shell

You can use sed or perl to get your file to just a list of numbers, then this will give you a running total in the form of "{current added value}, {current total}":
 perl -anle '$x+=$_ for(@F); print "$_, $x"; END {print $x}' filename.txt

## How to add a prefix to a set of files in Unix shell

There is a bash thing called [[http://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#Brace-Expansion Brace Expansion]] that just expands what's in the braces, so for example:
 for i in *.txt; do mv {,40_}$i; done;

If you had one file in the folder, "test.txt" the above would issue the command {{mv test.txt 40_test.txt}}. The comma at the beginning means no prefix, and if you wanted a bigger expansion for different purposes, you can add more {{\{,40_,50_\}}} etc

## How to pretty-print CSV files at the command line
From https://www.stefaanlippens.net/pretty-csv.html (follow the Mac directions)

Make a script with the special less flags and perl described in that website: in your user {{~/bin}} folder (create it if you haven't).

For example, create a zsh/bash script file called  {{~/bin/pretty-csv.sh}}, containing:

#!/bin/bash
  perl -pe 's/((?<=,)|(?<=^)),/ ,/g;' "$@" | column -t -s, | exec less  -F -S -X -K

Make it executable (chmod ug+x ~/bin/pretty-csv.sh) and create a bash alias for it (e.g in .zshrc or .zprofile)

alias pretty_csv='~/bin/pretty-csv.sh'

Afterwords, you can invoke it on a file:
  pretty-csv stops.txt

or pipe an output to it:
 sed -ne '/Line/p' routes.txt | pretty-csv

## How to prepend text into stdout output in Unix shell
There are two ways to do this:

* You can use semi-colon in bash and bash-like shells to, say prepend the columns from a CSV file to a grep. You may need to use paretheses as well.
** Example:   <tt>(head -1 routes.txt ; grep "[A-Z] Line" routes.txt ) | pretty-csv</tt>

* You can use a parenthesis with a ''"(prepend text && cat)"'' construct, as follows. I guess this is better if you are building from left to right and remember you need to prepend after you already started typing your pipe command
** Example:   <tt>grep "[A-Z] Line" routes.txt | (head -1 routes.txt && cat) |pretty-csv</tt>

Summary
Password