bash history grep alias

In a previous post, I introduced many useful bash shell aliases that I use. There's one that I've made a notable improvement to that I'd like to share. If you spend a lot of time on the command-line and use bash, you should really try this out.

'bhg', for "bash history grep," was a pretty simple alias to search through your bash history file. Previously, I used it like this:

alias bhg='cat $HISTFILE | grep'

The deficiency I found with it is that I often keep a shell open for days (even weeks!) at a time. So, I may be searching for a command that has not been written to the history file. What I'd like, then, is one 'bhg' alias that will find any command from my current shell history or my history file. The bash 'history' built-in will give you all of the current shell's command history. We just need to strip out the line numbers -- easy enough with 'sed' -- and then con-'cat'-enate that with the history file and we're in business. So, this is what I use now to accomplish all that:

alias bhg="history | sed 's/^ *[0-9]* *//' | cat $HISTFILE - | grep "

As I mentioned before, if you find this useful, you can make it even more useful by increasing the history size variables in your ".bash_profile". Namely, 'HISTSIZE', which controls how many lines of current history will be available via the history command, and 'HISTFILESIZE', which controls how many lines of command history will be written to your history file. They both default to 500, I think, which I've found to be way too small for my purposes.

alias or function? and bash 'time'

In a previous post (and a follow-up), I wrote about a number of useful and time-saving aliases that I use in the bash shell. Those posts are pretty old now. In the meantime, I've defined another interesting alias. While nothing special in itself, I've discovered that this is one task that can be done better in another way.

The new alias is this:

pig='pkg_info | grep '

I've found it to be very useful. I often want to know what version of some port is installed, or if I have a port installed at all, and this allows me to find out quickly without paging through the whole list. For example:

mybox$ pig swfdec
swfdec-0.6.8 Flash Rendering Library
swfdec-plugin-0.6.0_1 Flash rendering plugin

It's very much like the alias "psg='ps auxw | grep '" that I'd talked about previously.

But, on a box with a large amount of software installed -- which is pretty well any FreeBSD desktop now that we have modular X -- this new command is pretty slow. Let's use bash's time utility and see just how slow:

mybox$ time pig swfdec
swfdec-0.6.8 Flash Rendering Library
swfdec-plugin-0.6.0_1 Flash rendering plugin

real 0m13.245s
user 0m0.306s
sys 0m0.233s

Thirteen seconds, wow! Immediate, subsequent re-invocations should go much faster since the file information has been cached. See:

mybox$ time pig swfdec
swfdec-0.6.8 Flash Rendering Library
swfdec-plugin-0.6.0_1 Flash rendering plugin

real 0m0.416s
user 0m0.258s
sys 0m0.163s

But, the underlying problem remains: pkg_info spends a lot of processing time going through all the installed ports, and we're only interested in one (or a few). We can do better.

Knowing that the package information is found in the directory '/var/db/pkg/' and how it's structured, we can use filename expansion and a couple of other tricks to dramatically reduce these times. After some fiddling, this is what I've come up with:

pi2 ()
{
RND=$RANDOM;
cd /var/db/pkg/;
cat *${1}*/+COMMENT > /tmp/pig.$RND;
echo *${1}* | sed 's/ /\
/g' | paste - /tmp/pig.$RND;
rm /tmp/pig.$RND;
cd $OLDPWD
}

I call it 'pi2' here for the sake of clarity. Yes, that's a literal newline for the sed replacement string (might cause problems with some versions of sed :/ ).

This function dramatically reduces times. Compare these times with those above:

mybox$ time pi2 swfdec
swfdec-0.6.8 Flash Rendering Library
swfdec-plugin-0.6.0_1 Flash rendering plugin

real 0m0.015s
user 0m0.001s
sys 0m0.023s

And, these fast times stay nearly the same when the file information is not cached. Which takes away that pesky 10 plus second wait when you want to use this utility and the file information doesn't happen to be cached already.

The bash problem

If you use bash as a login shell you might have had the problem of it getting broken and then being unable to login. Now that's a problem! I've burnt myself on this a couple of times. But, in both cases it was not a big deal to fix. (In one case, though, I was locked out over the weekend until I could get to the box.) This is more of a problem in some operating systems (like FreeBSD) where bash is not part of the base system. In that respect, it's a risk worth having because of FreeBSD's great separation of the base system and third party software -- and the ability to easily update that software. And, bash is just a great shell.

As long as you've got console, it's easy enough to fix. You can just drop into (or boot to) single user mode and then run 'chsh [<username>]'. If you don't have console and all of your network enabled user accounts use bash you've got a real problem.

So, how to prevent this once and for all? I've got several suggestions. 1) Have a back-up account that uses a shell from the base system. This is simple and effective. But, having to remember another user name and having to remember and/or update another password is not ideal. 2) Use a 'toor'-like system on your own user account. This is interesting. I tested this out and it works alright. Just reverse your username (e.g., 'kace' -> 'ecak'), create that account, then use 'vipw' to change the uid on that account to your original uid and the home directory to your home directory. But, make this other username use a base system shell. This is really nearly the same as solution 1), except you'll not need to remember a new user name.

Finally, the one I like best: no new account, no second password: 3) Change your login shell to '/bin/csh' then add this code to the ".login" file in your home directory:

if ( { bash -c 'echo blah > /dev/null' } ) then
    exec bash -l
else
    echo '=== bash broken ==='
endif

This simply tests bash every time you login and then replaces csh with a bash login shell as long as it's working. Simple and effective. It may seem like too much overhead, but it's not really. This isn't some system task that will be repeated hundreds of times a day. You don't actually login that much. The small cost in computer time of running csh and then bash and then finally bash once more is a trifle in comparison to the human time that this measure may one day save you.

I first tested and implemented this solution with '/bin/sh' and it worked, though the code is slightly different and it goes into the ".profile" file. But, upon reflection that's a bad idea. 'sh' is a little faster than 'csh', but using 'sh' requires that you have a ".bash_profile" or a ".bash_login" file in your home directory. If you don't, or something happens to them, then bash will read ".profile" and you'll be in a loop, re-executing bash over and over. Ack! Let's just solve the problem instead of creating new, more interesting ones. :)

Update:

If you want to run any tests on these ideas, you may want to "break" bash or create and assign yourself a broken shell. (Be careful! Don't lock yourself out! Test on a box that you have console access to until you are satisfied.) One quick and easy way to break bash is this: "chmod -x /usr/local/bin/bash" (or your path to bash). On the other hand, the system seems to handle a non-executable shell slightly differently from a broken shell. For testing purposes it's broken enough I think. :) But, if you want to be a stickler, a more realistic way to emulate a broken shell might be to assign yourself as a login shell a utility that always returns a non-zero exit status, perhaps '/usr/bin/false'. (And, you'll have to put it in '/etc/shells/ first.) ... In particular, something like "ssh problem-box '/bin/sh' " may seem to work while your login shell is not broken, but, in fact, the command '/bin/sh' is being passed to your login shell, and if it's broken that won't work.

Make "Home" and "End" keys work

There was an article linked in the press section of the FreeBSD site that mentioned something pretty cool. It in turn linked to another article with lots of details. More details than suit me frankly. :p But, darned useful nonetheless.

If you have to switch back and forth between W1ndows and Unix often, as do I, you may sometimes reach for a key that doesn't work as you expect it to (or at all). Sometimes when I'm in a hurry I start typing vi commands while writing an email in Outlo0k. It doesn't work. (Too bad, because when you're editing in a hurry, vi commands are what you want!) Well, getting vi or vim to plug in as the editor for Outlo0k will have to wait for another day. :) But, there are some keys that have been non-functional for me on the Unix command line. Now, I've got them back.

Like I said, the above-linked article by Anne Baretta has a ton of details. She talks about settings to enable some of these keys in X, tcsh, KDE, and more. I just scanned down to the part on bash and found what I needed. Namely, you can get the "Home", "End", and "Delete" keys working as expected on the command line in bash by creating a file named ".inputrc" in your home directory that contains the following:

"\e[1~": beginning-of-line
"\e[4~": end-of-line
"\e[3~": delete-char

As far as I can tell, you'll have to log in again to make it effective.

Wait, there's more! The article mentioned the "Ctrl-v" trick: You hit Ctrl-v and then another key or key combination and then it will show you the escape sequence that gets sent to the shell. Using that trick I discovered that the escapes for "Ctrl-LeftArrow" and "Ctrl-RightArrow" (yeah, more W1ndows lapses) and then mapped those to the appropriate bash readline commands. Just add these two lines more to "~/.inputrc" and you'll be in business with the word hopping arrows:

"\eOD": backward-word
"\eOC": forward-word

I'm in switch-hitters heaven. Furthermore, none of these bindings override the ones that were already there. That is, "Ctrl-A" still goes to the beginning of the line too, and "Alt-F" still hops the cursor forward a word, etc.

Update: Commenter Eddie led me to an interesting article at 'nixCraft' on the same subject. Playing around, I found some interesting things. First, the codes above don't work correctly in all situations!! I was doing that through ssh/PuTty. On a regular terminal, these are the codes that worked right for me:

"\e[7~": beginning-of-line #home
"\e[8~": end-of-line #end
"\e[3~": delete-char #delete
"\eOd": backward-word #ctl-leftarrow
"\eOc": forward-word #ctl-rightarrow

Secondly, if you're on a machine with multiple (command-line) users, then it makes little sense for everyone to have to create the same "~/.inputrc" file. You need only create one file with those readline settings in it, I suggest "/usr/local/etc/inputrc", and then point everyone to it by putting this line into "/etc/profile":

export INPUTRC='/usr/local/etc/inputrc'

Script to alert on high disk utilization

Below I've got a simple script that will check your disk utilization every time it's run and then alert you if it exceeds a certain level. It could be run as often as you like from cron. This script is not pretty but it's effective. It should probably be modified for your particular situation before you use it.

Although it's pretty simple, some parts of it may be mysterious if you're not familiar with the various Unix utilities it uses. Let's take a look at the code first, then I'll briefly explain some of the parts which you may want to tweak, and I'll touch on some of the other parts that would be useful to play with on the command line to learn more. So, to the script:

#!/usr/bin/env bash

if ! df -t ufs |grep ' \(9[5-9]\|10[0-9]\)% ' > /dev/null ; then
   exit 0
fi

for MP in `df -t ufs | grep ' 9[5-9]% ' | awk '{ print $6 }'` ; do
   PU=`df -t ufs | grep " ${MP}$" | awk '{ print $5 }'`
   MSG="file system $MP at $PU"
   logger -p local3.warn -t `hostname | sed 's/\..*$//'` $MSG
done

for MP in `df -t ufs | grep ' 10[0-9]% ' | awk '{ print $6 }'` ; do
   PU=`df -t ufs | grep " ${MP}$" | awk '{ print $5 }'`
   MSG="${MP}: file system full, $PU"
   logger -p local3.crit -t `hostname | sed 's/\..*$//'` $MSG
done


What this script is doing is just checking the output of the df (disk free) command and then sending a message to the system log if it exceeds a certain level. I've got separate loops for high disk utilization and full disk utilization (100%+) so that you can use different messages. The first, small code block, the if clause, is to improve efficiency by running df just once when there is no alerting condition (presumably, the case most of the time).

One thing you might want to change in this script is the logging commands, which use logger. It may be preferable to you to send an email instead of a log message (or maybe both). So, for example, you might change the logger lines above to something like this:

   echo $MSG | mail -s "`hostname | sed 's/\..*$//'` disk warning" bsdguy@fake.net

Of course, you could also do both by just adding the mail line before or after the logger line. If you use email, though, consider carefully how often you will run the job and how long it might take you to get to the machine and correct the issue. Don't spam yourself with a ton of mail! A good approach would be to break this into two scripts which are run at different frequencies and perhaps with different alerting methods (this would also make the "if" block unnecessary).

Another thing you might want to change is the level at which the script will alert. Notice that I'm just using grep to identify the high percentages. So, the part of the regular expression that says '9[5-9]' is matching any number from 95 to 99. If I wanted to change that to match anything over 80% then I'd replace it with '[89][0-9]'. If I really had to match anything at 85% and higher then I'd need an alternation and would use this: '8[5-9]\|9[0-9]'.

The regular expression '10[0-9]' is not a mistake. FreeBSD's filesystem has a reserve which means it can go over 100% utilization in df! ... Be careful cutting and pasting: the spaces inside the quotes of the regular expressions are needed. ... If you're still learning Unix, try taking apart the pipelines in the script (several commands connected by '|') and running them on the command line in parts. For example, first run "df -t ufs" then run "df -t ufs | grep ' 9[5-9]% ' ", and so on. ... This script will run fine on Linux too (either as is or with very minor modifications).

Option-ize your shell scripts

If you're like me you may have started shell scripting very simply, maybe just to collect a number of other unwieldy commands into one executable file to call with just one short command (the name of the script). Over time (hopefully!), you will have discovered many of the other useful features of shells, like variables, redirections, and conditional expressions. As one's scripts become more complicated, one mistake a person might make is to copy and then modify a script to do something that's really only a little bit different. A better way to handle this may be to keep it all in one script and just use command line options to that script. Once you start using options to your scripts (as opposed to just arguments), all sorts of great possibilities open up.

I'm using the bash shell here and this probably won't work the same way with most other shells. Below, I've put an options demo' script. If you want to give it a try, you should have bash installed and then just paste the below into a file and 'chmod' it to be executable. This system of using options with your scripts will allow boolean options, defaults with option overrides, options with arguments, options in a blob (e.g., -aBbdg5) or singly, some crude syntax checking of the options, and required arguments as well. So, first the script, then some example output, and finally some discussion of the nuts and bolts for the curious.

#!/usr/bin/env bash
USAGE='usage: '`basename $0`' [-13ds] [-o out-file] filename'
snmpver=2c
sync=0
debug=0
while [[ ${1:0:1} = '-' ]] ; do
N=1
L=${#1}
while [[ $N -lt $L ]] ; do
  case ${1:$N:1} in
     'd') debug=1 ;;
     's') sync=1 ;;
     '1') snmpver=1 ;;
     '3') snmpver=3 ;;
     'o') if [[ $N -ne $(($L-1)) || ! -n ${2} ]] ; then
            echo $USAGE
            exit 1
          fi
          outfile=${2}
          shift ;;
     *) echo $USAGE
        exit 1 ;;
  esac
  N=$(($N+1))
done
shift
done
if [[ ! -n ${1} ]] ; then
echo $USAGE
exit 1
fi
infile=$1
echo -n "snmpver:$snmpver debug:$debug sync:$sync "
echo "outfile:$outfile infile:$infile"

I'd normally put more white space in a script, but I want to keep it tight here. Some example output with short comments:

$ optionize.bsh
usage: optionize.bsh [-13ds] [-o ] filename
$ #that last argument _is_ required
$
$ optionize.bsh blah
snmpver:2c debug:0 sync:0 outfile: infile:blah
$ #defaults
$
$ optionize.bsh -s1 -o yada blah
snmpver:1 debug:0 sync:1 outfile:yada infile:blah
$ #blobs are fine
$
$ optionize.bsh -sd1 -o yada -3 blah
snmpver:3 debug:1 sync:1 outfile:yada infile:blah
$ #last option on the command line overrides any previous
$
$ optionize.bsh -s -d -d1o yada -3o Yadaya blah
snmpver:3 debug:1 sync:1 outfile:Yadaya infile:blah
$ #options requiring arg.s can be in a blob, but ...
$
$ optionize.bsh -d -so1 yada blah
usage: optionize.bsh [-13ds] [-o ] filename
$ # ... not in the middle of a blob, obviously.

How this works is really pretty simple if you're familiar with bash's positional parameters and shell parameter expansions. For a script called from the command line like this, the positional parameters will be each of the arguments numbered from left to right, from 1 to the number of arguments. In my first while loop, each time I finished processing an option I called the shell command 'shift', which pops off that first positional parameter and shifts the remaining ones down one position each. In the case of the '-o' option I first checked that it wasn't stuck in the middle of an options blob, then I took the following positional parameter ("$2") as the value for the option "outfile" and performed an extra shift. BTW, I was thinking here that if "outfile" was not specified that we'd just send the output to standard out, but you could just as easily have specified a default out file.

The other confusing part might be the shell parameter expansions. Briefly, "${1:0:1}" is using bash substring expansion and the syntax is "${parameter:offset:length}". So, it's short for the first character of positional parameter 1 (yes, it is zero-indexed). Similarly, "${1:$N:1}" is taking one character at a time from an option blob. And, "${#1}" is short for the number of characters in positional parameter 1.

(Can you see which variable in the script wasn't necessary? :) )

If you want to learn more about these parameter expansions, or bash in general, look here: Shell Parameter Expansion.

More useful shell aliases

This article will expand on my previous post, Useful shell aliases. All of the below was done with the bash shell on a FreeBSD system, but ought to be largely adaptable to other shells and systems.

One of the aliases I defined in the previous article was this one:

alias psa='ps auwx'

which shows the entire process table with my preferred options. But, often you're just looking for one or a few specific processes and it's a bother to pick through the full process listing. Therefore, I developed this alternative alias:

alias psg='ps auxw | grep '

Now, say if I want to just check if apache is running or how many apache processes are up, I just invoke this alias along with the regular expression that I wish to match, in this case "httpd", like this (with some shrunken, example output):

$ psg httpd
root 3870 0.0 0.3 6740 5564 ?? Ss 22Aug06 3:13.62 /usr/local/sbin/httpd
nobody 3871 0.0 0.3 6808 5640 ?? I 22Aug06 0:18.72 /usr/local/sbin/httpd
nobody 3872 0.0 0.3 6844 5652 ?? I 22Aug06 0:11.94 /usr/local/sbin/httpd
nobody 3873 0.0 0.3 6844 5652 ?? I 22Aug06 0:03.08 /usr/local/sbin/httpd
kace 6523 0.0 0.0 1448 856 p0 S+ 10:59PM 0:00.00 grep httpd

That alias is so simple, yet it is a great time saver that I use almost daily.

Many times, I will write and run long "for" loops at the command line or some other command that is long or complicated. At the time, maybe I didn't expect to be using it again. Or, maybe I've forgotten the IP address of a machine I logged into recently. This next alias can help with all of that. I call it "bash history grep".

alias bhg='cat $HISTFILE | grep '

You just have to think of a good, specific match pattern to find the old command that you are thinking of. Like a go0gle search, you may have to refine your search and try again. :) And, if you've forgotten an IP address you might try "bhg ssh". If that turns out to give too much output, then try "bhg ssh | sort | uniq " to eliminate duplicates or maybe "bhg 'ssh 1' " if you remember the first digit of the address is '1'. I could hardly make it through the day without this alias. If you're like me and you work from the command line alot and find "bhg" to be useful, then you'll almost certainly want to change the HISTFILESIZE environment variable in your .bash_profile and make it much larger.

This next one I call "sc" for sans comments. (I orginally called it "nc" but that conflicted with the wonderful netcat utility, which now appears in the base system, BTW.) This is another one of my favorites. Please note that between the square brackets below is exactly one space and one tab! (... You can't cut and paste this one, I'm afraid.)

alias sc="grep -v '^[ ]*#\|^[ ]*$' "

What this one does is to use grep to ignore (-v) any lines that are empty, only whitespace, only comments, or some combination of those things. It is fantastic for reading various configuration files and various scripts as well. Many of these files have a lot of comments, which is fine, but sometimes you know what you're looking for and you don't have time to sift through pages of comments (cough! httpd.conf). Yet, you want to leave those comments where they are for other times when you need them.

Here is just one more, a kind of fun one. I don't get far in the morning without some hot tea. But, what happens is I bring the tea back to my desk and start on something and forget to watch the clock and remove the teabag from the hot water. This risks over-steeping and bitterness. Hence, this little alarm alias:

alias beep="echo -ne '\a' ; sleep 0.1 ; echo -ne '\a' ; sleep 0.1 ;echo -ne '\a' "

When I pour my tea I check my watch and then I check my watch again as soon as I get back to my desk. Then I run a sleep with the remaining steeping time followed by the beep (say "sleep 180 ; beep "). The beep is, as you can see, really three quick beeps, which helps distinguish it from the various actual system beeps that may be going on at any time in your busy day. Enjoy your tea.

[ tags: shell alias, bash, aliases, freebsd, shell, sysadmin, system administration, unix, unix shell ]

Useful shell aliases

In a shell, an alias is a user defined command that will execute some other predefined, and usually longer, command. It's a tremendously useful command line feature that allows you take the longer commands that you use often and make them very short. I also use aliases to, in effect, redefine basic commands to always use the options that I prefer. For example, two commands I often use in FreeBSD to get a quick look at the network status of a box are "netstat -rn" to see the routing tables, and "netstat -an" to see the network connections. I might type these commands many times when there is a 'situation'. So, I've defined these aliases to give me some much shorter, time saving alternatives:

alias nsr='netstat -rn '
alias nsa='netstat -an | sed -n "1,/Active UNIX domain sockets/ p"'

These two lines are in the file "~/.bash_profile" and are executed for me whenever I login. The sed command that I'm piping the "netstat -an" command through just cuts off the domain sockets listing which, on a busy machine, is usually long enough to cause the network connections to be beyond a page/screen long. Then again, on a machine with a really large number of network connections, the output from "nsa" may still be too long for your terminal. No problem -- "alias" to the rescue again:

alias nsa='netstat -an | sed -n "1,/Active UNIX domain sockets/ p" | more'

I put a lot of importance on the modification time of a file. So, the basic ls command does nothing for me. I always want to see the modification times and I always(*) want to see the files in modification time order. Furthermore, if I copy a file, I want to preserve the modification time -- the content still hasn't changed yet. Since I pretty much always want these options, I will use the alias command to redefine these basic commands to suit my own tastes:

alias ls='ls -lt'
alias cp='cp -p'

Here are few more basic aliases that I've found to be very useful with short explanations for each:

#"change back" to the last directory you were in.
# Automatic shell variable, OLDPWD, is from bash.
alias cb='cd $OLDPWD'

#This one is good one to help standardize your command line across different OS's.
# I think it was "alias psa='ps -ef'" on Sun
#"process status all"
alias psa='ps auwx'

#"list directories", lists all of the subdirectories (names not contents) in the current directory
alias lsd='ls -d */'

#I quit after a few pings, because forever is a long time. :)
# (Also, notice 5 pings is more than a certain other OS's default 4. :D )
alias ping='ping -nc 5 '

# * -- did I say "always"? Once in a while I want to find a specific
# file name in very full directory and alphabetical listing is better.
#"list alphabetcial" (aliases will expand inside other aliases, hence the absolute path)
alias la='/bin/ls -l'

I've also written a second article on this subject, "More useful shell aliases." There we'll find that shell aliases plus grep equals a happy sysadmin -- and a decent cup of tea, to boot!

[ tags: shell alias, bash, aliases, freebsd, shell, sysadmin, system administration, unix, unix shell ]