Redirection Wizardry

or How to make two parallel pipes

Hi all,

Goal: understand the following script and why it is usefull:

{
    print "Standard"
    print "Error" >&2
    print "Main" >&3
} > >(awk '{print "BUF1: "$0}') \
2> >(awk '{print "BUF2: "$0}') \
3> >(awk '{print "BUF3: "$0}')

Problem

I write a script with:

echo "first step"
launch a complex sub command

echo "second step"
launch a complex sub command

...

It’s output should be:

first step
hello, I'm a big long complex
command which output this
And I write many things.
Here is an error message.
second step
therefore
It is difficult to see where I end

My main logs are difficult to discern and therefore should be unnoticed . Most of time the sub command is not mine, and therefore, it is not easy to manage their appearance.

First solution: redirect command output

Prefix each output of the complex sub command

echo "first step"
complexSubcommand | awk '{print "\tcomplexSubcommand: "$0}'

echo "second step"
complexSubcommand2 | awk '{print "\tcomplexSubcommand2: "$0}'

OK, it is far better now:

first step
    complexSubcommand: hello, I'm a big long complex
    complexSubcommand: command which output this
    complexSubcommand: And I write many things.
Here is an error message.
second step
    complexSubcommand2: therefore
    complexSubcommand2: It is difficult to see where I end

Unfortunately the standard error is not redirected.

Second solution: redirect everything

One can do:

echo "first step"
complexSubcommand 2>&1 | awk '{print "\tcomplexSubcommand:"$0}'

echo "second step"
complexSubcommand2 2>&1 | awk '{print "\tcomplexSubcommand2:"$0}'

output is now:

first step
    complexSubcommand: hello, I'm a big long complex
    complexSubcommand: command which output this
    complexSubcommand: And I write many things.
    complexSubcommand: Here is an error message.
second step
    complexSubcommand2: therefore
    complexSubcommand2: It is difficult to see where I end

This is better, but error is lost in the flow of standard output. This is why it would be good to prefix the standard output by some string and standard error by another more visible string.

But it is not possible to implement that only with pipe or even redirection between standard output and error.

Complete solution: using named pipe

Then, a final solution is to use named pipe.
One named pipe for standard output, one for standard error and one for main messages.

Here is an example:

#!/usr/bin/env zsh

Creation of the fifos: /tmp/y/fifo1, fifo2, fifo3

fifo="/tmp/y/fifo"
mkdir -p $(dirname $fifo)
for n in 1 2 3; do
    if [[ ! -p "fifo$n" ]]; then
        mkfifo $fifo$n
    fi
done

# will print what is written in the fifo
# prefix all string written in fifoX by BUF X:
for n in 1 2 3; do
    awk "{print \"BUF $n:\"\$0}" < $fifo$n &
done

# here begins the program
{
    print "Standard"
    print "Error" >&2
    print "Main" > ${fifo}3
    # close the standard output and error
    2>&- >&-
# redirect all standard output to fifo1
# and all standard error to fifo2
} >${fifo}1 2>${fifo}2

# once the program ended
# delete the fifos
for n in 1 2 3; do
    \rm $fifo$n
done

Here is the result:

BUF 1:Standard
BUF 2:Erreur
BUF 3:Main

Final solution: use zsh syntax

Finally I recently found a way to do exactly the same with a much clearer and compact syntax. Thanks zsh:

{
    print "Standard"
    print "Error" >&2
    print "Main" >&3
} > >(awk '{print "BUF1: "$0}') \
2> >(awk '{print "BUF2: "$0}') \
3> >(awk '{print "BUF3: "$0}')

Here is the result:

BUF 1:Standard
BUF 2:Erreur
BUF 3:Main

Blogged with the Flock Browser

Add include directive to awk

With gawk, there exists the @include directive :
@include "fic"

But, when you work for a company which use a prehistoric version of awk, you could use my script (it is more a hack than a real program) :

#!/usr/bin/env zsh
if (($#<1)); then
    {
        echo "usage : $(basename $0) script.awk arg1 ... argN"
        echo "provide @include \"path/to/awk/file\" directive"
        echo "usefull for libraries"
    } >&2
    exit 1
fi

tmpScriptAwk="/tmp/$(basename $1)"

# see my blog entry about this function
function matche
{
    echo "$1" | egrep "$2" > /dev/null
}

# go into the same directory as the script (to find the good path)
cd $(dirname $1)

# read the script
IFS='
'
for ligne in $(cat $(basename $1) | sed 's/\\/\\\\/g'); do
    # transform @include "path/to/file" by the content of path/to/file.awk
     if matche "$ligne" "^@include \".*\"" ; then
         nomFic=$(echo $ligne | perl -p -e 's#\@include "(.*)"#$1#' )
        # verify if the file to include is readable
         if [ ! -r "$nomFic" ]; then
            # add the .awk prefix (as it is not necessary)
             nomFic="$nomFic.awk"
             if [ ! -r "$nomFic" ]; then
                 echo "$nomFic n'est pas accessible en lecture" >&2
                 exit 1
             fi
         fi
        # write the content of the included file
         cat $nomFic
     else
         echo $ligne
     fi
done > $tmpScriptAwk
shift
# run the script with the argument
awk -f $tmpScriptAwk "$@"

Hope this help,

Blogged with the Flock Browser

global search and replace on place with perl

This is a tip on how to modify many files (when sed don’t work):

perl -pi -e 's/toto/tata/g' file1 file2 ... fileN

If you want to make a test assign a suffix at all files (beware to use suffix no existing file already has):

perl -pi.orig -e 's/toto/tata/g' file1 file2 ... fileN

or to make a preview on the standard output:

perl -p -e 's/toto/tata/g' file1 file2 ... fileN

In short to make perl work like sed or awk simply use perl -pe instead of just perl. And use perl -pi -e if you want to replace files in place.

Blogged with the Flock Browser

Bash or Zsh pointer

Here is a way to implement pointer like behavior in shell script:

toto="tata"
ptr="toto"
echo $( eval echo \$$ptr )

tata

complex remote command with ssh

How to launch remote command ? Use ssh:

> ssh user@host "ls /home"

OK it works!

PROBLEM

With complex commands:

> remoteUser="toto" ;
> ssh user@host "for fic in /home/$remoteUser/???/*(/); do echo $fic | egrep -v "common$" ; done

It fails miserably! Because all special characters are interpreted within the command. And even if you protect all special characters, the remote shell should not be zsh and the command will not be interpreted correctly.

SOLUTION

remoteUser="toto"
ssh cmsrct <<END
zsh -s <<END2
        for fic in /home/$remoteUser/???/*(/); do
                echo \\\$fic | egrep -v "common\\\$"
        done
END2
END

All special character are no more interpreted except variables (such as $fic). This is why you need to put three backslashes before the $. The previous code will execute the following command on the remote host:

for fic in /home/toto/???/*(/); do
    echo $fic | egrep -v "common$"
done

Blogged with the Flock Browser

Bazaar (bzr) with existing projects

Bazaar is a distributed concurent versions system (DCVS). That means, you can focus on the development of one feature at a time. You “freeze” the main project, working in a totally isolated environment. Others should continue on the main project while you work on only one feature.
When you have finished you can pull your change into the main project.

Bazaar gives you many others benefits :

• security (backups),
• history (who as done what)
• focusing (work on one feature at a time)
• multi-user
• work on parallel versions of the same program (remove a bug on all versions)…

Unfortunately, most of time, project have an older versionning system not as elegant as Bazaar. But you can really easily handle this.

• Download sources using the versionning system used by the other (subversion for example)
• into the root of the new arborescence
  (said its name is main) do a:
  1. bzr init
  2. bzr ingnore **/.svn
  3. bzr add
  4. and then bzr commit.

  Now you have a complete new bazaar branch containing the code of theproject

• Now you want to add a new feature :
  1. bzr branch main new-feature-branch
  2. hack, hack, hack…
  3. svn update
  4. bzr branch ../new-feature-branch
  5. In case of conflicts resolve them and do not forget to put the conflict files into your
     new-feature-branch
  6. svn commit

It’s done!

How to write a counter in zsh

Here is a code to be able to write a counter in zsh. The advantage are it work alwas on the same line. The trick is to use backspace characters.

for example:

host> echo "123\b4"
124

What happened ?

1. write 1
2. write 2
3. write 3
4. backspace (delete 3 and position the cursor on the 3)
5. write 4

Now real usage:

n=1
echo
while true; do
    printf "\b\b\b\b\b%5s" $((n++))
    sleep 1
done

A faster way should be to use the ‘\r’ which erase the beginning of the line.

n=1
echo
while true; do
    printf "\r%5s" $((n++))
    sleep 1
done

My vision of twitter

The first time I heard about twitter and the buzz around it, my first reaction was why? What is the interest of saying what I’m doing? To know what others are doing? Furthermore I live in France and with my contract I cannot send free SMS to the English twitter’s server!

what decided me?

I wanted to know why so much people like twitter. Then I created an account and verified twitter can send me SMS for free. This way I can send SMS to my wife for free. Yeah a free service I didn’t have before!

notification system

I wanted to be able to have some reminder on my cell phone. Of course it already do it, but I’d hate to write it on the cell phone. I wanted to write it with my keyboard on my computer.

twitter can send me SMS for free, then I’d want an organisation mecanism twitter compliant. My choice goes to iwantsandy. I’ve configured twitter to work with my cell phone and also my IM (google talk).

Now I can order to Sandy via my IM desktop client.

Movement :

1. click on my IM client
2. click on the twitter@twitter.com user
3. write something like : “d s r do something at 6:30pm” where “d s r” stand for :

• d : direct message to
• s : sandy
• r : remind me to

Ok, now I’m happy

Third step (try out)

In two first step, I wanted to used twitter just for “me”. After that, I tried to look at some interesting people to follow. And I founded many of my favorite blogger.
Then I can be up to date exactly as I would be using RSS.

From now I can use twitter as in IM sending direct message to someone, and it can receive it directly to it’s cell phone.
I can use twitter like a forum sending it public replies.
I can use twitter to follow interresting people like RSS.
I can follow people given me interresting links like into a bookmark finder à la SumbleUpon.

What I don’t do for now is stay in touch with friends. As I’ve don’t have many friends working a lot on a computer. My french friends will not send SMS to update their twitter status because it will cost money. Some should update via the web. But most won’t want to lose such a time.

Conclusion (my perception of twitter)

I see twitter like a really adaptable “framework” to communicate with other. You can be a receptor only, an “emitor” only, could use it as an IM, as a forum, as a RSS, as a better filter to search the web.

That was the good sides of twitter.

Bad side : the interface of twitter does not take into account all these usages. Particularly, if you want to follow a blogger only to be alerted when his blog as a new entry, you don’t necessarily want to follow all his replies.

Then, there is some lack of visual features into the twitter’s web interface.

This is why I should think about a new interface. Particularly a desktop one (for mac).

Don’t get burned! Easy password managment

We all have to use many passwords. Unfortunately, most of time, we use only a few (if not one) password everywhere. Most of time it is really dangerous. Think of it:

I’m using an email account which belongs to my employer. If the password I use for my job is the same as my personnal gmail password. If he wants, he can try the same password to view all my personal mail or track my monster account…

This is a good reason enough to manage many different passwords. But how to do it? In general, best answers are:

1. Use a program containing all password
2. Write it on paper (use steganography with phone numbers for example)
3. Use a checksum of the concatenation of your password and the domain name of the site you want to visit.

The first method is the best when you work only on one computer, on one place. Unfortunately, if you use many computer, it is a pain to synchronize between each of them.The second method is the basic method, not as bad as it should appear. But it is also not a really secure way to do that ; think about physical destruction by error or by accident.The third method is the one I adopted. On most computer it is easy to have a sha1 or md5 checksum at hand. You just have to do it that way :

echo -n "myUniqueStrongPassword_DomainName" | openssl sha1

or use services such as :

• sha1 calculator
• many checksum calculator

This process gives you a string to use as password, from which it is almost impossible to recover your password. Now you only have to remember one strong password and to calculate the checksum each time you need the password.

Many people use this method, but in my humble opinion, this is not enough shared.

bash (or zsh) regular expression matching

I wanted to be able to write conditionnal on string matching in bash. One easy way to do that is :

if echo "$fic" | egrep "regexp" > /dev/null; then
    echo "it matches !"
else
    echo "it doesn't matches"
fi

But I found it a litle too long. This why I have written the small script match :

echo $1 | egrep $2 > /dev/null

in order to be able to write the more readable (in my point of vue) command :

if match "$fic" "regexp"; then
    echo "it matches !"
else
    echo "it doesn't matches"
fi