Redirection Wizardry

or How to make two parallel pipes

Hi all,

Goal: understand the following script and why it is usefull:

{
    print "Standard"
    print "Error" >&2
    print "Main" >&3
} > >(awk '{print "BUF1: "$0}') \
2> >(awk '{print "BUF2: "$0}') \
3> >(awk '{print "BUF3: "$0}')

Problem

I write a script with:

echo "first step"
launch a complex sub command

echo "second step"
launch a complex sub command

...

It’s output should be:

first step
hello, I'm a big long complex
command which output this
And I write many things.
Here is an error message.
second step
therefore
It is difficult to see where I end

My main logs are difficult to discern and therefore should be unnoticed . Most of time the sub command is not mine, and therefore, it is not easy to manage their appearance.

First solution: redirect command output

Prefix each output of the complex sub command

echo "first step"
complexSubcommand | awk '{print "\tcomplexSubcommand: "$0}'

echo "second step"
complexSubcommand2 | awk '{print "\tcomplexSubcommand2: "$0}'

OK, it is far better now:

first step
    complexSubcommand: hello, I'm a big long complex
    complexSubcommand: command which output this
    complexSubcommand: And I write many things.
Here is an error message.
second step
    complexSubcommand2: therefore
    complexSubcommand2: It is difficult to see where I end

Unfortunately the standard error is not redirected.

Second solution: redirect everything

One can do:

echo "first step"
complexSubcommand 2>&1 | awk '{print "\tcomplexSubcommand:"$0}'

echo "second step"
complexSubcommand2 2>&1 | awk '{print "\tcomplexSubcommand2:"$0}'

output is now:

first step
    complexSubcommand: hello, I'm a big long complex
    complexSubcommand: command which output this
    complexSubcommand: And I write many things.
    complexSubcommand: Here is an error message.
second step
    complexSubcommand2: therefore
    complexSubcommand2: It is difficult to see where I end

This is better, but error is lost in the flow of standard output. This is why it would be good to prefix the standard output by some string and standard error by another more visible string.

But it is not possible to implement that only with pipe or even redirection between standard output and error.

Complete solution: using named pipe

Then, a final solution is to use named pipe.
One named pipe for standard output, one for standard error and one for main messages.

Here is an example:

#!/usr/bin/env zsh

Creation of the fifos: /tmp/y/fifo1, fifo2, fifo3

fifo="/tmp/y/fifo"
mkdir -p $(dirname $fifo)
for n in 1 2 3; do
    if [[ ! -p "fifo$n" ]]; then
        mkfifo $fifo$n
    fi
done

# will print what is written in the fifo
# prefix all string written in fifoX by BUF X:
for n in 1 2 3; do
    awk "{print \"BUF $n:\"\$0}" < $fifo$n &
done

# here begins the program
{
    print "Standard"
    print "Error" >&2
    print "Main" > ${fifo}3
    # close the standard output and error
    2>&- >&-
# redirect all standard output to fifo1
# and all standard error to fifo2
} >${fifo}1 2>${fifo}2

# once the program ended
# delete the fifos
for n in 1 2 3; do
    \rm $fifo$n
done

Here is the result:

BUF 1:Standard
BUF 2:Erreur
BUF 3:Main

Final solution: use zsh syntax

Finally I recently found a way to do exactly the same with a much clearer and compact syntax. Thanks zsh:

{
    print "Standard"
    print "Error" >&2
    print "Main" >&3
} > >(awk '{print "BUF1: "$0}') \
2> >(awk '{print "BUF2: "$0}') \
3> >(awk '{print "BUF3: "$0}')

Here is the result:

BUF 1:Standard
BUF 2:Erreur
BUF 3:Main

Blogged with the Flock Browser