Originally published by Robert Beisert at fortcollinsprogram.robert-beisert.com

Linux + C – Pipes and Filters

We’ve seen how to pass arguments to programs, and we’ve seen how to write to files. Now we’ll look at how that is done from the Terminal.

Pipes

A pipe redirects the standard output (stdout) from one program to the standard input (stdin) of another. It acts like a tunnel (or pipe) between two programs, which allows us to create automatic chains of operation.

For a classic example, we have the program ls, which can print all the files in the directories and subdirectories you choose. If we type

ls -R

we get all the files in one blast – this often makes it impossible to scroll through.

However, we also have the program less, which allows us to view content one page at a time. To put the two together, we create a pipe between ls (which produces all the content) and less (which lets us view it more easily). That looks something like this:

ls -R | less

The symbol for a pipe in the UNIX operating systems (including all forms of Linux) is | (usually found above your ENTER or RETURN key).

Filters

A filter is pretty much any program which reads from standard input and writes to standard output. More specifically, a filter is a program which restructures or modifies data on its way to a further program.

One excellent example of a filter is the program cat, which reads a file to standard output. This can be very useful in certain pipe situations.

Redirects

Sometimes we don’t want to deal with all the output our programs produce at one time. Servers, for example, run continuously, and we need to look at hours or days of past outputs to spot problems.

These are the times when we employ redirects – operations that transfer data from files to standard input or from outputs to files.

There are three standard interfaces common to every C program:

  • stdin – the standard input stream
  • stdout – the standard output stream
  • stderr – the standard error stream

We can use filters to read a file into standard input. For example, if we want to read the file pandas.txt into the program print_pandas, we could write this:

pandas.txt > print_pandas

We can also use filters to write from standard output to a file. There are two ways to do this. The first way erases the contents of the file before filling it with the new content. That operation uses a single > character, which looks like this:

print_pandas > pandas.txt

The second method uses two >> characters, which tells the computer to append the contents to the original file. That operation looks like this:

print_pandas >> pandas.txt

Finally, we can redirect standard error to a file. It’s just like redirecting standard output, only we start the operation with the number 2 (which stands for the file descriptor 2, which is how the computer sees stderr).

print_pandas 2> pandas.txt

print_pandas 2>> pandas.txt

Example

For this example, we will use the following programs:

  • echo – a program which copies any input to standard output (a basic filter)
  • ./input – our input program we worked on earlier

Look at the image below for the exact details. Basically, we echo the EXACT inputs we would provide to our input program (age \n last name \n first name), pipe that to input, then store the results in the file proof.txt.

Note: We are using the append redirection to write to proof.txt. This means that every time you run this operation, you make the file bigger.

2015-09-06-101356_1920x1080_scrot

I wrote a script to automate this operation, but I think it’s best if you type it out and see how it works for yourself. Don’t worry – the script will show up again in the near future.

The contents of proof.txt should look something like this when you’re done:

Input your current age
Input your last name
Input your first name
Robert  Beisert
       5

No, I’m not 5. I just don’t feel a burning need to put my real age down.