LINUX CLASSES - LINUX BASICS

Will Linux Pipelines Make Me Filthy Rich?

Pumping a Program's Output into Another Program Linux provides you with a wide array of utilities to manipulate data. You can search, sort, slice, dice, and transform data stored in files in many different ways. A pipe (also called a pipeline) is a powerful shell feature that allows you to p ump the output of one program directly into another.

For example, say you have a file with information about a group of people, including a name, age, zip code, and phone number for each person, that looks like this:

Roosevelt Tommy 38 54579 555-1212

Nixon Edward 19 37583 246-3457

Roosevelt Freddie 47 11745 674-6972

Lincoln Albert 26 26452 916-5763

If you wanted to find all the Roosevelts and sort them by zip code, you could do it like this:

grep Roosevelt people.txt > grep.out
sort +3 grep.out
rm grep.out

Since I haven't introduced the grep and sort commands yet, here's an English translation of what's happening above:

Look for lines that contain Roosevelt in the people.txt file and put them in a file named grep.out. Then sort the grep.out file on the fourth column and display the results on the console befor e deleting the grep.out file. (Yes, it is odd that the +3 flag tells sort to use the fourth column!)

But you could avoid creating and deleting the intermediate file (grep.out) by combining the operation into one command like this:

grep Roosevelt people.txt | sort +3

The vertical bar tells the shell that the output of the program on the left (grep) should become the input for the program on the right (sort). Behind the scenes, the shell may be issuing the exact same three commands as in the previous example, but you don't really care--you've combined three commands into one.

You can have any number of steps in a pipeline, and you can even combine pipes with redirection, as shown here:

grep Roosevelt people.txt | sort +3 > sort-results

Here, the same thing happens, except that the end result is stored in a file called sort-results.

Previous Lesson: Redirection
Next Lesson: Processes

[ RETURN TO INDEX ]

Comments - most recent first
(Please feel free to answer questions posted by others!)

Don Kalyan (18 Nov 2012, 12:49)

Thanks a Ton...Bob..Nice little explanation..

Oniex (02 Mar 2012, 06:44)

thanks a lot...... avoided weight gain just by absorbing low fat linux tuts..........gr8 work.....hope you can come up with many more linux/unix based applications and services....

Mike (19 Feb 2012, 21:45)

Just wanted to say thanks for putting in what must have been a HUGE amount of time and work to make this. I am just starting with Linux/Unix and you have made it nearly painless!

cyril (17 Feb 2012, 01:48)

try grep Roosevelt people.txt | sort -k 3

Felipe (13 Feb 2012, 05:03)

This command is not working
open failed: +3 No such file or directory

Thanks for the Tutorial, is great

ram (06 Jul 2011, 07:38)

this command is not working and giving error open failed. kindly post the reason for this pls!

oasg (07 Apr 2011, 23:10)

Yes! Now I am understanding.
Thank you very much Dr. Bob!

Ulrich (31 Mar 2011, 13:32)

Hi,
I've got the same error:

open failed: +3: No such file or directory

Jelloso (19 Oct 2010, 11:07)

You're the best Dr. Bob!!! Everything I ever need to pass my Linux subject is right here!!!

aldiazo (10 Sep 2010, 16:53)

I am trying to do a telnet, run a shell to extrac data from an informix DB, and redirect de output to a text file...

Who can I do that?

Kevin (27 Jun 2010, 16:40)

When piped the sort command (sort +3) gave me the error message:
open failed: +3: No such file or directory
When you add a -4 (as the second position flag) the command works. Do you Bob still use this syntax or do you now use the -k flag?

umakanta (25 Jun 2010, 08:33)

thanks .. information is everything...

kapil pateria (08 Apr 2010, 04:40)

nformation is really very useful and in a very concise manner. best for a novice begginer.

Dennis (25 Feb 2010, 08:57)

thanks.
I just enter the linux world, and the information is very accessible.

Bob Rankin (25 Feb 2010, 08:30)

@Dennis, those commands are equivalent. Here's the only difference: In the first case, grep is passed the name of the input file, and grep opens the file. In the second, grep is not passed a filename, and reads from the standard input stream. Not really important from a user perspective, but that's the scoop.

Dennis (25 Feb 2010, 08:12)

grep Roosevelt people.txt | sort +3

does it overrides ?
grep Roosevelt < people.txt | sort +3

< - is missing

I welcome your comments. However... I am puzzled by many people who say "Please send me the Linux tutorial." This website *is* your Linux Tutorial! Read everything here, learn all you can, ask questions if you like. But don't ask me to send what you already have. :-)

NO SPAM! If you post garbage, it will be deleted, and you will be banned.

Will Linux Pipelines Make Me Filthy Rich?

Comments - most recent first(Please feel free to answer questions posted by others!)

Comments - most recent first
(Please feel free to answer questions posted by others!)