Input, Output and piping

In this Article let me explain you a little how the output and input of a program works, and how to take advantage of that to mix multiple programs.

One ear two mouths (Standard data streams)

When you run a program, by default it will have 1 stream for getting data (standard input) and 2 to print data (standard output and standard error).

Suppose you have python a program which reads text and prints it in upper case form:

print(input().upper())

And your boss come with this file asking you if there is a way to convert its words in upper case form

Consectetur obcaecati harum delectus perferendis numquam
Atque sit facere incidunt sequi totam Harum omnis dolore
a excepturi laudantium Cum dolore autem est maxime expedita,
repudiandae sint. Officiis provident repellat veritatis
vero ipsum Ratione temporibus accusamus quis harum odio
natus exercitationem

Standard Input

Well the answer to this problem is pretty easy, you just have tell your python program to redirect its input to your boss file as follows:

$ python upper_script.py < words_file

Standard Output

Now you need to grab the output of the program into a file to send it to your boss. To do that use the > which redirects the output of your program to the file you need.

$ python upper_script.py < words_file > upper_words_file

Standard Error

To explain this one I will use curl, a CLI program which can make http requests that are printed on the terminal. Suppose we want to download some text from a website to a file using output redirection.

$ curl https://example-files.online-convert.com/document/txt/example.txt > some_web_text.txt
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2574    0  2574    0     0   4093      0 --:--:-- --:--:-- --:--:--  4098

Wow, what happened here?, Why curl is printing text? It is not supposed that its output should be in the file some_web_text.txt?. Well if you look the file, it is very likely you will find the text you wanted to download, the printed text on the terminal corresponds to the other output every program has, which is called standard error.

Standard error is commonly used to show debugging information and it works pretty much the same as standard output, you can even redirect this output to another file (e.g download_info.txt) making use of 2> symbol, here is the same example saving the download info into a file.

$ curl https://example-files.online-convert.com/document/txt/example.txt > some_web_text.txt 2> download_info.txt

Mixing programs through Pipes

Knowing that, our boss ask us if there is a way to mix the curl program with your upper script program to save this file from www.w3.org website in upper case words.

Here the pipes come to rescue, it allows us to mix multiple programs connecting the output of a program to the input of another, so we need to connect the output of curl to the input of upper_script.py using the | symbol as follows:

$ curl https://www.w3.org/TR/2003/REC-PNG-20031110/iso_8859-1.txt | python upper_script.py

| is the pipe which connects curl standard output with the standard input of upper_script.py.

Finally to save the upper words output you only have to redirect the standard output of the last program:

$ curl https://www.w3.org/TR/2003/REC-PNG-20031110/iso_8859-1.txt | python upper_script.py > ISO_8859_UPPER.txt