This is something, that I always have to check. So today I'm writing it down.
Documentation to subprocess.Popen is here.
So pipelines are useful when you want to do something with output of command performed by Popen. What you would like to do with output:
- pass it to another bash command
- use it inside Python script
So you want to pass output of first bash command to another. This will be equivalent of this code in bash:
$ ls /etc | grep ntp ntp-restrict.conf ntp.conf ntp_opendirectory.conf
In Python you do that like this:
ls = subprocess.Popen('ls /etc'.split(), stdout=subprocess.PIPE) grep = subprocess.Popen('grep ntp'.split(), stdin=ls.stdout, stdout=subprocess.PIPE) output = grep.communicate()
Call ls.stdout.close() before grep.communicate() so that if grep dies prematurely, ls would exit sooner. And add ls.wait() at the end, to avoid creating a zombie:
ls = subprocess.Popen('ls /etc'.split(), stdout=subprocess.PIPE) grep = subprocess.Popen('grep ntp'.split(), stdin=ls.stdout, stdout=subprocess.PIPE) ls.stdout.close() output = grep.communicate() ls.wait()
grep.stdin.close() is called by grep.communicate().
Another way to write it:
grep = Popen('grep ntp'.split(), stdin=PIPE, stdout=PIPE) ls = Popen('ls /etc'.split(), stdout=grep.stdin) output = grep.communicate() ls.wait()
As you can see, declaration order of commands in pipe doesn't matter.
Instead of grep.communicate() function you can use grep.stdout.read(). But first you have to wait for end of subprocess call:
>>> grep.wait() >>> print grep.stdout.read()
Although this way have one disadvantage. It moves file pointer do the end of file. So every subsequent call will return empty string:
>>> grep.wait() >>> print grep.stdout.read() ntp-restrict.conf ntp.conf ntp_opendirectory.conf >>> p2.stdout.read() ''
Further more this has one more disadvantage. If you first wait() and then read() a program that produces a lot of output (more than 4 kilobytes), you'll get a deadlock.
Special thanks to Marius Gedminas, J.F. Sebastian for help with this article.
So what do you think? Did I miss something? Is any part unclear? Leave your comments below.