If you wait on one pipe, the other may overflow and block, preventing your wait on the other one from ever finishing. The only easy way to get around this is to create a thread to service each pipe.
The source to the module, especially communicate and its helpers, shows how to do it. I linked to 3. In your case, you can't use communicate , but fortunately you don't need more than one pipe. Stack Overflow for Teams — Collaborate and share knowledge with a private group. Create a free Team What is Teams?
Collectives on Stack Overflow. Learn more. Python: subprocess. Asked 8 years, 4 months ago. Active 4 years, 2 months ago. Viewed 59k times. I have a command line tool actually, several that I am writing a wrapper for in Python. I want to replicate this behavior, while also logging stderr the status messages to a file.
To recap, desired behavior is: use call , or subprocess direct stdout to a file direct stderr to a file, while also writing stderr to the screen in real time as if the tool had been called directly from the command line. EDIT: this only needs to work on Linux. Improve this question. Ben S. If not, there's an easier answer. Add a comment. Active Oldest Votes. PIPE for line in proc. But if you're on Unix, it might be simpler to use the tee command to do it for you. Something like this: subprocess.
What if you need to gather both stderr and stdout? Thread proc. Improve this answer. Great idea. I was having the same problem and this helped me solve it. Your method for doing cleanup though is wrong as you mentioned it might be. Basically, you need to close the write end of the pipes after passing them to the subprocess. That way when the child process exits and closes it's end of the pipes, the logging thread will get a SIGPIPE and return a zero length message as you expected.
Otherwise, the main process will keep the write end of the pipe open forever, causing readline to block indefinitely, which will cause your thread to live forever as well as the pipe. This becomes a major problem after a while because you'll reach the limit on the number of open file descriptors. Also, the thread shouldn't be a daemon thread because that creates the risk of losing log data during process shutdown. If you properly cleanup as a described, all the threads will exit properly removing the need to mark them as daemons.
I used different names in a couple of spots, but otherwise it's the same idea, except a little cleaner and more robust. We need the file descriptor to be closed in the parent process i. The two streams still end up not being synchronized correctly. I'm pretty sure the reason is that we're using two separate threads. I think if we only used one thread underneath for the logging, the problem would be solved.
The problem is that we're dealing with two different buffers pipes. Having two threads now I remember gives an approximate synchronization by writing the data as it becomes available. It's still a race condition, but there are two "servers", so it's normally not a big deal. With only one thread, there's only one "server" so the race condition shows up pretty bad in the form of unsynchronized output.
The only way I can think to solve the problem is to extend os. Sign up to join this community. The best answers are voted up and rise to the top. Stack Overflow for Teams — Collaborate and share knowledge with a private group. Create a free Team What is Teams? Since python 3. I found this approach to redirecting stderr particularly helpful. Essentially, it is necessary to understand if your output is stdout or stderr.
The difference? Stdout is any output posted by a shell command think an 'ls' list while sterr is any error output. It may be that you want to take a shell commands output and redirect it to a log file only if it is normal output. Using ls as an example here, with an all files flag:. If you wanted to make this an error log, you could do the same with stderr.
It's exactly the same code as stdout with stderr in its place. This pipes an error messages that get sent to the console to the log. Doing so actually keeps it from flooding your terminal window as well! Python will not execute your code if there is an error. But you can import your script in another script an catch exceptions. You can write several print statements in your script and you can stdout to a file, it will stop writing to the file when the error occurs.
To debug the code you could check the last logged output and check your script after that point. You might want something similar to raw string. You can check this thread for this. Stack Overflow for Teams — Collaborate and share knowledge with a private group. Create a free Team What is Teams?
Collectives on Stack Overflow. Learn more. How to redirect stderr in Python? Ask Question. Asked 12 years ago. Active 2 months ago. Viewed 70k times. I would like to log all the output of a Python script. How do I capture also the errors from Python interpreter? Improve this question. EcirH EcirH 1 1 gold badge 5 5 silver badges 9 9 bronze badges. Perhaps writer should implement writelines as well. Do not forget sys. However, how do you check whether your print is being logged?
You store stdout and stderr in a list. All your print results should go to the list.
0コメント