Gotcha of the Day: ssh command causes shell script to mysteriously exit

I recently added another step to a lengthy server init script I maintain for one of my customers. The command I added ssh's out to another server, tar's up a particular directory to stdout and then unpacks this tar stream locally. It's a trick I use all the time. The crazy part: adding this ssh command caused my shell script to exit after the tar command completed.

My first thought was that I'd introduced some sort of error, but no, running the script under sh -x revealed no unusual behavior.

The init script I'm working with is intended to be run on a remote server. To streamline things, I usually run it like so:

curl http://master.client.com/tools/init.php?server=foo | sudo sh

Where init.php generates a customized shell script on the fly.

The first hint of a solution came when I realized that I could run the command like so and have it succeed:

curl http://master.client.com/tools/init.php?server=foo > /tmp/init.sh ;  sudo sh /tmp/init.sh

And while it was nice to have a work-around, I was far from satisfied. What black magic was the ssh command doing that it was causing the script to fail?

I eventually narrowed down the entire problem to this trivial script:

#!/bin/sh

echo "Before"
ssh -p 222 -o "StrictHostKeyChecking no"  -i /home/me/id.pem source.client.com uname
echo "After"

Here's a session showing the problem:

$ cat sample.sh | sudo sh 
Before
Linux
$ sudo sh sample.sh 
Before
Linux
After

This confirmed to me that the problem had to do with ssh and not tar or other commands in the script.

I finally figured out the issue thanks to this stackoverflow post: SSH causes while loop to stop. In that post, it talked about how "SSH might be reading from standard input, eating up your actionlist." Aha! I'm feeding my init script the shell via standard input. If ssh consumed that stream then the rest of the commands wouldn't be there to execute. The script would end, not because it crashed, because there was nothing left to do.

I changed my sample script as follows:

#!/bin/sh

echo "Before"
ssh -p 222 -o "StrictHostKeyChecking no"  -i /home/me/id.pem source.client.com uname < /dev/null
echo "After"

And now the output is correct:

$ cat sample.sh | sudo sh 
Before
Linux
After

So there you go: ssh consume stdin, use with care. I can't believe I've been using Unix for over 20 years and never ran into this behavior.