Connecting the dots with scp

Today, one of my colleagues was having trouble copying some files between Linux boxes. As a result, I learnt something about scp that I didn’t know before — by default, using scp between two remote hosts from a third party host means that SSH connections “leapfrog” through the box where the source file is located.

Let me set the scene — imagine three hosts:

A host calledbastionthat scp copies are initiated from.
A host called sourcebox that has a file you want to copy (/tmp/source_file).
A host called destbox that has a place you want to copy the file to (/tmp/destfile).

Imagine that our private key (~/.ssh/id_rsa) is physically present on bastion, but not on either sourcebox or destbox.

Our corresponding public key (~/.ssh/id_rsa.pub) is configured in the~/.ssh/authorized_keys file of all three hosts. We can therefore authenticate to each of these three boxes via ssh by using our private key.

Our /tmp/source_file exists on sourcebox and is accessible from bastion.

drew@bastion:~$ ssh -i ~/.ssh/id_rsa sourcebox 'cat /tmp/source_file'
content of my file

Likewise, we can tell that there’s no file called /tmp/dest_file on destbox.

drew@bastion:~$ ssh -i ~/.ssh/id_rsa destbox 'ls -ld /tmp/dest_file'
ls: cannot access /tmp/dest_file: No such file or directory

Okay, let’s try and copy our file by using scp from bastion:

drew@bastion:~$ scp -i ~/.ssh/id_rsa sourcebox:/tmp/source_file sourcebox:/tmp/dest_file
Host key verification failed.
lost connection

Huh? But… ssh worked before between bastion and both of our other hosts... What gives? Both of these hosts are located in the ~/.ssh/known_hosts file of bastion, too…

drew@bastion:~$ ssh-keygen -F sourcebox | grep found
# Host sourcebox found: line 3 type ECDSA

drew@bastion:~$ ssh-keygen -F destbox | grep found
# Host destbox found: line 4 type ECDSA

Let’s dig a bit deeper with a bit of ssh's -vvv option… (I’ve omitted a lot of the noise with ellipsis to make it easier to read)

drew@bastion:~$ scp -vvv -i ~/.ssh/id_rsa sourcebox:/tmp/source_file destbox:/tmp/dest_file
...
debug1: Host 'sourcebox' is known and matches the ECDSA host key.
...
debug1: Authentication succeeded (publickey).
Authenticated to sourcebox ([192.168.10.12]:22).
...
debug1: Connecting to destbox [192.168.10.13] port 22.
...
debug1: Sending command: scp -v /tmp/source_file destbox:/tmp/dest_file
...
Host key verification failed.
lost connection

So, it seems like when we invoke scp from bastion, we first connect to sourcebox and then initiate a connection from there to destbox.

As it stands, sourcebox doesn’t know about destbox, so we can fix that by adding destbox's host keys…

drew@sourcebox:~$ ssh-keyscan destbox >> ~/.ssh/known_hosts
# destbox SSH-2.0-OpenSSH_6.6p1 Ubuntu-2ubuntu1
# destbox SSH-2.0-OpenSSH_6.6p1 Ubuntu-2ubuntu1

drew@sourcebox:~$ ssh-keygen -F destbox | grep found
# Host destbox found: line 1 type RSA
# Host destbox found: line 2 type ECDSA

Let’s try again and see where we stand…

drew@bastion:~$ scp -i ~/.ssh/id_rsa sourcebox:/tmp/source_file destbox:/tmp/dest_file
Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,password).
lost connection

Our host key problem is now resolved. Our sourcebox host now successfully validates the host key of destbox. But we have another problem…

Now we’re getting authentication failure messages. Basically, the private key that we’re using from bastion isn’t present on sourcebox, so we’re effectively trying to authenticate to destbox from sourceboxwithout a password, an attempt which is swiftly rebuffed by destbox.

You can fix this from bastion by using a feature of ssh called AgentForwarding. First, you load your private key into ssh-agentand tell scpto forward it by using -o ForwardAgent=yes...

drew@bastion:~$ eval $(ssh-agent)
Agent pid 3117

drew@bastion:~$ ssh-add
Identity added: /home/drew/.ssh/id_rsa (/home/drew/.ssh/id_rsa)

drew@bastion:~$ scp -o ForwardAgent=true sourcebox:/tmp/source_file destbox:/tmp/dest_file

drew@bastion:~$ ssh destbox 'cat /tmp/dest_file'
content of my file

Another way around this without using ssh-agent or having to configure anything on sourcebox would be to use the -3 option that comes with scp. From scp(1):

-3 Copies between two remote hosts are transferred through the local host. Without this option the data is copied directly between the two remote hosts. Note that this option disables the progress meter.

This would change the connections that are made by scp from this:

bastion -> sourcebox    # initiate connection to sourcebox
sourcebox -> destbox    # copy from sourcebox to destbox

to this:

bastion -> sourcebox    # copy from sourcebox to bastion
bastion -> destbox      # copy from bastion to destbox

Both of these are a bit more convenient than manually setting up SSH tunnels or anything like that. And both options help to reduce the number of places you have to plonk your private key, which is always good.

I’ve been using these tools for a long time now, but for some reason in my head I always assumed that connections were made from the place that the command was invoked. TIL, I guess.