Generating public keys from a SSH private key

If you just need to generate a Public Key from SSH Private Key, you will need to

# ssh-keygen -y -f ~/.ssh/id_rsa > ~/.ssh/id_rsa.pub

where  -y output the public key

References:

1. How do I retrieve the public key from a SSH private key?

Algorithm negotiation failed for SSH Secure Shell Client

If you are using the dated SSH Secure Shell Client 3.2.9, you may have issue connect to the more updated OpenSSH Server.

If you cannot change the client (which is recommended), you will have to update the OpenSSH Server on Linux. Add this in

# vim /etc/ssh/sshd_config

# Ciphers
Ciphers aes128-cbc,aes192-cbc,aes256-cbc,blowfish-cbc,arcfour
KexAlgorithms diffie-hellman-group1-sha1

*If you are using Centrify-OpenSSH, you have to modify /etc/centrifydc/ssh/sshd_config and do the same

References:

1. Bug 1228013 - Server responded "Algorithm negotiation failed"  

Calculate the fingerprint of a key file

# ssh-keygen -l -f id_rsa.pub
2048 ....................................................     yournode@headnode.com

Simple BASH script to setup shared SSH keys on Cluster

Do take a look at Simple BASH script to setup shared SSH keys on Cluster.

This can be run in the user directory to allow passwordless access through nodes in the cluster. 

SSH-keygen non-interactive command

If you want a non-interactive ssh-keygen command direct from ssh-keygen itself, see

# ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa

Encountering 'Write Failed: Broken Pipe" suring SSH connection on CentOS

I encountered this error Write Failed: Broken Pipe" during SSH connection. From what I know, it is caused by the Linux Server severing connections that have been idle too long ago.

To solve the issues, you can do the following

1. At your Linux Server Side, you can configure

# vim /etc/ssh/sshd_config

.....
ClientAliveInterval 60
.....

2. At your Client Side

# vim ~/.ssh/config 

.....
ServerAliveInterval 60
.....

Centrify Error - Not authenticated: while getting service credentials: No credentials found with supported encryption

I was not able to use authenticate with my password when I tried to logon with Putty. A closer look at the log file shows. Only the local account root was able to logon

Sep 17 12:00:00 node1 sshd[4725]: error: PAM: 
Authentication failure for user2 from 192.168.1.5
Sep 17 12:00:01 node1 adclient[7052]: WARN  audit User 'user2' not authenticated: 
while getting service credentials: 
No credentials found with supported encryption

The solution was very simple. Just restart the /etc/init.d/centrifydc and /etc/init.d/centrify-sshd

# service /etc/init.d/centrifydc restart
# service /etc/init.d/centrify-sshd restart

Helping users to SSH without password into the Compute Nodes manually

There are occasionally in a cluster environment that users accidentally delete their head node SSH keys and later cannot submit their jobs to the queue or their MPI jobs cannot scale beyond 1 node. The system you will see when you turn on the verbose method

To conduct a quick test,

# ssh -v remote-host

you will see an errors similar to  such as those below:

debug1: Unspecified GSS failure.  Minor code may provide more information
Unknown code krb5 195

OR

debug1: Miscellaneous failure
No credentials cache found

To reinstate the password-less access to compute nodes, you have to do the following. First thing first, please do backup files at your ~/.ssh/

Step 1: Regenerate the SSH keys
Auto SSH Login without Password

Step 2: Append the public keys ~/.ssh/id_rsa.pub and put into the ~/.ssh/authorized_keys

# cd ~/.ssh/

# cat id_rsa.pub >> authorized_keys

# chmod 400 /home/myuser/.ssh/authorized_keys

Step 3: Try ssh into the compute nodes. It should be clear password-less access to all nodes.

Unable to SSH due to failure of authentication with GSSAPI

I have a very slow or failed ssh connection due to failure with authentication with GSSAPI. See below my error

.....
debug1: Authentications that can continue: publickey,gssapi-with-mic,password
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information Unknown code krb5 195
debug1: Unspecified GSS failure.  Minor code may provide more information Unknown code krb5 195
debug1: Unspecified GSS failure.  Minor code may provide more information Unknown code krb5 195
debug1: Next authentication method: publickey
debug1: Trying private key: /home/user1/.ssh/identity
.....

It seems that when the ssh connection first try to authenticate with GSSAPI and when it failed, it switched to the publickey.

To change the ssh at user level

$ vim ~/.ssh/config

GSSAPIAuthentication no 

To change at global level

# /etc/ssh/sshd_config

GSSAPIAuthentication no 

More Information

1. Slow SSH connections – hanging at GSSAPI auth

Killing a SSH User Shell Session

As Administrator, when you do a notice that you will some of your users have idle and active ssh session. The idle ssh session could be be due to the hanged ssh session. So the question is how to remove the individual session without killing the active and genuine session.

First thing first, do

# w 

You may get some information like this

USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT 

.....
user1   pts/31   :24.0            08Oct12 22days  0.05s  0.05s -bash
user2   pts/24   :30              02Jan13  2days  0.66s  0.66s -bash
user3   pts/55   :17              12Nov12 59days  0.01s  0.01s -bash
.....

To get process id (PID) of the idle session, do the command

# ps -aux |grep 'pts/31'

Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.7/FAQ
root     27552  0.0  0.0  61172   776 pts/1    S+   00:41   0:00 grep pts/31
546      30050  0.0  0.0  64188  1516 pts/31   Ss+   2012   0:00 -bash

Kill the Process

# kill -9 30050 

The idle ssh session has been removed. You can verify with the command "w"

Encountering Warning No xauth data; using fake authentication data for X11 forwarding

I was encountering this error recently which trying to X forward to another remote site.

"Warning: No xauth data; using fake authentication data for X11 forwarding."
and there was no and doesn't display picture.

These are the steps I took to trouble-shoot

1. I checked my /etc/ssh/sshd_config and noted that the I have "X11Forwarding yes"
2. On my .ssh/config, I have the "ForwardX11 yes"
3. But one of my parameter /etc/ssh/sshd_config  "X11Uselocalhost yes". Apparently,I was able to X11 Forward for hosts specify in my /etc/hosts file, but those outside my host file, I was not able to display the picture.
4. But once I modified the  "X11Uselocalhost no", the issue was resolved.

There was this post that a user explained quite well. (http://www.authsecu.com/nntp/comp-security-ssh/19540-comp-security-ssh-what-does-%22x11uselocalhost-no%22-do.htm)

When doing X forwarding, sshd listens on a TCP socket for connections from X clients. Normally, it will accept connections addressed to the loopback address only (127.0.0.1), restricting it to clients on the local host. X11UseLocalhost no means it will accept connections from anywhere. 

Using sshpass to automate install ssh-copy-id to remote machines with CentOS 6.2

This is a follow-up of the writeup of  Tools to automate ssh-copy-id to remote servers. The Server OS used is CentOS 6.2. If you are automating scripts, you may have to modify the default settings SSH first and later write a simple bash scripts using sshpass to push the ssh-copy-id.

You may want to look at the
Automate pushing of ssh-copy-id to multiple servers from LinuxCluster

Tools to automate ssh-copy-id to remote servers

Well you can write scripts or if you prefer to use an open-source tools, you can use the sshpass from sourceforge.

According to ssh manpage

sshpass  is  a utility designed for running ssh using the mode referred to as "keyboard-interactive" password authentication, but in non-inter-active mode.

ssh  uses  direct  TTY  access to make sure that the password is indeed issued by an interactive keyboard user. Sshpass runs ssh in a dedicated tty, fooling it into thinking it is getting the password from an inter-active user.

The command to run is specified after sshpassâ own  options.  Typically it  will  be "ssh" with arguments, but it can just as well be any other command. The password prompt used by ssh is, however,  currently  hard-coded into sshpass.

Common example of use is

# sshpass -f password.txt ssh-copy-id user@remoteserver

Speeding up multiple ssh connections with ControlMaster

I'm assuming you are using OpenSSH 4.

If you are make multiple connections to the same server, you can enables the sharing of multiple sessions over a single network connections. In other words, the additional sessions will try to reuse the master instance's connection rather than initiating new ones.

Step 1: Create a config file in your ~/.ssh directory. Make sure the permission is readable and writable by the owner only ie permission of 600

Step 2: Add the following lines

Host *
   ControlMaster auto
   ControlPath ~/.ssh/master-%r@%h:%p

ControlMaster auto Tries to start a master if there is no existing connection or it will use an existing master connection. ControlPath is the location socket for the ssh processes to communicate among themselves. The %r, %h and %p are replaced with your user name, the host to which you're connecting and the port number—only ssh sessions from the same user to the same host on the same port can or should share a TCP connection, so each group of multiplexed ssh processes needs a separate socket.

Step 3a: To test the configuration, start an ssh session and keep it connected, you should see something like

...........
debug1: setting up multiplex master socket
debug1: channel 0: new [client-session]
...........

Step 3b: Launch another ssh connection to a the same server with the same userid

....................
debug1: auto-mux: Trying existing master
...................

Much of the materials come from  Speed Up Multiple SSH Connections to the Same Server (Linux Journal).

sys_copy and scp -rpb error captured on pbs_mom logs

I was encountering an interesting scp error on my log file regarding pbs_mom

.......pbs_mom: LOG_ERROR::sys_copy, 
command '/usr/bin/scp -rpB  2014.Head-Node.OU 
userid@headnode:/home/xxx' failed with status=1, 
giving up after 4 attempts

It seems that the error may be due to default MaxStartups 10 setting in the /etc/ssh/sshd_config which is too low a value and scp may be overwhelm

According to manual page
MaxStartups - Specifies the maximum number of concurrent unauthenticated connections to the sshd daemon.  Additional connections will be dropped until authentication succeeds or the LoginGraceTime
expires for a connection. The default is 10.

Try increasing the MaxStartups to 100 at /etc/ssh/sshd_config

MaxStartups 100 

Unspecified GSS failure from SSH causes slow login

I SSH into one of my server, But I encounter this error, but instead I encounter the follow error. Eventually, after waiting about 15-20 seconds, I'm able to connect to. This was far too long for a LAN-based machine 

$ ssh -v ip_of_remote_server

.....
debug1: Unspecified GSS failure.  Minor code may provide more information
Unknown code krb5 195

debug1: Unspecified GSS failure.  Minor code may provide more information
Unknown code krb5 195

debug1: Unspecified GSS failure.  Minor code may provide more information
Unknown code krb5 195
.....

I was quite puzzled. Although I am using IP address of the server to ssh and have tweaked "UseDNS = no" at /etc/sshd_config. See Resolving Slow SSH Login, In addition, I'm doing SSH public/private key authentication. See Auto SSH Login without Password


But the resolution for this issue was easier than I thought. I just need to ensure /etc/hosts contains both the servers I am ssh from and to and it became very quick.

If you are using DNS instead of /etc/hosts, do take a look that your DNS settings at /etc/resolv.conf

Other Issues on SSH, you may want to read about

1. SSH Error : Permission denied (publickey,gssapi-with-mic,password)
2. Resolving Slow SSH Login

Recommended sshd_config for OpenSSH

There are a few settings at /etc/ssh/sshd_config we can set to improve security, performance and user experience. Many of this information comes from SSH The Secure Shell, 2nd Edition from O'Reilly

1. Using SSH-2 Protocol and disable SSH-1 protocol altogether

Protocol 2

2. Ensure that the HostKey and PidFile are located on a machine's local disk and not over the NFS mount. The default setting should be in the machine local file like those below

HostKey /etc/ssh/ssh_host_key
PidFile /var/run/sshd.pid

3. File and directory permissions
The StrictModes value requires users to protect their SSH-related files and directories or else they will not authenticate.The default values is yes

StrictModes yes

4. Enable KeepAlive messages
Keepalive messages are enabled so that the connections to clients that have crashed or unreachable will terminate rather than be an orphaned process which require manual intervention by sysadmin to eliminate it.

Port 22 
ListenAddress 0.0.0.0
TcpKeepAlive yes

5. Disable Reverse DNS lookup

UseDNS no

6. Select a shorter grace login time
The default grace login is 2 minute which you might want to change. The value here is 30 seconds

LoginGraceTime 30

7. Authentication
The default setting are fine unless you wish to use Public-Key Authentication and wish to disabled Kerberos, Interactive and GSSAPIAuthentication

PubkeyAuthentication yes
PasswordAuthentication no
PermitEmptyPasswords no
RSAAuthentication yes
RhostsRSAAuthentication no
HostbasedAuthentication no
KerberosAuthentication no
ChallengeResponseAuthentication yes
GSSAPIAuthentication no
IgnoreRhosts yes

8. Access Control
If you wish to allow only selected users or groups to use ssh, you would like to use

AllowGroups users
AllowUsers me_only
DenyGroups black_list
DenyUsers hacker_id

For more information, see How do I permit specific users SSH access?


9. Securing TCP port forwarding and X forwarding

AllowTcpForwarding yes
X11Forwarding yes

Resolving Slow SSH Login

If you are facing slow login times, it might be due to reverse DNS is not responding quick enough. This system can show up on your log file

# tail -50 /var/log/secure

You will notice that there is a time lag from accepting the key to opening a session

Sep  6 10:15:42 santol-h00 sshd[4268]: 
Accepted password for root from 192.168.1.191 port 51109 ssh2

Sep  6 10:15:52 santol-h00 sshd[4268]: pam_unix(sshd:session): 
session opened for user root by (uid=0)

To fix the issue, you should modify the /etc/ssh/sshd_config file

# vim /etc/ssh/sshd_config

At /etc/ssh/sshd_config, change UseDNS  no

#ShowPatchLevel no
UseDNS no
#PidFile /var/run/sshd.pid

Restart the ssh service

# service sshd restart

Feel the login speed :)

Shutting down sFTP on CentOS

To disable sFTP, you have to disable under OpenSSH Subsystem option since sFTP protocol is NOT called directly but via OpenSSH

Step 1: Proceed to sshd_config file

# vim /etc/ssh/sshd_config

Step2: Comment out the line with a # inside /etc/ssh/sshd_config

# Subsystem sftp /usr/lib/openssh/sftp-server

Step 3: Restart OpenSSH service

# service sshd restart

Keeping OpenSSH from disconnecting

Sometimes whille connecting to another remote server via, there is this unplesant disconnection. How do we deal with it. You have to look at OpenSSH configuration

# /etc/ssh/sshd_config

ClientAliveInterval 15
ClientAliveCountMax 3

From the man page from ClientAliveInterval
Sets a timeout interval in seconds after which if no data has been    received from the client, sshd will send a message through the encrypted channel to request a response from the client.  The default
is 0, indicating that these messages will not be sent to the client.  This option applies to protocol version 2 only.

ClientAliveCountMax
Sets the number of client alive messages (see above) which may be sent without sshd receiving any messages back from the client. If this threshold is reached while client alive messages are being
sent, sshd will disconnect the client, terminating the session.

If ClientAliveInterval (above) is set to 15, and ClientAliveCountMax is left 3, unresponsive ssh clients will be disconnected after approximately 45 seconds.

Do note that the ClientAlive messages are sent through the encrypted channel. On the other hand, The TCPKeepAlive Messages is not sent through encrypted Channel and thus spoofable.