Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm trying to install bash as the default shell on a ARM Linux running on an embedded device (Synology DS212+ NAS). But there's something really wrong, and I can't figure out what it is.

Symptoms:

1) Root has /bin/bash as default shell, and can log in normally via SSH:

$ grep root /etc/passwd
root:x:0:0:root:/root:/bin/bash

$ ssh root@NAS
root@NAS's password:
Last login: Sun Dec 16 14:06:56 2012 from desktop
# 


2) joeuser has /bin/bash as default shell, and receives "Permission denied" when trying to log in via SSH:

$ grep joeuser /etc/passwd
joeuser:x:1029:100:Joe User:/home/joeuser:/bin/bash

$ ssh joeuser@localhost
joeuser@NAS's password:
Last login: Sun Dec 16 14:07:22 2012 from desktop
Permission denied, please try again.
Connection to localhost closed.


3) changing joeuser's shell back to /bin/sh:

$ grep joeuser /etc/passwd
joeuser:x:1029:100:Joe User:/home/joeuser:/bin/sh

$ ssh joeuser@localhost
Last login: Sun Dec 16 15:50:52 2012 from localhost
$ 


To make things even more strange, I can log in as joeuser using /bin/bash using the serial console (!). Also a su - joeuser as root works fine, so the bash binary itself is working fine.

In an act of despair, I changed joeuser's uid to 0 on /etc/passwd, but also didn't work, so it doesn't seem to be anything permission related.

Seems that bash is doing some extra checking that sshd didn't like, and blocking the connections for non-root users. Maybe some sort of sanity checking - or terminal emulation - that is triggering the SIGCHLD, but only when called via ssh.

I already went through every single item on sshd_config, and also put SSHD in debug mode, but didn't find anything strange. Here's my /etc/ssh/sshd_config:

LogLevel DEBUG
LoginGraceTime 2m
PermitRootLogin yes
RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile      %h/.ssh/authorized_keys
ChallengeResponseAuthentication no
UsePAM yes
AllowTcpForwarding no
ChrootDirectory none
Subsystem       sftp    internal-sftp -f DAEMON -u 000


And here's the output from /usr/syno/sbin/sshd -d, showing the failed attempt of joeuser trying to log in, with /bin/bash as the shell:

debug1: Config token is loglevel
debug1: Config token is logingracetime
debug1: Config token is permitrootlogin
debug1: Config token is rsaauthentication
debug1: Config token is pubkeyauthentication
debug1: Config token is authorizedkeysfile
debug1: Config token is challengeresponseauthentication
debug1: Config token is usepam
debug1: Config token is allowtcpforwarding
debug1: Config token is chrootdirectory
debug1: Config token is subsystem
debug1: HPN Buffer Size: 87380
debug1: sshd version OpenSSH_5.8p1-hpn13v11
debug1: read PEM private key done: type RSA
debug1: private host key: #0 type 1 RSA
debug1: read PEM private key done: type DSA
debug1: private host key: #1 type 2 DSA
debug1: read PEM private key done: type ECDSA
debug1: private host key: #2 type 3 ECDSA
debug1: rexec_argv[0]='/usr/syno/sbin/sshd'
debug1: rexec_argv[1]='-d'
Set /proc/self/oom_adj from 0 to -17
debug1: Bind to port 22 on ::.
debug1: Server TCP RWIN socket size: 87380
debug1: HPN Buffer Size: 87380
Server listening on :: port 22.
debug1: Bind to port 22 on 0.0.0.0.
debug1: Server TCP RWIN socket size: 87380
debug1: HPN Buffer Size: 87380
Server listening on 0.0.0.0 port 22.

debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 6 out 6 newsock 6 pipe -1 sock 9
debug1: inetd sockets after dupping: 4, 4
Connection from 127.0.0.1 port 52212
debug1: HPN Disabled: 0, HPN Buffer Size: 87380
debug1: Client protocol version 2.0; client software version OpenSSH_5.8p1-hpn13v11
SSH: Server;Ltype: Version;Remote: 127.0.0.1-52212;Protocol: 2.0;Client: OpenSSH_5.8p1-hpn13v11
debug1: match: OpenSSH_5.8p1-hpn13v11 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.8p1-hpn13v11
debug1: permanently_set_uid: 1024/100
debug1: MYFLAG IS 1
debug1: list_hostkey_types: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: AUTH STATE IS 0
debug1: REQUESTED ENC.NAME is 'aes128-ctr'
debug1: kex: client->server aes128-ctr hmac-md5 none
SSH: Server;Ltype: Kex;Remote: 127.0.0.1-52212;Enc: aes128-ctr;MAC: hmac-md5;Comp: none
debug1: REQUESTED ENC.NAME is 'aes128-ctr'
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: expecting SSH2_MSG_KEX_ECDH_INIT
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: KEX done
debug1: userauth-request for user joeuser service ssh-connection method none
SSH: Server;Ltype: Authname;Remote: 127.0.0.1-52212;Name: joeuser
debug1: attempt 0 failures 0
debug1: Config token is loglevel
debug1: Config token is logingracetime
debug1: Config token is permitrootlogin
debug1: Config token is rsaauthentication
debug1: Config token is pubkeyauthentication
debug1: Config token is authorizedkeysfile
debug1: Config token is challengeresponseauthentication
debug1: Config token is usepam
debug1: Config token is allowtcpforwarding
debug1: Config token is chrootdirectory
debug1: Config token is subsystem
debug1: PAM: initializing for "joeuser"
debug1: PAM: setting PAM_RHOST to "localhost"
debug1: PAM: setting PAM_TTY to "ssh"
debug1: userauth-request for user joeuser service ssh-connection method password
debug1: attempt 1 failures 0
debug1: do_pam_account: called
Accepted password for joeuser from 127.0.0.1 port 52212 ssh2
debug1: monitor_child_preauth: joeuser has been authenticated by privileged process
debug1: PAM: establishing credentials
User child is on pid 9129
debug1: Entering interactive session for SSH2.
debug1: server_init_dispatch_20
debug1: server_input_channel_open: ctype session rchan 0 win 65536 max 16384
debug1: input_session_request
debug1: channel 0: new [server-session]
debug1: session_new: session 0
debug1: session_open: channel 0
debug1: session_open: session 0: link with channel 0
debug1: server_input_channel_open: confirm session
debug1: server_input_global_request: rtype no-more-sessions@openssh.com want_reply 0
debug1: server_input_channel_req: channel 0 request pty-req reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req pty-req
debug1: Allocating pty.
debug1: session_new: session 0
debug1: session_pty_req: session 0 alloc /dev/pts/1
debug1: server_input_channel_req: channel 0 request shell reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req shell
debug1: Setting controlling tty using TIOCSCTTY.

debug1: Received SIGCHLD.
debug1: session_by_pid: pid 9130
debug1: session_exit_message: session 0 channel 0 pid 9130
debug1: session_exit_message: release channel 0
debug1: session_by_tty: session 0 tty /dev/pts/1
debug1: session_pty_cleanup: session 0 release /dev/pts/1
Received disconnect from 127.0.0.1: 11: disconnected by user
debug1: do_cleanup
debug1: do_cleanup
debug1: PAM: cleanup
debug1: PAM: closing session
debug1: PAM: deleting credentials


Here you have the full output of sshd -dd, together with ssh -vv.

Bash:

# bash --version
GNU bash, version 3.2.49(1)-release (arm-none-linux-gnueabi)
Copyright (C) 2007 Free Software Foundation, Inc.

The bash binary was cross compiled from source. I also tried using a pre-compiled binary from the Optware distribution, but had the exact same problem. I checked for missing shared libraries using objdump -x, but they're all there.

Any ideas what could be causing this "Permission denied, please try again."? I'm almost diving in the bash source code to investigate, but trying to avoid hours chasing something that may be silly.

EDIT: adding more information about bash and the system

$ ls -la /bin/bash
-rwxr-xr-x    1 root     root        724676 Dec 15 23:57 /bin/bash

$  file /bin/bash
/bin/bash: ELF 32-bit LSB executable, ARM, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.14, stripped

$  uname -a
Linux NAS 2.6.32.12 #2661 Mon Nov 12 23:10:15 CST 2012 armv5tel GNU/Linux synology_88f6282_212+

$ grep bash /etc/shells
/bin/bash
/bin/bash2
share|improve this question
2  
ls -l /bin/bash –  Michael Hampton Dec 16 '12 at 21:55
    
Is /bin/bash listed in /etc/shells ? –  tink Dec 17 '12 at 0:30
    
Yes, it's listed in /etc/shells. And permissions 0755, added above. –  Gui Ambros Dec 17 '12 at 0:41
    
Most likely there is something being run in his .bashrc file. Can you cat ~joeuser/.bashrc and poke around his profile scripts? You could also try running /bin/bash while logged in as him. –  vicfn Dec 17 '12 at 1:03
    
No ~joeuser/.ssh and no profile scripts. It's a blank user I just created to test. –  Gui Ambros Dec 17 '12 at 1:27
show 1 more comment

6 Answers 6

up vote 9 down vote accepted

For future reference: after way too many hours researching and debugging this issue, I finally discovered the root cause.

The OpenSSH version used by Synology is a highly customized version, that does not behave like the original code. It has lots of hacks and ad-hoc customizations - e.g., additional checking before accepting a login to see if the SSH service is enabled within the web interface, or stripping special chars (;, |, ') from rsync commands, or... wait for it... avoiding regular users to use a shell different than /bin/sh or /bin/ash. Yeah, hard coded within the binary.

Here's the piece of code from OpenSSH 5.8p1, as distributed by Synology on their source code (DSM4.1 - branch 2636), file session.c:

void do_child(Session *s, const char *command)
{
...

#ifdef MY_ABC_HERE
   char szValue[8];
   int RunSSH = 0;
   SSH_CMD SSHCmd = REQ_UNKNOWN;

   if (1 == GetKeyValue("/etc/synoinfo.conf", "runssh", szValue, sizeof(szValue))) {
           if (strcasecmp(szValue, "yes") == 0) {
                   RunSSH = 1;
           }
   }

   if (IsSFTPReq(command)){
           SSHCmd = REQ_SFTP;
   } else if (IsRsyncReq(command)){
           SSHCmd = REQ_RSYNC;
   } else if (IsTimebkpRequest(command)){
           SSHCmd = REQ_TIMEBKP;
   } else if (RunSSH && IsAllowShell(pw)){
           SSHCmd = REQ_SHELL;
   } else {
           goto Err;
   }

   if (REQ_RSYNC == SSHCmd) {
           pw = SYNOChgValForRsync(pw);
   }
   if (!SSHCanLogin(SSHCmd, pw)) {
           goto Err;
   }
   goto Pass;

 Err:
   fprintf(stderr, "Permission denied, please try again.\n");
   exit(1);

 Pass:
   #endif /* MY_ABC_HERE */
...
}


As you can imagine, the IsAllowShell(pw) was the culprit:

static int IsAllowShell(const struct passwd *pw)
{
     struct passwd *pUnPrivilege = NULL;
     char *szUserName = NULL;
     if (!pw || !pw->pw_name) {
             return 0;
     }
     szUserName = pw->pw_name;
     if(!strcmp(szUserName, "root") || !strcmp(szUserName, "admin")){
             return 1;
     }
     if (NULL != (pUnPrivilege = getpwnam(szUserName))){
             if (!strcmp(pUnPrivilege->pw_shell, "/bin/sh") || 
                     !strcmp(pUnPrivilege->pw_shell, "/bin/ash")) {
                     return 1;
             }
     }
     return 0;
}


No wonder why I was experiencing such an odd behavior. Only shells /bin/sh and /bin/ash would be accepted for users different than root or admin. And this regardless of the uid (I had tested also making joeuser uid=0, and it didn't work. Now it's obvious why).

Once identified the cause, the fix was easy: just remove the call to IsAllowShell(). It took me a while to the right configuration to cross-compile openssh and all its dependencies, but in the end it worked well.

If anyone is interested in doing the same (or trying to cross-compile other kernel modules or binaries for Synology), here's my version of Makefile. It was tested with OpenSSH-5.8p1 source, and works well with models running Marvell Kirkwood mv6281/mv6282 CPU (like DS212+). I used a host running Ubuntu 12.10 x64.

Bottom line: bad practice, terrible code, and a great example of what not to do. I understand sometimes OEMs need to develop special customizations, but they should think twice before digging too deep. Not only this results in unmaintainable code for them, but also creates all sorts of unforeseen issues down the road. Thankfully GPL exist to keep them honest - and open.

share|improve this answer
    
Stellar work sir, in investigating this issue and pulling the curtain back from the Synology stage. I must concede I had thought quite highly of my Diskstation till now as it has been very useful and once bootstrapped is now even running some scripts for me under the new task scheduler. But you've revealed some flaws in short-sighted design here. I am still pleased with the Diskstation but will keep your post in mind. Upvoted both post and answer. –  Bernard Dy Sep 10 '13 at 3:12
add comment

To circumvent the problem, and since I installed bash via ipkg and I can't be sure /opt will always be available (mounted correctly), I simply put the following in my .profile

[ -x /opt/bin/bash ] && exec /opt/bin/bash

while /etc/passwd contains /bin/ash as shell.

share|improve this answer
add comment

Let's see. It's isolated to a single shell, plus you're looking at sshd debug output, so it's not world writable permission problems with ~joeuser/.ssh. That's the one that gets most people.

Have you tried creating an additional normal user (i.e. not joeuser) to make sure it experiences the same problem? That would isolate it to the user's configuration vs. system-wide configuration.

If it's a system-wide issue, the next thing I'd take a look at are shared configuration files like /etc/profile that get sourced by everyone. There might be a conditional block that isn't firing if the username is root. (not effective userid, since you already tested for that)

Check dmesg for segmentation fault reports if you haven't already, just in case there's something even weirder going on.

share|improve this answer
    
Thanks Andrew. Also checked dmesg, and absolutely nothing. Creating a brand new user also didn't help, so clearly it is a system wide issue. And joeuser can log in normally if I change the shell back to /bin/ash, which rules out any /etc/profile or other (which I also checked, btw). It's just with non-root, using bash, via ssh. As this point it's not even a real issue (I can live with it as it is now), but it annoys me to admit that I wasn't able to find the cause for this weird behavior. It is a deterministic system after all; there must be some sort of explanation... –  Gui Ambros Jan 17 '13 at 7:01
add comment

try /etc/ssh/sshd_config
search for AllowUsers

if it's there try adding joeuser there, just username

also it may be blocked in pam... I do not recall which file is it...

share|improve this answer
    
Not the case. If it was an issue with AllowUsers, it wouldn't let me log in with joeuser when I change the shell to /bin/ash. This doesn't seem to be related to anything security-related, or any other config. It's probably some sort of how bash handles pseudo terminals via ssh, versus ash, csh, etc. –  Gui Ambros Jan 17 '13 at 7:04
add comment

Try reinstalling Bash and see if that helps.

share|improve this answer
    
Nops, I got bash from two different sources (one from Optware distribution, and I also compiled 3.2 from source myself), and same problem. I went to the extreme of even getting bash 4.2 and applying the patches (all 39 of them), and cross-compiling. Exact same behavior. Works with root, Permission denied for others. My last guess is the OpenSSH binary, which is the same installed by default by Synology's firmware. This is the only piece that I didn't touch yet. I'll try downloading the latest source and cross-compiling, to see what happens. –  Gui Ambros Dec 18 '12 at 5:22
    
Strange not sure but this seems to be authconfig issue. Is ldap also running on the server? Can you send me real time log output of secure and message file at the time of login failure. –  PratapSingh Dec 18 '12 at 9:28
    
Also login as root to the server and let the user shell be /bin/bash and try to switch to user using su - username. Show me the output of this too. –  PratapSingh Dec 18 '12 at 10:49
add comment

Their modified version of openssh looks for a /bin/sh shell ?

Easy solution then:

ln -fs /bin/bash /bin/sh

share|improve this answer
    
This would change the shell for all users, and not only those that needed bash. Also any new upgrade would remove the symbolic link (even though fixing openssh also has the same problem). Anyway, this was solved, as described above. –  Gui Ambros Apr 9 '13 at 17:46
    
OK true, but as I am the only user on my NAS this solution works for me. Anyway thanks for pointing out what you discovered. –  Ponytech Apr 9 '13 at 20:53
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.