Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Rsync cron works well, just today at 00:01 looks thats happend with ssh!!!
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Rsync cron works well, just today at 00:01 looks thats happend with ssh!!!

xrzxrz Member
edited February 2017 in Help

Cloudflare blocked my post so here it is - http://pastebin.com/raw/x6ENpyZf

Heeelp :(

Log is full of

Feb 5 07:07:01 sd-47323 sshd[28803]: error: Bad prime description in line 247
Feb 5 07:07:01 sd-47323 sshd[28803]: error: Bad prime description in line 248
Feb 5 07:07:01 sd-47323 sshd[28803]: error: Bad prime description in line 249
Feb 5 07:07:01 sd-47323 sshd[28803]: error: Bad prime description in line 250
Feb 5 07:07:01 sd-47323 sshd[28803]: error: Bad prime description in line 251
Feb 5 07:07:01 sd-47323 sshd[28803]: error: Bad prime description in line 252
Feb 5 07:07:01 sd-47323 sshd[28803]: error: Bad prime description in line 253
Feb 5 07:07:01 sd-47323 sshd[28803]: error: Bad prime description in line 254
Feb 5 07:07:01 sd-47323 sshd[28803]: error: Bad prime description in line 255
Feb 5 07:07:01 sd-47323 sshd[28803]: error: Bad prime description in line 256
Feb 5 07:07:01 sd-47323 sshd[28803]: error: Bad prime description in line 257
Feb 5 07:07:01 sd-47323 sshd[28803]: error: Bad prime description in line 258
Feb 5 07:07:01 sd-47323 sshd[28803]: error: Bad prime description in line 259
Feb 5 07:07:01 sd-47323 sshd[28803]: error: Bad prime description in line 260
Feb 5 07:07:01 sd-47323 sshd[28803]: error: Bad prime description in line 261
Feb 5 07:07:01 sd-47323 sshd[28803]: error: Bad prime description in line 262
Feb 5 07:07:01 sd-47323 sshd[28803]: error: Bad prime description in line 263
Feb 5 07:07:01 sd-47323 sshd[28803]: error: Bad prime description in line 264
Feb 5 07:07:01 sd-47323 sshd[28803]: error: Bad prime description in line 265
Feb 5 07:07:01 sd-47323 sshd[28803]: error: Bad prime description in line 266
Feb 5 07:07:01 sd-47323 sshd[28803]: error: Bad prime description in line 267

«1

Comments

  • What are permissions on sshd_config? Anything different?

    Thanked by 2xrz Basil
  • xrzxrz Member
    edited February 2017

    When i login i do not see anymore ubuntu welcome and how much space i have on /home ...

    cloudflare blocked angain my bad sry - http://pastebin.com/raw/gy0VNFTR

    when i do command who id does not print root ??

  • Yes, it's fucking annoying. What's your sshd look like remotely? Also, can you post your sshd_config on pastebin or elsewhere?

    Thanked by 1xrz
  • a u n a m e -a would be good to have as well.

    Thanked by 1xrz
  • I've never seen an sshd with everything commented out but the scp service before. I'm kind of stumped, because I have no idea what it'll actually do when setup like that.

  • What's a telnet to port 22 show for version, by chance?

  • xrzxrz Member
    edited February 2017

    Connected to ...
    Escape character is '^]'.
    SSH-2.0-OpenSSH_7.2p2

  • FalzoFalzo Member
    edited February 2017

    From the timestamp of the files it looks like someone or something messed around with ssh. Probably the bin got replaced and the config had been edited to not stand in the way... (that's also why you don't see your motd anymore)

    My guess would be the system has been compromised. Have you read the logfiles for that time and after?

    Boot in rescue mode mount the drives and search for files with that timestamp and newer to see what else differs and maybe work from there.

    Also you might try to force reinstall of openssh including overwrite of configs and such... but in the end might need to prepare for a reinstall of the system.

  • @WSS said:
    I've never seen an sshd with everything commented out but the scp service before. I'm kind of stumped, because I have no idea what it'll actually do when setup like that.

    Nothing special, this is how it works. It's specified in the file.

    The strategy used for options in the default sshd_config shipped OpenSSH is to specify options with their default value possible, but leave them commented. Uncommented options change a default value.

  • @putStrLn said:
    Nothing special, this is how it works. It's specified in the file.

    Do you remember if PermitRootLogin is allowed by default under 7.2/Ubuntian? I don't. :) That's my point. There are defaults, and there are defaults that have been changed, which is very common for Debian based systems. I'm too damn lazy to do a source unpack and look right now, but I might in a couple hours after a nap.

    @xrz said:

    Connected to ...
    Escape character is '^]'.
    SSH-2.0-OpenSSH_7.2p2

    The one thing that looks off here is that DebianBanner (Ubuntu) is not showing, and that is usually enabled by default:

    SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1

  • Also rsync couldn't connect to that server because the hostkey changed. That has to be related to the changes done to the sshd_config. So if OP didn't touch that at 00:01 someone else did. Supposed he is the only one using root or sudo...

    If it should have been cron the question is what job ran at that time that messes with ssh and their config? Any kind of upgrade? Should definitely not replace any configfiles esp. when unattended...

  • I'm going to say "Your ssh is not what is standardly installed", myself, with no further information. Look for modifications in your /root/.ssh/* files, and work from there.

  • @Falzo said:
    Also rsync couldn't connect to that server because the hostkey changed. That has to be related to the changes done to the sshd_config. So if OP didn't touch that at 00:01 someone else did. Supposed he is the only one using root or sudo...

    If it should have been cron the question is what job ran at that time that messes with ssh and their config? Any kind of upgrade? Should definitely not replace any configfiles esp. when unattended...

    There are not too many changes that will fuck up the host key, except for a skr1ptkiddy or a forced installation.. it's easier for them to just replace the binaries and leave the rest alone. So, unless this is a semi-managed host, I'd be concerned with anything/everything on there being compromised.

  • FalzoFalzo Member
    edited February 2017

    @WSS said:

    So, unless this is a semi-managed host, I'd be concerned with anything/everything on there being compromised.

    That's what I wrote in my first comment, which you might have overlooked ;-)

  • @Falzo said:

    @WSS said:

    So, unless this is a semi-managed host, I'd be concerned with anything/everything on there being compromised.

    That's what I wrote in my first comment, which you might have overlooked ;-)

    No, I didn't overlook it- I didn't consider it at all. I was doing my own assessment from the limited information I asked for and was given. It's a force of habit. :)

  • xrzxrz Member
    edited February 2017

    It all started 00:01 and see n sbin folder

    Feb 5 00:01:09 sd-47323 sshd[13120]: Server listening on 0.0.0.0 port 22.
    Feb 5 00:01:09 sd-47323 sshd[13120]: error: Bind to port 22 on :: failed: Address already in use.

    Hmmm ??

    lrwxrwxrwx 1 root root 30 févr. 4 23:58 sendmail-mta -> /etc/alternatives/sendmail-mta
    lrwxrwxrwx 1 root root 22 févr. 4 23:58 runq -> /etc/alternatives/runq
    lrwxrwxrwx 1 root root 28 févr. 4 23:58 newaliases -> /etc/alternatives/newaliases
    lrwxrwxrwx 1 root root 30 févr. 4 23:58 sendmail-msp -> /etc/alternatives/sendmail-msp
    lrwxrwxrwx 1 root root 26 févr. 4 23:58 sendmail -> /etc/alternatives/sendmail
    -rwxr-xr-x 1 root root 1362840 févr. 5 00:01 sshd

  • I had a similar situation a few months back when a server automatically upgraded cloud-init, which in turn regenerated SSH host keys.

    So before you suspect a compromised system, just check for any upgrade logs

  • in logs:

    /var/log$ cat alternatives.log
    update-alternatives 2017-02-04 23:58:17: run with --install /usr/sbin/sendmail-mta sendmail-mta /usr/lib/sm.bin/sendmail 25 --slave /usr/share/man/man8/sendmail-mta.8.gz sendmail-mta.8.gz /usr/share/man/man8/sendmail.sendmail.8.gz --slave /usr/sbin/runq runq /usr/share/sendmail/runq --slave /usr/share/man/man8/runq.8.gz runq.8.gz /usr/share/man/man8/runq.sendmail.8.gz --slave /usr/sbin/newaliases newaliases /usr/share/sendmail/newaliases --slave /usr/share/man/man8/newaliases.8.gz newaliases.8.gz /usr/share/man/man8/newaliases.sendmail.8.gz --slave /usr/bin/mailq mailq /usr/share/sendmail/mailq --slave /usr/share/man/man1/mailq.1.gz mailq.1.gz /usr/share/man/man1/mailq.sendmail.1.gz --slave /usr/share/man/man5/aliases.5.gz aliases.5.gz /usr/share/man/man5/aliases.sendmail.5.gz
    update-alternatives 2017-02-04 23:58:17: link group sendmail-mta updated to point to /usr/lib/sm.bin/sendmail
    update-alternatives 2017-02-04 23:58:17: run with --install /usr/sbin/sendmail-msp sendmail-msp /usr/lib/sm.bin/sendmail 25 --slave /usr/share/man/man8/sendmail-msp.8.gz sendmail-msp.8.gz /usr/share/man/man8/sendmail.sendmail.8.gz --slave /usr/sbin/sendmail sendmail /usr/lib/sm.bin/sendmail --slave /usr/lib/sendmail lib.sendmail /usr/lib/sm.bin/sendmail --slave /usr/share/man/man8/sendmail.8.gz sendmail.8.gz /usr/share/man/man8/sendmail.sendmail.8.gz
    update-alternatives 2017-02-04 23:58:17: link group sendmail-msp updated to point to /usr/lib/sm.bin/sendmail
    update-alternatives 2017-02-05 10:13:17: run with --install /usr/bin/maildirmake maildirmake /usr/bin/maildirmake.maildrop 5 --slave /usr/share/man/man1/maildirmake.1.gz maildirmake.1.gz /usr/share/man/man1/maildirmake.maildrop.1.gz
    update-alternatives 2017-02-05 10:13:17: link group maildirmake updated to point to /usr/bin/maildirmake.maildrop
    update-alternatives 2017-02-05 10:13:17: run with --install /usr/sbin/deliverquota deliverquota /usr/sbin/deliverquota.maildrop 10 --slave /usr/share/man/man8/deliverquota.8.gz deliverquota.8.gz /usr/share/man/man8/deliverquota.maildrop.8.gz
    update-alternatives 2017-02-05 10:13:17: link group deliverquota updated to point to /usr/sbin/deliverquota.maildrop
    update-alternatives 2017-02-05 10:13:17: run with --install /usr/bin/lockmail lockmail /usr/bin/lockmail.maildrop 5 --slave /usr/share/man/man1/lockmail.1.gz lockmail.1.gz /usr/share/man/man1/lockmail.maildrop.1.gz
    update-alternatives 2017-02-05 10:13:17: link group lockmail updated to point to /usr/bin/lockmail.maildrop
    update-alternatives 2017-02-05 10:13:17: run with --install /usr/share/man/man5/maildir.5.gz maildir.5.gz /usr/share/man/man5/maildir.maildrop.5.gz 5
    update-alternatives 2017-02-05 10:13:17: link group maildir.5.gz updated to point to /usr/share/man/man5/maildir.maildrop.5.gz
    update-alternatives 2017-02-05 10:13:17: run with --install /usr/share/man/man7/maildirquota.7.gz maildirquota.7.gz /usr/share/man/man7/maildirquota.maildrop.7.gz 5
    update-alternatives 2017-02-05 10:13:17: link group maildirquota.7.gz updated to point to /usr/share/man/man7/maildirquota.maildrop.7.gz
    update-alternatives 2017-02-05 10:13:17: run with --install /usr/bin/makedat makedat /usr/bin/makedat.maildrop 5 --slave /usr/share/man/man1/makedat.1.gz makedat.1.gz /usr/share/man/man1/makedat.maildrop.1.gz
    update-alternatives 2017-02-05 10:13:17: link group makedat updated to point to /usr/bin/makedat.maildrop
    update-alternatives 2017-02-05 10:13:32: run with --install /usr/bin/x-terminal-emulator x-terminal-emulator /usr/bin/xterm 20 --slave /usr/share/man/man1/x-terminal-emulator.1.gz x-terminal-emulator.1.gz /usr/share/man/man1/xterm.1.gz
    update-alternatives 2017-02-05 10:13:32: link group x-terminal-emulator updated to point to /usr/bin/xterm
    update-alternatives 2017-02-05 10:13:32: run with --install /usr/bin/x-terminal-emulator x-terminal-emulator /usr/bin/uxterm 20 --slave /usr/share/man/man1/x-terminal-emulator.1.gz x-terminal-emulator.1.gz /usr/share/man/man1/uxterm.1.gz
    update-alternatives 2017-02-05 10:13:32: run with --install /usr/bin/x-terminal-emulator x-terminal-emulator /usr/bin/koi8rxterm 20 --slave /usr/share/man/man1/x-terminal-emulator.1.gz x-terminal-emulator.1.gz /usr/share/man/man1/koi8rxterm.1.gz
    update-alternatives 2017-02-05 10:13:32: run with --install /usr/bin/x-terminal-emulator x-terminal-emulator /usr/bin/lxterm 30 --slave /usr/share/man/man1/x-terminal-emulator.1.gz x-terminal-emulator.1.gz /usr/share/man/man1/lxterm.1.gz
    update-alternatives 2017-02-05 10:13:32: link group x-terminal-emulator updated to point to /usr/bin/lxterm
    update-alternatives 2017-02-05 11:57:20: run with --quiet --remove rsh /usr/bin/ssh
    update-alternatives 2017-02-05 11:57:20: link group rsh fully removed
    update-alternatives 2017-02-05 11:57:20: run with --quiet --remove rlogin /usr/bin/slogin
    update-alternatives 2017-02-05 11:57:20: link group rlogin fully removed
    update-alternatives 2017-02-05 11:57:20: run with --quiet --remove rcp /usr/bin/scp
    update-alternatives 2017-02-05 11:57:20: link group rcp fully removed
    update-alternatives 2017-02-05 11:57:58: run with --quiet --install /usr/bin/rsh rsh /usr/bin/ssh 20 --slave /usr/share/man/man1/rsh.1.gz rsh.1.gz /usr/share/man/man1/ssh.1.gz
    update-alternatives 2017-02-05 11:57:58: link group rsh updated to point to /usr/bin/ssh
    update-alternatives 2017-02-05 11:57:58: run with --quiet --install /usr/bin/rlogin rlogin /usr/bin/slogin 20 --slave /usr/share/man/man1/rlogin.1.gz rlogin.1.gz /usr/share/man/man1/slogin.1.gz
    update-alternatives 2017-02-05 11:57:58: link group rlogin updated to point to /usr/bin/slogin
    update-alternatives 2017-02-05 11:57:58: run with --quiet --install /usr/bin/rcp rcp /usr/bin/scp 20 --slave /usr/share/man/man1/rcp.1.gz rcp.1.gz /usr/share/man/man1/scp.1.gz
    update-alternatives 2017-02-05 11:57:58: link group rcp updated to point to /usr/bin/scp

  • after all, something changed the ssh binary and the config. after that it obviously failed to restart maybe because it didn't stop properly or some other process hooked up on that port...

    yes I agree any (broken) update could cause such a mess, mentioned that earlier already... yet I do find it very... strange.

  • First of all I would check the system's uptime to see if the machine has been rebooted.

    Check who has been connecting via ssh with
    last

    or

    journalctl /usr/sbin/sshd|grep Accepted

    Check logs in /var/log/apt to see what has the package manager last done before the issue appeared.

    Thanked by 1Falzo
  • xrzxrz Member
    edited February 2017

    I removed the bins and installed new openssh-server so i dont have that in Accepted but in apt log see

    Start-Date: 2017-02-04 23:56:21
    Commandline: apt-get install sendmail
    Requested-By: xxx (1000)
    Install: liblockfile-bin:amd64 (1.09-6ubuntu1, automatic), m4:amd64 (1.4.17-5, automatic), sendmail-cf:amd64 (8.15.2-3, automatic), sendmail-bin:amd64 (8.15.2-3, automatic), sensible-mda:amd64 (8.15.2-3, automatic), sendmail:amd64 (8.15.2-3), sendmail-base:amd64 (8.15.2-3, automatic), liblockfile1:amd64 (1.09-6ubuntu1, automatic), procmail:amd64 (3.22-25, automatic), libsigsegv2:amd64 (2.10-4, automatic)
    End-Date: 2017-02-04 23:58:33

  • @xrz said:
    I removed the bins and installed new openssh-server so i dont have that in Accepted but in apt log see

    End-Date: 2017-02-04 23:58:33

    which ended before ssh was modified. did you install sendmail? if not I'd say this hardens the believe someone messed around with your server and probably installed sendmail to be able to run his spam-scripts or whatever... after that he also might have messed around with ssh for giving the bot control access - however your ssh has been configured before.

    it's about 12 hours after your first post... have you watched closely what processes are running and the network etc.? or is that box still in rescue mode? otherwise it may have send millions of spam mails already. lookup your IP in blacklist to see the outcome...

    Thanked by 1xrz
  • a) the offending line is "walking"/INC'd by 1 each time.
    b) please provide output of ssh -vv [username]@[your_server]

    Thanked by 1xrz
  • xrzxrz Member
    edited February 2017

    i have not intalled it thats a problem, i firewalled now everything even port 22, in netstat -tupen i only have 3 port opens i know about them and all firewalled, i reinstalled all bins but before i removed all ssh and sendmail anything with mail in bin sbin dirs so lets see

  • ssh -vv root@host
    OpenSSH_7.2p2 Ubuntu-4ubuntu2.1, OpenSSL 1.0.2g 1 Mar 2016
    debug1: Reading configuration data /etc/ssh/ssh_config
    debug1: /etc/ssh/ssh_config line 19: Applying options for *
    debug2: resolving "xxx" port 22
    debug2: ssh_connect_direct: needpriv 0
    debug1: Connecting to xxx [127.0.1.1] port 22.

  • You either left away most of the output (what you show is just the start) or you fucked up your setup big time.

    I'm particularly missing the KEX lines.

  • I had enabled firewall probably thats why, with disabled firewall its like this ->

    ssh -vv root@host
    OpenSSH_7.2p2 Ubuntu-4ubuntu2.1, OpenSSL 1.0.2g 1 Mar 2016
    debug1: Reading configuration data /etc/ssh/ssh_config
    debug1: /etc/ssh/ssh_config line 19: Applying options for *
    debug2: resolving "xxx" port 22
    debug2: ssh_connect_direct: needpriv 0
    debug1: Connecting to xxx [127.0.1.1] port 22.
    debug1: Connection established.
    debug1: permanently_set_uid: 0/0
    debug1: key_load_public: No such file or directory
    debug1: identity file /root/.ssh/id_rsa type -1
    debug1: key_load_public: No such file or directory
    debug1: identity file /root/.ssh/id_rsa-cert type -1
    debug1: key_load_public: No such file or directory
    debug1: identity file /root/.ssh/id_dsa type -1
    debug1: key_load_public: No such file or directory
    debug1: identity file /root/.ssh/id_dsa-cert type -1
    debug1: key_load_public: No such file or directory
    debug1: identity file /root/.ssh/id_ecdsa type -1
    debug1: key_load_public: No such file or directory
    debug1: identity file /root/.ssh/id_ecdsa-cert type -1
    debug1: key_load_public: No such file or directory
    debug1: identity file /root/.ssh/id_ed25519 type -1
    debug1: key_load_public: No such file or directory
    debug1: identity file /root/.ssh/id_ed25519-cert type -1
    debug1: Enabling compatibility mode for protocol 2.0
    debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1

    ssh_exchange_identification: read: Connection reset by peer

  • bsdguybsdguy Member
    edited February 2017

    Let me guess: You have configured the server sshd with key only authentication, right?

    Have a look into your local .ssh directory.

    So, again, as many have problems with that:

    First, while sshd_config is still default (and normal password login is enabled)
    scp .ssh/some_key**.pub** [user]@[server]:/home/[user] (with "some_key" standing for whatever you choose, say "id_rsa.pub")

    Then, on the server as that user (not as root unless your want to enable it for root access):

        mkdir .ssh
        chmod 700 .ssh
        mv [some_key.pub] .ssh/authorized_keys
        chmod 600 .ssh/authorized_keys
    

    In case .ssh already exists, don't care about any warning. In case .ssh/authorized_keys already exists, instead of mv ... you use cat [some_key.pub] >> .ssh/authorized_keys and then, if it works well, rm [some_key.pub]

    Thanked by 1xrz
  • xrzxrz Member
    edited February 2017

    Ops forgot to say i have it like this:

    iptables -> DROP others ALLOW only ME -> host.allow only me and host.deny all others -> user ssh password -> 2Factor Auth -> logged in as user

Sign In or Register to comment.