Recurring issues with DediRock VPS, and support asks for root password

slowservers · May 12

@zed said:
You have no way of knowing if "support" is just some outsourced fiverr monkey so the idea that it's ok for support to ask for your root creds because OBVIOUSLY they have root access to the host node is complete nonsense.

That is a good point. Not something that I would have considered.

Personally, I've never worked in hosting in a context where support was questionably outsourced, and nor would I be likely to trust a company that did. Outside of my own Slow Servers infrastructure, I have one server at ARP Networks ($10/month for 1GB memory VPS, I think), IRCNow Homestead VPS ($5/month for 2GB memory VPS), and a bit over $5/month VPS with OpenBSD.Amsterdam. I trust all three of them to be able to competently debug an issue if I have one. (I guess there's one straggler on Vultr, who I also trust to debug an issue, that I've yet to migrate into my Slow Servers infrastructure.)

None of those providers are cheap, by any means. I know there are cheaper hosts that are trustworthy -- I'm not aware of who they are. I will say that I'm extremely impressed at how active @DediRock is here, considering the extremely low prices.

Now the way I would debug this, with host access, is running tcpdump on the VM's interface. Attempt a SSH handshake with it, and see what happens.

If you see syn packets with no response, it's probably the customer's iptables settings. If the handshake gets part of the way there, things might be more interesting. A more privacy minded support tech might limit that tcpdump to just port 22, but either an inexperienced or a more experienced one might want to see everything. There could be an interesting ICMP unreachable message transmitted, or perhaps a DNS packet that goes out and is unanswered every time there's an SSH connection.

There are some issues that are much easier to debug with customer VM access, but a lot can be done from the outside. Now how much does it cost to have support who knows how to diagnose networking issues within a virtualized context? And/or to trust with host access?

DrNutella · May 13

@slowservers said:

@zed said:
You have no way of knowing if "support" is just some outsourced fiverr monkey so the idea that it's ok for support to ask for your root creds because OBVIOUSLY they have root access to the host node is complete nonsense.

That is a good point. Not something that I would have considered.

Personally, I've never worked in hosting in a context where support was questionably outsourced, and nor would I be likely to trust a company that did. Outside of my own Slow Servers infrastructure, I have one server at ARP Networks ($10/month for 1GB memory VPS, I think), IRCNow Homestead VPS ($5/month for 2GB memory VPS), and a bit over $5/month VPS with OpenBSD.Amsterdam. I trust all three of them to be able to competently debug an issue if I have one. (I guess there's one straggler on Vultr, who I also trust to debug an issue, that I've yet to migrate into my Slow Servers infrastructure.)

None of those providers are cheap, by any means. I know there are cheaper hosts that are trustworthy -- I'm not aware of who they are. I will say that I'm extremely impressed at how active @DediRock is here, considering the extremely low prices.

Now the way I would debug this, with host access, is running tcpdump on the VM's interface. Attempt a SSH handshake with it, and see what happens.

If you see syn packets with no response, it's probably the customer's iptables settings. If the handshake gets part of the way there, things might be more interesting. A more privacy minded support tech might limit that tcpdump to just port 22, but either an inexperienced or a more experienced one might want to see everything. There could be an interesting ICMP unreachable message transmitted, or perhaps a DNS packet that goes out and is unanswered every time there's an SSH connection.

There are some issues that are much easier to debug with customer VM access, but a lot can be done from the outside. Now how much does it cost to have support who knows how to diagnose networking issues within a virtualized context? And/or to trust with host access?

@DediRock has a real team folks. That’s all we need to hear.

JamesOakley · May 16

Having previously read this LET thread, I've just had the same thing, so recognised it.

I had opened a ticket to explain that my VPS had been dropping out its network connection repeatedly for a few minutes at a time, for several hours. I simply asked if there were any known issues with the DediRock network, and if so when they expected things to stabilise.

3 hours later I got a reply. “Please provide us the root password and SSH Port for further Investigation”

I suspect this is saved in WHMCS predefined replies.

Root login is disabled on this server, and I've run passwd - d root. There is no root password.
As others have said, the correct process is to give the customer the public SSH key of the support team, so we can save it in authorized_keys for an account in the wheel group.
They do not need to login to my VPS as root to answer the question. I know nothing has changed on the VPS, so the problem does not come from there. Either they have had some network instability, or they haven't. You don't need to log in to the customer's VPS to know the answer to that.

@DediRock says that they have the opportunity to improve the process here. It seems it's taking some time to make those changes, so here are my suggestions as to what to do:

Delete that from the WHMCS predefined replies
Tell all the support staff to triage tickets to see which need VPS access to troubleshoot. It may not be all support team asking for root access for everything, so maybe chat to “Scott P” first.
If they need access, do so via public key only
Explain to the customer “the reason we need this is so that I can check X, Y or Z which could be causing this”, rather than asking for such a high level of access without giving a reason.

As it stands, it seems that “Please provide us the root password and SSH Port for further Investigation” is a blanket sledgehammer response irrespective of what particular bug or nut is being asked about in the ticket.

ralf · May 16

@bmilk - if you don't trust the provider with your root password, then you shouldn't be using the server at all.

They don't need the root password to get all your data / sensitive whatevers / credentials etc from your VPS. If you're worried what they might do with the root password, then the logical conclusion is to not use that machine at all.

Given that you've decided to trust them as a provider, what can they do with a root password that they couldn't without it? Nothing, except do what most providers would expect you to do yourself - use the VNC console and log in and attempt to diagnose the problem. If you want them to diagnose something on the box, they could get the same access without the root password, but they'd have to do more invasive things like using guest agent, or maybe shutting it down, editing the password file in the disk image, rebooting and then diagnosing, or whatever. Giving them the password just makes it simpler.

As soon as they've done whatever they are doing, you should change the password immediately. If you're paranoid, you can audit what they've done, but again anything you might be paranoid they could have done with the root password, they could have done anyway without it if they really wanted.

Giving them the password is just effectively giving your consent for them to log into your box in a way that makes life easier for them.

Personally, I wouldn't, but then I wouldn't be troubling support with it anyway unless I'd proved (not just suspected) that there was packet loss and there were no firewall rules blocking those packets (which is the mostly likely outcome). For me, I'd want to reinstall my machine from scratch anyway in this kind of situation, and if you're doing that there's no point in bothering support at all.

TimboJones · May 16

@JamesOakley said:
Having previously read this LET thread, I've just had the same thing, so recognised it.

I had opened a ticket to explain that my VPS had been dropping out its network connection repeatedly for a few minutes at a time, for several hours. I simply asked if there were any known issues with the DediRock network, and if so when they expected things to stabilise.

3 hours later I got a reply. “Please provide us the root password and SSH Port for further Investigation”

I suspect this is saved in WHMCS predefined replies.

Root login is disabled on this server, and I've run passwd - d root. There is no root password.

As others have said, the correct process is to give the customer the public SSH key of the support team, so we can save it in authorized_keys for an account in the wheel group.

They do not need to login to my VPS as root to answer the question. I know nothing has changed on the VPS, so the problem does not come from there. Either they have had some network instability, or they haven't. You don't need to log in to the customer's VPS to know the answer to that.

@DediRock says that they have the opportunity to improve the process here. It seems it's taking some time to make those changes, so here are my suggestions as to what to do:

Delete that from the WHMCS predefined replies

Tell all the support staff to triage tickets to see which need VPS access to troubleshoot. It may not be all support team asking for root access for everything, so maybe chat to “Scott P” first.

If they need access, do so via public key only

Explain to the customer “the reason we need this is so that I can check X, Y or Z which could be causing this”, rather than asking for such a high level of access without giving a reason.

As it stands, it seems that “Please provide us the root password and SSH Port for further Investigation” is a blanket sledgehammer response irrespective of what particular bug or nut is being asked about in the ticket.

Would have saved time if you just provided debug details to them in the first place. I could see the tech wanting to just look himself out of frustration you didn't provide it in the first place.

ShalaWorks · May 16

My ny vps server keeps shutting down without any reason, i asked to support but they just restarted it without doing anything else. (also they requested me root password).

But I can't complain, it's very cheap for that.

Yachiyo · May 17

Strictly speaking, operations and maintenance do not fall within the scope of the obligations covered by the package.
It is already quite good that they are willing to help investigate the issue; elsewhere, the standard practice is usually to require an upgrade to a premium package that includes DevOps support.
Regarding the issue of the VPS throwing a 503 error every few days, I would be more inclined to investigate whether any programs running on the VPS are utilizing a compromised supply chain, or if unpatched vulnerabilities exist that have led to the VPS being compromised.

DediRock · May 17

@msatt said:
As an 'outsider' I would say - DR are being very generous to actually offer to look (internally) at your VPS. If they know their infrastructure is working then the problem is associated with the VPS. The VPS will very likely have been built from a template which means everyone using that template would be affected in the same way. So if you are the 'odd one out' then investigation needs to be made inside the VPS. This is outside the normal scope of support so credit to DR.
The fact you don't trust DR says more about you than them.
My advice would be backup and rebuild (not restore) - your problem will very likely disappear.

Thanks! @msatt,

We definitely try to give the best service we can

DediRock · May 17

@AlteredParadox said:
lol. just freaking lol. These threads lately, wtf. @DediRock you are a far more patient person than anyone I know if this is the kinda shit you deal with on a daily basis, my friend

op: if you have such huge trust issues with the host, then why are you there in the first place? Move somewhere else, im sure dedirock will got out of business losing your $10.

FWIW, i've had other providers ask for creds when troubleshooting, usually to rule out an issue on the vps before pivoting to their network/infra, which makes perfect sense given the differing knowledge/skill levels of customers. Generally ill just change the root pw, give it to them, get whatever the issue is fixed, and rebuild the box. Free backup testing

lol thanks @AlteredParadox Gotta smile and have fun, everything always works out well

DediRock · May 17

@DrNutella said:

@slowservers said:

@zed said:
You have no way of knowing if "support" is just some outsourced fiverr monkey so the idea that it's ok for support to ask for your root creds because OBVIOUSLY they have root access to the host node is complete nonsense.

That is a good point. Not something that I would have considered.

Personally, I've never worked in hosting in a context where support was questionably outsourced, and nor would I be likely to trust a company that did. Outside of my own Slow Servers infrastructure, I have one server at ARP Networks ($10/month for 1GB memory VPS, I think), IRCNow Homestead VPS ($5/month for 2GB memory VPS), and a bit over $5/month VPS with OpenBSD.Amsterdam. I trust all three of them to be able to competently debug an issue if I have one. (I guess there's one straggler on Vultr, who I also trust to debug an issue, that I've yet to migrate into my Slow Servers infrastructure.)

None of those providers are cheap, by any means. I know there are cheaper hosts that are trustworthy -- I'm not aware of who they are. I will say that I'm extremely impressed at how active @DediRock is here, considering the extremely low prices.

Now the way I would debug this, with host access, is running tcpdump on the VM's interface. Attempt a SSH handshake with it, and see what happens.

If you see syn packets with no response, it's probably the customer's iptables settings. If the handshake gets part of the way there, things might be more interesting. A more privacy minded support tech might limit that tcpdump to just port 22, but either an inexperienced or a more experienced one might want to see everything. There could be an interesting ICMP unreachable message transmitted, or perhaps a DNS packet that goes out and is unanswered every time there's an SSH connection.

There are some issues that are much easier to debug with customer VM access, but a lot can be done from the outside. Now how much does it cost to have support who knows how to diagnose networking issues within a virtualized context? And/or to trust with host access?

@DediRock has a real team folks. That’s all we need to hear.

Thank you @DrNutella

DediRock · May 17

@JamesOakley said:
Having previously read this LET thread, I've just had the same thing, so recognised it.

I had opened a ticket to explain that my VPS had been dropping out its network connection repeatedly for a few minutes at a time, for several hours. I simply asked if there were any known issues with the DediRock network, and if so when they expected things to stabilise.

3 hours later I got a reply. “Please provide us the root password and SSH Port for further Investigation”

I suspect this is saved in WHMCS predefined replies.

Root login is disabled on this server, and I've run passwd - d root. There is no root password.

As others have said, the correct process is to give the customer the public SSH key of the support team, so we can save it in authorized_keys for an account in the wheel group.

They do not need to login to my VPS as root to answer the question. I know nothing has changed on the VPS, so the problem does not come from there. Either they have had some network instability, or they haven't. You don't need to log in to the customer's VPS to know the answer to that.

@DediRock says that they have the opportunity to improve the process here. It seems it's taking some time to make those changes, so here are my suggestions as to what to do:

Delete that from the WHMCS predefined replies

Tell all the support staff to triage tickets to see which need VPS access to troubleshoot. It may not be all support team asking for root access for everything, so maybe chat to “Scott P” first.

If they need access, do so via public key only

Explain to the customer “the reason we need this is so that I can check X, Y or Z which could be causing this”, rather than asking for such a high level of access without giving a reason.

As it stands, it seems that “Please provide us the root password and SSH Port for further Investigation” is a blanket sledgehammer response irrespective of what particular bug or nut is being asked about in the ticket.

Hey @JamesOakley,

How are you? Sorry about that man. We had gone over this with support members, so it is getting grooved in. It takes some doing, but it is getting ironed out. Thanks for your suggestions, and thanks for your patience with us. I will message you directly to get the ticket number, as I would like to take a look. Have a great rest of your weekend, and thanks for your viewpoint on the matter.

DediRock Listens™

DediRock · May 17

@slowservers said:

@zed said:
You have no way of knowing if "support" is just some outsourced fiverr monkey so the idea that it's ok for support to ask for your root creds because OBVIOUSLY they have root access to the host node is complete nonsense.

That is a good point. Not something that I would have considered.

Personally, I've never worked in hosting in a context where support was questionably outsourced, and nor would I be likely to trust a company that did. Outside of my own Slow Servers infrastructure, I have one server at ARP Networks ($10/month for 1GB memory VPS, I think), IRCNow Homestead VPS ($5/month for 2GB memory VPS), and a bit over $5/month VPS with OpenBSD.Amsterdam. I trust all three of them to be able to competently debug an issue if I have one. (I guess there's one straggler on Vultr, who I also trust to debug an issue, that I've yet to migrate into my Slow Servers infrastructure.)

None of those providers are cheap, by any means. I know there are cheaper hosts that are trustworthy -- I'm not aware of who they are. I will say that I'm extremely impressed at how active @DediRock is here, considering the extremely low prices.

Now the way I would debug this, with host access, is running tcpdump on the VM's interface. Attempt a SSH handshake with it, and see what happens.

If you see syn packets with no response, it's probably the customer's iptables settings. If the handshake gets part of the way there, things might be more interesting. A more privacy minded support tech might limit that tcpdump to just port 22, but either an inexperienced or a more experienced one might want to see everything. There could be an interesting ICMP unreachable message transmitted, or perhaps a DNS packet that goes out and is unanswered every time there's an SSH connection.

There are some issues that are much easier to debug with customer VM access, but a lot can be done from the outside. Now how much does it cost to have support who knows how to diagnose networking issues within a virtualized context? And/or to trust with host access?

Hey @slowservers,

Thanks for that We are pushing, and things are going in the right direction, I can say. I do keep taking situations like these and learning from it. Thanks for the comment.

Thanks

DediRock · May 17

@ShalaWorks said:
My ny vps server keeps shutting down without any reason, i asked to support but they just restarted it without doing anything else. (also they requested me root password).

But I can't complain, it's very cheap for that.

Hey @ShalaWorks,

It is always good to hear from you We keep prices competitive I will DM you in a few minutes here.

The DediRock Family™

The DediRam Special™

DediRock · May 17

@Yachiyo said:
Strictly speaking, operations and maintenance do not fall within the scope of the obligations covered by the package.
It is already quite good that they are willing to help investigate the issue; elsewhere, the standard practice is usually to require an upgrade to a premium package that includes DevOps support.
Regarding the issue of the VPS throwing a 503 error every few days, I would be more inclined to investigate whether any programs running on the VPS are utilizing a compromised supply chain, or if unpatched vulnerabilities exist that have led to the VPS being compromised.

Hey @Yachiyo,

Thank you for that. Our support teams definitely try to help whenever they can. Thanks for your comment.

Thanks

JamesOakley · May 17

@DediRock said:
We had gone over this with support members, so it is getting grooved in. It takes some doing, but it is getting ironed out.

That's great to hear. Of course, these changes take time to embed, so the main thing to hear is that you've already been working on it.

@DediRock said:
I will message you directly to get the ticket number, as I would like to take a look.

I've got your DM, let's pick this up over there.

@DediRock said:
DediRock Listens™

Indeed, and for me that's what matters. Appreciate you taking the time to respond and to reach out over DM. I expect providers to get things wrong, we all do; good ones listen to clients and take ownership of problems. Keep at it™

Howdy, Stranger!

Categories

In this Discussion

Recurring issues with DediRock VPS, and support asks for root password

Comments

Howdy, Stranger!

Quick Links

Categories

In this Discussion

Recurring issues with DediRock VPS, and support asks for root password

Comments