Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Do leaseweb resellers have access to VPS/hypervisor?

2»

Comments

  • @luckypenguin @forest Can you please share basic steps about securing linux VPS.

    1. Install own ISO, how do you mount an iso in VPS, you download iso to VPS, then install from there?
    2. Encryption. Whole disk encryption? Then how do I add encryption password at boot?
    3. @luckypenguin you mentioned not leaving the console logged in. Doesn't VNC ask for password whenever you connect with console like RDP? How do I disconnect a VNC session securely? and you said NOT to use provider template, what did you mean by that?
    Thanked by 1HostDZire
  • LeviLevi Veteran

    @EvoSteven said:

    @Levi said:

    @forest said:

    @Levi said:
    So, customer disables qemu agent, encrypt vps disk. Then provider gets some abuse letters. What would normally happen?

    They press "suspend" and the VM shuts down.

    If I would be reseller, I would be extremelly skeptical of disk encryption and qemu disable actions. Since middleman has massive risk to be terminated by vendor for repeated offences. Csam and all that shit

    ... and I don't like that the barebones of privacy and security instantly implies someone has something to hide?

    It does imply that you are afraid of something. Resellers wants “normal” customers to shift liability if something happens.

    @forest said:

    @Levi said:

    @forest said:

    @Levi said:
    So, customer disables qemu agent, encrypt vps disk. Then provider gets some abuse letters. What would normally happen?

    They press "suspend" and the VM shuts down.

    If I would be reseller, I would be extremelly skeptical of disk encryption and qemu disable actions. Since middleman has massive risk to be terminated by vendor for repeated offences. Csam and all that shit

    It's not up to the reseller to snoop on private files even if someone is accused of possessing illegal pornography. If they get a legitimate court order, then they act based on that and that alone. They don't try to change your root password and log in and browse your media library looking for dirty things, so their inability to change your root password should raise no alarms.

    Up to reseller to guarantee that resold services won’t be used for abuse.

    P.s. there is no provider which bans encryption and qemu removal. So all of this is just hypothetical.

  • HostDZireHostDZire Patron Provider, Veteran

    @Obelous said:

    @Calypso said: Just curious... Why would it be an issue that a reseller has access, but not that someone at Leaseweb has it?

    Personally I trust Leaseweb way more than some guy reselling them.

    When it comes to trust, the reality is that trusting absolutely no one is the only way to eliminate trust-related risk completely.

    Even a company as large as Leaseweb, OVH, AWS, or any other major provider still has employees with different levels of access. If someone is storing extremely sensitive information inside a VPS, such as cryptocurrency private keys, there is always some degree of human risk. One dishonest employee, compromised account, or internal security failure can potentially create a serious problem.

    We have already seen many security incidents over the years where insider access, employee mistakes, stolen credentials, or abuse of privileged access contributed to a breach.

    This is also one of the reasons governments and organizations care so much about data sovereignty and data residency. Many countries require certain categories of data to remain within their own jurisdiction because they want greater control over which laws, governments, companies, and personnel may potentially have access to that data. It is not simply about whether the foreign company is large or reputable.

  • @Levi said:

    @EvoSteven said:

    @Levi said:

    @forest said:

    @Levi said:
    So, customer disables qemu agent, encrypt vps disk. Then provider gets some abuse letters. What would normally happen?

    They press "suspend" and the VM shuts down.

    If I would be reseller, I would be extremelly skeptical of disk encryption and qemu disable actions. Since middleman has massive risk to be terminated by vendor for repeated offences. Csam and all that shit

    ... and I don't like that the barebones of privacy and security instantly implies someone has something to hide?

    It does imply that you are afraid of something. Resellers wants “normal” customers to shift liability if something happens.

    @forest said:

    @Levi said:

    @forest said:

    @Levi said:
    So, customer disables qemu agent, encrypt vps disk. Then provider gets some abuse letters. What would normally happen?

    They press "suspend" and the VM shuts down.

    If I would be reseller, I would be extremelly skeptical of disk encryption and qemu disable actions. Since middleman has massive risk to be terminated by vendor for repeated offences. Csam and all that shit

    It's not up to the reseller to snoop on private files even if someone is accused of possessing illegal pornography. If they get a legitimate court order, then they act based on that and that alone. They don't try to change your root password and log in and browse your media library looking for dirty things, so their inability to change your root password should raise no alarms.

    Up to reseller to guarantee that resold services won’t be used for abuse.

    P.s. there is no provider which bans encryption and qemu removal. So all of this is just hypothetical.

    Yes, that's why I don't have doors. I could hide a crack lab in my apartment

  • @JustLookingAround

    1) Many providers offer custom ISO right from the panel, if not, they can mount it thru a support ticket, however this is less recommended. If they don't have this from the panel, you can still use netboot.xyz (iPXE mode) on most hypervisors.

    Hit ESC during boot, choose iPXE, then follow these steps:
    https://netboot.xyz/docs/quick-start/

    2) Yes, whole disk encryption. Depends on the OS, modern Linux/BSD semi-automate this
    process - you only need to configure it during initial OS setup, the bootloader will prompt
    you for password itself - unless you want to take it one step further and unlock remotely,
    without ever needing to VNC, refer to this:
    https://www.cyberciti.biz/security/how-to-unlock-luks-using-dropbear-ssh-keys-remotely-in-linux/

    3) VNC only asks for client/server auth password, it has nothing to do with your tty and has no knowledge about what state the current terminal is in. So when you do something,
    make sure to type "logout" or "exit" - you should see the login prompt, not your shell.
    You can then disconnect from VNC knowing that whoever connects will meet the login prompt, not your actual logged in shell. For Windows, you simply log off the user via GUI.

    A provider template is what the provider gives you to "auto-install". Outdated in most cases, and does not offer disk encryption since it's a ready to go template.

  • HostDZireHostDZire Patron Provider, Veteran

    @forest said:

    @HostDZire said: In leaseweb vps we do not get access to hypervisor, however there is option to reset root password, and since provider has access to the vps all the time, he can always reset password and get access to your vps,

    Is password reset all you can do? Because that likely requires QEMU Guest Agent. If the agent is disabled (or neutralized), what access do you as a Leaseweb reseller still have?

    Yes, we have all the features you guys see in the panel, plus some extra features like the Security Firewall tab, which is not available through the API yet, and upgrade options.

    @Levi have provided already screenshots of panel, you can see

    But since we are the owner of that VPS, we can ask Leaseweb to do certain things with it. Resetting the password is already available in the panel, but we may also be able to ask for more, like restoring the last backup Leaseweb created or something similar.

    I have never asked them for this before, so I can’t confirm exactly what they can or cannot do

    Thanked by 1JustLookingAround
  • HostDZireHostDZire Patron Provider, Veteran

    @LEmeINalr said:

    @kenjing789 said: If you worry about provider snooping your data, then buy it directly from LWs.

    LW doesn't sell directly to end users unless you have a big enough business to back it up.

    @JustLookingAround said: The reason I'm concerned is I'll have customers data on it, so I'd rather not take the additional risk.

    If you're genuinely worried about that then I dont think you should be going for a LET host and instead go for Amazon or Oracle or whatnot. I personally dont keep any critical data on any of my LET hosts because I know they are offering me cheap services and are not billion dollar businesses that can protect it from every hack.

    Exactly. Buying a VPS at low-end prices and expecting defense-level security will never go together. It really depends on how much security your data actually requires.

    Most hosting providers use WHMCS, and these days critical security updates seem to be coming out like tea and coffee. But that does not necessarily mean other panels are more secure. WHMCS is one of the most widely used billing panels, so naturally it gets more attention from attackers and security researchers. Other panels may also have vulnerabilities, but they may simply not be targeted as heavily.

    And it’s not just WHMCS. We’ve also seen vulnerabilities reported in cPanel, Plesk, Virtualizor, KVM, and many other platforms.

  • HostDZireHostDZire Patron Provider, Veteran

    @forest said:

    @Levi said: So yes, middleman has access to root, rescue console, basically all functions for self-managed server.

    Seems like you could protect from that by disabling the guest agent and encrypting the disk. Then the only way in would require the ability to directly modify memory state, which would require root on the node (which resellers don't have).

    Yes, a lot also depends on how you secure the VPS yourself. If you know what you’re doing, there are several measures you can apply to make it significantly harder for anyone to access your data.

  • @HostDZire said: And it’s not just WHMCS. We’ve also seen vulnerabilities reported in cPanel, Plesk, Virtualizor, KVM, and many other platforms

    This is why you follow the steps @forest and I mentioned earlier.
    A compromised panel, or a dishonest employee should not be able to access your data,
    even if it's a VPS. Or at least you should make it as technically hard as possible, which
    will require dumping memory as root on the hypervisor, looking for the encryption key,
    messing with the qcow2 image to unlock it, etc.
    As we seen, most opportunistic compromises are just ransomware attacks, in which case
    they go for low hanging fruits - you don't need to make it 100% proof, you just need to
    make it reasonably hard enough to make the effort. Same goes with any physical lock.

  • HostDZireHostDZire Patron Provider, Veteran
    edited September 9

    @Levi said:

    @forest said:

    @Levi said:
    So, customer disables qemu agent, encrypt vps disk. Then provider gets some abuse letters. What would normally happen?

    They press "suspend" and the VM shuts down.

    If I would be reseller, I would be extremelly skeptical of disk encryption and qemu disable actions. Since middleman has massive risk to be terminated by vendor for repeated offences. Csam and all that shit

    No there is no risk at all,
    The middleman doesn't own 5 VPs, so LSW sees his 3 VPs got multiple complaints, meaning they will suspend all vps.

    The middleman has already done all the KYC requirements, and has alot of service with them which has no complain at all, so they know the reseller is legit, and they only suspend exactly that VPS for which the complaint came.

    Leaseweb sends us the abuse email.
    If we dont reply in time, they null-route the IP.

    But we always act within time; we suspend vps or send forward abuse complain, depending on the complain type.

  • zedzed Veteran

    @Levi said:

    @forest said:

    @Levi said:
    So, customer disables qemu agent, encrypt vps disk. Then provider gets some abuse letters. What would normally happen?

    They press "suspend" and the VM shuts down.

    If I would be reseller, I would be extremelly skeptical of disk encryption and qemu disable actions. Since middleman has massive risk to be terminated by vendor for repeated offences. Csam and all that shit

    quoted for future referral.

    Thanked by 2forest Decicus
  • LeviLevi Veteran

    @HostDZire said:

    @Levi said:

    @forest said:

    @Levi said:
    So, customer disables qemu agent, encrypt vps disk. Then provider gets some abuse letters. What would normally happen?

    They press "suspend" and the VM shuts down.

    If I would be reseller, I would be extremelly skeptical of disk encryption and qemu disable actions. Since middleman has massive risk to be terminated by vendor for repeated offences. Csam and all that shit

    No there is no risk at all,
    The middleman doesn't own 5 VPs, so LSW sees his 3 VPs got multiple complaints, meaning they will suspend all vps.

    The middleman has already done all the KYC requirements, and has alot of service with them which has no complain at all, so they know the reseller is legit, and they only suspend exactly that VPS for which the complaint came.

    Leaseweb sends us the abuse email.
    If we dont reply in time, they null-route the IP.

    But we always act within time; we suspend vps or send forward abuse complain, depending on the complain type.

    And what if lw will make reseller “the source of repeated infringement”?

  • Which provider is this? Could you please share the provider’s name? @Levi

  • HostDZireHostDZire Patron Provider, Veteran
    edited September 9

    @Levi said:

    @HostDZire said:

    @Levi said:

    @forest said:

    @Levi said:
    So, customer disables qemu agent, encrypt vps disk. Then provider gets some abuse letters. What would normally happen?

    They press "suspend" and the VM shuts down.

    If I would be reseller, I would be extremelly skeptical of disk encryption and qemu disable actions. Since middleman has massive risk to be terminated by vendor for repeated offences. Csam and all that shit

    No there is no risk at all,
    The middleman doesn't own 5 VPs, so LSW sees his 3 VPs got multiple complaints, meaning they will suspend all vps.

    The middleman has already done all the KYC requirements, and has alot of service with them which has no complain at all, so they know the reseller is legit, and they only suspend exactly that VPS for which the complaint came.

    Leaseweb sends us the abuse email.
    If we dont reply in time, they null-route the IP.

    But we always act within time; we suspend vps or send forward abuse complain, depending on the complain type.

    And what if lw will make reseller “the source of repeated infringement”?

    Well, it ultimately depends on how the provider handles abuse and illegal activity on its services.

    For example, if a provider repeatedly ignores DMCA or abuse complaints, knowingly sells services specifically for illegal activities, or takes no action against repeat offenders, then yes, the provider itself can eventually get into trouble.

    If a provider has 100 VPSs and more than 50 of them are repeatedly receiving abuse complaints, it becomes pretty obvious that either the service is being marketed toward abusive users or the provider is simply not taking proper action.

    In that situation, Leaseweb will first warn the reseller and if still ignored then they can ask that reseller to leave leaseweb.

    In our case, our relationship with Leaseweb is much broader than just VPS. VPS represents only a very small portion of our overall billing with them compared to the dedicated servers we own, colocation, and other services we use.

    We also have more than 10 years of history with Leaseweb. As long as we continue handling abuse reports properly and enforcing our policies, our position with them is quite strong.

    If you knowingly allow illegal activities on your services, then Leaseweb is not the only thing you should be worried about. Your own company could face investigation from law enforcement, your infrastructure could be seized, and you could personally end up in serious legal trouble.

  • HostDZireHostDZire Patron Provider, Veteran

    @gooku said:

    Which provider is this? Could you please share the provider’s name? @Levi

    Its leaseweb panel

    Thanked by 1gooku
  • pretty sure most resellers do get console/VNC access and can trigger reinstalls through the provider's control panel API, so yeah they could technically get in regardless of your root password. changing the root pw only stops someone from logging in over SSH, not someone with hypervisor access doing a console attach or snapshot.

  • itzsenuitzsenu Member
    edited September 9

    @JustLookingAround said: securing linux VPS.

    “There is no such thing as a secure system if it has a network connection and an attacker has physical access.”

    I use a clean Arch or ubuntu install from ISO if the provider allows it. I usually avoid provider templates unless I’ve checked what they actually contain. Check cloud init, users, SSH keys, repos, systemd services, cron jobs, agents and listening ports.

    For SSH, use keys only, disable root login and passwords, use a separate admin account, and restrict users with AllowUsers/AllowGroups. For something more serious, keep SSH on a WireGuard/private management network and use FIDO2 keys.

    Firewall both IPv4 and IPv6. Default deny and only open what you need. ss -lntup is useful for checking what is actually listening. Dont leave things like MySQL, Redis, Docker API or random admin panels exposed to the internet.

    LUKS2 is good for data at rest, but on a VPS you still have to deal with unlocking it at boot. You can enter the password through the provider console or use a remote-unlock setup. Once the VPS is running, disk encryption wont help much against a compromised system. For extra hardening, AppArmor/SELinux and systemd sandboxing can be pretty useful. Things like NoNewPrivileges, ProtectSystem, ProtectHome, PrivateTmp, RestrictAddressFamilies, CapabilityBoundingSet, SystemCallFilter, etc. can limit what a compromised service can do.

    QEMU Guest Agent is a bit different. It gives the hypervisor extra ways to interact with the guest. If you dont need it, disabling it removes some attack surface, but it doesnt protect you from the provider. They still control the hypervisor, virtual hardware and console.

    But theres a nice guide by Forest - https://lowendtalk.com/discussion/220023/neutralizing-the-security-risk-of-qemu-guest-agent-without-removing-it

    For VNC/web console, it depends on the provider. Dont think closing the browser means the Linux session is logged out. actually exit the shell and close the console session properly.

  • @HostDZire said: But since we are the owner of that VPS, we can ask Leaseweb to do certain things with it.

    I doubt they'd give you a memory dump if you asked though, would they? That kind of thing is typically only done during forensic investigations or when law enforcement sends a data preservation request.

    @JustLookingAround said:
    @luckypenguin @forest Can you please share basic steps about securing linux VPS.

    1. Install own ISO, how do you mount an iso in VPS, you download iso to VPS, then install from there?

    You can either have the panel download and mount it if supported, or just run this script as root (with VNC open):

    #!/bin/bash
    
    mkdir /netboot
    wget -P /netboot https://deb.debian.org/debian/dists/stable/main/installer-amd64/current/images/netboot/debian-installer/amd64/{linux,initrd.gz}
    cat << EOF >> /etc/grub.d/40_custom
    menuentry "Debian netboot" {
        insmod part_gpt
        insmod ext2
        search --no-floppy --file --set=root /netboot/linux
        linux /netboot/linux priority=low nomodeset --- quiet
        initrd /netboot/initrd.gz
    }
    EOF
    update-grub
    grub-reboot "Debian netboot"
    reboot
    

    If you don't have VNC, you can connect over SSH but that's a bit more complicated. With example values, you'd add something like this to the linux line in the above GRUB config:

    anna/choose_modules=network-console network-console/password=hunter2 network-console/password-again=hunter2 netcfg/choose-interface=auto netcfg/disable_autoconfig=true netcfg/get_ipaddress=203.0.113.87 netcfg/get_netmask=255.255.255.0 netcfg/get_gateway=203.0.113.1 netcfg/get_nameservers=8.8.8.8 netcfg/confirm_static=true
    

    You can also use https://netboot.xyz.

    1. Encryption. Whole disk encryption? Then how do I add encryption password at boot?

    FDE doesn't always include the boot partition, and that's fine. If you use the Debian installer in expert mode (not sure about normal mode), you'll have the option to select "guided partitioning with encryption". Make sure the password is very strong because it must be able to withstand offline brute force and dictionary attacks.

    Install the cryptsetup-initramfs package and read its documentation if you don't want to have to connect over VNC to supply the password every time you reboot.

    1. @luckypenguin you mentioned not leaving the console logged in. Doesn't VNC ask for password whenever you connect with console like RDP? How do I disconnect a VNC session securely? and you said NOT to use provider template, what did you mean by that?

    The VNC password is really weak (with noVNC you usually don't even see the password and it authenticates you automatically), but that only lets you access the virtual console, which has another password: Your user password. If that's strong (or if you have password-based login disabled), VNC can't do much.

    You can hide potentially sensitive information from the reseller by adding quiet loglevel=0 to your boot options. That way, they won't see certain system messages that would otherwise get displayed and would be visible over VNC.

    Be aware that this only protects you from a reseller trying to compromise your VPS without user interaction. They could still, if they really cared, mount a rescue system and modify your unencrypted boot partition (or even bootloader) to capture your encryption password when you enter it next time. It also provides no protection against anyone with root access to the node, in this case Leaseweb. It only protects against a reseller who is not going to tamper with your bootloader/boot partition.

  • @HostDZire said:

    @LEmeINalr said:

    @JustLookingAround said: The reason I'm concerned is I'll have customers data on it, so I'd rather not take the additional risk.

    If you're genuinely worried about that then I dont think you should be going for a LET host and instead go for Amazon or Oracle or whatnot. I personally dont keep any critical data on any of my LET hosts because I know they are offering me cheap services and are not billion dollar businesses that can protect it from every hack.

    Exactly. Buying a VPS at low-end prices and expecting defense-level security will never go together. It really depends on how much security your data actually requires.

    Most hosting providers use WHMCS, and these days critical security updates seem to be coming out like tea and coffee.

    Absolutely. If provider supports custom ISO then that is already a huge bonus. Beyond that you just need to make sure anyone that's not LEA is gonna have a hard time trying to get your data. Encrypt the disk and only open stuff you need is fine.

    And yeah, I saw that email about the whmcs updates from 3rd Sept being applied. Funny how it mentioned many vps users were still using the default generated password and their ssh was publically accessible. Such basic things to follow that can protect one's stuff better.

  • If someone's threat model is really of an extreme need then they can just opt to go with Xentt's new offering out there. :D :D

  • forestforest Member
    edited September 11

    @LEmeINalr said:
    If someone's threat model is really of an extreme need then they can just opt to go with Xentt's new offering out there. :D :D

    It's honestly not that extreme, considering we've had numerous providers who violated users' privacy and numerous providers who have gotten hacked. If you're talking about extreme, it'd be AMD SEV-SNP with FDE and remote attestation, not "encrypt sensitive stuff and turn off guest agent".

    Thanked by 1JustLookingAround
  • Oh I was not talking about OP's needs but just in general if someone wanted bulletproof hosting like running the next white rabbit network and other morally illegal stuff then someone like Xentt would be better.

    On another note, you're right about SEV-SNP. I hardly see that in the offer threads but will make a mental note to look for it.

  • forestforest Member
    edited September 11

    @LEmeINalr said: On another note, you're right about SEV-SNP. I hardly see that in the offer threads but will make a mental note to look for it.

    Afaik only @onidel and @Servitro @servury have SEV-SNP with remote attestation. I think Infofractal also has it on some nodes, but they don't support remote attestation yet.

    Thanked by 1LEmeINalr
  • I need to read up on how to use that then. Never used it before. Next weekend activity sorted.

  • @LEmeINalr said:
    I need to read up on how to use that then. Never used it before. Next weekend activity sorted.

    https://kb.onidel.com/hc/kb/articles/1771943583-sev_snp-verified-boot-and-memory-encryption

    Thanked by 1LEmeINalr
  • PacketraOliverPacketraOliver Member, Patron Provider

    A true and respectful host/reseller doesn't dig into its clients hosting account no-matter who they are without a court order, full stop, and even then they do as the court order says to the letter.

    Thanked by 1forest
  • @PacketraOliver said:
    A true and respectful host/reseller doesn't dig into its clients hosting account no-matter who they are without a court order, full stop, and even then they do as the court order says to the letter.

    And even then, it's best to fight the court order legally (where possible).

  • LeviLevi Veteran

    @PacketraOliver said:
    A true and respectful host/reseller doesn't dig into its clients hosting account no-matter who they are without a court order, full stop, and even then they do as the court order says to the letter.

    Snooping is the only pleasure in web hosting.

    Thanked by 1buggedout
  • @PacketraOliver said: A true and respectful host/reseller doesn't dig into its clients hosting account no-matter who they are

    That's a "trust me bro" approach, the bigger risk is not from the hoster/reseller, it's from
    an attacker that manages to compromise Virtualizor, for example.

    Thanked by 1JustLookingAround
  • @luckypenguin said:

    @PacketraOliver said: A true and respectful host/reseller doesn't dig into its clients hosting account no-matter who they are

    That's a "trust me bro" approach, the bigger risk is not from the hoster/reseller, it's from
    an attacker that manages to compromise Virtualizor, for example.

    That's true, but it's also true that a respectful host doesn't snoop, even though that is of course not a guarantee of security.

Sign In or Register to comment.