Growing pains with amateurish customer service at startups ... x4b.net & crossbox.io

TimboJones

I will say, whenever Letsencrypt cert renewal fails for some random reason, IIRC, I've always had to rerun it with verbose to have any fucking clue what the actual problem was.

So I'm with op for problem 1 and provider for problem 2.

SplitIce

Sorry that we never responded to this thread, I've just been made aware of it care of a customer with AutoSSL issues (resolved, customer did not tick the "Enable AutoSSL" checkbox). I'm sure OP has moved on but in the interest of anyone who may find this thread here is the response from me (X4B).

I found the customers ticket. Read through the ticket, I'm pretty happy with the responses made by myself, considering the responses quoted are the worst the customer could find and are the kind of responses I would like to be on the receiving end of (provides next steps, provides indication of tests performed, well written) I struggle to fault myself here. Certainly for the limited time we have to allocate towards support I think they are very good (feedback is always welcome).

Its quite old now but fortunately there is extensive notes of what was done to troubleshoot the customers issue (mostly because being the first issue of its kind we needed to validate against library / system failure). But from that I'll cover a few key points.

1. Customer had issues with the initial generation of a AutoSSL (Lets Encrypt issued) certificate. Specifically the Lets Encrypt API call to convert an order into a certificate failed with an order rejected message. This rejection message was non-standard for a failure to resolve or HTTP related error. My guess was that either:

a) When queried by LE responded to an older location (or perhaps geodns etc) which served its own ACME challenge response (invalid) and hence LE rejected the order; or
b) the customers domain was on a LE blacklist

2. For troubleshooting purposes we provide to all customers the full error text provided by Lets Encrypt, this is the error as returned by them in response to the API call. Or in the event of internal errors the exception (e.g failure to communicate with LE).

The only item on our roadmap here is to offer advice (think like our error log system) on ratelimit related errors.

If anyone would like to suggest something extra that we can provide, I'm all ears. As far as I am aware this is all we have.

The process to generate a certificate from LE is the following:
1. Upload a challenge file to the edge servers (simplifying here, we don't actually do this)
2. Issue a couple API requests to LE
3. LE issues a HTTP request to your domain on port 80 retreiving this challenge file
4. We ask LE for the result, or an error if one occurs.
5. We distribute the certificate to all edge servers (if fails will be retried)

We use AcmePHP as our LE client, and custom web server integration and deployment systems. If an error occurs, we report it. An error can not occurs at any step its reported by us (and the process stops). To the best of our knowledge its incredibly reliable. There are currently 52k active certificates being managed by the system.

Asside from some documentation on some of the LE error codes we havent needed much else, is there something else we should have done here?

3. The LE generation and renewal system has been incredibly reliable since its upgrade in 2020, I was able to find this customers ticket due to it being one of 3 tickets since 2020 mentioning problems with it. We use it ourself for https://www.x4b.net and its held up nicely. The system we designed is far better than our original implementation in that:
   • It uses a LE account per customer, this way customers get emails before expiration (e.g on repeated renewal failure) and the rate limit boost from being allocated their own account
   • It uses multiple IP addresses for renewal / generation providing a huge rate limit increase
   • Certificates are renewed and generated on a reliable and clustered system
   • Certificate generation and renewal is directly integrated into the panel (domain in use, feedback on failures, renewal times and similar all being available)
   • Regex provided for filtering for allowed domains
   • It uses the system database for storage instead of a seperate file store
   • We have controls capable of doing revocation and re-issuance in the event of a major issue requiring bulk action
   • Being built into our main database it integrates with our standard backup procedures
   • Users can request activate and de-activate certificate renewal for a certificate (this also functions to enable re-activation of failed certificates)

All things that our previous standalone system was not (and I believe it to be incredibly reliable as a result).

4. I wouldnt mind writing a bit more documentation on AutoSSL and LE in general. We have a fair bit, but there is always room for improvement. Its just never been a priority due to a low number of tickets.

I will shedule a review of the documentation and if needed add some (small) additional documentation.

5. My regular reminder is always as a customer remember that the person on the other end is a human trying to help you. Remember to let them, expecially if you are dealing with senior support (the only type we provide). Remember an unmanaged service does not manage your product or service. We must try to be enablers of you, not do your job. If your DNS service needs changes we can't be doing that, but perhaps if you provided a screenshot of it or asked a relevant question you might get a relevant answer. Of course if its "not the problem" then who are we to say what the problem actually is.