Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Erratic email usage on SES causing a high billing
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Erratic email usage on SES causing a high billing

LisoLiso Member

I have been a aws ses customer from December last year— I'm using it to handle transactional email for my webapp that serve a small community of people, as the app rarely send an email, ses has been perfect for my use case because of its generous pricing. My usage has never reached $1 a month (I see from the log that I sent max ~3000 email a month).

Just several days ago, I received my September billing for my SES usage, it listed that I have send 221k of email. Amazon has charged me $24.62 (including VAT) for my usage.

I just found out about this through the invoice email they sent to me, needless to say, I don't believe I have sent that many email— I have taken action to temporarily pause ses service and opened a support case, asking them which verified identities send the most email.

Currently I have only three verified identities, all of them is a domain level identity (not email).


Upon investigating, it appear that this strange behavior started in 15 September 2022.

I also noticed that my ses have high activities at fixed time (09:00 to 18:00), and the email sent is ~1300 per hour ?!

Currently still waiting for aws support to reply, meanwhile is there anything I can do to diagnose this issue further ? Also has anyone face the same problem ?

Thanks ! Sorry for my bad english.

Comments

  • MikeAMikeA Member, Patron Provider

    No way to log emails sending history from your web app? Amazon doesn't really provide useful info in a friendly or easy to display way.

  • @MikeA said:
    No way to log emails sending history from your web app? Amazon doesn't really provide useful info in a friendly or easy to display way.

    Have looked into it, but did not found anything unusual.

  • hostdarehostdare Member, Patron Provider

    possibly misuse of your api or your software setup got hacked?

    Thanked by 2lentro Falzo
  • Replace api keys and see if it makes difference, possibly they are compromised.

  • @hostdare said: possibly misuse of your api or your software setup got hacked?

    @nvme said: Replace api keys and see if it makes difference, possibly they are compromised.

    Will do this, thanks for the tip.

    Thanked by 1hostdare
  • Immediately enable 2FA on Amazon acc and replace API keys. You are lucky that damages was only ~30 USD. Usually crackers goes to the max as fast as they can.

  • Had similar issues on the past. Someone hacked our API keys and were able to send spam using our domains.

    Amazon support were really helpful tbh. They were able to provide us with logs including sender and recipient email addresses, timestamps, email title, etc.

  • LisoLiso Member
    edited October 2022

    Have generated new smtp key for my app, also deleted older one. So far no issues, I guess I'll keep an eye to SES usage. Amazon support team hasn't replied and my case remain "unassigned" :(

    @LTniger said:
    Immediately enable 2FA on Amazon acc and replace API keys. You are lucky that damages was only ~30 USD. Usually crackers goes to the max as fast as they can.

    Thanks, the damn thing immediately charge my credit card-- I have no way to dispute the transaction now.

    @sanvit said:
    Had similar issues on the past. Someone hacked our API keys and were able to send spam using our domains.

    Amazon support were really helpful tbh. They were able to provide us with logs including sender and recipient email addresses, timestamps, email title, etc.

    That's nice, hopefully mine will get resolved soon. Still waiting for their response.

  • @Liso said:
    Have generated new smtp key for my app, also deleted older one. So far no issues, I guess I'll keep an eye to SES usage. Amazon support team hasn't replied and my case remain "unassigned" :(

    @LTniger said:
    Immediately enable 2FA on Amazon acc and replace API keys. You are lucky that damages was only ~30 USD. Usually crackers goes to the max as fast as they can.

    Thanks, the damn thing immediately charge my credit card-- I have no way to dispute the transaction now.

    @sanvit said:
    Had similar issues on the past. Someone hacked our API keys and were able to send spam using our domains.

    Amazon support were really helpful tbh. They were able to provide us with logs including sender and recipient email addresses, timestamps, email title, etc.

    That's nice, hopefully mine will get resolved soon. Still waiting for their response.

    Even if they charged your CC, they will be able to revert those. One of our client's account had faced similar issue, and the ones who got access used it for mining shit coins on EC2. When we complained, they were generous enough to refund those charges.
    I think you had just SES access on your API keys?

  • MikePTMikePT Moderator, Patron Provider, Veteran

    You're lucky it wasn't much of a damage.

    Do try to check your logs, as best as you can.

    Thanked by 1hostdare
  • hostdarehostdare Member, Patron Provider

    I think you can also ask them to reduce the daily sending limit

  • tjntjn Member

    I'd also recommend https://github.com/Nikeev/sesdashboard which will allow you to track all of your outgoing mail from SES.

    Very easy to setup and isn't very heavy.

  • please keep informing us about your SES status, is it decreasing or increasing?

  • Your credentials or access leys might be leaked (eg. Your source code in git). One of the friend I knew had similar issue on sendgrid and receive the bill of a few thousand but sorted out in joint investigation.

    You are lucky just 24$. Rotate your keys and trace back.

  • LisoLiso Member
    edited October 2022

    @iqbal said:
    please keep informing us about your SES status, is it decreasing or increasing?

    Still getting high volume of email, take place at the same fixed time.

    it's a php (CI) based webapp, the credentials stored in CI models. Guess I should strengthen the security. Meanwhile aws support hasn't replied yet.

    @mrclown said: Your credentials or access leys might be leaked (eg. Your source code in git). One of the friend I knew had similar issue on sendgrid and receive the bill of a few thousand but sorted out in joint investigation.

    We also utilize git repository, but it is a private selfhosted gitea repository hosted on vps. It shouldn't be accessible to others right ?

    @tjn said: I'd also recommend https://github.com/Nikeev/sesdashboard which will allow you to track all of your outgoing mail from SES.

    Thanks, I will try it out.

  • your .git folder might be leaked, if your website folder listing is not disable

  • NeoonNeoon Community Contributor, Veteran
    edited October 2022

    @Liso said:

    @iqbal said:
    please keep informing us about your SES status, is it decreasing or increasing?

    Still getting high volume of email, take place at the same fixed time.

    it's a php (CI) based webapp, the credentials stored in CI models. Guess I should strengthen the security. Meanwhile aws support hasn't replied yet.

    @mrclown said: Your credentials or access leys might be leaked (eg. Your source code in git). One of the friend I knew had similar issue on sendgrid and receive the bill of a few thousand but sorted out in joint investigation.

    We also utilize git repository, but it is a private selfhosted gitea repository hosted on vps. It shouldn't be accessible to others right ?

    as @chat263net said, if .git is accessible, someone could pull your git repo including your access keys.
    Never store any of such in the source code, also block .git or . in general.

  • Good news, since I once again generate new smtp credentials— also have put wonderful advice from LET member here into practice. Now my email usage is normal again.

    @tjn said: I'd also recommend https://github.com/Nikeev/sesdashboard which will allow you to track all of your outgoing mail from SES.

    Have installed this and instantly loved it, this really make it easier to monitor my outgoing mail usage. Setup was quick and easy.

    Thanked by 1iqbal
Sign In or Register to comment.