When I get spam email, I usually check the headers and if it's coming from a reputable service (Postmark, Sendgrid, etc.) they usually have a web form or an abuse@ email to send the headers to so that they can shut down the account.
Months ago I received spam from a Mailgun server and tried to use their web form[1] to report it, but it was broken. I reported both that bug and the spam email to their support, which acknowledged it. Weeks later I got another spam email from that same domain, popped open that report form and it was still broken (FWIW as of today it seems to be working again). So I followed up on my initial support request with that info but got no response. Just a few days ago I received another spam message from that domain.
I personally consider all that a very bad sign in an email service provider and wouldn't use Mailgun myself. In contrast, I've been very happy with Postmark.
Postmark's service is great but their new min $10/month pricing scheme is a retrograde step and penalises small companies sending less than 1000 emails a month.
Deeply unhappy with the change, and wish more companies would follow the Amazon AWS pricing model.
I didn't realize the new monthly plans were required for new accounts - that's unfortunate. I'm still on the old credits system (grandfathered I guess).
Do they have an overage charge for the free 100/month, like the 1.25/1000 they list for their non-free use? So you could be free most of the time with the occasional 1.25 charge if you have a busy month?
I don't believe this would even be an issue if they offered the option to not log sensitive data. I had requested that they provide something like this and someone quite senior reached out to me. He was very polite and professional. He explained that they had to keep this data for operational and compliance reasons and that all email providers are required to. However, that didn't resolve my security concern.
We ended up going with Mandrill which does offer the option to not log sensitive data ^1. Whether they log it somewhere else for the compliance reasons that Mailgun mentioned isn't mentioned anywhere in their docs or privacy policy, but doesn't seem to be accessible from everything I could find. You should never log or allow others to log password reset urls or other sensitive details.
This needs to be the #1 comment in the thread. If you use a transactional mailer, make sure you are not archiving emails with security-sensitive content.
That includes resets, username reminders, signin notifications, etc.
Also secure access to your transactional mailer account with 2FA and restrict access to those who need to be there (i.e. not your entire support team).
It's the opposite. That is a link to the section of the Mandrill docs, not the Mailgun docs. The view_content_link option fixes the security problem. (In theory, anyway).
Right. I understand that having that option lets you mitigate some problem. Can anyone expand on what this option does and how it mitigates the problem? Did I miss something from the blog post?
Sure -- AFAIK the problem was that Mailchimp was hacked, and the hacker was able to see and intercept the password reset links being sent to the customer by looking at Mailchimp log data. This option indicates that links should not be stored in log data, so even if an attacker has compromised your Mandrill account, they should be unable to see the exact reset links that are being sent.
edit: worth noting that there are obviously other ways a hacked Mandrill/Mailchimp account could be abused. This just shuts down one of the major abuses you could perform.
On 12/31, Reddit received several reports regarding password reset emails that were initiated and completed without the account owners’ requests.
We have been working to investigate the issue and coordinating with Mailgun, a third-party vendor we’ve been using to send some of our account emails including password reset emails. A malicious actor targeted Mailgun and gained access to Reddit’s password reset emails. The nature of the exploit meant that an unauthorized person was able to access the contents of the reset email. This individual did not have access to either Reddit’s systems or to a redditor’s email account.
As an immediate precautionary measure, we moved reset emails to an in-house mail server soon after we determined reset links were indeed being clicked without access to the user's email, and before Mailgun had confirmed to us that they were vulnerable. We know this is frustrating as a user, and we have put additional controls in place to help make sure it doesn’t happen again.
We are continuing to work with Mailgun to make sure we have identified all impacted accounts. At this time, the overall number of confirmed impacted users is less than twenty. For those affected, we have resolved the issue and assisted in account recovery.
Additional information about Mailgun’s security incident can be found on its blog here. We’re committed to keeping your Reddit account safe and will continue to monitor this situation carefully. u/sodypop, u/KeyserSosa, and I will be sitting around in the comments for any general questions.
This is a new class of attack. Instead of spear-phishing, it's spear-hacking.
It looks like the target was "bitcoin-ish tipped into /u/someredditor" and the hack/vuln was "intercept mail password resets in order to auth account in order to snatch crypto-currency"
ie: most people's reddit accounts (IMHO) are on the "not that important" on the scale of password protection. (Personal Email/Financial => Work => Medium Security [facebook, amazon, etc] => Low Security [discussion forums])
It's another way of saying that I would expect phpBB or reddit or pinterest to have lower password/server security than my gmail or bank websites.
However, because reddit is relatively high profile, and there was mixing of "cash and reddit", all of a sudden not just reddit was target of a hacking attack, but also reddit's 3rd party service providers.
I can choose to use reddit or not, but I can't choose that reddit uses or doesn't use some other random service provider that may or may not be vulnerable.
"I can choose to use reddit or not, but I can't choose that reddit uses or doesn't use some other random service provider that may or may not be vulnerable."
Which is similar to the same problem we all face of 'I can choose to work for company x' but I cant choose that they farm out background checks, HR, payroll, benefits etc. to random companies that may or may not be secure.
Many services state in the password reset emails that "if this was not initiated by you, ignore it", but it really should be the exact opposite - click the link below to report it!
Why would employees need access to client API keys, as opposed to just client ID?
Furthermore, this seems to indicate that the API keys are not hashed. I would expect some bits of the API key to work as an identifier and the rest of the bits treated as secret material (properly hashed).
As a former Rackspace employee, I had access to every customer secret IN PLAIN TEXT through multiple web-based systems with a click of a button (IE: business as usual).
Thanks for confirming this. I had my suspicions, especially after the last few years of using them and just seeing massive problems that seemed to be caused by the software at Rackspace.
1. How was the employee's account accessed? No 2FA?
2. Do employees ordinarily have access to customer secrets (e.g. API keys) or was there some further exploit?
3. The advice in OP for affected customers is to roll keys and SMTP logins. Couldn't/shouldn't you do that for them? Surely security should trump up-time/deliverability?
For 3. I'd say no way. There's no way for Mailgun to know what services are doing with those keys, how important those services are to their customers, how difficult it is for the service owners to rotate their keys, and how much bandwidth they have to do that right now.
In an ideal world, every customer would have a good setup where they can rotate third-party supplier API keys painlessly and have plenty of bandwidth to handle security emergencies. Alas, there's a lot of bad setups out there, and some of them are critical to their customers' operations.
Nothing I've personally worked with had a setup bad enough to make that painful, but I'd be very worried about how reckless a service is to rotate API keys that aren't being actively exploited to do something dangerous without getting a positive confirmation from the customer.
Does this only affect Mailgun's customers? If these customers hold data of third-party – let's call them "end-users" – in Mailgun accounts, Mailgun could/should communicate the total number of individuals affected. "1% of our customers/users" can affect millions of individuals.
In those security disclosures, I often read what I see as contradictory language.
For example, I'm confused by this kind of statement:
> Mailgun has now completed its diagnostic of accounts that were affected and has notified each of the affected users. At this time, we believe less than 1% of our customer base was potentially affected. If you were not directly notified by Mailgun regarding this incident, then your account was not affected.
If you believe that less than 1% of users were affected, it means you don't know for sure how many accounts were affected.
From there, how can you state that "If you were not directly notified by Mailgun regarding this incident, then your account was not affected"?
Doesn't this last statement mean you know for sure my account was not affected? Isn't it in direct contradiction with the previous statement?
Foremost, it was written by a human and unintended language contradictions are common. With that said, what you're suggesting isn't necessarily true -- the language can also indicate potential false positives, again because of the nuances of language.
Yes, definitely true. Although some contexts, like a security disclosure, might warrant a very carefully non-contradictory worded statement that leaves no doubts of interpretation.
> the language can also indicate potential false positives, again because of the nuances of language.
Yes, but in this context, false-positive aren't important to the audience of the disclosure. Nobody really cares if their account was "identified as affected, but in the end wasn't".
If you announce that 1% of your user base was affected, and it turns out that 50% of this 1% were false-positive, great! You were still right in announcing that 1% of your user base was affected. You can always correct this later and announce that things panned out better and only 0.5% of your users were impacted.
I like Mailgun so much because of its simplicity but last November 2017 the default postmaster account of one of our domain in Mailgun was hacked. (I don't know where it was hacked but i suspect it was on the Mailgun server because I kept the secret key in my server very well). We moved to Sendgrid because my account in Mailgun got a very bad reputation. One of the hacked smtp credentials was used to send spam.
> At this time, we believe less than 1% of our customer base was potentially affected. If you were not directly notified by Mailgun regarding this incident, then your account was not affected.
Former mailgun customer. Asked them to delete my personal data a couple of weeks ago (I was not able to do it myself... ) because I would rather they don't leak it in a security hiccup. They kindly refused to do so (as I don't believe any tech support can be that incompetent) and kept spamming my inbox instead. While the severity of this incident is not clear, never imagined curses can act on such a short notice.
This is because Mailgun is in the practice of spam. The number of spam campaigns I've seen with Mailgun as the conduit is high, second only to Mailchimp.
This is Chris from the Mailgun team. I'm sorry that this happened, this shouldn't have been the case. I'd be happy to help rectify this issue, would you be able to send an email to help@mailgun.com with details so I can review?
Months ago I received spam from a Mailgun server and tried to use their web form[1] to report it, but it was broken. I reported both that bug and the spam email to their support, which acknowledged it. Weeks later I got another spam email from that same domain, popped open that report form and it was still broken (FWIW as of today it seems to be working again). So I followed up on my initial support request with that info but got no response. Just a few days ago I received another spam message from that domain.
I personally consider all that a very bad sign in an email service provider and wouldn't use Mailgun myself. In contrast, I've been very happy with Postmark.
[1]: https://www.mailgun.com/receiving-spam-from-mailgun