Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
Information about a recent security incident at Mailchimp (mailchimp.com)
106 points by guiambros on Jan 21, 2023 | hide | past | favorite | 62 comments


Funnily enough I’m studying a cyber security course at the moment and had to find examples of social engineering for an activity, and came across this yesterday. I shared it with my classmates and they just couldn’t believe that MailChimp been hit 3 times in a year with what sounds like the same social engineering attack.

It does make you wonder when there’s going to be a 4th, I certainly don’t think I’d want my customers contact details in their system at the moment.


This problem is almost certainly Mailchimp-specific. The talent was decimated after the owners sold it to Intuit.


> This problem is almost certainly Mailchimp-specific.

It's not. Multiple marketing automation/email service provider platforms were affected by similar hacks last year.


Mailchimp was gutted.

You have to understand the sort of mythical place the company had in the nascent Atlanta tech scene. It was one of the first Atlanta tech unicorns and was growing and hiring at a rapid clip. The founders promised never to sell and that they would give annual bonuses in exchange for not offering employee equity. The office was brimming with art, music, and perks that were unusual for Atlanta at the time. They said they were going to build a generational Atlanta tech company.

I desperately tried to recruit my friends that worked at Mailchimp away, because I saw the writing on the wall. I had joined a promising startup offering significant equity, and more importantly, building real value in the world. To my disappointment, none of my Mailchimp friends joined me. They liked the culture too much, felt comfortable, and continued to tout the bonuses they were receiving despite Mailchimp offering lower total comp. I never stopped trying to pull them away, but I didn't succeed. I received counter offers and heard the Kool Aid stories. "The founders are so cool and so generous."

The job itself isn't that interesting, either. Marketing and email advertising isn't intellectually attractive. To top it off, Mailchimp has a disgusting PHP monolith that I've heard is like pulling teeth to work with.

I was right in the end. I walked away with a huge eight figure exit. Mailchimp employees got pay cuts, benefit cuts, and saw their culture ravaged. Intuit took what little niceties they had and threw it all to the wind. Macy's and Home Depot have better software jobs.

I really don't like Mailchimp. The company is rotten.


> said they were going to build a generational Atlanta tech company.

And you thought a company whose raison d'être was email marketing was going to be a “generational company”?

I worked in Atlanta until late last year.


Not my words. I've always disliked Mailchimp and their pretentious self-importance.


You worked at Autotrader?


No, I'm not familiar with them.


Once is a hack. Three times in a year is a systemic business problem.


That's the thing you can't outsource IT security because in order to be secure you need to have institutional understanding.


But if your devs are outsourced...?


The sad part is it's one of the hardest components to solve. High quality training can instill a healthy dose of paranoia, that is necessary, though.


It is. And then you receive mails from say Google or some other large company that look exactly as though they are very bad forgeries of what you'd expect. 'Action required', some weird vaguely related domain name, a bunch of links to click that definitely have nothing to do with the matter at hand.

With crap like that it isn't all that strange that some of these attempts get through and on the social engineering side it can't be all that much better. After all we're continuously conditioned to trust that which we really shouldn't be trusting. Right up to the point where the party on the other side is abusing that trust.

One example: my real bank calling me and then asking me to verify that it is really me by asking me all kinds of information about my account... And from a number that isn't associated with them. And yet, it was very real and when I refused to answer their questions they took it pretty badly as though I was a scammer...


I'm convinced there's no way to solve this except for regular "red shirt drills" linked to your job performance. (This is a term I borrowed from life-guarding where someone randomly pretends to drown to see if/how fast the guard will catch it and assess the skills of their response.)


> a term I borrowed from life-guarding where someone randomly pretends to drown to see if/how fast the guard will catch it and assess the skills of their response

They have a similar concept in cybersecurity.

The Red Team

> a group that plays the role of an enemy or competitor to provide security feedback from that perspective. Red teams are used in many fields, especially in cybersecurity, airport security, law enforcement, the military and intelligence agencies.

https://en.wikipedia.org/wiki/Red_team


Good detection and response is a must, but creating a security culture helps greatly. Training, test campaigns, and further training for those that fail, does help. Nothing is a silver bullet, that is why we emphasize "Defense in Depth."


Even that doesn't work - some companies do exactly that with regular phishing exercises etc.

Employees get really good at spotting the tests.


The technical ones may, but they're also less likely to fall victim to the phishing schemes.

It's those non-technical folks in admin roles that you tend to have to worry about most. Those folks aren't so good at picking out the tests.

I actually found KnowBe4 to work fairly well on a large scale.


The non technical staff seem to learn to spot the knowbe4 shit real quick lol.


Yep. All of KB4s stuff looks like it came from a single designer. You need to go through your spam filter and make custom templates from whatever you are receiving, and also make lookalikes from real business messages. Bankers doing wire transfers to wrong accounts because of a forged document has been a serious problem in my sphere lately.

It used to be that if a mailbox got compromised they would just send spam about lottery wins and boner pills. Now they watch your messages and reply to a real request with a good looking response. A correct expected reply in a chain from an authentic account, just some numbers have changed. Then they will steal your contacts and register a similar domain and try to impersonate you.


We had almost this exact scenario (look alike domain) play out with a customer. Their accounts payable department almost paid out half a million to a scammer. Fortunately the employee at the customer accidentally replied to our actual email address and our folks knew better and picked up the phone. The customer insisted up and down that their email system had not been compromised. It took telling their IT folks what and where to look before they finally realized they were compromised. Good IT/security teams make all the difference.


> A correct expected reply in a chain from an authentic account, just some numbers have changed.

Ooof, that'd be a tough one not to fall for.


It's all in how much effort you put into it. It's quite versatile and their customer success team is phenomenal. We had multiple pretty smart employees accidentally fall for it the first few campaigns and immediately reach out when they realized what they did.


If being able to spot the tests doesn't confer ability to spot real phishers, then the tests aren't very good.


> Employees get really good at spotting the tests.

Then surely either the test isn't good enough, or the problem is solved?


Again? This is the third incident in ~12 months IIRC.

Is every other company hiding their incidents, or are MC developing a habit of actually not "taking the security of our users' data seriously"?


The problem that people are overlooking is the profit motives of the people capable of penetrating systems. These motives are dictated by how companies treat security. By:

- Feet dragging on free work or extremely low paid work - Not hiring any competent person or not paying enough to attract that person. - Refusing to fix internal problems that create the issues.

The reality is, there are so many more people willing to use technology to scam you than there are people employed to stop those people. Probably because these currently bad people are, by design, hard to discover and thus go unaccounted for entirely.

Nobody pays for the skillset and what people hire for clearly doesn't work. Until companies see consequences, nothing will change. So expect this not to change.

Why on earth are you giving these companies your money and data anyway? It's truly not difficult to avoid breaches. Do a tiny bit of digging into the security model before you unload your list of customers into their servers. Take some bloody responsibility yourselves too.


The third case is very common.

I would not even use the word "refuse" Sometimes the legacy system is so legacy that fixing it is a many years project. Sometimes you have such a messy environment that rotating credentials means a general crash. Sometimes you have legacy software that must stay legacy and you cannot patch + that software is so ingrained into your system that you cannot isolate it. Sometimes ...

This is all bad design and bad architecture from scratch. Or "good architecture 25 years ago".


What exactly does Mailchimp publish about their security model that would allow anyone to predict this breach?


You should never trust the marketing department to accurately convey any security model and should assume anything published regarding security has been approved by marketing.

Avoid companies that require data which isn't required. Overlook companies with previous vulnerability disclosures that leave you facepalming. Ignore companies that pre check marketing anything. Blacklist the companies with GDPR ignorant cookie "consent." Black hole companies that kill the planet to bring you advertising while telling you you're killing the planet.

I would make a joke about what remains. It wouldn't be funny. There are still lots of companies left however and those are the ones that gave at least a signal about caring about you.

And for the love of all things. If your company is forcing employees to sign up for garbage data farm software, SAY SOMETHING. Your data is important.


>Blacklist the companies with GDPR ignorant cookie "consent."

Could you explain further?


As purely an outsider and not OP: it seems their contractors and support people have direct access to critical customer data simply by typing password and/or totp into a form. Seems it’s not always the correct one.

* Limiting access (maybe to email subject instead of all content?) might prevent some fallout as it might be more difficult to extract password reset emails sent by customers.

* Limiting access from certain IP sources might make it more difficult to use captured login credentials.

* Hardware key based authentication might prevent the type of phishing that seems to have happened here.


* Allowing customer to only allow MC staff access for fixed intervals when support tickets are raised.


What I've found is it isn't companies won't pay for good people, they very much will, as a sec engineer you can make 200k pretty easily with just a few years of experience in a lot of places.

The problem is once these people are hired they come in and aren't given power to do anything, or the "drive for security" isn't actually present in the organization once people realize it might actually force people to focus on things other than pushing features as fast as you can.

This issue is further exacerbated by the fact that there is not an insignificant portion of info sec guys that believe they have to come in and save the organization from themselves and that they are the heroic white knight valiantly protecting the company from the unwashed masses of wild wild west cowboy developers and incompetent sys admins.


I once worked with one of those heroic security guys before. He setup the firewall so that the public website was only accessible from a white list of known up addresses. New users would need to submit their IP address to him in person before they’d be allowed to browse the site.

He insisted that this was industry best practice and it took two weeks before the site was online again.


At that point just take it offline entirely. For security of course.


Well, considering how absurdly expensive they are compared to other ESPs, the money has to be going somewhere?


They’re expensive for their feature set but cheap compared to most good ESP’s.


Might it be that Mailchimp has Internet facing customer service facilities?

Our lot doesn't, if you got and admin user and password that, in itself, doesn't get you very far.


I find it hard to believe it was just 133 accounts. I just got an email saying mine was one of the accounts compromised, even though I haven't used Mailchimp in over 12 years, and don't remember having ever used them for anything material.

From what I remember either I deleted the account, or they closed for inactivity, so it's weird they still had my data.


Unless they expressly commit to deleting it, or you were paying per GB for storage, consider it saved forever.


True, although I think the OP was highlighting the oddness of such a minor account being one of those compromised.


I failed to convince them to undelete my account (the deadline is currently 2 years), but there is an HN support incident: https://news.ycombinator.com/item?id=18715866#18717279

Back in the day I thought "who better to retain a mailing list" but turns out I'd have been best off with a Google Sheet. Now I have to scrounge around in backups to see if I downloaded it anywhere... maybe I should start a support ticket here of my own!


> I failed to convince them to undelete my account (the deadline is currently 2 years), but there is an HN support incident: https://news.ycombinator.com/item?id=18715866#18717279.

A lot of companies have a habit of keeping data but not migrating dead customers to newer schemas and systems.


GDPR to the rescue! /s


> we take the security of our users’ data seriously

Clearly they don’t if staff are able to just access customer accounts.

SaaS companies really need to stop giving themselves access to customer accounts.


Do MailChimp employees not need a Yubikey or some other secure OTP generator to access customer data?

Or does the attack involve the user sending their OTP as well?

Either this is a pretty heavy handed attack, or MailChimp aren't following best practices in terms of security.


For a corporate setting where you can replace keys easily, I would choose Fido keys and SSO protecting all access into the network instead of OTP/VPN as that is generally immune to social engineering attacks / leaked passwords. Additionally, it’s trivial to manage those keys in a corporate setting centrally.

For personal use, I still prefer OTP because I’m not fully comfortable that I won’t lose some key and forget to revoke it everywhere and/or lose permanent access. Phones are getting very good at being that security key with backup but I’m a little hesitant to do that because I’m not sure of the interop story.


while most implementations don’t require 2 security keys, most services highly suggest that you enroll 2 keys, and that you keep one in a safe place for backup.

e.g. Apple’s new icloud Security Keys support requires 2 keys to enroll


The likelihood of me keeping track of both well isn’t great. And then of I lose one, how do I go about revoking that from all the services? Not to mention that enrolling 2 keys requires you to have both keys every time you enroll. So I’m not sure how practical that is and how you know when you’ll need both keys. Recovery codes seem like a better option but then you’re stuck figuring out how to keep them around securely… OTP + diligence against social engineering is so far the only thing I’ve found that I trust. It’s not the best but multiple Yubikey’s sounds wrong and impractical as well.


You just ask the target to install teamviewer or anydesk…

These are probably support staffers working from personal computers at home using some sort of remote desktop software, hard to enforce any security policies.


This is one of the places where ChromeOS is an interesting option. If the employee literally can't install software, it's very hard to get owned by malware.


What are savvy folks using instead of Mailchimp nowadays? I run a B2B news site and use Mailchimp for its daily and weekly newsletters. I like that MC handles the subscriber management and deliverability side of things, but they seem to be moving away from doing email well and towards random value added small biz marketing stuff. And my monthly bill has almost doubled since the Intuit acquisition.


2022 Ask HN: What service are you using to send marketing emails? https://news.ycombinator.com/item?id=32535397

https://hn.algolia.com/?query=mailchimp%20comments%3E10

2021 emailoctopus postmark mailersend|lite sendgrid campaign-monitor constant-contact https://news.ycombinator.com/item?id=28516219&p=2#28516522

2016 (self-hosted): mailtrain dada-mail-project+ses mailwizz! maybe:sendy | mailgun sparkpost https://news.ycombinator.com/item?id=11424189

2017 ses sendy mailblast mailgun postmark mailtrain mail-for-good kevy mailget https://news.ycombinator.com/item?id=15493127 (+per-recipient customization tools)


Aside from the above, Audienceful is a new company in the emailing space as well: https://www.audienceful.com


Does the scanning software in the update have any opinions about installations of LibreOffice? Asking because Microsoft really has a strong opinion about not using Microsoft Edge, whose shortcut icon they keep resurrecting from the dead to haunt my desktop. Twice just the past 10 days.


Off topic: what do people think of MailChimps quirky web design.

Like it or not?


Jarring, overwhelming, everything is too big, hard to process quickly. Not a fan.


While they don’t specify it sounds like they don’t even require 2FA to access their systems?


I just don't understand how this keeps happening?


I don’t like Mailchimp, they won’t allow any emails that even seem to mention anything crypto-related. Super annoying.


>> we take the security of our users’ data seriously

I take the war in Ukraine very seriously, and a few other things, but I'm still going on the piss tonight.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: