I live in China for a long time. Last night wife and I up waiting for lock to tick over to midnight. The shopping cart was loaded up with about 30 items and just waiting to hit the go button so that the 11/11 special pricing would kick in. And then kaboom....tried placing the orders and servers couldnt handle it- lots of comms errors, weird errors that didnt even make sense in Chinese, but at least the cart contents never got lost or mangled.
I would like to know how many people were doing the same but a conservative guess would be over 100m concurrent users. We had to retry about a dozen times, including pruning the cart because limited stock of certain items was already gone in the first few seconds. Eventually got the order thru about 00:10am with 1/3 of items bought successfully. Now i know for next 3-5 days the courier guys will be heaving huge loads at every city apartment block. And in case you are wondering the volume of online sales on normal days is so high that courier companies have dedicated teams of delivery guys servicing each group of a few apartment buildings. I live in an estate with 4 buildings of 32 floors and here there are 4-6 from one company alone....and there are multiple companies. As someone who works in ERP software this is logistics on steroids.
Any...just sharing a story from the coal face from a HN fanboy.
Some background info: I'm tech lead for a german shop on tmall.hk, and indeed this time, the numbers are crazy.
What you have to know though is that B2C shopping on Alibaba is extremely promotion driven. You have a very flat curve most of the time, and then on promotion day, you've got lots of sales. Everyone is waiting for the special deals.
The way this works is that people can buy or sign up for all kinds of different vouchers, rebates and coupons and whatnot, which they can only use during the promotion, which is typically 24h. This will of course affect your margins, but the volume is high, so it can be worth it.
On the technical side, the APIs are astonishingly bad. I suppose this is the result of building this very quickly, and it has grown ever since. I don't know how they manage to maintain it.
> On the technical side, the APIs are astonishingly bad. I suppose this is the result of building this very quickly, and it has grown ever since. I don't know how they manage to maintain it.
I visited Alibaba HQ last month. People seemed pretty worked and driven by deadlines which can decrease their code quality. The teams I talked to didn't do much testing.
That figures. China (Bejing) is 7 hours ahead of german time, yet I've noticed that my Tmall contacts are replying to messages when it is really late at night for them, sometimes excusing themselves that they have dinner right now. On 11.11 many are on duty for 24 hours.
Off topic, but in contrast to americans, german people are often frowning and complaining a lot - even though we really don't have much cause, in comparison. Friendliness and positivity are some of the things I really appreciate about the USA. Treasure it!
Exactly, sorry - should have explained this. While aliexpress etc. are similar to Amazon marketplace, the tmall.hk site is geared towards trusted, foreign brands and big shops. You can setup a very distinctively looking shop, maintain your brand and standing out completely from the surrounding hosting site.
You are not allowed to open a tmall.hk shop unless you are a globally recognized brand or a shop that makes at least xx million € of revenue a year, its more or less invite only. Its a premium platform.
The reason Alibaba did this is that chinese people don't trust their own brands and shops, so for certain goods where safety is important and luxury goods they flock to foreign brands.
Supposedly a big chunk of Alibaba's backend infrastructure runs on its custom-developed distributed datastore called Oceanbase, which is available on Github here[1] and here[2].
There are some presentations[3] and papers online (see links below), but I haven't found any information on people other than Alibaba running this software.
Would love to learn more about this if anyone has experience with it / more technical detail.
Here are some academic links describing Oceanbase. I would love to see some of these, if anyone has access; I haven't been able to download the academic papers. (Need Google translate, I imagine.)
The article is a little dated though, and some of the links to their open source repositories don't work anymore (maybe this is because is because they moved to GitHub [2]).
I assume it's the language barrier, but for such a huge company I could find surprisingly little information on their technology or infrastructure. If anyone has any more current information, I'm sure quite a few of us would be interested to hear about running software at that scale.
I work at a young company providing search as a service for multiple medium/big e-commerce sites (mostly those which don't have their own search teams) and 11/11 is indeed a crazy day. It's funny how till 2 years ago, Cyber Monday was the big event we looked at and we noticed major traffic spikes last year on veteran's day (11/11) only to later realize that this was in fact singles' day!
This year we were geared up and it's been a crazy rise in traffic as expected again! :)
Temperature Controlled Outlet 33.3 RMB https://detail.tmall.com/item.htm?spm=a230r.1.14.13.t80P3i&id=536295296729&cm_id=140105335569ed55e27b&abbucket=4&skuId=3200881016743
Water Heater 19.9 RMB https://detail.tmall.com/item.htm?spm=a230r.1.14.1.MjQ88E&id=527717128601&cm_id=140105335569ed55e27b&abbucket=4&skuId=3219050495205
Aquarium Wave Maker 15.9 RMB https://item.taobao.com/item.htm?spm=a230r.1.14.47.Wgm0TD&id=533814521124&ns=1&abbucket=4#detail
50 L Plastic Box 35.0 RMB https://detail.tmall.com/item.htm?spm=a230r.1.14.13.haJnJy&id=530500415822&cm_id=140105335569ed55e27b&abbucket=4&sku_properties=21433:648878820
Which, if i manage to get it working, makes a sous vide circulator (i hope). Also have a small hydroponics setup in an orgfile somewhere, which comes to about 150RMB.
YES!! You can buy just an egregious amount of (crappy) components for very little money. As a wannabe-inventor that burns through a lot of components, this is really one of my favorite things about it.
It's great for amateur electronics projects. Need 100 22AWG five-strand connectors and you're not in a rush? Hit up AliExpress, and they'll show up on your doorstep for less than the price of a frozen pizza in a few weeks.
I guess if they run a 0.001% risk of doing something disastrous then it doesn't cut it. If 0.001% of purchases fail or orders get lost, or they have a few minutes of complete downtime, it might not be worth adding another 9.
It says something interesting about how think about "high reliability." What it really means is "must never, ever fail" which kind of makes "% reliability" a sort of strange euphemism, I think.
High reliability means that it is much more likely that something else fails first. We could all be wiped out by a gamma ray burst for example, it makes little sense to make something last longer in expectation than the expected time between two major extinction events.
.001 of a years worth of minutes is 525 minutes (.001 * 525600) or about 8 hours and 40 minutes. If all of that time just so happened to occur on the $1B day then that's 356 million. Of course this is back of the napkin. That $1B is not evenly distributed throughout the day and especially in high availability systems the .001 downtime is many small outages over the entire infrastructure and not the entire apparatus crashing all at once. Though because load can be a factor in downtime, they are not completely without correlation.
On mobile but didn't you leave off the percentage sign? (which is another two zeros after the dot)
Of course this is taking op very literally
EDIT: I checked: google "0.001% * 1 year" and it tells you 5.259 minutes, so you were off due to dropping the % sign. That changes things dramatically: beyond the 100x difference, there is very high likelihood that someone would check back 2 minutes later (especially if they are waiting all year), so 5 minutes of downtime (spread over a whole day) likely objectively results in much lower lost sales than your calculation implied! For future envelope calculations don't forget that a % sign is exactly moving the decimal two more places! 55.2% is exactly .552 and so forth!
No, my point is that 99.999% uptime has to be scoped to the same timeline as your window that you're measuring. If you're focusing on availability for a single day, then achieving 99.999% availability for that day means less than a second of downtime.
I don't even monitor for sub-second downtime... A sub 500ms glitch would very likely go unnoticed on most of my production platforms. @$1B/5mins, that could represent ~$1.5million in (potentially) lost sales...
> and a weakening yuan may curb enthusiasm for foreign wares.
I don't get it. Aren't most brands these days made in China or south east Asia? (looking at you, fashion brands)
Now, I don't speak Chinese so I stick to AliExpress, but you don't typically find non-Chinese made products for sale there (at least in the realm of clothing or electronics)
Many things are made in what China calls "Special Export Zones" (SEZs). This is for goods that require lots of imported machinery and components to make, which are basically brought into the zones duty free. But as a consequence, what is made in these zones can only be exported.
So that iPhone you buy in China has to be imported into China with appropriate duty! Made in China but still imported on a dollar basis. Besides, the iPhone is < 10% Chinese (final assembly), a lot of the components are coming from ROK, Japan, Taiwan, and of course the IP is all American.
For clothes, the amount that foreign companies is paying on material and labor was already ridiculously low, and they still pay for their other costs in dollars (or rather, not yuan) so....
I seem to recall that before HK was turned over to China, HK companies would do all but the most final of assemblies in China and then ship it across the border so they could be sold as "made in Hong Kong". And this is how Shenzen go so massive, as it is right up stream from HK.
And Singapore as well before they turned from a city state to a city offshore bank. They exploited their WTO membership.
Most of operations were ran like that: company imports loose goods from China at few pennies, and puts in "value" into them, most of the times just packaging with writing in English that was supposed to "cost" few dollars each, but of course the packaging was made in China for less than a penny each. This also allowed the Sg intermediary to claim expense on the stuff, reducing effective tax to single digits.
Foreign goods are extremely popular on Tmall and Taobao. There are companies that specialize in shipping containers full of Costco Kirkland goods to China, and usually selling them for a good markup. For example: Ferrero Rocher chocolate boxes can be bought at Costco for between 9 and 13 dollars. They sell in China for 200-250 RMB (thats about 30 USD).
One of my wife's friends is a Chinese citizen who lives in Italy. She selects purses, backpacks, and occasionally wine and markets them in China. There is a lot of demand for showing up your friends with expensive imported goods. Even more benefit if you had to use a connection to purchase the goods.
> Aren't most brands these days made in China or south east Asia? (looking at you, fashion brands)
They are. And the highest, most expensive fashion brands are still made by underpaid Chinese workers in Italy or France, so they can stick a "Made in $country" logo on the final product.
As someone who has visited the factories of Brunello Cucinelli, Kiton, and Zenga -- that is patently untrue. I saw mostly skilled craftsmen doing handwork.
So either you are making this up or you definition of "highest" isn't quite on the mark.
"Sold to Gucci for €24 euros each (around $28.09) despite a retail rate of $1000, Aroldo Guidotti of the subcontractor, leather goods specialist Mondo Libero, was filmed discussing the cost saving strategy of replacing Italian workers with Chinese nationals, who are hired as part-time employees, and yet work more hours than those on a full-time contract."
A lot of dirty tricks are played by exploiting the rules of origin. You can mark something as "Made in Italy" if the last substantial transformation of the goods occurred there. Last substantial transformation is very poorly defined, so is open to creative interpretation. You can make 95% of a handbag in China or Bangladesh, do some final stitching in Italy and perfectly legally label it as "made in Italy".
The inverse manipulation is used by Pakistani manufacturers of surgical instruments. They buy stamped steel blanks from Germany, perform the labour-intensive grinding and sharpening in Pakistan, then label the instruments as "Made in Germany" by claiming that they did not undergo substantial transformation in Pakistan.
If they're buying stuff then it's not a DDOS. If it's millions of garbage IoT devices hitting your servers and running whatever the most expensive (CPU/Memory/IO wise) queries constantly then yes, that's a DDOS.
The DDOS extortion guys are getting really good at their job. They even broke Akamai a few weeks ago. It's a spiral where they extort money and use it to improve their infrastructure to be able to extort bigger and bigger players. I have zero doubt that Alibabba's servers were already straining under the load of that many shoppers. Adding a DDOS on top of that would be catastrophic.
There might be a DOS but it's not an attack. It's easier to leave off the the "attack" in "DDOS attack" so it's implied but without it the meaning changes.
It looks like AliExpress have put some of their prices up in the last week, so they can "reduce" them for this sale. The sale price seems to match exactly the price I was seeing last week.
I noticed this for a few random items I had open in tabs, but hand't committed to buying.
Also, I'm in the UTC−05:00 timezone, but for me the sale doesn't start until 3am, so midnight UTC−08:00.
Ma tried to get to the West with B2C really really hard, but only got burned with the 10 Main fad store, a wasted office lease, and around $100m. wasted on online ads alone.
Interestingly, their biggest operation abroad is in Russia, where they spent zero on ads.
retail type of bullshit, literally. stuffing all the sale into a single day when none of the physical or non-physical systems are designed or implemented to meet such peak demand. my current company does everything to eliminate such peaks, which makes sense considering customer satisfaction (service quality) and cost efficiency, and some guys like alibaba doing the opposite. nothing but marketing scam, prices getting bloated in the day before and discounts(?) exclusive to 11.11.
Singles' Day is Nov 11 (11/11), so I'm guessing you're in Pacific time zone? The big splash was when the sale started in China, which is quite a few hours ahead of the US.
Frankly I'm extraordinarily disappointed. I went to aliexpress.com and I don't see anything in their "Today's Deals" that looks anything like a deal. Add in the fact that I'd have to wait who knows how long for shipping and deal with them for an international return and I'd rather buy locally.
Aliexpress is US-based sellers with US-based warehousing so you get it directly in the US. The better deals are on alibaba outright for Chinese consumers.
Of my 140 orders over the last two years on Aliexpress, only one has been originated in the US (California, to be exact). All but a few of the remainder were from China. I reside in Colorado. Mostly I buy hobby electronic components, chemicals/substances, and the occasional household or automotive item.
I've used Alibaba.com as well, but the payment methods are far different and it's a more risky situation with fewer guarantees. That service I use for high-value orders after a thorough vetting of the seller. Mostly commodity items like a pallet of green coffee beans, or weird chemicals in unusual quantities.
Sometimes an order on Aliexpress will be large enough that I must consider negotiating with someone on Alibaba.com instead. Negotiations happen for me on Aliexpress as well, but the sellers on Alibaba.com seem to have greater latitude in not only price, but also packing, shipping, and handling to destination.
I would like to know how many people were doing the same but a conservative guess would be over 100m concurrent users. We had to retry about a dozen times, including pruning the cart because limited stock of certain items was already gone in the first few seconds. Eventually got the order thru about 00:10am with 1/3 of items bought successfully. Now i know for next 3-5 days the courier guys will be heaving huge loads at every city apartment block. And in case you are wondering the volume of online sales on normal days is so high that courier companies have dedicated teams of delivery guys servicing each group of a few apartment buildings. I live in an estate with 4 buildings of 32 floors and here there are 4-6 from one company alone....and there are multiple companies. As someone who works in ERP software this is logistics on steroids.
Any...just sharing a story from the coal face from a HN fanboy.