It’s important to plan ahead for times of high site traffic – especially following a TV premiere of a show that is based on your company like Shipping Wars, an A&E reality series that follows six carriers as they complete shipments they have competed for on uShip. While we were expecting a higher than normal level of traffic following this season’s premiere, the visits were impressive, and a good learning experience.
Nick Parker, uShip’s VP Engineering, brought us through how he and his team prepped for the burst of traffic that came following the premiere, and how they course corrected for future episodes.
“We had no idea what kind of traffic we were gonna get” Parker said, “I kind of assumed, completely wrongfully so, that everyone was gonna go to the Home and maybe the About Us page.”
In anticipation of what could happen, Parker sat down with Matt Chasen, CEO, to decide on the type of equipment they would need to handle the anticipated high-traffic immediately following the premiere. It came down to buying new equipment, or taking a risk with the equipment already in use. They decided to take the risky approach and stick with what was in-house.
To their dismay, it didn’t go over so well. “Everything was going fine until about five minutes after that first episode, and then it all went downhill real fast. We were completely offline for fifteen minutes. It was slow and somewhat unresponsive for about 30 min.”
Episode Two:
Parker’s team had a week to troubleshoot the issues before the next episode. At the time we only had three webservers. The load balancers, which are supposed to be the most resilient piece of equipment, went down but quickly got back online. Parker’s team of developers immediately purchased more servers, switched to a higher quality webserver and a completely virtualized environment. They focused additional energy on Find Shipments, which accounted for 80% of CPU traffic.
Crunch time hit as we didn’t actually receive the servers until Friday afternoon. With episode two of Shipping Wars to premiere the following Tuesday, Parker, a member of the IT staff, and a couple developers camped out all weekend at the colocation facility swapping out servers and testing capacity.
By the following Tuesday morning everything was stable – albeit with a little stress-induced hair loss. We were ready for episode two of Shipping Wars. We survived.
Iteration
Over the following weeks the Parker’s team continued to make massive improvements to the Find Shipments page as well as other heavy traffic pages across the site. A couple weeks later the show’s subject matter, Star Wars memorabilia, caused the highest peak of traffic the site has ever seen at one time. Turns out those improvements were well timed.
Scale
It seems to come down to company size, budget, risk assessment and data analysis.
“If you have a ton, ton of money and you can spend whatever you need to on infrastructure, that’s great. I think I would probably do the same thing again because we could have spent $300,000 on servers, [but] is 15-20 minutes of downtime worth that? I don’t think so,” said Parker. “We saw 9,000 registrations in that next hour. I think people that were interested in seeing it came back….if you have a Super Bowl ad obviously that’s a little bit different…One of the things we’ve really done since then is invest a lot more in on-demand capacity, so the ability to use cloud bursting, to spawn up instances in the cloud when we need extra CPU capacity, extra bandwidth.”
The question remains, how should companies plan for high-traffic events like this?
uShip takes a conservative trial and error approach. Have the necessary critical conversations but make calculated decisions. It’s important to plan conservatively and not stretch equipment budgets until you have a better sense of what will happen in conjunction with an event. It’s also important to understand what losses you can manage and those that would put you in high-risk situation.
“Plan for 30% increase in capacity and that’s something you can burst to over long-term…Don’t over invest in scalability or server architecture,” Parker advises.
If possible, fit in additional time to brainstorm the most popular spots across the site that people would visit as related to the event. In essence, know your audience; know where they frequent on the site and if their intent will continue to bring them back despite potential blips in user experience.
“Next time I would] try to maybe put more thought into it than I did, into what Pages people are actually gonna be visiting…If we had done more profiling on Find Shipments to find out how much CPU it needed and how to make improvements, I think we would have been better off,” says Parker.
Shipping Wars Season 5, Episode 5 premiered yesterday at 10/9 CST on A&E.
