WE SHIP INTERNATIONALLY 
All was going well on Friday and Saturday, then on Sunday... no website! Yes, I mean no website.... nothing. There is a message saying:
"This site has moved to a new server"
Corrupting database engine
So after sending several texts and make some phone calls, I get hold of Yann from Hexica. He tells me that the server had a failing and corrupting database engine. These database engines are normally incredibly reliable and to have one corrupt in this way was very rare. If left running it would corrupt and destroy all the databases on the server. So Yann took the difficult decision to move the commercial sites to a new server that was being prepared for use in the New Year.
Site moved?
So Unicycle.uk.com got moved to a new superfast server. The move should have been easy and reliable as it has been tested for the last 6 months with our Sandbox site.
DX records, Cloudflare and magic wands.
This all took a few hours, so by 1pm we only needed to wave the magic wands and change the DNS records and set up Cloudflare. This should have been straight forward, but we were getting lots of redirects and bounce back to the old IP address.
Ahhh, Help!
Nothing we could do would bring the website back up. Even though this was Sunday, we had lots of helpers as everyone knew how important it was to get the site up and running. At one point we had 4 teams of people working/advising on it at the same time.
Stripping the site
We started stripping the site, taking out the parts that would not run. We got the site so it would run, roughly. This is all done in full view of the public. We had people starting to try and make orders. Although the site was unstable and no one was able to make orders and we were having to fend off bits of the site to make it stable.
Site runs... but wont take any money!
With lots of help we eventually got the site up and running. Other than some styling problems it looks like the site was running by 11pm. It even appeared to taking orders, I ran several test orders and the site appeared to working... phew. There was one little problem, it was not actually taking any money and confirming the orders.
Monday morning and still no orders!
Monday morning it was back to work with more support, we were putting in tickets with one application after another, enlisted Paypals top level merchant support. Still nothing until we realised that a small security programme on the server that installed was .1 different in version than on the original server. We down graded it to the original version and Bam it all works!
What do we do now?
So what do we do now? we were down for 28 hours in total. Lots of customers missed the deals they were waiting for. The black Friday deals finished at Midnight on Sunday night....
Black Friday extended
So we extended black Friday for 4 more days. Within 2 minutes of the site coming back on line with the black Friday deals in place, we had 2 order!
A few words of thanks
It was a long stressful weekend and one I hope not to have to repeat. I would like to thank Sola from Productive Minds, Cpanel support team, Paypal MTS team, Cloudflare and particularly Yann from Hexica. I would also like to thank all the customers who offered screen shots to help us sort this out and for their patience.
