Syed Karim asked: "Where/how did you end up hosting the site..." so here are my "notes from the front":
I didn't find any "automagical" service or solution. Wordpress.com VIP Hosting, WP Engine, and a few others look pretty interesting, but didn't fit this bill. So we hosted directly on Amazon Web Services (AWS).
We used (a few, big) EC2 server instances + Elastic Load Balancer + S3/CloudFront offload of I/O + optimization/minification + caching + CloudWatch Monitoring + SNS notifications + Python/bash automation/scripting + replication based on Mercurial.
The TL;DR details:
This site was a flash-mob site for an event of 1M+ people. It had only nominal visitors 15 days before the event, but then started accelerating into a fierce-slope, aggressive climb at T-3 days or so. With either Google Analytics or site performance monitors, you could watch load notably increase hour by hour.
We used EC2 servers, running Ubuntu "Natty Narwhal," MySQL 5.1, and WordPress 3.x. They were fronted by an Elastic Load Balancer. We had to shift DNS to Amazon's Route 53 service to make ELB work right with our primary domain (e.g. domain.tld rather than www.domain.tld).
For the servers, we preferred to scale up more than out in order to avoid the many complexities and potential failure modes of shared/replicated/clustered MySQL and NFS/CFS file pools. We could and did scale out, but in a "eventually consistent" and "all nodes use their own local storage" way. Site changes were made to a master server, then file and database changes were replicated to the rest of the WP pool. Replication was handled by a combination of Mercurial and SCP (there are fancier ways, these are simple and worked great).
In this configuration, eventually consistent <= ~3 minutes. That is, some users could see older content for a few minutes after it had been updated.
WordPress under load wants CPU much faster than memory, so we ended up vastly over-configured on memory. But we needed the CPU, and the price/complexity tradeoff was right for the limited number of hours we were in peak operation.
We considered an autoscaling configuration, but knowing the short duration of peak loads and the likely demand curve, and having good performance metrics and alarms (starting with CloudWatch and SNS), we chose manual scaling--yet another successful KISS choice.
The most common scaling operation was activating the next-largest-size instance, auto-loading required software, auto-replicating site content to it, and adding it to the ELB group. Starting from scratch, instance activation typically completed in under 3 minutes. With human confirmation the server was operating correctly and adding it to the load balancer, scaling up took perhaps 5-10 minutes in total. We would typically then quiesce the server instance it replaced. In the same 5-10 minutes, we could have spun up a few dozen new servers; we tested but never actually needed to deploy that.
We ran instance size all the way up to High-Memory Quadruple Extra Large Instance for the final full-peak load. We considered the even higher-CPU, higher-network-bandwidth Cluster Compute nodes, but did not deploy there because they require a different Linux build (the Amazon AMI, based on Red Hat/CentOS rather than Ubuntu), and we didn't want to invest in build automation or QA for a second software foundation.
We used S3 for files, with clients accessing it through a CloudFront distribution (i.e., S3 fronted through a CDN) to offload I/O. Going by the numbers, aggressively using I/O offload is the most important thing we did. It was also the simplest.
We optimized/minified JS, CSS, and PHP files for WP themes and plugins as best we could. Such changes toy with breaking the code, so we used Mercurial to track changes, and immediately back out any changes that broke the site. The designers had, unfortunately, chosen some plugins for image galleries and displaying the Twitter feed that were great when you get 50 requests an hour, but not nice in a scaling config, zapping right through your CPU and Twitter API allotments. Seriously, Twitter does not want you banging on their door 100+ times a second! Failing to find a Twitter feed plugin or service that was guaranteed scalable, I wrote my own simple caching mechanism. The image gallery we could't fix without changing the look in at least subtle ways, so we just bought more CPU to compensate for its bad design. (Next time: Designers, please select one that uses JS only, not PHP!)
We used YSlow, FireBug, and Google Chrome's Developer Tools to guide the optimization process.
We used the W3 TotalCache to reduce site load. Unfortunately, site design wasn't compatible with full caching (broke some plugins). We didn't have time to fix those issues, so we limited caching to what could be done safely, and again, bought more CPU time to compensate.
Most of the automation was accomplished via either Bash shell scripts or Python programs using the excellent Boto module/API. We considered using higher-level provisioning enablers like Fabric, Chef, and Salt, but various glitches we encountered said "we're awesome, but come back and try us again a bit later, ok?"
The results were excellent. We never went down, and never bobbled under load. We were ready for and could have handled 5x or 50x the load we received. There are many things we could have done more elegantly. Some of those would be desirable or even necessary on services that have different requirements--for example, high load over longer timeframes, greater load variability or unpredictable spikes, etc. But the many KISS tradeoffs worked well, scoring a practical win with modest DevOps effort and low execution cost.
My biggest regret is we didn't automate the process of detecting content changes and replicating them throughout the fleet. We required manual coordination between site operators and content owners. This worked ok, given the short peak window--but it was onerous. If I had it all to do over, watching the master for file/database changes and auto-firing the replication process across the fleet would be the #1 upgrade. (That would have the secondary benefit of greatly increasing the scale-out-ability of the enterprise; we could then use more smaller, cheaper server instances. An autoscale configuration would then be more highly leveraged.)