Is there a way that I can objectively compare the performance of different CDNs? Will I need to purchase a plan and test it out myself?
A good way to objectively test a CDN is by using a monitoring service that's geographically diverse and on diverse networks that can mimic your application loads.
A good CDN will get content to your users at lower latency and with fewer hops than going all that way back to your origin servers.
For a web site, that typically means web page loads.
You didn't specify your application, so I'm assuming web site.
Companies like Gomez and Keynote (there are many others -- see some below, but those are the old time heavyweights-- most $$$$) offer services that track how well a webpage will load under a variety of parameters. For US focused sites, I usually like to check from 6 regions in the US. (Seattle, LA, Chicago, Dallas, Boston, Miami).. If you care a lot about international traffic you can set up probes abroad too.
You also want to setup probes on different ISPs. How well a CDN works for customers with Comcast might be very different from how it looks to customers on Level 3. Having a mix of top tier and 'eyeball' networks is good. (monitoring companies already have probes on a variety of networks)
Be careful with these services. They tend to charge per probe you hit, and while you may think that checking every 5 minutes is cool, it will soon cost you. Consider it a development tool to track long term trends and only turn up the frequency when you're A/B testing CDNs. (better yet, set up a different URLs for each CDN you are testing and let the same probes hits all of them.)
Cheaper alternatives to Gomez and Keynote: Pingdom WebpageTest Siteuptime AlertSite
I've had good luck with AlertSite in the recent past.