First, you can (most likely) use your registrar's DNS.
I haven't yet seen a registar that allowed you to buy a domain, but didn't offer free DNS hosting. If you want the least complicated solution possible, this is most likely it.
Secondly, you can use a free (or paid) DNS hosting service.
Just a simple search for 'free DNS hosting' yielded plenty of results. If you trust/like these options better than your registar's DNS option (or your registrar doesn't offer DNS hosting), then use one of them. Paid DNS hosting options are also readily available.
Finally, you can host your own DNS server.
You have a VPS, so you can always host your own DNS server. This is personally my favorite option, as it gives you full control over the DNS servers. Also, you stated you like open source, and most DNS servers are open source. If you would like help with this, I have experience setting up bind9 and would be more than willing to help, so just comment down below and I'll help out.
To answer your question of what the difference could be, in general paid DNS hosting can afford better and more servers, so your DNS will be more fault tolerant and resilient to DoS attacks. Of course, if you host your own DNS the stability will be determined by how powerful your servers are. However, as you are only serving your one domain name, you need significantly less power than the DNS hosters who server tons of domain names (I've been hosting my own DNS on 2 single core 1GB RAM VPSs fore a long time, and they work perfectly).