There is such a thing as "too many nameservers", but not for the reasons you're concerned with. The others have covered the irrelevance of latency concerns and I won't beat that horse to death.
The real problem with adding too many nameservers is authority bloat. A reply to a SOA record request should not exceed 512 bytes if at all possible, including both the authority and additional sections. TCP and EDNS can be used to overcome the 512 byte barrier in most cases, but some nameserver software still behaves poorly when the authority+additional information can't fit within a standard 512 byte response. (mostly in regards to zone transfers or DNS forwarders)
Taking the above into consideration, the answer on how many DNS servers is too many still boils down to "it depends". You should plan on your DNS servers supporting IPv6, so that's going to add overhead. (A+AAAA records)
For reference, here is what Yahoo's SOA reply looks like at this point in time:
$ dig @ns1.yahoo.com +norecurse yahoo.com soa
; <<>> DiG 9.9.5-9+deb8u2-Debian <<>> @ns1.yahoo.com +norecurse yahoo.com soa
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53960
;; flags: qr aa; QUERY: 1, ANSWER: 1, AUTHORITY: 6, ADDITIONAL: 11
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1272
;; QUESTION SECTION:
;yahoo.com. IN SOA
;; ANSWER SECTION:
yahoo.com. 1800 IN SOA ns1.yahoo.com. hostmaster.yahoo-inc.com. 2015100224 3600 300 1814400 600
;; AUTHORITY SECTION:
yahoo.com. 172800 IN NS ns2.yahoo.com.
yahoo.com. 172800 IN NS ns3.yahoo.com.
yahoo.com. 172800 IN NS ns6.yahoo.com.
yahoo.com. 172800 IN NS ns1.yahoo.com.
yahoo.com. 172800 IN NS ns4.yahoo.com.
yahoo.com. 172800 IN NS ns5.yahoo.com.
;; ADDITIONAL SECTION:
ns1.yahoo.com. 1209600 IN A 22.214.171.124
ns1.yahoo.com. 86400 IN AAAA 2001:4998:130::1001
ns2.yahoo.com. 1209600 IN A 126.96.36.199
ns2.yahoo.com. 86400 IN AAAA 2001:4998:140::1002
ns3.yahoo.com. 1209600 IN A 188.8.131.52
ns3.yahoo.com. 86400 IN AAAA 2406:8600:b8:fe03::1003
ns4.yahoo.com. 1209600 IN A 184.108.40.206
ns5.yahoo.com. 1209600 IN A 220.127.116.11
ns6.yahoo.com. 172800 IN A 18.104.22.168
ns6.yahoo.com. 1800 IN AAAA 2406:2000:108:4::1006
;; Query time: 27 msec
;; SERVER: 22.214.171.124#53(126.96.36.199)
;; WHEN: Fri Oct 02 19:03:45 EDT 2015
;; MSG SIZE rcvd: 411
As you can see above, six NS records along with their corresponding A and AAAA addresses are already weighing in at 411 bytes. If I were to add so much as six characters to the base domain (6 * 19 = 144, 411 + 144 = 555) I'd already be breaching 512 on mandatory records alone, nevermind optional bloat this usually adds to the right hand side of the SOA record.
Most companies don't use more than four NS records. Six is probably too much. There aren't many good reasons for extending into this number so long as you're following the usual guidelines, or using a well-respected DNS provider.
DNS answers are cached, possibly on multiple levels. I.e. I would bet most end users use their provider's DNS..