Yes, but it's not just because of your use of Telnet and your weak passwords, it's because of your attitude towards security.
Good security comes in layers. You should not assume that because you have a good firewall, your internal security can be weak. You should assume that at some point in time, your firewall will be compromised, the workstations will have viruses, and your switch will be hijacked. Possibly all at the same time. You should make sure that important things have good passwords, and less important things do too. You should also make use of strong encryption when possible for network traffic. It's simple to set up, and in the case of OpenSSH, makes your life easier with the use of public keys.
And then, you also have to watch out for the employees. Make sure everyone's not using the same account for any given function. This makes it a pain for everyone else when someone gets fired and you need to change all the passwords. You also have to make sure that they don't fall victim to phishing attacks through education (tell them that if you ever asked them for their password, it would be because you've just gotten fired and you don't have access anymore! Anyone else has even less reason to ask.), as well as segmenting access on a per-account basis.
Since this seems to be a new concept to you, it's probably a good idea for you to pick up a book on network/systems security. Chapter 7 of "The Practice of System and Network Administration" covers this topic a bit, as does "Essential Systems Administration", both of which I recommend reading anyway. There are also entire books dedicated to the subject.