Not exactly. You can't use any old storage medium. They do make USB Smart Sticks (Smart card in usb stick form really). The problem is in how verification with smart cards works.
This is the generic/simplified version of the process: The "server" application knows the smart card's public cert. It creates a nonce and encrypts it with the public cert. It then sends the encrypted nonce to the client; client forward it to the smart card. The smart card decrypts it with the private key; the nonce is then sent back to the server (usually re-encrypted, but that is trivial to this process).
Note that the client computer never sees the private certificate. This way the client computer can not make a copy of the private certificate and the token is always needed for authentication. If it could be copied then it would be insecure; a "criminal" could copy your card, return it, and you wouldn't know that they have your authentication mechanism.
Smart Cards for PKI Login (Windows AD Login) aren't terribly expensive* if you avoid the packaged solutions. Of course those packaged solutions make the whole process easier and require much less knowledge on your part.
*Athena makes a PKI Smart Card for $36 each (cheaper in bulk).
*Aladdin makes a eToken Pro USB for $65
Side note: I really wouldn't recommend using tokens to completely replace passwords. If nothing else I'd recommend issuing the private keys with a password and setting the expiry for a year; even a simple password would be very effective in this situation (though you should still avoid dictionary words and anything blatantly obvious).