go with XFS which is really great for storing large files, but be warned: never-ever pull USB cable out of computer while XFS pendrive is still mounted, it will destroy/damage your XFS filesystem at a high chance.
You should also consider encrypting the filesystem. Various methods existing for this, by my best practice is the script below:
$ cat bin/encmount
echo 0 `/sbin/blockdev --getsize $1` crypt aes-plain $HASH 0 $1 0 | /sbin/dmsetup create $2
$ sudo sh bin/encmount /dev/sdb1 encpen
you can then mount /dev/mapper/encpen as a block device, while /dev/sdb1 (my original pendrive path) are encrpyted. of course it's necessary to create an XFS filesystem on top of the newly created block device (/dev/mapper/encpen), and the original block device content (/dev/sdb1 in this case) gets shredded during doing so.
be sure you execute these commands before you pull out pendrive:
dmsetup remove encpen
Your other option is to use ext2/ext3, as there's Ext2 IFS For Windows to let Windows read/write your filesystem as ext2, but in this case you lost the performance gain on XFS large file handling and will not be able to encrypt the drive.
Always backup your data, for example, I always keep 2 backups of my data in 2 different place, neither of them is close to the site of the backup subject. Backups shall be encrypted also.