Yes, there is. Using a tap device causes context switches between the client process and the process that holds the tap device. It also causes additional copies, as data needs to be copied from the client process into the kernel, and then from the kernel into the tap holder.
However, Linux is pretty good at context switching, and in-cache copies are fairly cheap, so the overhead might be small enough. I would recommend that you implement your stack in user-space, using tap, and then perform some benchmarking and profiling to decide whether it is worth the effort to move your code into the kernel.