I've tried to setup nginx on my brand new linux-server together with Plesk 10.3 (latest) and created a new nginx subdomain for port 8226 - how can I configure Plesk to use nginx as a server for my subdomain created in Plesk?
You are looking for a so called "reversed proxy". You do not need to create another subdomain in your Plesk-Panel.
Just tell your apache, to proxy all requests from sub.mydomain.com to your predefined nginx-port (8226).
Create a new file
For further questions, I advise you to read the official Apache Virtual Host Documentation.