Security groups won't be visible, because VPC security groups have no meaning outide their VPC... and your Classic instance is (of course) outside the VPC.
Private VPC IP addresses of the instances won't work in the security config, either, since they also have no meaning outside the VPC.
The solution is that you have to open up access (in the RDS security group) for the public IP addresses that will be visible to RDS --
If the VPC machines are in a public subnet, you would use each machine's public IP address. (If the VPC machines are in a public subnet and don't have public IPs, that's an incorrect configuration).
If the VPC machines are in a private subnet, you need the public IP address(es) associated with the VPC's NAT instance(s) to be open in the RDS security group, because those VPC machines will be using that source address to contact the Classic RDS, and the address of the NAT instance is what will be visible to RDS.