Interesting question. You'd have to issue a certificate for the IP, which according to this question is possible, but I know Let's Encrypt who I use doesn't do it.
Once you have done it you would need to set up a default server for SSL that looks something like this (note that I haven't checked it so it may need tweaking)
listen 80 default_server;
listen 443 default_server; # not sure if you can / need to specify default server twice
access_log off; log_not_found off;
return 444; # This means "go away", effectively, but you can choose whatever HTTP status code you want
Update - as per Michael Hampton's insightful comment below, just use a self signed certificate.