So, you're trying to do something like curl http://example.com:22?
Yes, in that case, you will get the SSH banner SSH-2.0-OpenSSH_5.3 or whatever it is you're using, and the HTTP browser will error out because it's not an HTTP response.
There's nothing you can do about that: it's the service advertising itself. See the FAQ: http://www.openssh.org/faq.html#2.14
You are first making a TCP connection (level 4). Once that happens, the applications (server and client) start talking to establish their communications. The server says SSH-2.0-OpenSSH_5.3, 220 mail.example.com ESMTP Postfix or whatever is appropriate to the protocol, and waits for the client. The client either recognizes what the server is (in which case they start communicating), or it bails out with a protocol mismatch (e.g., Firefox says that it can't connect). You can't really keep the banner from displaying after TCP establishes.
If you want to "block direct port access", you'll have to block it below the application level, perhaps with a firewall rule that prevents all connections to that port except for a whitelisted IP address.