Suppose if I want to limit each upload to be at 25 MB via SFTP, how can I make this possible? Do I add a line in /etc/ssh/sshd_config for that? I tried to look for this information via Google search but I couldn't find it at all.
I don't think many sftpd packages support this by default, but I admit that I have not done exhaustive research. My recommendation would be to go beyond sftp and limit the max file size for a user on the system more generally.
The other thing to keep in mind is that if your sftp package is part of your ssh package, SFTP users usually have SSH access when they have sftp access.