"If you are using System State, you should be thinking about 10 to 15 gigabyte that is necessary on the protected server. However, for the BMR, you don't need any space on the protected server because the data is sent straight to the DPM server.
DPM itself will not calculate the size for BMR but assumes 30 GB should be enough. However, this is not always enough, but is an average for the size of all critical volumes if you haven't installed or placed anything else on it. Just the OS. So the best thing you can do is calculate yourself (roughly) what the size is of your critical volumes and resize the allocated space to that.
Critical volumes = Boot Volume + System Volume + Volume hosting system state data
Just my 2 cents,
I had the same issue and when I would run the "C:\Program Files\Microsoft Data Protection Manager\DPM\binBmrBackup.cmd" in a command prompt I would get a not enough free space. So I bumped up the DPM store from 30GB to 200GB since my c: drive on that server had 177GB of data on it.