The hint to the core rule set is good, but the actual rule you should use is
This rule is especially for your case: protect certain url from being brute forced and block the IP that initiates this brute force attack. You can configure this rule in the setup file
In this setup file modify the variable
tx.brute_force_protected_urls in rule 900014 with the urls you want to protect:
setvar:'tx.brute_force_protected_urls=#/error/404.html#, #/error/403.html#', \
tx.brute_force_burst_time_slice specifies the amount of time in seconds within this a certain threshold has to be exceeded. As I understand your question right, use 60. The variable
tx.brute_force_block_timeout defines the time the IP of the brute forcer is blocked. The variable
tx.brute_force_counter_threshold is a bit tricky. It defines how often you can access an url before you get blocked. It is not the exact number, but with the value 2 or 3 you should get a quite good result for your block-after-10-request-rule.