You very likely have a duplex mismatch on account of the ISP hard-coding their side to 100-Full essentially disabling auto-negotiation on the ISP Ethernet PHY.
With the ISP set to 100-Full and your side remaining at auto/auto (a hunch, but a common one), the auto-negotiation on your side will configure the interface to 100-Half -- a duplex mismatch as the ISP side will remain 100-Full.
You can fix the problem by hard-coding your Ethernet PHY to 100-Full -- or specifically whatever the ISP is set to. Most ISP's use 100-Full.
With the duplex mismatch of 100-Full to 100-Half, the 100-Full side disables CSMA/CD while CSMA/CD remains in effect on the 100-Half side. The 100-Full side transmits without regard to whether or not the medium is free. The 100-Half side perform CSMA/CD checks and backoff as defined by CSMA/CD. This is why you can only achieve 20 Mb/s on what should be a 50 Mb/s Internet circuit. The CSMA/CD backoff on account of the 100-Half side detecting collisions is limiting throughput.
By hard-coding the interface to 100-Full to match the ISP, both sides will have CSMA/CD disabled, therefore backoff and collision detection will be disabled and you should achieve numbers much closer to your 50 Mb/s Internet circuit data rate.
Many ISP's hard-code their Ethernet PHY handoffs as there was a time when it was much more reliable to do so. When the original 802.3u 100 Mb/s Fast Ethernet standard was released auto-negotiation of speed and duplex was present, but not required. It wasn't until 802.3z 1 Gb/s Gigabit Ethernet standard when auto-negotiation was required by the standard.
Many network engineers have misconceptions about auto-negotiation. The biggest misconception is that auto-negotiation can properly negotiate speed and duplex if only one side implements auto-negotiation. This is false -- as you have seen.
The reason for this likely stems from the following -- if one side is hard-coded at 100-Full, the other side running auto-negotiation always seems to figure out the 100 Mb/s part. Same if one side is hard-coded to 10-Full -- the other side running auto-negotiation can figure out the 10 Mb/s part. The ability to determine the link speed is from a feature called parallel detection which tries the received physical layer signal on all locally supported link speeds until a match is found. However, parallel detection only works for the speed, not for duplex matching. This is why duplex mismatches can occur -- as an interface will always fall back to half-duplex when it cannot determine the other side through auto-negotiation.
At one time there was spotty support for auto-negotiation and it caused just as many problems as it was intended to solve. That time, in this network engineer's opinion -- has passed. While auto-negotiation problems still exist, the number of problems I have seen due to auto-negotiation being configured in the last 5 years dwarfs the number of problems I have seen on account of auto-negotiation being disabled.
I have never had an ISP unwilling to change their Ethernet handoff to auto/auto upon being asked. With most cable and DSL modems and gateways this is not a problem. It is the NxT1 and fiber managed CPE routers with Ethernet handoff where this problem usually resides. The problem is that a network administrator has to ask in the first place.
With an ISP hard-coding to 100-Full they have given an obligation. An obligation that must be documented and continued. Auto-negotiation is technology that is now stable, has been for years, and takes care of this problem for us. As mentioned earlier, the number of problems caused by auto-negotiation is far outweighed by the number of issues that arise on account of it being disabled in 2011. Technology exists to solve this problem, use it. Perhaps we should be manually setting the initial TCP SYN's, MSS, and managing the Receive Window for every TCP virtual circuit as well? I kid.