LACP is negotiated link aggregation (LAG) using a protocol. It needs to be actively supported on both sides. If one of the sides doesn't support LACP the trunk doesn't come up and you've got two separate ports. If the modem uses a simple switch for both ports (plus the internal one) and there's no spanning tree protocol running, this effectively creates a bridge loop, killing the network.
You'll need to find out what exactly the modem does.
You should try connecting two hosts, one on each port. If they can "see" each other (ping etc) the modem uses a simple bridge. With a bridge, you should be able to get up to 1 Gbit/s from each port alone.
Setting up a static, non-protocol trunk on the downlink switch might work. The trunk will make the switch regard both links as a single, logical port. It won't care where a frame came in, and it will never forward it back to the modem.
In the reverse direction, the switch will likely use some kind of SA/DA traffic distribution. This needs to be on MAC or IP addresses alone, don't activate TCP or UDP port distribution - that'll mess up the modem's MAC table.
However, the modem will likely reflect traffic not addressed to the modem itself back to the network. You might have a problem with the modem reflecting broadcasts back through the trunk but maybe you can live with that (possibly minimizing the broadcast domain for that segment).
A cleaner solution would be to set up a load-balancing router with two uplinks.
