What is your memory_target set to? If it's set high, then I wouldn't be surprised to see it grow to the max size over time. Take a look here for more information on the new memory_target parameter in 11g. That will put a cap on how much memory Oracle will attempt to get.
The memory_target parameter configures Oracle to self manage memory parameters that used to require individual attention:
A note is that the STATISTICS_LEVEL must be set to TYPICAL.
Two parameters are actually involved here: memory_max_target and memory_target. The first sets the maximum value for memory_target and requires an instance restart to change:
ALTER SYSTEM SET memory_max_target = 1500M SCOPE=SPFILE;
Once in place, the memory_target parameter can be set dynamically without an instance restart:
ALTER SYSTEM SET memory_target = 1000M;
This is a really nice feature for folks who want a basic "set it and forget it" memory management setup. Set the highest value you think you'll need for the memory_max_target, and then play with the value for memory_target to tune to your needs. Let Oracle manage the individual parameters. Oracle will use the instance statistics to manage these values, so as your instance evolves in terms of usage patterns, Oracle will take this into account.
Another nice reference on these parameters can be found at the Oracle FAQ's Oracle Wiki.