1) Delete data
If you have 1M records and want to keep only 10K of them I suggest you select last 10K in a temporary table, truncate entire table with TRUNCATE command, then insert 10K records back (you may need to switch off constraints and identity before the operation and switch them on again after it)
2) Make database backup
3) Shrink database file and transaction log.
Try to shrink individual files with DBCC SHRINKFILE if DBCC SHRINKDATABASE is not that you want (it cannot shrink more then initial database file, but I think this is not the point of your question)
I dont know why you think that shinking will not help you, pls give us some arguments.
There is also option AUTO_SHRINK but I never tried it with SQL Express
Also - if your database is growing because you have lot of digital content there (documents, pictures) consider redesining your system to store this content in file system and store only file names or hyperlinks in the database.
In most cases databases grow too fast if they have a lot of audit information (when each change in any table is recorded somewhere) or when there are files stored in the database. If your database grows to 4Gb and all this actual important data, like transactions, customers data etc., then SQL Express is just not right tool for you and the best choice will be to upgrade as soon as possible