I started off by loading a day of data for one of our applications and spending a few weeks just coming up with questions about the data I wanted answered: how many transactions per second of a particular customer, how busy are the busiest times for different transaction types, how can I search the logs for SLA violations, stuff like that.
It's a bit surprising how easy a lot of things are to search, and the more I searched the more ideas I had for new searches. Before long you'll get quite a catalog of saved searches.
The gotcha that got me in the beginning is making sure the time and hostname data is correct at index time. Some of our custom logs were not timestamped in a friendly format and took a few iterations to get it indexed correctly. Be sure to index a few small samples first to ensure everything looks correct before indexing a large collection of logs.
But, yeah, just imagine the questions you want answered about your data.