For Web Apps, Always On means your app stays loaded in memory, vs being aged out (which prevents long delays when a user first visits after the app's cache has expired). It's not related to the allocated Web App resources being deployed. If you have an App Service Plan (and it's not at Free tier), you're paying for it.
With Web Apps, you can start and stop individual web apps under the app service plan, but you cannot just stop the app service plan. The ways to reduce cost-burn:
- scale back number of instances
- scale back size of instances
- scale back service tier (you can go as low as Shared and still use custom domain name, with higher tiers required for SSL)