The sort order behaviour of sort(1) is controlled by your locale settings (see
There are a number of different locale settings, for example:
To choose the desired sort behaviour, you need to choose the correct LC_COLLATE value. In this case, the standard built in C (POSIX) locale is suitable:
$ sort testcase
$ LC_COLLATE=C sort testcase
If you prefer, you can set all the locale settings (thus being more consistent) by setting LC_ALL=C. Since these are environment variables, you can permanently set your sort order, with
export LC_ALL=C or similar, in your shell startup script.