Can anyone suggest an optimal setting for what I'm doing? (Note that this is a one time job, after finishing my code, I'll put this back to development setting.)
I am setting up a 64bit Ubuntu 11.10 VM with 4 cores and 16GB RAM. I'm going to import 15MM+ rows into a single table and upon saving, the Python/Django code will trigger a look up to a second table, and pull some fields from the second table and update the large table.
The job has already failed once because I was trying to import 15MM+ in one short. So I have broken the file down into 16 files.
There will be some filtering and export off the first table after the import is completed.
The DB is InnoDB. I don't know how good the default my-large or my-huge .cnf settings are and not sure if it is suitable at all for me.
Thank you in advance!