I am quite new to using servers, but I recently ran into a bottle neck and had to do my computations on the cloud. After 2 weeks of using the server, there seems to be some memory issues.

I use the server for Jupyter notebooks and storage of large files. I load in large files into my Jupyter notebooks, and my files loaded without problem for the first 2 weeks. However today, I ran into memory problem after memory problem. And its so bad that even after I shut off all active notebooks and ran my first few note books, none of those even worked because of memory problems.

Using the TOP command, it says I have about 90% of the memory available. However, it won't let me run the code I have been running the the last 2 weeks without problems.

So I am wondering if I should restart the server regularly or something to solve this problem. Or if it is something else.



If you load a file in a Jupyter notebook and store its content in a variable, the underlying Python process will keep the memory for this data allocated as long as the variable exists and the notebook is running. Python's garbage collector will free the memory again (in most cases) if it detects that the data is not needed anylonger. This is the case if it is deleted, e.g. by using del, if the variable is overwritten with something else or if it goes out of scope (a local variable at the end of a function).

If you are storing large files in (different) variables over weeks, the data will stay in memory and eventually fill it up. In this case you actually might have to shutdown the notebook manually or use some other method to delete the (global) variables.

A completely different reason for the same kind of problem might be a bug in Jupyter. These type of bugs are called memory leak and often occur in server processes running for a long time. Even if they are less likely to happen in Python, there are some bug reports for Jupyter. In this case, the only work around might be restarting the Jupyter process. In other cases I would recommend doing so.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.