How to Use the Python Memory Profiler

Recently one of my coworkers was having an issue where some of our code running over at heroku was consuming a massive amount of memory. One of the tools I was looking at to help troubleshoot this was the python memory profiler. While I was loading a newer copy of our data set my co-worker identified the root cause of the issue, so we didn’t end up using this tool.  That said, I found the information in provided while profiling code incredibly interesting, and potentially useful in the future. 

Installation is pretty straight forward using our old friend pip:

pip install memory_profiler

After it’s installed, you need to decorate a function or two with the @profiler tag.  Here’s a small example:

from memory_profiler import profile
import random

@profile
def memory_test():
numbers = [random.randint(0, 1000) for x in range(100000)]
biggest = max(numbers)

return biggest

memory_test()

After that you get a pretty nifty–as well as high level–out put of memory consumption in your function that will look something like this:

python -m memory_profiler test.py
Filename: test.py

Line #    Mem usage    Increment   Line Contents
================================================
     4     11.3 MiB      0.0 MiB   @profile
     5                             def memory_test():
     6     16.9 MiB      5.6 MiB        numbers = [random.randint(0, 1000) for x in range(100000)]
     7     16.1 MiB     -0.8 MiB        biggest = max(numbers)
     8
     9     16.1 MiB      0.0 MiB        return biggest

One thing I like about this is how simple the out put is. Even with longer functions, or functions that call a lot of other functions its easy to find what line number that that memory growth is happening on.

October 7, 2016 • Tags: , • Posted in: Technology

Leave a Reply