Warning: "continue" targeting switch is equivalent to "break". Did you mean to use "continue 2"? in /nfs/c02/h02/mnt/21229/domains/perfectresolution.com/html/wp-includes/pomo/plural-forms.php on line 210
Over 5 Minutes » How to Use the Python Memory Profiler

How to Use the Python Memory Profiler

Recently one of my coworkers was having an issue where some of our code running over at heroku was consuming a massive amount of memory. One of the tools I was looking at to help troubleshoot this was the python memory profiler. While I was loading a newer copy of our data set my co-worker identified the root cause of the issue, so we didn’t end up using this tool.  That said, I found the information in provided while profiling code incredibly interesting, and potentially useful in the future. 

Installation is pretty straight forward using our old friend pip:

pip install memory_profiler

After it’s installed, you need to decorate a function or two with the @profiler tag.  Here’s a small example:

from memory_profiler import profile
import random

@profile
def memory_test():
numbers = [random.randint(0, 1000) for x in range(100000)]
biggest = max(numbers)

return biggest

memory_test()

After that you get a pretty nifty–as well as high level–out put of memory consumption in your function that will look something like this:

python -m memory_profiler test.py
Filename: test.py

Line #    Mem usage    Increment   Line Contents
================================================
     4     11.3 MiB      0.0 MiB   @profile
     5                             def memory_test():
     6     16.9 MiB      5.6 MiB        numbers = [random.randint(0, 1000) for x in range(100000)]
     7     16.1 MiB     -0.8 MiB        biggest = max(numbers)
     8
     9     16.1 MiB      0.0 MiB        return biggest

One thing I like about this is how simple the out put is. Even with longer functions, or functions that call a lot of other functions its easy to find what line number that that memory growth is happening on.

October 7, 2016 • Tags: , • Posted in: Technology

Leave a Reply