Comments on: Django query set iterator – for really large, querysets

By: Danilo

Danilo — Fri, 01 Apr 2011 08:43:42 +0000

I added another iterator function in order to return a list of size n containing query result rows, at the same time making use of the original queryset_iterator.

Feel free to improve it on gist: https://gist.github.com/897894

By: Danilo

Danilo — Fri, 01 Apr 2011 08:10:59 +0000

Aaron: Your improvement with the first PK is good, but the pk increment using `pk += chunksize` only works if you have consecutive primary keys.

By: Aron Griffis

Aron Griffis — Tue, 11 Jan 2011 21:56:05 +0000

Thanks, Thierry. This is exactly what I need in my application. I tweaked it slightly as you can see here: http://pastebin.com/6ukwatVs (setting pk to the first primary key in the queryset rather than 0, and incrementing pk once rather than for each row).

By: Very Large Result Sets in Django using PostgreSQL

Very Large Result Sets in Django using PostgreSQL — Tue, 14 Dec 2010 03:10:20 +0000

[...] also an example here of constructing an iterator that does much the same [...]

By: Christophe Pettus

Christophe Pettus — Tue, 14 Dec 2010 02:31:07 +0000

Ah, never mind: Psycopg2 grabs the whole result set even if you are doing .fetchmany().

By: Christophe Pettus

Christophe Pettus — Sun, 12 Dec 2010 03:17:20 +0000

Adding .iterator to your query set helps somewhat, but still loads the entire query result into memory.

Hm, that’s interesting. Looking at the Django code (1.2.3), it appears that it is not loading the whole result set into the memory. Instead, it’s calling .fetchmany for a series of chunks, each chunk being hard-coded to be 100 rows. Is your experience different, or could the memory usage be coming from elsewhere?

By: Django Memory Error – How-to work with large databases « Harbinger's Hollow

Fri, 25 Jun 2010 20:13:28 +0000

[...] the Memory efficient Django Queryset Iterator, written by Thierry [...]

By: Django query set iterator – for really large, querysets

Django query set iterator – for really large, querysets — Sat, 06 Mar 2010 17:08:02 +0000

[...] solution for dealing with very large querysets in django when memory is a limiting constraint, with some nice discussion in the comments about why limit and [...]

By: rick

rick — Fri, 05 Mar 2010 01:51:09 +0000

@Henrique: even with UUIDs this would still work, it works as long as you have some unique sortable identifier. Since UUIDs are unique they are by definition sortable so the method still works flawlessly.

The problem here was that the database client (Psycopg2) was putting all (or atleast, way too much) of the rows in memory so the application ran out of memory. For the record, we’re using Postgres 8.4.

@Oleg: using a limit is nice on the clientside, but the server doesn’t like it. When you’re using LIMIT/OFFSET queries the server has to get all the results and only return the requested ones.

So when doing `LIMIT 1 OFFSET 1000000` the server will fetch 1000001 rows and discard 1000000 of them. On a large table with a large offset this is a very slow process. That is also the reason that Google only shows the first 1000 results ;)

With this iterator your database is able to use the index on the primary key so it’s always fast to fetch the requested rows.

By: Thierry

Thierry — Thu, 04 Mar 2010 11:29:57 +0000

Well the PK based method above is quite similar to using slicing.
Slicing however gives a few problems.

Something like [100000:200000] is not very nice on your DB. The PK method is less heavy.
If you use slicing though, be sure to iterate from the back of the list to the beginning. Otherwise you’ll have problems when items are being removed from the list by the process you use it for.

By: Oleg

Oleg — Thu, 04 Mar 2010 11:26:14 +0000

Why not to use limit function on given queryset. Like qs[from:to] it will add LIMIT OFFSET construction to your query.

By: Thierry

Thierry — Thu, 04 Mar 2010 11:24:03 +0000

Hey Henrique, thanks for the input. As far as I can see its the actual reading of the resultset from the db into python’s memory which causes the problem. So the raw result from the DB, not actually Django’s orm layer.

Would probably need to adjust some things in the cursor handling within django’s orm to implement a fix like you suggest…

By: Henrique

Henrique — Thu, 04 Mar 2010 02:44:43 +0000

Django’s iterator() helps on avoiding the internal cache, but I think you’re hitting a limit on either the database adaptor on the database itself. Could benefit from some DB tuning. Are you using MySQL?

Now, the problem with the snippet is that it naturally won’t work for non-auto PKs – UUIDs for instance. As such, it means it won’t work with really big or sharded databases, that can’t use auto PKs. Those are the ones that would benefit the most from burst loading result sets.

If the problem is simply memory usage and is limited to batch tasks, maybe streaming the result set to disk and consuming on a queue later on could help.