Comments for Mellow Morning

Comment on Django query set iterator – for really large, querysets by Django query set iterator – for really large, querysets

Django query set iterator – for really large, querysets — Sat, 06 Mar 2010 17:08:02 +0000

[...] solution for dealing with very large querysets in django when memory is a limiting constraint, with some nice discussion in the comments about why limit and [...]

Comment on Django query set iterator – for really large, querysets by rick

rick — Fri, 05 Mar 2010 01:51:09 +0000

@Henrique: even with UUIDs this would still work, it works as long as you have some unique sortable identifier. Since UUIDs are unique they are by definition sortable so the method still works flawlessly.

The problem here was that the database client (Psycopg2) was putting all (or atleast, way too much) of the rows in memory so the application ran out of memory. For the record, we’re using Postgres 8.4.

@Oleg: using a limit is nice on the clientside, but the server doesn’t like it. When you’re using LIMIT/OFFSET queries the server has to get all the results and only return the requested ones.

So when doing `LIMIT 1 OFFSET 1000000` the server will fetch 1000001 rows and discard 1000000 of them. On a large table with a large offset this is a very slow process. That is also the reason that Google only shows the first 1000 results ;)

With this iterator your database is able to use the index on the primary key so it’s always fast to fetch the requested rows.

Comment on Django query set iterator – for really large, querysets by Thierry

Thierry — Thu, 04 Mar 2010 11:29:57 +0000

Well the PK based method above is quite similar to using slicing.
Slicing however gives a few problems.

Something like [100000:200000] is not very nice on your DB. The PK method is less heavy.
If you use slicing though, be sure to iterate from the back of the list to the beginning. Otherwise you’ll have problems when items are being removed from the list by the process you use it for.

Comment on Django query set iterator – for really large, querysets by Oleg

Oleg — Thu, 04 Mar 2010 11:26:14 +0000

Why not to use limit function on given queryset. Like qs[from:to] it will add LIMIT OFFSET construction to your query.

Comment on Django query set iterator – for really large, querysets by Thierry

Thierry — Thu, 04 Mar 2010 11:24:03 +0000

Hey Henrique, thanks for the input. As far as I can see its the actual reading of the resultset from the db into python’s memory which causes the problem. So the raw result from the DB, not actually Django’s orm layer.

Would probably need to adjust some things in the cursor handling within django’s orm to implement a fix like you suggest…

Comment on Django query set iterator – for really large, querysets by Henrique

Henrique — Thu, 04 Mar 2010 02:44:43 +0000

Django’s iterator() helps on avoiding the internal cache, but I think you’re hitting a limit on either the database adaptor on the database itself. Could benefit from some DB tuning. Are you using MySQL?

Now, the problem with the snippet is that it naturally won’t work for non-auto PKs – UUIDs for instance. As such, it means it won’t work with really big or sharded databases, that can’t use auto PKs. Those are the ones that would benefit the most from burst loading result sets.

If the problem is simply memory usage and is limited to batch tasks, maybe streaming the result set to disk and consuming on a queue later on could help.

Comment on Django vs Symfony by zoja

zoja — Fri, 05 Feb 2010 11:37:29 +0000

I’m a great Symfony fan and i really found this article very interesting. You got me interested in looking further at Django. Seems like the perfect solution for big applications with high server load. Thanks for the the comparison :)

Comment on Django vs Symfony by nihimu

nihimu — Mon, 28 Dec 2009 03:06:42 +0000

As languages go I can say that after learning both by heart, only one of the feel like a real language. And its not PHP!

Comment on YTM launch!! by VidJa

VidJa — Fri, 11 Dec 2009 22:14:13 +0000

Gave site!

Comment on Django vs Symfony by Drupal’s prominence is hurting the growth of the PHP frameworks, such as Symfony

Mon, 16 Nov 2009 00:15:47 +0000

[...] versus framework, which would be more fair than the current framework versus CMS comparisons. (When the Django versus Symfony comparison is made, Symfony is at least able to hold its own, with at least a few areas where it is superior to [...]

Comment on Introducing a cross site ajax plugin for Prototype by Icaro Dourado

Icaro Dourado — Thu, 30 Jul 2009 03:23:15 +0000

Hi everyone,
This solution helps me A LOT!

Comment on Updated cross site ajax plugin for Prototype by Confluence: Régis Leray

Confluence: Régis Leray — Wed, 15 Jul 2009 10:27:04 +0000

Emailvision documentation…

PRESENTATION Historiquement l’envoi d’email etait effectue par notre propre serveur SMTP, ne gerant pas en mode queue les messages (si erreur non reemission des messages), et aucune statistique concernant les envois. D’ou l’utilisation,……

Comment on Introducing a cross site ajax plugin for Prototype by Max Kiesler - Designer » Blog Archive » Learning AJAX & Javascript by Example - Tutorials, Source-Code and Documentation

Tue, 02 Jun 2009 07:08:55 +0000

[...] Introducing a Cross Site AJAX Plugin for Prototype “During cross site Ajax requests the standard XmlHttpRequest approach breaks down. The problem is that XmlHttpRequest is bounded by the same site policy. Fortunately the script tag has the freedom to do as it pleases.” [...]

Comment on Ten reasons why Symfony rocks – Part 2 by Kenneth van Rumste

Kenneth van Rumste — Tue, 26 May 2009 09:33:53 +0000

Any idea how it’s possible to get the i18n:extract to search class files too? In the Symfony, we still find: Unfortunately, the i18n:extract task does not yet parse form classes for untranslated strings. This is a huge problem… anyone an idea?

Comment on Django vs Symfony by Mellow Morning " Django vs Symfony

Mellow Morning " Django vs Symfony — Mon, 18 May 2009 23:42:47 +0000

[...] rest is here: Mellow Morning " Django vs Symfony Share and [...]