Archive for the 'Python' Category

Mixing Django with Jinja2 without losing template debugging

At Fashiolista we’ve build nearly the entire site with Jinja instead of the Django template engine.

There are a lot of reasons for choosing Jinja2 over Django for us. Better performance (atleast… it was a lot better with previous Django versions), way more options (named arguments, multiple arguments for filters, etc), macros and simply easier to extend. Writing custom tags is simply not needed anymore since you can just make any function callable from the templates.

But… during the conversion there are always moments when you need  a Django function in a Jinja template or vice versa. So… I created a few template tags to allow for Jinja code in Django templates (I’ve also created code to run Django code from Jinja, but I haven’t seen the need for it so I omitted it here).

A Jinja Include tag to include a template and let it be parsed by Jinja from a Django template:

from django import template
from coffin import shortcuts as jinja_shortcuts                                                                                                                                                      

register = template.Library()                                                                                                                                                                        

class JinjaInclude(template.Node):
    def __init__(self, filename):
        self.filename = filename                                                                                                                                                                     

    def render(self, context):
        return jinja_shortcuts.render_to_string(self.filename, context)                                                                                                                              

@register.tag
def jinja_include(parser, token):
    bits = token.contents.split()                                                                                                                                                                    

    '''Check if a filename was given'''
    if len(bits) != 2:
        raise template.TemplateSyntaxError('%r tag requires the name of the '
            'template to be included included ' % bits[0])
    filename = bits[1]                                                                                                                                                                               

    '''Remove quotes if used'''
    if filename[0] in ('"', "'") and filename[-1] == filename[0]:
        filename = bits[1:-1]                                                                                                                                                                        

    return JinjaInclude(filename)

Usage:

{% jinja_include "some_template.html" %}

A couple of noop nodes to make sure that when you convert your Jinja templates to be executed from Django, they won’t break because of the missing Django tag.

from django import template

class Empty(template.Node):
    def render(self, context):
        return ''                                                                                                                                                                                    

@register.tag
def django(parser, token):
    return Empty()                                                                                                                                                                                   

@register.tag
def end_django(parser, token):
    return Empty()

And the Jinja tag to allow Jinja blocks in Django templates.

from django import template
from coffin.template import Template

register = template.Library()

class Jinja(template.Node):
    def __init__(self, template):
        self.template = template                                                                                                                                                                     

    def render(self, context):
        return self.template.render(context)                                                                                                                                                         

@register.tag
def jinja(parser, token):
    '''Create a Jinja template block                                                                                                                                                                 

    Usage:
    {% jinja %}
    Although you're in a Django template, code here will be executed by Jinja
    {% end_jinja %}
    '''                                                                                                                                                                                              

    '''Generate the end tag from the currently used tag name'''
    end_tag = 'end_%s' % token.contents.split()[0]                                                                                                                                                   

    tokens = []
    '''Convert all tokens to the string representation of them
    That way we can keep Django template debugging with Jinja and feed the
    entire string to Jinja'''
    while parser.tokens:
        token = parser.next_token()
        if token.token_type == template.TOKEN_TEXT:
            tokens.append(token.contents)                                                                                                                                                            

        elif token.token_type == template.TOKEN_VAR:
            tokens.append(' '.join((
                template.VARIABLE_TAG_START,
                token.contents,
                template.VARIABLE_TAG_END,
            )))                                                                                                                                                                                      

        elif token.token_type == template.TOKEN_BLOCK:
            if token.contents == end_tag:
                break                                                                                                                                                                                

            tokens.append(' '.join((
                template.BLOCK_TAG_START,
                token.contents,
                template.BLOCK_TAG_END,
            )))                                                                                                                                                                                      

        elif token.token_type == template.TOKEN_COMMENT:
            pass                                                                                                                                                                                     

        else:
            raise template.TemplateSyntaxError('Unknown token type: "%s"' % token.token_type)                                                                                                        

    '''If our token has a `source` attribute than template_debugging is
    enabled. If it's enabled create a valid source attribute for the Django
    template debugger'''
    if hasattr(token, 'source'):
        source = token.source[0], (token.source[1][0], token.source[1][1])
    else:
        source = None                                                                                                                                                                                

    return Jinja(Template(''.join(tokens), source=source))

Do note that I have modified the “coffin.template.Template” to enable debugging completely. Just replace the “Template” class in “coffin/template/__init__.py” to make it work.

def _generate_django_exception(e, source=None):
    '''Generate a Django exception from a Jinja source'''
    from django.views.debug import linebreak_iter                                                                                                                                                    

    if source:
        exception = DjangoTemplateSyntaxError(e.message)
        exception_dict = e.__dict__
        del exception_dict['source']                                                                                                                                                                 

        '''Fetch the entire template in a string'''
        template_string = source[0].reload()                                                                                                                                                         

        '''Get the line number from the error message, if available'''
        match = re.match('.* at (\d+)$', e.message)                                                                                                                                                  

        start_index = 0
        stop_index = 0
        if match:
            '''Convert the position found in the stacktrace to a position
            the Django template debug system can use'''
            position = int(match.group(1)) + source[1][0] + 1                                                                                                                                        

            for index in linebreak_iter(template_string):
                if index >= position:
                    stop_index = min(index, position + 3)
                    start_index = min(index, position - 2)
                    break
                start_index = index                                                                                                                                                                  

        else:
            '''So there wasn't a matching error message, in that case we
            simply have to highlight the entire line instead of the specific
            words'''
            ignore_lines = 0
            for i, index in enumerate(linebreak_iter(template_string)):
                if source[1][0] > index:
                    ignore_lines += 1                                                                                                                                                                

                if i - ignore_lines == e.lineno:
                    stop_index = index
                    break                                                                                                                                                                            

                start_index = index                                                                                                                                                                  

        '''Convert the positions to a source that is compatible with the
        Django template debugger'''
        source = source[0], (
            start_index,
            stop_index,
        )
    else:
        '''No source available so we let Django fetch it for us'''
        lineno = e.lineno - 1
        template_string, source = django_loader.find_template_source(e.name)
        exception = DjangoTemplateSyntaxError(e.message)                                                                                                                                             

        '''Find the positions by the line number given in the exception'''
        start_index = 0
        for i in range(lineno):
            start_index = template_string.index('\n', start_index + 1)                                                                                                                               

        source = source, (
            start_index + 1,
            template_string.index('\n', start_index + 1) + 1,
        )                                                                                                                                                                                            

    exception.source = source
    return exception          

class Template(_Jinja2Template):
    """Fixes the incompabilites between Jinja2's template class and
    Django's.                                                                                                                                                                                        

    The end result should be a class that renders Jinja2 templates but
    is compatible with the interface specfied by Django.                                                                                                                                             

    This includes flattening a ``Context`` instance passed to render
    and making sure that this class will automatically use the global
    coffin environment.
    """                                                                                                                                                                                              

    def __new__(cls, template_string, origin=None, name=None, source=None):
        # We accept the "origin" and "name" arguments, but discard them
        # right away - Jinja's Template class (apparently) stores no
        # equivalent information.
        from coffin.common import env                                                                                                                                                                

        try:
            return env.from_string(template_string, template_class=cls)
        except JinjaTemplateSyntaxError, e:
            raise _generate_django_exception(e, source)                                                                                                                                              

    def __iter__(self):
        # TODO: Django allows iterating over the templates nodes. Should
        # be parse ourself and iterate over the AST?
        raise NotImplementedError()                                                                                                                                                                  

    def render(self, context=None):
        """Differs from Django's own render() slightly in that makes the
        ``context`` parameter optional. We try to strike a middle ground
        here between implementing Django's interface while still supporting
        Jinja's own call syntax as well.
        """
        if not context:
            context = {}
        else:
            context = dict_from_django_context(context)                                                                                                                                              

        try:
            return super(Template, self).render(context)
        except JinjaTemplateSyntaxError, e:
            raise _generate_django_exception(e)                                                                                                                                                      

def dict_from_django_context(context):
    """Flattens a Django :class:`django.template.context.Context` object.
    """
    if isinstance(context, DjangoContext):
        dict_ = {}
        # Newest dicts are up front, so update from oldest to newest.
        for subcontext in reversed(list(context)):
            dict_.update(dict_from_django_context(subcontext))
        return dict_
    else:
        return context

And you’re done, now you can just mix your Django and Jinja templates like this:

{% ifequal foo bar %}
Django style if...
{% endif %}

{% jinja %}
{% if foo == bar %}
Jinja style if...
{% endif %}
{% end_jinja %}

Django & Jinja2 & Python & Web Development Rick van Hattem 24 Aug 2010 8 Comments

Django open inviter – contact importer – python

Django open inviter is a python port of the PHP api client for openinviter.com’s contact importer to work with Django. I build it for our fashion community, Fashiolista.com, where it is currently in production usage and fully functional. If you are a member of Fashiolista (which I highly doubt given the different audiences) you can test it by clicking find friends in your profile.

Usage is extremly straight forward:

from django_open_inviter.open_inviter import OpenInviter
o = OpenInviter()
contacts = o.contacts('example@example.com', 'test')

Get the code here.

Django & Fashiolista & PHP & Python & Web Development & YouTellMe tschellenbach 09 Aug 2010 No Comments

Creating your own Digg/Facebook/Tweetmeme button

This quick walkthrough is going to bring you up to speed on how to create your own social bookmarking button. The three prime examples are the famous Digg button, Facebook’s like functionality and the tweetmeme button. For an implementation look slightly above this paragraph or check out mashable’s version on the left of their post.

Our button will be focusing on Fashiolista.com. Fashiolista is a social bookmarking site for fashion, which has seen rapid growth after launching at the next web. This tutorial explains the javascript (client side) aspects of the button. Feedback and improvements on the code would be greatly appreciated. You can find the full 450 lines of js on github.

This is what the end result looks like:

Compact Medium Large

Love it!

Love it!

Love it!

(If you are working on a shop in the fashion industry have a look at our installation instuctions.)

Step 1 – The markup

Its important to get the client side markup of the button right. Since other sites will be implementing this there is no way you can change it later on. The three major players each have their own way.

Facebook XFBML: Async script with XFBML or Iframe
Digg button: Async script with A elements
Tweetmeme: Normal script

<script type="text/javascript">
  //async script, fashiolista.com version
  (function() {
   var s = document.createElement('SCRIPT');
   var c = document.getElementsByTagName('script')[0];
   s.type = 'text/javascript';
   s.async = true;
   s.src = 'http://button.www.fashiolista.com/button/script/';
   c.parentNode.insertBefore(s, c);
  })();
</script>
<a class="fashiolista_button fashiolista_compact"
href="http://www.fashiolista.com/item_add_oe/">Love it!</a>

For Fashiolista we have chosen an async script approach with A elements. Normally loading a script element is a blocking operation for the browser. Loading the script async ensures faster page load times and a better experience if your site would ever go down. (Note that not all browsers support this option so it is still recommended to include the script tag at the bottom of the page). The function wrapped around the code ensures we don’t pollute the global scope. Furthermore the insertBefore in combination with a script tag technique is used by GA so should work in any scenario.

Step 2 – Creating the buttons, Iframe vs Script

The next step is to convert our A elements into actual buttons. We can choose to replace these A elements by our button’s html (digg, delicious approach) or load an iframe in their place (facebook, tweetmeme). The difference between these two approaches is actually pretty large. For Fashiolista you can see both an iframe and script approach. These are the most important differences I encountered.

Iframe vs Script

  • + Popup communication possible
    The script approach cannot communicate with popups it creates due to the same origin restrictions. The iframe however can be of the same domain as the popup and freely communicate. This gives a better user experience when for instance logging in.
  • + Easier to develop
    The iframe approach is easier to develop and requires less code.
  • + Parallel download in IE
    IE doesn’t download the count scripts in parallel, but it does do so for the IFRAMEs. Making this approach somewhat faster.
  • Independent CSS
    External sites don’t interfere with your button’s css if you use an iframe technique. The disadvantage is that it makes things likes hovers impossible to integrate with the other site. (For example Fashiolista’s compact button).
  • Independent
    The iframe approach makes it very hard for other sites to game the users like/love action. With a script approach a foreign site can simply call your javascript to fake someone loving the product. This freedom can be abused but also allows for mashups.
  • - Slower dom load
    Creating iframes takes a lot more time for the browser.
  • - Slower perceived load
    The script approach allows you to format the buttons before the data is loaded. Vastly increasing the perceived load speed.
  • - No shared functionality
    Buttons can’t share functionality. So when someone logs in for one button its is not possible to update the others.

The best choice differs for each project. For Fashiolista the more open script approach is currently the default.

Step 3 – Cross site scripting using JSONP

Essential to the bookmarking button is requesting the count for the given url. Cross site policies prevent us from using Ajax so we will do so by creating a script element.

_makeRequest: function (url) {
	//Simple create script element functionality
        var s = document.createElement('script');
        var b = document.body;

        s.setAttribute('type', 'text/javascript');
        s.setAttribute('async', 'true');
        s.setAttribute('src', url);

        b.appendChild(s);
}

The trouble with the script element is that you lack the nice APIs Ajax offers you. We work around this by using an url with a callback paramater, for example callback=button_loaded_3
The server side code then responds with something like this, executing the callback when the script is loaded.

button_loaded_3({"item_id": 26545, "url": "/item/26545/", "loves": 853})

This technique is often referred to as JSONP. We bind the response function to the global button_loaded_3 using the following code:

loadButtonInformation: function (buttonId) {
		//make a request to the script with the given callback
		var buttonInstance = this.buttonDict[buttonId];
		var buttonUrl = buttonInstance.lookupUrl;
		var path = '&url=' + encodeURIComponent(buttonUrl);
		var callbackFunctionName = 'button_loaded_' + buttonId;
		var scope = this;
		var callbackFunction = function(data) {
			//bind the scope and button id
			scope.buttonLoaded.call(scope, buttonId, data);
		};
		window[callbackFunctionName] = callbackFunction;
		this.makeRequest(this.countApi + path, callbackFunctionName, true);
}

Step 4 – Object oriented design

Since we are loading our code into someone else’s website we should be careful not to use similar variable names. We therefore hide as much code as possible in classes.

var fashiolistaClass = function(){ this.initialize.apply(this, arguments); };
fashiolistaClass.prototype = {
	//
	//Base class implementing the fashiolista button
	//
	initialize: function () {
		//load the buttons
		this.initializeCss();
		var fashiolistaButtons = this.findButtons();
		this.initializeButtons(fashiolistaButtons);
	}
}

Note that we are not simulating inheritance for these classes. Using them as simple namespaces is more than sufficient in this case.
The code is organized into 3 classes:

  • fashiolistaClass
  • fashiolistaUtilsClass
  • fashiolistaButtonClass

The first one acts as a manager (finding the buttons, instantiating fashiolistaButtonClasses and retrieving counts). Fashiolista button contains the logic for individual buttons and fashiolista utils contains some string parsing and dom load functionality.

Step 5 – Caching requests in the google app engine

appengineTo prevent our servers from getting flooded we are routing all traffic through google servers using the google app engine. button.www.fashiolista.com is connected to a google app engine account which forwards and caches requests to fashiolista.com. This setup enables your button to withstand great amounts of traffic without killing your servers. Furthermore it immediately also acts as a cdn for our web requests, speeding up load times for our international visitors. Setting up caching in the google app engine would require another blog post though. Let us know in the comments if you would like to know more about it.

Conclusion

The full client side code can be found here. This blog post covered the most essential parts. Code review and questions are more than welcome. Be sure to let us know in the comments. Furthermore if you are running a webshop in the fashion industry consider implementing the button.

More information

Improvements/ Request for code review

  • The domload technique is rather verbose, does anyone know a better method?
  • The popup communication or lack thereof is not ideal for users, is there a better method?
  • Script or Iframe what do you prefer?
  • Suggestions to make it faster?

Django & Fashiolista & Google app engine & JQuery & Javascript & Python & Web Development & YouTellMe tschellenbach 03 Aug 2010 3 Comments

Django Facebook – Open graph API implementation

For Fashiolista.com Fashiolista we needed to integrate with the Facebook Open Graph API. The open graph API is a very exciting facebook project, which you can read about more here and here. The code at fashiolista.com allows you to register/login via facebook using the Open Graph API (similar to the old Facebook connect, but registration, instead of only logging in). Before you go try it out, fashiolista.com is aimed primarily at females so your girl friends facebook account is probably a better fit.

Im releasing the source code for Django Facebook on github. Its a very early release, but it might help other developers trying to implement a facebook register/loging flow using the new open graph api. See Github for requirements and installation instructions.

Update:
Birthday formats are currently giving some troubles for some users.
Fixed and tests added to prevent future problems. (note Fashiolista.com may still give errors, will be resolved during our next release).

Django & Fashiolista & Javascript & Python & Web Development tschellenbach 17 May 2010 11 Comments

Django query set iterator – for really large, querysets

When you try to iterate over a query set with about 0.5 million items (a few hundred megs of db storage), the memory usage can become somewhat problematic. Adding .iterator to your query set helps somewhat, but still loads the entire query result into memory. Cronjobs at YouTellMe.nl where unfortunately starting to fail. My colleague Rick came up with the following fix.

This solution chunks up the querying in bits of 1000 (by default). While this is somewhat heavier on your database (multiple queries) it seriously reduces the memory usage. Curious to hear how other django developers have worked around this problem.

import gc

def queryset_iterator(queryset, chunksize=1000):
    '''
    Iterate over a Django Queryset ordered by the primary key

    This method loads a maximum of chunksize (default: 1000) rows in it's
    memory at the same time while django normally would load all rows in it's
    memory. Using the iterator() method only causes it to not preload all the
    classes.

    Note that the implementation of the iterator does not support ordered query sets.
    '''
    pk = 0
    last_pk = queryset.order_by('-pk')[0].pk
    queryset = queryset.order_by('pk')
    while pk < last_pk:
        for row in queryset.filter(pk__gt=pk)[:chunksize]:
            pk = row.pk
            yield row
        gc.collect()

#Some Examples:
#old
MyItem.objects.all()

#better
MyItem.objects.all().iterator()

#even better
queryset_iterator(MyItem.objects.all())

Django snippet here.

Django & Python & Web Development & YouTellMe tschellenbach 03 Mar 2010 7 Comments

YTM launch!!

No more beta for YouTellMe.nl
The website which is taking over the Dutch product comparison market is officially going out of beta @ 8 o clock.
Party in Amsterdam, Keizersgracht 182 :) Festivities starting right now!

13342_350348980430_784785430_9966158_5558110_n

Things are going well, looking very forward to international launch.
We’ve changed a lot since the first reviews!

13342_350352790430_784785430_9966172_7726367_n

Beter pictures coming after the event :P

PS. Thanks to Python and Django, for enabling us to beat the competition :)

PSS. Next2News, eduhub, come and join :)

Apache & Business & Css & Django & Dutch & Events & Javascript & PHP & Prototype & Python & Symfony & Web Development & YouTellMe tschellenbach 11 Dec 2009 1 Comment

Django template tags – Google chart – python 2.4 port

I just tried the django template tags for the google charting api by Jacob. Unfortunately they were python 2.5 only and I happen to still be stuck to 2.4. The changes to move it to 2.4 were minimal though. Still to save some of you googlers out there the hassle:

charts

I was just browsing the code a bit. There is something peculiar there:

args, varargs, varkw, defaults = inspect.getargspec(func)

I don’t get the point of using the inspect functionality. Anyone care to explain?

Django & Python & Web Development tschellenbach 02 Jan 2009 4 Comments