Saturday, November 15, 2008

Site Update: Added Memcache and RSS

Earlier today, my website went out because I exceeded the quota limit. After using memcache on several parts of my page, I noticed a big difference. The application runs a lot faster while using a much lesser CPU.

Another thing I added is the RSS feed section. I noticed that I'm receiving around 40 unique request a day trying to access the rss section which was removed after migrating to app engine.

 

Tuesday, November 11, 2008

Use Projection.sqlProjection For MySQL Specific Functions

While at work, I had a problem and I need to project my query to look like this:

CONCAT(CONCAT(YEAR(documentDate), "_"), MONTH(documentDate))

Since hibernate doesnt support mysql or database specific queries, I needed to use Project.sqlProjection to customize my projection list. Anyway the code looks like this:

final Criteria c = session.createCriteria(JobServiceBill.class);
              
final ProjectionList projectionList = Projections.projectionList();
                
final StringBuilder sb = new StringBuilder();
sb.append("CONCAT(CONCAT(YEAR(documentDate), \"_\"), MONTH(documentDate)) AS yearMonth");
                
final Projection distinctYearMonthProj = Projections.distinct(Projections.sqlProjection(sb.toString(),
                        new String[] {"yearMonth"}, new Type[] {Hibernate.STRING}));
               
projectionList.add(distinctYearMonthProj);
                
c.setProjection(projectionList);
                
 return c.list();

 

Monday, November 10, 2008

Custom Python Decorator

Decorators allow developers to use aspect oriented approach. Here is an example of a custom python/django decorator that checks if the user is a super user and redirects it to login page if not

def super_user(func):
    def call(request, *args, **kwds):
        user = request.user
        if user.is_authenticated() and user.is_superuser:
            return func(request, *args, **kwds)
        else:
            return utils.redirect(settings.LOGIN_URL)
    return call

Here is an example usage:

@super_user
def my_view(request):
    pass

 

Saturday, November 08, 2008

Django Tip: Cache List instead of QuerySet to minimize queries to the database

While playing with django debug toolbar, I noticed that some page on my application causes serious amount of queries that is being executed. One page actually executes around 580 queries so I investigated and noticed that this was caused probably by lazy initialization, not using select_related() on my queries and i'm caching QuerySet object instead of list.

An example of a getter function on my model:

@property
    def data(self):
        cache_data = getattr(self, 'cache_data', None)
        if cache_data:
            return cache_data
        else:
            self.cache_data = TimecardData.objects.filter(timecard_day=self)
            return self.cache_data

The code above retrieves all the associated data on my timecard but to prevent it from hitting the database everytime i call that data() method, i need to cache it. One mistake I did is to cache the returned query set instead of casting it to a list type because whenever the cache_data is retrieved, It will again hit the database since it's only a query set.

After refactoring the above code to this one:

@property
    def data(self):
        cache_data = getattr(self, 'cache_data', None)
        if cache_data:
            return cache_data
        else:
            self.cache_data = list(TimecardData.objects.select_related()
.filter(timecard_day=self))
            return self.cache_data

The queries was reduced from 560 to 300 queries.

Further looking at the problem, I refactored the code to this one:

I still have a long way debugging the problem but I hope this helps you aswell.

@property
    def data(self):
        cache_data = getattr(self, 'cache_data', None)
        if cache_data != None:
            return cache_data
        else:
            self.cache_data = list(TimecardData.objects.select_related().filter(timecard_day=self))
            return self.cache_data

The code above to this one checks using "if cache_data" but since empty list returns false if evaluated, it queries the database again instead of using cache. This one brought the queries from 580 to 120 (a very big difference)

 

Tuesday, November 04, 2008

Python 2.5 Date/Time Module Poorly Designed?

I'm not sure If I'm the only one who feel that the date/time module in python 2.5 is very hard to use. If any of you ever tried using joda time, you'll notice that it's way to easy compared to python's date/time library.

Correct me If i'm wrong but joda time was created by Google guys so I hope that in the near future, python will have a better date/time library (since Google is the main supporter of python).