Blog

During the course of the day I encounter lots of challenges, some of which take minutes to solve and others that take much longer. My goal for this section is to keep track of these challenges and their solutions. I will turn the longer ones into articles, while the shorter ones will stay as blog entries. You can expect topics to range from very specific programming challenges to broader topics like life.

Below is a list of the recent blog entries. You can also browse the blog by using the tags on the right side, or if you know what you are looking for then you can use the search box at the top right.

Ordering Django querysets by custom SQL based columns

As I was working on my website in Django I had a need for ordering a model based queryset on a custom SQL. Searching around the net I found this entry in the Django user group, but according that entry I have to create a suclass of the queryset and then a manager, in addition to other things. I didn’t like that solution so I continued looking around and finally came up with one that I think is cool and accomplishes the goal. Basically, we have to use the extra() method to create a new column based on custom SQL and then use that in the order by clause.

1
2
comments = Comment.objects.filter(node__id=node.id).extra(
  select={'thread_order': "SUBSTR(thread, 1, LENGTH(thread)-1)"}).order_by('thread_order')

In this case I needed to exclude the last character of the ‘thread’ column, and that’s what the SUBSTR does. So, there you have it, a quick way to order model querysets by custom SQL.

Please update your blog feed URL

Lots of changes are happening, I am moving my feed to FeedBurner and in the next few days I will be deleting the original feed URL. So, please update your readers to use the FeedBurner URL http://feeds.feedburner.com/TheBitGuru. I have actually used FeedBurner for a while now so there is a good chance that you are subscribing to the FeedBurner URL, but remember to update in case if you still have the original URL.

Unicode Fun!

Bottom line: If you keep getting “No JSON object could be decoded” when loading JSON objects in django that look perfectly valid then make sure that you are using ASCII encoding.

I have been experimenting with django’s test framework and I recently hit a wall while creating a JSON fixture. Django wouldn’t like my JSON file and keep spitting out…

Problem installing fixture 'thebitguru/nodes/fixtures/nofmt_nodes.json': No JSON object could be decoded

The fixture was very simple and for the longest time I couldn’t figure out what was wrong. I had used the code>dumpdata command to dump the JSON in the first place, but after Magus- on #django (IRC) asked me if that’s what I had done, I decided to do it again. This time somehow magically it worked! I used TortoiseMerge to diff the two files and it claimed that they were exactly the same. Obviously, they weren’t!

I, being I who I am, wasn’t satisfied without finding out really why my initial export was not working. So, off I went to do some investigation. After about fifteen minutes and several different hex dumps and google searches I was looking at this wikipedia entry. Yes, 0xFEFF! That was the key to this puzzle. So, somehow my initial export had ended up in UTF-16 Big Endian encoding. A quick encoding change in VIM, set fileencoding=ascii, and everything was back to normal.

I took a trip down the memory lane and realized that I was using PowerShell at the time, and considering that it is fairly new and fancy shell I guessed that it must probably is using Unicode as the default output. If you look through the PowerShell User Guide you will see this specifically mentioned…

By default, the Out-File cmdlet creates a Unicode file. This is the best default in the long run, but it means that tools that expect ASCII files will not work correctly with the default output format. You can change the default output format to ASCII by using the Encoding parameter:

PS> Get-Process | Out-File -FilePath C:\temp\processlist.txt -Encoding ASCII

Another quick test verified that this was in fact the cause. Whew! Finally, another mystery solved!

Windows Vista SP1 for multiple computers

UPDATE:

Nick White has an excellent post about SP1 on the Windows Vista Blog. Check it out, it lists several reasons why SP1 might not show up in Windows Update. Going the Windows Update route will be worth it because

  • The download size from Windows Update of Windows Vista SP1 for x86 is 65 MB (compared to 450 MB from the Microsoft Download Center).
  • The download size from Windows Update of Windows Vista SP1 for x64 is 125 MB (compared to 745 MB from the Microsoft Download Center).

Microsoft has made Windows Vista Service Pack 1 (SP1) available in the Windows Download Center. SP1 will show up as an optional update in Windows Update within the next few days, if it hasn’t already. If you have multiple computers running Windows Vista and you don’t want to wait for it to show up then you can get it from here.

As you can see on that page…

DO NOT CLICK DOWNLOAD IF YOU ARE UPDATING JUST ONE COMPUTER: A smaller, more appropriate download is available on Windows Update.

So, please don’t use this if you are going to apply this to only one computer. I have three so I think my download is justified 🙂

Using custom settings in django tests

I have been learning about django over the past two weeks or so. I am finally up to the point where I have started learning about its suggested test frameworks. The creators suggest using unittest, which is the standard python unit testing framework. I have liked this aspect of django: even though the core framework was built from ground up, many of its pieces still rely on standard python libraries.

Read full post… “Using custom settings in django tests”

Lambda functions in Python

Python

The lambda functions in Python always seemed very cool to me, but I never thought that I would actually get to use them in my projects. Guess what? Last night I was working on a side project when I had the perfect opportunity to use and very much appreciate this feature!

As part of my move of this site to django I have to move over the “custom” textile format parsing that I created for this site. Basically, it’s mostly textile with the additional feature of allowing you to easily embed the attached image or file. Below is the Python specific regular expression that defines this syntax.

pre.
\[(?Pinline|image):(?P[<>]?)(?P\S+?)(=(?P.*?))?\]

This allows me to easily reference an attached image and instructor the site to resize based on my needs. So, after I upload an image for a blog entry all I have to do is put [image:>0=thumb] (right aligned “standard” thumbnail sized, as predefined by me) or [image:0=200x100] (default alignment with 200×100 resize keeping the original image in proportion) wherever I want an embedded image to be displayed. With this I don’t have to worry about resizing the image or adding the img tag manually, all that is done automatically by my Node class.
Ruby has a useful notation where I could do this in a block notation as…

1
2
3
replaced = txt.gsub(/\[(inline|image):([<>]?)(\S+?)(=(.*?))?\]/m) {
  // do all the custom code in this block.
}

Since python doesn’t have the concept of blocks it uses the approach of passing in functions that return the replacement text.

1
2
3
4
5
6
7
8
def replacement_function(match):
  // interpret the match in this function
	return "...replacement for match." 

def interpret_attachments(text):
  attach = re.compile(r'\[(?P<type>inline|image):(?P<alignment>[<>]?)(?P<positionOrAlias>\S+?)(=(?P<size>.*?))?\]', re.MULTILINE | re.IGNORECASE)
  replaced = attach.sub(replacement_function, text)
  return replaced

Now, the problem comes in if replacement_function needs some extra parameter, which in my case was node_id. How can you pass that to replacement_function?

1
2
3
def replacement_function(node_id, match):
	// interpret the match in this function
	return "...replacement for match." 

That is where lambda function comes in. I ended up encapsulating the call in a lambda function and passing that to the sub() function.

1
2
3
4
def interpret_attachments(text, node_id):
  attach = re.compile(r'\[(?P<type>inline|image):(?P<alignment>[<>]?)(?P<positionOrAlias>\S+?)(=(?P<size>.*?))?\]', re.MULTILINE | re.IGNORECASE)
  replaced = attach.sub(lambda match: get_replacement(node_id, match), text)
  return replaced

I remembered seeing numerous references to lambda functions in the django framework, so now that I understood the need for these, I went back and looked at a few examples. They all make much more sense now!

Here is an excerpt from django/contrib/auth/decorators.py

1
2
3
4
5
6
def permission_required(perm, login_url=None):
    """
    Decorator for views that checks whether a user has a particular permission
    enabled, redirecting to the log-in page if necessary.
    """
    return user_passes_test(lambda u: u.has_perm(perm), login_url=login_url)

…and one more example from django/template/defaultfilters.py

1
2
3
4
5
def title(value):
    """Converts a string into titlecase."""
    return re.sub("([a-z])'([A-Z])", lambda m: m.group(0).lower(), value.title())
title.is_safe = True
title = stringfilter(title)

Awesome!

If you are confused about lambda functions then I would suggest reading through the filter, map and reduce section on the python site. That section demonstrates good usage of lambda functions.

Running django-admin in PowerShell

Continuing with my theme of tips, about what might seem like pretty much random stuff, today I am going to show how to setup your PowerShell profile to allow you to run python and django-admin in the active shell. Without this if you run django-admin using something like python django-admin.py then it will pop up a new command prompt window. Basically, you need to do two steps.

  1. Run PowerShell (as administrator in Windows Vista) and allow execution of local scripts without signing it: set-executionpolicy remotesigned
  2. Create UserProfile\Documents\WindowsPowershell\profile.ps1 (on Vista or UserProfile\My Documents\ in other versions) and put the following text in there, changing the paths as appropriate.


$env:PYTHONPATH=“R:\django\packages\django-trunk”
$env:PATH=$env:PATH + “;D:\Programs\Python25\;R:\django\packages\django-trunk\django\bin”

function django-admin {python (gcm django-admin.py | resolve-path) $args}

If you are curious about the execution policy then checkout the very useful Getting Started document that came with PowerShell (look in your Start Menu).

2008-03-31, Update: I had an issue with the way I had posted this originally. I needed to add ‘$args’ so that the arguments would be passed to the script.

Wireless Configuration in Windows XP

I just spent 45 minutes helping a relative get his wireless network connection to work. He was used to the normal Windows XP wireless network configuration utility, but recently somehow Dell’s wireless configuration utility decided that it would like to configure the wireless card instead. The weird part is that the utility has been on there since Windows XP was installed almost two years ago, and previously it didn’t bother him. What I believe was even worst was that the Windows configuration utility had a very cryptic message talking about the Wireless Zero Configuration service and pointing to a knowledge base article number. How can the user lookup this article if he/she is having wireless connection problems?

That aside, instead of pointing the user to Wireless Zero Configuration service, the Windows XP utility should have checked that the service was running, which it was, and eliminated that as an option. Furthermore, there should be an easy way for the user to go the utility that might have offered to configure the wireless, instead of just telling the user that another program is doing it. Thinking about it, this is just one more interaction between the operating system (OS) and the program offering assistance, i.e. the new utility should tell the OS where it lives and how to run it.

Doing these two step would have saved both of us time, and who knows it might even have prevented this call!

Mapping Caps Lock to Backspace in Linux

One of the first things that I do after installing the operating system is remap the Caps Lock key to the Backspace key. This is because I have rarely seen any need for the Caps Lock key, while the Backspace is much more useful. You can remap this in Windows using a very nice program called SharpKeys. In Linux there are several ways to accomplish this. You can change the underlying keyboard layout, or if you do not use the terminal directly and instead use the command line interface only through X then you can accomplish this very easily by using xmodmap.

Basically there are two steps.

remove lock = Caps_Lock
keysym Caps_Lock = BackSpace

Save the above code to a file (~/bin/map_caps_lock_to_backspace.txt
, may I suggest), and then execute xmodmap map_caps_lock_to_backspace.txt. Try it out, assuming you are in an X session, now the Caps Lock key will act as Backspace!

There you have it. Now, to do this every time that X starts, you can add this to your favorite window manager (XFCE > Settings > Autostarted Application).

Starting a new initiative: Understanding Windows Vista

Last night I posted the first section of the “Understanding Windows Vista” series. I have started this as a personal initiative to diffuse the bad rep. that Windows Vista has been getting. Check it out.