I’ve been banging my head against similar problems with HTML and also with processing syndicated feeds with feedparser. Some, but not all of them, seem to have been resolved by editing sitecustomize.py.

I’m nowhere near your level of gurudom (I don’t grok C much), but could you explain why you think editing sitecustomize.py is a bad idea? I understand that in some cases (like on a hosted web server), the file may not be accessible to users without root, but isn’t it fair to say that utf-8 *should* be the default encoding, if at all possible?

Interesting post, cheers…

If you’re using sitecustomize on an app for which you completely control the environment, that should work just fine. But, if you’re distributing the app, it’s really a lot better if your application can run properly in whatever environment it gets dropped into.

By the way, with some help from the python-list, I solved my problem:

http://www.blueskyonmars.com/archives/2005/02/03/dont_mix_unicode_and_encoded_strings.html

Take a look at that posting, and you may be able to get around using a sitecustomize file as well.

Kevin

This is actually the part where I’m not sure what best practice is, because I’m under the impression that in fact you *can’t* add sitecustomize.py anywhere in your path, because of some programming oddities. These are mentioned in Martin Doudoroff’s article (which you recently blogged about):

“You want this application to handle multi-lingual text, so you’re going to take advantage of Unicode. The first thing you will probably want to do is set up a sitecustomize.py file in the Lib directory of your python installation and designate a Unicode encoding (probably UTF-8) as the default encoding for Python.

import sys
sys.setdefaultencoding(”utf-8″)

Important: as of Python 2.2, as far as I can tell, you can only call the setdefaultencoding method from within sitecustomize.py. You cannot perform this step from within your application! I don’t understand why Guido set it up this way, but I’m sure he had his reasons. ”

I’m really glad you brought that article to my attention, because it’s the first time I’ve seen someone just come out and say that setting utf-8 as default in sitecustomize.py is the way to go. The number of headaches that it seems to solve, imho, are far greater than the ones involved in editing the file.

I see what you’re saying about creating apps for distribution. That’s something I never really do, though, the only “distribution” I do is by way of cgi apps or something like that.

Thanks for the discussion!

sitecustomize.py is definitely the easiest thing to do when you’re in complete control of what the PYTHONPATH will be at startup.

As for ElementTidy, an encoding attribute was added to the recent 1.0b1 release.