Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Note: I asked this question on StackOverflow, but I realize this might be a more proper place to ask this kind of question.

I'm trying to upload a file called 'Testaråäö.txt' via the Django admin app.

I'm running Django 1.3.1 with Gunicorn 0.13.4 and Nginx 0.7.6.7 on a Debian 6 server. Database is PostgreSQL 8.4.9. Other Unicode data is saved to the database with no problem, so I guess the problem must be with the filesystem somehow.

I've set

http {
    charset utf-8;
}

in my nginx.conf. LC_ALL and LANG is set to 'sv_SE.UTF-8'. Running 'locale' verifies this. I even tried setting LC_ALL and LANG in my nginx init script just to make sure locale is set properly.

Here's the traceback:

Traceback (most recent call last):

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/core/handlers/base.py", line 111, in get_response
response = callback(request, *callback_args, **callback_kwargs)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/contrib/admin/options.py", line 307, in wrapper
return self.admin_site.admin_view(view)(*args, **kwargs)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/utils/decorators.py", line 93, in _wrapped_view
response = view_func(request, *args, **kwargs)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/views/decorators/cache.py", line 79, in _wrapped_view_func
response = view_func(request, *args, **kwargs)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/contrib/admin/sites.py", line 197, in inner
return view(request, *args, **kwargs)

File "/srv/django/letebo/app/cms/admin.py", line 81, in change_view
return super(PageAdmin, self).change_view(request, obj_id)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/utils/decorators.py", line 28, in _wrapper
return bound_func(*args, **kwargs)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/utils/decorators.py", line 93, in _wrapped_view
response = view_func(request, *args, **kwargs)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/utils/decorators.py", line 24, in bound_func
return func(self, *args2, **kwargs2)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/db/transaction.py", line 217, in inner
res = func(*args, **kwargs)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/contrib/admin/options.py", line 985, in change_view
self.save_formset(request, form, formset, change=True)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/contrib/admin/options.py", line 677, in save_formset
formset.save()

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/forms/models.py", line 482, in save
return self.save_existing_objects(commit) + self.save_new_objects(commit)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/forms/models.py", line 613, in save_new_objects
self.new_objects.append(self.save_new(form, commit=commit))

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/forms/models.py", line 717, in save_new
obj.save()

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/db/models/base.py", line 460, in save
self.save_base(using=using, force_insert=force_insert, force_update=force_update)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/db/models/base.py", line 504, in save_base
self.save_base(cls=parent, origin=org, using=using)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/db/models/base.py", line 543, in save_base
for f in meta.local_fields if not isinstance(f, AutoField)]

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/db/models/fields/files.py", line 255, in pre_save
file.save(file.name, file, save=False)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/db/models/fields/files.py", line 92, in save
self.name = self.storage.save(name, content)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/core/files/storage.py", line 48, in save
name = self.get_available_name(name)

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/core/files/storage.py", line 74, in get_available_name
while self.exists(name):

File "/srv/.virtualenvs/letebo/lib/python2.6/site-packages/django/core/files/storage.py", line 218, in exists
return os.path.exists(self.path(name))

File "/srv/.virtualenvs/letebo/lib/python2.6/genericpath.py", line 18, in exists
st = os.stat(path)

UnicodeEncodeError: 'ascii' codec can't encode characters in position 52-54: ordinal not in range(128)

I tried running Gunicorn with debugging turned on, and the file uploads without any problem at all. I suppose this must mean that the issue is with Nginx. Still beats me where to look, though. Here are the raw response headers from Gunicorn and Nginx, if it makes any sense:

Gunicorn:

HTTP/1.1 302 FOUND
Server: gunicorn/0.13.4
Date: Thu, 09 Feb 2012 14:50:27 GMT
Connection: close
Transfer-Encoding: chunked
Expires: Thu, 09 Feb 2012 14:50:27 GMT
Vary: Cookie
Last-Modified: Thu, 09 Feb 2012 14:50:27 GMT
Location: http://my-server.se:8000/admin/cms/page/15/
Cache-Control: max-age=0
Content-Type: text/html; charset=utf-8
Set-Cookie: messages="yada yada yada"; Path=/

Nginx:

HTTP/1.1 500 INTERNAL SERVER ERROR
Server: nginx/0.7.67
Date: Thu, 09 Feb 2012 14:50:57 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: close
Vary: Cookie

500

UPDATE: Both locale.getpreferredencoding() and sys.getfilesystemencoding() outputs 'UTF-8'. locale.getdefaultlocale() outputs ('sv_SE', 'UTF8'). This seem correct to me, so I'm still not sure why I keep getting these errors.

share|improve this question
    
Check what you get for outputs from locale.getpreferredlocale() and sys.getfilesystemencoding() in your python code - something's not playing the UTF8 game, since it's trying to shoehorn into ASCII for the os.stat() call. –  Shane Madden Feb 14 '12 at 19:06
    
I do have the same exact problem. I hooked some code in django/db/models/fields/files.py to know what is the type of the string name in line 92. and it's a unicode. have you found the fix? –  bclermont Feb 26 '12 at 17:58
add comment

1 Answer 1

I had the same problem and I found the fixes for my cases (I run Ubuntu).

I found a good unanswered thread here, where Karen give some usefull information:

https://groups.google.com/forum/?fromgroups#!topic/django-users/hwNL7i6IeIY

which led me to this:

https://code.djangoproject.com/wiki/ExpectedTestFailures

I installed the missing "language-pack" for Ubuntu. and I ran in a shell as root:

locale-gen en_US.UTF-8

and it's now working.

Look like, in low level layers of Python, something call Glibc which need some compiled files generated by locale-gen to work correctly.

I've looked in http://packages.debian.org and there is no package like this in Debian.

Try to run the following and restart your daemon before trying again:

locale-gen sv_SE.UTF-8

or maybe install any Debian specifics locale/language packages, if there are.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.