I've been wracking my brain over this for the last few days now and haven't been able to find much in the way of help for my specific issue. Here's my setup.
Windows 7
IIS 7.5
Helicon Zoo 3.1.98.508
Django 1.6.3
Python 2.7.3
I should state that I'm new to Django and Helicon Zoo.
What's happening is that while I seem to have no issue uploading a file under the default of 2.5mb, I cannot get chunks to upload correctly for anything over this amount. While I'm sure I could up this limit in my settings file, I'd really rather not as for my purpose I could see needing to upload larger files than what I would really want being handled by memory.
What I'm seeing when I attempt this is a rather uninformative Error 500 from Helicon Zoo about some missing headers
Worker Status
The process was created
Windows error
The operation completed successfully. (ERROR CODE: 0)
Internal module error
message: HTTP-headers - are expected
type: ZooException
file: Jobs\JobFastCgi.cpp
line: 391
version: 3.1.98.508
STDERR
Empty stderr
At first, I thought maybe it was something to do with the temp folder so I set
FILE_UPLOAD_TEMP_DIR = [os.path.join(BASE_DIR, 'temp')]
in my settings file, but it didn't seem to do anything useful.
I then discovered how to enable logging for django which really opened things up to me. I know now where the error is coming from.. but I have no clue what I'm supposed to do to get around it..
[29/Apr/2014 11:37:00] ERROR [django.request:226] Internal Server Error: /upload/
Traceback (most recent call last):
File "E:\mysite\venv\lib\site-packages\django\core\handlers\base.py", line 107, in get_response
response = middleware_method(request, callback, callback_args, callback_kwargs)
File "E:\mysite\venv\lib\site-packages\django\middleware\csrf.py", line 170, in process_view
request_csrf_token = request.POST.get('csrfmiddlewaretoken', '')
File "E:\mysite\venv\lib\site-packages\django\core\handlers\wsgi.py", line 146, in _get_post
self._load_post_and_files()
File "E:\mysite\venv\lib\site-packages\django\http\request.py", line 215, in _load_post_and_files
self._post, self._files = self.parse_file_upload(self.META, data)
File "E:\mysite\venv\lib\site-packages\django\http\request.py", line 180, in parse_file_upload
return parser.parse()
File "E:\mysite\venv\lib\site-packages\django\http\multipartparser.py", line 197, in parse
charset)
File "E:\mysite\venv\lib\site-packages\django\core\files\uploadhandler.py", line 135, in new_file
self.file = TemporaryUploadedFile(self.file_name, self.content_type, 0, self.charset)
File "E:\mysite\venv\lib\site-packages\django\core\files\uploadedfile.py", line 61, in __init__
dir=settings.FILE_UPLOAD_TEMP_DIR)
File "E:\mysite\venv\lib\site-packages\django\core\files\temp.py", line 27, in __init__
dir=dir)
File "C:\python27\lib\tempfile.py", line 300, in mkstemp
return _mkstemp_inner(dir, prefix, suffix, flags)
File "C:\python27\lib\tempfile.py", line 235, in _mkstemp_inner
fd = _os.open(file, flags, 0600)
TypeError: coercing to Unicode: need string or buffer, list found
What does this mean? Are my chunks coming in as a list object somehow and I don't realize it?
Here's my code for the app, it's included in my Blog app I made since it's my main app and it seemed simpler than creating an app all it's own. Was that a mistake?
forms.py
from django import forms
class UploadFileForm(forms.Form):
title = forms.CharField(max_length=50)
file = forms.FileField()
views.py
from django.shortcuts import render, render_to_response
from django.http import HttpResponse, Http404, HttpResponseRedirect
from django.template import RequestContext, loader
from blog.forms import UploadFileForm
from blog.models import Blog
from blog.uploads import handle_uploaded_file
def index(request):
latest_blog_list = Blog.objects.order_by('-pub_date')[:5]
template = loader.get_template('blog/index.html')
context = RequestContext(request, {
'latest_blog_list': latest_blog_list,
})
return HttpResponse(template.render(context))
def detail(request, blog_id):
try:
blog = Blog.objects.get(pk=blog_id)
except:
raise Http404
return render(request, 'blog/detail.html', {'blog': blog})
def upload_file(request):
if request.method == 'POST':
form = UploadFileForm(request.POST, request.FILES)
if form.is_valid():
handle_uploaded_file(request.FILES['file'])
#form.save()
return HttpResponseRedirect('/upload/')
else:
form = UploadFileForm()
return render_to_response('upload/upload.html', {'form': form}, context_instance=RequestContext(request))
uploads.py
import os
saveDir = 'E:\\uploads\\'
def handle_uploaded_file(f):
#logging.debug('upload_here')
if f:
destination = open(saveDir + f.name, 'wb+')
for chunk in f.chunks():
destination.write(chunk)
destination.close()
app urls.py
from django.conf.urls import patterns, url
from blog import views
urlpatterns = patterns('',
url(r'^$', views.index, name='index'),
url(r'^(?P<blog_id>\d+)/$', views.detail, name='detail'),
url(r'^upload/$', views.upload_file, name='upload'),
)
project urls.py
from django.conf.urls import patterns, include, url
from django.conf import settings
from django.conf.urls.static import static
from django.contrib import admin
admin.autodiscover()
urlpatterns = patterns('',
# Examples:
# url(r'^$', 'project.views.home', name='home'),
# url(r'^blog/', include('blog.urls')),
url(r'^/?', include('blog.urls', namespace="blog")),
url(r'^admin/', include(admin.site.urls)),
url(r'^upload/', include('blog.urls', namespace="upload")),
)
I should also note that portions of these large files are actually appearing in my upload location, but nothing more than 1 meg in size. It's like the process stops before it's actually done for some reason. I then have to stop and restart IIS to get the site to even function again.
So there were a couple of different problems here. First off, I discovered that the
TypeError: coercing to Unicode: need string or buffer, list found
error was not because of the object being sent by my code, but rather due to the setting
FILE_UPLOAD_TEMP_DIR = [os.path.join(BASE_DIR, 'temp')]
needing to instead be
FILE_UPLOAD_TEMP_DIR = os.path.join(BASE_DIR, 'temp')
THAT is what was being sent as a list when it shouldn't have been, simple mistake in trying to replicate other fields within the django project settings file. I should have noticed the difference between DIRS and DIR, of course denoting a singular reference.
Secondly, the error continued after resolving that matter. After a couple days of beating my head trying to find maybe an IIS setting that could be truncating my uploads to 1MB in size, or potentially a django setting I was overlooking, and even testing a prebuilt app, only to still hit the same brick wall, it finally dawned on me when reading through the Helicon Zoo error report's list of configuration variables, that I noticed something...
postBuffer = 1024
Son of a... So after a little research I discovered I could modify this setting by using an environment variable inside of my Helicon Zoo web.config.
POST_BUFFER = "4096"
Viola.. I can suddenly upload anything over 1mb in size... anything over 4mb is truncated and I get an error!
So I finally found the issue, but its completely surprising to me that I can't find any sort of documentation regarding this as being an issue. Surely I'm not the only person whose come across this as it's a default setting! I wish I could remove it altogether as even being a limitation since django handles this just fine itself, I really don't need Helicon doing it for me. Unfortunately it looks like if I set the value to 0 then no uploads will work at all. So for now I'm going to set this value to something arbitrarily high just because frankly it's not needed and doesn't help me in any way.
UPDATE: Thanks to rukeba's comments above, his solution was the final answer to this issue as the Twisted engine does not have this problem.