[Bug 1915819] [NEW] 'NoneType' object has no attribute 'encode' in requestReceived() when multipart body doesn't include content-disposition

Tue Feb 16 12:54:42 UTC 2021

Public bug reported:

[impact]

python-twisted errors out with "'NoneType' object has no attribute
'encode' in requestReceived()" when it tries to parse a multipart mime
message and python3.7+ is used. This happens because before commit
cc3fa20 in cpython, cgi.parse_multipart ignored parts without a name
defined in "content-disposition" (or parts without headers for that
matter) but after 3.7+ the return of the function can now contain
NoneType keys, which fail to encode.

[scope]

This bug affects all releases

Fixed upstream with commit 310496249, available since 21.2.0rc1

[test case]

1. Save the following code as webserver.py

from twisted.application.internet import TCPServer
from twisted.application.service import Application
from twisted.web.resource import Resource
from twisted.web.server import Site

class Foo(Resource):
    def render_POST(self, request):
        newdata = request.content.getvalue()
        print(newdata)
        return ''

root = Resource()
root.putChild("foo", Foo())
application = Application("cgi.parse_multipart test")
TCPServer(8080, Site(root)).setServiceParent(application)

2. Save the following code as client.py (python3-httplib2 is required)

#!/usr/bin/env python
import httplib2

def http_request(url, method, body=None, headers=None, insecure=False):
    """Issue an http request."""
    http = httplib2.Http(disable_ssl_certificate_validation=insecure)
    if isinstance(url, bytes):
        url = url.decode("ascii")
    return http.request(url, method, body=body, headers=headers)

url = "http://localhost:8080"
method = "POST"
headers = {'Content-Type': 'multipart/form-data; boundary="8825899812428059282"'}
emptyh = '--8825899812428059282\n\n--8825899812428059282--'

print("== BODY: " + emptyh + "\n")
response, content = http_request(url, method, emptyh, headers)

3. Run the server with "twistd3 -y webserver.py"
4. Run the client
5. twistd will fail to encode the key and will drop this traceback in the log file (twistd.log)

2021-02-16T13:41:39+0100 [_GenericHTTPChannelProtocol,7,127.0.0.1] Unhandled Error
        Traceback (most recent call last):
          File "/usr/lib/python3/dist-packages/twisted/python/log.py", line 103, in callWithLogger
            return callWithContext({"system": lp}, func, *args, **kw)
          File "/usr/lib/python3/dist-packages/twisted/python/log.py", line 86, in callWithContext
            return context.call({ILogContext: newCtx}, func, *args, **kw)
          File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext
            return func(*args,**kw)
        --- <exception caught here> ---
          File "/usr/lib/python3/dist-packages/twisted/internet/posixbase.py", line 614, in _doReadOrWrite
            why = selectable.doRead()
          File "/usr/lib/python3/dist-packages/twisted/internet/tcp.py", line 243, in doRead
            return self._dataReceived(data)
          File "/usr/lib/python3/dist-packages/twisted/internet/tcp.py", line 249, in _dataReceived
            rval = self.protocol.dataReceived(data)
          File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2952, in dataReceived
            return self._channel.dataReceived(data)
          File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2245, in dataReceived
            return basic.LineReceiver.dataReceived(self, data)
          File "/usr/lib/python3/dist-packages/twisted/protocols/basic.py", line 579, in dataReceived
            why = self.rawDataReceived(data)
          File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2252, in rawDataReceived
            self._transferDecoder.dataReceived(data)
          File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 1699, in dataReceived
            finishCallback(data[contentLength:])
          File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2115, in _finishRequestBody
            self.allContentReceived()
          File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2224, in allContentReceived
            req.requestReceived(command, path, version)
          File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 898, in requestReceived
            self.args.update({
          File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 899, in <dictcomp>
            x.encode('iso-8859-1'): \
        builtins.AttributeError: 'NoneType' object has no attribute 'encode'

[regression potential]

This affects the returned dictionaries with non-str keys, which were
discarded in python3.6 or earlier before they reached twisted, so
patching this will make its behavior consistent.

** Affects: twisted (Ubuntu)
     Importance: Undecided
         Status: New

** Tags: sts

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to twisted in Ubuntu.
https://bugs.launchpad.net/bugs/1915819

Title:
  'NoneType' object has no attribute 'encode' in requestReceived() when
  multipart body doesn't include content-disposition

Status in twisted package in Ubuntu:
  New

Bug description:
  [impact]

  python-twisted errors out with "'NoneType' object has no attribute
  'encode' in requestReceived()" when it tries to parse a multipart mime
  message and python3.7+ is used. This happens because before commit
  cc3fa20 in cpython, cgi.parse_multipart ignored parts without a name
  defined in "content-disposition" (or parts without headers for that
  matter) but after 3.7+ the return of the function can now contain
  NoneType keys, which fail to encode.

  [scope]

  This bug affects all releases

  Fixed upstream with commit 310496249, available since 21.2.0rc1

  [test case]

  1. Save the following code as webserver.py

  from twisted.application.internet import TCPServer
  from twisted.application.service import Application
  from twisted.web.resource import Resource
  from twisted.web.server import Site

  class Foo(Resource):
      def render_POST(self, request):
          newdata = request.content.getvalue()
          print(newdata)
          return ''

  root = Resource()
  root.putChild("foo", Foo())
  application = Application("cgi.parse_multipart test")
  TCPServer(8080, Site(root)).setServiceParent(application)

  2. Save the following code as client.py (python3-httplib2 is required)

  #!/usr/bin/env python
  import httplib2

  def http_request(url, method, body=None, headers=None, insecure=False):
      """Issue an http request."""
      http = httplib2.Http(disable_ssl_certificate_validation=insecure)
      if isinstance(url, bytes):
          url = url.decode("ascii")
      return http.request(url, method, body=body, headers=headers)

  url = "http://localhost:8080"
  method = "POST"
  headers = {'Content-Type': 'multipart/form-data; boundary="8825899812428059282"'}
  emptyh = '--8825899812428059282\n\n--8825899812428059282--'

  print("== BODY: " + emptyh + "\n")
  response, content = http_request(url, method, emptyh, headers)

  3. Run the server with "twistd3 -y webserver.py"
  4. Run the client
  5. twistd will fail to encode the key and will drop this traceback in the log file (twistd.log)

  2021-02-16T13:41:39+0100 [_GenericHTTPChannelProtocol,7,127.0.0.1] Unhandled Error
          Traceback (most recent call last):
            File "/usr/lib/python3/dist-packages/twisted/python/log.py", line 103, in callWithLogger
              return callWithContext({"system": lp}, func, *args, **kw)
            File "/usr/lib/python3/dist-packages/twisted/python/log.py", line 86, in callWithContext
              return context.call({ILogContext: newCtx}, func, *args, **kw)
            File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
              return self.currentContext().callWithContext(ctx, func, *args, **kw)
            File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext
              return func(*args,**kw)
          --- <exception caught here> ---
            File "/usr/lib/python3/dist-packages/twisted/internet/posixbase.py", line 614, in _doReadOrWrite
              why = selectable.doRead()
            File "/usr/lib/python3/dist-packages/twisted/internet/tcp.py", line 243, in doRead
              return self._dataReceived(data)
            File "/usr/lib/python3/dist-packages/twisted/internet/tcp.py", line 249, in _dataReceived
              rval = self.protocol.dataReceived(data)
            File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2952, in dataReceived
              return self._channel.dataReceived(data)
            File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2245, in dataReceived
              return basic.LineReceiver.dataReceived(self, data)
            File "/usr/lib/python3/dist-packages/twisted/protocols/basic.py", line 579, in dataReceived
              why = self.rawDataReceived(data)
            File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2252, in rawDataReceived
              self._transferDecoder.dataReceived(data)
            File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 1699, in dataReceived
              finishCallback(data[contentLength:])
            File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2115, in _finishRequestBody
              self.allContentReceived()
            File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2224, in allContentReceived
              req.requestReceived(command, path, version)
            File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 898, in requestReceived
              self.args.update({
            File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 899, in <dictcomp>
              x.encode('iso-8859-1'): \
          builtins.AttributeError: 'NoneType' object has no attribute 'encode'

  [regression potential]

  This affects the returned dictionaries with non-str keys, which were
  discarded in python3.6 or earlier before they reached twisted, so
  patching this will make its behavior consistent.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/twisted/+bug/1915819/+subscriptions