[Bug 1915819] Re: 'NoneType' object has no attribute 'encode' in requestReceived() when multipart body doesn't include content-disposition

Thu Feb 18 22:18:39 UTC 2021

** Tags added: sts-sponsor-ddstreet

** Changed in: twisted (Ubuntu Hirsute)
     Assignee: (unassigned) => Victor Tapia (vtapia)

** Changed in: twisted (Ubuntu Groovy)
     Assignee: (unassigned) => Victor Tapia (vtapia)

** Changed in: twisted (Ubuntu Focal)
     Assignee: (unassigned) => Victor Tapia (vtapia)

** Changed in: twisted (Ubuntu Hirsute)
   Importance: Undecided => Medium

** Changed in: twisted (Ubuntu Groovy)
   Importance: Undecided => Medium

** Changed in: twisted (Ubuntu Focal)
   Importance: Undecided => Medium

** Changed in: twisted (Ubuntu Hirsute)
       Status: New => In Progress

** Changed in: twisted (Ubuntu Groovy)
       Status: New => In Progress

** Changed in: twisted (Ubuntu Focal)
       Status: New => In Progress

-- 
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1915819

Title:
  'NoneType' object has no attribute 'encode' in requestReceived() when
  multipart body doesn't include content-disposition

Status in twisted package in Ubuntu:
  In Progress
Status in twisted source package in Focal:
  In Progress
Status in twisted source package in Groovy:
  In Progress
Status in twisted source package in Hirsute:
  In Progress

Bug description:
  [impact]

  python-twisted errors out with "'NoneType' object has no attribute
  'encode' in requestReceived()" when it tries to parse a multipart mime
  message and python3.7+ is used. This happens because before commit
  cc3fa20 in cpython, cgi.parse_multipart ignored parts without a name
  defined in "content-disposition" (or parts without headers for that
  matter) but after 3.7+ the return of the function can now contain
  NoneType keys, which fail to encode.

  [scope]

  Even though this bug affects all python3-twisted releases, I'll
  backport the fix just to Focal, Groovy and Hirsute. Bionic and Xenial
  do not have Python 3.7 as the default interpreter (required to trigger
  the issue), and the delta in python3-twisted might be to big to
  backport as the current packages do not contemplate _PY37PLUS at all.

  Fixed upstream with commit 310496249, available since 21.2.0rc1

  [test case]

  1. Save the following code as webserver.py

  from twisted.application.internet import TCPServer
  from twisted.application.service import Application
  from twisted.web.resource import Resource
  from twisted.web.server import Site

  class Foo(Resource):
      def render_POST(self, request):
          newdata = request.content.getvalue()
          print(newdata)
          return ''

  root = Resource()
  root.putChild("foo", Foo())
  application = Application("cgi.parse_multipart test")
  TCPServer(8080, Site(root)).setServiceParent(application)

  2. Save the following code as client.py (python3-httplib2 is required)

  #!/usr/bin/env python
  import httplib2

  def http_request(url, method, body=None, headers=None, insecure=False):
      """Issue an http request."""
      http = httplib2.Http(disable_ssl_certificate_validation=insecure)
      if isinstance(url, bytes):
          url = url.decode("ascii")
      return http.request(url, method, body=body, headers=headers)

  url = "http://localhost:8080"
  method = "POST"
  headers = {'Content-Type': 'multipart/form-data; boundary="8825899812428059282"'}
  emptyh = '--8825899812428059282\n\n--8825899812428059282--'

  print("== BODY: " + emptyh + "\n")
  response, content = http_request(url, method, emptyh, headers)

  3. Run the server with "twistd3 -y webserver.py"
  4. Run the client
  5. twistd will fail to encode the key and will drop this traceback in the log file (twistd.log)

  2021-02-16T13:41:39+0100 [_GenericHTTPChannelProtocol,7,127.0.0.1] Unhandled Error
          Traceback (most recent call last):
            File "/usr/lib/python3/dist-packages/twisted/python/log.py", line 103, in callWithLogger
              return callWithContext({"system": lp}, func, *args, **kw)
            File "/usr/lib/python3/dist-packages/twisted/python/log.py", line 86, in callWithContext
              return context.call({ILogContext: newCtx}, func, *args, **kw)
            File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
              return self.currentContext().callWithContext(ctx, func, *args, **kw)
            File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext
              return func(*args,**kw)
          --- <exception caught here> ---
            File "/usr/lib/python3/dist-packages/twisted/internet/posixbase.py", line 614, in _doReadOrWrite
              why = selectable.doRead()
            File "/usr/lib/python3/dist-packages/twisted/internet/tcp.py", line 243, in doRead
              return self._dataReceived(data)
            File "/usr/lib/python3/dist-packages/twisted/internet/tcp.py", line 249, in _dataReceived
              rval = self.protocol.dataReceived(data)
            File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2952, in dataReceived
              return self._channel.dataReceived(data)
            File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2245, in dataReceived
              return basic.LineReceiver.dataReceived(self, data)
            File "/usr/lib/python3/dist-packages/twisted/protocols/basic.py", line 579, in dataReceived
              why = self.rawDataReceived(data)
            File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2252, in rawDataReceived
              self._transferDecoder.dataReceived(data)
            File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 1699, in dataReceived
              finishCallback(data[contentLength:])
            File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2115, in _finishRequestBody
              self.allContentReceived()
            File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2224, in allContentReceived
              req.requestReceived(command, path, version)
            File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 898, in requestReceived
              self.args.update({
            File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 899, in <dictcomp>
              x.encode('iso-8859-1'): \
          builtins.AttributeError: 'NoneType' object has no attribute 'encode'

  [regression potential]

  This affects the returned dictionaries with non-str keys, which were
  discarded in python3.6 or earlier before they reached twisted, so
  patching this will make its behavior consistent.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/twisted/+bug/1915819/+subscriptions