[Bug 1041387] [NEW] CR characters block/confuse Thunderbird body-contains filters

Fri Aug 24 21:58:49 UTC 2012

Public bug reported:

Thunderbird filters do not find some text strings that follow CR (x0d,
carriage return) characters in the body of the email.  This problem
occurs whether the filtering is done interactively or with filters saved
in msgFilterRules.dat using the Tools/Message Filters menu item.

The problem occurs for me in the context of cron jobs that call curl.
curl sends lines to stderr that are terminated with CR characters rather
than LF characters.  This blocks Thunderbird filtering of cron email
based on results that occur after the curl command.

The problem is occuring with Thunderbird 14.0 on Ubuntu 12.04.1 LTS.
The problem can be created all on one machine.  Postfix is the MTA using
virtual Maildir mailboxes, Dovecot is the imap server.

I have recreated the essential problem with the attached files.  There
are four versions of the same email body.

msg1-withCRline.txt is the worst offender and has the following contents:
=============== file starts on next line ============
FINDTARGETBEFOREmore
1\r2\r3
FINDOTHERextra
FINDTARGETAFTERstuff
=============== file ends on previous line ============
where the above text lines are terminated by a LF character (x0a, line feed) and the sequence '\r' denotes the CR character (x0d, carriage return).  Some programs recognize the CR character as a line termination character, others do not.  Body-contains filters will find 'TARGETBEFORE' but will not find 'OTHER' or 'TARGETAFTER'.

msg2-trCR2plus.txt is the same as msg1-withCRline.txt, but with the CR
characters replaced by '+' characters.  Body-contains filters work
correctly with this message body.  They find 'TARGETBEFORE', 'OTHER',
and 'TARGETAFTER'.

msg3-trCR2LF.txt is the same as msg1-withCRline.txt, but with the CR
characters replaced by LF characters.  Body-contains filters work
correctly with this message body.  They find 'TARGETBEFORE', 'OTHER',
and 'TARGETAFTER'.

msg4-shortCRline.txt is the same as msg1-withCRline.txt, but with the
two consecutive characters "\r3" deleted, so there is only one CR
character in the message body.  Body-contains filters will find
'TARGETBEFORE'.  They will also find 'OTHER' but will not find
'TARGETAFTER', which suggests the problem is more complicated than the
other examples show.

sendfiltertest is a simple bash script that will mail each message body using the mail command to a recipient.  It should be called with two arguments, the recipient and an optional test number, for example:
  $./sendfiltertest myemailaddress 7

msgFilterRules.dat is the file containing a stored body-contains filter
that looks for 'TARGETAFTER'.

The procedure for recreating the problem can be as simple as:

1.  Download and extract the above files to a directory.  Make sure the
sendfiltertest file has executable permissions.  Use the chmod command
if it does not.

2.  Start Thunderbird.

3.  Click on the Inbox of the recipient of the test emails.

4.  Open a terminal window and change the working directory using the cd
command to the directory where the above files have been downloaded.

5.  Run the sendfiltertest command.

6.  Return to Thunderbird and scroll/sort the Inbox listing to show the
four messages sent by the sendfiltertest command, each with a subject
that begins with 'Test #ID msgN-', where 'ID' is the second argument
given to the sendfiltertest command and 'N' is a digit 1-4 corresponding
to the message body file name.

7.  Click in the filter text field above the Inbox list of messages,
which initially has the prompt 'Filter these message... <Ctrl+Shift+k>'.

8.  Enter the text (without the quotes) 'targetb'.  All four messages
will be selected by the filter.

9.  Change the filter text to 'targeta'.  Of the four messages, only
msg2 and msg3 will be selected.  Messages msg1 and msg4 will not be
selected.

10. Change the filter text to 'other'.  Of the four messages, only msg2,
msg3, and msg4 will be selected. Message msg1 will not be selected.

My current workaround for this problem is to change the cron job script to redirect the curl stderr to a file and then copy the file to stderr after changing all CR characters to LF characters using the tr command:
  $tr '\r' '\n' < curl-stderr >&2

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: thunderbird 14.0+build1-0ubuntu0.12.04.1
ProcVersionSignature: Ubuntu 3.2.0-29.46-generic-pae 3.2.24
Uname: Linux 3.2.0-29-generic-pae i686
NonfreeKernelModules: wl
AddonCompatCheckDisabled: False
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.24.
ApportVersion: 2.0.1-0ubuntu12
Architecture: i386
ArecordDevices:
 **** List of CAPTURE Hardware Devices ****
 card 0: Intel [HDA Intel], device 0: ALC270 Analog [ALC270 Analog]
   Subdevices: 1/1
   Subdevice #0: subdevice #0
AudioDevicesInUse:
 USER        PID ACCESS COMMAND
 /dev/snd/controlC0:  webdev1    2925 F.... pulseaudio
BuildID: 20120714011110
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
Card0.Amixer.info:
 Card hw:0 'Intel'/'HDA Intel at 0xb4400000 irq 44'
   Mixer name	: 'Intel IbexPeak HDMI'
   Components	: 'HDA:10ec0270,103c1439,00100100 HDA:80862804,80860101,00100000'
   Controls      : 27
   Simple ctrls  : 13
Channel: Unavailable
Date: Fri Aug 24 14:24:55 2012
ExecutablePath: /usr/lib/thunderbird/thunderbird
ForcedLayersAccel: False
IfupdownConfig:
 auto lo
 iface lo inet loopback
InstallationMedia: Ubuntu 11.04 "Natty Narwhal" - Release i386 (20110427.1)
IpRoute:
 default via 192.168.1.254 dev eth0  proto static 
 169.254.0.0/16 dev eth0  scope link  metric 1000 
 192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.65  metric 1
PrefSources: prefs.js
Profiles: Profile0 (Default) - LastVersion=14.0/20120714011110 (In use)
RelatedPackageVersions:
 icedtea-6-plugin  1.2-2ubuntu1.1
 rhythmbox-mozilla 2.96-0ubuntu4.1
 totem-mozilla     3.0.1-0ubuntu21
RunningIncompatibleAddons: False
SourcePackage: thunderbird
UpgradeStatus: Upgraded to precise on 2012-06-02 (82 days ago)
WifiSyslog:
 Aug 24 08:55:32 HP10C dhclient: DHCPREQUEST of 192.168.1.65 on eth0 to 192.168.1.254 port 67
 Aug 24 08:55:32 HP10C dhclient: DHCPACK of 192.168.1.65 from 192.168.1.254
 Aug 24 08:55:32 HP10C dhclient: bound to 192.168.1.65 -- renewal in 34206 seconds.
 Aug 24 12:30:17 HP10C kernel: [870110.921074] [drm:drm_mode_getfb] *ERROR* invalid framebuffer id
 Aug 24 12:39:25 HP10C kernel: [870657.891846] [drm:drm_mode_getfb] *ERROR* invalid framebuffer id
dmi.bios.date: 09/06/2010
dmi.bios.vendor: Hewlett-Packard
dmi.bios.version: F.35
dmi.board.asset.tag: Base Board Asset Tag
dmi.board.name: 1439
dmi.board.vendor: Hewlett-Packard
dmi.board.version: 60.3E
dmi.chassis.asset.tag: Chassis Asset Tag
dmi.chassis.type: 10
dmi.chassis.vendor: Hewlett-Packard
dmi.chassis.version: Chassis Version
dmi.modalias: dmi:bvnHewlett-Packard:bvrF.35:bd09/06/2010:svnHewlett-Packard:pnHPG72NotebookPC:pvr0594110000252710001020100:rvnHewlett-Packard:rn1439:rvr60.3E:cvnHewlett-Packard:ct10:cvrChassisVersion:
dmi.product.name: HP G72 Notebook PC
dmi.product.version: 0594110000252710001020100
dmi.sys.vendor: Hewlett-Packard

** Affects: thunderbird (Ubuntu)
     Importance: Undecided
         Status: New

** Tags: apport-bug i386 precise running-unity

-- 
You received this bug notification because you are a member of Mozilla
Bugs, which is subscribed to thunderbird in Ubuntu.
https://bugs.launchpad.net/bugs/1041387

Title:
  CR characters block/confuse Thunderbird body-contains filters

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/thunderbird/+bug/1041387/+subscriptions