shell pipe "loses" parts of the data, was: help needed to debug Perl script

[Paul Sladen]

On Wed, 17 Oct 2018, M. Fioretti wrote:
> myscript.pl 2> error.log | tee datadump | grep ^csvfinal | sort | cut -c10- >  result.csv
> a) datadump was simply truncated.

  grep --line-buffered

> b) on top of that, grep did not extract all the lines

  grep --text  or  grep --null-data

> a) the volume of input data .. passed .. some threshold

4kB/8kB ?

> b) AND the data also contained, for the first time, non-ascii characters

Use UTF-8, and only UTF-8.

> every comment, and b tip

Please provide a .zip/tarball with the exact actual command being run,
and the exact input/intermediate file, and the exact minimally
source code example that exhibits the issue.

   -Paul