[Bug 672304] Re: Image abnmormally stretched when streaming with transcode activated (Integer overflow in aspect ratio calculation)

[Nicola Ferralis]

** Patch added: "transcode_aspect_ratio_maverick_2.diff"
   https://bugs.launchpad.net/ubuntu/+source/vlc/+bug/672304/+attachment/1738038/+files/transcode_aspect_ratio_maverick_2.diff

-- 
Image abnmormally stretched when streaming with transcode activated (Integer overflow in aspect ratio calculation)
https://bugs.launchpad.net/bugs/672304
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is a direct subscriber.

Status in VLC media player: Fix Released
Status in “vlc” package in Ubuntu: Fix Released
Status in “vlc” source package in Maverick: New
Status in “vlc” package in Debian: New

Bug description:
Binary package hint: vlc

Using VLC 1.1.4-1 on Ubuntu 10.10. When I try to stream and transcode from capturing with a webcam, the image in the stream is always stretched vertically. Note that if I select to stream without transcoding, the image, in either case, looks normal. Please keep in mind that this bug seems to be only reproducible with Ubuntu. In Windows everything works as expected. I tried several webcams, with similar results. As it is, streaming from a capture device like a webcam is completely unusable.

The cause of the bug (a comprehensive description is given in the upstream report) is due to a integer overflow during transcoding. Basically, the aspect ratio for a 640x320 webcam is given in absolute terms, for example 276480000:276480000 which is equivalent to 1:1, but when multiplied by 640 the result is > 32 bits.

The attached patch essentially prevents the overflow by reducing the value of the aspect ratio within 32 bit and by assigning a working value for the aspect ratio of the image after decoding.

TEST CASE to reproduce:
0. Make sure you have a webcam attached.
1. Launch vlc
2. press CTRL+C for capture from the webcam using v4l2
3. press ALT+S for streaming
4. select stream to a file (but http will do also)
5. Important: make sure you select a transcoding option. Anyone is
good, other than OGG (where this bug seem not to apply).
6. Start the stream and wait a few minutes, before turning the stream OFF.

Expected: the file (or http stream) should have an image with the
correct aspect ratio.

Actual: the video is deformed (in my case vertically elongated.

The patch has been extensively tested on test builds on VLC 1.1.4-1ubuntu1 in maverick. No collateral effects are expected.