Update the bench_dwt utility to have a -decode/-encode switch
Measured performance gains for DWT encoder on a
Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz (4 cores, hyper threaded)
Encoding time:
$ ./bin/bench_dwt -encode -num_threads 1
time for dwt_encode: total = 8.348 s, wallclock = 8.352 s
$ ./bin/bench_dwt -encode -num_threads 2
time for dwt_encode: total = 9.776 s, wallclock = 4.904 s
$ ./bin/bench_dwt -encode -num_threads 4
time for dwt_encode: total = 13.188 s, wallclock = 3.310 s
$ ./bin/bench_dwt -encode -num_threads 8
time for dwt_encode: total = 30.024 s, wallclock = 4.064 s
Scaling is probably limited by memory access patterns causing
memory access to be the bottleneck.
The slightly worse results with threads==8 than with thread==4
is due to hyperthreading being not appropriate here.
* -PLT switch added to opj_compress
* Add a opj_encoder_set_extra_options() function that
accepts a PLT=YES option, and could be expanded later
for other uses.
-------
Testing with a Sentinel2 10m band, T36JTT_20160914T074612_B02.jp2,
coming from S2A_MSIL1C_20160914T074612_N0204_R135_T36JTT_20160914T081456.SAFE
Decompress it to TIFF:
```
opj_uncompress -i T36JTT_20160914T074612_B02.jp2 -o T36JTT_20160914T074612_B02.tif
```
Recompress it with similar parameters as original:
```
opj_compress -n 5 -c [256,256],[256,256],[256,256],[256,256],[256,256] -t 1024,1024 -PLT -i T36JTT_20160914T074612_B02.tif -o T36JTT_20160914T074612_B02_PLT.jp2
```
Dump codestream detail with GDAL dump_jp2.py utility (https://github.com/OSGeo/gdal/blob/master/gdal/swig/python/samples/dump_jp2.py)
```
python dump_jp2.py T36JTT_20160914T074612_B02.jp2 > /tmp/dump_sentinel2_ori.txt
python dump_jp2.py T36JTT_20160914T074612_B02_PLT.jp2 > /tmp/dump_sentinel2_openjpeg_plt.txt
```
The diff between both show very similar structure, and identical number of packets in PLT markers
Now testing with Kakadu (KDU803_Demo_Apps_for_Linux-x86-64_200210)
Full file decompression:
```
kdu_expand -i T36JTT_20160914T074612_B02_PLT.jp2 -o tmp.tif
Consumed 121 tile-part(s) from a total of 121 tile(s).
Consumed 80,318,806 codestream bytes (excluding any file format) = 5.329697
bits/pel.
Processed using the multi-threaded environment, with
8 parallel threads of execution
```
Partial decompresson (presumably using PLT markers):
```
kdu_expand -i T36JTT_20160914T074612_B02.jp2 -o tmp.pgm -region "{0.5,0.5},{0.01,0.01}"
kdu_expand -i T36JTT_20160914T074612_B02_PLT.jp2 -o tmp2.pgm -region "{0.5,0.5},{0.01,0.01}"
diff tmp.pgm tmp2.pgm && echo "same !"
```
-------
Funded by ESA for S2-MPC project
Previously the multiple component transformation SGcod(C)
and wavelet transformation SPcod(H)/SPcoc(E) parameter
values were never checked, allowing for out of range values.
The lack of validation allowed the bit stream provided in
issue #1158 through. After this commit an error message
points to the marker segments' parameters as being out of
range.
input/nonregression/edf_c2_20.jp2 contains an SPcod(H) value
of 17, but according to Table A-20 of the specification only
values 0 and 1 are valid. input/nonregression/issue826.jp2
contains a SGcod(B) value of 2, but according to Table A-17
of the specification only values 0 and 1 are valid.
input/nonregression/oss-fuzz2785.jp2 contains a SGcod(B)
value of 32, but it is likewise limited to 0 or 1. These test
cases have been updated to consistently fail to parse the
headers since they contain out of bounds values.
This fixes issue #1210.
width/length dimensions read from bmp headers are not necessarily
valid. For instance they may have been maliciously set to very large
values with the intention to cause DoS (large memory allocation, stack
overflow). In these cases we want to detect the invalid size as early
as possible.
This commit introduces a counter which verifies that the number of
written bytes corresponds to the advertized width/length.
See commit 8ee335227b for details.
Signed-off-by: Young Xiao <YangX92@hotmail.com>
Fixes#1053 / CVE-2018-5727
Note: I don't consider this issue to be a security vulnerability, in
practice.
At least with gcc or clang compilers on x86_64 which generate the same
assembly code with or without that fix.
Previously the caller had to check whether each component data had
been decoded. This means duplicating the checking in every user of
openjpeg which is unnecessary. If the caller wantes to decode all
or a set of, or a specific component then openjpeg ought to error
out if it was unable to do so.
Fixes#1158.
width/length dimensions read from bmp headers are not necessarily
valid. For instance they may have been maliciously set to very large
values with the intention to cause DoS (large memory allocation, stack
overflow). In these cases we want to detect the invalid size as early
as possible.
This commit introduces a counter which verifies that the number of
written bytes corresponds to the advertized width/length.
Fixes#1059 (CVE-2018-6616).
When compressing a lot of slices (starting from 44 FullHD slices with 3 8bit components in our experiments) the rate values are high enough to cause an int overflow that leads to negative lengths and wrong results. The cast happens too late.
Tile components in a JP2 image might have null data pointer by defining a
zero component size (for example using large horizontal or vertical
sampling periods). This null data pointer leads to null image component
data pointer, causing crash when dereferenced without != null check in
imagetopnm.
Add != null check.
This commit addresses #1152 (CVE-2018-18088).
* Fix some potential overflow issues
Put sizeof to the beginning of the multiplication to enforce that
size_t instead of smaller integer types is used for the calculation.
This fixes warnings from LGTM:
Multiplication result may overflow 'unsigned int'
before it is converted to 'unsigned long'.
It also allows removing some type casts.
Signed-off-by: Stefan Weil <sw@weilnetz.de>
* Fix code indentation
Signed-off-by: Stefan Weil <sw@weilnetz.de>
This uses snprintf() with correct buffer length instead of sprintf(), which
prevents a buffer overflow when providing a long output prefix. Furthermore
the program exits with an error when the provided output prefix is too long.
Fixes#1088.
In the case where a BMP file declares compression 3 (BI_BITFIELDS)
with header size <= 56, all bitmask values keep their initialization
value 0. This may lead to various undefined behavior later e.g. when
doing 1 << (l_comp->prec - 1).
This issue does not affect files with bit count 16 because of a check
added in 16240e2 which sets default values to the color masks if they
are all 0.
This commit adds similar checks for the 32 bit case.
Also, if a BMP file declares compression 3 with header size >= 56 and
intentional 0 bitmasks, the same issue will be triggered in both the
16 and 32 bit count case.
This commit adds checks to bmp_read_info_header() rejecting BMP files
with "intentional" 0 bitmasks. These checks might be removed in the
future when proper handling of zero bitmasks will be available in
openjpeg2.
fixes#1057 (CVE-2018-5785)
Compiler warnings:
src/lib/openjp2/jp2.c:1008:35: warning:
too many arguments for format [-Wformat-extra-args]
src/lib/openjp2/j2k.c:1928:73: warning:
format ‘%d’ expects argument of type ‘int’, but argument 4 has type ‘OPJ_OFF_T {aka long int}’ [-Wformat=]
Signed-off-by: Stefan Weil <sw@weilnetz.de>
ssize_t is a POSIX type which is declared in POSIX include files.
Mingw-w64 provides it also for Windows.
Use the local declaration only with MS compilers.
Signed-off-by: Stefan Weil <sw@weilnetz.de>
This adds the appropriate variables to the invocation of fprintf(). They were
specified in the format string, but were missing in the actual call. This
fixes#1074 and #1075.
Coverity report:
CID 179466 (#1 of 1): Resource leak (RESOURCE_LEAK)
93. leaked_storage: Variable name going out of scope leaks the storage it points to.
Signed-off-by: Stefan Weil <sw@weilnetz.de>
This fixes warnings from the clang compiler:
/openjpeg/src/lib/openjp2/sparse_array.h:141:32: warning:
no newline at end of file [-Wnewline-eof]
Signed-off-by: Stefan Weil <sw@weilnetz.de>
This adds a opj_set_decoded_components(opj_codec_t *p_codec,
OPJ_UINT32 numcomps, const OPJ_UINT32* comps_indices) function,
and equivalent "opj_decompress -c compno[,compno]*" option.
When specified, neither the MCT transform nor JP2 channel transformations
will be applied.
Tests added for various combinations of whole image vs tiled-based decoding,
full or reduced resolution, use of decode area or not.
Even Rouault
szukw000
Eduardo Barretto
Robert Ancell
Sebastian Rasmussen
Young Xiao
Hugo Lefeuvre
Even Rouault
ichlubna
Stefan Weil
Nikola Forró
Karol Babioch
Robert Everson
Karol Babioch
setharnold
Campbell Barton
Andrew Murray
Gregory Fiumara
Antonin Descampe