Commit Graph

2972 Commits

Author SHA1 Message Date
Even Rouault 47943daa15
Speed-up 9x7 IDWD by ~20%
"bench_dwt -I" time goes from 2.8s to 2.2s
2020-05-21 15:42:51 +02:00
Even Rouault 0c09062464
bench_dwt.c: add a -I switch to test irreversible FWDT/IDWT 2020-05-20 23:20:48 +02:00
Even Rouault 19ef7f26c4
Merge pull request #1211 from sebras/master
Add check to validate SGcod/SPcoc/SPcod parameter values.
2020-05-20 21:10:55 +02:00
Even Rouault adccbc8336
Irreversible decoding: partially revert previous commit, to fix failures in test suite 2020-05-20 20:31:28 +02:00
Even Rouault 3cd1305596
Irreversible compression/decompression DWT: use 1/K constant as per standard
The previous constant opj_c13318 was mysteriously equal to 2/K , and in
the DWT, we had to divide K and opj_c13318 by 2... The issue was that the
band->stepsize computation in tcd.c didn't take into account the log2gain of
the band.

The effect of this change is expected to be mostly equivalent to the previous
situation, except some difference in rounding. But it leads to a dramatic
reduction of the mean square error and peak error in the irreversible encoding
of issue141.tif !
2020-05-20 20:31:28 +02:00
Even Rouault f38c069547
Irreversible decoding: align code more closely to the standard by avoid messing up with stepsize (no functional change) 2020-05-20 20:31:28 +02:00
Even Rouault e46e300de5
opj_dwt_encode_1_real(): avoid many bound comparisons, similarly to decoding side 2020-05-20 20:31:28 +02:00
Even Rouault 4ab2ed0907
opj_j2k_setup_encoder(): add validation of tile width and height to avoid potential division by zero 2020-05-20 20:31:28 +02:00
Even Rouault c6a413a423
opj_mct_encode_real(): add SSE optimization 2020-05-20 20:31:28 +02:00
Even Rouault fe4c15f12c
Testing: revise testing of lossy encoding by comparing PEAK and MSE with original image 2020-05-20 20:31:28 +02:00
Even Rouault c2b9d09c65
compare_images.c: code reformatting 2020-05-20 20:31:28 +02:00
Even Rouault 3d35d0f3af
tcd.c: add comment 2020-05-20 20:31:28 +02:00
Even Rouault 00cff6f5c0
Encoder: use floating-point operations for irreversible transformation 2020-05-20 20:31:28 +02:00
Even Rouault 99107d5e46
dwt.c: change sign of constants to match standard and compensate (no functional change) 2020-05-20 20:31:28 +02:00
Even Rouault 07d1f775a1
Add multithreaded support in the DWT encoder.
Update the bench_dwt utility to have a -decode/-encode switch

Measured performance gains for DWT encoder on a
Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz (4 cores, hyper threaded)

Encoding time:
$ ./bin/bench_dwt -encode -num_threads 1
time for dwt_encode: total = 8.348 s, wallclock = 8.352 s

$ ./bin/bench_dwt -encode -num_threads 2
time for dwt_encode: total = 9.776 s, wallclock = 4.904 s

$ ./bin/bench_dwt -encode -num_threads 4
time for dwt_encode: total = 13.188 s, wallclock = 3.310 s

$ ./bin/bench_dwt -encode -num_threads 8
time for dwt_encode: total = 30.024 s, wallclock = 4.064 s

Scaling is probably limited by memory access patterns causing
memory access to be the bottleneck.
The slightly worse results with threads==8 than with thread==4
is due to hyperthreading being not appropriate here.
2020-05-20 20:30:21 +02:00
Even Rouault 97eb7e0bf1
Add multithreading support in the T1 (entropy phase) encoder
- API wise, opj_codec_set_threads() can be used on the encoding side
- opj_compress has a -threads switch similar to opj_uncompress
2020-05-20 20:30:21 +02:00
Even Rouault 1d358f25c8
Merge pull request #1246 from rouault/write_plt
Add support for generation of PLT markers in encoder
2020-05-20 20:29:31 +02:00
Even Rouault 4edb8c8337
Add support for generation of PLT markers in encoder
* -PLT switch added to opj_compress
* Add a opj_encoder_set_extra_options() function that
  accepts a PLT=YES option, and could be expanded later
  for other uses.

-------

Testing with a Sentinel2 10m band, T36JTT_20160914T074612_B02.jp2,
coming from S2A_MSIL1C_20160914T074612_N0204_R135_T36JTT_20160914T081456.SAFE

Decompress it to TIFF:
```
opj_uncompress -i T36JTT_20160914T074612_B02.jp2 -o T36JTT_20160914T074612_B02.tif
```

Recompress it with similar parameters as original:
```
opj_compress -n 5 -c [256,256],[256,256],[256,256],[256,256],[256,256] -t 1024,1024 -PLT -i T36JTT_20160914T074612_B02.tif -o T36JTT_20160914T074612_B02_PLT.jp2
```

Dump codestream detail with GDAL dump_jp2.py utility (https://github.com/OSGeo/gdal/blob/master/gdal/swig/python/samples/dump_jp2.py)
```
python dump_jp2.py T36JTT_20160914T074612_B02.jp2 > /tmp/dump_sentinel2_ori.txt
python dump_jp2.py T36JTT_20160914T074612_B02_PLT.jp2 > /tmp/dump_sentinel2_openjpeg_plt.txt
```

The diff between both show very similar structure, and identical number of packets in PLT markers

Now testing with Kakadu (KDU803_Demo_Apps_for_Linux-x86-64_200210)

Full file decompression:
```
kdu_expand -i T36JTT_20160914T074612_B02_PLT.jp2 -o tmp.tif

Consumed 121 tile-part(s) from a total of 121 tile(s).
Consumed 80,318,806 codestream bytes (excluding any file format) = 5.329697
bits/pel.
Processed using the multi-threaded environment, with
    8 parallel threads of execution
```

Partial decompresson (presumably using PLT markers):
```
kdu_expand -i T36JTT_20160914T074612_B02.jp2 -o tmp.pgm -region "{0.5,0.5},{0.01,0.01}"
kdu_expand -i T36JTT_20160914T074612_B02_PLT.jp2 -o tmp2.pgm  -region "{0.5,0.5},{0.01,0.01}"
diff tmp.pgm tmp2.pgm && echo "same !"
```

-------

Funded by ESA for S2-MPC project
2020-04-21 15:55:44 +02:00
Even Rouault 64689d05df
struct opj_j2k: remove unused fields, and add some documentation 2020-04-18 18:25:44 +02:00
Even Rouault 774889a328
Merge pull request #1244 from rouault/fix_pi_warnings
Fix warnings about signed/unsigned casts in pi.c
2020-04-17 00:39:46 +02:00
szukw000 b6b7e96b0c
color_apply_icc_profile: add checks on the number of components (#1236) 2020-04-17 00:37:33 +02:00
Eduardo Barretto 040e142288
jp3d/jpwl/mj2/jpip: Fix resource leaks (#1226)
This issues were found by cppcheck and coverity.
2020-04-17 00:09:40 +02:00
Even Rouault 271a71ef0f
Fix warnings about signed/unsigned casts in pi.c 2020-04-16 23:34:10 +02:00
Even Rouault 221a801a97
Rename mis-named function opj_tcd_get_encoded_tile_size() to opj_tcd_get_encoder_input_buffer_size() 2020-04-16 20:33:22 +02:00
Even Rouault 9c1cfb034a
Merge pull request #1240 from rouault/fix_crash_opj_decompress
opj_decompress: add sanity checks to avoid segfault in case of decoding error
2020-04-01 22:00:19 +02:00
Even Rouault 1c54024165
opj_decompress: add sanity checks to avoid segfault in case of decoding error
Prevent crashes like:
opj_decompress -i 0722_5-1_2019.jp2 -o out.ppm -r 4 -t 0

where 0722_5-1_2019.jp2 is
https://drive.google.com/file/d/1ZxOUZg2-FKjYwa257VFLMpTXRWxEoP0a/view?usp=sharing
2020-04-01 21:11:36 +02:00
Even Rouault 563ecfb55c
opj_compress: improve help message regarding new IMF switch 2020-02-13 09:59:36 +01:00
Even Rouault 4e5501b3c7
Merge pull request #1235 from rouault/imf
Implement writing of IMF profiles
2020-02-13 09:54:20 +01:00
Even Rouault 84f3bebbff
Implement writing of IMF profiles
Add -IMF switch to opj_compress as well
2020-02-12 15:55:25 +01:00
Even Rouault fffe32adcb
openjpeg.h: fix values of OPJ_PROFILE_IMF_ constants 2020-02-12 15:55:02 +01:00
Even Rouault 28881453f6 Merge pull request #1234 from rouault/md5_libtiff_4_1
tests: add alternate checksums for libtiff 4.1
2020-02-10 11:20:20 +01:00
Even Rouault b5cb419faf
tests: add alternate checksums for libtiff 4.1
Fixes #1233

libtiff 4.1 slightly modifies the way it generates files. So
add the new expected md5sum.

Not super elegant solution admitedly.
2020-02-07 22:05:55 +01:00
Even Rouault 647f9b118d
Merge pull request #1232 from rouault/fix_1231
opj_tcd_init_tile(): avoid integer overflow
2020-01-30 13:07:31 +01:00
Even Rouault 05f9b91e60
opj_tcd_init_tile(): avoid integer overflow
That could lead to later assertion failures.

Fixes #1231 / CVE-2020-8112
2020-01-30 00:59:57 +01:00
Max Moroz b63a433ba1 tests/fuzzers: link fuzz binaries using $LIB_FUZZING_ENGINE. (#1230)
This was changed some time ago (https://google.github.io/oss-fuzz/getting-started/new-project-guide/) but the build didn't fail as there is a fallback mechanism. The main advantage of the new approach is that for libFuzzer this produces more performant binaries (as `$LIB_FUZZING_ENGINE` expands into `-fsanitize=fuzzer`, which links libFuzzer from the compiler-rt, allowing better optimization tricks).

I'm also experimenting with dataflow (https://github.com/google/oss-fuzz/issues/1632) on your project, and the dataflow config doesn't have a fallback (as it's a new configuration), therefore I'm proposing a change to migrate from `-lFuzzingEngine` to `$LIB_FUZZING_ENGINE`.
2020-01-13 18:07:54 +01:00
Even Rouault 46c1eff9e9
Merge pull request #1229 from rouault/fix_1228
opj_j2k_update_image_dimensions(): reject images whose coordinates are beyond INT_MAX (fixes #1228)
2020-01-11 11:29:11 +01:00
Even Rouault 024b840739
opj_j2k_update_image_dimensions(): reject images whose coordinates are beyond INT_MAX (fixes #1228) 2020-01-11 01:51:58 +01:00
Even Rouault ac3737372a
Merge pull request #1217 from rouault/fix_ossfuzz_18979
pi.c: avoid integer overflow, resulting in later invalid access to memory in opj_t2_decode_packets()
2019-11-17 13:08:41 +01:00
Robert Ancell 9701b3305d JPWL: convert: Fix buffer overflow reading an image file less than four characters (#1196)
Fixes #1068
2019-11-17 03:09:59 +01:00
Even Rouault cb332992a7
Merge pull request #1218 from rouault/fix_broken_abi_check
abi-check.sh: fix false postive ABI error, and display output error log
2019-11-17 02:47:26 +01:00
Even Rouault 016f80ae21
abi-check.sh: fix false postive ABI error, and display output error log
There is currently a false positive ABI check failure between v2.3.1
and current. It disappears when removing the generated reports of v2.3.1
and recreating them. It is likely that some tooling has evolved since
the initial v2.3.1 report generation.
2019-11-17 02:26:54 +01:00
Even Rouault 4cb1f66304
pi.c: avoid integer overflow, resulting in later invalid access to memory in opj_t2_decode_packets(). Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=18979 2019-11-17 01:18:26 +01:00
Even Rouault 5875a6b446
opj_tcd_mct_decode()/opj_mct_decode()/opj_mct_encode_real()/opj_mct_decode_real(): proper deal with a number of samples larger than 4 billion (refs #1151) 2019-10-03 11:04:30 +02:00
Sebastian Rasmussen f3ee448815 openjp2/j2k: Validate all SGcod/SPcod/SPcoc parameter values.
Previously the multiple component transformation SGcod(C)
and wavelet transformation SPcod(H)/SPcoc(E) parameter
values were never checked, allowing for out of range values.

The lack of validation allowed the bit stream provided in
issue #1158 through. After this commit an error message
points to the marker segments' parameters as being out of
range.

input/nonregression/edf_c2_20.jp2 contains an SPcod(H) value
of 17, but according to Table A-20 of the specification only
values 0 and 1 are valid. input/nonregression/issue826.jp2
contains a SGcod(B) value of 2, but according to Table A-17
of the specification only values 0 and 1 are valid.
input/nonregression/oss-fuzz2785.jp2 contains a SGcod(B)
value of 32, but it is likewise limited to 0 or 1. These test
cases have been updated to consistently fail to parse the
headers since they contain out of bounds values.

This fixes issue #1210.
2019-09-04 05:14:41 +02:00
Sebastian Rasmussen d801bd4e62 openjp2/j2k: Make comments adhere to specification.
The function is used to read both SPcod and SPcoc, so all
comments should refer to both marker segments' parameter names.
2019-09-04 05:14:30 +02:00
Even Rouault e66125fe26
Merge pull request #1164 from sebras/master
openjp2/j2k: Report error if all wanted components are not decoded.
2019-09-03 17:03:54 +02:00
Even Rouault 8db9d25dcf
opj_decompress_fuzzer: remove checks regarding input dimensions (fixes #1079) 2019-06-15 09:55:16 +02:00
Even Rouault f4d6578359
test_decode_area.c: assign tdy to *ptileh instead of *ptilew (fixes #1195) 2019-05-26 11:06:30 +02:00
Even Rouault 9b7620ee7a
Merge pull request #1185 from Young-X/fix
Fix several potential vulnerabilities
2019-04-26 19:52:52 +02:00
Even Rouault 4f447c6e18
Merge pull request #1192 from rouault/poc_fixes
compression: emit POC marker when only one single POC is requested (f…
2019-04-25 15:32:22 +02:00