Commit Graph

2960 Commits

Author SHA1 Message Date
Even Rouault 00cff6f5c0
Encoder: use floating-point operations for irreversible transformation 2020-05-20 20:31:28 +02:00
Even Rouault 99107d5e46
dwt.c: change sign of constants to match standard and compensate (no functional change) 2020-05-20 20:31:28 +02:00
Even Rouault 07d1f775a1
Add multithreaded support in the DWT encoder.
Update the bench_dwt utility to have a -decode/-encode switch

Measured performance gains for DWT encoder on a
Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz (4 cores, hyper threaded)

Encoding time:
$ ./bin/bench_dwt -encode -num_threads 1
time for dwt_encode: total = 8.348 s, wallclock = 8.352 s

$ ./bin/bench_dwt -encode -num_threads 2
time for dwt_encode: total = 9.776 s, wallclock = 4.904 s

$ ./bin/bench_dwt -encode -num_threads 4
time for dwt_encode: total = 13.188 s, wallclock = 3.310 s

$ ./bin/bench_dwt -encode -num_threads 8
time for dwt_encode: total = 30.024 s, wallclock = 4.064 s

Scaling is probably limited by memory access patterns causing
memory access to be the bottleneck.
The slightly worse results with threads==8 than with thread==4
is due to hyperthreading being not appropriate here.
2020-05-20 20:30:21 +02:00
Even Rouault 97eb7e0bf1
Add multithreading support in the T1 (entropy phase) encoder
- API wise, opj_codec_set_threads() can be used on the encoding side
- opj_compress has a -threads switch similar to opj_uncompress
2020-05-20 20:30:21 +02:00
Even Rouault 1d358f25c8
Merge pull request #1246 from rouault/write_plt
Add support for generation of PLT markers in encoder
2020-05-20 20:29:31 +02:00
Even Rouault 4edb8c8337
Add support for generation of PLT markers in encoder
* -PLT switch added to opj_compress
* Add a opj_encoder_set_extra_options() function that
  accepts a PLT=YES option, and could be expanded later
  for other uses.

-------

Testing with a Sentinel2 10m band, T36JTT_20160914T074612_B02.jp2,
coming from S2A_MSIL1C_20160914T074612_N0204_R135_T36JTT_20160914T081456.SAFE

Decompress it to TIFF:
```
opj_uncompress -i T36JTT_20160914T074612_B02.jp2 -o T36JTT_20160914T074612_B02.tif
```

Recompress it with similar parameters as original:
```
opj_compress -n 5 -c [256,256],[256,256],[256,256],[256,256],[256,256] -t 1024,1024 -PLT -i T36JTT_20160914T074612_B02.tif -o T36JTT_20160914T074612_B02_PLT.jp2
```

Dump codestream detail with GDAL dump_jp2.py utility (https://github.com/OSGeo/gdal/blob/master/gdal/swig/python/samples/dump_jp2.py)
```
python dump_jp2.py T36JTT_20160914T074612_B02.jp2 > /tmp/dump_sentinel2_ori.txt
python dump_jp2.py T36JTT_20160914T074612_B02_PLT.jp2 > /tmp/dump_sentinel2_openjpeg_plt.txt
```

The diff between both show very similar structure, and identical number of packets in PLT markers

Now testing with Kakadu (KDU803_Demo_Apps_for_Linux-x86-64_200210)

Full file decompression:
```
kdu_expand -i T36JTT_20160914T074612_B02_PLT.jp2 -o tmp.tif

Consumed 121 tile-part(s) from a total of 121 tile(s).
Consumed 80,318,806 codestream bytes (excluding any file format) = 5.329697
bits/pel.
Processed using the multi-threaded environment, with
    8 parallel threads of execution
```

Partial decompresson (presumably using PLT markers):
```
kdu_expand -i T36JTT_20160914T074612_B02.jp2 -o tmp.pgm -region "{0.5,0.5},{0.01,0.01}"
kdu_expand -i T36JTT_20160914T074612_B02_PLT.jp2 -o tmp2.pgm  -region "{0.5,0.5},{0.01,0.01}"
diff tmp.pgm tmp2.pgm && echo "same !"
```

-------

Funded by ESA for S2-MPC project
2020-04-21 15:55:44 +02:00
Even Rouault 64689d05df
struct opj_j2k: remove unused fields, and add some documentation 2020-04-18 18:25:44 +02:00
Even Rouault 774889a328
Merge pull request #1244 from rouault/fix_pi_warnings
Fix warnings about signed/unsigned casts in pi.c
2020-04-17 00:39:46 +02:00
szukw000 b6b7e96b0c
color_apply_icc_profile: add checks on the number of components (#1236) 2020-04-17 00:37:33 +02:00
Eduardo Barretto 040e142288
jp3d/jpwl/mj2/jpip: Fix resource leaks (#1226)
This issues were found by cppcheck and coverity.
2020-04-17 00:09:40 +02:00
Even Rouault 271a71ef0f
Fix warnings about signed/unsigned casts in pi.c 2020-04-16 23:34:10 +02:00
Even Rouault 221a801a97
Rename mis-named function opj_tcd_get_encoded_tile_size() to opj_tcd_get_encoder_input_buffer_size() 2020-04-16 20:33:22 +02:00
Even Rouault 9c1cfb034a
Merge pull request #1240 from rouault/fix_crash_opj_decompress
opj_decompress: add sanity checks to avoid segfault in case of decoding error
2020-04-01 22:00:19 +02:00
Even Rouault 1c54024165
opj_decompress: add sanity checks to avoid segfault in case of decoding error
Prevent crashes like:
opj_decompress -i 0722_5-1_2019.jp2 -o out.ppm -r 4 -t 0

where 0722_5-1_2019.jp2 is
https://drive.google.com/file/d/1ZxOUZg2-FKjYwa257VFLMpTXRWxEoP0a/view?usp=sharing
2020-04-01 21:11:36 +02:00
Even Rouault 563ecfb55c
opj_compress: improve help message regarding new IMF switch 2020-02-13 09:59:36 +01:00
Even Rouault 4e5501b3c7
Merge pull request #1235 from rouault/imf
Implement writing of IMF profiles
2020-02-13 09:54:20 +01:00
Even Rouault 84f3bebbff
Implement writing of IMF profiles
Add -IMF switch to opj_compress as well
2020-02-12 15:55:25 +01:00
Even Rouault fffe32adcb
openjpeg.h: fix values of OPJ_PROFILE_IMF_ constants 2020-02-12 15:55:02 +01:00
Even Rouault 28881453f6 Merge pull request #1234 from rouault/md5_libtiff_4_1
tests: add alternate checksums for libtiff 4.1
2020-02-10 11:20:20 +01:00
Even Rouault b5cb419faf
tests: add alternate checksums for libtiff 4.1
Fixes #1233

libtiff 4.1 slightly modifies the way it generates files. So
add the new expected md5sum.

Not super elegant solution admitedly.
2020-02-07 22:05:55 +01:00
Even Rouault 647f9b118d
Merge pull request #1232 from rouault/fix_1231
opj_tcd_init_tile(): avoid integer overflow
2020-01-30 13:07:31 +01:00
Even Rouault 05f9b91e60
opj_tcd_init_tile(): avoid integer overflow
That could lead to later assertion failures.

Fixes #1231 / CVE-2020-8112
2020-01-30 00:59:57 +01:00
Max Moroz b63a433ba1 tests/fuzzers: link fuzz binaries using $LIB_FUZZING_ENGINE. (#1230)
This was changed some time ago (https://google.github.io/oss-fuzz/getting-started/new-project-guide/) but the build didn't fail as there is a fallback mechanism. The main advantage of the new approach is that for libFuzzer this produces more performant binaries (as `$LIB_FUZZING_ENGINE` expands into `-fsanitize=fuzzer`, which links libFuzzer from the compiler-rt, allowing better optimization tricks).

I'm also experimenting with dataflow (https://github.com/google/oss-fuzz/issues/1632) on your project, and the dataflow config doesn't have a fallback (as it's a new configuration), therefore I'm proposing a change to migrate from `-lFuzzingEngine` to `$LIB_FUZZING_ENGINE`.
2020-01-13 18:07:54 +01:00
Even Rouault 46c1eff9e9
Merge pull request #1229 from rouault/fix_1228
opj_j2k_update_image_dimensions(): reject images whose coordinates are beyond INT_MAX (fixes #1228)
2020-01-11 11:29:11 +01:00
Even Rouault 024b840739
opj_j2k_update_image_dimensions(): reject images whose coordinates are beyond INT_MAX (fixes #1228) 2020-01-11 01:51:58 +01:00
Even Rouault ac3737372a
Merge pull request #1217 from rouault/fix_ossfuzz_18979
pi.c: avoid integer overflow, resulting in later invalid access to memory in opj_t2_decode_packets()
2019-11-17 13:08:41 +01:00
Robert Ancell 9701b3305d JPWL: convert: Fix buffer overflow reading an image file less than four characters (#1196)
Fixes #1068
2019-11-17 03:09:59 +01:00
Even Rouault cb332992a7
Merge pull request #1218 from rouault/fix_broken_abi_check
abi-check.sh: fix false postive ABI error, and display output error log
2019-11-17 02:47:26 +01:00
Even Rouault 016f80ae21
abi-check.sh: fix false postive ABI error, and display output error log
There is currently a false positive ABI check failure between v2.3.1
and current. It disappears when removing the generated reports of v2.3.1
and recreating them. It is likely that some tooling has evolved since
the initial v2.3.1 report generation.
2019-11-17 02:26:54 +01:00
Even Rouault 4cb1f66304
pi.c: avoid integer overflow, resulting in later invalid access to memory in opj_t2_decode_packets(). Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=18979 2019-11-17 01:18:26 +01:00
Even Rouault 5875a6b446
opj_tcd_mct_decode()/opj_mct_decode()/opj_mct_encode_real()/opj_mct_decode_real(): proper deal with a number of samples larger than 4 billion (refs #1151) 2019-10-03 11:04:30 +02:00
Sebastian Rasmussen f3ee448815 openjp2/j2k: Validate all SGcod/SPcod/SPcoc parameter values.
Previously the multiple component transformation SGcod(C)
and wavelet transformation SPcod(H)/SPcoc(E) parameter
values were never checked, allowing for out of range values.

The lack of validation allowed the bit stream provided in
issue #1158 through. After this commit an error message
points to the marker segments' parameters as being out of
range.

input/nonregression/edf_c2_20.jp2 contains an SPcod(H) value
of 17, but according to Table A-20 of the specification only
values 0 and 1 are valid. input/nonregression/issue826.jp2
contains a SGcod(B) value of 2, but according to Table A-17
of the specification only values 0 and 1 are valid.
input/nonregression/oss-fuzz2785.jp2 contains a SGcod(B)
value of 32, but it is likewise limited to 0 or 1. These test
cases have been updated to consistently fail to parse the
headers since they contain out of bounds values.

This fixes issue #1210.
2019-09-04 05:14:41 +02:00
Sebastian Rasmussen d801bd4e62 openjp2/j2k: Make comments adhere to specification.
The function is used to read both SPcod and SPcoc, so all
comments should refer to both marker segments' parameter names.
2019-09-04 05:14:30 +02:00
Even Rouault e66125fe26
Merge pull request #1164 from sebras/master
openjp2/j2k: Report error if all wanted components are not decoded.
2019-09-03 17:03:54 +02:00
Even Rouault 8db9d25dcf
opj_decompress_fuzzer: remove checks regarding input dimensions (fixes #1079) 2019-06-15 09:55:16 +02:00
Even Rouault f4d6578359
test_decode_area.c: assign tdy to *ptileh instead of *ptilew (fixes #1195) 2019-05-26 11:06:30 +02:00
Even Rouault 9b7620ee7a
Merge pull request #1185 from Young-X/fix
Fix several potential vulnerabilities
2019-04-26 19:52:52 +02:00
Even Rouault 4f447c6e18
Merge pull request #1192 from rouault/poc_fixes
compression: emit POC marker when only one single POC is requested (f…
2019-04-25 15:32:22 +02:00
Even Rouault a94cfbd533
Change opj_j2k_check_poc_val() to take into account tile number 2019-04-25 15:06:45 +02:00
Even Rouault bdec5ae272
Add test for previous commit 2019-04-25 14:40:56 +02:00
Even Rouault 6423163141
Fix POC in multi-tile scenarios: avoid almost endless loop when a tile has no POC settings 2019-04-25 14:40:56 +02:00
Even Rouault b86717fdd3
Add test for previous commit 2019-04-25 14:40:56 +02:00
Even Rouault 23883458b9
opj_j2k_check_poc_val(): prevent potential write outside of allocated array 2019-04-25 14:40:56 +02:00
Even Rouault 6589c609f6
opj_j2k_check_poc_val(): fix starting index for checking layer dimension
The standard mandates that the layer index always starts at zero for every
progression.
2019-04-25 14:40:55 +02:00
Even Rouault 1e3a57563d
compression: emit POC marker when only one single POC is requested (fixes #1191) 2019-04-25 14:40:55 +02:00
Even Rouault 5dd75f62e2
j2k.c: use correct naming convention for total_data_size variable 2019-04-23 16:52:21 +02:00
Young Xiao 3aef207f90 bmp_read_rle4_data(): avoid potential infinite loop 2019-04-15 16:10:18 +08:00
Young Xiao 21399f6b7d convertbmp: detect invalid file dimensions early
width/length dimensions read from bmp headers are not necessarily
valid. For instance they may have been maliciously set to very large
values with the intention to cause DoS (large memory allocation, stack
overflow). In these cases we want to detect the invalid size as early
as possible.

This commit introduces a counter which verifies that the number of
written bytes corresponds to the advertized width/length.

See commit 8ee335227b for details.

Signed-off-by: Young Xiao <YangX92@hotmail.com>
2019-04-15 16:10:18 +08:00
Antonin Descampe d0dd894ae2 Comment back opj_previous_version in abi_check.sh 2019-04-02 15:37:38 +02:00
Antonin Descampe 291e45bb04 Update version number for automatic abi check 2019-04-02 15:12:59 +02:00