Commit Graph

208 Commits

Author SHA1 Message Date
Even Rouault 98cfdd3134 opj_j2k_read_cod: remove check for 'No more than one COD marker per tile' (fixes #1043)
This check was added per daed8cc919
to fix https://github.com/uclouvain/openjpeg/issues/476 , but it does not seem
to be necessary with latest master (issue476.jp2 doesn't cause memory issues),
and breaks reading legit files.
2017-11-30 14:48:34 +01:00
Even Rouault 936910cf7a Fix typo in comments 2017-11-30 14:26:17 +01:00
Even Rouault be6ea90e13 opj_j2k_set_threads(): add sanity check to error out if called after opj_read_header() 2017-10-12 01:16:23 +02:00
Antonin Descampe d45ccb048b Merge pull request #1022 from rouault/partial_component_decoding
Add capability to decode only a subset of all components of an image.
2017-09-26 18:30:20 -03:00
Even Rouault 4c8aba2add Add assertion 2017-09-21 14:37:01 +02:00
Even Rouault 113e0976e2 Comment dead code (Coverity CID 94681) 2017-09-21 14:14:32 +02:00
Even Rouault 39082fc665 Workaround Coverity CID 113061 2017-09-21 14:13:16 +02:00
Even Rouault 19e157871f opj_j2k_get_default_thread_count(): validate value of OPJ_NUM_THREADS to fix Coverity 179465 and 179463 2017-09-21 14:06:03 +02:00
Even Rouault 68e596dada Fix copy&paste error (Coverity CID 169394) 2017-09-21 13:54:14 +02:00
Even Rouault b8c4b450c4 Use a #define J2K_MAX_POCS 32 to avoid hard-coded constant (#349) 2017-09-20 00:55:22 +02:00
Even Rouault 82ab7effe3 Remove redundant test (raised by cppcheck) 2017-09-19 18:34:49 +02:00
Even Rouault 7e2b6bebff Add capability to decode only a subset of all components of an image.
This adds a opj_set_decoded_components(opj_codec_t *p_codec,
OPJ_UINT32 numcomps, const OPJ_UINT32* comps_indices) function,
and equivalent "opj_decompress -c compno[,compno]*" option.

When specified, neither the MCT transform nor JP2 channel transformations
will be applied.

Tests added for various combinations of whole image vs tiled-based decoding,
full or reduced resolution, use of decode area or not.
2017-09-19 17:06:19 +02:00
Even Rouault 18f6696372 Fix compilation with AppleClang 8.1.0.8020042 (#1020) 2017-09-19 12:13:34 +02:00
Even Rouault 33167ddc13 opj_j2k_update_image_data(): restrict optimized path 2017-09-08 09:53:52 +02:00
Even Rouault c67e1cd73f Fix invalid access out of bounds, and bad behaviour, when calling repeatdly opj_get_decoded_tile() on an image with a color palette 2017-09-06 17:33:38 +02:00
Even Rouault 297f202104 Fix 2.2.0 regression when reading codestream with reperated calls to opj_get_decoded_tile() where tile parts of a same tile are not consecutive
This check was introduced per #939, but relied on the incorrect assumption
we decode all the tile parts of all tiles.
2017-09-06 16:49:28 +02:00
Even Rouault 968e36bbd9 Merge pull request #1010 from rouault/subtile_decoding_stage3
Subtile decoding: memory use reduction and perf improvements
2017-09-05 22:18:58 +02:00
Even Rouault e5ab1682a1 Improve error message when specifying a too big cp_reduce parameter (#474) 2017-09-02 09:10:53 +02:00
Even Rouault 676d4c807f opj_j2k_update_image_data(): avoid allocating image buffer if we can just reuse the tile buffer one 2017-09-01 22:23:29 +02:00
Even Rouault 7aa071aa27 opj_j2k_setup_encoder(): emit warnings if tcp_rates are not decreasing or tcp_distoratio are not increasing (#1009) 2017-09-01 19:49:01 +02:00
Even Rouault a538815c77 opj_j2k_setup_encoder(): avoid potential int overflow in computations related to max_cs_size 2017-09-01 19:27:56 +02:00
Even Rouault 09929bb615 opj_compress help: revert 3257261776 and indicate 1 again as being the value to get lossless for -r. In opj_j2k_setup_encoder(), make sure that ll rates[] <= 1.0 are set to 0. Document 0 as being lossless for -q / tcp_distoratio (#1009) 2017-09-01 19:16:35 +02:00
Even Rouault 1644665a91 opj_j2k_update_image_data(): avoid zero-ing the buffer if not needed 2017-09-01 16:30:52 +02:00
Even Rouault 0ae3cba340 Allow several repeated calls to opj_set_decode_area() and opj_decode() for single-tiled images
* Only works for single-tiled images --> will error out cleanly, as currently
  in other cases
* Save re-reading the codestream for the tile, and re-use code-blocks of the
  previous decoding pass.
* Future improvements might involve improving opj_decompress, and the image writing logic,
  to use this strategy.
2017-09-01 16:30:48 +02:00
Even Rouault 5d07d463fd opj_j2k_decode_tiles(): apply whole single tile image decoding optimization to reading at reduced resolution as well 2017-09-01 16:30:45 +02:00
Even Rouault 98b9310361 Various changes to allow tile buffers of more than 4giga pixels
Untested though, since that means a tile buffer of at least 16 GB. So
there might be places where uint32 overflow on multiplication still occur...
2017-09-01 16:30:44 +02:00
Even Rouault d5153ba404 Remove limitation that prevents from opening images bigger than 4 billion pixels
However the intermediate buffer for decoding must still be smaller than 4
billion pixels, so this is useful for decoding at a lower resolution level,
or subtile decoding.
2017-09-01 16:30:37 +02:00
Even Rouault f9e9942330 Sub-tile decoding: only allocate tile component buffer of the needed dimension
Instead of being the full tile size.

* Use a sparse array mechanism to store code-blocks and intermediate stages of
  IDWT.
* IDWT, DC level shift and MCT stages are done just on that smaller array.
* Improve copy of tile component array to final image, by saving an intermediate
  buffer.
* For full-tile decoding at reduced resolution, only allocate the tile buffer to
  the reduced size, instead of the full-resolution size.
2017-09-01 16:30:29 +02:00
Even Rouault 0a25dceca7 opj_j2k_setup_encoder(): validate code block width/height 2017-09-01 10:26:53 +02:00
Even Rouault 8f92fc9791 Make opj_set_decode_area() and opj_decode() take into account opj_set_decoded_resolution_factor() (#1006, affect API use)
* Better document usage of opj_set_decode_area(), ie expecting coordinates
  in full resolution/reference grid even if requesting at a lower resolution
  factor
* Make sure that image->comps[].factor is set by opj_set_decode_area() and
  opj_decode() from the value specified in opj_set_decoded_resolution_factor()
* opj_decompress: add 2 environmenet variables to test alternate ways of
  using the API, namely USE_OPJ_SET_DECODED_RESOLUTION_FACTOR=YES to use
  opj_set_decoded_resolution_factor() instead of parameters.cp_reduce, and
  SKIP_OPJ_SET_DECODE_AREA=YES to not call opj_set_decode_area() if -d is
  not specified.
2017-08-28 14:57:49 +02:00
Even Rouault c97666f72b j2k.c: fix comment, and remove FIXME 2017-08-21 19:02:04 +02:00
Even Rouault 5d12806091 opj_j2k_update_rates(): grow tile size buffer for some situations 2017-08-17 19:18:48 +02:00
Even Rouault fe338a057c Sub-tile decoding: only decode precincts and codeblocks that intersect the window specified in opj_set_decode_area() 2017-08-17 19:05:54 +02:00
Even Rouault c535531f03 opj_t2_encode_packet(): fix potential write heap buffer overflow (#992) 2017-08-16 17:20:29 +02:00
Even Rouault dcac91b8c7 opj_j2k_write_sot(): fix potential write heap buffer overflow (#991) 2017-08-16 17:09:10 +02:00
Even Rouault 4241ae6fbb Fix assertion in debug mode / heap-based buffer overflow in opj_write_bytes_LE for Cinema profiles with numresolutions = 1 (#985) 2017-08-15 11:55:58 +02:00
Stefan Weil 10e6ce2c2c Use const qualifier for j2k_prog_order_list
This allows more compiler optimizations.

Signed-off-by: Stefan Weil <sw@weilnetz.de>
2017-08-14 14:41:27 +02:00
Even Rouault 0b4fef6d19 Propagate event manager down to opj_t2_encode_packet() and use it to emit an error message when the output buffer is too small 2017-08-10 16:49:47 +02:00
Even Rouault a316f36dfc Fix crash on encoding if using opj_set_default_encoder_parameters() without defining tcp_numlayers 2017-08-10 14:43:16 +02:00
Even Rouault 4b16e8d27a Remove useless opj_tcd_t::enumcs field added per #975 2017-08-09 17:37:05 +02:00
Even Rouault ac375ac9f5 Partial revert BPC related check of #975 (#979)
PR #975 introduced a check that rejects images that have different bit depth/sign
per compoment in SIZ marker if the JP2 IHDR box has BPC != 255
This didn't work properly if decoding a .j2k file since the new bit added in
opj_cp_t wasn't initialized to the right value.
For clarity, tThis new bit has also been renamed to allow_different_bit_depth_sign

But looking closer at the code, it seems we were already tolerant to inconsistencies.
For example we parsed a JP2 BPCC box even if BPC != 255 (just a warning is emitted)
So failing hard in opj_j2k_read_siz() wouldn't be very inconsistent, and that
alone cannot protect against other issues, so just emit a warning if BPC != 255
and the SIZ marker contains different bit depth/sign per component.

Note: we could also check that the content of JP2 BPCC box is consistant with the one
of the SIZ marker.
2017-08-09 11:34:08 +02:00
Antonin Descampe 0394f8d0f1 Merge pull request #975 from szukw000/changes-for-afl-tests
Catch images broken by AFL
2017-08-08 16:51:54 -07:00
Even Rouault ca34d13e76 Decoding: do not allocate memory for the codestream of each codeblock
Currently we allocate at least 8192 bytes for each codeblock, and copy
the relevant parts of the codestream in that per-codeblock buffer as we
decode packets.
As the whole codestream for the tile is ingested in memory and alive
during the decoding, we can directly point to it instead of copying. But
to do that, we need an intermediate concept, a 'chunk' of code-stream segment,
given that segments may be made of data at different places in the code-stream
when quality layers are used.

With that change, the decoding of MAPA_005.jp2 goes down from the previous
improvement of 2.7 GB down to 1.9 GB.

New profile:

n4: 1885648469 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n1: 1610689344 0x4E78287: opj_aligned_malloc (opj_malloc.c:61)
  n1: 1610689344 0x4E71D7B: opj_alloc_tile_component_data (tcd.c:676)
   n1: 1610689344 0x4E7272C: opj_tcd_init_decode_tile (tcd.c:816)
    n1: 1610689344 0x4E4BDD9: opj_j2k_read_tile_header (j2k.c:8618)
     n1: 1610689344 0x4E4C8A2: opj_j2k_decode_tiles (j2k.c:10349)
      n1: 1610689344 0x4E4E36E: opj_j2k_decode (j2k.c:7847)
       n1: 1610689344 0x4E52FA2: opj_jp2_decode (jp2.c:1564)
        n0: 1610689344 0x40374E: main (opj_decompress.c:1459)
 n1: 219232541 0x4E4BBF0: opj_j2k_read_tile_header (j2k.c:4685)
  n1: 219232541 0x4E4C8A2: opj_j2k_decode_tiles (j2k.c:10349)
   n1: 219232541 0x4E4E36E: opj_j2k_decode (j2k.c:7847)
    n1: 219232541 0x4E52FA2: opj_jp2_decode (jp2.c:1564)
     n0: 219232541 0x40374E: main (opj_decompress.c:1459)
 n1: 39822000 0x4E727A9: opj_tcd_init_decode_tile (tcd.c:1219)
  n1: 39822000 0x4E4BDD9: opj_j2k_read_tile_header (j2k.c:8618)
   n1: 39822000 0x4E4C8A2: opj_j2k_decode_tiles (j2k.c:10349)
    n1: 39822000 0x4E4E36E: opj_j2k_decode (j2k.c:7847)
     n1: 39822000 0x4E52FA2: opj_jp2_decode (jp2.c:1564)
      n0: 39822000 0x40374E: main (opj_decompress.c:1459)
 n0: 15904584 in 52 places, all below massif's threshold (1.00%)
2017-08-07 18:32:52 +02:00
Even Rouault f58aab9d6a Add opj_image_data_alloc() / opj_image_data_free()
As bin/common/color.c used to directly call malloc()/free(), we need
to export functions dedicated to allocating/freeing image component data.
2017-08-07 18:32:52 +02:00
Even Rouault 61fb5dd7f8 Fix crash on Windows due to b7594c0fcb9dd3aa6356d72c4a525d76168da689
b7594c0fcb9dd3aa6356d72c4a525d76168da689 may put opj_tcd_tilecomp_t->data
allocated by opj_alloc_tile_component_data() as the image->comps[].data. As
opj_alloc_tile_component_data() use opj_aligned_malloc() we must be sure to
ue opj_alined_malloc()/_free() in all places where we alloc/free
image->comps[].data.

Note: this might have some compatibility impact in case user code does itself
the allocation/free of image->comps[].data
2017-08-07 18:32:49 +02:00
Even Rouault 793edc38e4 Decrease memory consumption for whole image single tile decoding.
We can use the same buffer for the tile decoding and the final image, and
save the intermediate buffer to transfer between those.

Effect on the decoding of MAPA (9944 x 13498 x 3 components of size byte)

Peak memory from 4.5 GB to 2.7 GB

Now:
n5: 2699708767 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n1: 1610689344 0x4E77E07: opj_aligned_malloc (opj_malloc.c:61) <-- final image
  n1: 1610689344 0x4E7195B: opj_alloc_tile_component_data (tcd.c:676)
   n1: 1610689344 0x4E722D2: opj_tcd_init_decode_tile (tcd.c:816)
    n1: 1610689344 0x4E4BCF1: opj_j2k_read_tile_header (j2k.c:8597)
     n1: 1610689344 0x4E4C742: opj_j2k_decode_tiles (j2k.c:10324)
      n1: 1610689344 0x4E4E20E: opj_j2k_decode (j2k.c:7826)
       n1: 1610689344 0x4E52E42: opj_jp2_decode (jp2.c:1564)
        n0: 1610689344 0x40369E: main (opj_decompress.c:1459)
 n1: 815554560 0x4E72231: opj_tcd_init_decode_tile (tcd.c:1217) <-- working memory for code blocks: 9944*13498/64/64*8192*3
  n1: 815554560 0x4E4BCF1: opj_j2k_read_tile_header (j2k.c:8597)
   n1: 815554560 0x4E4C742: opj_j2k_decode_tiles (j2k.c:10324)
    n1: 815554560 0x4E4E20E: opj_j2k_decode (j2k.c:7826)
     n1: 815554560 0x4E52E42: opj_jp2_decode (jp2.c:1564)
      n0: 815554560 0x40369E: main (opj_decompress.c:1459)
 n1: 219758391 0x4E4C0BF: opj_j2k_read_tile_header (j2k.c:4661) <-- ingestion of code stream
  n1: 219758391 0x4E4C742: opj_j2k_decode_tiles (j2k.c:10324)
   n1: 219758391 0x4E4E20E: opj_j2k_decode (j2k.c:7826)
    n1: 219758391 0x4E52E42: opj_jp2_decode (jp2.c:1564)
     n0: 219758391 0x40369E: main (opj_decompress.c:1459)
 n1: 39822000 0x4E7224F: opj_tcd_init_decode_tile (tcd.c:1224) <-- OPJ_J2K_DEFAULT_NB_SEGS*sizeof(opj_tcd_seg_t) per codeblock
  n1: 39822000 0x4E4BCF1: opj_j2k_read_tile_header (j2k.c:8597)
   n1: 39822000 0x4E4C742: opj_j2k_decode_tiles (j2k.c:10324)
    n1: 39822000 0x4E4E20E: opj_j2k_decode (j2k.c:7826)
     n1: 39822000 0x4E52E42: opj_jp2_decode (jp2.c:1564)
      n0: 39822000 0x40369E: main (opj_decompress.c:1459)
 n0: 13884472 in 49 places, all below massif's threshold (1.00%)

Before:
n5: 4493329848 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n2: 1610709160 0x4E77C87: opj_aligned_malloc (opj_malloc.c:61)
  n1: 1610689344 0x4E717DB: opj_alloc_tile_component_data (tcd.c:676)
   n1: 1610689344 0x4E72152: opj_tcd_init_decode_tile (tcd.c:816)
    n1: 1610689344 0x4E4BCF1: opj_j2k_read_tile_header (j2k.c:8597)
     n1: 1610689344 0x4E4C64A: opj_j2k_decode_tiles (j2k.c:10318)
      n1: 1610689344 0x4E4E08E: opj_j2k_decode (j2k.c:7826)
       n1: 1610689344 0x4E52CC2: opj_jp2_decode (jp2.c:1564)
        n0: 1610689344 0x40369E: main (opj_decompress.c:1459)
  n0: 19816 in 2 places, all below massif's threshold (1.00%)
 n1: 1610689344 0x4E43F36: opj_j2k_update_image_data.isra.7 (j2k.c:8743)
  n1: 1610689344 0x4E4C5C1: opj_j2k_decode_tiles (j2k.c:10358)
   n1: 1610689344 0x4E4E08E: opj_j2k_decode (j2k.c:7826)
    n1: 1610689344 0x4E52CC2: opj_jp2_decode (jp2.c:1564)
     n0: 1610689344 0x40369E: main (opj_decompress.c:1459)
 n1: 815554560 0x4E720B1: opj_tcd_init_decode_tile (tcd.c:1217)
  n1: 815554560 0x4E4BCF1: opj_j2k_read_tile_header (j2k.c:8597)
   n1: 815554560 0x4E4C64A: opj_j2k_decode_tiles (j2k.c:10318)
    n1: 815554560 0x4E4E08E: opj_j2k_decode (j2k.c:7826)
     n1: 815554560 0x4E52CC2: opj_jp2_decode (jp2.c:1564)
      n0: 815554560 0x40369E: main (opj_decompress.c:1459)
 n1: 402672336 0x4E4C545: opj_j2k_decode_tiles (j2k.c:10336)
  n1: 402672336 0x4E4E08E: opj_j2k_decode (j2k.c:7826)
   n1: 402672336 0x4E52CC2: opj_jp2_decode (jp2.c:1564)
    n0: 402672336 0x40369E: main (opj_decompress.c:1459)
 n0: 53704448 in 58 places, all below massif's threshold (1.00%)
2017-08-07 18:18:53 +02:00
Even Rouault 2fbd4bb0b9 opj_j2k_read_sot(): check current TPSot number regarding previous (non-zero) TNsot to avoid opj_j2k_merge_ppt() to be called several times. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=2851. Credit to OSS Fuzz 2017-08-04 18:02:10 +02:00
szukw000 57e36dbfeb First change on changes-for-afl-tests 2017-08-02 17:27:08 +02:00
szukw000 00f45684a8 Catch images broken by AFL 2017-07-31 13:58:08 +02:00
Even Rouault b716f86163 Fix breakage of 22bf99ce02 2017-07-30 19:26:47 +02:00