Commit Graph

112 Commits

Author SHA1 Message Date
Even Rouault c06632c6f6
Cleanup code related to quality layer allocation, and add a few safety checks 2022-08-11 18:12:07 +02:00
Even Rouault dd1a2d6480
opj_t1_encode_cblk(): avoid undefined behaviour on fuzzed input (fixes #1432) 2022-06-29 11:47:58 +02:00
Aous Naman 1e6c925eb5
Added support for high throughput (HTJ2K) decoding.
There are a few limitations:
- mixed mode (HT and regular code blocks) is not supported.
- ROI in HT blocks is not supported.
- Placeholder passes are not supported.
- MultiHT sets are not support, only a singleHT set.
- there are known issues with some compliance testing files related to
  the parsing of packet header.
2021-09-25 12:26:59 +02:00
Even Rouault 67ec360f48
Fix typo in internal function name 2020-11-23 13:38:27 +01:00
Even Rouault 491299eb07
Merge pull request #1253 from rouault/floating_point_irreversible_encoding
Single-threaded performance improvements in forward DWT for 5-3 and 9-7 (and other improvements)
2020-10-09 13:25:27 +02:00
Sebastian Rasmussen 79b199a8fe openjp2: Plug memory leak when setting data as TLS fails.
Previously the Tier 1 handle was not freed when setting it as
TLS failed.

This fixes issue #1257.
2020-06-23 03:21:31 +08:00
Sebastian Rasmussen 93b9f7236c openjp2: Error out if failing to create Tier 1 handle.
Previously when the handle failed to be created (e.g. when
opj_calloc returned NULL due to low memory), the code still
assumed that the t1 handle pointer was valid and dereferenced
NULL, causing a crash. After this commit OpenJPEG will instead
error out under this condition.

This fixes issue #1255.
2020-06-23 02:32:31 +08:00
Even Rouault 1c5627ee74
T1 encoder: speed-up by aggressive inlining and more cache friendly data organization
~ 9% speed improvement seen on 10980x10980 uint16 image, T36JTT_20160914T074612_B02.tif
opj_compress time from 17.2s to 15.8s
2020-05-24 15:38:48 +02:00
Even Rouault 3cd1305596
Irreversible compression/decompression DWT: use 1/K constant as per standard
The previous constant opj_c13318 was mysteriously equal to 2/K , and in
the DWT, we had to divide K and opj_c13318 by 2... The issue was that the
band->stepsize computation in tcd.c didn't take into account the log2gain of
the band.

The effect of this change is expected to be mostly equivalent to the previous
situation, except some difference in rounding. But it leads to a dramatic
reduction of the mean square error and peak error in the irreversible encoding
of issue141.tif !
2020-05-20 20:31:28 +02:00
Even Rouault f38c069547
Irreversible decoding: align code more closely to the standard by avoid messing up with stepsize (no functional change) 2020-05-20 20:31:28 +02:00
Even Rouault 00cff6f5c0
Encoder: use floating-point operations for irreversible transformation 2020-05-20 20:31:28 +02:00
Even Rouault 97eb7e0bf1
Add multithreading support in the T1 (entropy phase) encoder
- API wise, opj_codec_set_threads() can be used on the encoding side
- opj_compress has a -threads switch similar to opj_uncompress
2020-05-20 20:30:21 +02:00
Even Rouault a1d32a596a
opj_t1_encode_cblks: fix UBSAN signed integer overflow
Fixes #1053 / CVE-2018-5727

Note: I don't consider this issue to be a security vulnerability, in
practice.
At least with gcc or clang compilers on x86_64 which generate the same
assembly code with or without that fix.
2019-03-29 11:17:39 +01:00
Stefan Weil 948332e6ed Fix some potential overflow issues (#1161)
* Fix some potential overflow issues

Put sizeof to the beginning of the multiplication to enforce that
size_t instead of smaller integer types is used for the calculation.

This fixes warnings from LGTM:

    Multiplication result may overflow 'unsigned int'
    before it is converted to 'unsigned long'.

It also allows removing some type casts.

Signed-off-by: Stefan Weil <sw@weilnetz.de>

* Fix code indentation

Signed-off-by: Stefan Weil <sw@weilnetz.de>
2018-10-31 20:44:30 +01:00
Even Rouault 7711307d86 Workaround Coverity CID 169392 2017-09-21 14:10:49 +02:00
Even Rouault 8df07d5866 Avoid relying on operator precedence (raised by cppcheck) 2017-09-19 18:35:52 +02:00
Even Rouault fdef69b43c Fix warnings and errors when compiling with a c++ compiler (#1021) 2017-09-19 12:46:20 +02:00
Even Rouault 579b8937ea Replace uses of size_t by OPJ_SIZE_T 2017-09-04 17:35:52 +02:00
Even Rouault 4c7effa6bc opj_t1_clbl_decode_processor(): use SSE2 in subtile decoding code path, for irreversible 2017-09-01 22:23:29 +02:00
Even Rouault 559d16e8f4 opj_t1_decode_cblk(): move some code to codeblock processor for (theoretical) better multi-threading in subtile decoding 2017-09-01 22:23:29 +02:00
Even Rouault ccac773556 Tiny perf improvement in T1 stage for subtile decoding 2017-09-01 16:30:58 +02:00
Even Rouault 0ae3cba340 Allow several repeated calls to opj_set_decode_area() and opj_decode() for single-tiled images
* Only works for single-tiled images --> will error out cleanly, as currently
  in other cases
* Save re-reading the codestream for the tile, and re-use code-blocks of the
  previous decoding pass.
* Future improvements might involve improving opj_decompress, and the image writing logic,
  to use this strategy.
2017-09-01 16:30:48 +02:00
Even Rouault 98b9310361 Various changes to allow tile buffers of more than 4giga pixels
Untested though, since that means a tile buffer of at least 16 GB. So
there might be places where uint32 overflow on multiplication still occur...
2017-09-01 16:30:44 +02:00
Even Rouault f9e9942330 Sub-tile decoding: only allocate tile component buffer of the needed dimension
Instead of being the full tile size.

* Use a sparse array mechanism to store code-blocks and intermediate stages of
  IDWT.
* IDWT, DC level shift and MCT stages are done just on that smaller array.
* Improve copy of tile component array to final image, by saving an intermediate
  buffer.
* For full-tile decoding at reduced resolution, only allocate the tile buffer to
  the reduced size, instead of the full-resolution size.
2017-09-01 16:30:29 +02:00
Even Rouault 84bbb4a874 opj_t1_allocate_buffers(): remove useless overflow checks 2017-09-01 10:26:53 +02:00
Even Rouault 5d40325056 Subtile decoding: only do 5x3 IDWT computations on relevant areas of tile-component buffer.
This lowers 'bin/opj_decompress -i ../MAPA.jp2 -o out.tif -d 0,0,256,256'
down to 0.860s
2017-08-18 15:08:51 +02:00
Even Rouault 4b0bfbfabc Zero-initialize tile buffer regions of skipped code-blocks, so as to make Valgrind happy 2017-08-17 19:05:54 +02:00
Even Rouault fe338a057c Sub-tile decoding: only decode precincts and codeblocks that intersect the window specified in opj_set_decode_area() 2017-08-17 19:05:54 +02:00
Even Rouault 8e6c371e66 opj_t1_encode_cblk(): avoid uint32 overflow when numbps = 0 (which is well defined behaviour, and is properly handled here, but better avoid it to detect real issues) 2017-08-16 18:29:59 +02:00
Even Rouault 92114694a4 Slight improvement in management of code block chunks
Instead of having the chunk array at the segment level, we can move it down to
the codeblock itself since segments are filled in sequential order.
Limit the number of memory allocation, and decrease slightly the memory usage.

On MAPA_005.jp2

n4: 1871312549 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n1: 1610689344 0x4E781E7: opj_aligned_malloc (opj_malloc.c:61)
  n1: 1610689344 0x4E71D1B: opj_alloc_tile_component_data (tcd.c:676)
   n1: 1610689344 0x4E726CF: opj_tcd_init_decode_tile (tcd.c:816)
    n1: 1610689344 0x4E4BE39: opj_j2k_read_tile_header (j2k.c:8617)
     n1: 1610689344 0x4E4C902: opj_j2k_decode_tiles (j2k.c:10348)
      n1: 1610689344 0x4E4E3CE: opj_j2k_decode (j2k.c:7846)
       n1: 1610689344 0x4E53002: opj_jp2_decode (jp2.c:1564)
        n0: 1610689344 0x40374E: main (opj_decompress.c:1459)
 n1: 219232541 0x4E4BC50: opj_j2k_read_tile_header (j2k.c:4683)
  n1: 219232541 0x4E4C902: opj_j2k_decode_tiles (j2k.c:10348)
   n1: 219232541 0x4E4E3CE: opj_j2k_decode (j2k.c:7846)
    n1: 219232541 0x4E53002: opj_jp2_decode (jp2.c:1564)
     n0: 219232541 0x40374E: main (opj_decompress.c:1459)
 n1: 23893200 0x4E72735: opj_tcd_init_decode_tile (tcd.c:1225)
  n1: 23893200 0x4E4BE39: opj_j2k_read_tile_header (j2k.c:8617)
   n1: 23893200 0x4E4C902: opj_j2k_decode_tiles (j2k.c:10348)
    n1: 23893200 0x4E4E3CE: opj_j2k_decode (j2k.c:7846)
     n1: 23893200 0x4E53002: opj_jp2_decode (jp2.c:1564)
      n0: 23893200 0x40374E: main (opj_decompress.c:1459)
 n0: 17497464 in 52 places, all below massif's threshold (1.00%)
2017-08-07 18:32:52 +02:00
Even Rouault ca34d13e76 Decoding: do not allocate memory for the codestream of each codeblock
Currently we allocate at least 8192 bytes for each codeblock, and copy
the relevant parts of the codestream in that per-codeblock buffer as we
decode packets.
As the whole codestream for the tile is ingested in memory and alive
during the decoding, we can directly point to it instead of copying. But
to do that, we need an intermediate concept, a 'chunk' of code-stream segment,
given that segments may be made of data at different places in the code-stream
when quality layers are used.

With that change, the decoding of MAPA_005.jp2 goes down from the previous
improvement of 2.7 GB down to 1.9 GB.

New profile:

n4: 1885648469 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n1: 1610689344 0x4E78287: opj_aligned_malloc (opj_malloc.c:61)
  n1: 1610689344 0x4E71D7B: opj_alloc_tile_component_data (tcd.c:676)
   n1: 1610689344 0x4E7272C: opj_tcd_init_decode_tile (tcd.c:816)
    n1: 1610689344 0x4E4BDD9: opj_j2k_read_tile_header (j2k.c:8618)
     n1: 1610689344 0x4E4C8A2: opj_j2k_decode_tiles (j2k.c:10349)
      n1: 1610689344 0x4E4E36E: opj_j2k_decode (j2k.c:7847)
       n1: 1610689344 0x4E52FA2: opj_jp2_decode (jp2.c:1564)
        n0: 1610689344 0x40374E: main (opj_decompress.c:1459)
 n1: 219232541 0x4E4BBF0: opj_j2k_read_tile_header (j2k.c:4685)
  n1: 219232541 0x4E4C8A2: opj_j2k_decode_tiles (j2k.c:10349)
   n1: 219232541 0x4E4E36E: opj_j2k_decode (j2k.c:7847)
    n1: 219232541 0x4E52FA2: opj_jp2_decode (jp2.c:1564)
     n0: 219232541 0x40374E: main (opj_decompress.c:1459)
 n1: 39822000 0x4E727A9: opj_tcd_init_decode_tile (tcd.c:1219)
  n1: 39822000 0x4E4BDD9: opj_j2k_read_tile_header (j2k.c:8618)
   n1: 39822000 0x4E4C8A2: opj_j2k_decode_tiles (j2k.c:10349)
    n1: 39822000 0x4E4E36E: opj_j2k_decode (j2k.c:7847)
     n1: 39822000 0x4E52FA2: opj_jp2_decode (jp2.c:1564)
      n0: 39822000 0x40374E: main (opj_decompress.c:1459)
 n0: 15904584 in 52 places, all below massif's threshold (1.00%)
2017-08-07 18:32:52 +02:00
Even Rouault 83342f2aaf Fix Doxygen warnings (patch derived from Winfried's doxygen-dif.txt.zip, #849) 2017-07-30 18:18:59 +02:00
Even Rouault db9ef99f6d opj_t1_decode_cblk(): avoid undefined shift behaviour. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=2487. Credit to OSS Fuzz 2017-07-29 16:34:35 +02:00
Even Rouault f6551f822f opj_t1_clbl_decode_processor(): avoid undefined behaviour if roishift >= 31. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=2506. Credit to OSS Fuzz 2017-07-29 16:29:11 +02:00
Even Rouault 94c4b7300c T1 decoder: check code stream errors when predictable termination is enabled and emit a warning when errors are found 2017-07-26 21:43:32 +02:00
Even Rouault cdd3e83bae Fix clang warning about extraneous parentheses 2017-06-21 12:49:01 +02:00
Even Rouault cc07aec6c7 Fix warnings with recent GCC versions 2017-06-17 14:09:31 +02:00
Even Rouault 9cbc9903c3 Merge branch 't1_flag_optimizations' 2017-06-13 12:09:52 +02:00
Even Rouault 73d1510d47 Encoder: fix packet writing of empty sub-bands (#891, #892)
There are situations where, given a tile size, at a resolution level,
there are sub-bands with x0==x1 or y0==y1, that consequently don't have any
valid codeblocks, but the other sub-bands may be non-empty.
Given that we recycle the memory from one tile to another one, those
ghost codeblocks might be non-0 and thus candidate for packet inclusion.
2017-06-12 18:37:50 +02:00
Even Rouault 81c5311758 T1: fix BYPASS/LAZY, TERMALL/RESTART and PTERM/ERTERM encoding modes. (#674)
There were a number of defects regarding when and how the termination of
passes had to done and the computation of their rate.
2017-06-09 10:49:03 +02:00
Even Rouault 9a9b06911e opj_t1_dec_sigpass_raw/opj_t1_dec_refpass_raw: harmonize style with mqc methods 2017-06-02 19:22:15 +02:00
Even Rouault 532243f1fd MQC/RAW decoder: use an artificial 0xFF 0xFF terminating marker.
This saves comparing the current pointer with the end of buffer pointer.
This results at least in tiny speed improvement for raw decoding, and
smaller code size for MQC as well.

This kills the remains of the raw.h/.c files that were only used for
decoding. Encoding using the mqc structure already.
2017-06-02 18:24:07 +02:00
Even Rouault dde6cbabc0 Simplify VSC handling: instead of masking out bits when reading the 4th row.
Do not set them when updating flags of the 1st row
2017-06-02 18:23:38 +02:00
Even Rouault 3d9940a35b Force inlining of mqc decoding and pass steps through heavy use of macros, so as to get better register allocation 2017-06-02 18:23:20 +02:00
Even Rouault 2ba861c37c Optimize opj_t1_update_flags() 2017-06-02 18:22:42 +02:00
Even Rouault a0861855c1 T1: remove use of neghalf variable. It is useless since bpno is always > 0 2017-06-02 18:22:21 +02:00
Even Rouault 10410fe72e T1: avoid pointer indirection for mqc and raw members of opj_t1_t 2017-06-02 18:21:54 +02:00
Even Rouault a5003787ff T1: remove flags_stride variable from opj_t1_t 2017-06-02 18:21:39 +02:00
Even Rouault aa7a8a4398 T1: loop unrolling in dec_sigpass_raw and dec_refpass_raw 2017-06-02 18:20:58 +02:00
Even Rouault 68557ff503 T1: Transpose coder optimizations to decoder, and cleanup code 2017-06-02 18:20:35 +02:00