walkero
/
openjpeg
1
0
Fork 0
Code Issues Pull Requests Projects Releases 1 Wiki Activity
2,504 Commits 24 Branches 23 Tags 132 MiB
Commit Graph

164 Commits

Author SHA1 Message Date
Even Rouault ca34d13e76 Decoding: do not allocate memory for the codestream of each codeblock 
Currently we allocate at least 8192 bytes for each codeblock, and copy
the relevant parts of the codestream in that per-codeblock buffer as we
decode packets.
As the whole codestream for the tile is ingested in memory and alive
during the decoding, we can directly point to it instead of copying. But
to do that, we need an intermediate concept, a 'chunk' of code-stream segment,
given that segments may be made of data at different places in the code-stream
when quality layers are used.

With that change, the decoding of MAPA_005.jp2 goes down from the previous
improvement of 2.7 GB down to 1.9 GB.

New profile:

n4: 1885648469 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n1: 1610689344 0x4E78287: opj_aligned_malloc (opj_malloc.c:61)
  n1: 1610689344 0x4E71D7B: opj_alloc_tile_component_data (tcd.c:676)
   n1: 1610689344 0x4E7272C: opj_tcd_init_decode_tile (tcd.c:816)
    n1: 1610689344 0x4E4BDD9: opj_j2k_read_tile_header (j2k.c:8618)
     n1: 1610689344 0x4E4C8A2: opj_j2k_decode_tiles (j2k.c:10349)
      n1: 1610689344 0x4E4E36E: opj_j2k_decode (j2k.c:7847)
       n1: 1610689344 0x4E52FA2: opj_jp2_decode (jp2.c:1564)
        n0: 1610689344 0x40374E: main (opj_decompress.c:1459)
 n1: 219232541 0x4E4BBF0: opj_j2k_read_tile_header (j2k.c:4685)
  n1: 219232541 0x4E4C8A2: opj_j2k_decode_tiles (j2k.c:10349)
   n1: 219232541 0x4E4E36E: opj_j2k_decode (j2k.c:7847)
    n1: 219232541 0x4E52FA2: opj_jp2_decode (jp2.c:1564)
     n0: 219232541 0x40374E: main (opj_decompress.c:1459)
 n1: 39822000 0x4E727A9: opj_tcd_init_decode_tile (tcd.c:1219)
  n1: 39822000 0x4E4BDD9: opj_j2k_read_tile_header (j2k.c:8618)
   n1: 39822000 0x4E4C8A2: opj_j2k_decode_tiles (j2k.c:10349)
    n1: 39822000 0x4E4E36E: opj_j2k_decode (j2k.c:7847)
     n1: 39822000 0x4E52FA2: opj_jp2_decode (jp2.c:1564)
      n0: 39822000 0x40374E: main (opj_decompress.c:1459)
 n0: 15904584 in 52 places, all below massif's threshold (1.00%)
 2017-08-07 18:32:52 +02:00
Even Rouault f58aab9d6a Add opj_image_data_alloc() / opj_image_data_free() 
As bin/common/color.c used to directly call malloc()/free(), we need
to export functions dedicated to allocating/freeing image component data.
 2017-08-07 18:32:52 +02:00
Even Rouault 61fb5dd7f8 Fix crash on Windows due to b7594c0fcb9dd3aa6356d72c4a525d76168da689 
b7594c0fcb9dd3aa6356d72c4a525d76168da689 may put opj_tcd_tilecomp_t->data
allocated by opj_alloc_tile_component_data() as the image->comps[].data. As
opj_alloc_tile_component_data() use opj_aligned_malloc() we must be sure to
ue opj_alined_malloc()/_free() in all places where we alloc/free
image->comps[].data.

Note: this might have some compatibility impact in case user code does itself
the allocation/free of image->comps[].data
 2017-08-07 18:32:49 +02:00
Even Rouault 793edc38e4 Decrease memory consumption for whole image single tile decoding. 
We can use the same buffer for the tile decoding and the final image, and
save the intermediate buffer to transfer between those.

Effect on the decoding of MAPA (9944 x 13498 x 3 components of size byte)

Peak memory from 4.5 GB to 2.7 GB

Now:
n5: 2699708767 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n1: 1610689344 0x4E77E07: opj_aligned_malloc (opj_malloc.c:61) <-- final image
  n1: 1610689344 0x4E7195B: opj_alloc_tile_component_data (tcd.c:676)
   n1: 1610689344 0x4E722D2: opj_tcd_init_decode_tile (tcd.c:816)
    n1: 1610689344 0x4E4BCF1: opj_j2k_read_tile_header (j2k.c:8597)
     n1: 1610689344 0x4E4C742: opj_j2k_decode_tiles (j2k.c:10324)
      n1: 1610689344 0x4E4E20E: opj_j2k_decode (j2k.c:7826)
       n1: 1610689344 0x4E52E42: opj_jp2_decode (jp2.c:1564)
        n0: 1610689344 0x40369E: main (opj_decompress.c:1459)
 n1: 815554560 0x4E72231: opj_tcd_init_decode_tile (tcd.c:1217) <-- working memory for code blocks: 9944*13498/64/64*8192*3
  n1: 815554560 0x4E4BCF1: opj_j2k_read_tile_header (j2k.c:8597)
   n1: 815554560 0x4E4C742: opj_j2k_decode_tiles (j2k.c:10324)
    n1: 815554560 0x4E4E20E: opj_j2k_decode (j2k.c:7826)
     n1: 815554560 0x4E52E42: opj_jp2_decode (jp2.c:1564)
      n0: 815554560 0x40369E: main (opj_decompress.c:1459)
 n1: 219758391 0x4E4C0BF: opj_j2k_read_tile_header (j2k.c:4661) <-- ingestion of code stream
  n1: 219758391 0x4E4C742: opj_j2k_decode_tiles (j2k.c:10324)
   n1: 219758391 0x4E4E20E: opj_j2k_decode (j2k.c:7826)
    n1: 219758391 0x4E52E42: opj_jp2_decode (jp2.c:1564)
     n0: 219758391 0x40369E: main (opj_decompress.c:1459)
 n1: 39822000 0x4E7224F: opj_tcd_init_decode_tile (tcd.c:1224) <-- OPJ_J2K_DEFAULT_NB_SEGS*sizeof(opj_tcd_seg_t) per codeblock
  n1: 39822000 0x4E4BCF1: opj_j2k_read_tile_header (j2k.c:8597)
   n1: 39822000 0x4E4C742: opj_j2k_decode_tiles (j2k.c:10324)
    n1: 39822000 0x4E4E20E: opj_j2k_decode (j2k.c:7826)
     n1: 39822000 0x4E52E42: opj_jp2_decode (jp2.c:1564)
      n0: 39822000 0x40369E: main (opj_decompress.c:1459)
 n0: 13884472 in 49 places, all below massif's threshold (1.00%)

Before:
n5: 4493329848 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n2: 1610709160 0x4E77C87: opj_aligned_malloc (opj_malloc.c:61)
  n1: 1610689344 0x4E717DB: opj_alloc_tile_component_data (tcd.c:676)
   n1: 1610689344 0x4E72152: opj_tcd_init_decode_tile (tcd.c:816)
    n1: 1610689344 0x4E4BCF1: opj_j2k_read_tile_header (j2k.c:8597)
     n1: 1610689344 0x4E4C64A: opj_j2k_decode_tiles (j2k.c:10318)
      n1: 1610689344 0x4E4E08E: opj_j2k_decode (j2k.c:7826)
       n1: 1610689344 0x4E52CC2: opj_jp2_decode (jp2.c:1564)
        n0: 1610689344 0x40369E: main (opj_decompress.c:1459)
  n0: 19816 in 2 places, all below massif's threshold (1.00%)
 n1: 1610689344 0x4E43F36: opj_j2k_update_image_data.isra.7 (j2k.c:8743)
  n1: 1610689344 0x4E4C5C1: opj_j2k_decode_tiles (j2k.c:10358)
   n1: 1610689344 0x4E4E08E: opj_j2k_decode (j2k.c:7826)
    n1: 1610689344 0x4E52CC2: opj_jp2_decode (jp2.c:1564)
     n0: 1610689344 0x40369E: main (opj_decompress.c:1459)
 n1: 815554560 0x4E720B1: opj_tcd_init_decode_tile (tcd.c:1217)
  n1: 815554560 0x4E4BCF1: opj_j2k_read_tile_header (j2k.c:8597)
   n1: 815554560 0x4E4C64A: opj_j2k_decode_tiles (j2k.c:10318)
    n1: 815554560 0x4E4E08E: opj_j2k_decode (j2k.c:7826)
     n1: 815554560 0x4E52CC2: opj_jp2_decode (jp2.c:1564)
      n0: 815554560 0x40369E: main (opj_decompress.c:1459)
 n1: 402672336 0x4E4C545: opj_j2k_decode_tiles (j2k.c:10336)
  n1: 402672336 0x4E4E08E: opj_j2k_decode (j2k.c:7826)
   n1: 402672336 0x4E52CC2: opj_jp2_decode (jp2.c:1564)
    n0: 402672336 0x40369E: main (opj_decompress.c:1459)
 n0: 53704448 in 58 places, all below massif's threshold (1.00%)
 2017-08-07 18:18:53 +02:00
Even Rouault 2fbd4bb0b9 opj_j2k_read_sot(): check current TPSot number regarding previous (non-zero) TNsot to avoid opj_j2k_merge_ppt() to be called several times. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=2851. Credit to OSS Fuzz 2017-08-04 18:02:10 +02:00
Even Rouault b716f86163 Fix breakage of 22bf99ce02 2017-07-30 19:26:47 +02:00
Even Rouault 22bf99ce02 Test return value of opj_j2k_setup_decoding_tile() (commit ec31fa0c7f by ak-dxdy, #561) 2017-07-30 19:07:16 +02:00
Even Rouault 4748318136 j2k.c: remove hardcoded constants related to m_state, and useless FIXME 2017-07-30 17:26:03 +02:00
Even Rouault 1ed8d67797 opj_j2k_set_decode_area: replace assertions by runtime checks. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=2795. Credit to OSS Fuzz 2017-07-30 15:35:47 +02:00
Even Rouault 3fbe713690 opj_tcd_get_decoded_tile_size(): fix potential UINT32 overflow (#854, CVE-2016-5152) 
Fix derived from https://pdfium.googlesource.com/pdfium.git/+/d8cc503575463ff3d81b22dad292665f2c88911e/third_party/libopenjpeg20/0018-tcd_get_decoded_tile_size.patch
 2017-07-29 18:38:16 +02:00
Even Rouault 9906fbf737 Avoid assertion in opj_j2k_merge_ppt() in case premature EOC is encountered in opj_j2k_read_tile_header(). Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=2785. Credit to OSS Fuzz 2017-07-29 16:22:36 +02:00
Even Rouault 16aeb9282f Fix null pointer dereference in opj_j2k_add_mct() (#895) 
Fixes openjeg-crashes-2017-07-27/issue879-poc1.j2k of #895
 2017-07-28 21:39:30 +02:00
Even Rouault c5bf5ef4d6 Avoid use-after-free when a MCT marker is found after a MCC one (#895) 
Fixes openjeg-crashes-2017-07-27/issue880-poc2.j2k of #895
 2017-07-28 21:29:55 +02:00
Even Rouault e03e947466 Avoid undefined shift behaviour if bit depth == 32 (#895) 
Fixes openjeg-crashes-2017-07-27/id:000000,sig:11,src:003798,op:ext_AO,pos:128.jp2
 2017-07-27 22:29:17 +02:00
Even Rouault 820fcfe8bb opj_j2k_update_image_data / opj_tcd_update_tile_data: fix unaligned load/store (#895) 
When components don't have the same width, unaligned load/store are possible.

Fixes openjeg-crashes-2017-07-27/id:000000,sig:11,src:001342,op:flip4,pos:162.jp2 of #895
 2017-07-27 19:35:35 +02:00
Even Rouault d6fa300997 Avoids undefined shift behaviour in m_dc_level_shift computation 
Fixes warning found on clusterfuzz-testcase-minimized-5146316340461568
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=2495

Credit to OSS Fuzz
 2017-07-27 18:10:03 +02:00
Even Rouault 8d2e69e37d Fix assertion / memory leak in opj_j2k_merge_ppt() on corrupted images (#939) 
Fixes issue on id:000020,sig:06,src:001958,op:flip4,pos:149 that has two
SOT markers for the same tile with the same tile part number, causing
opj_j2k_merge_ppt() to be called several times.
 2017-07-26 19:49:38 +02:00
Even Rouault c308de39ed opj_j2k_read_header_procedure(): validate marker size to avoid excessive memory allocation attempt 2017-07-03 14:33:57 +02:00
Even Rouault a8ca7c51f3 CMake: add stronger warnings for openjp2 lib/bin by default, and error out on declaration-after-statement 
And remove occurences of unused arguments in src/lib/openjp2
 2017-05-23 15:47:57 +02:00
Even Rouault 563bd8499e Reformat whole codebase with astyle.options (#128) 2017-05-09 20:46:20 +02:00
Matthieu Darbois 0954bc11e3 Fix some warnings (#838) 
Fix warnings introduced by uclouvain/openjpeg#786
 2016-09-14 00:12:43 +02:00
Even Rouault 48c16b2c19 Merge branch 'master' of https://github.com/uclouvain/openjpeg into tier1_optimizations_multithreading_2 
Conflicts:
	src/lib/openjp2/t1.c
 2016-09-08 10:30:09 +02:00
Matthieu Darbois 734d57d5f7 fix incrementing of "l_tcp->m_nb_mcc_records" in opj_j2k_read_mcc (#820) 2016-09-06 22:33:26 +02:00
Matthieu Darbois ccd9ced49e Add overflow check in opj_j2k_update_image_data (#817) 2016-09-06 00:50:44 +02:00
Even Rouault d4b7f03cfa Add opj_codec_set_threads() in public API and propagate resulting thread pool to tcd level 
By default, only the main thread is used. If opj_codec_set_threads() is not used,
but the OPJ_NUM_THREADS environment variable is set, its value will be
used to initialize the number of threads. The value can be either an integer
number, or "ALL_CPUS". If OPJ_NUM_THREADS is set and this function is called,
this function will override the behaviour of the environment variable.
 2016-05-25 21:02:07 +02:00
mayeut 1a8318f6c2 Fix Out-of-Bounds Access in function opj_tgt_reset 
Fix uclouvain/openjpeg#775
 2016-05-08 20:10:13 +02:00
mayeut cd77b6003b Fix some coverity warnings 2016-04-30 12:15:22 +02:00
Matthieu Darbois be42e72d22 Fix warnings (#763) 2016-04-30 01:12:16 +02:00
julienmalik 319fc971fe cppcheck fix for openjp2 (#740) 2016-04-29 23:49:17 +02:00
Matthieu Darbois e166e4a209 Check SSIZ is valid in opj_j2k_read_siz (#762) 2016-04-28 19:20:14 +02:00
Matthieu Darbois 29313eb5f1 Fix unsigned int overflow reported by UBSan (#761) 
* Fix unsigned int overflow reported by UBSan

Please add -DOPJ_UBSAN_BUILD to CFLAGS when building with
-fsanitize=undefined,unsigned-integer-overflow

It seems clang/gcc do not allow to disable checking for block of code
other than function or file.
 2016-04-28 13:16:43 +02:00
julienmalik e982d03966 Fix implicit float-to-bool conversions (#752) 2016-04-28 11:52:45 +02:00
Antonin Descampe 5c5ae1d51a fixed tile numbering 
Fixes #245
 2016-01-25 17:30:43 +01:00
Matthieu Darbois a42d03df79 Merge pull request #551 from mayeut/coc-qcc 
Add COC/QCC in main header when needed
 2016-01-11 00:23:44 +01:00
Stefan Weil 99c4f621bd Fix duplicate article in comments 
Signed-off-by: Stefan Weil <sw@weilnetz.de>
 2016-01-06 08:55:29 +01:00
mayeut 80be580d50 Merge branch 'master' into coc-qcc 2015-10-18 03:23:38 +02:00
mayeut 83249c318f Fixed crash on encoding 
Update #624
Update #625
 2015-10-17 01:30:23 +02:00
mayeut 2dab5fed59 Merge branch 'master' into coc-qcc 2015-10-06 20:55:36 +02:00
Stephan Mühlstrasser d6b51b7041 Added missing casts for return values of opj_malloc()/opj_calloc(). 2015-10-06 15:33:46 +02:00
mayeut db23831d57 Add check for seek support before trying TPsot==TNsot workaround 2015-10-02 22:33:44 +02:00
Stefan Weil 99fc1ab306 Fix typos in comments and string 
Most typos were found by codespell.

Signed-off-by: Stefan Weil <sw@weilnetz.de>
 2015-09-29 05:55:43 +02:00
mayeut 55dbf8acff Fix resource leak in opj_j2k_create_cstr_index 
Fixes #599
 2015-09-17 12:46:42 +02:00
mayeut c16c91797f Fix resource leak in opj_j2k_encode 
Fixes #598
 2015-09-17 12:41:41 +02:00
mayeut a4e93c3b0a fix resource leak in opj_j2k_decode_one_tile 
Fixes #597
 2015-09-17 12:31:18 +02:00
mayeut 66d890bf22 Merge remote-tracking branch 'uclouvain/master' into coc-qcc 2015-09-08 22:33:07 +02:00
mayeut 940100c28a Fix potential use-after-free in opj_j2k_write_mco function 
Fixes #563
 2015-09-06 17:24:03 +02:00
mayeut 60301842c7 Sync coc-qcc branch with master 2015-07-31 01:07:14 +02:00
mayeut 0df90afdf7 Remove printf/fprintf to stdout/stderr throughout openjp2 lib 
Update uclouvain/openjpeg#246
 2015-07-30 23:26:31 +02:00
mayeut c3d9719cd4 Write COC/QCC in Main Header 
Update uclouvain/openjpeg#459
 2015-07-29 00:29:49 +02:00
mayeut 2263b6bacc Start support for qcc/coc 2015-07-28 23:58:10 +02:00