Commit Graph

78 Commits

Author SHA1 Message Date
Even Rouault 4edb8c8337
Add support for generation of PLT markers in encoder
* -PLT switch added to opj_compress
* Add a opj_encoder_set_extra_options() function that
  accepts a PLT=YES option, and could be expanded later
  for other uses.

-------

Testing with a Sentinel2 10m band, T36JTT_20160914T074612_B02.jp2,
coming from S2A_MSIL1C_20160914T074612_N0204_R135_T36JTT_20160914T081456.SAFE

Decompress it to TIFF:
```
opj_uncompress -i T36JTT_20160914T074612_B02.jp2 -o T36JTT_20160914T074612_B02.tif
```

Recompress it with similar parameters as original:
```
opj_compress -n 5 -c [256,256],[256,256],[256,256],[256,256],[256,256] -t 1024,1024 -PLT -i T36JTT_20160914T074612_B02.tif -o T36JTT_20160914T074612_B02_PLT.jp2
```

Dump codestream detail with GDAL dump_jp2.py utility (https://github.com/OSGeo/gdal/blob/master/gdal/swig/python/samples/dump_jp2.py)
```
python dump_jp2.py T36JTT_20160914T074612_B02.jp2 > /tmp/dump_sentinel2_ori.txt
python dump_jp2.py T36JTT_20160914T074612_B02_PLT.jp2 > /tmp/dump_sentinel2_openjpeg_plt.txt
```

The diff between both show very similar structure, and identical number of packets in PLT markers

Now testing with Kakadu (KDU803_Demo_Apps_for_Linux-x86-64_200210)

Full file decompression:
```
kdu_expand -i T36JTT_20160914T074612_B02_PLT.jp2 -o tmp.tif

Consumed 121 tile-part(s) from a total of 121 tile(s).
Consumed 80,318,806 codestream bytes (excluding any file format) = 5.329697
bits/pel.
Processed using the multi-threaded environment, with
    8 parallel threads of execution
```

Partial decompresson (presumably using PLT markers):
```
kdu_expand -i T36JTT_20160914T074612_B02.jp2 -o tmp.pgm -region "{0.5,0.5},{0.01,0.01}"
kdu_expand -i T36JTT_20160914T074612_B02_PLT.jp2 -o tmp2.pgm  -region "{0.5,0.5},{0.01,0.01}"
diff tmp.pgm tmp2.pgm && echo "same !"
```

-------

Funded by ESA for S2-MPC project
2020-04-21 15:55:44 +02:00
Even Rouault 1e3a57563d
compression: emit POC marker when only one single POC is requested (fixes #1191) 2019-04-25 14:40:55 +02:00
Even Rouault 07d526e4cb opj_t2_encode_packet(): disable setting empty packet header bit to 1 when there is an empty packet
This effectively reverts commit 2609fb8077
since it has been reported that
such packets cause decoding issues with cinema J2K hardware
decoders: https://groups.google.com/forum/#!topic/openjpeg/M7M_fLX_Bco
2018-01-08 09:38:44 +01:00
Even Rouault fe338a057c Sub-tile decoding: only decode precincts and codeblocks that intersect the window specified in opj_set_decode_area() 2017-08-17 19:05:54 +02:00
Even Rouault c535531f03 opj_t2_encode_packet(): fix potential write heap buffer overflow (#992) 2017-08-16 17:20:29 +02:00
Even Rouault 9624b2fa47 opj_t2_encode_packet(): only emit an error about insufficiently large output buffer in FINAL_PASS mode. Fixes (master-only) regression added in 0b4fef6d19 2017-08-16 12:52:33 +02:00
Even Rouault a35b489134 Fix argument order in error message of previous commit 2017-08-10 16:58:36 +02:00
Even Rouault 0b4fef6d19 Propagate event manager down to opj_t2_encode_packet() and use it to emit an error message when the output buffer is too small 2017-08-10 16:49:47 +02:00
Even Rouault 92114694a4 Slight improvement in management of code block chunks
Instead of having the chunk array at the segment level, we can move it down to
the codeblock itself since segments are filled in sequential order.
Limit the number of memory allocation, and decrease slightly the memory usage.

On MAPA_005.jp2

n4: 1871312549 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n1: 1610689344 0x4E781E7: opj_aligned_malloc (opj_malloc.c:61)
  n1: 1610689344 0x4E71D1B: opj_alloc_tile_component_data (tcd.c:676)
   n1: 1610689344 0x4E726CF: opj_tcd_init_decode_tile (tcd.c:816)
    n1: 1610689344 0x4E4BE39: opj_j2k_read_tile_header (j2k.c:8617)
     n1: 1610689344 0x4E4C902: opj_j2k_decode_tiles (j2k.c:10348)
      n1: 1610689344 0x4E4E3CE: opj_j2k_decode (j2k.c:7846)
       n1: 1610689344 0x4E53002: opj_jp2_decode (jp2.c:1564)
        n0: 1610689344 0x40374E: main (opj_decompress.c:1459)
 n1: 219232541 0x4E4BC50: opj_j2k_read_tile_header (j2k.c:4683)
  n1: 219232541 0x4E4C902: opj_j2k_decode_tiles (j2k.c:10348)
   n1: 219232541 0x4E4E3CE: opj_j2k_decode (j2k.c:7846)
    n1: 219232541 0x4E53002: opj_jp2_decode (jp2.c:1564)
     n0: 219232541 0x40374E: main (opj_decompress.c:1459)
 n1: 23893200 0x4E72735: opj_tcd_init_decode_tile (tcd.c:1225)
  n1: 23893200 0x4E4BE39: opj_j2k_read_tile_header (j2k.c:8617)
   n1: 23893200 0x4E4C902: opj_j2k_decode_tiles (j2k.c:10348)
    n1: 23893200 0x4E4E3CE: opj_j2k_decode (j2k.c:7846)
     n1: 23893200 0x4E53002: opj_jp2_decode (jp2.c:1564)
      n0: 23893200 0x40374E: main (opj_decompress.c:1459)
 n0: 17497464 in 52 places, all below massif's threshold (1.00%)
2017-08-07 18:32:52 +02:00
Even Rouault ca34d13e76 Decoding: do not allocate memory for the codestream of each codeblock
Currently we allocate at least 8192 bytes for each codeblock, and copy
the relevant parts of the codestream in that per-codeblock buffer as we
decode packets.
As the whole codestream for the tile is ingested in memory and alive
during the decoding, we can directly point to it instead of copying. But
to do that, we need an intermediate concept, a 'chunk' of code-stream segment,
given that segments may be made of data at different places in the code-stream
when quality layers are used.

With that change, the decoding of MAPA_005.jp2 goes down from the previous
improvement of 2.7 GB down to 1.9 GB.

New profile:

n4: 1885648469 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n1: 1610689344 0x4E78287: opj_aligned_malloc (opj_malloc.c:61)
  n1: 1610689344 0x4E71D7B: opj_alloc_tile_component_data (tcd.c:676)
   n1: 1610689344 0x4E7272C: opj_tcd_init_decode_tile (tcd.c:816)
    n1: 1610689344 0x4E4BDD9: opj_j2k_read_tile_header (j2k.c:8618)
     n1: 1610689344 0x4E4C8A2: opj_j2k_decode_tiles (j2k.c:10349)
      n1: 1610689344 0x4E4E36E: opj_j2k_decode (j2k.c:7847)
       n1: 1610689344 0x4E52FA2: opj_jp2_decode (jp2.c:1564)
        n0: 1610689344 0x40374E: main (opj_decompress.c:1459)
 n1: 219232541 0x4E4BBF0: opj_j2k_read_tile_header (j2k.c:4685)
  n1: 219232541 0x4E4C8A2: opj_j2k_decode_tiles (j2k.c:10349)
   n1: 219232541 0x4E4E36E: opj_j2k_decode (j2k.c:7847)
    n1: 219232541 0x4E52FA2: opj_jp2_decode (jp2.c:1564)
     n0: 219232541 0x40374E: main (opj_decompress.c:1459)
 n1: 39822000 0x4E727A9: opj_tcd_init_decode_tile (tcd.c:1219)
  n1: 39822000 0x4E4BDD9: opj_j2k_read_tile_header (j2k.c:8618)
   n1: 39822000 0x4E4C8A2: opj_j2k_decode_tiles (j2k.c:10349)
    n1: 39822000 0x4E4E36E: opj_j2k_decode (j2k.c:7847)
     n1: 39822000 0x4E52FA2: opj_jp2_decode (jp2.c:1564)
      n0: 39822000 0x40374E: main (opj_decompress.c:1459)
 n0: 15904584 in 52 places, all below massif's threshold (1.00%)
2017-08-07 18:32:52 +02:00
Even Rouault 373520db30 Add documentation for magic values in the code 2017-08-07 18:32:52 +02:00
Even Rouault 83342f2aaf Fix Doxygen warnings (patch derived from Winfried's doxygen-dif.txt.zip, #849) 2017-07-30 18:18:59 +02:00
Even Rouault 6c5fe9407b Avoid potential undefined shift behaviour in opj_bio_read() from opj_t2_read_packet_header()
Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=2485
Credit to OSS Fuzz
2017-07-26 22:22:44 +02:00
Even Rouault 9cbc9903c3 Merge branch 't1_flag_optimizations' 2017-06-13 12:09:52 +02:00
Even Rouault 2609fb8077 Packet header writing: set empty packet header bit to 0 when appropriate (small optimization) 2017-06-12 18:38:11 +02:00
Even Rouault 73d1510d47 Encoder: fix packet writing of empty sub-bands (#891, #892)
There are situations where, given a tile size, at a resolution level,
there are sub-bands with x0==x1 or y0==y1, that consequently don't have any
valid codeblocks, but the other sub-bands may be non-empty.
Given that we recycle the memory from one tile to another one, those
ghost codeblocks might be non-0 and thus candidate for packet inclusion.
2017-06-12 18:37:50 +02:00
Even Rouault 532243f1fd MQC/RAW decoder: use an artificial 0xFF 0xFF terminating marker.
This saves comparing the current pointer with the end of buffer pointer.
This results at least in tiny speed improvement for raw decoding, and
smaller code size for MQC as well.

This kills the remains of the raw.h/.c files that were only used for
decoding. Encoding using the mqc structure already.
2017-06-02 18:24:07 +02:00
Even Rouault 563bd8499e Reformat whole codebase with astyle.options (#128) 2017-05-09 20:46:20 +02:00
mayeut 1a8318f6c2 Fix Out-of-Bounds Access in function opj_tgt_reset
Fix uclouvain/openjpeg#775
2016-05-08 20:10:13 +02:00
Stephan Mühlstrasser a1fc83cc25 Fix HP compiler warning about redeclaration of function (#640)
HP compiler warns:
cc: "dwt.c", line 798: warning 562: Redeclaration of "opj_v4dwt_decode"
with a different storage class specifier: "opj_v4dwt_decode" will have
internal linkage.
cc: "t2.c", line 1341: warning 562: Redeclaration of "opj_t2_init_seg"
with a different storage class specifier: "opj_t2_init_seg" will have
internal linkage.
2015-10-19 12:14:27 +02:00
mayeut 0df90afdf7 Remove printf/fprintf to stdout/stderr throughout openjp2 lib
Update uclouvain/openjpeg#246
2015-07-30 23:26:31 +02:00
mayeut ae4799ad07 Add some missing static
Still needs to check j2k.c & jp2.c
Update uclouvain/openjpeg#243
2015-07-18 02:39:32 +02:00
Matthieu Darbois a2c2d86d3d [trunk] fixed memleak in abnormal condition (fixes issue 437) 2014-11-20 20:13:47 +00:00
Antonin Descampe f8796711e8 [trunk] add a check on progression order (fixes issue 413) 2014-10-15 10:14:54 +00:00
Antonin Descampe eb7c6d295a [trunk] fixed potential negative size params (fixes issue 390) 2014-10-06 21:05:32 +00:00
Antonin Descampe 48832d944f [trunk]fixes buffer overflow in t2.c
Update issue 390
Summary: check for negative-size params in code
Patch from Matthieu applied. Left open to apply this to other places in
code.
2014-10-03 10:15:28 +00:00
Antonin Descampe 45f33cdbd6 [trunk] check possible overflow on size in opj_t2_read_packet_data
Update issue 390
Labels: Priority-Critical
Should be rechecked with ASan and pdfium
2014-09-30 10:07:09 +00:00
Antonin Descampe 2f3fd1194e [trunk] prevent overflow in opj_t2_read_packet_header (update issue 389) 2014-09-30 09:26:44 +00:00
Antonin Descampe f08af3ed82 [trunk] Ensure cblk->data pointer is correctly initialized (updates issue 391). Still need to check if it does not fail anymore with ASan. 2014-09-29 12:03:50 +00:00
Antonin Descampe 6868ee373e added memory allocation checks (fixes issue 355) 2014-09-19 10:26:35 +00:00
Antonin Descampe 50a205d20f [trunk] improve memory management (fixes issue 359) 2014-09-16 15:48:04 +00:00
Antonin Descampe 7199d9b5d3 [trunk] refactoring of rsiz, profiles, and extensions management 2014-04-17 09:31:37 +00:00
Antonin Descampe d19a4ab676 [trunk] updated copyright and added copyright notice required by ISO, in each file; updated AUTHORS, NEWS 2014-04-03 15:30:57 +00:00
Mathieu Malaterre f4707da97a [trunk] Revert r2764. It breaks ~230 tests as seen on continuous. Since no dataset is available, the issue 296 will be left open.
Update issue 296
2014-03-18 15:19:29 +00:00
Mathieu Malaterre 6d9e8a5a63 [trunk] Import git commit fc884aee2b69c78500e65c3d05bf216791a9ea4a from ghostscript team
Fixes issue 296
2014-03-18 14:43:47 +00:00
Mathieu Malaterre 008de86533 [trunk] Import git commit 99a6f1af177c15f4db475186b79d169c993494ef from ghostscript team
Update issue 296
2014-03-18 14:43:18 +00:00
Mathieu Malaterre fd3f7ca2bb [trunk] Remove warnings about unused values introduced in r2710
Thanks to Matthieu Darbois for patch
Fixes issue 290
2014-03-14 08:44:53 +00:00
Mathieu Malaterre 320784659d [trunk] Just in case this would be helpful for other, here is the code used to track issue 80
Update issue 80
2014-03-13 10:56:48 +00:00
Mathieu Malaterre 51ab8cd67d [trunk] Tweak documentation in t2.c. Be more verbose in case of failure. 2014-03-13 10:46:54 +00:00
Mathieu Malaterre 59270c39a7 [trunk] Re-active old warning about missing SOP marker 2014-03-11 15:21:12 +00:00
Mathieu Malaterre db82cf7778 [trunk] Add debug info to track issue 80
Update issue 80
2014-03-11 15:17:39 +00:00
Mathieu Malaterre ddb1d57766 [trunk] Rework assertion to work on 32bits system 2014-03-11 12:56:52 +00:00
Mathieu Malaterre 5dd770746b [trunk] Fix remainings warnings on linux/32bits arch 2014-03-11 10:28:04 +00:00
Mathieu Malaterre 1778e2576e [trunk] Another round of fixes for sign conversion warnings.
Update issue 256
2014-03-10 16:17:12 +00:00
Mathieu Malaterre 581d2ba8b4 [trunk] Rework r2597, make sure test suite is passing now.
Update issue 256
2014-03-07 13:50:58 +00:00
Mathieu Malaterre 33c472d734 [trunk] Redo r2623, and remove guilty section
Update issue 256
2014-03-07 13:29:35 +00:00
Mathieu Malaterre a8eb122c1f [trunk] Revert r2623 for now 2014-03-07 13:14:26 +00:00
Mathieu Malaterre 01b4bb5447 [trunk] Another round of fixes for sign conversion warnings.
Update issue 256
2014-03-07 11:57:37 +00:00
Mathieu Malaterre 44ca89eff7 [trunk] Remove remainings conversion issues in t2.c + clock.c 2014-03-03 16:08:03 +00:00
Mathieu Malaterre 765c4ab4f4 [trunk] rework code to avoid a warning. this also remove a division and prefer bit-shift 2014-03-03 16:02:16 +00:00