Add a -allow-partial option to opj_decompress utility and a opj_decoder_set_strict_mode() option to the API
Co-authored-by: Chris Hafey <chafey@gmail.com>
There are a few limitations:
- mixed mode (HT and regular code blocks) is not supported.
- ROI in HT blocks is not supported.
- Placeholder passes are not supported.
- MultiHT sets are not support, only a singleHT set.
- there are known issues with some compliance testing files related to
the parsing of packet header.
* -PLT switch added to opj_compress
* Add a opj_encoder_set_extra_options() function that
accepts a PLT=YES option, and could be expanded later
for other uses.
-------
Testing with a Sentinel2 10m band, T36JTT_20160914T074612_B02.jp2,
coming from S2A_MSIL1C_20160914T074612_N0204_R135_T36JTT_20160914T081456.SAFE
Decompress it to TIFF:
```
opj_uncompress -i T36JTT_20160914T074612_B02.jp2 -o T36JTT_20160914T074612_B02.tif
```
Recompress it with similar parameters as original:
```
opj_compress -n 5 -c [256,256],[256,256],[256,256],[256,256],[256,256] -t 1024,1024 -PLT -i T36JTT_20160914T074612_B02.tif -o T36JTT_20160914T074612_B02_PLT.jp2
```
Dump codestream detail with GDAL dump_jp2.py utility (https://github.com/OSGeo/gdal/blob/master/gdal/swig/python/samples/dump_jp2.py)
```
python dump_jp2.py T36JTT_20160914T074612_B02.jp2 > /tmp/dump_sentinel2_ori.txt
python dump_jp2.py T36JTT_20160914T074612_B02_PLT.jp2 > /tmp/dump_sentinel2_openjpeg_plt.txt
```
The diff between both show very similar structure, and identical number of packets in PLT markers
Now testing with Kakadu (KDU803_Demo_Apps_for_Linux-x86-64_200210)
Full file decompression:
```
kdu_expand -i T36JTT_20160914T074612_B02_PLT.jp2 -o tmp.tif
Consumed 121 tile-part(s) from a total of 121 tile(s).
Consumed 80,318,806 codestream bytes (excluding any file format) = 5.329697
bits/pel.
Processed using the multi-threaded environment, with
8 parallel threads of execution
```
Partial decompresson (presumably using PLT markers):
```
kdu_expand -i T36JTT_20160914T074612_B02.jp2 -o tmp.pgm -region "{0.5,0.5},{0.01,0.01}"
kdu_expand -i T36JTT_20160914T074612_B02_PLT.jp2 -o tmp2.pgm -region "{0.5,0.5},{0.01,0.01}"
diff tmp.pgm tmp2.pgm && echo "same !"
```
-------
Funded by ESA for S2-MPC project
Currently we allocate at least 8192 bytes for each codeblock, and copy
the relevant parts of the codestream in that per-codeblock buffer as we
decode packets.
As the whole codestream for the tile is ingested in memory and alive
during the decoding, we can directly point to it instead of copying. But
to do that, we need an intermediate concept, a 'chunk' of code-stream segment,
given that segments may be made of data at different places in the code-stream
when quality layers are used.
With that change, the decoding of MAPA_005.jp2 goes down from the previous
improvement of 2.7 GB down to 1.9 GB.
New profile:
n4: 1885648469 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
n1: 1610689344 0x4E78287: opj_aligned_malloc (opj_malloc.c:61)
n1: 1610689344 0x4E71D7B: opj_alloc_tile_component_data (tcd.c:676)
n1: 1610689344 0x4E7272C: opj_tcd_init_decode_tile (tcd.c:816)
n1: 1610689344 0x4E4BDD9: opj_j2k_read_tile_header (j2k.c:8618)
n1: 1610689344 0x4E4C8A2: opj_j2k_decode_tiles (j2k.c:10349)
n1: 1610689344 0x4E4E36E: opj_j2k_decode (j2k.c:7847)
n1: 1610689344 0x4E52FA2: opj_jp2_decode (jp2.c:1564)
n0: 1610689344 0x40374E: main (opj_decompress.c:1459)
n1: 219232541 0x4E4BBF0: opj_j2k_read_tile_header (j2k.c:4685)
n1: 219232541 0x4E4C8A2: opj_j2k_decode_tiles (j2k.c:10349)
n1: 219232541 0x4E4E36E: opj_j2k_decode (j2k.c:7847)
n1: 219232541 0x4E52FA2: opj_jp2_decode (jp2.c:1564)
n0: 219232541 0x40374E: main (opj_decompress.c:1459)
n1: 39822000 0x4E727A9: opj_tcd_init_decode_tile (tcd.c:1219)
n1: 39822000 0x4E4BDD9: opj_j2k_read_tile_header (j2k.c:8618)
n1: 39822000 0x4E4C8A2: opj_j2k_decode_tiles (j2k.c:10349)
n1: 39822000 0x4E4E36E: opj_j2k_decode (j2k.c:7847)
n1: 39822000 0x4E52FA2: opj_jp2_decode (jp2.c:1564)
n0: 39822000 0x40374E: main (opj_decompress.c:1459)
n0: 15904584 in 52 places, all below massif's threshold (1.00%)
There are situations where, given a tile size, at a resolution level,
there are sub-bands with x0==x1 or y0==y1, that consequently don't have any
valid codeblocks, but the other sub-bands may be non-empty.
Given that we recycle the memory from one tile to another one, those
ghost codeblocks might be non-0 and thus candidate for packet inclusion.
This saves comparing the current pointer with the end of buffer pointer.
This results at least in tiny speed improvement for raw decoding, and
smaller code size for MQC as well.
This kills the remains of the raw.h/.c files that were only used for
decoding. Encoding using the mqc structure already.
HP compiler warns:
cc: "dwt.c", line 798: warning 562: Redeclaration of "opj_v4dwt_decode"
with a different storage class specifier: "opj_v4dwt_decode" will have
internal linkage.
cc: "t2.c", line 1341: warning 562: Redeclaration of "opj_t2_init_seg"
with a different storage class specifier: "opj_t2_init_seg" will have
internal linkage.
Even Rouault
Robert Gabriel Jakabosky
Stefan Weil
Aous Naman
mayeut
Stephan Mühlstrasser
Antonin Descampe
Mathieu Malaterre