PR #975 introduced a check that rejects images that have different bit depth/sign
per compoment in SIZ marker if the JP2 IHDR box has BPC != 255
This didn't work properly if decoding a .j2k file since the new bit added in
opj_cp_t wasn't initialized to the right value.
For clarity, tThis new bit has also been renamed to allow_different_bit_depth_sign
But looking closer at the code, it seems we were already tolerant to inconsistencies.
For example we parsed a JP2 BPCC box even if BPC != 255 (just a warning is emitted)
So failing hard in opj_j2k_read_siz() wouldn't be very inconsistent, and that
alone cannot protect against other issues, so just emit a warning if BPC != 255
and the SIZ marker contains different bit depth/sign per component.
Note: we could also check that the content of JP2 BPCC box is consistant with the one
of the SIZ marker.
Currently we allocate at least 8192 bytes for each codeblock, and copy
the relevant parts of the codestream in that per-codeblock buffer as we
decode packets.
As the whole codestream for the tile is ingested in memory and alive
during the decoding, we can directly point to it instead of copying. But
to do that, we need an intermediate concept, a 'chunk' of code-stream segment,
given that segments may be made of data at different places in the code-stream
when quality layers are used.
With that change, the decoding of MAPA_005.jp2 goes down from the previous
improvement of 2.7 GB down to 1.9 GB.
New profile:
n4: 1885648469 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
n1: 1610689344 0x4E78287: opj_aligned_malloc (opj_malloc.c:61)
n1: 1610689344 0x4E71D7B: opj_alloc_tile_component_data (tcd.c:676)
n1: 1610689344 0x4E7272C: opj_tcd_init_decode_tile (tcd.c:816)
n1: 1610689344 0x4E4BDD9: opj_j2k_read_tile_header (j2k.c:8618)
n1: 1610689344 0x4E4C8A2: opj_j2k_decode_tiles (j2k.c:10349)
n1: 1610689344 0x4E4E36E: opj_j2k_decode (j2k.c:7847)
n1: 1610689344 0x4E52FA2: opj_jp2_decode (jp2.c:1564)
n0: 1610689344 0x40374E: main (opj_decompress.c:1459)
n1: 219232541 0x4E4BBF0: opj_j2k_read_tile_header (j2k.c:4685)
n1: 219232541 0x4E4C8A2: opj_j2k_decode_tiles (j2k.c:10349)
n1: 219232541 0x4E4E36E: opj_j2k_decode (j2k.c:7847)
n1: 219232541 0x4E52FA2: opj_jp2_decode (jp2.c:1564)
n0: 219232541 0x40374E: main (opj_decompress.c:1459)
n1: 39822000 0x4E727A9: opj_tcd_init_decode_tile (tcd.c:1219)
n1: 39822000 0x4E4BDD9: opj_j2k_read_tile_header (j2k.c:8618)
n1: 39822000 0x4E4C8A2: opj_j2k_decode_tiles (j2k.c:10349)
n1: 39822000 0x4E4E36E: opj_j2k_decode (j2k.c:7847)
n1: 39822000 0x4E52FA2: opj_jp2_decode (jp2.c:1564)
n0: 39822000 0x40374E: main (opj_decompress.c:1459)
n0: 15904584 in 52 places, all below massif's threshold (1.00%)
b7594c0fcb9dd3aa6356d72c4a525d76168da689 may put opj_tcd_tilecomp_t->data
allocated by opj_alloc_tile_component_data() as the image->comps[].data. As
opj_alloc_tile_component_data() use opj_aligned_malloc() we must be sure to
ue opj_alined_malloc()/_free() in all places where we alloc/free
image->comps[].data.
Note: this might have some compatibility impact in case user code does itself
the allocation/free of image->comps[].data
When components don't have the same width, unaligned load/store are possible.
Fixes openjeg-crashes-2017-07-27/id:000000,sig:11,src:001342,op:flip4,pos:162.jp2 of #895
Fixes issue on id:000020,sig:06,src:001958,op:flip4,pos:149 that has two
SOT markers for the same tile with the same tile part number, causing
opj_j2k_merge_ppt() to be called several times.
Thanks to our macros that abstract SSE use, the functions can use
AVX2 when available (at compile time)
This brings an extra 23% speed improvement on bench_dwt in 64bit builds
with AVX2 compared to SSE2.
* Use single-pass lifting inverse wavelet transform.
* For vertical pass, use SSE2 when available so as to process 8 columns
in parallel. This is the most beneficial improvement, since the
vertical pass involves a lot of cache trashing.
With the bench_dwt utility with default arguments (16383x16383 image),
time goes from 4.064 s to 1.212 s.
Commit 29313eb5 introduced those flags to avoid issues with
-fsanitize=unsigned-integer-overflow
However it is better just to rewrite the loop to avoid such condition
to occur.
There are situations where, given a tile size, at a resolution level,
there are sub-bands with x0==x1 or y0==y1, that consequently don't have any
valid codeblocks, but the other sub-bands may be non-empty.
Given that we recycle the memory from one tile to another one, those
ghost codeblocks might be non-0 and thus candidate for packet inclusion.
This saves comparing the current pointer with the end of buffer pointer.
This results at least in tiny speed improvement for raw decoding, and
smaller code size for MQC as well.
This kills the remains of the raw.h/.c files that were only used for
decoding. Encoding using the mqc structure already.
Ported from Carl Hetherington work (actually through Matthieu Darbois's port
on top of OpenJPEG 2.1.0)
Can reduce total encoding time by 10-15%
WARNING: VSC mode is not implemented, and so is a temporary regression
that must be fixed.
Decoding some valid .jp2 files like Sentinel2 datasets leads to warnings like:
No incltree created.
tgt_create tree->numnodes == 0, no tree created.
No imsbtree created.
tgt_create tree->numnodes == 0, no tree created.
Besides that, the image is correctly decoded. So there is no reason to emit
those warnings.
* test_tile_decoder: Fix potential buffer overflow (coverity)
CID 1190155 (#1 of 1): Unbounded source buffer (STRING_SIZE)
Using a pointer instead of buffer of fixed size avoids the limit
for the length of the input file name.
Signed-off-by: Stefan Weil <sw@weilnetz.de>
* test_tile_encoder: Fix potential buffer overflow (coverity)
CID 1190154 (#1 of 1): Unbounded source buffer (STRING_SIZE)
Using a pointer instead of buffer of fixed size avoids the limit
for the length of the output file name. This implies that the length
can exceed 255, so the data type for variable len had to be fixed, too.
Signed-off-by: Stefan Weil <sw@weilnetz.de>
* openjpip: Initialize data before returning it
This fixes an error reported by Coverity:
CID 1190143 (#1 of 1): Uninitialized scalar variable (UNINIT)
Signed-off-by: Stefan Weil <sw@weilnetz.de>
* query_parser: Fix potential out-of-bounds read (coverity)
CID 1190207 (#1 of 1): Out-of-bounds read (OVERRUN)
Variable i must be checked before testing query_param.box_type.
Signed-off-by: Stefan Weil <sw@weilnetz.de>
* jpip_parser: Fix potential out-of-bounds read (coverity)
CID 1190206 (#1 of 1): Out-of-bounds read (OVERRUN)
Variable i must be checked before testing query_param.box_type.
Signed-off-by: Stefan Weil <sw@weilnetz.de>
Visual Studio 2015 does not pass regression tests with `__restrict` so kept disabled for MSVC.
Need to check proper usage of OPJ_RESTRICT (if correct then there’s
probably a bug in vc14)
Closes#661
The definition of bit-fields with type OPJ_UINT32 caused complilation errors
on IBM iSeries, because OPJ_UINT32 is defined as uint32_t, and
uint32_t is defined as unsigned long in <stdint.h>. The definition of
bit-fields with an integer type of a specific size doesn't make sense
anyway.
The type casts which used this data type can be removed by changing
the signature of function swap16. As this function is called with
unsigned variables, this change is reasonable.
Signed-off-by: Stefan Weil <sw@weilnetz.de>
openjpeg provides libopenjp2.pc, so the require statements must refer to
libopenjp2 instead of openjp2.
Fixes#594
Signed-off-by: Stefan Weil <sw@weilnetz.de>
By default, only the main thread is used. If opj_codec_set_threads() is not used,
but the OPJ_NUM_THREADS environment variable is set, its value will be
used to initialize the number of threads. The value can be either an integer
number, or "ALL_CPUS". If OPJ_NUM_THREADS is set and this function is called,
this function will override the behaviour of the environment variable.
Addition flag array such that colflags[1+0] is for state of col=0,row=0..3,
colflags[1+1] for col=1, row=0..3, colflags[1+flags_stride] for col=0,row=4..7, ...
This array avoids too much cache trashing when processing by 4 vertical samples
as done in the various decoding steps.
Add a opj_t1_dec_clnpass_step_only_if_flag_not_sig_visit() method that
does the job of opj_t1_dec_clnpass_step_only() assuming the conditions
are met. And use it in opj_t1_dec_clnpass(). The compiler generates
more efficient code.
Fix issue 135
The fix is legal regarding the standard but I did not manage to find out if it covers a bug in opj_t2_read_packet_data or if the file is corrupted
42x Images with an odd x0/y0 lead to subsampled component starting at the
2nd column/line.
That is offset = comp->dx * comp->x0 - image->x0 = 1
Fix#726
When all components do not have the same dx/dy, components buffer are
read beyond their end.
Do not convert in this case.
Update uclouvain/openjpeg#725
* Fix unsigned int overflow reported by UBSan
Please add -DOPJ_UBSAN_BUILD to CFLAGS when building with
-fsanitize=undefined,unsigned-integer-overflow
It seems clang/gcc do not allow to disable checking for block of code
other than function or file.
dwt_interleave_h.gsr105.jp2 now has the same output as kakadu
issue399 is corrupted. Only the corrupted part changes.
Update known failures for x86 MD5
NR-DEC-kodak_2layers_lrcp.j2c-31-decode-md5
NR-DEC-kodak_2layers_lrcp.j2c-32-decode-md5
NR-DEC-issue135.j2k-68-decode-md5
Follow-up of #757
This shall have no performance impact on 2’s complement machine where
the compiler replaces the multiplication by power of two (constant) by
a left shift.
Verified at least on MacOS Xcode 7.3, same assembly generated after fix.
This shall have no performance impact on 2’s complement machine where
the compiler replaces the multiplication by power of two (constant) by
a left shift.
Verified at least on MacOS Xcode 7.3, same assembly generated after fix.
When trying the GDAL OpenJPEG driver against openjpeg current master HEAD,
I get failures when trying to create .jp2 files. The driver uses
opj_write_tile() and in some tests numresolutions = 1.
In openjp2/dwt.c:410, l_data_size = opj_dwt_max_resolution( tilec->resolutions,tilec->numresolutions) * (OPJ_UINT32)sizeof(OPJ_INT32);
is called and returns l_data_size = 0. Now in git opj_malloc() has a special case
for 0 to return a NULL pointer whereas previously it relied on system malloc(),
which in my case didn't return NULL.
So only test the pointer value if l_data_size != 0. This makes the GDAL
autotest suite to pass again.
By default, OpenJPEG uses the function memalign to allocate aligned
memory on Linux systems. That function needs malloc.h which was
missing. This results in a compiler warning:
openjpeg/src/lib/openjp2/opj_malloc.c:63:3: warning:
implicit declaration of function ‘memalign’
[-Wimplicit-function-declaration]
On hosts where sizeof(int) < sizeof(void *) the return value of memalign
will be truncated which results in an invalid pointer.
That caused "make test" to produce lots of segmentation faults when
running on a 64 bit Linux host.
Signed-off-by: Stefan Weil <sw@weilnetz.de>
posix_memalign is only declared conditionally in stdlib.h,
so add one of the possible definitions to get the declaration.
Signed-off-by: Stefan Weil <sw@weilnetz.de>
The png format is also supported, so add it to the message.
Remove also the unneeded blank character before \n.
Signed-off-by: Stefan Weil <sw@weilnetz.de>
Fix this and other similar compiler warnings:
src/bin/jp2/convert.c: In function ‘tga_readheader’:
src/bin/jp2/convert.c:595:5: warning:
dereferencing type-punned pointer will break strict-aliasing rules
[-Wstrict-aliasing]
cmap_len = get_ushort(*(unsigned short*)(&tga[5]));
Signed-off-by: Stefan Weil <sw@weilnetz.de>
HP compiler warns:
cc: "dwt.c", line 798: warning 562: Redeclaration of "opj_v4dwt_decode"
with a different storage class specifier: "opj_v4dwt_decode" will have
internal linkage.
cc: "t2.c", line 1341: warning 562: Redeclaration of "opj_t2_init_seg"
with a different storage class specifier: "opj_t2_init_seg" will have
internal linkage.
The static code analyzer cppcheck warns about unsigned integers
which use "%d" in the format string.
It also warns about an unneeded assignment.
Fix both issues.
Signed-off-by: Stefan Weil <sw@weilnetz.de>
if (!JPWL_ASSUME)
return false;
meaning that once JPWL_ASSUME=1 the code will pass over such errors and try to decode anyway (just paired with the other JPWL_ASSUME on line 1112).
Fixes#596
g++ complains about invalid conversions like these ones:
error: invalid conversion from ‘const char*’ to ‘char*’ [-fpermissive]
error: invalid conversion from ‘void*’ to ‘opj_precision* {aka opj_prec*}’ [-fpermissive]
Signed-off-by: Stefan Weil <sw@weilnetz.de>
Even Rouault
Antonin Descampe
Antonin Descampe
szukw000
Jeroen
Even Rouault
Russell McOrmond
Mathieu Malaterre
Matthieu Darbois
Stefan Weil
trylab
Stephan Mühlstrasser
Stephan Mühlstrasser
maddin200
Kal Conley
Julien Malik
julienmalik
Aaron Boxer
Jakub Wilk
Giuseppe Baruffa