Reducing storage of similar PNGs by compressing them into a video and retrieving them losslessly--possibility or dumb idea?

What sort of images are they? Are they continuous-tone, or paletted?

I have had good experience using JXL, which has a lossless mode. Its lossy mode with a high quality setting could also be useful.

Re video codecs, you should investigate what people are using for screen recording.

You again! That's three for three now; thank you for always responding to me.

What sort of images are they? Are they continuous-tone, or paletted?

They're regular PNG images, and they have to be PNG because JPEGs don't support alpha and HEVCs are just too computationally expensive for my purposes.

I have had good experience using JXL, which has a lossless mode. Its lossy mode with a high quality setting could also be useful.

Assuming you're referring to JPEG-XL, it's likely not going to work as speedily with UIImage, which is my main use case. Are you suggesting that when I should convert to JXL when I need to minimize storage and then convert back to PNG?

Re video codecs, you should investigate what people are using for screen recording.

Would you mind elaborating on that? I don't see your point

@endecotp I did some reading on JXL and it looks promising. The format seems to have some inter-frame compression for animation, which might be what I'm looking for. On top of that the encode/decode speed seems good, but that depends on how well the iOS implementation runs. I need to run tests.

What do you use to incorporate JXL into your iOS projects? Unless something is by a trusted or official source, I have to go through every single line, which means something like https://github.com/awxkee/jxl-coder-swift is prohibitively long to get through.

I'm using the JPEG XL reference implementation - https://github.com/libjxl

I'm creating GPU textures from the decompressed images, so UIImage is not a concern.

I've found this to have reasonably good performance, certainly better than the open-source JPEG200 implementations, but not as good as libjpeg-turbo for regular JPEGs. I've not compared it with decoding PNGs.

@endecotp Thank you. I'm going to try to compile it for iOS and then incorporate wrappers around it. So much to learn, so little time.

Does the build instructions in the docs work for iOS/arm?

cd libjxl
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=OFF ..
cmake --build . -- -j$(nproc)

That cmake invocation will build macOS libraries I think.

I'm not a cmake expert. Search github for "ios-cmake".

@endecotp

If something builds for arm64 macOS, shouldn't it work with iOS/iPadOS, which is also arm64?

Anyways, I might have misunderstood you earlier--are you saying you use libjxl during development but don't include it as part of your runtime? Because I thought you included it and that making libjxl work on iOS is an easy/supported process.

I'm scratching my head trying to figure out how to compile libjxl to include in my binary but the more I look at the problem, the more it seems like this is far more than what a complete novice at cmake should even attempt, especially when a professional says "This coder does not supports JPEG-XL encoding (Because I have no time :))"

Looks like I need to go back to using video as storage, or maybe try AVIF/webp/jpeg2000

If something builds for arm64 macOS, shouldn't it work with iOS/iPadOS, which is also arm64?

No, unfortunately not.

I thought you included it and that making libjxl work on iOS is an easy/supported process.

I do include it in my iOS app at runtime. Doing so was not a particularly easy process. I used something from github called "ios-cmake", which is a pile of scripts and/or cmake configuration fragments.

I do include it in my iOS app at runtime. Doing so was not a particularly easy process. I used something from github called "ios-cmake", which is a pile of scripts and/or cmake configuration fragments.

0  comments

I'm experiencing first-hand how uneasy this is. Especially since I'm new to CMake. I was able to prepare the compile after making minor adjustments to the CMakeLists.txt files of outdated dependencies, but after that I've been unable to do the actual build process using ios-cmake due to errors

cmake -B build -G Xcode -DCMAKE_TOOLCHAIN_FILE=../../ios.toolchain.cmake -DPLATFORM=OS64 -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=OFF .. cmake --build . --config Release

Yields the error

** BUILD FAILED **

The following build commands failed:
	Ld /Users/username/Desktop/libjxl-main/build/build/lib/Release-iphoneos/libjxl_cms.0.10.2.dylib normal (in target 'jxl_cms' from project 'LIBJXL')
(1 failure)

It feels like I'm close, but I can't even gauge whether I'm close to figuring this out or whether I'm actually miles away--there's much uncertainty when you're new to something.

I normally tell cmake to build Makefiles (which is possibly the default?), rather than using -G Xcode.

Is that error telling you that it is trying to build a dynamic library? For iOS builds, you probably want to tell it to build only a static library, and not to build executables i.e. the cjxl / djxl utilities. Something like -DBUILD_SHARED_LIBS=false ? If you're lucky, there will be a cmake invocation that will list all the available flags. You're already turning off testing, see what else there is.

I don't know what I'm doing, so I'm using -G Xcode simply because that's what's in the ios-cmake toolchain example. I took your advice and looked for flags to set, and this is my new cmake invocation (tl;dr everything off except static flag, which invokes -DBUILD_SHARED_LIBS=OFF and required files):

cmake -B build \
-DJPEGXL_ENABLE_FUZZERS=OFF \
-DJPEGXL_ENABLE_DEVTOOLS=OFF \
-DJPEGXL_ENABLE_TOOLS=OFF \
-DJPEGXL_ENABLE_JPEGLI=OFF \
-DJPEGXL_INSTALL_JPEGLI_LIBJPEG=OFF \
-DJPEGXL_ENABLE_DOXYGEN=OFF \
-DJPEGXL_ENABLE_BENCHMARK=OFF \
-DJPEGXL_ENABLE_EXAMPLES=OFF \
-DJPEGXL_BUNDLE_LIBPNG=OFF \
-DJPEGXL_ENABLE_JNI=OFF \
-DJPEGXL_ENABLE_SJPEG=OFF \
-DJPEGXL_ENABLE_OPENEXR=OFF \
-DJPEGXL_ENABLE_SKCMS=ON \
-DJPEGXL_ENABLE_VIEWERS=OFF \
-DJPEGXL_ENABLE_PLUGINS=OFF \
-DJPEGXL_ENABLE_COVERAGE=OFF \
-DJPEGXL_ENABLE_TRANSCODE_JPEG=OFF \
-DJPEGXL_STATIC=ON \
-DJPEGXL_TEST_TOOLS=OFF \
-DJPEGXL_ENABLE_AVX512=OFF \
-DJPEGXL_ENABLE_AVX512_SPR=OFF \
-DJPEGXL_ENABLE_AVX512_ZEN4=OFF \
-DJPEGXL_FORCE_SYSTEM_BROTLI=OFF \
-DJPEGXL_FORCE_SYSTEM_GTEST=ON \
-DJPEGXL_FORCE_SYSTEM_LCMS2=ON \
-DJPEGXL_FORCE_SYSTEM_HWY=OFF \
-DBUILD_SHARED_LIBS=OFF \
-DCMAKE_TOOLCHAIN_FILE=../../ios.toolchain.cmake \
-DPLATFORM=OS64 \
-DCMAKE_BUILD_TYPE=Release \
-DBUILD_TESTING=OFF ..

With -G Xcode I'm back to the same errors as above with highway and brotli, which means this is a dependency issue even with flags set.

Without it, I get the following error:

[  6%] Linking CXX executable hwy_list_targets.app/hwy_list_targets
ld: library not found for -lcrt0.o
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [third_party/highway/hwy_list_targets.app/hwy_list_targets] Error 1
make[1]: *** [third_party/highway/CMakeFiles/hwy_list_targets.dir/all] Error 2
make: *** [all] Error 2

At this point, I have to say that this has been a learning experience but this is too much to just evaluate whether JXL fits my needs. I need to get back to productive work.

Can I make you an offer? If you compiled JXL yourself so it works on your iOS runtime, with encode/decode and animation functions intact and performant, I will gladly pay you for detailed instructions on how to replicate your success, and also a bit more to thank you for your help here and elsewhere. We can discuss the details elsewhere.

Note that I'm not asking for a compiled product (security issues), but for the instructions on how to get from the reference repository to iOS runtime with the requested functions.

% cd /usr/local/src/
% git clone https://github.com/leetal/ios-cmake.git

% cd /usr/local/src/
% git clone https://github.com/libjxl/libjxl.git --recursive --shallow-submodules
% cd libjxl

At this point I hacked third_party/brotli/CMakeLists.txt to disable three lines around line 214:

#  install(
#    TARGETS brotli
#    RUNTIME DESTINATION "${CMAKE_INSTALL_BINDIR}"
#  )

Without this, I was getting this error:

CMake Error at third_party/brotli/CMakeLists.txt:214 (install):
  install TARGETS given no BUNDLE DESTINATION for MACOSX_BUNDLE executable
  target "brotli".

The clue is the word "executable" in the message; we don't want it to try to build any executables, but it seemed to be trying to build a brotli executable. My change is a hack. Anyway:

% mkdir build.ios
% cd build.ios
% cmake .. \
  -DCMAKE_TOOLCHAIN_FILE=../../ios-cmake/ios.toolchain.cmake \
  -DPLATFORM=OS64 \
  -DENABLE_BITCODE=false \
  -DENABLE_VISIBILITY=true \
  -DCMAKE_INSTALL_PREFIX=/some/path/ \
  -DBUILD_SHARED_LIBS=false \
  -DCMAKE_BUILD_TYPE=Release \
  -DBUILD_TESTING=OFF \
  -DJPEGXL_ENABLE_TOOLS=false \
  -DJPEGXL_ENABLE_MANPAGES=false \
  -DJPEGXL_ENABLE_BENCHMARK=false \
  -DJPEGXL_ENABLE_EXAMPLES=false \
  -DJPEGXL_ENABLE_JNI=false \
  -DJPEGXL_ENABLE_FUZZERS=false
% make
% make install

You should now have .a static libraries in /some/path/lib

Let me know if that works for you.

Thank you so very much.

In the interim I had already opened an account on Upwork, waded through LLM spam and grifters, found someone willing to do it for $150, had money persistently rejected by Upwork, found its support intentionally disabled, then closed my account (takeaway: Upwork is a dumpster fire). That was 6 hours down the drain.

In comparison, this is a godsend, even if I might need to put in some work myself

wmk's Misadventures in Compiling C++, Episode 1

I used to get around the install TARGETS given no BUNDLE DESTINATION error by adding BUNDLE DESTINATION after RUNTIME DESTINATION but I had to do it for tools/CMakeLists.txt as well.

I cleared everything and followed your instructions from scratch, verbatim.

My first attempt after git cloning and modifying the brotli/CMakeLists.txt, I was getting the error:

CMake Error at tools/CMakeLists.txt:340 (install):
  install TARGETS given no BUNDLE DESTINATION for MACOSX_BUNDLE executable
  target "cjxl".

Maybe your invocation was supposed to disable the building of tools, so I double-checked that mine's a verbatim copy of yours (it was) and considered whether your libjxl might be older. Since it seems tools was still being built for some reason, I appended the following to your command:

-DJPEGXL_ENABLE_DEVTOOLS=false \
-DJPEGXL_ENABLE_VIEWERS=false \
-DJPEGXL_ENABLE_PLUGINS=false \
-DJPEGXL_TEST_TOOLS=false \
-DJPEGXL_STATIC=true

Same error. So I applied your fix by removing the offending install call from tools/CMakeLists.txt and reverted to your original command. The config succeeds but always has zsh: no such file or directory: -DCMAKE_INSTALL_PREFIX=/Users/username/Desktop/compiled at the end (Note: I clear build.ios between every attempt).

I then ran make:

[ 61%] Linking CXX executable tests/codec_test.app/tests/codec_test
ld: open() failed, errno=2 for 'tests/codec_test.app/tests/codec_test'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [lib/tests/codec_test.app/tests/codec_test] Error 1
make[1]: *** [lib/CMakeFiles/codec_test.dir/all] Error 2
make: *** [all] Error 2

Little did I know, I'll be yearning to see this error in an hour.

I saw "executable" and "test" and realized it was still compiling executables for tests, so I went back and added the extra flags above, then ran make again.

[ 20%] Linking CXX shared library libjxl_dec.dylib
ld: unknown options: --exclude-libs=ALL 
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [lib/libjxl_dec.0.10.2.dylib] Error 1
make[1]: *** [lib/CMakeFiles/jxl_dec.dir/all] Error 2
make: *** [all] Error 2

Why is it building a dylib? Is there an issue with the flags? I tried variations of the invocation

your flags + -DJPEGXL_STATIC=true:

Same error

I decided to give your vanilla command another try:

Same error?

Clearly some state is being saved somewhere, so I wiped the entire libjxl folder and copied in the untouched original, then reapplied CMakeLists.txt changes:

Still the same error!?

This was where grand attempt 1 was reset since I was stuck with the dylib error (I'm writing this log as I retrace my steps with grand attempt 2)

I tried various changes, including retracing my precise steps above, but I couldn't return to the previous error. I suspected CMake has some sort of cache/state set somewhere, so I looked online, found nothing, then did what any good techie did and reset my mac. Still there.

Grand attempt 3 was basically the same except I never saw ld: open() failed, errno=2 .

env: macOS 14.5, CMake 3.29.6

The adventure continues...

wmk's Misadventures in Compiling C++, Episode 2

After a break, it became apparent that tools had to be purged from the process entirely. This was accomplished by removing add_subdirectory(tools) from the main CMakeLists.txt.

[ 73%] Linking CXX executable tests/codec_test.app/tests/codec_test
ld: open() failed, errno=2 for 'tests/codec_test.app/tests/codec_test'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [lib/tests/codec_test.app/tests/codec_test] Error 1
make[1]: *** [lib/CMakeFiles/codec_test.dir/all] Error 2
make: *** [all] Error 2

I'm finally back to where I started--question now is why this is being built at all.

Since I made progress by removing a part of the main CMakeLists.txt, I decided to go all-in and started hacking away entire sections of the file like I was chopping a piece of wood. So this is why they call it "hacking". My mental image of a hacker became a bit more like a lumberjack.

Everything below if (JPEGXL_ENABLE_MANPAGES) was hacked off. This means no testing, but at this point I was willing to accept this tradeoff just to see some progress.

The result was counterintuitive: we're back to ld: unknown options: --exclude-libs=ALL at a dylib. It was here I realized just how finnicky CMake is and restarted my mac.

Starting with a fresh ios-cmake and libjxl, I went through all the steps again, removing the bottom chunk of CMakeLists.txt.

Success!

[ 95%] Built target jxl_enc-obj
[ 95%] Linking CXX static library libjxl_dec-internal.a
[ 95%] Built target jxl_dec-internal
[ 96%] Linking CXX static library libjxl-internal.a
[ 96%] Built target jxl-internal
[ 96%] Linking CXX static library libjxl.a
[ 96%] Built target jxl
[ 97%] Linking CXX static library libjxl_dec.a
[ 97%] Built target jxl_dec
[ 98%] Building CXX object lib/CMakeFiles/jxl_threads.dir/threads/resizable_parallel_runner.cc.o
[ 98%] Building CXX object lib/CMakeFiles/jxl_threads.dir/threads/thread_parallel_runner.cc.o
[100%] Building CXX object lib/CMakeFiles/jxl_threads.dir/threads/thread_parallel_runner_internal.cc.o
[100%] Linking CXX static library libjxl_threads.a
[100%] Built target jxl_threads

Now to see whether this is a stillbirth, an abomination, or a little angel that poops out JXL. My expectations are low; what I've done so far is trivial, especially for a professional, and we do have a professional working at TikTok's parent company saying, "This coder does not supports JPEG-XL encoding (Because I have no time :))",.

The adventure continues...

Maybe we have different cmakes?

% cmake --version
cmake version 3.24.1     
 
CMake suite maintained and supported by Kitware (kitware.com/cmake).
% which cmake
/opt/homebrew/bin/cmake

I'm not aware of cmake keeping any state outside of the directory tree where it is run.

That's probably it, given how touchy CMake seems to be

I'm currently stuck because I can't get jpegli to compile, and apparently its wired into the package. It just absolutely refuses to be compiled. I suspect it has something to do with

-- Could NOT find JPEG (missing: JPEG_LIBRARY JPEG_INCLUDE_DIR) 

But I don't know where to begin because, while installing libjpeg-turbo on this would probably lead to the warning going away, it would likely lead to cross-compiling issues (that thing Quinn was talking about).

Apparently I'm not the only one with issues compiling jpegli: https://github.com/libjxl/libjxl/issues/3440

I've also tried a slew of different hacks, including isolating it and forcing it to compile, but I'm stuck on this error:

CMake Error at jpegli.cmake:46 (target_link_libraries):
  Target "jpegli-static" links to:

    Threads::Threads

  but the target was not found.  Possible reasons include:

    * There is a typo in the target name.
    * A find_package call is missing for an IMPORTED target.
    * An ALIAS target is missing.

Anyways, I'll figure something out, like precisely excising it from libjxl

wmk's Misadventures in Compiling C++, Finale (?)

JXL is working on my iPhone after I got rid of libjpegli. The weeklong tumble down the rabbit hole of C++ compilation is finally over; and thus ends this impromptu blog I started writing for debugging purposes and to keep myself sane.

At the end of this I look back and realize how much I've been spoiled by Xcode, where compiling is just a mindless habit. Having to figure out CMake after being raised on Xcode is like getting chucked into the slums after having lived all your life in an ivory tower being pampered by servants who aren't perfect, but get the job done.

I've done some preliminary tests on encoding, decoding, and animation and everything seems to be working fine. The code in https://github.com/awxkee/jxl-coder-swift works even with TaskGroups, which allows me to turbocharge everything.

JXL so far seems to be absolutely worth the effort.

Now I just need to fiddle with the parameters to optimize performance for my app, and actually examine the animation output.

@endecotp I cannot thank you enough for introducing me to JXL and sticking with me until I got it working. I was about to spend perhaps hundreds of dollars, but in the end all I spent was $2.99 on an Upwork job post. If the opportunity ever arises, I'd definitely return the favor.

P.S. Upwork is a dumpster fire.

Great that you finally got it working!