Fixes https://github.com/harfbuzz/harfbuzz/issues/2780
Also make sure we only download it once per CI run, not three times...