Hungary 🇭🇺🇪🇺

Developer behind the Eternity for Lemmy android app.

@bazsalanszky@lemmy.ml is my old account, migrated to my own instance in 2023.

  • 19 Posts
  • 164 Comments
Joined 1 year ago
cake
Cake day: July 2nd, 2023

help-circle







  • From what I’ve seen, it’s definitely worth quantizing. I’ve used llama 3 8B (fp16) and llama 3 70B (q2_XS). The 70B version was way better, even with this quantization and it fits perfectly in 24 GB of VRAM. There’s also this comparison showing the quantization option and their benchmark scores:

    1000029570

    Source

    To run this particular model though, you would need about 45GB of RAM just for the q2_K quant according to Ollama. I think I could run this with my GPU and offload the rest of the layers to the CPU, but the performance wouldn’t be that great(e.g. less than 1 t/s).













  • Just wanted to add here that you can test the release early with the nightly builds or with the Google Play Beta program (it’s still under review, but hopefully it will be out there soon).

    If you find any issues with it, please let me know so I can fix it before this release.

    UPDATE: It should be available on the Google Play Beta now. Unfortunately, I’ve found some issues with posting comments/posts so I will need to fix that before the release.