Llama 3.1 8B Instruct, 4-bit MLX, for Lantern
Built with Llama
A copy of mlx-community/Meta-Llama-3.1-8B-Instruct-4bit, kept here for Lantern, an offline, private AI app for iPhone, iPad and Mac. The model runs entirely on the device: no account, no server, and nothing you type leaves it. The only network use is downloading these files once.
The weights are unchanged from the mlx-community conversion. The original model is meta-llama/Llama-3.1-8B-Instruct.
| Parameters | 8 billion |
| Precision | 4-bit, MLX format |
| Download | about 4.5 GB |
| Takes in | text |
| Runs on | 8 GB iPhones only with a warning, and it may be closed by iOS in long chats. Comfortable on Macs with 16 GB or more. |
Good for: The most capable model Lantern allows. Lantern stops at 8 billion parameters, above which a phone closes the app.
Use it in Lantern
Lantern already lists this model. To download this copy instead, open Settings, turn on Show advanced options, choose Add a model from Hugging Face and enter RabiatS/Llama-3.1-8B-Instruct-4bit-Lantern. Lantern checks it against your device's memory before anything downloads.
Use it with mlx-lm on a Mac
pip install mlx-lm
mlx_lm.generate --model RabiatS/Llama-3.1-8B-Instruct-4bit-Lantern --prompt "Name two things to pack for a rainy hike."
Licence
Llama 3.1 is licensed under the Llama 3.1 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved. The licence and Meta's Acceptable Use Policy are in LICENSE and USE_POLICY.md in this repository. Use of this model must follow both.
Credit for the model goes to its authors, and for the MLX conversion to mlx-community.