Llama-3.3-70B-Instruct-FP8-Dynamic

This is a quantization of the Llama-3.3-70B-Instruct.

The Meta Llama 3.3 is a state-of-the-art multilingual large language model (LLM) with 70 billion parameters, pretrained and instruction-tuned for exceptional performance in generative text-based tasks. Optimized for multilingual dialogue, it supports English and seven additional languages: French, German, Hindi, Italian, Portuguese, Spanish, and Thai, enabling seamless communication across diverse audiences. The model consistently outperforms both open-source and proprietary chat models on key industry benchmarks, delivering superior quality, safety, and helpfulness. Its advanced features and multilingual support position Llama 3.3 as a powerful tool for building innovative AI applications.

Evaluations

This model provides an accuracy recovery of 99.67%.

We did not check for data contamination. Evaluation was done using Eval. Harness with limit=1000.

Usage

Install vLLM and run the server:

python -m vllm.entrypoints.openai.api_server --model cortecs/Llama-3.3-70B-Instruct-FP8-Dynamic --max-model-len 9000 --gpu-memory-utilization 0.95

Access the model:

curl http://localhost:8000/v1/completions     -H "Content-Type: application/json"     -d ' {
        "model": "cortecs/Llama-3.3-70B-Instruct-FP8-Dynamic",
        "prompt": "San Francisco is a"
    } '

⚡ This model is optimized to handle heavy workloads providing a total throughput of ️1485 tokens per second using one NVIDIA H100 ⚡