DialoGPT-medium-distill-Kimi-K2-Instruct
This model is a fine-tuned version of microsoft/DialoGPT-medium, specialized for a custom persona and critical knowledge injection. It has been trained to balance conversational flexibility with specific factual recall.
Model Description
- Model type: Causal Language Model
- Language(s): English
- Base Model: DialoGPT-medium
- Total Parameters: 406.3M (Post-fine-tune expanded state)
Intended Uses & Limitations
This model is designed for creative assistant tasks and casual conversation.
- Direct Use: Chatbots, creative storytelling, and persona-driven interactions.
- Limitations: Due to the small dataset size and "creative" training, the model may occasionally hallucinate or provide non-literal answers (e.g., creative definitions of common objects).
Training Procedure
The model underwent a full fine-tune on a custom dataset consisting of critical facts and casual chat examples.
Training Hyperparameters
- Learning Rate: 2e-5
- Epochs: 5
- Batch Size: 4 (with gradient accumulation)
- Precision: Mixed Precision (FP16)
- Loss achieved: 3.264037
Weight Analysis
Post-training analysis showed a significant shift in the LM Head weights (Absolute Shift: 4.4164), indicating a strong adaptation to the new conversational style while maintaining structural grammar stability in the transformer layers.
How to Use
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("Fu01978/DialoGPT-medium-distill-Kimi-K2-Instruct")
model = AutoModelForCausalLM.from_pretrained("Fu01978/DialoGPT-medium-distill-Kimi-K2-Instruct")
# For best results, use a temperature between 0.7 and 0.85