We are training models efficiently not racing to be the biggest, but big things come in optimizing small packages.
Cutting these layers out do not change the output for my model at all individually, but cutting all of these redundant layers at once breaks the model completely. So strategically balance extractions and see if we can recover after removing this redundancy.
Please try my Mermaid-Llama-3-8B and This Mermaid-Llama-3-Pruned-6B, you might be surprised.
24/32 Layers Model.
license: cc-by-4.0
Mermaid-Llama-3-6B
Introducing Mermaid-LLama-3-6B, a robust language model designed for Python code understanding and crafting captivating story flow maps. Pruned down to 6 billion parameter to show we dont need the bloat.
See MergeKit Notes And Try Triming my model yourself and explore my world of trimming models to fit SMARTER Models with lower requirements f or specific tasks. Mermaid is just a start, Hire me to solve your problem and I will build the smallest footprint model that solves just that problem.
I wish to specialize in packing models on Edge Devices.
Open For Hire See my links to my Linkedin for more.
Key Features
-
Code Understanding:
- Masters Python intricacies with finesse.
- Generates clear and accurate Mermaid Diagram Flow Charts.
- Ideal for developers seeking visual representations of their code logic.
-
Storytelling Capabilities:
- Converts narrative inputs into captivating Mermaid Diagrams.
- Maps character interactions, plot developments, and narrative arcs.
-
Unmatched Performance:
- Surpasses GPT-4 in generating well-organized Mermaid Diagrams.
-
Training Insights:
- Trained on a diverse dataset, including 800 unique, hand-curated Mermaid Graph examples utilizing 478 complete Python programs.
- Exhibits emergent properties in story-to-flow map translations and step-by-step instruction flow maps.
Collaboration
Interested in enhancing Mermaid's capabilities? Contact troydoesai@gmail.com for collaboration opportunities.
Example Use Cases
- Retrieval-Augmented Generation (RAG): Utilize Mermaid-LLama-3-8B to create condensed knowledge graphs. This model excels in generating flow diagrams that enhance the retrieval process. These knowledge graphs are stored in a vector database, which allows for quick and efficient retrieval of contextually relevant information. When a query is received, the system retrieves a pertinent knowledge graph, appending it as context to the model. This enriched context enables Mermaid-LLama-3-8B to deliver more accurate and nuanced responses. This approach is particularly beneficial in applications requiring deep, context-aware interactions, such as sophisticated Q&A systems, dynamic data analysis, and complex decision-making tasks.
- Code Documentation: Automatic visual flow charts from Python code.
- Storyboarding: Visually appealing diagrams for storytelling.
- Project Planning: Visual project flow maps for effective team communication.
- Learning Python: Helps students visually understand Python code structures.
- Game Design: Visualizing game storylines for coherent narrative structure.
Proof of Concept
Stay tuned for the release of the VSCode Extension that displays the Live Flow Map every time a user stops typing for more than 10 seconds.
Training Specifications
- LoRA Rank: 2048
- LoRA Alpha: 4096
- Batch Size: 1
- Micro Batch Size: 1
- Cutoff Length: 4096
- Save every n steps: 1000
- Epochs: 3
- Learning Rate: 1e-6
- LR Scheduler: Cosine
Target Modules:
- Enable q_proj
- Enable v_proj
- Enable k_proj
- Enable o_proj
- Enable gate_proj
- Enable down_proj
- Enable up_proj
Getting Started
Start by downloading one of my models.
Load the model.
Use my prompt template to generate a Mermaid code block, which can be viewed in the Mermaid Live Editor or using the Mermaid CLI tool.
Here we open the VLLM GUI Program while still running in Vram the Mermaid-Llama-8B to compare the flow diagram to the actual program and show the lightweight capabilites of small models on consumer hardware.
More on my VLLM Class and inference GUI : https://github.com/Troys-Code/VLLM
Note: This model should be treated as an Auto-Complete Model, Do not try talking to it in chat you are gonna get garbage, those layers have been pruned and replaced, that is all you will hear of my secret sauce on training on small < 1000 entry datasets.