DistillSupra-0.2M
DistillSupra-0.2M is an ultra-compact causal language model with approximately 0.2 million parameters, produced by knowledge distillation from Supra-Mini-v4-2M.
It was trained 500 steps(1 Epoch) for 30 minutes on a GTX 750 Ti 4GB using generated text from the teacher.
The model was 10x compressed! That's crazy!
Architecture
| Parameter | Teacher | Student |
|---|---|---|
| hidden_size | 64 | 48 |
| intermediate_size | 128 | 96 |
| num_hidden_layers | 5 | 4 |
| num_attention_heads | 8 | 6 |
| vocab_size | 4096 | 4096 |
| Parameters | ~468k | ~289k |
Some outputs:
Prompt : Throughout history, great civilizations
Output: Throughout history, great civilizations to in, a be polrain for is with more the the be the for. of be of on (I.er The b M.A-R and or have that not is and the is this they, can for to to. is of a a, to ofs the for and the a. in the is to as of is that an that of and you the which is, the, for in be a are by’ of. and to a m
Prompt : The human brain is capable of
Output: The human brain is capable ofs in an more that in a new can is the this the a of the pS, the a to the other in not it... and with a to that be are of to for in of of ass. The be of the,.F-s be the of dLal. ins of be and of Sin: and or that a one that to and a a bFed, asRal., the, is a and as
Prompt : The most important principle in science is
The most important principle in science is a is a this are not for that the to of be digels-LC. to the in a the to, on to,
Why did supra created this trash?
We are currently researching knowledge distillation and this was the first step! Things will better up!
Benchmarks
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| arc_easy | 1 | none | 0 | acc | ↑ | 0.2517 | ± | 0.0089 |
| none | 0 | acc_norm | ↑ | 0.2677 | ± | 0.0091 | ||
| blimp | 2 | none | acc | ↑ | 0.5184 | ± | 0.0017 | |
| - blimp_adjunct_island | 1 | none | 0 | acc | ↑ | 0.7610 | ± | 0.0135 |
| - blimp_anaphor_gender_agreement | 1 | none | 0 | acc | ↑ | 0.1920 | ± | 0.0125 |
| - blimp_anaphor_number_agreement | 1 | none | 0 | acc | ↑ | 0.3650 | ± | 0.0152 |
| - blimp_animate_subject_passive | 1 | none | 0 | acc | ↑ | 0.5790 | ± | 0.0156 |
| - blimp_animate_subject_trans | 1 | none | 0 | acc | ↑ | 0.6450 | ± | 0.0151 |
| - blimp_causative | 1 | none | 0 | acc | ↑ | 0.5210 | ± | 0.0158 |
| - blimp_complex_NP_island | 1 | none | 0 | acc | ↑ | 0.5150 | ± | 0.0158 |
| - blimp_coordinate_structure_constraint_complex_left_branch | 1 | none | 0 | acc | ↑ | 0.2970 | ± | 0.0145 |
| - blimp_coordinate_structure_constraint_object_extraction | 1 | none | 0 | acc | ↑ | 0.3530 | ± | 0.0151 |
| - blimp_determiner_noun_agreement_1 | 1 | none | 0 | acc | ↑ | 0.5080 | ± | 0.0158 |
| - blimp_determiner_noun_agreement_2 | 1 | none | 0 | acc | ↑ | 0.4900 | ± | 0.0158 |
| - blimp_determiner_noun_agreement_irregular_1 | 1 | none | 0 | acc | ↑ | 0.4510 | ± | 0.0157 |
| - blimp_determiner_noun_agreement_irregular_2 | 1 | none | 0 | acc | ↑ | 0.5310 | ± | 0.0158 |
| - blimp_determiner_noun_agreement_with_adj_2 | 1 | none | 0 | acc | ↑ | 0.4730 | ± | 0.0158 |
| - blimp_determiner_noun_agreement_with_adj_irregular_1 | 1 | none | 0 | acc | ↑ | 0.4710 | ± | 0.0158 |
| - blimp_determiner_noun_agreement_with_adj_irregular_2 | 1 | none | 0 | acc | ↑ | 0.5370 | ± | 0.0158 |
| - blimp_determiner_noun_agreement_with_adjective_1 | 1 | none | 0 | acc | ↑ | 0.5220 | ± | 0.0158 |
| - blimp_distractor_agreement_relational_noun | 1 | none | 0 | acc | ↑ | 0.4940 | ± | 0.0158 |
| - blimp_distractor_agreement_relative_clause | 1 | none | 0 | acc | ↑ | 0.4800 | ± | 0.0158 |
| - blimp_drop_argument | 1 | none | 0 | acc | ↑ | 0.6900 | ± | 0.0146 |
| - blimp_ellipsis_n_bar_1 | 1 | none | 0 | acc | ↑ | 0.2580 | ± | 0.0138 |
| - blimp_ellipsis_n_bar_2 | 1 | none | 0 | acc | ↑ | 0.2810 | ± | 0.0142 |
| - blimp_existential_there_object_raising | 1 | none | 0 | acc | ↑ | 0.6180 | ± | 0.0154 |
| - blimp_existential_there_quantifiers_1 | 1 | none | 0 | acc | ↑ | 0.5550 | ± | 0.0157 |
| - blimp_existential_there_quantifiers_2 | 1 | none | 0 | acc | ↑ | 0.0750 | ± | 0.0083 |
| - blimp_existential_there_subject_raising | 1 | none | 0 | acc | ↑ | 0.6750 | ± | 0.0148 |
| - blimp_expletive_it_object_raising | 1 | none | 0 | acc | ↑ | 0.5820 | ± | 0.0156 |
| - blimp_inchoative | 1 | none | 0 | acc | ↑ | 0.4010 | ± | 0.0155 |
| - blimp_intransitive | 1 | none | 0 | acc | ↑ | 0.5550 | ± | 0.0157 |
| - blimp_irregular_past_participle_adjectives | 1 | none | 0 | acc | ↑ | 0.2340 | ± | 0.0134 |
| - blimp_irregular_past_participle_verbs | 1 | none | 0 | acc | ↑ | 0.7760 | ± | 0.0132 |
| - blimp_irregular_plural_subject_verb_agreement_1 | 1 | none | 0 | acc | ↑ | 0.4810 | ± | 0.0158 |
| - blimp_irregular_plural_subject_verb_agreement_2 | 1 | none | 0 | acc | ↑ | 0.4370 | ± | 0.0157 |
| - blimp_left_branch_island_echo_question | 1 | none | 0 | acc | ↑ | 0.9310 | ± | 0.0080 |
| - blimp_left_branch_island_simple_question | 1 | none | 0 | acc | ↑ | 0.3260 | ± | 0.0148 |
| - blimp_matrix_question_npi_licensor_present | 1 | none | 0 | acc | ↑ | 0.0200 | ± | 0.0044 |
| - blimp_npi_present_1 | 1 | none | 0 | acc | ↑ | 0.6120 | ± | 0.0154 |
| - blimp_npi_present_2 | 1 | none | 0 | acc | ↑ | 0.5830 | ± | 0.0156 |
| - blimp_only_npi_licensor_present | 1 | none | 0 | acc | ↑ | 0.0000 | ± | 0 |
| - blimp_only_npi_scope | 1 | none | 0 | acc | ↑ | 0.0270 | ± | 0.0051 |
| - blimp_passive_1 | 1 | none | 0 | acc | ↑ | 0.5190 | ± | 0.0158 |
| - blimp_passive_2 | 1 | none | 0 | acc | ↑ | 0.5210 | ± | 0.0158 |
| - blimp_principle_A_c_command | 1 | none | 0 | acc | ↑ | 0.6470 | ± | 0.0151 |
| - blimp_principle_A_case_1 | 1 | none | 0 | acc | ↑ | 1.0000 | ± | 0 |
| - blimp_principle_A_case_2 | 1 | none | 0 | acc | ↑ | 0.5290 | ± | 0.0158 |
| - blimp_principle_A_domain_1 | 1 | none | 0 | acc | ↑ | 1.0000 | ± | 0 |
| - blimp_principle_A_domain_2 | 1 | none | 0 | acc | ↑ | 0.5880 | ± | 0.0156 |
| - blimp_principle_A_domain_3 | 1 | none | 0 | acc | ↑ | 0.4790 | ± | 0.0158 |
| - blimp_principle_A_reconstruction | 1 | none | 0 | acc | ↑ | 0.3470 | ± | 0.0151 |
| - blimp_regular_plural_subject_verb_agreement_1 | 1 | none | 0 | acc | ↑ | 0.5750 | ± | 0.0156 |
| - blimp_regular_plural_subject_verb_agreement_2 | 1 | none | 0 | acc | ↑ | 0.4710 | ± | 0.0158 |
| - blimp_sentential_negation_npi_licensor_present | 1 | none | 0 | acc | ↑ | 1.0000 | ± | 0 |
| - blimp_sentential_negation_npi_scope | 1 | none | 0 | acc | ↑ | 0.4020 | ± | 0.0155 |
| - blimp_sentential_subject_island | 1 | none | 0 | acc | ↑ | 0.3030 | ± | 0.0145 |
| - blimp_superlative_quantifiers_1 | 1 | none | 0 | acc | ↑ | 0.5090 | ± | 0.0158 |
| - blimp_superlative_quantifiers_2 | 1 | none | 0 | acc | ↑ | 0.6150 | ± | 0.0154 |
| - blimp_tough_vs_raising_1 | 1 | none | 0 | acc | ↑ | 0.3080 | ± | 0.0146 |
| - blimp_tough_vs_raising_2 | 1 | none | 0 | acc | ↑ | 0.6980 | ± | 0.0145 |
| - blimp_transitive | 1 | none | 0 | acc | ↑ | 0.5210 | ± | 0.0158 |
| - blimp_wh_island | 1 | none | 0 | acc | ↑ | 0.4050 | ± | 0.0155 |
| - blimp_wh_questions_object_gap | 1 | none | 0 | acc | ↑ | 0.9970 | ± | 0.0017 |
| - blimp_wh_questions_subject_gap | 1 | none | 0 | acc | ↑ | 0.9990 | ± | 0.0010 |
| - blimp_wh_questions_subject_gap_long_distance | 1 | none | 0 | acc | ↑ | 0.9980 | ± | 0.0014 |
| - blimp_wh_vs_that_no_gap | 1 | none | 0 | acc | ↑ | 1.0000 | ± | 0 |
| - blimp_wh_vs_that_no_gap_long_distance | 1 | none | 0 | acc | ↑ | 1.0000 | ± | 0 |
| - blimp_wh_vs_that_with_gap | 1 | none | 0 | acc | ↑ | 0.0000 | ± | 0 |
| - blimp_wh_vs_that_with_gap_long_distance | 1 | none | 0 | acc | ↑ | 0.0000 | ± | 0 |
| wikitext | 2 | none | 0 | bits_per_byte | ↓ | 3.0125 | ± | N/A |
| none | 0 | byte_perplexity | ↓ | 8.0696 | ± | N/A | ||
| none | 0 | word_perplexity | ↓ | 70687.5638 | ± | N/A |
| Groups | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| blimp | 2 | none | acc | ↑ | 0.5184 | ± | 0.0017 |
Final Thought
Knowledge distillation is a promising thing for us, we believe that LLMs can be helpful even being so small!