Model Card for Model ID
This model is based on LLaVA1.5-7b. The model is finetuned with LoRA on OpenCOLE1.0 dataset to generate text layouts.
Model Details
Model Description
This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-
Language(s) (NLP): English
-
License: Llama 2 is licensed under the LLAMA 2 Community License, Copyright (c) Meta Platforms, Inc. All Rights Reserved.
-
Finetuned from model: LLaVA1.5-7b
Model Sources [optional]
- Repository: CyberAgentAILab/OpenCOLE
- Paper: OpenCOLE: Towards Reproducible Automatic Graphic Design Generation
Uses
Please refer to OpenCOLE.
Training Details
Training Data
- About 18k image-text extracted automatically from OpenCOLE
Below is an example.
[
{
"id": "592d203395a7a863ddcd9df1",
"image": "images/592/592d203395a7a863ddcd9df1.png",
"conversations": [
{
"from": "human",
"value": "<image>\nGiven an image and text input including set of keywords to be placed on the image and its properties (optional), plan the layout of the texts. The output should be formatted as a JSON instance that conforms to the JSON schema below.\n\nAs an example, for the schema {\"properties\": {\"foo\": {\"title\": \"Foo\", \"description\": \"a list of strings\", \"type\": \"array\", \"items\": {\"type\": \"string\"}}}, \"required\": [\"foo\"]}\nthe object {\"foo\": [\"bar\", \"baz\"]} is a well-formatted instance of the schema. The object {\"properties\": {\"foo\": [\"bar\", \"baz\"]}} is not well-formatted.\n\nHere is the output schema:\n```\n{\"properties\": {\"elements\": {\"title\": \"Elements\", \"default\": [], \"type\": \"array\", \"items\": {\"$ref\": \"#/definitions/Element\"}}}, \"definitions\": {\"Element\": {\"title\": \"Element\", \"type\": \"object\", \"properties\": {\"text\": {\"title\": \"Text\", \"description\": \"Dummy text\", \"type\": \"string\"}, \"width\": {\"title\": \"Width\", \"description\": \"range: 0 <= width <= 127\", \"type\": \"integer\"}, \"height\": {\"title\": \"Height\", \"description\": \"range: 0 <= height <= 127\", \"type\": \"integer\"}, \"left\": {\"title\": \"Left\", \"description\": \"range: 0 <= left <= 127\", \"type\": \"integer\"}, \"top\": {\"title\": \"Top\", \"description\": \"range: 0 <= top <= 127\", \"type\": \"integer\"}, \"font\": {\"title\": \"Font\", \"type\": \"string\"}, \"color\": {\"title\": \"Color\", \"description\": \"range: 0 <= color <= 127\", \"type\": \"integer\"}, \"text_align\": {\"title\": \"Text Align\", \"description\": \"choices: \\\"\\\", \\\"left\\\", \\\"center\\\", \\\"right\\\"\", \"type\": \"string\"}, \"capitalize\": {\"title\": \"Capitalize\", \"description\": \"choices: \\\"false\\\", \\\"true\\\"\", \"type\": \"string\"}, \"font_size\": {\"title\": \"Font Size\", \"description\": \"range: 0 <= font_size <= 127\", \"type\": \"integer\"}, \"angle\": {\"title\": \"Angle\", \"description\": \"range: 0 <= angle <= 127\", \"type\": \"integer\"}, \"letter_spacing\": {\"title\": \"Letter Spacing\", \"description\": \"range: 0 <= letter_spacing <= 127\", \"type\": \"integer\"}, \"line_height\": {\"title\": \"Line Height\", \"description\": \"range: 0 <= line_height <= 127\", \"type\": \"integer\"}}, \"required\": [\"text\", \"width\", \"height\", \"left\", \"top\", \"font\", \"color\", \"text_align\", \"capitalize\", \"font_size\", \"angle\", \"letter_spacing\", \"line_height\"]}}}\n``` Input: [\"WE DON'T HAVE\\nANOTHER PLANET\", \"GREEN\", \"GO\"]"
},
{
"from": "gpt",
"value": "{\"elements\": [{\"text\": \"GO\", \"width\": 62, \"height\": 40, \"left\": 11, \"top\": 43, \"font\": \"Cormorant Infant\", \"color\": 38, \"text_align\": \"center\", \"capitalize\": \"false\", \"font_size\": 79, \"angle\": 0, \"letter_spacing\": 61, \"line_height\": 27}, {\"text\": \"GREEN\", \"width\": 69, \"height\": 30, \"left\": 6, \"top\": 60, \"font\": \"Cormorant Infant\", \"color\": 56, \"text_align\": \"center\", \"capitalize\": \"false\", \"font_size\": 67, \"angle\": 0, \"letter_spacing\": 50, \"line_height\": 27}, {\"text\": \"WE DON'T HAVE\\nANOTHER PLANET\", \"width\": 71, \"height\": 37, \"left\": 3, \"top\": 74, \"font\": \"Cormorant Infant\", \"color\": 56, \"text_align\": \"center\", \"capitalize\": \"false\", \"font_size\": 39, \"angle\": 0, \"letter_spacing\": 29, \"line_height\": 47}]}"
}
]
},
...
Citation
@inproceedings{inoue2024opencole,
title={{OpenCOLE: Towards Reproducible Automatic Graphic Design Generation}},
author={Naoto Inoue and Kento Masui and Wataru Shimoda and Kota Yamaguchi},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)},
year={2024},
}