LED-Large for Amazon All_Beauty Multi-Review Summarization

A fine-tuned allenai/led-large-16384 model for product-level multi-review summarization in the Amazon Reviews 2023 All_Beauty category.

The model takes multiple customer reviews for one product and generates a single abstractive summary that captures the main review consensus, recurring strengths, and meaningful complaints.

Input: product title + multiple review blocks
Output: one product-level summary

The product's numeric rating is not generated by the model. It is computed separately from the original review ratings.

Quick Start

Fastest option: open AllBeauty_LED_Large_Inference.ipynb if you want a ready-to-run example without reading the full training notebook.

1. Install dependencies

pip install -U transformers torch sentencepiece

2. Load the model

import torch
from transformers import AutoTokenizer, LEDForConditionalGeneration

MODEL_ID = "vltruong01/amazon-all-beauty-led-large-summarization"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

model = LEDForConditionalGeneration.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)
model.eval()

3. Prepare the input

The model was trained on inputs with this structure:

Product: Example Beauty Product

[Review 1]
Rating: 5/5
Title: Works very well
Text: I have used this product for several weeks and it works well...

[Review 2]
Rating: 4/5
Title: Good but expensive
Text: The quality is good, although the price is higher than expected...

[Review 3]
Rating: 2/5
Title: Not for me
Text: I experienced irritation and stopped using it...

Keep each review as a complete block whenever possible.

4. Generate a summary

LED uses sparse attention. The inference code below reproduces the global-attention pattern used in training: the first token, the product-title line, each review header, and each rating line receive global attention.

import re
import torch

MAX_SOURCE_LENGTH = 4096

def build_global_attention_mask(source, input_ids, attention_mask, tokenizer):
    global_attention = torch.zeros_like(input_ids)

    # Always give global attention to the first non-padding token.
    global_attention[:, 0] = 1

    lines = source.splitlines()
    important_prefixes = ("Product:", "[Review ", "Rating:")

    important_texts = [
        line.strip()
        for line in lines
        if line.strip().startswith(important_prefixes)
    ]

    for text in important_texts:
        token_ids = tokenizer(
            text,
            add_special_tokens=False,
        )["input_ids"]

        if not token_ids:
            continue

        first_token = token_ids[0]

        matches = (
            (input_ids[0] == first_token)
            & (attention_mask[0] == 1)
        ).nonzero(as_tuple=False)

        for match in matches:
            global_attention[0, match.item()] = 1

    return global_attention


source = '''Product: Example Beauty Product

[Review 1]
Rating: 5/5
Title: Works very well
Text: I have used this product for several weeks and it works well.

[Review 2]
Rating: 4/5
Title: Good but expensive
Text: The quality is good, although the price is higher than expected.

[Review 3]
Rating: 2/5
Title: Not for me
Text: I experienced irritation and stopped using it.
'''

inputs = tokenizer(
    source,
    return_tensors="pt",
    max_length=MAX_SOURCE_LENGTH,
    truncation=True,
    padding=False,
)

input_ids = inputs["input_ids"].to(device)
attention_mask = inputs["attention_mask"].to(device)

global_attention_mask = build_global_attention_mask(
    source,
    input_ids,
    attention_mask,
    tokenizer,
).to(device)

with torch.inference_mode():
    generated = model.generate(
        input_ids=input_ids,
        attention_mask=attention_mask,
        global_attention_mask=global_attention_mask,
        num_beams=6,
        max_new_tokens=192,
        no_repeat_ngram_size=3,
        repetition_penalty=1.05,
        early_stopping=True,
    )

summary = tokenizer.decode(
    generated[0],
    skip_special_tokens=True,
)

print(summary)

Inference Notebook

If you only want to use the trained model, open:

AllBeauty_LED_Large_Inference.ipynb

The notebook is intentionally much shorter than the training notebook and contains only the steps required for inference:

install dependencies
        ↓
load tokenizer + fine-tuned LED-Large
        ↓
paste one product's review collection
        ↓
build structured global attention
        ↓
generate one product-level summary
        ↓
optionally compute the numeric rating separately

The inference notebook reproduces the same structural global-attention pattern used during training.

Global attention is assigned to:

• the first source token,
• the Product: line,
• every [Review N] header,
• every Rating: line.

This matters because LED uses sparse attention for long inputs. These structural anchors help the model connect product identity, review boundaries, and ratings across a long multi-review document.

The notebook also keeps the numeric product rating outside the model. LED generates only the summary.

What the Model Does

The model performs multi-review aggregation, not single-review summarization.

It is intended to learn patterns such as:

• what customers most consistently agree on,
• recurring positive experiences,
• recurring negative experiences,
• meaningful minority complaints,
• the overall product-level impression.

The model should not simply concatenate or extract individual review sentences.

Dataset

Training data comes from:

vltruong01/amazon-all-beauty-led-summarization

The dataset was derived from Amazon Reviews 2023 All_Beauty product reviews.

Each example contains fields such as:

parent_asin
product_title
rating
num_reviews
source
target_summary
target_word_count
target_qc_ok
source_title_mismatch
mismatch

Before training, rows failing quality control or marked as source/title mismatches were excluded.

The final training experiment used:

The split was deterministic with seed 42.

Synthetic Reference Summaries

The target summaries used for training are synthetic reference summaries generated by a teacher model, not human-written gold summaries.

They were produced under constraints intended to:

• preserve product identity,
• summarize the review consensus,
• retain recurring strengths,
• retain meaningful minority complaints,
• avoid unsupported claims,
• avoid numeric rating generation,
• produce one natural paragraph.

This distinction is important when interpreting the evaluation results below: automatic metrics compare LED outputs against these synthetic teacher-generated references.

Training Configuration

This was a full fine-tuning of LED-Large, not LoRA.

Global attention

Sparse global attention was assigned to:

• the first source token,
• the product-title line,
• the first token of every [Review N] block,
• every Rating: line.

This gives LED explicit anchors for the product identity and review boundaries across long inputs.

Held-Out Test Results

Evaluation was performed on the 196-example held-out test split.

Validation loss for the final run:

2.6452

Automatic metrics are useful for reproducibility, but they do not fully measure factual consistency, consensus weighting, or product-identity preservation.

Qualitative Behavior

A manual audit of the 196 held-out generations found several strengths:

• summaries generally reflect the overall review consensus,
• product identity is usually preserved,
• outputs are concise and readable,
• recurring positive and negative opinions are often combined effectively.

Observed limitations include:

• repetitive summary openings,
• occasional local word or phrase repetition,
• product or specification drift,
• occasional polarity or factual drift,
• isolated opinions sometimes being promoted to consensus,
• minority complaints sometimes being over- or under-weighted.

These limitations are important for downstream use.

Example Application

A simple application can combine the deterministic product rating with the generated review summary:

Rating: 4.3/5
Summary: Customers generally appreciate ...

The rating should be computed from the original review ratings in application code.

Do not ask LED to generate the numeric rating.

Reproduce the Training

The full experiment notebook is included in this repository:

AllBeauty_LED_Large_Training.ipynb

The training workflow is:

prepared product-level review dataset
        ↓
quality filtering
        ↓
deterministic train / validation / test split
        ↓
LED-Large tokenization
        ↓
structured global attention
        ↓
full fine-tuning
        ↓
best-model selection
        ↓
held-out generation
        ↓
ROUGE + BERTScore + qualitative diagnostics

Experiment artifacts are stored under:

artifacts/

Typical files include:

experiment_config.json
final_metrics.json
split_manifest.json
test_predictions.csv

Repository Structure

amazon-all-beauty-led-large-summarization/
├── README.md
├── AllBeauty_LED_Large_Training.ipynb
├── AllBeauty_LED_Large_Inference.ipynb
├── config.json
├── generation_config.json
├── model.safetensors
├── tokenizer.json
├── tokenizer_config.json
├── special_tokens_map.json
├── merges.txt
├── vocab.json
└── artifacts/
    ├── experiment_config.json
    ├── final_metrics.json
    ├── split_manifest.json
    └── test_predictions.csv

The model files are stored at the repository root so the model can be loaded directly with:

LEDForConditionalGeneration.from_pretrained(
    "vltruong01/amazon-all-beauty-led-large-summarization"
)

Which File Should I Use?

Expected Input

Use the same structured format as training:

Product: <product title>

[Review 1]
Rating: <rating>/5
Title: <review title>
Text: <review text>

[Review 2]
Rating: <rating>/5
Title: <review title>
Text: <review text>

For best results:

• keep the product title at the beginning,
• preserve complete review blocks,
• include review rating lines,
• keep the full input within 4096 tokens,
• avoid mixing reviews from different products.

Expected Output

The intended output is:

• one paragraph,
• product-level rather than review-by-review,
• focused on recurring customer opinions,
• balanced across positives and meaningful complaints,
• no bullet list,
• no numeric product rating.

Intended Use

Suitable uses include:

• research on long-context review summarization,
• product-level multi-review summarization,
• comparison of LED against text-only or multimodal variants,
• studying consensus aggregation from many customer reviews,
• educational and academic experiments.

Limitations

This model has several important limitations:

• It is trained only on the Amazon All_Beauty domain.
• Training targets are synthetic teacher-generated references rather than human-written gold summaries.
• The model can hallucinate or distort product details.
• It can occasionally reverse or flatten sentiment.
• It may over-emphasize isolated opinions or miss minority complaints.
• Summary phrasing can become repetitive.
• Inputs longer than the configured context budget require review selection or truncation before inference.
• Automatic metrics do not guarantee factual faithfulness.

For high-stakes or user-facing deployment, summaries should be treated as generated text and checked against the source reviews.

Related Dataset

Training dataset:

vltruong01/amazon-all-beauty-led-summarization

Source-only review dataset:

vltruong01/amazon-all-beauty-led-reviews

Citation and Attribution

Base model:

• allenai/led-large-16384

Dataset source:

• Amazon Reviews 2023, All_Beauty category

This repository is provided for research and educational use.