Model Card for Vera
Vera is a commonsense statement verification model. See our paper at: https://arxiv.org/abs/2305.03695.
Model Details
Model Description
Given a commonsense statement as input, Vera predicts the plausibility of this statement. Vera outputs a real-valued score in the range [0, 1]. A score of 1 means the statement is correct according to commonsense, and a score of 0 means the statement is incorrect. This score is calibrated, so a score between 0 and 1 can be interpreted as Vera's confidence that the statement is correct.
- Developed by: Jiacheng Liu, Wenya Wang, Dianzhuo Wang, Noah A. Smith, Yejin Choi, Hannaneh Hajishirzi
- Shared by: Jiacheng (Gary) Liu
- Model type: Transformers
- Language(s) (NLP): English
- License: MIT
- Finetuned from model: T5-v1.1-XXL
Model Sources
- Repository: https://github.com/liujch1998/vera
- Paper: https://arxiv.org/abs/2305.03695
- Demo: https://huggingface.co/spaces/liujch1998/vera
Uses
Direct Use
Vera is intended to predict the correctness of commonsense statements.
Downstream Use
Vera can be used to detect commonsense errors made by generative LMs (e.g., ChatGPT), or filter noisy commonsense knowledge generated by other LMs (e.g., Rainier).
Out-of-Scope Use
Vera is a research prototype and may make mistakes. Do not use for making critical decisions. It is intended to predict the correctness of commonsense statements, and may be unreliable when taking input out of this scope.
Bias, Risks, and Limitations
See the Limitations and Ethics Statement section of our paper.
How to Get Started with the Model
Use the code below to get started with the model.
tokenizer = transformers.AutoTokenizer.from_pretrained('liujch1998/vera')
model = transformers.T5EncoderModel.from_pretrained('liujch1998/vera')
model.D = model.shared.embedding_dim
linear = torch.nn.Linear(model.D, 1, dtype=model.dtype)
linear.weight = torch.nn.Parameter(model.shared.weight[32099, :].unsqueeze(0))
linear.bias = torch.nn.Parameter(model.shared.weight[32098, 0].unsqueeze(0))
model.eval()
t = model.shared.weight[32097, 0].item() # temperature for calibration
statement = 'Please enter a commonsense statement here.'
input_ids = tokenizer.batch_encode_plus([statement], return_tensors='pt', padding='longest', truncation='longest_first', max_length=128).input_ids
with torch.no_grad():
output = model(input_ids)
last_hidden_state = output.last_hidden_state
hidden = last_hidden_state[0, -1, :]
logit = linear(hidden).squeeze(-1)
logit_calibrated = logit / t
score_calibrated = logit_calibrated.sigmoid()
# score_calibrated is Vera's final output plausibility score
You may also refer to https://huggingface.co/spaces/liujch1998/vera/blob/main/app.py#L27-L98 for implementation.
Citation
BibTeX:
@article{Liu2023VeraAG,
title={Vera: A General-Purpose Plausibility Estimation Model for Commonsense Statements},
author={Jiacheng Liu and Wenya Wang and Dianzhuo Wang and Noah A. Smith and Yejin Choi and Hanna Hajishirzi},
journal={ArXiv},
year={2023},
volume={abs/2305.03695}
}