LayoutLMv3 Q/A Inference

nielsr
Mon, 23 Jan 2023 13:14:30 +0000

Hi,

The model outputs start_scores and end_scores, which are logits (unnormalized scores) that indicate which token is at the start, and which token is at the end of the answer. One can normalize them using softmax to turn them into probabilities.

However, to turn these scores into actual predictions, we just need to take the highest score (for both start_scores and end_scores), which will give us the index of the respective start and end token.

answer_start_index = outputs.start_logits.argmax()
answer_end_index = outputs.end_logits.argmax()

Next, we can use the processor (actually the tokenizer, which is used behind the scenes) to decode the predicted answer:

predict_answer_tokens = encoding.input_ids[0, answer_start_index : answer_end_index + 1]
print(processor.decode(predict_answer_tokens, skip_special_tokens=True))