Tokens and Tokenization Quiz

5 questions Pass: 70% +25 pts (50 with Pro)

Quiz covering Transformers and Attention

What you'll be asked

  1. What is a 'token' in the context of Large Language Models?
  2. Why do modern LLMs use sub-word tokenization methods like Byte Pair Encoding (BPE) instead of whole-word tokenization?
  3. If you are using a model with a context window of 8,000 tokens, what happens if your input prompt plus the expected output exceeds 8,000 tokens?
  4. How does the choice of tokenizer affect the cost and performance of a language model?
  5. When analyzing how a BPE tokenizer handles the word 'unbelievable', which of the following best describes the process?

Answers and explanations come after you submit.