Seed text → Predict next word

• Probability distribution of possible next words.

Read input (does not have to be purely text, could be any input including sound, images).

• Helpful is input and output are real numbers; as therefore can do calculus and gradients.

Lookup Table

• Create a lookup table, associating each word with others words in the same context, such as in the same sentence.
  • Encode the position of a word in the context.
  • Are there nouns at specific positions?
  • For each word, are there any adjectives in certain positions, in front of the word?
• Determine what is the probability of that word being the next word, from the list of possible words.
• Use back-propagation to adjust weights.

Weights = which words are associated with other words.

• Bigger numbers matter more.
• Smaller numbers are unrelated; matter less.
• Use dot.product; as cheaper
  • v1 * w1 = v1w1
  • v1w1 + v2w2 + v3w3 = dot.product
    • Measures whether this points in the same direction.
      • Positive value means it points (aligns).
      • Zero means it is perpendicular (unrelated).
      • Negative value means it is opposit.

Activation

• Use softmax
  • Converts weights to all be between zero and one; and add up to one.

Aim for low cost; in determining the next word.

• cost formula is: cost = -log(probability).
  • A low cost, close to zero, is where the probability is closer to one.
  • A low probability (i.e. not predicting the next word), increases cost very steeply.
• There is a cost per word, but same applies to entire network.
  • Cost of entire network will be sum or avg of each word cost.