Seed text → Predict next word

• Probability distribution of possible next words.

Read input (does not have to be purely text, could be any input including sound, images).

• Helpful is input and output are real numbers; as therefore can do calculus and gradients.

Create a lookup table, associating each word with others words in the same context, such as in the same sentence.

• Determine what is the probability of that word being the next word, from the list of possible words.
• Use back-propagation to adjust weights.

Aim for low cost; in determining the next word.

• cost formula is: cost = -log(probability).
  • A low cost, close to zero, is where the probability is closer to one.
  • A low probability (i.e. not predicting the next word), increases cost very steeply.
• There is a cost per word, but same applies to entire network.
  • Cost of entire network will be sum or avg of each word cost.