| Global Context-Aware Neural Language Model | To compute the score of local context, scorel, we use a neural network with one hidden layer: |
| Global Context-Aware Neural Language Model | where [£131,132, ...,;vm] is the concatenation of the m word embeddings representing sequence 8, f is an element-wise activation function such as tanh, a1 6 Rh“ is the activation of the hidden layer with h hidden nodes, W1 6 WW) and W2 6 nlxh are respectively the first and second layer weights of the neural network, and b1, ()2 are the biases of each layer. |
| Global Context-Aware Neural Language Model | the hidden layer with h(9) hidden nodes, ng) E Rh(g)x(2n) and ng) 6 Emmy) are respectively the first and second layer weights of the neural network, and big), bég) are the biases of each layer. |