Dynamics analysis of language processing systems (1)
Chinese link: https://zhuanlan.zhihu.com/p/717409759
1. Temporal Event Encoding in Dynamical Systems
Time is the basis of many interesting human behaviors [1].
In connectionist methods, such as neural networks, explicitly encoding temporal events based on their absolute positions can pose problems for recognizing similar patterns. A similar example of this issue is pointed out in work [1]. Pattern vectors $[0,0,0,1,1,1,0,0]$ and $[0,1,1,1,0,0,0,0]$ (where $1$ represent temporal events) exhibit similar patterns, even though the relative positions of the temporal events differ. However, the explicit encoding of absolute positions causes these vectors to have large differences.
Moreover, since it is difficult to explain how biological systems might use mechanisms similar to shift registers to process patterns with differences in relative positions, this encoding method lacks some degree of biological interpretability [1].
In Jeffrey L. Elman et al.’s work in 1990 [1], a method was proposed for implicitly representing the influence of time on temporal data processing during sequence processing. Specifically, a recurrent neural network was used, where the network’s internal hidden states encoded input events at each moment.