Dynamics analysis of language processing systems (1)
Chinese link: https://zhuanlan.zhihu.com/p/717409759
1. Temporal Event Encoding in Dynamical Systems
Time is the basis of many interesting human behaviors [1].
In connectionist methods, such as neural networks, explicitly encoding temporal events based on their absolute positions can pose problems for recognizing similar patterns. A similar example of this issue is pointed out in work [1]. Pattern vectors $[0,0,0,1,1,1,0,0]$ and $[0,1,1,1,0,0,0,0]$ (where $1$ represent temporal events) exhibit similar patterns, even though the relative positions of the temporal events differ. However, the explicit encoding of absolute positions causes these vectors to have large differences.
Moreover, since it is difficult to explain how biological systems might use mechanisms similar to shift registers to process patterns with differences in relative positions, this encoding method lacks some degree of biological interpretability [1].
In Jeffrey L. Elman et al.’s work in 1990 [1], a method was proposed for implicitly representing the influence of time on temporal data processing during sequence processing. Specifically, a recurrent neural network was used, where the network’s internal hidden states encoded input events at each moment.
An abstract, discretized dynamical system can be represented by the Equation $1$ and $2$:
$$x_{t+1}=f(x_t,u_t,\mathcal P) \tag1$$
$$y_t=g(x_t,u_t,\mathcal P) \tag 2$$
where $u_t$ is the input to the system at time $t$, $x_t$ is the system’s state at time $t$, $y_t$ is the system’s output at time $t$, and $\mathcal P$ represents the system parameters. $f(\cdot)$ and $g(\cdot)$ are two function. In which, $f(\cdot)$ evolve the system’s states and $g(\cdot)$ generate output from input and state
All possible states $x_t$ form the phase space of the system.
In a dynamical system model, the encoding of a temporal event $u_t$ depends not only on the time event $u_t$ itself but also on the context in which the event occurs.
2. Geometric Encoding of Words in Phase Space
Specifically, let the sequence of temporal events ${u_t}_t$ represent a sequence of words in a given language.
When we enter the domain of dynamical systems, the encoding of the input $u_t$ at time $t$ no longer depends solely on $u_t$ itself, but also on the state of the language processing system at time $t$.
As mentioned in work [1], the implicit modeling of time within the state space of a dynamical system results in multiple different representations for a given input $u$.
More specifically, the representation of a word input depends not only on the word itself but also on the state of the language processing system when that word is processed.
From the perspective of dynamical systems, let the system’s phase space be $\mathcal{X}$, and for a given input $u_t$, the set of possible system states is $S(u_t)$, with $S(u_t)⊆\mathcal X$.
For a specific word $u_t$, all feasible solutions (i.e., $S(u_t)$) to the differential equation $\dot{x} = f(x, u_t, \theta)$ form an algebraic geometry within the phase space $\mathcal{X}$.
Here, the word input $u_t$ can be viewed as a vector field driving the system’s state evolution, and the state evolves into a flow according to the differential equation. All possible flows under fixed inputs forms a manifold.
A dynamical interpretation of $u_t$ would view it as a flow on the manifold.
As mentioned in work [1], studying the geometry within the state space (phase space) is meaningful.
One of its applications can be found in explaining linguistics-related theories. For example, under the same parameters of a dynamical system, the encoding of a single symbol in phase space can exhibit different representations [1] (such as in the earlier mentioned $S(u_t)$). The dynamical system model provides a certain degree of interpretability for the arbitrariness of the signifier-signified mapping, as pointed out in Saussure’s linguistic theory [3].
For different language processing systems (such as different human brains), we may perform phase space reconstruction (PSR) on brain imaging data (e.g., EEG, MEG, fMRI) to create a representation of the brain’s dynamics. This allows us to reconstruct the geometry of the same word in the brain’s phase space and analyze the dynamics of different brains in processing the same word. Moreover, the direct mapping of the dynamic geometry of word encoding under different dynamical systems might also be an interesting topic of study.
3. Discussion
3.1 The Connection Between Brain Dynamics and Language Processing System Dynamics
In 2017, Pillai AS et al.’s work [2] explored the dynamics of behavior, establishing a connection between brain dynamics and behavioral dynamics. Specifically, it shows that the dynamics of behavior may be modeled as a structured flow on a manifold (SFM) in a low-dimensional space of brain dynamics, where the flow is shaped by certain constraints.
Considering language processing behavior as one type of behavior, modeling the dynamics of a language processing system using SFM might be an interesting direction to pursue.
3.2 Trajectories in Generative Grammar
Let $G$ represent the set of grammar rules in a given language $L$, and let $\mathcal{C}(G_i)$ denote the corpus that can be generated by a certain grammar $G_i\in G$.
There exists a dynamical language processing system $\mathcal{D}_L$ for the language $L$, with an initial state $x_0$.
At this point, a particular word sequence $\bold c={c_1,c_2,…c_k}∈C(G_i)$ causes the system’s state to evolve, producing a state sequence $x_0,x_1,…,x_k,x_{k+1}$.
This state sequence forms a trajectory in the phase space of system $\mathcal{D}_L$.
Similarly, all possible trajectories generated by grammar $G_1$ also form a geometry within the phase space. This allows us to analyze the dynamics of the language processing system for different grammars by studying the geometries corresponding to these different grammars.
【Reference】
[1] Jeffrey L. Elman (1990). Finding structure in time*. Cognitive Science,14(2), 179-211.*
[2] Pillai AS, Jirsa VK. Symmetry Breaking in Space-Time Hierarchies Shapes Brain Dynamics and Behavior. Neuron. 2017 Jun 7;94(5):1010-1026. doi: 10.1016/j.neuron.2017.05.013. PMID: 28595045.
[3] De Saussure, Ferdinand. (1966). Course in General Linguistics (Edited by Charles Bally and Albert Sechehaye, Translated by Wade Baskin). New York, Toronto, London: McGraw-Hill Book Company.