fully connected readout layer

[[concept]]

fully connected readout layer

In a fully connected readout layer we define $y=\rho (C.\text{vec}(x_{\ell }))$ Where

• $C\in {\mathbb{R}}^{d\times n{d}_L}$
• $\text{vec}(x_{\ell })\in {\mathbb{R}}^{n{d}_L}$
  • in general $\text{vec}(\cdot )$ vectorizes ${\mathbb{R}}^{m\times n}$ matrices into ${\mathbb{R}}^{mn}$ vectors
• $\rho$ can be the identity or some other pointwise nonlinearity (ReLU, softmax, etc)

Note

There are some downsides of a fully connected readout layer

• The number of parameters depends on $n$ - adding $n{d}_{L}d$ learning parameters, which grows with the graph size. This is not amenable to groups of large graphs

• No longer permutation invariant because of the $\text{vec}(\cdot )$ operation

fully connected readout layers are no longer permutation invariant

Verify that

• No longer transferrable across graphs.
  • unlike in GNNs, $C$ depends on $n$. So if the number of nodes $n$ changes, we have to relearn $C$

These make this a not-so-attractive option, so we usually use an aggregation readout layer

Mentions

TABLE
FROM [[]]
 
FLATTEN choice(contains(artist, this.file.link), 1, "") + choice(contains(author, this.file.link), 1, "") + choice(contains(director, this.file.link), 1, "") + choice(contains(source, this.file.link), 1, "") as direct_source
 
WHERE !direct_source