Paying in Attention

Search

SearchSearch

D-Separation

Nov 14, 20243 min read

BayesianMachineLearningMachineLearningProbabilityTheoryStatistics Given a Bayesian Network over , the edges of specify the dependencies between the ‘s explicitly. The edges also implicitly define indirect dependencies via Conditional Independence relations.

A subset of nodes D-separates from if for every (undirected) path from to there exists a node such that either:

  • are not observed and is Head-Head
  • is observed and is Tail-Tail or Tail-Head

Below shows the (length two) configurations that lead to various conditional independence relations:

  • Tail-Tail - have a common cause Active path: Which shows that i.e. and are not (marginally) independent in general via the calculation
Misplaced & \mathbb{P}(Y,Z) &= \sum_{X} \mathbb{P}(X,Y,Z)\\ &= \sum_X \mathbb{P}(Y \space | \space X) \cdot \mathbb{P}(Z \space | \space X) \cdot \mathbb{P}(X) \\ &\neq \mathbb{P}(Y) \mathbb{P}(Z) \end{aligned}$$ Inactive path: When conditioning on $X$ (i.e. it is observed), the path is blocked ![[tail_to_tail_observed.svg | 250]] and we see that $$\begin{aligned} \mathbb{P}(Y,Z \space | \space X) &= \frac{\mathbb{P}(X,Y,Z)}{\mathbb{P}(X)} \\ &= \frac{\mathbb{P}(Y \space | \space X) \cdot \mathbb{P}(Z \space | \space X) \cdot \mathbb{P}(X)}{\mathbb{P}(X)} \\ &= \mathbb{P}(Y \space | \space X) \cdot \mathbb{P}(Z \space | \space X) \end{aligned}$$ so $Y \perp\!\!\!\perp Z \space | \space X$. - **Tail-Head** Active path: ![[tail_to_head.svg|250]] and again $Y \centernot{\perp\!\!\!\perp} Z$, in general, by the calculation $$\begin{aligned} \mathbb{P}(Y,Z) &= \sum_{X} \mathbb{P}(X,Y,Z)\\ &= \sum_X \mathbb{P}(Y) \cdot \mathbb{P}(X \space | \space Y) \cdot \mathbb{P}(Z \space | \space X) \\ &\neq \mathbb{P}(Y) \mathbb{P}(Z) \end{aligned}$$ Inactive path: When $X$ is observed, the path is blocked ![[tail_to_head_observed.svg|250]] we obtain the factorization by Bayes rule: $$\begin{aligned} \mathbb{P}(Y,Z \space | \space X) &= \frac{\mathbb{P}(X,Y,Z)}{\mathbb{P}(X)} \\ &= \frac{\mathbb{P}(Y) \cdot \mathbb{P}(X \space | \space Y) \cdot \mathbb{P}(Z \space | \space X)}{\mathbb{P}(X)} \\ &= \frac{\mathbb{P}(X, Y)}{\mathbb{P}(X)}\cdot \mathbb{P}(Z \space | \space X) \\ &= \mathbb{P}(Y \space | \space X) \cdot \mathbb{P}(Z \space | \space X) \end{aligned}$$ and $Y \perp\!\!\!\perp Z \space | \space X$. - **Head-Head** ![[head_to_head.svg|250]] The flow of causality here shows that $Y \perp\!\!\!\perp Z$: $$\begin{aligned} \mathbb{P}(Y,Z) &= \sum_{X} \mathbb{P}(X,Y,Z)\\ &= \sum_X \mathbb{P}(Y) \cdot \mathbb{P}(Z) \cdot \mathbb{P}(X \space | \space Y, Z) \\ &= \mathbb{P}(Y) \cdot \mathbb{P}(Z) \sum_X \cdot \mathbb{P}(X \space | \space Y, Z) \\ &= \mathbb{P}(Y) \mathbb{P}(Z) \end{aligned}$$ When observing $X$ ![[head_to_head_observed.svg|250]] $Y \centernot{\perp\!\!\!\perp} Z \space | \space X$ as $$\begin{aligned} \mathbb{P}(Y,Z \space | \space X) &= \frac{\mathbb{P}(X,Y,Z)}{\mathbb{P}(X)} \\ &= \frac{\mathbb{P}(Y) \cdot \mathbb{P}(Z) \cdot \mathbb{P}(X \space | \space Y, Z)}{\mathbb{P}(X)} \\ &\neq \mathbb{P}(Y \space | \space X) \cdot \mathbb{P}(Z \space | \space X) \end{aligned}$$ so that observing $X$ introduces a dependency between $Y$ and $Z$. This is referred to as **explaining away** $Y$ or $Z$ (as observing one explains the other away).

Graph View

Backlinks