TABLE

Matrix TypeDefinitionKey Properties (Eigenvalues $\lambda$, etc.)
Symmetric$A^T = A$$\lambda \in \mathbb{R}$;eigenvectors are orthogonal; always diagonalizable $A=U\Lambda U^T$.
Orthogonal$Q^T Q = I$$Q^{-1} = Q^T$; preserves norm $
Idempotent$P^2 = P$$\lambda \in \{0, 1\}$; $Rank(P) = Trace(P)$; Singular (unless $I$).
PSD$v^T A v \ge 0$$\lambda_i \ge 0$; all principal minors $\ge 0$; has square root $A^{1/2}$.
Diagonal$d_{ij}=0 (i \ne j)$$\lambda_i = d_{ii}$; Det = $\prod d_{ii}$; Inverse = $1/d_{ii}$.
TriangularUpper/Lower$\lambda_i$ are diagonal entries; Det = $\prod d_{ii}$.
  • Hat Matrix $\subset$ Projection Matrix $\subset$ Idempotent & Symmetric.
  • Covariance Matrix $\subset$ PSD Matrix $\subset$ Symmetric Matrix.

1. Matrix Classification & Properties

  • Symmetric Matrix: Defined as $A^T = A$. Eigenvalues are real ($\lambda \in \mathbb{R}$). Eigenvectors are orthogonal. It is always diagonalizable ($A=U\Lambda U^T$).
  • Orthogonal Matrix: Defined as $Q^T Q = I$. Inverse equals transpose ($Q^{-1} = Q^T$). It preserves the norm ($||Qx|| = ||x||$).
  • Idempotent Matrix: Defined as $P^2 = P$. Eigenvalues are either 0 or 1 ($\lambda \in \{0, 1\}$). Trace equals Rank ($Rank(P) = Trace(P)$). It is singular unless it is the Identity matrix.
  • Positive Semi-Definite (PSD): Defined as $v^T A v \ge 0$. Eigenvalues are non-negative ($\lambda_i \ge 0$). All principal minors are $\ge 0$. It possesses a square root ($A^{1/2}$).
    • Note on PD vs PSD: Positive Definite (PD) requires strict positivity ($>0$) and checking Leading Principal Minors is sufficient (Sylvester’s criterion). PSD allows 0, requiring the check of all Principal Minors.
  • Diagonal Matrix: Entries $d_{ij} = 0$ for $i \ne j$. Eigenvalues are diagonal entries ($\lambda_i = d_{ii}$). Determinant is the product of diagonal entries. Inverse is $1/d_{ii}$.
  • Triangular Matrix (Upper/Lower): Eigenvalues are the diagonal entries. Determinant is the product of the diagonal entries.
  • Relationship Chain:
    • Hat Matrix $\subset$ Projection Matrix $\subset$ Idempotent & Symmetric.
    • Covariance Matrix $\subset$ PSD Matrix $\subset$ Symmetric Matrix.

2. Functional Matrices & Operations

  • Gram Matrix: $G = X^T X$. Always Symmetric and PSD.
  • Covariance Matrix ($S$): $S = \frac{1}{n}X_c^T X_c$. Symmetric and PSD.
  • Correlation Matrix ($R$): Covariance of standardized data. Diagonal elements are 1; other elements $\in [-1, 1]$.
  • Projection Matrix ($P$): Symmetric ($P^T = P$) and Idempotent ($P^2 = P$). The orthogonal complement $P_{\perp} = I - P$ is also a projection matrix (onto $\mathcal{C}(X)^{\perp}$).
    • Hat Matrix ($H$): $H = X(X^T X)^{-1} X^T$. If $X$ is column-orthogonal, $H = XX^T$. Projects onto the Column Space of $X$. $Tr(H) = p$ (Rank).
    • Centering Matrix ($C$): $C = I - \frac{1}{n}11^T$. Property: $C1 = 0$.
  • Centering Operations (Kernel Context):
    • Right Multiplication ($KC$): Row Centering (Row sums become 0).
    • Left Multiplication ($CK$): Column Centering.
    • Double Centering ($CKC$): Adjusts for over-subtraction.
  • Matrix Properties of $X$:
    • Column-Centered: Mean of each column is 0 ($1^T X = 0^T$). Operation: $X_{centered} = CX$.
    • Column-Orthogonal: Columns are perpendicular and unit length ($X^T X = I$). Implies unit variance ($S_{ii}=1$) and uncorrelated features ($S_{ij}=0$).
    • Outer Product: $uv^T$. Result is a matrix ($n \times m$). Rank is 1 (all columns multiples of $u$, all rows multiples of $v$).

3. Trace & Matrix Tricks

  • Scalar Trick: If the result is a scalar, trace can be applied for derivatives: $\nabla_{A}tr(AB) = B^T$.
  • Frobenius Norm: $||A||_F^2 = \sum A_{ij}^2 = tr(A^T A)$.
  • Cyclic Property: $tr(ABC) = tr(BCA) = tr(CAB)$. Useful for rearranging terms like $tr(Axx^T)$.
  • Spectral Property: $tr(A) = \sum \lambda_i$. (e.g., used to calculate degrees of freedom for Hat Matrix). Total Variance = Trace of Covariance Matrix.

4. Linear Regression & Distributions

  • Standard Model: $y = X\beta + \epsilon$, where $\epsilon \sim \mathcal{N}_n(0, \sigma^2 I_n)$.
  • Estimators:
    • OLS Estimator: $\hat{\beta} = (X^T X)^{-1} X^T y$. If $X$ is column-orthogonal ($X^T X = I$), then $\hat{\beta} = X^T y$.
    • Sampling Distribution: $\hat{\beta} \sim \mathcal{N}_p(\beta, \sigma^2 (X^T X)^{-1})$.
  • Ridge Regression:
    • Formula: $\hat{\beta}_R = (X^T X + \lambda I)^{-1} X^T y$.
    • Ridge Hat Matrix: $A_{\lambda} = X(X^T X + \lambda I)^{-1} X^T$. When $\lambda=0$, it reverts to the OLS Hat Matrix.
    • Role of $\lambda$: It is a Penalty term. It modifies eigenvalues from $d_i^2$ to $d_i^2 + \lambda$, preventing division by zero (singularity).
    • Properties: $(X^T X + \lambda I)^{-1}$ is Symmetric. Commutativity applies between $(X^T X)$ and its inverse/polynomials (e.g., $(X^T X + \lambda I)^{-1}$).
  • Assumptions & Properties:
    • Homoscedasticity: Scalar $\sigma^2$ implies diagonal elements of error covariance are equal. If variance is not constant (Heteroscedasticity), $\Sigma \neq \sigma^2 I$, requiring GLS/WLS.
    • Independence: Off-diagonal elements of error covariance are 0.
    • Why $X^T X \neq I$: Data features are usually correlated and not whitened. If $X^T X = I$, Ridge is unnecessary.
    • Equivalence: Maximizing Log-likelihood for Normal errors is equivalent to Least Squares (minimizing $||y - X\beta||^2$).
  • Residuals & Independence:
    • Fitted values $\hat{y} = Py \sim \mathcal{N}(X\beta, \sigma^2 P)$.
    • Residuals $r = (I-P)y \sim \mathcal{N}(0, \sigma^2 (I-P))$.
    • General Covariance: $Cov(Ay, By) = A Cov(y) B^T$.
    • Independence Proof: $Cov(\hat{y}, r) = \sigma^2(P - P^2) = 0$. For Multivariate Normal (MVN), Uncorrelated ($Cov=0$) implies Independent.
  • Hypothesis Testing ($H_0: \beta = 0$):
    • Test Statistic ($\sigma^2$ known): $\chi^2 = \frac{||\hat{\beta}||^2}{\sigma^2} \sim \chi^2_p$.
    • Test Statistic ($\sigma^2$ unknown): Use F-test involving SSE.
    • Rejection Meaning: At least one $\beta_i \neq 0$.

5. Decomposition & Approximation

  • Eckart-Young-Mirsky Theorem:
    • $A_k = \sum_{i=1}^k \sigma_i u_i v_i^T$ is the best rank-$k$ approximation of $A$ under the Frobenius norm.
    • Approximation Error: $||A - A_k||_F = \sqrt{\sum_{i=k+1}^r \sigma_i^2}$ (Sum of squared discarded singular values).

6. Principal Component Analysis (PCA) Family

  • Standard PCA ($X^T X$):
    • Focus: Features (Columns). Analyzes Covariance Matrix $S \propto X^T X$ ($p \times p$).
    • Geometry: Finds direction $v$ maximizing variance. If points lie on line $y=x$, PC1 is along (1,1) and PC2 (orthogonal) has 0 variance.
    • Calculations: Scores $z = Xv$. Variance Explained $\lambda = Var(z)$.
    • Scale Sensitivity: PCA is not scale-invariant. Scaling a feature (e.g., meters to cm) drastically changes $\lambda$ and eigenvectors.
    • Interpretation: It is a lossy compression method minimizing reconstruction error.
  • Kernel PCA ($XX^T$):
    • Focus: Samples (Rows). Analyzes Kernel Matrix $K = \Phi \Phi^T$ ($n \times n$).
    • Kernel Trick: $K(x,y) = \langle \phi(x), \phi(y) \rangle$. Computes inner product in high-dimensional feature space using low-dimensional inputs.
    • Example: $K=(x^T y)^2$ implies a quadratic feature map with $\sqrt{2}$ coefficients.
    • Linear Kernel: $K(x,y) = x^T y$ implies $\phi(x)=x$. In this case, Linear KPCA is equivalent to Standard PCA.
    • SVD Connection: Let $X = U\Sigma V^T$. Standard PCA diagonalizes $X^T X$ (Eigenvectors $V$). KPCA diagonalizes $XX^T$ (Eigenvectors $U$). They share non-zero eigenvalues $\lambda_i = \sigma_i^2$.
    • Property: Kernel Matrix $K$ is always Positive Semi-Definite (PSD).

7. Latent Variable Models

  • Factor Analysis (FA):
    • Assumption: $E[zz^T] = I$. Derived from $z \sim N(0, I)$ where diagonal variance is 1 and off-diagonal covariance is 0.
    • Variance Decomposition: Total Variance ($\Sigma_{ii}$) = Communality ($h_i$) + Uniqueness ($\Psi_i$). $h_i = (LL^T)_{ii} = \sum l_{ij}^2$.
    • Rotation:
      • Isotropy: If $L^T L = I$, rotation is ineffective.
      • Varimax: Used when $L^T L \neq I$ to maximize identifiability. Uses Kaiser Normalization to adjust for communality differences.
    • Scoring:
      • Weighted (Bartlett/MLE): $\hat{z} = (L^T \Psi^{-1} L)^{-1} L^T \Psi^{-1} x$. Uses $\Psi^{-1}$ as weights (trusts low-noise variables more).
      • Unweighted (PCA approach): $\hat{z} = (L^T L)^{-1} L^T x$. Implicitly assumes $\Psi = I$ (OLS).
  • Independent Component Analysis (ICA):
    • Concept: Uncorrelated $\neq$ Independent. $Cov(X,Y)=0$ does not imply Independence (e.g., $Y=X^2$).
    • Goal: De-mixing to maximize Non-Gaussianity (Reverse CLT logic).
    • Measures:
      • Kurtosis ($y^4$): Sensitive to outliers ($4y^3$ derivative).
      • Negentropy ($\log \cosh y$): Used in FastICA. Robust (derivative $\tanh y$ is bounded).
    • FastICA Steps: 1. Preprocessing (Centering + Whitening $\Sigma^{-1/2}X$). 2. Deflationary iteration (Find $w_1$, then find $w_2$ orthogonal to $w_1$). 3. $W$ is the unmixing matrix where $s = Wx$.

8. Canonical Correlation Analysis (CCA)

  • Goal: Find vectors $u, v$ to maximize correlation $\rho = Corr(Xu, Yv)$.
  • Constraint: $Var(U) = 1, Var(V) = 1$ (Scale Invariance). This allows simplifying the denominator to 1, maximizing the numerator $u^T S_{XY} v$.
  • Calculation (SVD Method):
    • Matrices involved: $S_X, S_Y, S_{XY}$. If columns are orthonormal/uncorrelated features, $S_X=I, S_Y=I$.
    • Decompose Matrix $K = S_{XX}^{-1/2} S_{XY} S_{YY}^{-1/2}$.
    • 1st Canonical Correlation ($\rho_1$): Largest singular value. Represents the primary coupled mode.
    • 2nd Canonical Correlation ($\rho_2$): Second largest singular value. Represents the secondary independent mode (orthogonal to the first). $\rho_1$ and $\rho_2$ coexist for a given dataset; $\rho_2$ is found by finding the next orthogonal direction, not by removing data.

9. Statistical Estimation Principles

  • MMSE Estimator: Conditional Expectation $E[z|x]$ is the Minimum Mean Squared Error estimator.
  • Derivation: Decomposition of MSE shows that setting $f(x) = E[z|x]$ minimizes the error term to 0.
  • Projection: Geometrically, $E[z|x]$ is the orthogonal projection of $z$ onto the space spanned by $x$.
  • Independence Property: If $P(XY) = P(X)P(Y)$, then $E[XY] = E[X]E[Y]$. Consequently, $Var(X+Y) = Var(X) + Var(Y)$ (Cross-terms vanish).

没问题，Alex。既然你需要最严谨、不省略步骤的推导，我们将不仅仅停留在图片的文字解释上，而是要把背后的数学推导 (Derivation) 完整写出来。

这道题的本质是考察 CCA 的优化形式以及它与OLS 回归的数学联系。

以下是针对 Part (a), (b), (c) 的完整解析和公式推导。

Part (a): The Objective (CCA Optimization Problem)

目标： 写出 CCA 的优化问题数学表达。

1. 定义变量与协方差矩阵： 假设我们有两个去中心化（Centered, mean=0）的数据矩阵 $X \in \mathbb{R}^{n \times p}$ 和 $Y \in \mathbb{R}^{n \times q}$。 样本协方差矩阵定义如下：

$$S_{XX} = \frac{1}{n} X^\top X, \quad S_{YY} = \frac{1}{n} Y^\top Y, \quad S_{XY} = \frac{1}{n} X^\top Y$$

我们要寻找线性投影方向 $u \in \mathbb{R}^p$ 和 $v \in \mathbb{R}^q$，使得投影后的变量 $U = Xu$ 和 $V = Yv$ 的相关系数最大。

2. 原始目标函数 (Unconstrained Objective)： 我们要最大化相关系数 $\rho$：

$$\max_{u, v} \rho(u, v) = \frac{\text{Cov}(Xu, Yv)}{\sqrt{\text{Var}(Xu)} \sqrt{\text{Var}(Yv)}} = \frac{u^\top S_{XY} v}{\sqrt{u^\top S_{XX} u} \sqrt{v^\top S_{YY} v}}$$

3. 处理尺度不变性 (Scale Invariance)： 注意到，如果我们把 $u$ 替换为 $\alpha u$ ($\alpha > 0$)，分子分母同时扩大 $\alpha$ 倍，$\rho$ 的值不变。 为了消除这种不确定性并简化分母，我们引入约束条件 (Constraints)，强制投影后的方差为 1：

$$u^\top S_{XX} u = 1$$

$$v^\top S_{YY} v = 1$$

4. 最终优化问题 (Final Optimization Problem)： 结合分子最大化和分母约束，得到标准形式：

  • Maximize: $$u^\top S_{XY} v$$
  • Subject to: $$u^\top S_{XX} u = 1$$ $$v^\top S_{YY} v = 1$$

Part (b): Independence

情景： 假设 $X$ 和 $Y$ 统计独立 (Statistically Independent)。

1. 理论推导： 如果 $X$ 和 $Y$ 独立，意味着它们之间不存在线性关系（也不存在非线性关系）。 在数学期望层面，总体交叉协方差矩阵 (Population Cross-Covariance Matrix) $\Sigma_{XY}$ 为零矩阵：

$$\Sigma_{XY} = \mathbb{E}[(X - \mu_X)(Y - \mu_Y)^\top] = \mathbf{0}$$

2. 对优化目标的影响： 回到 Part (a) 的目标函数，此时我们要最大化的量变成了：

$$u^\top \Sigma_{XY} v = u^\top \mathbf{0} v = 0$$

3. 结论：

  • Theoretical Maximum Canonical Correlation: 严格为 0。
  • Finite Sample Note (有限样本情况): 在实际数据中，由于样本量 $n$ 是有限的，样本协方差矩阵 $S_{XY} = \frac{1}{n}X^\top Y$ 通常不会是完美的零矩阵（由于随机噪声/Spurious Correlation）。因此计算出的 Sample Canonical Correlation 会是一个很小的非零正数，但在统计意义上它应该接近于 0。

Part (c): Relation to Regression (OLS)

目标： 证明当 $Y$ 是单变量 ($y$) 时，CCA 的方向 $u$ 与 OLS 回归系数 $\hat{\beta}$ 成比例 ($u \propto \hat{\beta}$)。

这里图片中给出了几何证明，我为你补充代数推导，这才是最严谨的“不省略”步骤。

设定环境

  • $Y$ 退化为单列向量 $y \in \mathbb{R}^{n \times 1}$。
  • $v$ 退化为标量 $v \in \mathbb{R}$ (因为是一维，方向只有正负，其模长被归一化限制，实际上 $v=1$ 或 $v$ 被吸收到缩放中，我们可以忽略 $v$ 的优化，只关注 $u$)。
  • OLS 回归系数公式： $\hat{\beta}_{OLS} = (X^\top X)^{-1} X^\top y$。
  • CCA 目标 (针对单变量 $y$)： $$\max_u \text{Corr}(Xu, y) = \frac{u^\top S_{Xy}}{\sqrt{u^\top S_{XX} u} \sqrt{S_{yy}}}$$

严谨代数推导 (Lagrange Multipliers Method)

Step 1: 简化目标函数 在最大化相关系数时，$\sqrt{S_{yy}}$ 是常数，可以忽略。 CCA 问题转化为：

  • Maximize: $f(u) = u^\top S_{Xy}$
  • Subject to: $g(u) = u^\top S_{XX} u - 1 = 0$

Step 2: 构造拉格朗日函数 (Lagrangian) 引入拉格朗日乘子 $\lambda$：

$$\mathcal{L}(u, \lambda) = u^\top S_{Xy} - \frac{\lambda}{2} (u^\top S_{XX} u - 1)$$

(注：用 $\frac{\lambda}{2}$ 是为了求导方便，不影响结果)

Step 3: 求导并置零 (FOC) 对向量 $u$ 求偏导：

$$\frac{\partial \mathcal{L}}{\partial u} = \frac{\partial}{\partial u}(u^\top S_{Xy}) - \frac{\lambda}{2} \frac{\partial}{\partial u}(u^\top S_{XX} u)$$

利用矩阵求导公式 $\frac{\partial (a^\top x)}{\partial x} = a$ 和 $\frac{\partial (x^\top A x)}{\partial x} = 2Ax$ (当 A 对称时)：

$$S_{Xy} - \frac{\lambda}{2} (2 S_{XX} u) = 0$$

$$S_{Xy} - \lambda S_{XX} u = 0$$

Step 4: 求解 $u$ 移项得到：

$$\lambda S_{XX} u = S_{Xy}$$

假设 $S_{XX}$ 可逆 (Invertible)，两边左乘 $S_{XX}^{-1}$：

$$u = \frac{1}{\lambda} S_{XX}^{-1} S_{Xy}$$

这意味着 $u$ 的方向与 $S_{XX}^{-1} S_{Xy}$ 相同，常数 $\frac{1}{\lambda}$ 只是缩放因子。所以：

$$u \propto S_{XX}^{-1} S_{Xy}$$

Step 5: 代入样本矩阵定义 将 $S_{XX} = \frac{1}{n} X^\top X$ 和 $S_{Xy} = \frac{1}{n} X^\top y$ 代入：

$$u \propto (\frac{1}{n} X^\top X)^{-1} (\frac{1}{n} X^\top y)$$

常数 $\frac{1}{n}$ 互相抵消：

$$u \propto (X^\top X)^{-1} X^\top y$$

Step 6: 与 OLS 对比 我们要对比的对象是 OLS 回归系数：

$$\hat{\beta}_{OLS} = (X^\top X)^{-1} X^\top y$$

观察 Step 5 和 Step 6 的结果，完全一致。

结论：

$$u \propto \hat{\beta}_{OLS}$$

即：当 $Y$ 为一维时，CCA 求出的投影方向 $u$ 正好就是 OLS 回归系数的方向。

Summary for Exam

  1. Optimization: $\max u^\top S_{XY} v$ s.t. $u^\top S_{XX} u = 1, v^\top S_{YY} v = 1$.
  2. Independence: $S_{XY} \approx 0 \implies \text{Correlation} \approx 0$.
  3. Regression: $u \propto S_{XX}^{-1} S_{Xy} = (X^\top X)^{-1} X^\top y = \hat{\beta}_{OLS}$.