4. Unsupervised learning

• 4.1. Gaussian mixture models
  • 4.1.1. GMM classifier
    • 4.1.1.1. Pros and cons of class GMM: expectation-maximization inference
      • 4.1.1.1.1. Pros
      • 4.1.1.1.2. Cons
    • 4.1.1.2. Selecting the number of components in a classical GMM
    • 4.1.1.3. Estimation algorithm Expectation-maximization
  • 4.1.2. VBGMM classifier: variational Gaussian mixtures
    • 4.1.2.1. Pros and cons of class VBGMM: variational inference
      • 4.1.2.1.1. Pros
      • 4.1.2.1.2. Cons
    • 4.1.2.2. Estimation algorithm: variational inference
  • 4.1.3. DPGMM classifier: Infinite Gaussian mixtures
    • 4.1.3.1. Pros and cons of class DPGMM: Diriclet process mixture model
      • 4.1.3.1.1. Pros
      • 4.1.3.1.2. Cons
    • 4.1.3.2. The Dirichlet Process
      • 4.1.3.2.1. Variational Gaussian Mixture Models
      • 4.1.3.2.2. Update rules for VB inference
        • 4.1.3.2.2.1. The spherical model
          • 4.1.3.2.2.1.1. The bound
          • 4.1.3.2.2.1.2. The updates
        • 4.1.3.2.2.2. The diagonal model
          • 4.1.3.2.2.2.1. The lower bound
          • 4.1.3.2.2.2.2. The updates
        • 4.1.3.2.2.3. The tied model
          • 4.1.3.2.2.3.1. The lower bound
          • 4.1.3.2.2.3.2. The updates
        • 4.1.3.2.2.4. The full model
          • 4.1.3.2.2.4.1. The lower bound
          • 4.1.3.2.2.4.2. The updates
• 4.2. Manifold learning
  • 4.2.1. Introduction
  • 4.2.2. Isomap
    • 4.2.2.1. Complexity
  • 4.2.3. Locally Linear Embedding
    • 4.2.3.1. Complexity
  • 4.2.4. Modified Locally Linear Embedding
    • 4.2.4.1. Complexity
  • 4.2.5. Hessian Eigenmapping
    • 4.2.5.1. Complexity
  • 4.2.6. Spectral Embedding
    • 4.2.6.1. Complexity
  • 4.2.7. Local Tangent Space Alignment
    • 4.2.7.1. Complexity
  • 4.2.8. Multi-dimensional Scaling (MDS)
    • 4.2.8.1. Metric MDS
    • 4.2.8.2. Nonmetric MDS
  • 4.2.9. Tips on practical use
• 4.3. Clustering
  • 4.3.1. Overview of clustering methods
  • 4.3.2. K-means
    • 4.3.2.1. Mini Batch K-Means
  • 4.3.3. Affinity Propagation
  • 4.3.4. Mean Shift
  • 4.3.5. Spectral clustering
    • 4.3.5.1. Different label assignement strategies
  • 4.3.6. Hierarchical clustering
    • 4.3.6.1. Adding connectivity constraints
  • 4.3.7. DBSCAN
  • 4.3.8. Clustering performance evaluation
    • 4.3.8.1. Inertia
      • 4.3.8.1.1. Presentation and usage
      • 4.3.8.1.2. Advantages
      • 4.3.8.1.3. Drawbacks
    • 4.3.8.2. Adjusted Rand index
      • 4.3.8.2.1. Presentation and usage
      • 4.3.8.2.2. Advantages
      • 4.3.8.2.3. Drawbacks
      • 4.3.8.2.4. Mathematical formulation
    • 4.3.8.3. Mutual Information based scores
      • 4.3.8.3.1. Presentation and usage
      • 4.3.8.3.2. Advantages
      • 4.3.8.3.3. Drawbacks
      • 4.3.8.3.4. Mathematical formulation
    • 4.3.8.4. Homogeneity, completeness and V-measure
      • 4.3.8.4.1. Presentation and usage
      • 4.3.8.4.2. Advantages
      • 4.3.8.4.3. Drawbacks
      • 4.3.8.4.4. Mathematical formulation
    • 4.3.8.5. Silhouette Coefficient
      • 4.3.8.5.1. Presentation and usage
      • 4.3.8.5.2. Advantages
      • 4.3.8.5.3. Drawbacks
• 4.4. Decomposing signals in components (matrix factorization problems)
  • 4.4.1. Principal component analysis (PCA)
    • 4.4.1.1. Exact PCA and probabilistic interpretation
    • 4.4.1.2. Approximate PCA
    • 4.4.1.3. Kernel PCA
    • 4.4.1.4. Sparse Principal Components Analysis (SparsePCA and MiniBatchSparsePCA)
  • 4.4.2. Dictionary Learning
    • 4.4.2.1. Sparse coding with a precomputed dictionary
    • 4.4.2.2. Generic dictionary learning
    • 4.4.2.3. Mini-batch dictionary learning
  • 4.4.3. Factor Analysis
  • 4.4.4. Independent component analysis (ICA)
  • 4.4.5. Non-negative matrix factorization (NMF or NNMF)
• 4.5. Covariance estimation
  • 4.5.1. Empirical covariance
  • 4.5.2. Shrunk Covariance
    • 4.5.2.1. Basic shrinkage
    • 4.5.2.2. Ledoit-Wolf shrinkage
    • 4.5.2.3. Oracle Approximating Shrinkage
  • 4.5.3. Sparse inverse covariance
  • 4.5.4. Robust Covariance Estimation
    • 4.5.4.1. Minimum Covariance Determinant
• 4.6. Novelty and Outlier Detection
  • 4.6.1. Novelty Detection
  • 4.6.2. Outlier Detection
    • 4.6.2.1. Fitting an elliptic envelop
    • 4.6.2.2. One-class SVM versus elliptic envelop
• 4.7. Hidden Markov Models
  • 4.7.1. Using HMM
    • 4.7.1.1. Building HMM and generating samples
    • 4.7.1.2. Training HMM parameters and infering the hidden states
    • 4.7.1.3. Implementing HMMs with custom emission probabilities