Week7

Week 7 #

Topic: The Canonical Correlation Analysis Family in Self-Supervised Learning

Keynote Speaker: Xunyi Jiang, Langtian Ma

Time: Aug 3, 19:30 - 21:00 pm

Venue: Lecture Hall 3, 302 (SUSTech)

Online Link: TencentMeeting

Compendium #

I. Traditional CCA

  • Generalized CCA framework
  • Traditional Nonlinear CCA
  • A compressed representatoin approach for CCA
  • Kernel CCA

II. Deep CCA and its variates

  • Deep canonical correlation analysis
  • Deep canonically correlated autoencoders

III. CCA in Self-supervised Learning

There are 4 major categories of self-supervise methods, including information maximization, clustering, distillation techniques, and contrastive method. In this week, I will introduce 3 methods(W-MSE/Barlow Twins/VICReg) lying in information maximization, and 2 methods(SeLa/SwAV) in clustering.

The story line is as following:

  • Overview of SSL methods
  • Information maximization methods: W-MSE/Barlow Twins/VICReg
  • Clustering methods: SeLa/SwAV
  • Summary of these methods

Material #

Slides 1 and 2 for Canonical Correlation Analysis Family from Xunyi Jiang and Langtian Ma.

References #

  1. Breiman, L et al, Estimating Optimal Transformations for Multiple Regression and Correlation

  2. Painsky A et al, Nonlinear Canonical Correlation Analysis:A Compressed Representation Approach

  3. D. R. Hardoon et al, Canonical correlation analysis: An overview with application to learning methods

  4. Andrew, G et al Deep canonical correlation analysis

  5. Wang, W et al On deep multi-view representation learning

  6. A. Ermolov, A. Siarohin, E. Sangineto, and N. Sebe, Whitening for Self-Supervised Representation Learning

  7. J. Zbontar, L. Jing, I. Misra, Y. Lecun, and S. Deny, Barlow Twins: Self-Supervised Learning via Redundancy Reduction

  8. A. Bardes, J. Ponce, and Y. Lecun, VICREG: VARIANCE-INVARIANCE-COVARIANCE RE- GULARIZATION FOR SELF-SUPERVISED LEARNING

  9. Y. Asano, C. Rupprecht, and A. Vedaldi, SELF-LABELLING VIA SIMULTANEOUS CLUSTERING AND REPRESENTATION LEARNING

  10. M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, Unsupervised Learning of Visual Features by Contrasting Cluster Assignments