Designing BERT for convolutional networks: sparse and hierarchical masked modeling Keyu Tian, Yi Jiang, Qishuai Diao, Chen Lin, Liwei Wang, Zehuan Yuan tl;dr: create image, which looks to CNN same, as transformers -> MIM starts working arxiv.org/abs/2301.03580…

🤩The first successful BERT-style #SelfSupervisedLearning on any convolutional network! #ResNet now enjoys masked autoencoding! 🚀A breakthrough paper "Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling" by Keyu Tian et al. deepai.org/publication/de…

[CV] Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling K Tian, Y Jiang, Q Diao, C Lin, L Wang, Z Yuan [Peking University & Bytedance Inc & University of Oxford] (2023) arxiv.org/abs/2301.03580 #MachineLearning #ML #AI #CV [1/2]

画像認識の事前学習で、ViTは一部の入力をMaskし、それを予測するタスクが成功していたがConvは隣り合うパッチ間で重複があり情報が漏れ成功してなかった。SparKは出力も入力と同じマスクを維持するSparseConvを使うことで問題を解決、また復号器側も階層を持ちさらに改善 openreview.net/forum?id=NRxyd…

How can we leverage successful pretraining techniques from transformers to improve purely convolutional networks? The answer is *Sparse Convolutions*! Let's see what happens when purely convolutional networks are pretrained with 1.28 million unlabeled images ... 1/7

: The first BERT-style pretraining on CNNs! #nlp #cnn reddit.com/r/MachineLearn…

SparK - [ICLR'23 Spotlight] The first successful BERT-style pretraining on any *convolutional network*; Pytorch impl. of "Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling" github.com/keyu-tian/SparK