profile-img
Nils Rethmeier

@Nils_Rethmeier

PhD candidate. continual SSL, low-resource SSL, unsupervised XAI, data-efficient multi-modal learning @Copenhagen University, @DFKI.

calendar_today10-11-2017 09:51:10

202 Tweets

315 Followers

210 Following

Nils Rethmeier(@Nils_Rethmeier) 's Twitter Profile Photo
Nils Rethmeier
@Nils_Rethmeier

2 years ago

Some LM pretraining tricks for the knowledge tool box.

Primer-EZ:
Use a better activation function (RelU**2) + a CNN layer in MHSA.

Makes large language model pretraining more efficient and thus a bit more accessible. This is great news.

account_circle