Experimenting in an Infinite Data Regime

Most machine learning tutorials gear toward defined datasets that can fit in the memory of most machines. These datasets are great for benchmarking new algorithms and for learning. However, newer SOTA models have many more...

Scaling Laws

My research direction for the OpenAI Scholar’s program is heavily influenced by the scaling laws paper by OpenAI on language models, and autoregressive models published last year. Scaling laws exist for cross-entropy loss in five...

Keeping Things Regular

In this post, I will introduce the direction of my OpenAI scholars’ project. Inspired by the Scaling Laws papers by OpenAI on language models, and autoregressive models published last year, I’m interested in learning more...

How to Navigate Conferences

The state of machine learning research moves incredibly fast. There are dozens of new papers published on arxiv every day, and it’s overwhelming trying to keep up to date. Conferences are a great way to...

Research Tools

One of the hardest things from switching from software engineering to research has been the open-ended nature of research and measuring progress. For me, software engineering usually has more explicit objectives, and it’s easy to...

Transformers, Roll Out!

I’ve spent a good majority of my time (when not constantly refreshing for election results) thinking about transformers. There are already many articles describing transformers and implementing them, so I won’t go into too much...

Hello from OpenAI

I’m excited to be joining the Fall 2020 cohort of OpenAI’s Scholars program. I’ll be writing semi-regularly as a log of what I’m learning and thinking about. I’m excited to be part of the scholars’...

The NLP Papers to Read Before ICLR 2020

Ahead of next week’s ICLR 2020 virtual conference, I went through the 687 accepted papers (out of 2594 submitted - up 63% since 2019!) and identified 9 papers with the potential to advance the use of deep...

Grounded Language Learning

Shaping representations through communication: community size effect in artificial learning systems

paper summary of Shaping Representations Through Communication: Community Size Effect in Artificial Learning Systems

Learning Machine Learning

Written for some coworkers who wanted to learn deep learning

On NMT Search Errors and Model Errors: Cat Got Your Tongue?

paper summary of On NMT Search Errors and Model Errors: Cat Got Your Tongue?

Beam Search

Negated: LAMA Birds cannot fly

paper summary of Negated LAMA: Birds cannot fly

Language Models as a Knowledge Base?

paper summary of Language Models as Knowledge Bases?

Backpropagation

Skip Connections and Residual Blocks

Human-Like Decision Making: Document-level Aspect Sentiment Classification via Hierarchical Reinforcement Learning

paper summary of Human-Like Decision Making: Document-level Aspect Sentiment Classification via Hierarchical Reinforcement Learning

Natural Language Understanding

Natural Language Understanding has been a focus in AI since the birth of AI as a field was defined. The 60s and 70s saw a burst of knowledge bases, ontologies, math word problem solvers. NLU...

Systems Sharing

A friend asked me the other day, “What are some “systems” that you have incorporated in your life that you felt have been super good ROI?” Here was my response: Weekly review Every week I...

Big Habits

Disclaimer: This post was written in less than 30 mins. I’m writing this for the sake of exploring an idea and am not convinced this is a good idea. Take this as a stream of...