How to Dissolve It

Table of contents
End in Mind
Backward Chaining
Qualia of Discovery

In the last month and a half, I’ve had more (of what I believe to be) profound, creative insights to technical problems than in the five years prior. For example, I independently came up with the core insight behind DenseNets during the second lecture on convolutional neural nets in my Deep Learning class. I’ve noticed that these insights occur as byproducts of good processes and having a conducive mindset and physiology at that point in time. I’m going to focus on the former in this post.

Often, when I’m confused about a technical problem, I realize that I don’t even know what a solution would look like. Sure, I could verify a solution if one were handed to me on a platter, but in these situations, I generally lack a detailed mental model of how I’d want a solution to behave. This may sound rather obvious once brought to your attention, but I’d like to emphasize how easy it is to spend hours in murky, muddled, pattern-matching gradient descent, mindlessly tweaking the first ideas that came to you.

Incredibly, generating a detailed solution model is often enough to quickly resolve (or at least reduce) even formidable-seeming problems.

The aforementioned process is backward chaining. This contrasts with forward chaining, which involves working forward from what you currently have. Take the time to actually visualize what a solution to the problem would look like. How would it behave in response to different inputs? Are you offloading all of the work onto a weasel word (like “emergence”)?

When you’re done, map-territory confusions should be ironed out, it should be clear whether current approaches truly tackle the core issue or just something similar-looking, and black-box terms should be disintegrated. If you can’t do this, you probably can’t solve the problem yet—you should spend more time working on your understanding. I encourage you to read this multiple times—the bulk of this post’s value resides in this section.

For example, yesterday I thought about a smaller problem for about 15 minutes, and had nothing to show for it. I almost resigned to asking a more mathematically experienced friend if he had any ideas, but instead forced myself to go through the above process. The answer was immediately obvious.

In my experience, proper solution formulations are accompanied by a satisfying mental “click.” Upon clicking, you eagerly chase down all the freshly exposed avenues of attack. Push the frontier of new understanding as far as it will go, repeat the process when you notice your thought process again becoming muddled, think with paper if the problem is complicated, and enjoy the rush.

I can’t impart to you the exact feeling of using this technique properly, but I can make its fruits concrete by selecting a few things I’ve dissolved recently:

I’ve observed $k$ samples from a normal distribution and recorded the mean and variance. New observations are going to come in, and I want to be able to assign some scalar $c \in [- 1, 1]$ quantifying how unusual this looks for the distribution.
A binary classifier has observed a bunch of “dog” pictures, and we want it to be able to robustly communicate when it sees something new—knowing what it doesn’t know.
I’m a reinforcement-learning agent named redacted, and I’d love to fetch things from other parts of your house for you—as quickly as possible.