202006071613

Next Steps for Penalty Paper

tags: [ proj:penalty ]

  1. Matrix Completion -> ?

Our procedure works so well for matrix completion. However, how dependent is it to our data generating process? Matrix completion is basically a different view of link prediction – I wonder how well it does in that setting?

Can we use this in other settings: collaborative filtering, noisy matrix completion, network analysis.

  1. Speed/Initialization

This thing is pretty slow, especially compared to the more direct approaches. Is it possible to take what we’ve learnt from this method and apply it to other methods and get state-of-the-art and very high speed?

The nice thing now is that we really don’t have to worry about speed since we’re mostly just interested in implicit regularization in deep learning

  1. Theory

It would be really nice to be able to do some theory here. The main problem is that the theory for Adam is pretty unwieldy. It’s an element-wise calculation, and so it’s not particularly amenable to study through matrix calculations (the Hadamard doesn’t really help here).

  1. Noisy Matrix Completion

This is more in the realm of statistics, and, as such, usually requires considerable theory (though we can always the help of our collegues). It looks like our algorithm isn’t that great in the noisy case, as it’s esentially assuming that the data is noise-less. If we alter the loss function to be truncated, what happens then? Can we better handle the noisy case?

  1. Collaborative Filtering

The most important use-case for matrix completion is collaborative filtering. Except there it’s not as exact as saying that the matrix is low-rank. You’re instead looking for features, and some looser form of sparsity. I suspect our algorithm relies on the nice structure of matrices a little too much to be that useful in collaborative filtering, though, my intuition could be misguided here. Might even be able to do this with dictionary learning (see https://arxiv.org/pdf/1905.12091.pdf).

  1. More General Setting

See [[extending-the-penalty]].