- Why
upper bound vs uniform? - Relation between depth separation and Fourier uncertainty -
must be heavy-tailed so is localized. Is there any hint towards a NN uncertainty principle? - Generalized role of symmetries? Function composition?
- Form of the product measure - implication of independence of coordinates in input?
- Why is the density of Eldan & Shamir not included in the class of measures here?
- Difference between Maiorov & Meir’s Sobolev result (with logarithmic tightness) vs Yarotsky’s?
- Brief explanation of how number of parameters relates to approximation rate? (Barron gives
upper bound - why does this imply parameters for approximation rate. Furthermore, where is the in Eldan & Shamir’s original result? - Why a bound on both number of parameters and the magnitudes? Possible that this should be the width not the magnitude…
- For the lower bound, are there constraints on the number of parameters besides super-polynomial? (Think I found this in the appendix)
- How does normalization of the functions (looking at balls in
) guarantee significance in the approximation rates? - How does
bound on the weight magnitude imply low rate of oscillation of inner layers?
Open questions:
-
Can we get a result for Gaussian measures? Uniform measures?
-
Generalizing the idea of Eldan & Shamir to higher depths? What harmonic analysis intuition can we get for deeper networks? Can we use the intuition provided by approximation of deep networks by shallow ones of this paper?
-
Depth separation lower bounds for uniform measure in general domains? Identification of gaps in each?
-
Deep depth separation may be hopeless
-
Introducing symmetries more hopeful:
-
Lower bounds in optimization - algorithmic
-
Uniform approximation of constant Lipschitz functions?
-
Clayton & Su
What have Gunturk and I been interested in answering?
- Understanding the class of functions well-approximated by DNNs
- Getting a relationship between approximation error, depth, and regularity
- Understanding the source of gaps between upper and lower bounds
- Understanding the curse of dimensionality and how depth helps
- Deriving upper bounds using the integral transform view of NNs