This can't be how neurons work, can it?

[tim1e9]

I find this code (and accompanying YouTube video) really fascinating. This is such a clever way to come up with a solution! If you gave me years of free time, I doubt I could come up with something so elegant. Very impressive, indeed!

Now that I've had a week or two to digest it, I'm wondering if someone could clarify something for me. I'll start out with a couple of observations, and then move to the question.

Note: I'm new to this stuff. This post could have some silly mistakes, or some incorrect assumptions. I am not presenting myself as an expert, but a novice who has encountered a confusing situation. I'm hoping for clarity - not validation. With that as background, I'll proceed.

In summary, backpropogation provides a way to come up with a solution (e.g. set of weights) that allows the neural net to provide the correct answers to the set of inputs (or "equations"). However, it is only a single solution. There are, it appears, an infinite number weights which will yield a valid solution. (And with a little Gen AI help, it seems obvious why: We have 41 parameters, but only four equations.)

Okay, fine: There are an infinite number of solutions to the four equations. So what? Well, what happens if you throw in an equation which adds a new aspect that hasn't been seen before? For example, what if you take this:

xs = [
  [2.0, 3.0, -1.0],
  [3.0, -1.0, 0.5],
  [0.5, 1.0, 1.0],
  [1.0, 1.0, -1.0]
]
ys = [1.0, -1.0, -1.0, 1.0] # desired targets

And make it this:

xs = [
  [2.0, 3.0, -1.0],
  [3.0, -1.0, 0.5],
  [0.5, 1.0, 1.0],
  [1.0, 1.0, -1.0],
  [1.0, 2.0, -3.0]
]
ys = [1.0, -1.0, -1.0, 1.0, 1.0] # desired targets

What happens to the previously calculated weights?

Stated another way: If I solve (e.g. determine collective weights) for 4 equations, does it bring me any closer to solving for 5 equations, or must I start all over again?

From what I can tell, you have to start all over again, and solving for 4 means nothing with respect to solving for 5. Here's my thinking:

1. The number of solutions for 4 equations is infinite.
2. The number of solutions for 5 equations is also infinite, but a subset of the solutions for 4. (If I understood Cantor's different types of infinity better, I'd say something really smart here. But I don't, so I can't.)
3. We solved for one such value for 4 equations. The odds of that being a correct one for 5 equations approaches zero.
4. To solve for 5 equations, we have to start all over again, and we gain "nothing" from first solving for 4 equations. And by "nothing", I mean - in light of the new equation, all of the weights must be completely reconsidered. The fifth equation invalidates everything we understood for solving for four.

This is the point where I'd kinda like to be wrong. And if I am, I will be in your debt. Because if it is true, then I can't imagine this is how our own neural nets (brains) work.

Regarding our brains: GenAI tells me we have something like 86 billion neurons, with each neuron (on average) having something like 10,000 synapses. In other words, there are far more neurons than necessary to "solve" for my current set of knowledge (or data points). However, if I come across something new, which contradicts the existing assumptions I've made, I don't have to start learning all over again. Instead, my brain simply incorporates this new nugget of data, and keeps going. Conversely, if it worked as described above, then the new information would invalidate all of the other "weights" of all of my neurons, and I'd have to re-learn everything.

To cite an absurd example: Today I learned something new about rhythm: if you tap along to a beat, your taps actually precede the sound by a few milliseconds — you’re not following, you’re predicting. That's cool! However, despite learning something completely new and surprising, I still know how to tie my shoes.

So... If this is how neurons work, then why didn't I have to re-learn to tie my shoes?

I welcome your comments and insights.

Best regards,
Tim

[kvishalv]

Let me chime in here with my understanding of this.

You're absolutely right about the infinite solutions part. 41 parameters and only 4 equations, there are indeed infinitely many weight configurations that could give you the correct outputs for those 4 examples. The backpropagation algorithm finds one solution, not "the" solution.
But here's where it gets interesting: When you add a 5th training example, you actually don't have to start from scratch, and in fact, you shouldn't. This is where neural network training differs from solving a static system of equations.

In practice, when you add new data:

• Warm starting works: You can (potentially should) initialize your network with the weights from the previous training run. The network will then adjust those weights through gradient descent to accommodate the new example. This is called "warm starting" and it's computationally much faster than retraining from scratch.

• The loss landscape matters: Neural networks aren't trying to solve exact equations, they're trying to minimize a loss function. When you add new data, you're changing the loss landscape slightly, but the previous solution is often already in a pretty good neighborhood. A few gradient descent steps can adjust the weights to fit the new data point.

• Generalization over memorization: The goal isn't to find weights that perfectly fit your training data, it's to find weights that generalize well to unseen data. Having more parameters than training examples (overparameterization) is actually common and can help with generalization through something called implicit regularization.

About your brain analogy: You're totally right that your brain doesn't have to relearn everything when you learn something new! And neural networks work similarly. When you learn that tapping precedes the beat, you don't forget how to tie shoes because, Your brain has massive overparameterization, just like you noted (86 billion neurons!). Different concepts are often represented in different parts of the network, there's some modularity. New learning often happens through incremental weight updates, not complete retraining.

The key insight is that gradient descent doesn't find a rigid, unique solution to a system of equations. It finds a point in parameter space that works well for the current data, and when new data arrives, it can incrementally adjust from that point. The weights don't all become invalid, they just need small tweaks to incorporate the new information.

In the micrograd example with just 4 training points, retraining from scratch is so fast it doesn't matter. But in real applications with millions of parameters and thousands/millions of training examples, people absolutely use warm starting, continual learning, and fine tuning techniques to avoid throwing away previous learning.

This connects to a whole area of research called continual learning or lifelong learning that's trying to make neural networks better at learning new things without forgetting old things (the "catastrophic forgetting" problem).

https://en.wikipedia.org/wiki/Catastrophic_interference

[tim1e9]

Okay, so it has taken me the better part of six months to get back to it. I took you up on your suggestion and tested two different starting points:

Start with 4. Expand to 5:

XS4 = [[2.0, 3.0, -1.0], [3.0, -1.0, 0.5], [0.5, 1.0, 1.0], [1.0, 1.0, -1.0]]
YS4 = [1.0, -1.0, -1.0, 1.0]

XS5 = [[2.0, 3.0, -1.0], [3.0, -1.0, 0.5], [0.5, 1.0, 1.0], [1.0, 1.0, -1.0], [1.0, 2.0, -3.0]]
YS5 = [1.0, -1.0, -1.0, 1.0, 1.0]

And for this particular set of equations, you are right that a warm start helps. But that's just because I happened to pick a fifth value that was consistent with the prior data. (Call it an unlucky guess.)

I did another test where the fifth equation was this:

XS5 = [[2.0, 3.0, -1.0], [3.0, -1.0, 0.5], [0.5, 1.0, 1.0], [1.0, 1.0, -1.0], [1.0, 2.0, 3.0]]
YS5 = [1.0, -1.0, -1.0, 1.0, 1.0]

The only difference: the fifth parameter's last value is 3.0 instead of -3.0.

When I did that, the data does not support your response. I ran 1000 tests and examined all 41 weights throughout the process - including the number of iterations before the loss was less than 0.001. Here are the details:

The v2 run finished. Here's the comparison:

	v1 (5th point: [1,2,-3])	v2 (5th point: [1,2,3])
From scratch mean iters	594	990
Warm mean iters	63	1020
Warm faster	999/1000 (99.9%)	433/911 (47.5%)

Changing the last feature from -3.0 to 3.0 completely destroyed the warm-start advantage. With the new 5th point:

• The problem is ~16x harder (1020 vs 63 iterations for warm)
• Warm-start and scratch are essentially a coin flip (47.5%)
• Neither reliably converges within 2000 steps

When I asked my friendly AI to explain why, I got the following:

This directly answers your earlier question about why the 4-point problem was harder than the 5-point problem in v1 — the specific 5th point [1,2,-3] happened to be very "compatible" with what the 4-point trained network already learned, making both the 5-point problem easy AND the warm-start very effective. Changing it to [1,2,3] breaks that compatibility.

So, given that this example definitively disproves that "warm starting works" in this case, I would like to give you the opportunity to respond. If warm starting works, then why did the v2 data / equations fail to converge faster than from scratch?

[log-prathm]

The model may have learned a shortcut from the first four examples: when x3 is negative, y tends to be positive, and when x3 is positive, y tends to be negative.

That would explain why the warm start helped in the first 5-point case but failed in the second. The new point [1, 2, -3] -> +1 is compatible with that shortcut, so the previous weights were already useful. But [1, 2, 3] -> +1 contradicts that shortcut, so the warm-started model begins from a representation that may now be actively misleading.

So I think you are correct that warm starting is not guaranteed to help. But the experiment does not show that the previous four-point solution is useless. It shows that the usefulness of a warm start depends on whether the new data is compatible with what the model had previously learned.

Another important point is dataset size. With only four or five examples, the model cannot reliably know whether the new point is an outlier, a contradiction, or evidence of a better underlying pattern. On such a tiny dataset, small changes can drastically change the loss landscape and the usefulness of the previous solution.