Your first neural network will probably disappoint you. Not with a crash or a stack trace, but with a loss curve that flattens somewhere useless and a validation score that looks suspiciously close to a coin flip. That is normal. What separates people who get working models from people who keep starting over is rarely talent — it is knowing which handful of mistakes cause most of the pain.

Here are the seven that appear again and again in first projects, and how to handle each one.

1. Data Leakage: The Accuracy That Isn't Real

Leakage means information from the test set, or from the future, has slipped into training. The model looks brilliant in your notebook and falls apart the moment it meets real data.

The usual suspects are boring. Fitting a scaler or an imputer on the whole dataset before splitting it. Duplicate rows landing in both train and test. Columns that quietly encode the answer — if you are predicting whether a customer cancels, a field called cancellation_reason has no business being a feature.

The fix is a habit rather than a trick: split first, transform second. Fit every scaler, encoder and imputer on the training split only, then apply those already-fitted objects to validation and test data. For anything time-dependent, split chronologically and never shuffle. A model trained on next month's rows to predict this month's is reading the answers.

2. Choosing a Learning Rate by Guessing

A learning rate that is too high makes the loss bounce around or turn into nan. Too low, and the model crawls — fifty epochs in, it still looks like it has learned nothing.

Start somewhere sensible: around 1e-3 for Adam-style optimizers, and somewhere in the 0.01–0.1 range for plain stochastic gradient descent. Then spend ten minutes actually testing it. Train on a small subset for a few hundred steps at three or four different rates and compare the shape of the loss curves, not just the final number. If the curve spikes upward, halve the rate. If it descends smoothly but glacially, multiply it by three and try again.

Once training is stable, add a scheduler — reducing the rate when validation loss stalls, or a cosine decay over the run. If the first few hundred steps keep blowing up, a short warmup period often settles things.

3. Treating One Train/Test Split as Ground Truth

A single split gives you a noisy estimate. Change the random seed and your accuracy might move by several points, which is enough to make you chase improvements that do not exist.

Use k-fold cross-validation to tune decisions, especially with small datasets. Keep a genuinely untouched test set aside and look at it once, at the end — repeatedly checking it while tuning is just leakage with extra steps. If your classes are imbalanced, stratify the splits so each fold sees a similar mix. Before blaming the architecture, ask whether your evaluation is steady enough to measure anything at all.

4. Ignoring Overfitting Until It's Too Late

Overfitting has a signature: training loss keeps dropping while validation loss bottoms out and then climbs. The gap between the two curves is the whole story, and it is visible within minutes if you plot them together.

Ways to close that gap:

• Get more data, or augment what you have if the task allows it.
• Shrink the model — fewer layers, fewer units per layer.
• Add regularization such as weight decay or dropout.
• Stop training when validation loss stops improving, and keep the weights from that point.

Also watch for the opposite problem. If both curves sit flat and high, you are underfitting, and no amount of dropout will save you. That model needs more capacity, better features, or a longer, better-tuned training run.

5. Feeding Raw, Unscaled Inputs

Neural networks are sensitive to the scale of their inputs. If one feature ranges from 0 to 1 and another from 0 to 500,000, gradient updates get pulled in conflicting directions and training becomes slow and unstable.

Standardize numeric features to roughly zero mean and unit variance, or min-max scale them into a fixed range. Badly skewed features such as income or event counts often behave better after a log transform. Categorical variables need proper encoding — integer codes imply an ordering that usually does not exist, so use one-hot encoding or learned embeddings for high-cardinality columns.

Remember that the scaling statistics must come from the training data. Otherwise you have reintroduced the leakage from mistake number one.

6. Staring Only at the Loss Number

Loss tells you the model is optimizing something. It does not tell you the model is useful.

Track at least one metric that reflects the real goal, and inspect the confusion matrix rather than a single accuracy figure — a model can hit 95% accuracy on imbalanced data by predicting the majority class every time. Read twenty individual predictions by hand. You will spot patterns no aggregate number shows you: labels flipped, near-duplicate rows, a whole category the model has never seen.

And always compare against a trivial baseline. "Always predict the most common class" is a surprisingly stubborn opponent. If your network cannot beat it, the problem is upstream, in the data or the framing.

7. Changing Five Things at Once

When a run disappoints, the temptation is to rewrite the architecture, swap the optimizer, adjust the learning rate and rebuild the data pipeline in one afternoon. If the next result improves, you have no idea why. If it gets worse, you no longer have a baseline to return to.

Change one variable per experiment. Fix your random seeds so runs are comparable, and keep a plain log — a spreadsheet is plenty — with a row per run: what changed, the validation metric, and one line of notes. Twenty logged runs will teach you more about your problem than twenty clever guesses.

A Practical Checklist Before Your Next Run

1. Split the data before touching any preprocessing step.
2. Check for duplicates and for columns that encode the target.
3. Scale numeric inputs and encode categoricals properly.
4. Spend ten minutes testing a few learning rates, then add a scheduler.
5. Plot training and validation loss together on every single run.
6. Compare against a trivial baseline before celebrating anything.
7. Change one thing at a time, and write down what happened.

None of this is glamorous, and none of it requires a bigger GPU. It requires patience with the unglamorous parts — splitting cleanly, watching two curves instead of one, and resisting the urge to change everything at once. Do that, and your second neural network will go considerably better than your first.

Photo: Polina Zimmerman / Pexels