Proposal: Supervised Training of Feed-Forward Networks in a Future Cognia Version

Context

While preparing Example 03 (plant-watering decisions), I wanted to build a conventional classification pipeline:

9 inputs -> nonlinear hidden layer -> 3 logits -> softmax -> decision

The inputs are soil moisture, temperature, air humidity, sunlight, rain forecast, time since the last watering, and a one-hot plant type. The dataset contains 600 synthetic, exactly balanced examples: 200 x NO_WATER, 200 x LIGHT_WATER, and 200 x DEEP_WATER. Both the generator and the network use seed 103.

Parsing, topology construction, training, network saving/loading, and REST inference all work. The problem is the quality and semantics of supervised learning across multiple layers.

What Was Measured

All variants were trained on the same dataset using:

cognia_train ... watering.dataset 5000 <network> --softmax

1. Fixed random hidden layer, 9 -> 24 -> 3:

   accuracy 408/600 = 68.0%
   

2. Fixed one-to-one transform, 9 -> 9 -> 3, without a direct shortcut:

   accuracy 314/600 = 52.3%
   

3. The same transform with a direct input -> output shortcut:

   accuracy 403/600 = 67.2%
   MSE      0.4355
   

4. Diagnostic shallow network, 9 -> 3, without a hidden layer:

   accuracy 400/600 = 66.7%
   MSE      0.3947
   

5. The original imbalanced random dataset reached 76.3%, but NO_WATER was the majority class. Accuracy fell after balancing the classes. Accuracy alone, without a per-class breakdown, therefore gave an overly optimistic result.

Conclusions from the measurements:

• adding a hidden layer produced no demonstrable improvement over the shallow variant;
• the result is sensitive to gain, leak, hidden-layer size, and hidden-layer type;
• a fixed one-to-one intermediate layer can substantially degrade performance;
• a direct skip connection roughly restores shallow-readout performance, but does not use the potential of the hidden representation;
• the problem is not only general deep credit assignment: even the shallow three-class ordinal classifier remained near two-thirds accuracy;
• the balanced dataset exposed a problem hidden by imbalanced accuracy.

What Is Confirmed and What Remains a Hypothesis

Confirmed behavior:

• Cognia has no general backpropagation/autodiff across a feed-forward DAG;
• --softmax trains the final readout, but the hidden transform receives no end-to-end credit based on classification error;
• REST returns raw activations, so the client must recompute probabilities;
• output activations pass through tanh/leak dynamics and are therefore not unbounded logits;
• the trainer prints every sample before and after training, overwhelming the output for larger datasets;
• it reports only aggregate accuracy and MSE, not a confusion matrix, per-class recall, or a separate validation result.

Likely causes that must be verified with unit tests:

• the softmax cross-entropy gradient may not be consistent with the value over which evaluation actually computes softmax (net versus tanh activation);
• the readout update may omit the output activation derivative, or may combine a cross-entropy gradient with a derivative that should not be applied to logits;
• normalization of sum(weighted) by in-degree changes the effective logit scale and forces users to guess gain manually;
• leak and repeated settling steps may make the training forward pass differ from the REST forward pass;
• output neurons do not have trainable biases. A bias is essential for the LIGHT_WATER class, which must win in the interval between two thresholds;
• the meaning of "5000 iterations" is insufficiently clear in CLI output: it is not obvious whether it means epochs, samples, or individual update steps.

These are diagnostic hypotheses, not confirmed implementation bugs.

Proposal for the Next Version

P0 — Unambiguous Classification Output

Introduce a dedicated neuron or output head whose state consists of real logits:

output classification from area.cells as watering {
    vocab: "no_water", "light_water", "deep_water";
    loss: softmax_cross_entropy;
}

The following contract must hold for this output:

• softmax is computed from net/logits before tanh, leak, and fire mechanics;
• the cross-entropy gradient with respect to a logit is exactly p - target;
• the same value is used for training, evaluation, accuracy, and the REST response;
• REST returns probabilities and selected_index/selected_label alongside logits;
• the classification head has one trainable bias per class;
• in-degree normalization is either not applied to logits or is explicit and included in the trainer's gradient. Users must not compensate gain manually according to the number of incoming edges.

P0 — Shallow-Softmax Regression Test

Add a small deterministic test for three-class ordinal classification. For example, use one input x and three classes LOW/MID/HIGH, with MID lying between two thresholds. A network with learned weights and biases must reach at least 95% on a balanced dataset.

The test must verify:

• numerical agreement between the analytical and implemented gradients;
• decreasing cross-entropy;
• successful learning of all three classes, not only the two extremes;
• matching predictions from the trainer, cognia_run, and REST /evaluate;
• unchanged results after save/load;
• reproducibility with the same seed.

P1 — Supervised Contract for a Feed-Forward DAG

Extend the language with an explicit training region so it is unambiguous which edges participate in gradient propagation:

training FeedForwardWatering {
    objective: supervised;
    input: sensors.environment, sensors.plant_type;
    output: brain.decision;
    loss: softmax_cross_entropy;
    optimizer: adam;
    owns: learnable path;
}

Minimum supported scope:

• an acyclic subnetwork;
• dense/full and one_to_one edges;
• tanh, sigmoid, ReLU, and linear activations;
• weights and biases;
• topological forward and reverse passes;
• a clear error for a cycle, gate, stochastic fire, or another unsupported operation;
• finite-difference gradient checks on small networks;
• optional save/load of optimizer state.

Cognia does not need general autodiff over the entire language. A bounded supervised DAG is enough to provide a reliable way to train ordinary feed-forward classification heads alongside local biologically inspired mechanisms.

P1 — If Backpropagation Is Undesirable

If Cognia should not add reverse-mode gradients even for a DAG, it must provide another explicit, tested path:

• a trainable random-feature/ELM head with a closed-form ridge solution;
• local target propagation with a defined contract;
• staged controller training with a target for every layer;
• CHL only where the required feedback symmetry and convergence are guaranteed.

The documentation must distinguish "feed-forward inference through multiple layers" from an "end-to-end trained feed-forward network." A fixed hidden layer with a learned readout must not be presented as deep supervised learning.

P1 — Metrics and Validation

Extend cognia_train with:

--validation validation.dataset
--metrics-json result.json
--quiet
--report-every 100
--early-stopping 20

Minimum report:

• training and validation loss;
• accuracy;
• confusion matrix;
• precision/recall for every class;
• sample counts for individual classes;
• best epoch and final epoch;
• learning rate and the actual number of update steps.

The trainer should warn about a strongly imbalanced dataset. In the watering example, such a warning would immediately reveal why 76.3% was not better than 68% on the balanced dataset.

P2 — Calibration and Decision Policy

Argmax alone is insufficient for practical decision-making. A REST classification output should return:

logits
probabilities
selected_label
confidence
entropy

Optional temperature scaling may be added using a validation set. Safety rules such as "do not water wet soil" should remain outside the neural network in ordinary control logic. The model recommends; a deterministic policy decides whether the action is permitted.

P2 — Dataset and CLI Diagnostics

• on a dimension mismatch, print both the expected and actual dimensions;
• for an empty dataset, print the first invalid line number and the reason;
• document the cognia-dataset <input_dim> <output_dim> header;
• provide --dry-run-dataset;
• print the mapping from sensors and outputs to dataset columns;
• define iterations, epochs, and batch size unambiguously.

Proposed Acceptance Criteria

1. The shallow LOW/MID/HIGH test reaches >= 95% on a balanced test set.
2. A 9 -> 16 -> 3 watering network reaches >= 90% training accuracy and >= 85% on a separate deterministic test set generated with a different seed.
3. Hidden weights demonstrably change during end-to-end training.
4. A numerical gradient check has relative error < 1e-4 on a small network.
5. The trainer, cognia_run, and REST return the same label and probabilities within tolerance.
6. Save/load does not change predictions or metrics.
7. A confusion matrix confirms that all three classes are learned.
8. For an unsupported cycle or stochastic operation, the trainer exits with a clear diagnostic instead of silently training only the final layer.

Recommended Implementation Order

1. Fix and numerically test the shallow softmax classification head, including biases.
2. Unify logits/probabilities between the trainer and REST.
3. Add a confusion matrix, validation, and quiet/JSON output.
4. Implement bounded backpropagation for an acyclic supervised subnetwork, or explicitly select and test a non-backpropagation alternative.
5. Only then present Example 03 as an end-to-end trained feed-forward network.

The highest priority is item 1. The measured 66.7% for the shallow variant shows that solving arbitrary deep graphs first would be premature. The final three-class classification head must first be reliable and mathematically consistent.