Architectural Orthogonality

Proving the gain is independent of the host network

Experimental argumentation strategy for papers proposing a pluggable technique (a layer, a preprocessing step, a loss): demonstrating that the gain replicates across several host architectures. Complements the ablation-study, which isolates internal components, within the structure described in redacao-academica-guia.

What it is

When the paper’s contribution is a piece that attaches to existing systems, a single successful experiment is not enough: the reader may suspect the gain is luck of that specific combination. Demonstrating orthogonality consists of applying the technique to $N$ architectures of varied design and showing that all of them improve in the same direction — the technique is “orthogonal” to the host network’s design.

The typical argumentative move has three steps:

1. Vary the host deliberately: pick baselines that differ from each other in depth, filter counts, strides, and feature map sizes — diversity is what gives strength to the generalization.
2. Report pair by pair: for each architecture, show the baseline without the technique side by side with the result with the technique, with the delta made explicit (the paper uses parentheses in the table: the gain is visible without cross-referencing).
3. Turn it into a stated conjecture: finish by claiming that, if the technique improves such different architectures, “it is reasonable to conjecture” it will improve future, deeper architectures — an explicit invitation for community adoption.

Anatomy: the SPP-net case

The SPP-net paper (He et al., 2014) is the canonical exemplar. In Table 2, it applies the SPP layer to four architectures (ZF-5, Convnet*-5, Overfeat-5, Overfeat-7) and shows top-1 error dropping in all of them (gains from 0.55% to 1.65%). An elegant rhetorical detail: the authors note the largest gain came from the most accurate architecture — suggesting the technique scales with host quality.

The paper goes further and includes a control experiment against the alternative explanation:

That third row disarms in advance the objection “it improved because the network got bigger”: the model with fewer parameters than the baseline still beats it. It is the analogue of the “informative negative result” from ablation studies.

Best practices

• Declare the experiment design: state explicitly why the chosen architectures differ (“various filter numbers/sizes, strides, depths”) — that is what turns 4 numbers into evidence of generalization.
• Separate orthogonality from ablation: ablation answers “is each of my pieces necessary?”; orthogonality answers “does my piece work in other systems?”. Strong papers do both, in distinct sections.
• Include the control for the alternative explanation: if the technique adds parameters, compute, or any other resource, show a variant that removes that confounder and keeps the gain.
• Conclude with the transfer conjecture: the value of a pluggable technique lies precisely in third-party use — close the section inviting that use.

⚠️ Common pitfall: testing only on trivial variations of the same network (same family, changing only width). That demonstrates internal robustness, not orthogonality — hosts need to come from different publications or families.

References and tools

• He, K., Zhang, X., Ren, S., Sun, J. “Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition” (2014). arXiv:1406.4729 — section 3.1 and Table 2 as an exemplar of orthogonality demonstration + confounder control.

Related: ablation-study · redacao-academica-guia · spp-net