A Fun Experiment: The Nitro-Protocol (CyLU)

What this is: Just a project I was tinkering with because I think physics is cool.

1. The Idea

I've always liked the Brachistochrone curve—it's the path that gets you from point A to B the fastest using gravity. I wondered if we could use that same "fastest descent" logic for the start of training an AI model. Standard activations like SiLU are great, but I wanted to see if a "High-Gravity" start would do anything interesting.

2. Tinkering with Nitro-SwiGLU (CyLU)

I put together something I called CyLU. It's basically just a 3-segment approximation of that cycloidal curve. I used it for the first 10% of training tokens just to see what would happen.

I used these numbers for the setup (mostly just trial and error):

• Weights (w_p): [1.5, -0.7, 0.4]
• Biases (b_p): [0.0, -1.2, -2.5]

3. The "Morphing" Part

One thing I noticed is that if I kept the "Nitro" phase going too long, the optimizer would get a bit overwhelmed (I called it "burning out"). So, I made a simple scheduler that slowly transitions it back to normal SiLU/SwiGLU by the time 15% of the training is done. It's like a plane taking off fast and then just gliding.

4. What I Saw (Take this with a grain of salt)

In the few tests I ran, I noticed:

• The attention heads seemed to line up a little bit faster (maybe around 2.4%?).
• The loss was a tiny bit lower at the end (about 0.26%), but honestly, that could just be random luck with the seed.
• It's definitely heavier on the computer—it takes about 12-18% more work during that first phase.

5. Thoughts

Look, I'm fully aware this isn't some groundbreaking discovery. You could probably get the same result by just bumping up the learning rate a bit. Plus, it's a bit of a pain to optimize the code for actual hardware since everything is built for standard SiLU.

But I did it because I like the math behind it. It's just a curious way to start a model, and I enjoyed seeing the physics-inspired curve in the code.

import torch
import torch.nn as nn
import torch.nn.functional as F

class NitroSwiGLU(nn.Module):
    def __init__(self, d_in, d_hidden, total_steps):
        super().__init__()
        self.w_gate = nn.Linear(d_in, d_hidden, bias=False)
        self.w_val = nn.Linear(d_in, d_hidden, bias=False)
        self.total_steps = total_steps
        
        # These are just the constants I was playing with 
        # to mimic that Brachistochrone drop.
        self.register_buffer("wp", torch.tensor([1.5, -0.7, 0.4]))
        self.register_buffer("bp", torch.tensor([0.0, -1.2, -2.5]))

    def nitro_activation(self, x):
        # A rough 3-segment cycloidal approximation. 
        # It's a bit heavy on the trig, but it gets that "High-Gravity" feel.
        return (self.wp[0] * torch.sin(x + self.bp[0]) + 
                self.wp[1] * torch.sin(2 * x + self.bp[1]) + 
                self.wp[2] * torch.sin(3 * x + self.bp[2]))

    def forward(self, x, current_step):
        g = self.w_gate(x)
        v = self.w_val(x)
        
        # Calculate our training progress
        progress = current_step / self.total_steps
        
        # The Morphing Logic: 
        # 0.0 means "Full Nitro", 1.0 means "Full SiLU"
        if progress < 0.10:
            alpha = 0.0
        elif progress < 0.15:
            # The "Glide" transition phase
            alpha = (progress - 0.10) / 0.05
        else:
            alpha = 1.0
            
        # Get both versions and blend them
        nitro_out = self.nitro_activation(g) * v
        silu_out = F.silu(g) * v
        
        return (1 - alpha) * nitro_out + alpha * silu_out