Nemotron’s Context Window and Function-Calling Support

“What is Nemotron’s context window” has no answer, because Nemotron is a name Nvidia has attached to at least three architecturally unrelated model lines with documented context lengths differing by a factor of thirty-two. The question you can answer is which line your checkpoint belongs to, and this page is about telling them apart.

Nemotron is a programme, not a model

Nvidia uses Nemotron for its own model work broadly, and the releases fall into three groups that share almost nothing but branding.

• From-scratch dense models. Nemotron-4, released in 2024, including a 340B model published together with a reward model and a large synthetic-data-generation pipeline. Trained by Nvidia from the ground up, under the Nvidia Open Model License.
• Llama derivatives. The Llama-Nemotron line takes a Meta Llama checkpoint and applies Nvidia’s post-training — preference tuning, distillation, and in the later ones a neural architecture search that prunes the base model. Names in this line carry Llama in them, as in Llama-3.1-Nemotron-70B-Instruct or the Nano, Super and Ultra sizes. These inherit Llama’s architecture and, importantly, Llama’s licence conditions on top of Nvidia’s.
• Nemotron-H. The hybrid line, in which the majority of self-attention layers are replaced with Mamba-2 state-space layers, for the same throughput and memory reasons described for Jamba.

If somebody hands you “a Nemotron model”, the first useful question is which of those three it is, because everything below branches on it.

The Llama-derived line has one property that trips people up. Nvidia does not only post-train these checkpoints; in the Super and Ultra variants it applies architecture search that removes and restructures layers, so the result is not a Llama with different weights but a model with a different layer count and different attention shapes. It loads through Llama-compatible tooling and it is not, structurally, the model it started as. Consequences: throughput and memory figures for the base Llama do not transfer, and anything you built that indexes into particular layers — an adapter, a probe, a layer-skipping serving trick — will not line up.

Context length by line

The dominant fact is that the Llama-derived models inherit their base model’s window. Llama 3.1 and later carry a documented 128K-token context, so the Nemotron models built on them do too, and this is stated on the corresponding cards on Nvidia’s Hugging Face organisation. The from-scratch Nemotron-4 340B models were documented with a 4,096-token window, which is far shorter and is the figure most likely to surprise somebody who assumed the largest model had the largest window.

Nemotron-4 340B (2024)              4,096 tokens
Llama-3.1-Nemotron-70B-Instruct   131,072 tokens (inherited)
Llama-3.3-Nemotron Super / Nano   131,072 tokens (inherited)
Nemotron-H                        long-context; check the card

The Nemotron-4 340B figure deserves the attention, because a 4,096-token window on a 340-billion-parameter model looks like a typo and is not. That model was built for a specific job — generating synthetic training data at scale, with a companion reward model for filtering it — and synthetic data generation does not need a long context. It needs throughput and it needs the output to be good. Nvidia released the generation pipeline alongside the weights, and the window is sized for that pipeline rather than for a chat product. Judging it as a general-purpose assistant and finding the context short is judging it against a use it was not shaped for.

How tool calls are expressed

There are two distinct answers and they correspond to two distinct things you might mean by “function calling support”.

Through Nvidia’s hosted endpoints and NIM

Nvidia serves these models behind an OpenAI-compatible chat completions interface. That means tools with a JSON Schema under function.parameters, tool_choice, and responses carrying tool_calls with an id, a name, and arguments as a JSON string. The wire format is the one you already have code for:

curl https://integrate.api.nvidia.com/v1/chat/completions \
  -H "Authorization: Bearer $NVIDIA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/llama-3.1-nemotron-70b-instruct",
    "messages": [
      {"role": "user", "content": "What is the weather in Abu Dhabi?"}
    ],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Current weather for a city",
        "parameters": {
          "type": "object",
          "properties": {"city": {"type": "string"}},
          "required": ["city"]
        }
      }
    }],
    "tool_choice": "auto"
  }'

Support for tools is per model on that endpoint, not universal across the catalogue. A model that does not advertise it will typically ignore the field rather than error, which is the worst of the available behaviours because it looks like the model simply declined to call anything.

Running the weights yourself

Here the OpenAI shape does not exist. What exists is the chat template in tokenizer_config.json, and the model emits tool calls as text in whatever form that template trained it to. For the Llama-derived models this is Llama’s own convention — see Llama 3’s function-calling format and how tool results are fed back. Your serving layer, not the model, is what turns that text into a structured tool_calls field, and different servers parse it differently. If tool calling “works in the API and not locally”, this is nearly always why.

The practical test is cheap and worth running before you plan around tool use: send a request that should obviously trigger a call, and look at the raw completion rather than at your client’s parsed object. If the model emitted a recognisable call in text and your stack returned it as ordinary content, the model supports tools and your parser does not. If the model answered the question in prose without attempting a call, the checkpoint was not trained for it and no amount of parser work will help.

The reasoning toggle in the system prompt

The later Llama-Nemotron models expose reasoning as a system-prompt switch rather than as an API parameter: a system message of detailed thinking on or detailed thinking off selects whether the model produces an extended reasoning trace before its answer. This is unusual and it has two consequences worth planning for.

• If you overwrite the system message with your own instructions and drop the toggle, you change the model’s behaviour in a way that looks like a quality regression rather than a configuration change.
• With the toggle on, the reasoning text arrives in the ordinary content stream and counts against max_tokens. A budget sized for the answer alone will truncate mid-reasoning and return nothing usable. The same failure appears with DeepSeek R1’s think tags and Qwen 3’s thinking mode, and the parsing you write for one is close to what the others need.

The wider lesson is that reasoning controls have not converged on an API shape. Anthropic exposes a token budget as a request field, Google exposes a thinking budget, OpenAI exposes an effort level, Qwen exposes a boolean, and Nvidia exposes a phrase you put in the system prompt. These are five encodings of nearly the same decision, and none of them is portable. If you route across models, this is one of the parameters that has to be translated rather than passed through, and the system-prompt encoding is the awkward one because it collides with the slot your own instructions occupy.

Reading the right model card

1. Identify the line from the name. A name containing Llama is a derivative and inherits Meta’s licence terms in addition to Nvidia’s; a name containing H is a hybrid; Nemotron-4 is from-scratch.
2. Take context length from config.json — max_position_embeddings — on the exact repository, and then check what your deployment actually configures, which may be lower.
3. For tool calling, check whether you are going through an OpenAI-compatible endpoint or through raw weights. Those are different questions with different answers.
4. If self-hosting, read the chat_template in tokenizer_config.json. It is the ground truth for how tools, tool results and system messages must be laid out.