The maxtoken value of completionOptions in the configuration file

[who52023]

Validations

• I believe this is a way to improve. I'll try to join the Continue Discord for questions
• I'm not able to find an open issue that requests the same enhancement

Problem

The maxtoken value of completionOptions in the configuration file is applied to the dialog at the same time, resulting in incomplete dialog output

Solution

It is recommended to separate the configuration, code completion and the output token of the session.

[who52023]

in OpenAI.ts file

protected _convertArgs(options: any, messages: ChatMessage[]) {
    const url = new URL(this.apiBase!);
    const finalOptions: any = {
      messages: messages.map(this._convertMessage),
      model: this._convertModelName(options.model),
      max_tokens: options.maxTokens,
      temperature: options.temperature,
      top_p: options.topP,
      frequency_penalty: options.frequencyPenalty,
      presence_penalty: options.presencePenalty,
      stream: options.stream ?? true,
      stop:
        // Jan + Azure OpenAI don't truncate and will throw an error
        this.maxStopWords !== undefined
          ? options.stop?.slice(0, this.maxStopWords)
          : url.host === "api.deepseek.com"
            ? options.stop?.slice(0, 16)
            : url.port === "1337" ||
                url.host === "api.openai.com" ||
                url.host === "api.groq.com" ||
                this.apiType === "azure"
              ? options.stop?.slice(0, 4)
              : options.stop,
    };

    // OpenAI o1-preview and o1-mini:
    if (this.isO1Model(options.model)) {
      // a) use max_completion_tokens instead of max_tokens
      finalOptions.max_completion_tokens = options.maxTokens;
      finalOptions.max_tokens = undefined;

      // b) don't support streaming currently
      finalOptions.stream = false;

      // c) don't support system message
      finalOptions.messages = finalOptions.messages?.filter(
        (message: any) => message?.role !== "system",
      );
    }

    return finalOptions;
  }

propose modify value ： max_tokens: options.maxTokens

[tomasz-stefaniak]

@who52023 could you share your config.json?

My guess is that you might be able to achieve this by setting per-model token limits: https://docs.continue.dev/reference#models
https://docs.continue.dev/reference#completionoptions