English
Chinese (Simplified)
French
German
Italian
Japanese
Russian
Spanish
Vietnamese
Question regarding AI model "Temperature"
-
In the documenation, the temperature scale seems to be from 0 to 2, but, when reviewing documentation on various LLMs and suggestions, you often encounter a scale of 0-1 for temperature.
So my question is, would a "1" in SecurityGateway equate to a .5 on the LLM scale? Also does this setting get passed via the API call to the model so it overridges whatever the model's defaul setting is?
Just trying to get a feel for how this works.
-
Actually upon further review it doesn't appear that this setting does anything or "sticks," I change it to 0 or 2, save, and re-open and it's back to 1.
-
Actually upon further review it doesn't appear that this setting does anything or "sticks," I change it to 0 or 2, save, and re-open and it's back to 1.
I was able to reproduce the issue and have submitted a bug on it.
So my question is, would a "1" in SecurityGateway equate to a .5 on the LLM scale? Also does this setting get passed via the API call to the model so it overridges whatever the model's defaul setting is?
No. When the above issue is fixed, If you set it to 1, then it will pass 1.
-
Do you know if the "temperature" setting passed via the API overrides whatever is set on the model default? If so, wouldn't you also need to adjust the top P sampling as well?
-
I'm not sure what you mean.
If you configure the model in SG to use a temperature of .75, then when a request is sent to that model, the temperature will be set to use .75
If you are using a model that allow the temperature to be 0-1 and you set it to 1.5, I would assume the model will return an error or just use the default value, but that is entirely dependent upon the model.
There are models that support a temperature of 0-2, which is why the highest allowed value is 2.
-
The backstory here is that I noticed I was having an issue where, let's say the same e-mail gets sent to 3 different recipients; exactly the same e-mail, the AI classification would be "legitimate" for one, "spam" for the next, and "phishing" for the third. In trying to figure this out I learned that this was likely caused by too high of a temperature. I also learned that there is another setting called Top P Sampling (I think it can also be called "nucleus sampling") which is also closely related and by having a temperature set low and a Top P Sampling set high you could also induce "strange" and inconsistent results.
I think, at least in LM Studio, that you can override whatever comes in via the API becuase I've changed both and the inconsistent results stopped.
-
According to the Open AI documentation you should adjust either temperature or top p, not both. This likely varies by model/provider.
An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens comprising the top 10% probability mass are considered.We generally recommend altering this or
temperaturebut not both.SecurityGateway does not currently allow you to configure the TOP P value. The API's default value for TOP P will be used.
I'll add a wish list item to allow TOP P to be configured.
-
The issue with temperature not being saved has been fixed in SecurityGateway 11.0.3a which is currently in beta.
