Streaming Feature

content/en/docs/genai/v2/reference-guide/agent-commons.md

@@ -115,6 +115,10 @@ Users can create two types of agents:
 
  {{< figure src="/attachments/genai/agentcommons/agent_builder_task.png" alt="" >}}
 
+#### Control model behavior {#model-settings}
+
+By clicking the icon next to the selected model, you can control model behavior by configuring the temperature, top P, and maximum number of tokens on the version. If the selected model and connector support it (currently only Mendix Cloud GenAI), you can also enable backend streaming to use the model's streaming API, and frontend streaming to send generated chunks to the user interface as they become available (see [ConversationalUI](/agents/agents-kit-2/reference-guide/conversational-ui/#streaming) documentation for more). UI streaming is supported only for chat agents and for a chat interface from the Conversational UI module. For task agents only backend streaming is possible because the UI is part of the custom implementation.
+
 #### Defining Context Entity {#define-context-entity}
 
 If your agent's prompt includes variables, your app must define an entity with attributes that match the variable names. An object of this entity serves as the context object, which holds the context data that will be passed when the **call agent** operation is triggered. For more details, see the [Use the agent in the app logic](#app-logic) section below.

content/en/docs/genai/v2/reference-guide/agent-editor.md

@@ -225,6 +225,10 @@ When these conditions are met, you can use the test functionality to validate pr
 
 If a call fails during testing, a generic error message is shown in the Agent Editor UI. Detailed error information is available in the running app console in Studio Pro (the **Console** pane), similar to errors you would inspect while testing the app itself.
 
+{{% alert color="info" %}}
+While it is possible to configure streaming as well as user permissions for tools and knowledge bases, these settings are currently not reflected in the playground in Studio Pro, but will be considered at runtime when calling an agent defined in Studio Pro.
+{{% /alert %}}
+
 ### Including the Agent in the App Logic {#call-agent}
 
 Include an agent in the app logic by calling it from a microflow. Agent Editor provides **Call Agent** toolbox actions in the **Agent Editor** category:
@@ -265,6 +269,7 @@ Use version control to view and restore previous agent versions. This lets you i
 * MCP tool support is limited to whole-server integration. Selecting individual tools from a consumed MCP service to be added to an agent is not yet supported. That also means that the tool choice option `Tool` can only refer to a microflow tool currently.
 * If a document referenced by an Agent document is excluded, Studio Pro shows a consistency error. These consistency errors may not be resolved automatically when you include the excluded document again. Resolve this by synchronizing the app directory (<kbd>F4</kbd>) or by making a small change in any agent-related document (for example, add a character to a system prompt and remove it again).
 * The extension creates a `/agenteditor` log folder in the app directory. This folder is not excluded from version control automatically when you include the module from Marketplace. Add this folder to `.gitignore` manually, as described in the [First-time setup](#setup) section.
+* While it is possible to configure streaming as well as user permissions for tools and knowledge bases, these settings are currently not reflected in the playground in Studio Pro, but will be considered at runtime when calling an agent defined in Studio Pro.
 
 ## Troubleshooting {#troubleshooting}
 

content/en/docs/genai/v2/reference-guide/conversational-ui.md

@@ -220,6 +220,15 @@ This changes how action microflows are used, because they are called each time a
 
 If no [user-visibility](/agents/agents-kit-2/reference-guide/commons/#enum-useraccessapproval) is configured for tools and you want to avoid storing tool messages, you can change the Boolean `SaveToolCallHistory` to *false* on the [Request](/agents/agents-kit-2/reference-guide/commons/#request). Note that [knowledge base retrievals](/agents/agents-kit-2/reference-guide/commons/#add-knowledge-base-to-request) are set to `HiddenForUser` by default.
 
+### Streaming {#streaming}
+
+Conversational UI chats can also be streamed to the UI, which expands the assistant response as the LLM generates new content. To use streaming, use [AgentCommons](/agents/agents-kit-2/reference-guide/agent-commons/) to configure a chat agent. In the [model settings](/agents/agents-kit-2/reference-guide/agent-commons/#model-settings), enable the applicable streaming options, and use that agent when calling the `New Chat for Agent` action and in your action microflow. Not all models and connectors support streaming. Currently, streaming is supported only for Mendix Cloud GenAI.
+
+Note that the following limitations apply for streaming because the microflows are called in isolated transactions:
+* Tool microflows can only read objects from the database and changes to objects are only stored within the microflow if not committed. Additionally, user interaction is limited, for example, opening a page or showing a message to the user.
+* This similarly applies for action microflows.
+A workaround to these limitations would be to make changes to persistent objects, for example the ChatContext which is available in both the action microflow and on the page.
+
 ### Human in the Loop {#human-in-the-loop}
 
 When using the [Function Calling](/agents/function-calling/) pattern by adding tools to the request, you can control when those tools get executed and if they are visible to the user by setting [user access approval](/agents/agents-kit-2/reference-guide/commons/#enum-useraccessapproval) per tool. [Human in the loop](/agents/glossary/#human-in-the-loop) describes a pattern where the AI can perform powerful tasks, but still requires humans to take certain decisions and oversee the agent's behavior. When using the ConversationalUI module, its basic action microflow pattern to execute requests with history and UI snippets to display the chat, human in the loop works out of the box. Note that action microflows are called until there is a final assistant's response as described in [Using Tool or Knowledge Base Calling](#action-microflow-tool-calling) section above, even if all tools are executed without user interaction.

content/en/docs/genai/v2/reference-guide/genai-commons.md

@@ -80,9 +80,10 @@ The `DeployedModel` entity replaces the capabilities that the `Connection` entit
 | `Model` | The model identifier of the LLM provider. |
 | `OutputModality` | The type of information the model returns. |
 | `Microflow` |  The microflow to execute for the specified model and modality. |
-| `SupportsSystemPrompt` | Enum to specify if the model supports system prompts. |
 | `SupportsConversationsWithHistory` | Enum to specify if the model supports conversation with history. |
 | `SupportsFunctionCalling` | Enum to specify if the model supports function calling. |
+| `SupportsStreaming` | Enum to specify if the model supports calling the streaming API. |
+| `SupportsSystemPrompt` | Enum to specify if the model supports system prompts. |
 | `IsActive` | Boolean to specify if the model is active/usable with the current authentication settings and user preference. |
 
 #### `ConsumedKnowledgeBase` {#consumed-knowledge-base}
@@ -233,6 +234,8 @@ The `Request` is an input object for the chat completions operations defined in
 | `ToolChoice` | Controls which (if any) tool is called by the model. For more information, see the [ENUM_ToolChoice](#enum-toolchoice) section containing a description of the possible values. |
 | `_AgentVersionId` | The `AgentVersionId`, set if the execution of the request was called from an Agent. |
 | `SaveToolCallHistory` | Indicates if the tool calls are stored for later continuation (must be implemented).  |
+| `UseStreamingAPI` | Indicates if the LLM call should be called with streaming. |
+| `StreamingResponseWriterId` | Indicates if streaming data should be pushed to the UI. Must not be set manually! This is only set in backend if a ConversationalUI chat interface is being used with an agent that has streaming enabled. |
 
 #### `Message` {#message}