(improvement) remove separate shell and support start llmParser service (#83)

* (improvement)(chat) remove group by dimension in join case

* (improvement)(chat) add FunctionAliasReplaceVisitor in dsl

---------

CHANGELOG.md

@@ -4,6 +4,25 @@
 - "Breaking Changes" describes any changes that may break existing functionality or cause
   compatibility issues with previous versions.
   
+  ## SuperSonic [0.7.4] - 2023-09-10
+  
+  ### Added
+  - add llm parser config
+  - add datasource agg_time option
+  - add function name adaptor in clickhouse
+  - add dimension and metric show in dsl
+  
+  
+  ### Updated
+  - update user guide doc
+  - update query building of plugin in default model
+  - update some core API constructs to keep naming consistency
+  - update ConfigureDemo config
+  - update the association mechanism so that invisible dimensions and metrics will no longer be associated
+  
+  ### Fixed
+  - fix hasAggregateFunction logic in SqlParserSelectHelper
+
 ## SuperSonic [0.7.3] - 2023-08-29
 
 ### Added

README.md

@@ -1,4 +1,4 @@
-English | [中文](README_CN.md)
+[中文介绍](README_CN.md) | [文档中心](https://github.com/tencentmusic/supersonic/wiki)
 
 # SuperSonic (超音数)
 
@@ -21,7 +21,7 @@ With these ideas in mind, we develop SuperSonic as a practical reference impleme
 
 - Built-in CUI(Chat User Interface) for *business users* to enter data queries
 - Built-in GUI(Graphical User Interface) for *analytics engineers* to build semantic models
-- Built-in GUI for *system administrators* to manage chat plugins and agents
+- Built-in GUI for *system administrators* to manage chat agents and third-party plugins
 - Support input auto-completion as well as query recommendation
 - Support multi-turn conversation and history context management 
 - Support four-level permission control: domain-level, model-level, column-level and row-level
@@ -49,27 +49,15 @@ The high-level architecture and main process flow is as follows:
 SuperSonic comes with sample semantic models as well as chat conversations that can be used as a starting point. Please follow the steps: 
 
 - Download the latest prebuilt binary from the [release page](https://github.com/tencentmusic/supersonic/releases)
-- Run script "bin/start-standalone.sh" to start a standalone server
+- Run script "bin/start-standalone.sh" to start services (one java process and one python process)
 - Visit http://localhost:9080 in the browser to start exploration
 
-## How to Build
+## Build and Delopment
 
-SuperSonic can be deployed in two modes: standalone (for a quick demo) and distributed (for production use). 
+Please refer to project [wiki](https://github.com/tencentmusic/supersonic/wiki). 
-
-### Build for Standalone Mode 
- 
-Pull the source code and run script "assembly/bin/build-standalone.sh" to build a single packages.
-
-### Build for Distributed Mode 
-
-Pull the source code and run scripts "assembly/bin/build-chat.sh" and "assembly/bin/build-semantic.sh" separately to build packages.
-
-### Build for Local Development
-
-Pull the source code and run script "assembly/bin/build-ide.sh" and run bootstrap class "StandaloneLauncher" in the IDE.
 
 ## WeChat Contact
 
 Please join the chat group to suggest feedbacks or ideas:
 
-<img src="./docs/images/wechat_contact.jpeg" height="30%" width="30%" align="center"/> 
+<img src="./docs/images/wechat_contact.jpeg" height="40%" width="40%" align="center"/> 

README_CN.md

@@ -19,7 +19,7 @@
 
 - 内置对话界面以便*业务用户*输入数据查询。
 - 内置图形界面以便*分析工程师*构建语义模型。
-- 内置图形界面以便*系统管理员*管理问答插件和助理。
+- 内置图形界面以便*系统管理员*管理第三方插件和对话助理。
 - 支持文本输入的联想和查询问题的推荐。
 - 支持多轮对话，根据语境自动切换上下文。
 - 支持四级权限控制：主题域级、模型级、列级、行级。
@@ -47,27 +47,15 @@
 超音数自带样例的语义模型和问答对话，只需以下三步即可快速体验：
 
 - 从[release page](https://github.com/tencentmusic/supersonic/releases)下载预先构建好的发行包
-- 运行 "bin/start-standalone.sh"启动服务
+- 运行 "bin/start-standalone.sh"启动服务（一个Java进程和一个Python进程）
 - 在浏览器访问http://localhost:9080 开启探索
 
-## 如何构建
+## 如何构建和部署
 
-超音数可以运行在两个模式：standalone（一般用于快速演示）和distributed（一般用于生产环境）。
+请参考项目[wiki](https://github.com/tencentmusic/supersonic/wiki)。
-
-### Standalone模式构建
-
-下载源码包，运行脚本"assembly/bin/build-standalone.sh"，将所有服务一起编译打包。
-
-### Distributed模式构建
-
-下载源码包，分别运行脚本"assembly/bin/build-chat.sh"、"assembly/bin/build-semantic.sh"，为问答层服务和语义层服务编译打包。
-
-### 本地开发构建
-
-下载源码包，运行脚本"assembly/bin/build-ide.sh"，然后在本地IDE运行启动类"StandaloneLauncher"。
 
 ## 微信联系方式
 
 欢迎加入微信群反馈建议：
 
-<img src="./docs/images/wechat_contact.jpeg" height="30%" width="30%" align="center"/> 
+<img src="./docs/images/wechat_contact.jpeg" height="40%" width="40%" align="center"/> 

assembly/bin/build-chat.sh

@@ -1,12 +0,0 @@
-#!/usr/bin/env bash
-
-sbinDir=$(cd "$(dirname "$0")"; pwd)
-baseDir=$(cd "$sbinDir/.." && pwd -P)
-runtimeDir=$baseDir/runtime
-buildDir=$baseDir/build
-
-cd $baseDir
-
-#1. move package to build
-cp $baseDir/../launchers/chat/target/*.tar.gz ${buildDir}/supersonic-chat.tar.gz
-

assembly/bin/build-ide.sh

@@ -1,13 +0,0 @@
-#!/usr/bin/env bash
-
-sbinDir=$(cd "$(dirname "$0")"; pwd)
-baseDir=$(cd "$sbinDir/.." && pwd -P)
-buildDir=$baseDir/build
-
-cd $baseDir/bin
-sh build-standalone.sh
-
-cd $buildDir
-tar xvf supersonic-webapp.tar.gz
-mv supersonic-webapp webapp
-mv webapp ../../launchers/standalone/target/classes

assembly/bin/build-semantic.sh

@@ -1,23 +0,0 @@
-#!/usr/bin/env bash
-
-sbinDir=$(cd "$(dirname "$0")"; pwd)
-baseDir=$(cd "$sbinDir/.." && pwd -P)
-runtimeDir=$baseDir/runtime
-buildDir=$baseDir/build
-
-cd $baseDir
-
-#1. build semantic service
-rm -fr ${buildDir}/*.tar.gz
-rm -fr dist
-
-mvn -f $baseDir/../ clean package -DskipTests
-
-#2. move package to build
-cp $baseDir/../launchers/semantic/target/*.tar.gz ${buildDir}/supersonic-semantic.tar.gz
-
-#3. build webapp
-chmod +x $baseDir/../webapp/start-fe-prod.sh
-cd ../webapp
-sh ./start-fe-prod.sh
-cp -fr  ./supersonic-webapp.tar.gz ${buildDir}/

assembly/bin/start-chat.sh

@@ -1,35 +0,0 @@
-#!/usr/bin/env bash
-
-sbinDir=$(cd "$(dirname "$0")"; pwd)
-baseDir=$(cd "$sbinDir/.." && pwd -P)
-runtimeDir=$baseDir/../runtime
-buildDir=$baseDir/build
-
-cd $baseDir
-
-#2. package lib
-
-tar -zxvf ${buildDir}/supersonic-chat.tar.gz  -C ${runtimeDir}
-
-mv ${runtimeDir}/launchers-chat-* ${runtimeDir}/supersonic-chat
-
-tar -zxvf ${buildDir}/supersonic-webapp.tar.gz  -C ${buildDir}
-
-mkdir -p ${runtimeDir}/supersonic-chat/webapp
-
-cp -fr  ${buildDir}/supersonic-webapp/* ${runtimeDir}/supersonic-chat/webapp
-
-rm -fr  ${buildDir}/supersonic-webapp
-
-json=$(cat ${runtimeDir}/supersonic-chat/webapp/supersonic.config.json)
-json=$(echo $json | jq '.env="chat"')
-echo $json > ${runtimeDir}/supersonic-chat/webapp/supersonic.config.json
-
-#3. start service
-#3.1 start chat service
-echo ${runtimeDir}
-sh ${runtimeDir}/supersonic-chat/bin/service.sh restart
-
-#3.2 start llm service
-sh ${runtimeDir}/supersonic-chat/llm/bin/service.sh restart
-

assembly/bin/start-semantic.sh

@@ -1,36 +0,0 @@
-#!/usr/bin/env bash
-
-sbinDir=$(cd "$(dirname "$0")"; pwd)
-baseDir=$(cd "$sbinDir/.." && pwd -P)
-runtimeDir=$baseDir/../runtime
-buildDir=$baseDir/build
-
-cd $baseDir
-
-#1. clear file
-mkdir -p ${runtimeDir}
-rm -fr ${runtimeDir}/*
-
-#2. package lib
-
-tar -zxvf ${buildDir}/supersonic-semantic.tar.gz  -C ${runtimeDir}
-
-mv ${runtimeDir}/launchers-semantic-* ${runtimeDir}/supersonic-semantic
-
-tar -zxvf ${buildDir}/supersonic-webapp.tar.gz  -C ${buildDir}
-
-mkdir -p ${runtimeDir}/supersonic-semantic/webapp
-
-cp -fr  ${buildDir}/supersonic-webapp/* ${runtimeDir}/supersonic-semantic/webapp
-
-rm -fr  ${buildDir}/supersonic-webapp
-
-
-json=$(cat ${runtimeDir}/supersonic-semantic/webapp/supersonic.config.json)
-json=$(echo $json | jq '.env="semantic"')
-echo $json > ${runtimeDir}/supersonic-semantic/webapp/supersonic.config.json
-
-#3. start service
-sh ${runtimeDir}/supersonic-semantic/bin/service.sh restart
-
-

assembly/bin/start-standalone.sh

@@ -1,32 +0,0 @@
-#!/usr/bin/env bash
-
-sbinDir=$(cd "$(dirname "$0")"; pwd)
-baseDir=$(cd "$sbinDir/.." && pwd -P)
-runtimeDir=$baseDir/../runtime
-buildDir=$baseDir/build
-
-cd $baseDir
-
-#1. clear file
-mkdir -p ${runtimeDir}
-rm -fr ${runtimeDir}/*
-
-#2. package lib
-
-tar -zxvf ${buildDir}/supersonic.tar.gz  -C ${runtimeDir}
-
-mv ${runtimeDir}/launchers-standalone-* ${runtimeDir}/supersonic-standalone
-
-tar -zxvf ${buildDir}/supersonic-webapp.tar.gz  -C ${buildDir}
-
-mkdir -p ${runtimeDir}/supersonic-standalone/webapp
-
-cp -fr  ${buildDir}/supersonic-webapp/* ${runtimeDir}/supersonic-standalone/webapp
-
-rm -fr  ${buildDir}/supersonic-webapp
-
-#3. start service
-#start standalone service
-sh ${runtimeDir}/supersonic-standalone/bin/service.sh restart
-#start llm service
-sh ${runtimeDir}/supersonic-standalone/llm/bin/service.sh restart

assembly/bin/supersonic-build.bat

@@ -0,0 +1,30 @@
+@echo off
+setlocal
+
+set "sbinDir=%~dp0"
+set "baseDir=%~dp0.."
+set "buildDir=%baseDir%\build"
+
+
+rem 1. build semantic chat service
+del /q "%buildDir%\*.tar.gz" 2>NUL
+
+call mvn -f "%baseDir%\..\pom.xml" clean package -DskipTests
+
+rem 2. move package to build
+echo f|xcopy "%baseDir%\..\launchers\standalone\target\*.tar.gz" "%buildDir%\supersonic-standalone.tar.gz"
+echo f|xcopy "%baseDir%\..\launchers\semantic\target\*.tar.gz" "%buildDir%\supersonic-semantic.tar.gz"
+echo f|xcopy "%baseDir%\..\launchers\chat\target\*.tar.gz" "%buildDir%\supersonic-chat.tar.gz"
+
+rem 3. build webapp
+cd "%baseDir%\..\webapp"
+call start-fe-prod.bat
+copy /y "%baseDir%\..\webapp\supersonic-webapp.tar.gz" "%buildDir%\"
+
+
+cd "%buildDir%"
+tar -zxvf supersonic-webapp.tar.gz
+move supersonic-webapp webapp
+move webapp ..\..\launchers\standalone\target\classes
+
+endlocal

assembly/bin/supersonic-build.sh

@@ -21,3 +21,9 @@ chmod +x $baseDir/../webapp/start-fe-prod.sh
 cd ../webapp
 sh ./start-fe-prod.sh
 cp -fr  ./supersonic-webapp.tar.gz ${buildDir}/
+
+
+cd $buildDir
+tar xvf supersonic-webapp.tar.gz
+mv supersonic-webapp webapp
+mv webapp ../../launchers/standalone/target/classes

assembly/bin/supersonic-daemon.bat

@@ -0,0 +1,120 @@
+@echo off
+setlocal
+
+set "sbinDir=%~dp0"
+set "baseDir=%~dp0.."
+set "runtimeDir=%baseDir%\..\runtime"
+set "buildDir=%baseDir%\build"
+
+set "command=%~1"
+set "module=%~2"
+
+set "APP_NAME=standalone-service"
+set "MAIN_CLASS=com.tencent.supersonic.StandaloneLauncher"
+
+set "python_path=python"
+set "pip_path=pip3.9"
+set "llm_host=127.0.0.1"
+set "llm_port=9092"
+set "start_name=api_service"
+
+set "llm_path=%baseDir%\..\chat\core\src\main\python"
+
+if "%module%"=="" (
+   set "module=standalone"
+) else if "%module%"=="semantic" (
+   set "APP_NAME=semantic-service"
+   set "MAIN_CLASS=com.tencent.supersonic.SemanticLauncher"
+) else if "%module%"=="chat" (
+   set "APP_NAME=chat-service"
+   set "MAIN_CLASS=com.tencent.supersonic.ChatLauncher"
+)
+
+if "%command%"=="" (
+   set "command=restart"
+)
+
+set "libDir=%runtimeDir%\supersonic-%module%\lib"
+set "confDir=%runtimeDir%\supersonic-%module%\conf"
+set "webDir=%runtimeDir%\supersonic-%module%\webapp"
+set "CLASSPATH=%confDir%;%webDir%;%libDir%\*"
+set "java-command=-Dfile.encoding=UTF-8 -Duser.language=Zh -Duser.region=CN -Duser.timezone=GMT+08 -Xms1024m -Xmx2048m -cp %CLASSPATH% %MAIN_CLASS%"
+
+
+if "%command%"=="stop" (
+    call:STOP
+    goto :EOF
+)
+
+if "%command%"=="restart" (
+     call:STOP
+)
+
+::1. clear file
+rd /s /q "%runtimeDir%"
+mkdir "%runtimeDir%"
+
+if "%module%"=="llmparser" (
+   tar -zxvf "%buildDir%\supersonic-standalone.tar.gz" -C "%runtimeDir%"
+   for /d %%f in ("%runtimeDir%\launchers-standalone-*") do (
+       move "%%f" "%runtimeDir%\supersonic-standalone"
+   )
+   cd "%runtimeDir%"
+   "%pip_path%" install -r "%llm_path%\requirements.txt"
+   "%python_path%" -c "import langchain,fastapi,chromadb,tiktoken,uvicorn" >nul 2>&1
+   cd "%runtimeDir%\supersonic-standalone\llm\llm"
+   start "" /B uvicorn %start_name%:app --port %llm_port% --host %llm_host%  > "%runtimeDir%\supersonic-standalone\llm\llm.log" 2>&1
+   echo "llm service started, see logs/error with logs/error command"
+   goto :EOF
+)
+
+tar -zxvf "%buildDir%\supersonic-%module%.tar.gz" -C "%runtimeDir%"
+for /d %%f in ("%runtimeDir%\launchers-%module%-*") do (
+    move "%%f" "%runtimeDir%\supersonic-%module%"
+)
+
+if not exist "%runtimeDir%\supersonic-%module%\logs" mkdir "%runtimeDir%\supersonic-%module%\logs"
+
+tar -zxvf "%buildDir%\supersonic-webapp.tar.gz" -C "%buildDir%"
+if not exist "%runtimeDir%\supersonic-%module%\webapp" mkdir "%runtimeDir%\supersonic-%module%\webapp"
+xcopy /s /e /h /y "%buildDir%\supersonic-webapp\*" "%runtimeDir%\supersonic-%module%\webapp"
+if not exist "%runtimeDir%\supersonic-%module%\conf\webapp" mkdir "%runtimeDir%\supersonic-%module%\conf\webapp"
+xcopy /s /e /h /y "%runtimeDir%\supersonic-%module%\webapp\*" "%runtimeDir%\supersonic-%module%\conf\webapp"
+rd /s /q "%buildDir%\supersonic-webapp"
+
+::3. start service
+::start standalone service
+if "%command%"=="start" (
+    call:START
+    goto :EOF
+)
+
+if "%command%"=="restart" (
+    call:START
+    goto :EOF
+)
+
+:START
+    if "%module%"=="standalone" (
+          cd "%runtimeDir%"
+          "%pip_path%" install -r "%llm_path%\requirements.txt"
+          "%python_path%" -c "import langchain,fastapi,chromadb,tiktoken,uvicorn" >nul 2>&1
+          cd "%runtimeDir%\supersonic-standalone\llm\llm"
+          start "" /B uvicorn %start_name%:app --port %llm_port% --host %llm_host%  > "%runtimeDir%\supersonic-standalone\llm\llm.log" 2>&1
+          echo "llm service started, see logs/error with logs/error command"
+    )
+    start "supersonic" /B java %java-command%>"%runtimeDir%\supersonic-%module%\logs\info-%module%.log" 2>&1
+    echo "%module% service started, see logs/error with logs/error command"
+    goto :EOF
+
+
+:STOP
+    for /f "tokens=2" %%i in ('tasklist ^| findstr /i "python"') do (
+                     taskkill /PID %%i /F
+                     echo "llm Process (PID = %%i) is killed."
+    )
+    for /f "tokens=2" %%i in ('tasklist ^| findstr /i "java"') do (
+                taskkill /PID %%i /F
+                echo "%module% Process (PID = %%i) is killed."
+    )
+    goto :EOF

assembly/bin/supersonic-daemon.sh

@@ -0,0 +1,98 @@
+#!/usr/bin/env bash
+
+sbinDir=$(cd "$(dirname "$0")"; pwd)
+baseDir=$(cd "$sbinDir/.." && pwd -P)
+runtimeDir=$baseDir/../runtime
+buildDir=$baseDir/build
+
+command=$1
+service=$2
+
+cd $baseDir
+if [[ "$service" == "semantic" || -z "$service"  ]] && [ "$command" != "stop"  ]; then
+   #1. clear file
+   mkdir -p ${runtimeDir}
+   rm -fr ${runtimeDir}/*
+
+   #2. package lib
+   tar -zxvf ${buildDir}/supersonic.tar.gz  -C ${runtimeDir}
+   mv ${runtimeDir}/launchers-standalone-* ${runtimeDir}/supersonic-standalone
+   tar -zxvf ${buildDir}/supersonic-webapp.tar.gz  -C ${buildDir}
+   mkdir -p ${runtimeDir}/supersonic-standalone/webapp
+   cp -fr  ${buildDir}/supersonic-webapp/* ${runtimeDir}/supersonic-standalone/webapp
+   rm -fr  ${buildDir}/supersonic-webapp
+fi
+if [[ "$service" == "semantic"  ]]; then
+   json=$(cat ${runtimeDir}/supersonic-semantic/webapp/supersonic.config.json)
+   json=$(echo $json | jq '.env="semantic"')
+   echo $json > ${runtimeDir}/supersonic-semantic/webapp/supersonic.config.json
+fi
+
+if [[ "$service" == "chat"  ]]; then
+   json=$(cat ${runtimeDir}/supersonic-chat/webapp/supersonic.config.json)
+   json=$(echo $json | jq '.env="chat"')
+   echo $json > ${runtimeDir}/supersonic-chat/webapp/supersonic.config.json
+fi
+echo $command
+echo $service
+case "$command" in
+  start)
+        if [[ "$service" == "semantic"  ]];then
+          echo -e "Starting semantic"
+          sh ${runtimeDir}/supersonic-semantic/bin/service.sh start
+        elif [[ "$service" == "chat"  ]];then
+          echo -e "Starting chat"
+          sh ${runtimeDir}/supersonic-chat/bin/service.sh start
+        elif [[ "$service" == "llmparser"  ]];then
+          echo -e "Starting LLM"
+          sh ${runtimeDir}/supersonic-standalone/llm/bin/service.sh  start
+        elif [[ -z "$service"  ]]; then
+          echo -e "Starting supersonic"
+          sh ${runtimeDir}/supersonic-standalone/bin/service.sh start
+          sh ${runtimeDir}/supersonic-standalone/llm/bin/service.sh start
+        else
+          echo "Use command {semantic|semantic||} to run."
+        fi
+        ;;
+  stop)
+        if [[ "$service" == "semantic"  ]];then
+          echo -e "Stopping semantic"
+          sh ${runtimeDir}/supersonic-semantic/bin/service.sh stop
+        elif [[ "$service" == "chat"  ]];then
+          echo -e "Stopping chat"
+          sh ${runtimeDir}/supersonic-chat/bin/service.sh stop
+        elif [[ "$service" == "llmparser"  ]];then
+          echo -e "Stopping LLM"
+          sh ${runtimeDir}/supersonic-standalone/llm/bin/service.sh  stop
+        elif [[ -z "$service"  ]]; then
+          echo -e "Stopping supersonic"
+          sh ${runtimeDir}/supersonic-standalone/bin/service.sh stop
+          sh ${runtimeDir}/supersonic-standalone/llm/bin/service.sh stop
+        else
+          echo "Use command {semantic|semantic||} to run."
+        fi
+        ;;
+  restart)
+        if [[ "$service" == "semantic"  ]];then
+          echo -e "Restarting semantic"
+          sh ${runtimeDir}/supersonic-semantic/bin/service.sh restart
+        elif [[ "$service" == "chat"  ]];then
+          echo -e "Restarting chat"
+          sh ${runtimeDir}/supersonic-chat/bin/service.sh restart
+        elif [[ "$service" == "llmparser"  ]];then
+          echo -e "Restarting LLM"
+          sh ${runtimeDir}/supersonic-standalone/llm/bin/service.sh  restart
+        elif [[ -z "$service"  ]]; then
+          echo -e "Restarting supersonic"
+          sh ${runtimeDir}/supersonic-standalone/bin/service.sh restart
+          sh ${runtimeDir}/supersonic-standalone/llm/bin/service.sh restart
+        else
+          echo "Use command {semantic|semantic||} to run."
+        fi
+        ;;
+  *)
+        echo "Use command {start|stop|status|restart} to run."
+        exit 1
+esac
+
+exit 0

auth/api/src/main/java/com/tencent/supersonic/auth/api/authentication/adaptor/UserAdaptor.java

@@ -3,6 +3,7 @@ package com.tencent.supersonic.auth.api.authentication.adaptor;
 import com.tencent.supersonic.auth.api.authentication.pojo.Organization;
 import com.tencent.supersonic.auth.api.authentication.pojo.User;
 import com.tencent.supersonic.auth.api.authentication.request.UserReq;
+
 import java.util.List;
 import java.util.Set;
 

auth/api/src/main/java/com/tencent/supersonic/auth/api/authentication/config/AuthenticationConfig.java

@@ -9,10 +9,12 @@ import org.springframework.context.annotation.Configuration;
 @Configuration
 public class AuthenticationConfig {
 
-
     @Value("${authentication.exclude.path:XXX}")
     private String excludePath;
 
+    @Value("${authentication.include.path:/api}")
+    private String includePath;
+
     @Value("${authentication.enable:false}")
     private boolean enabled;
 
@@ -22,5 +24,4 @@ public class AuthenticationConfig {
     @Value("${authentication.token.http.header.key:Authorization}")
     private String tokenHttpHeaderKey;
 
-
 }

auth/api/src/main/java/com/tencent/supersonic/auth/api/authentication/pojo/Organization.java

@@ -1,9 +1,10 @@
 package com.tencent.supersonic.auth.api.authentication.pojo;
 
 import com.google.common.collect.Lists;
-import java.util.List;
 import lombok.Data;
 
+import java.util.List;
+
 @Data
 public class Organization {
 

auth/api/src/main/java/com/tencent/supersonic/auth/api/authentication/service/UserService.java

@@ -3,6 +3,7 @@ package com.tencent.supersonic.auth.api.authentication.service;
 import com.tencent.supersonic.auth.api.authentication.pojo.Organization;
 import com.tencent.supersonic.auth.api.authentication.pojo.User;
 import com.tencent.supersonic.auth.api.authentication.request.UserReq;
+
 import java.util.List;
 import java.util.Set;
 

auth/api/src/main/java/com/tencent/supersonic/auth/api/authentication/service/UserStrategy.java

@@ -1,6 +1,7 @@
 package com.tencent.supersonic.auth.api.authentication.service;
 
 import com.tencent.supersonic.auth.api.authentication.pojo.User;
+
 import javax.servlet.http.HttpServletRequest;
 import javax.servlet.http.HttpServletResponse;
 

auth/api/src/main/java/com/tencent/supersonic/auth/api/authorization/request/QueryAuthResReq.java

@@ -3,6 +3,7 @@ package com.tencent.supersonic.auth.api.authorization.request;
 import com.tencent.supersonic.auth.api.authorization.pojo.AuthRes;
 import java.util.ArrayList;
 import java.util.List;
+
 import lombok.Data;
 import lombok.ToString;
 

auth/api/src/main/java/com/tencent/supersonic/auth/api/authorization/response/AuthorizedResourceResp.java

@@ -4,6 +4,7 @@ import com.tencent.supersonic.auth.api.authorization.pojo.AuthResGrp;
 import com.tencent.supersonic.auth.api.authorization.pojo.DimensionFilter;
 import java.util.ArrayList;
 import java.util.List;
+
 import lombok.Data;
 
 @Data

auth/api/src/main/java/com/tencent/supersonic/auth/api/authorization/service/AuthService.java

@@ -6,7 +6,6 @@ import com.tencent.supersonic.auth.api.authorization.request.QueryAuthResReq;
 import com.tencent.supersonic.auth.api.authorization.response.AuthorizedResourceResp;
 import java.util.List;
 
-
 public interface AuthService {
 
     List<AuthGroup> queryAuthGroups(String domainId, Integer groupId);

auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/adaptor/DefaultUserAdaptor.java

@@ -11,10 +11,10 @@ import com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO;
 import com.tencent.supersonic.auth.authentication.persistence.repository.UserRepository;
 import com.tencent.supersonic.auth.authentication.utils.UserTokenUtils;
 import com.tencent.supersonic.common.util.ContextUtils;
+import org.springframework.beans.BeanUtils;
 import java.util.List;
 import java.util.Set;
 import java.util.stream.Collectors;
-import org.springframework.beans.BeanUtils;
 
 public class DefaultUserAdaptor implements UserAdaptor {
 

auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/interceptor/AuthenticationInterceptor.java

@@ -42,6 +42,17 @@ public abstract class AuthenticationInterceptor implements HandlerInterceptor {
         return excludePaths.stream().anyMatch(uri::startsWith);
     }
 
+    protected boolean isIncludedUri(String uri) {
+        String includePathStr = authenticationConfig.getIncludePath();
+        if (Strings.isEmpty(includePathStr)) {
+            return false;
+        }
+        List<String> includePaths = Arrays.asList(includePathStr.split(","));
+        if (CollectionUtils.isEmpty(includePaths)) {
+            return false;
+        }
+        return includePaths.stream().anyMatch(uri::startsWith);
+    }
 
     protected boolean isInternalRequest(HttpServletRequest request) {
         String internal = request.getHeader(UserConstants.INTERNAL);

auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/interceptor/DefaultAuthenticationInterceptor.java

@@ -46,6 +46,10 @@ public class DefaultAuthenticationInterceptor extends AuthenticationInterceptor
         }
 
         String uri = request.getServletPath();
+        if (!isIncludedUri(uri)) {
+            return true;
+        }
+
         if (isExcludedUri(uri)) {
             return true;
         }

auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/interceptor/InterceptorFactory.java

@@ -1,11 +1,12 @@
 package com.tencent.supersonic.auth.authentication.interceptor;
 
-import java.util.List;
 import org.springframework.context.annotation.Configuration;
 import org.springframework.core.io.support.SpringFactoriesLoader;
 import org.springframework.web.servlet.config.annotation.InterceptorRegistry;
 import org.springframework.web.servlet.config.annotation.WebMvcConfigurer;
 
+import java.util.List;
+
 @Configuration
 public class InterceptorFactory implements WebMvcConfigurer {
 

auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/persistence/repository/Impl/UserRepositoryImpl.java

@@ -3,8 +3,8 @@ package com.tencent.supersonic.auth.authentication.persistence.repository.impl;
 
 import com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO;
 import com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDOExample;
-import com.tencent.supersonic.auth.authentication.persistence.mapper.UserDOMapper;
 import com.tencent.supersonic.auth.authentication.persistence.repository.UserRepository;
+import com.tencent.supersonic.auth.authentication.persistence.mapper.UserDOMapper;
 import java.util.List;
 import java.util.Optional;
 import org.springframework.stereotype.Component;

auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/service/UserServiceImpl.java

@@ -7,6 +7,7 @@ import com.tencent.supersonic.auth.api.authentication.service.UserService;
 import com.tencent.supersonic.auth.authentication.utils.ComponentFactory;
 import java.util.List;
 import java.util.Set;
+
 import org.springframework.stereotype.Service;
 
 @Service

auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/utils/ComponentFactory.java

@@ -1,8 +1,8 @@
 package com.tencent.supersonic.auth.authentication.utils;
 
 import com.tencent.supersonic.auth.api.authentication.adaptor.UserAdaptor;
-import java.util.Objects;
 import org.springframework.core.io.support.SpringFactoriesLoader;
+import java.util.Objects;
 
 public class ComponentFactory {
 

auth/authentication/src/main/resources/mapper/UserDOMapper.xml

@@ -1,14 +1,12 @@
 <?xml version="1.0" encoding="UTF-8"?>
-<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN"
+<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
-        "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
 <mapper namespace="com.tencent.supersonic.auth.authentication.persistence.mapper.UserDOMapper">
-    <resultMap id="BaseResultMap"
+  <resultMap id="BaseResultMap" type="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO">
-            type="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO">
+    <id column="id" jdbcType="BIGINT" property="id" />
-        <id column="id" jdbcType="BIGINT" property="id"/>
+    <result column="name" jdbcType="VARCHAR" property="name" />
-        <result column="name" jdbcType="VARCHAR" property="name"/>
+    <result column="password" jdbcType="VARCHAR" property="password" />
-        <result column="password" jdbcType="VARCHAR" property="password"/>
+    <result column="display_name" jdbcType="VARCHAR" property="displayName" />
-        <result column="display_name" jdbcType="VARCHAR" property="displayName"/>
+    <result column="email" jdbcType="VARCHAR" property="email" />
-        <result column="email" jdbcType="VARCHAR" property="email"/>
   </resultMap>
   <sql id="Example_Where_Clause">
     <where>
@@ -24,13 +22,11 @@
                   and ${criterion.condition} #{criterion.value}
                 </when>
                 <when test="criterion.betweenValue">
-                                    and ${criterion.condition} #{criterion.value} and
+                  and ${criterion.condition} #{criterion.value} and #{criterion.secondValue}
-                                    #{criterion.secondValue}
                 </when>
                 <when test="criterion.listValue">
                   and ${criterion.condition}
-                                    <foreach close=")" collection="criterion.value" item="listItem"
+                  <foreach close=")" collection="criterion.value" item="listItem" open="(" separator=",">
-                                            open="(" separator=",">
                     #{listItem}
                   </foreach>
                 </when>
@@ -42,20 +38,17 @@
     </where>
   </sql>
   <sql id="Base_Column_List">
-        id
+    id, name, password, display_name, email
-        , name, password, display_name, email
   </sql>
-    <select id="selectByExample"
+  <select id="selectByExample" parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDOExample" resultMap="BaseResultMap">
-            parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDOExample"
-            resultMap="BaseResultMap">
     select
     <if test="distinct">
       distinct
     </if>
-        <include refid="Base_Column_List"/>
+    <include refid="Base_Column_List" />
     from s2_user
     <if test="_parameter != null">
-            <include refid="Example_Where_Clause"/>
+      <include refid="Example_Where_Clause" />
     </if>
     <if test="orderByClause != null">
       order by ${orderByClause}
@@ -66,24 +59,21 @@
   </select>
   <select id="selectByPrimaryKey" parameterType="java.lang.Long" resultMap="BaseResultMap">
     select 
-        <include refid="Base_Column_List"/>
+    <include refid="Base_Column_List" />
     from s2_user
     where id = #{id,jdbcType=BIGINT}
   </select>
   <delete id="deleteByPrimaryKey" parameterType="java.lang.Long">
-        delete
+    delete from s2_user
-        from s2_user
     where id = #{id,jdbcType=BIGINT}
   </delete>
-    <insert id="insert"
+  <insert id="insert" parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO">
-            parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO">
     insert into s2_user (id, name, password, 
       display_name, email)
     values (#{id,jdbcType=BIGINT}, #{name,jdbcType=VARCHAR}, #{password,jdbcType=VARCHAR}, 
       #{displayName,jdbcType=VARCHAR}, #{email,jdbcType=VARCHAR})
   </insert>
-    <insert id="insertSelective"
+  <insert id="insertSelective" parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO">
-            parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO">
     insert into s2_user
     <trim prefix="(" suffix=")" suffixOverrides=",">
       <if test="id != null">
@@ -120,16 +110,13 @@
       </if>
     </trim>
   </insert>
-    <select id="countByExample"
+  <select id="countByExample" parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDOExample" resultType="java.lang.Long">
-            parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDOExample"
-            resultType="java.lang.Long">
     select count(*) from s2_user
     <if test="_parameter != null">
-            <include refid="Example_Where_Clause"/>
+      <include refid="Example_Where_Clause" />
     </if>
   </select>
-    <update id="updateByPrimaryKeySelective"
+  <update id="updateByPrimaryKeySelective" parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO">
-            parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO">
     update s2_user
     <set>
       <if test="name != null">
@@ -147,8 +134,7 @@
     </set>
     where id = #{id,jdbcType=BIGINT}
   </update>
-    <update id="updateByPrimaryKey"
+  <update id="updateByPrimaryKey" parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO">
-            parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO">
     update s2_user
     set name = #{name,jdbcType=VARCHAR},
       password = #{password,jdbcType=VARCHAR},

auth/authorization/pom.xml

@@ -35,10 +35,6 @@
             <groupId>com.google.code.gson</groupId>
             <artifactId>gson</artifactId>
         </dependency>
-        <dependency>
-            <groupId>com.google.guava</groupId>
-            <artifactId>guava</artifactId>
-        </dependency>
         <dependency>
             <groupId>com.tencent.supersonic</groupId>
             <artifactId>auth-authentication</artifactId>

chat/api/src/main/java/com/tencent/supersonic/chat/api/component/SchemaMapper.java

@@ -3,11 +3,8 @@ package com.tencent.supersonic.chat.api.component;
 import com.tencent.supersonic.chat.api.pojo.QueryContext;
 
 /**
- * This interface defines the contract for a schema mapper that identifies references to schema
+ * A schema mapper identifies references to schema elements(metrics/dimensions/entities/values)
- * elements in natural language queries.
+ * in user queries. It matches the query text against the knowledge base.
- *
- * The schema mapper matches queries against the knowledge base which is constructed using the
- * schema of semantic models.
  */
 public interface SchemaMapper {
 

chat/api/src/main/java/com/tencent/supersonic/chat/api/component/SemanticCorrector.java

@@ -1,9 +1,13 @@
 package com.tencent.supersonic.chat.api.component;
 
-import com.tencent.supersonic.chat.api.pojo.CorrectionInfo;
+import com.tencent.supersonic.chat.api.pojo.SemanticCorrectInfo;
 import net.sf.jsqlparser.JSQLParserException;
 
+/**
+ * A semantic corrector checks validity of extracted semantic information and
+ * performs correction and optimization if needed.
+ */
 public interface SemanticCorrector {
-    CorrectionInfo corrector(CorrectionInfo correctionInfo) throws JSQLParserException;
 
+    void correct(SemanticCorrectInfo semanticCorrectInfo) throws JSQLParserException;
 }

chat/api/src/main/java/com/tencent/supersonic/chat/api/component/SemanticLayer.java

@@ -18,10 +18,9 @@ import com.tencent.supersonic.semantic.api.query.request.QueryStructReq;
 import java.util.List;
 
 /**
- * This interface defines the contract for a semantic layer that provides a simplified and
+ * A semantic layer provides a simplified and consistent view of data from multiple sources.
- * consistent view of data from multiple sources.
+ * It abstracts away the complexity of the underlying data sources and provides a unified view
- * The semantic layer abstracts away the complexity of the underlying data sources and provides
+ * of the data that is easier to understand and use.
- * a unified view of the data that is easier to understand and use.
  * <p>
  * The interface defines methods for getting metadata as well as querying data in the semantic layer.
  * Implementations of this interface should provide concrete implementations that interact with the

chat/api/src/main/java/com/tencent/supersonic/chat/api/component/SemanticParser.java

@@ -5,11 +5,9 @@ import com.tencent.supersonic.chat.api.pojo.ChatContext;
 import com.tencent.supersonic.chat.api.pojo.QueryContext;
 
 /**
- * This interface defines the contract for a semantic parser that can analyze natural language query
+ * A semantic parser understands user queries and extracts semantic information.
- * and extract meaning from it.
+ * It could leverage either rule-based or LLM-based approach to identify query intent
- *
+ * and extract related semantic items from the query.
- * The semantic parser uses either rule-based or model-based algorithms to identify query intent
- * and related semantic items described in the query.
  */
 public interface SemanticParser {
 

chat/api/src/main/java/com/tencent/supersonic/chat/api/component/SemanticQuery.java

@@ -6,8 +6,7 @@ import com.tencent.supersonic.chat.api.pojo.response.QueryResult;
 import org.apache.calcite.sql.parser.SqlParseException;
 
 /**
- * This class defines the contract for a semantic query that executes specific type of
+ * A semantic query executes specific type of query based on the results of semantic parsing.
- * query based on the results of semantic parsing.
  */
 public interface SemanticQuery {
 

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/ModelSchema.java

@@ -1,9 +1,10 @@
 package com.tencent.supersonic.chat.api.pojo;
 
+import lombok.Data;
+
 import java.util.HashSet;
 import java.util.Optional;
 import java.util.Set;
-import lombok.Data;
 
 @Data
 public class ModelSchema {

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/QueryContext.java

@@ -2,11 +2,12 @@ package com.tencent.supersonic.chat.api.pojo;
 
 import com.tencent.supersonic.chat.api.component.SemanticQuery;
 import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
-import java.util.ArrayList;
-import java.util.List;
 import lombok.Data;
 import lombok.NoArgsConstructor;
 
+import java.util.ArrayList;
+import java.util.List;
+
 @Data
 @NoArgsConstructor
 public class QueryContext {

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/SchemaElement.java

@@ -1,39 +1,30 @@
 package com.tencent.supersonic.chat.api.pojo;
 
 import com.google.common.base.Objects;
-
 import java.io.Serializable;
 import java.util.List;
-
+import lombok.AllArgsConstructor;
+import lombok.Builder;
 import lombok.Data;
 import lombok.Getter;
-import lombok.Builder;
 import lombok.NoArgsConstructor;
 
 @Data
 @Getter
 @Builder
+@AllArgsConstructor
 @NoArgsConstructor
 public class SchemaElement implements Serializable {
-
     private Long model;
     private Long id;
     private String name;
     private String bizName;
     private Long useCnt;
     private SchemaElementType type;
+
     private List<String> alias;
 
-    public SchemaElement(Long model, Long id, String name, String bizName,
+    private List<SchemaValueMap> schemaValueMaps;
-                         Long useCnt, SchemaElementType type, List<String> alias) {
-        this.model = model;
-        this.id = id;
-        this.name = name;
-        this.bizName = bizName;
-        this.useCnt = useCnt;
-        this.type = type;
-        this.alias = alias;
-    }
 
     @Override
     public boolean equals(Object o) {
@@ -54,4 +45,5 @@ public class SchemaElement implements Serializable {
     public int hashCode() {
         return Objects.hashCode(model, id, name, bizName, useCnt, type);
     }
+
 }

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/SchemaMapInfo.java

@@ -21,6 +21,10 @@ public class SchemaMapInfo {
         return modelElementMatches;
     }
 
+    public void setModelElementMatches(Map<Long, List<SchemaElementMatch>> modelElementMatches) {
+        this.modelElementMatches = modelElementMatches;
+    }
+
     public void setMatchedElements(Long model, List<SchemaElementMatch> elementMatches) {
         modelElementMatches.put(model, elementMatches);
     }

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/SchemaValueMap.java

@@ -0,0 +1,24 @@
+package com.tencent.supersonic.chat.api.pojo;
+
+import java.util.ArrayList;
+import java.util.List;
+import lombok.Data;
+
+@Data
+public class SchemaValueMap {
+
+    /**
+     * dimension value in db
+     */
+    private String techName;
+
+    /**
+     * dimension value for result show
+     */
+    private String bizName;
+
+    /**
+     * dimension value for user query
+     */
+    private List<String> alias = new ArrayList<>();
+}

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/SemanticCorrectInfo.java

@@ -10,7 +10,7 @@ import lombok.NoArgsConstructor;
 @AllArgsConstructor
 @NoArgsConstructor
 @Builder
-public class CorrectionInfo {
+public class SemanticCorrectInfo {
 
     private QueryFilters queryFilters;
 
@@ -18,4 +18,5 @@ public class CorrectionInfo {
 
     private String sql;
 
+    private String preSql;
 }

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/SemanticSchema.java

@@ -7,7 +7,6 @@ import java.util.Map;
 import java.util.stream.Collectors;
 
 public class SemanticSchema implements Serializable {
-
     private List<ModelSchema> modelSchemaList;
 
     public SemanticSchema(List<ModelSchema> modelSchemaList) {

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/ChatAggConfigReq.java

@@ -1,8 +1,9 @@
 package com.tencent.supersonic.chat.api.pojo.request;
 
-import java.util.List;
 import lombok.Data;
 
+import java.util.List;
+
 @Data
 public class ChatAggConfigReq {
 

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/ChatConfigBaseReq.java

@@ -1,10 +1,12 @@
 package com.tencent.supersonic.chat.api.pojo.request;
 
 import com.tencent.supersonic.common.pojo.enums.StatusEnum;
-import java.util.List;
+
 import lombok.Data;
 import lombok.ToString;
 
+import java.util.List;
+
 /**
  * extended information command about model
  */

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/ChatDefaultConfigReq.java

@@ -2,9 +2,10 @@ package com.tencent.supersonic.chat.api.pojo.request;
 
 
 import com.tencent.supersonic.common.pojo.Constants;
+import lombok.Data;
+
 import java.util.ArrayList;
 import java.util.List;
-import lombok.Data;
 
 @Data
 public class ChatDefaultConfigReq {

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/ChatDetailConfigReq.java

@@ -1,8 +1,9 @@
 package com.tencent.supersonic.chat.api.pojo.request;
 
-import java.util.List;
 import lombok.Data;
 
+import java.util.List;
+
 @Data
 public class ChatDetailConfigReq {
 

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/DimensionValueReq.java

@@ -0,0 +1,12 @@
+package com.tencent.supersonic.chat.api.pojo.request;
+
+import lombok.Data;
+
+@Data
+public class DimensionValueReq {
+    private Long modelId;
+
+    private String bizName;
+
+    private Object value;
+}

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/ExecuteQueryReq.java

@@ -7,13 +7,12 @@ import lombok.Data;
 
 @Data
 public class ExecuteQueryReq {
-
     private User user;
     private Integer agentId;
     private Integer chatId;
     private String queryText;
-    private Long queryId;
+    private Long queryId = 7L;
-    private Integer parseId;
+    private Integer parseId = 2;
     private SemanticParseInfo parseInfo;
     private boolean saveAnswer = true;
 }

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/ItemNameVisibility.java

@@ -0,0 +1,13 @@
+package com.tencent.supersonic.chat.api.pojo.request;
+
+import lombok.Data;
+import lombok.ToString;
+
+@Data
+@ToString
+public class ItemNameVisibility {
+
+    private ItemNameVisibilityInfo aggVisibilityInfo;
+
+    private ItemNameVisibilityInfo detailVisibilityInfo;
+}

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/ItemNameVisibilityInfo.java

@@ -0,0 +1,23 @@
+package com.tencent.supersonic.chat.api.pojo.request;
+
+import lombok.Data;
+import lombok.ToString;
+
+import java.util.ArrayList;
+import java.util.List;
+
+
+@Data
+@ToString
+public class ItemNameVisibilityInfo {
+
+    /**
+     * invisible dimensions
+     */
+    private List<String> blackDimNameList = new ArrayList<>();
+
+    /**
+     * invisible metrics
+     */
+    private List<String> blackMetricNameList = new ArrayList<>();
+}

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/KnowledgeAdvancedConfig.java

@@ -1,8 +1,9 @@
 package com.tencent.supersonic.chat.api.pojo.request;
 
+import lombok.Data;
+
 import java.util.ArrayList;
 import java.util.List;
-import lombok.Data;
 
 /**
  * advanced knowledge config

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/KnowledgeInfoReq.java

@@ -1,7 +1,9 @@
 package com.tencent.supersonic.chat.api.pojo.request;
 
 import com.tencent.supersonic.common.pojo.enums.TypeEnums;
+
 import javax.validation.constraints.NotNull;
+
 import lombok.Data;
 
 /**
@@ -30,6 +32,4 @@ public class KnowledgeInfoReq {
      * advanced knowledge config for single item
      */
     private KnowledgeAdvancedConfig knowledgeAdvancedConfig;
-
-
 }

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/QueryDataReq.java

@@ -4,19 +4,20 @@ package com.tencent.supersonic.chat.api.pojo.request;
 import com.tencent.supersonic.chat.api.pojo.SchemaElement;
 import com.tencent.supersonic.common.pojo.DateConf;
 import com.tencent.supersonic.common.pojo.Order;
+import com.tencent.supersonic.common.pojo.enums.AggregateTypeEnum;
 import java.util.HashSet;
 import java.util.Set;
 import lombok.Data;
 
 @Data
 public class QueryDataReq {
-
     String queryMode;
     SchemaElement model;
     Set<SchemaElement> metrics = new HashSet<>();
     Set<SchemaElement> dimensions = new HashSet<>();
     Set<QueryFilter> dimensionFilters = new HashSet<>();
     Set<QueryFilter> metricFilters = new HashSet<>();
+    private AggregateTypeEnum aggType = AggregateTypeEnum.NONE;
     private Set<Order> orders = new HashSet<>();
     private DateConf dateInfo;
     private Long limit;

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/QueryFilters.java

@@ -1,14 +1,13 @@
 package com.tencent.supersonic.chat.api.pojo.request;
 
+import lombok.Data;
 import java.util.ArrayList;
 import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
-import lombok.Data;
 
 @Data
 public class QueryFilters {
-
     private List<QueryFilter> filters = new ArrayList<>();
     private Map<String, Object> params = new HashMap<>();
 }

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/QueryReq.java

@@ -5,7 +5,6 @@ import lombok.Data;
 
 @Data
 public class QueryReq {
-
     private String queryText;
     private Integer chatId;
     private Long modelId = 0L;

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/ChatAggRichConfigResp.java

@@ -2,9 +2,10 @@ package com.tencent.supersonic.chat.api.pojo.response;
 
 import com.tencent.supersonic.chat.api.pojo.request.KnowledgeAdvancedConfig;
 import com.tencent.supersonic.chat.api.pojo.request.KnowledgeInfoReq;
-import java.util.List;
 import lombok.Data;
 
+import java.util.List;
+
 @Data
 public class ChatAggRichConfigResp {
 

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/ChatConfigResp.java

@@ -4,8 +4,10 @@ import com.tencent.supersonic.chat.api.pojo.request.ChatAggConfigReq;
 import com.tencent.supersonic.chat.api.pojo.request.ChatDetailConfigReq;
 import com.tencent.supersonic.chat.api.pojo.request.RecommendedQuestionReq;
 import com.tencent.supersonic.common.pojo.enums.StatusEnum;
+
 import java.util.Date;
 import java.util.List;
+
 import lombok.Data;
 
 @Data

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/ChatConfigRichResp.java

@@ -4,6 +4,7 @@ import com.tencent.supersonic.chat.api.pojo.request.RecommendedQuestionReq;
 import com.tencent.supersonic.common.pojo.enums.StatusEnum;
 import java.util.Date;
 import java.util.List;
+
 import lombok.Data;
 
 @Data

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/ChatDefaultRichConfigResp.java

@@ -4,9 +4,10 @@ package com.tencent.supersonic.chat.api.pojo.response;
 import com.tencent.supersonic.chat.api.pojo.SchemaElement;
 import com.tencent.supersonic.chat.api.pojo.request.ChatDefaultConfigReq;
 import com.tencent.supersonic.common.pojo.Constants;
-import java.util.List;
 import lombok.Data;
 
+import java.util.List;
+
 @Data
 public class ChatDefaultRichConfigResp {
 

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/ChatDetailRichConfigResp.java

@@ -2,9 +2,10 @@ package com.tencent.supersonic.chat.api.pojo.response;
 
 import com.tencent.supersonic.chat.api.pojo.request.KnowledgeAdvancedConfig;
 import com.tencent.supersonic.chat.api.pojo.request.KnowledgeInfoReq;
-import java.util.List;
 import lombok.Data;
 
+import java.util.List;
+
 @Data
 public class ChatDetailRichConfigResp {
 

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/EntityRichInfoResp.java

@@ -1,12 +1,12 @@
 package com.tencent.supersonic.chat.api.pojo.response;
 
 import com.tencent.supersonic.chat.api.pojo.SchemaElement;
+
 import java.util.List;
 import lombok.Data;
 
 @Data
 public class EntityRichInfoResp {
-
     /**
      *  entity alias
      */

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/RecommendQuestionResp.java

@@ -1,14 +1,14 @@
 package com.tencent.supersonic.chat.api.pojo.response;
 
 import com.tencent.supersonic.chat.api.pojo.request.RecommendedQuestionReq;
-import java.util.List;
 import lombok.AllArgsConstructor;
 import lombok.Data;
 
+import java.util.List;
+
 @Data
 @AllArgsConstructor
 public class RecommendQuestionResp {
-
     private Long modelId;
     private List<RecommendedQuestionReq> recommendedQuestions;
 }

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/RecommendResp.java

@@ -1,12 +1,12 @@
 package com.tencent.supersonic.chat.api.pojo.response;
 
 import com.tencent.supersonic.chat.api.pojo.SchemaElement;
-import java.util.List;
 import lombok.Data;
 
+import java.util.List;
+
 @Data
 public class RecommendResp {
-
     private List<SchemaElement> dimensions;
     private List<SchemaElement> metrics;
 }

chat/core/pom.xml

@@ -136,6 +136,14 @@
             <artifactId>xk-time</artifactId>
             <version>${xk.time.version}</version>
         </dependency>
+
+        <dependency>
+            <groupId>org.mockito</groupId>
+            <artifactId>mockito-inline</artifactId>
+            <version>${mockito-inline.version}</version>
+            <scope>test</scope>
+        </dependency>
+
     </dependencies>
 
 </project>

chat/core/src/main/java/com/tencent/supersonic/chat/agent/tool/AgentTool.java

@@ -9,8 +9,9 @@ import lombok.NoArgsConstructor;
 @AllArgsConstructor
 public class AgentTool {
 
+    private String id;
+
     private String name;
 
     private AgentToolType type;
-
 }

chat/core/src/main/java/com/tencent/supersonic/chat/aspect/TimeCostAspect.java

@@ -1,99 +0,0 @@
-/*
-//package com.tencent.supersonic.chat.aspect;
-//
-//import lombok.extern.slf4j.Slf4j;
-//import org.aspectj.lang.JoinPoint;
-//import org.aspectj.lang.ProceedingJoinPoint;
-//import org.aspectj.lang.annotation.*;
-//import org.springframework.stereotype.Component;
-//
-//import java.util.HashMap;
-//import java.util.Map;
-//
-//@Aspect
-//@Component
-//@Slf4j
-//public class TimeCostAspect {
-//
-//    ThreadLocal<Long> startTime = new ThreadLocal<>();
-//
-//    ThreadLocal<Map<String, Long>> map = new ThreadLocal<>();
-//
-//    @Pointcut("execution(public * com.tencent.supersonic.chat.mapper.HanlpDictMapper.*(*))")
-//    //@Pointcut("execution(* public com.tencent.supersonic.chat.parser.*.*(..))")
-//    //@Pointcut("execution(* com.tencent.supersonic.chat.mapper.*Mapper.map(..)) ")
-//    //@Pointcut("execution(* com.tencent.supersonic.chat.mapper.HanlpDictMapper.map(..)) ")
-//    //@Pointcut("execution(* com.tencent.supersonic.chat.parser.rule.QueryModeParser.*(..)) ")
-//    public void point() {
-//    }
-//
-//    @Around("point()")
-//    public void doAround(ProceedingJoinPoint joinPoint) throws Throwable {
-//        long start = System.currentTimeMillis();
-//        try {
-//            log.info("切面开始");
-//            Object result = joinPoint.proceed();
-//            log.info("切面开始");
-//            if (result == null) {
-//                //如果切到了 没有返回类型的void方法，这里直接返回
-//                //return null;
-//            }
-//            long end = System.currentTimeMillis();
-//            log.info("===================");
-//            String targetClassName = joinPoint.getSignature().getDeclaringTypeName();
-//            String MethodName = joinPoint.getSignature().getName();
-//            String typeStr = joinPoint.getSignature().getDeclaringType().toString().split(" ")[0];
-//            log.info("类/接口:" + targetClassName + "(" + typeStr + ")");
-//            log.info("方法:" + MethodName);
-//            Long total = end - start;
-//            log.info("耗时: " + total + " ms!");
-//            map.get().put(targetClassName + "_" + MethodName, total);
-//            //return result;
-//        } catch (Throwable e) {
-//            long end = System.currentTimeMillis();
-//            log.info("====around " + joinPoint + "\tUse time : " + (end - start) + " ms with exception : "
-//                    + e.getMessage());
-//            throw e;
-//        }
-//    }
-//
-////    //对Controller下面的方法执行前进行切入，初始化开始时间
-////    @Before(value = "execution(* com.appleyk.controller.*.*(..))")
-////    public void beforMehhod(JoinPoint jp) {
-////        startTime.set(System.currentTimeMillis());
-////    }
-////
-////    //对Controller下面的方法执行后进行切入，统计方法执行的次数和耗时情况
-////    //注意，这里的执行方法统计的数据不止包含Controller下面的方法，也包括环绕切入的所有方法的统计信息
-////    @AfterReturning(value = "execution(* com.appleyk.controller.*.*(..))")
-////    public void afterMehhod(JoinPoint jp) {
-////        long end = System.currentTimeMillis();
-////        long total =  end - startTime.get();
-////        String methodName = jp.getSignature().getName();
-////        log.info("连接点方法为：" + methodName + ",执行总耗时为：" +total+"ms");
-////
-////        //重新new一个map
-////        Map<String, Long> map = new HashMap<>();
-//////从map2中将最后的 连接点方法给移除了，替换成最终的，避免连接点方法多次进行叠加计算
-////        //由于map2受ThreadLocal的保护，这里不支持remove，因此，需要单开一个map进行数据交接
-////        for(Map.Entry<String, Long> entry:map2.get().entrySet()){
-////            if(entry.getKey().equals(methodName)){
-////                map.put(methodName, total);
-////
-////            }else{
-////                map.put(entry.getKey(), entry.getValue());
-////            }
-////        }
-////
-////        for (Map.Entry<String, Long> entry :map1.get().entrySet()) {
-////            for(Map.Entry<String, Long> entry2 :map.entrySet()){
-////                if(entry.getKey().equals(entry2.getKey())){
-////                    System.err.println(entry.getKey()+",被调用次数："+entry.getValue()+",综合耗时："+entry2.getValue()+"ms");
-////                }
-////            }
-////
-////        }
-////    }
-//
-//}
-*/

chat/core/src/main/java/com/tencent/supersonic/chat/config/ChatConfig.java

@@ -3,12 +3,13 @@ package com.tencent.supersonic.chat.config;
 import com.tencent.supersonic.chat.api.pojo.request.ChatAggConfigReq;
 import com.tencent.supersonic.chat.api.pojo.request.ChatDetailConfigReq;
 import com.tencent.supersonic.chat.api.pojo.request.RecommendedQuestionReq;
-import com.tencent.supersonic.common.pojo.RecordInfo;
 import com.tencent.supersonic.common.pojo.enums.StatusEnum;
-import java.util.List;
+import com.tencent.supersonic.common.pojo.RecordInfo;
 import lombok.Data;
 import lombok.ToString;
 
+import java.util.List;
+
 @Data
 @ToString
 public class ChatConfig {

chat/core/src/main/java/com/tencent/supersonic/chat/config/FunctionCallInfoConfig.java

@@ -7,7 +7,9 @@ import org.springframework.context.annotation.Configuration;
 @Configuration
 @Data
 public class FunctionCallInfoConfig {
-
     @Value("${functionCall.url:}")
     private String url;
+
+    @Value("${funtionCall.plugin.select.path:/plugin_selection}")
+    private String pluginSelectPath;
 }

chat/core/src/main/java/com/tencent/supersonic/chat/config/LLMConfig.java

@@ -13,7 +13,7 @@ public class LLMConfig {
     @Value("${llm.url:}")
     private String url;
 
-    @Value("${query2sql.path:query2sql}")
+    @Value("${query2sql.path:/query2sql}")
     private String queryToSqlPath;
 
 }

chat/core/src/main/java/com/tencent/supersonic/chat/config/OptimizationConfig.java

@@ -0,0 +1,43 @@
+package com.tencent.supersonic.chat.config;
+
+import lombok.Data;
+import org.springframework.beans.factory.annotation.Value;
+import org.springframework.context.annotation.Configuration;
+import org.springframework.context.annotation.PropertySource;
+
+@Configuration
+@Data
+@PropertySource("classpath:optimization.properties")
+//@ComponentScan(basePackages = "com.tencent.supersonic.chat")
+public class OptimizationConfig {
+
+    @Value("${one.detection.size}")
+    private Integer oneDetectionSize;
+    @Value("${one.detection.max.size}")
+    private Integer oneDetectionMaxSize;
+
+    @Value("${metric.dimension.min.threshold}")
+    private Double metricDimensionMinThresholdConfig;
+
+    @Value("${metric.dimension.threshold}")
+    private Double metricDimensionThresholdConfig;
+
+    @Value("${dimension.value.threshold}")
+    private Double dimensionValueThresholdConfig;
+
+    @Value("${function.bonus.threshold}")
+    private Double functionBonusThreshold;
+
+    @Value("${long.text.threshold}")
+    private Double longTextThreshold;
+
+    @Value("${short.text.threshold}")
+    private Double shortTextThreshold;
+
+    @Value("${query.text.length.threshold}")
+    private Integer queryTextLengthThreshold;
+
+    @Value("${candidate.threshold}")
+    private Double candidateThreshold;
+
+}

chat/core/src/main/java/com/tencent/supersonic/chat/corrector/DateFieldCorrector.java

@@ -1,6 +1,6 @@
 package com.tencent.supersonic.chat.corrector;
 
-import com.tencent.supersonic.chat.api.pojo.CorrectionInfo;
+import com.tencent.supersonic.chat.api.pojo.SemanticCorrectInfo;
 import com.tencent.supersonic.chat.parser.llm.dsl.DSLDateHelper;
 import com.tencent.supersonic.common.util.jsqlparser.SqlParserSelectHelper;
 import com.tencent.supersonic.common.util.jsqlparser.SqlParserUpdateHelper;
@@ -12,16 +12,16 @@ import org.springframework.util.CollectionUtils;
 public class DateFieldCorrector extends BaseSemanticCorrector {
 
     @Override
-    public CorrectionInfo corrector(CorrectionInfo correctionInfo) {
+    public void correct(SemanticCorrectInfo semanticCorrectInfo) {
 
-        String sql = correctionInfo.getSql();
+        String sql = semanticCorrectInfo.getSql();
         List<String> whereFields = SqlParserSelectHelper.getWhereFields(sql);
         if (CollectionUtils.isEmpty(whereFields) || !whereFields.contains(DATE_FIELD)) {
-            String currentDate = DSLDateHelper.getCurrentDate(correctionInfo.getParseInfo().getModelId());
+            String currentDate = DSLDateHelper.getReferenceDate(semanticCorrectInfo.getParseInfo().getModelId());
             sql = SqlParserUpdateHelper.addWhere(sql, DATE_FIELD, currentDate);
         }
-        correctionInfo.setSql(sql);
+        semanticCorrectInfo.setPreSql(semanticCorrectInfo.getSql());
-        return correctionInfo;
+        semanticCorrectInfo.setSql(sql);
     }
 
 }

chat/core/src/main/java/com/tencent/supersonic/chat/corrector/FieldCorrector.java

@@ -1,6 +1,6 @@
 package com.tencent.supersonic.chat.corrector;
 
-import com.tencent.supersonic.chat.api.pojo.CorrectionInfo;
+import com.tencent.supersonic.chat.api.pojo.SemanticCorrectInfo;
 import com.tencent.supersonic.common.util.jsqlparser.SqlParserUpdateHelper;
 import lombok.extern.slf4j.Slf4j;
 
@@ -8,10 +8,11 @@ import lombok.extern.slf4j.Slf4j;
 public class FieldCorrector extends BaseSemanticCorrector {
 
     @Override
-    public CorrectionInfo corrector(CorrectionInfo correctionInfo) {
+    public void correct(SemanticCorrectInfo semanticCorrectInfo) {
-        String replaceFields = SqlParserUpdateHelper.replaceFields(correctionInfo.getSql(),
+        String preSql = semanticCorrectInfo.getSql();
-                getFieldToBizName(correctionInfo.getParseInfo().getModelId()));
+        semanticCorrectInfo.setPreSql(preSql);
-        correctionInfo.setSql(replaceFields);
+        String sql = SqlParserUpdateHelper.replaceFields(preSql,
-        return correctionInfo;
+                getFieldToBizName(semanticCorrectInfo.getParseInfo().getModelId()));
+        semanticCorrectInfo.setSql(sql);
     }
 }

chat/core/src/main/java/com/tencent/supersonic/chat/corrector/FieldNameCorrector.java

@@ -0,0 +1,49 @@
+package com.tencent.supersonic.chat.corrector;
+
+import com.tencent.supersonic.chat.api.pojo.SemanticCorrectInfo;
+import com.tencent.supersonic.chat.parser.llm.dsl.DSLParseResult;
+import com.tencent.supersonic.chat.query.llm.dsl.LLMReq;
+import com.tencent.supersonic.chat.query.llm.dsl.LLMReq.ElementValue;
+import com.tencent.supersonic.common.pojo.Constants;
+import com.tencent.supersonic.common.util.JsonUtil;
+import com.tencent.supersonic.common.util.jsqlparser.SqlParserUpdateHelper;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import lombok.extern.slf4j.Slf4j;
+import org.springframework.util.CollectionUtils;
+
+@Slf4j
+public class FieldNameCorrector extends BaseSemanticCorrector {
+
+    @Override
+    public void correct(SemanticCorrectInfo semanticCorrectInfo) {
+
+        Object context = semanticCorrectInfo.getParseInfo().getProperties().get(Constants.CONTEXT);
+        if (Objects.isNull(context)) {
+            return;
+        }
+
+        DSLParseResult dslParseResult = JsonUtil.toObject(JsonUtil.toString(context), DSLParseResult.class);
+        if (Objects.isNull(dslParseResult) || Objects.isNull(dslParseResult.getLlmReq())) {
+            return;
+        }
+        LLMReq llmReq = dslParseResult.getLlmReq();
+        List<ElementValue> linking = llmReq.getLinking();
+        if (CollectionUtils.isEmpty(linking)) {
+            return;
+        }
+
+        Map<String, Set<String>> fieldValueToFieldNames = linking.stream().collect(
+                Collectors.groupingBy(ElementValue::getFieldValue,
+                        Collectors.mapping(ElementValue::getFieldName, Collectors.toSet())));
+
+        String preSql = semanticCorrectInfo.getSql();
+        semanticCorrectInfo.setPreSql(preSql);
+        String sql = SqlParserUpdateHelper.replaceFieldNameByValue(preSql, fieldValueToFieldNames);
+        semanticCorrectInfo.setSql(sql);
+    }
+
+}

chat/core/src/main/java/com/tencent/supersonic/chat/corrector/FieldValueCorrector.java

@@ -1,48 +1,81 @@
 package com.tencent.supersonic.chat.corrector;
 
-import com.tencent.supersonic.chat.api.pojo.CorrectionInfo;
+import com.tencent.supersonic.chat.api.pojo.SemanticCorrectInfo;
-import com.tencent.supersonic.chat.parser.llm.dsl.DSLParseResult;
+import com.tencent.supersonic.chat.api.pojo.SchemaElement;
-import com.tencent.supersonic.chat.query.llm.dsl.LLMReq;
+import com.tencent.supersonic.chat.api.pojo.SchemaValueMap;
-import com.tencent.supersonic.chat.query.llm.dsl.LLMReq.ElementValue;
+import com.tencent.supersonic.chat.api.pojo.SemanticSchema;
-import com.tencent.supersonic.common.pojo.Constants;
+import com.tencent.supersonic.common.util.ContextUtils;
-import com.tencent.supersonic.common.util.JsonUtil;
 import com.tencent.supersonic.common.util.jsqlparser.SqlParserUpdateHelper;
+import com.tencent.supersonic.knowledge.service.SchemaService;
+import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
 import java.util.Objects;
-import java.util.Set;
 import java.util.stream.Collectors;
 import lombok.extern.slf4j.Slf4j;
+import org.apache.logging.log4j.util.Strings;
 import org.springframework.util.CollectionUtils;
 
 @Slf4j
 public class FieldValueCorrector extends BaseSemanticCorrector {
 
     @Override
-    public CorrectionInfo corrector(CorrectionInfo correctionInfo) {
+    public void correct(SemanticCorrectInfo semanticCorrectInfo) {
+        SemanticSchema semanticSchema = ContextUtils.getBean(SchemaService.class).getSemanticSchema();
+        Long modelId = semanticCorrectInfo.getParseInfo().getModel().getId();
+        List<SchemaElement> dimensions = semanticSchema.getDimensions().stream()
+                .filter(schemaElement -> modelId.equals(schemaElement.getModel()))
+                .collect(Collectors.toList());
 
-        Object context = correctionInfo.getParseInfo().getProperties().get(Constants.CONTEXT);
+        if (CollectionUtils.isEmpty(dimensions)) {
-        if (Objects.isNull(context)) {
+            return;
-            return correctionInfo;
         }
 
-        DSLParseResult dslParseResult = JsonUtil.toObject(JsonUtil.toString(context), DSLParseResult.class);
+        Map<String, Map<String, String>> aliasAndBizNameToTechName = getAliasAndBizNameToTechName(dimensions);
-        if (Objects.isNull(dslParseResult) || Objects.isNull(dslParseResult.getLlmReq())) {
+        String preSql = semanticCorrectInfo.getSql();
-            return correctionInfo;
+        semanticCorrectInfo.setPreSql(preSql);
-        }
+        String sql = SqlParserUpdateHelper.replaceValue(preSql, aliasAndBizNameToTechName);
-        LLMReq llmReq = dslParseResult.getLlmReq();
+        semanticCorrectInfo.setSql(sql);
-        List<ElementValue> linking = llmReq.getLinking();
+        return;
-        if (CollectionUtils.isEmpty(linking)) {
-            return correctionInfo;
     }
 
-        Map<String, Set<String>> fieldValueToFieldNames = linking.stream().collect(
-                Collectors.groupingBy(ElementValue::getFieldValue,
-                        Collectors.mapping(ElementValue::getFieldName, Collectors.toSet())));
 
-        String sql = SqlParserUpdateHelper.replaceValueFields(correctionInfo.getSql(), fieldValueToFieldNames);
+    private Map<String, Map<String, String>> getAliasAndBizNameToTechName(List<SchemaElement> dimensions) {
-        correctionInfo.setSql(sql);
+        if (CollectionUtils.isEmpty(dimensions)) {
-        return correctionInfo;
+            return new HashMap<>();
         }
 
+        Map<String, Map<String, String>> result = new HashMap<>();
+
+        for (SchemaElement dimension : dimensions) {
+            if (Objects.isNull(dimension)
+                    || Strings.isEmpty(dimension.getBizName())
+                    || CollectionUtils.isEmpty(dimension.getSchemaValueMaps())) {
+                continue;
+            }
+            String bizName = dimension.getBizName();
+
+            Map<String, String> aliasAndBizNameToTechName = new HashMap<>();
+
+            for (SchemaValueMap valueMap : dimension.getSchemaValueMaps()) {
+                if (Objects.isNull(valueMap) || Strings.isEmpty(valueMap.getTechName())) {
+                    continue;
+                }
+                if (Strings.isNotEmpty(valueMap.getBizName())) {
+                    aliasAndBizNameToTechName.put(valueMap.getBizName(), valueMap.getTechName());
+                }
+                if (!CollectionUtils.isEmpty(valueMap.getAlias())) {
+                    valueMap.getAlias().stream().forEach(alias -> {
+                        if (Strings.isNotEmpty(alias)) {
+                            aliasAndBizNameToTechName.put(alias, valueMap.getTechName());
+                        }
+                    });
+                }
+            }
+            if (!CollectionUtils.isEmpty(aliasAndBizNameToTechName)) {
+                result.put(bizName, aliasAndBizNameToTechName);
+            }
+        }
+        return result;
+    }
 }

chat/core/src/main/java/com/tencent/supersonic/chat/corrector/FunctionAliasCorrector.java

@@ -0,0 +1,16 @@
+package com.tencent.supersonic.chat.corrector;
+
+import com.tencent.supersonic.chat.api.pojo.SemanticCorrectInfo;
+import com.tencent.supersonic.common.util.jsqlparser.SqlParserUpdateHelper;
+import lombok.extern.slf4j.Slf4j;
+
+@Slf4j
+public class FunctionAliasCorrector extends BaseSemanticCorrector {
+
+    @Override
+    public void correct(SemanticCorrectInfo semanticCorrectInfo) {
+        String replaceAlias = SqlParserUpdateHelper.replaceAlias(semanticCorrectInfo.getSql());
+        semanticCorrectInfo.setSql(replaceAlias);
+    }
+
+}

chat/core/src/main/java/com/tencent/supersonic/chat/corrector/FunctionCorrector.java

@@ -1,6 +1,6 @@
 package com.tencent.supersonic.chat.corrector;
 
-import com.tencent.supersonic.chat.api.pojo.CorrectionInfo;
+import com.tencent.supersonic.chat.api.pojo.SemanticCorrectInfo;
 import com.tencent.supersonic.common.util.jsqlparser.SqlParserUpdateHelper;
 import lombok.extern.slf4j.Slf4j;
 
@@ -8,9 +8,10 @@ import lombok.extern.slf4j.Slf4j;
 public class FunctionCorrector extends BaseSemanticCorrector {
 
     @Override
-    public CorrectionInfo corrector(CorrectionInfo correctionInfo) {
+    public void correct(SemanticCorrectInfo semanticCorrectInfo) {
-        String replaceFunction = SqlParserUpdateHelper.replaceFunction(correctionInfo.getSql());
+        String preSql = semanticCorrectInfo.getSql();
-        correctionInfo.setSql(replaceFunction);
+        semanticCorrectInfo.setPreSql(preSql);
-        return correctionInfo;
+        String sql = SqlParserUpdateHelper.replaceFunction(preSql);
+        semanticCorrectInfo.setSql(sql);
     }
 }

chat/core/src/main/java/com/tencent/supersonic/chat/corrector/QueryFilterAppend.java

@@ -1,6 +1,6 @@
 package com.tencent.supersonic.chat.corrector;
 
-import com.tencent.supersonic.chat.api.pojo.CorrectionInfo;
+import com.tencent.supersonic.chat.api.pojo.SemanticCorrectInfo;
 import com.tencent.supersonic.chat.api.pojo.request.QueryFilters;
 import com.tencent.supersonic.common.pojo.Constants;
 import com.tencent.supersonic.common.util.StringUtil;
@@ -18,17 +18,17 @@ import org.apache.commons.lang3.StringUtils;
 public class QueryFilterAppend extends BaseSemanticCorrector {
 
     @Override
-    public CorrectionInfo corrector(CorrectionInfo correctionInfo) throws JSQLParserException {
+    public void correct(SemanticCorrectInfo semanticCorrectInfo) throws JSQLParserException {
-        String queryFilter = getQueryFilter(correctionInfo.getQueryFilters());
+        String queryFilter = getQueryFilter(semanticCorrectInfo.getQueryFilters());
-        String sql = correctionInfo.getSql();
+        String preSql = semanticCorrectInfo.getSql();
 
         if (StringUtils.isNotEmpty(queryFilter)) {
-            log.info("add queryFilter to sql :{}", queryFilter);
+            log.info("add queryFilter to preSql :{}", queryFilter);
             Expression expression = CCJSqlParserUtil.parseCondExpression(queryFilter);
-            sql = SqlParserUpdateHelper.addWhere(sql, expression);
+            String sql = SqlParserUpdateHelper.addWhere(preSql, expression);
+            semanticCorrectInfo.setPreSql(preSql);
+            semanticCorrectInfo.setSql(sql);
         }
-        correctionInfo.setSql(sql);
-        return correctionInfo;
     }
 
     private String getQueryFilter(QueryFilters queryFilters) {

chat/core/src/main/java/com/tencent/supersonic/chat/corrector/SelectFieldAppendCorrector.java

@@ -1,6 +1,6 @@
 package com.tencent.supersonic.chat.corrector;
 
-import com.tencent.supersonic.chat.api.pojo.CorrectionInfo;
+import com.tencent.supersonic.chat.api.pojo.SemanticCorrectInfo;
 import com.tencent.supersonic.common.util.jsqlparser.SqlParserSelectHelper;
 import com.tencent.supersonic.common.util.jsqlparser.SqlParserUpdateHelper;
 import com.tencent.supersonic.semantic.api.model.enums.TimeDimensionEnum;
@@ -14,26 +14,25 @@ import org.springframework.util.CollectionUtils;
 public class SelectFieldAppendCorrector extends BaseSemanticCorrector {
 
     @Override
-    public CorrectionInfo corrector(CorrectionInfo correctionInfo) {
+    public void correct(SemanticCorrectInfo semanticCorrectInfo) {
-        String sql = correctionInfo.getSql();
+        String preSql = semanticCorrectInfo.getSql();
-        if (SqlParserSelectHelper.hasAggregateFunction(sql)) {
+        if (SqlParserSelectHelper.hasAggregateFunction(preSql)) {
-            return correctionInfo;
+            return;
         }
-        Set<String> selectFields = new HashSet<>(SqlParserSelectHelper.getSelectFields(sql));
+        Set<String> selectFields = new HashSet<>(SqlParserSelectHelper.getSelectFields(preSql));
-        Set<String> whereFields = new HashSet<>(SqlParserSelectHelper.getWhereFields(sql));
+        Set<String> whereFields = new HashSet<>(SqlParserSelectHelper.getWhereFields(preSql));
 
         if (CollectionUtils.isEmpty(selectFields) || CollectionUtils.isEmpty(whereFields)) {
-            return correctionInfo;
+            return;
         }
 
-        whereFields.addAll(SqlParserSelectHelper.getOrderByFields(sql));
+        whereFields.addAll(SqlParserSelectHelper.getOrderByFields(preSql));
         whereFields.removeAll(selectFields);
         whereFields.remove(TimeDimensionEnum.DAY.getName());
         whereFields.remove(TimeDimensionEnum.WEEK.getName());
         whereFields.remove(TimeDimensionEnum.MONTH.getName());
-
+        String replaceFields = SqlParserUpdateHelper.addFieldsToSelect(preSql, new ArrayList<>(whereFields));
-        String replaceFields = SqlParserUpdateHelper.addFieldsToSelect(sql, new ArrayList<>(whereFields));
+        semanticCorrectInfo.setPreSql(preSql);
-        correctionInfo.setSql(replaceFields);
+        semanticCorrectInfo.setSql(replaceFields);
-        return correctionInfo;
     }
 }

chat/core/src/main/java/com/tencent/supersonic/chat/corrector/TableNameCorrector.java

@@ -1,6 +1,6 @@
 package com.tencent.supersonic.chat.corrector;
 
-import com.tencent.supersonic.chat.api.pojo.CorrectionInfo;
+import com.tencent.supersonic.chat.api.pojo.SemanticCorrectInfo;
 import com.tencent.supersonic.common.util.jsqlparser.SqlParserUpdateHelper;
 import lombok.extern.slf4j.Slf4j;
 
@@ -10,12 +10,12 @@ public class TableNameCorrector extends BaseSemanticCorrector {
     public static final String TABLE_PREFIX = "t_";
 
     @Override
-    public CorrectionInfo corrector(CorrectionInfo correctionInfo) {
+    public void correct(SemanticCorrectInfo semanticCorrectInfo) {
-        Long modelId = correctionInfo.getParseInfo().getModelId();
+        Long modelId = semanticCorrectInfo.getParseInfo().getModelId();
-        String sqlOutput = correctionInfo.getSql();
+        String preSql = semanticCorrectInfo.getSql();
-        String replaceTable = SqlParserUpdateHelper.replaceTable(sqlOutput, TABLE_PREFIX + modelId);
+        semanticCorrectInfo.setPreSql(preSql);
-        correctionInfo.setSql(replaceTable);
+        String sql = SqlParserUpdateHelper.replaceTable(preSql, TABLE_PREFIX + modelId);
-        return correctionInfo;
+        semanticCorrectInfo.setSql(sql);
     }
 
 }

chat/core/src/main/java/com/tencent/supersonic/chat/mapper/FuzzyNameMapper.java

@@ -8,6 +8,7 @@ import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
 import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
 import com.tencent.supersonic.chat.api.pojo.SemanticSchema;
 import com.tencent.supersonic.chat.api.pojo.QueryContext;
+import com.tencent.supersonic.chat.config.OptimizationConfig;
 import com.tencent.supersonic.common.util.ContextUtils;
 import com.tencent.supersonic.knowledge.service.SchemaService;
 import com.tencent.supersonic.knowledge.utils.HanlpHelper;
@@ -105,8 +106,9 @@ public class FuzzyNameMapper implements SchemaMapper {
 
     private Double getThreshold(QueryContext queryContext, MapperHelper mapperHelper) {
 
-        Double metricDimensionThresholdConfig = mapperHelper.getMetricDimensionThresholdConfig();
+        OptimizationConfig optimizationConfig = ContextUtils.getBean(OptimizationConfig.class);
-        Double metricDimensionMinThresholdConfig = mapperHelper.getMetricDimensionMinThresholdConfig();
+        Double metricDimensionThresholdConfig = optimizationConfig.getMetricDimensionThresholdConfig();
+        Double metricDimensionMinThresholdConfig = optimizationConfig.getMetricDimensionMinThresholdConfig();
 
         Map<Long, List<SchemaElementMatch>> modelElementMatches = queryContext.getMapInfo()
                 .getModelElementMatches();

chat/core/src/main/java/com/tencent/supersonic/chat/mapper/HanlpDictMapper.java

@@ -9,13 +9,13 @@ import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
 import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
 import com.tencent.supersonic.chat.api.pojo.SchemaMapInfo;
 import com.tencent.supersonic.chat.service.SemanticService;
-import com.tencent.supersonic.knowledge.utils.NatureHelper;
 import com.tencent.supersonic.common.util.ContextUtils;
-import com.tencent.supersonic.knowledge.dictionary.DictWordType;
+import com.tencent.supersonic.common.pojo.enums.DictWordType;
 import com.tencent.supersonic.knowledge.dictionary.MapResult;
 import com.tencent.supersonic.knowledge.dictionary.builder.BaseWordBuilder;
 import com.tencent.supersonic.knowledge.dictionary.builder.WordBuilderFactory;
 import com.tencent.supersonic.knowledge.utils.HanlpHelper;
+import com.tencent.supersonic.knowledge.utils.NatureHelper;
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Map;
@@ -25,6 +25,8 @@ import java.util.Set;
 import java.util.stream.Collectors;
 import lombok.extern.slf4j.Slf4j;
 import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.springframework.beans.BeanUtils;
 
 @Slf4j
 public class HanlpDictMapper implements SchemaMapper {
@@ -83,11 +85,14 @@ public class HanlpDictMapper implements SchemaMapper {
                 Long elementID = baseWordBuilder.getElementID(nature);
                 Long frequency = wordNatureToFrequency.get(mapResult.getName() + nature);
 
-                SchemaElement element = modelSchema.getElement(elementType, elementID);
+                SchemaElement elementDb = modelSchema.getElement(elementType, elementID);
-                if (Objects.isNull(element)) {
+                if (Objects.isNull(elementDb)) {
                     log.info("element is null, elementType:{},elementID:{}", elementType, elementID);
                     continue;
                 }
+                SchemaElement element = new SchemaElement();
+                BeanUtils.copyProperties(elementDb, element);
+                element.setAlias(getAlias(elementDb));
                 if (element.getType().equals(SchemaElementType.VALUE)) {
                     element.setName(mapResult.getName());
                 }
@@ -124,4 +129,16 @@ public class HanlpDictMapper implements SchemaMapper {
         }
         return matches;
     }
+
+    public List<String> getAlias(SchemaElement element) {
+        if (!SchemaElementType.VALUE.equals(element.getType())) {
+            return element.getAlias();
+        }
+        if (CollectionUtils.isNotEmpty(element.getAlias()) && StringUtils.isNotEmpty(element.getName())) {
+            return element.getAlias().stream()
+                    .filter(aliasItem -> aliasItem.contains(element.getName()))
+                    .collect(Collectors.toList());
+        }
+        return element.getAlias();
+    }
 }

chat/core/src/main/java/com/tencent/supersonic/chat/mapper/MapperHelper.java

@@ -2,6 +2,7 @@ package com.tencent.supersonic.chat.mapper;
 
 import com.hankcs.hanlp.algorithm.EditDistance;
 import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
+import com.tencent.supersonic.chat.config.OptimizationConfig;
 import com.tencent.supersonic.chat.service.AgentService;
 import com.tencent.supersonic.common.util.ContextUtils;
 import com.tencent.supersonic.knowledge.utils.NatureHelper;
@@ -13,7 +14,7 @@ import java.util.Set;
 import java.util.stream.Collectors;
 import lombok.Data;
 import lombok.extern.slf4j.Slf4j;
-import org.springframework.beans.factory.annotation.Value;
+import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.stereotype.Service;
 
 /**
@@ -25,17 +26,8 @@ import org.springframework.stereotype.Service;
 @Slf4j
 public class MapperHelper {
 
-    @Value("${one.detection.size:8}")
+    @Autowired
-    private Integer oneDetectionSize;
+    private OptimizationConfig optimizationConfig;
-    @Value("${one.detection.max.size:20}")
-    private Integer oneDetectionMaxSize;
-    @Value("${metric.dimension.threshold:0.3}")
-    private Double metricDimensionThresholdConfig;
-
-    @Value("${metric.dimension.min.threshold:0.3}")
-    private Double metricDimensionMinThresholdConfig;
-    @Value("${dimension.value.threshold:0.5}")
-    private Double dimensionValueThresholdConfig;
 
     public Integer getStepIndex(Map<Integer, Integer> regOffsetToLength, Integer index) {
         Integer subRegLength = regOffsetToLength.get(index);
@@ -58,9 +50,9 @@ public class MapperHelper {
 
     public double getThresholdMatch(List<String> natures) {
         if (existDimensionValues(natures)) {
-            return dimensionValueThresholdConfig;
+            return optimizationConfig.getDimensionValueThresholdConfig();
         }
-        return metricDimensionThresholdConfig;
+        return optimizationConfig.getMetricDimensionThresholdConfig();
     }
 
     /***
@@ -97,9 +89,20 @@ public class MapperHelper {
         AgentService agentService = ContextUtils.getBean(AgentService.class);
 
         Set<Long> detectModelIds = agentService.getDslToolsModelIds(request.getAgentId(), null);
+        //contains all
+        if (agentService.containsAllModel(detectModelIds)) {
+            if (Objects.nonNull(modelId) && modelId > 0) {
+                Set<Long> result = new HashSet<>();
+                result.add(modelId);
+                return result;
+            }
+            return new HashSet<>();
+        }
+
         if (Objects.nonNull(detectModelIds)) {
             detectModelIds = detectModelIds.stream().filter(entry -> entry > 0).collect(Collectors.toSet());
         }
+
         if (Objects.nonNull(modelId) && modelId > 0 && Objects.nonNull(detectModelIds)) {
             if (detectModelIds.contains(modelId)) {
                 Set<Long> result = new HashSet<>();
@@ -109,5 +112,4 @@ public class MapperHelper {
         }
         return detectModelIds;
     }
-
 }

chat/core/src/main/java/com/tencent/supersonic/chat/mapper/QueryMatchStrategy.java

@@ -2,6 +2,7 @@ package com.tencent.supersonic.chat.mapper;
 
 import com.hankcs.hanlp.seg.common.Term;
 import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
+import com.tencent.supersonic.chat.config.OptimizationConfig;
 import com.tencent.supersonic.common.pojo.Constants;
 import com.tencent.supersonic.knowledge.dictionary.MapResult;
 import com.tencent.supersonic.knowledge.service.SearchService;
@@ -31,6 +32,9 @@ public class QueryMatchStrategy implements MatchStrategy {
     @Autowired
     private MapperHelper mapperHelper;
 
+    @Autowired
+    private OptimizationConfig optimizationConfig;
+
     @Override
     public Map<MatchText, List<MapResult>> match(QueryReq queryReq, List<Term> terms, Set<Long> detectModelIds) {
         String text = queryReq.getQueryText();
@@ -111,7 +115,7 @@ public class QueryMatchStrategy implements MatchStrategy {
         String detectSegment = text.substring(index, i);
 
         // step1. pre search
-        Integer oneDetectionMaxSize = mapperHelper.getOneDetectionMaxSize();
+        Integer oneDetectionMaxSize = optimizationConfig.getOneDetectionMaxSize();
         LinkedHashSet<MapResult> mapResults = SearchService.prefixSearch(detectSegment, oneDetectionMaxSize, agentId,
                 detectModelIds).stream().collect(Collectors.toCollection(LinkedHashSet::new));
         // step2. suffix search
@@ -153,7 +157,7 @@ public class QueryMatchStrategy implements MatchStrategy {
         if (CollectionUtils.isNotEmpty(dimensionMetrics)) {
             return dimensionMetrics;
         } else {
-            return mapResults.stream().limit(mapperHelper.getOneDetectionSize()).collect(Collectors.toList());
+            return mapResults.stream().limit(optimizationConfig.getOneDetectionSize()).collect(Collectors.toList());
         }
     }
 }

chat/core/src/main/java/com/tencent/supersonic/chat/mapper/SearchMatchStrategy.java

@@ -3,7 +3,7 @@ package com.tencent.supersonic.chat.mapper;
 import com.google.common.collect.Lists;
 import com.hankcs.hanlp.seg.common.Term;
 import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
-import com.tencent.supersonic.knowledge.dictionary.DictWordType;
+import com.tencent.supersonic.common.pojo.enums.DictWordType;
 import com.tencent.supersonic.knowledge.dictionary.MapResult;
 import com.tencent.supersonic.knowledge.service.SearchService;
 import java.util.List;

chat/core/src/main/java/com/tencent/supersonic/chat/parser/SatisfactionChecker.java

@@ -4,7 +4,9 @@ package com.tencent.supersonic.chat.parser;
 import com.tencent.supersonic.chat.api.component.SemanticQuery;
 import com.tencent.supersonic.chat.api.pojo.QueryContext;
 import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
+import com.tencent.supersonic.chat.config.OptimizationConfig;
 import com.tencent.supersonic.chat.query.llm.dsl.DslQuery;
+import com.tencent.supersonic.common.util.ContextUtils;
 import lombok.extern.slf4j.Slf4j;
 
 /**
@@ -15,10 +17,6 @@ import lombok.extern.slf4j.Slf4j;
 @Slf4j
 public class SatisfactionChecker {
 
-    private static final double LONG_TEXT_THRESHOLD = 0.8;
-    private static final double SHORT_TEXT_THRESHOLD = 0.5;
-    private static final int QUERY_TEXT_LENGTH_THRESHOLD = 10;
-
     // check all the parse info in candidate
     public static boolean check(QueryContext queryContext) {
         for (SemanticQuery query : queryContext.getCandidateQueries()) {
@@ -35,11 +33,12 @@ public class SatisfactionChecker {
     private static boolean checkThreshold(String queryText, SemanticParseInfo semanticParseInfo) {
         int queryTextLength = queryText.replaceAll(" ", "").length();
         double degree = semanticParseInfo.getScore() / queryTextLength;
-        if (queryTextLength > QUERY_TEXT_LENGTH_THRESHOLD) {
+        OptimizationConfig optimizationConfig = ContextUtils.getBean(OptimizationConfig.class);
-            if (degree < LONG_TEXT_THRESHOLD) {
+        if (queryTextLength > optimizationConfig.getQueryTextLengthThreshold()) {
+            if (degree < optimizationConfig.getLongTextThreshold()) {
                 return false;
             }
-        } else if (degree < SHORT_TEXT_THRESHOLD) {
+        } else if (degree < optimizationConfig.getShortTextThreshold()) {
             return false;
         }
         log.info("queryMode:{}, degree:{}, parse info:{}",

chat/core/src/main/java/com/tencent/supersonic/chat/parser/llm/dsl/DSLDateHelper.java

@@ -1,11 +1,55 @@
 package com.tencent.supersonic.chat.parser.llm.dsl;
 
+import com.tencent.supersonic.chat.api.pojo.request.ChatConfigFilter;
+import com.tencent.supersonic.chat.api.pojo.request.ChatDefaultConfigReq;
+import com.tencent.supersonic.chat.api.pojo.response.ChatConfigResp;
+import com.tencent.supersonic.chat.service.ConfigService;
+import com.tencent.supersonic.common.util.ContextUtils;
+import com.tencent.supersonic.common.util.DatePeriodEnum;
 import com.tencent.supersonic.common.util.DateUtils;
+import java.util.List;
+import java.util.Objects;
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.lang3.StringUtils;
 
 public class DSLDateHelper {
 
-    public static String getCurrentDate(Long modelId) {
+    public static String getReferenceDate(Long modelId) {
-        return DateUtils.getBeforeDate(4);
+        String chatDetailDate = getChatDetailDate(modelId);
+        if (StringUtils.isNotBlank(chatDetailDate)) {
+            return chatDetailDate;
+        }
+        return DateUtils.getBeforeDate(0);
+    }
 
+    private static String getChatDetailDate(Long modelId) {
+        if (Objects.isNull(modelId)) {
+            return null;
+        }
+        ChatConfigFilter filter = new ChatConfigFilter();
+        filter.setModelId(modelId);
+
+        List<ChatConfigResp> configResps = ContextUtils.getBean(ConfigService.class).search(filter, null);
+        if (CollectionUtils.isEmpty(configResps)) {
+            return null;
+        }
+        ChatConfigResp chatConfigResp = configResps.get(0);
+        if (Objects.isNull(chatConfigResp.getChatDetailConfig()) || Objects.isNull(
+                chatConfigResp.getChatDetailConfig().getChatDefaultConfig())) {
+            return null;
+        }
+
+        ChatDefaultConfigReq chatDefaultConfig = chatConfigResp.getChatDetailConfig().getChatDefaultConfig();
+        Integer unit = chatDefaultConfig.getUnit();
+        String period = chatDefaultConfig.getPeriod();
+        if (Objects.nonNull(unit)) {
+            DatePeriodEnum datePeriodEnum = DatePeriodEnum.get(period);
+            if (Objects.isNull(datePeriodEnum)) {
+                return DateUtils.getBeforeDate(unit);
+            } else {
+                return DateUtils.getBeforeDate(unit, datePeriodEnum);
+            }
+        }
+        return null;
     }
 }

chat/core/src/main/java/com/tencent/supersonic/chat/parser/llm/dsl/LLMDslParser.java

@@ -6,11 +6,11 @@ import com.tencent.supersonic.chat.agent.tool.DslTool;
 import com.tencent.supersonic.chat.api.component.SemanticCorrector;
 import com.tencent.supersonic.chat.api.component.SemanticParser;
 import com.tencent.supersonic.chat.api.pojo.ChatContext;
-import com.tencent.supersonic.chat.api.pojo.CorrectionInfo;
 import com.tencent.supersonic.chat.api.pojo.QueryContext;
 import com.tencent.supersonic.chat.api.pojo.SchemaElement;
 import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
 import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
+import com.tencent.supersonic.chat.api.pojo.SemanticCorrectInfo;
 import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
 import com.tencent.supersonic.chat.api.pojo.SemanticSchema;
 import com.tencent.supersonic.chat.api.pojo.request.QueryFilter;
@@ -40,6 +40,7 @@ import com.tencent.supersonic.semantic.api.query.enums.FilterOperatorEnum;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.HashMap;
+import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Objects;
@@ -66,8 +67,12 @@ public class LLMDslParser implements SemanticParser {
     public void parse(QueryContext queryCtx, ChatContext chatCtx) {
         QueryReq request = queryCtx.getRequest();
         LLMConfig llmConfig = ContextUtils.getBean(LLMConfig.class);
-        if (StringUtils.isEmpty(llmConfig.getUrl()) || SatisfactionChecker.check(queryCtx)) {
+        if (StringUtils.isEmpty(llmConfig.getUrl())) {
-            log.info("llmConfig:{}, skip function parser, queryText:{}", llmConfig, request.getQueryText());
+            log.info("llm url is empty, skip dsl parser, llmConfig:{}", llmConfig);
+            return;
+        }
+        if (SatisfactionChecker.check(queryCtx)) {
+            log.info("skip dsl parser, queryText:{}", request.getQueryText());
             return;
         }
         try {
@@ -88,32 +93,51 @@ public class LLMDslParser implements SemanticParser {
             if (Objects.isNull(llmResp)) {
                 return;
             }
-            DSLParseResult dslParseResult = DSLParseResult.builder().request(request).dslTool(dslTool).llmReq(llmReq)
+            DSLParseResult dslParseResult = DSLParseResult.builder().request(request)
-                    .llmResp(llmResp).build();
+                    .dslTool(dslTool).llmReq(llmReq).llmResp(llmResp).build();
 
             SemanticParseInfo parseInfo = getParseInfo(queryCtx, modelId, dslTool, dslParseResult);
 
-            String correctorSql = getCorrectorSql(queryCtx, parseInfo, llmResp.getSqlOutput());
+            SemanticCorrectInfo semanticCorrectInfo = getCorrectorSql(queryCtx, parseInfo, llmResp.getSqlOutput());
 
-            llmResp.setCorrectorSql(correctorSql);
+            llmResp.setCorrectorSql(semanticCorrectInfo.getSql());
 
-            setFilter(correctorSql, modelId, parseInfo);
+            updateParseInfo(semanticCorrectInfo, modelId, parseInfo);
 
         } catch (Exception e) {
             log.error("LLMDSLParser error", e);
         }
     }
 
-    public void setFilter(String correctorSql, Long modelId, SemanticParseInfo parseInfo) {
+    private Set<SchemaElement> getElements(Long modelId, List<String> allFields, List<SchemaElement> elements) {
-
+        return elements.stream()
-        List<FilterExpression> expressions = SqlParserSelectHelper.getFilterExpression(correctorSql);
+                .filter(schemaElement -> modelId.equals(schemaElement.getModel())
-        if (CollectionUtils.isEmpty(expressions)) {
+                        && allFields.contains(schemaElement.getBizName())
-            return;
+                ).collect(Collectors.toSet());
     }
+
+    private List<String> getFieldsExceptDate(List<String> allFields) {
+        if (CollectionUtils.isEmpty(allFields)) {
+            return new ArrayList<>();
+        }
+        return allFields.stream()
+                .filter(entry -> !TimeDimensionEnum.getNameList().contains(entry))
+                .collect(Collectors.toList());
+    }
+
+    public void updateParseInfo(SemanticCorrectInfo semanticCorrectInfo, Long modelId, SemanticParseInfo parseInfo) {
+
+        String correctorSql = semanticCorrectInfo.getPreSql();
+        if (StringUtils.isEmpty(correctorSql)) {
+            correctorSql = semanticCorrectInfo.getSql();
+        }
+        List<FilterExpression> expressions = SqlParserSelectHelper.getFilterExpression(correctorSql);
         //set dataInfo
         try {
+            if (!CollectionUtils.isEmpty(expressions)) {
                 DateConf dateInfo = getDateInfo(expressions);
                 parseInfo.setDateInfo(dateInfo);
+            }
         } catch (Exception e) {
             log.error("set dateInfo error :", e);
         }
@@ -126,6 +150,28 @@ public class LLMDslParser implements SemanticParser {
         } catch (Exception e) {
             log.error("set dimensionFilter error :", e);
         }
+
+        SemanticSchema semanticSchema = ContextUtils.getBean(SchemaService.class).getSemanticSchema();
+
+        if (Objects.isNull(semanticSchema)) {
+            return;
+        }
+        List<String> allFields = getFieldsExceptDate(SqlParserSelectHelper.getAllFields(semanticCorrectInfo.getSql()));
+
+        Set<SchemaElement> metrics = getElements(modelId, allFields, semanticSchema.getMetrics());
+        parseInfo.setMetrics(metrics);
+
+        if (SqlParserSelectHelper.hasAggregateFunction(semanticCorrectInfo.getSql())) {
+            parseInfo.setNativeQuery(false);
+            List<String> groupByFields = SqlParserSelectHelper.getGroupByFields(semanticCorrectInfo.getSql());
+            List<String> groupByDimensions = getFieldsExceptDate(groupByFields);
+            parseInfo.setDimensions(getElements(modelId, groupByDimensions, semanticSchema.getDimensions()));
+        } else {
+            parseInfo.setNativeQuery(true);
+            List<String> selectFields = SqlParserSelectHelper.getSelectFields(semanticCorrectInfo.getSql());
+            List<String> selectDimensions = getFieldsExceptDate(selectFields);
+            parseInfo.setDimensions(getElements(modelId, selectDimensions, semanticSchema.getDimensions()));
+        }
     }
 
     private List<QueryFilter> getDimensionFilter(Map<String, SchemaElement> bizNameToElement,
@@ -200,9 +246,9 @@ public class LLMDslParser implements SemanticParser {
         return dateExpressions.size() > 1 && Objects.nonNull(dateExpressions.get(1).getFieldValue());
     }
 
-    private String getCorrectorSql(QueryContext queryCtx, SemanticParseInfo parseInfo, String sql) {
+    private SemanticCorrectInfo getCorrectorSql(QueryContext queryCtx, SemanticParseInfo parseInfo, String sql) {
 
-        CorrectionInfo correctionInfo = CorrectionInfo.builder()
+        SemanticCorrectInfo correctInfo = SemanticCorrectInfo.builder()
                 .queryFilters(queryCtx.getRequest().getQueryFilters()).sql(sql)
                 .parseInfo(parseInfo).build();
 
@@ -210,14 +256,13 @@ public class LLMDslParser implements SemanticParser {
 
         dslCorrections.forEach(dslCorrection -> {
             try {
-                dslCorrection.corrector(correctionInfo);
+                dslCorrection.correct(correctInfo);
-                log.info("sqlCorrection:{} sql:{}", dslCorrection.getClass().getSimpleName(),
+                log.info("sqlCorrection:{} sql:{}", dslCorrection.getClass().getSimpleName(), correctInfo.getSql());
-                        correctionInfo.getSql());
             } catch (Exception e) {
-                log.error("sqlCorrection:{} execute error,correctionInfo:{}", dslCorrection, correctionInfo, e);
+                log.error("sqlCorrection:{} correct error,correctInfo:{}", dslCorrection, correctInfo, e);
             }
         });
-        return correctionInfo.getSql();
+        return correctInfo;
     }
 
     private SemanticParseInfo getParseInfo(QueryContext queryCtx, Long modelId, DslTool dslTool,
@@ -250,7 +295,14 @@ public class LLMDslParser implements SemanticParser {
     private DslTool getDslTool(QueryReq request, Long modelId) {
         AgentService agentService = ContextUtils.getBean(AgentService.class);
         List<DslTool> dslTools = agentService.getDslTools(request.getAgentId(), AgentToolType.DSL);
-        Optional<DslTool> dslToolOptional = dslTools.stream().filter(tool -> tool.getModelIds().contains(modelId))
+        Optional<DslTool> dslToolOptional = dslTools.stream()
+                .filter(tool -> {
+                    List<Long> modelIds = tool.getModelIds();
+                    if (agentService.containsAllModel(new HashSet<>(modelIds))) {
+                        return true;
+                    }
+                    return modelIds.contains(modelId);
+                })
                 .findFirst();
         return dslToolOptional.orElse(null);
     }
@@ -258,6 +310,9 @@ public class LLMDslParser implements SemanticParser {
     private Long getModelId(QueryContext queryCtx, ChatContext chatCtx, Integer agentId) {
         AgentService agentService = ContextUtils.getBean(AgentService.class);
         Set<Long> distinctModelIds = agentService.getDslToolsModelIds(agentId, AgentToolType.DSL);
+        if (agentService.containsAllModel(distinctModelIds)) {
+            distinctModelIds = new HashSet<>();
+        }
         ModelResolver modelResolver = ComponentFactory.getModelResolver();
         Long modelId = modelResolver.resolve(queryCtx, chatCtx, distinctModelIds);
         log.info("resolve modelId:{},dslModels:{}", modelId, distinctModelIds);
@@ -301,12 +356,12 @@ public class LLMDslParser implements SemanticParser {
         List<ElementValue> linking = new ArrayList<>();
         linking.addAll(getValueList(queryCtx, modelId, semanticSchema));
         llmReq.setLinking(linking);
-        String currentDate = DSLDateHelper.getCurrentDate(modelId);
+        String currentDate = DSLDateHelper.getReferenceDate(modelId);
         llmReq.setCurrentDate(currentDate);
         return llmReq;
     }
 
-    private List<ElementValue> getValueList(QueryContext queryCtx, Long modelId, SemanticSchema semanticSchema) {
+    protected List<ElementValue> getValueList(QueryContext queryCtx, Long modelId, SemanticSchema semanticSchema) {
         Map<Long, String> itemIdToName = getItemIdToName(modelId, semanticSchema);
 
         List<SchemaElementMatch> matchedElements = queryCtx.getMapInfo().getMatchedElements(modelId);
@@ -344,7 +399,7 @@ public class LLMDslParser implements SemanticParser {
     }
 
 
-    private List<String> getFieldNameList(QueryContext queryCtx, Long modelId, SemanticSchema semanticSchema) {
+    protected List<String> getFieldNameList(QueryContext queryCtx, Long modelId, SemanticSchema semanticSchema) {
         Map<Long, String> itemIdToName = getItemIdToName(modelId, semanticSchema);
 
         List<SchemaElementMatch> matchedElements = queryCtx.getMapInfo().getMatchedElements(modelId);
@@ -371,7 +426,7 @@ public class LLMDslParser implements SemanticParser {
         return new ArrayList<>(fieldNameList);
     }
 
-    private Map<Long, String> getItemIdToName(Long modelId, SemanticSchema semanticSchema) {
+    protected Map<Long, String> getItemIdToName(Long modelId, SemanticSchema semanticSchema) {
         return semanticSchema.getDimensions().stream()
                 .filter(entry -> modelId.equals(entry.getModel()))
                 .collect(Collectors.toMap(SchemaElement::getId, SchemaElement::getName, (value1, value2) -> value2));

chat/core/src/main/java/com/tencent/supersonic/chat/parser/llm/interpret/MetricInterpretParser.java

@@ -17,7 +17,7 @@ import com.tencent.supersonic.chat.api.pojo.ModelSchema;
 import com.tencent.supersonic.chat.api.pojo.request.QueryFilter;
 import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
 import com.tencent.supersonic.chat.parser.SatisfactionChecker;
-import com.tencent.supersonic.chat.query.metricinterpret.MetricInterpretQuery;
+import com.tencent.supersonic.chat.query.llm.interpret.MetricInterpretQuery;
 import com.tencent.supersonic.chat.query.QueryManager;
 import com.tencent.supersonic.chat.query.plugin.PluginSemanticQuery;
 import com.tencent.supersonic.chat.service.AgentService;

chat/core/src/main/java/com/tencent/supersonic/chat/parser/plugin/PluginParser.java

@@ -0,0 +1,114 @@
+package com.tencent.supersonic.chat.parser.plugin;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+import com.tencent.supersonic.chat.api.component.SemanticParser;
+import com.tencent.supersonic.chat.api.pojo.ChatContext;
+import com.tencent.supersonic.chat.api.pojo.QueryContext;
+import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
+import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
+import com.tencent.supersonic.chat.api.pojo.request.QueryFilter;
+import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
+import com.tencent.supersonic.chat.api.pojo.SchemaElement;
+import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
+import com.tencent.supersonic.chat.plugin.Plugin;
+import com.tencent.supersonic.chat.plugin.PluginManager;
+import com.tencent.supersonic.chat.plugin.PluginParseResult;
+import com.tencent.supersonic.chat.plugin.PluginRecallResult;
+import com.tencent.supersonic.chat.query.QueryManager;
+import com.tencent.supersonic.chat.query.plugin.PluginSemanticQuery;
+import com.tencent.supersonic.common.pojo.Constants;
+import com.tencent.supersonic.semantic.api.query.enums.FilterOperatorEnum;
+import org.springframework.util.CollectionUtils;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+public abstract class PluginParser implements SemanticParser {
+
+    @Override
+    public void parse(QueryContext queryContext, ChatContext chatContext) {
+        if (!checkPreCondition(queryContext)) {
+            return;
+        }
+        PluginRecallResult pluginRecallResult = recallPlugin(queryContext);
+        if (pluginRecallResult == null) {
+            return;
+        }
+        buildQuery(queryContext, pluginRecallResult);
+    }
+
+    public abstract boolean checkPreCondition(QueryContext queryContext);
+
+    public abstract PluginRecallResult recallPlugin(QueryContext queryContext);
+
+    public void buildQuery(QueryContext queryContext, PluginRecallResult pluginRecallResult) {
+        Plugin plugin = pluginRecallResult.getPlugin();
+        Set<Long> modelIds = pluginRecallResult.getModelIds();
+        if (plugin.isContainsAllModel()) {
+            modelIds = Sets.newHashSet(-1L);
+        }
+        for (Long modelId : modelIds) {
+            PluginSemanticQuery pluginQuery = QueryManager.createPluginQuery(plugin.getType());
+            SemanticParseInfo semanticParseInfo = buildSemanticParseInfo(modelId, plugin, queryContext.getRequest(),
+                    queryContext.getMapInfo().getMatchedElements(modelId), pluginRecallResult.getDistance());
+            semanticParseInfo.setQueryMode(pluginQuery.getQueryMode());
+            semanticParseInfo.setScore(pluginRecallResult.getScore());
+            pluginQuery.setParseInfo(semanticParseInfo);
+            queryContext.getCandidateQueries().add(pluginQuery);
+        }
+    }
+
+    protected List<Plugin> getPluginList(QueryContext queryContext) {
+        return PluginManager.getPluginAgentCanSupport(queryContext.getRequest().getAgentId());
+    }
+
+    protected SemanticParseInfo buildSemanticParseInfo(Long modelId, Plugin plugin, QueryReq queryReq,
+                                                       List<SchemaElementMatch> schemaElementMatches, double distance) {
+        if (modelId == null && !CollectionUtils.isEmpty(plugin.getModelList())) {
+            modelId = plugin.getModelList().get(0);
+        }
+        if (schemaElementMatches == null) {
+            schemaElementMatches = Lists.newArrayList();
+        }
+        SchemaElement model = new SchemaElement();
+        model.setModel(modelId);
+        model.setId(modelId);
+        SemanticParseInfo semanticParseInfo = new SemanticParseInfo();
+        semanticParseInfo.setElementMatches(schemaElementMatches);
+        semanticParseInfo.setModel(model);
+        Map<String, Object> properties = new HashMap<>();
+        PluginParseResult pluginParseResult = new PluginParseResult();
+        pluginParseResult.setPlugin(plugin);
+        pluginParseResult.setRequest(queryReq);
+        pluginParseResult.setDistance(distance);
+        properties.put(Constants.CONTEXT, pluginParseResult);
+        properties.put("type", "plugin");
+        properties.put("name", plugin.getName());
+        semanticParseInfo.setProperties(properties);
+        semanticParseInfo.setScore(distance);
+        fillSemanticParseInfo(semanticParseInfo);
+        return semanticParseInfo;
+    }
+
+
+    private void fillSemanticParseInfo(SemanticParseInfo semanticParseInfo) {
+        List<SchemaElementMatch> schemaElementMatches = semanticParseInfo.getElementMatches();
+        if (CollectionUtils.isEmpty(schemaElementMatches)) {
+            return;
+        }
+        schemaElementMatches.stream().filter(schemaElementMatch ->
+                        SchemaElementType.VALUE.equals(schemaElementMatch.getElement().getType())
+                                || SchemaElementType.ID.equals(schemaElementMatch.getElement().getType()))
+                .forEach(schemaElementMatch -> {
+                    QueryFilter queryFilter = new QueryFilter();
+                    queryFilter.setValue(schemaElementMatch.getWord());
+                    queryFilter.setElementID(schemaElementMatch.getElement().getId());
+                    queryFilter.setName(schemaElementMatch.getElement().getName());
+                    queryFilter.setOperator(FilterOperatorEnum.EQUALS);
+                    queryFilter.setBizName(schemaElementMatch.getElement().getBizName());
+                    semanticParseInfo.getDimensionFilters().add(queryFilter);
+                });
+    }
+}

chat/core/src/main/java/com/tencent/supersonic/chat/parser/plugin/embedding/EmbeddingBasedParser.java

@@ -1,63 +1,58 @@
 package com.tencent.supersonic.chat.parser.plugin.embedding;
 
 import com.google.common.collect.Lists;
-import com.tencent.supersonic.chat.api.component.SemanticParser;
+import com.tencent.supersonic.chat.api.component.SemanticQuery;
 import com.tencent.supersonic.chat.api.pojo.QueryContext;
-import com.tencent.supersonic.chat.api.pojo.SchemaElement;
-import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
-import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
-import com.tencent.supersonic.chat.api.pojo.ChatContext;
-import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
-import com.tencent.supersonic.chat.api.pojo.ModelSchema;
-import com.tencent.supersonic.chat.api.pojo.request.QueryFilter;
-import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
 import com.tencent.supersonic.chat.parser.ParseMode;
+import com.tencent.supersonic.chat.parser.plugin.PluginParser;
 import com.tencent.supersonic.chat.plugin.Plugin;
 import com.tencent.supersonic.chat.plugin.PluginManager;
-import com.tencent.supersonic.chat.plugin.PluginParseResult;
+import com.tencent.supersonic.chat.plugin.PluginRecallResult;
-import com.tencent.supersonic.chat.query.QueryManager;
-import com.tencent.supersonic.chat.query.llm.dsl.DslQuery;
-import com.tencent.supersonic.chat.query.plugin.PluginSemanticQuery;
-import com.tencent.supersonic.chat.service.SemanticService;
-import com.tencent.supersonic.common.pojo.Constants;
 import com.tencent.supersonic.common.util.ContextUtils;
+import java.util.Comparator;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
-import java.util.HashMap;
-import java.util.Comparator;
 import java.util.stream.Collectors;
-import com.tencent.supersonic.semantic.api.query.enums.FilterOperatorEnum;
 import lombok.extern.slf4j.Slf4j;
 import org.apache.commons.lang3.StringUtils;
 import org.apache.commons.lang3.tuple.Pair;
 import org.springframework.util.CollectionUtils;
 
 @Slf4j
-public class EmbeddingBasedParser implements SemanticParser {
+public class EmbeddingBasedParser extends PluginParser {
 
     @Override
-    public void parse(QueryContext queryContext, ChatContext chatContext) {
+    public boolean checkPreCondition(QueryContext queryContext) {
         EmbeddingConfig embeddingConfig = ContextUtils.getBean(EmbeddingConfig.class);
         if (StringUtils.isBlank(embeddingConfig.getUrl())) {
-            return;
+            return false;
         }
-        log.info("EmbeddingBasedParser parser query ctx: {}, chat ctx: {}", queryContext, chatContext);
+        List<Plugin> plugins = getPluginList(queryContext);
-        String text = queryContext.getRequest().getQueryText();
+        if (CollectionUtils.isEmpty(plugins)) {
-        List<RecallRetrieval> embeddingRetrievals = recallResult(text);
+            return false;
-        choosePlugin(embeddingRetrievals, queryContext);
+        }
+        List<SemanticQuery> semanticQueries = queryContext.getCandidateQueries();
+        for (SemanticQuery semanticQuery : semanticQueries) {
+            if (queryContext.getRequest().getQueryText().length() <= semanticQuery.getParseInfo().getScore()) {
+                return false;
+            }
+        }
+        return true;
     }
 
-    private void choosePlugin(List<RecallRetrieval> embeddingRetrievals,
+    @Override
-            QueryContext queryContext) {
+    public PluginRecallResult recallPlugin(QueryContext queryContext) {
+        String text = queryContext.getRequest().getQueryText();
+        List<RecallRetrieval> embeddingRetrievals = embeddingRecall(text);
         if (CollectionUtils.isEmpty(embeddingRetrievals)) {
-            return;
+            return null;
         }
         List<Plugin> plugins = getPluginList(queryContext);
         Map<Long, Plugin> pluginMap = plugins.stream().collect(Collectors.toMap(Plugin::getId, p -> p));
         for (RecallRetrieval embeddingRetrieval : embeddingRetrievals) {
             Plugin plugin = pluginMap.get(Long.parseLong(embeddingRetrieval.getId()));
-            if (plugin == null || DslQuery.QUERY_MODE.equalsIgnoreCase(plugin.getType())) {
+            if (plugin == null) {
                 continue;
             }
             Pair<Boolean, Set<Long>> pair = PluginManager.resolve(plugin, queryContext);
@@ -65,69 +60,19 @@ public class EmbeddingBasedParser implements SemanticParser {
             if (pair.getLeft()) {
                 Set<Long> modelList = pair.getRight();
                 if (CollectionUtils.isEmpty(modelList)) {
-                    return;
+                    continue;
                 }
-                for (Long modelId : modelList) {
-                    buildQuery(plugin, Double.parseDouble(embeddingRetrieval.getDistance()), modelId, queryContext,
-                            queryContext.getMapInfo().getMatchedElements(modelId));
-                    if (plugin.isContainsAllModel()) {
-                        break;
-                    }
-                }
-                return;
-            }
-        }
-    }
-
-    private void buildQuery(Plugin plugin, double distance, Long modelId,
-            QueryContext queryContext, List<SchemaElementMatch> schemaElementMatches) {
-        log.info("EmbeddingBasedParser Model: {} choose plugin: [{} {}]", modelId, plugin.getId(), plugin.getName());
-        PluginSemanticQuery pluginQuery = QueryManager.createPluginQuery(plugin.getType());
                 plugin.setParseMode(ParseMode.EMBEDDING_RECALL);
-        SemanticParseInfo semanticParseInfo = buildSemanticParseInfo(modelId, plugin, queryContext.getRequest(),
+                double distance = Double.parseDouble(embeddingRetrieval.getDistance());
-                schemaElementMatches, distance);
                 double score = queryContext.getRequest().getQueryText().length() * (1 - distance);
-        semanticParseInfo.setQueryMode(pluginQuery.getQueryMode());
+                return PluginRecallResult.builder()
-        semanticParseInfo.setScore(score);
+                        .plugin(plugin).modelIds(modelList).score(score).distance(distance).build();
-        pluginQuery.setParseInfo(semanticParseInfo);
+            }
-        queryContext.getCandidateQueries().add(pluginQuery);
+        }
+        return null;
     }
 
-    private SemanticParseInfo buildSemanticParseInfo(Long modelId, Plugin plugin, QueryReq queryReq,
+    public List<RecallRetrieval> embeddingRecall(String embeddingText) {
-            List<SchemaElementMatch> schemaElementMatches, double distance) {
-        if (modelId == null && !CollectionUtils.isEmpty(plugin.getModelList())) {
-            modelId = plugin.getModelList().get(0);
-        }
-        SchemaElement model = new SchemaElement();
-        model.setModel(modelId);
-        model.setId(modelId);
-        SemanticParseInfo semanticParseInfo = new SemanticParseInfo();
-        semanticParseInfo.setElementMatches(schemaElementMatches);
-        semanticParseInfo.setModel(model);
-        Map<String, Object> properties = new HashMap<>();
-        PluginParseResult pluginParseResult = new PluginParseResult();
-        pluginParseResult.setPlugin(plugin);
-        pluginParseResult.setRequest(queryReq);
-        pluginParseResult.setDistance(distance);
-        properties.put(Constants.CONTEXT, pluginParseResult);
-        properties.put("type", "plugin");
-        properties.put("name", plugin.getName());
-        semanticParseInfo.setProperties(properties);
-        semanticParseInfo.setScore(distance);
-        fillSemanticParseInfo(semanticParseInfo);
-        setEntity(modelId, semanticParseInfo);
-        return semanticParseInfo;
-    }
-
-    private void setEntity(Long modelId, SemanticParseInfo semanticParseInfo) {
-        SemanticService semanticService = ContextUtils.getBean(SemanticService.class);
-        ModelSchema modelSchema = semanticService.getModelSchema(modelId);
-        if (modelSchema != null && modelSchema.getEntity() != null) {
-            semanticParseInfo.setEntity(modelSchema.getEntity());
-        }
-    }
-
-    public List<RecallRetrieval> recallResult(String embeddingText) {
         try {
             PluginManager pluginManager = ContextUtils.getBean(PluginManager.class);
             EmbeddingResp embeddingResp = pluginManager.recognize(embeddingText);
@@ -144,26 +89,4 @@ public class EmbeddingBasedParser implements SemanticParser {
         return Lists.newArrayList();
     }
 
-    private void fillSemanticParseInfo(SemanticParseInfo semanticParseInfo) {
-        List<SchemaElementMatch> schemaElementMatches = semanticParseInfo.getElementMatches();
-        if (!CollectionUtils.isEmpty(schemaElementMatches)) {
-            schemaElementMatches.stream().filter(schemaElementMatch ->
-                            SchemaElementType.VALUE.equals(schemaElementMatch.getElement().getType())
-                                    || SchemaElementType.ID.equals(schemaElementMatch.getElement().getType()))
-                    .forEach(schemaElementMatch -> {
-                        QueryFilter queryFilter = new QueryFilter();
-                        queryFilter.setValue(schemaElementMatch.getWord());
-                        queryFilter.setElementID(schemaElementMatch.getElement().getId());
-                        queryFilter.setName(schemaElementMatch.getElement().getName());
-                        queryFilter.setOperator(FilterOperatorEnum.EQUALS);
-                        queryFilter.setBizName(schemaElementMatch.getElement().getBizName());
-                        semanticParseInfo.getDimensionFilters().add(queryFilter);
-                    });
-        }
-    }
-
-    protected List<Plugin> getPluginList(QueryContext queryContext) {
-        return PluginManager.getPluginAgentCanSupport(queryContext.getRequest().getAgentId());
-    }
-
 }

chat/core/src/main/java/com/tencent/supersonic/chat/parser/plugin/embedding/EmbeddingConfig.java

@@ -11,13 +11,13 @@ public class EmbeddingConfig {
     @Value("${embedding.url:}")
     private String url;
 
-    @Value("${embedding.recognize.path:preset_query_retrival}")
+    @Value("${embedding.recognize.path:/preset_query_retrival}")
     private String recognizePath;
 
-    @Value("${embedding.delete.path:preset_delete_by_ids}")
+    @Value("${embedding.delete.path:/preset_delete_by_ids}")
     private String deletePath;
 
-    @Value("${embedding.add.path:preset_query_add}")
+    @Value("${embedding.add.path:/preset_query_add}")
     private String addPath;
 
     @Value("${embedding.nResult:1}")

chat/core/src/main/java/com/tencent/supersonic/chat/parser/plugin/embedding/EmbeddingResp.java

@@ -1,9 +1,10 @@
 package com.tencent.supersonic.chat.parser.plugin.embedding;
 
 
-import java.util.List;
 import lombok.Data;
 
+import java.util.List;
+
 @Data
 public class EmbeddingResp {
 

chat/core/src/main/java/com/tencent/supersonic/chat/parser/plugin/function/FunctionBasedParser.java

@@ -1,32 +1,23 @@
 package com.tencent.supersonic.chat.parser.plugin.function;
 
 import com.alibaba.fastjson.JSON;
-import com.tencent.supersonic.chat.api.component.SemanticParser;
-import com.tencent.supersonic.chat.api.pojo.ChatContext;
 import com.tencent.supersonic.chat.api.pojo.QueryContext;
-import com.tencent.supersonic.chat.api.pojo.SchemaElement;
-import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
 import com.tencent.supersonic.chat.config.FunctionCallInfoConfig;
 import com.tencent.supersonic.chat.parser.ParseMode;
 import com.tencent.supersonic.chat.parser.SatisfactionChecker;
+import com.tencent.supersonic.chat.parser.plugin.PluginParser;
 import com.tencent.supersonic.chat.plugin.Plugin;
 import com.tencent.supersonic.chat.plugin.PluginManager;
 import com.tencent.supersonic.chat.plugin.PluginParseConfig;
-import com.tencent.supersonic.chat.plugin.PluginParseResult;
+import com.tencent.supersonic.chat.plugin.PluginRecallResult;
-import com.tencent.supersonic.chat.query.QueryManager;
-import com.tencent.supersonic.chat.query.plugin.PluginSemanticQuery;
 import com.tencent.supersonic.chat.query.llm.dsl.DslQuery;
 import com.tencent.supersonic.chat.service.PluginService;
-import com.tencent.supersonic.chat.utils.ComponentFactory;
-import com.tencent.supersonic.common.pojo.Constants;
 import com.tencent.supersonic.common.util.ContextUtils;
 import java.net.URI;
 import java.util.List;
 import java.util.Optional;
 import java.util.Set;
 import java.util.Objects;
-import java.util.Map;
-import java.util.HashMap;
 import java.util.stream.Collectors;
 import com.tencent.supersonic.common.util.JsonUtil;
 import lombok.extern.slf4j.Slf4j;
@@ -42,85 +33,73 @@ import org.springframework.web.client.RestTemplate;
 import org.springframework.web.util.UriComponentsBuilder;
 
 @Slf4j
-public class FunctionBasedParser implements SemanticParser {
+public class FunctionBasedParser extends PluginParser {
 
     @Override
-    public void parse(QueryContext queryCtx, ChatContext chatCtx) {
+    public boolean checkPreCondition(QueryContext queryContext) {
         FunctionCallInfoConfig functionCallConfig = ContextUtils.getBean(FunctionCallInfoConfig.class);
-        PluginService pluginService = ContextUtils.getBean(PluginService.class);
         String functionUrl = functionCallConfig.getUrl();
-        if (StringUtils.isBlank(functionUrl) || SatisfactionChecker.check(queryCtx)) {
+        if (StringUtils.isBlank(functionUrl) || SatisfactionChecker.check(queryContext)) {
             log.info("functionUrl:{}, skip function parser, queryText:{}", functionUrl,
-                    queryCtx.getRequest().getQueryText());
+                    queryContext.getRequest().getQueryText());
-            return;
+            return false;
         }
-        List<PluginParseConfig> functionDOList = getFunctionDO(queryCtx.getRequest().getModelId(), queryCtx);
+        List<Plugin> plugins = getPluginList(queryContext);
-        if (CollectionUtils.isEmpty(functionDOList)) {
+        return !CollectionUtils.isEmpty(plugins);
-            log.info("function call parser, plugin is empty, skip");
-            return;
     }
-        FunctionResp functionResp = new FunctionResp();
+
-        if (functionDOList.size() == 1) {
+    @Override
-            functionResp.setToolSelection(functionDOList.iterator().next().getName());
+    public PluginRecallResult recallPlugin(QueryContext queryContext) {
-        } else {
+        PluginService pluginService = ContextUtils.getBean(PluginService.class);
-            FunctionReq functionReq = FunctionReq.builder()
+        FunctionResp functionResp = functionCall(queryContext);
-                    .queryText(queryCtx.getRequest().getQueryText())
+        if (skipFunction(functionResp)) {
-                    .pluginConfigs(functionDOList).build();
+            return null;
-            functionResp = requestFunction(functionUrl, functionReq);
         }
         log.info("requestFunction result:{}", functionResp.getToolSelection());
-        if (skipFunction(functionResp)) {
-            return;
-        }
-        PluginParseResult functionCallParseResult = new PluginParseResult();
         String toolSelection = functionResp.getToolSelection();
         Optional<Plugin> pluginOptional = pluginService.getPluginByName(toolSelection);
         if (!pluginOptional.isPresent()) {
             log.info("pluginOptional is not exist:{}, skip the parse", toolSelection);
-            return;
+            return null;
         }
         Plugin plugin = pluginOptional.get();
         plugin.setParseMode(ParseMode.FUNCTION_CALL);
-        toolSelection = plugin.getType();
+        Pair<Boolean, Set<Long>> pluginResolveResult = PluginManager.resolve(plugin, queryContext);
-        functionCallParseResult.setPlugin(plugin);
+        if (pluginResolveResult.getLeft()) {
-        log.info("QueryManager PluginQueryModes:{}", QueryManager.getPluginQueryModes());
+            Set<Long> modelList = pluginResolveResult.getRight();
-        PluginSemanticQuery semanticQuery = QueryManager.createPluginQuery(toolSelection);
+            if (CollectionUtils.isEmpty(modelList)) {
-        ModelResolver modelResolver = ComponentFactory.getModelResolver();
+                return null;
-        log.info("plugin ModelList:{}", plugin.getModelList());
-        Pair<Boolean, Set<Long>> pluginResolveResult = PluginManager.resolve(plugin, queryCtx);
-        Long modelId = modelResolver.resolve(queryCtx, chatCtx, pluginResolveResult.getRight());
-        log.info("FunctionBasedParser modelId:{}", modelId);
-        if ((Objects.isNull(modelId) || modelId <= 0) && !plugin.isContainsAllModel()) {
-            log.info("Model is null, skip the parse, select tool: {}", toolSelection);
-            return;
             }
-        if (!plugin.getModelList().contains(modelId) && !plugin.isContainsAllModel()) {
+            double score = queryContext.getRequest().getQueryText().length();
-            return;
+            return PluginRecallResult.builder().plugin(plugin).modelIds(modelList).score(score).build();
         }
-        SemanticParseInfo parseInfo = semanticQuery.getParseInfo();
+        return null;
-        if (Objects.nonNull(modelId) && modelId > 0) {
-            parseInfo.getElementMatches().addAll(queryCtx.getMapInfo().getMatchedElements(modelId));
     }
-        functionCallParseResult.setRequest(queryCtx.getRequest());
+
-        Map<String, Object> properties = new HashMap<>();
+    public FunctionResp functionCall(QueryContext queryContext) {
-        properties.put(Constants.CONTEXT, functionCallParseResult);
+        List<PluginParseConfig> pluginToFunctionCall =
-        properties.put("type", "plugin");
+                getPluginToFunctionCall(queryContext.getRequest().getModelId(), queryContext);
-        properties.put("name", plugin.getName());
+        if (CollectionUtils.isEmpty(pluginToFunctionCall)) {
-        parseInfo.setProperties(properties);
+            log.info("function call parser, plugin is empty, skip");
-        parseInfo.setScore(queryCtx.getRequest().getQueryText().length());
+            return null;
-        parseInfo.setQueryMode(semanticQuery.getQueryMode());
+        }
-        SchemaElement model = new SchemaElement();
+        FunctionResp functionResp = new FunctionResp();
-        model.setModel(modelId);
+        if (pluginToFunctionCall.size() == 1) {
-        model.setId(modelId);
+            functionResp.setToolSelection(pluginToFunctionCall.iterator().next().getName());
-        parseInfo.setModel(model);
+        } else {
-        queryCtx.getCandidateQueries().add(semanticQuery);
+            FunctionReq functionReq = FunctionReq.builder()
+                    .queryText(queryContext.getRequest().getQueryText())
+                    .pluginConfigs(pluginToFunctionCall).build();
+            functionResp = requestFunction(functionReq);
+        }
+        return functionResp;
     }
 
     private boolean skipFunction(FunctionResp functionResp) {
         return Objects.isNull(functionResp) || StringUtils.isBlank(functionResp.getToolSelection());
     }
 
-    private List<PluginParseConfig> getFunctionDO(Long modelId, QueryContext queryContext) {
+    private List<PluginParseConfig> getPluginToFunctionCall(Long modelId, QueryContext queryContext) {
         log.info("user decide Model:{}", modelId);
         List<Plugin> plugins = getPluginList(queryContext);
         List<PluginParseConfig> functionDOList = plugins.stream().filter(plugin -> {
@@ -150,11 +129,13 @@ public class FunctionBasedParser implements SemanticParser {
                 return true;
             }
         }).map(o -> JsonUtil.toObject(o.getParseModeConfig(), PluginParseConfig.class)).collect(Collectors.toList());
-        log.info("getFunctionDO:{}", JsonUtil.toString(functionDOList));
+        log.info("PluginToFunctionCall: {}", JsonUtil.toString(functionDOList));
         return functionDOList;
     }
 
-    public FunctionResp requestFunction(String url, FunctionReq functionReq) {
+    public FunctionResp requestFunction(FunctionReq functionReq) {
+        FunctionCallInfoConfig functionCallInfoConfig = ContextUtils.getBean(FunctionCallInfoConfig.class);
+        String url = functionCallInfoConfig.getUrl() + functionCallInfoConfig.getPluginSelectPath();
         HttpHeaders headers = new HttpHeaders();
         long startTime = System.currentTimeMillis();
         headers.setContentType(MediaType.APPLICATION_JSON);
@@ -173,8 +154,4 @@ public class FunctionBasedParser implements SemanticParser {
         }
         return null;
     }
-
-    protected List<Plugin> getPluginList(QueryContext queryContext) {
-        return PluginManager.getPluginAgentCanSupport(queryContext.getRequest().getAgentId());
-    }
 }

chat/core/src/main/java/com/tencent/supersonic/chat/parser/plugin/function/FunctionReq.java

@@ -1,7 +1,8 @@
 package com.tencent.supersonic.chat.parser.plugin.function;
 
-import com.tencent.supersonic.chat.plugin.PluginParseConfig;
 import java.util.List;
+
+import com.tencent.supersonic.chat.plugin.PluginParseConfig;
 import lombok.Builder;
 import lombok.Data;
 

chat/core/src/main/java/com/tencent/supersonic/chat/parser/plugin/function/HeuristicModelResolver.java

@@ -137,7 +137,10 @@ public class HeuristicModelResolver implements ModelResolver {
     public Long resolve(QueryContext queryContext, ChatContext chatCtx, Set<Long> restrictiveModels) {
         Long modelId = queryContext.getRequest().getModelId();
         if (Objects.nonNull(modelId) && modelId > 0) {
-            if (CollectionUtils.isNotEmpty(restrictiveModels) && restrictiveModels.contains(modelId)) {
+            if (CollectionUtils.isEmpty(restrictiveModels)) {
+                return modelId;
+            }
+            if (restrictiveModels.contains(modelId)) {
                 return modelId;
             } else {
                 return null;

chat/core/src/main/java/com/tencent/supersonic/chat/parser/plugin/function/ModelMatchResult.java

@@ -4,7 +4,6 @@ import lombok.Data;
 
 @Data
 public class ModelMatchResult {
-
     private Integer count = 0;
     private double maxSimilarity;
 }

chat/core/src/main/java/com/tencent/supersonic/chat/parser/plugin/function/Parameters.java

@@ -1,8 +1,8 @@
 package com.tencent.supersonic.chat.parser.plugin.function;
 
+import lombok.Data;
 import java.util.List;
 import java.util.Map;
-import lombok.Data;
 
 @Data
 public class Parameters {

chat/core/src/main/java/com/tencent/supersonic/chat/parser/rule/AgentCheckParser.java

@@ -43,6 +43,9 @@ public class AgentCheckParser implements SemanticParser {
                 if (!tool.getQueryModes().contains(query.getQueryMode())) {
                     return true;
                 }
+                if (CollectionUtils.isEmpty(tool.getModelIds())) {
+                    return true;
+                }
                 if (tool.isContainsAllModel() || tool.getModelIds().contains(query.getParseInfo().getModelId())) {
                     return false;
                 }

chat/core/src/main/java/com/tencent/supersonic/chat/parser/rule/AggregateTypeParser.java

@@ -68,6 +68,7 @@ public class AggregateTypeParser implements SemanticParser {
         Map<AggregateTypeEnum, Integer> aggregateCount = new HashMap<>(REGX_MAP.size());
         Map<AggregateTypeEnum, String> aggregateWord = new HashMap<>(REGX_MAP.size());
 
+
         for (Map.Entry<AggregateTypeEnum, Pattern> entry : REGX_MAP.entrySet()) {
             Matcher matcher = entry.getValue().matcher(queryText);
             int count = 0;
@@ -90,7 +91,6 @@ public class AggregateTypeParser implements SemanticParser {
 
     @AllArgsConstructor
     class AggregateConf {
-
         public AggregateTypeEnum type;
         public String detectWord;
     }

chat/core/src/main/java/com/tencent/supersonic/chat/parser/rule/ContextInheritParser.java

@@ -1,5 +1,12 @@
 package com.tencent.supersonic.chat.parser.rule;
 
+import static com.tencent.supersonic.chat.api.pojo.SchemaElementType.DIMENSION;
+import static com.tencent.supersonic.chat.api.pojo.SchemaElementType.ENTITY;
+import static com.tencent.supersonic.chat.api.pojo.SchemaElementType.ID;
+import static com.tencent.supersonic.chat.api.pojo.SchemaElementType.METRIC;
+import static com.tencent.supersonic.chat.api.pojo.SchemaElementType.MODEL;
+import static com.tencent.supersonic.chat.api.pojo.SchemaElementType.VALUE;
+
 import com.tencent.supersonic.chat.api.component.SemanticParser;
 import com.tencent.supersonic.chat.api.component.SemanticQuery;
 import com.tencent.supersonic.chat.api.pojo.ChatContext;
@@ -8,8 +15,8 @@ import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
 import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
 import com.tencent.supersonic.chat.query.QueryManager;
 import com.tencent.supersonic.chat.query.rule.RuleSemanticQuery;
-import com.tencent.supersonic.chat.query.rule.metric.MetricModelQuery;
 import com.tencent.supersonic.chat.query.rule.metric.MetricEntityQuery;
+import com.tencent.supersonic.chat.query.rule.metric.MetricModelQuery;
 import com.tencent.supersonic.chat.query.rule.metric.MetricSemanticQuery;
 import java.util.AbstractMap;
 import java.util.ArrayList;
@@ -21,13 +28,6 @@ import java.util.stream.Collectors;
 import java.util.stream.Stream;
 import lombok.extern.slf4j.Slf4j;
 
-import static com.tencent.supersonic.chat.api.pojo.SchemaElementType.METRIC;
-import static com.tencent.supersonic.chat.api.pojo.SchemaElementType.DIMENSION;
-import static com.tencent.supersonic.chat.api.pojo.SchemaElementType.VALUE;
-import static com.tencent.supersonic.chat.api.pojo.SchemaElementType.ENTITY;
-import static com.tencent.supersonic.chat.api.pojo.SchemaElementType.MODEL;
-import static com.tencent.supersonic.chat.api.pojo.SchemaElementType.ID;
-
 @Slf4j
 public class ContextInheritParser implements SemanticParser {
 
@@ -97,10 +97,10 @@ public class ContextInheritParser implements SemanticParser {
     }
 
     protected boolean shouldInherit(QueryContext queryContext, ChatContext chatContext) {
-        Long contextmodelId = chatContext.getParseInfo().getModelId();
+        Long contextModelId = chatContext.getParseInfo().getModelId();
         // if map info doesn't contain the same Model of the context,
         // no inheritance could be done
-        if (queryContext.getMapInfo().getMatchedElements(contextmodelId) == null) {
+        if (queryContext.getMapInfo().getMatchedElements(contextModelId) == null) {
             return false;
         }