(feature)(chat) optimized code in GroupByCorrector (#226)

* (feature)(webapp) use nativeQuery field to determine whether it is a selection

* (feature)(webapp) modify chat sdk version

* (feature)(webapp) modify setupProxy target

* (feature)(webapp) modify setupProxy target

* (feature)(webapp) modify setupProxy target

* (feature)(webapp) modify chat sdk setupProxy config

.gitignore

@@ -15,4 +15,6 @@ assembly/runtime/*
 /assembly/deploy
 /runtime
 **/.flattened-pom.xml
-__pycache__/
+chm_db/
+__pycache__/
+/dict

CHANGELOG.md

@@ -3,25 +3,40 @@
 - All notable changes to this project will be documented in this file.
 - "Breaking Changes" describes any changes that may break existing functionality or cause
   compatibility issues with previous versions.
+
+## SuperSonic [0.7.5] - 2023-10-13
+
+### Added
+- add SQL generation improvement optimization, support LLM SQL, Logic SQL, and Physical SQL display.
+- add showcase functionality to support recommending similar questions.
+- add frontend modification of filtering conditions and re-querying feature.
+- support nested query functionality in semantic.
+- support switching queries between multiple parsers in the frontend.
+
+### Updated
+- optimizing the build and deployment of the project.
+- overall optimization of the SQL Corrector functionality.
+
+### Fixed
+- fix execute error on mysql <=5.7
   
-  ## SuperSonic [0.7.4] - 2023-09-10
+## SuperSonic [0.7.4] - 2023-09-10
   
-  ### Added
-  - add llm parser config
-  - add datasource agg_time option
-  - add function name adaptor in clickhouse
-  - add dimension and metric show in dsl
+### Added
+- add llm parser config
+- add datasource agg_time option
+- add function name adaptor in clickhouse
+- add dimension and metric show in dsl
   
-  
-  ### Updated
-  - update user guide doc
-  - update query building of plugin in default model
-  - update some core API constructs to keep naming consistency
-  - update ConfigureDemo config
-  - update the association mechanism so that invisible dimensions and metrics will no longer be associated
-  
-  ### Fixed
-  - fix hasAggregateFunction logic in SqlParserSelectHelper
+### Updated
+- update user guide doc
+- update query building of plugin in default model
+- update some core API constructs to keep naming consistency
+- update ConfigureDemo config
+- update the association mechanism so that invisible dimensions and metrics will no longer be associated
+
+### Fixed
+- fix hasAggregateFunction logic in SqlParserSelectHelper
 
 ## SuperSonic [0.7.3] - 2023-08-29
 

README.md

@@ -2,20 +2,20 @@
 
 # SuperSonic (超音数)
 
-**SuperSonic is an out-of-the-box yet highly extensible framework for building a data chatbot**. SuperSonic provides a chat interface that empowers users to query data using natural language and visualize the results with suitable charts. To enable such experience, the only thing necessary is to build logical semantic models (definition of metrics/dimensions/entities, along with their meaning, context and relationships) on top of physical data models, and no data modification or copying is required. Meanwhile, SuperSonic is designed to be pluggable, allowing new functionalities to be added through plugins and core components to be integrated with other systems.
+**SuperSonic is an out-of-the-box yet highly extensible framework for building ChatBI**. SuperSonic provides a chat interface that empowers users to query data using natural language and visualize the results with suitable charts. To enable such experience, the only thing necessary is to build logical semantic models (definition of metrics/dimensions/entities, along with their meaning, context and relationships) on top of physical data models, and no data modification or copying is required. Meanwhile, SuperSonic is designed to be pluggable, allowing new functionalities to be added through plugins and core components to be integrated with other systems.
 
 <img src="./docs/images/supersonic_demo.gif" height="100%" width="100%" align="center"/>
 
 ## Motivation
 
-The emergence of Large Language Model (LLM) like ChatGPT is reshaping the way information is retrieved. In the field of data analytics, both academia and industry are primarily focused on leveraging LLM to convert natural language queries into SQL queries. While some works show promising results, they are still not applicable to real-world scenarios.
+The emergence of Large Language Model (LLM) like ChatGPT is reshaping the way information is retrieved. In the field of data analytics, both academia and industry are primarily focused on leveraging LLM to convert natural language into SQL (so called text2sql or nl2sql). While some works exhibit promising results, their **reliability** is inadequate for real-world applications.  
 
 From our perspective, the key to filling the real-world gap lies in three aspects: 
-1. Complement the LLM-based semantic parser with rule-based semantic parsers to improve **efficiency**(in terms of latency and cost).
-2. Augment semantic parsing with schema mappers(as a kind of preprocessor) and semantic correctors(as a kind of postprocessor) to improve **accuracy** and **stability**.
-3. Introduce a semantic layer encapsulating underlying data context(joins, formulas, etc) to reduce **complexity**.
+1. Introduce a semantic layer encapsulating underlying data context(joins, formulas, etc) to reduce **complexity**.
+2. Augment the LLM with schema mappers(as a kind of preprocessor) and semantic correctors(as a kind of postprocessor) to mitigate **hallucination**.
+3. Utilize heuristic rules when necessary to improve **efficiency**(in terms of latency and cost).
 
-With these ideas in mind, we develop SuperSonic as a practical reference implementation and use it to power our real-world products. Additionally, to facilitate further development of data chatbot, we decide to open source SuperSonic as an extensible framework.
+With these ideas in mind, we develop SuperSonic as a practical reference implementation and use it to power our real-world products. Additionally, to facilitate further development of ChatBI, we decide to open source SuperSonic as an extensible framework.
 
 ## Out-of-the-box Features
 
@@ -40,7 +40,7 @@ The high-level architecture and main process flow is as follows:
 
 - **Semantic Corrector:** checks validity of extracted semantic information and performs correction and optimization if needed.
 
-- **Semantic Layer:** performs execution according to extracted semantic information. It generates SQL queries and executes them against physical data models.
+- **Semantic Interpreter:** performs execution according to extracted semantic information. It generates SQL statements and executes them against physical data models.
 
 - **Chat Plugin:** extends functionality with third-party tools. The LLM is going to select the most suitable one, given all configured plugins with function description and sample questions.
 
@@ -49,15 +49,15 @@ The high-level architecture and main process flow is as follows:
 SuperSonic comes with sample semantic models as well as chat conversations that can be used as a starting point. Please follow the steps: 
 
 - Download the latest prebuilt binary from the [release page](https://github.com/tencentmusic/supersonic/releases)
-- Run script "bin/start-standalone.sh" to start services (one java process and one python process)
+- Run script "bin/supersonic-daemon.sh" to start services (one java process and one python process)
 - Visit http://localhost:9080 in the browser to start exploration
 
-## Build and Delopment
+## Build and Development
 
 Please refer to project [wiki](https://github.com/tencentmusic/supersonic/wiki). 
 
 ## WeChat Contact
 
-Please join the chat group to suggest feedbacks or ideas:
+Please follow SuperSonic wechat official account:
 
-<img src="./docs/images/wechat_contact.jpeg" height="40%" width="40%" align="center"/> 
+<img src="./docs/images/supersonic_wechat_oa.png" height="50%" width="50%" align="center"/> 

README_CN.md

@@ -6,12 +6,12 @@
 
 ## 项目动机
 
-大型语言模型（LLMs）如ChatGPT的出现正在重塑信息检索的方式。在数据分析领域，学术界和工业界主要关注利用深度学习模型将自然语言查询转换为SQL查询。虽然一些工作显示出有前景的结果，但它们还并不适用于实际场景。
+大型语言模型（LLMs）如ChatGPT的出现正在重塑信息检索的方式。在数据分析领域，学术界和工业界主要关注利用深度学习模型将自然语言查询转换为SQL查询。虽然一些工作显示出有前景的结果，但它们的可靠性还达不到生产可用的要求。
 
 在我们看来，为了在实际场景发挥价值，有三个关键点：
-1. 在基于大模型语义解析器基础上，增加基于规则的解析器，提升语义解析的**效率**。
-2. 加入模式映射器和语义修正器，来增强语义解析能力，提升语义解析的**准确性**和**稳定性**。
-3. 引入语义模型层，封装底层数据的上下文（关联、公式等），降低语义解析的**复杂性**。
+1. 引入语义模型层，封装底层数据的上下文（关联、公式等），降低SQL生成的**复杂度**。
+2. 通过一前一后的模式映射器和语义修正器，来缓解LLM常见的**幻觉**现象。
+3. 设计启发式的规则，在一些特定场景提升语义解析的**效率**。
 
 为了验证上述想法，我们开发了超音数项目，并将其应用在实际的内部产品中。与此同时，我们将超音数作为一个可扩展的框架开源，希望能够促进数据问答对话领域的进一步发展。
 
@@ -30,7 +30,7 @@
 
 <img src="./docs/images/supersonic_components.png" height="65%" width="65%" align="center"/> 
 
-- **知识库(Knowledge Base)：** 定期从语义模型中提取相关的模式信息，构建词典和索引，以便后续的模式映射。
+- **模型知识库(Knowledge Base)：** 定期从语义模型中提取相关的模式信息，构建词典和索引，以便后续的模式映射。
 
 - **模式映射器(Schema Mapper)：** 将自然语言文本在知识库中进行匹配，为后续的语义解析提供相关信息。
 
@@ -38,7 +38,7 @@
 
 - **语义修正器(Semantic Corrector)：** 检查语义信息的合法性，对不合法的信息做修正和优化处理。
 
-- **语义模型层(Semantic Layer)：** 根据语义信息生成物理SQL执行查询。
+- **语义解释器(Semantic Interpreter)：** 根据语义信息生成物理SQL执行查询。
 
 - **问答插件(Chat Plugin)：** 通过第三方工具扩展功能。给定所有配置的插件及其功能描述和示例问题，大语言模型将选择最合适的插件。
 
@@ -47,7 +47,7 @@
 超音数自带样例的语义模型和问答对话，只需以下三步即可快速体验：
 
 - 从[release page](https://github.com/tencentmusic/supersonic/releases)下载预先构建好的发行包
-- 运行 "bin/start-standalone.sh"启动服务（一个Java进程和一个Python进程）
+- 运行 "bin/supersonic-daemon.sh"启动服务（一个Java进程和一个Python进程）
 - 在浏览器访问http://localhost:9080 开启探索
 
 ## 如何构建和部署
@@ -56,6 +56,6 @@
 
 ## 微信联系方式
 
-欢迎加入微信群反馈建议：
+欢迎关注微信公众号：
 
-<img src="./docs/images/wechat_contact.jpeg" height="40%" width="40%" align="center"/> 
+<img src="./docs/images/supersonic_wechat_oa.png" height="50%" width="50%" align="center"/> 

assembly/bin/supersonic-build.bat

@@ -1,30 +1,53 @@
 @echo off
 setlocal
-
+chcp 65001
 set "sbinDir=%~dp0"
 set "baseDir=%~dp0.."
 set "buildDir=%baseDir%\build"
+set "runtimeDir=%baseDir%\..\runtime"
+set "pip_path=pip3"
 
-
-rem 1. build semantic chat service
+rem 1. build backend java modules
 del /q "%buildDir%\*.tar.gz" 2>NUL
-
 call mvn -f "%baseDir%\..\pom.xml" clean package -DskipTests
 
 rem 2. move package to build
 echo f|xcopy "%baseDir%\..\launchers\standalone\target\*.tar.gz" "%buildDir%\supersonic-standalone.tar.gz"
-echo f|xcopy "%baseDir%\..\launchers\semantic\target\*.tar.gz" "%buildDir%\supersonic-semantic.tar.gz"
-echo f|xcopy "%baseDir%\..\launchers\chat\target\*.tar.gz" "%buildDir%\supersonic-chat.tar.gz"
 
-rem 3. build webapp
+rem 3. build frontend webapp
 cd "%baseDir%\..\webapp"
 call start-fe-prod.bat
 copy /y "%baseDir%\..\webapp\supersonic-webapp.tar.gz" "%buildDir%\"
 
-
+rem 4. copy webapp to java classpath
 cd "%buildDir%"
 tar -zxvf supersonic-webapp.tar.gz
 move supersonic-webapp webapp
 move webapp ..\..\launchers\standalone\target\classes
 
+rem 5. build backend python modules
+echo "start installing python modules with pip: ${pip_path}"
+set requirementPath="%baseDir%/../chat/core/src/main/python/requirements.txt"
+%pip_path% install -r %requirementPath%
+echo "install python modules success"
+
+call :BUILD_RUNTIME
+
+:BUILD_RUNTIME
+  rem 6. reset runtime
+  rd /s /q "%runtimeDir%"
+  mkdir "%runtimeDir%"
+  tar -zxvf "%buildDir%\supersonic-standalone.tar.gz" -C "%runtimeDir%"
+  for /d %%f in ("%runtimeDir%\launchers-standalone-*") do (
+      move "%%f" "%runtimeDir%\supersonic-standalone"
+  )
+
+  rem 7. copy webapp to runtime
+  tar -zxvf "%buildDir%\supersonic-webapp.tar.gz" -C "%buildDir%"
+  if not exist "%runtimeDir%\supersonic-standalone\webapp" mkdir "%runtimeDir%\supersonic-standalone\webapp"
+  xcopy /s /e /h /y "%buildDir%\supersonic-webapp\*" "%runtimeDir%\supersonic-standalone\webapp"
+  if not exist "%runtimeDir%\supersonic-standalone\conf\webapp" mkdir "%runtimeDir%\supersonic-standalone\conf\webapp"
+  xcopy /s /e /h /y "%runtimeDir%\supersonic-standalone\webapp\*" "%runtimeDir%\supersonic-standalone\conf\webapp"
+  rd /s /q "%buildDir%\supersonic-webapp"
+
 endlocal

assembly/bin/supersonic-build.sh

@@ -1,29 +1,48 @@
 #!/usr/bin/env bash
 
+set -x
 sbinDir=$(cd "$(dirname "$0")"; pwd)
-baseDir=$(cd "$sbinDir/.." && pwd -P)
-runtimeDir=$baseDir/runtime
-buildDir=$baseDir/build
+chmod +x $sbinDir/supersonic-common.sh
+source $sbinDir/supersonic-common.sh
 
 cd $baseDir
 
-#1. build semantic chat service
+#1. build backend java modules
 rm -fr ${buildDir}/*.tar.gz
 rm -fr dist
 
+set +x
+
 mvn -f $baseDir/../ clean package -DskipTests
 
 #2. move package to build
-cp $baseDir/../launchers/standalone/target/*.tar.gz ${buildDir}/supersonic.tar.gz
+cp $baseDir/../launchers/semantic/target/*.tar.gz ${buildDir}/supersonic-semantic.tar.gz
+cp $baseDir/../launchers/chat/target/*.tar.gz ${buildDir}/supersonic-chat.tar.gz
+cp $baseDir/../launchers/standalone/target/*.tar.gz ${buildDir}/supersonic-standalone.tar.gz
 
-#3. build webapp
+#3. build frontend webapp
 chmod +x $baseDir/../webapp/start-fe-prod.sh
 cd ../webapp
 sh ./start-fe-prod.sh
 cp -fr  ./supersonic-webapp.tar.gz ${buildDir}/
 
-
+#4. copy webapp to java classpath
 cd $buildDir
 tar xvf supersonic-webapp.tar.gz
 mv supersonic-webapp webapp
-mv webapp ../../launchers/standalone/target/classes
+cp -fr webapp ../../launchers/semantic/target/classes
+cp -fr webapp ../../launchers/chat/target/classes
+cp -fr webapp ../../launchers/standalone/target/classes
+rm -fr  ${buildDir}/webapp
+
+#5. build backend python modules
+echo "start installing python modules with pip: ${pip_path}"
+requirementPath=$baseDir/../chat/core/src/main/python/requirements.txt
+${pip_path} install -r ${requirementPath}
+echo "install python modules success"
+
+#6. reset runtime
+rm -fr $runtimeDir/*
+moveAllToRuntime
+setEnvToWeb chat
+setEnvToWeb semantic

assembly/bin/supersonic-common.sh

@@ -0,0 +1,105 @@
+#!/usr/bin/env bash
+
+# environment parameters
+python_path=${PYTHON_PATH:-"python3"}
+pip_path=${PIP_PATH:-"pip3"}
+
+sbinDir=$(cd "$(dirname "$0")"; pwd)
+baseDir=$(cd "$sbinDir/.." && pwd -P)
+runtimeDir=$baseDir/../runtime
+buildDir=$baseDir/build
+
+readonly CHAT_APP_NAME="supersonic_chat"
+readonly SEMANTIC_APP_NAME="supersonic_semantic"
+readonly LLMPARSER_APP_NAME="supersonic_llmparser"
+readonly STANDALONE_APP_NAME="supersonic_standalone"
+readonly CHAT_SERVICE="chat"
+readonly SEMANTIC_SERVICE="semantic"
+readonly LLMPARSER_SERVICE="llmparser"
+readonly STANDALONE_SERVICE="standalone"
+readonly LLMPARSER_HOST="127.0.0.1"
+readonly LLMPARSER_PORT="9092"
+
+function setEnvToWeb {
+   model_name=$1
+   json='{"env": "'$model_name'"}'
+   echo $json > ${runtimeDir}/supersonic-${model_name}/webapp/supersonic.config.json
+   echo $json > $baseDir/../launchers/${model_name}/target/classes/webapp/supersonic.config.json
+}
+
+function moveToRuntime {
+  model_name=$1
+  tar -zxvf ${buildDir}/supersonic-${model_name}.tar.gz  -C ${runtimeDir}
+  mv ${runtimeDir}/launchers-${model_name}-* ${runtimeDir}/supersonic-${model_name}
+
+  mkdir -p ${runtimeDir}/supersonic-${model_name}/webapp
+  cp -fr  ${buildDir}/webapp/* ${runtimeDir}/supersonic-${model_name}/webapp
+}
+
+function moveAllToRuntime {
+  mkdir -p ${runtimeDir}
+  tar xvf  ${buildDir}/supersonic-webapp.tar.gz  -C ${buildDir}
+  mv ${buildDir}/supersonic-webapp ${buildDir}/webapp
+
+  moveToRuntime chat
+  moveToRuntime semantic
+  moveToRuntime standalone
+  rm -fr  ${buildDir}/webapp
+}
+
+# run java service
+function runJavaService {
+  javaRunDir=${runtimeDir}/supersonic-${model_name}
+  local_app_name=$1
+  libDir=$javaRunDir/lib
+  confDir=$javaRunDir/conf
+
+  CLASSPATH=""
+  CLASSPATH=$CLASSPATH:$confDir
+
+  for jarPath in $libDir/*.jar; do
+   CLASSPATH=$CLASSPATH:$jarPath
+  done
+
+  export CLASSPATH
+  export LANG="zh_CN.UTF-8"
+
+  cd $javaRunDir
+  if [[ "$JAVA_HOME" == "" ]]; then
+    JAVA_HOME=$(ls /usr/jdk64/jdk* -d 2>/dev/null | xargs | awk '{print "'$local_app_name'"}')
+  fi
+  export PATH=$JAVA_HOME/bin:$PATH
+  command="-Dfile.encoding="UTF-8"  -Duser.language="Zh" -Duser.region="CN" -Duser.timezone="GMT+08" -Dapp_name=${local_app_name} -Xms1024m -Xmx2048m "$main_class
+
+  mkdir -p $javaRunDir/logs
+  if [[ "$is_test" == "true" ]]; then
+    java -Dspring.profiles.active="dev" $command >/dev/null 2>$javaRunDir/logs/error.log &
+  else
+    java  $command $javaRunDir >/dev/null 2>$javaRunDir/logs/error.log &
+  fi
+}
+
+# run python service
+function runPythonService {
+  pythonRunDir=${runtimeDir}/supersonic-${model_name}/llmparser
+  cd $pythonRunDir
+  nohup ${python_path} supersonic_llmparser.py  > $pythonRunDir/llmparser.log  2>&1   &
+  # add health check
+  for i in {1..10}
+  do
+    echo "llmparser health check attempt $i..."
+    response=$(curl -s http://${LLMPARSER_HOST}:${LLMPARSER_PORT}/health)
+    echo "llmparser health check response: $response"
+    status_ok="Healthy"
+    if [[ $response == *$status_ok* ]] ; then
+      echo "llmparser Health check passed."
+      break
+    else
+      if [ "$i" -eq 10 ]; then
+        echo "llmparser Health check failed after 10 attempts. Exiting."
+      fi
+      echo "Retrying after 5 seconds..."
+      sleep 5
+    fi
+  done
+}

assembly/bin/supersonic-daemon.bat

@@ -1,120 +1,118 @@
 @echo off
 setlocal
-
+chcp 65001
 set "sbinDir=%~dp0"
 set "baseDir=%~dp0.."
 set "runtimeDir=%baseDir%\..\runtime"
 set "buildDir=%baseDir%\build"
+set "main_class=com.tencent.supersonic.StandaloneLauncher"
+set "python_path=python"
+set "pip_path=pip3"
+set "standalone_service=standalone"
+set "llmparser_service=llmparser"
+
+set "javaRunDir=%runtimeDir%\supersonic-standalone"
+set "pythonRunDir=%runtimeDir%\supersonic-standalone\llmparser"
 
 set "command=%~1"
-set "module=%~2"
+set "service=%~2"
 
-set "APP_NAME=standalone-service"
-set "MAIN_CLASS=com.tencent.supersonic.StandaloneLauncher"
-
-set "python_path=python"
-set "pip_path=pip3.9"
-set "llm_host=127.0.0.1"
-set "llm_port=9092"
-set "start_name=api_service"
-
-set "llm_path=%baseDir%\..\chat\core\src\main\python"
-
-if "%module%"=="" (
-   set "module=standalone"
-) else if "%module%"=="semantic" (
-   set "APP_NAME=semantic-service"
-   set "MAIN_CLASS=com.tencent.supersonic.SemanticLauncher"
-) else if "%module%"=="chat" (
-   set "APP_NAME=chat-service"
-   set "MAIN_CLASS=com.tencent.supersonic.ChatLauncher"
+if "%service%"=="" (
+   set "service=%standalone_service%"
 )
 
-if "%command%"=="" (
-   set "command=restart"
-)
-
-set "libDir=%runtimeDir%\supersonic-%module%\lib"
-set "confDir=%runtimeDir%\supersonic-%module%\conf"
-set "webDir=%runtimeDir%\supersonic-%module%\webapp"
-set "CLASSPATH=%confDir%;%webDir%;%libDir%\*"
-set "java-command=-Dfile.encoding=UTF-8 -Duser.language=Zh -Duser.region=CN -Duser.timezone=GMT+08 -Xms1024m -Xmx2048m -cp %CLASSPATH% %MAIN_CLASS%"
-
-
-if "%command%"=="stop" (
-    call:STOP
-    goto :EOF
-)
+call :BUILD_RUNTIME
 
 if "%command%"=="restart" (
-     call:STOP
-)
-
-::1. clear file
-rd /s /q "%runtimeDir%"
-mkdir "%runtimeDir%"
-
-if "%module%"=="llmparser" (
-   tar -zxvf "%buildDir%\supersonic-standalone.tar.gz" -C "%runtimeDir%"
-   for /d %%f in ("%runtimeDir%\launchers-standalone-*") do (
-       move "%%f" "%runtimeDir%\supersonic-standalone"
-   )
-   cd "%runtimeDir%"
-   "%pip_path%" install -r "%llm_path%\requirements.txt"
-   "%python_path%" -c "import langchain,fastapi,chromadb,tiktoken,uvicorn" >nul 2>&1
-   cd "%runtimeDir%\supersonic-standalone\llm\llm"
-   start "" /B uvicorn %start_name%:app --port %llm_port% --host %llm_host%  > "%runtimeDir%\supersonic-standalone\llm\llm.log" 2>&1
-   echo "llm service started, see logs/error with logs/error command"
+   call :STOP
+   call :START
+   goto :EOF
+) else if "%command%"=="start" (
+   call :START
+   goto :EOF
+) else if "%command%"=="stop" (
+  call :STOP
+  goto :EOF
+) else if "%command%"=="reload" (
+  call :RELOAD_EXAMPLE
+  goto :EOF
+) else (
+   echo "Use command {start|stop|restart} to run."
    goto :EOF
 )
 
-tar -zxvf "%buildDir%\supersonic-%module%.tar.gz" -C "%runtimeDir%"
-for /d %%f in ("%runtimeDir%\launchers-%module%-*") do (
-    move "%%f" "%runtimeDir%\supersonic-%module%"
-)
-
-if not exist "%runtimeDir%\supersonic-%module%\logs" mkdir "%runtimeDir%\supersonic-%module%\logs"
-
-tar -zxvf "%buildDir%\supersonic-webapp.tar.gz" -C "%buildDir%"
-if not exist "%runtimeDir%\supersonic-%module%\webapp" mkdir "%runtimeDir%\supersonic-%module%\webapp"
-xcopy /s /e /h /y "%buildDir%\supersonic-webapp\*" "%runtimeDir%\supersonic-%module%\webapp"
-if not exist "%runtimeDir%\supersonic-%module%\conf\webapp" mkdir "%runtimeDir%\supersonic-%module%\conf\webapp"
-xcopy /s /e /h /y "%runtimeDir%\supersonic-%module%\webapp\*" "%runtimeDir%\supersonic-%module%\conf\webapp"
-rd /s /q "%buildDir%\supersonic-webapp"
-
-::3. start service
-::start standalone service
-if "%command%"=="start" (
-    call:START
-    goto :EOF
-)
-
-if "%command%"=="restart" (
-    call:START
-    goto :EOF
-)
-
 :START
-    if "%module%"=="standalone" (
-          cd "%runtimeDir%"
-          "%pip_path%" install -r "%llm_path%\requirements.txt"
-          "%python_path%" -c "import langchain,fastapi,chromadb,tiktoken,uvicorn" >nul 2>&1
-          cd "%runtimeDir%\supersonic-standalone\llm\llm"
-          start "" /B uvicorn %start_name%:app --port %llm_port% --host %llm_host%  > "%runtimeDir%\supersonic-standalone\llm\llm.log" 2>&1
-          echo "llm service started, see logs/error with logs/error command"
+    if "%service%"=="%llmparser_service%" (
+         call :START_PYTHON
+         goto :EOF
     )
-    start "supersonic" /B java %java-command%>"%runtimeDir%\supersonic-%module%\logs\info-%module%.log" 2>&1
-    echo "%module% service started, see logs/error with logs/error command"
+    call :START_PYTHON
+    call :START_JAVA
     goto :EOF
 
-
 :STOP
-    for /f "tokens=2" %%i in ('tasklist ^| findstr /i "python"') do (
-                     taskkill /PID %%i /F
-                     echo "llm Process (PID = %%i) is killed."
+    if "%service%"=="%llmparser_service%" (
+        call :STOP_PYTHON
+        goto :EOF
     )
-    for /f "tokens=2" %%i in ('tasklist ^| findstr /i "java"') do (
-                taskkill /PID %%i /F
-                echo "%module% Process (PID = %%i) is killed."
-    )
-    goto :EOF
+    call :STOP_PYTHON
+    call :STOP_JAVA
+    goto :EOF
+
+:START_PYTHON
+   echo 'python service starting, see logs in llmparser/llmparser.log'
+   cd "%pythonRunDir%"
+   start /B %python_path% supersonic_llmparser.py  > %pythonRunDir%\llmparser.log 2>&1
+   timeout /t 10 >nul
+   echo 'python service started'
+   goto :EOF
+
+:START_JAVA
+  echo 'java service starting, see logs in logs/'
+   cd "%javaRunDir%"
+   if not exist "%runtimeDir%\supersonic-standalone\logs" mkdir "%runtimeDir%\supersonic-standalone\logs"
+   set "libDir=%runtimeDir%\supersonic-%service%\lib"
+   set "confDir=%runtimeDir%\supersonic-%service%\conf"
+   set "webDir=%runtimeDir%\supersonic-%service%\webapp"
+   set "classpath=%confDir%;%webDir%;%libDir%\*"
+   set "java-command=-Dfile.encoding=UTF-8 -Duser.language=Zh -Duser.region=CN -Duser.timezone=GMT+08 -Xms1024m -Xmx2048m -cp %CLASSPATH% %MAIN_CLASS%"
+   start /B java %java-command% >nul 2>&1
+   timeout /t 10 >nul
+   echo 'java service started'
+   goto :EOF
+
+:STOP_PYTHON
+   for /f "tokens=2" %%i in ('tasklist ^| findstr /i "python"') do (
+           taskkill /PID %%i /F
+           echo "python service (PID = %%i) is killed."
+   )
+   goto :EOF
+
+:STOP_JAVA
+   for /f "tokens=2" %%i in ('tasklist ^| findstr /i "java"') do (
+            taskkill /PID %%i /F
+            echo "java service (PID = %%i) is killed."
+   )
+   goto :EOF
+
+:RELOAD_EXAMPLE
+   cd "%runtimeDir%\supersonic-standalone\llmparser\sql"
+   start  %python_path% examples_reload_run.py
+   goto :EOF
+
+:BUILD_RUNTIME
+  rem 6. reset runtime
+  if exist "%runtimeDir%" goto :EOF
+  mkdir "%runtimeDir%"
+  tar -zxvf "%buildDir%\supersonic-standalone.tar.gz" -C "%runtimeDir%"
+  for /d %%f in ("%runtimeDir%\launchers-standalone-*") do (
+      move "%%f" "%runtimeDir%\supersonic-standalone"
+  )
+
+  rem 7. copy webapp to runtime
+  tar -zxvf "%buildDir%\supersonic-webapp.tar.gz" -C "%buildDir%"
+  if not exist "%runtimeDir%\supersonic-standalone\webapp" mkdir "%runtimeDir%\supersonic-standalone\webapp"
+  xcopy /s /e /h /y "%buildDir%\supersonic-webapp\*" "%runtimeDir%\supersonic-standalone\webapp"
+  if not exist "%runtimeDir%\supersonic-standalone\conf\webapp" mkdir "%runtimeDir%\supersonic-standalone\conf\webapp"
+  xcopy /s /e /h /y "%runtimeDir%\supersonic-standalone\webapp\*" "%runtimeDir%\supersonic-standalone\conf\webapp"
+  rd /s /q "%buildDir%\supersonic-webapp"

assembly/bin/supersonic-daemon.sh

@@ -1,98 +1,147 @@
 #!/usr/bin/env bash
 
+set -x
 sbinDir=$(cd "$(dirname "$0")"; pwd)
-baseDir=$(cd "$sbinDir/.." && pwd -P)
-runtimeDir=$baseDir/../runtime
-buildDir=$baseDir/build
+chmod +x $sbinDir/supersonic-common.sh
+source $sbinDir/supersonic-common.sh
+
+# 1.init environment parameters
+if [ ! -d "$runtimeDir" ]; then
+    echo "the runtime dir does not exist move all to runtime"
+    moveAllToRuntime
+fi
+set +x
 
 command=$1
 service=$2
+if [ -z "$service"  ]; then
+  service=${STANDALONE_SERVICE}
+fi
+
+app_name=$STANDALONE_APP_NAME
+main_class="com.tencent.supersonic.StandaloneLauncher"
+model_name=$service
+
+if [ "$service" == "llmparser" ]; then
+  model_name=${STANDALONE_SERVICE}
+fi
 
 cd $baseDir
-if [[ "$service" == "semantic" || -z "$service"  ]] && [ "$command" != "stop"  ]; then
-   #1. clear file
-   mkdir -p ${runtimeDir}
-   rm -fr ${runtimeDir}/*
 
-   #2. package lib
-   tar -zxvf ${buildDir}/supersonic.tar.gz  -C ${runtimeDir}
-   mv ${runtimeDir}/launchers-standalone-* ${runtimeDir}/supersonic-standalone
-   tar -zxvf ${buildDir}/supersonic-webapp.tar.gz  -C ${buildDir}
-   mkdir -p ${runtimeDir}/supersonic-standalone/webapp
-   cp -fr  ${buildDir}/supersonic-webapp/* ${runtimeDir}/supersonic-standalone/webapp
-   rm -fr  ${buildDir}/supersonic-webapp
-fi
-if [[ "$service" == "semantic"  ]]; then
-   json=$(cat ${runtimeDir}/supersonic-semantic/webapp/supersonic.config.json)
-   json=$(echo $json | jq '.env="semantic"')
-   echo $json > ${runtimeDir}/supersonic-semantic/webapp/supersonic.config.json
-fi
+# 2.set main class
+function setMainClass {
+  if [ "$service" == $CHAT_SERVICE ]; then
+    main_class="com.tencent.supersonic.ChatLauncher"
+  elif [ "$service" == $SEMANTIC_SERVICE ]; then
+    main_class="com.tencent.supersonic.SemanticLauncher"
+  fi
+}
+setMainClass
+# 3.set app name
+function setAppName {
+  if [ "$service" == $CHAT_SERVICE ]; then
+    app_name=$CHAT_APP_NAME
+  elif [ "$service" == $SEMANTIC_SERVICE ]; then
+    app_name=$SEMANTIC_APP_NAME
+  elif [ "$service" == $LLMPARSER_SERVICE ]; then
+    app_name=$LLMPARSER_APP_NAME
+  fi
+}
+setAppName
 
-if [[ "$service" == "chat"  ]]; then
-   json=$(cat ${runtimeDir}/supersonic-chat/webapp/supersonic.config.json)
-   json=$(echo $json | jq '.env="chat"')
-   echo $json > ${runtimeDir}/supersonic-chat/webapp/supersonic.config.json
-fi
-echo $command
-echo $service
+function reloadExamples {
+  pythonRunDir=${runtimeDir}/supersonic-${model_name}/llmparser
+  cd $pythonRunDir/sql
+  ${python_path} examples_reload_run.py
+}
+
+
+function start()
+{
+  local_app_name=$1
+  pid=$(ps aux |grep ${local_app_name} | grep -v grep | awk '{print $2}')
+  if [[ "$pid" == "" ]]; then
+    if [[ ${local_app_name} == $LLMPARSER_APP_NAME ]]; then
+      runPythonService ${local_app_name}
+    else
+      runJavaService ${local_app_name}
+    fi
+  else
+    echo "Process (PID = $pid) is running."
+    return 1
+  fi
+}
+
+function stop()
+{
+  pid=$(ps aux | grep $1 | grep -v grep | awk '{print $2}')
+  if [[ "$pid" == "" ]]; then
+    echo "Process $1 is not running !"
+    return 1
+  else
+    kill -9 $pid
+    echo "Process (PID = $pid) is killed !"
+    return 0
+  fi
+}
+
+function reload()
+{
+  if [[ $1 == $LLMPARSER_APP_NAME ]]; then
+    reloadExamples
+  fi
+}
+
+# 4. execute command operation
 case "$command" in
   start)
-        if [[ "$service" == "semantic"  ]];then
-          echo -e "Starting semantic"
-          sh ${runtimeDir}/supersonic-semantic/bin/service.sh start
-        elif [[ "$service" == "chat"  ]];then
-          echo -e "Starting chat"
-          sh ${runtimeDir}/supersonic-chat/bin/service.sh start
-        elif [[ "$service" == "llmparser"  ]];then
-          echo -e "Starting LLM"
-          sh ${runtimeDir}/supersonic-standalone/llm/bin/service.sh  start
-        elif [[ -z "$service"  ]]; then
-          echo -e "Starting supersonic"
-          sh ${runtimeDir}/supersonic-standalone/bin/service.sh start
-          sh ${runtimeDir}/supersonic-standalone/llm/bin/service.sh start
+        if [ "$service" == $STANDALONE_SERVICE ]; then
+          echo  "Starting $LLMPARSER_APP_NAME"
+          start $LLMPARSER_APP_NAME
+          echo  "Starting $app_name"
+          start $app_name
         else
-          echo "Use command {semantic|semantic||} to run."
+          echo  "Starting $app_name"
+          start $app_name
         fi
+        echo  "Start success"
         ;;
   stop)
-        if [[ "$service" == "semantic"  ]];then
-          echo -e "Stopping semantic"
-          sh ${runtimeDir}/supersonic-semantic/bin/service.sh stop
-        elif [[ "$service" == "chat"  ]];then
-          echo -e "Stopping chat"
-          sh ${runtimeDir}/supersonic-chat/bin/service.sh stop
-        elif [[ "$service" == "llmparser"  ]];then
-          echo -e "Stopping LLM"
-          sh ${runtimeDir}/supersonic-standalone/llm/bin/service.sh  stop
-        elif [[ -z "$service"  ]]; then
-          echo -e "Stopping supersonic"
-          sh ${runtimeDir}/supersonic-standalone/bin/service.sh stop
-          sh ${runtimeDir}/supersonic-standalone/llm/bin/service.sh stop
+        if [ "$service" == $STANDALONE_SERVICE ]; then
+          echo  "Stopping $LLMPARSER_APP_NAME"
+          stop $LLMPARSER_APP_NAME
+          echo  "Stopping $app_name"
+          stop $app_name
         else
-          echo "Use command {semantic|semantic||} to run."
+          echo  "Stopping $app_name"
+          stop ${app_name}
         fi
+        echo  "Stop success"
+        ;;
+  reload)
+        echo  "Reloading ${app_name}"
+        reload ${app_name}
+        echo  "Reload success"
         ;;
   restart)
-        if [[ "$service" == "semantic"  ]];then
-          echo -e "Restarting semantic"
-          sh ${runtimeDir}/supersonic-semantic/bin/service.sh restart
-        elif [[ "$service" == "chat"  ]];then
-          echo -e "Restarting chat"
-          sh ${runtimeDir}/supersonic-chat/bin/service.sh restart
-        elif [[ "$service" == "llmparser"  ]];then
-          echo -e "Restarting LLM"
-          sh ${runtimeDir}/supersonic-standalone/llm/bin/service.sh  restart
-        elif [[ -z "$service"  ]]; then
-          echo -e "Restarting supersonic"
-          sh ${runtimeDir}/supersonic-standalone/bin/service.sh restart
-          sh ${runtimeDir}/supersonic-standalone/llm/bin/service.sh restart
+        if [ "$service" == $STANDALONE_SERVICE ]; then
+          echo  "Stopping ${app_name}"
+          stop ${app_name}
+          echo  "Stopping ${LLMPARSER_APP_NAME}"
+          stop $LLMPARSER_APP_NAME
+          echo  "Starting ${LLMPARSER_APP_NAME}"
+          start $LLMPARSER_APP_NAME
+          echo  "Starting ${app_name}"
+          start ${app_name}
         else
-          echo "Use command {semantic|semantic||} to run."
+          echo  "Stopping ${app_name}"
+          stop ${app_name}
+          echo  "Starting ${app_name}"
+          start ${app_name}
         fi
+        echo  "Restart success"
         ;;
   *)
-        echo "Use command {start|stop|status|restart} to run."
+        echo "Use command {start|stop|restart} to run."
         exit 1
 esac
-
-exit 0

assembly/build/build.xml

@@ -6,14 +6,6 @@
         <format>tar.gz</format>
     </formats>
     <fileSets>
-
-        <fileSet>
-            <directory>${project.basedir}/src/main/bin</directory>
-            <outputDirectory>bin</outputDirectory>
-            <fileMode>0777</fileMode>
-            <directoryMode>0755</directoryMode>
-        </fileSet>
-
         <fileSet>
             <directory>${project.basedir}/src/main/resources</directory>
             <outputDirectory>conf</outputDirectory>
@@ -30,7 +22,7 @@
         </fileSet>
         <fileSet>
             <directory>${project.basedir}/../../chat/core/src/main/python</directory>
-            <outputDirectory>llm</outputDirectory>
+            <outputDirectory>llmparser</outputDirectory>
             <fileMode>0777</fileMode>
             <directoryMode>0755</directoryMode>
         </fileSet>

auth/api/src/main/java/com/tencent/supersonic/auth/api/authentication/constant/UserConstants.java

@@ -12,6 +12,8 @@ public class UserConstants {
 
     public static final String TOKEN_USER_EMAIL = "token_user_email";
 
+    public static final String TOKEN_IS_ADMIN = "token_is_admin";
+
     public static final String TOKEN_ALGORITHM = "HS512";
 
     public static final String TOKEN_CREATE_TIME = "token_create_time";

auth/api/src/main/java/com/tencent/supersonic/auth/api/authentication/pojo/User.java

@@ -18,17 +18,22 @@ public class User {
 
     private String email;
 
-    public static User get(Long id, String name, String displayName, String email) {
-        return new User(id, name, displayName, email);
+    private Integer isAdmin;
+
+    public static User get(Long id, String name, String displayName, String email, Integer isAdmin) {
+        return new User(id, name, displayName, email, isAdmin);
     }
 
     public static User getFakeUser() {
-        return new User(1L, "admin", "admin", "admin@email");
+        return new User(1L, "admin", "admin", "admin@email", 1);
     }
 
     public String getDisplayName() {
         return StringUtils.isBlank(displayName) ? name : displayName;
     }
 
+    public boolean isSuperAdmin() {
+        return isAdmin != null && isAdmin == 1;
+    }
 
 }

auth/api/src/main/java/com/tencent/supersonic/auth/api/authentication/pojo/UserWithPassword.java

@@ -9,13 +9,14 @@ public class UserWithPassword extends User {
 
     private String password;
 
-    public UserWithPassword(Long id, String name, String displayName, String email, String password) {
-        super(id, name, displayName, email);
+    public UserWithPassword(Long id, String name, String displayName, String email, String password, Integer isAdmin) {
+        super(id, name, displayName, email, isAdmin);
         this.password = password;
     }
 
-    public static UserWithPassword get(Long id, String name, String displayName, String email, String password) {
-        return new UserWithPassword(id, name, displayName, email, password);
+    public static UserWithPassword get(Long id, String name, String displayName,
+                                       String email, String password, Integer isAdmin) {
+        return new UserWithPassword(id, name, displayName, email, password, isAdmin);
     }
 
 }

auth/api/src/main/java/com/tencent/supersonic/auth/api/authorization/service/AuthService.java

@@ -10,7 +10,7 @@ public interface AuthService {
 
     List<AuthGroup> queryAuthGroups(String domainId, Integer groupId);
 
-    void updateAuthGroup(AuthGroup group);
+    void addOrUpdateAuthGroup(AuthGroup group);
 
     void removeAuthGroup(AuthGroup group);
 

auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/adaptor/DefaultUserAdaptor.java

@@ -71,7 +71,7 @@ public class DefaultUserAdaptor implements UserAdaptor {
         }
         if (userDO.getPassword().equals(userReq.getPassword())) {
             UserWithPassword user = UserWithPassword.get(userDO.getId(), userDO.getName(), userDO.getDisplayName(),
-                    userDO.getEmail(), userDO.getPassword());
+                    userDO.getEmail(), userDO.getPassword(), userDO.getIsAdmin());
             return userTokenUtils.generateToken(user);
         }
         throw new RuntimeException("password not correct, please try again");

auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/persistence/dataobject/UserDO.java

@@ -1,99 +1,129 @@
 package com.tencent.supersonic.auth.authentication.persistence.dataobject;
 
 public class UserDO {
-
     /**
-     *
+     * 
      */
     private Long id;
 
     /**
-     *
+     * 
      */
     private String name;
 
     /**
-     *
+     * 
      */
     private String password;
 
     /**
-     *
+     * 
      */
     private String displayName;
 
     /**
-     *
+     * 
      */
     private String email;
 
     /**
-     * @return id
+     * 
+     */
+    private Integer isAdmin;
+
+    /**
+     * 
+     * @return id 
      */
     public Long getId() {
         return id;
     }
 
     /**
-     * @param id
+     * 
+     * @param id 
      */
     public void setId(Long id) {
         this.id = id;
     }
 
     /**
-     * @return name
+     * 
+     * @return name 
      */
     public String getName() {
         return name;
     }
 
     /**
-     * @param name
+     * 
+     * @param name 
      */
     public void setName(String name) {
         this.name = name == null ? null : name.trim();
     }
 
     /**
-     * @return password
+     * 
+     * @return password 
      */
     public String getPassword() {
         return password;
     }
 
     /**
-     * @param password
+     * 
+     * @param password 
      */
     public void setPassword(String password) {
         this.password = password == null ? null : password.trim();
     }
 
     /**
-     * @return display_name
+     * 
+     * @return display_name 
      */
     public String getDisplayName() {
         return displayName;
     }
 
     /**
-     * @param displayName
+     * 
+     * @param displayName 
      */
     public void setDisplayName(String displayName) {
         this.displayName = displayName == null ? null : displayName.trim();
     }
 
     /**
-     * @return email
+     * 
+     * @return email 
      */
     public String getEmail() {
         return email;
     }
 
     /**
-     * @param email
+     * 
+     * @param email 
      */
     public void setEmail(String email) {
         this.email = email == null ? null : email.trim();
     }
+
+    /**
+     * 
+     * @return is_admin 
+     */
+    public Integer getIsAdmin() {
+        return isAdmin;
+    }
+
+    /**
+     * 
+     * @param isAdmin 
+     */
+    public void setIsAdmin(Integer isAdmin) {
+        this.isAdmin = isAdmin;
+    }
 }

auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/persistence/dataobject/UserDOExample.java

@@ -4,7 +4,6 @@ import java.util.ArrayList;
 import java.util.List;
 
 public class UserDOExample {
-
     /**
      * s2_user
      */
@@ -31,6 +30,7 @@ public class UserDOExample {
     protected Integer limitEnd;
 
     /**
+     *
      * @mbg.generated
      */
     public UserDOExample() {
@@ -38,13 +38,7 @@ public class UserDOExample {
     }
 
     /**
-     * @mbg.generated
-     */
-    public String getOrderByClause() {
-        return orderByClause;
-    }
-
-    /**
+     *
      * @mbg.generated
      */
     public void setOrderByClause(String orderByClause) {
@@ -52,13 +46,15 @@ public class UserDOExample {
     }
 
     /**
+     *
      * @mbg.generated
      */
-    public boolean isDistinct() {
-        return distinct;
+    public String getOrderByClause() {
+        return orderByClause;
     }
 
     /**
+     *
      * @mbg.generated
      */
     public void setDistinct(boolean distinct) {
@@ -66,6 +62,15 @@ public class UserDOExample {
     }
 
     /**
+     *
+     * @mbg.generated
+     */
+    public boolean isDistinct() {
+        return distinct;
+    }
+
+    /**
+     *
      * @mbg.generated
      */
     public List<Criteria> getOredCriteria() {
@@ -73,6 +78,7 @@ public class UserDOExample {
     }
 
     /**
+     *
      * @mbg.generated
      */
     public void or(Criteria criteria) {
@@ -80,6 +86,7 @@ public class UserDOExample {
     }
 
     /**
+     *
      * @mbg.generated
      */
     public Criteria or() {
@@ -89,6 +96,7 @@ public class UserDOExample {
     }
 
     /**
+     *
      * @mbg.generated
      */
     public Criteria createCriteria() {
@@ -100,6 +108,7 @@ public class UserDOExample {
     }
 
     /**
+     *
      * @mbg.generated
      */
     protected Criteria createCriteriaInternal() {
@@ -108,6 +117,7 @@ public class UserDOExample {
     }
 
     /**
+     *
      * @mbg.generated
      */
     public void clear() {
@@ -117,6 +127,15 @@ public class UserDOExample {
     }
 
     /**
+     *
+     * @mbg.generated
+     */
+    public void setLimitStart(Integer limitStart) {
+        this.limitStart=limitStart;
+    }
+
+    /**
+     *
      * @mbg.generated
      */
     public Integer getLimitStart() {
@@ -124,31 +143,25 @@ public class UserDOExample {
     }
 
     /**
+     *
      * @mbg.generated
      */
-    public void setLimitStart(Integer limitStart) {
-        this.limitStart = limitStart;
+    public void setLimitEnd(Integer limitEnd) {
+        this.limitEnd=limitEnd;
     }
 
     /**
+     *
      * @mbg.generated
      */
     public Integer getLimitEnd() {
         return limitEnd;
     }
 
-    /**
-     * @mbg.generated
-     */
-    public void setLimitEnd(Integer limitEnd) {
-        this.limitEnd = limitEnd;
-    }
-
     /**
      * s2_user null
      */
     protected abstract static class GeneratedCriteria {
-
         protected List<Criterion> criteria;
 
         protected GeneratedCriteria() {
@@ -528,6 +541,66 @@ public class UserDOExample {
             addCriterion("email not between", value1, value2, "email");
             return (Criteria) this;
         }
+
+        public Criteria andIsAdminIsNull() {
+            addCriterion("is_admin is null");
+            return (Criteria) this;
+        }
+
+        public Criteria andIsAdminIsNotNull() {
+            addCriterion("is_admin is not null");
+            return (Criteria) this;
+        }
+
+        public Criteria andIsAdminEqualTo(Integer value) {
+            addCriterion("is_admin =", value, "isAdmin");
+            return (Criteria) this;
+        }
+
+        public Criteria andIsAdminNotEqualTo(Integer value) {
+            addCriterion("is_admin <>", value, "isAdmin");
+            return (Criteria) this;
+        }
+
+        public Criteria andIsAdminGreaterThan(Integer value) {
+            addCriterion("is_admin >", value, "isAdmin");
+            return (Criteria) this;
+        }
+
+        public Criteria andIsAdminGreaterThanOrEqualTo(Integer value) {
+            addCriterion("is_admin >=", value, "isAdmin");
+            return (Criteria) this;
+        }
+
+        public Criteria andIsAdminLessThan(Integer value) {
+            addCriterion("is_admin <", value, "isAdmin");
+            return (Criteria) this;
+        }
+
+        public Criteria andIsAdminLessThanOrEqualTo(Integer value) {
+            addCriterion("is_admin <=", value, "isAdmin");
+            return (Criteria) this;
+        }
+
+        public Criteria andIsAdminIn(List<Integer> values) {
+            addCriterion("is_admin in", values, "isAdmin");
+            return (Criteria) this;
+        }
+
+        public Criteria andIsAdminNotIn(List<Integer> values) {
+            addCriterion("is_admin not in", values, "isAdmin");
+            return (Criteria) this;
+        }
+
+        public Criteria andIsAdminBetween(Integer value1, Integer value2) {
+            addCriterion("is_admin between", value1, value2, "isAdmin");
+            return (Criteria) this;
+        }
+
+        public Criteria andIsAdminNotBetween(Integer value1, Integer value2) {
+            addCriterion("is_admin not between", value1, value2, "isAdmin");
+            return (Criteria) this;
+        }
     }
 
     /**
@@ -544,7 +617,6 @@ public class UserDOExample {
      * s2_user null
      */
     public static class Criterion {
-
         private String condition;
 
         private Object value;
@@ -561,6 +633,38 @@ public class UserDOExample {
 
         private String typeHandler;
 
+        public String getCondition() {
+            return condition;
+        }
+
+        public Object getValue() {
+            return value;
+        }
+
+        public Object getSecondValue() {
+            return secondValue;
+        }
+
+        public boolean isNoValue() {
+            return noValue;
+        }
+
+        public boolean isSingleValue() {
+            return singleValue;
+        }
+
+        public boolean isBetweenValue() {
+            return betweenValue;
+        }
+
+        public boolean isListValue() {
+            return listValue;
+        }
+
+        public String getTypeHandler() {
+            return typeHandler;
+        }
+
         protected Criterion(String condition) {
             super();
             this.condition = condition;
@@ -596,37 +700,5 @@ public class UserDOExample {
         protected Criterion(String condition, Object value, Object secondValue) {
             this(condition, value, secondValue, null);
         }
-
-        public String getCondition() {
-            return condition;
-        }
-
-        public Object getValue() {
-            return value;
-        }
-
-        public Object getSecondValue() {
-            return secondValue;
-        }
-
-        public boolean isNoValue() {
-            return noValue;
-        }
-
-        public boolean isSingleValue() {
-            return singleValue;
-        }
-
-        public boolean isBetweenValue() {
-            return betweenValue;
-        }
-
-        public boolean isListValue() {
-            return listValue;
-        }
-
-        public String getTypeHandler() {
-            return typeHandler;
-        }
     }
 }

auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/utils/UserTokenUtils.java

@@ -2,6 +2,7 @@ package com.tencent.supersonic.auth.authentication.utils;
 
 import static com.tencent.supersonic.auth.api.authentication.constant.UserConstants.TOKEN_ALGORITHM;
 import static com.tencent.supersonic.auth.api.authentication.constant.UserConstants.TOKEN_CREATE_TIME;
+import static com.tencent.supersonic.auth.api.authentication.constant.UserConstants.TOKEN_IS_ADMIN;
 import static com.tencent.supersonic.auth.api.authentication.constant.UserConstants.TOKEN_PREFIX;
 import static com.tencent.supersonic.auth.api.authentication.constant.UserConstants.TOKEN_TIME_OUT;
 import static com.tencent.supersonic.auth.api.authentication.constant.UserConstants.TOKEN_USER_DISPLAY_NAME;
@@ -42,6 +43,7 @@ public class UserTokenUtils {
         claims.put(TOKEN_USER_PASSWORD, StringUtils.isEmpty(user.getPassword()) ? "" : user.getPassword());
         claims.put(TOKEN_USER_DISPLAY_NAME, user.getDisplayName());
         claims.put(TOKEN_CREATE_TIME, System.currentTimeMillis());
+        claims.put(TOKEN_IS_ADMIN, user.getIsAdmin());
         return generate(claims);
     }
 
@@ -52,6 +54,7 @@ public class UserTokenUtils {
         claims.put(TOKEN_USER_PASSWORD, "admin");
         claims.put(TOKEN_USER_DISPLAY_NAME, "admin");
         claims.put(TOKEN_CREATE_TIME, System.currentTimeMillis());
+        claims.put(TOKEN_IS_ADMIN, 1);
         return generate(claims);
     }
 
@@ -63,7 +66,9 @@ public class UserTokenUtils {
         String userName = String.valueOf(claims.get(TOKEN_USER_NAME));
         String email = String.valueOf(claims.get(TOKEN_USER_EMAIL));
         String displayName = String.valueOf(claims.get(TOKEN_USER_DISPLAY_NAME));
-        return User.get(userId, userName, displayName, email);
+        Integer isAdmin = claims.get(TOKEN_IS_ADMIN) == null
+                ? 0 : Integer.parseInt(claims.get(TOKEN_IS_ADMIN).toString());
+        return User.get(userId, userName, displayName, email, isAdmin);
     }
 
     public UserWithPassword getUserWithPassword(HttpServletRequest request) {
@@ -79,7 +84,9 @@ public class UserTokenUtils {
         String email = String.valueOf(claims.get(TOKEN_USER_EMAIL));
         String displayName = String.valueOf(claims.get(TOKEN_USER_DISPLAY_NAME));
         String password = String.valueOf(claims.get(TOKEN_USER_PASSWORD));
-        return UserWithPassword.get(userId, userName, displayName, email, password);
+        Integer isAdmin = claims.get(TOKEN_IS_ADMIN) == null
+                ? 0 : Integer.parseInt(claims.get(TOKEN_IS_ADMIN).toString());
+        return UserWithPassword.get(userId, userName, displayName, email, password, isAdmin);
     }
 
     private Claims getClaims(String token) {

auth/authentication/src/main/resources/mapper/UserDOMapper.xml

@@ -2,11 +2,12 @@
 <!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
 <mapper namespace="com.tencent.supersonic.auth.authentication.persistence.mapper.UserDOMapper">
   <resultMap id="BaseResultMap" type="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO">
-    <id column="id" jdbcType="BIGINT" property="id" />
+    <result column="id" jdbcType="BIGINT" property="id" />
     <result column="name" jdbcType="VARCHAR" property="name" />
     <result column="password" jdbcType="VARCHAR" property="password" />
     <result column="display_name" jdbcType="VARCHAR" property="displayName" />
     <result column="email" jdbcType="VARCHAR" property="email" />
+    <result column="is_admin" jdbcType="INTEGER" property="isAdmin" />
   </resultMap>
   <sql id="Example_Where_Clause">
     <where>
@@ -38,7 +39,7 @@
     </where>
   </sql>
   <sql id="Base_Column_List">
-    id, name, password, display_name, email
+    id, name, password, display_name, email, is_admin
   </sql>
   <select id="selectByExample" parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDOExample" resultMap="BaseResultMap">
     select
@@ -57,21 +58,13 @@
       limit #{limitStart} , #{limitEnd}
     </if>
   </select>
-  <select id="selectByPrimaryKey" parameterType="java.lang.Long" resultMap="BaseResultMap">
-    select 
-    <include refid="Base_Column_List" />
-    from s2_user
-    where id = #{id,jdbcType=BIGINT}
-  </select>
-  <delete id="deleteByPrimaryKey" parameterType="java.lang.Long">
-    delete from s2_user
-    where id = #{id,jdbcType=BIGINT}
-  </delete>
   <insert id="insert" parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO">
     insert into s2_user (id, name, password, 
-      display_name, email)
+      display_name, email, is_admin
+      )
     values (#{id,jdbcType=BIGINT}, #{name,jdbcType=VARCHAR}, #{password,jdbcType=VARCHAR}, 
-      #{displayName,jdbcType=VARCHAR}, #{email,jdbcType=VARCHAR})
+      #{displayName,jdbcType=VARCHAR}, #{email,jdbcType=VARCHAR}, #{isAdmin,jdbcType=INTEGER}
+      )
   </insert>
   <insert id="insertSelective" parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO">
     insert into s2_user
@@ -91,6 +84,9 @@
       <if test="email != null">
         email,
       </if>
+      <if test="isAdmin != null">
+        is_admin,
+      </if>
     </trim>
     <trim prefix="values (" suffix=")" suffixOverrides=",">
       <if test="id != null">
@@ -108,6 +104,9 @@
       <if test="email != null">
         #{email,jdbcType=VARCHAR},
       </if>
+      <if test="isAdmin != null">
+        #{isAdmin,jdbcType=INTEGER},
+      </if>
     </trim>
   </insert>
   <select id="countByExample" parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDOExample" resultType="java.lang.Long">
@@ -116,30 +115,4 @@
       <include refid="Example_Where_Clause" />
     </if>
   </select>
-  <update id="updateByPrimaryKeySelective" parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO">
-    update s2_user
-    <set>
-      <if test="name != null">
-        name = #{name,jdbcType=VARCHAR},
-      </if>
-      <if test="password != null">
-        password = #{password,jdbcType=VARCHAR},
-      </if>
-      <if test="displayName != null">
-        display_name = #{displayName,jdbcType=VARCHAR},
-      </if>
-      <if test="email != null">
-        email = #{email,jdbcType=VARCHAR},
-      </if>
-    </set>
-    where id = #{id,jdbcType=BIGINT}
-  </update>
-  <update id="updateByPrimaryKey" parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO">
-    update s2_user
-    set name = #{name,jdbcType=VARCHAR},
-      password = #{password,jdbcType=VARCHAR},
-      display_name = #{displayName,jdbcType=VARCHAR},
-      email = #{email,jdbcType=VARCHAR}
-    where id = #{id,jdbcType=BIGINT}
-  </update>
 </mapper>

auth/authorization/src/main/java/com/tencent/supersonic/auth/authorization/application/AuthServiceImpl.java

@@ -53,7 +53,7 @@ public class AuthServiceImpl implements AuthService {
     }
 
     @Override
-    public void updateAuthGroup(AuthGroup group) {
+    public void addOrUpdateAuthGroup(AuthGroup group) {
         Gson g = new Gson();
         if (group.getGroupId() == null) {
             int nextGroupId = 1;

auth/authorization/src/main/java/com/tencent/supersonic/auth/authorization/rest/AuthController.java

@@ -40,7 +40,7 @@ public class AuthController {
     @PostMapping("/createGroup")
     public void newAuthGroup(@RequestBody AuthGroup group) {
         group.setGroupId(null);
-        authService.updateAuthGroup(group);
+        authService.addOrUpdateAuthGroup(group);
     }
 
     @PostMapping("/removeGroup")
@@ -58,7 +58,7 @@ public class AuthController {
         if (group.getGroupId() == null || group.getGroupId() == 0) {
             throw new RuntimeException("groupId is empty");
         }
-        authService.updateAuthGroup(group);
+        authService.addOrUpdateAuthGroup(group);
     }
 
     /**

chat/api/src/main/java/com/tencent/supersonic/chat/api/component/SemanticInterpreter.java

@@ -8,9 +8,12 @@ import com.tencent.supersonic.semantic.api.model.request.PageDimensionReq;
 import com.tencent.supersonic.semantic.api.model.request.PageMetricReq;
 import com.tencent.supersonic.semantic.api.model.response.DomainResp;
 import com.tencent.supersonic.semantic.api.model.response.DimensionResp;
+import com.tencent.supersonic.semantic.api.model.response.ExplainResp;
 import com.tencent.supersonic.semantic.api.model.response.ModelResp;
 import com.tencent.supersonic.semantic.api.model.response.MetricResp;
 import com.tencent.supersonic.semantic.api.model.response.QueryResultWithSchemaResp;
+import com.tencent.supersonic.semantic.api.query.request.ExplainSqlReq;
+import com.tencent.supersonic.semantic.api.query.request.QueryDimValueReq;
 import com.tencent.supersonic.semantic.api.query.request.QueryDslReq;
 import com.tencent.supersonic.semantic.api.query.request.QueryMultiStructReq;
 import com.tencent.supersonic.semantic.api.query.request.QueryStructReq;
@@ -28,16 +31,30 @@ import java.util.List;
  * as proxy to a remote semantic service.
  * </p>
  */
-public interface SemanticLayer {
+public interface SemanticInterpreter {
 
     QueryResultWithSchemaResp queryByStruct(QueryStructReq queryStructReq, User user);
+
     QueryResultWithSchemaResp queryByMultiStruct(QueryMultiStructReq queryMultiStructReq, User user);
+
     QueryResultWithSchemaResp queryByDsl(QueryDslReq queryDslReq, User user);
+
+    QueryResultWithSchemaResp queryDimValue(QueryDimValueReq queryDimValueReq, User user);
+
     List<ModelSchema> getModelSchema();
+
     List<ModelSchema> getModelSchema(List<Long> ids);
+
     ModelSchema getModelSchema(Long model, Boolean cacheEnable);
+
     PageInfo<DimensionResp> getDimensionPage(PageDimensionReq pageDimensionCmd);
-    PageInfo<MetricResp> getMetricPage(PageMetricReq pageMetricCmd);
+
+    PageInfo<MetricResp> getMetricPage(PageMetricReq pageMetricCmd, User user);
+
     List<DomainResp> getDomainList(User user);
+
     List<ModelResp> getModelList(AuthType authType, Long domainId, User user);
+
+    <T> ExplainResp explain(ExplainSqlReq<T> explainSqlReq, User user) throws Exception;
+
 }

chat/api/src/main/java/com/tencent/supersonic/chat/api/component/SemanticQuery.java

@@ -3,6 +3,7 @@ package com.tencent.supersonic.chat.api.component;
 import com.tencent.supersonic.auth.api.authentication.pojo.User;
 import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
 import com.tencent.supersonic.chat.api.pojo.response.QueryResult;
+import com.tencent.supersonic.semantic.api.model.response.ExplainResp;
 import org.apache.calcite.sql.parser.SqlParseException;
 
 /**
@@ -14,6 +15,8 @@ public interface SemanticQuery {
 
     QueryResult execute(User user) throws SqlParseException;
 
+    ExplainResp explain(User user);
+
     SemanticParseInfo getParseInfo();
 
     void setParseInfo(SemanticParseInfo parseInfo);

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/SchemaElement.java

@@ -15,6 +15,7 @@ import lombok.NoArgsConstructor;
 @AllArgsConstructor
 @NoArgsConstructor
 public class SchemaElement implements Serializable {
+
     private Long model;
     private Long id;
     private String name;
@@ -26,6 +27,8 @@ public class SchemaElement implements Serializable {
 
     private List<SchemaValueMap> schemaValueMaps;
 
+    private String defaultAgg;
+
     @Override
     public boolean equals(Object o) {
         if (this == o) {

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/SemanticParseInfo.java

@@ -1,20 +1,21 @@
 package com.tencent.supersonic.chat.api.pojo;
 
 
-import java.util.List;
-import java.util.Set;
-import java.util.TreeSet;
-import java.util.LinkedHashSet;
-import java.util.ArrayList;
-import java.util.Map;
-import java.util.HashMap;
-import java.util.Comparator;
-
 import com.tencent.supersonic.chat.api.pojo.request.QueryFilter;
 import com.tencent.supersonic.chat.api.pojo.response.EntityInfo;
+import com.tencent.supersonic.chat.api.pojo.response.SqlInfo;
 import com.tencent.supersonic.common.pojo.DateConf;
 import com.tencent.supersonic.common.pojo.Order;
 import com.tencent.supersonic.common.pojo.enums.AggregateTypeEnum;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeSet;
+import com.tencent.supersonic.common.pojo.enums.FilterType;
 import lombok.Data;
 
 @Data
@@ -27,6 +28,7 @@ public class SemanticParseInfo {
     private Set<SchemaElement> dimensions = new LinkedHashSet();
     private SchemaElement entity;
     private AggregateTypeEnum aggType = AggregateTypeEnum.NONE;
+    private FilterType filterType = FilterType.UNION;
     private Set<QueryFilter> dimensionFilters = new LinkedHashSet();
     private Set<QueryFilter> metricFilters = new LinkedHashSet();
     private Set<Order> orders = new LinkedHashSet();
@@ -37,6 +39,8 @@ public class SemanticParseInfo {
     private List<SchemaElementMatch> elementMatches = new ArrayList<>();
     private Map<String, Object> properties = new HashMap<>();
     private EntityInfo entityInfo;
+    private SqlInfo sqlInfo = new SqlInfo();
+
     public Long getModelId() {
         return model != null ? model.getId() : 0L;
     }
@@ -46,6 +50,7 @@ public class SemanticParseInfo {
     }
 
     private static class SchemaNameLengthComparator implements Comparator<SchemaElement> {
+
         @Override
         public int compare(SchemaElement o1, SchemaElement o2) {
             int len1 = o1.getName().length();

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/SemanticSchema.java

@@ -7,6 +7,7 @@ import java.util.Map;
 import java.util.stream.Collectors;
 
 public class SemanticSchema implements Serializable {
+
     private List<ModelSchema> modelSchemaList;
 
     public SemanticSchema(List<ModelSchema> modelSchemaList) {
@@ -34,12 +35,28 @@ public class SemanticSchema implements Serializable {
         return dimensions;
     }
 
+    public List<SchemaElement> getDimensions(Long modelId) {
+        List<SchemaElement> dimensions = getDimensions();
+        return getElementsByModelId(modelId, dimensions);
+    }
+
     public List<SchemaElement> getMetrics() {
         List<SchemaElement> metrics = new ArrayList<>();
         modelSchemaList.stream().forEach(d -> metrics.addAll(d.getMetrics()));
         return metrics;
     }
 
+    public List<SchemaElement> getMetrics(Long modelId) {
+        List<SchemaElement> metrics = getMetrics();
+        return getElementsByModelId(modelId, metrics);
+    }
+
+    private List<SchemaElement> getElementsByModelId(Long modelId, List<SchemaElement> elements) {
+        return elements.stream()
+                .filter(schemaElement -> modelId.equals(schemaElement.getModel()))
+                .collect(Collectors.toList());
+    }
+
     public List<SchemaElement> getModels() {
         List<SchemaElement> models = new ArrayList<>();
         modelSchemaList.stream().forEach(d -> models.add(d.getModel()));

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/DictLatestTaskReq.java

@@ -0,0 +1,23 @@
+package com.tencent.supersonic.chat.api.pojo.request;
+
+import lombok.Data;
+import lombok.NoArgsConstructor;
+import lombok.ToString;
+
+import javax.validation.constraints.NotNull;
+import java.util.List;
+
+import static java.time.LocalDate.now;
+
+@ToString
+@Data
+@NoArgsConstructor
+public class DictLatestTaskReq {
+
+    @NotNull
+    private Long modelId;
+
+    private List<Long> dimIds;
+
+    private String createdAt = now().plusDays(-4).toString();
+}

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/DictTaskFilterReq.java

@@ -0,0 +1,20 @@
+package com.tencent.supersonic.chat.api.pojo.request;
+
+import com.tencent.supersonic.common.pojo.enums.TaskStatusEnum;
+import lombok.Data;
+import lombok.ToString;
+
+@ToString
+@Data
+public class DictTaskFilterReq {
+
+    private Long id;
+
+    private String name;
+
+    private String createdBy;
+
+    private String createdAt;
+
+    private TaskStatusEnum status;
+}

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/DimensionValueReq.java

@@ -4,6 +4,10 @@ import lombok.Data;
 
 @Data
 public class DimensionValueReq {
+    private Integer agentId;
+
+    private Long elementID;
+
     private Long modelId;
 
     private String bizName;

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/PageQueryInfoReq.java

@@ -1,6 +1,7 @@
 package com.tencent.supersonic.chat.api.pojo.request;
 
 import lombok.Data;
+import java.util.List;
 
 @Data
 public class PageQueryInfoReq {
@@ -11,27 +12,5 @@ public class PageQueryInfoReq {
 
     private String userName;
 
-    public int getPageSize() {
-        return pageSize;
-    }
-
-    public void setPageSize(int pageSize) {
-        this.pageSize = pageSize;
-    }
-
-    public int getCurrent() {
-        return current;
-    }
-
-    public void setCurrent(int current) {
-        this.current = current;
-    }
-
-    public String getUserName() {
-        return userName;
-    }
-
-    public void setUserName(String userName) {
-        this.userName = userName;
-    }
+    private List<Long> ids;
 }

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/QueryDataReq.java

@@ -1,25 +1,21 @@
 package com.tencent.supersonic.chat.api.pojo.request;
 
 
+import com.tencent.supersonic.auth.api.authentication.pojo.User;
 import com.tencent.supersonic.chat.api.pojo.SchemaElement;
 import com.tencent.supersonic.common.pojo.DateConf;
-import com.tencent.supersonic.common.pojo.Order;
-import com.tencent.supersonic.common.pojo.enums.AggregateTypeEnum;
 import java.util.HashSet;
 import java.util.Set;
 import lombok.Data;
 
 @Data
 public class QueryDataReq {
-    String queryMode;
-    SchemaElement model;
-    Set<SchemaElement> metrics = new HashSet<>();
-    Set<SchemaElement> dimensions = new HashSet<>();
-    Set<QueryFilter> dimensionFilters = new HashSet<>();
-    Set<QueryFilter> metricFilters = new HashSet<>();
-    private AggregateTypeEnum aggType = AggregateTypeEnum.NONE;
-    private Set<Order> orders = new HashSet<>();
+    private User user;
+    private Set<SchemaElement> metrics = new HashSet<>();
+    private Set<SchemaElement> dimensions = new HashSet<>();
+    private Set<QueryFilter> dimensionFilters = new HashSet<>();
+    private Set<QueryFilter> metricFilters = new HashSet<>();
     private DateConf dateInfo;
-    private Long limit;
-    private Boolean nativeQuery = false;
+    private Long queryId = 7L;
+    private Integer parseId = 2;
 }

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/QueryFilter.java

@@ -1,7 +1,7 @@
 package com.tencent.supersonic.chat.api.pojo.request;
 
+import com.google.common.base.Objects;
 import com.tencent.supersonic.semantic.api.query.enums.FilterOperatorEnum;
-import java.util.Objects;
 import lombok.Data;
 import lombok.ToString;
 
@@ -19,6 +19,8 @@ public class QueryFilter {
 
     private Long elementID;
 
+    private String function;
+
     @Override
     public boolean equals(Object o) {
         if (this == o) {
@@ -27,14 +29,15 @@ public class QueryFilter {
         if (o == null || getClass() != o.getClass()) {
             return false;
         }
-        QueryFilter filter = (QueryFilter) o;
-        return Objects.equals(bizName, filter.bizName) && Objects.equals(name, filter.name)
-                && operator == filter.operator && Objects.equals(value, filter.value) && Objects.equals(
-                elementID, filter.elementID);
+        QueryFilter that = (QueryFilter) o;
+        return Objects.equal(bizName, that.bizName) && Objects.equal(name,
+                that.name) && operator == that.operator && Objects.equal(value, that.value)
+                && Objects.equal(elementID, that.elementID) && Objects.equal(
+                function, that.function);
     }
 
     @Override
     public int hashCode() {
-        return Objects.hash(bizName, name, operator, value, elementID);
+        return Objects.hashCode(bizName, name, operator, value, elementID, function);
     }
 }

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/SolvedQueryReq.java

@@ -0,0 +1,25 @@
+package com.tencent.supersonic.chat.api.pojo.request;
+
+
+import lombok.AllArgsConstructor;
+import lombok.Builder;
+import lombok.Data;
+import lombok.NoArgsConstructor;
+
+@Data
+@NoArgsConstructor
+@AllArgsConstructor
+@Builder
+public class SolvedQueryReq {
+
+    private Long queryId;
+
+    private Integer parseId;
+
+    private String queryText;
+
+    private Long modelId;
+
+    private Integer agentId;
+
+}

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/DictLatestTaskResp.java

@@ -0,0 +1,30 @@
+package com.tencent.supersonic.chat.api.pojo.response;
+
+import com.tencent.supersonic.common.pojo.enums.TaskStatusEnum;
+import lombok.Data;
+import lombok.ToString;
+
+import java.util.Date;
+
+@ToString
+@Data
+public class DictLatestTaskResp {
+
+    private Long dimId;
+
+    private Long id;
+
+    private String name;
+
+    private String description;
+
+    private String command;
+
+    private TaskStatusEnum status;
+
+    private String createdBy;
+
+    private Date createdAt;
+
+    private Long elapsedMs;
+}

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/ModelInfo.java

@@ -8,5 +8,6 @@ import lombok.Data;
 public class ModelInfo extends DataInfo implements Serializable {
 
     private List<String> words;
+    private String primaryEntityName;
     private String primaryEntityBizName;
 }

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/ParseResp.java

@@ -21,6 +21,7 @@ public class ParseResp {
     private ParseState state;
     private List<SemanticParseInfo> selectedParses;
     private List<SemanticParseInfo> candidateParses;
+    private List<SolvedQueryRecallResp> similarSolvedQuery;
 
     public enum ParseState {
         COMPLETED,

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/SolvedQueryRecallResp.java

@@ -0,0 +1,17 @@
+package com.tencent.supersonic.chat.api.pojo.response;
+
+
+import lombok.Builder;
+import lombok.Data;
+
+@Data
+@Builder
+public class SolvedQueryRecallResp {
+
+    private Long queryId;
+
+    private Integer parseId;
+
+    private String queryText;
+
+}

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/SqlInfo.java

@@ -0,0 +1,11 @@
+package com.tencent.supersonic.chat.api.pojo.response;
+
+import lombok.Data;
+
+@Data
+public class SqlInfo {
+
+    private String llmParseSql;
+    private String logicSql;
+    private String querySql;
+}

chat/core/src/main/java/com/tencent/supersonic/chat/config/LLMParserConfig.java

@@ -7,13 +7,19 @@ import org.springframework.context.annotation.Configuration;
 
 @Configuration
 @Data
-public class LLMConfig {
+public class LLMParserConfig {
 
 
-    @Value("${llm.url:}")
+    @Value("${llm.parser.url:}")
     private String url;
 
     @Value("${query2sql.path:/query2sql}")
     private String queryToSqlPath;
 
+    @Value("${dimension.topn:5}")
+    private Integer dimensionTopN;
+
+    @Value("${metric.topn:5}")
+    private Integer metricTopN;
+
 }

chat/core/src/main/java/com/tencent/supersonic/chat/corrector/BaseSemanticCorrector.java

@@ -2,20 +2,32 @@ package com.tencent.supersonic.chat.corrector;
 
 import com.tencent.supersonic.chat.api.component.SemanticCorrector;
 import com.tencent.supersonic.chat.api.pojo.SchemaElement;
+import com.tencent.supersonic.chat.api.pojo.SemanticCorrectInfo;
 import com.tencent.supersonic.chat.api.pojo.SemanticSchema;
+import com.tencent.supersonic.common.pojo.enums.AggregateTypeEnum;
 import com.tencent.supersonic.common.util.ContextUtils;
+import com.tencent.supersonic.common.util.DateUtils;
+import com.tencent.supersonic.common.util.jsqlparser.SqlParserAddHelper;
+import com.tencent.supersonic.common.util.jsqlparser.SqlParserSelectHelper;
 import com.tencent.supersonic.knowledge.service.SchemaService;
-import com.tencent.supersonic.semantic.api.model.enums.TimeDimensionEnum;
 import java.util.ArrayList;
+import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
 import java.util.stream.Collectors;
 import lombok.extern.slf4j.Slf4j;
+import org.springframework.util.CollectionUtils;
 
 @Slf4j
 public abstract class BaseSemanticCorrector implements SemanticCorrector {
-    public static final String DATE_FIELD = "数据日期";
-    protected Map<String, String> getFieldToBizName(Long modelId) {
+
+    public void correct(SemanticCorrectInfo semanticCorrectInfo) {
+        semanticCorrectInfo.setPreSql(semanticCorrectInfo.getSql());
+    }
+
+    protected Map<String, String> getFieldNameMap(Long modelId) {
 
         SemanticSchema semanticSchema = ContextUtils.getBean(SchemaService.class).getSemanticSchema();
 
@@ -25,9 +37,56 @@ public abstract class BaseSemanticCorrector implements SemanticCorrector {
 
         Map<String, String> result = dbAllFields.stream()
                 .filter(entry -> entry.getModel().equals(modelId))
-                .collect(Collectors.toMap(SchemaElement::getName, a -> a.getBizName(), (k1, k2) -> k1));
-        result.put(DATE_FIELD, TimeDimensionEnum.DAY.getName());
+                .collect(Collectors.toMap(SchemaElement::getName, a -> a.getName(), (k1, k2) -> k1));
+        result.put(DateUtils.DATE_FIELD, DateUtils.DATE_FIELD);
         return result;
     }
 
+    protected void addFieldsToSelect(SemanticCorrectInfo semanticCorrectInfo, String sql) {
+        Set<String> selectFields = new HashSet<>(SqlParserSelectHelper.getSelectFields(sql));
+        Set<String> needAddFields = new HashSet<>(SqlParserSelectHelper.getGroupByFields(sql));
+        needAddFields.addAll(SqlParserSelectHelper.getOrderByFields(sql));
+
+        if (CollectionUtils.isEmpty(selectFields) || CollectionUtils.isEmpty(needAddFields)) {
+            return;
+        }
+
+        needAddFields.removeAll(selectFields);
+        needAddFields.remove(DateUtils.DATE_FIELD);
+        String replaceFields = SqlParserAddHelper.addFieldsToSelect(sql, new ArrayList<>(needAddFields));
+        semanticCorrectInfo.setSql(replaceFields);
+    }
+
+    protected void addAggregateToMetric(SemanticCorrectInfo semanticCorrectInfo) {
+        //add aggregate to all metric
+        String sql = semanticCorrectInfo.getSql();
+        Long modelId = semanticCorrectInfo.getParseInfo().getModel().getModel();
+
+        List<SchemaElement> metrics = getMetricElements(modelId);
+
+        Map<String, String> metricToAggregate = metrics.stream()
+                .map(schemaElement -> {
+                    if (Objects.isNull(schemaElement.getDefaultAgg())) {
+                        schemaElement.setDefaultAgg(AggregateTypeEnum.SUM.name());
+                    }
+                    return schemaElement;
+                }).collect(Collectors.toMap(a -> a.getName(), a -> a.getDefaultAgg(), (k1, k2) -> k1));
+
+        if (CollectionUtils.isEmpty(metricToAggregate)) {
+            return;
+        }
+
+        String aggregateSql = SqlParserAddHelper.addAggregateToField(sql, metricToAggregate);
+        semanticCorrectInfo.setSql(aggregateSql);
+    }
+
+    protected List<SchemaElement> getMetricElements(Long modelId) {
+        SemanticSchema semanticSchema = ContextUtils.getBean(SchemaService.class).getSemanticSchema();
+        return semanticSchema.getMetrics(modelId);
+    }
+
+    protected List<SchemaElement> getDimensionElements(Long modelId) {
+        SemanticSchema semanticSchema = ContextUtils.getBean(SchemaService.class).getSemanticSchema();
+        return semanticSchema.getDimensions(modelId);
+    }
 }

chat/core/src/main/java/com/tencent/supersonic/chat/corrector/DateFieldCorrector.java

@@ -1,27 +0,0 @@
-package com.tencent.supersonic.chat.corrector;
-
-import com.tencent.supersonic.chat.api.pojo.SemanticCorrectInfo;
-import com.tencent.supersonic.chat.parser.llm.dsl.DSLDateHelper;
-import com.tencent.supersonic.common.util.jsqlparser.SqlParserSelectHelper;
-import com.tencent.supersonic.common.util.jsqlparser.SqlParserUpdateHelper;
-import java.util.List;
-import lombok.extern.slf4j.Slf4j;
-import org.springframework.util.CollectionUtils;
-
-@Slf4j
-public class DateFieldCorrector extends BaseSemanticCorrector {
-
-    @Override
-    public void correct(SemanticCorrectInfo semanticCorrectInfo) {
-
-        String sql = semanticCorrectInfo.getSql();
-        List<String> whereFields = SqlParserSelectHelper.getWhereFields(sql);
-        if (CollectionUtils.isEmpty(whereFields) || !whereFields.contains(DATE_FIELD)) {
-            String currentDate = DSLDateHelper.getReferenceDate(semanticCorrectInfo.getParseInfo().getModelId());
-            sql = SqlParserUpdateHelper.addWhere(sql, DATE_FIELD, currentDate);
-        }
-        semanticCorrectInfo.setPreSql(semanticCorrectInfo.getSql());
-        semanticCorrectInfo.setSql(sql);
-    }
-
-}

chat/core/src/main/java/com/tencent/supersonic/chat/corrector/FieldCorrector.java

@@ -1,18 +0,0 @@
-package com.tencent.supersonic.chat.corrector;
-
-import com.tencent.supersonic.chat.api.pojo.SemanticCorrectInfo;
-import com.tencent.supersonic.common.util.jsqlparser.SqlParserUpdateHelper;
-import lombok.extern.slf4j.Slf4j;
-
-@Slf4j
-public class FieldCorrector extends BaseSemanticCorrector {
-
-    @Override
-    public void correct(SemanticCorrectInfo semanticCorrectInfo) {
-        String preSql = semanticCorrectInfo.getSql();
-        semanticCorrectInfo.setPreSql(preSql);
-        String sql = SqlParserUpdateHelper.replaceFields(preSql,
-                getFieldToBizName(semanticCorrectInfo.getParseInfo().getModelId()));
-        semanticCorrectInfo.setSql(sql);
-    }
-}

chat/core/src/main/java/com/tencent/supersonic/chat/corrector/FieldNameCorrector.java

@@ -1,49 +0,0 @@
-package com.tencent.supersonic.chat.corrector;
-
-import com.tencent.supersonic.chat.api.pojo.SemanticCorrectInfo;
-import com.tencent.supersonic.chat.parser.llm.dsl.DSLParseResult;
-import com.tencent.supersonic.chat.query.llm.dsl.LLMReq;
-import com.tencent.supersonic.chat.query.llm.dsl.LLMReq.ElementValue;
-import com.tencent.supersonic.common.pojo.Constants;
-import com.tencent.supersonic.common.util.JsonUtil;
-import com.tencent.supersonic.common.util.jsqlparser.SqlParserUpdateHelper;
-import java.util.List;
-import java.util.Map;
-import java.util.Objects;
-import java.util.Set;
-import java.util.stream.Collectors;
-import lombok.extern.slf4j.Slf4j;
-import org.springframework.util.CollectionUtils;
-
-@Slf4j
-public class FieldNameCorrector extends BaseSemanticCorrector {
-
-    @Override
-    public void correct(SemanticCorrectInfo semanticCorrectInfo) {
-
-        Object context = semanticCorrectInfo.getParseInfo().getProperties().get(Constants.CONTEXT);
-        if (Objects.isNull(context)) {
-            return;
-        }
-
-        DSLParseResult dslParseResult = JsonUtil.toObject(JsonUtil.toString(context), DSLParseResult.class);
-        if (Objects.isNull(dslParseResult) || Objects.isNull(dslParseResult.getLlmReq())) {
-            return;
-        }
-        LLMReq llmReq = dslParseResult.getLlmReq();
-        List<ElementValue> linking = llmReq.getLinking();
-        if (CollectionUtils.isEmpty(linking)) {
-            return;
-        }
-
-        Map<String, Set<String>> fieldValueToFieldNames = linking.stream().collect(
-                Collectors.groupingBy(ElementValue::getFieldValue,
-                        Collectors.mapping(ElementValue::getFieldName, Collectors.toSet())));
-
-        String preSql = semanticCorrectInfo.getSql();
-        semanticCorrectInfo.setPreSql(preSql);
-        String sql = SqlParserUpdateHelper.replaceFieldNameByValue(preSql, fieldValueToFieldNames);
-        semanticCorrectInfo.setSql(sql);
-    }
-
-}

chat/core/src/main/java/com/tencent/supersonic/chat/corrector/FieldValueCorrector.java

@@ -1,81 +0,0 @@
-package com.tencent.supersonic.chat.corrector;
-
-import com.tencent.supersonic.chat.api.pojo.SemanticCorrectInfo;
-import com.tencent.supersonic.chat.api.pojo.SchemaElement;
-import com.tencent.supersonic.chat.api.pojo.SchemaValueMap;
-import com.tencent.supersonic.chat.api.pojo.SemanticSchema;
-import com.tencent.supersonic.common.util.ContextUtils;
-import com.tencent.supersonic.common.util.jsqlparser.SqlParserUpdateHelper;
-import com.tencent.supersonic.knowledge.service.SchemaService;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-import java.util.Objects;
-import java.util.stream.Collectors;
-import lombok.extern.slf4j.Slf4j;
-import org.apache.logging.log4j.util.Strings;
-import org.springframework.util.CollectionUtils;
-
-@Slf4j
-public class FieldValueCorrector extends BaseSemanticCorrector {
-
-    @Override
-    public void correct(SemanticCorrectInfo semanticCorrectInfo) {
-        SemanticSchema semanticSchema = ContextUtils.getBean(SchemaService.class).getSemanticSchema();
-        Long modelId = semanticCorrectInfo.getParseInfo().getModel().getId();
-        List<SchemaElement> dimensions = semanticSchema.getDimensions().stream()
-                .filter(schemaElement -> modelId.equals(schemaElement.getModel()))
-                .collect(Collectors.toList());
-
-        if (CollectionUtils.isEmpty(dimensions)) {
-            return;
-        }
-
-        Map<String, Map<String, String>> aliasAndBizNameToTechName = getAliasAndBizNameToTechName(dimensions);
-        String preSql = semanticCorrectInfo.getSql();
-        semanticCorrectInfo.setPreSql(preSql);
-        String sql = SqlParserUpdateHelper.replaceValue(preSql, aliasAndBizNameToTechName);
-        semanticCorrectInfo.setSql(sql);
-        return;
-    }
-
-
-    private Map<String, Map<String, String>> getAliasAndBizNameToTechName(List<SchemaElement> dimensions) {
-        if (CollectionUtils.isEmpty(dimensions)) {
-            return new HashMap<>();
-        }
-
-        Map<String, Map<String, String>> result = new HashMap<>();
-
-        for (SchemaElement dimension : dimensions) {
-            if (Objects.isNull(dimension)
-                    || Strings.isEmpty(dimension.getBizName())
-                    || CollectionUtils.isEmpty(dimension.getSchemaValueMaps())) {
-                continue;
-            }
-            String bizName = dimension.getBizName();
-
-            Map<String, String> aliasAndBizNameToTechName = new HashMap<>();
-
-            for (SchemaValueMap valueMap : dimension.getSchemaValueMaps()) {
-                if (Objects.isNull(valueMap) || Strings.isEmpty(valueMap.getTechName())) {
-                    continue;
-                }
-                if (Strings.isNotEmpty(valueMap.getBizName())) {
-                    aliasAndBizNameToTechName.put(valueMap.getBizName(), valueMap.getTechName());
-                }
-                if (!CollectionUtils.isEmpty(valueMap.getAlias())) {
-                    valueMap.getAlias().stream().forEach(alias -> {
-                        if (Strings.isNotEmpty(alias)) {
-                            aliasAndBizNameToTechName.put(alias, valueMap.getTechName());
-                        }
-                    });
-                }
-            }
-            if (!CollectionUtils.isEmpty(aliasAndBizNameToTechName)) {
-                result.put(bizName, aliasAndBizNameToTechName);
-            }
-        }
-        return result;
-    }
-}

chat/core/src/main/java/com/tencent/supersonic/chat/corrector/FunctionAliasCorrector.java

@@ -1,16 +0,0 @@
-package com.tencent.supersonic.chat.corrector;
-
-import com.tencent.supersonic.chat.api.pojo.SemanticCorrectInfo;
-import com.tencent.supersonic.common.util.jsqlparser.SqlParserUpdateHelper;
-import lombok.extern.slf4j.Slf4j;
-
-@Slf4j
-public class FunctionAliasCorrector extends BaseSemanticCorrector {
-
-    @Override
-    public void correct(SemanticCorrectInfo semanticCorrectInfo) {
-        String replaceAlias = SqlParserUpdateHelper.replaceAlias(semanticCorrectInfo.getSql());
-        semanticCorrectInfo.setSql(replaceAlias);
-    }
-
-}

chat/core/src/main/java/com/tencent/supersonic/chat/corrector/FunctionCorrector.java

@@ -1,17 +0,0 @@
-package com.tencent.supersonic.chat.corrector;
-
-import com.tencent.supersonic.chat.api.pojo.SemanticCorrectInfo;
-import com.tencent.supersonic.common.util.jsqlparser.SqlParserUpdateHelper;
-import lombok.extern.slf4j.Slf4j;
-
-@Slf4j
-public class FunctionCorrector extends BaseSemanticCorrector {
-
-    @Override
-    public void correct(SemanticCorrectInfo semanticCorrectInfo) {
-        String preSql = semanticCorrectInfo.getSql();
-        semanticCorrectInfo.setPreSql(preSql);
-        String sql = SqlParserUpdateHelper.replaceFunction(preSql);
-        semanticCorrectInfo.setSql(sql);
-    }
-}

chat/core/src/main/java/com/tencent/supersonic/chat/corrector/GlobalAfterCorrector.java

@@ -0,0 +1,30 @@
+package com.tencent.supersonic.chat.corrector;
+
+import com.tencent.supersonic.chat.api.pojo.SemanticCorrectInfo;
+import com.tencent.supersonic.common.util.jsqlparser.SqlParserAddHelper;
+import com.tencent.supersonic.common.util.jsqlparser.SqlParserSelectFunctionHelper;
+import com.tencent.supersonic.common.util.jsqlparser.SqlParserSelectHelper;
+import java.util.Objects;
+import lombok.extern.slf4j.Slf4j;
+import net.sf.jsqlparser.expression.Expression;
+
+@Slf4j
+public class GlobalAfterCorrector extends BaseSemanticCorrector {
+
+    @Override
+    public void correct(SemanticCorrectInfo semanticCorrectInfo) {
+
+        super.correct(semanticCorrectInfo);
+        String sql = semanticCorrectInfo.getSql();
+        if (!SqlParserSelectFunctionHelper.hasAggregateFunction(sql)) {
+            return;
+        }
+        Expression havingExpression = SqlParserSelectHelper.getHavingExpression(sql);
+        if (Objects.nonNull(havingExpression)) {
+            String replaceSql = SqlParserAddHelper.addFunctionToSelect(sql, havingExpression);
+            semanticCorrectInfo.setSql(replaceSql);
+        }
+        return;
+    }
+
+}

chat/core/src/main/java/com/tencent/supersonic/chat/corrector/GlobalBeforeCorrector.java

@@ -0,0 +1,96 @@
+package com.tencent.supersonic.chat.corrector;
+
+import com.tencent.supersonic.chat.api.pojo.SemanticCorrectInfo;
+import com.tencent.supersonic.chat.parser.llm.dsl.DSLParseResult;
+import com.tencent.supersonic.chat.query.llm.dsl.LLMReq;
+import com.tencent.supersonic.chat.query.llm.dsl.LLMReq.ElementValue;
+import com.tencent.supersonic.common.pojo.Constants;
+import com.tencent.supersonic.common.util.JsonUtil;
+import com.tencent.supersonic.common.util.jsqlparser.SqlParserReplaceHelper;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import lombok.extern.slf4j.Slf4j;
+import org.springframework.util.CollectionUtils;
+
+@Slf4j
+public class GlobalBeforeCorrector extends BaseSemanticCorrector {
+
+    @Override
+    public void correct(SemanticCorrectInfo semanticCorrectInfo) {
+
+        super.correct(semanticCorrectInfo);
+
+        replaceAlias(semanticCorrectInfo);
+
+        updateFieldNameByLinkingValue(semanticCorrectInfo);
+
+        updateFieldValueByLinkingValue(semanticCorrectInfo);
+
+        correctFieldName(semanticCorrectInfo);
+    }
+
+    private void replaceAlias(SemanticCorrectInfo semanticCorrectInfo) {
+        String replaceAlias = SqlParserReplaceHelper.replaceAlias(semanticCorrectInfo.getSql());
+        semanticCorrectInfo.setSql(replaceAlias);
+    }
+
+    private void correctFieldName(SemanticCorrectInfo semanticCorrectInfo) {
+
+        Map<String, String> fieldNameMap = getFieldNameMap(semanticCorrectInfo.getParseInfo().getModelId());
+
+        String sql = SqlParserReplaceHelper.replaceFields(semanticCorrectInfo.getSql(), fieldNameMap);
+
+        semanticCorrectInfo.setSql(sql);
+    }
+
+    private void updateFieldNameByLinkingValue(SemanticCorrectInfo semanticCorrectInfo) {
+        List<ElementValue> linking = getLinkingValues(semanticCorrectInfo);
+        if (CollectionUtils.isEmpty(linking)) {
+            return;
+        }
+
+        Map<String, Set<String>> fieldValueToFieldNames = linking.stream().collect(
+                Collectors.groupingBy(ElementValue::getFieldValue,
+                        Collectors.mapping(ElementValue::getFieldName, Collectors.toSet())));
+
+        String sql = SqlParserReplaceHelper.replaceFieldNameByValue(semanticCorrectInfo.getSql(),
+                fieldValueToFieldNames);
+        semanticCorrectInfo.setSql(sql);
+    }
+
+    private List<ElementValue> getLinkingValues(SemanticCorrectInfo semanticCorrectInfo) {
+        Object context = semanticCorrectInfo.getParseInfo().getProperties().get(Constants.CONTEXT);
+        if (Objects.isNull(context)) {
+            return null;
+        }
+
+        DSLParseResult dslParseResult = JsonUtil.toObject(JsonUtil.toString(context), DSLParseResult.class);
+        if (Objects.isNull(dslParseResult) || Objects.isNull(dslParseResult.getLlmReq())) {
+            return null;
+        }
+        LLMReq llmReq = dslParseResult.getLlmReq();
+        return llmReq.getLinking();
+    }
+
+
+    private void updateFieldValueByLinkingValue(SemanticCorrectInfo semanticCorrectInfo) {
+        List<ElementValue> linking = getLinkingValues(semanticCorrectInfo);
+        if (CollectionUtils.isEmpty(linking)) {
+            return;
+        }
+
+        Map<String, Map<String, String>> filedNameToValueMap = linking.stream().collect(
+                Collectors.groupingBy(ElementValue::getFieldName,
+                        Collectors.mapping(ElementValue::getFieldValue, Collectors.toMap(
+                                oldValue -> oldValue,
+                                newValue -> newValue,
+                                (existingValue, newValue) -> newValue)
+                        )));
+
+        String sql = SqlParserReplaceHelper.replaceValue(semanticCorrectInfo.getSql(), filedNameToValueMap, false);
+        semanticCorrectInfo.setSql(sql);
+    }
+}

chat/core/src/main/java/com/tencent/supersonic/chat/corrector/GroupByCorrector.java

@@ -0,0 +1,63 @@
+package com.tencent.supersonic.chat.corrector;
+
+import com.tencent.supersonic.chat.api.pojo.SemanticCorrectInfo;
+import com.tencent.supersonic.chat.api.pojo.SemanticSchema;
+import com.tencent.supersonic.common.util.ContextUtils;
+import com.tencent.supersonic.common.util.DateUtils;
+import com.tencent.supersonic.common.util.jsqlparser.SqlParserAddHelper;
+import com.tencent.supersonic.common.util.jsqlparser.SqlParserSelectHelper;
+import com.tencent.supersonic.knowledge.service.SchemaService;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+import lombok.extern.slf4j.Slf4j;
+import org.springframework.util.CollectionUtils;
+
+@Slf4j
+public class GroupByCorrector extends BaseSemanticCorrector {
+
+    @Override
+    public void correct(SemanticCorrectInfo semanticCorrectInfo) {
+
+        super.correct(semanticCorrectInfo);
+
+        addGroupByFields(semanticCorrectInfo);
+
+        addAggregate(semanticCorrectInfo);
+    }
+
+    private void addGroupByFields(SemanticCorrectInfo semanticCorrectInfo) {
+        Long modelId = semanticCorrectInfo.getParseInfo().getModel().getModel();
+
+        //add dimension group by
+        String sql = semanticCorrectInfo.getSql();
+        SemanticSchema semanticSchema = ContextUtils.getBean(SchemaService.class).getSemanticSchema();
+        Set<String> dimensions = semanticSchema.getDimensions(modelId).stream()
+                .map(schemaElement -> schemaElement.getName()).collect(Collectors.toSet());
+        dimensions.add(DateUtils.DATE_FIELD);
+        List<String> selectFields = SqlParserSelectHelper.getSelectFields(sql);
+
+        if (CollectionUtils.isEmpty(selectFields) || CollectionUtils.isEmpty(dimensions)) {
+            return;
+        }
+        List<String> aggregateFields = SqlParserSelectHelper.getAggregateFields(sql);
+        Set<String> groupByFields = selectFields.stream()
+                .filter(field -> dimensions.contains(field))
+                .filter(field -> {
+                    if (!CollectionUtils.isEmpty(aggregateFields) && aggregateFields.contains(field)) {
+                        return false;
+                    }
+                    return true;
+                })
+                .collect(Collectors.toSet());
+        semanticCorrectInfo.setSql(SqlParserAddHelper.addGroupBy(sql, groupByFields));
+    }
+
+    private void addAggregate(SemanticCorrectInfo semanticCorrectInfo) {
+        List<String> sqlGroupByFields = SqlParserSelectHelper.getGroupByFields(semanticCorrectInfo.getSql());
+        if (CollectionUtils.isEmpty(sqlGroupByFields)) {
+            return;
+        }
+        addAggregateToMetric(semanticCorrectInfo);
+    }
+}

chat/core/src/main/java/com/tencent/supersonic/chat/corrector/HavingCorrector.java

@@ -0,0 +1,37 @@
+package com.tencent.supersonic.chat.corrector;
+
+import com.tencent.supersonic.chat.api.pojo.SemanticCorrectInfo;
+import com.tencent.supersonic.chat.api.pojo.SemanticSchema;
+import com.tencent.supersonic.common.util.ContextUtils;
+import com.tencent.supersonic.common.util.jsqlparser.SqlParserAddHelper;
+import com.tencent.supersonic.knowledge.service.SchemaService;
+import java.util.Set;
+import java.util.stream.Collectors;
+import lombok.extern.slf4j.Slf4j;
+import org.springframework.util.CollectionUtils;
+
+@Slf4j
+public class HavingCorrector extends BaseSemanticCorrector {
+
+    @Override
+    public void correct(SemanticCorrectInfo semanticCorrectInfo) {
+
+        super.correct(semanticCorrectInfo);
+
+        //add aggregate to all metric
+        semanticCorrectInfo.setPreSql(semanticCorrectInfo.getSql());
+        Long modelId = semanticCorrectInfo.getParseInfo().getModel().getModel();
+
+        SemanticSchema semanticSchema = ContextUtils.getBean(SchemaService.class).getSemanticSchema();
+
+        Set<String> metrics = semanticSchema.getMetrics(modelId).stream()
+                .map(schemaElement -> schemaElement.getName()).collect(Collectors.toSet());
+
+        if (CollectionUtils.isEmpty(metrics)) {
+            return;
+        }
+        String havingSql = SqlParserAddHelper.addHaving(semanticCorrectInfo.getSql(), metrics);
+        semanticCorrectInfo.setSql(havingSql);
+    }
+
+}

chat/core/src/main/java/com/tencent/supersonic/chat/corrector/QueryFilterAppend.java

@@ -1,48 +0,0 @@
-package com.tencent.supersonic.chat.corrector;
-
-import com.tencent.supersonic.chat.api.pojo.SemanticCorrectInfo;
-import com.tencent.supersonic.chat.api.pojo.request.QueryFilters;
-import com.tencent.supersonic.common.pojo.Constants;
-import com.tencent.supersonic.common.util.StringUtil;
-import com.tencent.supersonic.common.util.jsqlparser.SqlParserUpdateHelper;
-import java.util.Objects;
-import java.util.stream.Collectors;
-import lombok.extern.slf4j.Slf4j;
-import net.sf.jsqlparser.JSQLParserException;
-import net.sf.jsqlparser.expression.Expression;
-import net.sf.jsqlparser.parser.CCJSqlParserUtil;
-import org.apache.commons.collections.CollectionUtils;
-import org.apache.commons.lang3.StringUtils;
-
-@Slf4j
-public class QueryFilterAppend extends BaseSemanticCorrector {
-
-    @Override
-    public void correct(SemanticCorrectInfo semanticCorrectInfo) throws JSQLParserException {
-        String queryFilter = getQueryFilter(semanticCorrectInfo.getQueryFilters());
-        String preSql = semanticCorrectInfo.getSql();
-
-        if (StringUtils.isNotEmpty(queryFilter)) {
-            log.info("add queryFilter to preSql :{}", queryFilter);
-            Expression expression = CCJSqlParserUtil.parseCondExpression(queryFilter);
-            String sql = SqlParserUpdateHelper.addWhere(preSql, expression);
-            semanticCorrectInfo.setPreSql(preSql);
-            semanticCorrectInfo.setSql(sql);
-        }
-    }
-
-    private String getQueryFilter(QueryFilters queryFilters) {
-        if (Objects.isNull(queryFilters) || CollectionUtils.isEmpty(queryFilters.getFilters())) {
-            return null;
-        }
-        return queryFilters.getFilters().stream()
-                .map(filter -> {
-                    String bizNameWrap = StringUtil.getSpaceWrap(filter.getBizName());
-                    String operatorWrap = StringUtil.getSpaceWrap(filter.getOperator().getValue());
-                    String valueWrap = StringUtil.getCommaWrap(filter.getValue().toString());
-                    return bizNameWrap + operatorWrap + valueWrap;
-                })
-                .collect(Collectors.joining(Constants.AND_UPPER));
-    }
-
-}

chat/core/src/main/java/com/tencent/supersonic/chat/corrector/SelectCorrector.java

@@ -0,0 +1,26 @@
+package com.tencent.supersonic.chat.corrector;
+
+import com.tencent.supersonic.chat.api.pojo.SemanticCorrectInfo;
+import com.tencent.supersonic.common.util.jsqlparser.SqlParserSelectHelper;
+import java.util.List;
+import lombok.extern.slf4j.Slf4j;
+import org.springframework.util.CollectionUtils;
+
+@Slf4j
+public class SelectCorrector extends BaseSemanticCorrector {
+
+    @Override
+    public void correct(SemanticCorrectInfo semanticCorrectInfo) {
+        super.correct(semanticCorrectInfo);
+        String sql = semanticCorrectInfo.getSql();
+        List<String> aggregateFields = SqlParserSelectHelper.getAggregateFields(sql);
+        List<String> selectFields = SqlParserSelectHelper.getSelectFields(sql);
+        // If the number of aggregated fields is equal to the number of queried fields, do not add fields to select.
+        if (!CollectionUtils.isEmpty(aggregateFields)
+                && !CollectionUtils.isEmpty(selectFields)
+                && aggregateFields.size() == selectFields.size()) {
+            return;
+        }
+        addFieldsToSelect(semanticCorrectInfo, sql);
+    }
+}

chat/core/src/main/java/com/tencent/supersonic/chat/corrector/SelectFieldAppendCorrector.java

@@ -1,38 +0,0 @@
-package com.tencent.supersonic.chat.corrector;
-
-import com.tencent.supersonic.chat.api.pojo.SemanticCorrectInfo;
-import com.tencent.supersonic.common.util.jsqlparser.SqlParserSelectHelper;
-import com.tencent.supersonic.common.util.jsqlparser.SqlParserUpdateHelper;
-import com.tencent.supersonic.semantic.api.model.enums.TimeDimensionEnum;
-import java.util.ArrayList;
-import java.util.HashSet;
-import java.util.Set;
-import lombok.extern.slf4j.Slf4j;
-import org.springframework.util.CollectionUtils;
-
-@Slf4j
-public class SelectFieldAppendCorrector extends BaseSemanticCorrector {
-
-    @Override
-    public void correct(SemanticCorrectInfo semanticCorrectInfo) {
-        String preSql = semanticCorrectInfo.getSql();
-        if (SqlParserSelectHelper.hasAggregateFunction(preSql)) {
-            return;
-        }
-        Set<String> selectFields = new HashSet<>(SqlParserSelectHelper.getSelectFields(preSql));
-        Set<String> whereFields = new HashSet<>(SqlParserSelectHelper.getWhereFields(preSql));
-
-        if (CollectionUtils.isEmpty(selectFields) || CollectionUtils.isEmpty(whereFields)) {
-            return;
-        }
-
-        whereFields.addAll(SqlParserSelectHelper.getOrderByFields(preSql));
-        whereFields.removeAll(selectFields);
-        whereFields.remove(TimeDimensionEnum.DAY.getName());
-        whereFields.remove(TimeDimensionEnum.WEEK.getName());
-        whereFields.remove(TimeDimensionEnum.MONTH.getName());
-        String replaceFields = SqlParserUpdateHelper.addFieldsToSelect(preSql, new ArrayList<>(whereFields));
-        semanticCorrectInfo.setPreSql(preSql);
-        semanticCorrectInfo.setSql(replaceFields);
-    }
-}

chat/core/src/main/java/com/tencent/supersonic/chat/corrector/TableNameCorrector.java

@@ -1,21 +0,0 @@
-package com.tencent.supersonic.chat.corrector;
-
-import com.tencent.supersonic.chat.api.pojo.SemanticCorrectInfo;
-import com.tencent.supersonic.common.util.jsqlparser.SqlParserUpdateHelper;
-import lombok.extern.slf4j.Slf4j;
-
-@Slf4j
-public class TableNameCorrector extends BaseSemanticCorrector {
-
-    public static final String TABLE_PREFIX = "t_";
-
-    @Override
-    public void correct(SemanticCorrectInfo semanticCorrectInfo) {
-        Long modelId = semanticCorrectInfo.getParseInfo().getModelId();
-        String preSql = semanticCorrectInfo.getSql();
-        semanticCorrectInfo.setPreSql(preSql);
-        String sql = SqlParserUpdateHelper.replaceTable(preSql, TABLE_PREFIX + modelId);
-        semanticCorrectInfo.setSql(sql);
-    }
-
-}

chat/core/src/main/java/com/tencent/supersonic/chat/corrector/WhereCorrector.java

@@ -0,0 +1,151 @@
+package com.tencent.supersonic.chat.corrector;
+
+import com.tencent.supersonic.chat.api.pojo.SchemaElement;
+import com.tencent.supersonic.chat.api.pojo.SchemaValueMap;
+import com.tencent.supersonic.chat.api.pojo.SemanticCorrectInfo;
+import com.tencent.supersonic.chat.api.pojo.SemanticSchema;
+import com.tencent.supersonic.chat.api.pojo.request.QueryFilters;
+import com.tencent.supersonic.chat.parser.llm.dsl.DSLDateHelper;
+import com.tencent.supersonic.common.pojo.Constants;
+import com.tencent.supersonic.common.util.ContextUtils;
+import com.tencent.supersonic.common.util.DateUtils;
+import com.tencent.supersonic.common.util.StringUtil;
+import com.tencent.supersonic.common.util.jsqlparser.SqlParserAddHelper;
+import com.tencent.supersonic.common.util.jsqlparser.SqlParserReplaceHelper;
+import com.tencent.supersonic.common.util.jsqlparser.SqlParserSelectHelper;
+import com.tencent.supersonic.knowledge.service.SchemaService;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import lombok.extern.slf4j.Slf4j;
+import net.sf.jsqlparser.JSQLParserException;
+import net.sf.jsqlparser.expression.Expression;
+import net.sf.jsqlparser.parser.CCJSqlParserUtil;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.logging.log4j.util.Strings;
+import org.springframework.util.CollectionUtils;
+
+@Slf4j
+public class WhereCorrector extends BaseSemanticCorrector {
+
+    @Override
+    public void correct(SemanticCorrectInfo semanticCorrectInfo) {
+
+        super.correct(semanticCorrectInfo);
+
+        addDateIfNotExist(semanticCorrectInfo);
+
+        parserDateDiffFunction(semanticCorrectInfo);
+
+        addQueryFilter(semanticCorrectInfo);
+
+        updateFieldValueByTechName(semanticCorrectInfo);
+    }
+
+    private void addQueryFilter(SemanticCorrectInfo semanticCorrectInfo) {
+        String queryFilter = getQueryFilter(semanticCorrectInfo.getQueryFilters());
+
+        String preSql = semanticCorrectInfo.getSql();
+
+        if (StringUtils.isNotEmpty(queryFilter)) {
+            log.info("add queryFilter to preSql :{}", queryFilter);
+            Expression expression = null;
+            try {
+                expression = CCJSqlParserUtil.parseCondExpression(queryFilter);
+            } catch (JSQLParserException e) {
+                log.error("parseCondExpression", e);
+            }
+            String sql = SqlParserAddHelper.addWhere(preSql, expression);
+            semanticCorrectInfo.setSql(sql);
+        }
+    }
+
+    private void parserDateDiffFunction(SemanticCorrectInfo semanticCorrectInfo) {
+        String sql = semanticCorrectInfo.getSql();
+        sql = SqlParserReplaceHelper.replaceFunction(sql);
+        semanticCorrectInfo.setSql(sql);
+    }
+
+    private void addDateIfNotExist(SemanticCorrectInfo semanticCorrectInfo) {
+        String sql = semanticCorrectInfo.getSql();
+        List<String> whereFields = SqlParserSelectHelper.getWhereFields(sql);
+        if (CollectionUtils.isEmpty(whereFields) || !whereFields.contains(DateUtils.DATE_FIELD)) {
+            String currentDate = DSLDateHelper.getReferenceDate(semanticCorrectInfo.getParseInfo().getModelId());
+            sql = SqlParserAddHelper.addParenthesisToWhere(sql);
+            sql = SqlParserAddHelper.addWhere(sql, DateUtils.DATE_FIELD, currentDate);
+        }
+        semanticCorrectInfo.setSql(sql);
+    }
+
+    private String getQueryFilter(QueryFilters queryFilters) {
+        if (Objects.isNull(queryFilters) || CollectionUtils.isEmpty(queryFilters.getFilters())) {
+            return null;
+        }
+        return queryFilters.getFilters().stream()
+                .map(filter -> {
+                    String bizNameWrap = StringUtil.getSpaceWrap(filter.getName());
+                    String operatorWrap = StringUtil.getSpaceWrap(filter.getOperator().getValue());
+                    String valueWrap = StringUtil.getCommaWrap(filter.getValue().toString());
+                    return bizNameWrap + operatorWrap + valueWrap;
+                })
+                .collect(Collectors.joining(Constants.AND_UPPER));
+    }
+
+    private void updateFieldValueByTechName(SemanticCorrectInfo semanticCorrectInfo) {
+        SemanticSchema semanticSchema = ContextUtils.getBean(SchemaService.class).getSemanticSchema();
+        Long modelId = semanticCorrectInfo.getParseInfo().getModel().getId();
+        List<SchemaElement> dimensions = semanticSchema.getDimensions().stream()
+                .filter(schemaElement -> modelId.equals(schemaElement.getModel()))
+                .collect(Collectors.toList());
+
+        if (CollectionUtils.isEmpty(dimensions)) {
+            return;
+        }
+
+        Map<String, Map<String, String>> aliasAndBizNameToTechName = getAliasAndBizNameToTechName(dimensions);
+        String sql = SqlParserReplaceHelper.replaceValue(semanticCorrectInfo.getSql(), aliasAndBizNameToTechName);
+        semanticCorrectInfo.setSql(sql);
+        return;
+    }
+
+    private Map<String, Map<String, String>> getAliasAndBizNameToTechName(List<SchemaElement> dimensions) {
+        if (CollectionUtils.isEmpty(dimensions)) {
+            return new HashMap<>();
+        }
+
+        Map<String, Map<String, String>> result = new HashMap<>();
+
+        for (SchemaElement dimension : dimensions) {
+            if (Objects.isNull(dimension)
+                    || Strings.isEmpty(dimension.getName())
+                    || CollectionUtils.isEmpty(dimension.getSchemaValueMaps())) {
+                continue;
+            }
+            String name = dimension.getName();
+
+            Map<String, String> aliasAndBizNameToTechName = new HashMap<>();
+
+            for (SchemaValueMap valueMap : dimension.getSchemaValueMaps()) {
+                if (Objects.isNull(valueMap) || Strings.isEmpty(valueMap.getTechName())) {
+                    continue;
+                }
+                if (Strings.isNotEmpty(valueMap.getBizName())) {
+                    aliasAndBizNameToTechName.put(valueMap.getBizName(), valueMap.getTechName());
+                }
+                if (!CollectionUtils.isEmpty(valueMap.getAlias())) {
+                    valueMap.getAlias().stream().forEach(alias -> {
+                        if (Strings.isNotEmpty(alias)) {
+                            aliasAndBizNameToTechName.put(alias, valueMap.getTechName());
+                        }
+                    });
+                }
+            }
+            if (!CollectionUtils.isEmpty(aliasAndBizNameToTechName)) {
+                result.put(name, aliasAndBizNameToTechName);
+            }
+        }
+        return result;
+    }
+}

chat/core/src/main/java/com/tencent/supersonic/chat/mapper/HanlpDictMapper.java

@@ -37,14 +37,12 @@ public class HanlpDictMapper implements SchemaMapper {
         String queryText = queryContext.getRequest().getQueryText();
         List<Term> terms = HanlpHelper.getTerms(queryText);
 
-        for (Term term : terms) {
-            log.info("word:{},nature:{},frequency:{}", term.word, term.nature.toString(), term.getFrequency());
-        }
-
         QueryMatchStrategy matchStrategy = ContextUtils.getBean(QueryMatchStrategy.class);
         MapperHelper mapperHelper = ContextUtils.getBean(MapperHelper.class);
         Set<Long> detectModelIds = mapperHelper.getModelIds(queryContext.getRequest());
 
+        terms = filterByModelIds(terms, detectModelIds);
+
         Map<MatchText, List<MapResult>> matchResult = matchStrategy.match(queryContext.getRequest(), terms,
                 detectModelIds);
 
@@ -57,6 +55,26 @@ public class HanlpDictMapper implements SchemaMapper {
         convertTermsToSchemaMapInfo(matches, queryContext.getMapInfo(), terms);
     }
 
+    private List<Term> filterByModelIds(List<Term> terms, Set<Long> detectModelIds) {
+        for (Term term : terms) {
+            log.info("before word:{},nature:{},frequency:{}", term.word, term.nature.toString(), term.getFrequency());
+        }
+        if (CollectionUtils.isNotEmpty(detectModelIds)) {
+            terms = terms.stream().filter(term -> {
+                Long modelId = NatureHelper.getModelId(term.getNature().toString());
+                if (Objects.nonNull(modelId)) {
+                    return detectModelIds.contains(modelId);
+                }
+                return false;
+            }).collect(Collectors.toList());
+        }
+        for (Term term : terms) {
+            log.info("after filter word:{},nature:{},frequency:{}", term.word, term.nature.toString(),
+                    term.getFrequency());
+        }
+        return terms;
+    }
+
 
     private void convertTermsToSchemaMapInfo(List<MapResult> mapResults, SchemaMapInfo schemaMap, List<Term> terms) {
         if (CollectionUtils.isEmpty(mapResults)) {

chat/core/src/main/java/com/tencent/supersonic/chat/parser/llm/dsl/HeuristicModelResolver.java

@@ -1,23 +1,23 @@
-package com.tencent.supersonic.chat.parser.plugin.function;
+package com.tencent.supersonic.chat.parser.llm.dsl;
 
-import com.tencent.supersonic.chat.api.pojo.QueryContext;
-import com.tencent.supersonic.chat.api.pojo.SchemaMapInfo;
+import com.tencent.supersonic.chat.api.component.SemanticQuery;
 import com.tencent.supersonic.chat.api.pojo.ChatContext;
-import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
+import com.tencent.supersonic.chat.api.pojo.QueryContext;
 import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
 import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
+import com.tencent.supersonic.chat.api.pojo.SchemaMapInfo;
+import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
 import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
-import com.tencent.supersonic.chat.api.component.SemanticQuery;
-import lombok.extern.slf4j.Slf4j;
-
-import java.util.Map;
+import java.util.Comparator;
 import java.util.HashMap;
-import java.util.Objects;
-import java.util.List;
 import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
 import java.util.Set;
 import java.util.stream.Collectors;
-
+import lombok.extern.slf4j.Slf4j;
 import org.apache.commons.collections.CollectionUtils;
 
 @Slf4j
@@ -25,6 +25,13 @@ public class HeuristicModelResolver implements ModelResolver {
 
     protected static Long selectModelBySchemaElementCount(Map<Long, SemanticQuery> modelQueryModes,
             SchemaMapInfo schemaMap) {
+        //model count priority
+        Long modelIdByModelCount = getModelIdByModelCount(schemaMap);
+        if (Objects.nonNull(modelIdByModelCount)) {
+            log.info("selectModel by model count:{}", modelIdByModelCount);
+            return modelIdByModelCount;
+        }
+
         Map<Long, ModelMatchResult> modelTypeMap = getModelTypeMap(schemaMap);
         if (modelTypeMap.size() == 1) {
             Long modelSelect = modelTypeMap.entrySet().stream().collect(Collectors.toList()).get(0).getKey();
@@ -33,6 +40,7 @@ public class HeuristicModelResolver implements ModelResolver {
                 return modelSelect;
             }
         } else {
+
             Map.Entry<Long, ModelMatchResult> maxModel = modelTypeMap.entrySet().stream()
                     .filter(entry -> modelQueryModes.containsKey(entry.getKey()))
                     .sorted((o1, o2) -> {
@@ -51,6 +59,31 @@ public class HeuristicModelResolver implements ModelResolver {
         return 0L;
     }
 
+    private static Long getModelIdByModelCount(SchemaMapInfo schemaMap) {
+        Map<Long, List<SchemaElementMatch>> modelElementMatches = schemaMap.getModelElementMatches();
+        Map<Long, Integer> modelIdToModelCount = new HashMap<>();
+        if (Objects.nonNull(modelElementMatches)) {
+            for (Entry<Long, List<SchemaElementMatch>> modelElementMatch : modelElementMatches.entrySet()) {
+                Long modelId = modelElementMatch.getKey();
+                List<SchemaElementMatch> modelMatches = modelElementMatch.getValue().stream().filter(
+                        elementMatch -> SchemaElementType.MODEL.equals(elementMatch.getElement().getType())
+                ).collect(Collectors.toList());
+
+                if (!CollectionUtils.isEmpty(modelMatches)) {
+                    Integer count = modelMatches.size();
+                    modelIdToModelCount.put(modelId, count);
+                }
+            }
+            Entry<Long, Integer> maxModelCount = modelIdToModelCount.entrySet().stream()
+                    .max(Comparator.comparingInt(o -> o.getValue())).orElse(null);
+            log.info("maxModelCount:{},modelIdToModelCount:{}", maxModelCount, modelIdToModelCount);
+            if (Objects.nonNull(maxModelCount)) {
+                return maxModelCount.getKey();
+            }
+        }
+        return null;
+    }
+
     /**
      * to check can switch Model if context exit Model
      *

chat/core/src/main/java/com/tencent/supersonic/chat/parser/llm/dsl/LLMDslParser.java

@@ -15,10 +15,8 @@ import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
 import com.tencent.supersonic.chat.api.pojo.SemanticSchema;
 import com.tencent.supersonic.chat.api.pojo.request.QueryFilter;
 import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
-import com.tencent.supersonic.chat.config.LLMConfig;
-import com.tencent.supersonic.chat.corrector.BaseSemanticCorrector;
+import com.tencent.supersonic.chat.config.LLMParserConfig;
 import com.tencent.supersonic.chat.parser.SatisfactionChecker;
-import com.tencent.supersonic.chat.parser.plugin.function.ModelResolver;
 import com.tencent.supersonic.chat.query.QueryManager;
 import com.tencent.supersonic.chat.query.llm.dsl.DslQuery;
 import com.tencent.supersonic.chat.query.llm.dsl.LLMReq;
@@ -31,14 +29,16 @@ import com.tencent.supersonic.common.pojo.Constants;
 import com.tencent.supersonic.common.pojo.DateConf;
 import com.tencent.supersonic.common.pojo.DateConf.DateMode;
 import com.tencent.supersonic.common.util.ContextUtils;
+import com.tencent.supersonic.common.util.DateUtils;
 import com.tencent.supersonic.common.util.JsonUtil;
 import com.tencent.supersonic.common.util.jsqlparser.FilterExpression;
+import com.tencent.supersonic.common.util.jsqlparser.SqlParserSelectFunctionHelper;
 import com.tencent.supersonic.common.util.jsqlparser.SqlParserSelectHelper;
 import com.tencent.supersonic.knowledge.service.SchemaService;
-import com.tencent.supersonic.semantic.api.model.enums.TimeDimensionEnum;
 import com.tencent.supersonic.semantic.api.query.enums.FilterOperatorEnum;
 import java.util.ArrayList;
 import java.util.Arrays;
+import java.util.Comparator;
 import java.util.HashMap;
 import java.util.HashSet;
 import java.util.List;
@@ -61,14 +61,12 @@ import org.springframework.web.client.RestTemplate;
 @Slf4j
 public class LLMDslParser implements SemanticParser {
 
-    public static final double function_bonus_threshold = 201;
-
     @Override
     public void parse(QueryContext queryCtx, ChatContext chatCtx) {
         QueryReq request = queryCtx.getRequest();
-        LLMConfig llmConfig = ContextUtils.getBean(LLMConfig.class);
-        if (StringUtils.isEmpty(llmConfig.getUrl())) {
-            log.info("llm url is empty, skip dsl parser, llmConfig:{}", llmConfig);
+        LLMParserConfig llmParserConfig = ContextUtils.getBean(LLMParserConfig.class);
+        if (StringUtils.isEmpty(llmParserConfig.getUrl())) {
+            log.info("llm parser url is empty, skip dsl parser, llmParserConfig:{}", llmParserConfig);
             return;
         }
         if (SatisfactionChecker.check(queryCtx)) {
@@ -87,8 +85,8 @@ public class LLMDslParser implements SemanticParser {
                 return;
             }
 
-            LLMReq llmReq = getLlmReq(queryCtx, modelId);
-            LLMResp llmResp = requestLLM(llmReq, modelId, llmConfig);
+            LLMReq llmReq = getLlmReq(queryCtx, modelId, llmParserConfig);
+            LLMResp llmResp = requestLLM(llmReq, modelId, llmParserConfig);
 
             if (Objects.isNull(llmResp)) {
                 return;
@@ -112,7 +110,7 @@ public class LLMDslParser implements SemanticParser {
     private Set<SchemaElement> getElements(Long modelId, List<String> allFields, List<SchemaElement> elements) {
         return elements.stream()
                 .filter(schemaElement -> modelId.equals(schemaElement.getModel())
-                        && allFields.contains(schemaElement.getBizName())
+                        && allFields.contains(schemaElement.getName())
                 ).collect(Collectors.toSet());
     }
 
@@ -121,16 +119,15 @@ public class LLMDslParser implements SemanticParser {
             return new ArrayList<>();
         }
         return allFields.stream()
-                .filter(entry -> !TimeDimensionEnum.getNameList().contains(entry))
+                .filter(entry -> !DateUtils.DATE_FIELD.equalsIgnoreCase(entry))
                 .collect(Collectors.toList());
     }
 
     public void updateParseInfo(SemanticCorrectInfo semanticCorrectInfo, Long modelId, SemanticParseInfo parseInfo) {
 
-        String correctorSql = semanticCorrectInfo.getPreSql();
-        if (StringUtils.isEmpty(correctorSql)) {
-            correctorSql = semanticCorrectInfo.getSql();
-        }
+        String correctorSql = semanticCorrectInfo.getSql();
+        parseInfo.getSqlInfo().setLogicSql(correctorSql);
+
         List<FilterExpression> expressions = SqlParserSelectHelper.getFilterExpression(correctorSql);
         //set dataInfo
         try {
@@ -144,8 +141,8 @@ public class LLMDslParser implements SemanticParser {
 
         //set filter
         try {
-            Map<String, SchemaElement> bizNameToElement = getBizNameToElement(modelId);
-            List<QueryFilter> result = getDimensionFilter(bizNameToElement, expressions);
+            Map<String, SchemaElement> fieldNameToElement = getNameToElement(modelId);
+            List<QueryFilter> result = getDimensionFilter(fieldNameToElement, expressions);
             parseInfo.getDimensionFilters().addAll(result);
         } catch (Exception e) {
             log.error("set dimensionFilter error :", e);
@@ -161,7 +158,7 @@ public class LLMDslParser implements SemanticParser {
         Set<SchemaElement> metrics = getElements(modelId, allFields, semanticSchema.getMetrics());
         parseInfo.setMetrics(metrics);
 
-        if (SqlParserSelectHelper.hasAggregateFunction(semanticCorrectInfo.getSql())) {
+        if (SqlParserSelectFunctionHelper.hasAggregateFunction(semanticCorrectInfo.getSql())) {
             parseInfo.setNativeQuery(false);
             List<String> groupByFields = SqlParserSelectHelper.getGroupByFields(semanticCorrectInfo.getSql());
             List<String> groupByDimensions = getFieldsExceptDate(groupByFields);
@@ -174,24 +171,23 @@ public class LLMDslParser implements SemanticParser {
         }
     }
 
-    private List<QueryFilter> getDimensionFilter(Map<String, SchemaElement> bizNameToElement,
+    private List<QueryFilter> getDimensionFilter(Map<String, SchemaElement> fieldNameToElement,
             List<FilterExpression> filterExpressions) {
         List<QueryFilter> result = Lists.newArrayList();
         for (FilterExpression expression : filterExpressions) {
             QueryFilter dimensionFilter = new QueryFilter();
             dimensionFilter.setValue(expression.getFieldValue());
-            String bizName = expression.getFieldName();
-            SchemaElement schemaElement = bizNameToElement.get(bizName);
+            SchemaElement schemaElement = fieldNameToElement.get(expression.getFieldName());
             if (Objects.isNull(schemaElement)) {
                 continue;
             }
-            String fieldName = schemaElement.getName();
-            dimensionFilter.setName(fieldName);
-            dimensionFilter.setBizName(bizName);
+            dimensionFilter.setName(schemaElement.getName());
+            dimensionFilter.setBizName(schemaElement.getBizName());
             dimensionFilter.setElementID(schemaElement.getId());
 
             FilterOperatorEnum operatorEnum = FilterOperatorEnum.getSqlOperator(expression.getOperator());
             dimensionFilter.setOperator(operatorEnum);
+            dimensionFilter.setFunction(expression.getFunction());
             result.add(dimensionFilter);
         }
         return result;
@@ -199,13 +195,8 @@ public class LLMDslParser implements SemanticParser {
 
     private DateConf getDateInfo(List<FilterExpression> filterExpressions) {
         List<FilterExpression> dateExpressions = filterExpressions.stream()
-                .filter(expression -> {
-                    List<String> nameList = TimeDimensionEnum.getNameList();
-                    if (StringUtils.isEmpty(expression.getFieldName())) {
-                        return false;
-                    }
-                    return nameList.contains(expression.getFieldName().toLowerCase());
-                }).collect(Collectors.toList());
+                .filter(expression -> DateUtils.DATE_FIELD.equalsIgnoreCase(expression.getFieldName()))
+                .collect(Collectors.toList());
         if (CollectionUtils.isEmpty(dateExpressions)) {
             return new DateConf();
         }
@@ -259,7 +250,7 @@ public class LLMDslParser implements SemanticParser {
                 dslCorrection.correct(correctInfo);
                 log.info("sqlCorrection:{} sql:{}", dslCorrection.getClass().getSimpleName(), correctInfo.getSql());
             } catch (Exception e) {
-                log.error("sqlCorrection:{} correct error,correctInfo:{}", dslCorrection, correctInfo, e);
+                log.error(String.format("correct error,correctInfo:%s", correctInfo), e);
             }
         });
         return correctInfo;
@@ -277,8 +268,9 @@ public class LLMDslParser implements SemanticParser {
         properties.put("name", dslTool.getName());
 
         parseInfo.setProperties(properties);
-        parseInfo.setScore(function_bonus_threshold);
+        parseInfo.setScore(queryCtx.getRequest().getQueryText().length());
         parseInfo.setQueryMode(semanticQuery.getQueryMode());
+        parseInfo.getSqlInfo().setLlmParseSql(dslParseResult.getLlmResp().getSqlOutput());
 
         SemanticSchema semanticSchema = ContextUtils.getBean(SchemaService.class).getSemanticSchema();
         Map<Long, String> modelIdToName = semanticSchema.getModelIdToName();
@@ -319,8 +311,8 @@ public class LLMDslParser implements SemanticParser {
         return modelId;
     }
 
-    private LLMResp requestLLM(LLMReq llmReq, Long modelId, LLMConfig llmConfig) {
-        String questUrl = llmConfig.getUrl() + llmConfig.getQueryToSqlPath();
+    private LLMResp requestLLM(LLMReq llmReq, Long modelId, LLMParserConfig llmParserConfig) {
+        String questUrl = llmParserConfig.getUrl() + llmParserConfig.getQueryToSqlPath();
         long startTime = System.currentTimeMillis();
         log.info("requestLLM request, modelId:{},llmReq:{}", modelId, llmReq);
         RestTemplate restTemplate = ContextUtils.getBean(RestTemplate.class);
@@ -340,22 +332,28 @@ public class LLMDslParser implements SemanticParser {
         return null;
     }
 
-    private LLMReq getLlmReq(QueryContext queryCtx, Long modelId) {
+    private LLMReq getLlmReq(QueryContext queryCtx, Long modelId, LLMParserConfig llmParserConfig) {
         SemanticSchema semanticSchema = ContextUtils.getBean(SchemaService.class).getSemanticSchema();
         Map<Long, String> modelIdToName = semanticSchema.getModelIdToName();
         String queryText = queryCtx.getRequest().getQueryText();
+
         LLMReq llmReq = new LLMReq();
         llmReq.setQueryText(queryText);
+
         LLMReq.LLMSchema llmSchema = new LLMReq.LLMSchema();
         llmSchema.setModelName(modelIdToName.get(modelId));
         llmSchema.setDomainName(modelIdToName.get(modelId));
-        List<String> fieldNameList = getFieldNameList(queryCtx, modelId, semanticSchema);
-        fieldNameList.add(BaseSemanticCorrector.DATE_FIELD);
+
+        List<String> fieldNameList = getFieldNameList(queryCtx, modelId, semanticSchema, llmParserConfig);
+
+        fieldNameList.add(DateUtils.DATE_FIELD);
         llmSchema.setFieldNameList(fieldNameList);
         llmReq.setSchema(llmSchema);
+
         List<ElementValue> linking = new ArrayList<>();
         linking.addAll(getValueList(queryCtx, modelId, semanticSchema));
         llmReq.setLinking(linking);
+
         String currentDate = DSLDateHelper.getReferenceDate(modelId);
         llmReq.setCurrentDate(currentDate);
         return llmReq;
@@ -385,7 +383,7 @@ public class LLMDslParser implements SemanticParser {
     }
 
 
-    protected Map<String, SchemaElement> getBizNameToElement(Long modelId) {
+    protected Map<String, SchemaElement> getNameToElement(Long modelId) {
         SemanticSchema semanticSchema = ContextUtils.getBean(SchemaService.class).getSemanticSchema();
         List<SchemaElement> dimensions = semanticSchema.getDimensions();
         List<SchemaElement> metrics = semanticSchema.getMetrics();
@@ -395,16 +393,26 @@ public class LLMDslParser implements SemanticParser {
         allElements.addAll(metrics);
         return allElements.stream()
                 .filter(schemaElement -> schemaElement.getModel().equals(modelId))
-                .collect(Collectors.toMap(SchemaElement::getBizName, Function.identity(), (value1, value2) -> value2));
+                .collect(Collectors.toMap(SchemaElement::getName, Function.identity(), (value1, value2) -> value2));
     }
 
 
-    protected List<String> getFieldNameList(QueryContext queryCtx, Long modelId, SemanticSchema semanticSchema) {
-        Map<Long, String> itemIdToName = getItemIdToName(modelId, semanticSchema);
+    protected List<String> getFieldNameList(QueryContext queryCtx, Long modelId, SemanticSchema semanticSchema,
+            LLMParserConfig llmParserConfig) {
 
+        Set<String> results = getTopNFieldNames(modelId, semanticSchema, llmParserConfig);
+
+        Set<String> fieldNameList = getMatchedFieldNames(queryCtx, modelId, semanticSchema);
+
+        results.addAll(fieldNameList);
+        return new ArrayList<>(results);
+    }
+
+    protected Set<String> getMatchedFieldNames(QueryContext queryCtx, Long modelId, SemanticSchema semanticSchema) {
+        Map<Long, String> itemIdToName = getItemIdToName(modelId, semanticSchema);
         List<SchemaElementMatch> matchedElements = queryCtx.getMapInfo().getMatchedElements(modelId);
         if (CollectionUtils.isEmpty(matchedElements)) {
-            return new ArrayList<>();
+            return new HashSet<>();
         }
         Set<String> fieldNameList = matchedElements.stream()
                 .filter(schemaElementMatch -> {
@@ -423,12 +431,29 @@ public class LLMDslParser implements SemanticParser {
                 })
                 .filter(name -> StringUtils.isNotEmpty(name) && !name.contains("%"))
                 .collect(Collectors.toSet());
-        return new ArrayList<>(fieldNameList);
+        return fieldNameList;
+    }
+
+    private Set<String> getTopNFieldNames(Long modelId, SemanticSchema semanticSchema,
+            LLMParserConfig llmParserConfig) {
+        Set<String> results = semanticSchema.getDimensions(modelId).stream()
+                .sorted(Comparator.comparing(SchemaElement::getUseCnt).reversed())
+                .limit(llmParserConfig.getDimensionTopN())
+                .map(entry -> entry.getName())
+                .collect(Collectors.toSet());
+
+        Set<String> metrics = semanticSchema.getMetrics(modelId).stream()
+                .sorted(Comparator.comparing(SchemaElement::getUseCnt).reversed())
+                .limit(llmParserConfig.getMetricTopN())
+                .map(entry -> entry.getName())
+                .collect(Collectors.toSet());
+
+        results.addAll(metrics);
+        return results;
     }
 
     protected Map<Long, String> getItemIdToName(Long modelId, SemanticSchema semanticSchema) {
-        return semanticSchema.getDimensions().stream()
-                .filter(entry -> modelId.equals(entry.getModel()))
+        return semanticSchema.getDimensions(modelId).stream()
                 .collect(Collectors.toMap(SchemaElement::getId, SchemaElement::getName, (value1, value2) -> value2));
     }
 

chat/core/src/main/java/com/tencent/supersonic/chat/parser/llm/dsl/ModelMatchResult.java

@@ -1,4 +1,4 @@
-package com.tencent.supersonic.chat.parser.plugin.function;
+package com.tencent.supersonic.chat.parser.llm.dsl;
 
 import lombok.Data;
 

chat/core/src/main/java/com/tencent/supersonic/chat/parser/llm/dsl/ModelResolver.java

@@ -1,4 +1,4 @@
-package com.tencent.supersonic.chat.parser.plugin.function;
+package com.tencent.supersonic.chat.parser.llm.dsl;
 
 
 import com.tencent.supersonic.chat.api.pojo.ChatContext;

chat/core/src/main/java/com/tencent/supersonic/chat/parser/llm/interpret/MetricInterpretParser.java

@@ -5,7 +5,7 @@ import com.google.common.collect.Sets;
 import com.tencent.supersonic.chat.agent.Agent;
 import com.tencent.supersonic.chat.agent.tool.AgentToolType;
 import com.tencent.supersonic.chat.agent.tool.MetricInterpretTool;
-import com.tencent.supersonic.chat.api.component.SemanticLayer;
+import com.tencent.supersonic.chat.api.component.SemanticInterpreter;
 import com.tencent.supersonic.chat.api.component.SemanticParser;
 import com.tencent.supersonic.chat.api.pojo.QueryContext;
 import com.tencent.supersonic.chat.api.pojo.ChatContext;
@@ -82,8 +82,8 @@ public class MetricInterpretParser implements SemanticParser {
     }
 
     public Set<SchemaElement> getMetrics(List<Long> metricIds, Long modelId) {
-        SemanticLayer semanticLayer = ComponentFactory.getSemanticLayer();
-        ModelSchema modelSchema = semanticLayer.getModelSchema(modelId, true);
+        SemanticInterpreter semanticInterpreter = ComponentFactory.getSemanticLayer();
+        ModelSchema modelSchema = semanticInterpreter.getModelSchema(modelId, true);
         Set<SchemaElement> metrics = modelSchema.getMetrics();
         return metrics.stream().filter(schemaElement -> metricIds.contains(schemaElement.getId()))
                 .collect(Collectors.toSet());

chat/core/src/main/java/com/tencent/supersonic/chat/parser/plugin/embedding/EmbeddingConfig.java

@@ -23,4 +23,13 @@ public class EmbeddingConfig {
     @Value("${embedding.nResult:1}")
     private String nResult;
 
+    @Value("${embedding.solvedQuery.recall.path:/solved_query_retrival}")
+    private String solvedQueryRecallPath;
+
+    @Value("${embedding.solvedQuery.add.path:/solved_query_add}")
+    private String solvedQueryAddPath;
+
+    @Value("${embedding.solved.query.nResult:5}")
+    private String solvedQueryResultNum;
+
 }

chat/core/src/main/java/com/tencent/supersonic/chat/parser/plugin/embedding/EmbeddingEntityResolver.java

@@ -1,98 +0,0 @@
-package com.tencent.supersonic.chat.parser.plugin.embedding;
-
-import com.alibaba.fastjson.JSONObject;
-import com.tencent.supersonic.chat.api.pojo.QueryContext;
-import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
-import com.tencent.supersonic.chat.api.pojo.SchemaMapInfo;
-import com.tencent.supersonic.chat.api.pojo.ChatContext;
-import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
-import com.tencent.supersonic.chat.api.pojo.request.QueryFilter;
-import com.tencent.supersonic.chat.api.pojo.request.QueryFilters;
-import com.tencent.supersonic.chat.service.ConfigService;
-import lombok.extern.slf4j.Slf4j;
-import org.apache.commons.collections.CollectionUtils;
-import org.apache.commons.lang3.StringUtils;
-import org.springframework.stereotype.Component;
-
-import java.util.List;
-import java.util.Objects;
-import java.util.Set;
-
-@Slf4j
-@Component("EmbeddingEntityResolver")
-public class EmbeddingEntityResolver {
-    private ConfigService configService;
-
-    public EmbeddingEntityResolver(ConfigService configService) {
-        this.configService = configService;
-    }
-
-
-    private Long getEntityValue(Long modelId, Long entityElementId, QueryContext queryCtx, ChatContext chatCtx) {
-        Long entityId = null;
-        QueryFilters queryFilters = queryCtx.getRequest().getQueryFilters();
-        if (queryFilters != null) {
-            entityId = getEntityValueFromQueryFilter(queryFilters.getFilters());
-            if (entityId != null) {
-                log.info("get entity id:{} model id:{} from  query filter :{} ", entityId, modelId, queryFilters);
-                return entityId;
-            }
-        }
-        entityId = getEntityValueFromSchemaMapInfo(modelId, queryCtx.getMapInfo(), entityElementId);
-        log.info("get entity id:{} from  schema map Info :{} ",
-                entityId, JSONObject.toJSONString(queryCtx.getMapInfo()));
-        if (entityId == null || entityId == 0) {
-            Long entityIdFromChat = getEntityValueFromParseInfo(chatCtx.getParseInfo(), entityElementId);
-            if (entityIdFromChat != null && entityIdFromChat > 0) {
-                entityId = entityIdFromChat;
-            }
-        }
-        return entityId;
-    }
-
-    private Long getEntityValueFromQueryFilter(List<QueryFilter> queryFilters) {
-        if (CollectionUtils.isEmpty(queryFilters)) {
-            return null;
-        }
-        QueryFilter filter = queryFilters.get(0);
-        String value = String.valueOf(filter.getValue());
-        if (StringUtils.isNumeric(value)) {
-            return Long.parseLong(value);
-        }
-        return null;
-    }
-
-    private Long getEntityValueFromParseInfo(SemanticParseInfo semanticParseInfo, Long entityElementId) {
-        Set<QueryFilter> filters = semanticParseInfo.getDimensionFilters();
-        if (CollectionUtils.isEmpty(filters)) {
-            return null;
-        }
-        for (QueryFilter filter : filters) {
-            if (entityElementId.equals(filter.getElementID())) {
-                String value = String.valueOf(filter.getValue());
-                if (StringUtils.isNumeric(value)) {
-                    return Long.parseLong(value);
-                }
-            }
-        }
-        return null;
-    }
-
-
-    private Long getEntityValueFromSchemaMapInfo(Long modelId, SchemaMapInfo schemaMapInfo, Long entityElementId) {
-        List<SchemaElementMatch> schemaElementMatchList = schemaMapInfo.getMatchedElements(modelId);
-        if (CollectionUtils.isEmpty(schemaElementMatchList)) {
-            return null;
-        }
-        for (SchemaElementMatch schemaElementMatch : schemaElementMatchList) {
-            if (Objects.equals(schemaElementMatch.getElement().getId(), entityElementId)) {
-                if (StringUtils.isNumeric(schemaElementMatch.getWord())) {
-                    return Long.parseLong(schemaElementMatch.getWord());
-                }
-            }
-
-        }
-        return null;
-    }
-
-}

chat/core/src/main/java/com/tencent/supersonic/chat/parser/plugin/embedding/RecallRetrieval.java

@@ -14,4 +14,6 @@ public class RecallRetrieval {
 
     private String presetId;
 
+    private String query;
+
 }

chat/core/src/main/java/com/tencent/supersonic/chat/parser/plugin/function/FunctionBasedParser.java

@@ -2,7 +2,6 @@ package com.tencent.supersonic.chat.parser.plugin.function;
 
 import com.alibaba.fastjson.JSON;
 import com.tencent.supersonic.chat.api.pojo.QueryContext;
-import com.tencent.supersonic.chat.config.FunctionCallInfoConfig;
 import com.tencent.supersonic.chat.parser.ParseMode;
 import com.tencent.supersonic.chat.parser.SatisfactionChecker;
 import com.tencent.supersonic.chat.parser.plugin.PluginParser;
@@ -37,7 +36,7 @@ public class FunctionBasedParser extends PluginParser {
 
     @Override
     public boolean checkPreCondition(QueryContext queryContext) {
-        FunctionCallInfoConfig functionCallConfig = ContextUtils.getBean(FunctionCallInfoConfig.class);
+        FunctionCallConfig functionCallConfig = ContextUtils.getBean(FunctionCallConfig.class);
         String functionUrl = functionCallConfig.getUrl();
         if (StringUtils.isBlank(functionUrl) || SatisfactionChecker.check(queryContext)) {
             log.info("functionUrl:{}, skip function parser, queryText:{}", functionUrl,
@@ -134,7 +133,7 @@ public class FunctionBasedParser extends PluginParser {
     }
 
     public FunctionResp requestFunction(FunctionReq functionReq) {
-        FunctionCallInfoConfig functionCallInfoConfig = ContextUtils.getBean(FunctionCallInfoConfig.class);
+        FunctionCallConfig functionCallInfoConfig = ContextUtils.getBean(FunctionCallConfig.class);
         String url = functionCallInfoConfig.getUrl() + functionCallInfoConfig.getPluginSelectPath();
         HttpHeaders headers = new HttpHeaders();
         long startTime = System.currentTimeMillis();

chat/core/src/main/java/com/tencent/supersonic/chat/parser/plugin/function/FunctionCallConfig.java

@@ -1,4 +1,4 @@
-package com.tencent.supersonic.chat.config;
+package com.tencent.supersonic.chat.parser.plugin.function;
 
 import lombok.Data;
 import org.springframework.beans.factory.annotation.Value;
@@ -6,7 +6,7 @@ import org.springframework.context.annotation.Configuration;
 
 @Configuration
 @Data
-public class FunctionCallInfoConfig {
+public class FunctionCallConfig {
     @Value("${functionCall.url:}")
     private String url;
 

chat/core/src/main/java/com/tencent/supersonic/chat/persistence/repository/ChatQueryRepository.java

@@ -15,7 +15,7 @@ import java.util.List;
 
 public interface ChatQueryRepository {
 
-    PageInfo<QueryResp> getChatQuery(PageQueryInfoReq pageQueryInfoCommend, long chatId);
+    PageInfo<QueryResp> getChatQuery(PageQueryInfoReq pageQueryInfoCommend, Long chatId);
 
     List<QueryResp> queryShowCase(PageQueryInfoReq pageQueryInfoCommend, int agentId);
 

chat/core/src/main/java/com/tencent/supersonic/chat/persistence/repository/impl/ChatQueryRepositoryImpl.java

@@ -52,15 +52,21 @@ public class ChatQueryRepositoryImpl implements ChatQueryRepository {
     }
 
     @Override
-    public PageInfo<QueryResp> getChatQuery(PageQueryInfoReq pageQueryInfoCommend, long chatId) {
+    public PageInfo<QueryResp> getChatQuery(PageQueryInfoReq pageQueryInfoReq, Long chatId) {
         ChatQueryDOExample example = new ChatQueryDOExample();
         example.setOrderByClause("question_id desc");
         Criteria criteria = example.createCriteria();
-        criteria.andChatIdEqualTo(chatId);
-        criteria.andUserNameEqualTo(pageQueryInfoCommend.getUserName());
-
-        PageInfo<ChatQueryDO> pageInfo = PageHelper.startPage(pageQueryInfoCommend.getCurrent(),
-                        pageQueryInfoCommend.getPageSize())
+        if (chatId != null) {
+            criteria.andChatIdEqualTo(chatId);
+        }
+        if (StringUtils.isNotBlank(pageQueryInfoReq.getUserName())) {
+            criteria.andUserNameEqualTo(pageQueryInfoReq.getUserName());
+        }
+        if (!CollectionUtils.isEmpty(pageQueryInfoReq.getIds())) {
+            criteria.andQuestionIdIn(pageQueryInfoReq.getIds());
+        }
+        PageInfo<ChatQueryDO> pageInfo = PageHelper.startPage(pageQueryInfoReq.getCurrent(),
+                        pageQueryInfoReq.getPageSize())
                 .doSelectPageInfo(() -> chatQueryDOMapper.selectByExampleWithBLOBs(example));
 
         PageInfo<QueryResp> chatQueryVOPageInfo = PageUtils.pageInfo2PageInfoVo(pageInfo);
@@ -118,9 +124,7 @@ public class ChatQueryRepositoryImpl implements ChatQueryRepository {
         } catch (Exception e) {
             log.info("database insert has an exception:{}", e.toString());
         }
-
-        ChatQueryDO lastChatQuery = getLastChatQuery(chatCtx.getChatId());
-        Long queryId = lastChatQuery.getQuestionId();
+        Long queryId = chatQueryDO.getQuestionId();
         parseResult.setQueryId(queryId);
         return queryId;
     }

chat/core/src/main/java/com/tencent/supersonic/chat/query/QueryManager.java

@@ -77,6 +77,10 @@ public class QueryManager {
         return ruleQueryMap.get(queryMode) instanceof EntitySemanticQuery;
     }
 
+    public static boolean isPluginQuery(String queryMode) {
+        return queryMode != null && pluginQueryMap.containsKey(queryMode);
+    }
+
     public static RuleSemanticQuery getRuleQuery(String queryMode) {
         if (queryMode == null) {
             return null;
@@ -92,4 +96,4 @@ public class QueryManager {
         return new ArrayList<>(pluginQueryMap.keySet());
     }
 
-}
+}

chat/core/src/main/java/com/tencent/supersonic/chat/query/llm/dsl/DslQuery.java

@@ -1,21 +1,21 @@
 package com.tencent.supersonic.chat.query.llm.dsl;
 
 import com.tencent.supersonic.auth.api.authentication.pojo.User;
-import com.tencent.supersonic.chat.api.component.SemanticLayer;
-import com.tencent.supersonic.chat.api.pojo.response.EntityInfo;
+import com.tencent.supersonic.chat.api.component.SemanticInterpreter;
 import com.tencent.supersonic.chat.api.pojo.response.QueryResult;
 import com.tencent.supersonic.chat.api.pojo.response.QueryState;
 import com.tencent.supersonic.chat.parser.llm.dsl.DSLParseResult;
 import com.tencent.supersonic.chat.query.QueryManager;
 import com.tencent.supersonic.chat.query.plugin.PluginSemanticQuery;
-import com.tencent.supersonic.chat.service.SemanticService;
 import com.tencent.supersonic.chat.utils.ComponentFactory;
 import com.tencent.supersonic.chat.utils.QueryReqBuilder;
 import com.tencent.supersonic.common.pojo.Constants;
 import com.tencent.supersonic.common.pojo.QueryColumn;
-import com.tencent.supersonic.common.util.ContextUtils;
 import com.tencent.supersonic.common.util.JsonUtil;
+import com.tencent.supersonic.semantic.api.model.enums.QueryTypeEnum;
+import com.tencent.supersonic.semantic.api.model.response.ExplainResp;
 import com.tencent.supersonic.semantic.api.model.response.QueryResultWithSchemaResp;
+import com.tencent.supersonic.semantic.api.query.request.ExplainSqlReq;
 import com.tencent.supersonic.semantic.api.query.request.QueryDslReq;
 import java.util.ArrayList;
 import java.util.List;
@@ -29,7 +29,7 @@ import org.springframework.stereotype.Component;
 public class DslQuery extends PluginSemanticQuery {
 
     public static final String QUERY_MODE = "DSL";
-    protected SemanticLayer semanticLayer = ComponentFactory.getSemanticLayer();
+    protected SemanticInterpreter semanticInterpreter = ComponentFactory.getSemanticLayer();
 
     public DslQuery() {
         QueryManager.register(this);
@@ -42,13 +42,11 @@ public class DslQuery extends PluginSemanticQuery {
 
     @Override
     public QueryResult execute(User user) {
-        String json = JsonUtil.toString(parseInfo.getProperties().get(Constants.CONTEXT));
-        DSLParseResult dslParseResult = JsonUtil.toObject(json, DSLParseResult.class);
-        LLMResp llmResp = dslParseResult.getLlmResp();
+        LLMResp llmResp = getLlmResp();
 
         long startTime = System.currentTimeMillis();
-        QueryDslReq queryDslReq = QueryReqBuilder.buildDslReq(llmResp.getCorrectorSql(), parseInfo.getModelId());
-        QueryResultWithSchemaResp queryResp = semanticLayer.queryByDsl(queryDslReq, user);
+        QueryDslReq queryDslReq = getQueryDslReq(llmResp);
+        QueryResultWithSchemaResp queryResp = semanticInterpreter.queryByDsl(queryDslReq, user);
 
         log.info("queryByDsl cost:{},querySql:{}", System.currentTimeMillis() - startTime, llmResp.getSqlOutput());
 
@@ -65,10 +63,33 @@ public class DslQuery extends PluginSemanticQuery {
         queryResult.setQueryMode(QUERY_MODE);
         queryResult.setQueryState(QueryState.SUCCESS);
 
-        // add model info
-        EntityInfo entityInfo = ContextUtils.getBean(SemanticService.class).getEntityInfo(parseInfo, user);
-        queryResult.setEntityInfo(entityInfo);
         parseInfo.setProperties(null);
         return queryResult;
     }
+
+    private LLMResp getLlmResp() {
+        String json = JsonUtil.toString(parseInfo.getProperties().get(Constants.CONTEXT));
+        DSLParseResult dslParseResult = JsonUtil.toObject(json, DSLParseResult.class);
+        return dslParseResult.getLlmResp();
+    }
+
+    private QueryDslReq getQueryDslReq(LLMResp llmResp) {
+        QueryDslReq queryDslReq = QueryReqBuilder.buildDslReq(llmResp.getCorrectorSql(), parseInfo.getModelId());
+        return queryDslReq;
+    }
+
+    @Override
+    public ExplainResp explain(User user) {
+        ExplainSqlReq explainSqlReq = null;
+        try {
+            explainSqlReq = ExplainSqlReq.builder()
+                    .queryTypeEnum(QueryTypeEnum.SQL)
+                    .queryReq(getQueryDslReq(getLlmResp()))
+                    .build();
+            return semanticInterpreter.explain(explainSqlReq, user);
+        } catch (Exception e) {
+            log.error("explain error explainSqlReq:{}", explainSqlReq, e);
+        }
+        return null;
+    }
 }

chat/core/src/main/java/com/tencent/supersonic/chat/query/llm/interpret/MetricInterpretQuery.java

@@ -3,7 +3,7 @@ package com.tencent.supersonic.chat.query.llm.interpret;
 import com.alibaba.fastjson.JSONObject;
 import com.google.common.collect.Lists;
 import com.tencent.supersonic.auth.api.authentication.pojo.User;
-import com.tencent.supersonic.chat.api.component.SemanticLayer;
+import com.tencent.supersonic.chat.api.component.SemanticInterpreter;
 import com.tencent.supersonic.chat.api.pojo.SchemaElement;
 import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
 import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
@@ -54,8 +54,8 @@ public class MetricInterpretQuery extends PluginSemanticQuery {
         QueryStructReq queryStructReq = QueryReqBuilder.buildStructReq(parseInfo);
         fillAggregator(queryStructReq, parseInfo.getMetrics());
         queryStructReq.setNativeQuery(true);
-        SemanticLayer semanticLayer = ComponentFactory.getSemanticLayer();
-        QueryResultWithSchemaResp queryResultWithSchemaResp = semanticLayer.queryByStruct(queryStructReq, user);
+        SemanticInterpreter semanticInterpreter = ComponentFactory.getSemanticLayer();
+        QueryResultWithSchemaResp queryResultWithSchemaResp = semanticInterpreter.queryByStruct(queryStructReq, user);
         String text = generateTableText(queryResultWithSchemaResp);
         Map<String, Object> properties = parseInfo.getProperties();
         Map<String, String> replacedMap = new HashMap<>();

chat/core/src/main/java/com/tencent/supersonic/chat/query/plugin/PluginSemanticQuery.java

@@ -1,7 +1,9 @@
 package com.tencent.supersonic.chat.query.plugin;
 
+import com.tencent.supersonic.auth.api.authentication.pojo.User;
 import com.tencent.supersonic.chat.api.component.SemanticQuery;
 import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
+import com.tencent.supersonic.semantic.api.model.response.ExplainResp;
 import lombok.extern.slf4j.Slf4j;
 
 @Slf4j
@@ -17,5 +19,8 @@ public abstract class PluginSemanticQuery implements SemanticQuery {
         return parseInfo;
     }
 
-
+    @Override
+    public ExplainResp explain(User user) {
+        return null;
+    }
 }

chat/core/src/main/java/com/tencent/supersonic/chat/query/plugin/webpage/WebPageQuery.java

@@ -8,7 +8,6 @@ import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
 import com.tencent.supersonic.chat.api.pojo.request.QueryFilter;
 import com.tencent.supersonic.chat.api.pojo.request.QueryFilters;
 import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
-import com.tencent.supersonic.chat.api.pojo.response.EntityInfo;
 import com.tencent.supersonic.chat.api.pojo.response.QueryResult;
 import com.tencent.supersonic.chat.api.pojo.response.QueryState;
 import com.tencent.supersonic.chat.plugin.Plugin;
@@ -57,8 +56,6 @@ public class WebPageQuery extends PluginSemanticQuery {
         SemanticService semanticService = ContextUtils.getBean(SemanticService.class);
         ModelSchema modelSchema = semanticService.getModelSchema(parseInfo.getModelId());
         parseInfo.setModel(modelSchema.getModel());
-        EntityInfo entityInfo = semanticService.getEntityInfo(parseInfo, user);
-        queryResult.setEntityInfo(entityInfo);
         queryResult.setQueryState(QueryState.SUCCESS);
         return queryResult;
     }

chat/core/src/main/java/com/tencent/supersonic/chat/query/rule/RuleSemanticQuery.java

@@ -2,7 +2,7 @@
 package com.tencent.supersonic.chat.query.rule;
 
 import com.tencent.supersonic.auth.api.authentication.pojo.User;
-import com.tencent.supersonic.chat.api.component.SemanticLayer;
+import com.tencent.supersonic.chat.api.component.SemanticInterpreter;
 import com.tencent.supersonic.chat.api.component.SemanticQuery;
 import com.tencent.supersonic.chat.api.pojo.ChatContext;
 import com.tencent.supersonic.chat.api.pojo.ModelSchema;
@@ -12,7 +12,6 @@ import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
 import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
 import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
 import com.tencent.supersonic.chat.api.pojo.request.QueryFilter;
-import com.tencent.supersonic.chat.api.pojo.response.EntityInfo;
 import com.tencent.supersonic.chat.api.pojo.response.QueryResult;
 import com.tencent.supersonic.chat.api.pojo.response.QueryState;
 import com.tencent.supersonic.chat.query.QueryManager;
@@ -21,8 +20,11 @@ import com.tencent.supersonic.chat.utils.ComponentFactory;
 import com.tencent.supersonic.chat.utils.QueryReqBuilder;
 import com.tencent.supersonic.common.pojo.QueryColumn;
 import com.tencent.supersonic.common.util.ContextUtils;
+import com.tencent.supersonic.semantic.api.model.enums.QueryTypeEnum;
+import com.tencent.supersonic.semantic.api.model.response.ExplainResp;
 import com.tencent.supersonic.semantic.api.model.response.QueryResultWithSchemaResp;
 import com.tencent.supersonic.semantic.api.query.enums.FilterOperatorEnum;
+import com.tencent.supersonic.semantic.api.query.request.ExplainSqlReq;
 import com.tencent.supersonic.semantic.api.query.request.QueryMultiStructReq;
 import com.tencent.supersonic.semantic.api.query.request.QueryStructReq;
 import java.io.Serializable;
@@ -42,7 +44,7 @@ public abstract class RuleSemanticQuery implements SemanticQuery, Serializable {
 
     protected SemanticParseInfo parseInfo = new SemanticParseInfo();
     protected QueryMatcher queryMatcher = new QueryMatcher();
-    protected SemanticLayer semanticLayer = ComponentFactory.getSemanticLayer();
+    protected SemanticInterpreter semanticInterpreter = ComponentFactory.getSemanticLayer();
 
     public RuleSemanticQuery() {
         QueryManager.register(this);
@@ -193,7 +195,7 @@ public abstract class RuleSemanticQuery implements SemanticQuery, Serializable {
         }
 
         QueryResult queryResult = new QueryResult();
-        QueryResultWithSchemaResp queryResp = semanticLayer.queryByStruct(convertQueryStruct(), user);
+        QueryResultWithSchemaResp queryResp = semanticInterpreter.queryByStruct(convertQueryStruct(), user);
 
         if (queryResp != null) {
             queryResult.setQueryAuthorization(queryResp.getQueryAuthorization());
@@ -208,13 +210,30 @@ public abstract class RuleSemanticQuery implements SemanticQuery, Serializable {
         queryResult.setQueryMode(queryMode);
         queryResult.setQueryState(QueryState.SUCCESS);
 
-        // add Model info
-        EntityInfo entityInfo = ContextUtils.getBean(SemanticService.class)
-                .getEntityInfo(parseInfo, user);
-        queryResult.setEntityInfo(entityInfo);
         return queryResult;
     }
 
+
+    @Override
+    public ExplainResp explain(User user) {
+        ExplainSqlReq explainSqlReq = null;
+        try {
+            explainSqlReq = ExplainSqlReq.builder()
+                    .queryTypeEnum(QueryTypeEnum.STRUCT)
+                    .queryReq(isMultiStructQuery()
+                            ? convertQueryMultiStruct() : convertQueryStruct())
+                    .build();
+            return semanticInterpreter.explain(explainSqlReq, user);
+        } catch (Exception e) {
+            log.error("explain error explainSqlReq:{}", explainSqlReq, e);
+        }
+        return null;
+    }
+
+    protected boolean isMultiStructQuery() {
+        return false;
+    }
+
     public QueryResult multiStructExecute(User user) {
         String queryMode = parseInfo.getQueryMode();
 
@@ -227,7 +246,7 @@ public abstract class RuleSemanticQuery implements SemanticQuery, Serializable {
 
         QueryResult queryResult = new QueryResult();
         QueryMultiStructReq queryMultiStructReq = convertQueryMultiStruct();
-        QueryResultWithSchemaResp queryResp = semanticLayer.queryByMultiStruct(queryMultiStructReq, user);
+        QueryResultWithSchemaResp queryResp = semanticInterpreter.queryByMultiStruct(queryMultiStructReq, user);
         if (queryResp != null) {
             queryResult.setQueryAuthorization(queryResp.getQueryAuthorization());
         }
@@ -241,10 +260,6 @@ public abstract class RuleSemanticQuery implements SemanticQuery, Serializable {
         queryResult.setQueryMode(queryMode);
         queryResult.setQueryState(QueryState.SUCCESS);
 
-        // add Model info
-        EntityInfo entityInfo = ContextUtils.getBean(SemanticService.class)
-                .getEntityInfo(parseInfo, user);
-        queryResult.setEntityInfo(entityInfo);
         return queryResult;
     }
 

chat/core/src/main/java/com/tencent/supersonic/chat/query/rule/entity/EntityListQuery.java

@@ -1,9 +1,9 @@
 package com.tencent.supersonic.chat.query.rule.entity;
 
-import com.tencent.supersonic.chat.api.pojo.SchemaElement;
-import com.tencent.supersonic.chat.api.pojo.QueryContext;
 import com.tencent.supersonic.chat.api.pojo.ChatContext;
 import com.tencent.supersonic.chat.api.pojo.ModelSchema;
+import com.tencent.supersonic.chat.api.pojo.QueryContext;
+import com.tencent.supersonic.chat.api.pojo.SchemaElement;
 import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
 import com.tencent.supersonic.chat.api.pojo.response.ChatConfigRichResp;
 import com.tencent.supersonic.chat.api.pojo.response.ChatDefaultRichConfigResp;
@@ -12,10 +12,10 @@ import com.tencent.supersonic.chat.service.SemanticService;
 import com.tencent.supersonic.common.pojo.Constants;
 import com.tencent.supersonic.common.pojo.Order;
 import com.tencent.supersonic.common.util.ContextUtils;
-
 import java.util.LinkedHashSet;
 import java.util.Objects;
 import java.util.Set;
+import org.apache.commons.collections.CollectionUtils;
 
 public abstract class EntityListQuery extends EntitySemanticQuery {
 
@@ -41,13 +41,17 @@ public abstract class EntityListQuery extends EntitySemanticQuery {
                 ChatDefaultRichConfigResp chatDefaultConfig = chaConfigRichDesc
                         .getChatDetailRichConfig().getChatDefaultConfig();
                 if (chatDefaultConfig != null) {
-                    chatDefaultConfig.getMetrics().stream()
-                            .forEach(metric -> {
-                                metrics.add(metric);
-                                orders.add(new Order(metric.getBizName(), Constants.DESC_UPPER));
-                            });
-                    chatDefaultConfig.getDimensions().stream()
-                            .forEach(dimension -> dimensions.add(dimension));
+                    if (CollectionUtils.isNotEmpty(chatDefaultConfig.getMetrics())) {
+                        chatDefaultConfig.getMetrics().stream()
+                                .forEach(metric -> {
+                                    metrics.add(metric);
+                                    orders.add(new Order(metric.getBizName(), Constants.DESC_UPPER));
+                                });
+                    }
+                    if (CollectionUtils.isNotEmpty(chatDefaultConfig.getDimensions())) {
+                        chatDefaultConfig.getDimensions().stream()
+                                .forEach(dimension -> dimensions.add(dimension));
+                    }
 
                 }
 

chat/core/src/main/java/com/tencent/supersonic/chat/query/rule/metric/MetricEntityQuery.java

@@ -2,6 +2,7 @@ package com.tencent.supersonic.chat.query.rule.metric;
 
 import com.tencent.supersonic.auth.api.authentication.pojo.User;
 import com.tencent.supersonic.chat.api.pojo.response.QueryResult;
+import com.tencent.supersonic.common.pojo.enums.FilterType;
 import com.tencent.supersonic.semantic.api.query.enums.FilterOperatorEnum;
 import com.tencent.supersonic.semantic.api.query.pojo.Filter;
 import com.tencent.supersonic.semantic.api.query.request.QueryMultiStructReq;
@@ -52,7 +53,7 @@ public class MetricEntityQuery extends MetricSemanticQuery {
         parseInfo.getDimensionFilters().stream()
                 .filter(filter -> filter.getElementID() != null)
                 .forEach(filter -> filterBizName.add(filter.getBizName()));
-        return filterBizName.size() > 1;
+        return FilterType.UNION.equals(parseInfo.getFilterType()) && filterBizName.size() > 1;
     }
 
     @Override

chat/core/src/main/java/com/tencent/supersonic/chat/query/rule/metric/MetricFilterQuery.java

@@ -6,6 +6,7 @@ import static com.tencent.supersonic.chat.query.rule.QueryMatchOption.RequireNum
 
 import com.tencent.supersonic.auth.api.authentication.pojo.User;
 import com.tencent.supersonic.chat.api.pojo.response.QueryResult;
+import com.tencent.supersonic.common.pojo.enums.FilterType;
 import com.tencent.supersonic.semantic.api.query.enums.FilterOperatorEnum;
 import com.tencent.supersonic.semantic.api.query.pojo.Filter;
 import com.tencent.supersonic.semantic.api.query.request.QueryMultiStructReq;
@@ -48,7 +49,7 @@ public class MetricFilterQuery extends MetricSemanticQuery {
         Set<String> filterBizName = new HashSet<>();
         parseInfo.getDimensionFilters().forEach(filter ->
                 filterBizName.add(filter.getBizName()));
-        return filterBizName.size() > 1;
+        return FilterType.UNION.equals(parseInfo.getFilterType()) && filterBizName.size() > 1;
     }
 
     @Override

chat/core/src/main/java/com/tencent/supersonic/chat/responder/execute/EntityInfoExecuteResponder.java

@@ -0,0 +1,59 @@
+package com.tencent.supersonic.chat.responder.execute;
+
+import com.tencent.supersonic.auth.api.authentication.pojo.User;
+import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
+import com.tencent.supersonic.chat.api.pojo.request.ExecuteQueryReq;
+import com.tencent.supersonic.chat.api.pojo.response.EntityInfo;
+import com.tencent.supersonic.chat.api.pojo.response.QueryResult;
+import com.tencent.supersonic.chat.query.QueryManager;
+import com.tencent.supersonic.chat.query.llm.dsl.DslQuery;
+import com.tencent.supersonic.chat.service.SemanticService;
+import com.tencent.supersonic.common.util.ContextUtils;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.lang3.StringUtils;
+
+public class EntityInfoExecuteResponder implements ExecuteResponder {
+
+    @Override
+    public void fillResponse(QueryResult queryResult, SemanticParseInfo semanticParseInfo, ExecuteQueryReq queryReq) {
+        if (semanticParseInfo == null || semanticParseInfo.getModelId() <= 0L) {
+            return;
+        }
+        String queryMode = semanticParseInfo.getQueryMode();
+        if (QueryManager.isPluginQuery(queryMode) && !DslQuery.QUERY_MODE.equals(queryMode)) {
+            return;
+        }
+        SemanticService semanticService = ContextUtils.getBean(SemanticService.class);
+        User user = queryReq.getUser();
+        EntityInfo entityInfo = semanticService.getEntityInfo(semanticParseInfo, user);
+        queryResult.setEntityInfo(entityInfo);
+
+        String primaryEntityBizName = semanticService.getPrimaryEntityBizName(entityInfo);
+        if (StringUtils.isEmpty(primaryEntityBizName)
+                || CollectionUtils.isEmpty(queryResult.getQueryColumns())) {
+            return;
+        }
+        boolean existPrimaryEntityName = queryResult.getQueryColumns().stream()
+                .anyMatch(queryColumn -> primaryEntityBizName.equals(queryColumn.getNameEn()));
+
+        semanticParseInfo.setNativeQuery(existPrimaryEntityName);
+
+        if (!existPrimaryEntityName) {
+            return;
+        }
+        List<Map<String, Object>> queryResults = queryResult.getQueryResults();
+        List<String> entities = queryResults.stream()
+                .map(entry -> entry.get(primaryEntityBizName))
+                .filter(Objects::nonNull)
+                .map(String::valueOf)
+                .collect(Collectors.toList());
+        if (CollectionUtils.isEmpty(entities)) {
+            return;
+        }
+    }
+
+}

chat/core/src/main/java/com/tencent/supersonic/chat/responder/execute/ExecuteResponder.java

@@ -0,0 +1,11 @@
+package com.tencent.supersonic.chat.responder.execute;
+
+import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
+import com.tencent.supersonic.chat.api.pojo.request.ExecuteQueryReq;
+import com.tencent.supersonic.chat.api.pojo.response.QueryResult;
+
+public interface ExecuteResponder {
+
+    void fillResponse(QueryResult queryResult, SemanticParseInfo semanticParseInfo, ExecuteQueryReq queryReq);
+
+}

chat/core/src/main/java/com/tencent/supersonic/chat/responder/parse/EntityInfoParseResponder.java

@@ -0,0 +1,47 @@
+package com.tencent.supersonic.chat.responder.parse;
+
+import com.tencent.supersonic.chat.api.pojo.QueryContext;
+import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
+import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
+import com.tencent.supersonic.chat.api.pojo.response.EntityInfo;
+import com.tencent.supersonic.chat.api.pojo.response.ParseResp;
+import com.tencent.supersonic.chat.query.QueryManager;
+import com.tencent.supersonic.chat.query.llm.dsl.DslQuery;
+import com.tencent.supersonic.chat.service.SemanticService;
+import com.tencent.supersonic.common.util.ContextUtils;
+import java.util.List;
+import org.apache.commons.lang3.StringUtils;
+import org.springframework.util.CollectionUtils;
+
+public class EntityInfoParseResponder implements ParseResponder {
+
+    @Override
+    public void fillResponse(ParseResp parseResp, QueryContext queryContext) {
+        List<SemanticParseInfo> selectedParses = parseResp.getSelectedParses();
+        if (CollectionUtils.isEmpty(selectedParses)) {
+            return;
+        }
+        QueryReq queryReq = queryContext.getRequest();
+        selectedParses.forEach(parseInfo -> {
+            if (QueryManager.isPluginQuery(parseInfo.getQueryMode())
+                    && !DslQuery.QUERY_MODE.equals(parseInfo.getQueryMode())) {
+                return;
+            }
+            //1. set entity info
+            SemanticService semanticService = ContextUtils.getBean(SemanticService.class);
+            EntityInfo entityInfo = semanticService.getEntityInfo(parseInfo, queryReq.getUser());
+            if (QueryManager.isEntityQuery(parseInfo.getQueryMode())
+                    || QueryManager.isMetricQuery(parseInfo.getQueryMode())) {
+                parseInfo.setEntityInfo(entityInfo);
+            }
+            //2. set native value
+            String primaryEntityBizName = semanticService.getPrimaryEntityBizName(entityInfo);
+            if (StringUtils.isNotEmpty(primaryEntityBizName)) {
+                //if exist primaryEntityBizName in parseInfo's dimensions, set nativeQuery to true
+                boolean existPrimaryEntityBizName = parseInfo.getDimensions().stream()
+                        .anyMatch(schemaElement -> primaryEntityBizName.equalsIgnoreCase(schemaElement.getBizName()));
+                parseInfo.setNativeQuery(existPrimaryEntityBizName);
+            }
+        });
+    }
+}

chat/core/src/main/java/com/tencent/supersonic/chat/responder/parse/ExplainSqlParseResponder.java

@@ -0,0 +1,45 @@
+package com.tencent.supersonic.chat.responder.parse;
+
+import com.tencent.supersonic.chat.api.component.SemanticQuery;
+import com.tencent.supersonic.chat.api.pojo.QueryContext;
+import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
+import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
+import com.tencent.supersonic.chat.api.pojo.response.ParseResp;
+import com.tencent.supersonic.chat.query.QueryManager;
+import com.tencent.supersonic.semantic.api.model.response.ExplainResp;
+import org.springframework.util.CollectionUtils;
+import java.util.List;
+import java.util.Objects;
+
+public class ExplainSqlParseResponder implements ParseResponder {
+
+    @Override
+    public void fillResponse(ParseResp parseResp, QueryContext queryContext) {
+        QueryReq queryReq = queryContext.getRequest();
+        addExplainSql(queryReq, parseResp.getSelectedParses());
+        addExplainSql(queryReq, parseResp.getCandidateParses());
+    }
+
+    private void addExplainSql(QueryReq queryReq, List<SemanticParseInfo> semanticParseInfos) {
+        if (CollectionUtils.isEmpty(semanticParseInfos)) {
+            return;
+        }
+        semanticParseInfos.forEach(parseInfo -> {
+            addExplainSql(queryReq, parseInfo);
+        });
+    }
+
+    private void addExplainSql(QueryReq queryReq, SemanticParseInfo parseInfo) {
+        SemanticQuery semanticQuery = QueryManager.createQuery(parseInfo.getQueryMode());
+        if (Objects.isNull(semanticQuery)) {
+            return;
+        }
+        semanticQuery.setParseInfo(parseInfo);
+        ExplainResp explain = semanticQuery.explain(queryReq.getUser());
+        if (Objects.isNull(explain)) {
+            return;
+        }
+        parseInfo.getSqlInfo().setQuerySql(explain.getSql());
+    }
+
+}

chat/core/src/main/java/com/tencent/supersonic/chat/responder/parse/ParseResponder.java

@@ -0,0 +1,10 @@
+package com.tencent.supersonic.chat.responder.parse;
+
+import com.tencent.supersonic.chat.api.pojo.QueryContext;
+import com.tencent.supersonic.chat.api.pojo.response.ParseResp;
+
+public interface ParseResponder {
+
+    void fillResponse(ParseResp parseResp, QueryContext queryContext);
+
+}

chat/core/src/main/java/com/tencent/supersonic/chat/rest/ChatConfigController.java

@@ -3,7 +3,7 @@ package com.tencent.supersonic.chat.rest;
 import com.github.pagehelper.PageInfo;
 import com.tencent.supersonic.auth.api.authentication.pojo.User;
 import com.tencent.supersonic.auth.api.authentication.utils.UserHolder;
-import com.tencent.supersonic.chat.api.component.SemanticLayer;
+import com.tencent.supersonic.chat.api.component.SemanticInterpreter;
 import com.tencent.supersonic.chat.api.pojo.request.ChatConfigBaseReq;
 import com.tencent.supersonic.chat.api.pojo.request.ChatConfigEditReqReq;
 import com.tencent.supersonic.chat.api.pojo.request.ChatConfigFilter;
@@ -41,7 +41,7 @@ public class ChatConfigController {
     private ConfigService configService;
 
 
-    private SemanticLayer semanticLayer = ComponentFactory.getSemanticLayer();
+    private SemanticInterpreter semanticInterpreter = ComponentFactory.getSemanticLayer();
 
 
     @PostMapping
@@ -85,42 +85,43 @@ public class ChatConfigController {
                                         HttpServletRequest request,
                                         HttpServletResponse response) {
         User user = UserHolder.findUser(request, response);
-        return semanticLayer.getModelList(AuthType.ADMIN, domainId, user);
+        return semanticInterpreter.getModelList(AuthType.ADMIN, domainId, user);
     }
 
     @GetMapping("/modelList")
     public List<ModelResp> getModelList(HttpServletRequest request,
                                         HttpServletResponse response) {
         User user = UserHolder.findUser(request, response);
-        return semanticLayer.getModelList(AuthType.ADMIN, null, user);
+        return semanticInterpreter.getModelList(AuthType.ADMIN, null, user);
     }
 
     @GetMapping("/domainList")
     public List<DomainResp> getDomainList(HttpServletRequest request,
                                           HttpServletResponse response) {
         User user = UserHolder.findUser(request, response);
-        return semanticLayer.getDomainList(user);
+        return semanticInterpreter.getDomainList(user);
     }
 
     @GetMapping("/modelList/view")
     public List<ModelResp> getModelListVisible(HttpServletRequest request,
                                                HttpServletResponse response) {
         User user = UserHolder.findUser(request, response);
-        return semanticLayer.getModelList(AuthType.VISIBLE, null, user);
+        return semanticInterpreter.getModelList(AuthType.VISIBLE, null, user);
     }
 
     @PostMapping("/dimension/page")
-    public PageInfo<DimensionResp> getDimension(@RequestBody PageDimensionReq pageDimensionCmd,
+    public PageInfo<DimensionResp> getDimension(@RequestBody PageDimensionReq pageDimensionReq,
                                                 HttpServletRequest request,
                                                 HttpServletResponse response) {
-        return semanticLayer.getDimensionPage(pageDimensionCmd);
+        return semanticInterpreter.getDimensionPage(pageDimensionReq);
     }
 
     @PostMapping("/metric/page")
-    public PageInfo<MetricResp> getMetric(@RequestBody PageMetricReq pageMetrricCmd,
+    public PageInfo<MetricResp> getMetric(@RequestBody PageMetricReq pageMetricReq,
                                           HttpServletRequest request,
                                           HttpServletResponse response) {
-        return semanticLayer.getMetricPage(pageMetrricCmd);
+        User user = UserHolder.findUser(request, response);
+        return semanticInterpreter.getMetricPage(pageMetricReq, user);
     }
 
 

chat/core/src/main/java/com/tencent/supersonic/chat/rest/ChatController.java

@@ -4,6 +4,7 @@ package com.tencent.supersonic.chat.rest;
 import com.github.pagehelper.PageInfo;
 import com.tencent.supersonic.auth.api.authentication.utils.UserHolder;
 import com.tencent.supersonic.chat.api.pojo.response.ShowCaseResp;
+import com.tencent.supersonic.chat.api.pojo.response.SolvedQueryRecallResp;
 import com.tencent.supersonic.chat.persistence.dataobject.ChatDO;
 import com.tencent.supersonic.chat.api.pojo.response.QueryResp;
 import com.tencent.supersonic.chat.api.pojo.request.PageQueryInfoReq;
@@ -85,4 +86,10 @@ public class ChatController {
         return chatService.queryShowCase(pageQueryInfoCommand, agentId);
     }
 
+    @RequestMapping("/getSolvedQuery")
+    public List<SolvedQueryRecallResp> getSolvedQuery(@RequestParam(value = "queryText") String queryText,
+                                                      @RequestParam(value = "agentId") Integer agentId) {
+        return chatService.getSolvedQuery(queryText, agentId);
+    }
+
 }

chat/core/src/main/java/com/tencent/supersonic/chat/rest/ChatQueryController.java

@@ -1,6 +1,7 @@
 package com.tencent.supersonic.chat.rest;
 
 
+import com.tencent.supersonic.auth.api.authentication.pojo.User;
 import com.tencent.supersonic.auth.api.authentication.utils.UserHolder;
 import com.tencent.supersonic.chat.api.pojo.request.DimensionValueReq;
 import com.tencent.supersonic.chat.api.pojo.request.ExecuteQueryReq;
@@ -34,7 +35,7 @@ public class ChatQueryController {
 
     @PostMapping("search")
     public Object search(@RequestBody QueryReq queryCtx, HttpServletRequest request,
-            HttpServletResponse response) {
+                         HttpServletResponse response) {
         queryCtx.setUser(UserHolder.findUser(request, response));
         return searchService.search(queryCtx);
     }
@@ -54,24 +55,25 @@ public class ChatQueryController {
     }
 
     @PostMapping("execute")
-    public Object execute(@RequestBody ExecuteQueryReq queryCtx,
-            HttpServletRequest request, HttpServletResponse response)
+    public Object execute(@RequestBody ExecuteQueryReq queryReq,
+                          HttpServletRequest request, HttpServletResponse response)
             throws Exception {
-        queryCtx.setUser(UserHolder.findUser(request, response));
-        return queryService.performExecution(queryCtx);
+        queryReq.setUser(UserHolder.findUser(request, response));
+        return queryService.performExecution(queryReq);
     }
 
     @PostMapping("queryContext")
     public Object queryContext(@RequestBody QueryReq queryCtx, HttpServletRequest request,
-            HttpServletResponse response) throws Exception {
+                               HttpServletResponse response) throws Exception {
         queryCtx.setUser(UserHolder.findUser(request, response));
         return queryService.queryContext(queryCtx);
     }
 
     @PostMapping("queryData")
     public Object queryData(@RequestBody QueryDataReq queryData,
-            HttpServletRequest request, HttpServletResponse response)
+                            HttpServletRequest request, HttpServletResponse response)
             throws Exception {
+        queryData.setUser(UserHolder.findUser(request, response));
         return queryService.executeDirectQuery(queryData, UserHolder.findUser(request, response));
     }
 
@@ -82,4 +84,12 @@ public class ChatQueryController {
         return queryService.queryDimensionValue(dimensionValueReq, UserHolder.findUser(request, response));
     }
 
+    @RequestMapping("/getEntityInfo")
+    public Object getEntityInfo(Long queryId, Integer parseId,
+                                HttpServletRequest request,
+                                HttpServletResponse response) {
+        User user = UserHolder.findUser(request, response);
+        return queryService.getEntityInfo(queryId, parseId, user);
+    }
 }
+

chat/core/src/main/java/com/tencent/supersonic/chat/rest/KnowledgeController.java

@@ -3,22 +3,24 @@ package com.tencent.supersonic.chat.rest;
 
 import com.tencent.supersonic.auth.api.authentication.pojo.User;
 import com.tencent.supersonic.auth.api.authentication.utils.UserHolder;
-import com.tencent.supersonic.chat.service.DictionaryService;
+import com.tencent.supersonic.chat.api.pojo.request.DictLatestTaskReq;
+import com.tencent.supersonic.chat.api.pojo.response.DictLatestTaskResp;
+import com.tencent.supersonic.chat.service.ChatKnowledgeService;
 import com.tencent.supersonic.knowledge.listener.ApplicationStartedListener;
-import com.tencent.supersonic.knowledge.dictionary.DictTaskFilter;
+import com.tencent.supersonic.chat.api.pojo.request.DictTaskFilterReq;
 import com.tencent.supersonic.knowledge.dictionary.DimValue2DictCommand;
 import com.tencent.supersonic.knowledge.dictionary.DimValueDictInfo;
 
 import java.util.List;
 import javax.servlet.http.HttpServletRequest;
 import javax.servlet.http.HttpServletResponse;
+import javax.validation.Valid;
 
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.web.bind.annotation.RestController;
 import org.springframework.web.bind.annotation.RequestMapping;
 import org.springframework.web.bind.annotation.PostMapping;
 import org.springframework.web.bind.annotation.RequestBody;
-import org.springframework.web.bind.annotation.DeleteMapping;
 import org.springframework.web.bind.annotation.GetMapping;
 import org.springframework.web.bind.annotation.PutMapping;
 
@@ -28,13 +30,14 @@ import org.springframework.web.bind.annotation.PutMapping;
 public class KnowledgeController {
 
     @Autowired
-    private DictionaryService dictApplicationService;
+    private ChatKnowledgeService knowledgeService;
 
     @Autowired
     private ApplicationStartedListener applicationStartedListener;
 
     /**
      * addDictInfo
+     * write specific dimension values to the knowledge base
      *
      * @param dimValue2DictCommend
      */
@@ -43,20 +46,21 @@ public class KnowledgeController {
                             HttpServletRequest request,
                             HttpServletResponse response) {
         User user = UserHolder.findUser(request, response);
-        return dictApplicationService.addDictTask(dimValue2DictCommend, user);
+        return knowledgeService.addDictTask(dimValue2DictCommend, user);
     }
 
     /**
      * deleteDictInfo
+     * remove specific dimension values from the knowledge base
      *
      * @param dimValue2DictCommend
      */
-    @DeleteMapping("/task")
+    @PostMapping("/task/delete")
     public Long deleteDictTask(@RequestBody DimValue2DictCommand dimValue2DictCommend,
                                HttpServletRequest request,
                                HttpServletResponse response) {
         User user = UserHolder.findUser(request, response);
-        return dictApplicationService.deleteDictTask(dimValue2DictCommend, user);
+        return knowledgeService.deleteDictTask(dimValue2DictCommend, user);
     }
 
     /**
@@ -65,19 +69,44 @@ public class KnowledgeController {
      * @param filter
      */
     @PostMapping("/task/search")
-    public List<DimValueDictInfo> searchDictTaskList(@RequestBody DictTaskFilter filter,
+    public List<DimValueDictInfo> searchDictTaskList(@RequestBody DictTaskFilterReq filter,
                                                      HttpServletRequest request,
                                                      HttpServletResponse response) {
         User user = UserHolder.findUser(request, response);
-        return dictApplicationService.searchDictTaskList(filter, user);
+        return knowledgeService.searchDictTaskList(filter, user);
     }
 
+    /**
+     * searchDictLatestTaskList
+     */
+    @PostMapping("/task/search/latest")
+    public List<DictLatestTaskResp> searchDictLatestTaskList(@RequestBody @Valid DictLatestTaskReq filter,
+                                                             HttpServletRequest request,
+                                                             HttpServletResponse response) {
+        User user = UserHolder.findUser(request, response);
+        return knowledgeService.searchDictLatestTaskList(filter, user);
+    }
+
+    /**
+     * getDictRootPath
+     * get knowledge base file directory
+     *
+     * @return
+     */
     @GetMapping("/rootPath")
     public String getDictRootPath(HttpServletRequest request,
                                   HttpServletResponse response) {
-        return dictApplicationService.getDictRootPath();
+        return knowledgeService.getDictRootPath();
     }
 
+    /**
+     * updateDimValue
+     * update in-memory dictionary files in real time
+     *
+     * @param request
+     * @param response
+     * @return
+     */
     @PutMapping("/knowledge/dimValue")
     public Boolean updateDimValue(HttpServletRequest request,
                                   HttpServletResponse response) {

chat/core/src/main/java/com/tencent/supersonic/chat/service/ChatKnowledgeService.java

@@ -1,21 +1,25 @@
 package com.tencent.supersonic.chat.service;
 
 import com.tencent.supersonic.auth.api.authentication.pojo.User;
+import com.tencent.supersonic.chat.api.pojo.request.DictLatestTaskReq;
+import com.tencent.supersonic.chat.api.pojo.response.DictLatestTaskResp;
 import com.tencent.supersonic.knowledge.dictionary.DictConfig;
-import com.tencent.supersonic.knowledge.dictionary.DictTaskFilter;
+import com.tencent.supersonic.chat.api.pojo.request.DictTaskFilterReq;
 import com.tencent.supersonic.knowledge.dictionary.DimValue2DictCommand;
 import com.tencent.supersonic.knowledge.dictionary.DimValueDictInfo;
 
 import java.util.List;
 
-public interface DictionaryService {
+public interface ChatKnowledgeService {
     Long addDictTask(DimValue2DictCommand dimValue2DictCommend, User user);
 
     Long deleteDictTask(DimValue2DictCommand dimValue2DictCommend, User user);
 
-    List<DimValueDictInfo> searchDictTaskList(DictTaskFilter filter, User user);
+    List<DimValueDictInfo> searchDictTaskList(DictTaskFilterReq filter, User user);
 
     DictConfig getDictInfoByModelId(Long modelId);
 
     String getDictRootPath();
+
+    List<DictLatestTaskResp> searchDictLatestTaskList(DictLatestTaskReq filter, User user);
 }

chat/core/src/main/java/com/tencent/supersonic/chat/service/ChatService.java

@@ -8,6 +8,7 @@ import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
 import com.tencent.supersonic.chat.api.pojo.response.ParseResp;
 import com.tencent.supersonic.chat.api.pojo.response.QueryResult;
 import com.tencent.supersonic.chat.api.pojo.response.ShowCaseResp;
+import com.tencent.supersonic.chat.api.pojo.response.SolvedQueryRecallResp;
 import com.tencent.supersonic.chat.persistence.dataobject.ChatDO;
 import com.tencent.supersonic.chat.persistence.dataobject.ChatParseDO;
 import com.tencent.supersonic.chat.persistence.dataobject.ChatQueryDO;
@@ -22,23 +23,23 @@ public interface ChatService {
      * @param chatId
      * @return
      */
-    public Long getContextModel(Integer chatId);
+    Long getContextModel(Integer chatId);
 
-    public ChatContext getOrCreateContext(int chatId);
+    ChatContext getOrCreateContext(int chatId);
 
-    public void updateContext(ChatContext chatCtx);
+    void updateContext(ChatContext chatCtx);
 
-    public void switchContext(ChatContext chatCtx);
+    void switchContext(ChatContext chatCtx);
 
-    public Boolean addChat(User user, String chatName, Integer agentId);
+    Boolean addChat(User user, String chatName, Integer agentId);
 
-    public List<ChatDO> getAll(String userName, Integer agentId);
+    List<ChatDO> getAll(String userName, Integer agentId);
 
-    public boolean updateChatName(Long chatId, String chatName, String userName);
+    boolean updateChatName(Long chatId, String chatName, String userName);
 
-    public boolean updateFeedback(Integer id, Integer score, String feedback);
+    boolean updateFeedback(Integer id, Integer score, String feedback);
 
-    public boolean updateChatIsTop(Long chatId, int isTop);
+    boolean updateChatIsTop(Long chatId, int isTop);
 
     Boolean deleteChat(Long chatId, String userName);
 
@@ -46,20 +47,22 @@ public interface ChatService {
 
     ShowCaseResp queryShowCase(PageQueryInfoReq pageQueryInfoCommend, int agentId);
 
-    public void addQuery(QueryResult queryResult, ChatContext chatCtx);
+    void addQuery(QueryResult queryResult, ChatContext chatCtx);
 
-    public void batchAddParse(ChatContext chatCtx, QueryReq queryReq,
+    void batchAddParse(ChatContext chatCtx, QueryReq queryReq,
                          ParseResp parseResult,
                          List<SemanticParseInfo> candidateParses,
                          List<SemanticParseInfo> selectedParses);
 
-    public ChatQueryDO getLastQuery(long chatId);
+    ChatQueryDO getLastQuery(long chatId);
 
-    public int updateQuery(ChatQueryDO chatQueryDO);
+    int updateQuery(ChatQueryDO chatQueryDO);
 
-    public Boolean updateQuery(Long questionId, QueryResult queryResult, ChatContext chatCtx);
+    Boolean updateQuery(Long questionId, QueryResult queryResult, ChatContext chatCtx);
 
-    public ChatParseDO getParseInfo(Long questionId, String userName, int parseId);
+    ChatParseDO getParseInfo(Long questionId, String userName, int parseId);
 
-    public Boolean deleteChatQuery(Long questionId);
+    Boolean deleteChatQuery(Long questionId);
+
+    List<SolvedQueryRecallResp> getSolvedQuery(String queryText, Integer agentId);
 }

chat/core/src/main/java/com/tencent/supersonic/chat/service/QueryService.java

@@ -5,6 +5,7 @@ import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
 import com.tencent.supersonic.chat.api.pojo.request.DimensionValueReq;
 import com.tencent.supersonic.chat.api.pojo.request.ExecuteQueryReq;
 import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
+import com.tencent.supersonic.chat.api.pojo.response.EntityInfo;
 import com.tencent.supersonic.chat.api.pojo.response.ParseResp;
 import com.tencent.supersonic.chat.api.pojo.response.QueryResult;
 import com.tencent.supersonic.chat.api.pojo.request.QueryDataReq;
@@ -25,5 +26,8 @@ public interface QueryService {
 
     QueryResult executeDirectQuery(QueryDataReq queryData, User user) throws SqlParseException;
 
+    EntityInfo getEntityInfo(Long queryId, Integer parseId, User user);
+
     Object queryDimensionValue(DimensionValueReq dimensionValueReq, User user) throws Exception;
 }
+

chat/core/src/main/java/com/tencent/supersonic/chat/service/SemanticService.java

@@ -12,7 +12,7 @@ import static com.tencent.supersonic.common.pojo.Constants.TIME_FORMAT;
 import static com.tencent.supersonic.common.pojo.Constants.WEEK;
 
 import com.tencent.supersonic.auth.api.authentication.pojo.User;
-import com.tencent.supersonic.chat.api.component.SemanticLayer;
+import com.tencent.supersonic.chat.api.component.SemanticInterpreter;
 import com.tencent.supersonic.chat.api.pojo.ModelSchema;
 import com.tencent.supersonic.chat.api.pojo.SchemaElement;
 import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
@@ -26,9 +26,9 @@ import com.tencent.supersonic.chat.api.pojo.response.ChatConfigResp;
 import com.tencent.supersonic.chat.api.pojo.response.ChatConfigRichResp;
 import com.tencent.supersonic.chat.api.pojo.response.ChatDefaultRichConfigResp;
 import com.tencent.supersonic.chat.api.pojo.response.DataInfo;
-import com.tencent.supersonic.chat.api.pojo.response.ModelInfo;
 import com.tencent.supersonic.chat.api.pojo.response.EntityInfo;
 import com.tencent.supersonic.chat.api.pojo.response.MetricInfo;
+import com.tencent.supersonic.chat.api.pojo.response.ModelInfo;
 import com.tencent.supersonic.chat.config.AggregatorConfig;
 import com.tencent.supersonic.chat.utils.ComponentFactory;
 import com.tencent.supersonic.chat.utils.QueryReqBuilder;
@@ -51,6 +51,7 @@ import java.time.YearMonth;
 import java.time.format.DateTimeFormatter;
 import java.util.ArrayList;
 import java.util.Arrays;
+import java.util.Collections;
 import java.util.Comparator;
 import java.util.HashMap;
 import java.util.LinkedHashSet;
@@ -62,6 +63,7 @@ import java.util.Set;
 import java.util.concurrent.CompletableFuture;
 import java.util.stream.Collectors;
 import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
 import org.springframework.beans.BeanUtils;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.stereotype.Service;
@@ -78,7 +80,7 @@ public class SemanticService {
     @Autowired
     private AggregatorConfig aggregatorConfig;
 
-    private SemanticLayer semanticLayer = ComponentFactory.getSemanticLayer();
+    private SemanticInterpreter semanticInterpreter = ComponentFactory.getSemanticLayer();
 
     public ModelSchema getModelSchema(Long id) {
         ModelSchema modelSchema = schemaService.getModelSchema(id);
@@ -113,15 +115,12 @@ public class SemanticService {
                         }
                     }
                 }
-                if (!"".equals(modelInfoId)) {
-                    try {
-                        setMainModel(entityInfo, parseInfo.getModelId(),
-                                modelInfoId, user);
-
-                        return entityInfo;
-                    } catch (Exception e) {
-                        log.error("setMaintModel error {}", e);
-                    }
+                try {
+                    setMainModel(entityInfo, parseInfo.getModelId(),
+                            modelInfoId, user);
+                    return entityInfo;
+                } catch (Exception e) {
+                    log.error("setMainModel error {}", e);
                 }
             }
         }
@@ -152,6 +151,7 @@ public class SemanticService {
             modelInfo.setWords(modelSchema.getModel().getAlias());
             modelInfo.setBizName(modelSchema.getModel().getBizName());
             if (Objects.nonNull(modelSchema.getEntity())) {
+                modelInfo.setPrimaryEntityName(modelSchema.getEntity().getName());
                 modelInfo.setPrimaryEntityBizName(modelSchema.getEntity().getBizName());
             }
 
@@ -189,10 +189,48 @@ public class SemanticService {
         return entityInfo;
     }
 
-    public void setMainModel(EntityInfo modelInfo, Long model, String entity, User user) {
-        ModelSchema modelSchema = schemaService.getModelSchema(model);
+    public String getPrimaryEntityBizName(EntityInfo entityInfo) {
+        if (Objects.isNull(entityInfo) || Objects.isNull(entityInfo.getModelInfo())) {
+            return null;
+        }
+        return entityInfo.getModelInfo().getPrimaryEntityBizName();
+    }
 
-        modelInfo.setEntityId(entity);
+    public void setMainModel(EntityInfo modelInfo, Long model, String entity, User user) {
+        if (StringUtils.isEmpty(entity)) {
+            return;
+        }
+
+        List<String> entities = Collections.singletonList(entity);
+
+        QueryResultWithSchemaResp queryResultWithColumns = getQueryResultWithSchemaResp(modelInfo, model, entities,
+                user);
+
+        if (queryResultWithColumns != null) {
+            if (!CollectionUtils.isEmpty(queryResultWithColumns.getResultList())
+                    && queryResultWithColumns.getResultList().size() > 0) {
+                Map<String, Object> result = queryResultWithColumns.getResultList().get(0);
+                for (Map.Entry<String, Object> entry : result.entrySet()) {
+                    String entryKey = getEntryKey(entry);
+                    if (entry.getValue() == null || entryKey == null) {
+                        continue;
+                    }
+                    modelInfo.getDimensions().stream().filter(i -> entryKey.equals(i.getBizName()))
+                            .forEach(i -> i.setValue(entry.getValue().toString()));
+                    modelInfo.getMetrics().stream().filter(i -> entryKey.equals(i.getBizName()))
+                            .forEach(i -> i.setValue(entry.getValue().toString()));
+                }
+            }
+        }
+    }
+
+    public QueryResultWithSchemaResp getQueryResultWithSchemaResp(EntityInfo modelInfo, Long model,
+            List<String> entities, User user) {
+        if (CollectionUtils.isEmpty(entities)) {
+            return null;
+        }
+        ModelSchema modelSchema = schemaService.getModelSchema(model);
+        modelInfo.setEntityId(entities.get(0));
         SemanticParseInfo semanticParseInfo = new SemanticParseInfo();
         semanticParseInfo.setModel(modelSchema.getModel());
         semanticParseInfo.setNativeQuery(true);
@@ -217,38 +255,32 @@ public class SemanticService {
         semanticParseInfo.setDateInfo(dateInfo);
 
         // add filter
-        QueryFilter chatFilter = new QueryFilter();
-        chatFilter.setValue(String.valueOf(entity));
-        chatFilter.setOperator(FilterOperatorEnum.EQUALS);
-        chatFilter.setBizName(getEntityPrimaryName(modelInfo));
+        QueryFilter chatFilter = getQueryFilter(modelInfo, entities);
         Set<QueryFilter> chatFilters = new LinkedHashSet();
         chatFilters.add(chatFilter);
         semanticParseInfo.setDimensionFilters(chatFilters);
 
         QueryResultWithSchemaResp queryResultWithColumns = null;
         try {
-            queryResultWithColumns = semanticLayer.queryByStruct(QueryReqBuilder.buildStructReq(semanticParseInfo),
-                    user);
+            queryResultWithColumns = semanticInterpreter.queryByStruct(
+                    QueryReqBuilder.buildStructReq(semanticParseInfo), user);
         } catch (Exception e) {
             log.warn("setMainModel queryByStruct error, e:", e);
         }
+        return queryResultWithColumns;
+    }
 
-        if (queryResultWithColumns != null) {
-            if (!CollectionUtils.isEmpty(queryResultWithColumns.getResultList())
-                    && queryResultWithColumns.getResultList().size() > 0) {
-                Map<String, Object> result = queryResultWithColumns.getResultList().get(0);
-                for (Map.Entry<String, Object> entry : result.entrySet()) {
-                    String entryKey = getEntryKey(entry);
-                    if (entry.getValue() == null || entryKey == null) {
-                        continue;
-                    }
-                    modelInfo.getDimensions().stream().filter(i -> entryKey.equals(i.getBizName()))
-                            .forEach(i -> i.setValue(entry.getValue().toString()));
-                    modelInfo.getMetrics().stream().filter(i -> entryKey.equals(i.getBizName()))
-                            .forEach(i -> i.setValue(entry.getValue().toString()));
-                }
-            }
+    private QueryFilter getQueryFilter(EntityInfo modelInfo, List<String> entities) {
+        QueryFilter chatFilter = new QueryFilter();
+        if (entities.size() == 1) {
+            chatFilter.setValue(entities.get(0));
+            chatFilter.setOperator(FilterOperatorEnum.EQUALS);
+        } else {
+            chatFilter.setValue(entities);
+            chatFilter.setOperator(FilterOperatorEnum.IN);
         }
+        chatFilter.setBizName(getEntityPrimaryName(modelInfo));
+        return chatFilter;
     }
 
     private Set<SchemaElement> getDimensions(EntityInfo modelInfo) {
@@ -332,7 +364,7 @@ public class SemanticService {
     }
 
     public AggregateInfo getAggregateInfo(User user, SemanticParseInfo semanticParseInfo,
-                                          QueryResultWithSchemaResp result) {
+            QueryResultWithSchemaResp result) {
         if (CollectionUtils.isEmpty(semanticParseInfo.getMetrics()) || !aggregatorConfig.getEnableRatio()) {
             return new AggregateInfo();
         }
@@ -384,7 +416,7 @@ public class SemanticService {
     }
 
     private MetricInfo queryRatio(User user, SemanticParseInfo semanticParseInfo, SchemaElement metric,
-                                  AggOperatorEnum aggOperatorEnum, QueryResultWithSchemaResp results) {
+            AggOperatorEnum aggOperatorEnum, QueryResultWithSchemaResp results) {
         MetricInfo metricInfo = new MetricInfo();
         metricInfo.setStatistics(new HashMap<>());
         QueryStructReq queryStructReq = QueryReqBuilder.buildStructRatioReq(semanticParseInfo, metric, aggOperatorEnum);
@@ -393,7 +425,7 @@ public class SemanticService {
 
         queryStructReq.setGroups(new ArrayList<>(Arrays.asList(dateField)));
         queryStructReq.setDateInfo(getRatioDateConf(aggOperatorEnum, semanticParseInfo, results));
-        QueryResultWithSchemaResp queryResp = semanticLayer.queryByStruct(queryStructReq, user);
+        QueryResultWithSchemaResp queryResp = semanticInterpreter.queryByStruct(queryStructReq, user);
         if (Objects.nonNull(queryResp) && !CollectionUtils.isEmpty(queryResp.getResultList())) {
 
             Map<String, Object> result = queryResp.getResultList().get(0);
@@ -432,7 +464,7 @@ public class SemanticService {
     }
 
     private DateConf getRatioDateConf(AggOperatorEnum aggOperatorEnum, SemanticParseInfo semanticParseInfo,
-                                      QueryResultWithSchemaResp results) {
+            QueryResultWithSchemaResp results) {
         String dateField = QueryReqBuilder.getDateField(semanticParseInfo.getDateInfo());
         Optional<String> lastDayOp = results.getResultList().stream()
                 .map(r -> r.get(dateField).toString())

chat/core/src/main/java/com/tencent/supersonic/chat/service/impl/ChatKnowledgeServiceImpl.java

@@ -0,0 +1,268 @@
+package com.tencent.supersonic.chat.service.impl;
+
+import com.tencent.supersonic.auth.api.authentication.pojo.User;
+import com.tencent.supersonic.chat.api.pojo.request.DictLatestTaskReq;
+import com.tencent.supersonic.chat.api.pojo.response.DictLatestTaskResp;
+import com.tencent.supersonic.chat.config.DefaultMetric;
+import com.tencent.supersonic.chat.config.Dim4Dict;
+import com.tencent.supersonic.chat.persistence.dataobject.DimValueDO;
+import com.tencent.supersonic.chat.service.ChatKnowledgeService;
+import com.tencent.supersonic.chat.utils.DictMetaHelper;
+import com.tencent.supersonic.chat.utils.DictQueryHelper;
+import com.tencent.supersonic.common.pojo.Constants;
+import com.tencent.supersonic.common.pojo.enums.TaskStatusEnum;
+import com.tencent.supersonic.common.util.JsonUtil;
+import com.tencent.supersonic.knowledge.dictionary.FileHandler;
+import com.tencent.supersonic.knowledge.listener.ApplicationStartedListener;
+import com.tencent.supersonic.knowledge.persistence.dataobject.DictTaskDO;
+import com.tencent.supersonic.knowledge.utils.DictTaskConverter;
+import com.tencent.supersonic.knowledge.dictionary.DictConfig;
+import com.tencent.supersonic.chat.api.pojo.request.DictTaskFilterReq;
+import com.tencent.supersonic.knowledge.dictionary.DictUpdateMode;
+import com.tencent.supersonic.knowledge.dictionary.DimValue2DictCommand;
+import com.tencent.supersonic.knowledge.dictionary.DimValueDictInfo;
+import com.tencent.supersonic.knowledge.persistence.repository.DictRepository;
+
+
+
+import lombok.extern.slf4j.Slf4j;
+import org.springframework.beans.BeanUtils;
+import org.springframework.beans.factory.annotation.Value;
+import org.springframework.scheduling.annotation.Scheduled;
+import org.springframework.stereotype.Service;
+import org.springframework.util.CollectionUtils;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.ArrayList;
+import java.util.Objects;
+import java.util.Set;
+import java.util.Map;
+import java.util.HashMap;
+
+@Slf4j
+@Service
+public class ChatKnowledgeServiceImpl implements ChatKnowledgeService {
+
+    private final DictMetaHelper metaUtils;
+    private final DictQueryHelper dictQueryHelper;
+    private final FileHandler fileHandler;
+    private final DictRepository dictRepository;
+    private final ApplicationStartedListener applicationStartedListener;
+
+    @Value("${dict.flush.enable:true}")
+    private Boolean dictFlushEnable;
+    @Value("${dict.flush.daily.enable:true}")
+    private Boolean dictFlushDailyEnable;
+    @Value("${dict.file.type:txt}")
+    private String dictFileType;
+    private String dimValue = "DimValue_%d_%d";
+
+    public ChatKnowledgeServiceImpl(DictMetaHelper metaUtils,
+                                    DictQueryHelper dictQueryHelper,
+                                    FileHandler fileHandler,
+                                    DictRepository dictRepository,
+                                    ApplicationStartedListener applicationStartedListener) {
+        this.metaUtils = metaUtils;
+        this.dictQueryHelper = dictQueryHelper;
+        this.fileHandler = fileHandler;
+        this.dictRepository = dictRepository;
+        this.applicationStartedListener = applicationStartedListener;
+    }
+
+    @Scheduled(cron = "${knowledge.dimension.value.cron:0 0 0 * * ?}")
+    public Boolean dailyDictTask() {
+        log.info("[dailyDictTask] start");
+        if (!dictFlushDailyEnable) {
+            log.info("dictFlushDailyEnable is false, now finish dailyDictTask");
+        }
+        DimValue2DictCommand dimValue2DictCommend = new DimValue2DictCommand();
+        dimValue2DictCommend.setUpdateMode(DictUpdateMode.OFFLINE_FULL);
+
+        User user = User.getFakeUser();
+        addDictTask(dimValue2DictCommend, user);
+        log.info("[dailyDictTask] finish");
+        return true;
+    }
+
+    @Override
+    public Long addDictTask(DimValue2DictCommand dimValue2DictCommend, User user) {
+        if (!dictFlushEnable) {
+            return 0L;
+        }
+
+        if (DictUpdateMode.REALTIME_DELETE.equals(dimValue2DictCommend.getUpdateMode())) {
+            return deleteDictTask(dimValue2DictCommend, user);
+        }
+
+        DictTaskDO dictTaskDO = DictTaskConverter.generateDimValueDictTaskDO(dimValue2DictCommend, user);
+        log.info("[addDictTask] dictTaskDO:{}", dictTaskDO);
+        // todo check dimension can not be searched
+
+        dictRepository.createDimValueDictTask(dictTaskDO);
+        runDictTask(dictTaskDO, user);
+
+        return dictTaskDO.getId();
+    }
+
+    public Long runDictTask(DictTaskDO dictTaskDO, User user) {
+        if (Objects.isNull(dictTaskDO)) {
+            return -1L;
+        }
+        DimValue2DictCommand command = JsonUtil.toObject(dictTaskDO.getCommand(), DimValue2DictCommand.class);
+        try {
+            //1. construct internal dictionary requirements
+            List<DimValueDO> dimValueDOList = metaUtils.generateDimValueInfo(command);
+            Set<Long> dimIds = generateDimSet(dimValueDOList);
+            dictTaskDO.setDimIds(JsonUtil.toString(dimIds));
+            dictRepository.updateDictTaskStatus(TaskStatusEnum.RUNNING.getCode(), dictTaskDO);
+            log.debug("dimValueDOList:{}", dimValueDOList);
+            //2. query dimension value information
+            for (DimValueDO dimValueDO : dimValueDOList) {
+                Long modelId = dimValueDO.getModelId();
+                DefaultMetric defaultMetricDesc = dimValueDO.getDefaultMetricDescList().get(0);
+                for (Dim4Dict dim4Dict : dimValueDO.getDimensions()) {
+                    List<String> data = dictQueryHelper.fetchDimValueSingle(modelId, defaultMetricDesc, dim4Dict, user);
+                    //3. local file changes
+                    String fileName = String.format(dimValue + Constants.DOT + dictFileType, modelId,
+                            dim4Dict.getDimId());
+                    fileHandler.writeFile(data, fileName, false);
+                }
+            }
+            applicationStartedListener.updateKnowledgeDimValue();
+            log.debug("updateDictTaskStatus to SUCCESS");
+            dictRepository.updateDictTaskStatus(TaskStatusEnum.SUCCESS.getCode(), dictTaskDO);
+        } catch (Exception e) {
+            log.warn("addDictInfo exception:", e);
+            dictRepository.updateDictTaskStatus(TaskStatusEnum.ERROR.getCode(), dictTaskDO);
+        }
+        return 1L;
+    }
+
+    private Set<Long> generateDimSet(List<DimValueDO> dimValueDOList) {
+        Set<Long> dimIds = new HashSet<>();
+        if (!CollectionUtils.isEmpty(dimValueDOList)) {
+            dimValueDOList.stream().forEach(dimValueDO -> {
+                if (!CollectionUtils.isEmpty(dimValueDO.getDimensions())) {
+                    dimValueDO.getDimensions().stream().forEach(dim4Dict -> dimIds.add(dim4Dict.getDimId()));
+                }
+            });
+        }
+        return dimIds;
+    }
+
+    @Override
+    public Long deleteDictTask(DimValue2DictCommand dimValue2DictCommand, User user) {
+        if (!dictFlushEnable) {
+            return 0L;
+        }
+        if (Objects.isNull(dimValue2DictCommand) || !DictUpdateMode.REALTIME_DELETE.equals(
+                dimValue2DictCommand.getUpdateMode())) {
+            throw new RuntimeException("illegal parameter");
+        }
+
+        DictTaskDO dictTaskDO = DictTaskConverter.generateDimValueDictTaskDO(dimValue2DictCommand, user);
+        log.info("[deleteDictTask] dictTaskDO:{}", dictTaskDO);
+        Set<Long> dimIds = generateDimSetFromCommand(dimValue2DictCommand.getModelAndDimPair());
+        dictTaskDO.setDimIds(JsonUtil.toString(dimIds));
+        dictRepository.createDimValueDictTask(dictTaskDO);
+
+        Map<Long, List<Long>> modelAndDimPair = dimValue2DictCommand.getModelAndDimPair();
+        if (CollectionUtils.isEmpty(modelAndDimPair)) {
+            return 0L;
+        }
+        for (Long modelId : modelAndDimPair.keySet()) {
+            if (CollectionUtils.isEmpty(modelAndDimPair.get(modelId))) {
+                continue;
+            }
+            for (Long dimId : modelAndDimPair.get(modelId)) {
+                String fileName = String.format(dimValue + Constants.DOT + dictFileType, modelId, dimId);
+                fileHandler.deleteDictFile(fileName);
+            }
+        }
+        applicationStartedListener.updateKnowledgeDimValue();
+        dictRepository.updateDictTaskStatus(TaskStatusEnum.SUCCESS.getCode(), dictTaskDO);
+        applicationStartedListener.updateKnowledgeDimValue();
+
+        return 1L;
+    }
+
+    private Set<Long> generateDimSetFromCommand(Map<Long, List<Long>> modelAndDimPair) {
+        Set<Long> dimIds = new HashSet<>();
+        if (!CollectionUtils.isEmpty(modelAndDimPair)) {
+            modelAndDimPair.forEach((k, v) -> dimIds.addAll(v));
+        }
+        return dimIds;
+    }
+
+    @Override
+    public String getDictRootPath() {
+        return fileHandler.getDictRootPath();
+    }
+
+    @Override
+    public List<DictLatestTaskResp> searchDictLatestTaskList(DictLatestTaskReq latestFilter, User user) {
+        DictTaskFilterReq filter = new DictTaskFilterReq();
+        BeanUtils.copyProperties(latestFilter, filter);
+        List<DimValueDictInfo> dimValueDictInfoList = searchDictTaskList(filter, user);
+        return extractLatestTask(dimValueDictInfoList, latestFilter.getDimIds());
+    }
+
+    private List<DictLatestTaskResp> extractLatestTask(List<DimValueDictInfo> dimValueDictInfoList, List<Long> dimIds) {
+        List<DictLatestTaskResp> dictLatestTaskRespList = new ArrayList<>();
+        Map<Long, DictLatestTaskResp> dimAndTaskPair = new HashMap<>(50);
+        for (DimValueDictInfo dimValueDictInfo : dimValueDictInfoList) {
+            //1. filter
+            if (Objects.isNull(dimValueDictInfo) || CollectionUtils.isEmpty(dimValueDictInfo.getDimIds())) {
+                continue;
+            }
+            if (!CollectionUtils.isEmpty(dimIds)) {
+                Set<Long> tmp = dimValueDictInfo.getDimIds();
+                tmp.retainAll(dimIds);
+                dimValueDictInfo.setDimIds(tmp);
+                if (tmp.size() <= 0) {
+                    continue;
+                }
+            }
+
+            // 2. extract
+            Set<Long> dimIdList = dimValueDictInfo.getDimIds();
+            for (Long dimId : dimIdList) {
+                DictLatestTaskResp dictLatestTaskResp = new DictLatestTaskResp();
+                if (!dimAndTaskPair.containsKey(dimId)) {
+                    BeanUtils.copyProperties(dimValueDictInfo, dictLatestTaskResp);
+                    dictLatestTaskResp.setDimId(dimId);
+                } else {
+                    DictLatestTaskResp dictLatestTaskExist = dimAndTaskPair.get(dimId);
+                    if (dictLatestTaskExist.getCreatedAt().before(dimValueDictInfo.getCreatedAt())) {
+                        BeanUtils.copyProperties(dimValueDictInfo, dictLatestTaskResp);
+                        dictLatestTaskResp.setDimId(dimId);
+                    } else {
+                        dictLatestTaskResp = dictLatestTaskExist;
+                    }
+                }
+                dimAndTaskPair.put(dimId, dictLatestTaskResp);
+            }
+
+        }
+
+        if (dimAndTaskPair.size() >= 0 && !CollectionUtils.isEmpty(dimAndTaskPair.values())) {
+            dimAndTaskPair.values().stream()
+                    .filter(v -> !v.getCommand().contains(DictUpdateMode.REALTIME_DELETE.name()))
+                    .forEach(v -> dictLatestTaskRespList.add(v));
+        }
+
+
+        return dictLatestTaskRespList;
+    }
+
+    @Override
+    public List<DimValueDictInfo> searchDictTaskList(DictTaskFilterReq filter, User user) {
+        return dictRepository.searchDictTaskList(filter);
+    }
+
+    @Override
+    public DictConfig getDictInfoByModelId(Long modelId) {
+        return dictRepository.getDictInfoByModelId(modelId);
+    }
+}

chat/core/src/main/java/com/tencent/supersonic/chat/service/impl/ChatServiceImpl.java

@@ -8,6 +8,7 @@ import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
 import com.tencent.supersonic.chat.api.pojo.response.ParseResp;
 import com.tencent.supersonic.chat.api.pojo.response.QueryResult;
 import com.tencent.supersonic.chat.api.pojo.response.ShowCaseResp;
+import com.tencent.supersonic.chat.api.pojo.response.SolvedQueryRecallResp;
 import com.tencent.supersonic.chat.persistence.dataobject.ChatDO;
 import com.tencent.supersonic.chat.persistence.dataobject.ChatParseDO;
 import com.tencent.supersonic.chat.persistence.dataobject.ChatQueryDO;
@@ -22,13 +23,17 @@ import java.text.SimpleDateFormat;
 import java.util.List;
 import java.util.Map;
 import java.util.Objects;
+import java.util.Set;
 import java.util.stream.Collectors;
 
 import com.tencent.supersonic.chat.service.ChatService;
+import com.tencent.supersonic.chat.utils.SolvedQueryManager;
 import com.tencent.supersonic.common.util.JsonUtil;
 import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.compress.utils.Lists;
 import org.springframework.context.annotation.Primary;
 import org.springframework.stereotype.Service;
+import org.springframework.util.CollectionUtils;
 
 @Service("ChatService")
 @Primary
@@ -38,12 +43,14 @@ public class ChatServiceImpl implements ChatService {
     private ChatContextRepository chatContextRepository;
     private ChatRepository chatRepository;
     private ChatQueryRepository chatQueryRepository;
+    private SolvedQueryManager solvedQueryManager;
 
     public ChatServiceImpl(ChatContextRepository chatContextRepository, ChatRepository chatRepository,
-                           ChatQueryRepository chatQueryRepository) {
+            ChatQueryRepository chatQueryRepository, SolvedQueryManager solvedQueryManager) {
         this.chatContextRepository = chatContextRepository;
         this.chatRepository = chatRepository;
         this.chatQueryRepository = chatQueryRepository;
+        this.solvedQueryManager = solvedQueryManager;
     }
 
     @Override
@@ -125,8 +132,8 @@ public class ChatServiceImpl implements ChatService {
     }
 
     @Override
-    public PageInfo<QueryResp> queryInfo(PageQueryInfoReq pageQueryInfoCommend, long chatId) {
-        return chatQueryRepository.getChatQuery(pageQueryInfoCommend, chatId);
+    public PageInfo<QueryResp> queryInfo(PageQueryInfoReq pageQueryInfoReq, long chatId) {
+        return chatQueryRepository.getChatQuery(pageQueryInfoReq, chatId);
     }
 
     @Override
@@ -167,9 +174,9 @@ public class ChatServiceImpl implements ChatService {
 
     @Override
     public void batchAddParse(ChatContext chatCtx, QueryReq queryReq,
-                              ParseResp parseResult,
-                              List<SemanticParseInfo> candidateParses,
-                              List<SemanticParseInfo> selectedParses) {
+            ParseResp parseResult,
+            List<SemanticParseInfo> candidateParses,
+            List<SemanticParseInfo> selectedParses) {
         chatQueryRepository.batchSaveParseInfo(chatCtx, queryReq, parseResult, candidateParses, selectedParses);
 
     }
@@ -192,4 +199,32 @@ public class ChatServiceImpl implements ChatService {
         return chatQueryRepository.deleteChatQuery(questionId);
     }
 
+    @Override
+    public List<SolvedQueryRecallResp> getSolvedQuery(String queryText, Integer agentId) {
+        //1. recall solved query by queryText
+        List<SolvedQueryRecallResp> solvedQueryRecallResps = solvedQueryManager.recallSolvedQuery(queryText, agentId);
+        if (CollectionUtils.isEmpty(solvedQueryRecallResps)) {
+            return Lists.newArrayList();
+        }
+        List<Long> queryIds = solvedQueryRecallResps.stream()
+                .map(SolvedQueryRecallResp::getQueryId).collect(Collectors.toList());
+        PageQueryInfoReq pageQueryInfoReq = new PageQueryInfoReq();
+        pageQueryInfoReq.setIds(queryIds);
+        pageQueryInfoReq.setPageSize(100);
+        pageQueryInfoReq.setCurrent(1);
+        //2. remove low score query
+        int lowScoreThreshold = 3;
+        PageInfo<QueryResp> queryRespPageInfo = chatQueryRepository.getChatQuery(pageQueryInfoReq, null);
+        List<QueryResp> queryResps = queryRespPageInfo.getList();
+        if (CollectionUtils.isEmpty(queryResps)) {
+            return Lists.newArrayList();
+        }
+        Set<Long> lowScoreQueryIds = queryResps.stream().filter(queryResp ->
+                        queryResp.getScore() != null && queryResp.getScore() <= lowScoreThreshold)
+                .map(QueryResp::getQuestionId).collect(Collectors.toSet());
+        return solvedQueryRecallResps.stream().filter(solvedQueryRecallResp ->
+                        !lowScoreQueryIds.contains(solvedQueryRecallResp.getQueryId()))
+                .collect(Collectors.toList());
+    }
+
 }

chat/core/src/main/java/com/tencent/supersonic/chat/service/impl/ConfigServiceImpl.java

@@ -2,7 +2,7 @@ package com.tencent.supersonic.chat.service.impl;
 
 
 import com.tencent.supersonic.auth.api.authentication.pojo.User;
-import com.tencent.supersonic.chat.api.component.SemanticLayer;
+import com.tencent.supersonic.chat.api.component.SemanticInterpreter;
 import com.tencent.supersonic.chat.api.pojo.ModelSchema;
 import com.tencent.supersonic.chat.api.pojo.SchemaElement;
 import com.tencent.supersonic.chat.api.pojo.request.ItemVisibility;
@@ -63,7 +63,7 @@ public class ConfigServiceImpl implements ConfigService {
     @Autowired
     private ApplicationEventPublisher applicationEventPublisher;
 
-    private SemanticLayer semanticLayer = ComponentFactory.getSemanticLayer();
+    private SemanticInterpreter semanticInterpreter = ComponentFactory.getSemanticLayer();
 
 
     public ConfigServiceImpl(ChatConfigRepository chatConfigRepository,
@@ -353,7 +353,7 @@ public class ConfigServiceImpl implements ConfigService {
     @Override
     public List<ChatConfigRichResp> getAllChatRichConfig() {
         List<ChatConfigRichResp> chatConfigRichInfoList = new ArrayList<>();
-        List<ModelSchema> modelSchemas = semanticLayer.getModelSchema();
+        List<ModelSchema> modelSchemas = semanticInterpreter.getModelSchema();
         modelSchemas.stream().forEach(modelSchema -> {
             ChatConfigRichResp chatConfigRichInfo = getConfigRichInfo(modelSchema.getModel().getId());
             if (Objects.nonNull(chatConfigRichInfo)) {

chat/core/src/main/java/com/tencent/supersonic/chat/service/impl/DictionaryServiceImpl.java

@@ -1,124 +0,0 @@
-package com.tencent.supersonic.chat.service.impl;
-
-import com.tencent.supersonic.auth.api.authentication.pojo.User;
-import com.tencent.supersonic.chat.config.DefaultMetric;
-import com.tencent.supersonic.chat.config.Dim4Dict;
-import com.tencent.supersonic.chat.persistence.dataobject.DimValueDO;
-import com.tencent.supersonic.chat.service.DictionaryService;
-import com.tencent.supersonic.chat.utils.DictMetaHelper;
-import com.tencent.supersonic.chat.utils.DictQueryHelper;
-import com.tencent.supersonic.common.pojo.Constants;
-import com.tencent.supersonic.common.pojo.enums.TaskStatusEnum;
-import com.tencent.supersonic.knowledge.dictionary.FileHandler;
-import com.tencent.supersonic.knowledge.persistence.dataobject.DictTaskDO;
-import com.tencent.supersonic.knowledge.utils.DictTaskConverter;
-import com.tencent.supersonic.knowledge.dictionary.DictConfig;
-import com.tencent.supersonic.knowledge.dictionary.DictTaskFilter;
-import com.tencent.supersonic.knowledge.dictionary.DictUpdateMode;
-import com.tencent.supersonic.knowledge.dictionary.DimValue2DictCommand;
-import com.tencent.supersonic.knowledge.dictionary.DimValueDictInfo;
-import com.tencent.supersonic.knowledge.persistence.repository.DictRepository;
-import java.util.List;
-import java.util.Map;
-import java.util.Objects;
-import lombok.extern.slf4j.Slf4j;
-import org.springframework.beans.factory.annotation.Value;
-import org.springframework.stereotype.Service;
-import org.springframework.util.CollectionUtils;
-
-@Slf4j
-@Service
-public class DictionaryServiceImpl implements DictionaryService {
-
-    private final DictMetaHelper metaUtils;
-    private final DictQueryHelper dictQueryHelper;
-    private final FileHandler fileHandler;
-    private final DictRepository dictRepository;
-    @Value("${dict.flush.enable:true}")
-    private Boolean dictFlushEnable;
-    @Value("${dict.file.type:txt}")
-    private String dictFileType;
-    private String dimValue = "DimValue_%d_%d";
-
-    public DictionaryServiceImpl(DictMetaHelper metaUtils,
-                                 DictQueryHelper dictQueryHelper,
-                                 FileHandler fileHandler,
-                                 DictRepository dictRepository) {
-        this.metaUtils = metaUtils;
-        this.dictQueryHelper = dictQueryHelper;
-        this.fileHandler = fileHandler;
-        this.dictRepository = dictRepository;
-    }
-
-    public Long addDictTask(DimValue2DictCommand dimValue2DictCommend, User user) {
-        if (!dictFlushEnable) {
-            return 0L;
-        }
-        DictTaskDO dictTaskDO = DictTaskConverter.generateDimValueDictTaskPO(dimValue2DictCommend,
-                user);
-        log.info("[addDictTask] dictTaskDO:{}", dictTaskDO);
-        dictRepository.createDimValueDictTask(dictTaskDO);
-        TaskStatusEnum finalStatus = TaskStatusEnum.SUCCESS;
-        try {
-            //1. construct internal dictionary requirements
-            List<DimValueDO> dimValueDOList = metaUtils.generateDimValueInfo(dimValue2DictCommend);
-            log.info("dimValueDOList:{}", dimValueDOList);
-            //2. query dimension value information
-            for (DimValueDO dimValueDO : dimValueDOList) {
-                Long modelId = dimValueDO.getModelId();
-                DefaultMetric defaultMetricDesc = dimValueDO.getDefaultMetricDescList().get(0);
-                for (Dim4Dict dim4Dict : dimValueDO.getDimensions()) {
-                    List<String> data = dictQueryHelper.fetchDimValueSingle(modelId, defaultMetricDesc, dim4Dict, user);
-                    //3. local file changes
-                    String fileName = String.format(dimValue + Constants.DOT + dictFileType, modelId,
-                            dim4Dict.getDimId());
-                    fileHandler.writeFile(data, fileName, false);
-                }
-            }
-        } catch (Exception e) {
-            log.warn("addDictInfo exception:", e);
-            finalStatus = TaskStatusEnum.ERROR;
-        }
-        dictRepository.updateDictTaskStatus(finalStatus.getCode(),
-                dictTaskDO);
-        return 1L;
-    }
-
-
-    public Long deleteDictTask(DimValue2DictCommand dimValue2DictCommend, User user) {
-        if (!dictFlushEnable) {
-            return 0L;
-        }
-        if (Objects.isNull(dimValue2DictCommend) || DictUpdateMode.REALTIME_DELETE.equals(
-                dimValue2DictCommend.getUpdateMode())) {
-            throw new RuntimeException("illegal parameter");
-        }
-        Map<Long, List<Long>> modelAndDimPair = dimValue2DictCommend.getModelAndDimPair();
-        if (CollectionUtils.isEmpty(modelAndDimPair)) {
-            return 0L;
-        }
-        for (Long modelId : modelAndDimPair.keySet()) {
-            if (CollectionUtils.isEmpty(modelAndDimPair.get(modelId))) {
-                continue;
-            }
-            for (Long dimId : modelAndDimPair.get(modelId)) {
-                String fileName = String.format(dimValue + Constants.DOT + dictFileType, modelId, dimId);
-                fileHandler.deleteDictFile(fileName);
-            }
-        }
-
-        return 1L;
-    }
-
-    public String getDictRootPath() {
-        return fileHandler.getDictRootPath();
-    }
-
-    public List<DimValueDictInfo> searchDictTaskList(DictTaskFilter filter, User user) {
-        return dictRepository.searchDictTaskList(filter);
-    }
-
-    public DictConfig getDictInfoByModelId(Long modelId) {
-        return dictRepository.getDictInfoByModelId(modelId);
-    }
-}

chat/core/src/main/java/com/tencent/supersonic/chat/service/impl/PluginServiceImpl.java

@@ -2,7 +2,7 @@ package com.tencent.supersonic.chat.service.impl;
 
 import com.google.common.collect.Lists;
 import com.tencent.supersonic.auth.api.authentication.pojo.User;
-import com.tencent.supersonic.chat.api.component.SemanticLayer;
+import com.tencent.supersonic.chat.api.component.SemanticInterpreter;
 import com.tencent.supersonic.chat.api.pojo.request.PluginQueryReq;
 import com.tencent.supersonic.chat.persistence.dataobject.PluginDO;
 import com.tencent.supersonic.chat.persistence.dataobject.PluginDOExample;
@@ -152,8 +152,8 @@ public class PluginServiceImpl implements PluginService {
     }
 
     private List<Plugin> authCheck(List<Plugin> plugins, User user) {
-        SemanticLayer semanticLayer = ComponentFactory.getSemanticLayer();
-        List<Long> modelIdAuthorized = semanticLayer.getModelList(AuthType.ADMIN, null, user).stream()
+        SemanticInterpreter semanticInterpreter = ComponentFactory.getSemanticLayer();
+        List<Long> modelIdAuthorized = semanticInterpreter.getModelList(AuthType.ADMIN, null, user).stream()
                 .map(ModelResp::getId).collect(Collectors.toList());
         plugins = plugins.stream().filter(plugin -> {
             if (CollectionUtils.isEmpty(plugin.getModelList()) || plugin.isContainsAllModel()) {

chat/core/src/main/java/com/tencent/supersonic/chat/service/impl/QueryServiceImpl.java

@@ -3,50 +3,67 @@ package com.tencent.supersonic.chat.service.impl;
 
 import com.tencent.supersonic.auth.api.authentication.pojo.User;
 import com.tencent.supersonic.chat.api.component.SchemaMapper;
-import com.tencent.supersonic.chat.api.component.SemanticQuery;
+import com.tencent.supersonic.chat.api.component.SemanticInterpreter;
 import com.tencent.supersonic.chat.api.component.SemanticParser;
+import com.tencent.supersonic.chat.api.component.SemanticQuery;
 import com.tencent.supersonic.chat.api.pojo.ChatContext;
 import com.tencent.supersonic.chat.api.pojo.QueryContext;
 import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
 import com.tencent.supersonic.chat.api.pojo.request.DimensionValueReq;
 import com.tencent.supersonic.chat.api.pojo.request.ExecuteQueryReq;
+import com.tencent.supersonic.chat.api.pojo.request.QueryDataReq;
+import com.tencent.supersonic.chat.api.pojo.request.QueryFilter;
 import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
+import com.tencent.supersonic.chat.api.pojo.request.SolvedQueryReq;
 import com.tencent.supersonic.chat.api.pojo.response.EntityInfo;
 import com.tencent.supersonic.chat.api.pojo.response.ParseResp;
 import com.tencent.supersonic.chat.api.pojo.response.QueryResult;
 import com.tencent.supersonic.chat.api.pojo.response.QueryState;
+import com.tencent.supersonic.chat.parser.llm.dsl.DSLParseResult;
 import com.tencent.supersonic.chat.persistence.dataobject.ChatParseDO;
+import com.tencent.supersonic.chat.persistence.dataobject.ChatQueryDO;
 import com.tencent.supersonic.chat.persistence.dataobject.CostType;
 import com.tencent.supersonic.chat.persistence.dataobject.StatisticsDO;
-import com.tencent.supersonic.chat.query.QuerySelector;
-import com.tencent.supersonic.chat.api.pojo.request.QueryDataReq;
 import com.tencent.supersonic.chat.query.QueryManager;
+import com.tencent.supersonic.chat.query.QuerySelector;
+import com.tencent.supersonic.chat.query.llm.dsl.DslQuery;
+import com.tencent.supersonic.chat.query.llm.dsl.LLMResp;
+import com.tencent.supersonic.chat.responder.execute.ExecuteResponder;
+import com.tencent.supersonic.chat.responder.parse.ParseResponder;
 import com.tencent.supersonic.chat.service.ChatService;
 import com.tencent.supersonic.chat.service.QueryService;
 import com.tencent.supersonic.chat.service.SemanticService;
 import com.tencent.supersonic.chat.service.StatisticsService;
 import com.tencent.supersonic.chat.utils.ComponentFactory;
-
-import java.util.List;
-import java.util.ArrayList;
-import java.util.Set;
-import java.util.HashSet;
-import java.util.Comparator;
-import java.util.Objects;
-import java.util.stream.Collectors;
-
-//import com.tencent.supersonic.common.pojo.Aggregator;
+import com.tencent.supersonic.chat.utils.SolvedQueryManager;
+import com.tencent.supersonic.common.pojo.Constants;
 import com.tencent.supersonic.common.pojo.DateConf;
-//import com.tencent.supersonic.common.pojo.enums.AggOperatorEnum;
+import com.tencent.supersonic.common.pojo.QueryColumn;
 import com.tencent.supersonic.common.util.ContextUtils;
+import com.tencent.supersonic.common.util.DateUtils;
 import com.tencent.supersonic.common.util.JsonUtil;
+import com.tencent.supersonic.common.util.jsqlparser.FilterExpression;
+import com.tencent.supersonic.common.util.jsqlparser.SqlParserReplaceHelper;
+import com.tencent.supersonic.common.util.jsqlparser.SqlParserSelectHelper;
+import com.tencent.supersonic.knowledge.dictionary.MapResult;
+import com.tencent.supersonic.knowledge.service.SearchService;
+import com.tencent.supersonic.semantic.api.model.response.ExplainResp;
 import com.tencent.supersonic.semantic.api.model.response.QueryResultWithSchemaResp;
 import com.tencent.supersonic.semantic.api.query.enums.FilterOperatorEnum;
-import com.tencent.supersonic.semantic.api.query.pojo.Filter;
 import com.tencent.supersonic.semantic.api.query.request.QueryStructReq;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
 import lombok.extern.slf4j.Slf4j;
 import org.apache.calcite.sql.parser.SqlParseException;
-import org.springframework.beans.BeanUtils;
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.lang3.StringUtils;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.beans.factory.annotation.Value;
 import org.springframework.context.annotation.Primary;
@@ -63,8 +80,8 @@ public class QueryServiceImpl implements QueryService {
     private ChatService chatService;
     @Autowired
     private StatisticsService statisticsService;
-
-    private final String entity = "ENTITY";
+    @Autowired
+    private SolvedQueryManager solvedQueryManager;
 
     @Value("${time.threshold: 100}")
     private Integer timeThreshold;
@@ -72,6 +89,8 @@ public class QueryServiceImpl implements QueryService {
     private List<SchemaMapper> schemaMappers = ComponentFactory.getSchemaMappers();
     private List<SemanticParser> semanticParsers = ComponentFactory.getSemanticParsers();
     private QuerySelector querySelector = ComponentFactory.getQuerySelector();
+    private List<ParseResponder> parseResponders = ComponentFactory.getParseResponders();
+    private List<ExecuteResponder> executeResponders = ComponentFactory.getExecuteResponders();
 
     @Override
     public ParseResp performParsing(QueryReq queryReq) {
@@ -82,20 +101,16 @@ public class QueryServiceImpl implements QueryService {
         schemaMappers.stream().forEach(mapper -> {
             Long startTime = System.currentTimeMillis();
             mapper.map(queryCtx);
-            Long endTime = System.currentTimeMillis();
-            String className = mapper.getClass().getSimpleName();
-            timeCostDOList.add(StatisticsDO.builder().cost((int) (endTime - startTime))
-                    .interfaceName(className).type(CostType.MAPPER.getType()).build());
-            log.info("{} result:{}", className, JsonUtil.toString(queryCtx));
+            timeCostDOList.add(StatisticsDO.builder().cost((int) (System.currentTimeMillis() - startTime))
+                    .interfaceName(mapper.getClass().getSimpleName()).type(CostType.MAPPER.getType()).build());
+            log.info("{} result:{}", mapper.getClass().getSimpleName(), JsonUtil.toString(queryCtx));
         });
         semanticParsers.stream().forEach(parser -> {
             Long startTime = System.currentTimeMillis();
             parser.parse(queryCtx, chatCtx);
-            Long endTime = System.currentTimeMillis();
-            String className = parser.getClass().getSimpleName();
-            timeCostDOList.add(StatisticsDO.builder().cost((int) (endTime - startTime))
-                    .interfaceName(className).type(CostType.PARSER.getType()).build());
-            log.info("{} result:{}", className, JsonUtil.toString(queryCtx));
+            timeCostDOList.add(StatisticsDO.builder().cost((int) (System.currentTimeMillis() - startTime))
+                    .interfaceName(parser.getClass().getSimpleName()).type(CostType.PARSER.getType()).build());
+            log.info("{} result:{}", parser.getClass().getSimpleName(), JsonUtil.toString(queryCtx));
         });
         ParseResp parseResult;
         if (queryCtx.getCandidateQueries().size() > 0) {
@@ -105,19 +120,9 @@ public class QueryServiceImpl implements QueryService {
             log.debug("pick after [{}]", selectedQueries.stream().collect(
                     Collectors.toList()));
 
-            List<SemanticParseInfo> selectedParses = selectedQueries.stream()
-                    .map(SemanticQuery::getParseInfo)
-                    .sorted(Comparator.comparingDouble(SemanticParseInfo::getScore).reversed())
-                    .collect(Collectors.toList());
-            selectedParses.forEach(parseInfo -> {
-                if (parseInfo.getQueryMode().contains(entity)) {
-                    EntityInfo entityInfo = ContextUtils.getBean(SemanticService.class)
-                            .getEntityInfo(parseInfo, queryReq.getUser());
-                    parseInfo.setEntityInfo(entityInfo);
-                }
-            });
-            List<SemanticParseInfo> candidateParses = queryCtx.getCandidateQueries().stream()
-                    .map(SemanticQuery::getParseInfo).collect(Collectors.toList());
+            List<SemanticParseInfo> selectedParses = convertParseInfo(selectedQueries);
+            List<SemanticParseInfo> candidateParses = convertParseInfo(queryCtx.getCandidateQueries());
+            candidateParses = getTop5CandidateParseInfo(selectedParses, candidateParses);
             parseResult = ParseResp.builder()
                     .chatId(queryReq.getChatId())
                     .queryText(queryReq.getQueryText())
@@ -135,13 +140,47 @@ public class QueryServiceImpl implements QueryService {
                     .state(ParseResp.ParseState.FAILED)
                     .build();
         }
+        for (ParseResponder parseResponder : parseResponders) {
+            parseResponder.fillResponse(parseResult, queryCtx);
+        }
         return parseResult;
     }
 
+    private List<SemanticParseInfo> convertParseInfo(List<SemanticQuery> semanticQueries) {
+        return semanticQueries.stream()
+                .map(SemanticQuery::getParseInfo)
+                .sorted(Comparator.comparingDouble(SemanticParseInfo::getScore).reversed())
+                .collect(Collectors.toList());
+    }
+
+    private List<SemanticParseInfo> getTop5CandidateParseInfo(List<SemanticParseInfo> selectedParses,
+            List<SemanticParseInfo> candidateParses) {
+        if (CollectionUtils.isEmpty(selectedParses) || CollectionUtils.isEmpty(candidateParses)) {
+            return candidateParses;
+        }
+        int selectParseSize = selectedParses.size();
+        Set<Double> selectParseScoreSet = selectedParses.stream()
+                .map(SemanticParseInfo::getScore).collect(Collectors.toSet());
+        int candidateParseSize = 5 - selectParseSize;
+        candidateParses = candidateParses.stream()
+                .filter(candidateParse -> !selectParseScoreSet.contains(candidateParse.getScore()))
+                .collect(Collectors.toList());
+        SemanticParseInfo semanticParseInfo = selectedParses.get(0);
+        Long modelId = semanticParseInfo.getModelId();
+        if (modelId == null || modelId <= 0) {
+            return candidateParses;
+        }
+        return candidateParses.stream()
+                .sorted(Comparator.comparing(parse -> !parse.getModelId().equals(modelId)))
+                .limit(candidateParseSize)
+                .collect(Collectors.toList());
+    }
+
     @Override
     public QueryResult performExecution(ExecuteQueryReq queryReq) throws Exception {
         ChatParseDO chatParseDO = chatService.getParseInfo(queryReq.getQueryId(),
                 queryReq.getUser().getName(), queryReq.getParseId());
+        ChatQueryDO chatQueryDO = chatService.getLastQuery(queryReq.getChatId());
         List<StatisticsDO> timeCostDOList = new ArrayList<>();
         SemanticParseInfo parseInfo = JsonUtil.toObject(chatParseDO.getParseInfo(), SemanticParseInfo.class);
         SemanticQuery semanticQuery = QueryManager.createQuery(parseInfo.getQueryMode());
@@ -155,9 +194,9 @@ public class QueryServiceImpl implements QueryService {
         chatCtx.setAgentId(queryReq.getAgentId());
         Long startTime = System.currentTimeMillis();
         QueryResult queryResult = semanticQuery.execute(queryReq.getUser());
-        Long endTime = System.currentTimeMillis();
+
         if (queryResult != null) {
-            timeCostDOList.add(StatisticsDO.builder().cost((int) (endTime - startTime))
+            timeCostDOList.add(StatisticsDO.builder().cost((int) (System.currentTimeMillis() - startTime))
                     .interfaceName(semanticQuery.getClass().getSimpleName()).type(CostType.QUERY.getType()).build());
             saveInfo(timeCostDOList, queryReq.getQueryText(), queryReq.getQueryId(),
                     queryReq.getUser().getName(), queryReq.getChatId().longValue());
@@ -166,11 +205,14 @@ public class QueryServiceImpl implements QueryService {
             if (queryReq.isSaveAnswer() && QueryState.SUCCESS.equals(queryResult.getQueryState())) {
                 chatCtx.setParseInfo(parseInfo);
                 chatService.updateContext(chatCtx);
+                saveSolvedQuery(queryReq, parseInfo, chatQueryDO, queryResult);
             }
             chatCtx.setQueryText(queryReq.getQueryText());
             chatCtx.setUser(queryReq.getUser().getName());
-            //chatService.addQuery(queryResult, chatCtx);
             chatService.updateQuery(queryReq.getQueryId(), queryResult, chatCtx);
+            for (ExecuteResponder executeResponder : executeResponders) {
+                executeResponder.fillResponse(queryResult, parseInfo, queryReq);
+            }
         } else {
             chatService.deleteChatQuery(queryReq.getQueryId());
         }
@@ -179,8 +221,8 @@ public class QueryServiceImpl implements QueryService {
     }
 
     public void saveInfo(List<StatisticsDO> timeCostDOList,
-                         String queryText, Long queryId,
-                         String userName, Long chatId) {
+            String queryText, Long queryId,
+            String userName, Long chatId) {
         List<StatisticsDO> list = timeCostDOList.stream()
                 .filter(o -> o.getCost() > timeThreshold).collect(Collectors.toList());
         list.forEach(o -> {
@@ -196,6 +238,18 @@ public class QueryServiceImpl implements QueryService {
         }
     }
 
+    private void saveSolvedQuery(ExecuteQueryReq queryReq, SemanticParseInfo parseInfo,
+                                 ChatQueryDO chatQueryDO, QueryResult queryResult) {
+        if (queryResult.getResponse() == null && CollectionUtils.isEmpty(queryResult.getQueryResults())) {
+            return;
+        }
+        solvedQueryManager.saveSolvedQuery(SolvedQueryReq.builder().parseId(queryReq.getParseId())
+                .queryId(queryReq.getQueryId())
+                .agentId(chatQueryDO.getAgentId())
+                .modelId(parseInfo.getModelId())
+                .queryText(queryReq.getQueryText()).build());
+    }
+
     @Override
     public QueryResult executeQuery(QueryReq queryReq) throws Exception {
         QueryContext queryCtx = new QueryContext(queryReq);
@@ -245,17 +299,148 @@ public class QueryServiceImpl implements QueryService {
 
     @Override
     public QueryResult executeDirectQuery(QueryDataReq queryData, User user) throws SqlParseException {
-        SemanticQuery semanticQuery = QueryManager.createRuleQuery(queryData.getQueryMode());
-        BeanUtils.copyProperties(queryData, semanticQuery.getParseInfo());
+        ChatParseDO chatParseDO = chatService.getParseInfo(queryData.getQueryId(),
+                queryData.getUser().getName(), queryData.getParseId());
+        SemanticParseInfo parseInfo = getSemanticParseInfo(queryData, chatParseDO);
+
+        SemanticQuery semanticQuery = QueryManager.createQuery(parseInfo.getQueryMode());
+
+        if (DslQuery.QUERY_MODE.equals(parseInfo.getQueryMode())) {
+            Map<String, Map<String, String>> filedNameToValueMap = new HashMap<>();
+            Map<String, Map<String, String>> havingFiledNameToValueMap = new HashMap<>();
+            String json = JsonUtil.toString(parseInfo.getProperties().get(Constants.CONTEXT));
+            DSLParseResult dslParseResult = JsonUtil.toObject(json, DSLParseResult.class);
+            LLMResp llmResp = dslParseResult.getLlmResp();
+            String correctorSql = llmResp.getCorrectorSql();
+            log.info("correctorSql before replacing:{}", correctorSql);
+
+            List<FilterExpression> filterExpressionList = SqlParserSelectHelper.getFilterExpression(correctorSql);
+
+            updateFilters(filedNameToValueMap, filterExpressionList, queryData.getDimensionFilters(),
+                    parseInfo.getDimensionFilters());
+
+            updateFilters(havingFiledNameToValueMap, filterExpressionList, queryData.getDimensionFilters(),
+                    parseInfo.getDimensionFilters());
+
+            updateDateInfo(queryData, parseInfo, filedNameToValueMap, filterExpressionList);
+
+            log.info("filedNameToValueMap:{}", filedNameToValueMap);
+            correctorSql = SqlParserReplaceHelper.replaceValue(correctorSql, filedNameToValueMap);
+            log.info("havingFiledNameToValueMap:{}", havingFiledNameToValueMap);
+            correctorSql = SqlParserReplaceHelper.replaceHavingValue(correctorSql, havingFiledNameToValueMap);
+            log.info("correctorSql after replacing:{}", correctorSql);
+            llmResp.setCorrectorSql(correctorSql);
+            dslParseResult.setLlmResp(llmResp);
+            Map<String, Object> properties = new HashMap<>();
+            properties.put(Constants.CONTEXT, dslParseResult);
+            parseInfo.setProperties(properties);
+            parseInfo.getSqlInfo().setLogicSql(correctorSql);
+            semanticQuery.setParseInfo(parseInfo);
+            ExplainResp explain = semanticQuery.explain(user);
+            if (!Objects.isNull(explain)) {
+                parseInfo.getSqlInfo().setQuerySql(explain.getSql());
+            }
+        }
+        log.info("parseInfo:{}", JsonUtil.toString(semanticQuery.getParseInfo().getProperties()));
+        semanticQuery.setParseInfo(parseInfo);
         QueryResult queryResult = semanticQuery.execute(user);
         queryResult.setChatContext(semanticQuery.getParseInfo());
         return queryResult;
     }
 
+    @Override
+    public EntityInfo getEntityInfo(Long queryId, Integer parseId, User user) {
+        ChatParseDO chatParseDO = chatService.getParseInfo(queryId, user.getName(), parseId);
+        SemanticParseInfo parseInfo = JsonUtil.toObject(chatParseDO.getParseInfo(), SemanticParseInfo.class);
+        SemanticService semanticService = ContextUtils.getBean(SemanticService.class);
+        return semanticService.getEntityInfo(parseInfo, user);
+    }
+
+    private void updateDateInfo(QueryDataReq queryData, SemanticParseInfo parseInfo,
+            Map<String, Map<String, String>> filedNameToValueMap, List<FilterExpression> filterExpressionList) {
+        if (Objects.isNull(queryData.getDateInfo())) {
+            return;
+        }
+        Map<String, String> map = new HashMap<>();
+        String dateField = DateUtils.DATE_FIELD;
+        if (queryData.getDateInfo().getStartDate().equals(queryData.getDateInfo().getEndDate())) {
+            for (FilterExpression filterExpression : filterExpressionList) {
+                if (DateUtils.DATE_FIELD.equals(filterExpression.getFieldName())) {
+                    dateField = filterExpression.getFieldName();
+                    map.put(filterExpression.getFieldValue().toString(),
+                            queryData.getDateInfo().getStartDate());
+                    break;
+                }
+            }
+        } else {
+            for (FilterExpression filterExpression : filterExpressionList) {
+                if (DateUtils.DATE_FIELD.equals(filterExpression.getFieldName())) {
+                    dateField = filterExpression.getFieldName();
+                    if (FilterOperatorEnum.GREATER_THAN_EQUALS.getValue().equals(filterExpression.getOperator())
+                            || FilterOperatorEnum.GREATER_THAN.getValue().equals(filterExpression.getOperator())) {
+                        map.put(filterExpression.getFieldValue().toString(),
+                                queryData.getDateInfo().getStartDate());
+                    }
+                    if (FilterOperatorEnum.MINOR_THAN_EQUALS.getValue().equals(filterExpression.getOperator())
+                            || FilterOperatorEnum.MINOR_THAN.getValue().equals(filterExpression.getOperator())) {
+                        map.put(filterExpression.getFieldValue().toString(),
+                                queryData.getDateInfo().getEndDate());
+                    }
+                }
+            }
+        }
+        filedNameToValueMap.put(dateField, map);
+        parseInfo.setDateInfo(queryData.getDateInfo());
+    }
+
+    private void updateFilters(Map<String, Map<String, String>> filedNameToValueMap,
+            List<FilterExpression> filterExpressionList, Set<QueryFilter> metricFilters,
+            Set<QueryFilter> contextMetricFilters) {
+        if (CollectionUtils.isEmpty(metricFilters)) {
+            return;
+        }
+        for (QueryFilter dslQueryFilter : metricFilters) {
+            Map<String, String> map = new HashMap<>();
+            for (FilterExpression filterExpression : filterExpressionList) {
+                if (filterExpression.getFieldName() != null
+                        && filterExpression.getFieldName().contains(dslQueryFilter.getName())
+                        && dslQueryFilter.getOperator().getValue().equals(filterExpression.getOperator())) {
+                    map.put(filterExpression.getFieldValue().toString(), dslQueryFilter.getValue().toString());
+                    contextMetricFilters.stream().forEach(o -> {
+                        if (o.getName().equals(dslQueryFilter.getName())) {
+                            o.setValue(dslQueryFilter.getValue());
+                        }
+                    });
+                    break;
+                }
+            }
+            filedNameToValueMap.put(dslQueryFilter.getName(), map);
+        }
+    }
+
+
+    private SemanticParseInfo getSemanticParseInfo(QueryDataReq queryData, ChatParseDO chatParseDO) {
+        SemanticParseInfo parseInfo = JsonUtil.toObject(chatParseDO.getParseInfo(), SemanticParseInfo.class);
+        if (DslQuery.QUERY_MODE.equals(parseInfo.getQueryMode())) {
+            return parseInfo;
+        }
+        if (CollectionUtils.isNotEmpty(queryData.getDimensions())) {
+            parseInfo.setDimensions(queryData.getDimensions());
+        }
+        if (CollectionUtils.isNotEmpty(queryData.getMetrics())) {
+            parseInfo.setMetrics(queryData.getMetrics());
+        }
+        if (CollectionUtils.isNotEmpty(queryData.getDimensionFilters())) {
+            parseInfo.setDimensionFilters(queryData.getDimensionFilters());
+        }
+        if (Objects.nonNull(queryData.getDateInfo())) {
+            parseInfo.setDateInfo(queryData.getDateInfo());
+        }
+        return parseInfo;
+    }
+
     @Override
     public Object queryDimensionValue(DimensionValueReq dimensionValueReq, User user) throws Exception {
-        com.tencent.supersonic.semantic.query.service.QueryService queryService =
-                ContextUtils.getBean(com.tencent.supersonic.semantic.query.service.QueryService.class);
         QueryStructReq queryStructReq = new QueryStructReq();
 
         DateConf dateConf = new DateConf();
@@ -264,38 +449,52 @@ public class QueryServiceImpl implements QueryService {
         dateConf.setPeriod("DAY");
         queryStructReq.setDateInfo(dateConf);
         queryStructReq.setLimit(20L);
-
-        //        List<Aggregator> aggregators = new ArrayList<>();
-        //        Aggregator aggregator = new Aggregator(dimensionValueReq.getQueryFilter().getBizName(),
-        //                AggOperatorEnum.DISTINCT);
-        //        aggregators.add(aggregator);
-        //        queryStructReq.setAggregators(aggregators);
-
         queryStructReq.setModelId(dimensionValueReq.getModelId());
-        queryStructReq.setNativeQuery(true);
+        queryStructReq.setNativeQuery(false);
         List<String> groups = new ArrayList<>();
         groups.add(dimensionValueReq.getBizName());
         queryStructReq.setGroups(groups);
-        if (!Objects.isNull(dimensionValueReq.getValue())) {
-            List<Filter> dimensionFilters = new ArrayList<>();
-            Filter dimensionFilter = new Filter();
-            dimensionFilter.setOperator(FilterOperatorEnum.LIKE);
-            dimensionFilter.setRelation(Filter.Relation.FILTER);
-            dimensionFilter.setBizName(dimensionValueReq.getBizName());
-            dimensionFilter.setValue(dimensionValueReq.getValue());
-            dimensionFilters.add(dimensionFilter);
-            queryStructReq.setDimensionFilters(dimensionFilters);
+        if ((!Objects.isNull(dimensionValueReq.getValue()))
+                && StringUtils.isNotBlank(dimensionValueReq.getValue().toString())) {
+            return queryHanlpDimensionValue(dimensionValueReq, user);
         }
-        QueryResultWithSchemaResp queryResultWithSchemaResp = queryService.queryByStructWithAuth(queryStructReq, user);
-        Set<String> dimensionValues = new HashSet<>();
-        queryResultWithSchemaResp.getResultList().removeIf(o -> {
-            if (dimensionValues.contains(o.get(dimensionValueReq.getBizName()))) {
-                return true;
-            } else {
-                dimensionValues.add(o.get(dimensionValueReq.getBizName()).toString());
-                return false;
+        SemanticInterpreter semanticInterpreter = ComponentFactory.getSemanticLayer();
+        QueryResultWithSchemaResp queryResultWithSchemaResp = semanticInterpreter.queryByStruct(queryStructReq, user);
+        return queryResultWithSchemaResp;
+    }
+
+    public Object queryHanlpDimensionValue(DimensionValueReq dimensionValueReq, User user) throws Exception {
+        QueryResultWithSchemaResp queryResultWithSchemaResp = new QueryResultWithSchemaResp();
+        Set<Long> detectModelIds = new HashSet<>();
+        detectModelIds.add(dimensionValueReq.getModelId());
+        List<MapResult> mapResultList = SearchService.prefixSearch(dimensionValueReq.getValue().toString(),
+                2000, dimensionValueReq.getAgentId(), detectModelIds);
+        log.info("mapResultList:{}", mapResultList);
+        mapResultList = mapResultList.stream().filter(o -> {
+            for (String nature : o.getNatures()) {
+                String[] natureArray = nature.split("_");
+                if (natureArray[2].equals(dimensionValueReq.getElementID().toString())) {
+                    return true;
+                }
             }
+            return false;
+        }).collect(Collectors.toList());
+        log.info("mapResultList:{}", mapResultList);
+        List<QueryColumn> columns = new ArrayList<>();
+        QueryColumn queryColumn = new QueryColumn();
+        queryColumn.setNameEn(dimensionValueReq.getBizName());
+        queryColumn.setShowType("CATEGORY");
+        queryColumn.setAuthorized(true);
+        queryColumn.setType("CHAR");
+        columns.add(queryColumn);
+        List<Map<String, Object>> resultList = new ArrayList<>();
+        mapResultList.stream().forEach(o -> {
+            Map<String, Object> map = new HashMap<>();
+            map.put(dimensionValueReq.getBizName(), o.getName());
+            resultList.add(map);
         });
+        queryResultWithSchemaResp.setColumns(columns);
+        queryResultWithSchemaResp.setResultList(resultList);
         return queryResultWithSchemaResp;
     }
 

chat/core/src/main/java/com/tencent/supersonic/chat/utils/ComponentFactory.java

@@ -1,7 +1,7 @@
 package com.tencent.supersonic.chat.utils;
 
 import com.tencent.supersonic.chat.api.component.SchemaMapper;
-import com.tencent.supersonic.chat.api.component.SemanticLayer;
+import com.tencent.supersonic.chat.api.component.SemanticInterpreter;
 import com.tencent.supersonic.chat.api.component.SemanticParser;
 
 import com.tencent.supersonic.chat.api.component.SemanticCorrector;
@@ -9,8 +9,10 @@ import java.util.ArrayList;
 import java.util.List;
 import java.util.Objects;
 
-import com.tencent.supersonic.chat.parser.plugin.function.ModelResolver;
+import com.tencent.supersonic.chat.parser.llm.dsl.ModelResolver;
 import com.tencent.supersonic.chat.query.QuerySelector;
+import com.tencent.supersonic.chat.responder.execute.ExecuteResponder;
+import com.tencent.supersonic.chat.responder.parse.ParseResponder;
 import org.apache.commons.collections.CollectionUtils;
 import org.springframework.core.io.support.SpringFactoriesLoader;
 
@@ -18,9 +20,10 @@ public class ComponentFactory {
 
     private static List<SchemaMapper> schemaMappers = new ArrayList<>();
     private static List<SemanticParser> semanticParsers = new ArrayList<>();
-
     private static List<SemanticCorrector> dslCorrections = new ArrayList<>();
-    private static SemanticLayer semanticLayer;
+    private static SemanticInterpreter semanticInterpreter;
+    private static List<ParseResponder> parseResponders = new ArrayList<>();
+    private static List<ExecuteResponder> executeResponders = new ArrayList<>();
     private static QuerySelector querySelector;
     private static ModelResolver modelResolver;
     public static List<SchemaMapper> getSchemaMappers() {
@@ -35,16 +38,24 @@ public class ComponentFactory {
         return CollectionUtils.isEmpty(dslCorrections) ? init(SemanticCorrector.class, dslCorrections) : dslCorrections;
     }
 
-
-    public static SemanticLayer getSemanticLayer() {
-        if (Objects.isNull(semanticLayer)) {
-            semanticLayer = init(SemanticLayer.class);
-        }
-        return semanticLayer;
+    public static List<ParseResponder> getParseResponders() {
+        return CollectionUtils.isEmpty(parseResponders) ? init(ParseResponder.class, parseResponders) : parseResponders;
     }
 
-    public static void setSemanticLayer(SemanticLayer layer) {
-        semanticLayer = layer;
+    public static List<ExecuteResponder> getExecuteResponders() {
+        return CollectionUtils.isEmpty(executeResponders)
+                ? init(ExecuteResponder.class, executeResponders) : executeResponders;
+    }
+
+    public static SemanticInterpreter getSemanticLayer() {
+        if (Objects.isNull(semanticInterpreter)) {
+            semanticInterpreter = init(SemanticInterpreter.class);
+        }
+        return semanticInterpreter;
+    }
+
+    public static void setSemanticLayer(SemanticInterpreter layer) {
+        semanticInterpreter = layer;
     }
 
     public static QuerySelector getQuerySelector() {

chat/core/src/main/java/com/tencent/supersonic/chat/utils/DictMetaHelper.java

@@ -3,7 +3,8 @@ package com.tencent.supersonic.chat.utils;
 import static com.tencent.supersonic.common.pojo.Constants.DAY;
 import static com.tencent.supersonic.common.pojo.Constants.UNDERLINE;
 
-import com.tencent.supersonic.chat.api.component.SemanticLayer;
+import com.github.pagehelper.PageInfo;
+import com.tencent.supersonic.chat.api.component.SemanticInterpreter;
 import com.tencent.supersonic.chat.api.pojo.ModelSchema;
 import com.tencent.supersonic.chat.api.pojo.SchemaElement;
 import com.tencent.supersonic.chat.api.pojo.request.KnowledgeAdvancedConfig;
@@ -26,7 +27,8 @@ import java.util.Objects;
 import java.util.Set;
 import java.util.stream.Collectors;
 
-import org.apache.logging.log4j.util.Strings;
+import com.tencent.supersonic.semantic.api.model.request.PageDimensionReq;
+import com.tencent.supersonic.semantic.api.model.response.DimensionResp;
 import org.springframework.beans.BeanUtils;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.beans.factory.annotation.Value;
@@ -40,7 +42,9 @@ public class DictMetaHelper {
     private ConfigService configService;
     @Value("${model.internal.metric.suffix:internal_cnt}")
     private String internalMetricNameSuffix;
-    private SemanticLayer semanticLayer = ComponentFactory.getSemanticLayer();
+    @Value("${model.internal.day.number:2}")
+    private Integer internalMetricDays;
+    private SemanticInterpreter semanticInterpreter = ComponentFactory.getSemanticLayer();
 
     public List<DimValueDO> generateDimValueInfo(DimValue2DictCommand dimValue2DictCommend) {
         List<DimValueDO> dimValueDOList = new ArrayList<>();
@@ -52,7 +56,7 @@ public class DictMetaHelper {
                 dimValueDOList = generateDimValueInfoByModel(modelIds);
                 break;
             case OFFLINE_FULL:
-                List<ModelSchema> modelSchemaDescList = semanticLayer.getModelSchema();
+                List<ModelSchema> modelSchemaDescList = semanticInterpreter.getModelSchema();
                 if (CollectionUtils.isEmpty(modelSchemaDescList)) {
                     break;
                 }
@@ -83,7 +87,7 @@ public class DictMetaHelper {
             return dimValueDOList;
         }
 
-        List<ModelSchema> modelSchemaDescList = semanticLayer.getModelSchema();
+        List<ModelSchema> modelSchemaDescList = semanticInterpreter.getModelSchema();
         if (CollectionUtils.isEmpty(modelSchemaDescList)) {
             return dimValueDOList;
         }
@@ -112,7 +116,7 @@ public class DictMetaHelper {
 
     private List<DimValueDO> generateDimValueInfoByModel(Set<Long> modelIds) {
         List<DimValueDO> dimValueDOList = new ArrayList<>();
-        List<ModelSchema> modelSchemaDescList = semanticLayer.getModelSchema(new ArrayList<>(modelIds));
+        List<ModelSchema> modelSchemaDescList = semanticInterpreter.getModelSchema(new ArrayList<>(modelIds));
         if (CollectionUtils.isEmpty(modelSchemaDescList)) {
             return dimValueDOList;
         }
@@ -134,14 +138,21 @@ public class DictMetaHelper {
 
             ChatDefaultRichConfigResp chatDefaultConfig =
                     chaConfigRichDesc.getChatAggRichConfig().getChatDefaultConfig();
+
+            KnowledgeAdvancedConfig globalKnowledgeConfigAgg = chaConfigRichDesc.getChatAggRichConfig()
+                    .getGlobalKnowledgeConfig();
             List<KnowledgeInfoReq> knowledgeAggInfo =
                     chaConfigRichDesc.getChatAggRichConfig().getKnowledgeInfos();
 
+            KnowledgeAdvancedConfig globalKnowledgeConfigDetail = chaConfigRichDesc.getChatDetailRichConfig()
+                    .getGlobalKnowledgeConfig();
             List<KnowledgeInfoReq> knowledgeDetailInfo =
                     chaConfigRichDesc.getChatDetailRichConfig().getKnowledgeInfos();
 
-            fillKnowledgeDimValue(knowledgeDetailInfo, chatDefaultConfig, dimValueDOList, dimIdAndDescPair, modelId);
-            fillKnowledgeDimValue(knowledgeAggInfo, chatDefaultConfig, dimValueDOList, dimIdAndDescPair, modelId);
+            fillKnowledgeDimValue(knowledgeDetailInfo, chatDefaultConfig, dimValueDOList, dimIdAndDescPair,
+                    modelId, globalKnowledgeConfigDetail);
+            fillKnowledgeDimValue(knowledgeAggInfo, chatDefaultConfig, dimValueDOList, dimIdAndDescPair,
+                    modelId, globalKnowledgeConfigAgg);
 
 
         }
@@ -150,7 +161,8 @@ public class DictMetaHelper {
     private void fillKnowledgeDimValue(List<KnowledgeInfoReq> knowledgeInfos,
                                        ChatDefaultRichConfigResp chatDefaultConfig,
                                        List<DimValueDO> dimValueDOList,
-                                       Map<Long, SchemaElement> dimIdAndDescPair, Long modelId) {
+                                       Map<Long, SchemaElement> dimIdAndDescPair, Long modelId,
+                                       KnowledgeAdvancedConfig globalKnowledgeConfigDetail) {
         if (!CollectionUtils.isEmpty(knowledgeInfos)) {
             List<Dim4Dict> dimensions = new ArrayList<>();
             List<DefaultMetric> defaultMetricDescList = new ArrayList<>();
@@ -159,36 +171,42 @@ public class DictMetaHelper {
                             && !CollectionUtils.isEmpty(dimIdAndDescPair)
                             && dimIdAndDescPair.containsKey(knowledgeInfo.getItemId()))
                     .forEach(knowledgeInfo -> {
-                        if (dimIdAndDescPair.containsKey(knowledgeInfo.getItemId())) {
-                            SchemaElement dimensionDesc = dimIdAndDescPair.get(knowledgeInfo.getItemId());
-
-                            //default cnt
-                            if (Objects.isNull(chatDefaultConfig)
-                                    || CollectionUtils.isEmpty(chatDefaultConfig.getMetrics())) {
-                                String datasourceBizName = dimensionDesc.getBizName();
-                                if (Strings.isNotEmpty(datasourceBizName)) {
-                                    String internalMetricName =
-                                            datasourceBizName + UNDERLINE + internalMetricNameSuffix;
-                                    defaultMetricDescList.add(new DefaultMetric(internalMetricName, 2, DAY));
-                                }
-                            } else {
-                                SchemaElement schemaItem = chatDefaultConfig.getMetrics().get(0);
-                                defaultMetricDescList.add(new DefaultMetric(schemaItem.getBizName(),
-                                        chatDefaultConfig.getUnit(), chatDefaultConfig.getPeriod()));
+                        SchemaElement dimensionDesc = dimIdAndDescPair.get(knowledgeInfo.getItemId());
 
+                        //default cnt
+                        if (Objects.isNull(chatDefaultConfig)
+                                || CollectionUtils.isEmpty(chatDefaultConfig.getMetrics())) {
+                            Long dimId = dimensionDesc.getId();
+                            if (Objects.nonNull(dimId)) {
+                                String datasourceBizName = queryDataSourceByDimId(dimId);
+                                String internalMetricName =
+                                        datasourceBizName + UNDERLINE + internalMetricNameSuffix;
+                                defaultMetricDescList.add(new DefaultMetric(internalMetricName,
+                                        internalMetricDays, DAY));
                             }
+                        } else {
+                            SchemaElement schemaItem = chatDefaultConfig.getMetrics().get(0);
+                            defaultMetricDescList.add(new DefaultMetric(schemaItem.getBizName(),
+                                    chatDefaultConfig.getUnit(), chatDefaultConfig.getPeriod()));
 
-                            String bizName = dimensionDesc.getBizName();
-                            Dim4Dict dim4Dict = new Dim4Dict();
-                            dim4Dict.setDimId(knowledgeInfo.getItemId());
-                            dim4Dict.setBizName(bizName);
-                            if (Objects.nonNull(knowledgeInfo.getKnowledgeAdvancedConfig())) {
-                                KnowledgeAdvancedConfig knowledgeAdvancedConfig
-                                        = knowledgeInfo.getKnowledgeAdvancedConfig();
-                                BeanUtils.copyProperties(knowledgeAdvancedConfig, dim4Dict);
-                            }
-                            dimensions.add(dim4Dict);
                         }
+
+                        String bizName = dimensionDesc.getBizName();
+                        Dim4Dict dim4Dict = new Dim4Dict();
+                        dim4Dict.setDimId(knowledgeInfo.getItemId());
+                        dim4Dict.setBizName(bizName);
+                        if (Objects.nonNull(knowledgeInfo.getKnowledgeAdvancedConfig())) {
+                            KnowledgeAdvancedConfig knowledgeAdvancedConfig
+                                    = knowledgeInfo.getKnowledgeAdvancedConfig();
+                            BeanUtils.copyProperties(knowledgeAdvancedConfig, dim4Dict);
+
+                            if (Objects.nonNull(globalKnowledgeConfigDetail)
+                                    && !CollectionUtils.isEmpty(globalKnowledgeConfigDetail.getRuleList())) {
+                                dim4Dict.getRuleList().addAll(globalKnowledgeConfigDetail.getRuleList());
+                            }
+                        }
+                        dimensions.add(dim4Dict);
+
                     });
 
             if (!CollectionUtils.isEmpty(dimensions)) {
@@ -200,4 +218,15 @@ public class DictMetaHelper {
             }
         }
     }
+
+    private String queryDataSourceByDimId(Long id) {
+        PageDimensionReq pageDimensionCmd = new PageDimensionReq();
+        pageDimensionCmd.setId(id.toString());
+        PageInfo<DimensionResp> dimensionPage = semanticInterpreter.getDimensionPage(pageDimensionCmd);
+        if (Objects.nonNull(dimensionPage) && !CollectionUtils.isEmpty(dimensionPage.getList())) {
+            List<DimensionResp> list = dimensionPage.getList();
+            return list.get(0).getDatasourceBizName();
+        }
+        return "";
+    }
 }

chat/core/src/main/java/com/tencent/supersonic/chat/utils/DictQueryHelper.java

@@ -1,12 +1,7 @@
 package com.tencent.supersonic.chat.utils;
 
-import static com.tencent.supersonic.common.pojo.Constants.AND_UPPER;
-import static com.tencent.supersonic.common.pojo.Constants.APOSTROPHE;
-import static com.tencent.supersonic.common.pojo.Constants.COMMA;
-import static com.tencent.supersonic.common.pojo.Constants.UNDERLINE_DOUBLE;
-
 import com.tencent.supersonic.auth.api.authentication.pojo.User;
-import com.tencent.supersonic.chat.api.component.SemanticLayer;
+import com.tencent.supersonic.chat.api.component.SemanticInterpreter;
 import com.tencent.supersonic.chat.config.DefaultMetric;
 import com.tencent.supersonic.chat.config.Dim4Dict;
 import com.tencent.supersonic.common.pojo.QueryColumn;
@@ -34,29 +29,38 @@ import org.springframework.beans.factory.annotation.Value;
 import org.springframework.stereotype.Component;
 import org.springframework.util.CollectionUtils;
 
+import static com.tencent.supersonic.common.pojo.Constants.SPACE;
+import static com.tencent.supersonic.common.pojo.Constants.AND_UPPER;
+import static com.tencent.supersonic.common.pojo.Constants.COMMA;
+import static com.tencent.supersonic.common.pojo.Constants.APOSTROPHE;
+import static com.tencent.supersonic.common.pojo.Constants.UNDERLINE_DOUBLE;
+
 @Slf4j
 @Component
 public class DictQueryHelper {
 
     private static final Long MAX_FREQUENCY = 99999999L;
-    private SemanticLayer semanticLayer = ComponentFactory.getSemanticLayer();
+    private SemanticInterpreter semanticInterpreter = ComponentFactory.getSemanticLayer();
     @Value("${dimension.multi.value.split:#}")
     private String dimMultiValueSplit;
     @Value("${dimension.value.show:50}")
     private Integer printDataShow;
     @Value("${dimension.max.limit:3000000}")
     private Long dimMaxLimit;
+    @Value("${dimension.white.weight:60000000}")
+    private Long dimensionWhiteWeight;
 
     public List<String> fetchDimValueSingle(Long modelId, DefaultMetric defaultMetricDesc, Dim4Dict dim4Dict,
                                             User user) {
         List<String> data = new ArrayList<>();
         QueryStructReq queryStructCmd = generateQueryStructCmd(modelId, defaultMetricDesc, dim4Dict);
         try {
-            QueryResultWithSchemaResp queryResultWithColumns = semanticLayer.queryByStruct(queryStructCmd, user);
+            QueryResultWithSchemaResp queryResultWithColumns = semanticInterpreter.queryByStruct(queryStructCmd, user);
+            log.info("fetchDimValueSingle sql:{}", queryResultWithColumns.getSql());
             String nature = String.format("_%d_%d", modelId, dim4Dict.getDimId());
             String dimNameRewrite = rewriteDimName(queryResultWithColumns.getColumns(), dim4Dict.getBizName());
             data = generateFileData(queryResultWithColumns.getResultList(), nature, dimNameRewrite,
-                    defaultMetricDesc.getBizName());
+                    defaultMetricDesc.getBizName(), dim4Dict);
             if (!CollectionUtils.isEmpty(data)) {
                 int size = (data.size() > printDataShow) ? printDataShow : data.size();
                 log.info("data:{}", data.subList(0, size - 1));
@@ -91,7 +95,7 @@ public class DictQueryHelper {
     }
 
     private List<String> generateFileData(List<Map<String, Object>> resultList, String nature, String dimName,
-                                          String metricName) {
+                                          String metricName, Dim4Dict dim4Dict) {
         List<String> data = new ArrayList<>();
         if (CollectionUtils.isEmpty(resultList)) {
             return data;
@@ -111,17 +115,26 @@ public class DictQueryHelper {
             }
 
         }
-        constructDataLines(valueAndFrequencyPair, nature, data);
+        constructDataLines(valueAndFrequencyPair, nature, data, dim4Dict);
         return data;
     }
 
-    private void constructDataLines(Map<String, Long> valueAndFrequencyPair, String nature, List<String> data) {
+    private void constructDataLines(Map<String, Long> valueAndFrequencyPair, String nature,
+                                    List<String> data, Dim4Dict dim4Dict) {
         valueAndFrequencyPair.forEach((dimValue, metric) -> {
             if (metric > MAX_FREQUENCY) {
                 metric = MAX_FREQUENCY;
             }
+            if (Strings.isNotEmpty(dimValue) && dimValue.contains(SPACE)) {
+                dimValue = dimValue.replace(SPACE, "#");
+            }
             data.add(String.format("%s %s %s", dimValue, nature, metric));
         });
+
+        if (Objects.nonNull(dim4Dict) && !CollectionUtils.isEmpty(dim4Dict.getWhiteList())) {
+            dim4Dict.getWhiteList().stream()
+                    .forEach(white -> data.add(String.format("%s %s %s", white, nature, dimensionWhiteWeight)));
+        }
     }
 
     private void mergeMultivaluedValue(Map<String, Long> valueAndFrequencyPair, String dimValue, Long metric) {
@@ -185,7 +198,7 @@ public class DictQueryHelper {
         if (Objects.isNull(dim4Dict)) {
             return "";
         }
-        StringJoiner joiner = new StringJoiner(AND_UPPER);
+        StringJoiner joiner = new StringJoiner(SPACE + AND_UPPER + SPACE);
 
         String dimName = dim4Dict.getBizName();
         if (!CollectionUtils.isEmpty(dim4Dict.getBlackList())) {

chat/core/src/main/java/com/tencent/supersonic/chat/utils/SolvedQueryManager.java

@@ -0,0 +1,158 @@
+package com.tencent.supersonic.chat.utils;
+
+import com.alibaba.fastjson.JSONObject;
+import com.alibaba.fastjson.serializer.SerializerFeature;
+import com.google.common.collect.Lists;
+import com.tencent.supersonic.chat.api.pojo.request.SolvedQueryReq;
+import com.tencent.supersonic.chat.api.pojo.response.SolvedQueryRecallResp;
+import com.tencent.supersonic.chat.parser.plugin.embedding.EmbeddingConfig;
+import com.tencent.supersonic.chat.parser.plugin.embedding.EmbeddingResp;
+import com.tencent.supersonic.chat.parser.plugin.embedding.RecallRetrieval;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.logging.log4j.util.Strings;
+import org.springframework.core.ParameterizedTypeReference;
+import org.springframework.http.HttpEntity;
+import org.springframework.http.HttpHeaders;
+import org.springframework.http.HttpMethod;
+import org.springframework.http.MediaType;
+import org.springframework.http.ResponseEntity;
+import org.springframework.stereotype.Component;
+import org.springframework.web.client.RestTemplate;
+import org.springframework.web.util.UriComponentsBuilder;
+import java.net.URI;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@Slf4j
+@Component
+public class SolvedQueryManager {
+
+    private EmbeddingConfig embeddingConfig;
+
+    public SolvedQueryManager(EmbeddingConfig embeddingConfig) {
+        this.embeddingConfig = embeddingConfig;
+    }
+
+    public void saveSolvedQuery(SolvedQueryReq solvedQueryReq) {
+        if (StringUtils.isBlank(embeddingConfig.getUrl())) {
+            return;
+        }
+        String queryText = solvedQueryReq.getQueryText();
+        try {
+            String uniqueId = generateUniqueId(solvedQueryReq.getQueryId(), solvedQueryReq.getParseId());
+            Map<String, Object> requestMap = new HashMap<>();
+            requestMap.put("query", queryText);
+            requestMap.put("query_id", uniqueId);
+            Map<String, Object> metaData = new HashMap<>();
+            metaData.put("modelId", String.valueOf(solvedQueryReq.getModelId()));
+            metaData.put("agentId", String.valueOf(solvedQueryReq.getAgentId()));
+            requestMap.put("metadata", metaData);
+            doRequest(embeddingConfig.getSolvedQueryAddPath(),
+                    JSONObject.toJSONString(Lists.newArrayList(requestMap)));
+        } catch (Exception e) {
+            log.warn("save history question to embedding failed, queryText:{}", queryText, e);
+        }
+    }
+
+    public List<SolvedQueryRecallResp> recallSolvedQuery(String queryText, Integer agentId) {
+        if (StringUtils.isBlank(embeddingConfig.getUrl())) {
+            return Lists.newArrayList();
+        }
+        List<SolvedQueryRecallResp> solvedQueryRecallResps = Lists.newArrayList();
+        try {
+            String url = embeddingConfig.getUrl() + embeddingConfig.getSolvedQueryRecallPath() + "?n_results="
+                    + embeddingConfig.getSolvedQueryResultNum();
+            HttpHeaders headers = new HttpHeaders();
+            headers.setContentType(MediaType.APPLICATION_JSON);
+            headers.setLocation(URI.create(url));
+            URI requestUrl = UriComponentsBuilder
+                    .fromHttpUrl(url).build().encode().toUri();
+            Map<String, Object> map = new HashMap<>();
+            map.put("queryTextsList", Lists.newArrayList(queryText));
+            Map<String, Object> filterCondition = new HashMap<>();
+            filterCondition.put("agentId", String.valueOf(agentId));
+            map.put("filterCondition", filterCondition);
+            String jsonBody = JSONObject.toJSONString(map, SerializerFeature.WriteMapNullValue);
+            HttpEntity<String> entity = new HttpEntity<>(jsonBody, headers);
+            log.info("[embedding] request body:{}, url:{}", jsonBody, url);
+            RestTemplate restTemplate = new RestTemplate();
+            ResponseEntity<List<EmbeddingResp>> embeddingResponseEntity =
+                    restTemplate.exchange(requestUrl, HttpMethod.POST, entity,
+                            new ParameterizedTypeReference<List<EmbeddingResp>>() {
+                            });
+            log.info("[embedding] recognize result body:{}", embeddingResponseEntity);
+            List<EmbeddingResp> embeddingResps = embeddingResponseEntity.getBody();
+            Set<String> querySet = new HashSet<>();
+            if (CollectionUtils.isNotEmpty(embeddingResps)) {
+                for (EmbeddingResp embeddingResp : embeddingResps) {
+                    List<RecallRetrieval> embeddingRetrievals = embeddingResp.getRetrieval();
+                    for (RecallRetrieval embeddingRetrieval : embeddingRetrievals) {
+                        if (queryText.equalsIgnoreCase(embeddingRetrieval.getQuery())) {
+                            continue;
+                        }
+                        if (querySet.contains(embeddingRetrieval.getQuery())) {
+                            continue;
+                        }
+                        String id = embeddingRetrieval.getId();
+                        SolvedQueryRecallResp solvedQueryRecallResp = SolvedQueryRecallResp.builder()
+                                .queryText(embeddingRetrieval.getQuery())
+                                .queryId(getQueryId(id)).parseId(getParseId(id))
+                                .build();
+                        solvedQueryRecallResps.add(solvedQueryRecallResp);
+                        querySet.add(embeddingRetrieval.getQuery());
+                    }
+                }
+            }
+        } catch (Exception e) {
+            log.warn("recall similar solved query failed, queryText:{}", queryText);
+        }
+        return solvedQueryRecallResps;
+    }
+
+    private String generateUniqueId(Long queryId, Integer parseId) {
+        String uniqueId = queryId + String.valueOf(parseId);
+        if (parseId < 10) {
+            uniqueId = queryId + String.format("0%s", parseId);
+        }
+        return uniqueId;
+    }
+
+    private Long getQueryId(String uniqueId) {
+        return Long.parseLong(uniqueId) / 100;
+    }
+
+    private Integer getParseId(String uniqueId) {
+        return Integer.parseInt(uniqueId) % 100;
+    }
+
+    private ResponseEntity<String> doRequest(String path, String jsonBody) {
+        if (Strings.isEmpty(embeddingConfig.getUrl())) {
+            return ResponseEntity.of(Optional.empty());
+        }
+        String url = embeddingConfig.getUrl() + path;
+        try {
+            HttpHeaders headers = new HttpHeaders();
+            headers.setContentType(MediaType.APPLICATION_JSON);
+            headers.setLocation(URI.create(url));
+            URI requestUrl = UriComponentsBuilder
+                    .fromHttpUrl(url).build().encode().toUri();
+            HttpEntity<String> entity = new HttpEntity<>(jsonBody, headers);
+            log.info("[embedding] request body :{}, url:{}", jsonBody, url);
+            RestTemplate restTemplate = new RestTemplate();
+            ResponseEntity<String> responseEntity = restTemplate.exchange(requestUrl,
+                    HttpMethod.POST, entity, new ParameterizedTypeReference<String>() {});
+            log.info("[embedding] result body:{}", responseEntity);
+            return responseEntity;
+        } catch (Exception e) {
+            log.warn("connect to embedding service failed, url:{}", url);
+        }
+        return ResponseEntity.of(Optional.empty());
+    }
+
+}