(improvement)(project) Upgrade to version 0.9.2. (#1072)

Co-authored-by: jolunoluo

.gitignore

@@ -8,6 +8,7 @@ log/
 *.bin
 *.log
 *.tar.gz
+*.zip
 *.lib
 assembly/runtime/*
 **/dist/
@@ -17,4 +18,5 @@ assembly/runtime/*
 **/.flattened-pom.xml
 chm_db/
 __pycache__/
-/dict
+/dict
+assembly/build/*-SNAPSHOT

CHANGELOG.md

@@ -4,6 +4,20 @@
 - "Breaking Changes" describes any changes that may break existing functionality or cause
   compatibility issues with previous versions.
 
+## SuperSonic [0.9.2] - 2024-06-01
+
+### Added
+- support multiple rounds of dialogue
+- add term configuration and identification to help LLM learn private domain knowledge
+- support configuring LLM parameters in the agent
+- metric market supports searching in natural language
+
+### Updated
+- introducing WorkFlow, Mapper, Parser, and Corrector support jump execution
+- Introducing the concept of Model-Set to simplify Domain management
+- overall optimization and upgrade of system pages
+- optimize startup script
+
 ## SuperSonic [0.9.0] - 2024-04-03
 
 ### Added

README.md

@@ -32,9 +32,9 @@ With these ideas in mind, we develop SuperSonic as a practical reference impleme
 
 - Built-in Chat BI interface for *business users* to enter natural language queries
 - Built-in Headless BI interface for *analytics engineers* to build semantic data models
-- Built-in rule-based semantic parser to improve efficiency in certain scenarios
-- Support input auto-completion as well as query recommendation
-- Support four-level permission control: domain-level, model-level, column-level and row-level
+- Built-in rule-based semantic parser to improve efficiency in certain scenarios (e.g. demonstration, integration testing)
+- Built-in support for input auto-completion, multi-turn conversation as well as post-query recommendation
+- Built-in support for three-level data access control: dataset-level, column-level and row-level
 
 ## Extensible Components
 
@@ -46,11 +46,11 @@ The high-level architecture and main process flow is as follows:
 
 - **Schema Mapper:** identifies references to schema elements(metrics/dimensions/entities/values) in user queries. It matches the query text against the knowledge base.
 
-- **Semantic Parser:** understands user queries and extracts semantic information. It consists of a combination of rule-based and model-based parsers, each of which deals with specific scenarios.
+- **Semantic Parser:** understands user queries and generates semantic query statement. It consists of a combination of rule-based and model-based parsers, each of which deals with specific scenarios.
 
-- **Semantic Corrector:** checks validity of extracted semantic information and performs correction and optimization if needed.
+- **Semantic Corrector:** checks validity of semantic query statement and performs correction and optimization if needed.
 
-- **Semantic Interpreter:** performs execution according to extracted semantic information. It generates SQL statements and executes them against physical data models.
+- **Semantic Translator:** converts semantic query statement into SQL statement that can be executed against physical data models.
 
 - **Chat Plugin:** extends functionality with third-party tools. The LLM is going to select the most suitable one, given all configured plugins with function description and sample questions.
 

README_CN.md

@@ -28,9 +28,9 @@
 
 - 内置Chat BI界面以便*业务用户*输入数据查询。
 - 内置Headless BI界面以便*分析工程师*构建语义模型。
-- 内置基于规则的语义解析器，在特定场景可以提升运行效率。
-- 支持文本输入的联想和查询问题的推荐。
-- 支持四级权限控制：主题域级、模型级、列级、行级。
+- 内置基于规则的语义解析器，在特定场景（比如DEMO演示、集成测试）可以提升推理效率。
+- 支持文本输入联想、多轮对话、查询后问题推荐等高级特征。
+- 支持三级权限控制：数据集级、列级、行级。
 
 ## 易于扩展的组件
 
@@ -42,11 +42,11 @@ SuperSonic的整体架构和主流程如下图所示：
 
 - **模式映射器(Schema Mapper)：** 将自然语言文本在知识库中进行匹配，为后续的语义解析提供相关信息。
 
-- **语义解析器(Semantic Parser)：** 理解用户查询并抽取语义信息，其由一组基于规则和基于模型的解析器组成，每个解析器可应对不同的特定场景。
+- **语义解析器(Semantic Parser)：** 理解用户查询并抽取语义信息，生成语义查询语句S2SQL。
 
-- **语义修正器(Semantic Corrector)：** 检查语义信息的合法性，对不合法的信息做修正和优化处理。
+- **语义修正器(Semantic Corrector)：** 检查语义查询语句的合法性，对不合法的信息做修正和优化处理。
 
-- **语义解释器(Semantic Interpreter)：** 根据语义信息生成物理SQL执行查询。
+- **语义翻译器(Semantic Translator)：** 将语义查询语句翻译成可在物理数据模型上执行的SQL语句。
 
 - **问答插件(Chat Plugin)：** 通过第三方工具扩展功能。给定所有配置的插件及其功能描述和示例问题，大语言模型将选择最合适的插件。
 

assembly/bin/supersonic-build.bat

@@ -1,72 +1,98 @@
 @echo off
-setlocal
+setlocal enabledelayedexpansion
 chcp 65001
-set "sbinDir=%~dp0"
-set "baseDir=%~dp0.."
-set "buildDir=%baseDir%\build"
-set "runtimeDir=%baseDir%\..\runtime"
-set "pip_path=pip3"
+
+call supersonic-common.bat %*
+
 set "service=%~1"
 
-
-rem 1. build backend java modules
-del /q "%buildDir%\*.tar.gz" 2>NUL
-call mvn -f "%baseDir%\..\pom.xml" clean package -DskipTests
-
-IF ERRORLEVEL 1 (
-    ECHO Failed to build backend Java modules.
-    EXIT /B 1
+cd %projectDir%
+if "%service%"=="" (
+    set service=%standalone_service%
 )
 
-rem 2. move package to build
-echo f|xcopy "%baseDir%\..\launchers\standalone\target\*.tar.gz" "%buildDir%\supersonic-standalone.tar.gz"
+call mvn help:evaluate -Dexpression=project.version > temp.txt
+for /f "delims=" %%i in (temp.txt) do (
+    set line=%%i
+    if not "!line:~0,1!"=="[" (
+        set MVN_VERSION=!line!
+    )
+)
+del temp.txt
+cd %baseDir%
 
-rem 3. build frontend webapp
-cd "%baseDir%\..\webapp"
-call start-fe-prod.bat
-copy /y "%baseDir%\..\webapp\supersonic-webapp.tar.gz" "%buildDir%\"
 
-IF ERRORLEVEL 1 (
-    ECHO Failed to build frontend webapp.
-    EXIT /B 1
+if "%service%"=="%pyllm_service%" (
+   echo start installing python modules required by supersonic-pyllm: %pip_path%
+   %pip_path% install -r %projectDir%\headless\python\requirements.txt"
+   echo install python modules success
+   goto :EOF
+) else if "%service%"=="webapp" (
+   call :buildWebapp
+   tar xvf supersonic-webapp.tar.gz
+   move /y supersonic-webapp webapp
+   move /y webapp %projectDir%\launchers\%STANDALONE_SERVICE%\target\classes
+   goto :EOF
+) else (
+   call :buildJavaService
+   call :buildWebapp
+   call :packageRelease
+   goto :EOF
 )
 
-rem 4. copy webapp to java classpath
-cd "%buildDir%"
-tar -zxvf supersonic-webapp.tar.gz
-move supersonic-webapp webapp
-move webapp ..\..\launchers\standalone\target\classes
 
-rem 5. build backend python modules
-if "%service%"=="pyllm" (
-    echo "start installing python modules with pip: ${pip_path}"
-    set requirementPath="%baseDir%/../headless/python/requirements.txt"
-    %pip_path% install -r %requirementPath%
-    echo "install python modules success"
-)
+:buildJavaService
+   set "model_name=%service%"
+   echo "starting building supersonic-%model_name% service"
+   call mvn -f %projectDir%\launchers\%model_name% clean package -DskipTests
+   IF ERRORLEVEL 1 (
+      ECHO Failed to build backend Java modules.
+      EXIT /B 1
+   )
+   copy /y %projectDir%\launchers\%model_name%\target\*.tar.gz %buildDir%\
+   echo "finished building supersonic-%model_name% service"
+   goto :EOF
 
-call :BUILD_RUNTIME
 
-:BUILD_RUNTIME
-  rem 6. reset runtime
-  IF EXIST "%runtimeDir%" (
-      echo begin to delete dir : %runtimeDir%
-      rd /s /q "%runtimeDir%"
-  ) ELSE (
-      echo %runtimeDir% does not exist, create directly
-  )
-  mkdir "%runtimeDir%"
-  tar -zxvf "%buildDir%\supersonic-standalone.tar.gz" -C "%runtimeDir%"
-  for /d %%f in ("%runtimeDir%\launchers-standalone-*") do (
-      move "%%f" "%runtimeDir%\supersonic-standalone"
-  )
+:buildWebapp
+   echo "starting building supersonic webapp"
+   cd %projectDir%\webapp
+   call start-fe-prod.bat
+   copy /y supersonic-webapp.tar.gz %buildDir%\
+   rem check build result
+   IF ERRORLEVEL 1 (
+        ECHO Failed to build frontend webapp.
+        EXIT /B 1
+   )
+   echo "finished building supersonic webapp"
+   goto :EOF
 
-  rem 7. copy webapp to runtime
-  tar -zxvf "%buildDir%\supersonic-webapp.tar.gz" -C "%buildDir%"
-  if not exist "%runtimeDir%\supersonic-standalone\webapp" mkdir "%runtimeDir%\supersonic-standalone\webapp"
-  xcopy /s /e /h /y "%buildDir%\supersonic-webapp\*" "%runtimeDir%\supersonic-standalone\webapp"
-  if not exist "%runtimeDir%\supersonic-standalone\conf\webapp" mkdir "%runtimeDir%\supersonic-standalone\conf\webapp"
-  xcopy /s /e /h /y "%runtimeDir%\supersonic-standalone\webapp\*" "%runtimeDir%\supersonic-standalone\conf\webapp"
-  rd /s /q "%buildDir%\supersonic-webapp"
+
+:packageRelease
+   set "model_name=%service%"
+   set "release_dir=supersonic-%model_name%-%MVN_VERSION%"
+   set "service_name=launchers-%model_name%-%MVN_VERSION%"
+   echo "starting packaging supersonic release"
+   cd %buildDir%
+   if exist %release_dir% rmdir /s /q %release_dir%
+   if exist %release_dir%.zip del %release_dir%.zip
+   mkdir %release_dir%
+   rem package webapp
+   tar xvf supersonic-webapp.tar.gz
+   move /y supersonic-webapp webapp
+   echo {"env": ""} > webapp\supersonic.config.json
+   move /y webapp %release_dir%
+   rem package java service
+   tar xvf %service_name%-bin.tar.gz
+   for /d %%D in ("%service_name%\*") do (
+       move "%%D" "%release_dir%"
+   )
+   rem generate zip file
+   powershell Compress-Archive -Path %release_dir% -DestinationPath %release_dir%.zip
+   del %service_name%-bin.tar.gz
+   del supersonic-webapp.tar.gz
+   rmdir /s /q %service_name%
+   echo "finished packaging supersonic release"
+   goto :EOF
 
 endlocal

assembly/bin/supersonic-build.sh

@@ -1,58 +1,80 @@
 #!/usr/bin/env bash
 
-set -x
 sbinDir=$(cd "$(dirname "$0")"; pwd)
 chmod +x $sbinDir/supersonic-common.sh
 source $sbinDir/supersonic-common.sh
+cd $projectDir
+MVN_VERSION=$(mvn help:evaluate -Dexpression=project.version | grep -e '^[^\[]')
+
 cd $baseDir
-
 service=$1
-#1. build backend java modules
-rm -fr ${buildDir}/*.tar.gz
-rm -fr dist
-set +x
-mvn -f $baseDir/../ clean package -DskipTests
-# check build result
-if [ $? -ne 0 ]; then
-    echo "Failed to build backend Java modules."
-    exit 1
+if [ -z "$service"  ]; then
+  service=${STANDALONE_SERVICE}
 fi
 
-#2. move package to build
-cp $baseDir/../launchers/headless/target/*.tar.gz ${buildDir}/supersonic-headless.tar.gz
-cp $baseDir/../launchers/chat/target/*.tar.gz ${buildDir}/supersonic-chat.tar.gz
-cp $baseDir/../launchers/standalone/target/*.tar.gz ${buildDir}/supersonic-standalone.tar.gz
+function buildJavaService {
+  model_name=$1
+  echo "starting building supersonic-${model_name} service"
+  mvn -f $projectDir clean package -DskipTests
+  if [ $? -ne 0 ]; then
+      echo "Failed to build backend Java modules."
+      exit 1
+  fi
+  cp $projectDir/launchers/${model_name}/target/*.tar.gz ${buildDir}/
+  echo "finished building supersonic-${model_name} service"
+}
 
-#3. build frontend webapp
-chmod +x $baseDir/../webapp/start-fe-prod.sh
-cd ../webapp
-sh ./start-fe-prod.sh
-cp -fr  ./supersonic-webapp.tar.gz ${buildDir}/
+function buildWebapp {
+  echo "starting building supersonic webapp"
+  chmod +x $projectDir/webapp/start-fe-prod.sh
+  cd $projectDir/webapp
+  sh ./start-fe-prod.sh
+  cp -fr  ./supersonic-webapp.tar.gz ${buildDir}/
+  # check build result
+  if [ $? -ne 0 ]; then
+      echo "Failed to build frontend webapp."
+      exit 1
+  fi
+  echo "finished building supersonic webapp"
+}
 
-# check build result
-if [ $? -ne 0 ]; then
-    echo "Failed to build frontend webapp."
-    exit 1
-fi
-#4. copy webapp to java classpath
-cd $buildDir
-tar xvf supersonic-webapp.tar.gz
-mv supersonic-webapp webapp
-cp -fr webapp ../../launchers/headless/target/classes
-cp -fr webapp ../../launchers/chat/target/classes
-cp -fr webapp ../../launchers/standalone/target/classes
-rm -fr  ${buildDir}/webapp
+function packageRelease {
+  model_name=$1
+  release_dir=supersonic-${model_name}-${MVN_VERSION}
+  service_name=launchers-${model_name}-${MVN_VERSION}
+  echo "starting packaging supersonic release"
+  cd $buildDir
+  mkdir $release_dir
+  # package webapp
+  tar xvf supersonic-webapp.tar.gz
+  mv supersonic-webapp webapp
+  json='{"env": "''"}'
+  echo $json > webapp/supersonic.config.json
+  mv webapp $release_dir/
+  # package java service
+  tar xvf $service_name-bin.tar.gz
+  mv $service_name/* $release_dir/
+  # generate zip file
+  zip -r $release_dir.zip $release_dir
+  # delete intermediate files
+  rm supersonic-webapp.tar.gz $service_name-bin.tar.gz
+  rm -rf webapp $service_name $release_dir
+  echo "finished packaging supersonic release"
+}
 
-#5. build backend python modules
-if [ "$service" == "pyllm" ]; then
-  echo "start installing python modules with pip: ${pip_path}"
-  requirementPath=$baseDir/../headless/python/requirements.txt
+#1. build backend services
+if [ "$service" == $PYLLM_SERVICE ]; then
+  echo "start installing python modules required by supersonic-pyllm: ${pip_path}"
+  requirementPath=$projectDir/headless/python/requirements.txt
   ${pip_path} install -r ${requirementPath}
   echo "install python modules success"
-fi
-
-#6. reset runtime
-rm -fr $runtimeDir/supersonic*
-moveAllToRuntime
-setEnvToWeb chat
-setEnvToWeb headless
+elif [ "$service" == "webapp" ]; then
+  buildWebapp
+  target_path=$projectDir/launchers/$STANDALONE_SERVICE/target/classes
+  tar xvf $projectDir/webapp/supersonic-webapp.tar.gz -C $target_path
+  mv $target_path/supersonic-webapp $target_path/webapp
+else
+  buildJavaService $service
+  buildWebapp
+  packageRelease $service
+fi

assembly/bin/supersonic-common.bat

@@ -0,0 +1,9 @@
+set "sbinDir=%~dp0"
+set "baseDir=%~dp0.."
+set "buildDir=%baseDir%\build"
+set "main_class=com.tencent.supersonic.StandaloneLauncher"
+set "python_path=python"
+set "pip_path=pip3"
+set "standalone_service=standalone"
+set "pyllm_service=pyllm"
+set "projectDir=%baseDir%\.."

assembly/bin/supersonic-common.sh

@@ -6,105 +6,19 @@ pip_path=${PIP_PATH:-"pip3"}
 
 sbinDir=$(cd "$(dirname "$0")"; pwd)
 baseDir=$(cd "$sbinDir/.." && pwd -P)
-runtimeDir=$baseDir/../runtime
+runtimeDir=$baseDir/runtime
 buildDir=$baseDir/build
+projectDir=$baseDir/..
 
 readonly CHAT_APP_NAME="supersonic_chat"
 readonly HEADLESS_APP_NAME="supersonic_headless"
 readonly PYLLM_APP_NAME="supersonic_pyllm"
 readonly STANDALONE_APP_NAME="supersonic_standalone"
+
 readonly CHAT_SERVICE="chat"
 readonly HEADLESS_SERVICE="headless"
 readonly PYLLM_SERVICE="pyllm"
 readonly STANDALONE_SERVICE="standalone"
+
 readonly PYLLM_HOST="127.0.0.1"
 readonly PYLLM_PORT="9092"
-
-function setEnvToWeb {
-   model_name=$1
-   json='{"env": "'$model_name'"}'
-   echo $json > ${runtimeDir}/supersonic-${model_name}/webapp/supersonic.config.json
-   echo $json > $baseDir/../launchers/${model_name}/target/classes/webapp/supersonic.config.json
-}
-
-function moveToRuntime {
-  model_name=$1
-  file="${buildDir}/supersonic-${model_name}.tar.gz"
-  if [ -f "$file" ]; then
-    tar -zxvf "$file" -C ${runtimeDir}
-    mv ${runtimeDir}/launchers-${model_name}-* ${runtimeDir}/supersonic-${model_name}
-    mkdir -p ${runtimeDir}/supersonic-${model_name}/webapp
-    cp -fr ${buildDir}/webapp/* ${runtimeDir}/supersonic-${model_name}/webapp
-  else
-    echo "File $file does not exist. Skipping the move to runtime."
-  fi
-}
-
-function moveAllToRuntime {
-  mkdir -p ${runtimeDir}
-  tar xvf  ${buildDir}/supersonic-webapp.tar.gz  -C ${buildDir}
-  mv ${buildDir}/supersonic-webapp ${buildDir}/webapp
-
-  moveToRuntime chat
-  moveToRuntime headless
-  moveToRuntime standalone
-  rm -fr  ${buildDir}/webapp
-}
-
-# run java service
-function runJavaService {
-  javaRunDir=${runtimeDir}/supersonic-${model_name}
-  local_app_name=$1
-  libDir=$javaRunDir/lib
-  confDir=$javaRunDir/conf
-
-  CLASSPATH=""
-  CLASSPATH=$CLASSPATH:$confDir
-
-  for jarPath in $libDir/*.jar; do
-   CLASSPATH=$CLASSPATH:$jarPath
-  done
-
-  export CLASSPATH
-  export LANG="zh_CN.UTF-8"
-
-  cd $javaRunDir
-  if [[ "$JAVA_HOME" == "" ]]; then
-    JAVA_HOME=$(ls /usr/jdk64/jdk* -d 2>/dev/null | xargs | awk '{print "'$local_app_name'"}')
-  fi
-  export PATH=$JAVA_HOME/bin:$PATH
-  command="-Dfile.encoding="UTF-8"  -Duser.language="Zh" -Duser.region="CN" -Duser.timezone="GMT+08" -Dapp_name=${local_app_name} -Xms1024m -Xmx2048m "$main_class
-
-  mkdir -p $javaRunDir/logs
-  if [[ "$is_test" == "true" ]]; then
-    java -Dspring.profiles.active="dev" $command >/dev/null 2>$javaRunDir/logs/error.log &
-  else
-    java  $command $javaRunDir >/dev/null 2>$javaRunDir/logs/error.log &
-  fi
-}
-
-# run python service
-function runPythonService {
-  pythonRunDir=${runtimeDir}/supersonic-${model_name}/pyllm
-  cd $pythonRunDir
-  nohup ${python_path} supersonic_pyllm.py  > $pythonRunDir/pyllm.log  2>&1   &
-  # add health check
-  for i in {1..10}
-  do
-    echo "pyllm health check attempt $i..."
-    response=$(curl -s http://${PYLLM_HOST}:${PYLLM_PORT}/health)
-    echo "pyllm health check response: $response"
-    status_ok="Healthy"
-    if [[ $response == *$status_ok* ]] ; then
-      echo "pyllm Health check passed."
-      break
-    else
-      if [ "$i" -eq 10 ]; then
-        echo "pyllm Health check failed after 10 attempts."
-        echo "May still downloading model files. Please check pyllm.log in runtime directory."
-      fi
-      echo "Retrying after 5 seconds..."
-      sleep 5
-    fi
-  done
-}

assembly/bin/supersonic-daemon.bat

@@ -1,118 +1,102 @@
 @echo off
 setlocal
 chcp 65001
-set "sbinDir=%~dp0"
-set "baseDir=%~dp0.."
-set "runtimeDir=%baseDir%\..\runtime"
-set "buildDir=%baseDir%\build"
-set "main_class=com.tencent.supersonic.StandaloneLauncher"
-set "python_path=python"
-set "pip_path=pip3"
-set "standalone_service=standalone"
-set "pyllm_service=pyllm"
 
-set "javaRunDir=%runtimeDir%\supersonic-standalone"
-set "pythonRunDir=%runtimeDir%\supersonic-standalone\pyllm"
+call supersonic-common.bat %*
+call %sbinDir%/../conf/supersonic-env.bat %*
 
 set "command=%~1"
 set "service=%~2"
-
 if "%service%"=="" (
    set "service=%standalone_service%"
 )
-
+set "model_name=%service%"
 IF "%service%"=="pyllm" (
-  SET "llmProxy=PythonLLMProxy"
+  set "llmProxy=PythonLLMProxy"
+  set "model_name=%standalone_service%"
 )
 
-call :BUILD_RUNTIME
+cd %baseDir%
 
 if "%command%"=="restart" (
-   call :STOP
-   call :START
+   call :stop
+   call :start
    goto :EOF
 ) else if "%command%"=="start" (
-   call :START
+   call :start
    goto :EOF
 ) else if "%command%"=="stop" (
-  call :STOP
-  goto :EOF
+   call :stop
+   goto :EOF
 ) else if "%command%"=="reload" (
-  call :RELOAD_EXAMPLE
-  goto :EOF
+   call :reloadExamples
+   goto :EOF
 ) else (
    echo "Use command {start|stop|restart} to run."
    goto :EOF
 )
 
-:START
-    if "%service%"=="%pyllm_service%" (
-         call :START_PYTHON
-         call :START_JAVA
+
+: start
+   if "%service%"=="%pyllm_service%" (
+         call :runPythonService
+         call :runJavaService
          goto :EOF
-    )
-    call :START_JAVA
-    goto :EOF
-
-:STOP
-    call :STOP_PYTHON
-    call :STOP_JAVA
-    goto :EOF
-
-:START_PYTHON
-   echo 'python service starting, see logs in pyllm/pyllm.log'
-   cd "%pythonRunDir%"
-   start /B %python_path% supersonic_pyllm.py  > %pythonRunDir%\pyllm.log 2>&1
-   timeout /t 10 >nul
-   echo 'python service started'
+   )
+   call :runJavaService
    goto :EOF
 
-:START_JAVA
-  echo 'java service starting, see logs in logs/'
-   cd "%javaRunDir%"
-   if not exist "%runtimeDir%\supersonic-standalone\logs" mkdir "%runtimeDir%\supersonic-standalone\logs"
-   set "libDir=%runtimeDir%\supersonic-standalone\lib"
-   set "confDir=%runtimeDir%\supersonic-standalone\conf"
-   set "webDir=%runtimeDir%\supersonic-standalone\webapp"
-   set "classpath=%confDir%;%webDir%;%libDir%\*"
+
+: stop
+   call :stopPythonService
+   call :stopJavaService
+   goto :EOF
+
+
+: reloadExamples
+   set "pythonRunDir=%baseDir%\pyllm"
+   cd "%pythonRunDir%\sql"
+   start  %python_path% examples_reload_run.py
+   goto :EOF
+
+
+: runJavaService
+   echo 'java service starting, see logs in logs/'
+   set "libDir=%baseDir%\lib"
+   set "confDir=%baseDir%\conf"
+   set "webDir=%baseDir%\webapp"
+   set "logDir=%baseDir%\logs"
+   set "classpath=%baseDir%;%webDir%;%libDir%\*;%confDir%"
    set "java-command=-Dfile.encoding=UTF-8 -Duser.language=Zh -Duser.region=CN -Duser.timezone=GMT+08 -Xms1024m -Xmx2048m -cp %CLASSPATH% %MAIN_CLASS%"
+   if not exist %logDir% mkdir %logDir%
    start /B java %java-command% >nul 2>&1
    timeout /t 10 >nul
    echo 'java service started'
    goto :EOF
 
-:STOP_PYTHON
+
+: runPythonService
+   echo 'python service starting, see logs in pyllm\pyllm.log'
+   set "pythonRunDir=%baseDir%\pyllm"
+   start /B %python_path% %pythonRunDir%\supersonic_pyllm.py  > %pythonRunDir%\pyllm.log 2>&1
+   timeout /t 10 >nul
+   echo 'python service started'
+   goto :EOF
+
+
+: stopPythonService
    for /f "tokens=2" %%i in ('tasklist ^| findstr /i "python"') do (
            taskkill /PID %%i /F
            echo "python service (PID = %%i) is killed."
    )
    goto :EOF
 
-:STOP_JAVA
+
+: stopJavaService
    for /f "tokens=2" %%i in ('tasklist ^| findstr /i "java"') do (
             taskkill /PID %%i /F
             echo "java service (PID = %%i) is killed."
    )
    goto :EOF
 
-:RELOAD_EXAMPLE
-   cd "%runtimeDir%\supersonic-standalone\pyllm\sql"
-   start  %python_path% examples_reload_run.py
-   goto :EOF
-
-:BUILD_RUNTIME
-  rem 6. reset runtime
-  if exist "%runtimeDir%" goto :EOF
-  mkdir "%runtimeDir%"
-  tar -zxvf "%buildDir%\supersonic-standalone.tar.gz" -C "%runtimeDir%"
-  for /d %%f in ("%runtimeDir%\launchers-standalone-*") do (
-      move "%%f" "%runtimeDir%\supersonic-standalone"
-  )
-
-  rem 7. copy webapp to runtime
-  tar -zxvf "%buildDir%\supersonic-webapp.tar.gz" -C "%buildDir%"
-  if not exist "%runtimeDir%\supersonic-standalone\webapp" mkdir "%runtimeDir%\supersonic-standalone\webapp"
-  xcopy /s /e /h /y "%buildDir%\supersonic-webapp\*" "%runtimeDir%\supersonic-standalone\webapp"
-  if not exist "%runtimeDir%\supersonic-standalone\conf\webapp" mkdir "%runtimeDir%\supersonic-standalone\conf\webapp"
-  xcopy /s /e /h /y "%runtimeDir%\supersonic-standalone\webapp\*" "%runtimeDir%\supersonic-standalone\conf\webapp"
-  rd /s /q "%buildDir%\supersonic-webapp"
+endlocal

assembly/bin/supersonic-daemon.sh

@@ -1,16 +1,11 @@
 #!/usr/bin/env bash
 
-set -x
 sbinDir=$(cd "$(dirname "$0")"; pwd)
-chmod +x $sbinDir/supersonic-common.sh
 source $sbinDir/supersonic-common.sh
 
-# 1.init environment parameters
-if [ ! -d "$runtimeDir" ]; then
-    echo "the runtime dir does not exist move all to runtime"
-    moveAllToRuntime
-fi
-set +x
+set -a
+source $sbinDir/../conf/supersonic-env.sh
+set +a
 
 command=$1
 service=$2
@@ -18,44 +13,93 @@ if [ -z "$service"  ]; then
   service=${STANDALONE_SERVICE}
 fi
 
-app_name=$STANDALONE_APP_NAME
-main_class="com.tencent.supersonic.StandaloneLauncher"
 model_name=$service
-
 if [ "$service" == "pyllm" ]; then
   model_name=${STANDALONE_SERVICE}
   export llmProxy=PythonLLMProxy
 fi
-
 cd $baseDir
 
-# 2.set main class
 function setMainClass {
   if [ "$service" == $CHAT_SERVICE ]; then
     main_class="com.tencent.supersonic.ChatLauncher"
   elif [ "$service" == $HEADLESS_SERVICE ]; then
     main_class="com.tencent.supersonic.HeadlessLauncher"
+  else
+    main_class="com.tencent.supersonic.StandaloneLauncher"
   fi
 }
-setMainClass
-# 3.set app name
+
 function setAppName {
   if [ "$service" == $CHAT_SERVICE ]; then
     app_name=$CHAT_APP_NAME
   elif [ "$service" == $HEADLESS_SERVICE ]; then
     app_name=$HEADLESS_APP_NAME
-  elif [ "$service" == $PYLLM_SERVICE ]; then
-    app_name=$PYLLM_APP_NAME
+  else
+    app_name=$STANDALONE_APP_NAME
   fi
 }
-setAppName
 
 function reloadExamples {
-  pythonRunDir=${runtimeDir}/supersonic-${model_name}/pyllm
-  cd $pythonRunDir/sql
+  cd $baseDir/pyllm/sql
   ${python_path} examples_reload_run.py
 }
 
+function runJavaService {
+  javaRunDir=$baseDir
+  local_app_name=$1
+  libDir=$baseDir/lib
+  confDir=$baseDir/conf
+
+  CLASSPATH=""
+  CLASSPATH=$CLASSPATH:$confDir
+
+  for jarPath in $libDir/*.jar; do
+   CLASSPATH=$CLASSPATH:$jarPath
+  done
+
+  export CLASSPATH
+  export LANG="zh_CN.UTF-8"
+
+  cd $javaRunDir
+  if [[ "$JAVA_HOME" == "" ]]; then
+    JAVA_HOME=$(ls /usr/jdk64/jdk* -d 2>/dev/null | xargs | awk '{print "'$local_app_name'"}')
+  fi
+  export PATH=$JAVA_HOME/bin:$PATH
+  command="-Dfile.encoding="UTF-8"  -Duser.language="Zh" -Duser.region="CN" -Duser.timezone="GMT+08" -Dapp_name=${local_app_name} -Xms1024m -Xmx2048m "$main_class
+
+  mkdir -p $javaRunDir/logs
+  if [[ "$is_test" == "true" ]]; then
+    java -Dspring.profiles.active="dev" $command >/dev/null 2>$javaRunDir/logs/error.log &
+  else
+    java  $command $javaRunDir >/dev/null 2>$javaRunDir/logs/error.log &
+  fi
+}
+
+function runPythonService {
+  pythonRunDir=$baseDir/pyllm
+  cd $pythonRunDir
+  nohup ${python_path} supersonic_pyllm.py  > $pythonRunDir/pyllm.log  2>&1   &
+  # add health check
+  for i in {1..10}
+  do
+    echo "pyllm health check attempt $i..."
+    response=$(curl -s http://${PYLLM_HOST}:${PYLLM_PORT}/health)
+    echo "pyllm health check response: $response"
+    status_ok="Healthy"
+    if [[ $response == *$status_ok* ]] ; then
+      echo "pyllm Health check passed."
+      break
+    else
+      if [ "$i" -eq 10 ]; then
+        echo "pyllm Health check failed after 10 attempts."
+        echo "May still downloading model files. Please check pyllm.log in runtime directory."
+      fi
+      echo "Retrying after 5 seconds..."
+      sleep 5
+    fi
+  done
+}
 
 function start()
 {
@@ -93,18 +137,16 @@ function reload()
   fi
 }
 
-# 4. execute command operation
+setMainClass
+setAppName
 case "$command" in
   start)
         if [ "$service" == $PYLLM_SERVICE ]; then
-          echo  "Starting $app_name"
-          start $app_name
-          echo  "Starting $STANDALONE_APP_NAME"
-          start $STANDALONE_APP_NAME
-        else
-          echo  "Starting $app_name"
-          start $app_name
+          echo  "Starting $PYLLM_APP_NAME"
+          start $PYLLM_APP_NAME
         fi
+        echo  "Starting ${app_name}"
+        start ${app_name}
         echo  "Start success"
         ;;
   stop)
@@ -121,20 +163,15 @@ case "$command" in
         ;;
   restart)
         if [ "$service" == $PYLLM_SERVICE ]; then
-          echo  "Stopping ${app_name}"
-          stop ${app_name}
-          echo  "Stopping ${STANDALONE_APP_NAME}"
-          stop $STANDALONE_APP_NAME
-          echo  "Starting ${app_name}"
-          start ${app_name}
-          echo  "Starting ${STANDALONE_APP_NAME}"
-          start $STANDALONE_APP_NAME
-        else
-          echo  "Stopping ${app_name}"
-          stop ${app_name}
-          echo  "Starting ${app_name}"
-          start ${app_name}
+          echo  "Stopping $PYLLM_APP_NAME"
+          stop $PYLLM_APP_NAME
+          echo  "Starting $PYLLM_APP_NAME"
+          start $PYLLM_APP_NAME
         fi
+        echo  "Stopping ${app_name}"
+        stop ${app_name}
+        echo  "Starting ${app_name}"
+        start ${app_name}
         echo  "Restart success"
         ;;
   *)

assembly/build/build.xml

@@ -21,11 +21,17 @@
             </includes>
         </fileSet>
         <fileSet>
-            <directory>${project.basedir}/../../chat/python</directory>
+            <directory>${project.basedir}/../../headless/python</directory>
             <outputDirectory>pyllm</outputDirectory>
             <fileMode>0777</fileMode>
             <directoryMode>0755</directoryMode>
         </fileSet>
+        <fileSet>
+            <directory>${project.basedir}/../../assembly/bin</directory>
+            <outputDirectory>bin</outputDirectory>
+            <fileMode>0777</fileMode>
+            <directoryMode>0755</directoryMode>
+        </fileSet>
     </fileSets>
 
     <dependencySets>

auth/api/src/main/java/com/tencent/supersonic/auth/api/authentication/utils/UserHolder.java

@@ -2,8 +2,8 @@ package com.tencent.supersonic.auth.api.authentication.utils;
 
 import com.tencent.supersonic.auth.api.authentication.pojo.User;
 import com.tencent.supersonic.auth.api.authentication.service.UserStrategy;
-import com.tencent.supersonic.common.pojo.SysParameter;
-import com.tencent.supersonic.common.service.SysParameterService;
+import com.tencent.supersonic.common.pojo.SystemConfig;
+import com.tencent.supersonic.common.service.SystemConfigService;
 import com.tencent.supersonic.common.util.ContextUtils;
 import org.springframework.util.CollectionUtils;
 
@@ -20,10 +20,10 @@ public final class UserHolder {
 
     public static User findUser(HttpServletRequest request, HttpServletResponse response) {
         User user = REPO.findUser(request, response);
-        SysParameterService sysParameterService = ContextUtils.getBean(SysParameterService.class);
-        SysParameter sysParameter = sysParameterService.getSysParameter();
-        if (!CollectionUtils.isEmpty(sysParameter.getAdmins())
-                && sysParameter.getAdmins().contains(user.getName())) {
+        SystemConfigService sysParameterService = ContextUtils.getBean(SystemConfigService.class);
+        SystemConfig systemConfig = sysParameterService.getSystemConfig();
+        if (!CollectionUtils.isEmpty(systemConfig.getAdmins())
+                && systemConfig.getAdmins().contains(user.getName())) {
             user.setIsAdmin(1);
         }
         return user;

auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/service/UserServiceImpl.java

@@ -6,8 +6,8 @@ import com.tencent.supersonic.auth.api.authentication.request.UserReq;
 import com.tencent.supersonic.auth.api.authentication.service.UserService;
 import com.tencent.supersonic.auth.api.authentication.utils.UserHolder;
 import com.tencent.supersonic.auth.authentication.utils.ComponentFactory;
-import com.tencent.supersonic.common.pojo.SysParameter;
-import com.tencent.supersonic.common.service.SysParameterService;
+import com.tencent.supersonic.common.pojo.SystemConfig;
+import com.tencent.supersonic.common.service.SystemConfigService;
 import org.springframework.stereotype.Service;
 import org.springframework.util.CollectionUtils;
 import javax.servlet.http.HttpServletRequest;
@@ -18,9 +18,9 @@ import java.util.Set;
 @Service
 public class UserServiceImpl implements UserService {
 
-    private SysParameterService sysParameterService;
+    private SystemConfigService sysParameterService;
 
-    public UserServiceImpl(SysParameterService sysParameterService) {
+    public UserServiceImpl(SystemConfigService sysParameterService) {
         this.sysParameterService = sysParameterService;
     }
 
@@ -28,9 +28,9 @@ public class UserServiceImpl implements UserService {
     public User getCurrentUser(HttpServletRequest httpServletRequest, HttpServletResponse httpServletResponse) {
         User user = UserHolder.findUser(httpServletRequest, httpServletResponse);
         if (user != null) {
-            SysParameter sysParameter = sysParameterService.getSysParameter();
-            if (!CollectionUtils.isEmpty(sysParameter.getAdmins())
-                    && sysParameter.getAdmins().contains(user.getName())) {
+            SystemConfig systemConfig = sysParameterService.getSystemConfig();
+            if (!CollectionUtils.isEmpty(systemConfig.getAdmins())
+                    && systemConfig.getAdmins().contains(user.getName())) {
                 user.setIsAdmin(1);
             }
         }

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/ChatParseReq.java

@@ -10,6 +10,7 @@ public class ChatParseReq {
     private String queryText;
     private Integer chatId;
     private Integer agentId;
+    private Integer topN = 10;
     private User user;
     private QueryFilters queryFilters;
     private boolean saveAnswer = true;

chat/server/src/main/java/com/tencent/supersonic/chat/server/agent/Agent.java

@@ -4,6 +4,7 @@ package com.tencent.supersonic.chat.server.agent;
 import com.alibaba.fastjson.JSONObject;
 import com.google.common.collect.Lists;
 import com.google.common.collect.Sets;
+import com.tencent.supersonic.headless.api.pojo.LLMConfig;
 import com.tencent.supersonic.common.pojo.RecordInfo;
 import lombok.Data;
 import org.springframework.util.CollectionUtils;
@@ -30,6 +31,8 @@ public class Agent extends RecordInfo {
     private Integer status;
     private List<String> examples;
     private String agentConfig;
+    private LLMConfig llmConfig;
+    private MultiTurnConfig multiTurnConfig;
 
     public List<String> getTools(AgentToolType type) {
         Map map = JSONObject.parseObject(agentConfig, Map.class);
@@ -70,6 +73,10 @@ public class Agent extends RecordInfo {
         return !CollectionUtils.isEmpty(getParserTools(AgentToolType.NL2SQL_LLM));
     }
 
+    public boolean containsRuleTool() {
+        return !CollectionUtils.isEmpty(getParserTools(AgentToolType.NL2SQL_RULE));
+    }
+
     public boolean containsNL2SQLTool() {
         return !CollectionUtils.isEmpty(getParserTools(AgentToolType.NL2SQL_LLM))
                 || !CollectionUtils.isEmpty(getParserTools(AgentToolType.NL2SQL_RULE));

chat/server/src/main/java/com/tencent/supersonic/chat/server/agent/MultiTurnConfig.java

@@ -0,0 +1,15 @@
+package com.tencent.supersonic.chat.server.agent;
+
+
+import lombok.AllArgsConstructor;
+import lombok.Data;
+import lombok.NoArgsConstructor;
+
+@Data
+@AllArgsConstructor
+@NoArgsConstructor
+public class MultiTurnConfig {
+
+    private boolean enableMultiTurn;
+
+}

chat/server/src/main/java/com/tencent/supersonic/chat/server/executor/SqlExecutor.java

@@ -2,6 +2,7 @@ package com.tencent.supersonic.chat.server.executor;
 
 import com.tencent.supersonic.chat.server.plugin.PluginQueryManager;
 import com.tencent.supersonic.chat.server.pojo.ChatExecuteContext;
+import com.tencent.supersonic.chat.server.util.ResultFormatter;
 import com.tencent.supersonic.common.util.ContextUtils;
 import com.tencent.supersonic.headless.api.pojo.SemanticParseInfo;
 import com.tencent.supersonic.headless.api.pojo.request.ExecuteQueryReq;
@@ -20,7 +21,11 @@ public class SqlExecutor implements ChatExecutor {
         }
         ExecuteQueryReq executeQueryReq = buildExecuteReq(chatExecuteContext);
         ChatQueryService chatQueryService = ContextUtils.getBean(ChatQueryService.class);
-        return chatQueryService.performExecution(executeQueryReq);
+        QueryResult queryResult = chatQueryService.performExecution(executeQueryReq);
+        String textResult = ResultFormatter.transform2TextNew(queryResult.getQueryColumns(),
+                queryResult.getQueryResults());
+        queryResult.setTextResult(textResult);
+        return queryResult;
     }
 
     private ExecuteQueryReq buildExecuteReq(ChatExecuteContext chatExecuteContext) {

chat/server/src/main/java/com/tencent/supersonic/chat/server/parser/MultiTurnParser.java

@@ -0,0 +1,154 @@
+package com.tencent.supersonic.chat.server.parser;
+
+import com.tencent.supersonic.chat.server.agent.MultiTurnConfig;
+import com.tencent.supersonic.chat.server.persistence.repository.ChatQueryRepository;
+import com.tencent.supersonic.chat.server.pojo.ChatParseContext;
+import com.tencent.supersonic.chat.server.util.QueryReqConverter;
+import com.tencent.supersonic.common.util.ContextUtils;
+import com.tencent.supersonic.headless.api.pojo.LLMConfig;
+import com.tencent.supersonic.headless.api.pojo.SchemaElementMatch;
+import com.tencent.supersonic.headless.api.pojo.SchemaElementType;
+import com.tencent.supersonic.headless.api.pojo.request.QueryReq;
+import com.tencent.supersonic.headless.api.pojo.response.MapResp;
+import com.tencent.supersonic.headless.api.pojo.response.ParseResp;
+import com.tencent.supersonic.headless.core.utils.S2ChatModelProvider;
+import com.tencent.supersonic.headless.server.service.ChatQueryService;
+import dev.langchain4j.data.message.AiMessage;
+import dev.langchain4j.model.chat.ChatLanguageModel;
+import dev.langchain4j.model.input.Prompt;
+import dev.langchain4j.model.input.PromptTemplate;
+import dev.langchain4j.model.output.Response;
+import lombok.Builder;
+import lombok.Data;
+import lombok.extern.slf4j.Slf4j;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Map;
+import java.util.HashMap;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.Collections;
+
+import static com.tencent.supersonic.chat.server.parser.ParserConfig.PARSER_MULTI_TURN_ENABLE;
+
+@Slf4j
+public class MultiTurnParser implements ChatParser {
+
+    private static final Logger keyPipelineLog = LoggerFactory.getLogger("keyPipeline");
+
+    private static final PromptTemplate promptTemplate = PromptTemplate.from(
+            "You are a data product manager experienced in data requirements."
+                    + "Your will be provided with current and history questions asked by a user,"
+                    + "along with their mapped schema elements(metric, dimension and value), "
+                    + "please try understanding the semantics and rewrite a question"
+                    + "(keep relevant entities, metrics, dimensions, values and date ranges)."
+                    + "Current Question: {{curtQuestion}} "
+                    + "Current Mapped Schema: {{curtSchema}} "
+                    + "History Question: {{histQuestion}} "
+                    + "History Mapped Schema: {{histSchema}} "
+                    + "Rewritten Question: ");
+
+    @Override
+    public void parse(ChatParseContext chatParseContext, ParseResp parseResp) {
+        ParserConfig parserConfig = ContextUtils.getBean(ParserConfig.class);
+        MultiTurnConfig agentMultiTurnConfig = chatParseContext.getAgent().getMultiTurnConfig();
+        Boolean globalMultiTurnConfig = Boolean.valueOf(parserConfig.getParameterValue(PARSER_MULTI_TURN_ENABLE));
+
+        Boolean multiTurnConfig = agentMultiTurnConfig != null
+                ? agentMultiTurnConfig.isEnableMultiTurn() : globalMultiTurnConfig;
+        if (!Boolean.TRUE.equals(multiTurnConfig)) {
+            return;
+        }
+
+        // derive mapping result of current question and parsing result of last question.
+        ChatQueryService chatQueryService = ContextUtils.getBean(ChatQueryService.class);
+        QueryReq queryReq = QueryReqConverter.buildText2SqlQueryReq(chatParseContext);
+        MapResp currentMapResult = chatQueryService.performMapping(queryReq);
+
+        List<ParseResp> historyParseResults = getHistoryParseResult(chatParseContext.getChatId(), 1);
+        if (historyParseResults.size() == 0) {
+            return;
+        }
+        ParseResp lastParseResult = historyParseResults.get(0);
+        Long dataId = lastParseResult.getSelectedParses().get(0).getDataSetId();
+
+        String curtMapStr = generateSchemaPrompt(currentMapResult.getMapInfo().getMatchedElements(dataId));
+        String histMapStr = generateSchemaPrompt(lastParseResult.getSelectedParses().get(0).getElementMatches());
+        String rewrittenQuery = rewriteQuery(RewriteContext.builder()
+                        .curtQuestion(currentMapResult.getQueryText())
+                        .histQuestion(lastParseResult.getQueryText())
+                        .curtSchema(curtMapStr)
+                        .histSchema(histMapStr)
+                        .llmConfig(queryReq.getLlmConfig())
+                        .build());
+        chatParseContext.setQueryText(rewrittenQuery);
+        log.info("Last Query: {} Current Query: {}, Rewritten Query: {}",
+                lastParseResult.getQueryText(), currentMapResult.getQueryText(), rewrittenQuery);
+    }
+
+    private String rewriteQuery(RewriteContext context) {
+        Map<String, Object> variables = new HashMap<>();
+        variables.put("curtQuestion", context.getCurtQuestion());
+        variables.put("histQuestion", context.getHistQuestion());
+        variables.put("curtSchema", context.getCurtSchema());
+        variables.put("histSchema", context.getHistSchema());
+
+        Prompt prompt = promptTemplate.apply(variables);
+        keyPipelineLog.info("MultiTurnParser reqPrompt:{}", prompt.toSystemMessage());
+
+        ChatLanguageModel chatLanguageModel = S2ChatModelProvider.provide(context.getLlmConfig());
+        Response<AiMessage> response = chatLanguageModel.generate(prompt.toSystemMessage());
+
+        String result = response.content().text();
+        keyPipelineLog.info("MultiTurnParser modelResp:{}", result);
+        return response.content().text();
+    }
+
+    private String generateSchemaPrompt(List<SchemaElementMatch> elementMatches) {
+        List<String> metrics = new ArrayList<>();
+        List<String> dimensions = new ArrayList<>();
+        List<String> values = new ArrayList<>();
+
+        for (SchemaElementMatch match : elementMatches) {
+            if (match.getElement().getType().equals(SchemaElementType.METRIC)) {
+                metrics.add(match.getWord());
+            } else if (match.getElement().getType().equals(SchemaElementType.DIMENSION)) {
+                dimensions.add(match.getWord());
+            } else if (match.getElement().getType().equals(SchemaElementType.VALUE)) {
+                values.add(match.getWord());
+            }
+        }
+
+        StringBuilder prompt = new StringBuilder();
+        prompt.append(String.format("'metrics:':[%s]", String.join(",", metrics)));
+        prompt.append(",");
+        prompt.append(String.format("'dimensions:':[%s]", String.join(",", dimensions)));
+        prompt.append(",");
+        prompt.append(String.format("'values:':[%s]", String.join(",", values)));
+
+        return prompt.toString();
+    }
+
+    private List<ParseResp> getHistoryParseResult(int chatId, int multiNum) {
+        ChatQueryRepository chatQueryRepository = ContextUtils.getBean(ChatQueryRepository.class);
+        List<ParseResp> contextualParseInfoList = chatQueryRepository.getContextualParseInfo(chatId)
+                .stream().filter(p -> p.getState() != ParseResp.ParseState.FAILED).collect(Collectors.toList());
+
+        List<ParseResp> contextualList = contextualParseInfoList.subList(0,
+                Math.min(multiNum, contextualParseInfoList.size()));
+        Collections.reverse(contextualList);
+        return contextualList;
+    }
+
+    @Data
+    @Builder
+    public static class RewriteContext {
+        private String curtQuestion;
+        private String histQuestion;
+        private String curtSchema;
+        private String histSchema;
+        private LLMConfig llmConfig;
+    }
+}

chat/server/src/main/java/com/tencent/supersonic/chat/server/parser/NL2PluginParser.java

@@ -17,8 +17,8 @@ public class NL2PluginParser implements ChatParser {
     public void parse(ChatParseContext chatParseContext, ParseResp parseResp) {
         pluginRecognizers.forEach(pluginRecognizer -> {
             pluginRecognizer.recognize(chatParseContext, parseResp);
-            log.info("{} context:{} result:{}", pluginRecognizer.getClass().getSimpleName(),
-                    JsonUtil.toString(chatParseContext), JsonUtil.toString(parseResp));
+            log.info("{} recallResult:{}", pluginRecognizer.getClass().getSimpleName(),
+                    JsonUtil.toString(parseResp));
         });
     }
 

chat/server/src/main/java/com/tencent/supersonic/chat/server/parser/NL2SQLParser.java

@@ -1,14 +1,24 @@
 package com.tencent.supersonic.chat.server.parser;
 
+import com.tencent.supersonic.chat.server.plugin.PluginQueryManager;
 import com.tencent.supersonic.chat.server.pojo.ChatParseContext;
 import com.tencent.supersonic.chat.server.util.QueryReqConverter;
 import com.tencent.supersonic.common.util.ContextUtils;
+import com.tencent.supersonic.headless.api.pojo.SchemaElement;
 import com.tencent.supersonic.headless.api.pojo.SemanticParseInfo;
+import com.tencent.supersonic.headless.api.pojo.request.QueryFilter;
 import com.tencent.supersonic.headless.api.pojo.request.QueryReq;
 import com.tencent.supersonic.headless.api.pojo.response.ParseResp;
 import com.tencent.supersonic.headless.server.service.ChatQueryService;
+import lombok.extern.slf4j.Slf4j;
+import org.springframework.util.CollectionUtils;
 import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
 
+@Slf4j
 public class NL2SQLParser implements ChatParser {
 
     @Override
@@ -20,12 +30,14 @@ public class NL2SQLParser implements ChatParser {
             return;
         }
         QueryReq queryReq = QueryReqConverter.buildText2SqlQueryReq(chatParseContext);
+
         ChatQueryService chatQueryService = ContextUtils.getBean(ChatQueryService.class);
         ParseResp text2SqlParseResp = chatQueryService.performParsing(queryReq);
         if (!ParseResp.ParseState.FAILED.equals(text2SqlParseResp.getState())) {
             parseResp.getSelectedParses().addAll(text2SqlParseResp.getSelectedParses());
         }
         parseResp.getParseTimeCost().setSqlTime(text2SqlParseResp.getParseTimeCost().getSqlTime());
+        formatParseResult(parseResp);
     }
 
     private boolean checkSkip(ParseResp parseResp) {
@@ -38,4 +50,49 @@ public class NL2SQLParser implements ChatParser {
         return false;
     }
 
+    private void formatParseResult(ParseResp parseResp) {
+        List<SemanticParseInfo> selectedParses = parseResp.getSelectedParses();
+        for (SemanticParseInfo parseInfo : selectedParses) {
+            formatParseInfo(parseInfo);
+        }
+    }
+
+    private void formatParseInfo(SemanticParseInfo parseInfo) {
+        if (!PluginQueryManager.isPluginQuery(parseInfo.getQueryMode())) {
+            formatNL2SQLParseInfo(parseInfo);
+        }
+    }
+
+    private void formatNL2SQLParseInfo(SemanticParseInfo parseInfo) {
+        StringBuilder textBuilder = new StringBuilder();
+        textBuilder.append("**数据集:** ").append(parseInfo.getDataSet().getName()).append(" ");
+        Optional<SchemaElement> metric = parseInfo.getMetrics().stream().findFirst();
+        metric.ifPresent(schemaElement ->
+                textBuilder.append("**指标:** ").append(schemaElement.getName()).append(" "));
+        List<String> dimensionNames = parseInfo.getDimensions().stream()
+                .map(SchemaElement::getName).filter(Objects::nonNull).collect(Collectors.toList());
+        if (!CollectionUtils.isEmpty(dimensionNames)) {
+            textBuilder.append("**维度:** ").append(String.join(",", dimensionNames));
+        }
+        textBuilder.append("\n\n**筛选条件:** \n");
+        if (parseInfo.getDateInfo() != null) {
+            textBuilder.append("**数据时间:** ").append(parseInfo.getDateInfo().getStartDate()).append("~")
+                    .append(parseInfo.getDateInfo().getEndDate()).append(" ");
+        }
+        if (!CollectionUtils.isEmpty(parseInfo.getDimensionFilters())
+                || CollectionUtils.isEmpty(parseInfo.getMetricFilters())) {
+            Set<QueryFilter> queryFilters = parseInfo.getDimensionFilters();
+            queryFilters.addAll(parseInfo.getMetricFilters());
+            for (QueryFilter queryFilter : queryFilters) {
+                textBuilder.append("**").append(queryFilter.getName()).append("**")
+                        .append(" ")
+                        .append(queryFilter.getOperator().getValue())
+                        .append(" ")
+                        .append(queryFilter.getValue())
+                        .append(" ");
+            }
+        }
+        parseInfo.setTextInfo(textBuilder.toString());
+    }
+
 }

chat/server/src/main/java/com/tencent/supersonic/chat/server/parser/ParserConfig.java

@@ -0,0 +1,27 @@
+package com.tencent.supersonic.chat.server.parser;
+
+import com.google.common.collect.Lists;
+import com.tencent.supersonic.common.pojo.ParameterConfig;
+import com.tencent.supersonic.common.pojo.Parameter;
+import lombok.extern.slf4j.Slf4j;
+import org.springframework.stereotype.Service;
+
+import java.util.List;
+
+@Service("ChatParserConfig")
+@Slf4j
+public class ParserConfig extends ParameterConfig {
+
+    public static final Parameter PARSER_MULTI_TURN_ENABLE =
+            new Parameter("s2.parser.multi-turn.enable", "false",
+                    "是否开启多轮对话", "开启多轮对话将消耗更多token",
+                    "bool", "Parser相关配置");
+
+    @Override
+    public List<Parameter> getSysParameters() {
+        return Lists.newArrayList(
+                PARSER_MULTI_TURN_ENABLE
+        );
+    }
+
+}

chat/server/src/main/java/com/tencent/supersonic/chat/server/persistence/dataobject/AgentDO.java

@@ -1,10 +1,18 @@
 package com.tencent.supersonic.chat.server.persistence.dataobject;
 
+import com.baomidou.mybatisplus.annotation.IdType;
+import com.baomidou.mybatisplus.annotation.TableId;
+import com.baomidou.mybatisplus.annotation.TableName;
+import lombok.Data;
+
 import java.util.Date;
 
+@Data
+@TableName("s2_agent")
 public class AgentDO {
     /**
      */
+    @TableId(type = IdType.AUTO)
     private Integer id;
 
     /**
@@ -48,159 +56,8 @@ public class AgentDO {
      */
     private Integer enableSearch;
 
-    /**
-     * @return id
-     */
-    public Integer getId() {
-        return id;
-    }
+    private String llmConfig;
 
-    /**
-     * @param id
-     */
-    public void setId(Integer id) {
-        this.id = id;
-    }
+    private String multiTurnConfig;
 
-    /**
-     * @return name
-     */
-    public String getName() {
-        return name;
-    }
-
-    /**
-     * @param name
-     */
-    public void setName(String name) {
-        this.name = name == null ? null : name.trim();
-    }
-
-    /**
-     * @return description
-     */
-    public String getDescription() {
-        return description;
-    }
-
-    /**
-     * @param description
-     */
-    public void setDescription(String description) {
-        this.description = description == null ? null : description.trim();
-    }
-
-    /**
-     * 0 offline, 1 online
-     * @return status 0 offline, 1 online
-     */
-    public Integer getStatus() {
-        return status;
-    }
-
-    /**
-     * 0 offline, 1 online
-     * @param status 0 offline, 1 online
-     */
-    public void setStatus(Integer status) {
-        this.status = status;
-    }
-
-    /**
-     * @return examples
-     */
-    public String getExamples() {
-        return examples;
-    }
-
-    /**
-     * @param examples
-     */
-    public void setExamples(String examples) {
-        this.examples = examples == null ? null : examples.trim();
-    }
-
-    /**
-     * @return config
-     */
-    public String getConfig() {
-        return config;
-    }
-
-    /**
-     * @param config
-     */
-    public void setConfig(String config) {
-        this.config = config == null ? null : config.trim();
-    }
-
-    /**
-     * @return created_by
-     */
-    public String getCreatedBy() {
-        return createdBy;
-    }
-
-    /**
-     * @param createdBy
-     */
-    public void setCreatedBy(String createdBy) {
-        this.createdBy = createdBy == null ? null : createdBy.trim();
-    }
-
-    /**
-     * @return created_at
-     */
-    public Date getCreatedAt() {
-        return createdAt;
-    }
-
-    /**
-     * @param createdAt
-     */
-    public void setCreatedAt(Date createdAt) {
-        this.createdAt = createdAt;
-    }
-
-    /**
-     * @return updated_by
-     */
-    public String getUpdatedBy() {
-        return updatedBy;
-    }
-
-    /**
-     * @param updatedBy
-     */
-    public void setUpdatedBy(String updatedBy) {
-        this.updatedBy = updatedBy == null ? null : updatedBy.trim();
-    }
-
-    /**
-     * @return updated_at
-     */
-    public Date getUpdatedAt() {
-        return updatedAt;
-    }
-
-    /**
-     * @param updatedAt
-     */
-    public void setUpdatedAt(Date updatedAt) {
-        this.updatedAt = updatedAt;
-    }
-
-    /**
-     * @return enable_search
-     */
-    public Integer getEnableSearch() {
-        return enableSearch;
-    }
-
-    /**
-     * @param enableSearch
-     */
-    public void setEnableSearch(Integer enableSearch) {
-        this.enableSearch = enableSearch;
-    }
 }

chat/server/src/main/java/com/tencent/supersonic/chat/server/persistence/mapper/AgentDOMapper.java

@@ -1,71 +1,10 @@
 package com.tencent.supersonic.chat.server.persistence.mapper;
 
+import com.baomidou.mybatisplus.core.mapper.BaseMapper;
 import com.tencent.supersonic.chat.server.persistence.dataobject.AgentDO;
-import com.tencent.supersonic.chat.server.persistence.dataobject.AgentDOExample;
 import org.apache.ibatis.annotations.Mapper;
-import org.apache.ibatis.annotations.Param;
-
-import java.util.List;
 
 @Mapper
-public interface AgentDOMapper {
-    /**
-     *
-     * @mbg.generated
-     */
-    long countByExample(AgentDOExample example);
+public interface AgentDOMapper extends BaseMapper<AgentDO> {
 
-    /**
-     *
-     * @mbg.generated
-     */
-    int deleteByPrimaryKey(Integer id);
-
-    /**
-     *
-     * @mbg.generated
-     */
-    int insert(AgentDO record);
-
-    /**
-     *
-     * @mbg.generated
-     */
-    int insertSelective(AgentDO record);
-
-    /**
-     *
-     * @mbg.generated
-     */
-    List<AgentDO> selectByExample(AgentDOExample example);
-
-    /**
-     *
-     * @mbg.generated
-     */
-    AgentDO selectByPrimaryKey(Integer id);
-
-    /**
-     *
-     * @mbg.generated
-     */
-    int updateByExampleSelective(@Param("record") AgentDO record, @Param("example") AgentDOExample example);
-
-    /**
-     *
-     * @mbg.generated
-     */
-    int updateByExample(@Param("record") AgentDO record, @Param("example") AgentDOExample example);
-
-    /**
-     *
-     * @mbg.generated
-     */
-    int updateByPrimaryKeySelective(AgentDO record);
-
-    /**
-     *
-     * @mbg.generated
-     */
-    int updateByPrimaryKey(AgentDO record);
 }

chat/server/src/main/java/com/tencent/supersonic/chat/server/persistence/mapper/ChatParseMapper.java

@@ -18,4 +18,6 @@ public interface ChatParseMapper {
 
     List<ChatParseDO> getParseInfoList(List<Long> questionIds);
 
+    List<ChatParseDO> getContextualParseInfo(Integer chatId);
+
 }

chat/server/src/main/java/com/tencent/supersonic/chat/server/persistence/repository/AgentRepository.java

@@ -1,18 +0,0 @@
-package com.tencent.supersonic.chat.server.persistence.repository;
-
-import com.tencent.supersonic.chat.server.persistence.dataobject.AgentDO;
-
-import java.util.List;
-
-public interface AgentRepository {
-
-    List<AgentDO> getAgents();
-
-    void createAgent(AgentDO agentDO);
-
-    void updateAgent(AgentDO agentDO);
-
-    AgentDO getAgent(Integer id);
-
-    void deleteAgent(Integer id);
-}

chat/server/src/main/java/com/tencent/supersonic/chat/server/persistence/repository/ChatQueryRepository.java

@@ -35,4 +35,7 @@ public interface ChatQueryRepository {
     List<ChatParseDO> getParseInfoList(List<Long> questionIds);
 
     Boolean deleteChatQuery(Long questionId);
+
+    List<ParseResp> getContextualParseInfo(Integer chatId);
+
 }

chat/server/src/main/java/com/tencent/supersonic/chat/server/persistence/repository/ChatRepository.java

@@ -6,7 +6,7 @@ import java.util.List;
 
 public interface ChatRepository {
 
-    boolean createChat(ChatDO chatDO);
+    Long createChat(ChatDO chatDO);
 
     List<ChatDO> getAll(String creator, Integer agentId);
 

chat/server/src/main/java/com/tencent/supersonic/chat/server/persistence/repository/impl/AgentRepositoryImpl.java

@@ -1,43 +0,0 @@
-package com.tencent.supersonic.chat.server.persistence.repository.impl;
-
-import com.tencent.supersonic.chat.server.persistence.dataobject.AgentDO;
-import com.tencent.supersonic.chat.server.persistence.dataobject.AgentDOExample;
-import com.tencent.supersonic.chat.server.persistence.mapper.AgentDOMapper;
-import com.tencent.supersonic.chat.server.persistence.repository.AgentRepository;
-import org.springframework.stereotype.Repository;
-import java.util.List;
-
-@Repository
-public class AgentRepositoryImpl implements AgentRepository {
-
-    private AgentDOMapper agentDOMapper;
-
-    public AgentRepositoryImpl(AgentDOMapper agentDOMapper) {
-        this.agentDOMapper = agentDOMapper;
-    }
-
-    @Override
-    public List<AgentDO> getAgents() {
-        return agentDOMapper.selectByExample(new AgentDOExample());
-    }
-
-    @Override
-    public void createAgent(AgentDO agentDO) {
-        agentDOMapper.insert(agentDO);
-    }
-
-    @Override
-    public void updateAgent(AgentDO agentDO) {
-        agentDOMapper.updateByPrimaryKey(agentDO);
-    }
-
-    @Override
-    public AgentDO getAgent(Integer id) {
-        return agentDOMapper.selectByPrimaryKey(id);
-    }
-
-    @Override
-    public void deleteAgent(Integer id) {
-        agentDOMapper.deleteByPrimaryKey(id);
-    }
-}

chat/server/src/main/java/com/tencent/supersonic/chat/server/persistence/repository/impl/ChatQueryRepositoryImpl.java

@@ -26,6 +26,7 @@ import org.springframework.beans.BeanUtils;
 import org.springframework.context.annotation.Primary;
 import org.springframework.stereotype.Repository;
 import org.springframework.util.CollectionUtils;
+
 import java.util.ArrayList;
 import java.util.Comparator;
 import java.util.List;
@@ -44,8 +45,8 @@ public class ChatQueryRepositoryImpl implements ChatQueryRepository {
     private final ShowCaseCustomMapper showCaseCustomMapper;
 
     public ChatQueryRepositoryImpl(ChatQueryDOMapper chatQueryDOMapper,
-            ChatParseMapper chatParseMapper,
-            ShowCaseCustomMapper showCaseCustomMapper) {
+                                   ChatParseMapper chatParseMapper,
+                                   ShowCaseCustomMapper showCaseCustomMapper) {
         this.chatQueryDOMapper = chatQueryDOMapper;
         this.chatParseMapper = chatParseMapper;
         this.showCaseCustomMapper = showCaseCustomMapper;
@@ -131,7 +132,7 @@ public class ChatQueryRepositoryImpl implements ChatQueryRepository {
 
     @Override
     public List<ChatParseDO> batchSaveParseInfo(ChatParseReq chatParseReq,
-            ParseResp parseResult, List<SemanticParseInfo> candidateParses) {
+                                                ParseResp parseResult, List<SemanticParseInfo> candidateParses) {
         List<ChatParseDO> chatParseDOList = new ArrayList<>();
         getChatParseDO(chatParseReq, parseResult.getQueryId(), candidateParses, chatParseDOList);
         if (!CollectionUtils.isEmpty(candidateParses)) {
@@ -141,7 +142,7 @@ public class ChatQueryRepositoryImpl implements ChatQueryRepository {
     }
 
     public void getChatParseDO(ChatParseReq chatParseReq, Long queryId,
-            List<SemanticParseInfo> parses, List<ChatParseDO> chatParseDOList) {
+                               List<SemanticParseInfo> parses, List<ChatParseDO> chatParseDOList) {
         for (int i = 0; i < parses.size(); i++) {
             ChatParseDO chatParseDO = new ChatParseDO();
             chatParseDO.setChatId(Long.valueOf(chatParseReq.getChatId()));
@@ -192,4 +193,18 @@ public class ChatQueryRepositoryImpl implements ChatQueryRepository {
     public Boolean deleteChatQuery(Long questionId) {
         return chatQueryDOMapper.deleteByPrimaryKey(questionId);
     }
+
+    @Override
+    public List<ParseResp> getContextualParseInfo(Integer chatId) {
+        List<ChatParseDO> chatParseDOList = chatParseMapper.getContextualParseInfo(chatId);
+        List<ParseResp> semanticParseInfoList = chatParseDOList.stream().map(parseInfo -> {
+            ParseResp parseResp = new ParseResp(chatId, parseInfo.getQueryText());
+            List<SemanticParseInfo> selectedParses = new ArrayList<>();
+            selectedParses.add(JSONObject.parseObject(parseInfo.getParseInfo(), SemanticParseInfo.class));
+            parseResp.setSelectedParses(selectedParses);
+            return parseResp;
+        }).collect(Collectors.toList());
+        return semanticParseInfoList;
+    }
+
 }

chat/server/src/main/java/com/tencent/supersonic/chat/server/persistence/repository/impl/ChatRepositoryImpl.java

@@ -1,14 +1,15 @@
 package com.tencent.supersonic.chat.server.persistence.repository.impl;
 
-import com.tencent.supersonic.chat.server.persistence.dataobject.QueryDO;
 import com.tencent.supersonic.chat.server.persistence.dataobject.ChatDO;
+import com.tencent.supersonic.chat.server.persistence.dataobject.QueryDO;
 import com.tencent.supersonic.chat.server.persistence.mapper.ChatMapper;
 import com.tencent.supersonic.chat.server.persistence.repository.ChatRepository;
-import java.util.List;
 import lombok.extern.slf4j.Slf4j;
 import org.springframework.context.annotation.Primary;
 import org.springframework.stereotype.Repository;
 
+import java.util.List;
+
 @Repository
 @Primary
 @Slf4j
@@ -21,8 +22,9 @@ public class ChatRepositoryImpl implements ChatRepository {
     }
 
     @Override
-    public boolean createChat(ChatDO chatDO) {
-        return chatMapper.createChat(chatDO);
+    public Long createChat(ChatDO chatDO) {
+        chatMapper.createChat(chatDO);
+        return chatDO.getChatId();
     }
 
     @Override

chat/server/src/main/java/com/tencent/supersonic/chat/server/plugin/build/PluginSemanticQuery.java

@@ -11,9 +11,11 @@ import com.tencent.supersonic.headless.api.pojo.response.QueryResult;
 import lombok.extern.slf4j.Slf4j;
 import org.springframework.util.CollectionUtils;
 
+import java.util.Comparator;
 import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
+import java.util.stream.Collectors;
 
 @Slf4j
 public abstract class PluginSemanticQuery {
@@ -41,7 +43,10 @@ public abstract class PluginSemanticQuery {
     protected Map<String, Object> getElementMap(PluginParseResult pluginParseResult) {
         Map<String, Object> elementValueMap = new HashMap<>();
         Map<Long, Object> filterValueMap = getFilterMap(pluginParseResult);
-        List<SchemaElementMatch> schemaElementMatchList = parseInfo.getElementMatches();
+        List<SchemaElementMatch> schemaElementMatchList = parseInfo.getElementMatches()
+                .stream().filter(schemaElementMatch -> schemaElementMatch.getFrequency() != null)
+                .sorted(Comparator.comparingLong(SchemaElementMatch::getFrequency).reversed())
+                .collect(Collectors.toList());
         if (!CollectionUtils.isEmpty(schemaElementMatchList)) {
             schemaElementMatchList.stream().filter(schemaElementMatch ->
                             SchemaElementType.VALUE.equals(schemaElementMatch.getElement().getType())

chat/server/src/main/java/com/tencent/supersonic/chat/server/plugin/build/webservice/WebServiceQuery.java

@@ -55,8 +55,13 @@ public class WebServiceQuery extends PluginSemanticQuery {
         log.info("webServiceResponse result:{}", JsonUtil.toString(object));
         try {
             Map<String, Object> data = JsonUtil.toMap(JsonUtil.toString(object), String.class, Object.class);
-            queryResult.setQueryResults((List<Map<String, Object>>) data.get("resultList"));
-            queryResult.setQueryColumns((List<QueryColumn>) data.get("columns"));
+            if (data.get("resultList") != null) {
+                queryResult.setQueryResults((List<Map<String, Object>>) data.get("resultList"));
+            }
+            if (data.get("columns") != null) {
+                queryResult.setQueryColumns((List<QueryColumn>) data.get("columns"));
+            }
+            queryResult.setTextResult(String.valueOf(data.get("textInfo")));
             queryResult.setQueryState(QueryState.SUCCESS);
         } catch (Exception e) {
             log.info("webServiceResponse result has an exception:{}", e.getMessage());

chat/server/src/main/java/com/tencent/supersonic/chat/server/plugin/recognize/PluginRecognizer.java

@@ -86,6 +86,7 @@ public abstract class PluginRecognizer {
         properties.put("name", plugin.getName());
         semanticParseInfo.setProperties(properties);
         semanticParseInfo.setScore(distance);
+        semanticParseInfo.setTextInfo(String.format("将由插件工具**%s**来解答", plugin.getName()));
         fillSemanticParseInfo(semanticParseInfo);
         return semanticParseInfo;
     }

chat/server/src/main/java/com/tencent/supersonic/chat/server/plugin/recognize/embedding/EmbeddingRecallRecognizer.java

@@ -11,7 +11,7 @@ import com.tencent.supersonic.common.config.EmbeddingConfig;
 import com.tencent.supersonic.common.util.ContextUtils;
 import com.tencent.supersonic.common.util.embedding.Retrieval;
 import com.tencent.supersonic.common.util.embedding.RetrieveQueryResult;
-import com.tencent.supersonic.headless.core.chat.parser.PythonLLMProxy;
+import com.tencent.supersonic.headless.core.chat.parser.llm.PythonLLMProxy;
 import com.tencent.supersonic.headless.core.utils.ComponentFactory;
 import lombok.extern.slf4j.Slf4j;
 import org.apache.commons.lang3.StringUtils;

chat/server/src/main/java/com/tencent/supersonic/chat/server/rest/AgentController.java

@@ -5,6 +5,7 @@ import com.tencent.supersonic.auth.api.authentication.utils.UserHolder;
 import com.tencent.supersonic.chat.server.agent.Agent;
 import com.tencent.supersonic.chat.server.agent.AgentToolType;
 import com.tencent.supersonic.chat.server.service.AgentService;
+import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.web.bind.annotation.DeleteMapping;
 import org.springframework.web.bind.annotation.PathVariable;
 import org.springframework.web.bind.annotation.PostMapping;
@@ -21,12 +22,9 @@ import java.util.Map;
 @RequestMapping({"/api/chat/agent", "/openapi/chat/agent"})
 public class AgentController {
 
+    @Autowired
     private AgentService agentService;
 
-    public AgentController(AgentService agentService) {
-        this.agentService = agentService;
-    }
-
     @PostMapping
     public boolean createAgent(@RequestBody Agent agent,
                                 HttpServletRequest httpServletRequest,

chat/server/src/main/java/com/tencent/supersonic/chat/server/rest/ChatController.java

@@ -8,6 +8,7 @@ import com.tencent.supersonic.chat.api.pojo.response.QueryResp;
 import com.tencent.supersonic.chat.api.pojo.response.ShowCaseResp;
 import com.tencent.supersonic.chat.server.persistence.dataobject.ChatDO;
 import com.tencent.supersonic.chat.server.service.ChatManageService;
+import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.web.bind.annotation.GetMapping;
 import org.springframework.web.bind.annotation.PathVariable;
 import org.springframework.web.bind.annotation.PostMapping;
@@ -24,17 +25,15 @@ import java.util.List;
 @RequestMapping({"/api/chat/manage", "/openapi/chat/manage"})
 public class ChatController {
 
-    private final ChatManageService chatService;
-
-    public ChatController(ChatManageService chatService) {
-        this.chatService = chatService;
-    }
+    @Autowired
+    private ChatManageService chatService;
 
     @PostMapping("/save")
     public Boolean save(@RequestParam(value = "chatName") String chatName,
                         @RequestParam(value = "agentId", required = false) Integer agentId,
             HttpServletRequest request, HttpServletResponse response) {
-        return chatService.addChat(UserHolder.findUser(request, response), chatName, agentId);
+        chatService.addChat(UserHolder.findUser(request, response), chatName, agentId);
+        return true;
     }
 
     @GetMapping("/getAll")

chat/server/src/main/java/com/tencent/supersonic/chat/server/rest/ChatQueryController.java

@@ -1,13 +1,19 @@
 package com.tencent.supersonic.chat.server.rest;
 
 
+import com.tencent.supersonic.auth.api.authentication.pojo.User;
 import com.tencent.supersonic.auth.api.authentication.utils.UserHolder;
 import com.tencent.supersonic.chat.api.pojo.request.ChatExecuteReq;
 import com.tencent.supersonic.chat.api.pojo.request.ChatParseReq;
 import com.tencent.supersonic.chat.api.pojo.request.ChatQueryDataReq;
 import com.tencent.supersonic.chat.server.service.ChatService;
+import com.tencent.supersonic.common.pojo.exception.InvalidArgumentException;
+import com.tencent.supersonic.headless.api.pojo.SemanticParseInfo;
 import com.tencent.supersonic.headless.api.pojo.request.DimensionValueReq;
 import com.tencent.supersonic.headless.api.pojo.request.QueryReq;
+import com.tencent.supersonic.headless.api.pojo.response.ParseResp;
+import org.apache.commons.collections4.CollectionUtils;
+import org.springframework.beans.BeanUtils;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.web.bind.annotation.PostMapping;
 import org.springframework.web.bind.annotation.RequestBody;
@@ -35,13 +41,6 @@ public class ChatQueryController {
         return chatService.search(chatParseReq);
     }
 
-    @PostMapping("map")
-    public Object map(@RequestBody ChatParseReq chatParseReq,
-                        HttpServletRequest request, HttpServletResponse response) throws Exception {
-        chatParseReq.setUser(UserHolder.findUser(request, response));
-        return chatService.performMapping(chatParseReq);
-    }
-
     @PostMapping("parse")
     public Object parse(@RequestBody ChatParseReq chatParseReq,
                         HttpServletRequest request, HttpServletResponse response) throws Exception {
@@ -57,6 +56,25 @@ public class ChatQueryController {
         return chatService.performExecution(chatExecuteReq);
     }
 
+    @PostMapping("/")
+    public Object query(@RequestBody ChatParseReq chatParseReq,
+                        HttpServletRequest request, HttpServletResponse response)
+            throws Exception {
+        User user = UserHolder.findUser(request, response);
+        chatParseReq.setUser(user);
+        ParseResp parseResp = chatService.performParsing(chatParseReq);
+
+        if (CollectionUtils.isEmpty(parseResp.getSelectedParses())) {
+            throw new InvalidArgumentException("parser error,no selectedParses");
+        }
+        SemanticParseInfo semanticParseInfo = parseResp.getSelectedParses().get(0);
+        ChatExecuteReq chatExecuteReq = ChatExecuteReq.builder().build();
+        BeanUtils.copyProperties(chatParseReq, chatExecuteReq);
+        chatExecuteReq.setQueryId(parseResp.getQueryId());
+        chatExecuteReq.setParseId(semanticParseInfo.getId());
+        return chatService.performExecution(chatExecuteReq);
+    }
+
     @PostMapping("queryContext")
     public Object queryContext(@RequestBody QueryReq queryCtx,
                                HttpServletRequest request, HttpServletResponse response) {

chat/server/src/main/java/com/tencent/supersonic/chat/server/service/AgentService.java

@@ -8,7 +8,7 @@ public interface AgentService {
 
     List<Agent> getAgents();
 
-    void createAgent(Agent agent, User user);
+    Integer createAgent(Agent agent, User user);
 
     void updateAgent(Agent agent, User user);
 

chat/server/src/main/java/com/tencent/supersonic/chat/server/service/ChatManageService.java

@@ -17,7 +17,7 @@ import com.tencent.supersonic.headless.api.pojo.response.QueryResult;
 import java.util.List;
 
 public interface ChatManageService {
-    Boolean addChat(User user, String chatName, Integer agentId);
+    Long addChat(User user, String chatName, Integer agentId);
 
     List<ChatDO> getAll(String userName, Integer agentId);
 

chat/server/src/main/java/com/tencent/supersonic/chat/server/service/ChatService.java

@@ -6,7 +6,6 @@ import com.tencent.supersonic.chat.api.pojo.request.ChatParseReq;
 import com.tencent.supersonic.chat.api.pojo.request.ChatQueryDataReq;
 import com.tencent.supersonic.headless.api.pojo.SemanticParseInfo;
 import com.tencent.supersonic.headless.api.pojo.request.DimensionValueReq;
-import com.tencent.supersonic.headless.api.pojo.response.MapResp;
 import com.tencent.supersonic.headless.api.pojo.response.ParseResp;
 import com.tencent.supersonic.headless.api.pojo.response.QueryResult;
 import com.tencent.supersonic.headless.api.pojo.response.SearchResult;
@@ -17,8 +16,6 @@ public interface ChatService {
 
     List<SearchResult> search(ChatParseReq chatParseReq);
 
-    MapResp performMapping(ChatParseReq chatParseReq);
-
     ParseResp performParsing(ChatParseReq chatParseReq);
 
     QueryResult performExecution(ChatExecuteReq chatExecuteReq) throws Exception;

chat/server/src/main/java/com/tencent/supersonic/chat/server/service/impl/AgentServiceImpl.java

@@ -1,25 +1,22 @@
 package com.tencent.supersonic.chat.server.service.impl;
 
-import com.alibaba.fastjson.JSONObject;
+import com.baomidou.mybatisplus.extension.service.impl.ServiceImpl;
 import com.tencent.supersonic.auth.api.authentication.pojo.User;
 import com.tencent.supersonic.chat.server.agent.Agent;
+import com.tencent.supersonic.chat.server.agent.MultiTurnConfig;
 import com.tencent.supersonic.chat.server.persistence.dataobject.AgentDO;
-import com.tencent.supersonic.chat.server.persistence.repository.AgentRepository;
+import com.tencent.supersonic.chat.server.persistence.mapper.AgentDOMapper;
 import com.tencent.supersonic.chat.server.service.AgentService;
-import java.util.Date;
-import java.util.List;
-import java.util.stream.Collectors;
+import com.tencent.supersonic.common.util.JsonUtil;
+import com.tencent.supersonic.headless.api.pojo.LLMConfig;
 import org.springframework.beans.BeanUtils;
 import org.springframework.stereotype.Service;
+import java.util.List;
+import java.util.stream.Collectors;
 
 @Service
-public class AgentServiceImpl implements AgentService {
-
-    private AgentRepository agentRepository;
-
-    public AgentServiceImpl(AgentRepository agentRepository) {
-        this.agentRepository = agentRepository;
-    }
+public class AgentServiceImpl extends ServiceImpl<AgentDOMapper, AgentDO>
+        implements AgentService {
 
     @Override
     public List<Agent> getAgents() {
@@ -28,13 +25,17 @@ public class AgentServiceImpl implements AgentService {
     }
 
     @Override
-    public void createAgent(Agent agent, User user) {
-        agentRepository.createAgent(convert(agent, user));
+    public Integer createAgent(Agent agent, User user) {
+        agent.createdBy(user.getName());
+        AgentDO agentDO = convert(agent);
+        save(agentDO);
+        return agentDO.getId();
     }
 
     @Override
     public void updateAgent(Agent agent, User user) {
-        agentRepository.updateAgent(convert(agent, user));
+        agent.updatedBy(user.getName());
+        updateById(convert(agent));
     }
 
     @Override
@@ -42,16 +43,16 @@ public class AgentServiceImpl implements AgentService {
         if (id == null) {
             return null;
         }
-        return convert(agentRepository.getAgent(id));
+        return convert(getById(id));
     }
 
     @Override
     public void deleteAgent(Integer id) {
-        agentRepository.deleteAgent(id);
+        removeById(id);
     }
 
     private List<AgentDO> getAgentDOList() {
-        return agentRepository.getAgents();
+        return list();
     }
 
     private Agent convert(AgentDO agentDO) {
@@ -61,19 +62,19 @@ public class AgentServiceImpl implements AgentService {
         Agent agent = new Agent();
         BeanUtils.copyProperties(agentDO, agent);
         agent.setAgentConfig(agentDO.getConfig());
-        agent.setExamples(JSONObject.parseArray(agentDO.getExamples(), String.class));
+        agent.setExamples(JsonUtil.toList(agentDO.getExamples(), String.class));
+        agent.setLlmConfig(JsonUtil.toObject(agentDO.getLlmConfig(), LLMConfig.class));
+        agent.setMultiTurnConfig(JsonUtil.toObject(agentDO.getMultiTurnConfig(), MultiTurnConfig.class));
         return agent;
     }
 
-    private AgentDO convert(Agent agent, User user) {
+    private AgentDO convert(Agent agent) {
         AgentDO agentDO = new AgentDO();
         BeanUtils.copyProperties(agent, agentDO);
         agentDO.setConfig(agent.getAgentConfig());
-        agentDO.setExamples(JSONObject.toJSONString(agent.getExamples()));
-        agentDO.setCreatedAt(new Date());
-        agentDO.setCreatedBy(user.getName());
-        agentDO.setUpdatedAt(new Date());
-        agentDO.setUpdatedBy(user.getName());
+        agentDO.setExamples(JsonUtil.toString(agent.getExamples()));
+        agentDO.setLlmConfig(JsonUtil.toString(agent.getLlmConfig()));
+        agentDO.setMultiTurnConfig(JsonUtil.toString(agent.getMultiTurnConfig()));
         if (agentDO.getStatus() == null) {
             agentDO.setStatus(1);
         }

chat/server/src/main/java/com/tencent/supersonic/chat/server/service/impl/ChatManageServiceImpl.java

@@ -43,7 +43,7 @@ public class ChatManageServiceImpl implements ChatManageService {
     private ChatQueryRepository chatQueryRepository;
 
     @Override
-    public Boolean addChat(User user, String chatName, Integer agentId) {
+    public Long addChat(User user, String chatName, Integer agentId) {
         ChatDO chatDO = new ChatDO();
         chatDO.setChatName(chatName);
         chatDO.setCreator(user.getName());

chat/server/src/main/java/com/tencent/supersonic/chat/server/service/impl/ChatServiceImpl.java

@@ -34,6 +34,7 @@ import com.tencent.supersonic.headless.server.service.SearchService;
 import lombok.extern.slf4j.Slf4j;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.stereotype.Service;
+
 import java.util.List;
 
 
@@ -61,11 +62,6 @@ public class ChatServiceImpl implements ChatService {
         return searchService.search(queryReq);
     }
 
-    @Override
-    public MapResp performMapping(ChatParseReq chatParseReq) {
-        return getMapResp(chatParseReq);
-    }
-
     @Override
     public ParseResp performParsing(ChatParseReq chatParseReq) {
         ParseResp parseResp = new ParseResp(chatParseReq.getChatId(), chatParseReq.getQueryText());
@@ -77,6 +73,8 @@ public class ChatServiceImpl implements ChatService {
         for (ParseResultProcessor processor : parseResultProcessors) {
             processor.process(chatParseContext, parseResp);
         }
+        chatParseReq.setQueryText(chatParseContext.getQueryText());
+        parseResp.setQueryText(chatParseContext.getQueryText());
         chatManageService.batchAddParse(chatParseReq, parseResp);
         return parseResp;
     }
@@ -110,16 +108,6 @@ public class ChatServiceImpl implements ChatService {
         return chatParseContext;
     }
 
-    private MapResp getMapResp(ChatParseReq chatParseReq) {
-        ChatParseContext chatParseContext = new ChatParseContext();
-        BeanMapper.mapper(chatParseReq, chatParseContext);
-        AgentService agentService = ContextUtils.getBean(AgentService.class);
-        Agent agent = agentService.getAgent(chatParseReq.getAgentId());
-        chatParseContext.setAgent(agent);
-        QueryReq queryReq = QueryReqConverter.buildText2SqlQueryReq(chatParseContext);
-        return chatQueryService.performMapping(queryReq);
-    }
-
     private ChatExecuteContext buildExecuteContext(ChatExecuteReq chatExecuteReq) {
         ChatExecuteContext chatExecuteContext = new ChatExecuteContext();
         BeanMapper.mapper(chatExecuteReq, chatExecuteContext);

chat/server/src/main/java/com/tencent/supersonic/chat/server/util/QueryReqConverter.java

@@ -2,8 +2,12 @@ package com.tencent.supersonic.chat.server.util;
 
 import com.tencent.supersonic.chat.server.agent.Agent;
 import com.tencent.supersonic.chat.server.pojo.ChatParseContext;
+import com.tencent.supersonic.common.pojo.enums.Text2SQLType;
 import com.tencent.supersonic.common.util.BeanMapper;
 import com.tencent.supersonic.headless.api.pojo.request.QueryReq;
+import org.apache.commons.collections.MapUtils;
+
+import java.util.Objects;
 
 public class QueryReqConverter {
 
@@ -14,10 +18,19 @@ public class QueryReqConverter {
         if (agent == null) {
             return queryReq;
         }
-        if (agent.containsLLMParserTool()) {
-            queryReq.setEnableLLM(true);
+        if (agent.containsLLMParserTool() && agent.containsRuleTool()) {
+            queryReq.setText2SQLType(Text2SQLType.RULE_AND_LLM);
+        } else if (agent.containsLLMParserTool()) {
+            queryReq.setText2SQLType(Text2SQLType.ONLY_LLM);
+        } else if (agent.containsRuleTool()) {
+            queryReq.setText2SQLType(Text2SQLType.ONLY_RULE);
         }
         queryReq.setDataSetIds(agent.getDataSetIds());
+        if (Objects.nonNull(queryReq.getMapInfo())
+                && MapUtils.isNotEmpty(queryReq.getMapInfo().getDataSetElementMatches())) {
+            queryReq.setMapInfo(queryReq.getMapInfo());
+        }
+        queryReq.setLlmConfig(agent.getLlmConfig());
         return queryReq;
     }
 

chat/server/src/main/java/com/tencent/supersonic/chat/server/util/ResultFormatter.java

@@ -0,0 +1,39 @@
+package com.tencent.supersonic.chat.server.util;
+
+import com.tencent.supersonic.common.pojo.QueryColumn;
+import org.springframework.util.CollectionUtils;
+
+import java.util.List;
+import java.util.Map;
+
+public class ResultFormatter {
+
+    public static String transform2TextNew(List<QueryColumn> queryColumns, List<Map<String, Object>> queryResults) {
+        if (CollectionUtils.isEmpty(queryColumns)) {
+            return "";
+        }
+        StringBuilder table = new StringBuilder();
+        for (QueryColumn column : queryColumns) {
+            String columnName = column.getName();
+            table.append("| ").append(columnName).append(" ");
+        }
+        table.append("|\n");
+        for (int i = 0; i < queryColumns.size(); i++) {
+            table.append("|:---:");
+        }
+        table.append("|\n");
+        if (queryResults == null) {
+            return table.toString();
+        }
+        for (Map<String, Object> row : queryResults) {
+            for (QueryColumn column : queryColumns) {
+                String columnKey = column.getNameEn();
+                Object value = row.get(columnKey);
+                table.append("| ").append(value != null ? value.toString() : "").append(" ");
+            }
+            table.append("|\n");
+        }
+        return table.toString();
+    }
+
+}

chat/server/src/main/resources/mapper/AgentDOMapper.xml

@@ -1,303 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
-<mapper namespace="com.tencent.supersonic.chat.server.persistence.mapper.AgentDOMapper">
-  <resultMap id="BaseResultMap" type="com.tencent.supersonic.chat.server.persistence.dataobject.AgentDO">
-    <id column="id" jdbcType="INTEGER" property="id" />
-    <result column="name" jdbcType="VARCHAR" property="name" />
-    <result column="description" jdbcType="VARCHAR" property="description" />
-    <result column="status" jdbcType="INTEGER" property="status" />
-    <result column="examples" jdbcType="VARCHAR" property="examples" />
-    <result column="config" jdbcType="VARCHAR" property="config" />
-    <result column="created_by" jdbcType="VARCHAR" property="createdBy" />
-    <result column="created_at" jdbcType="TIMESTAMP" property="createdAt" />
-    <result column="updated_by" jdbcType="VARCHAR" property="updatedBy" />
-    <result column="updated_at" jdbcType="TIMESTAMP" property="updatedAt" />
-    <result column="enable_search" jdbcType="INTEGER" property="enableSearch" />
-  </resultMap>
-  <sql id="Example_Where_Clause">
-    <where>
-      <foreach collection="oredCriteria" item="criteria" separator="or">
-        <if test="criteria.valid">
-          <trim prefix="(" prefixOverrides="and" suffix=")">
-            <foreach collection="criteria.criteria" item="criterion">
-              <choose>
-                <when test="criterion.noValue">
-                  and ${criterion.condition}
-                </when>
-                <when test="criterion.singleValue">
-                  and ${criterion.condition} #{criterion.value}
-                </when>
-                <when test="criterion.betweenValue">
-                  and ${criterion.condition} #{criterion.value} and #{criterion.secondValue}
-                </when>
-                <when test="criterion.listValue">
-                  and ${criterion.condition}
-                  <foreach close=")" collection="criterion.value" item="listItem" open="(" separator=",">
-                    #{listItem}
-                  </foreach>
-                </when>
-              </choose>
-            </foreach>
-          </trim>
-        </if>
-      </foreach>
-    </where>
-  </sql>
-  <sql id="Update_By_Example_Where_Clause">
-    <where>
-      <foreach collection="example.oredCriteria" item="criteria" separator="or">
-        <if test="criteria.valid">
-          <trim prefix="(" prefixOverrides="and" suffix=")">
-            <foreach collection="criteria.criteria" item="criterion">
-              <choose>
-                <when test="criterion.noValue">
-                  and ${criterion.condition}
-                </when>
-                <when test="criterion.singleValue">
-                  and ${criterion.condition} #{criterion.value}
-                </when>
-                <when test="criterion.betweenValue">
-                  and ${criterion.condition} #{criterion.value} and #{criterion.secondValue}
-                </when>
-                <when test="criterion.listValue">
-                  and ${criterion.condition}
-                  <foreach close=")" collection="criterion.value" item="listItem" open="(" separator=",">
-                    #{listItem}
-                  </foreach>
-                </when>
-              </choose>
-            </foreach>
-          </trim>
-        </if>
-      </foreach>
-    </where>
-  </sql>
-  <sql id="Base_Column_List">
-    id, name, description, status, examples, config, created_by, created_at, updated_by, 
-    updated_at, enable_search
-  </sql>
-  <select id="selectByExample" parameterType="com.tencent.supersonic.chat.server.persistence.dataobject.AgentDOExample" resultMap="BaseResultMap">
-    select
-    <if test="distinct">
-      distinct
-    </if>
-    <include refid="Base_Column_List" />
-    from s2_agent
-    <if test="_parameter != null">
-      <include refid="Example_Where_Clause" />
-    </if>
-    <if test="orderByClause != null">
-      order by ${orderByClause}
-    </if>
-    <if test="limitStart != null and limitStart>=0">
-      limit #{limitStart} , #{limitEnd}
-    </if>
-  </select>
-  <select id="selectByPrimaryKey" parameterType="java.lang.Integer" resultMap="BaseResultMap">
-    select 
-    <include refid="Base_Column_List" />
-    from s2_agent
-    where id = #{id,jdbcType=INTEGER}
-  </select>
-  <delete id="deleteByPrimaryKey" parameterType="java.lang.Integer">
-    delete from s2_agent
-    where id = #{id,jdbcType=INTEGER}
-  </delete>
-  <insert id="insert" parameterType="com.tencent.supersonic.chat.server.persistence.dataobject.AgentDO">
-    insert into s2_agent (id, name, description, 
-      status, examples, config, 
-      created_by, created_at, updated_by, 
-      updated_at, enable_search)
-    values (#{id,jdbcType=INTEGER}, #{name,jdbcType=VARCHAR}, #{description,jdbcType=VARCHAR}, 
-      #{status,jdbcType=INTEGER}, #{examples,jdbcType=VARCHAR}, #{config,jdbcType=VARCHAR}, 
-      #{createdBy,jdbcType=VARCHAR}, #{createdAt,jdbcType=TIMESTAMP}, #{updatedBy,jdbcType=VARCHAR}, 
-      #{updatedAt,jdbcType=TIMESTAMP}, #{enableSearch,jdbcType=INTEGER})
-  </insert>
-  <insert id="insertSelective" parameterType="com.tencent.supersonic.chat.server.persistence.dataobject.AgentDO">
-    insert into s2_agent
-    <trim prefix="(" suffix=")" suffixOverrides=",">
-      <if test="id != null">
-        id,
-      </if>
-      <if test="name != null">
-        name,
-      </if>
-      <if test="description != null">
-        description,
-      </if>
-      <if test="status != null">
-        status,
-      </if>
-      <if test="examples != null">
-        examples,
-      </if>
-      <if test="config != null">
-        config,
-      </if>
-      <if test="createdBy != null">
-        created_by,
-      </if>
-      <if test="createdAt != null">
-        created_at,
-      </if>
-      <if test="updatedBy != null">
-        updated_by,
-      </if>
-      <if test="updatedAt != null">
-        updated_at,
-      </if>
-      <if test="enableSearch != null">
-        enable_search,
-      </if>
-    </trim>
-    <trim prefix="values (" suffix=")" suffixOverrides=",">
-      <if test="id != null">
-        #{id,jdbcType=INTEGER},
-      </if>
-      <if test="name != null">
-        #{name,jdbcType=VARCHAR},
-      </if>
-      <if test="description != null">
-        #{description,jdbcType=VARCHAR},
-      </if>
-      <if test="status != null">
-        #{status,jdbcType=INTEGER},
-      </if>
-      <if test="examples != null">
-        #{examples,jdbcType=VARCHAR},
-      </if>
-      <if test="config != null">
-        #{config,jdbcType=VARCHAR},
-      </if>
-      <if test="createdBy != null">
-        #{createdBy,jdbcType=VARCHAR},
-      </if>
-      <if test="createdAt != null">
-        #{createdAt,jdbcType=TIMESTAMP},
-      </if>
-      <if test="updatedBy != null">
-        #{updatedBy,jdbcType=VARCHAR},
-      </if>
-      <if test="updatedAt != null">
-        #{updatedAt,jdbcType=TIMESTAMP},
-      </if>
-      <if test="enableSearch != null">
-        #{enableSearch,jdbcType=INTEGER},
-      </if>
-    </trim>
-  </insert>
-  <select id="countByExample" parameterType="com.tencent.supersonic.chat.server.persistence.dataobject.AgentDOExample" resultType="java.lang.Long">
-    select count(*) from s2_agent
-    <if test="_parameter != null">
-      <include refid="Example_Where_Clause" />
-    </if>
-  </select>
-  <update id="updateByExampleSelective" parameterType="map">
-    update s2_agent
-    <set>
-      <if test="record.id != null">
-        id = #{record.id,jdbcType=INTEGER},
-      </if>
-      <if test="record.name != null">
-        name = #{record.name,jdbcType=VARCHAR},
-      </if>
-      <if test="record.description != null">
-        description = #{record.description,jdbcType=VARCHAR},
-      </if>
-      <if test="record.status != null">
-        status = #{record.status,jdbcType=INTEGER},
-      </if>
-      <if test="record.examples != null">
-        examples = #{record.examples,jdbcType=VARCHAR},
-      </if>
-      <if test="record.config != null">
-        config = #{record.config,jdbcType=VARCHAR},
-      </if>
-      <if test="record.createdBy != null">
-        created_by = #{record.createdBy,jdbcType=VARCHAR},
-      </if>
-      <if test="record.createdAt != null">
-        created_at = #{record.createdAt,jdbcType=TIMESTAMP},
-      </if>
-      <if test="record.updatedBy != null">
-        updated_by = #{record.updatedBy,jdbcType=VARCHAR},
-      </if>
-      <if test="record.updatedAt != null">
-        updated_at = #{record.updatedAt,jdbcType=TIMESTAMP},
-      </if>
-      <if test="record.enableSearch != null">
-        enable_search = #{record.enableSearch,jdbcType=INTEGER},
-      </if>
-    </set>
-    <if test="_parameter != null">
-      <include refid="Update_By_Example_Where_Clause" />
-    </if>
-  </update>
-  <update id="updateByExample" parameterType="map">
-    update s2_agent
-    set id = #{record.id,jdbcType=INTEGER},
-      name = #{record.name,jdbcType=VARCHAR},
-      description = #{record.description,jdbcType=VARCHAR},
-      status = #{record.status,jdbcType=INTEGER},
-      examples = #{record.examples,jdbcType=VARCHAR},
-      config = #{record.config,jdbcType=VARCHAR},
-      created_by = #{record.createdBy,jdbcType=VARCHAR},
-      created_at = #{record.createdAt,jdbcType=TIMESTAMP},
-      updated_by = #{record.updatedBy,jdbcType=VARCHAR},
-      updated_at = #{record.updatedAt,jdbcType=TIMESTAMP},
-      enable_search = #{record.enableSearch,jdbcType=INTEGER}
-    <if test="_parameter != null">
-      <include refid="Update_By_Example_Where_Clause" />
-    </if>
-  </update>
-  <update id="updateByPrimaryKeySelective" parameterType="com.tencent.supersonic.chat.server.persistence.dataobject.AgentDO">
-    update s2_agent
-    <set>
-      <if test="name != null">
-        name = #{name,jdbcType=VARCHAR},
-      </if>
-      <if test="description != null">
-        description = #{description,jdbcType=VARCHAR},
-      </if>
-      <if test="status != null">
-        status = #{status,jdbcType=INTEGER},
-      </if>
-      <if test="examples != null">
-        examples = #{examples,jdbcType=VARCHAR},
-      </if>
-      <if test="config != null">
-        config = #{config,jdbcType=VARCHAR},
-      </if>
-      <if test="createdBy != null">
-        created_by = #{createdBy,jdbcType=VARCHAR},
-      </if>
-      <if test="createdAt != null">
-        created_at = #{createdAt,jdbcType=TIMESTAMP},
-      </if>
-      <if test="updatedBy != null">
-        updated_by = #{updatedBy,jdbcType=VARCHAR},
-      </if>
-      <if test="updatedAt != null">
-        updated_at = #{updatedAt,jdbcType=TIMESTAMP},
-      </if>
-      <if test="enableSearch != null">
-        enable_search = #{enableSearch,jdbcType=INTEGER},
-      </if>
-    </set>
-    where id = #{id,jdbcType=INTEGER}
-  </update>
-  <update id="updateByPrimaryKey" parameterType="com.tencent.supersonic.chat.server.persistence.dataobject.AgentDO">
-    update s2_agent
-    set name = #{name,jdbcType=VARCHAR},
-      description = #{description,jdbcType=VARCHAR},
-      status = #{status,jdbcType=INTEGER},
-      examples = #{examples,jdbcType=VARCHAR},
-      config = #{config,jdbcType=VARCHAR},
-      created_by = #{createdBy,jdbcType=VARCHAR},
-      created_at = #{createdAt,jdbcType=TIMESTAMP},
-      updated_by = #{updatedBy,jdbcType=VARCHAR},
-      updated_at = #{updatedAt,jdbcType=TIMESTAMP},
-      enable_search = #{enableSearch,jdbcType=INTEGER}
-    where id = #{id,jdbcType=INTEGER}
-  </update>
-</mapper>

chat/server/src/main/resources/mapper/ChatMapper.xml

@@ -49,7 +49,7 @@
         where chat_id = #{chatId}
     </update>
 
-    <insert id="createChat" parameterType="com.tencent.supersonic.chat.server.persistence.dataobject.ChatDO">
+    <insert id="createChat" parameterType="com.tencent.supersonic.chat.server.persistence.dataobject.ChatDO"  useGeneratedKeys="true" keyProperty="chatId">
         insert into s2_chat
         (agent_id, chat_name, create_time, last_time, creator, last_question, is_delete, is_top)
         values (#{agentId}, #{chatName}, #{createTime}, #{lastTime}, #{creator}, #{lastQuestion}, #{isDelete}, #{isTop})

chat/server/src/main/resources/mapper/ChatParseMapper.xml

@@ -46,4 +46,10 @@
         </foreach>
     </select>
 
+    <select id="getContextualParseInfo" resultMap="ChatParse">
+        select *
+        from s2_chat_parse
+        where chat_id = #{chatId}  order by question_id desc  limit 10
+    </select>
+
 </mapper>

common/pom.xml

@@ -111,20 +111,6 @@
             <version>${transmittable.version}</version>
             <scope>compile</scope>
         </dependency>
-
-
-        <dependency>
-            <groupId>com.github.plexpt</groupId>
-            <artifactId>chatgpt</artifactId>
-            <version>4.1.2</version>
-            <exclusions>
-                <exclusion>
-                    <groupId>com.knuddels</groupId>
-                    <artifactId>jtokkit</artifactId>
-                </exclusion>
-            </exclusions>
-        </dependency>
-
         <dependency>
             <groupId>com.github.pagehelper</groupId>
             <artifactId>pagehelper-spring-boot-starter</artifactId>
@@ -172,6 +158,10 @@
             <groupId>dev.langchain4j</groupId>
             <artifactId>langchain4j-open-ai</artifactId>
         </dependency>
+        <dependency>
+            <groupId>dev.langchain4j</groupId>
+            <artifactId>langchain4j-local-ai</artifactId>
+        </dependency>
         <dependency>
             <groupId>dev.langchain4j</groupId>
             <artifactId>langchain4j</artifactId>

common/src/main/java/com/tencent/supersonic/common/config/DataBaseConfig.java

@@ -13,6 +13,7 @@ import javax.sql.DataSource;
 public class DataBaseConfig {
 
     @Bean("h2")
+    @Primary
     @ConfigurationProperties("spring.datasource")
     public DataSource dataSource() {
         return new DruidDataSource();

common/src/main/java/com/tencent/supersonic/common/config/EmbeddingConfig.java

@@ -8,37 +8,37 @@ import org.springframework.context.annotation.Configuration;
 @Data
 public class EmbeddingConfig {
 
-    @Value("${embedding.url:}")
+    @Value("${s2.embedding.url:}")
     private String url;
 
-    @Value("${embedding.recognize.path:/preset_query_retrival}")
+    @Value("${s2.embedding.persistent.path:/tmp}")
+    private String embeddingStorePersistentPath;
+
+    @Value("${s2.embedding.recognize.path:/preset_query_retrival}")
     private String recognizePath;
 
-    @Value("${embedding.preset.collection:preset_query_collection}")
+    @Value("${s2.embedding.preset.collection:preset_query_collection}")
     private String presetCollection;
 
-    @Value("${embedding.meta.collection:meta_collection}")
+    @Value("${s2.embedding.meta.collection:meta_collection}")
     private String metaCollectionName;
 
-    @Value("${embedding.nResult:1}")
+    @Value("${s2.embedding.nResult:1}")
     private int nResult;
 
-    @Value("${embedding.solved.query.collection:solved_query_collection}")
+    @Value("${s2.embedding.solved.query.collection:solved_query_collection}")
     private String solvedQueryCollection;
 
-    @Value("${embedding.solved.query.nResult:5}")
+    @Value("${s2.embedding.solved.query.nResult:5}")
     private int solvedQueryResultNum;
 
-    @Value("${embedding.metric.analyzeQuery.collection:solved_query_collection}")
+    @Value("${s2.embedding.metric.analyzeQuery.collection:solved_query_collection}")
     private String metricAnalyzeQueryCollection;
 
     @Value("${text2sql.collection.name:text2dsl_agent_collection}")
     private String text2sqlCollectionName;
 
-    @Value("${embedding.metric.analyzeQuery.nResult:5}")
+    @Value("${s2.embedding.metric.analyzeQuery.nResult:5}")
     private int metricAnalyzeQueryResultNum;
 
-    @Value("${inMemoryEmbeddingStore.persistent.path:/tmp}")
-    private String embeddingStorePersistentPath;
-
 }

common/src/main/java/com/tencent/supersonic/common/persistence/dataobject/SystemConfigDO.java

@@ -6,8 +6,8 @@ import com.baomidou.mybatisplus.annotation.TableName;
 import lombok.Data;
 
 @Data
-@TableName("s2_sys_parameter")
-public class SysParameterDO {
+@TableName("s2_system_config")
+public class SystemConfigDO {
     @TableId(type = IdType.AUTO)
     private Integer id;
 

common/src/main/java/com/tencent/supersonic/common/persistence/mapper/SystemConfigMapper.java

@@ -1,11 +1,11 @@
 package com.tencent.supersonic.common.persistence.mapper;
 
 import com.baomidou.mybatisplus.core.mapper.BaseMapper;
-import com.tencent.supersonic.common.persistence.dataobject.SysParameterDO;
+import com.tencent.supersonic.common.persistence.dataobject.SystemConfigDO;
 import org.apache.ibatis.annotations.Mapper;
 
 @Mapper
-public interface SysParameterMapper extends BaseMapper<SysParameterDO> {
+public interface SystemConfigMapper extends BaseMapper<SystemConfigDO> {
 
 
 }

common/src/main/java/com/tencent/supersonic/common/pojo/Parameter.java

@@ -3,6 +3,7 @@ package com.tencent.supersonic.common.pojo;
 import lombok.AllArgsConstructor;
 import lombok.Data;
 import lombok.NoArgsConstructor;
+import org.apache.commons.lang3.StringUtils;
 import java.util.List;
 
 @Data
@@ -10,19 +11,40 @@ import java.util.List;
 @NoArgsConstructor
 public class Parameter {
     private String name;
-    private String value;
+    private String defaultValue;
     private String comment;
     private String description;
     private String dataType;
     private String module;
+    private String value;
     private List<Object> candidateValues;
-    public Parameter(String name, String value, String comment, String description, String dataType, String module) {
+
+    public Parameter(String name, String defaultValue, String comment,
+                     String description, String dataType, String module) {
         this.name = name;
-        this.value = value;
+        this.defaultValue = defaultValue;
         this.comment = comment;
         this.description = description;
         this.dataType = dataType;
         this.module = module;
     }
 
+    public Parameter(String name, String defaultValue, String comment, String description,
+                     String dataType, String module, List<Object> candidateValues) {
+        this.name = name;
+        this.defaultValue = defaultValue;
+        this.comment = comment;
+        this.description = description;
+        this.dataType = dataType;
+        this.module = module;
+        this.candidateValues = candidateValues;
+    }
+
+    public String getValue() {
+        if (StringUtils.isBlank(value)) {
+            return defaultValue;
+        }
+        return value;
+    }
+
 }

common/src/main/java/com/tencent/supersonic/common/pojo/ParameterConfig.java

@@ -0,0 +1,49 @@
+package com.tencent.supersonic.common.pojo;
+
+import com.tencent.supersonic.common.service.SystemConfigService;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.core.env.Environment;
+import org.springframework.stereotype.Service;
+
+import java.util.List;
+
+@Service
+@Slf4j
+public abstract class ParameterConfig {
+
+    @Autowired
+    private SystemConfigService sysConfigService;
+
+    @Autowired
+    private Environment environment;
+
+    protected abstract List<Parameter> getSysParameters();
+
+    /**
+     * Parameter value will be derived by following orders:
+     * 1. `system config` set with user interface
+     * 2. `system property` set with application.yaml
+     * 3. `default value` set with parameter declaration
+     * @param parameter
+     * @return parameter value
+     */
+    public String getParameterValue(Parameter parameter) {
+        String paramName = parameter.getName();
+        String value = sysConfigService.getSystemConfig().getParameterByName(paramName);
+        try {
+            if (StringUtils.isBlank(value)) {
+                if (environment.containsProperty(paramName)) {
+                    value = environment.getProperty(paramName);
+                } else {
+                    value = parameter.getDefaultValue();
+                }
+            }
+        } catch (Exception e) {
+            log.error("Failed to get parameter value for {} with exception: {}", paramName, e);
+        }
+
+        return value;
+    }
+}

common/src/main/java/com/tencent/supersonic/common/pojo/QueryColumn.java

@@ -16,6 +16,7 @@ public class QueryColumn {
     private Boolean authorized = true;
     private String dataFormatType;
     private DataFormat dataFormat;
+    private String comment;
 
     public QueryColumn(String nameEn, String type) {
         this.type = type;

common/src/main/java/com/tencent/supersonic/common/pojo/SysParameter.java

@@ -1,106 +0,0 @@
-package com.tencent.supersonic.common.pojo;
-
-import com.google.common.collect.Lists;
-import lombok.Data;
-import org.apache.commons.lang3.StringUtils;
-import org.springframework.util.CollectionUtils;
-import java.util.Arrays;
-import java.util.List;
-import java.util.Map;
-import java.util.stream.Collectors;
-
-@Data
-public class SysParameter {
-
-    private Integer id;
-
-    private List<String> admins;
-
-    private List<Parameter> parameters;
-
-    public String getAdmin() {
-        if (CollectionUtils.isEmpty(admins)) {
-            return "";
-        }
-        return StringUtils.join(admins, ",");
-    }
-
-    public String getParameterByName(String name) {
-        if (StringUtils.isBlank(name)) {
-            return "";
-        }
-        Map<String, String> nameToValue = parameters.stream()
-                .collect(Collectors.toMap(Parameter::getName, Parameter::getValue, (k1, k2) -> k1));
-        return nameToValue.get(name);
-    }
-
-    public void setAdminList(String admin) {
-        if (StringUtils.isNotBlank(admin)) {
-            admins = Arrays.asList(admin.split(","));
-        } else {
-            admins = Lists.newArrayList();
-        }
-    }
-
-    public void init() {
-        parameters = Lists.newArrayList();
-        admins = Lists.newArrayList("admin");
-
-        //detect config
-        parameters.add(new Parameter("one.detection.size", "8",
-                "一次探测返回结果个数", "在每次探测后, 将前后缀匹配的结果合并, 并根据相似度阈值过滤后的结果个数",
-                "number", "Mapper相关配置"));
-        parameters.add(new Parameter("one.detection.max.size", "20",
-                "一次探测前后缀匹配结果返回个数", "单次前后缀匹配返回的结果个数", "number", "Mapper相关配置"));
-
-        //mapper config
-        parameters.add(new Parameter("metric.dimension.threshold", "0.3",
-                "指标名、维度名文本相似度阈值", "文本片段和匹配到的指标、维度名计算出来的编辑距离阈值, 若超出该阈值, 则舍弃",
-                "number", "Mapper相关配置"));
-        parameters.add(new Parameter("metric.dimension.min.threshold", "0.3",
-                "指标名、维度名最小文本相似度阈值",
-                "最小编辑距离阈值, 在FuzzyNameMapper中, 如果上面设定的编辑距离阈值的1/2大于该最小编辑距离, 则取上面设定阈值的1/2作为阈值, 否则取该阈值",
-                "number", "Mapper相关配置"));
-        parameters.add(new Parameter("dimension.value.threshold", "0.5",
-                "维度值最小文本相似度阈值", "文本片段和匹配到的维度值计算出来的编辑距离阈值, 若超出该阈值, 则舍弃",
-                "number", "Mapper相关配置"));
-
-        //embedding mapper config
-        parameters.add(new Parameter("embedding.mapper.word.min",
-                "4", "用于向量召回最小的文本长度", "为提高向量召回效率, 小于该长度的文本不进行向量语义召回", "number", "Mapper相关配置"));
-        parameters.add(new Parameter("embedding.mapper.word.max", "5",
-                "用于向量召回最大的文本长度", "为提高向量召回效率, 大于该长度的文本不进行向量语义召回", "number", "Mapper相关配置"));
-        parameters.add(new Parameter("embedding.mapper.batch", "50",
-                "批量向量召回文本请求个数", "每次进行向量语义召回的原始文本片段个数", "number", "Mapper相关配置"));
-        parameters.add(new Parameter("embedding.mapper.number", "5",
-                "批量向量召回文本返回结果个数", "每个文本进行向量语义召回的文本结果个数", "number", "Mapper相关配置"));
-        parameters.add(new Parameter("embedding.mapper.distance.threshold",
-                "0.01", "向量召回相似度阈值", "相似度大于该阈值的则舍弃", "number", "Mapper相关配置"));
-
-        //parser config
-        Parameter s2SQLParameter = new Parameter("s2SQL.generation", "TWO_PASS_AUTO_COT",
-                "S2SQL生成方式", "ONE_PASS_AUTO_COT: 通过思维链方式一步生成sql"
-                + "\nONE_PASS_AUTO_COT_SELF_CONSISTENCY: 通过思维链且投票方式一步生成sql"
-                + "\nTWO_PASS_AUTO_COT: 通过思维链方式两步生成sql"
-                + "\nTWO_PASS_AUTO_COT_SELF_CONSISTENCY: 通过思维链且投票方式两步生成sql", "list", "Parser相关配置");
-
-        s2SQLParameter.setCandidateValues(Lists.newArrayList("ONE_PASS_AUTO_COT", "ONE_PASS_AUTO_COT_SELF_CONSISTENCY",
-                "TWO_PASS_AUTO_COT", "TWO_PASS_AUTO_COT_SELF_CONSISTENCY"));
-        parameters.add(s2SQLParameter);
-        parameters.add(new Parameter("s2SQL.linking.value.switch", "true",
-                "是否将Mapper探测识别到的维度值提供给大模型", "为了数据安全考虑, 这里可进行开关选择",
-                "bool", "Parser相关配置"));
-        parameters.add(new Parameter("query.text.length.threshold", "10",
-                "用户输入文本长短阈值", "文本超过该阈值为长文本", "number", "Parser相关配置"));
-        parameters.add(new Parameter("short.text.threshold", "0.5",
-                "短文本匹配阈值", "由于请求大模型耗时较长, 因此如果有规则类型的Query得分达到阈值,则跳过大模型的调用,\n如果是短文本, 若query得分/文本长度>该阈值, 则跳过当前parser",
-                "number", "Parser相关配置"));
-        parameters.add(new Parameter("long.text.threshold", "0.8",
-                "长文本匹配阈值", "如果是长文本, 若query得分/文本长度>该阈值, 则跳过当前parser",
-                "number", "Parser相关配置"));
-        parameters.add(new Parameter("parse.show.count", "3",
-                "解析结果个数", "前端展示的解析个数",
-                "number", "Parser相关配置"));
-    }
-
-}

common/src/main/java/com/tencent/supersonic/common/pojo/SystemConfig.java

@@ -0,0 +1,77 @@
+package com.tencent.supersonic.common.pojo;
+
+import com.google.common.collect.Lists;
+import com.tencent.supersonic.common.util.ContextUtils;
+import lombok.Data;
+import org.apache.commons.lang3.StringUtils;
+import org.springframework.util.CollectionUtils;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+@Data
+public class SystemConfig {
+
+    private Integer id;
+
+    private List<String> admins;
+
+    private List<Parameter> parameters;
+
+    public String getAdmin() {
+        if (CollectionUtils.isEmpty(admins)) {
+            return "";
+        }
+        return StringUtils.join(admins, ",");
+    }
+
+    public String getParameterByName(String name) {
+        if (StringUtils.isBlank(name)) {
+            return "";
+        }
+        Map<String, String> nameToValue = getParameters().stream()
+                .collect(Collectors.toMap(Parameter::getName, Parameter::getValue, (k1, k2) -> k1));
+        return nameToValue.get(name);
+    }
+
+    public void setAdminList(String admin) {
+        if (StringUtils.isNotBlank(admin)) {
+            admins = Arrays.asList(admin.split(","));
+        } else {
+            admins = Lists.newArrayList();
+        }
+    }
+
+    public void init() {
+        parameters = buildDefaultParameters();
+        admins = Lists.newArrayList("admin");
+    }
+
+    private List<Parameter> buildDefaultParameters() {
+        List<Parameter> defaultParameters = Lists.newArrayList();
+        Collection<ParameterConfig> configurableParameters =
+                ContextUtils.getBeansOfType(ParameterConfig.class).values();
+        for (ParameterConfig configParameters : configurableParameters) {
+            defaultParameters.addAll(configParameters.getSysParameters());
+        }
+        return defaultParameters;
+    }
+
+    public List<Parameter> getParameters() {
+        List<Parameter> defaultParameters = buildDefaultParameters();
+        if (CollectionUtils.isEmpty(parameters)) {
+            return defaultParameters;
+        }
+        Map<String, String> parameterNameValueMap = parameters.stream()
+                .collect(Collectors.toMap(Parameter::getName, Parameter::getValue, (v1, v2) -> v2));
+        for (Parameter parameter : defaultParameters) {
+            parameter.setValue(parameterNameValueMap.getOrDefault(parameter.getName(),
+                    parameter.getDefaultValue()));
+        }
+        return defaultParameters;
+    }
+
+}

common/src/main/java/com/tencent/supersonic/common/pojo/enums/DictWordType.java

@@ -1,7 +1,7 @@
 package com.tencent.supersonic.common.pojo.enums;
 
-import java.util.Objects;
 import org.apache.commons.lang3.StringUtils;
+import java.util.Objects;
 
 /***
  * nature type
@@ -23,6 +23,8 @@ public enum DictWordType {
 
     TAG("tag"),
 
+    TERM("term"),
+
     SUFFIX("suffix");
 
     public static final String NATURE_SPILT = "_";

common/src/main/java/com/tencent/supersonic/common/pojo/enums/QueryType.java

@@ -11,13 +11,13 @@ public enum QueryType {
     /**
      * queries with tag-based entity targeting
      */
-    TAG,
+    DETAIL,
     /**
      * queries with ID-based entity selection
      */
     ID;
 
     public boolean isNativeAggQuery() {
-        return TAG.equals(this);
+        return DETAIL.equals(this);
     }
 }

common/src/main/java/com/tencent/supersonic/common/pojo/enums/S2ModelProvider.java

@@ -0,0 +1,9 @@
+package com.tencent.supersonic.common.pojo.enums;
+
+public enum S2ModelProvider {
+
+    OPEN_AI,
+    HUGGING_FACE,
+    LOCAL_AI,
+    IN_PROCESS
+}

common/src/main/java/com/tencent/supersonic/common/pojo/enums/Text2SQLType.java

@@ -0,0 +1,15 @@
+package com.tencent.supersonic.common.pojo.enums;
+
+public enum Text2SQLType {
+
+    ONLY_RULE, ONLY_LLM, RULE_AND_LLM;
+
+    public boolean enableRule() {
+        return this.equals(ONLY_RULE) || this.equals(RULE_AND_LLM);
+    }
+
+    public boolean enableLLM() {
+        return this.equals(ONLY_LLM) || this.equals(RULE_AND_LLM);
+    }
+
+}

common/src/main/java/com/tencent/supersonic/common/pojo/enums/TimeDimensionEnum.java

@@ -1,6 +1,7 @@
 package com.tencent.supersonic.common.pojo.enums;
 
-import cn.hutool.core.collection.CollectionUtil;
+import org.springframework.util.CollectionUtils;
+
 import java.util.Arrays;
 import java.util.List;
 import java.util.Map;
@@ -58,17 +59,15 @@ public enum TimeDimensionEnum {
     }
 
     /**
-     * Determine if a time dimension field is included in a Chinese text field
+     * Determine if a time dimension field is included in a Chinese/English text field
      *
      * @param fields field
      * @return true/false
      */
     public static boolean containsZhTimeDimension(List<String> fields) {
-        if (CollectionUtil.isEmpty(fields)) {
+        if (CollectionUtils.isEmpty(fields)) {
             return false;
         }
-        return fields.contains(TimeDimensionEnum.DAY.getChName())
-                || fields.contains(TimeDimensionEnum.WEEK.getChName())
-                || fields.contains(TimeDimensionEnum.MONTH.getChName());
+        return fields.stream().anyMatch(field -> containsTimeDimension(field));
     }
 }

common/src/main/java/com/tencent/supersonic/common/rest/SystemConfigController.java

@@ -1,7 +1,7 @@
 package com.tencent.supersonic.common.rest;
 
-import com.tencent.supersonic.common.pojo.SysParameter;
-import com.tencent.supersonic.common.service.SysParameterService;
+import com.tencent.supersonic.common.pojo.SystemConfig;
+import com.tencent.supersonic.common.service.SystemConfigService;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.web.bind.annotation.GetMapping;
 import org.springframework.web.bind.annotation.PostMapping;
@@ -11,20 +11,20 @@ import org.springframework.web.bind.annotation.RestController;
 
 @RestController
 @RequestMapping({"/api/semantic/parameter"})
-public class SysParameterController {
+public class SystemConfigController {
 
     @Autowired
-    private SysParameterService sysParameterService;
+    private SystemConfigService sysConfigService;
 
     @PostMapping
-    public Boolean save(@RequestBody SysParameter sysParameter) {
-        sysParameterService.save(sysParameter);
+    public Boolean save(@RequestBody SystemConfig systemConfig) {
+        sysConfigService.save(systemConfig);
         return true;
     }
 
     @GetMapping
-    public SysParameter get() {
-        return sysParameterService.getSysParameter();
+    public SystemConfig get() {
+        return sysConfigService.getSystemConfig();
     }
 
 }

common/src/main/java/com/tencent/supersonic/common/service/SysParameterService.java

@@ -1,13 +0,0 @@
-package com.tencent.supersonic.common.service;
-
-import com.baomidou.mybatisplus.extension.service.IService;
-import com.tencent.supersonic.common.persistence.dataobject.SysParameterDO;
-import com.tencent.supersonic.common.pojo.SysParameter;
-
-public interface SysParameterService extends IService<SysParameterDO> {
-
-    SysParameter getSysParameter();
-
-    void save(SysParameter sysParameter);
-
-}

common/src/main/java/com/tencent/supersonic/common/service/SystemConfigService.java

@@ -0,0 +1,13 @@
+package com.tencent.supersonic.common.service;
+
+import com.baomidou.mybatisplus.extension.service.IService;
+import com.tencent.supersonic.common.persistence.dataobject.SystemConfigDO;
+import com.tencent.supersonic.common.pojo.SystemConfig;
+
+public interface SystemConfigService extends IService<SystemConfigDO> {
+
+    SystemConfig getSystemConfig();
+
+    void save(SystemConfig systemConfig);
+
+}

common/src/main/java/com/tencent/supersonic/common/service/impl/SysParameterServiceImpl.java

@@ -1,58 +0,0 @@
-package com.tencent.supersonic.common.service.impl;
-
-import com.alibaba.fastjson.JSONObject;
-import com.baomidou.mybatisplus.extension.service.impl.ServiceImpl;
-import com.fasterxml.jackson.core.type.TypeReference;
-import com.tencent.supersonic.common.persistence.dataobject.SysParameterDO;
-import com.tencent.supersonic.common.persistence.mapper.SysParameterMapper;
-import com.tencent.supersonic.common.pojo.Parameter;
-import com.tencent.supersonic.common.pojo.SysParameter;
-import com.tencent.supersonic.common.service.SysParameterService;
-import com.tencent.supersonic.common.util.JsonUtil;
-import java.util.List;
-import org.springframework.stereotype.Service;
-import org.springframework.util.CollectionUtils;
-
-@Service
-public class SysParameterServiceImpl
-        extends ServiceImpl<SysParameterMapper, SysParameterDO> implements SysParameterService {
-
-    @Override
-    public SysParameter getSysParameter() {
-        List<SysParameterDO> list = list();
-        if (CollectionUtils.isEmpty(list)) {
-            SysParameter sysParameter = new SysParameter();
-            sysParameter.setId(1);
-            sysParameter.init();
-            save(sysParameter);
-            return sysParameter;
-        }
-        return convert(list.iterator().next());
-    }
-
-    @Override
-    public void save(SysParameter sysParameter) {
-        SysParameterDO sysParameterDO = convert(sysParameter);
-        saveOrUpdate(sysParameterDO);
-    }
-
-    private SysParameter convert(SysParameterDO sysParameterDO) {
-        SysParameter sysParameter = new SysParameter();
-        sysParameter.setId(sysParameterDO.getId());
-        List<Parameter> parameters = JsonUtil.toObject(sysParameterDO.getParameters(),
-                new TypeReference<List<Parameter>>() {
-                });
-        sysParameter.setParameters(parameters);
-        sysParameter.setAdminList(sysParameterDO.getAdmin());
-        return sysParameter;
-    }
-
-    private SysParameterDO convert(SysParameter sysParameter) {
-        SysParameterDO sysParameterDO = new SysParameterDO();
-        sysParameterDO.setId(sysParameter.getId());
-        sysParameterDO.setParameters(JSONObject.toJSONString(sysParameter.getParameters()));
-        sysParameterDO.setAdmin(sysParameter.getAdmin());
-        return sysParameterDO;
-    }
-
-}

common/src/main/java/com/tencent/supersonic/common/service/impl/SystemConfigServiceImpl.java

@@ -0,0 +1,58 @@
+package com.tencent.supersonic.common.service.impl;
+
+import com.alibaba.fastjson.JSONObject;
+import com.baomidou.mybatisplus.extension.service.impl.ServiceImpl;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.tencent.supersonic.common.persistence.dataobject.SystemConfigDO;
+import com.tencent.supersonic.common.persistence.mapper.SystemConfigMapper;
+import com.tencent.supersonic.common.pojo.Parameter;
+import com.tencent.supersonic.common.pojo.SystemConfig;
+import com.tencent.supersonic.common.service.SystemConfigService;
+import com.tencent.supersonic.common.util.JsonUtil;
+import org.springframework.stereotype.Service;
+import org.springframework.util.CollectionUtils;
+import java.util.List;
+
+@Service
+public class SystemConfigServiceImpl
+        extends ServiceImpl<SystemConfigMapper, SystemConfigDO> implements SystemConfigService {
+
+    @Override
+    public SystemConfig getSystemConfig() {
+        List<SystemConfigDO> list = list();
+        if (CollectionUtils.isEmpty(list)) {
+            SystemConfig systemConfig = new SystemConfig();
+            systemConfig.setId(1);
+            systemConfig.init();
+            save(systemConfig);
+            return systemConfig;
+        }
+        return convert(list.iterator().next());
+    }
+
+    @Override
+    public void save(SystemConfig sysConfig) {
+        SystemConfigDO systemConfigDO = convert(sysConfig);
+        saveOrUpdate(systemConfigDO);
+    }
+
+    private SystemConfig convert(SystemConfigDO systemConfigDO) {
+        SystemConfig sysParameter = new SystemConfig();
+        sysParameter.setId(systemConfigDO.getId());
+        List<Parameter> parameters = JsonUtil.toObject(systemConfigDO.getParameters(),
+                new TypeReference<List<Parameter>>() {
+                });
+        sysParameter.setParameters(parameters);
+        sysParameter.setAdminList(systemConfigDO.getAdmin());
+        return sysParameter;
+    }
+
+    private SystemConfigDO convert(SystemConfig sysParameter) {
+        SystemConfigDO sysParameterDO = new SystemConfigDO();
+        sysParameterDO.setId(sysParameter.getId());
+        sysParameterDO.setParameters(JSONObject.toJSONString(sysParameter.getParameters()));
+        sysParameterDO.setAdmin(sysParameter.getAdmin());
+        return sysParameterDO;
+    }
+
+}

common/src/main/java/com/tencent/supersonic/common/util/ChatGptHelper.java

@@ -1,149 +0,0 @@
-package com.tencent.supersonic.common.util;
-
-
-import com.plexpt.chatgpt.ChatGPT;
-import com.plexpt.chatgpt.entity.chat.ChatCompletion;
-import com.plexpt.chatgpt.entity.chat.ChatCompletionResponse;
-import com.plexpt.chatgpt.entity.chat.Message;
-import com.plexpt.chatgpt.util.Proxys;
-import lombok.extern.slf4j.Slf4j;
-import org.springframework.beans.factory.annotation.Value;
-import org.springframework.stereotype.Component;
-
-import java.net.Proxy;
-import java.text.SimpleDateFormat;
-import java.util.Arrays;
-import java.util.Date;
-
-
-@Component
-@Slf4j
-public class ChatGptHelper {
-
-    @Value("${llm.chatgpt.apikey:}")
-    private String apiKey;
-
-    @Value("${llm.chatgpt.apiHost:}")
-    private String apiHost;
-
-    @Value("${llm.chatgpt.proxyIp:default}")
-    private String proxyIp;
-
-    @Value("${llm.chatgpt.proxyPort:}")
-    private Integer proxyPort;
-
-    public ChatGPT getChatGPT() {
-        Proxy proxy = null;
-        if (!"default".equals(proxyIp)) {
-            proxy = Proxys.http(proxyIp, proxyPort);
-        }
-        return ChatGPT.builder()
-                .apiKey(apiKey)
-                .proxy(proxy)
-                .timeout(900)
-                .apiHost(apiHost) //反向代理地址
-                .build()
-                .init();
-    }
-
-    public Message getChatCompletion(Message system, Message message) {
-        ChatCompletion chatCompletion = ChatCompletion.builder()
-                .model(ChatCompletion.Model.GPT_3_5_TURBO_16K.getName())
-                .messages(Arrays.asList(system, message))
-                .maxTokens(10000)
-                .temperature(0.9)
-                .build();
-        ChatCompletionResponse response = getChatGPT().chatCompletion(chatCompletion);
-        return response.getChoices().get(0).getMessage();
-    }
-
-    public String inferredTime(String queryText) {
-        long nowTime = System.currentTimeMillis();
-        Date date = new Date(nowTime);
-        SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd");
-        String formattedDate = sdf.format(date);
-        Message system = Message.ofSystem("现在时间 " + formattedDate + "，你是一个专业的数据分析师，你的任务是基于数据，专业的解答用户的问题。"
-                + "你需要遵守以下规则：\n"
-                + "1.返回规范的数据格式，json，如： 输入：近 10 天的日活跃数，输出：{\"start\":\"2023-07-21\",\"end\":\"2023-07-31\"}"
-                + "2.你对时间数据要求规范，能从近 10 天，国庆节，端午节，获取到相应的时间，填写到 json 中。\n"
-                + "3.你的数据时间，只有当前及之前时间即可,超过则回复去年\n"
-                + "4.只需要解析出时间，时间可以是时间月和年或日、日历采用公历\n"
-                + "5.时间给出要是绝对正确，不能瞎编\n"
-        );
-        Message message = Message.of("输入：" + queryText + "，输出：");
-        Message res = getChatCompletion(system, message);
-        return res.getContent();
-    }
-
-    public String mockAlias(String mockType,
-                            String name,
-                            String bizName,
-                            String table,
-                            String desc,
-                            Boolean isPercentage) {
-        String msg = "Assuming you are a professional data analyst specializing in metrics and dimensions, "
-                + "you have a vast amount of data analysis metrics content. You are familiar with the basic"
-                + " format of the content,Now, Construct your answer Based on the following json-schema.\n"
-                + "{\n"
-                + "\"$schema\": \"http://json-schema.org/draft-07/schema#\",\n"
-                + "\"type\": \"array\",\n"
-                + "\"minItems\": 2,\n"
-                + "\"maxItems\": 4,\n"
-                + "\"items\": {\n"
-                + "\"type\": \"string\",\n"
-                + "\"description\": \"Assuming you are a data analyst and give a defined "
-                + mockType
-                + " name: "
-                + name + ","
-                + "this "
-                + mockType
-                + " is from database and table: "
-                + table + ",This "
-                + mockType
-                + " calculates the field source: "
-                + bizName
-                + ", The description of this metrics is: "
-                + desc
-                + ", provide some aliases for this, please take chinese or english,"
-                + "You must adhere to the following rules:\n"
-                + "1. Please do not generate aliases like xxx1, xxx2, xxx3.\n"
-                + "2. Please do not generate aliases that are the same as the original names of metrics/dimensions.\n"
-                + "3. Please pay attention to the quality of the generated aliases and "
-                + "   avoid creating aliases that look like test data.\n"
-                + "4. Please generate more Chinese aliases."
-                + "},\n"
-                + "\"additionalProperties\":false}\n"
-                + "Please double-check whether the answer conforms to the format described in the JSON-schema.\n"
-                + "ANSWER JSON:";
-        log.info("msg:{}", msg);
-        Message system = Message.ofSystem("");
-        Message message = Message.of(msg);
-        Message res = getChatCompletion(system, message);
-        return res.getContent();
-    }
-
-    public String mockDimensionValueAlias(String json) {
-        String msg = "Assuming you are a professional data analyst specializing in indicators,for you a json list，"
-                + "the required content to follow is as follows: "
-                + "1. The format of JSON,"
-                + "2. Only return in JSON format,"
-                + "3. the array item > 1 and < 5,more alias,"
-                + "for example：input:[\"qq_music\",\"kugou_music\"],"
-                + "out:{\"tran\":[\"qq音乐\",\"酷狗音乐\"],\"alias\":{\"qq_music\":[\"q音\",\"qq音乐\"],"
-                + "\"kugou_music\":[\"kugou\",\"酷狗\"]}},"
-                + "now input: "
-                + json + ","
-                + "answer json:";
-        log.info("msg:{}", msg);
-        Message system = Message.ofSystem("");
-        Message message = Message.of(msg);
-        Message res = getChatCompletion(system, message);
-        return res.getContent();
-    }
-
-    public static void main(String[] args) {
-        ChatGptHelper chatGptHelper = new ChatGptHelper();
-        System.out.println(chatGptHelper.mockAlias("", "", "", "", "", false));
-    }
-
-}

common/src/main/java/com/tencent/supersonic/common/util/DateModeUtils.java

@@ -1,17 +1,15 @@
 package com.tencent.supersonic.common.util;
 
-import static com.tencent.supersonic.common.pojo.Constants.APOSTROPHE;
-import static com.tencent.supersonic.common.pojo.Constants.COMMA;
-import static com.tencent.supersonic.common.pojo.Constants.DAY;
-import static com.tencent.supersonic.common.pojo.Constants.DAY_FORMAT;
-import static com.tencent.supersonic.common.pojo.Constants.MONTH;
-import static com.tencent.supersonic.common.pojo.Constants.MONTH_FORMAT;
-import static com.tencent.supersonic.common.pojo.Constants.WEEK;
-import static com.tencent.supersonic.common.pojo.Constants.YEAR;
-
 import com.google.common.base.Strings;
+import com.tencent.supersonic.common.pojo.Constants;
 import com.tencent.supersonic.common.pojo.DateConf;
 import com.tencent.supersonic.common.pojo.ItemDateResp;
+import lombok.Data;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.tuple.ImmutablePair;
+import org.springframework.beans.factory.annotation.Value;
+import org.springframework.stereotype.Component;
+import org.springframework.util.CollectionUtils;
 
 import java.time.LocalDate;
 import java.time.format.DateTimeFormatter;
@@ -24,12 +22,14 @@ import java.util.Objects;
 import java.util.StringJoiner;
 import java.util.regex.Pattern;
 
-import lombok.Data;
-import lombok.extern.slf4j.Slf4j;
-import org.apache.commons.lang3.tuple.ImmutablePair;
-import org.springframework.beans.factory.annotation.Value;
-import org.springframework.stereotype.Component;
-import org.springframework.util.CollectionUtils;
+import static com.tencent.supersonic.common.pojo.Constants.APOSTROPHE;
+import static com.tencent.supersonic.common.pojo.Constants.COMMA;
+import static com.tencent.supersonic.common.pojo.Constants.DAY;
+import static com.tencent.supersonic.common.pojo.Constants.DAY_FORMAT;
+import static com.tencent.supersonic.common.pojo.Constants.MONTH;
+import static com.tencent.supersonic.common.pojo.Constants.MONTH_FORMAT;
+import static com.tencent.supersonic.common.pojo.Constants.WEEK;
+import static com.tencent.supersonic.common.pojo.Constants.YEAR;
 
 
 @Slf4j
@@ -253,6 +253,11 @@ public class DateModeUtils {
      */
     public String betweenDateStr(ItemDateResp dateDate, DateConf dateInfo) {
         if (MONTH.equalsIgnoreCase(dateInfo.getPeriod())) {
+            // startDate YYYYMM
+            if (!dateInfo.getStartDate().contains(Constants.MINUS)) {
+                return String.format("%s >= '%s' and %s <= '%s'",
+                        sysDateMonthCol, dateInfo.getStartDate(), sysDateMonthCol, dateInfo.getEndDate());
+            }
             LocalDate endData = LocalDate.parse(dateInfo.getEndDate(),
                     DateTimeFormatter.ofPattern(DAY_FORMAT));
             LocalDate startData = LocalDate.parse(dateInfo.getStartDate(),

common/src/main/java/com/tencent/supersonic/common/util/DateUtils.java

@@ -1,6 +1,8 @@
 package com.tencent.supersonic.common.util;
 
+import com.google.common.collect.Lists;
 import com.tencent.supersonic.common.pojo.Constants;
+import lombok.extern.slf4j.Slf4j;
 import java.text.DateFormat;
 import java.text.SimpleDateFormat;
 import java.time.LocalDate;
@@ -16,7 +18,6 @@ import java.util.Calendar;
 import java.util.Date;
 import java.util.List;
 import java.util.Objects;
-import lombok.extern.slf4j.Slf4j;
 
 @Slf4j
 public class DateUtils {
@@ -147,25 +148,29 @@ public class DateUtils {
     }
 
     public static List<String> getDateList(String startDateStr, String endDateStr, String period) {
-        LocalDate startDate = LocalDate.parse(startDateStr);
-        LocalDate endDate = LocalDate.parse(endDateStr);
-
-        List<String> datesInRange = new ArrayList<>();
-        LocalDate currentDate = startDate;
-        DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyy-MM");
-        while (!currentDate.isAfter(endDate)) {
-            if (Constants.MONTH.equals(period)) {
-                datesInRange.add(currentDate.format(formatter));
-                currentDate = currentDate.plusMonths(1);
-            } else if (Constants.WEEK.equals(period)) {
-                datesInRange.add(currentDate.format(DateTimeFormatter.ISO_DATE));
-                currentDate = currentDate.plusWeeks(1);
-            } else {
-                datesInRange.add(currentDate.format(DateTimeFormatter.ISO_DATE));
-                currentDate = currentDate.plusDays(1);
+        try {
+            LocalDate startDate = LocalDate.parse(startDateStr);
+            LocalDate endDate = LocalDate.parse(endDateStr);
+            List<String> datesInRange = new ArrayList<>();
+            LocalDate currentDate = startDate;
+            DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyy-MM");
+            while (!currentDate.isAfter(endDate)) {
+                if (Constants.MONTH.equals(period)) {
+                    datesInRange.add(currentDate.format(formatter));
+                    currentDate = currentDate.plusMonths(1);
+                } else if (Constants.WEEK.equals(period)) {
+                    datesInRange.add(currentDate.format(DateTimeFormatter.ISO_DATE));
+                    currentDate = currentDate.plusWeeks(1);
+                } else {
+                    datesInRange.add(currentDate.format(DateTimeFormatter.ISO_DATE));
+                    currentDate = currentDate.plusDays(1);
+                }
             }
+            return datesInRange;
+        } catch (Exception e) {
+            log.info("parse date failed, startDate:{}, endDate:{}", startDateStr, endDateStr, e);
         }
-        return datesInRange;
+        return Lists.newArrayList();
     }
 
     public static boolean isAnyDateString(String value) {

common/src/main/java/com/tencent/supersonic/common/util/FileUtils.java

@@ -0,0 +1,89 @@
+package com.tencent.supersonic.common.util;
+
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+/**
+ * * tools functions to file
+ */
+public class FileUtils {
+
+    public static Boolean exit(String path) {
+        File file = new File(path);
+        return file.exists();
+    }
+
+    public static long getLastModified(String path) {
+        if (!exit(path)) {
+            return -1;
+        }
+        File file = new File(path);
+        Optional<Long> lastModified = Arrays.stream(file.listFiles()).filter(f -> f.isFile())
+                .map(f -> f.lastModified()).sorted(Collections.reverseOrder()).findFirst();
+
+        if (lastModified.isPresent()) {
+            return lastModified.get();
+        }
+        return -1;
+    }
+
+    public static File[] getDirFiles(File file) {
+        if (file.isDirectory()) {
+            return file.listFiles();
+        }
+        return null;
+    }
+
+    public static void scanDirectory(File file, int maxLevel, Map<Integer, List<File>> directories) {
+        if (maxLevel < 0) {
+            return;
+        }
+        if (!file.exists() || !file.isDirectory()) {
+            return;
+        }
+        if (!directories.containsKey(maxLevel)) {
+            directories.put(maxLevel, new ArrayList<>());
+        }
+        for (File f : file.listFiles()) {
+            if (f.isDirectory()) {
+                directories.get(maxLevel).add(f);
+                scanDirectory(f, maxLevel - 1, directories);
+            }
+        }
+    }
+
+    public static Map<String, Map<String, List<String>>> getTop3Directory(String path) {
+        Map<String, Map<String, List<String>>> result = new HashMap<>();
+        File file = new File(path);
+        if (!file.exists() || !file.isDirectory()) {
+            return result;
+        }
+        Map<Integer, List<File>> directories = new HashMap<>();
+        scanDirectory(file, 2, directories);
+        for (int i = 2; i >= 0; i--) {
+            for (File f : directories.getOrDefault(i, new ArrayList<>())) {
+                if (i == 2) {
+                    result.put(f.getName(), new HashMap<>());
+                    continue;
+                }
+                if (i == 1 && result.containsKey(f.getParentFile().getName())) {
+                    result.get(f.getParentFile().getName()).put(f.getName(), new ArrayList<>());
+                    continue;
+                }
+                String parent = f.getParentFile().getParentFile().getName();
+                if (result.containsKey(parent) && result.get(parent).containsKey(f.getParentFile().getName())) {
+                    result.get(parent).get(f.getParentFile().getName()).add(f.getName());
+                }
+            }
+        }
+        return result;
+    }
+
+}

common/src/main/java/com/tencent/supersonic/common/util/StringUtil.java

@@ -26,10 +26,9 @@ public class StringUtil {
     }
 
     /**
-     *
      * @param v1
      * @param v2
-     * @return  value 0 if v1 equal to v2; less than 0 if v1 is less than v2; greater than 0 if v1 is greater than v2
+     * @return value 0 if v1 equal to v2; less than 0 if v1 is less than v2; greater than 0 if v1 is greater than v2
      */
     public static int compareVersion(String v1, String v2) {
         String[] v1s = v1.split("\\.");
@@ -46,4 +45,7 @@ public class StringUtil {
         return v1s.length - v2s.length;
     }
 
+    public static String replaceBackticks(String sql) {
+        return sql.replaceAll("`", "");
+    }
 }

common/src/main/java/com/tencent/supersonic/common/util/embedding/EmbeddingPersistentTask.java

@@ -1,32 +0,0 @@
-package com.tencent.supersonic.common.util.embedding;
-
-import com.tencent.supersonic.common.util.ComponentFactory;
-import javax.annotation.PreDestroy;
-import lombok.extern.slf4j.Slf4j;
-import org.springframework.scheduling.annotation.Scheduled;
-import org.springframework.stereotype.Component;
-
-@Component
-@Slf4j
-public class EmbeddingPersistentTask {
-
-    private S2EmbeddingStore s2EmbeddingStore = ComponentFactory.getS2EmbeddingStore();
-
-    @PreDestroy
-    public void onShutdown() {
-        embeddingStorePersistentToFile();
-    }
-
-    private void embeddingStorePersistentToFile() {
-        if (s2EmbeddingStore instanceof InMemoryS2EmbeddingStore) {
-            log.info("start persistentToFile");
-            ((InMemoryS2EmbeddingStore) s2EmbeddingStore).persistentToFile();
-            log.info("end persistentToFile");
-        }
-    }
-
-    @Scheduled(cron = "${inMemoryEmbeddingStore.persistent.cron:0 0 * * * ?}")
-    public void executeTask() {
-        embeddingStorePersistentToFile();
-    }
-}

common/src/main/java/com/tencent/supersonic/common/util/embedding/EmbeddingQuery.java

@@ -1,10 +1,13 @@
 package com.tencent.supersonic.common.util.embedding;
 
 
+import com.alibaba.fastjson.JSONObject;
+import com.tencent.supersonic.common.pojo.DataItem;
 import lombok.Data;
 
 import java.util.List;
 import java.util.Map;
+import java.util.stream.Collectors;
 
 @Data
 public class EmbeddingQuery {
@@ -17,4 +20,17 @@ public class EmbeddingQuery {
 
     private List<Double> queryEmbedding;
 
+    public static List<EmbeddingQuery> convertToEmbedding(List<DataItem> dataItems) {
+        return dataItems.stream().map(dataItem -> {
+            EmbeddingQuery embeddingQuery = new EmbeddingQuery();
+            embeddingQuery.setQueryId(
+                    dataItem.getId() + dataItem.getType().name().toLowerCase());
+            embeddingQuery.setQuery(dataItem.getName());
+            Map meta = JSONObject.parseObject(JSONObject.toJSONString(dataItem), Map.class);
+            embeddingQuery.setMetadata(meta);
+            embeddingQuery.setQueryEmbedding(null);
+            return embeddingQuery;
+        }).collect(Collectors.toList());
+    }
+
 }

common/src/main/java/com/tencent/supersonic/common/util/embedding/InMemoryS2EmbeddingStore.java

@@ -1,10 +1,5 @@
 package com.tencent.supersonic.common.util.embedding;
 
-import static dev.langchain4j.internal.Utils.randomUUID;
-import static java.nio.file.StandardOpenOption.CREATE;
-import static java.nio.file.StandardOpenOption.TRUNCATE_EXISTING;
-import static java.util.Comparator.comparingDouble;
-
 import com.tencent.supersonic.common.config.EmbeddingConfig;
 import com.tencent.supersonic.common.util.ContextUtils;
 import dev.langchain4j.data.embedding.Embedding;
@@ -13,6 +8,10 @@ import dev.langchain4j.store.embedding.CosineSimilarity;
 import dev.langchain4j.store.embedding.EmbeddingMatch;
 import dev.langchain4j.store.embedding.EmbeddingStore;
 import dev.langchain4j.store.embedding.RelevanceScore;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.collections4.MapUtils;
+import org.apache.commons.lang3.StringUtils;
+
 import java.io.IOException;
 import java.nio.file.Files;
 import java.nio.file.Path;
@@ -20,6 +19,7 @@ import java.nio.file.Paths;
 import java.util.ArrayList;
 import java.util.Collections;
 import java.util.Comparator;
+import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
 import java.util.Map.Entry;
@@ -29,9 +29,11 @@ import java.util.Set;
 import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.CopyOnWriteArraySet;
 import java.util.stream.Collectors;
-import lombok.extern.slf4j.Slf4j;
-import org.apache.commons.collections4.MapUtils;
-import org.apache.commons.lang3.StringUtils;
+
+import static dev.langchain4j.internal.Utils.randomUUID;
+import static java.nio.file.StandardOpenOption.CREATE;
+import static java.nio.file.StandardOpenOption.TRUNCATE_EXISTING;
+import static java.util.Comparator.comparingDouble;
 
 /***
  * Implementation of S2EmbeddingStore within the Java process's in-memory.
@@ -134,7 +136,11 @@ public class InMemoryS2EmbeddingStore implements S2EmbeddingStore {
                 retrieval.setDistance(1 - embeddingMatch.score());
                 retrieval.setId(embeddingMatch.embeddingId());
                 retrieval.setQuery(embeddingMatch.embedded().getQuery());
-                Map<String, Object> metadata = embeddingMatch.embedded().getMetadata();
+                Map<String, Object> metadata = new HashMap<>();
+                if (Objects.nonNull(embeddingMatch.embedded())
+                        && MapUtils.isNotEmpty(embeddingMatch.embedded().getMetadata())) {
+                    metadata.putAll(embeddingMatch.embedded().getMetadata());
+                }
                 if (filterRetrieval(filterCondition, metadata)) {
                     continue;
                 }
@@ -179,9 +185,9 @@ public class InMemoryS2EmbeddingStore implements S2EmbeddingStore {
      * Uses a brute force approach by iterating over all embeddings to find the best matches.
      *
      * @param <Embedded> The class of the object that has been embedded.
-     *         Typically, it is {@link dev.langchain4j.data.segment.TextSegment}.
-     *         copy from dev.langchain4j.store.embedding.inmemory.InMemoryEmbeddingStore
-     *         and fix concurrentModificationException in a multi-threaded environment
+     *                   Typically, it is {@link dev.langchain4j.data.segment.TextSegment}.
+     *                   copy from dev.langchain4j.store.embedding.inmemory.InMemoryEmbeddingStore
+     *                   and fix concurrentModificationException in a multi-threaded environment
      */
     public static class InMemoryEmbeddingStore<Embedded> implements EmbeddingStore<Embedded> {
 
@@ -267,7 +273,7 @@ public class InMemoryS2EmbeddingStore implements S2EmbeddingStore {
 
         @Override
         public List<EmbeddingMatch<Embedded>> findRelevant(Embedding referenceEmbedding, int maxResults,
-                double minScore) {
+                                                           double minScore) {
 
             Comparator<EmbeddingMatch<Embedded>> comparator = comparingDouble(EmbeddingMatch::score);
             PriorityQueue<EmbeddingMatch<Embedded>> matches = new PriorityQueue<>(comparator);

common/src/main/java/com/tencent/supersonic/common/util/jsqlparser/DateFunctionHelper.java

@@ -2,18 +2,18 @@ package com.tencent.supersonic.common.util.jsqlparser;
 
 import com.tencent.supersonic.common.util.DatePeriodEnum;
 import com.tencent.supersonic.common.util.DateUtils;
-import java.util.List;
 import lombok.extern.slf4j.Slf4j;
 import net.sf.jsqlparser.expression.DoubleValue;
 import net.sf.jsqlparser.expression.Expression;
 import net.sf.jsqlparser.expression.LongValue;
 import net.sf.jsqlparser.expression.StringValue;
 import net.sf.jsqlparser.expression.operators.relational.ComparisonOperator;
+import net.sf.jsqlparser.expression.operators.relational.ExpressionList;
 
 @Slf4j
 public class DateFunctionHelper {
 
-    public static String getStartDateStr(ComparisonOperator minorThanEquals, List<Expression> expressions) {
+    public static String getStartDateStr(ComparisonOperator minorThanEquals, ExpressionList<?> expressions) {
         String unitValue = getUnit(expressions);
         String dateValue = getEndDateValue(expressions);
         String dateStr = "";
@@ -38,12 +38,12 @@ public class DateFunctionHelper {
         return "<=";
     }
 
-    public static String getEndDateValue(List<Expression> leftExpressions) {
+    public static String getEndDateValue(ExpressionList<?> leftExpressions) {
         StringValue date = (StringValue) leftExpressions.get(2);
         return date.getValue();
     }
 
-    private static String getUnit(List<Expression> expressions) {
+    private static String getUnit(ExpressionList<?> expressions) {
         StringValue unit = (StringValue) expressions.get(0);
         return unit.getValue();
     }

common/src/main/java/com/tencent/supersonic/common/util/jsqlparser/ExpressionReplaceVisitor.java

@@ -1,7 +1,5 @@
 package com.tencent.supersonic.common.util.jsqlparser;
 
-import java.util.ArrayList;
-import java.util.List;
 import java.util.Map;
 import java.util.Objects;
 import net.sf.jsqlparser.expression.BinaryExpression;
@@ -9,6 +7,7 @@ import net.sf.jsqlparser.expression.Expression;
 import net.sf.jsqlparser.expression.ExpressionVisitorAdapter;
 import net.sf.jsqlparser.expression.Function;
 import net.sf.jsqlparser.expression.WhenClause;
+import net.sf.jsqlparser.expression.operators.relational.ExpressionList;
 import net.sf.jsqlparser.schema.Column;
 
 public class ExpressionReplaceVisitor extends ExpressionVisitorAdapter {
@@ -80,12 +79,12 @@ public class ExpressionReplaceVisitor extends ExpressionVisitorAdapter {
             Expression expression = QueryExpressionReplaceVisitor.getExpression(
                     QueryExpressionReplaceVisitor.getReplaceExpr(function, fieldExprMap));
             if (Objects.nonNull(expression)) {
-                List<Expression> expressions = new ArrayList<>();
+                ExpressionList<Expression> expressions = new ExpressionList<>();
                 expressions.add(expression);
-                for (int i = 1; i < function.getParameters().getExpressions().size(); i++) {
-                    expressions.add(function.getParameters().getExpressions().get(i));
+                for (int i = 1; i < function.getParameters().size(); i++) {
+                    expressions.add((Expression) function.getParameters().get(i));
                 }
-                function.getParameters().setExpressions(expressions);
+                function.setParameters(expressions);
                 return true;
             }
         }

common/src/main/java/com/tencent/supersonic/common/util/jsqlparser/FieldAndValueAcquireVisitor.java

@@ -19,7 +19,6 @@ import net.sf.jsqlparser.expression.operators.relational.ExpressionList;
 import net.sf.jsqlparser.expression.operators.relational.GreaterThan;
 import net.sf.jsqlparser.expression.operators.relational.GreaterThanEquals;
 import net.sf.jsqlparser.expression.operators.relational.InExpression;
-import net.sf.jsqlparser.expression.operators.relational.ItemsList;
 import net.sf.jsqlparser.expression.operators.relational.LikeExpression;
 import net.sf.jsqlparser.expression.operators.relational.MinorThan;
 import net.sf.jsqlparser.expression.operators.relational.MinorThanEquals;
@@ -58,7 +57,7 @@ public class FieldAndValueAcquireVisitor extends ExpressionVisitorAdapter {
         }
         fieldExpression.setFieldName(((Column) leftExpression).getColumnName());
         fieldExpression.setOperator(JsqlConstants.IN);
-        ItemsList rightItemsList = expr.getRightItemsList();
+        Expression rightItemsList = expr.getRightExpression();
         fieldExpression.setFieldValue(rightItemsList);
         List<Object> result = new ArrayList<>();
         if (rightItemsList instanceof ExpressionList) {
@@ -143,7 +142,8 @@ public class FieldAndValueAcquireVisitor extends ExpressionVisitorAdapter {
     }
 
     private Column getColumn(Function leftExpressionFunction) {
-        List<Expression> leftExpressions = leftExpressionFunction.getParameters().getExpressions();
+        //List<Expression> leftExpressions = leftExpressionFunction.getParameters().getExpressions();
+        ExpressionList<?> leftExpressions = leftExpressionFunction.getParameters();
         if (CollectionUtils.isEmpty(leftExpressions)) {
             return null;
         }

common/src/main/java/com/tencent/supersonic/common/util/jsqlparser/FieldlValueReplaceVisitor.java

@@ -1,11 +1,10 @@
 package com.tencent.supersonic.common.util.jsqlparser;
 
-import java.util.List;
+import com.tencent.supersonic.common.util.JsonUtil;
 import java.util.ArrayList;
+import java.util.List;
 import java.util.Map;
 import java.util.Objects;
-
-import com.tencent.supersonic.common.util.JsonUtil;
 import lombok.extern.slf4j.Slf4j;
 import net.sf.jsqlparser.expression.DoubleValue;
 import net.sf.jsqlparser.expression.Expression;
@@ -15,12 +14,12 @@ import net.sf.jsqlparser.expression.LongValue;
 import net.sf.jsqlparser.expression.StringValue;
 import net.sf.jsqlparser.expression.operators.relational.ComparisonOperator;
 import net.sf.jsqlparser.expression.operators.relational.EqualsTo;
+import net.sf.jsqlparser.expression.operators.relational.ExpressionList;
 import net.sf.jsqlparser.expression.operators.relational.GreaterThan;
 import net.sf.jsqlparser.expression.operators.relational.GreaterThanEquals;
+import net.sf.jsqlparser.expression.operators.relational.InExpression;
 import net.sf.jsqlparser.expression.operators.relational.MinorThan;
 import net.sf.jsqlparser.expression.operators.relational.MinorThanEquals;
-import net.sf.jsqlparser.expression.operators.relational.InExpression;
-import net.sf.jsqlparser.expression.operators.relational.ExpressionList;
 import net.sf.jsqlparser.schema.Column;
 import org.apache.commons.lang3.StringUtils;
 import org.springframework.util.CollectionUtils;
@@ -64,7 +63,7 @@ public class FieldlValueReplaceVisitor extends ExpressionVisitorAdapter {
         }
         Column column = (Column) inExpression.getLeftExpression();
         Map<String, String> valueMap = filedNameToValueMap.get(column.getColumnName());
-        ExpressionList rightItemsList = (ExpressionList) inExpression.getRightItemsList();
+        ExpressionList rightItemsList = (ExpressionList) inExpression.getRightExpression();
         List<Expression> expressions = rightItemsList.getExpressions();
         List<String> values = new ArrayList<>();
         expressions.stream().forEach(o -> {
@@ -86,7 +85,7 @@ public class FieldlValueReplaceVisitor extends ExpressionVisitorAdapter {
             newExpressions.add(stringValue);
         });
         rightItemsList.setExpressions(newExpressions);
-        inExpression.setRightItemsList(rightItemsList);
+        inExpression.setRightExpression(rightItemsList);
     }
 
     public <T extends Expression> void replaceComparisonExpression(T expression) {

common/src/main/java/com/tencent/supersonic/common/util/jsqlparser/FiledFilterReplaceVisitor.java

@@ -11,6 +11,7 @@ import net.sf.jsqlparser.expression.ExpressionVisitorAdapter;
 import net.sf.jsqlparser.expression.Function;
 import net.sf.jsqlparser.expression.operators.relational.ComparisonOperator;
 import net.sf.jsqlparser.expression.operators.relational.EqualsTo;
+import net.sf.jsqlparser.expression.operators.relational.ExpressionList;
 import net.sf.jsqlparser.expression.operators.relational.GreaterThan;
 import net.sf.jsqlparser.expression.operators.relational.GreaterThanEquals;
 import net.sf.jsqlparser.expression.operators.relational.MinorThan;
@@ -85,7 +86,8 @@ public class FiledFilterReplaceVisitor extends ExpressionVisitorAdapter {
             return result;
         }
 
-        List<Expression> leftExpressions = leftExpressionFunction.getParameters().getExpressions();
+        //List<Expression> leftExpressions = leftExpressionFunction.getParameters().getExpressions();
+        ExpressionList<?> leftExpressions = leftExpressionFunction.getParameters();
         if (CollectionUtils.isEmpty(leftExpressions)) {
             return result;
         }

common/src/main/java/com/tencent/supersonic/common/util/jsqlparser/FunctionAliasReplaceVisitor.java

@@ -4,7 +4,7 @@ import java.util.HashMap;
 import java.util.Map;
 import java.util.Objects;
 import net.sf.jsqlparser.expression.Function;
-import net.sf.jsqlparser.statement.select.SelectExpressionItem;
+import net.sf.jsqlparser.statement.select.SelectItem;
 import net.sf.jsqlparser.statement.select.SelectItemVisitorAdapter;
 
 public class FunctionAliasReplaceVisitor extends SelectItemVisitorAdapter {
@@ -12,7 +12,7 @@ public class FunctionAliasReplaceVisitor extends SelectItemVisitorAdapter {
     private Map<String, String> aliasToActualExpression = new HashMap<>();
 
     @Override
-    public void visit(SelectExpressionItem selectExpressionItem) {
+    public void visit(SelectItem selectExpressionItem) {
         if (selectExpressionItem.getExpression() instanceof Function) {
             Function function = (Function) selectExpressionItem.getExpression();
             String columnName = SqlSelectHelper.getColumnName(function);

common/src/main/java/com/tencent/supersonic/common/util/jsqlparser/FunctionNameReplaceVisitor.java

@@ -1,24 +1,41 @@
 package com.tencent.supersonic.common.util.jsqlparser;
 
 import java.util.Map;
+import java.util.Objects;
+import java.util.function.UnaryOperator;
 import lombok.extern.slf4j.Slf4j;
 import net.sf.jsqlparser.expression.ExpressionVisitorAdapter;
 import net.sf.jsqlparser.expression.Function;
+import net.sf.jsqlparser.expression.operators.relational.ExpressionList;
 import org.apache.commons.lang3.StringUtils;
 
 @Slf4j
 public class FunctionNameReplaceVisitor extends ExpressionVisitorAdapter {
 
     private Map<String, String> functionMap;
+    private Map<String, UnaryOperator> functionCallMap;
 
     public FunctionNameReplaceVisitor(Map<String, String> functionMap) {
         this.functionMap = functionMap;
     }
 
+    public FunctionNameReplaceVisitor(Map<String, String> functionMap, Map<String, UnaryOperator> functionCallMap) {
+        this.functionMap = functionMap;
+        this.functionCallMap = functionCallMap;
+    }
+
     public void visit(Function function) {
-        String replaceFunctionName = functionMap.get(function.getName().toLowerCase());
+        String functionName = function.getName().toLowerCase();
+        String replaceFunctionName = functionMap.get(functionName);
         if (StringUtils.isNotBlank(replaceFunctionName)) {
             function.setName(replaceFunctionName);
+            if (Objects.nonNull(functionCallMap) && functionCallMap.containsKey(functionName)) {
+                Object ret = functionCallMap.get(functionName).apply(function.getParameters());
+                if (Objects.nonNull(ret) && ret instanceof ExpressionList) {
+                    ExpressionList expressionList = (ExpressionList) ret;
+                    function.setParameters(expressionList);
+                }
+            }
         }
     }
 

common/src/main/java/com/tencent/supersonic/common/util/jsqlparser/FunctionReplaceVisitor.java

@@ -12,6 +12,7 @@ import net.sf.jsqlparser.expression.Function;
 import net.sf.jsqlparser.expression.LongValue;
 import net.sf.jsqlparser.expression.operators.relational.ComparisonOperator;
 import net.sf.jsqlparser.expression.operators.relational.EqualsTo;
+import net.sf.jsqlparser.expression.operators.relational.ExpressionList;
 import net.sf.jsqlparser.expression.operators.relational.GreaterThan;
 import net.sf.jsqlparser.expression.operators.relational.GreaterThanEquals;
 import net.sf.jsqlparser.expression.operators.relational.MinorThan;
@@ -79,7 +80,8 @@ public class FunctionReplaceVisitor extends ExpressionVisitorAdapter {
         if (!leftExpressionFunction.toString().contains(JsqlConstants.DATE_FUNCTION)) {
             return result;
         }
-        List<Expression> leftExpressions = leftExpressionFunction.getParameters().getExpressions();
+        //List<Expression> leftExpressions = leftExpressionFunction.getParameters().getExpressions();
+        ExpressionList<?> leftExpressions = leftExpressionFunction.getParameters();
         if (CollectionUtils.isEmpty(leftExpressions) || leftExpressions.size() < 3) {
             return result;
         }

common/src/main/java/com/tencent/supersonic/common/util/jsqlparser/GroupByFunctionReplaceVisitor.java

@@ -2,6 +2,8 @@ package com.tencent.supersonic.common.util.jsqlparser;
 
 import java.util.List;
 import java.util.Map;
+import java.util.Objects;
+import java.util.function.UnaryOperator;
 import lombok.extern.slf4j.Slf4j;
 import net.sf.jsqlparser.expression.Expression;
 import net.sf.jsqlparser.expression.Function;
@@ -14,11 +16,17 @@ import org.apache.commons.lang3.StringUtils;
 public class GroupByFunctionReplaceVisitor implements GroupByVisitor {
 
     private Map<String, String> functionMap;
+    private Map<String, UnaryOperator> functionCallMap;
 
     public GroupByFunctionReplaceVisitor(Map<String, String> functionMap) {
         this.functionMap = functionMap;
     }
 
+    public GroupByFunctionReplaceVisitor(Map<String, String> functionMap, Map<String, UnaryOperator> functionCallMap) {
+        this.functionMap = functionMap;
+        this.functionCallMap = functionCallMap;
+    }
+
     public void visit(GroupByElement groupByElement) {
         groupByElement.getGroupByExpressionList();
         ExpressionList groupByExpressionList = groupByElement.getGroupByExpressionList();
@@ -28,9 +36,17 @@ public class GroupByFunctionReplaceVisitor implements GroupByVisitor {
             Expression expression = groupByExpressions.get(i);
             if (expression instanceof Function) {
                 Function function = (Function) expression;
-                String replaceName = functionMap.get(function.getName().toLowerCase());
+                String functionName = function.getName().toLowerCase();
+                String replaceName = functionMap.get(functionName);
                 if (StringUtils.isNotBlank(replaceName)) {
                     function.setName(replaceName);
+                    if (Objects.nonNull(functionCallMap) && functionCallMap.containsKey(functionName)) {
+                        Object ret = functionCallMap.get(functionName).apply(function.getParameters());
+                        if (Objects.nonNull(ret) && ret instanceof ExpressionList) {
+                            ExpressionList expressionList = (ExpressionList) ret;
+                            function.setParameters(expressionList);
+                        }
+                    }
                 }
             }
         }

common/src/main/java/com/tencent/supersonic/common/util/jsqlparser/GroupByReplaceVisitor.java

@@ -47,7 +47,14 @@ public class GroupByReplaceVisitor implements GroupByVisitor {
                 if (expression instanceof Function) {
                     try {
                         Expression element = CCJSqlParserUtil.parseExpression(replaceColumn);
-                        ((Function) expression).getParameters().getExpressions().set(0, element);
+                        ExpressionList<Expression> expressionList = new ExpressionList<Expression>();
+                        expressionList.add(element);
+                        if (((Function) expression).getParameters().size() > 1) {
+                            ((Function) expression).getParameters().stream().skip(1).forEach(e -> {
+                                expressionList.add((Function) e);
+                            });
+                        }
+                        ((Function) expression).setParameters(expressionList);
                     } catch (JSQLParserException e) {
                         log.error("e", e);
                     }

common/src/main/java/com/tencent/supersonic/common/util/jsqlparser/OrderByAcquireVisitor.java

@@ -1,10 +1,10 @@
 package com.tencent.supersonic.common.util.jsqlparser;
 
 import com.tencent.supersonic.common.pojo.Constants;
-import java.util.List;
 import java.util.Set;
 import net.sf.jsqlparser.expression.Expression;
 import net.sf.jsqlparser.expression.Function;
+import net.sf.jsqlparser.expression.operators.relational.ExpressionList;
 import net.sf.jsqlparser.schema.Column;
 import net.sf.jsqlparser.statement.select.OrderByElement;
 import net.sf.jsqlparser.statement.select.OrderByVisitorAdapter;
@@ -26,7 +26,8 @@ public class OrderByAcquireVisitor extends OrderByVisitorAdapter {
         }
         if (expression instanceof Function) {
             Function function = (Function) expression;
-            List<Expression> expressions = function.getParameters().getExpressions();
+            //List<Expression> expressions = function.getParameters().getExpressions();
+            ExpressionList<?> expressions = function.getParameters();
             for (Expression column : expressions) {
                 if (column instanceof Column) {
                     fieldExpression.setFieldName(((Column) column).getColumnName());

common/src/main/java/com/tencent/supersonic/common/util/jsqlparser/OrderByReplaceVisitor.java

@@ -1,9 +1,9 @@
 package com.tencent.supersonic.common.util.jsqlparser;
 
-import java.util.List;
 import java.util.Map;
 import net.sf.jsqlparser.expression.Expression;
 import net.sf.jsqlparser.expression.Function;
+import net.sf.jsqlparser.expression.operators.relational.ExpressionList;
 import net.sf.jsqlparser.schema.Column;
 import net.sf.jsqlparser.statement.select.OrderByElement;
 import net.sf.jsqlparser.statement.select.OrderByVisitorAdapter;
@@ -27,7 +27,8 @@ public class OrderByReplaceVisitor extends OrderByVisitorAdapter {
         }
         if (expression instanceof Function) {
             Function function = (Function) expression;
-            List<Expression> expressions = function.getParameters().getExpressions();
+            //List<Expression> expressions = function.getParameters().getExpressions();
+            ExpressionList<?> expressions = function.getParameters();
             for (Expression column : expressions) {
                 if (column instanceof Column) {
                     parseVisitorHelper.replaceColumn((Column) column, fieldNameMap, exactReplace);

common/src/main/java/com/tencent/supersonic/common/util/jsqlparser/QueryExpressionReplaceVisitor.java

@@ -9,7 +9,7 @@ import net.sf.jsqlparser.expression.ExpressionVisitorAdapter;
 import net.sf.jsqlparser.expression.Function;
 import net.sf.jsqlparser.parser.CCJSqlParserUtil;
 import net.sf.jsqlparser.schema.Column;
-import net.sf.jsqlparser.statement.select.SelectExpressionItem;
+import net.sf.jsqlparser.statement.select.SelectItem;
 
 public class QueryExpressionReplaceVisitor extends ExpressionVisitorAdapter {
 
@@ -43,7 +43,7 @@ public class QueryExpressionReplaceVisitor extends ExpressionVisitorAdapter {
 
     }
 
-    public void visit(SelectExpressionItem selectExpressionItem) {
+    public void visit(SelectItem selectExpressionItem) {
 
         Expression expression = selectExpressionItem.getExpression();
         String toReplace = "";

common/src/main/java/com/tencent/supersonic/common/util/jsqlparser/SqlAddHelper.java

@@ -1,10 +1,12 @@
 package com.tencent.supersonic.common.util.jsqlparser;
 
+import java.util.ArrayList;
 import java.util.List;
 import java.util.Map;
 import java.util.Objects;
 import java.util.Set;
-import java.util.ArrayList;
+
+import com.tencent.supersonic.common.pojo.enums.TimeDimensionEnum;
 import lombok.extern.slf4j.Slf4j;
 import net.sf.jsqlparser.expression.Expression;
 import net.sf.jsqlparser.expression.Function;
@@ -12,6 +14,7 @@ import net.sf.jsqlparser.expression.LongValue;
 import net.sf.jsqlparser.expression.Parenthesis;
 import net.sf.jsqlparser.expression.StringValue;
 import net.sf.jsqlparser.expression.operators.conditional.AndExpression;
+import net.sf.jsqlparser.expression.operators.conditional.OrExpression;
 import net.sf.jsqlparser.expression.operators.relational.ComparisonOperator;
 import net.sf.jsqlparser.expression.operators.relational.EqualsTo;
 import net.sf.jsqlparser.schema.Column;
@@ -19,11 +22,10 @@ import net.sf.jsqlparser.statement.select.GroupByElement;
 import net.sf.jsqlparser.statement.select.OrderByElement;
 import net.sf.jsqlparser.statement.select.PlainSelect;
 import net.sf.jsqlparser.statement.select.Select;
-import net.sf.jsqlparser.statement.select.SelectBody;
-import net.sf.jsqlparser.statement.select.SelectExpressionItem;
 import net.sf.jsqlparser.statement.select.SelectItem;
 import net.sf.jsqlparser.statement.select.SelectVisitorAdapter;
 import net.sf.jsqlparser.statement.select.SetOperationList;
+import net.sf.jsqlparser.statement.select.ParenthesedSelect;
 import org.apache.commons.lang3.StringUtils;
 import org.springframework.util.CollectionUtils;
 
@@ -39,21 +41,23 @@ public class SqlAddHelper {
         if (selectStatement == null) {
             return null;
         }
-        SelectBody selectBody = selectStatement.getSelectBody();
-        if (selectBody instanceof PlainSelect) {
-            PlainSelect plainSelect = (PlainSelect) selectBody;
+        //SelectBody selectBody = selectStatement.getSelectBody();
+        if (selectStatement instanceof PlainSelect) {
+            PlainSelect plainSelect = (PlainSelect) selectStatement;
             fields.stream().filter(Objects::nonNull).forEach(field -> {
-                SelectExpressionItem selectExpressionItem = new SelectExpressionItem(new Column(field));
+                //SelectExpressionItem selectExpressionItem = new SelectExpressionItem(new Column(field));
+                SelectItem<Column> selectExpressionItem = new SelectItem(new Column(field));
                 plainSelect.addSelectItems(selectExpressionItem);
             });
 
-        } else if (selectBody instanceof SetOperationList) {
-            SetOperationList setOperationList = (SetOperationList) selectBody;
+        } else if (selectStatement instanceof SetOperationList) {
+            SetOperationList setOperationList = (SetOperationList) selectStatement;
             if (!CollectionUtils.isEmpty(setOperationList.getSelects())) {
                 setOperationList.getSelects().forEach(subSelectBody -> {
                     PlainSelect subPlainSelect = (PlainSelect) subSelectBody;
                     fields.stream().forEach(field -> {
-                        SelectExpressionItem selectExpressionItem = new SelectExpressionItem(new Column(field));
+                        //SelectExpressionItem selectExpressionItem = new SelectExpressionItem(new Column(field));
+                        SelectItem<Column> selectExpressionItem = new SelectItem(new Column(field));
                         subPlainSelect.addSelectItems(selectExpressionItem);
                     });
                 });
@@ -67,14 +71,14 @@ public class SqlAddHelper {
         if (selectStatement == null) {
             return null;
         }
-        SelectBody selectBody = selectStatement.getSelectBody();
+        //SelectBody selectBody = selectStatement.getSelectBody();
 
         List<PlainSelect> plainSelectList = new ArrayList<>();
-        if (selectBody instanceof PlainSelect) {
-            PlainSelect plainSelect = (PlainSelect) selectBody;
+        if (selectStatement instanceof PlainSelect) {
+            PlainSelect plainSelect = (PlainSelect) selectStatement.getPlainSelect();
             plainSelectList.add(plainSelect);
-        } else if (selectBody instanceof SetOperationList) {
-            SetOperationList setOperationList = (SetOperationList) selectBody;
+        } else if (selectStatement instanceof SetOperationList) {
+            SetOperationList setOperationList = (SetOperationList) selectStatement.getSetOperationList();
             if (!CollectionUtils.isEmpty(setOperationList.getSelects())) {
                 setOperationList.getSelects().forEach(subSelectBody -> {
                     PlainSelect subPlainSelect = (PlainSelect) subSelectBody;
@@ -87,16 +91,16 @@ public class SqlAddHelper {
             return sql;
         }
         for (PlainSelect plainSelect : plainSelectList) {
-            List<SelectItem> selectItems = plainSelect.getSelectItems();
+            List<SelectItem<?>> selectItems = plainSelect.getSelectItems();
             if (CollectionUtils.isEmpty(selectItems)) {
                 continue;
             }
             boolean existFunction = false;
             for (Expression expression : expressionList) {
                 for (SelectItem selectItem : selectItems) {
-                    SelectExpressionItem expressionItem = (SelectExpressionItem) selectItem;
-                    if (expressionItem.getExpression() instanceof Function) {
-                        Function expressionFunction = (Function) expressionItem.getExpression();
+                    //SelectExpressionItem expressionItem = (SelectExpressionItem) selectItem;
+                    if (selectItem.getExpression() instanceof Function) {
+                        Function expressionFunction = (Function) selectItem.getExpression();
                         if (expression.toString().equalsIgnoreCase(expressionFunction.toString())) {
                             existFunction = true;
                             break;
@@ -104,7 +108,8 @@ public class SqlAddHelper {
                     }
                 }
                 if (!existFunction) {
-                    SelectExpressionItem sumExpressionItem = new SelectExpressionItem(expression);
+                    //SelectExpressionItem sumExpressionItem = new SelectExpressionItem(expression);
+                    SelectItem sumExpressionItem = new SelectItem(expression);
                     selectItems.add(sumExpressionItem);
                 }
             }
@@ -117,12 +122,12 @@ public class SqlAddHelper {
             return sql;
         }
         Select selectStatement = SqlSelectHelper.getSelect(sql);
-        SelectBody selectBody = selectStatement.getSelectBody();
+        //SelectBody selectBody = selectStatement.getSelectBody();
 
-        if (!(selectBody instanceof PlainSelect)) {
+        if (!(selectStatement instanceof PlainSelect)) {
             return sql;
         }
-        PlainSelect plainSelect = (PlainSelect) selectBody;
+        PlainSelect plainSelect = (PlainSelect) selectStatement;
         Expression where = plainSelect.getWhere();
 
         Expression right = new StringValue(value.toString());
@@ -140,12 +145,30 @@ public class SqlAddHelper {
 
     public static String addWhere(String sql, Expression expression) {
         Select selectStatement = SqlSelectHelper.getSelect(sql);
-        SelectBody selectBody = selectStatement.getSelectBody();
+        //SelectBody selectBody = selectStatement.getSelectBody();
 
-        if (!(selectBody instanceof PlainSelect)) {
+        if (!(selectStatement instanceof PlainSelect)) {
             return sql;
         }
-        PlainSelect plainSelect = (PlainSelect) selectBody;
+        PlainSelect plainSelect = (PlainSelect) selectStatement;
+        List<String> chNameList = TimeDimensionEnum.getChNameList();
+        Boolean dateWhere = false;
+        for (String chName : chNameList) {
+            if (expression.toString().contains(chName)) {
+                dateWhere = true;
+            }
+        }
+        if (plainSelect.getFromItem() instanceof ParenthesedSelect && dateWhere) {
+            ParenthesedSelect parenthesedSelect = (ParenthesedSelect) plainSelect.getFromItem();
+            PlainSelect subPlainSelect = parenthesedSelect.getPlainSelect();
+            Expression subWhere = subPlainSelect.getWhere();
+            if (subWhere == null) {
+                subPlainSelect.setWhere(expression);
+            } else {
+                subPlainSelect.setWhere(new AndExpression(subWhere, expression));
+            }
+            return selectStatement.toString();
+        }
         Expression where = plainSelect.getWhere();
 
         if (where == null) {
@@ -158,9 +181,9 @@ public class SqlAddHelper {
 
     public static String addWhere(String sql, List<Expression> expressionList) {
         Select selectStatement = SqlSelectHelper.getSelect(sql);
-        SelectBody selectBody = selectStatement.getSelectBody();
+        //SelectBody selectBody = selectStatement.getSelectBody();
 
-        if (!(selectBody instanceof PlainSelect)) {
+        if (!(selectStatement instanceof PlainSelect)) {
             return sql;
         }
         if (CollectionUtils.isEmpty(expressionList)) {
@@ -170,7 +193,7 @@ public class SqlAddHelper {
         for (int i = 1; i < expressionList.size(); i++) {
             expression = new AndExpression(expression, expressionList.get(i));
         }
-        PlainSelect plainSelect = (PlainSelect) selectBody;
+        PlainSelect plainSelect = (PlainSelect) selectStatement;
         Expression where = plainSelect.getWhere();
 
         if (where == null) {
@@ -183,12 +206,12 @@ public class SqlAddHelper {
 
     public static String addAggregateToField(String sql, Map<String, String> fieldNameToAggregate) {
         Select selectStatement = SqlSelectHelper.getSelect(sql);
-        SelectBody selectBody = selectStatement.getSelectBody();
+        //SelectBody selectBody = selectStatement.getSelectBody();
 
-        if (!(selectBody instanceof PlainSelect)) {
+        if (!(selectStatement instanceof PlainSelect)) {
             return sql;
         }
-        selectBody.accept(new SelectVisitorAdapter() {
+        selectStatement.accept(new SelectVisitorAdapter() {
             @Override
             public void visit(PlainSelect plainSelect) {
                 addAggregateToSelectItems(plainSelect.getSelectItems(), fieldNameToAggregate);
@@ -205,13 +228,13 @@ public class SqlAddHelper {
             return sql;
         }
         Select selectStatement = SqlSelectHelper.getSelect(sql);
-        SelectBody selectBody = selectStatement.getSelectBody();
+        //SelectBody selectBody = selectStatement.getSelectBody();
 
-        if (!(selectBody instanceof PlainSelect)) {
+        if (!(selectStatement instanceof PlainSelect)) {
             return sql;
         }
 
-        PlainSelect plainSelect = (PlainSelect) selectBody;
+        PlainSelect plainSelect = (PlainSelect) selectStatement;
         GroupByElement groupByElement = new GroupByElement();
         List<String> originalGroupByFields = SqlSelectHelper.getGroupByFields(sql);
         if (!CollectionUtils.isEmpty(originalGroupByFields)) {
@@ -224,23 +247,20 @@ public class SqlAddHelper {
         return selectStatement.toString();
     }
 
-    private static void addAggregateToSelectItems(List<SelectItem> selectItems,
+    private static void addAggregateToSelectItems(List<SelectItem<?>> selectItems,
             Map<String, String> fieldNameToAggregate) {
         for (SelectItem selectItem : selectItems) {
-            if (selectItem instanceof SelectExpressionItem) {
-                SelectExpressionItem selectExpressionItem = (SelectExpressionItem) selectItem;
-                Expression expression = selectExpressionItem.getExpression();
-                Function function = SqlSelectFunctionHelper.getFunction(expression, fieldNameToAggregate);
-                if (function == null) {
-                    continue;
-                }
-                selectExpressionItem.setExpression(function);
+            Expression expression = selectItem.getExpression();
+            Function function = SqlSelectFunctionHelper.getFunction(expression, fieldNameToAggregate);
+            if (function == null) {
+                continue;
             }
+            selectItem.setExpression(function);
         }
     }
 
     private static void addAggregateToOrderByItems(List<OrderByElement> orderByElements,
-            Map<String, String> fieldNameToAggregate) {
+                                                   Map<String, String> fieldNameToAggregate) {
         if (orderByElements == null) {
             return;
         }
@@ -255,11 +275,12 @@ public class SqlAddHelper {
     }
 
     private static void addAggregateToGroupByItems(GroupByElement groupByElement,
-            Map<String, String> fieldNameToAggregate) {
+                                                   Map<String, String> fieldNameToAggregate) {
         if (groupByElement == null) {
             return;
         }
-        for (Expression expression : groupByElement.getGroupByExpressions()) {
+        for (int i = 0; i < groupByElement.getGroupByExpressionList().size(); i++) {
+            Expression expression = (Expression) groupByElement.getGroupByExpressionList().get(i);
             Function function = SqlSelectFunctionHelper.getFunction(expression, fieldNameToAggregate);
             if (function == null) {
                 continue;
@@ -276,13 +297,22 @@ public class SqlAddHelper {
     }
 
     private static void modifyWhereExpression(Expression whereExpression,
-            Map<String, String> fieldNameToAggregate) {
+                                              Map<String, String> fieldNameToAggregate) {
         if (SqlSelectHelper.isLogicExpression(whereExpression)) {
-            AndExpression andExpression = (AndExpression) whereExpression;
-            Expression leftExpression = andExpression.getLeftExpression();
-            Expression rightExpression = andExpression.getRightExpression();
-            modifyWhereExpression(leftExpression, fieldNameToAggregate);
-            modifyWhereExpression(rightExpression, fieldNameToAggregate);
+            if (whereExpression instanceof AndExpression) {
+                AndExpression andExpression = (AndExpression) whereExpression;
+                Expression leftExpression = andExpression.getLeftExpression();
+                Expression rightExpression = andExpression.getRightExpression();
+                modifyWhereExpression(leftExpression, fieldNameToAggregate);
+                modifyWhereExpression(rightExpression, fieldNameToAggregate);
+            }
+            if (whereExpression instanceof OrExpression) {
+                OrExpression orExpression = (OrExpression) whereExpression;
+                Expression leftExpression = orExpression.getLeftExpression();
+                Expression rightExpression = orExpression.getRightExpression();
+                modifyWhereExpression(leftExpression, fieldNameToAggregate);
+                modifyWhereExpression(rightExpression, fieldNameToAggregate);
+            }
         } else if (whereExpression instanceof Parenthesis) {
             modifyWhereExpression(((Parenthesis) whereExpression).getExpression(), fieldNameToAggregate);
         } else {
@@ -315,13 +345,13 @@ public class SqlAddHelper {
 
     public static String addHaving(String sql, Set<String> fieldNames) {
         Select selectStatement = SqlSelectHelper.getSelect(sql);
-        SelectBody selectBody = selectStatement.getSelectBody();
+        //SelectBody selectBody = selectStatement.getSelectBody();
 
-        if (!(selectBody instanceof PlainSelect)) {
+        if (!(selectStatement instanceof PlainSelect)) {
             return sql;
         }
 
-        PlainSelect plainSelect = (PlainSelect) selectBody;
+        PlainSelect plainSelect = (PlainSelect) selectStatement;
         //replace metric to 1 and 1 and add having metric
         Expression where = plainSelect.getWhere();
         FiledFilterReplaceVisitor visitor = new FiledFilterReplaceVisitor(fieldNames);
@@ -344,9 +374,9 @@ public class SqlAddHelper {
 
     public static String addHaving(String sql, List<Expression> expressionList) {
         Select selectStatement = SqlSelectHelper.getSelect(sql);
-        SelectBody selectBody = selectStatement.getSelectBody();
+        //SelectBody selectBody = selectStatement.getSelectBody();
 
-        if (!(selectBody instanceof PlainSelect)) {
+        if (!(selectStatement instanceof PlainSelect)) {
             return sql;
         }
         if (CollectionUtils.isEmpty(expressionList)) {
@@ -356,7 +386,7 @@ public class SqlAddHelper {
         for (int i = 1; i < expressionList.size(); i++) {
             expression = new AndExpression(expression, expressionList.get(i));
         }
-        PlainSelect plainSelect = (PlainSelect) selectBody;
+        PlainSelect plainSelect = (PlainSelect) selectStatement;
         Expression having = plainSelect.getHaving();
 
         if (having == null) {
@@ -369,12 +399,12 @@ public class SqlAddHelper {
 
     public static String addParenthesisToWhere(String sql) {
         Select selectStatement = SqlSelectHelper.getSelect(sql);
-        SelectBody selectBody = selectStatement.getSelectBody();
+        //SelectBody selectBody = selectStatement.getSelectBody();
 
-        if (!(selectBody instanceof PlainSelect)) {
+        if (!(selectStatement instanceof PlainSelect)) {
             return sql;
         }
-        PlainSelect plainSelect = (PlainSelect) selectBody;
+        PlainSelect plainSelect = (PlainSelect) selectStatement;
         Expression where = plainSelect.getWhere();
         if (Objects.nonNull(where)) {
             Parenthesis parenthesis = new Parenthesis(where);

common/src/main/java/com/tencent/supersonic/common/util/jsqlparser/SqlRemoveHelper.java

@@ -1,5 +1,9 @@
 package com.tencent.supersonic.common.util.jsqlparser;
 
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
 import lombok.extern.slf4j.Slf4j;
 import net.sf.jsqlparser.JSQLParserException;
 import net.sf.jsqlparser.expression.BinaryExpression;
@@ -10,7 +14,6 @@ import net.sf.jsqlparser.expression.operators.conditional.AndExpression;
 import net.sf.jsqlparser.expression.operators.conditional.OrExpression;
 import net.sf.jsqlparser.expression.operators.relational.ComparisonOperator;
 import net.sf.jsqlparser.expression.operators.relational.EqualsTo;
-import net.sf.jsqlparser.expression.operators.relational.NotEqualsTo;
 import net.sf.jsqlparser.expression.operators.relational.ExpressionList;
 import net.sf.jsqlparser.expression.operators.relational.GreaterThan;
 import net.sf.jsqlparser.expression.operators.relational.GreaterThanEquals;
@@ -18,21 +21,16 @@ import net.sf.jsqlparser.expression.operators.relational.InExpression;
 import net.sf.jsqlparser.expression.operators.relational.LikeExpression;
 import net.sf.jsqlparser.expression.operators.relational.MinorThan;
 import net.sf.jsqlparser.expression.operators.relational.MinorThanEquals;
+import net.sf.jsqlparser.expression.operators.relational.NotEqualsTo;
 import net.sf.jsqlparser.parser.CCJSqlParserUtil;
 import net.sf.jsqlparser.schema.Column;
 import net.sf.jsqlparser.statement.select.GroupByElement;
 import net.sf.jsqlparser.statement.select.PlainSelect;
 import net.sf.jsqlparser.statement.select.Select;
-import net.sf.jsqlparser.statement.select.SelectBody;
-import net.sf.jsqlparser.statement.select.SelectExpressionItem;
 import net.sf.jsqlparser.statement.select.SelectItem;
 import net.sf.jsqlparser.statement.select.SelectVisitorAdapter;
 import org.springframework.util.CollectionUtils;
 
-import java.util.List;
-import java.util.Set;
-import java.util.Objects;
-
 /**
  * Sql Parser remove Helper
  */
@@ -44,30 +42,49 @@ public class SqlRemoveHelper {
         if (selectStatement == null) {
             return sql;
         }
-        SelectBody selectBody = selectStatement.getSelectBody();
-        if (!(selectBody instanceof PlainSelect)) {
+        //SelectBody selectBody = selectStatement.getSelectBody();
+        if (!(selectStatement instanceof PlainSelect)) {
             return sql;
         }
-        List<SelectItem> selectItems = ((PlainSelect) selectBody).getSelectItems();
+        List<SelectItem<?>> selectItems = ((PlainSelect) selectStatement).getSelectItems();
         selectItems.removeIf(selectItem -> {
-            if (selectItem instanceof SelectExpressionItem) {
-                SelectExpressionItem selectExpressionItem = (SelectExpressionItem) selectItem;
-                String columnName = SqlSelectHelper.getColumnName(selectExpressionItem.getExpression());
-                return fields.contains(columnName);
+            String columnName = SqlSelectHelper.getColumnName(selectItem.getExpression());
+            return fields.contains(columnName);
+        });
+        return selectStatement.toString();
+    }
+
+    public static String removeSameFieldFromSelect(String sql) {
+        Select selectStatement = SqlSelectHelper.getSelect(sql);
+        if (selectStatement == null) {
+            return sql;
+        }
+        //SelectBody selectBody = selectStatement.getSelectBody();
+        if (!(selectStatement instanceof PlainSelect)) {
+            return sql;
+        }
+        List<SelectItem<?>> selectItems = ((PlainSelect) selectStatement).getSelectItems();
+        Set<String> fields = new HashSet<>();
+        selectItems.removeIf(selectItem -> {
+            String field = selectItem.getExpression().toString();
+            if (fields.contains(field)) {
+                return true;
             }
+            fields.add(field);
             return false;
         });
+        ((PlainSelect) selectStatement).setSelectItems(selectItems);
         return selectStatement.toString();
     }
 
     public static String removeWhereCondition(String sql, Set<String> removeFieldNames) {
         Select selectStatement = SqlSelectHelper.getSelect(sql);
-        SelectBody selectBody = selectStatement.getSelectBody();
+        //SelectBody selectBody = selectStatement.getSelectBody();
 
-        if (!(selectBody instanceof PlainSelect)) {
+        if (!(selectStatement instanceof PlainSelect)) {
             return sql;
         }
-        selectBody.accept(new SelectVisitorAdapter() {
+        selectStatement.accept(new SelectVisitorAdapter() {
             @Override
             public void visit(PlainSelect plainSelect) {
                 removeWhereCondition(plainSelect.getWhere(), removeFieldNames);
@@ -88,16 +105,16 @@ public class SqlRemoveHelper {
         if (selectStatement == null) {
             return sql;
         }
-        SelectBody selectBody = selectStatement.getSelectBody();
+        //SelectBody selectBody = selectStatement.getSelectBody();
 
-        if (!(selectBody instanceof PlainSelect)) {
+        if (!(selectStatement instanceof PlainSelect)) {
             return sql;
         }
-        Expression where = ((PlainSelect) selectBody).getWhere();
-        Expression having = ((PlainSelect) selectBody).getHaving();
+        Expression where = ((PlainSelect) selectStatement).getWhere();
+        Expression having = ((PlainSelect) selectStatement).getHaving();
         try {
-            ((PlainSelect) selectBody).setWhere(filteredExpression(where, SqlEditEnum.NUMBER_FILTER));
-            ((PlainSelect) selectBody).setHaving(filteredExpression(having, SqlEditEnum.NUMBER_FILTER));
+            ((PlainSelect) selectStatement).setWhere(filteredExpression(where, SqlEditEnum.NUMBER_FILTER));
+            ((PlainSelect) selectStatement).setHaving(filteredExpression(having, SqlEditEnum.NUMBER_FILTER));
         } catch (Exception e) {
             log.info("replaceFunction has an exception:{}", e.toString());
         }
@@ -167,7 +184,8 @@ public class SqlRemoveHelper {
                 InExpression constantExpression = (InExpression) CCJSqlParserUtil.parseCondExpression(
                         JsqlConstants.IN_CONSTANT);
                 inExpression.setLeftExpression(constantExpression.getLeftExpression());
-                inExpression.setRightItemsList(constantExpression.getRightItemsList());
+                //inExpression.setRightItemsList(constantExpression.getRightItemsList());
+                inExpression.setRightExpression(constantExpression.getRightExpression());
                 inExpression.setASTNode(constantExpression.getASTNode());
             } catch (JSQLParserException e) {
                 log.error("JSQLParserException", e);
@@ -193,12 +211,12 @@ public class SqlRemoveHelper {
 
     public static String removeHavingCondition(String sql, Set<String> removeFieldNames) {
         Select selectStatement = SqlSelectHelper.getSelect(sql);
-        SelectBody selectBody = selectStatement.getSelectBody();
+        //SelectBody selectBody = selectStatement.getSelectBody();
 
-        if (!(selectBody instanceof PlainSelect)) {
+        if (!(selectStatement instanceof PlainSelect)) {
             return sql;
         }
-        selectBody.accept(new SelectVisitorAdapter() {
+        selectStatement.accept(new SelectVisitorAdapter() {
             @Override
             public void visit(PlainSelect plainSelect) {
                 removeWhereCondition(plainSelect.getHaving(), removeFieldNames);
@@ -212,11 +230,11 @@ public class SqlRemoveHelper {
         if (selectStatement == null) {
             return sql;
         }
-        SelectBody selectBody = selectStatement.getSelectBody();
-        if (!(selectBody instanceof PlainSelect)) {
+        //SelectBody selectBody = selectStatement.getSelectBody();
+        if (!(selectStatement instanceof PlainSelect)) {
             return sql;
         }
-        GroupByElement groupByElement = ((PlainSelect) selectBody).getGroupBy();
+        GroupByElement groupByElement = ((PlainSelect) selectStatement).getGroupBy();
         if (groupByElement == null) {
             return sql;
         }
@@ -229,7 +247,7 @@ public class SqlRemoveHelper {
             return false;
         });
         if (CollectionUtils.isEmpty(groupByExpressionList.getExpressions())) {
-            ((PlainSelect) selectBody).setGroupByElement(null);
+            ((PlainSelect) selectStatement).setGroupByElement(null);
         }
         return selectStatement.toString();
     }

common/src/main/java/com/tencent/supersonic/common/util/jsqlparser/SqlReplaceHelper.java

@@ -3,19 +3,23 @@ package com.tencent.supersonic.common.util.jsqlparser;
 import com.tencent.supersonic.common.pojo.enums.AggOperatorEnum;
 import com.tencent.supersonic.common.util.StringUtil;
 import java.util.ArrayList;
+import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
 import java.util.Objects;
 import java.util.Set;
+import java.util.function.UnaryOperator;
 import lombok.extern.slf4j.Slf4j;
 import net.sf.jsqlparser.JSQLParserException;
 import net.sf.jsqlparser.expression.Alias;
 import net.sf.jsqlparser.expression.Expression;
 import net.sf.jsqlparser.expression.Function;
+import net.sf.jsqlparser.expression.LongValue;
 import net.sf.jsqlparser.expression.operators.conditional.AndExpression;
 import net.sf.jsqlparser.expression.operators.conditional.OrExpression;
 import net.sf.jsqlparser.expression.operators.relational.ComparisonOperator;
 import net.sf.jsqlparser.expression.operators.relational.EqualsTo;
+import net.sf.jsqlparser.expression.operators.relational.ExpressionList;
 import net.sf.jsqlparser.expression.operators.relational.GreaterThan;
 import net.sf.jsqlparser.expression.operators.relational.GreaterThanEquals;
 import net.sf.jsqlparser.expression.operators.relational.MinorThan;
@@ -27,14 +31,13 @@ import net.sf.jsqlparser.schema.Table;
 import net.sf.jsqlparser.statement.select.GroupByElement;
 import net.sf.jsqlparser.statement.select.Join;
 import net.sf.jsqlparser.statement.select.OrderByElement;
+import net.sf.jsqlparser.statement.select.ParenthesedFromItem;
+import net.sf.jsqlparser.statement.select.ParenthesedSelect;
 import net.sf.jsqlparser.statement.select.PlainSelect;
 import net.sf.jsqlparser.statement.select.Select;
-import net.sf.jsqlparser.statement.select.SelectBody;
-import net.sf.jsqlparser.statement.select.SelectExpressionItem;
 import net.sf.jsqlparser.statement.select.SelectItem;
 import net.sf.jsqlparser.statement.select.SelectVisitorAdapter;
 import net.sf.jsqlparser.statement.select.SetOperationList;
-import net.sf.jsqlparser.statement.select.SubSelect;
 import org.apache.commons.lang3.StringUtils;
 import org.apache.commons.lang3.tuple.Pair;
 import org.springframework.util.CollectionUtils;
@@ -47,12 +50,13 @@ public class SqlReplaceHelper {
 
     public static String replaceSelectFields(String sql, Map<String, String> fieldNameMap) {
         Select selectStatement = SqlSelectHelper.getSelect(sql);
-        SelectBody selectBody = selectStatement.getSelectBody();
-        if (!(selectBody instanceof PlainSelect)) {
+        //SelectBody selectBody = selectStatement.getSelectBody();
+        if (!(selectStatement instanceof PlainSelect)) {
             return sql;
         }
-        ((PlainSelect) selectBody).getSelectItems().stream().forEach(o -> {
-            SelectExpressionItem selectExpressionItem = (SelectExpressionItem) o;
+        ((PlainSelect) selectStatement).getSelectItems().stream().forEach(o -> {
+            //SelectExpressionItem selectExpressionItem = (SelectExpressionItem) o;
+            SelectItem selectExpressionItem = (SelectItem) o;
             String alias = "";
             if (selectExpressionItem.getExpression() instanceof Function) {
                 Function function = (Function) selectExpressionItem.getExpression();
@@ -60,9 +64,7 @@ public class SqlReplaceHelper {
                 if (fieldNameMap.containsKey(column.getColumnName())) {
                     String value = fieldNameMap.get(column.getColumnName());
                     alias = value;
-                    List<Expression> expressions = new ArrayList<>();
-                    expressions.add(new Column(value));
-                    function.getParameters().setExpressions(expressions);
+                    function.withParameters(new Column(value));
                 }
             }
             if (selectExpressionItem.getExpression() instanceof Column) {
@@ -85,12 +87,12 @@ public class SqlReplaceHelper {
 
     public static String replaceAggFields(String sql, Map<String, Pair<String, String>> fieldNameToAggMap) {
         Select selectStatement = SqlSelectHelper.getSelect(sql);
-        SelectBody selectBody = selectStatement.getSelectBody();
-        if (!(selectBody instanceof PlainSelect)) {
+
+        if (!(selectStatement instanceof PlainSelect)) {
             return sql;
         }
-        ((PlainSelect) selectBody).getSelectItems().stream().forEach(o -> {
-            SelectExpressionItem selectExpressionItem = (SelectExpressionItem) o;
+        ((PlainSelect) selectStatement).getSelectItems().stream().forEach(o -> {
+            SelectItem selectExpressionItem = (SelectItem) o;
             if (selectExpressionItem.getExpression() instanceof Function) {
                 Function function = (Function) selectExpressionItem.getExpression();
                 Column column = (Column) function.getParameters().getExpressions().get(0);
@@ -104,9 +106,7 @@ public class SqlReplaceHelper {
                     } else {
                         function.setName(func);
                     }
-                    List<Expression> expressions = new ArrayList<>();
-                    expressions.add(new Column(field));
-                    function.getParameters().setExpressions(expressions);
+                    function.withParameters(new Column(field));
                     if (Objects.nonNull(selectExpressionItem.getAlias()) && StringUtils.isNotBlank(field)) {
                         selectExpressionItem.getAlias().setName(field);
                     }
@@ -123,12 +123,12 @@ public class SqlReplaceHelper {
     public static String replaceValue(String sql, Map<String, Map<String, String>> filedNameToValueMap,
             boolean exactReplace) {
         Select selectStatement = SqlSelectHelper.getSelect(sql);
-        SelectBody selectBody = selectStatement.getSelectBody();
-        if (!(selectBody instanceof PlainSelect)) {
+        //SelectBody selectBody = selectStatement.getSelectBody();
+        if (!(selectStatement instanceof PlainSelect)) {
             return sql;
         }
         List<PlainSelect> plainSelectList = new ArrayList<>();
-        plainSelectList.add((PlainSelect) selectBody);
+        plainSelectList.add((PlainSelect) selectStatement);
         List<PlainSelect> plainSelects = SqlSelectHelper.getPlainSelects(plainSelectList);
         for (PlainSelect plainSelect : plainSelects) {
             Expression where = plainSelect.getWhere();
@@ -142,12 +142,12 @@ public class SqlReplaceHelper {
 
     public static String replaceFieldNameByValue(String sql, Map<String, Set<String>> fieldValueToFieldNames) {
         Select selectStatement = SqlSelectHelper.getSelect(sql);
-        SelectBody selectBody = selectStatement.getSelectBody();
-        if (!(selectBody instanceof PlainSelect)) {
+        //SelectBody selectBody = selectStatement.getSelectBody();
+        if (!(selectStatement instanceof PlainSelect)) {
             return sql;
         }
         List<PlainSelect> plainSelectList = new ArrayList<>();
-        plainSelectList.add((PlainSelect) selectBody);
+        plainSelectList.add((PlainSelect) selectStatement);
         List<PlainSelect> plainSelects = SqlSelectHelper.getPlainSelects(plainSelectList);
         for (PlainSelect plainSelect : plainSelects) {
             Expression where = plainSelect.getWhere();
@@ -166,20 +166,17 @@ public class SqlReplaceHelper {
     public static String replaceFields(String sql, Map<String, String> fieldNameMap, boolean exactReplace) {
         Select selectStatement = SqlSelectHelper.getSelect(sql);
         System.out.println(selectStatement.getSelectBody());
-        SelectBody selectBody = selectStatement.getSelectBody();
         List<PlainSelect> plainSelectList = new ArrayList<>();
-        if (selectBody instanceof PlainSelect) {
-            plainSelectList.add((PlainSelect) selectBody);
-        } else if (selectBody instanceof SetOperationList) {
-            SetOperationList setOperationList = (SetOperationList) selectBody;
-            //replace select
+        if (selectStatement instanceof PlainSelect) {
+            plainSelectList.add((PlainSelect) selectStatement);
+        } else if (selectStatement instanceof SetOperationList) {
+            SetOperationList setOperationList = (SetOperationList) selectStatement;
             if (!CollectionUtils.isEmpty(setOperationList.getSelects())) {
                 setOperationList.getSelects().forEach(subSelectBody -> {
                     PlainSelect subPlainSelect = (PlainSelect) subSelectBody;
                     plainSelectList.add(subPlainSelect);
                 });
             }
-            //replace order by
             List<OrderByElement> orderByElements = setOperationList.getOrderByElements();
             if (!CollectionUtils.isEmpty(orderByElements)) {
                 for (OrderByElement orderByElement : orderByElements) {
@@ -211,6 +208,12 @@ public class SqlReplaceHelper {
             replaceAsName(fieldNameMap, selectItem);
         }
 
+        if (plainSelect.getFromItem() instanceof ParenthesedSelect) {
+            ParenthesedSelect parenthesedSelect = (ParenthesedSelect) plainSelect.getFromItem();
+            PlainSelect subPlainSelect = parenthesedSelect.getPlainSelect();
+            replaceFieldsInPlainOneSelect(fieldNameMap, exactReplace, subPlainSelect);
+        }
+
         //3. replace oder by fields
         List<OrderByElement> orderByElements = plainSelect.getOrderByElements();
         if (!CollectionUtils.isEmpty(orderByElements)) {
@@ -232,12 +235,12 @@ public class SqlReplaceHelper {
         if (!CollectionUtils.isEmpty(joins)) {
             for (Join join : joins) {
                 join.getOnExpression().accept(visitor);
-                if (!(join.getRightItem() instanceof SubSelect)) {
+                if (!(join.getRightItem() instanceof ParenthesedSelect)) {
                     continue;
                 }
-                SelectBody subSelectBody = ((SubSelect) join.getRightItem()).getSelectBody();
+                // SelectBody subSelectBody = ((SubSelect) join.getRightItem()).getSelectBody();
                 List<PlainSelect> plainSelectList = new ArrayList<>();
-                plainSelectList.add((PlainSelect) subSelectBody);
+                plainSelectList.add((PlainSelect) join.getRightItem());
                 List<PlainSelect> subPlainSelects = SqlSelectHelper.getPlainSelects(plainSelectList);
                 for (PlainSelect subPlainSelect : subPlainSelects) {
                     replaceFieldsInPlainOneSelect(fieldNameMap, exactReplace, subPlainSelect);
@@ -247,47 +250,52 @@ public class SqlReplaceHelper {
     }
 
     private static void replaceAsName(Map<String, String> fieldNameMap, SelectItem selectItem) {
-        if (selectItem instanceof SelectExpressionItem) {
-            SelectExpressionItem expressionItem = (SelectExpressionItem) selectItem;
-            Alias alias = expressionItem.getAlias();
-            if (Objects.isNull(alias)) {
-                return;
-            }
-            String aliasName = alias.getName();
-            String replaceFieldName = fieldNameMap.get(aliasName);
-            if (StringUtils.isNotBlank(replaceFieldName)) {
-                alias.setName(replaceFieldName);
-            }
+
+        Alias alias = selectItem.getAlias();
+        if (Objects.isNull(alias)) {
+            return;
         }
+        String aliasName = alias.getName();
+        String replaceFieldName = fieldNameMap.get(aliasName);
+        if (StringUtils.isNotBlank(replaceFieldName)) {
+            alias.setName(replaceFieldName);
+        }
+
     }
 
     public static String replaceFunction(String sql, Map<String, String> functionMap) {
+        return replaceFunction(sql, functionMap, null);
+    }
+
+    public static String replaceFunction(String sql, Map<String, String> functionMap,
+            Map<String, UnaryOperator> functionCall) {
         Select selectStatement = SqlSelectHelper.getSelect(sql);
-        SelectBody selectBody = selectStatement.getSelectBody();
-        if (!(selectBody instanceof PlainSelect)) {
+        //SelectBody selectBody = selectStatement.getSelectBody();
+        if (!(selectStatement instanceof PlainSelect)) {
             return sql;
         }
         List<PlainSelect> plainSelectList = new ArrayList<>();
-        plainSelectList.add((PlainSelect) selectBody);
+        plainSelectList.add((PlainSelect) selectStatement);
         List<PlainSelect> plainSelects = SqlSelectHelper.getPlainSelects(plainSelectList);
         for (PlainSelect plainSelect : plainSelects) {
-            replaceFunction(functionMap, plainSelect);
+            replaceFunction(functionMap, functionCall, plainSelect);
         }
         return selectStatement.toString();
     }
 
-    private static void replaceFunction(Map<String, String> functionMap, PlainSelect selectBody) {
+    private static void replaceFunction(Map<String, String> functionMap, Map<String, UnaryOperator> functionCall,
+            PlainSelect selectBody) {
         PlainSelect plainSelect = selectBody;
         //1. replace where dataDiff function
         Expression where = plainSelect.getWhere();
 
-        FunctionNameReplaceVisitor visitor = new FunctionNameReplaceVisitor(functionMap);
+        FunctionNameReplaceVisitor visitor = new FunctionNameReplaceVisitor(functionMap, functionCall);
         if (Objects.nonNull(where)) {
             where.accept(visitor);
         }
         GroupByElement groupBy = plainSelect.getGroupBy();
         if (Objects.nonNull(groupBy)) {
-            GroupByFunctionReplaceVisitor replaceVisitor = new GroupByFunctionReplaceVisitor(functionMap);
+            GroupByFunctionReplaceVisitor replaceVisitor = new GroupByFunctionReplaceVisitor(functionMap, functionCall);
             groupBy.accept(replaceVisitor);
         }
 
@@ -304,14 +312,14 @@ public class SqlReplaceHelper {
 
     public static String replaceFunction(String sql) {
         Select selectStatement = SqlSelectHelper.getSelect(sql);
-        SelectBody selectBody = selectStatement.getSelectBody();
-        if (!(selectBody instanceof PlainSelect)) {
+        //SelectBody selectBody = selectStatement.getSelectBody();
+        if (!(selectStatement instanceof PlainSelect)) {
             return sql;
         }
-        Expression where = ((PlainSelect) selectBody).getWhere();
+        Expression where = ((PlainSelect) selectStatement).getWhere();
         try {
             Expression expression = SqlRemoveHelper.filteredExpression(where, SqlEditEnum.DATEDIFF);
-            ((PlainSelect) selectBody).setWhere(expression);
+            ((PlainSelect) selectStatement).setWhere(expression);
         } catch (Exception e) {
             log.info("replaceFunction has an exception:{}", e.toString());
         }
@@ -385,16 +393,18 @@ public class SqlReplaceHelper {
             return sql;
         }
         Select selectStatement = SqlSelectHelper.getSelect(sql);
-        SelectBody selectBody = selectStatement.getSelectBody();
-        if (selectBody instanceof PlainSelect) {
-            PlainSelect plainSelect = (PlainSelect) selectBody;
+        //SelectBody selectBody = selectStatement.getSelectBody();
+        if (selectStatement instanceof PlainSelect) {
+            PlainSelect plainSelect = (PlainSelect) selectStatement;
             replaceSingleTable(plainSelect, tableName);
-        } else if (selectBody instanceof SetOperationList) {
-            SetOperationList setOperationList = (SetOperationList) selectBody;
+            replaceSubTable(plainSelect, tableName);
+        } else if (selectStatement instanceof SetOperationList) {
+            SetOperationList setOperationList = (SetOperationList) selectStatement;
             if (!CollectionUtils.isEmpty(setOperationList.getSelects())) {
                 setOperationList.getSelects().forEach(subSelectBody -> {
                     PlainSelect subPlainSelect = (PlainSelect) subSelectBody;
                     replaceSingleTable(subPlainSelect, tableName);
+                    replaceSubTable(subPlainSelect, tableName);
                 });
             }
         }
@@ -402,6 +412,14 @@ public class SqlReplaceHelper {
         return selectStatement.toString();
     }
 
+    public static void replaceSubTable(PlainSelect plainSelect, String tableName) {
+        if (plainSelect.getFromItem() instanceof ParenthesedSelect) {
+            ParenthesedSelect parenthesedSelect = (ParenthesedSelect) plainSelect.getFromItem();
+            PlainSelect subPlainSelect = parenthesedSelect.getPlainSelect();
+            replaceSingleTable(subPlainSelect, tableName);
+        }
+    }
+
     public static void replaceSingleTable(PlainSelect plainSelect, String tableName) {
         // replace table name
         List<PlainSelect> plainSelects = new ArrayList<>();
@@ -418,10 +436,10 @@ public class SqlReplaceHelper {
             List<Join> joins = painSelect.getJoins();
             if (!CollectionUtils.isEmpty(joins)) {
                 for (Join join : joins) {
-                    if (join.getRightItem() instanceof SubSelect) {
-                        SelectBody subSelectBody = ((SubSelect) join.getRightItem()).getSelectBody();
+                    if (join.getRightItem() instanceof ParenthesedFromItem) {
+                        //SelectBody subSelectBody = ((SubSelect) join.getRightItem()).getSelectBody();
                         List<PlainSelect> plainSelectList = new ArrayList<>();
-                        plainSelectList.add((PlainSelect) subSelectBody);
+                        plainSelectList.add((PlainSelect) join.getRightItem());
                         List<PlainSelect> subPlainSelects = SqlSelectHelper.getPlainSelects(plainSelectList);
                         for (PlainSelect subPlainSelect : subPlainSelects) {
                             subPlainSelect.getFromItem().accept(new TableNameReplaceVisitor(tableName));
@@ -437,11 +455,11 @@ public class SqlReplaceHelper {
 
     public static String replaceAlias(String sql) {
         Select selectStatement = SqlSelectHelper.getSelect(sql);
-        SelectBody selectBody = selectStatement.getSelectBody();
-        if (!(selectBody instanceof PlainSelect)) {
+        //SelectBody selectBody = selectStatement.getSelectBody();
+        if (!(selectStatement instanceof PlainSelect)) {
             return sql;
         }
-        PlainSelect plainSelect = (PlainSelect) selectBody;
+        PlainSelect plainSelect = (PlainSelect) selectStatement;
         FunctionAliasReplaceVisitor visitor = new FunctionAliasReplaceVisitor();
         for (SelectItem selectItem : plainSelect.getSelectItems()) {
             selectItem.accept(visitor);
@@ -455,11 +473,11 @@ public class SqlReplaceHelper {
 
     public static String replaceHavingValue(String sql, Map<String, Map<String, String>> filedNameToValueMap) {
         Select selectStatement = SqlSelectHelper.getSelect(sql);
-        SelectBody selectBody = selectStatement.getSelectBody();
-        if (!(selectBody instanceof PlainSelect)) {
+        //SelectBody selectBody = selectStatement.getSelectBody();
+        if (!(selectStatement instanceof PlainSelect)) {
             return sql;
         }
-        PlainSelect plainSelect = (PlainSelect) selectBody;
+        PlainSelect plainSelect = (PlainSelect) selectStatement;
         Expression having = plainSelect.getHaving();
         FieldlValueReplaceVisitor visitor = new FieldlValueReplaceVisitor(false, filedNameToValueMap);
         if (Objects.nonNull(having)) {
@@ -473,7 +491,7 @@ public class SqlReplaceHelper {
             Function function = (Function) leftExpression;
             if (function.getName().equals(JsqlConstants.DATE_FUNCTION)) {
                 ComparisonOperator comparisonOperator = (ComparisonOperator) expression;
-                List<Expression> leftExpressions = function.getParameters().getExpressions();
+                ExpressionList<?> leftExpressions = function.getParameters();
                 Column field = (Column) function.getParameters().getExpressions().get(1);
                 String columnName = field.getColumnName();
                 try {
@@ -509,6 +527,50 @@ public class SqlReplaceHelper {
         }
     }
 
+    private static Select replaceAggAliasOrderItem(Select selectStatement) {
+        if (selectStatement instanceof PlainSelect) {
+            PlainSelect plainSelect = (PlainSelect) selectStatement;
+            if (Objects.nonNull(plainSelect.getOrderByElements())) {
+                Map<String, String> selectNames = new HashMap<>();
+                for (int i = 0; i < plainSelect.getSelectItems().size(); i++) {
+                    SelectItem<?> f = plainSelect.getSelectItem(i);
+                    if (Objects.nonNull(f.getAlias()) && f.getExpression() instanceof Function) {
+                        Function function = (Function) f.getExpression();
+                        String alias = f.getAlias().getName();
+                        if (function.getParameters().size() == 1 && function.getParameters().get(0) instanceof Column) {
+                            Column column = (Column) function.getParameters().get(0);
+                            if (column.getColumnName().equalsIgnoreCase(alias)) {
+                                selectNames.put(alias, String.valueOf(i + 1));
+                            }
+                        }
+                    }
+                }
+                plainSelect.getOrderByElements().stream().forEach(o -> {
+                    if (o.getExpression() instanceof Function) {
+                        Function function = (Function) o.getExpression();
+                        if (function.getParameters().size() == 1 && function.getParameters().get(0) instanceof Column) {
+                            Column column = (Column) function.getParameters().get(0);
+                            if (selectNames.containsKey(column.getColumnName())) {
+                                o.setExpression(new LongValue(selectNames.get(column.getColumnName())));
+                            }
+                        }
+                    }
+                });
+            }
+            if (plainSelect.getFromItem() instanceof ParenthesedSelect) {
+                ParenthesedSelect parenthesedSelect = (ParenthesedSelect) plainSelect.getFromItem();
+                parenthesedSelect.setSelect(replaceAggAliasOrderItem(parenthesedSelect.getSelect()));
+            }
+            return selectStatement;
+        }
+        return selectStatement;
+    }
+
+    public static String replaceAggAliasOrderItem(String sql) {
+        Select selectStatement = replaceAggAliasOrderItem(SqlSelectHelper.getSelect(sql));
+        return selectStatement.toString();
+    }
+
     public static String replaceExpression(String expr, Map<String, String> replace) {
         Expression expression = QueryExpressionReplaceVisitor.getExpression(expr);
         if (Objects.nonNull(expression)) {
@@ -524,12 +586,12 @@ public class SqlReplaceHelper {
 
     public static String replaceSqlByExpression(String sql, Map<String, String> replace) {
         Select selectStatement = SqlSelectHelper.getSelect(sql);
-        SelectBody selectBody = selectStatement.getSelectBody();
+        //SelectBody selectBody = selectStatement.getSelectBody();
         List<PlainSelect> plainSelectList = new ArrayList<>();
-        if (selectBody instanceof PlainSelect) {
-            plainSelectList.add((PlainSelect) selectBody);
-        } else if (selectBody instanceof SetOperationList) {
-            SetOperationList setOperationList = (SetOperationList) selectBody;
+        if (selectStatement instanceof PlainSelect) {
+            plainSelectList.add((PlainSelect) selectStatement);
+        } else if (selectStatement instanceof SetOperationList) {
+            SetOperationList setOperationList = (SetOperationList) selectStatement;
             if (!CollectionUtils.isEmpty(setOperationList.getSelects())) {
                 setOperationList.getSelects().forEach(subSelectBody -> {
                     PlainSelect subPlainSelect = (PlainSelect) subSelectBody;
@@ -569,5 +631,30 @@ public class SqlReplaceHelper {
             }
         }
     }
+
+    public static String dealAliasToOrderBy(String querySql) {
+        Select selectStatement = SqlSelectHelper.getSelect(querySql);
+        PlainSelect plainSelect = selectStatement.getPlainSelect();
+        List<SelectItem<?>> selectItemList = plainSelect.getSelectItems();
+        List<OrderByElement> orderByElementList = plainSelect.getOrderByElements();
+        if (CollectionUtils.isEmpty(orderByElementList)) {
+            return querySql;
+        }
+        Map<String, Expression> map = new HashMap<>();
+        for (int i = 0; i < selectItemList.size(); i++) {
+            if (!Objects.isNull(selectItemList.get(i).getAlias())) {
+                map.put(selectItemList.get(i).getAlias().getName(), selectItemList.get(i).getExpression());
+                selectItemList.get(i).setAlias(null);
+            }
+        }
+        for (OrderByElement orderByElement : orderByElementList) {
+            if (map.containsKey(orderByElement.getExpression().toString())) {
+                orderByElement.setExpression(map.get(orderByElement.getExpression().toString()));
+            }
+        }
+        plainSelect.setOrderByElements(orderByElementList);
+        return plainSelect.toString();
+    }
+
 }
 

common/src/main/java/com/tencent/supersonic/common/util/jsqlparser/SqlSelectFunctionHelper.java

@@ -1,6 +1,14 @@
 package com.tencent.supersonic.common.util.jsqlparser;
 
 import com.tencent.supersonic.common.pojo.enums.AggOperatorEnum;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
 import lombok.extern.slf4j.Slf4j;
 import net.sf.jsqlparser.expression.Expression;
 import net.sf.jsqlparser.expression.Function;
@@ -8,23 +16,18 @@ import net.sf.jsqlparser.expression.operators.relational.ExpressionList;
 import net.sf.jsqlparser.schema.Column;
 import net.sf.jsqlparser.statement.select.PlainSelect;
 import net.sf.jsqlparser.statement.select.Select;
-import net.sf.jsqlparser.statement.select.SelectBody;
 import net.sf.jsqlparser.statement.select.SelectItem;
 import org.apache.commons.lang3.StringUtils;
 import org.springframework.util.CollectionUtils;
 
-import java.util.HashSet;
-import java.util.List;
-import java.util.Map;
-import java.util.Objects;
-import java.util.Set;
-
 /**
  * Sql Parser Select function Helper
  */
 @Slf4j
 public class SqlSelectFunctionHelper {
 
+    public static List<String> aggregateFunctionName = Arrays.asList("SUM", "COUNT", "MAX", "MIN", "AVG");
+
     public static boolean hasAggregateFunction(String sql) {
         if (!CollectionUtils.isEmpty(getFunctions(sql))) {
             return true;
@@ -42,13 +45,13 @@ public class SqlSelectFunctionHelper {
 
     public static Set<String> getFunctions(String sql) {
         Select selectStatement = SqlSelectHelper.getSelect(sql);
-        SelectBody selectBody = selectStatement.getSelectBody();
+        //SelectBody selectBody = selectStatement.getSelectBody();
 
-        if (!(selectBody instanceof PlainSelect)) {
+        if (!(selectStatement instanceof PlainSelect)) {
             return new HashSet<>();
         }
-        PlainSelect plainSelect = (PlainSelect) selectBody;
-        List<SelectItem> selectItems = plainSelect.getSelectItems();
+        PlainSelect plainSelect = (PlainSelect) selectStatement;
+        List<SelectItem<?>> selectItems = plainSelect.getSelectItems();
         FunctionVisitor visitor = new FunctionVisitor();
         for (SelectItem selectItem : selectItems) {
             selectItem.accept(visitor);
@@ -86,5 +89,22 @@ public class SqlSelectFunctionHelper {
         return sumFunction;
     }
 
+    public static String getFirstAggregateFunctions(String expr) {
+        List<String> functions = getAggregateFunctions(expr);
+        return CollectionUtils.isEmpty(functions) ? "" : functions.get(0);
+    }
+
+    public static List<String> getAggregateFunctions(String expr) {
+        Expression expression = QueryExpressionReplaceVisitor.getExpression(expr);
+        if (Objects.nonNull(expression)) {
+            FunctionVisitor visitor = new FunctionVisitor();
+            expression.accept(visitor);
+            Set<String> functions = visitor.getFunctionNames();
+            return functions.stream()
+                    .filter(t -> aggregateFunctionName.contains(t.toUpperCase())).collect(Collectors.toList());
+        }
+        return new ArrayList<>();
+    }
+
 }
 

common/src/main/java/com/tencent/supersonic/common/util/jsqlparser/SqlSelectHelper.java

@@ -1,8 +1,11 @@
 package com.tencent.supersonic.common.util.jsqlparser;
 
+import com.tencent.supersonic.common.util.StringUtil;
 import java.util.ArrayList;
+import java.util.HashMap;
 import java.util.HashSet;
 import java.util.List;
+import java.util.Map;
 import java.util.Objects;
 import java.util.Set;
 import java.util.stream.Collectors;
@@ -21,22 +24,29 @@ import net.sf.jsqlparser.expression.WhenClause;
 import net.sf.jsqlparser.expression.operators.conditional.AndExpression;
 import net.sf.jsqlparser.expression.operators.conditional.OrExpression;
 import net.sf.jsqlparser.expression.operators.conditional.XorExpression;
+import net.sf.jsqlparser.expression.operators.relational.Between;
 import net.sf.jsqlparser.expression.operators.relational.ComparisonOperator;
+import net.sf.jsqlparser.expression.operators.relational.ExpressionList;
+import net.sf.jsqlparser.expression.operators.relational.InExpression;
+import net.sf.jsqlparser.expression.operators.relational.IsBooleanExpression;
+import net.sf.jsqlparser.expression.operators.relational.IsNullExpression;
 import net.sf.jsqlparser.parser.CCJSqlParserUtil;
 import net.sf.jsqlparser.schema.Column;
 import net.sf.jsqlparser.schema.Table;
 import net.sf.jsqlparser.statement.Statement;
 import net.sf.jsqlparser.statement.select.Distinct;
 import net.sf.jsqlparser.statement.select.GroupByElement;
+import net.sf.jsqlparser.statement.select.Join;
+import net.sf.jsqlparser.statement.select.LateralView;
 import net.sf.jsqlparser.statement.select.OrderByElement;
+import net.sf.jsqlparser.statement.select.ParenthesedSelect;
 import net.sf.jsqlparser.statement.select.PlainSelect;
 import net.sf.jsqlparser.statement.select.Select;
-import net.sf.jsqlparser.statement.select.SelectBody;
-import net.sf.jsqlparser.statement.select.SelectExpressionItem;
 import net.sf.jsqlparser.statement.select.SelectItem;
 import net.sf.jsqlparser.statement.select.SelectVisitorAdapter;
 import net.sf.jsqlparser.statement.select.SetOperationList;
-import net.sf.jsqlparser.statement.select.SubSelect;
+import net.sf.jsqlparser.statement.select.WithItem;
+import net.sf.jsqlparser.statement.select.Limit;
 import org.apache.commons.lang3.StringUtils;
 import org.springframework.util.CollectionUtils;
 
@@ -98,7 +108,7 @@ public class SqlSelectHelper {
     public static Set<String> getSelectFields(List<PlainSelect> plainSelectList) {
         Set<String> result = new HashSet<>();
         plainSelectList.stream().forEach(plainSelect -> {
-            List<SelectItem> selectItems = plainSelect.getSelectItems();
+            List<SelectItem<?>> selectItems = plainSelect.getSelectItems();
             for (SelectItem selectItem : selectItems) {
                 selectItem.accept(new FieldAcquireVisitor(result));
             }
@@ -111,24 +121,33 @@ public class SqlSelectHelper {
         if (selectStatement == null) {
             return null;
         }
-        SelectBody selectBody = selectStatement.getSelectBody();
+        //SelectBody selectBody = selectStatement.getSelectBody();
 
         List<PlainSelect> plainSelectList = new ArrayList<>();
-        if (selectBody instanceof PlainSelect) {
-            PlainSelect plainSelect = (PlainSelect) selectBody;
-            plainSelectList.add(plainSelect);
-        } else if (selectBody instanceof SetOperationList) {
-            SetOperationList setOperationList = (SetOperationList) selectBody;
+        if (selectStatement instanceof PlainSelect) {
+            PlainSelect plainSelect = (PlainSelect) selectStatement;
+            getSubPlainSelect(plainSelect, plainSelectList);
+        } else if (selectStatement instanceof SetOperationList) {
+            SetOperationList setOperationList = (SetOperationList) selectStatement;
             if (!CollectionUtils.isEmpty(setOperationList.getSelects())) {
                 setOperationList.getSelects().forEach(subSelectBody -> {
                     PlainSelect subPlainSelect = (PlainSelect) subSelectBody;
-                    plainSelectList.add(subPlainSelect);
+                    getSubPlainSelect(subPlainSelect, plainSelectList);
                 });
             }
         }
         return plainSelectList;
     }
 
+    public static void getSubPlainSelect(PlainSelect plainSelect, List<PlainSelect> plainSelectList) {
+        plainSelectList.add(plainSelect);
+        if (plainSelect.getFromItem() instanceof ParenthesedSelect) {
+            ParenthesedSelect parenthesedSelect = (ParenthesedSelect) plainSelect.getFromItem();
+            PlainSelect subPlainSelect = parenthesedSelect.getPlainSelect();
+            getSubPlainSelect(subPlainSelect, plainSelectList);
+        }
+    }
+
     public static Select getSelect(String sql) {
         Statement statement = null;
         try {
@@ -138,6 +157,11 @@ public class SqlSelectHelper {
             return null;
         }
 
+        if (statement instanceof ParenthesedSelect) {
+            ParenthesedSelect parenthesedSelect = (ParenthesedSelect) statement;
+            return parenthesedSelect.getSelect();
+        }
+
         if (!(statement instanceof Select)) {
             return null;
         }
@@ -150,10 +174,12 @@ public class SqlSelectHelper {
             plainSelects.add(plainSelect);
             ExpressionVisitorAdapter expressionVisitor = new ExpressionVisitorAdapter() {
                 @Override
-                public void visit(SubSelect subSelect) {
-                    SelectBody subSelectBody = subSelect.getSelectBody();
-                    if (subSelectBody instanceof PlainSelect) {
-                        plainSelects.add((PlainSelect) subSelectBody);
+                public void visit(Select subSelect) {
+                    if (subSelect instanceof ParenthesedSelect) {
+                        ParenthesedSelect parenthesedSelect = (ParenthesedSelect) subSelect;
+                        if (parenthesedSelect.getSelect() instanceof PlainSelect) {
+                            plainSelects.add(parenthesedSelect.getPlainSelect());
+                        }
                     }
                 }
             };
@@ -169,7 +195,7 @@ public class SqlSelectHelper {
                     if (Objects.nonNull(having)) {
                         having.accept(expressionVisitor);
                     }
-                    List<SelectItem> selectItems = plainSelect.getSelectItems();
+                    List<SelectItem<?>> selectItems = plainSelect.getSelectItems();
                     if (!CollectionUtils.isEmpty(selectItems)) {
                         for (SelectItem selectItem : selectItems) {
                             selectItem.accept(expressionVisitor);
@@ -207,6 +233,8 @@ public class SqlSelectHelper {
 
         getHavingFields(plainSelect, result);
 
+        getLateralViewsFields(plainSelect, result);
+
         return new ArrayList<>(result);
     }
 
@@ -218,6 +246,17 @@ public class SqlSelectHelper {
 
     }
 
+    private static void getLateralViewsFields(PlainSelect plainSelect, Set<String> result) {
+        List<LateralView> lateralViews = plainSelect.getLateralViews();
+        if (!CollectionUtils.isEmpty(lateralViews)) {
+            lateralViews.stream().forEach(l -> {
+                if (Objects.nonNull(l.getGeneratorFunction())) {
+                    l.getGeneratorFunction().accept(new FieldAcquireVisitor(result));
+                }
+            });
+        }
+    }
+
     public static List<Expression> getHavingExpression(String sql) {
         List<PlainSelect> plainSelectList = getPlainSelect(sql);
         List<Expression> expressionList = new ArrayList<>();
@@ -249,6 +288,14 @@ public class SqlSelectHelper {
             if (Objects.nonNull(where)) {
                 where.accept(new FieldAndValueAcquireVisitor(result));
             }
+            if (plainSelect.getFromItem() instanceof ParenthesedSelect) {
+                ParenthesedSelect parenthesedSelect = (ParenthesedSelect) plainSelect.getFromItem();
+                PlainSelect subPlainSelect = parenthesedSelect.getPlainSelect();
+                Expression subWhere = subPlainSelect.getWhere();
+                if (Objects.nonNull(subWhere)) {
+                    subWhere.accept(new FieldAndValueAcquireVisitor(result));
+                }
+            }
         }
         return new ArrayList<>(result);
     }
@@ -339,7 +386,7 @@ public class SqlSelectHelper {
 
     public static String getTableName(String sql) {
         Table table = getTable(sql);
-        return table.getName();
+        return StringUtil.replaceBackticks(table.getName());
     }
 
     public static List<String> getAggregateFields(String sql) {
@@ -349,19 +396,18 @@ public class SqlSelectHelper {
             if (Objects.isNull(plainSelect)) {
                 continue;
             }
-            List<SelectItem> selectItems = plainSelect.getSelectItems();
+            List<SelectItem<?>> selectItems = plainSelect.getSelectItems();
             for (SelectItem selectItem : selectItems) {
-                if (selectItem instanceof SelectExpressionItem) {
-                    SelectExpressionItem expressionItem = (SelectExpressionItem) selectItem;
-                    if (expressionItem.getExpression() instanceof Function) {
-                        Function function = (Function) expressionItem.getExpression();
-                        if (Objects.nonNull(function.getParameters())
-                                && !CollectionUtils.isEmpty(function.getParameters().getExpressions())) {
-                            String columnName = function.getParameters().getExpressions().get(0).toString();
-                            result.add(columnName);
-                        }
+
+                if (selectItem.getExpression() instanceof Function) {
+                    Function function = (Function) selectItem.getExpression();
+                    if (Objects.nonNull(function.getParameters())
+                            && !CollectionUtils.isEmpty(function.getParameters().getExpressions())) {
+                        String columnName = function.getParameters().getExpressions().get(0).toString();
+                        result.add(columnName);
                     }
                 }
+
             }
         }
         return new ArrayList<>(result);
@@ -374,24 +420,23 @@ public class SqlSelectHelper {
             if (Objects.isNull(plainSelect)) {
                 continue;
             }
-            List<SelectItem> selectItems = plainSelect.getSelectItems();
+            List<SelectItem<?>> selectItems = plainSelect.getSelectItems();
             for (SelectItem selectItem : selectItems) {
-                if (selectItem instanceof SelectExpressionItem) {
-                    SelectExpressionItem expressionItem = (SelectExpressionItem) selectItem;
-                    if (expressionItem.getExpression() instanceof Function) {
-                        Function function = (Function) expressionItem.getExpression();
-                        Alias alias = expressionItem.getAlias();
-                        if (alias != null && StringUtils.isNotBlank(alias.getName())) {
-                            result.add(alias.getName());
-                        } else {
-                            if (Objects.nonNull(function.getParameters())
-                                    && !CollectionUtils.isEmpty(function.getParameters().getExpressions())) {
-                                String columnName = function.getParameters().getExpressions().get(0).toString();
-                                result.add(columnName);
-                            }
+
+                if (selectItem.getExpression() instanceof Function) {
+                    Function function = (Function) selectItem.getExpression();
+                    Alias alias = selectItem.getAlias();
+                    if (alias != null && StringUtils.isNotBlank(alias.getName())) {
+                        result.add(alias.getName());
+                    } else {
+                        if (Objects.nonNull(function.getParameters())
+                                && !CollectionUtils.isEmpty(function.getParameters().getExpressions())) {
+                            String columnName = function.getParameters().getExpressions().get(0).toString();
+                            result.add(columnName);
                         }
                     }
                 }
+
             }
         }
         return new ArrayList<>(result);
@@ -399,12 +444,12 @@ public class SqlSelectHelper {
 
     public static boolean hasGroupBy(String sql) {
         Select selectStatement = getSelect(sql);
-        SelectBody selectBody = selectStatement.getSelectBody();
+        //SelectBody selectBody = selectStatement.getSelectBody();
 
-        if (!(selectBody instanceof PlainSelect)) {
+        if (!(selectStatement instanceof PlainSelect)) {
             return false;
         }
-        PlainSelect plainSelect = (PlainSelect) selectBody;
+        PlainSelect plainSelect = (PlainSelect) selectStatement;
         GroupByElement groupBy = plainSelect.getGroupBy();
         if (Objects.nonNull(groupBy)) {
             GroupByVisitor replaceVisitor = new GroupByVisitor();
@@ -416,12 +461,12 @@ public class SqlSelectHelper {
 
     public static boolean hasDistinct(String sql) {
         Select selectStatement = getSelect(sql);
-        SelectBody selectBody = selectStatement.getSelectBody();
+        //SelectBody selectBody = selectStatement.getSelectBody();
 
-        if (!(selectBody instanceof PlainSelect)) {
+        if (!(selectStatement instanceof PlainSelect)) {
             return false;
         }
-        PlainSelect plainSelect = (PlainSelect) selectBody;
+        PlainSelect plainSelect = (PlainSelect) selectStatement;
         Distinct distinct = plainSelect.getDistinct();
         return Objects.nonNull(distinct);
     }
@@ -436,7 +481,8 @@ public class SqlSelectHelper {
             return ((Column) leftExpression).getColumnName();
         }
         if (leftExpression instanceof Function) {
-            List<Expression> expressionList = ((Function) leftExpression).getParameters().getExpressions();
+            ExpressionList<?> expressionList = ((Function) leftExpression).getParameters();
+
             if (!CollectionUtils.isEmpty(expressionList) && expressionList.get(0) instanceof Column) {
                 return ((Column) expressionList.get(0)).getColumnName();
             }
@@ -454,8 +500,8 @@ public class SqlSelectHelper {
         }
         if (leftExpression instanceof Function) {
             Function function = (Function) leftExpression;
-            if (!CollectionUtils.isEmpty(function.getParameters().getExpressions())) {
-                Expression expression = function.getParameters().getExpressions().get(0);
+            if (!CollectionUtils.isEmpty(function.getParameters())) {
+                Expression expression = (Expression) function.getParameters().get(0);
                 if (expression instanceof Column) {
                     return ((Column) expression).getColumnName();
                 }
@@ -481,20 +527,21 @@ public class SqlSelectHelper {
         if (selectStatement == null) {
             return null;
         }
-        SelectBody selectBody = selectStatement.getSelectBody();
-        if (selectBody instanceof PlainSelect) {
-            PlainSelect plainSelect = (PlainSelect) selectBody;
+        //SelectBody selectBody = selectStatement.getSelectBody();
+        if (selectStatement instanceof PlainSelect) {
+            PlainSelect plainSelect = (PlainSelect) selectStatement;
             if (plainSelect.getFromItem() instanceof Table) {
                 return (Table) plainSelect.getFromItem();
             }
-            if (plainSelect.getFromItem() instanceof SubSelect) {
+            if (plainSelect.getFromItem() instanceof ParenthesedSelect) {
 
-                SubSelect subSelect = (SubSelect) plainSelect.getFromItem();
+                //ParenthesedFromItem subSelect = (ParenthesedFromItem) plainSelect.getFromItem();
+                PlainSelect subSelect = ((ParenthesedSelect) plainSelect.getFromItem()).getPlainSelect();
                 return getTable(subSelect.getSelectBody().toString());
             }
 
-        } else if (selectBody instanceof SetOperationList) {
-            SetOperationList setOperationList = (SetOperationList) selectBody;
+        } else if (selectStatement instanceof SetOperationList) {
+            SetOperationList setOperationList = (SetOperationList) selectStatement;
             if (!CollectionUtils.isEmpty(setOperationList.getSelects())) {
                 return (Table) ((PlainSelect) setOperationList.getSelects().get(0)).getFromItem();
             }
@@ -521,7 +568,8 @@ public class SqlSelectHelper {
             columns.add(((Column) expression).getColumnName());
         }
         if (expression instanceof Function) {
-            List<Expression> expressionList = ((Function) expression).getParameters().getExpressions();
+            //List<Expression> expressionList = ((Function) expression).getParameters().getExpressions();
+            ExpressionList<?> expressionList = ((Function) expression).getParameters();
             for (Expression expr : expressionList) {
                 getColumnFromExpr(expr, columns);
             }
@@ -543,10 +591,84 @@ public class SqlSelectHelper {
             getColumnFromExpr(expr.getLeftExpression(), columns);
             getColumnFromExpr(expr.getRightExpression(), columns);
         }
+        if (expression instanceof InExpression) {
+            InExpression inExpression = (InExpression) expression;
+            getColumnFromExpr(inExpression.getLeftExpression(), columns);
+        }
+        if (expression instanceof Between) {
+            Between between = (Between) expression;
+            getColumnFromExpr(between.getLeftExpression(), columns);
+        }
+        if (expression instanceof IsBooleanExpression) {
+            IsBooleanExpression isBooleanExpression = (IsBooleanExpression) expression;
+            getColumnFromExpr(isBooleanExpression.getLeftExpression(), columns);
+        }
+        if (expression instanceof IsNullExpression) {
+            IsNullExpression isNullExpression = (IsNullExpression) expression;
+            getColumnFromExpr(isNullExpression.getLeftExpression(), columns);
+        }
         if (expression instanceof Parenthesis) {
             Parenthesis expr = (Parenthesis) expression;
             getColumnFromExpr(expr.getExpression(), columns);
         }
     }
+
+    public static Boolean hasLimit(String querySql) {
+        Select selectStatement = SqlSelectHelper.getSelect(querySql);
+        PlainSelect plainSelect = selectStatement.getPlainSelect();
+        Limit limit = plainSelect.getLimit();
+        if (Objects.nonNull(limit)) {
+            return true;
+        } else {
+            return false;
+        }
+    }
+
+    public static Map<String, Set<String>> getFieldsWithSubQuery(String sql) {
+        List<PlainSelect> plainSelects = getPlainSelects(getPlainSelect(sql));
+        Map<String, Set<String>> results = new HashMap<>();
+        for (PlainSelect plainSelect : plainSelects) {
+            getFieldsWithSubQuery(plainSelect, results);
+        }
+        return results;
+    }
+
+    private static void getFieldsWithSubQuery(PlainSelect plainSelect, Map<String, Set<String>> fields) {
+        if (plainSelect.getFromItem() instanceof Table) {
+            boolean isWith = false;
+            if (!CollectionUtils.isEmpty(plainSelect.getWithItemsList())) {
+                for (WithItem withItem : plainSelect.getWithItemsList()) {
+                    if (Objects.nonNull(withItem.getSelect())) {
+                        getFieldsWithSubQuery(withItem.getSelect().getPlainSelect(), fields);
+                        isWith = true;
+                    }
+                }
+            }
+            if (!isWith) {
+                Table table = (Table) plainSelect.getFromItem();
+                if (!fields.containsKey(table.getFullyQualifiedName())) {
+                    fields.put(table.getFullyQualifiedName(), new HashSet<>());
+                }
+                List<String> sqlFields = getFieldsByPlainSelect(plainSelect).stream().map(f -> f.replaceAll("`", ""))
+                        .collect(
+                                Collectors.toList());
+                fields.get(table.getFullyQualifiedName()).addAll(sqlFields);
+            }
+        }
+        if (plainSelect.getFromItem() instanceof ParenthesedSelect) {
+            ParenthesedSelect parenthesedSelect = (ParenthesedSelect) plainSelect.getFromItem();
+            getFieldsWithSubQuery(parenthesedSelect.getPlainSelect(), fields);
+            if (!CollectionUtils.isEmpty(plainSelect.getJoins())) {
+                for (Join join : plainSelect.getJoins()) {
+                    if (join.getRightItem() instanceof ParenthesedSelect) {
+                        getFieldsWithSubQuery(((ParenthesedSelect) join.getRightItem()).getPlainSelect(), fields);
+                    }
+                    if (join.getFromItem() instanceof ParenthesedSelect) {
+                        getFieldsWithSubQuery(((ParenthesedSelect) join.getFromItem()).getPlainSelect(), fields);
+                    }
+                }
+            }
+        }
+    }
 }
 

common/src/test/java/com/tencent/supersonic/common/util/jsqlparser/SqlEqualHelperTest.java

@@ -1,7 +1,7 @@
 package com.tencent.supersonic.common.util.jsqlparser;
 
 
-import cn.hutool.core.lang.Assert;
+import org.junit.Assert;
 import org.junit.jupiter.api.Test;
 
 /**
@@ -14,27 +14,29 @@ class SqlEqualHelperTest {
     void testEquals() {
         String sql1 = "SELECT * FROM table1 WHERE column1 = 1 AND column2 = 2";
         String sql2 = "SELECT * FROM table1 WHERE column2 = 2 AND column1 = 1";
-        Assert.equals(SqlEqualHelper.equals(sql1, sql2), true);
+        Assert.assertEquals(SqlEqualHelper.equals(sql1, sql2), true);
 
         sql1 = "SELECT a,b,c,d FROM table1 WHERE column1 = 1 AND column2 = 2 order by a";
         sql2 = "SELECT d,c,b,a FROM table1 WHERE column2 = 2 AND column1 = 1 order by a";
-        Assert.equals(SqlEqualHelper.equals(sql1, sql2), true);
+        Assert.assertEquals(SqlEqualHelper.equals(sql1, sql2), true);
 
         sql1 = "SELECT a,sum(b),sum(c),sum(d) FROM table1 WHERE column1 = 1 AND column2 = 2 group by a order by a";
 
         sql2 = "SELECT sum(d),sum(c),sum(b),a FROM table1 WHERE column2 = 2 AND column1 = 1 group by a order by a";
 
-        Assert.equals(SqlEqualHelper.equals(sql1, sql2), true);
+        Assert.assertEquals(SqlEqualHelper.equals(sql1, sql2), true);
 
         sql1 = "SELECT a,sum(b),sum(c),sum(d) FROM table1 WHERE column1 = 1 AND column2 = 2 group by a order by a";
 
         sql2 = "SELECT sum(d),sum(c),sum(b),a FROM table1 WHERE column2 = 2 AND column1 = 1 group by a order by a";
 
-        Assert.equals(SqlEqualHelper.equals(sql1, sql2), true);
+        Assert.assertEquals(SqlEqualHelper.equals(sql1, sql2), true);
+
 
         sql1 = "SELECT a,b,c,d FROM table1 WHERE column1 = 1 AND column2 = 2 order by a";
         sql2 = "SELECT d,c,b,f FROM table1 WHERE column2 = 2 AND column1 = 1 order by a";
-        Assert.equals(SqlEqualHelper.equals(sql1, sql2), false);
+        Assert.assertEquals(SqlEqualHelper.equals(sql1, sql2), false);
+
 
         sql1 = "SELECT\n"
                 + "页面,\n"
@@ -63,7 +65,6 @@ class SqlEqualHelperTest {
                 + "页面\n"
                 + "LIMIT\n"
                 + "365";
-
-        Assert.equals(SqlEqualHelper.equals(sql1, sql2), true);
+        Assert.assertEquals(SqlEqualHelper.equals(sql1, sql2), true);
     }
 }

common/src/test/java/com/tencent/supersonic/common/util/jsqlparser/SqlRemoveHelperTest.java

@@ -1,6 +1,7 @@
 package com.tencent.supersonic.common.util.jsqlparser;
 
 import java.util.HashSet;
+import java.util.List;
 import java.util.Set;
 import org.junit.Assert;
 import org.junit.jupiter.api.Test;
@@ -10,6 +11,21 @@ import org.junit.jupiter.api.Test;
  */
 class SqlRemoveHelperTest {
 
+    @Test
+    void testRemoveSameFieldFromSelect() {
+        String sql = "select 歌曲名,歌手名,粉丝数,粉丝数,sum(粉丝数),sum(粉丝数),avg(播放量),avg(播放量)"
+                + " from 歌曲库 where sum(粉丝数) > 20000 and  2>1 and "
+                + "sum(播放量) > 20000 and 1=1  HAVING sum(播放量) > 20000 and 3>1";
+        sql = SqlRemoveHelper.removeSameFieldFromSelect(sql);
+        System.out.println(sql);
+        sql = "SELECT 结算播放量 FROM 艺人 WHERE (歌手名 IN ('林俊杰', '陈奕迅')) AND (数据日期 >= '2024-04-04' AND 数据日期 <= '2024-04-04')";
+        List<FieldExpression> fieldExpressionList = SqlSelectHelper.getWhereExpressions(sql);
+        fieldExpressionList.stream().forEach(fieldExpression -> {
+            System.out.println(fieldExpression.toString());
+        });
+
+    }
+
     @Test
     void testRemoveWhereHavingCondition() {
         String sql = "select 歌曲名 from 歌曲库 where sum(粉丝数) > 20000 and  2>1 and "

common/src/test/java/com/tencent/supersonic/common/util/jsqlparser/SqlReplaceHelperTest.java

@@ -1,15 +1,14 @@
 package com.tencent.supersonic.common.util.jsqlparser;
 
 import com.tencent.supersonic.common.pojo.enums.AggOperatorEnum;
-import org.apache.commons.lang3.tuple.Pair;
-import org.junit.Assert;
-import org.junit.jupiter.api.Test;
-
 import java.util.Collections;
 import java.util.HashMap;
 import java.util.HashSet;
 import java.util.Map;
 import java.util.Set;
+import org.apache.commons.lang3.tuple.Pair;
+import org.junit.Assert;
+import org.junit.jupiter.api.Test;
 
 /**
  * SqlParserReplaceHelperTest
@@ -479,6 +478,17 @@ class SqlReplaceHelperTest {
 
     }
 
+    @Test
+    void testReplaceAggAliasOrderItem() {
+        String sql = "SELECT SUM(访问次数) AS top10总播放量 FROM (SELECT 部门, SUM(访问次数) AS 访问次数 FROM 超音数  "
+                + "GROUP BY 部门 ORDER BY SUM(访问次数) DESC LIMIT 10) AS top10";
+        String replaceSql = SqlReplaceHelper.replaceAggAliasOrderItem(sql);
+        Assert.assertEquals(
+                "SELECT SUM(访问次数) AS top10总播放量 FROM (SELECT 部门, SUM(访问次数) AS 访问次数 FROM 超音数 "
+                        + "GROUP BY 部门 ORDER BY 2 DESC LIMIT 10) AS top10",
+                replaceSql);
+    }
+
     private Map<String, String> initParams() {
         Map<String, String> fieldToBizName = new HashMap<>();
         fieldToBizName.put("部门", "department");

common/src/test/java/com/tencent/supersonic/common/util/jsqlparser/SqlSelectHelperTest.java

@@ -1,11 +1,12 @@
 package com.tencent.supersonic.common.util.jsqlparser;
 
-import java.util.List;
 import net.sf.jsqlparser.expression.Expression;
 import net.sf.jsqlparser.statement.select.Select;
 import org.junit.Assert;
 import org.junit.jupiter.api.Test;
 
+import java.util.List;
+
 /**
  * SqlParserSelectHelper Test
  */
@@ -272,4 +273,13 @@ class SqlSelectHelperTest {
 
     }
 
+    @Test
+    void testGetTableName() {
+
+        String sql = "select 部门,sum (访问次数) from `超音数` where 数据日期 = '2023-08-08'"
+                + " and 用户 = 'alice' and 发布日期 ='11' group by 部门 limit 1";
+        String tableName = SqlSelectHelper.getTableName(sql);
+        Assert.assertEquals(tableName, "超音数");
+    }
+
 }

evaluation/build_models.py

@@ -32,7 +32,7 @@ def get_list(url):
         return None
 def build_domain():
     dict_info={}
-    json_data='{"name":"DuSQL_互联网企业","bizName":"internet","sensitiveLevel":0,"parentId":0,"isOpen":0,"viewers":["admin","tom","jack"],"viewOrgs":["1"],"admins":["admin"],"adminOrgs":[],"admin":"admin","viewer":"admin,tom,jack","viewOrg":"1","adminOrg":""}'
+    json_data='{"name":"DuSQL_互联网企业","bizName":"internet","sensitiveLevel":0,"parentId":0,"isOpen":0,"viewers":["admin","tom","jack"],"viewOrgs":["1"],"admins":["admin"],"adminOrgs":[],"viewer":"admin,tom,jack","viewOrg":"1","adminOrg":"","admin":"admin"}'
     json_dict=json.loads(json_data)
     url=get_url_pre()+"/api/semantic/domain/getDomainList"
     domain_list=get_list(url)
@@ -60,7 +60,7 @@ def build_domain():
     dict_info["domain_id"]=domain_id
     return dict_info
 def build_model_1(domain_id):
-    json_data='{"name":"公司","bizName":"company","description":"公司","sensitiveLevel":0,"databaseId":1,"domainId":4,"modelDetail":{"queryType":"sql_query","sqlQuery":"SELECT imp_date,company_id,company_name,headquarter_address,company_established_time,founder,ceo,annual_turnover,employee_count FROM company","identifiers":[{"name":"公司id","type":"primary","bizName":"company_id","isCreateDimension":0,"fieldName":"company_id"}],"dimensions":[{"name":"","type":"time","dateFormat":"yyyy-MM-dd","typeParams":{"isPrimary":"false","timeGranularity":"none"},"isCreateDimension":0,"bizName":"imp_date","isTag":0,"fieldName":"imp_date"},{"name":"公司名称","type":"categorical","dateFormat":"yyyy-MM-dd","isCreateDimension":1,"bizName":"company_name","isTag":0,"fieldName":"company_name"},{"name":"总部地点","type":"categorical","dateFormat":"yyyy-MM-dd","isCreateDimension":1,"bizName":"headquarter_address","isTag":0,"fieldName":"headquarter_address"},{"name":"公司成立时间","type":"categorical","dateFormat":"yyyy-MM-dd","isCreateDimension":1,"bizName":"company_established_time","isTag":0,"fieldName":"company_established_time"},{"name":"创始人","type":"categorical","dateFormat":"yyyy-MM-dd","isCreateDimension":1,"bizName":"founder","isTag":0,"fieldName":"founder"},{"name":"首席执行官","type":"categorical","dateFormat":"yyyy-MM-dd","isCreateDimension":1,"bizName":"ceo","isTag":0,"fieldName":"ceo"}],"measures":[{"name":"年营业额","agg":"SUM","bizName":"annual_turnover","isCreateMetric":1},{"name":"员工数","agg":"SUM","bizName":"employee_count","isCreateMetric":1}],"fields":[{"fieldName":"company_id"},{"fieldName":"imp_date"},{"fieldName":"company_established_time"},{"fieldName":"founder"},{"fieldName":"headquarter_address"},{"fieldName":"ceo"},{"fieldName":"company_name"}]},"viewers":["admin","tom","jack"],"viewOrgs":["1"],"admins":["admin"],"adminOrgs":[],"admin":"admin","viewer":"admin,tom,jack","viewOrg":"1","timeDimension":[{"name":"","type":"time","dateFormat":"yyyy-MM-dd","typeParams":{"isPrimary":"false","timeGranularity":"none"},"isCreateDimension":0,"bizName":"imp_date","isTag":0,"fieldName":"imp_date"}],"adminOrg":""}'
+    json_data='{"name":"公司","bizName":"company","description":"公司","sensitiveLevel":0,"databaseId":1,"domainId":4,"modelDetail":{"queryType":"sql_query","sqlQuery":"SELECT imp_date,company_id,company_name,headquarter_address,company_established_time,founder,ceo,annual_turnover,employee_count FROM company","identifiers":[{"name":"公司id","type":"primary","bizName":"company_id","isCreateDimension":0,"fieldName":"company_id"}],"dimensions":[{"name":"","type":"time","dateFormat":"yyyy-MM-dd","typeParams":{"isPrimary":"false","timeGranularity":"none"},"isCreateDimension":0,"bizName":"imp_date","isTag":0,"fieldName":"imp_date"},{"name":"公司名称","type":"categorical","dateFormat":"yyyy-MM-dd","isCreateDimension":1,"bizName":"company_name","isTag":0,"fieldName":"company_name"},{"name":"总部地点","type":"categorical","dateFormat":"yyyy-MM-dd","isCreateDimension":1,"bizName":"headquarter_address","isTag":0,"fieldName":"headquarter_address"},{"name":"公司成立时间","type":"categorical","dateFormat":"yyyy-MM-dd","isCreateDimension":1,"bizName":"company_established_time","isTag":0,"fieldName":"company_established_time"},{"name":"创始人","type":"categorical","dateFormat":"yyyy-MM-dd","isCreateDimension":1,"bizName":"founder","isTag":0,"fieldName":"founder"},{"name":"首席执行官","type":"categorical","dateFormat":"yyyy-MM-dd","isCreateDimension":1,"bizName":"ceo","isTag":0,"fieldName":"ceo"}],"measures":[{"name":"年营业额","agg":"SUM","bizName":"annual_turnover","isCreateMetric":1},{"name":"员工数","agg":"SUM","bizName":"employee_count","isCreateMetric":1}],"fields":[{"fieldName":"company_id"},{"fieldName":"imp_date"},{"fieldName":"company_established_time"},{"fieldName":"founder"},{"fieldName":"headquarter_address"},{"fieldName":"ceo"},{"fieldName":"company_name"}],"sqlVariables":[]},"viewers":["admin","tom","jack"],"viewOrgs":["1"],"admins":["admin"],"adminOrgs":[],"viewer":"admin,tom,jack","viewOrg":"1","timeDimension":[{"name":"","type":"time","dateFormat":"yyyy-MM-dd","typeParams":{"isPrimary":"false","timeGranularity":"none"},"isCreateDimension":0,"bizName":"imp_date","isTag":0,"fieldName":"imp_date"}],"adminOrg":"","admin":"admin"}'
     json_dict=json.loads(json_data)
     json_dict["domainId"]=domain_id
     url=get_url_pre()+"/api/semantic/model/getModelList/"+str(domain_id)
@@ -88,7 +88,7 @@ def build_model_1(domain_id):
     return model_id
 
 def build_model_2(domain_id):
-    json_data='{"name":"品牌","bizName":"brand","description":"品牌","sensitiveLevel":0,"databaseId":1,"domainId":4,"modelDetail":{"queryType":"sql_query","sqlQuery":"SELECT  imp_date,brand_id,brand_name,brand_established_time,company_id,legal_representative,registered_capital FROM brand","identifiers":[{"name":"品牌id","type":"primary","bizName":"brand_id","isCreateDimension":0,"fieldName":"brand_id"},{"name":"公司id","type":"foreign","bizName":"company_id","isCreateDimension":0,"fieldName":"company_id"}],"dimensions":[{"name":"","type":"time","dateFormat":"yyyy-MM-dd","typeParams":{"isPrimary":"false","timeGranularity":"none"},"isCreateDimension":0,"bizName":"imp_date","isTag":0,"fieldName":"imp_date"},{"name":"品牌名称","type":"categorical","dateFormat":"yyyy-MM-dd","isCreateDimension":1,"bizName":"brand_name","isTag":0,"fieldName":"brand_name"},{"name":"品牌成立时间","type":"categorical","dateFormat":"yyyy-MM-dd","isCreateDimension":1,"bizName":"brand_established_time","isTag":0,"fieldName":"brand_established_time"},{"name":"法定代表人","type":"categorical","dateFormat":"yyyy-MM-dd","isCreateDimension":1,"bizName":"legal_representative","isTag":0,"fieldName":"legal_representative"}],"measures":[{"name":"注册资本","agg":"SUM","bizName":"registered_capital","isCreateMetric":1}],"fields":[{"fieldName":"company_id"},{"fieldName":"brand_id"},{"fieldName":"brand_name"},{"fieldName":"imp_date"},{"fieldName":"brand_established_time"},{"fieldName":"legal_representative"}]},"viewers":["admin","tom","jack"],"viewOrgs":["1"],"admins":["admin"],"adminOrgs":[],"admin":"admin","viewer":"admin,tom,jack","viewOrg":"1","timeDimension":[{"name":"","type":"time","dateFormat":"yyyy-MM-dd","typeParams":{"isPrimary":"false","timeGranularity":"none"},"isCreateDimension":0,"bizName":"imp_date","isTag":0,"fieldName":"imp_date"}],"adminOrg":""}'
+    json_data='{"name":"品牌","bizName":"brand","description":"品牌","sensitiveLevel":0,"databaseId":1,"domainId":4,"modelDetail":{"queryType":"sql_query","sqlQuery":"SELECT  imp_date,brand_id,brand_name,brand_established_time,company_id,legal_representative,registered_capital FROM brand","identifiers":[{"name":"品牌id","type":"primary","bizName":"brand_id","isCreateDimension":0,"fieldName":"brand_id"},{"name":"公司id","type":"foreign","bizName":"company_id","isCreateDimension":0,"fieldName":"company_id"}],"dimensions":[{"name":"","type":"time","dateFormat":"yyyy-MM-dd","typeParams":{"isPrimary":"false","timeGranularity":"none"},"isCreateDimension":0,"bizName":"imp_date","isTag":0,"fieldName":"imp_date"},{"name":"品牌名称","type":"categorical","dateFormat":"yyyy-MM-dd","isCreateDimension":1,"bizName":"brand_name","isTag":0,"fieldName":"brand_name"},{"name":"品牌成立时间","type":"categorical","dateFormat":"yyyy-MM-dd","isCreateDimension":1,"bizName":"brand_established_time","isTag":0,"fieldName":"brand_established_time"},{"name":"法定代表人","type":"categorical","dateFormat":"yyyy-MM-dd","isCreateDimension":1,"bizName":"legal_representative","isTag":0,"fieldName":"legal_representative"}],"measures":[{"name":"注册资本","agg":"SUM","bizName":"registered_capital","isCreateMetric":1}],"fields":[{"fieldName":"company_id"},{"fieldName":"brand_id"},{"fieldName":"brand_name"},{"fieldName":"imp_date"},{"fieldName":"brand_established_time"},{"fieldName":"legal_representative"}],"sqlVariables":[]},"viewers":["admin","tom","jack"],"viewOrgs":["1"],"admins":["admin"],"adminOrgs":[],"viewer":"admin,tom,jack","viewOrg":"1","timeDimension":[{"name":"","type":"time","dateFormat":"yyyy-MM-dd","typeParams":{"isPrimary":"false","timeGranularity":"none"},"isCreateDimension":0,"bizName":"imp_date","isTag":0,"fieldName":"imp_date"}],"adminOrg":"","admin":"admin"}'
     json_dict=json.loads(json_data)
     json_dict["domainId"]=domain_id
     url=get_url_pre()+"/api/semantic/model/getModelList/"+str(domain_id)
@@ -116,7 +116,7 @@ def build_model_2(domain_id):
     return model_id
 
 def build_model_3(domain_id):
-    json_data='{"name":"公司各品牌收入排名","bizName":"company_revenue","description":"公司各品牌收入排名","sensitiveLevel":0,"databaseId":1,"domainId":4,"modelDetail":{"queryType":"sql_query","sqlQuery":"SELECT imp_date,company_id,brand_id,revenue_proportion,profit_proportion,expenditure_proportion FROM company_revenue","identifiers":[{"name":"公司id","type":"foreign","bizName":"company_id","isCreateDimension":0,"fieldName":"company_id"},{"name":"品牌id","type":"foreign","bizName":"brand_id","isCreateDimension":0,"fieldName":"brand_id"}],"dimensions":[{"name":"","type":"time","dateFormat":"yyyy-MM-dd","typeParams":{"isPrimary":"false","timeGranularity":"none"},"isCreateDimension":0,"bizName":"imp_date","isTag":0,"fieldName":"imp_date"}],"measures":[{"name":"营收占比","agg":"SUM","bizName":"revenue_proportion","isCreateMetric":1},{"name":"利润占比","agg":"SUM","bizName":"profit_proportion","isCreateMetric":1},{"name":"支出占比","agg":"SUM","bizName":"expenditure_proportion","isCreateMetric":1}],"fields":[{"fieldName":"company_id"},{"fieldName":"brand_id"},{"fieldName":"imp_date"}]},"viewers":["admin","tom","jack"],"viewOrgs":["1"],"admins":["admin"],"adminOrgs":[],"admin":"admin","viewer":"admin,tom,jack","viewOrg":"1","timeDimension":[{"name":"","type":"time","dateFormat":"yyyy-MM-dd","typeParams":{"isPrimary":"false","timeGranularity":"none"},"isCreateDimension":0,"bizName":"imp_date","isTag":0,"fieldName":"imp_date"}],"adminOrg":""}'
+    json_data='{"createdBy":"admin","updatedBy":"admin","createdAt":1713260936677,"updatedAt":1713260936677,"name":"公司各品牌收入排名","bizName":"company_revenue","description":"公司各品牌收入排名","sensitiveLevel":0,"databaseId":1,"domainId":4,"modelDetail":{"queryType":"sql_query","sqlQuery":"SELECT imp_date,company_id,brand_id,revenue_proportion,profit_proportion,expenditure_proportion FROM company_revenue","identifiers":[{"name":"公司id","type":"foreign","bizName":"company_id","isCreateDimension":0,"fieldName":"company_id"},{"name":"品牌id","type":"foreign","bizName":"brand_id","isCreateDimension":0,"fieldName":"brand_id"}],"dimensions":[{"name":"","type":"time","dateFormat":"yyyy-MM-dd","typeParams":{"isPrimary":"false","timeGranularity":"none"},"isCreateDimension":0,"bizName":"imp_date","isTag":0,"fieldName":"imp_date"}],"measures":[{"name":"营收占比","agg":"SUM","bizName":"revenue_proportion","isCreateMetric":1,"fieldName":"revenue_proportion"},{"name":"利润占比","agg":"SUM","bizName":"profit_proportion","isCreateMetric":1,"fieldName":"profit_proportion"},{"name":"支出占比","agg":"SUM","bizName":"expenditure_proportion","isCreateMetric":1,"fieldName":"expenditure_proportion"}],"fields":[{"fieldName":"company_id"},{"fieldName":"brand_id"},{"fieldName":"imp_date"},{"fieldName":"expenditure_proportion"},{"fieldName":"revenue_proportion"},{"fieldName":"profit_proportion"}],"sqlVariables":[]},"viewers":["admin","tom","jack"],"viewOrgs":["1"],"admins":["admin"],"adminOrgs":[],"viewer":"admin,tom,jack","viewOrg":"1","timeDimension":[{"name":"","type":"time","dateFormat":"yyyy-MM-dd","typeParams":{"isPrimary":"false","timeGranularity":"none"},"isCreateDimension":0,"bizName":"imp_date","isTag":0,"fieldName":"imp_date"}],"adminOrg":"","admin":"admin"}'
     json_dict=json.loads(json_data)
     json_dict["domainId"]=domain_id
     url=get_url_pre()+"/api/semantic/model/getModelList/"+str(domain_id)
@@ -144,7 +144,7 @@ def build_model_3(domain_id):
     return model_id
 
 def build_model_4(domain_id):
-    json_data='{"name":"公司品牌历年收入","bizName":"company_brand_revenue","description":"公司品牌历年收入","sensitiveLevel":0,"databaseId":1,"domainId":4,"modelDetail":{"queryType":"sql_query","sqlQuery":"SELECT imp_date,year_time,brand_id,revenue,profit,revenue_growth_year_on_year,profit_growth_year_on_year FROM company_brand_revenue","identifiers":[{"name":"品牌id","type":"foreign","bizName":"brand_id","isCreateDimension":0,"fieldName":"brand_id"}],"dimensions":[{"name":"","type":"time","dateFormat":"yyyy-MM-dd","typeParams":{"isPrimary":"false","timeGranularity":"none"},"isCreateDimension":0,"bizName":"imp_date","isTag":0,"fieldName":"imp_date"},{"name":"年份","type":"categorical","dateFormat":"yyyy-MM-dd","isCreateDimension":1,"bizName":"year_time","isTag":0,"fieldName":"year_time"}],"measures":[{"name":"营收","agg":"SUM","bizName":"revenue","isCreateMetric":1},{"name":"利润","agg":"SUM","bizName":"profit","isCreateMetric":1},{"name":"营收同比增长","agg":"SUM","bizName":"revenue_growth_year_on_year","isCreateMetric":1},{"name":"利润同比增长","agg":"SUM","bizName":"profit_growth_year_on_year","isCreateMetric":1}],"fields":[{"fieldName":"brand_id"},{"fieldName":"imp_date"},{"fieldName":"year_time"}]},"viewers":["admin","tom","jack"],"viewOrgs":["1"],"admins":["admin"],"adminOrgs":[],"admin":"admin","viewer":"admin,tom,jack","viewOrg":"1","timeDimension":[{"name":"","type":"time","dateFormat":"yyyy-MM-dd","typeParams":{"isPrimary":"false","timeGranularity":"none"},"isCreateDimension":0,"bizName":"imp_date","isTag":0,"fieldName":"imp_date"}],"adminOrg":""}'
+    json_data='{"name":"公司品牌历年收入","bizName":"company_brand_revenue","description":"公司品牌历年收入","sensitiveLevel":0,"databaseId":1,"domainId":4,"modelDetail":{"queryType":"sql_query","sqlQuery":"SELECT imp_date,year_time,brand_id,revenue,profit,revenue_growth_year_on_year,profit_growth_year_on_year FROM company_brand_revenue","identifiers":[{"name":"品牌id","type":"foreign","bizName":"brand_id","isCreateDimension":0,"fieldName":"brand_id"}],"dimensions":[{"name":"","type":"time","dateFormat":"yyyy-MM-dd","typeParams":{"isPrimary":"false","timeGranularity":"none"},"isCreateDimension":0,"bizName":"imp_date","isTag":0,"fieldName":"imp_date"},{"name":"年份","type":"categorical","dateFormat":"yyyy-MM-dd","isCreateDimension":1,"bizName":"year_time","isTag":0,"fieldName":"year_time"}],"measures":[{"name":"营收","agg":"SUM","bizName":"revenue","isCreateMetric":1},{"name":"利润","agg":"SUM","bizName":"profit","isCreateMetric":1},{"name":"营收同比增长","agg":"SUM","bizName":"revenue_growth_year_on_year","isCreateMetric":1},{"name":"利润同比增长","agg":"SUM","bizName":"profit_growth_year_on_year","isCreateMetric":1}],"fields":[{"fieldName":"brand_id"},{"fieldName":"imp_date"},{"fieldName":"year_time"}],"sqlVariables":[]},"viewers":["admin","tom","jack"],"viewOrgs":["1"],"admins":["admin"],"adminOrgs":[],"viewer":"admin,tom,jack","viewOrg":"1","timeDimension":[{"name":"","type":"time","dateFormat":"yyyy-MM-dd","typeParams":{"isPrimary":"false","timeGranularity":"none"},"isCreateDimension":0,"bizName":"imp_date","isTag":0,"fieldName":"imp_date"}],"adminOrg":"","admin":"admin"}'
     json_dict=json.loads(json_data)
     json_dict["domainId"]=domain_id
     url=get_url_pre()+"/api/semantic/model/getModelList/"+str(domain_id)
@@ -224,7 +224,7 @@ def get_id_list(data_list):
         for data in data_list:
             id_list.append(data["id"])
     return id_list
-def build_view(domain_id,model_id1,model_id2,model_id3,model_id4):
+def build_dataSet(domain_id,model_id1,model_id2,model_id3,model_id4):
     url=get_url_pre()+"/api/semantic/dimension/getDimensionList/"+str(model_id1)
     dimension_list1=get_id_list(get_list(url))
     url=get_url_pre()+"/api/semantic/dimension/getDimensionList/"+str(model_id2)
@@ -251,13 +251,21 @@ def build_view(domain_id,model_id1,model_id2,model_id3,model_id4):
     {"id":model_id4,"includesAll":False,"metrics":metric_list4,"dimensions":dimension_list4}]},"queryConfig":{"tagTypeDefaultConfig":
     {"dimensionIds":[],"metricIds":[]},"metricTypeDefaultConfig":{"timeDefaultConfig":{"unit":1,"period":"DAY","timeMode":"RECENT"}}},"admins":["admin"],"admin":"admin"}
 
-    url=get_url_pre()+"/api/semantic/view"
+    json_dict={"name":"DuSQL 互联网企业","bizName":"internet","description":"DuSQL互联网企业数据源相关的指标和维度等","typeEnum":"DATASET","sensitiveLevel":0,"domainId":domain_id,
+               "dataSetDetail":{"dataSetModelConfigs":[
+                   {"id":model_id1,"includesAll":False,"metrics":metric_list1,"dimensions":dimension_list1},
+                   {"id":model_id2,"includesAll":False,"metrics":metric_list2,"dimensions":dimension_list2},
+                   {"id":model_id3,"includesAll":False,"metrics":metric_list3,"dimensions":dimension_list3},
+                   {"id":model_id4,"includesAll":False,"metrics":metric_list4,"dimensions":dimension_list4}
+               ]},
+               "queryConfig":{"tagTypeDefaultConfig":{},"metricTypeDefaultConfig":{"timeDefaultConfig":{"unit":0,"period":"DAY","timeMode":"RECENT"}}},"admins":["admin"],"admin":"admin"}
+    url=get_url_pre()+"/api/semantic/dataSet"
     authorization=get_authorization()
     header = {}
     header["Authorization"] =authorization
     resp=requests.post(url=url, headers=header,json=json_dict)
 
-    url=get_url_pre()+"/api/semantic/view/getViewList?domainId="+str(domain_id)
+    url=get_url_pre()+"/api/semantic/dataSet/getDataSetList?domainId="+str(domain_id)
     print(url)
     resp=get_list(url)
     data={}
@@ -271,22 +279,36 @@ def build_view(domain_id,model_id1,model_id2,model_id3,model_id4):
     return  data
 
 
-def build_agent(view_id):
-    json_dict={
-        "id":10,
-        "enableSearch":1,
-        "name":"DuSQL 互联网企业",
-        "description":"DuSQL",
-        "status":1,
-        "examples":[],
-        "agentConfig":json.dumps({
-            "tools":[{
-                "id":1,
-                "type":"NL2SQL_LLM",
-                "viewIds":[view_id]
-            }]
-        })
-    }
+def build_agent(dataSetId):
+    # json_dict={
+    #     "id":10,
+    #     "enableSearch":1,
+    #     "name":"DuSQL 互联网企业",
+    #     "description":"DuSQL",
+    #     "status":1,
+    #     "examples":[],
+    #     "agentConfig":json.dumps({
+    #         "tools":[{
+    #             "id":1,
+    #             "type":"NL2SQL_LLM",
+    #             "viewIds":[view_id]
+    #         }]
+    #     })
+    # }
+    json_dict={"id":10,
+               "enableSearch":1,
+               "name":"DuSQL 互联网企业",
+               "description":"DuSQL",
+               "status":1,
+               "examples":[],
+               "agentConfig":json.dumps({
+                   "tools":[{
+                       "id":1,
+                       "type":"NL2SQL_LLM",
+                       "dataSetIds":[dataSetId]
+                   }]
+               }),
+               "dataSetIds":[dataSetId]}
     url=get_url_pre()+"/api/chat/agent"
     authorization=get_authorization()
     header = {}
@@ -324,12 +346,12 @@ def build():
         model_id2=build_model_2(domain_id)
         model_id3=build_model_3(domain_id)
         model_id4=build_model_4(domain_id)
-        view_id=build_view(domain_id,model_id1,model_id2,model_id3,model_id4)
+        dataSet_id=build_dataSet(domain_id,model_id1,model_id2,model_id3,model_id4)
         build_model_rela1(domain_id,model_id1,model_id2)
         build_model_rela2(domain_id,model_id1,model_id3)
         build_model_rela3(domain_id,model_id2,model_id3)
         build_model_rela4(domain_id,model_id2,model_id4)
-        build_agent(view_id["id"])
+        build_agent(dataSet_id["id"])
         agentId=10
         chat_id=build_chat(agentId)
         dict={}