(feature)(headless) Add tag query api (#790)

(improvement)(project) merge master to dev-0.9

.github/workflows/ci.yml

@@ -0,0 +1,31 @@
+name: supersonic CI
+
+on:
+  push:
+    branches:
+      - master
+  pull_request:
+    branches:
+      - master
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v2
+    - name: Set up JDK 8
+      uses: actions/setup-java@v2
+      with:
+        java-version: '8'
+        distribution: 'adopt'
+    - name: Cache Maven packages
+      uses: actions/cache@v2
+      with:
+        path: ~/.m2
+        key: ${{ runner.os }}-m2-${{ hashFiles('**/pom.xml') }}
+        restore-keys: ${{ runner.os }}-m2
+    - name: Build with Maven
+      run: mvn -B package --file pom.xml
+    - name: Test with Maven
+      run: mvn test

CHANGELOG.md

@@ -4,6 +4,31 @@
 - "Breaking Changes" describes any changes that may break existing functionality or cause
   compatibility issues with previous versions.
 
+## SuperSonic [0.8.6] - 2024-02-23
+
+### Added
+- support view abstraction to Headless.
+- add the Metric API to Headless and optimizing the Headless API.
+- add integration tests to Headless.
+- add TimeCorrector to Chat.
+
+## SuperSonic [0.8.4] - 2024-01-19
+
+### Added
+- support creating derived metrics.
+  - Support creating metrics using three methods: by measure, metric, and field expressions.
+- added support for postgresql data source.
+- code adjustment and abstract optimization for chat and headless.
+
+## SuperSonic [0.8.2] - 2023-12-18
+
+### Added
+- rewrite Python service with Java project, default to Java implementation.
+- support setting the SQL generation method for large models in the interface.
+- optimization of metric market experience.
+- optimization of semantic modeling canvas experience.
+- code structure adjustment and abstraction optimization for chat.
+
 ## SuperSonic [0.7.5] - 2023-10-13
 
 ### Added

README.md

@@ -1,26 +1,28 @@
 [中文介绍](README_CN.md) | [文档中心](https://github.com/tencentmusic/supersonic/wiki)
 
+![Java CI](https://github.com/tencentmusic/supersonic/workflows/supersonic%20CI/badge.svg)
+
 # SuperSonic (超音数)
 
-**SuperSonic is an out-of-the-box yet highly extensible framework for building ChatBI**. SuperSonic provides a chat interface that empowers users to query data using natural language and visualize the results with suitable charts. To enable such experience, the only thing necessary is to build logical semantic models (definition of metrics/dimensions/entities, along with their meaning, context and relationships) on top of physical data models, and no data modification or copying is required. Meanwhile, SuperSonic is designed to be pluggable, allowing new functionalities to be added through plugins and core components to be integrated with other systems.
+**SuperSonic is the next-generation LLM-powered data analytics platform that integrates ChatBI and HeadlessBI**. SuperSonic provides a chat interface that empowers users to query data using natural language and visualize the results with suitable charts. To enable such experience, the only thing necessary is to build logical semantic models (definition of entities/metrics/dimensions/tags, along with their meaning, context and relationships) on top of physical data models, and **no data modification or copying** is required. Meanwhile, SuperSonic is designed to be **highly extensible**, allowing custom functionalities to be added and configured with Java SPI.
 
 <img src="./docs/images/supersonic_demo.gif" height="100%" width="100%" align="center"/>
 
 ## Motivation
 
-The emergence of Large Language Model (LLM) like ChatGPT is reshaping the way information is retrieved. In the field of data analytics, both academia and industry are primarily focused on leveraging LLM to convert natural language into SQL (so called text2sql or nl2sql). While some works exhibit promising results, their **reliability** is inadequate for real-world applications.  
+The emergence of Large Language Model (LLM) like ChatGPT is reshaping the way information is retrieved. In the field of data analytics, both academia and industry are primarily focused on leveraging LLM to convert natural language into SQL (so called Text2SQL or NL2SQL). While some approaches exhibit promising results, their **reliability** and **efficiency** are insufficient for real-world applications.  
 
 From our perspective, the key to filling the real-world gap lies in three aspects: 
-1. Introduce a semantic layer encapsulating underlying data context(joins, formulas, etc) to reduce **complexity**.
-2. Augment the LLM with schema mappers(as a kind of preprocessor) and semantic correctors(as a kind of postprocessor) to mitigate **hallucination**.
-3. Utilize heuristic rules when necessary to improve **efficiency**(in terms of latency and cost).
+1. Integrate ChatBI with HeadlessBI encapsulating underlying data context (joins, keys, formulas, etc) to **reduce complexity**.
+2. Augment the LLM with schema mappers(as a kind of preprocessor) and semantic correctors(as a kind of postprocessor) to **mitigate hallucination**.
+3. Utilize rule-based schema parsers when necessary to **improve efficiency**(in terms of latency and cost).
 
 With these ideas in mind, we develop SuperSonic as a practical reference implementation and use it to power our real-world products. Additionally, to facilitate further development of ChatBI, we decide to open source SuperSonic as an extensible framework.
 
 ## Out-of-the-box Features
 
-- Built-in CUI(Chat User Interface) for *business users* to enter data queries
-- Built-in GUI(Graphical User Interface) for *analytics engineers* to build semantic models
+- Built-in ChatBI interface for *business users* to enter natural language queries
+- Built-in HeadlessBI interface for *analytics engineers* to build semantic models
 - Built-in GUI for *system administrators* to manage chat agents and third-party plugins
 - Support input auto-completion as well as query recommendation
 - Support multi-turn conversation and history context management 
@@ -49,7 +51,7 @@ The high-level architecture and main process flow is as follows:
 SuperSonic comes with sample semantic models as well as chat conversations that can be used as a starting point. Please follow the steps: 
 
 - Download the latest prebuilt binary from the [release page](https://github.com/tencentmusic/supersonic/releases)
-- Run script "bin/supersonic-daemon.sh" to start services (one java process and one python process)
+- Run script "assembly/bin/supersonic-daemon.sh start" to start a standalone Java service
 - Visit http://localhost:9080 in the browser to start exploration
 
 ## Build and Development

README_CN.md

@@ -1,6 +1,6 @@
-# 超音数（SuperSonic）
+# SuperSonic (超音数)
 
-**超音数是一个开箱即用且易于扩展的数据问答对话框架**。通过超音数的问答对话界面，用户能够使用自然语言查询数据，系统会选择合适的可视化图表呈现结果。超音数不需要修改或复制数据，只需要在物理数据模型之上构建逻辑语义模型（指标/维度/实体的定义，以及他们的业务含义、相互间关系等），即可开启数据问答体验。与此同时，超音数被设计为可插拔式的框架，允许以插件形式来扩展新功能，或者将核心组件与其他系统集成。
+**SuperSonic融合ChatBI和HeadlessBI打造新一代的数据分析平台**。通过SuperSonic的问答对话界面，用户能够使用自然语言查询数据，系统会选择合适的可视化图表呈现结果。SuperSonic不需要修改或复制数据，只需要在物理数据模型之上构建逻辑语义模型（指标/维度/实体的定义，以及他们的业务含义、相互间关系等），即可开启数据问答体验。与此同时，SuperSonic被设计为可插拔的框架，采用Java SPI机制来扩展定制功能。
 
 <img src="./docs/images/supersonic_demo.gif" height="100%" width="100%" align="center"/>
 
@@ -9,24 +9,24 @@
 大型语言模型（LLMs）如ChatGPT的出现正在重塑信息检索的方式。在数据分析领域，学术界和工业界主要关注利用深度学习模型将自然语言查询转换为SQL查询。虽然一些工作显示出有前景的结果，但它们的可靠性还达不到生产可用的要求。
 
 在我们看来，为了在实际场景发挥价值，有三个关键点：
-1. 引入语义模型层，封装底层数据的上下文（关联、公式等），降低SQL生成的**复杂度**。
+1. 融合HeadlessBI，通过统一语义层封装底层数据细节（关联、键值、公式等），降低SQL生成的**复杂度**。
 2. 通过一前一后的模式映射器和语义修正器，来缓解LLM常见的**幻觉**现象。
 3. 设计启发式的规则，在一些特定场景提升语义解析的**效率**。
 
-为了验证上述想法，我们开发了超音数项目，并将其应用在实际的内部产品中。与此同时，我们将超音数作为一个可扩展的框架开源，希望能够促进数据问答对话领域的进一步发展。
+为了验证上述想法，我们开发了SuperSonic项目，并将其应用在实际的内部产品中。与此同时，我们将SuperSonic作为一个可扩展的框架开源，希望能够促进数据问答对话领域的进一步发展。
 
 ## 开箱即用的特性
 
-- 内置对话界面以便*业务用户*输入数据查询。
-- 内置图形界面以便*分析工程师*构建语义模型。
-- 内置图形界面以便*系统管理员*管理第三方插件和对话助理。
+- 内置ChatBI界面以便*业务用户*输入数据查询。
+- 内置HeadlessBI界面以便*分析工程师*构建语义模型。
+- 内置图形用户界面以便*系统管理员*管理第三方插件和对话助理。
 - 支持文本输入的联想和查询问题的推荐。
 - 支持多轮对话，根据语境自动切换上下文。
 - 支持四级权限控制：主题域级、模型级、列级、行级。
 
 ## 易于扩展的组件
 
-超音数的整体架构和主流程如下图所示：
+SuperSonic的整体架构和主流程如下图所示：
 
 <img src="./docs/images/supersonic_components.png" height="65%" width="65%" align="center"/> 
 
@@ -44,10 +44,10 @@
 
 ## 快速体验
 
-超音数自带样例的语义模型和问答对话，只需以下三步即可快速体验：
+SuperSonic自带样例的语义模型和问答对话，只需以下三步即可快速体验：
 
 - 从[release page](https://github.com/tencentmusic/supersonic/releases)下载预先构建好的发行包
-- 运行 "bin/supersonic-daemon.sh"启动服务（一个Java进程和一个Python进程）
+- 运行 "assembly/bin/supersonic-daemon.sh start"启动standalone模式的Java服务
 - 在浏览器访问http://localhost:9080 开启探索
 
 ## 如何构建和部署

assembly/bin/supersonic-build.bat

@@ -6,11 +6,18 @@ set "baseDir=%~dp0.."
 set "buildDir=%baseDir%\build"
 set "runtimeDir=%baseDir%\..\runtime"
 set "pip_path=pip3"
+set "service=%~1"
+
 
 rem 1. build backend java modules
 del /q "%buildDir%\*.tar.gz" 2>NUL
 call mvn -f "%baseDir%\..\pom.xml" clean package -DskipTests
 
+IF ERRORLEVEL 1 (
+    ECHO Failed to build backend Java modules.
+    EXIT /B 1
+)
+
 rem 2. move package to build
 echo f|xcopy "%baseDir%\..\launchers\standalone\target\*.tar.gz" "%buildDir%\supersonic-standalone.tar.gz"
 
@@ -19,6 +26,11 @@ cd "%baseDir%\..\webapp"
 call start-fe-prod.bat
 copy /y "%baseDir%\..\webapp\supersonic-webapp.tar.gz" "%buildDir%\"
 
+IF ERRORLEVEL 1 (
+    ECHO Failed to build frontend webapp.
+    EXIT /B 1
+)
+
 rem 4. copy webapp to java classpath
 cd "%buildDir%"
 tar -zxvf supersonic-webapp.tar.gz
@@ -26,16 +38,23 @@ move supersonic-webapp webapp
 move webapp ..\..\launchers\standalone\target\classes
 
 rem 5. build backend python modules
+if "%service%"=="pyllm" (
     echo "start installing python modules with pip: ${pip_path}"
     set requirementPath="%baseDir%/../chat/python/requirements.txt"
     %pip_path% install -r %requirementPath%
     echo "install python modules success"
+)
 
 call :BUILD_RUNTIME
 
 :BUILD_RUNTIME
   rem 6. reset runtime
+  IF EXIST "%runtimeDir%" (
+      echo begin to delete dir : %runtimeDir%
       rd /s /q "%runtimeDir%"
+  ) ELSE (
+      echo %runtimeDir% does not exist, create directly
+  )
   mkdir "%runtimeDir%"
   tar -zxvf "%buildDir%\supersonic-standalone.tar.gz" -C "%runtimeDir%"
   for /d %%f in ("%runtimeDir%\launchers-standalone-*") do (

assembly/bin/supersonic-build.sh

@@ -4,19 +4,22 @@ set -x
 sbinDir=$(cd "$(dirname "$0")"; pwd)
 chmod +x $sbinDir/supersonic-common.sh
 source $sbinDir/supersonic-common.sh
-
 cd $baseDir
 
+service=$1
 #1. build backend java modules
 rm -fr ${buildDir}/*.tar.gz
 rm -fr dist
-
 set +x
-
 mvn -f $baseDir/../ clean package -DskipTests
+# check build result
+if [ $? -ne 0 ]; then
+    echo "Failed to build backend Java modules."
+    exit 1
+fi
 
 #2. move package to build
-cp $baseDir/../launchers/semantic/target/*.tar.gz ${buildDir}/supersonic-semantic.tar.gz
+cp $baseDir/../launchers/headless/target/*.tar.gz ${buildDir}/supersonic-headless.tar.gz
 cp $baseDir/../launchers/chat/target/*.tar.gz ${buildDir}/supersonic-chat.tar.gz
 cp $baseDir/../launchers/standalone/target/*.tar.gz ${buildDir}/supersonic-standalone.tar.gz
 
@@ -26,23 +29,30 @@ cd ../webapp
 sh ./start-fe-prod.sh
 cp -fr  ./supersonic-webapp.tar.gz ${buildDir}/
 
+# check build result
+if [ $? -ne 0 ]; then
+    echo "Failed to build frontend webapp."
+    exit 1
+fi
 #4. copy webapp to java classpath
 cd $buildDir
 tar xvf supersonic-webapp.tar.gz
 mv supersonic-webapp webapp
-cp -fr webapp ../../launchers/semantic/target/classes
+cp -fr webapp ../../launchers/headless/target/classes
 cp -fr webapp ../../launchers/chat/target/classes
 cp -fr webapp ../../launchers/standalone/target/classes
 rm -fr  ${buildDir}/webapp
 
 #5. build backend python modules
+if [ "$service" == "pyllm" ]; then
   echo "start installing python modules with pip: ${pip_path}"
   requirementPath=$baseDir/../chat/python/requirements.txt
   ${pip_path} install -r ${requirementPath}
   echo "install python modules success"
+fi
 
 #6. reset runtime
-rm -fr $runtimeDir/*
+rm -fr $runtimeDir/supersonic*
 moveAllToRuntime
 setEnvToWeb chat
-setEnvToWeb semantic
+setEnvToWeb headless

assembly/bin/supersonic-common.sh

@@ -10,15 +10,15 @@ runtimeDir=$baseDir/../runtime
 buildDir=$baseDir/build
 
 readonly CHAT_APP_NAME="supersonic_chat"
-readonly SEMANTIC_APP_NAME="supersonic_semantic"
-readonly LLMPARSER_APP_NAME="supersonic_llmparser"
+readonly HEADLESS_APP_NAME="supersonic_headless"
+readonly PYLLM_APP_NAME="supersonic_pyllm"
 readonly STANDALONE_APP_NAME="supersonic_standalone"
 readonly CHAT_SERVICE="chat"
-readonly SEMANTIC_SERVICE="semantic"
-readonly LLMPARSER_SERVICE="llmparser"
+readonly HEADLESS_SERVICE="headless"
+readonly PYLLM_SERVICE="pyllm"
 readonly STANDALONE_SERVICE="standalone"
-readonly LLMPARSER_HOST="127.0.0.1"
-readonly LLMPARSER_PORT="9092"
+readonly PYLLM_HOST="127.0.0.1"
+readonly PYLLM_PORT="9092"
 
 function setEnvToWeb {
    model_name=$1
@@ -29,11 +29,15 @@ function setEnvToWeb {
 
 function moveToRuntime {
   model_name=$1
-  tar -zxvf ${buildDir}/supersonic-${model_name}.tar.gz  -C ${runtimeDir}
+  file="${buildDir}/supersonic-${model_name}.tar.gz"
+  if [ -f "$file" ]; then
+    tar -zxvf "$file" -C ${runtimeDir}
     mv ${runtimeDir}/launchers-${model_name}-* ${runtimeDir}/supersonic-${model_name}
-
     mkdir -p ${runtimeDir}/supersonic-${model_name}/webapp
     cp -fr ${buildDir}/webapp/* ${runtimeDir}/supersonic-${model_name}/webapp
+  else
+    echo "File $file does not exist. Skipping the move to runtime."
+  fi
 }
 
 function moveAllToRuntime {
@@ -42,7 +46,7 @@ function moveAllToRuntime {
   mv ${buildDir}/supersonic-webapp ${buildDir}/webapp
 
   moveToRuntime chat
-  moveToRuntime semantic
+  moveToRuntime headless
   moveToRuntime standalone
   rm -fr  ${buildDir}/webapp
 }
@@ -81,23 +85,23 @@ function runJavaService {
 
 # run python service
 function runPythonService {
-  pythonRunDir=${runtimeDir}/supersonic-${model_name}/llmparser
+  pythonRunDir=${runtimeDir}/supersonic-${model_name}/pyllm
   cd $pythonRunDir
-  nohup ${python_path} supersonic_llmparser.py  > $pythonRunDir/llmparser.log  2>&1   &
+  nohup ${python_path} supersonic_pyllm.py  > $pythonRunDir/pyllm.log  2>&1   &
   # add health check
   for i in {1..10}
   do
-    echo "llmparser health check attempt $i..."
-    response=$(curl -s http://${LLMPARSER_HOST}:${LLMPARSER_PORT}/health)
-    echo "llmparser health check response: $response"
+    echo "pyllm health check attempt $i..."
+    response=$(curl -s http://${PYLLM_HOST}:${PYLLM_PORT}/health)
+    echo "pyllm health check response: $response"
     status_ok="Healthy"
     if [[ $response == *$status_ok* ]] ; then
-      echo "llmparser Health check passed."
+      echo "pyllm Health check passed."
       break
     else
       if [ "$i" -eq 10 ]; then
-        echo "llmparser Health check failed after 10 attempts."
-        echo "May still downloading model files. Please check llmparser.log in runtime directory."
+        echo "pyllm Health check failed after 10 attempts."
+        echo "May still downloading model files. Please check pyllm.log in runtime directory."
       fi
       echo "Retrying after 5 seconds..."
       sleep 5

assembly/bin/supersonic-daemon.bat

@@ -9,10 +9,10 @@ set "main_class=com.tencent.supersonic.StandaloneLauncher"
 set "python_path=python"
 set "pip_path=pip3"
 set "standalone_service=standalone"
-set "llmparser_service=llmparser"
+set "pyllm_service=pyllm"
 
 set "javaRunDir=%runtimeDir%\supersonic-standalone"
-set "pythonRunDir=%runtimeDir%\supersonic-standalone\llmparser"
+set "pythonRunDir=%runtimeDir%\supersonic-standalone\pyllm"
 
 set "command=%~1"
 set "service=%~2"
@@ -21,6 +21,10 @@ if "%service%"=="" (
    set "service=%standalone_service%"
 )
 
+IF "%service%"=="pyllm" (
+  SET "llmProxy=PythonLLMProxy"
+)
+
 call :BUILD_RUNTIME
 
 if "%command%"=="restart" (
@@ -42,27 +46,23 @@ if "%command%"=="restart" (
 )
 
 :START
-    if "%service%"=="%llmparser_service%" (
+    if "%service%"=="%pyllm_service%" (
          call :START_PYTHON
+         call :START_JAVA
          goto :EOF
     )
-    call :START_PYTHON
     call :START_JAVA
     goto :EOF
 
 :STOP
-    if "%service%"=="%llmparser_service%" (
-        call :STOP_PYTHON
-        goto :EOF
-    )
     call :STOP_PYTHON
     call :STOP_JAVA
     goto :EOF
 
 :START_PYTHON
-   echo 'python service starting, see logs in llmparser/llmparser.log'
+   echo 'python service starting, see logs in pyllm/pyllm.log'
    cd "%pythonRunDir%"
-   start /B %python_path% supersonic_llmparser.py  > %pythonRunDir%\llmparser.log 2>&1
+   start /B %python_path% supersonic_pyllm.py  > %pythonRunDir%\pyllm.log 2>&1
    timeout /t 10 >nul
    echo 'python service started'
    goto :EOF
@@ -71,9 +71,9 @@ if "%command%"=="restart" (
   echo 'java service starting, see logs in logs/'
    cd "%javaRunDir%"
    if not exist "%runtimeDir%\supersonic-standalone\logs" mkdir "%runtimeDir%\supersonic-standalone\logs"
-   set "libDir=%runtimeDir%\supersonic-%service%\lib"
-   set "confDir=%runtimeDir%\supersonic-%service%\conf"
-   set "webDir=%runtimeDir%\supersonic-%service%\webapp"
+   set "libDir=%runtimeDir%\supersonic-standalone\lib"
+   set "confDir=%runtimeDir%\supersonic-standalone\conf"
+   set "webDir=%runtimeDir%\supersonic-standalone\webapp"
    set "classpath=%confDir%;%webDir%;%libDir%\*"
    set "java-command=-Dfile.encoding=UTF-8 -Duser.language=Zh -Duser.region=CN -Duser.timezone=GMT+08 -Xms1024m -Xmx2048m -cp %CLASSPATH% %MAIN_CLASS%"
    start /B java %java-command% >nul 2>&1
@@ -96,7 +96,7 @@ if "%command%"=="restart" (
    goto :EOF
 
 :RELOAD_EXAMPLE
-   cd "%runtimeDir%\supersonic-standalone\llmparser\sql"
+   cd "%runtimeDir%\supersonic-standalone\pyllm\sql"
    start  %python_path% examples_reload_run.py
    goto :EOF
 

assembly/bin/supersonic-daemon.sh

@@ -22,8 +22,9 @@ app_name=$STANDALONE_APP_NAME
 main_class="com.tencent.supersonic.StandaloneLauncher"
 model_name=$service
 
-if [ "$service" == "llmparser" ]; then
+if [ "$service" == "pyllm" ]; then
   model_name=${STANDALONE_SERVICE}
+  export llmProxy=PythonLLMProxy
 fi
 
 cd $baseDir
@@ -32,8 +33,8 @@ cd $baseDir
 function setMainClass {
   if [ "$service" == $CHAT_SERVICE ]; then
     main_class="com.tencent.supersonic.ChatLauncher"
-  elif [ "$service" == $SEMANTIC_SERVICE ]; then
-    main_class="com.tencent.supersonic.SemanticLauncher"
+  elif [ "$service" == $HEADLESS_SERVICE ]; then
+    main_class="com.tencent.supersonic.HeadlessLauncher"
   fi
 }
 setMainClass
@@ -41,16 +42,16 @@ setMainClass
 function setAppName {
   if [ "$service" == $CHAT_SERVICE ]; then
     app_name=$CHAT_APP_NAME
-  elif [ "$service" == $SEMANTIC_SERVICE ]; then
-    app_name=$SEMANTIC_APP_NAME
-  elif [ "$service" == $LLMPARSER_SERVICE ]; then
-    app_name=$LLMPARSER_APP_NAME
+  elif [ "$service" == $HEADLESS_SERVICE ]; then
+    app_name=$HEADLESS_APP_NAME
+  elif [ "$service" == $PYLLM_SERVICE ]; then
+    app_name=$PYLLM_APP_NAME
   fi
 }
 setAppName
 
 function reloadExamples {
-  pythonRunDir=${runtimeDir}/supersonic-${model_name}/llmparser
+  pythonRunDir=${runtimeDir}/supersonic-${model_name}/pyllm
   cd $pythonRunDir/sql
   ${python_path} examples_reload_run.py
 }
@@ -61,7 +62,7 @@ function start()
   local_app_name=$1
   pid=$(ps aux |grep ${local_app_name} | grep -v grep | awk '{print $2}')
   if [[ "$pid" == "" ]]; then
-    if [[ ${local_app_name} == $LLMPARSER_APP_NAME ]]; then
+    if [[ ${local_app_name} == $PYLLM_APP_NAME ]]; then
       runPythonService ${local_app_name}
     else
       runJavaService ${local_app_name}
@@ -87,7 +88,7 @@ function stop()
 
 function reload()
 {
-  if [[ $1 == $LLMPARSER_APP_NAME ]]; then
+  if [[ $1 == $PYLLM_APP_NAME ]]; then
     reloadExamples
   fi
 }
@@ -95,11 +96,11 @@ function reload()
 # 4. execute command operation
 case "$command" in
   start)
-        if [ "$service" == $STANDALONE_SERVICE ]; then
-          echo  "Starting $LLMPARSER_APP_NAME"
-          start $LLMPARSER_APP_NAME
+        if [ "$service" == $PYLLM_SERVICE ]; then
           echo  "Starting $app_name"
           start $app_name
+          echo  "Starting $STANDALONE_APP_NAME"
+          start $STANDALONE_APP_NAME
         else
           echo  "Starting $app_name"
           start $app_name
@@ -107,15 +108,10 @@ case "$command" in
         echo  "Start success"
         ;;
   stop)
-        if [ "$service" == $STANDALONE_SERVICE ]; then
-          echo  "Stopping $LLMPARSER_APP_NAME"
-          stop $LLMPARSER_APP_NAME
         echo  "Stopping $app_name"
         stop $app_name
-        else
-          echo  "Stopping $app_name"
-          stop ${app_name}
-        fi
+        echo  "Stopping $PYLLM_APP_NAME"
+        stop $PYLLM_APP_NAME
         echo  "Stop success"
         ;;
   reload)
@@ -124,15 +120,15 @@ case "$command" in
         echo  "Reload success"
         ;;
   restart)
-        if [ "$service" == $STANDALONE_SERVICE ]; then
+        if [ "$service" == $PYLLM_SERVICE ]; then
           echo  "Stopping ${app_name}"
           stop ${app_name}
-          echo  "Stopping ${LLMPARSER_APP_NAME}"
-          stop $LLMPARSER_APP_NAME
-          echo  "Starting ${LLMPARSER_APP_NAME}"
-          start $LLMPARSER_APP_NAME
+          echo  "Stopping ${STANDALONE_APP_NAME}"
+          stop $STANDALONE_APP_NAME
           echo  "Starting ${app_name}"
           start ${app_name}
+          echo  "Starting ${STANDALONE_APP_NAME}"
+          start $STANDALONE_APP_NAME
         else
           echo  "Stopping ${app_name}"
           stop ${app_name}

assembly/build/build.xml

@@ -22,7 +22,7 @@
         </fileSet>
         <fileSet>
             <directory>${project.basedir}/../../chat/python</directory>
-            <outputDirectory>llmparser</outputDirectory>
+            <outputDirectory>pyllm</outputDirectory>
             <fileMode>0777</fileMode>
             <directoryMode>0755</directoryMode>
         </fileSet>

auth/api/src/main/java/com/tencent/supersonic/auth/api/authentication/adaptor/UserAdaptor.java

@@ -7,6 +7,9 @@ import com.tencent.supersonic.auth.api.authentication.request.UserReq;
 import java.util.List;
 import java.util.Set;
 
+/**
+ * UserAdaptor defines some interfaces for obtaining user and organization information
+ */
 public interface UserAdaptor {
 
     List<String> getUserNames();

auth/api/src/main/java/com/tencent/supersonic/auth/api/authentication/config/AuthenticationConfig.java

@@ -24,4 +24,13 @@ public class AuthenticationConfig {
     @Value("${authentication.token.http.header.key:Authorization}")
     private String tokenHttpHeaderKey;
 
+    @Value("${authentication.app.appId:appId}")
+    private String appId;
+
+    @Value("${authentication.app.timestamp:timestamp}")
+    private String timestamp;
+
+    @Value("${authentication.app.signature:signature}")
+    private String signature;
+
 }

auth/api/src/main/java/com/tencent/supersonic/auth/api/authentication/pojo/Organization.java

@@ -1,11 +1,15 @@
 package com.tencent.supersonic.auth.api.authentication.pojo;
 
 import com.google.common.collect.Lists;
+import lombok.AllArgsConstructor;
 import lombok.Data;
+import lombok.NoArgsConstructor;
 
 import java.util.List;
 
 @Data
+@AllArgsConstructor
+@NoArgsConstructor
 public class Organization {
 
     private String id;

auth/api/src/main/java/com/tencent/supersonic/auth/api/authentication/pojo/User.java

@@ -24,10 +24,19 @@ public class User {
         return new User(id, name, displayName, email, isAdmin);
     }
 
+    public static User get(Long id, String name) {
+        return new User(id, name, name, name, 0);
+    }
+
     public static User getFakeUser() {
         return new User(1L, "admin", "admin", "admin@email", 1);
     }
 
+    public static User getAppUser(int appId) {
+        String name = String.format("app_%s", appId);
+        return new User(1L, name, name, "", 1);
+    }
+
     public String getDisplayName() {
         return StringUtils.isBlank(displayName) ? name : displayName;
     }

auth/api/src/main/java/com/tencent/supersonic/auth/api/authentication/service/UserService.java

@@ -4,11 +4,15 @@ import com.tencent.supersonic.auth.api.authentication.pojo.Organization;
 import com.tencent.supersonic.auth.api.authentication.pojo.User;
 import com.tencent.supersonic.auth.api.authentication.request.UserReq;
 
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
 import java.util.List;
 import java.util.Set;
 
 public interface UserService {
 
+    User getCurrentUser(HttpServletRequest httpServletRequest, HttpServletResponse httpServletResponse);
+
     List<String> getUserNames();
 
     List<User> getUserList();

auth/api/src/main/java/com/tencent/supersonic/auth/api/authentication/utils/UserHolder.java

@@ -2,6 +2,11 @@ package com.tencent.supersonic.auth.api.authentication.utils;
 
 import com.tencent.supersonic.auth.api.authentication.pojo.User;
 import com.tencent.supersonic.auth.api.authentication.service.UserStrategy;
+import com.tencent.supersonic.common.pojo.SysParameter;
+import com.tencent.supersonic.common.service.SysParameterService;
+import com.tencent.supersonic.common.util.ContextUtils;
+import org.springframework.util.CollectionUtils;
+
 import javax.servlet.http.HttpServletRequest;
 import javax.servlet.http.HttpServletResponse;
 
@@ -14,7 +19,14 @@ public final class UserHolder {
     }
 
     public static User findUser(HttpServletRequest request, HttpServletResponse response) {
-        return REPO.findUser(request, response);
+        User user = REPO.findUser(request, response);
+        SysParameterService sysParameterService = ContextUtils.getBean(SysParameterService.class);
+        SysParameter sysParameter = sysParameterService.getSysParameter();
+        if (!CollectionUtils.isEmpty(sysParameter.getAdmins())
+                && sysParameter.getAdmins().contains(user.getName())) {
+            user.setIsAdmin(1);
+        }
+        return user;
     }
 
 }

auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/adaptor/DefaultUserAdaptor.java

@@ -16,6 +16,9 @@ import java.util.List;
 import java.util.Set;
 import java.util.stream.Collectors;
 
+/**
+ * DefaultUserAdaptor provides a default method to obtain user and organization information
+ */
 public class DefaultUserAdaptor implements UserAdaptor {
 
     private List<UserDO> getUserDOList() {
@@ -41,7 +44,17 @@ public class DefaultUserAdaptor implements UserAdaptor {
 
     @Override
     public List<Organization> getOrganizationTree() {
-        return Lists.newArrayList();
+        Organization superSonic = new Organization("1", "0",
+                "SuperSonic", "SuperSonic", Lists.newArrayList(), true);
+        Organization hr = new Organization("2", "1",
+                "Hr", "SuperSonic/Hr", Lists.newArrayList(), false);
+        Organization sales = new Organization("3", "1",
+                "Sales", "SuperSonic/Sales", Lists.newArrayList(), false);
+        Organization marketing = new Organization("4", "1",
+                "Marketing", "SuperSonic/Marketing", Lists.newArrayList(), false);
+        List<Organization> subOrganization = Lists.newArrayList(hr, sales, marketing);
+        superSonic.setSubOrganizations(subOrganization);
+        return Lists.newArrayList(superSonic);
     }
 
     private User convert(UserDO userDO) {

auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/interceptor/AuthenticationInterceptor.java

@@ -5,16 +5,17 @@ import com.tencent.supersonic.auth.api.authentication.constant.UserConstants;
 import com.tencent.supersonic.auth.authentication.service.UserServiceImpl;
 import com.tencent.supersonic.auth.authentication.utils.UserTokenUtils;
 import com.tencent.supersonic.common.util.S2ThreadContext;
-import java.lang.reflect.Field;
-import java.util.Arrays;
-import java.util.List;
-import javax.servlet.http.HttpServletRequest;
 import org.apache.catalina.connector.RequestFacade;
+import org.apache.commons.lang3.StringUtils;
 import org.apache.logging.log4j.util.Strings;
 import org.apache.tomcat.util.http.MimeHeaders;
 import org.springframework.util.CollectionUtils;
 import org.springframework.web.multipart.support.StandardMultipartHttpServletRequest;
 import org.springframework.web.servlet.HandlerInterceptor;
+import javax.servlet.http.HttpServletRequest;
+import java.lang.reflect.Field;
+import java.util.Arrays;
+import java.util.List;
 
 
 public abstract class AuthenticationInterceptor implements HandlerInterceptor {
@@ -29,7 +30,6 @@ public abstract class AuthenticationInterceptor implements HandlerInterceptor {
 
     protected S2ThreadContext s2ThreadContext;
 
-
     protected boolean isExcludedUri(String uri) {
         String excludePathStr = authenticationConfig.getExcludePath();
         if (Strings.isEmpty(excludePathStr)) {
@@ -59,6 +59,10 @@ public abstract class AuthenticationInterceptor implements HandlerInterceptor {
         return "true".equalsIgnoreCase(internal);
     }
 
+    protected boolean isAppRequest(HttpServletRequest request) {
+        String appId = request.getHeader(authenticationConfig.getAppId());
+        return StringUtils.isNotBlank(appId);
+    }
 
     protected void reflectSetparam(HttpServletRequest request, String key, String value) {
         try {

auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/interceptor/DefaultAuthenticationInterceptor.java

@@ -10,12 +10,12 @@ import com.tencent.supersonic.common.pojo.exception.AccessException;
 import com.tencent.supersonic.common.util.ContextUtils;
 import com.tencent.supersonic.common.util.S2ThreadContext;
 import com.tencent.supersonic.common.util.ThreadContext;
-import java.lang.reflect.Method;
-import javax.servlet.http.HttpServletRequest;
-import javax.servlet.http.HttpServletResponse;
 import lombok.extern.slf4j.Slf4j;
 import org.apache.commons.lang3.StringUtils;
 import org.springframework.web.method.HandlerMethod;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import java.lang.reflect.Method;
 
 @Slf4j
 public class DefaultAuthenticationInterceptor extends AuthenticationInterceptor {
@@ -35,7 +35,10 @@ public class DefaultAuthenticationInterceptor extends AuthenticationInterceptor
             setFakerUser(request);
             return true;
         }
-
+        if (isAppRequest(request)) {
+            setFakerUser(request);
+            return true;
+        }
         if (handler instanceof HandlerMethod) {
             HandlerMethod handlerMethod = (HandlerMethod) handler;
             Method method = handlerMethod.getMethod();
@@ -76,5 +79,4 @@ public class DefaultAuthenticationInterceptor extends AuthenticationInterceptor
         s2ThreadContext.set(threadContext);
     }
 
-
 }

auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/persistence/repository/Impl/UserRepositoryImpl.java

@@ -20,7 +20,6 @@ public class UserRepositoryImpl implements UserRepository {
         this.userDOMapper = userDOMapper;
     }
 
-
     @Override
     public List<UserDO> getUserList() {
         return userDOMapper.selectByExample(new UserDOExample());
@@ -40,5 +39,4 @@ public class UserRepositoryImpl implements UserRepository {
         return userDOOptional.orElse(null);
     }
 
-
 }

auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/rest/UserController.java

@@ -5,18 +5,18 @@ import com.tencent.supersonic.auth.api.authentication.pojo.Organization;
 import com.tencent.supersonic.auth.api.authentication.pojo.User;
 import com.tencent.supersonic.auth.api.authentication.request.UserReq;
 import com.tencent.supersonic.auth.api.authentication.service.UserService;
-import com.tencent.supersonic.auth.api.authentication.utils.UserHolder;
-import java.util.List;
-import java.util.Set;
-import javax.servlet.http.HttpServletRequest;
-import javax.servlet.http.HttpServletResponse;
 import lombok.extern.slf4j.Slf4j;
-import org.springframework.web.bind.annotation.RestController;
-import org.springframework.web.bind.annotation.RequestMapping;
 import org.springframework.web.bind.annotation.GetMapping;
+import org.springframework.web.bind.annotation.PathVariable;
 import org.springframework.web.bind.annotation.PostMapping;
 import org.springframework.web.bind.annotation.RequestBody;
-import org.springframework.web.bind.annotation.PathVariable;
+import org.springframework.web.bind.annotation.RequestMapping;
+import org.springframework.web.bind.annotation.RestController;
+
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import java.util.List;
+import java.util.Set;
 
 @RestController
 @RequestMapping("/api/auth/user")
@@ -29,10 +29,9 @@ public class UserController {
         this.userService = userService;
     }
 
-
     @GetMapping("/getCurrentUser")
     public User getCurrentUser(HttpServletRequest httpServletRequest, HttpServletResponse httpServletResponse) {
-        return UserHolder.findUser(httpServletRequest, httpServletResponse);
+        return userService.getCurrentUser(httpServletRequest, httpServletResponse);
     }
 
     @GetMapping("/getUserNames")
@@ -70,5 +69,4 @@ public class UserController {
         return userService.login(userCmd);
     }
 
-
 }

auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/service/UserServiceImpl.java

@@ -4,15 +4,38 @@ import com.tencent.supersonic.auth.api.authentication.pojo.Organization;
 import com.tencent.supersonic.auth.api.authentication.pojo.User;
 import com.tencent.supersonic.auth.api.authentication.request.UserReq;
 import com.tencent.supersonic.auth.api.authentication.service.UserService;
+import com.tencent.supersonic.auth.api.authentication.utils.UserHolder;
 import com.tencent.supersonic.auth.authentication.utils.ComponentFactory;
+import com.tencent.supersonic.common.pojo.SysParameter;
+import com.tencent.supersonic.common.service.SysParameterService;
+import org.springframework.stereotype.Service;
+import org.springframework.util.CollectionUtils;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
 import java.util.List;
 import java.util.Set;
 
-import org.springframework.stereotype.Service;
-
 @Service
 public class UserServiceImpl implements UserService {
 
+    private SysParameterService sysParameterService;
+
+    public UserServiceImpl(SysParameterService sysParameterService) {
+        this.sysParameterService = sysParameterService;
+    }
+
+    @Override
+    public User getCurrentUser(HttpServletRequest httpServletRequest, HttpServletResponse httpServletResponse) {
+        User user = UserHolder.findUser(httpServletRequest, httpServletResponse);
+        if (user != null) {
+            SysParameter sysParameter = sysParameterService.getSysParameter();
+            if (!CollectionUtils.isEmpty(sysParameter.getAdmins())
+                    && sysParameter.getAdmins().contains(user.getName())) {
+                user.setIsAdmin(1);
+            }
+        }
+        return user;
+    }
 
     @Override
     public List<String> getUserNames() {

auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/strategy/FakeUserStrategy.java

@@ -20,5 +20,4 @@ public class FakeUserStrategy implements UserStrategy {
         return User.getFakeUser();
     }
 
-
 }

auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/utils/UserTokenUtils.java

@@ -58,7 +58,6 @@ public class UserTokenUtils {
         return generate(claims);
     }
 
-
     public User getUser(HttpServletRequest request) {
         String token = request.getHeader(authenticationConfig.getTokenHttpHeaderKey());
         final Claims claims = getClaims(token);
@@ -120,5 +119,4 @@ public class UserTokenUtils {
                 .compact();
     }
 
-
 }

auth/authorization/src/main/java/com/tencent/supersonic/auth/authorization/service/AuthServiceImpl.java

@@ -1,6 +1,7 @@
-package com.tencent.supersonic.auth.authorization.application;
+package com.tencent.supersonic.auth.authorization.service;
 
 import com.google.common.base.Strings;
+import com.google.common.collect.Lists;
 import com.google.gson.Gson;
 import com.tencent.supersonic.auth.api.authentication.pojo.User;
 import com.tencent.supersonic.auth.api.authentication.service.UserService;
@@ -75,7 +76,6 @@ public class AuthServiceImpl implements AuthService {
         jdbcTemplate.update("delete from s2_auth_groups where group_id = ?", group.getGroupId());
     }
 
-
     @Override
     public AuthorizedResourceResp queryAuthorizedResources(QueryAuthResReq req, User user) {
         Set<String> userOrgIds = userService.getUserAllOrgId(user.getName());
@@ -109,8 +109,11 @@ public class AuthServiceImpl implements AuthService {
             }
         }
 
-        if (req.getModelId() != null) {
-            List<AuthGroup> authGroups = authGroupsByModelId.get(req.getModelId());
+        if (!CollectionUtils.isEmpty(req.getModelIds())) {
+            List<AuthGroup> authGroups = Lists.newArrayList();
+            for (Long modelId : authGroupsByModelId.keySet()) {
+                authGroups.addAll(authGroupsByModelId.getOrDefault(modelId, Lists.newArrayList()));
+            }
             if (!CollectionUtils.isEmpty(authGroups)) {
                 for (AuthGroup group : authGroups) {
                     if (group.getDimensionFilters() != null

chat/api/pom.xml

@@ -23,7 +23,7 @@
         </dependency>
         <dependency>
             <groupId>com.tencent.supersonic</groupId>
-            <artifactId>semantic-api</artifactId>
+            <artifactId>headless-api</artifactId>
             <version>${project.version}</version>
             <scope>compile</scope>
         </dependency>

chat/api/src/main/java/com/tencent/supersonic/chat/api/component/SemanticInterpreter.java

@@ -1,63 +0,0 @@
-package com.tencent.supersonic.chat.api.component;
-
-import com.github.pagehelper.PageInfo;
-import com.tencent.supersonic.auth.api.authentication.pojo.User;
-import com.tencent.supersonic.chat.api.pojo.ModelSchema;
-import com.tencent.supersonic.common.pojo.enums.AuthType;
-import com.tencent.supersonic.semantic.api.model.request.PageDimensionReq;
-import com.tencent.supersonic.semantic.api.model.request.PageMetricReq;
-import com.tencent.supersonic.semantic.api.model.response.DomainResp;
-import com.tencent.supersonic.semantic.api.model.response.DimensionResp;
-import com.tencent.supersonic.semantic.api.model.response.ExplainResp;
-import com.tencent.supersonic.semantic.api.model.response.MetricResp;
-import com.tencent.supersonic.semantic.api.model.response.ModelResp;
-import com.tencent.supersonic.semantic.api.model.response.ModelSchemaResp;
-import com.tencent.supersonic.semantic.api.model.response.QueryResultWithSchemaResp;
-import com.tencent.supersonic.semantic.api.query.request.ExplainSqlReq;
-import com.tencent.supersonic.semantic.api.query.request.QueryDimValueReq;
-import com.tencent.supersonic.semantic.api.query.request.QueryS2SQLReq;
-import com.tencent.supersonic.semantic.api.query.request.QueryMultiStructReq;
-import com.tencent.supersonic.semantic.api.query.request.QueryStructReq;
-
-import java.util.List;
-
-/**
- * A semantic layer provides a simplified and consistent view of data from multiple sources.
- * It abstracts away the complexity of the underlying data sources and provides a unified view
- * of the data that is easier to understand and use.
- * <p>
- * The interface defines methods for getting metadata as well as querying data in the semantic layer.
- * Implementations of this interface should provide concrete implementations that interact with the
- * underlying data sources and return results in a consistent format. Or it can be implemented
- * as proxy to a remote semantic service.
- * </p>
- */
-public interface SemanticInterpreter {
-
-    QueryResultWithSchemaResp queryByStruct(QueryStructReq queryStructReq, User user);
-
-    QueryResultWithSchemaResp queryByMultiStruct(QueryMultiStructReq queryMultiStructReq, User user);
-
-    QueryResultWithSchemaResp queryByS2SQL(QueryS2SQLReq queryS2SQLReq, User user);
-
-    QueryResultWithSchemaResp queryDimValue(QueryDimValueReq queryDimValueReq, User user);
-
-    List<ModelSchema> getModelSchema();
-
-    List<ModelSchema> getModelSchema(List<Long> ids);
-
-    ModelSchema getModelSchema(Long model, Boolean cacheEnable);
-
-    PageInfo<DimensionResp> getDimensionPage(PageDimensionReq pageDimensionReq);
-
-    PageInfo<MetricResp> getMetricPage(PageMetricReq pageDimensionReq, User user);
-
-    List<DomainResp> getDomainList(User user);
-
-    List<ModelResp> getModelList(AuthType authType, Long domainId, User user);
-
-    <T> ExplainResp explain(ExplainSqlReq<T> explainSqlReq, User user) throws Exception;
-
-    List<ModelSchemaResp> fetchModelSchema(List<Long> ids, Boolean cacheEnable);
-
-}

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/QueryContext.java

@@ -1,23 +0,0 @@
-package com.tencent.supersonic.chat.api.pojo;
-
-import com.tencent.supersonic.chat.api.component.SemanticQuery;
-import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
-import lombok.Data;
-import lombok.NoArgsConstructor;
-
-import java.util.ArrayList;
-import java.util.List;
-
-@Data
-@NoArgsConstructor
-public class QueryContext {
-
-    private QueryReq request;
-    private List<SemanticQuery> candidateQueries = new ArrayList<>();
-    private SchemaMapInfo mapInfo = new SchemaMapInfo();
-    private SchemaModelClusterMapInfo modelClusterMapInfo = new SchemaModelClusterMapInfo();
-
-    public QueryContext(QueryReq request) {
-        this.request = request;
-    }
-}

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/SchemaElementMatch.java

@@ -1,5 +1,6 @@
 package com.tencent.supersonic.chat.api.pojo;
 
+import com.tencent.supersonic.headless.api.pojo.SchemaElement;
 import lombok.AllArgsConstructor;
 import lombok.Builder;
 import lombok.Data;

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/SchemaMapInfo.java

@@ -1,5 +1,7 @@
 package com.tencent.supersonic.chat.api.pojo;
 
+import com.google.common.collect.Lists;
+
 import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
@@ -7,25 +9,25 @@ import java.util.Set;
 
 public class SchemaMapInfo {
 
-    private Map<Long, List<SchemaElementMatch>> modelElementMatches = new HashMap<>();
+    private Map<Long, List<SchemaElementMatch>> viewElementMatches = new HashMap<>();
 
-    public Set<Long> getMatchedModels() {
-        return modelElementMatches.keySet();
+    public Set<Long> getMatchedViewInfos() {
+        return viewElementMatches.keySet();
     }
 
-    public List<SchemaElementMatch> getMatchedElements(Long model) {
-        return modelElementMatches.get(model);
+    public List<SchemaElementMatch> getMatchedElements(Long view) {
+        return viewElementMatches.getOrDefault(view, Lists.newArrayList());
     }
 
-    public Map<Long, List<SchemaElementMatch>> getModelElementMatches() {
-        return modelElementMatches;
+    public Map<Long, List<SchemaElementMatch>> getViewElementMatches() {
+        return viewElementMatches;
     }
 
-    public void setModelElementMatches(Map<Long, List<SchemaElementMatch>> modelElementMatches) {
-        this.modelElementMatches = modelElementMatches;
+    public void setViewElementMatches(Map<Long, List<SchemaElementMatch>> viewElementMatches) {
+        this.viewElementMatches = viewElementMatches;
     }
 
-    public void setMatchedElements(Long model, List<SchemaElementMatch> elementMatches) {
-        modelElementMatches.put(model, elementMatches);
+    public void setMatchedElements(Long view, List<SchemaElementMatch> elementMatches) {
+        viewElementMatches.put(view, elementMatches);
     }
 }

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/SchemaModelClusterMapInfo.java

@@ -1,61 +0,0 @@
-package com.tencent.supersonic.chat.api.pojo;
-
-import com.clickhouse.client.internal.apache.commons.compress.utils.Lists;
-import com.tencent.supersonic.common.pojo.ModelCluster;
-import lombok.Data;
-import org.springframework.util.CollectionUtils;
-
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-
-@Data
-public class SchemaModelClusterMapInfo {
-
-    private Map<String, List<SchemaElementMatch>> modelElementMatches = new HashMap<>();
-
-    public Set<String> getMatchedModelClusters() {
-        return modelElementMatches.keySet();
-    }
-
-    public List<SchemaElementMatch> getMatchedElements(Long modelId) {
-        for (String key : modelElementMatches.keySet()) {
-            if (ModelCluster.getModelIdFromKey(key).contains(modelId)) {
-                return modelElementMatches.get(key);
-            }
-        }
-        return Lists.newArrayList();
-    }
-
-    public List<SchemaElementMatch> getMatchedElements(String modelCluster) {
-        return modelElementMatches.get(modelCluster);
-    }
-
-    public Map<String, List<SchemaElementMatch>> getModelElementMatches() {
-        return modelElementMatches;
-    }
-
-    public Map<String, List<SchemaElementMatch>> getElementMatchesByModelIds(Set<Long> modelIds) {
-        if (CollectionUtils.isEmpty(modelIds)) {
-            return modelElementMatches;
-        }
-        Map<String, List<SchemaElementMatch>> modelElementMatchesFiltered = new HashMap<>();
-        for (String key : modelElementMatches.keySet()) {
-            for (Long modelId : modelIds) {
-                if (ModelCluster.getModelIdFromKey(key).contains(modelId)) {
-                    modelElementMatchesFiltered.put(key, modelElementMatches.get(key));
-                }
-            }
-        }
-        return modelElementMatchesFiltered;
-    }
-
-    public void setModelElementMatches(Map<String, List<SchemaElementMatch>> modelElementMatches) {
-        this.modelElementMatches = modelElementMatches;
-    }
-
-    public void setMatchedElements(String modelCluster, List<SchemaElementMatch> elementMatches) {
-        modelElementMatches.put(modelCluster, elementMatches);
-    }
-}

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/SemanticParseInfo.java

@@ -1,15 +1,15 @@
 package com.tencent.supersonic.chat.api.pojo;
 
 
+import com.tencent.supersonic.chat.api.pojo.response.SqlInfo;
 import com.tencent.supersonic.chat.api.pojo.request.QueryFilter;
 import com.tencent.supersonic.chat.api.pojo.response.EntityInfo;
-import com.tencent.supersonic.chat.api.pojo.response.SqlInfo;
 import com.tencent.supersonic.common.pojo.DateConf;
-import com.tencent.supersonic.common.pojo.ModelCluster;
 import com.tencent.supersonic.common.pojo.Order;
-import com.tencent.supersonic.common.pojo.QueryType;
+import com.tencent.supersonic.common.pojo.enums.QueryType;
 import com.tencent.supersonic.common.pojo.enums.AggregateTypeEnum;
 import com.tencent.supersonic.common.pojo.enums.FilterType;
+import com.tencent.supersonic.headless.api.pojo.SchemaElement;
 import lombok.Data;
 
 import java.util.ArrayList;
@@ -26,7 +26,7 @@ public class SemanticParseInfo {
 
     private Integer id;
     private String queryMode;
-    private ModelCluster model = new ModelCluster();
+    private SchemaElement view;
     private Set<SchemaElement> metrics = new TreeSet<>(new SchemaNameLengthComparator());
     private Set<SchemaElement> dimensions = new LinkedHashSet();
     private SchemaElement entity;
@@ -42,21 +42,7 @@ public class SemanticParseInfo {
     private Map<String, Object> properties = new HashMap<>();
     private EntityInfo entityInfo;
     private SqlInfo sqlInfo = new SqlInfo();
-    private QueryType queryType = QueryType.OTHER;
-
-    public String getModelClusterKey() {
-        if (model == null) {
-            return "";
-        }
-        return model.getKey();
-    }
-
-    public String getModelName() {
-        if (model == null) {
-            return "";
-        }
-        return model.getName();
-    }
+    private QueryType queryType = QueryType.ID;
 
     private static class SchemaNameLengthComparator implements Comparator<SchemaElement> {
 
@@ -86,26 +72,15 @@ public class SemanticParseInfo {
         return metrics;
     }
 
-    private Map<Long, Integer> getModelElementCountMap() {
-        Map<Long, Integer> elementCountMap = new HashMap<>();
-        elementMatches.forEach(element -> {
-            int count = elementCountMap.getOrDefault(element.getElement().getModel(), 0);
-            elementCountMap.put(element.getElement().getModel(), count + 1);
-        });
-        return elementCountMap;
+    public Long getViewId() {
+        if (view == null) {
+            return null;
+        }
+        return view.getView();
     }
 
-    public Long getModelId() {
-        Map<Long, Integer> elementCountMap = getModelElementCountMap();
-        Long modelId = -1L;
-        int maxCnt = 0;
-        for (Long model : elementCountMap.keySet()) {
-            if (elementCountMap.get(model) > maxCnt) {
-                maxCnt = elementCountMap.get(model);
-                modelId = model;
-            }
-        }
-        return modelId;
+    public SchemaElement getModel() {
+        return view;
     }
 
 }

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/SemanticSchema.java

@@ -1,26 +1,26 @@
 package com.tencent.supersonic.chat.api.pojo;
 
-import org.springframework.util.CollectionUtils;
-
+import com.tencent.supersonic.headless.api.pojo.SchemaElement;
+import com.tencent.supersonic.headless.api.pojo.SchemaElementType;
 import java.io.Serializable;
 import java.util.ArrayList;
 import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
 import java.util.Optional;
-import java.util.Set;
 import java.util.stream.Collectors;
+import org.springframework.util.CollectionUtils;
 
 public class SemanticSchema implements Serializable {
 
-    private List<ModelSchema> modelSchemaList;
+    private List<ViewSchema> viewSchemaList;
 
-    public SemanticSchema(List<ModelSchema> modelSchemaList) {
-        this.modelSchemaList = modelSchemaList;
+    public SemanticSchema(List<ViewSchema> viewSchemaList) {
+        this.viewSchemaList = viewSchemaList;
     }
 
-    public void add(ModelSchema schema) {
-        modelSchemaList.add(schema);
+    public void add(ViewSchema schema) {
+        viewSchemaList.add(schema);
     }
 
     public SchemaElement getElement(SchemaElementType elementType, long elementID) {
@@ -30,8 +30,8 @@ public class SemanticSchema implements Serializable {
             case ENTITY:
                 element = getElementsById(elementID, getEntities());
                 break;
-            case MODEL:
-                element = getElementsById(elementID, getModels());
+            case VIEW:
+                element = getElementsById(elementID, getViews());
                 break;
             case METRIC:
                 element = getElementsById(elementID, getMetrics());
@@ -52,58 +52,29 @@ public class SemanticSchema implements Serializable {
         }
     }
 
-    public SchemaElement getElementByName(SchemaElementType elementType, String name) {
-        Optional<SchemaElement> element = Optional.empty();
-
-        switch (elementType) {
-            case ENTITY:
-                element = getElementsByName(name, getEntities());
-                break;
-            case MODEL:
-                element = getElementsByName(name, getModels());
-                break;
-            case METRIC:
-                element = getElementsByName(name, getMetrics());
-                break;
-            case DIMENSION:
-                element = getElementsByName(name, getDimensions());
-                break;
-            case VALUE:
-                element = getElementsByName(name, getDimensionValues());
-                break;
-            default:
-        }
-
-        if (element.isPresent()) {
-            return element.get();
-        } else {
-            return null;
-        }
-    }
-
-    public Map<Long, String> getModelIdToName() {
-        return modelSchemaList.stream()
-                .collect(Collectors.toMap(a -> a.getModel().getId(), a -> a.getModel().getName(), (k1, k2) -> k1));
+    public Map<Long, String> getViewIdToName() {
+        return viewSchemaList.stream()
+                .collect(Collectors.toMap(a -> a.getView().getId(), a -> a.getView().getName(), (k1, k2) -> k1));
     }
 
     public List<SchemaElement> getDimensionValues() {
         List<SchemaElement> dimensionValues = new ArrayList<>();
-        modelSchemaList.stream().forEach(d -> dimensionValues.addAll(d.getDimensionValues()));
+        viewSchemaList.stream().forEach(d -> dimensionValues.addAll(d.getDimensionValues()));
         return dimensionValues;
     }
 
     public List<SchemaElement> getDimensions() {
         List<SchemaElement> dimensions = new ArrayList<>();
-        modelSchemaList.stream().forEach(d -> dimensions.addAll(d.getDimensions()));
+        viewSchemaList.stream().forEach(d -> dimensions.addAll(d.getDimensions()));
         return dimensions;
     }
 
-    public List<SchemaElement> getDimensions(Set<Long> modelIds) {
+    public List<SchemaElement> getDimensions(Long viewId) {
         List<SchemaElement> dimensions = getDimensions();
-        return getElementsByModelId(modelIds, dimensions);
+        return getElementsByViewId(viewId, dimensions);
     }
 
-    public SchemaElement getDimensions(Long id) {
+    public SchemaElement getDimension(Long id) {
         List<SchemaElement> dimensions = getDimensions();
         Optional<SchemaElement> dimension = getElementsById(id, dimensions);
         return dimension.orElse(null);
@@ -111,37 +82,43 @@ public class SemanticSchema implements Serializable {
 
     public List<SchemaElement> getTags() {
         List<SchemaElement> tags = new ArrayList<>();
-        modelSchemaList.stream().forEach(d -> tags.addAll(d.getTags()));
+        viewSchemaList.stream().forEach(d -> tags.addAll(d.getTags()));
         return tags;
     }
 
-    public List<SchemaElement> getTags(Set<Long> modelIds) {
+    public List<SchemaElement> getTags(Long viewId) {
         List<SchemaElement> tags = new ArrayList<>();
-        modelSchemaList.stream().filter(schemaElement -> modelIds.contains(schemaElement.getModel()))
+        viewSchemaList.stream().filter(schemaElement ->
+                        viewId.equals(schemaElement.getView().getView()))
                 .forEach(d -> tags.addAll(d.getTags()));
         return tags;
     }
 
     public List<SchemaElement> getMetrics() {
         List<SchemaElement> metrics = new ArrayList<>();
-        modelSchemaList.stream().forEach(d -> metrics.addAll(d.getMetrics()));
+        viewSchemaList.stream().forEach(d -> metrics.addAll(d.getMetrics()));
         return metrics;
     }
 
-    public List<SchemaElement> getMetrics(Set<Long> modelIds) {
+    public List<SchemaElement> getMetrics(Long viewId) {
         List<SchemaElement> metrics = getMetrics();
-        return getElementsByModelId(modelIds, metrics);
+        return getElementsByViewId(viewId, metrics);
     }
 
     public List<SchemaElement> getEntities() {
         List<SchemaElement> entities = new ArrayList<>();
-        modelSchemaList.stream().forEach(d -> entities.add(d.getEntity()));
+        viewSchemaList.stream().forEach(d -> entities.add(d.getEntity()));
         return entities;
     }
 
-    private List<SchemaElement> getElementsByModelId(Set<Long> modelIds, List<SchemaElement> elements) {
+    public List<SchemaElement> getEntities(Long viewId) {
+        List<SchemaElement> entities = getEntities();
+        return getElementsByViewId(viewId, entities);
+    }
+
+    private List<SchemaElement> getElementsByViewId(Long viewId, List<SchemaElement> elements) {
         return elements.stream()
-                .filter(schemaElement -> modelIds.contains(schemaElement.getModel()))
+                .filter(schemaElement -> viewId.equals(schemaElement.getView()))
                 .collect(Collectors.toList());
     }
 
@@ -151,31 +128,30 @@ public class SemanticSchema implements Serializable {
                 .findFirst();
     }
 
-    private Optional<SchemaElement> getElementsByName(String name, List<SchemaElement> elements) {
-        return elements.stream()
-                .filter(schemaElement -> name.equals(schemaElement.getName()))
-                .findFirst();
+    public SchemaElement getView(Long viewId) {
+        List<SchemaElement> views = getViews();
+        return getElementsById(viewId, views).orElse(null);
     }
 
-    public List<SchemaElement> getModels() {
-        List<SchemaElement> models = new ArrayList<>();
-        modelSchemaList.stream().forEach(d -> models.add(d.getModel()));
-        return models;
+    public List<SchemaElement> getViews() {
+        List<SchemaElement> views = new ArrayList<>();
+        viewSchemaList.stream().forEach(d -> views.add(d.getView()));
+        return views;
     }
 
-    public Map<String, String> getBizNameToName(Set<Long> modelIds) {
+    public Map<String, String> getBizNameToName(Long viewId) {
         List<SchemaElement> allElements = new ArrayList<>();
-        allElements.addAll(getDimensions(modelIds));
-        allElements.addAll(getMetrics(modelIds));
+        allElements.addAll(getDimensions(viewId));
+        allElements.addAll(getMetrics(viewId));
         return allElements.stream()
                 .collect(Collectors.toMap(SchemaElement::getBizName, SchemaElement::getName, (k1, k2) -> k1));
     }
 
-    public Map<Long, ModelSchema> getModelSchemaMap() {
-        if (CollectionUtils.isEmpty(modelSchemaList)) {
+    public Map<Long, ViewSchema> getViewSchemaMap() {
+        if (CollectionUtils.isEmpty(viewSchemaList)) {
             return new HashMap<>();
         }
-        return modelSchemaList.stream().collect(Collectors.toMap(modelSchema
-                -> modelSchema.getModel().getModel(), modelSchema -> modelSchema));
+        return viewSchemaList.stream().collect(Collectors.toMap(viewSchema
+                -> viewSchema.getView().getView(), viewSchema -> viewSchema));
     }
 }

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/ViewSchema.java

@@ -1,26 +1,26 @@
 package com.tencent.supersonic.chat.api.pojo;
 
-import com.google.common.collect.Sets;
-import com.tencent.supersonic.common.pojo.ModelRela;
+import com.tencent.supersonic.headless.api.pojo.QueryConfig;
+import com.tencent.supersonic.headless.api.pojo.SchemaElement;
+import com.tencent.supersonic.headless.api.pojo.SchemaElementType;
+import com.tencent.supersonic.headless.api.pojo.TagTypeDefaultConfig;
+import com.tencent.supersonic.headless.api.pojo.TimeDefaultConfig;
 import lombok.Data;
-import org.springframework.util.CollectionUtils;
 
-import java.util.ArrayList;
 import java.util.HashSet;
-import java.util.List;
 import java.util.Optional;
 import java.util.Set;
 
 @Data
-public class ModelSchema {
+public class ViewSchema {
 
-    private SchemaElement model;
+    private SchemaElement view;
     private Set<SchemaElement> metrics = new HashSet<>();
     private Set<SchemaElement> dimensions = new HashSet<>();
     private Set<SchemaElement> dimensionValues = new HashSet<>();
     private Set<SchemaElement> tags = new HashSet<>();
     private SchemaElement entity = new SchemaElement();
-    private List<ModelRela> modelRelas = new ArrayList<>();
+    private QueryConfig queryConfig;
 
     public SchemaElement getElement(SchemaElementType elementType, long elementID) {
         Optional<SchemaElement> element = Optional.empty();
@@ -29,8 +29,8 @@ public class ModelSchema {
             case ENTITY:
                 element = Optional.ofNullable(entity);
                 break;
-            case MODEL:
-                element = Optional.of(model);
+            case VIEW:
+                element = Optional.of(view);
                 break;
             case METRIC:
                 element = metrics.stream().filter(e -> e.getId() == elementID).findFirst();
@@ -61,8 +61,8 @@ public class ModelSchema {
             case ENTITY:
                 element = Optional.ofNullable(entity);
                 break;
-            case MODEL:
-                element = Optional.of(model);
+            case VIEW:
+                element = Optional.of(view);
                 break;
             case METRIC:
                 element = metrics.stream().filter(e -> name.equals(e.getName())).findFirst();
@@ -83,16 +83,31 @@ public class ModelSchema {
         }
     }
 
-    public Set<Long> getModelClusterSet() {
-        if (CollectionUtils.isEmpty(modelRelas)) {
-            return Sets.newHashSet();
+    public TimeDefaultConfig getTagTypeTimeDefaultConfig() {
+        if (queryConfig == null) {
+            return null;
         }
-        Set<Long> modelClusterSet = new HashSet<>();
-        modelRelas.forEach(modelRela -> {
-            modelClusterSet.add(modelRela.getToModelId());
-            modelClusterSet.add(modelRela.getFromModelId());
-        });
-        return modelClusterSet;
+        if (queryConfig.getTagTypeDefaultConfig() == null) {
+            return null;
+        }
+        return queryConfig.getTagTypeDefaultConfig().getTimeDefaultConfig();
+    }
+
+    public TimeDefaultConfig getMetricTypeTimeDefaultConfig() {
+        if (queryConfig == null) {
+            return null;
+        }
+        if (queryConfig.getMetricTypeDefaultConfig() == null) {
+            return null;
+        }
+        return queryConfig.getMetricTypeDefaultConfig().getTimeDefaultConfig();
+    }
+
+    public TagTypeDefaultConfig getTagTypeDefaultConfig() {
+        if (queryConfig == null) {
+            return null;
+        }
+        return queryConfig.getTagTypeDefaultConfig();
     }
 
 }

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/ChatDefaultConfigReq.java

@@ -1,7 +1,6 @@
 package com.tencent.supersonic.chat.api.pojo.request;
 
 
-import com.tencent.supersonic.common.pojo.Constants;
 import lombok.Data;
 
 import java.util.ArrayList;
@@ -13,26 +12,5 @@ public class ChatDefaultConfigReq {
     private List<Long> dimensionIds = new ArrayList<>();
     private List<Long> metricIds = new ArrayList<>();
 
-    /**
-     * default time span unit
-     */
-    private Integer unit = 1;
-
-    /**
-     * default time type: day
-     * DAY, WEEK, MONTH, YEAR
-     */
-    private String period = Constants.DAY;
-
-    private TimeMode timeMode = TimeMode.LAST;
-
-    public enum TimeMode {
-        /**
-         * date mode
-         * LAST   - a certain time
-         * RECENT - a period time
-         */
-        LAST, RECENT
-    }
 
 }

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/DictLatestTaskReq.java

@@ -1,23 +0,0 @@
-package com.tencent.supersonic.chat.api.pojo.request;
-
-import lombok.Data;
-import lombok.NoArgsConstructor;
-import lombok.ToString;
-
-import javax.validation.constraints.NotNull;
-import java.util.List;
-
-import static java.time.LocalDate.now;
-
-@ToString
-@Data
-@NoArgsConstructor
-public class DictLatestTaskReq {
-
-    @NotNull
-    private Long modelId;
-
-    private List<Long> dimIds;
-
-    private String createdAt = now().plusDays(-4).toString();
-}

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/DictTaskFilterReq.java

@@ -1,20 +0,0 @@
-package com.tencent.supersonic.chat.api.pojo.request;
-
-import com.tencent.supersonic.common.pojo.enums.TaskStatusEnum;
-import lombok.Data;
-import lombok.ToString;
-
-@ToString
-@Data
-public class DictTaskFilterReq {
-
-    private Long id;
-
-    private String name;
-
-    private String createdBy;
-
-    private String createdAt;
-
-    private TaskStatusEnum status;
-}

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/PluginQueryReq.java

@@ -13,7 +13,7 @@ public class PluginQueryReq {
 
     private String type;
 
-    private String model;
+    private String view;
 
     private String pattern;
 

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/QueryDataReq.java

@@ -2,7 +2,7 @@ package com.tencent.supersonic.chat.api.pojo.request;
 
 
 import com.tencent.supersonic.auth.api.authentication.pojo.User;
-import com.tencent.supersonic.chat.api.pojo.SchemaElement;
+import com.tencent.supersonic.headless.api.pojo.SchemaElement;
 import com.tencent.supersonic.common.pojo.DateConf;
 import java.util.HashSet;
 import java.util.Set;

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/SimilarQueryReq.java

@@ -10,7 +10,7 @@ import lombok.NoArgsConstructor;
 @NoArgsConstructor
 @AllArgsConstructor
 @Builder
-public class SolvedQueryReq {
+public class SimilarQueryReq {
 
     private Long queryId;
 
@@ -18,7 +18,7 @@ public class SolvedQueryReq {
 
     private String queryText;
 
-    private String modelId;
+    private Long viewId;
 
     private Integer agentId;
 

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/ChatDefaultRichConfigResp.java

@@ -1,9 +1,9 @@
 package com.tencent.supersonic.chat.api.pojo.response;
 
 
-import com.tencent.supersonic.chat.api.pojo.SchemaElement;
-import com.tencent.supersonic.chat.api.pojo.request.ChatDefaultConfigReq;
+import com.tencent.supersonic.headless.api.pojo.SchemaElement;
 import com.tencent.supersonic.common.pojo.Constants;
+import com.tencent.supersonic.common.pojo.enums.TimeMode;
 import lombok.Data;
 
 import java.util.List;
@@ -21,11 +21,11 @@ public class ChatDefaultRichConfigResp {
     private Integer unit = 1;
 
     /**
-     * default time type: day
+     * default time type:
      * DAY, WEEK, MONTH, YEAR
      */
     private String period = Constants.DAY;
 
-    private ChatDefaultConfigReq.TimeMode timeMode;
+    private TimeMode timeMode;
 
 }

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/DataInfo.java

@@ -1,8 +1,12 @@
 package com.tencent.supersonic.chat.api.pojo.response;
 
+import lombok.AllArgsConstructor;
 import lombok.Data;
+import lombok.NoArgsConstructor;
 
 @Data
+@AllArgsConstructor
+@NoArgsConstructor
 public class DataInfo {
 
     private Integer itemId;

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/EntityInfo.java

@@ -1,14 +1,16 @@
 package com.tencent.supersonic.chat.api.pojo.response;
 
+import lombok.Data;
+
 import java.util.ArrayList;
 import java.util.List;
-import lombok.Data;
 
 @Data
 public class EntityInfo {
 
-    private ModelInfo modelInfo = new ModelInfo();
+    private ViewInfo viewInfo = new ViewInfo();
     private List<DataInfo> dimensions = new ArrayList<>();
     private List<DataInfo> metrics = new ArrayList<>();
     private String entityId;
+
 }

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/EntityRichInfoResp.java

@@ -1,6 +1,6 @@
 package com.tencent.supersonic.chat.api.pojo.response;
 
-import com.tencent.supersonic.chat.api.pojo.SchemaElement;
+import com.tencent.supersonic.headless.api.pojo.SchemaElement;
 
 import java.util.List;
 import lombok.Data;

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/ParseResp.java

@@ -12,7 +12,6 @@ public class ParseResp {
     private Long queryId;
     private ParseState state;
     private List<SemanticParseInfo> selectedParses = Lists.newArrayList();
-    private List<SemanticParseInfo> candidateParses = Lists.newArrayList();
     private ParseTimeCostDO parseTimeCost = new ParseTimeCostDO();
 
     public enum ParseState {

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/QueryRecallResp.java

@@ -6,6 +6,6 @@ import java.util.List;
 
 @Data
 public class QueryRecallResp {
-    private List<SolvedQueryRecallResp> solvedQueryRecallRespList;
+    private List<SimilarQueryRecallResp> solvedQueryRecallRespList;
     private Long queryTimeCost;
 }

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/QueryResp.java

@@ -1,9 +1,9 @@
 package com.tencent.supersonic.chat.api.pojo.response;
 
-import java.util.Date;
-import java.util.List;
 import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
 import lombok.Data;
+import java.util.Date;
+import java.util.List;
 
 @Data
 public class QueryResp {
@@ -16,4 +16,7 @@ public class QueryResp {
     private String queryText;
     private QueryResult queryResult;
     private List<SemanticParseInfo> parseInfos;
+    private List<SimilarQueryRecallResp> similarQueries;
+
+
 }

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/QueryResult.java

@@ -1,11 +1,12 @@
 package com.tencent.supersonic.chat.api.pojo.response;
 
+import com.tencent.supersonic.headless.api.pojo.SchemaElement;
 import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
 import com.tencent.supersonic.common.pojo.QueryAuthorization;
 import com.tencent.supersonic.common.pojo.QueryColumn;
+import lombok.Data;
 import java.util.List;
 import java.util.Map;
-import lombok.Data;
 
 @Data
 public class QueryResult {
@@ -22,4 +23,5 @@ public class QueryResult {
     private Object response;
     private List<Map<String, Object>> queryResults;
     private Long queryTimeCost;
+    private List<SchemaElement> recommendedDimensions;
 }

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/RecommendResp.java

@@ -1,6 +1,6 @@
 package com.tencent.supersonic.chat.api.pojo.response;
 
-import com.tencent.supersonic.chat.api.pojo.SchemaElement;
+import com.tencent.supersonic.headless.api.pojo.SchemaElement;
 import lombok.Data;
 
 import java.util.List;

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/SearchResult.java

@@ -1,6 +1,6 @@
 package com.tencent.supersonic.chat.api.pojo.response;
 
-import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
+import com.tencent.supersonic.headless.api.pojo.SchemaElementType;
 import java.util.Objects;
 import lombok.Builder;
 import lombok.Data;

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/SimilarQueryRecallResp.java

@@ -6,7 +6,7 @@ import lombok.Data;
 
 @Data
 @Builder
-public class SolvedQueryRecallResp {
+public class SimilarQueryRecallResp {
 
     private Long queryId;
 

chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/ViewInfo.java

@@ -6,7 +6,7 @@ import java.io.Serializable;
 import java.util.List;
 
 @Data
-public class ModelInfo extends DataInfo implements Serializable {
+public class ViewInfo extends DataInfo implements Serializable {
 
     private List<String> words;
     private String primaryKey;

chat/core/pom.xml

@@ -21,7 +21,6 @@
             <groupId>org.springframework</groupId>
             <artifactId>spring-context</artifactId>
         </dependency>
-
         <dependency>
             <groupId>org.testng</groupId>
             <artifactId>testng</artifactId>
@@ -40,11 +39,6 @@
             <scope>compile</scope>
         </dependency>
 
-        <dependency>
-            <groupId>org.junit.jupiter</groupId>
-            <artifactId>junit-jupiter</artifactId>
-            <scope>test</scope>
-        </dependency>
         <dependency>
             <groupId>org.springframework.boot</groupId>
             <artifactId>spring-boot-starter-test</artifactId>
@@ -77,32 +71,20 @@
         </dependency>
         <dependency>
             <groupId>com.tencent.supersonic</groupId>
-            <artifactId>chat-knowledge</artifactId>
+            <artifactId>headless-api</artifactId>
             <version>${project.version}</version>
         </dependency>
         <dependency>
             <groupId>com.tencent.supersonic</groupId>
-            <artifactId>semantic-api</artifactId>
+            <artifactId>headless-core</artifactId>
             <version>${project.version}</version>
         </dependency>
-        <dependency>
-            <groupId>com.tencent.supersonic</groupId>
-            <artifactId>semantic-query</artifactId>
-            <version>${project.version}</version>
-            <scope>test</scope>
-        </dependency>
         <dependency>
             <groupId>com.tencent.supersonic</groupId>
             <artifactId>chat-api</artifactId>
             <version>${project.version}</version>
             <scope>compile</scope>
         </dependency>
-        <dependency>
-            <groupId>com.tencent.supersonic</groupId>
-            <artifactId>semantic-query</artifactId>
-            <version>${project.version}</version>
-            <scope>compile</scope>
-        </dependency>
 
         <dependency>
             <groupId>com.github.xkzhangsan</groupId>
@@ -116,6 +98,12 @@
             <version>${mockito-inline.version}</version>
             <scope>test</scope>
         </dependency>
+        <dependency>
+            <groupId>com.tencent.supersonic</groupId>
+            <artifactId>headless-server</artifactId>
+            <version>${project.version}</version>
+            <scope>compile</scope>
+        </dependency>
 
     </dependencies>
 

chat/core/src/main/java/com/tencent/supersonic/chat/agent/tool/AgentToolType.java

@@ -1,8 +0,0 @@
-package com.tencent.supersonic.chat.agent.tool;
-
-public enum AgentToolType {
-    RULE,
-    LLM_S2SQL,
-    PLUGIN,
-    INTERPRET
-}

chat/core/src/main/java/com/tencent/supersonic/chat/agent/tool/LLMParserTool.java

@@ -1,12 +0,0 @@
-package com.tencent.supersonic.chat.agent.tool;
-
-import lombok.Data;
-
-import java.util.List;
-
-@Data
-public class LLMParserTool extends CommonAgentTool {
-
-    private List<String> exampleQuestions;
-
-}

chat/core/src/main/java/com/tencent/supersonic/chat/agent/tool/MetricInterpretTool.java

@@ -1,16 +0,0 @@
-package com.tencent.supersonic.chat.agent.tool;
-
-import com.tencent.supersonic.chat.parser.llm.interpret.MetricOption;
-import lombok.Data;
-
-import java.util.List;
-
-
-@Data
-public class MetricInterpretTool extends AgentTool {
-
-    private Long modelId;
-
-    private List<MetricOption> metricOptions;
-
-}

chat/core/src/main/java/com/tencent/supersonic/chat/config/EntityInternalDetail.java

@@ -1,14 +0,0 @@
-package com.tencent.supersonic.chat.config;
-
-
-import com.tencent.supersonic.semantic.api.model.response.DimSchemaResp;
-import com.tencent.supersonic.semantic.api.model.response.MetricSchemaResp;
-import java.util.List;
-import lombok.Data;
-
-@Data
-public class EntityInternalDetail {
-
-    List<DimSchemaResp> dimensionList;
-    List<MetricSchemaResp> metricList;
-}

chat/core/src/main/java/com/tencent/supersonic/chat/core/agent/Agent.java

@@ -1,15 +1,18 @@
-package com.tencent.supersonic.chat.agent;
+package com.tencent.supersonic.chat.core.agent;
 
 
 import com.alibaba.fastjson.JSONObject;
 import com.google.common.collect.Lists;
-import com.tencent.supersonic.chat.agent.tool.AgentToolType;
 import com.tencent.supersonic.common.pojo.RecordInfo;
-import java.util.Objects;
 import lombok.Data;
 import org.springframework.util.CollectionUtils;
+
+import java.util.Collection;
+import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
 import java.util.stream.Collectors;
 
 @Data
@@ -20,10 +23,13 @@ public class Agent extends RecordInfo {
     private String name;
     private String description;
 
-    //0 offline, 1 online
+    /**
+     * 0 offline, 1 online
+     */
     private Integer status;
     private List<String> examples;
     private String agentConfig;
+
     public List<String> getTools(AgentToolType type) {
         Map map = JSONObject.parseObject(agentConfig, Map.class);
         if (CollectionUtils.isEmpty(map) || map.get("tools") == null) {
@@ -46,4 +52,27 @@ public class Agent extends RecordInfo {
         return enableSearch != null && enableSearch == 1;
     }
 
+    public static boolean containsAllModel(Set<Long> detectViewIds) {
+        return !CollectionUtils.isEmpty(detectViewIds) && detectViewIds.contains(-1L);
+    }
+
+    public List<NL2SQLTool> getParserTools(AgentToolType agentToolType) {
+        List<String> tools = this.getTools(agentToolType);
+        if (CollectionUtils.isEmpty(tools)) {
+            return Lists.newArrayList();
+        }
+        return tools.stream().map(tool -> JSONObject.parseObject(tool, NL2SQLTool.class))
+                .collect(Collectors.toList());
+    }
+
+    public Set<Long> getViewIds(AgentToolType agentToolType) {
+        List<NL2SQLTool> commonAgentTools = getParserTools(agentToolType);
+        if (CollectionUtils.isEmpty(commonAgentTools)) {
+            return new HashSet<>();
+        }
+        return commonAgentTools.stream().map(NL2SQLTool::getViewIds)
+                .filter(modelIds -> !CollectionUtils.isEmpty(modelIds))
+                .flatMap(Collection::stream)
+                .collect(Collectors.toSet());
+    }
 }

chat/core/src/main/java/com/tencent/supersonic/chat/core/agent/AgentConfig.java

@@ -1,7 +1,6 @@
-package com.tencent.supersonic.chat.agent;
+package com.tencent.supersonic.chat.core.agent;
 
 import com.google.common.collect.Lists;
-import com.tencent.supersonic.chat.agent.tool.AgentTool;
 import lombok.AllArgsConstructor;
 import lombok.Data;
 import lombok.NoArgsConstructor;

chat/core/src/main/java/com/tencent/supersonic/chat/core/agent/AgentTool.java

@@ -1,4 +1,4 @@
-package com.tencent.supersonic.chat.agent.tool;
+package com.tencent.supersonic.chat.core.agent;
 
 import lombok.AllArgsConstructor;
 import lombok.Data;

chat/core/src/main/java/com/tencent/supersonic/chat/core/agent/AgentToolType.java

@@ -0,0 +1,25 @@
+package com.tencent.supersonic.chat.core.agent;
+
+import java.util.HashMap;
+import java.util.Map;
+
+public enum AgentToolType {
+    NL2SQL_RULE("基于规则Text-to-SQL"),
+    NL2SQL_LLM("基于大模型Text-to-SQL"),
+    PLUGIN("第三方插件");
+
+    private String title;
+
+    AgentToolType(String title) {
+        this.title = title;
+    }
+
+    public static Map<AgentToolType, String> getToolTypes() {
+        Map<AgentToolType, String> map = new HashMap<>();
+        map.put(NL2SQL_RULE, NL2SQL_RULE.title);
+        map.put(NL2SQL_LLM, NL2SQL_LLM.title);
+        map.put(PLUGIN, PLUGIN.title);
+        return map;
+    }
+
+}

chat/core/src/main/java/com/tencent/supersonic/chat/core/agent/LLMParserTool.java

@@ -0,0 +1,12 @@
+package com.tencent.supersonic.chat.core.agent;
+
+import lombok.Data;
+
+import java.util.List;
+
+@Data
+public class LLMParserTool extends NL2SQLTool {
+
+    private List<String> exampleQuestions;
+
+}

chat/core/src/main/java/com/tencent/supersonic/chat/core/agent/NL2SQLTool.java

@@ -1,16 +1,17 @@
-package com.tencent.supersonic.chat.agent.tool;
+package com.tencent.supersonic.chat.core.agent;
 
 
-import java.util.List;
 import lombok.AllArgsConstructor;
 import lombok.Data;
 import lombok.NoArgsConstructor;
 
+import java.util.List;
+
 @Data
 @NoArgsConstructor
 @AllArgsConstructor
-public class CommonAgentTool extends AgentTool {
+public class NL2SQLTool extends AgentTool {
 
-    protected List<Long> modelIds;
+    protected List<Long> viewIds;
 
 }

chat/core/src/main/java/com/tencent/supersonic/chat/core/agent/PluginTool.java

@@ -1,4 +1,4 @@
-package com.tencent.supersonic.chat.agent.tool;
+package com.tencent.supersonic.chat.core.agent;
 
 
 import lombok.Data;

chat/core/src/main/java/com/tencent/supersonic/chat/core/agent/RuleParserTool.java

@@ -1,4 +1,4 @@
-package com.tencent.supersonic.chat.agent.tool;
+package com.tencent.supersonic.chat.core.agent;
 
 
 import lombok.Data;
@@ -7,7 +7,7 @@ import org.apache.commons.collections.CollectionUtils;
 import java.util.List;
 
 @Data
-public class RuleQueryTool extends CommonAgentTool {
+public class RuleParserTool extends NL2SQLTool {
 
 
     private List<String> queryModes;
@@ -15,7 +15,7 @@ public class RuleQueryTool extends CommonAgentTool {
     private List<String> queryTypes;
 
     public boolean isContainsAllModel() {
-        return CollectionUtils.isNotEmpty(modelIds) && modelIds.contains(-1L);
+        return CollectionUtils.isNotEmpty(viewIds) && viewIds.contains(-1L);
     }
 
 }

chat/core/src/main/java/com/tencent/supersonic/chat/core/config/AggregatorConfig.java

@@ -1,4 +1,4 @@
-package com.tencent.supersonic.chat.config;
+package com.tencent.supersonic.chat.core.config;
 
 import lombok.Data;
 import org.springframework.beans.factory.annotation.Value;

chat/core/src/main/java/com/tencent/supersonic/chat/core/config/ChatLocalFileConfig.java

@@ -1,7 +1,9 @@
-package com.tencent.supersonic.knowledge.dictionary;
+package com.tencent.supersonic.chat.core.config;
+
+import com.tencent.supersonic.headless.core.knowledge.helper.HanlpHelper;
 
-import com.tencent.supersonic.knowledge.utils.HanlpHelper;
 import java.io.FileNotFoundException;
+
 import lombok.Data;
 import lombok.extern.slf4j.Slf4j;
 import org.springframework.beans.factory.annotation.Value;
@@ -11,7 +13,7 @@ import org.springframework.context.annotation.Configuration;
 @Data
 @Configuration
 @Slf4j
-public class LocalFileConfig {
+public class ChatLocalFileConfig {
 
 
     @Value("${dict.directory.latest:/data/dictionary/custom}")

chat/core/src/main/java/com/tencent/supersonic/chat/core/config/DefaultMetric.java

@@ -1,4 +1,4 @@
-package com.tencent.supersonic.chat.config;
+package com.tencent.supersonic.chat.core.config;
 
 import lombok.AllArgsConstructor;
 import lombok.Data;

chat/core/src/main/java/com/tencent/supersonic/chat/core/config/DefaultMetricInfo.java

@@ -1,4 +1,4 @@
-package com.tencent.supersonic.chat.config;
+package com.tencent.supersonic.chat.core.config;
 
 import com.tencent.supersonic.common.pojo.Constants;
 import lombok.Data;

chat/core/src/main/java/com/tencent/supersonic/chat/core/config/DefaultSemanticConfig.java

@@ -1,4 +1,4 @@
-package com.tencent.supersonic.knowledge.semantic;
+package com.tencent.supersonic.chat.core.config;
 
 import lombok.Data;
 import org.springframework.beans.factory.annotation.Value;

chat/core/src/main/java/com/tencent/supersonic/chat/core/config/Dim4Dict.java

@@ -1,4 +1,4 @@
-package com.tencent.supersonic.chat.config;
+package com.tencent.supersonic.chat.core.config;
 
 import java.util.List;
 import lombok.AllArgsConstructor;

chat/core/src/main/java/com/tencent/supersonic/chat/core/config/EntityDetailData.java

@@ -1,4 +1,4 @@
-package com.tencent.supersonic.chat.config;
+package com.tencent.supersonic.chat.core.config;
 
 import java.util.List;
 import lombok.Data;

chat/core/src/main/java/com/tencent/supersonic/chat/core/config/EntityInternalDetail.java

@@ -0,0 +1,14 @@
+package com.tencent.supersonic.chat.core.config;
+
+
+import com.tencent.supersonic.headless.api.pojo.response.DimSchemaResp;
+import com.tencent.supersonic.headless.api.pojo.response.MetricSchemaResp;
+import java.util.List;
+import lombok.Data;
+
+@Data
+public class EntityInternalDetail {
+
+    List<DimSchemaResp> dimensionList;
+    List<MetricSchemaResp> metricList;
+}

chat/core/src/main/java/com/tencent/supersonic/chat/core/config/LLMParserConfig.java

@@ -1,4 +1,4 @@
-package com.tencent.supersonic.chat.config;
+package com.tencent.supersonic.chat.core.config;
 
 
 import lombok.Data;
@@ -16,10 +16,12 @@ public class LLMParserConfig {
     @Value("${query2sql.path:/query2sql}")
     private String queryToSqlPath;
 
-    @Value("${dimension.topn:5}")
+    @Value("${dimension.topn:10}")
     private Integer dimensionTopN;
 
-    @Value("${metric.topn:5}")
+    @Value("${metric.topn:10}")
     private Integer metricTopN;
 
+    @Value("${all.model:false}")
+    private Boolean allModel;
 }

chat/core/src/main/java/com/tencent/supersonic/chat/core/config/OptimizationConfig.java

@@ -1,5 +1,6 @@
-package com.tencent.supersonic.chat.config;
+package com.tencent.supersonic.chat.core.config;
 
+import com.tencent.supersonic.chat.core.query.llm.s2sql.LLMReq.SqlGenerationMode;
 import com.tencent.supersonic.common.service.SysParameterService;
 import lombok.Data;
 import lombok.extern.slf4j.Slf4j;
@@ -51,16 +52,19 @@ public class OptimizationConfig {
     @Value("${embedding.mapper.round.number:10}")
     private int embeddingMapperRoundNumber;
 
-    @Value("${embedding.mapper.distance.threshold:0.58}")
+    @Value("${embedding.mapper.distance.threshold:0.01}")
     private Double embeddingMapperDistanceThreshold;
 
     @Value("${s2SQL.linking.value.switch:true}")
     private boolean useLinkingValueSwitch;
 
+    @Value("${s2SQL.generation:TWO_PASS_AUTO_COT}")
+    private SqlGenerationMode sqlGenerationMode;
+
     @Value("${s2SQL.use.switch:true}")
     private boolean useS2SqlSwitch;
 
-    @Value("${text2sql.example.num:10}")
+    @Value("${text2sql.example.num:15}")
     private int text2sqlExampleNum;
 
     @Value("${text2sql.fewShots.num:10}")
@@ -69,8 +73,8 @@ public class OptimizationConfig {
     @Value("${text2sql.self.consistency.num:5}")
     private int text2sqlSelfConsistencyNum;
 
-    @Value("${text2sql.collection.name:text2dsl_agent_collection}")
-    private String text2sqlCollectionName;
+    @Value("${parse.show.count:3}")
+    private Integer parseShowCount;
 
     @Autowired
     private SysParameterService sysParameterService;
@@ -139,6 +143,14 @@ public class OptimizationConfig {
         return convertValue("s2SQL.linking.value.switch", Boolean.class, useLinkingValueSwitch);
     }
 
+    public SqlGenerationMode getSqlGenerationMode() {
+        return convertValue("s2SQL.generation", SqlGenerationMode.class, sqlGenerationMode);
+    }
+
+    public Integer getParseShowCount() {
+        return convertValue("parse.show.count", Integer.class, parseShowCount);
+    }
+
     public <T> T convertValue(String paramName, Class<T> targetType, T defaultValue) {
         try {
             String value = sysParameterService.getSysParameter().getParameterByName(paramName);
@@ -151,6 +163,8 @@ public class OptimizationConfig {
                 return targetType.cast(Integer.parseInt(value));
             } else if (targetType == Boolean.class) {
                 return targetType.cast(Boolean.parseBoolean(value));
+            } else if (targetType == SqlGenerationMode.class) {
+                return targetType.cast(SqlGenerationMode.valueOf(value));
             }
         } catch (Exception e) {
             log.error("convertValue", e);

chat/core/src/main/java/com/tencent/supersonic/chat/core/corrector/BaseSemanticCorrector.java

@@ -1,19 +1,19 @@
-package com.tencent.supersonic.chat.corrector;
+package com.tencent.supersonic.chat.core.corrector;
 
-import com.tencent.supersonic.chat.api.component.SemanticCorrector;
-import com.tencent.supersonic.chat.api.pojo.SchemaElement;
+import com.tencent.supersonic.common.util.ContextUtils;
+import com.tencent.supersonic.headless.api.pojo.SchemaElement;
 import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
 import com.tencent.supersonic.chat.api.pojo.SemanticSchema;
-import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
+import com.tencent.supersonic.chat.core.pojo.QueryContext;
 import com.tencent.supersonic.common.pojo.enums.AggregateTypeEnum;
 import com.tencent.supersonic.common.pojo.enums.TimeDimensionEnum;
-import com.tencent.supersonic.common.util.ContextUtils;
-import com.tencent.supersonic.common.util.jsqlparser.SqlParserAddHelper;
-import com.tencent.supersonic.common.util.jsqlparser.SqlParserSelectFunctionHelper;
-import com.tencent.supersonic.common.util.jsqlparser.SqlParserSelectHelper;
-import com.tencent.supersonic.knowledge.service.SchemaService;
+import com.tencent.supersonic.common.util.jsqlparser.SqlAddHelper;
+import com.tencent.supersonic.common.util.jsqlparser.SqlSelectFunctionHelper;
+import com.tencent.supersonic.common.util.jsqlparser.SqlSelectHelper;
 import lombok.extern.slf4j.Slf4j;
 import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.springframework.core.env.Environment;
 import org.springframework.util.CollectionUtils;
 
 import java.util.ArrayList;
@@ -31,24 +31,23 @@ import java.util.stream.Collectors;
 @Slf4j
 public abstract class BaseSemanticCorrector implements SemanticCorrector {
 
-    public void correct(QueryReq queryReq, SemanticParseInfo semanticParseInfo) {
+    public void correct(QueryContext queryContext, SemanticParseInfo semanticParseInfo) {
         try {
             if (StringUtils.isBlank(semanticParseInfo.getSqlInfo().getCorrectS2SQL())) {
                 return;
             }
-            doCorrect(queryReq, semanticParseInfo);
+            doCorrect(queryContext, semanticParseInfo);
             log.info("sqlCorrection:{} sql:{}", this.getClass().getSimpleName(), semanticParseInfo.getSqlInfo());
         } catch (Exception e) {
             log.error(String.format("correct error,sqlInfo:%s", semanticParseInfo.getSqlInfo()), e);
         }
     }
 
+    public abstract void doCorrect(QueryContext queryContext, SemanticParseInfo semanticParseInfo);
 
-    public abstract void doCorrect(QueryReq queryReq, SemanticParseInfo semanticParseInfo);
+    protected Map<String, String> getFieldNameMap(QueryContext queryContext, Long viewId) {
 
-    protected Map<String, String> getFieldNameMap(Set<Long> modelIds) {
-
-        SemanticSchema semanticSchema = ContextUtils.getBean(SchemaService.class).getSemanticSchema();
+        SemanticSchema semanticSchema = queryContext.getSemanticSchema();
 
         List<SchemaElement> dbAllFields = new ArrayList<>();
         dbAllFields.addAll(semanticSchema.getMetrics());
@@ -56,7 +55,7 @@ public abstract class BaseSemanticCorrector implements SemanticCorrector {
 
         // support fieldName and field alias
         Map<String, String> result = dbAllFields.stream()
-                .filter(entry -> modelIds.contains(entry.getModel()))
+                .filter(entry -> viewId.equals(entry.getView()))
                 .flatMap(schemaElement -> {
                     Set<String> elements = new HashSet<>();
                     elements.add(schemaElement.getName());
@@ -78,14 +77,20 @@ public abstract class BaseSemanticCorrector implements SemanticCorrector {
     }
 
     protected void addFieldsToSelect(SemanticParseInfo semanticParseInfo, String correctS2SQL) {
-        Set<String> selectFields = new HashSet<>(SqlParserSelectHelper.getSelectFields(correctS2SQL));
-        Set<String> needAddFields = new HashSet<>(SqlParserSelectHelper.getGroupByFields(correctS2SQL));
-        needAddFields.addAll(SqlParserSelectHelper.getOrderByFields(correctS2SQL));
+        Set<String> selectFields = new HashSet<>(SqlSelectHelper.getSelectFields(correctS2SQL));
+        Set<String> needAddFields = new HashSet<>(SqlSelectHelper.getGroupByFields(correctS2SQL));
+
+        //decide whether add order by expression field to select
+        Environment environment = ContextUtils.getBean(Environment.class);
+        String correctorAdditionalInfo = environment.getProperty("corrector.additional.information");
+        if (StringUtils.isNotBlank(correctorAdditionalInfo) && Boolean.parseBoolean(correctorAdditionalInfo)) {
+            needAddFields.addAll(SqlSelectHelper.getOrderByFields(correctS2SQL));
+        }
 
         // If there is no aggregate function in the S2SQL statement and
         // there is a data field in 'WHERE' statement, add the field to the 'SELECT' statement.
-        if (!SqlParserSelectFunctionHelper.hasAggregateFunction(correctS2SQL)) {
-            List<String> whereFields = SqlParserSelectHelper.getWhereFields(correctS2SQL);
+        if (!SqlSelectFunctionHelper.hasAggregateFunction(correctS2SQL)) {
+            List<String> whereFields = SqlSelectHelper.getWhereFields(correctS2SQL);
             List<String> timeChNameList = TimeDimensionEnum.getChNameList();
             Set<String> timeFields = whereFields.stream().filter(field -> timeChNameList.contains(field))
                     .collect(Collectors.toSet());
@@ -97,16 +102,15 @@ public abstract class BaseSemanticCorrector implements SemanticCorrector {
         }
 
         needAddFields.removeAll(selectFields);
-        String replaceFields = SqlParserAddHelper.addFieldsToSelect(correctS2SQL, new ArrayList<>(needAddFields));
+        String replaceFields = SqlAddHelper.addFieldsToSelect(correctS2SQL, new ArrayList<>(needAddFields));
         semanticParseInfo.getSqlInfo().setCorrectS2SQL(replaceFields);
     }
 
-    protected void addAggregateToMetric(SemanticParseInfo semanticParseInfo) {
+    protected void addAggregateToMetric(QueryContext queryContext, SemanticParseInfo semanticParseInfo) {
         //add aggregate to all metric
         String correctS2SQL = semanticParseInfo.getSqlInfo().getCorrectS2SQL();
-        Set<Long> modelIds = semanticParseInfo.getModel().getModelIds();
-
-        List<SchemaElement> metrics = getMetricElements(modelIds);
+        Long viewId = semanticParseInfo.getView().getView();
+        List<SchemaElement> metrics = getMetricElements(queryContext, viewId);
 
         Map<String, String> metricToAggregate = metrics.stream()
                 .map(schemaElement -> {
@@ -114,18 +118,26 @@ public abstract class BaseSemanticCorrector implements SemanticCorrector {
                         schemaElement.setDefaultAgg(AggregateTypeEnum.SUM.name());
                     }
                     return schemaElement;
-                }).collect(Collectors.toMap(a -> a.getName(), a -> a.getDefaultAgg(), (k1, k2) -> k1));
+                }).flatMap(schemaElement -> {
+                    Set<String> elements = new HashSet<>();
+                    elements.add(schemaElement.getName());
+                    if (!CollectionUtils.isEmpty(schemaElement.getAlias())) {
+                        elements.addAll(schemaElement.getAlias());
+                    }
+                    return elements.stream().map(element -> Pair.of(element, schemaElement.getDefaultAgg())
+                    );
+                }).collect(Collectors.toMap(Pair::getLeft, Pair::getRight, (k1, k2) -> k1));
 
         if (CollectionUtils.isEmpty(metricToAggregate)) {
             return;
         }
-        String aggregateSql = SqlParserAddHelper.addAggregateToField(correctS2SQL, metricToAggregate);
+        String aggregateSql = SqlAddHelper.addAggregateToField(correctS2SQL, metricToAggregate);
         semanticParseInfo.getSqlInfo().setCorrectS2SQL(aggregateSql);
     }
 
-    protected List<SchemaElement> getMetricElements(Set<Long> modelIds) {
-        SemanticSchema semanticSchema = ContextUtils.getBean(SchemaService.class).getSemanticSchema();
-        return semanticSchema.getMetrics(modelIds);
+    protected List<SchemaElement> getMetricElements(QueryContext queryContext, Long viewId) {
+        SemanticSchema semanticSchema = queryContext.getSemanticSchema();
+        return semanticSchema.getMetrics(viewId);
     }
 
 }

chat/core/src/main/java/com/tencent/supersonic/chat/core/corrector/GroupByCorrector.java

@@ -0,0 +1,132 @@
+package com.tencent.supersonic.chat.core.corrector;
+
+import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
+import com.tencent.supersonic.chat.api.pojo.SemanticSchema;
+import com.tencent.supersonic.chat.api.pojo.response.SqlInfo;
+import com.tencent.supersonic.chat.core.pojo.QueryContext;
+import com.tencent.supersonic.common.pojo.enums.TimeDimensionEnum;
+import com.tencent.supersonic.common.util.ContextUtils;
+import com.tencent.supersonic.common.util.jsqlparser.SqlAddHelper;
+import com.tencent.supersonic.common.util.jsqlparser.SqlSelectHelper;
+import com.tencent.supersonic.headless.api.pojo.Dim;
+import com.tencent.supersonic.headless.api.pojo.response.ModelResp;
+import com.tencent.supersonic.headless.api.pojo.response.ViewResp;
+import com.tencent.supersonic.headless.server.pojo.MetaFilter;
+import com.tencent.supersonic.headless.server.service.ModelService;
+import com.tencent.supersonic.headless.server.service.ViewService;
+import lombok.extern.slf4j.Slf4j;
+import org.springframework.util.CollectionUtils;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Perform SQL corrections on the "Group by" section in S2SQL.
+ */
+@Slf4j
+public class GroupByCorrector extends BaseSemanticCorrector {
+
+    @Override
+    public void doCorrect(QueryContext queryContext, SemanticParseInfo semanticParseInfo) {
+        Boolean needAddGroupBy = needAddGroupBy(queryContext, semanticParseInfo);
+        if (!needAddGroupBy) {
+            return;
+        }
+        addGroupByFields(queryContext, semanticParseInfo);
+    }
+
+    private Boolean needAddGroupBy(QueryContext queryContext, SemanticParseInfo semanticParseInfo) {
+        Long viewId = semanticParseInfo.getViewId();
+        ViewService viewService = ContextUtils.getBean(ViewService.class);
+        ModelService modelService = ContextUtils.getBean(ModelService.class);
+        ViewResp viewResp = viewService.getView(viewId);
+        List<Long> modelIds = viewResp.getViewDetail().getViewModelConfigs().stream().map(config -> config.getId())
+                .collect(Collectors.toList());
+        MetaFilter metaFilter = new MetaFilter();
+        metaFilter.setIds(modelIds);
+        List<ModelResp> modelRespList = modelService.getModelList(metaFilter);
+        for (ModelResp modelResp : modelRespList) {
+            List<Dim> dimList = modelResp.getModelDetail().getDimensions();
+            for (Dim dim : dimList) {
+                if (Objects.nonNull(dim.getTypeParams()) && dim.getTypeParams().getTimeGranularity().equals("none")) {
+                    return false;
+                }
+            }
+        }
+        //add dimension group by
+        SqlInfo sqlInfo = semanticParseInfo.getSqlInfo();
+        String correctS2SQL = sqlInfo.getCorrectS2SQL();
+        SemanticSchema semanticSchema = queryContext.getSemanticSchema();
+        // check has distinct
+        if (SqlSelectHelper.hasDistinct(correctS2SQL)) {
+            log.info("not add group by ,exist distinct in correctS2SQL:{}", correctS2SQL);
+            return false;
+        }
+        //add alias field name
+        Set<String> dimensions = getDimensions(viewId, semanticSchema);
+        List<String> selectFields = SqlSelectHelper.getSelectFields(correctS2SQL);
+        if (CollectionUtils.isEmpty(selectFields) || CollectionUtils.isEmpty(dimensions)) {
+            return false;
+        }
+        // if only date in select not add group by.
+        if (selectFields.size() == 1 && selectFields.contains(TimeDimensionEnum.DAY.getChName())) {
+            return false;
+        }
+        if (SqlSelectHelper.hasGroupBy(correctS2SQL)) {
+            log.info("not add group by ,exist group by in correctS2SQL:{}", correctS2SQL);
+            return false;
+        }
+        return true;
+    }
+
+    private Set<String> getDimensions(Long viewId, SemanticSchema semanticSchema) {
+        Set<String> dimensions = semanticSchema.getDimensions(viewId).stream()
+                .flatMap(
+                        schemaElement -> {
+                            Set<String> elements = new HashSet<>();
+                            elements.add(schemaElement.getName());
+                            if (!CollectionUtils.isEmpty(schemaElement.getAlias())) {
+                                elements.addAll(schemaElement.getAlias());
+                            }
+                            return elements.stream();
+                        }
+                ).collect(Collectors.toSet());
+        dimensions.add(TimeDimensionEnum.DAY.getChName());
+        return dimensions;
+    }
+
+    private void addGroupByFields(QueryContext queryContext, SemanticParseInfo semanticParseInfo) {
+        Long viewId = semanticParseInfo.getViewId();
+        //add dimension group by
+        SqlInfo sqlInfo = semanticParseInfo.getSqlInfo();
+        String correctS2SQL = sqlInfo.getCorrectS2SQL();
+        SemanticSchema semanticSchema = queryContext.getSemanticSchema();
+        //add alias field name
+        Set<String> dimensions = getDimensions(viewId, semanticSchema);
+        List<String> selectFields = SqlSelectHelper.getSelectFields(correctS2SQL);
+        List<String> aggregateFields = SqlSelectHelper.getAggregateFields(correctS2SQL);
+        Set<String> groupByFields = selectFields.stream()
+                .filter(field -> dimensions.contains(field))
+                .filter(field -> {
+                    if (!CollectionUtils.isEmpty(aggregateFields) && aggregateFields.contains(field)) {
+                        return false;
+                    }
+                    return true;
+                })
+                .collect(Collectors.toSet());
+        semanticParseInfo.getSqlInfo().setCorrectS2SQL(SqlAddHelper.addGroupBy(correctS2SQL, groupByFields));
+        addAggregate(queryContext, semanticParseInfo);
+    }
+
+    private void addAggregate(QueryContext queryContext, SemanticParseInfo semanticParseInfo) {
+        List<String> sqlGroupByFields = SqlSelectHelper.getGroupByFields(
+                semanticParseInfo.getSqlInfo().getCorrectS2SQL());
+        if (CollectionUtils.isEmpty(sqlGroupByFields)) {
+            return;
+        }
+        addAggregateToMetric(queryContext, semanticParseInfo);
+    }
+}

chat/core/src/main/java/com/tencent/supersonic/chat/core/corrector/HavingCorrector.java

@@ -0,0 +1,69 @@
+package com.tencent.supersonic.chat.core.corrector;
+
+import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
+import com.tencent.supersonic.chat.api.pojo.SemanticSchema;
+import com.tencent.supersonic.chat.core.pojo.QueryContext;
+import com.tencent.supersonic.common.util.ContextUtils;
+import com.tencent.supersonic.common.util.jsqlparser.SqlAddHelper;
+import com.tencent.supersonic.common.util.jsqlparser.SqlSelectFunctionHelper;
+import com.tencent.supersonic.common.util.jsqlparser.SqlSelectHelper;
+import lombok.extern.slf4j.Slf4j;
+import net.sf.jsqlparser.expression.Expression;
+import org.apache.commons.lang3.StringUtils;
+import org.springframework.core.env.Environment;
+import org.springframework.util.CollectionUtils;
+
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Perform SQL corrections on the "Having" section in S2SQL.
+ */
+@Slf4j
+public class HavingCorrector extends BaseSemanticCorrector {
+
+    @Override
+    public void doCorrect(QueryContext queryContext, SemanticParseInfo semanticParseInfo) {
+
+        //add aggregate to all metric
+        addHaving(queryContext, semanticParseInfo);
+
+        //decide whether add having expression field to select
+        Environment environment = ContextUtils.getBean(Environment.class);
+        String correctorAdditionalInfo = environment.getProperty("corrector.additional.information");
+        if (StringUtils.isNotBlank(correctorAdditionalInfo) && Boolean.parseBoolean(correctorAdditionalInfo)) {
+            addHavingToSelect(semanticParseInfo);
+        }
+
+    }
+
+    private void addHaving(QueryContext queryContext, SemanticParseInfo semanticParseInfo) {
+        Long viewId = semanticParseInfo.getView().getView();
+
+        SemanticSchema semanticSchema = queryContext.getSemanticSchema();
+
+        Set<String> metrics = semanticSchema.getMetrics(viewId).stream()
+                .map(schemaElement -> schemaElement.getName()).collect(Collectors.toSet());
+
+        if (CollectionUtils.isEmpty(metrics)) {
+            return;
+        }
+        String havingSql = SqlAddHelper.addHaving(semanticParseInfo.getSqlInfo().getCorrectS2SQL(), metrics);
+        semanticParseInfo.getSqlInfo().setCorrectS2SQL(havingSql);
+    }
+
+    private void addHavingToSelect(SemanticParseInfo semanticParseInfo) {
+        String correctS2SQL = semanticParseInfo.getSqlInfo().getCorrectS2SQL();
+        if (!SqlSelectFunctionHelper.hasAggregateFunction(correctS2SQL)) {
+            return;
+        }
+        List<Expression> havingExpressionList = SqlSelectHelper.getHavingExpression(correctS2SQL);
+        if (!CollectionUtils.isEmpty(havingExpressionList)) {
+            String replaceSql = SqlAddHelper.addFunctionToSelect(correctS2SQL, havingExpressionList);
+            semanticParseInfo.getSqlInfo().setCorrectS2SQL(replaceSql);
+        }
+        return;
+    }
+
+}

chat/core/src/main/java/com/tencent/supersonic/chat/core/corrector/SchemaCorrector.java

@@ -1,30 +1,31 @@
-package com.tencent.supersonic.chat.corrector;
+package com.tencent.supersonic.chat.core.corrector;
 
 import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
-import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
 import com.tencent.supersonic.chat.api.pojo.response.SqlInfo;
-import com.tencent.supersonic.chat.parser.llm.s2sql.ParseResult;
-import com.tencent.supersonic.chat.query.llm.s2sql.LLMReq.ElementValue;
+import com.tencent.supersonic.chat.core.parser.sql.llm.ParseResult;
+import com.tencent.supersonic.chat.core.pojo.QueryContext;
+import com.tencent.supersonic.chat.core.query.llm.s2sql.LLMReq.ElementValue;
 import com.tencent.supersonic.common.pojo.Constants;
 import com.tencent.supersonic.common.util.JsonUtil;
 import com.tencent.supersonic.common.util.jsqlparser.AggregateEnum;
-import com.tencent.supersonic.common.util.jsqlparser.SqlParserReplaceHelper;
+import com.tencent.supersonic.common.util.jsqlparser.SqlReplaceHelper;
+import lombok.extern.slf4j.Slf4j;
+import org.springframework.util.CollectionUtils;
+
 import java.util.List;
 import java.util.Map;
 import java.util.Objects;
 import java.util.Set;
 import java.util.stream.Collectors;
-import lombok.extern.slf4j.Slf4j;
-import org.springframework.util.CollectionUtils;
 
 /**
- * Perform schema corrections on the Schema information in S2QL.
+ * Perform schema corrections on the Schema information in S2SQL.
  */
 @Slf4j
 public class SchemaCorrector extends BaseSemanticCorrector {
 
     @Override
-    public void doCorrect(QueryReq queryReq, SemanticParseInfo semanticParseInfo) {
+    public void doCorrect(QueryContext queryContext, SemanticParseInfo semanticParseInfo) {
 
         correctAggFunction(semanticParseInfo);
 
@@ -34,26 +35,26 @@ public class SchemaCorrector extends BaseSemanticCorrector {
 
         updateFieldValueByLinkingValue(semanticParseInfo);
 
-        correctFieldName(semanticParseInfo);
+        correctFieldName(queryContext, semanticParseInfo);
     }
 
     private void correctAggFunction(SemanticParseInfo semanticParseInfo) {
         Map<String, String> aggregateEnum = AggregateEnum.getAggregateEnum();
         SqlInfo sqlInfo = semanticParseInfo.getSqlInfo();
-        String sql = SqlParserReplaceHelper.replaceFunction(sqlInfo.getCorrectS2SQL(), aggregateEnum);
+        String sql = SqlReplaceHelper.replaceFunction(sqlInfo.getCorrectS2SQL(), aggregateEnum);
         sqlInfo.setCorrectS2SQL(sql);
     }
 
     private void replaceAlias(SemanticParseInfo semanticParseInfo) {
         SqlInfo sqlInfo = semanticParseInfo.getSqlInfo();
-        String replaceAlias = SqlParserReplaceHelper.replaceAlias(sqlInfo.getCorrectS2SQL());
+        String replaceAlias = SqlReplaceHelper.replaceAlias(sqlInfo.getCorrectS2SQL());
         sqlInfo.setCorrectS2SQL(replaceAlias);
     }
 
-    private void correctFieldName(SemanticParseInfo semanticParseInfo) {
-        Map<String, String> fieldNameMap = getFieldNameMap(semanticParseInfo.getModel().getModelIds());
+    private void correctFieldName(QueryContext queryContext, SemanticParseInfo semanticParseInfo) {
+        Map<String, String> fieldNameMap = getFieldNameMap(queryContext, semanticParseInfo.getViewId());
         SqlInfo sqlInfo = semanticParseInfo.getSqlInfo();
-        String sql = SqlParserReplaceHelper.replaceFields(sqlInfo.getCorrectS2SQL(), fieldNameMap);
+        String sql = SqlReplaceHelper.replaceFields(sqlInfo.getCorrectS2SQL(), fieldNameMap);
         sqlInfo.setCorrectS2SQL(sql);
     }
 
@@ -69,7 +70,7 @@ public class SchemaCorrector extends BaseSemanticCorrector {
 
         SqlInfo sqlInfo = semanticParseInfo.getSqlInfo();
 
-        String sql = SqlParserReplaceHelper.replaceFieldNameByValue(sqlInfo.getCorrectS2SQL(), fieldValueToFieldNames);
+        String sql = SqlReplaceHelper.replaceFieldNameByValue(sqlInfo.getCorrectS2SQL(), fieldValueToFieldNames);
         sqlInfo.setCorrectS2SQL(sql);
     }
 
@@ -86,7 +87,6 @@ public class SchemaCorrector extends BaseSemanticCorrector {
         return parseResult.getLinkingValues();
     }
 
-
     private void updateFieldValueByLinkingValue(SemanticParseInfo semanticParseInfo) {
         List<ElementValue> linking = getLinkingValues(semanticParseInfo);
         if (CollectionUtils.isEmpty(linking)) {
@@ -102,7 +102,7 @@ public class SchemaCorrector extends BaseSemanticCorrector {
                         )));
 
         SqlInfo sqlInfo = semanticParseInfo.getSqlInfo();
-        String sql = SqlParserReplaceHelper.replaceValue(sqlInfo.getCorrectS2SQL(), filedNameToValueMap, false);
+        String sql = SqlReplaceHelper.replaceValue(sqlInfo.getCorrectS2SQL(), filedNameToValueMap, false);
         sqlInfo.setCorrectS2SQL(sql);
     }
 }

chat/core/src/main/java/com/tencent/supersonic/chat/core/corrector/SelectCorrector.java

@@ -1,8 +1,8 @@
-package com.tencent.supersonic.chat.corrector;
+package com.tencent.supersonic.chat.core.corrector;
 
+import com.tencent.supersonic.chat.core.pojo.QueryContext;
 import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
-import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
-import com.tencent.supersonic.common.util.jsqlparser.SqlParserSelectHelper;
+import com.tencent.supersonic.common.util.jsqlparser.SqlSelectHelper;
 import java.util.List;
 import lombok.extern.slf4j.Slf4j;
 import org.springframework.util.CollectionUtils;
@@ -14,10 +14,10 @@ import org.springframework.util.CollectionUtils;
 public class SelectCorrector extends BaseSemanticCorrector {
 
     @Override
-    public void doCorrect(QueryReq queryReq, SemanticParseInfo semanticParseInfo) {
+    public void doCorrect(QueryContext queryContext, SemanticParseInfo semanticParseInfo) {
         String correctS2SQL = semanticParseInfo.getSqlInfo().getCorrectS2SQL();
-        List<String> aggregateFields = SqlParserSelectHelper.getAggregateFields(correctS2SQL);
-        List<String> selectFields = SqlParserSelectHelper.getSelectFields(correctS2SQL);
+        List<String> aggregateFields = SqlSelectHelper.getAggregateFields(correctS2SQL);
+        List<String> selectFields = SqlSelectHelper.getSelectFields(correctS2SQL);
         // If the number of aggregated fields is equal to the number of queried fields, do not add fields to select.
         if (!CollectionUtils.isEmpty(aggregateFields)
                 && !CollectionUtils.isEmpty(selectFields)

chat/core/src/main/java/com/tencent/supersonic/chat/core/corrector/SemanticCorrector.java

@@ -1,7 +1,7 @@
-package com.tencent.supersonic.chat.api.component;
+package com.tencent.supersonic.chat.core.corrector;
 
+import com.tencent.supersonic.chat.core.pojo.QueryContext;
 import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
-import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
 
 /**
  * A semantic corrector checks validity of extracted semantic information and
@@ -9,5 +9,5 @@ import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
  */
 public interface SemanticCorrector {
 
-    void correct(QueryReq queryReq, SemanticParseInfo semanticParseInfo);
+    void correct(QueryContext queryContext, SemanticParseInfo semanticParseInfo);
 }

chat/core/src/main/java/com/tencent/supersonic/chat/core/corrector/TimeCorrector.java

@@ -0,0 +1,57 @@
+package com.tencent.supersonic.chat.core.corrector;
+
+import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
+import com.tencent.supersonic.chat.core.pojo.QueryContext;
+import com.tencent.supersonic.common.util.jsqlparser.DateVisitor.DateBoundInfo;
+import com.tencent.supersonic.common.util.jsqlparser.SqlAddHelper;
+import com.tencent.supersonic.common.util.jsqlparser.SqlDateSelectHelper;
+import com.tencent.supersonic.common.util.jsqlparser.SqlReplaceHelper;
+import java.util.Objects;
+import lombok.extern.slf4j.Slf4j;
+import net.sf.jsqlparser.JSQLParserException;
+import net.sf.jsqlparser.parser.CCJSqlParserUtil;
+import org.apache.commons.lang3.StringUtils;
+
+/**
+ * Perform SQL corrections on the time in S2SQL.
+ */
+@Slf4j
+public class TimeCorrector extends BaseSemanticCorrector {
+
+    @Override
+    public void doCorrect(QueryContext queryContext, SemanticParseInfo semanticParseInfo) {
+
+        parserDateDiffFunction(semanticParseInfo);
+
+        addLowerBoundDate(semanticParseInfo);
+
+    }
+
+    private void addLowerBoundDate(SemanticParseInfo semanticParseInfo) {
+        String correctS2SQL = semanticParseInfo.getSqlInfo().getCorrectS2SQL();
+        DateBoundInfo dateBoundInfo = SqlDateSelectHelper.getDateBoundInfo(correctS2SQL);
+        if (Objects.isNull(dateBoundInfo)) {
+            return;
+        }
+        if (StringUtils.isBlank(dateBoundInfo.getLowerBound())
+                && StringUtils.isNotBlank(dateBoundInfo.getUpperBound())
+                && StringUtils.isNotBlank(dateBoundInfo.getUpperDate())) {
+            String upperDate = dateBoundInfo.getUpperDate();
+            try {
+                correctS2SQL = SqlAddHelper.addParenthesisToWhere(correctS2SQL);
+                String condExpr = dateBoundInfo.getColumName() + " >= '" + upperDate + "'";
+                correctS2SQL = SqlAddHelper.addWhere(correctS2SQL, CCJSqlParserUtil.parseCondExpression(condExpr));
+            } catch (JSQLParserException e) {
+                log.error("parseCondExpression", e);
+            }
+            semanticParseInfo.getSqlInfo().setCorrectS2SQL(correctS2SQL);
+        }
+    }
+
+    private void parserDateDiffFunction(SemanticParseInfo semanticParseInfo) {
+        String correctS2SQL = semanticParseInfo.getSqlInfo().getCorrectS2SQL();
+        correctS2SQL = SqlReplaceHelper.replaceFunction(correctS2SQL);
+        semanticParseInfo.getSqlInfo().setCorrectS2SQL(correctS2SQL);
+    }
+
+}

chat/core/src/main/java/com/tencent/supersonic/chat/core/corrector/WhereCorrector.java

@@ -1,25 +1,24 @@
-package com.tencent.supersonic.chat.corrector;
+package com.tencent.supersonic.chat.core.corrector;
 
-import com.tencent.supersonic.chat.api.pojo.SchemaElement;
-import com.tencent.supersonic.chat.api.pojo.SchemaValueMap;
+import com.tencent.supersonic.headless.api.pojo.SchemaElement;
+import com.tencent.supersonic.headless.api.pojo.SchemaValueMap;
 import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
 import com.tencent.supersonic.chat.api.pojo.SemanticSchema;
 import com.tencent.supersonic.chat.api.pojo.request.QueryFilters;
-import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
-import com.tencent.supersonic.chat.parser.llm.s2sql.S2SQLDateHelper;
+import com.tencent.supersonic.chat.core.utils.S2SqlDateHelper;
+import com.tencent.supersonic.chat.core.pojo.QueryContext;
 import com.tencent.supersonic.common.pojo.Constants;
 import com.tencent.supersonic.common.pojo.enums.TimeDimensionEnum;
-import com.tencent.supersonic.common.util.ContextUtils;
 import com.tencent.supersonic.common.util.StringUtil;
-import com.tencent.supersonic.common.util.jsqlparser.SqlParserAddHelper;
-import com.tencent.supersonic.common.util.jsqlparser.SqlParserReplaceHelper;
-import com.tencent.supersonic.common.util.jsqlparser.SqlParserSelectHelper;
-import com.tencent.supersonic.knowledge.service.SchemaService;
+import com.tencent.supersonic.common.util.jsqlparser.SqlAddHelper;
+import com.tencent.supersonic.common.util.jsqlparser.SqlReplaceHelper;
+import com.tencent.supersonic.common.util.jsqlparser.SqlSelectHelper;
 import lombok.extern.slf4j.Slf4j;
 import net.sf.jsqlparser.JSQLParserException;
 import net.sf.jsqlparser.expression.Expression;
 import net.sf.jsqlparser.parser.CCJSqlParserUtil;
 import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.tuple.Pair;
 import org.apache.logging.log4j.util.Strings;
 import org.springframework.util.CollectionUtils;
 
@@ -27,7 +26,6 @@ import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
 import java.util.Objects;
-import java.util.Set;
 import java.util.stream.Collectors;
 
 /**
@@ -37,19 +35,17 @@ import java.util.stream.Collectors;
 public class WhereCorrector extends BaseSemanticCorrector {
 
     @Override
-    public void doCorrect(QueryReq queryReq, SemanticParseInfo semanticParseInfo) {
+    public void doCorrect(QueryContext queryContext, SemanticParseInfo semanticParseInfo) {
 
-        addDateIfNotExist(semanticParseInfo);
+        addDateIfNotExist(queryContext, semanticParseInfo);
 
-        parserDateDiffFunction(semanticParseInfo);
+        addQueryFilter(queryContext, semanticParseInfo);
 
-        addQueryFilter(queryReq, semanticParseInfo);
-
-        updateFieldValueByTechName(semanticParseInfo);
+        updateFieldValueByTechName(queryContext, semanticParseInfo);
     }
 
-    private void addQueryFilter(QueryReq queryReq, SemanticParseInfo semanticParseInfo) {
-        String queryFilter = getQueryFilter(queryReq.getQueryFilters());
+    private void addQueryFilter(QueryContext queryContext, SemanticParseInfo semanticParseInfo) {
+        String queryFilter = getQueryFilter(queryContext.getQueryFilters());
 
         String correctS2SQL = semanticParseInfo.getSqlInfo().getCorrectS2SQL();
 
@@ -61,26 +57,29 @@ public class WhereCorrector extends BaseSemanticCorrector {
             } catch (JSQLParserException e) {
                 log.error("parseCondExpression", e);
             }
-            correctS2SQL = SqlParserAddHelper.addWhere(correctS2SQL, expression);
+            correctS2SQL = SqlAddHelper.addWhere(correctS2SQL, expression);
             semanticParseInfo.getSqlInfo().setCorrectS2SQL(correctS2SQL);
         }
     }
 
-    private void parserDateDiffFunction(SemanticParseInfo semanticParseInfo) {
+    private void addDateIfNotExist(QueryContext queryContext, SemanticParseInfo semanticParseInfo) {
         String correctS2SQL = semanticParseInfo.getSqlInfo().getCorrectS2SQL();
-        correctS2SQL = SqlParserReplaceHelper.replaceFunction(correctS2SQL);
-        semanticParseInfo.getSqlInfo().setCorrectS2SQL(correctS2SQL);
-    }
-
-    private void addDateIfNotExist(SemanticParseInfo semanticParseInfo) {
-        String correctS2SQL = semanticParseInfo.getSqlInfo().getCorrectS2SQL();
-        List<String> whereFields = SqlParserSelectHelper.getWhereFields(correctS2SQL);
+        List<String> whereFields = SqlSelectHelper.getWhereFields(correctS2SQL);
         if (CollectionUtils.isEmpty(whereFields) || !TimeDimensionEnum.containsZhTimeDimension(whereFields)) {
-            String currentDate = S2SQLDateHelper.getReferenceDate(semanticParseInfo.getModelId());
-            if (StringUtils.isNotBlank(currentDate)) {
-                correctS2SQL = SqlParserAddHelper.addParenthesisToWhere(correctS2SQL);
-                correctS2SQL = SqlParserAddHelper.addWhere(
-                        correctS2SQL, TimeDimensionEnum.DAY.getChName(), currentDate);
+            Pair<String, String> startEndDate = S2SqlDateHelper.getStartEndDate(queryContext,
+                    semanticParseInfo.getViewId(), semanticParseInfo.getQueryType());
+            if (StringUtils.isNotBlank(startEndDate.getLeft())
+                    && StringUtils.isNotBlank(startEndDate.getRight())) {
+                correctS2SQL = SqlAddHelper.addParenthesisToWhere(correctS2SQL);
+                String dateChName = TimeDimensionEnum.DAY.getChName();
+                String condExpr = String.format(" ( %s >= '%s'  and %s <= '%s' )", dateChName,
+                        startEndDate.getLeft(), dateChName, startEndDate.getRight());
+                try {
+                    Expression expression = CCJSqlParserUtil.parseCondExpression(condExpr);
+                    correctS2SQL = SqlAddHelper.addWhere(correctS2SQL, expression);
+                } catch (JSQLParserException e) {
+                    log.error("parseCondExpression:{}", e);
+                }
             }
         }
         semanticParseInfo.getSqlInfo().setCorrectS2SQL(correctS2SQL);
@@ -100,17 +99,17 @@ public class WhereCorrector extends BaseSemanticCorrector {
                 .collect(Collectors.joining(Constants.AND_UPPER));
     }
 
-    private void updateFieldValueByTechName(SemanticParseInfo semanticParseInfo) {
-        SemanticSchema semanticSchema = ContextUtils.getBean(SchemaService.class).getSemanticSchema();
-        Set<Long> modelIds = semanticParseInfo.getModel().getModelIds();
-        List<SchemaElement> dimensions = semanticSchema.getDimensions(modelIds);
+    private void updateFieldValueByTechName(QueryContext queryContext, SemanticParseInfo semanticParseInfo) {
+        SemanticSchema semanticSchema = queryContext.getSemanticSchema();
+        Long viewId = semanticParseInfo.getViewId();
+        List<SchemaElement> dimensions = semanticSchema.getDimensions(viewId);
 
         if (CollectionUtils.isEmpty(dimensions)) {
             return;
         }
 
         Map<String, Map<String, String>> aliasAndBizNameToTechName = getAliasAndBizNameToTechName(dimensions);
-        String correctS2SQL = SqlParserReplaceHelper.replaceValue(semanticParseInfo.getSqlInfo().getCorrectS2SQL(),
+        String correctS2SQL = SqlReplaceHelper.replaceValue(semanticParseInfo.getSqlInfo().getCorrectS2SQL(),
                 aliasAndBizNameToTechName);
         semanticParseInfo.getSqlInfo().setCorrectS2SQL(correctS2SQL);
     }

chat/core/src/main/java/com/tencent/supersonic/chat/core/mapper/BaseMapper.java

@@ -1,27 +1,23 @@
-package com.tencent.supersonic.chat.mapper;
+package com.tencent.supersonic.chat.core.mapper;
 
-import com.tencent.supersonic.chat.api.component.SchemaMapper;
-import com.tencent.supersonic.chat.api.pojo.ModelSchema;
-import com.tencent.supersonic.chat.api.pojo.QueryContext;
-import com.tencent.supersonic.chat.api.pojo.SchemaElement;
+import com.tencent.supersonic.headless.api.pojo.SchemaElement;
 import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
-import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
+import com.tencent.supersonic.headless.api.pojo.SchemaElementType;
 import com.tencent.supersonic.chat.api.pojo.SchemaMapInfo;
-import com.tencent.supersonic.chat.service.SemanticService;
-import com.tencent.supersonic.common.util.ContextUtils;
+import com.tencent.supersonic.chat.api.pojo.SemanticSchema;
+import com.tencent.supersonic.chat.api.pojo.ViewSchema;
+import com.tencent.supersonic.chat.core.pojo.QueryContext;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.springframework.beans.BeanUtils;
+
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Map;
 import java.util.Objects;
 import java.util.concurrent.atomic.AtomicBoolean;
 import java.util.stream.Collectors;
-import lombok.extern.slf4j.Slf4j;
-import org.apache.commons.lang3.StringUtils;
-import org.springframework.beans.BeanUtils;
 
-/**
- * base Mapper
- */
 @Slf4j
 public abstract class BaseMapper implements SchemaMapper {
 
@@ -30,7 +26,7 @@ public abstract class BaseMapper implements SchemaMapper {
 
         String simpleName = this.getClass().getSimpleName();
         long startTime = System.currentTimeMillis();
-        log.info("before {},mapInfo:{}", simpleName, queryContext.getMapInfo().getModelElementMatches());
+        log.debug("before {},mapInfo:{}", simpleName, queryContext.getMapInfo().getViewElementMatches());
 
         try {
             doMap(queryContext);
@@ -39,14 +35,13 @@ public abstract class BaseMapper implements SchemaMapper {
         }
 
         long cost = System.currentTimeMillis() - startTime;
-        log.info("after {},cost:{},mapInfo:{}", simpleName, cost, queryContext.getMapInfo().getModelElementMatches());
+        log.debug("after {},cost:{},mapInfo:{}", simpleName, cost, queryContext.getMapInfo().getViewElementMatches());
     }
 
     public abstract void doMap(QueryContext queryContext);
 
-
     public void addToSchemaMap(SchemaMapInfo schemaMap, Long modelId, SchemaElementMatch newElementMatch) {
-        Map<Long, List<SchemaElementMatch>> modelElementMatches = schemaMap.getModelElementMatches();
+        Map<Long, List<SchemaElementMatch>> modelElementMatches = schemaMap.getViewElementMatches();
         List<SchemaElementMatch> schemaElementMatches = modelElementMatches.putIfAbsent(modelId, new ArrayList<>());
         if (schemaElementMatches == null) {
             schemaElementMatches = modelElementMatches.get(modelId);
@@ -72,14 +67,14 @@ public abstract class BaseMapper implements SchemaMapper {
         }
     }
 
-    public SchemaElement getSchemaElement(Long modelId, SchemaElementType elementType, Long elementID) {
+    public SchemaElement getSchemaElement(Long viewId, SchemaElementType elementType, Long elementID,
+            SemanticSchema semanticSchema) {
         SchemaElement element = new SchemaElement();
-        SemanticService schemaService = ContextUtils.getBean(SemanticService.class);
-        ModelSchema modelSchema = schemaService.getModelSchema(modelId);
-        if (Objects.isNull(modelSchema)) {
+        ViewSchema viewSchema = semanticSchema.getViewSchemaMap().get(viewId);
+        if (Objects.isNull(viewSchema)) {
             return null;
         }
-        SchemaElement elementDb = modelSchema.getElement(elementType, elementID);
+        SchemaElement elementDb = viewSchema.getElement(elementType, elementID);
         if (Objects.isNull(elementDb)) {
             log.info("element is null, elementType:{},elementID:{}", elementType, elementID);
             return null;

chat/core/src/main/java/com/tencent/supersonic/chat/core/mapper/BaseMatchStrategy.java

@@ -1,8 +1,8 @@
-package com.tencent.supersonic.chat.mapper;
+package com.tencent.supersonic.chat.core.mapper;
 
-import com.hankcs.hanlp.seg.common.Term;
-import com.tencent.supersonic.chat.api.pojo.QueryContext;
-import com.tencent.supersonic.knowledge.utils.NatureHelper;
+import com.tencent.supersonic.chat.core.pojo.QueryContext;
+import com.tencent.supersonic.headless.api.pojo.response.S2Term;
+import com.tencent.supersonic.headless.core.knowledge.helper.NatureHelper;
 import java.util.ArrayList;
 import java.util.Comparator;
 import java.util.HashMap;
@@ -19,9 +19,6 @@ import org.apache.commons.lang3.StringUtils;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.stereotype.Service;
 
-/**
- * Base Match Strategy
- */
 @Service
 @Slf4j
 public abstract class BaseMatchStrategy<T> implements MatchStrategy<T> {
@@ -30,24 +27,25 @@ public abstract class BaseMatchStrategy<T> implements MatchStrategy<T> {
     private MapperHelper mapperHelper;
 
     @Override
-    public Map<MatchText, List<T>> match(QueryContext queryContext, List<Term> terms, Set<Long> detectModelIds) {
-        String text = queryContext.getRequest().getQueryText();
+    public Map<MatchText, List<T>> match(QueryContext queryContext, List<S2Term> terms,
+            Set<Long> detectViewIds) {
+        String text = queryContext.getQueryText();
         if (Objects.isNull(terms) || StringUtils.isEmpty(text)) {
             return null;
         }
 
-        log.debug("terms:{},,detectModelIds:{}", terms, detectModelIds);
+        log.debug("terms:{},,detectViewIds:{}", terms, detectViewIds);
 
-        List<T> detects = detect(queryContext, terms, detectModelIds);
+        List<T> detects = detect(queryContext, terms, detectViewIds);
         Map<MatchText, List<T>> result = new HashMap<>();
 
         result.put(MatchText.builder().regText(text).detectSegment(text).build(), detects);
         return result;
     }
 
-    public List<T> detect(QueryContext queryContext, List<Term> terms, Set<Long> detectModelIds) {
+    public List<T> detect(QueryContext queryContext, List<S2Term> terms, Set<Long> detectViewIds) {
         Map<Integer, Integer> regOffsetToLength = getRegOffsetToLength(terms);
-        String text = queryContext.getRequest().getQueryText();
+        String text = queryContext.getQueryText();
         Set<T> results = new HashSet<>();
 
         Set<String> detectSegments = new HashSet<>();
@@ -58,25 +56,26 @@ public abstract class BaseMatchStrategy<T> implements MatchStrategy<T> {
                 int offset = mapperHelper.getStepOffset(terms, startIndex);
                 index = mapperHelper.getStepIndex(regOffsetToLength, index);
                 if (index <= text.length()) {
-                    String detectSegment = text.substring(startIndex, index);
+                    String detectSegment = text.substring(startIndex, index).trim();
                     detectSegments.add(detectSegment);
-                    detectByStep(queryContext, results, detectModelIds, startIndex, index, offset);
+                    detectByStep(queryContext, results, detectViewIds, detectSegment, offset);
                 }
             }
             startIndex = mapperHelper.getStepIndex(regOffsetToLength, startIndex);
         }
-        detectByBatch(queryContext, results, detectModelIds, detectSegments);
+        detectByBatch(queryContext, results, detectViewIds, detectSegments);
         return new ArrayList<>(results);
     }
 
-    protected void detectByBatch(QueryContext queryContext, Set<T> results, Set<Long> detectModelIds,
+    protected void detectByBatch(QueryContext queryContext, Set<T> results, Set<Long> detectViewIds,
             Set<String> detectSegments) {
         return;
     }
 
-    public Map<Integer, Integer> getRegOffsetToLength(List<Term> terms) {
-        return terms.stream().sorted(Comparator.comparing(Term::length))
-                .collect(Collectors.toMap(Term::getOffset, term -> term.word.length(), (value1, value2) -> value2));
+    public Map<Integer, Integer> getRegOffsetToLength(List<S2Term> terms) {
+        return terms.stream().sorted(Comparator.comparing(S2Term::length))
+                .collect(Collectors.toMap(S2Term::getOffset, term -> term.word.length(),
+                        (value1, value2) -> value2));
     }
 
     public void selectResultInOneRound(Set<T> existResults, List<T> oneRoundResults) {
@@ -104,10 +103,10 @@ public abstract class BaseMatchStrategy<T> implements MatchStrategy<T> {
         }
     }
 
-    public List<T> getMatches(QueryContext queryContext, List<Term> terms) {
-        Set<Long> detectModelIds = mapperHelper.getModelIds(queryContext.getRequest());
-        terms = filterByModelIds(terms, detectModelIds);
-        Map<MatchText, List<T>> matchResult = match(queryContext, terms, detectModelIds);
+    public List<T> getMatches(QueryContext queryContext, List<S2Term> terms) {
+        Set<Long> viewIds = mapperHelper.getViewIds(queryContext.getViewId(), queryContext.getAgent());
+        terms = filterByViewId(terms, viewIds);
+        Map<MatchText, List<T>> matchResult = match(queryContext, terms, viewIds);
         List<T> matches = new ArrayList<>();
         if (Objects.isNull(matchResult)) {
             return matches;
@@ -122,27 +121,27 @@ public abstract class BaseMatchStrategy<T> implements MatchStrategy<T> {
         return matches;
     }
 
-    public List<Term> filterByModelIds(List<Term> terms, Set<Long> detectModelIds) {
+    public List<S2Term> filterByViewId(List<S2Term> terms, Set<Long> viewIds) {
         logTerms(terms);
-        if (CollectionUtils.isNotEmpty(detectModelIds)) {
+        if (CollectionUtils.isNotEmpty(viewIds)) {
             terms = terms.stream().filter(term -> {
-                Long modelId = NatureHelper.getModelId(term.getNature().toString());
-                if (Objects.nonNull(modelId)) {
-                    return detectModelIds.contains(modelId);
+                Long viewId = NatureHelper.getViewId(term.getNature().toString());
+                if (Objects.nonNull(viewId)) {
+                    return viewIds.contains(viewId);
                 }
                 return false;
             }).collect(Collectors.toList());
-            log.info("terms filter by modelIds:{}", detectModelIds);
+            log.info("terms filter by viewId:{}", viewIds);
             logTerms(terms);
         }
         return terms;
     }
 
-    public void logTerms(List<Term> terms) {
+    public void logTerms(List<S2Term> terms) {
         if (CollectionUtils.isEmpty(terms)) {
             return;
         }
-        for (Term term : terms) {
+        for (S2Term term : terms) {
             log.debug("word:{},nature:{},frequency:{}", term.word, term.nature.toString(), term.getFrequency());
         }
     }
@@ -151,8 +150,7 @@ public abstract class BaseMatchStrategy<T> implements MatchStrategy<T> {
 
     public abstract String getMapKey(T a);
 
-    public abstract void detectByStep(QueryContext queryContext, Set<T> results,
-            Set<Long> detectModelIds, Integer startIndex, Integer index, int offset);
-
+    public abstract void detectByStep(QueryContext queryContext, Set<T> existResults, Set<Long> detectViewIds,
+            String detectSegment, int offset);
 
 }

chat/core/src/main/java/com/tencent/supersonic/chat/core/mapper/DatabaseMatchStrategy.java

@@ -1,13 +1,18 @@
-package com.tencent.supersonic.chat.mapper;
+package com.tencent.supersonic.chat.core.mapper;
 
-import com.hankcs.hanlp.seg.common.Term;
-import com.tencent.supersonic.chat.api.pojo.QueryContext;
-import com.tencent.supersonic.chat.api.pojo.SchemaElement;
+import com.tencent.supersonic.headless.api.pojo.SchemaElement;
 import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
-import com.tencent.supersonic.chat.config.OptimizationConfig;
+import com.tencent.supersonic.chat.core.config.OptimizationConfig;
+import com.tencent.supersonic.headless.api.pojo.response.S2Term;
+import com.tencent.supersonic.headless.core.knowledge.DatabaseMapResult;
+import com.tencent.supersonic.chat.core.pojo.QueryContext;
 import com.tencent.supersonic.common.pojo.Constants;
-import com.tencent.supersonic.knowledge.dictionary.FuzzyResult;
-import com.tencent.supersonic.knowledge.service.SchemaService;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Service;
+import org.springframework.util.CollectionUtils;
+
 import java.util.ArrayList;
 import java.util.HashSet;
 import java.util.List;
@@ -15,57 +20,47 @@ import java.util.Map;
 import java.util.Map.Entry;
 import java.util.Set;
 import java.util.stream.Collectors;
-import lombok.extern.slf4j.Slf4j;
-import org.apache.commons.lang3.StringUtils;
-import org.springframework.beans.factory.annotation.Autowired;
-import org.springframework.stereotype.Service;
-import org.springframework.util.CollectionUtils;
 
 /**
- * Fuzzy Name Match Strategy
+ * DatabaseMatchStrategy uses SQL LIKE operator to match schema elements.
+ * It currently supports fuzzy matching against names and aliases.
  */
 @Service
 @Slf4j
-public class FuzzyNameMatchStrategy extends BaseMatchStrategy<FuzzyResult> {
+public class DatabaseMatchStrategy extends BaseMatchStrategy<DatabaseMapResult> {
 
     @Autowired
     private OptimizationConfig optimizationConfig;
     @Autowired
     private MapperHelper mapperHelper;
-    @Autowired
-    private SchemaService schemaService;
     private List<SchemaElement> allElements;
 
-
     @Override
-    public Map<MatchText, List<FuzzyResult>> match(QueryContext queryContext, List<Term> terms,
-            Set<Long> detectModelIds) {
-        this.allElements = getSchemaElements();
-        return super.match(queryContext, terms, detectModelIds);
+    public Map<MatchText, List<DatabaseMapResult>> match(QueryContext queryContext, List<S2Term> terms,
+            Set<Long> detectViewIds) {
+        this.allElements = getSchemaElements(queryContext);
+        return super.match(queryContext, terms, detectViewIds);
     }
 
     @Override
-    public boolean needDelete(FuzzyResult oneRoundResult, FuzzyResult existResult) {
+    public boolean needDelete(DatabaseMapResult oneRoundResult, DatabaseMapResult existResult) {
         return getMapKey(oneRoundResult).equals(getMapKey(existResult))
                 && existResult.getDetectWord().length() < oneRoundResult.getDetectWord().length();
     }
 
     @Override
-    public String getMapKey(FuzzyResult a) {
+    public String getMapKey(DatabaseMapResult a) {
         return a.getName() + Constants.UNDERLINE + a.getSchemaElement().getId()
                 + Constants.UNDERLINE + a.getSchemaElement().getName();
     }
 
-    public void detectByStep(QueryContext queryContext, Set<FuzzyResult> existResults, Set<Long> detectModelIds,
-            Integer startIndex, Integer index, int offset) {
-        String detectSegment = queryContext.getRequest().getQueryText().substring(startIndex, index);
+    public void detectByStep(QueryContext queryContext, Set<DatabaseMapResult> existResults, Set<Long> detectViewIds,
+            String detectSegment, int offset) {
         if (StringUtils.isBlank(detectSegment)) {
             return;
         }
-        Set<Long> modelIds = mapperHelper.getModelIds(queryContext.getRequest());
 
         Double metricDimensionThresholdConfig = getThreshold(queryContext);
-
         Map<String, Set<SchemaElement>> nameToItems = getNameToItems(allElements);
 
         for (Entry<String, Set<SchemaElement>> entry : nameToItems.entrySet()) {
@@ -75,34 +70,33 @@ public class FuzzyNameMatchStrategy extends BaseMatchStrategy<FuzzyResult> {
                 continue;
             }
             Set<SchemaElement> schemaElements = entry.getValue();
-            if (!CollectionUtils.isEmpty(modelIds)) {
+            if (!CollectionUtils.isEmpty(detectViewIds)) {
                 schemaElements = schemaElements.stream()
-                        .filter(schemaElement -> modelIds.contains(schemaElement.getModel()))
+                        .filter(schemaElement -> detectViewIds.contains(schemaElement.getView()))
                         .collect(Collectors.toSet());
             }
             for (SchemaElement schemaElement : schemaElements) {
-                FuzzyResult fuzzyResult = new FuzzyResult();
-                fuzzyResult.setDetectWord(detectSegment);
-                fuzzyResult.setName(schemaElement.getName());
-                fuzzyResult.setSchemaElement(schemaElement);
-                existResults.add(fuzzyResult);
+                DatabaseMapResult databaseMapResult = new DatabaseMapResult();
+                databaseMapResult.setDetectWord(detectSegment);
+                databaseMapResult.setName(schemaElement.getName());
+                databaseMapResult.setSchemaElement(schemaElement);
+                existResults.add(databaseMapResult);
             }
         }
     }
 
-    private List<SchemaElement> getSchemaElements() {
+    private List<SchemaElement> getSchemaElements(QueryContext queryContext) {
         List<SchemaElement> allElements = new ArrayList<>();
-        allElements.addAll(schemaService.getSemanticSchema().getDimensions());
-        allElements.addAll(schemaService.getSemanticSchema().getMetrics());
+        allElements.addAll(queryContext.getSemanticSchema().getDimensions());
+        allElements.addAll(queryContext.getSemanticSchema().getMetrics());
         return allElements;
     }
 
-
     private Double getThreshold(QueryContext queryContext) {
         Double metricDimensionThresholdConfig = optimizationConfig.getMetricDimensionThresholdConfig();
         Double metricDimensionMinThresholdConfig = optimizationConfig.getMetricDimensionMinThresholdConfig();
 
-        Map<Long, List<SchemaElementMatch>> modelElementMatches = queryContext.getMapInfo().getModelElementMatches();
+        Map<Long, List<SchemaElementMatch>> modelElementMatches = queryContext.getMapInfo().getViewElementMatches();
 
         boolean existElement = modelElementMatches.entrySet().stream().anyMatch(entry -> entry.getValue().size() >= 1);
 

chat/core/src/main/java/com/tencent/supersonic/chat/core/mapper/EmbeddingMapper.java

@@ -1,21 +1,22 @@
-package com.tencent.supersonic.chat.mapper;
+package com.tencent.supersonic.chat.core.mapper;
 
-import com.alibaba.fastjson.JSONObject;
-import com.hankcs.hanlp.seg.common.Term;
-import com.tencent.supersonic.chat.api.pojo.QueryContext;
-import com.tencent.supersonic.chat.api.pojo.SchemaElement;
 import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
+import com.tencent.supersonic.chat.core.pojo.QueryContext;
 import com.tencent.supersonic.common.util.ContextUtils;
 import com.tencent.supersonic.common.util.embedding.Retrieval;
-import com.tencent.supersonic.knowledge.dictionary.EmbeddingResult;
-import com.tencent.supersonic.knowledge.dictionary.builder.BaseWordBuilder;
-import com.tencent.supersonic.knowledge.utils.HanlpHelper;
-import java.util.List;
+import com.tencent.supersonic.headless.api.pojo.SchemaElement;
+import com.tencent.supersonic.headless.api.pojo.SchemaElementType;
+import com.tencent.supersonic.headless.api.pojo.response.S2Term;
+import com.tencent.supersonic.headless.core.knowledge.EmbeddingResult;
+import com.tencent.supersonic.headless.core.knowledge.builder.BaseWordBuilder;
+import com.tencent.supersonic.headless.core.knowledge.helper.HanlpHelper;
+import com.tencent.supersonic.headless.server.service.KnowledgeService;
 import lombok.extern.slf4j.Slf4j;
-import org.apache.commons.lang3.StringUtils;
+import java.util.List;
+import java.util.Objects;
 
 /***
- * A mapper that is capable of semantic understanding of text.
+ * A mapper that recognizes schema elements with vector embedding.
  */
 @Slf4j
 public class EmbeddingMapper extends BaseMapper {
@@ -23,9 +24,9 @@ public class EmbeddingMapper extends BaseMapper {
     @Override
     public void doMap(QueryContext queryContext) {
         //1. query from embedding by queryText
-
-        String queryText = queryContext.getRequest().getQueryText();
-        List<Term> terms = HanlpHelper.getTerms(queryText);
+        String queryText = queryContext.getQueryText();
+        KnowledgeService knowledgeService = ContextUtils.getBean(KnowledgeService.class);
+        List<S2Term> terms = knowledgeService.getTerms(queryText);
 
         EmbeddingMatchStrategy matchStrategy = ContextUtils.getBean(EmbeddingMatchStrategy.class);
         List<EmbeddingResult> matchResults = matchStrategy.getMatches(queryContext, terms);
@@ -35,16 +36,13 @@ public class EmbeddingMapper extends BaseMapper {
         //2. build SchemaElementMatch by info
         for (EmbeddingResult matchResult : matchResults) {
             Long elementId = Retrieval.getLongId(matchResult.getId());
-
-            SchemaElement schemaElement = JSONObject.parseObject(JSONObject.toJSONString(matchResult.getMetadata()),
-                    SchemaElement.class);
-
-            if (StringUtils.isBlank(matchResult.getMetadata().get("modelId"))) {
+            Long viewId = Retrieval.getLongId(matchResult.getMetadata().get("viewId"));
+            if (Objects.isNull(viewId)) {
                 continue;
             }
-            long modelId = Long.parseLong(matchResult.getMetadata().get("modelId"));
-
-            schemaElement = getSchemaElement(modelId, schemaElement.getType(), elementId);
+            SchemaElementType elementType = SchemaElementType.valueOf(matchResult.getMetadata().get("type"));
+            SchemaElement schemaElement = getSchemaElement(viewId, elementType, elementId,
+                    queryContext.getSemanticSchema());
             if (schemaElement == null) {
                 continue;
             }
@@ -56,7 +54,7 @@ public class EmbeddingMapper extends BaseMapper {
                     .detectWord(matchResult.getDetectWord())
                     .build();
             //3. add to mapInfo
-            addToSchemaMap(queryContext.getMapInfo(), modelId, schemaElementMatch);
+            addToSchemaMap(queryContext.getMapInfo(), viewId, schemaElementMatch);
         }
     }
 }

chat/core/src/main/java/com/tencent/supersonic/chat/core/mapper/EmbeddingMatchStrategy.java

@@ -1,17 +1,16 @@
-package com.tencent.supersonic.chat.mapper;
+package com.tencent.supersonic.chat.core.mapper;
 
 import com.google.common.collect.Lists;
-import com.tencent.supersonic.chat.api.pojo.QueryContext;
-import com.tencent.supersonic.chat.config.OptimizationConfig;
+import com.tencent.supersonic.chat.core.config.OptimizationConfig;
+import com.tencent.supersonic.chat.core.pojo.QueryContext;
 import com.tencent.supersonic.common.pojo.Constants;
-import com.tencent.supersonic.common.util.embedding.EmbeddingUtils;
 import com.tencent.supersonic.common.util.embedding.Retrieval;
 import com.tencent.supersonic.common.util.embedding.RetrieveQuery;
 import com.tencent.supersonic.common.util.embedding.RetrieveQueryResult;
-import com.tencent.supersonic.knowledge.dictionary.EmbeddingResult;
-import com.tencent.supersonic.semantic.model.domain.listener.MetaEmbeddingListener;
+import com.tencent.supersonic.headless.core.knowledge.EmbeddingResult;
+import com.tencent.supersonic.headless.server.service.MetaEmbeddingService;
+import java.util.ArrayList;
 import java.util.Comparator;
-import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
@@ -24,7 +23,8 @@ import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.stereotype.Service;
 
 /**
- * match strategy implement
+ * EmbeddingMatchStrategy uses vector database to perform
+ * similarity search against the embeddings of schema elements.
  */
 @Service
 @Slf4j
@@ -32,8 +32,9 @@ public class EmbeddingMatchStrategy extends BaseMatchStrategy<EmbeddingResult> {
 
     @Autowired
     private OptimizationConfig optimizationConfig;
+
     @Autowired
-    private EmbeddingUtils embeddingUtils;
+    private MetaEmbeddingService metaEmbeddingService;
 
     @Override
     public boolean needDelete(EmbeddingResult oneRoundResult, EmbeddingResult existResult) {
@@ -46,9 +47,14 @@ public class EmbeddingMatchStrategy extends BaseMatchStrategy<EmbeddingResult> {
         return a.getName() + Constants.UNDERLINE + a.getId();
     }
 
+    @Override
+    public void detectByStep(QueryContext queryContext, Set<EmbeddingResult> existResults, Set<Long> detectViewIds,
+            String detectSegment, int offset) {
+
+    }
 
     @Override
-    protected void detectByBatch(QueryContext queryContext, Set<EmbeddingResult> results, Set<Long> detectModelIds,
+    protected void detectByBatch(QueryContext queryContext, Set<EmbeddingResult> results, Set<Long> detectViewIds,
             Set<String> detectSegments) {
 
         List<String> queryTextsList = detectSegments.stream()
@@ -62,49 +68,29 @@ public class EmbeddingMatchStrategy extends BaseMatchStrategy<EmbeddingResult> {
                 optimizationConfig.getEmbeddingMapperBatch());
 
         for (List<String> queryTextsSub : queryTextsSubList) {
-            detectByQueryTextsSub(results, detectModelIds, queryTextsSub);
+            detectByQueryTextsSub(results, detectViewIds, queryTextsSub);
         }
     }
 
-    private void detectByQueryTextsSub(Set<EmbeddingResult> results, Set<Long> detectModelIds,
+    private void detectByQueryTextsSub(Set<EmbeddingResult> results, Set<Long> detectViewIds,
             List<String> queryTextsSub) {
         int embeddingNumber = optimizationConfig.getEmbeddingMapperNumber();
         Double distance = optimizationConfig.getEmbeddingMapperDistanceThreshold();
-        Map<String, String> filterCondition = null;
         // step1. build query params
-        // if only one modelId, add to filterCondition
-        if (CollectionUtils.isNotEmpty(detectModelIds) && detectModelIds.size() == 1) {
-            filterCondition = new HashMap<>();
-            filterCondition.put("modelId", detectModelIds.stream().findFirst().get().toString());
-        }
-
-        RetrieveQuery retrieveQuery = RetrieveQuery.builder()
-                .queryTextsList(queryTextsSub)
-                .filterCondition(filterCondition)
-                .queryEmbeddings(null)
-                .build();
+        RetrieveQuery retrieveQuery = RetrieveQuery.builder().queryTextsList(queryTextsSub).build();
         // step2. retrieveQuery by detectSegment
-        List<RetrieveQueryResult> retrieveQueryResults = embeddingUtils.retrieveQuery(
-                MetaEmbeddingListener.COLLECTION_NAME, retrieveQuery, embeddingNumber);
+        List<RetrieveQueryResult> retrieveQueryResults = metaEmbeddingService.retrieveQuery(
+                new ArrayList<>(detectViewIds), retrieveQuery, embeddingNumber);
 
         if (CollectionUtils.isEmpty(retrieveQueryResults)) {
             return;
         }
-        // step3. build EmbeddingResults. filter by modelId
+        // step3. build EmbeddingResults
         List<EmbeddingResult> collect = retrieveQueryResults.stream()
                 .map(retrieveQueryResult -> {
                     List<Retrieval> retrievals = retrieveQueryResult.getRetrieval();
                     if (CollectionUtils.isNotEmpty(retrievals)) {
                         retrievals.removeIf(retrieval -> retrieval.getDistance() > distance.doubleValue());
-                        if (CollectionUtils.isNotEmpty(detectModelIds)) {
-                            retrievals.removeIf(retrieval -> {
-                                String modelIdStr = retrieval.getMetadata().get("modelId");
-                                if (StringUtils.isBlank(modelIdStr)) {
-                                    return true;
-                                }
-                                return detectModelIds.contains(Long.parseLong(modelIdStr));
-                            });
-                        }
                     }
                     return retrieveQueryResult;
                 })
@@ -115,6 +101,9 @@ public class EmbeddingMatchStrategy extends BaseMatchStrategy<EmbeddingResult> {
                             BeanUtils.copyProperties(retrieval, embeddingResult);
                             embeddingResult.setDetectWord(retrieveQueryResult.getQuery());
                             embeddingResult.setName(retrieval.getQuery());
+                            Map<String, String> convertedMap = retrieval.getMetadata().entrySet().stream()
+                                    .collect(Collectors.toMap(Map.Entry::getKey, entry -> entry.getValue().toString()));
+                            embeddingResult.setMetadata(convertedMap);
                             return embeddingResult;
                         }))
                 .collect(Collectors.toList());
@@ -128,9 +117,4 @@ public class EmbeddingMatchStrategy extends BaseMatchStrategy<EmbeddingResult> {
         selectResultInOneRound(results, oneRoundResults);
     }
 
-    @Override
-    public void detectByStep(QueryContext queryContext, Set<EmbeddingResult> existResults, Set<Long> detectModelIds,
-            Integer startIndex, Integer index, int offset) {
-        return;
-    }
 }

chat/core/src/main/java/com/tencent/supersonic/chat/core/mapper/EntityMapper.java

@@ -1,14 +1,13 @@
-package com.tencent.supersonic.chat.mapper;
+package com.tencent.supersonic.chat.core.mapper;
 
 
-import com.tencent.supersonic.chat.api.pojo.ModelSchema;
-import com.tencent.supersonic.chat.api.pojo.QueryContext;
-import com.tencent.supersonic.chat.api.pojo.SchemaElement;
+import com.tencent.supersonic.headless.api.pojo.SchemaElement;
 import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
-import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
+import com.tencent.supersonic.headless.api.pojo.SchemaElementType;
 import com.tencent.supersonic.chat.api.pojo.SchemaMapInfo;
-import com.tencent.supersonic.chat.service.SemanticService;
-import com.tencent.supersonic.common.util.ContextUtils;
+import com.tencent.supersonic.chat.api.pojo.SemanticSchema;
+import com.tencent.supersonic.chat.api.pojo.ViewSchema;
+import com.tencent.supersonic.chat.core.pojo.QueryContext;
 import lombok.extern.slf4j.Slf4j;
 import org.springframework.beans.BeanUtils;
 import org.springframework.util.CollectionUtils;
@@ -25,12 +24,12 @@ public class EntityMapper extends BaseMapper {
     @Override
     public void doMap(QueryContext queryContext) {
         SchemaMapInfo schemaMapInfo = queryContext.getMapInfo();
-        for (Long modelId : schemaMapInfo.getMatchedModels()) {
-            List<SchemaElementMatch> schemaElementMatchList = schemaMapInfo.getMatchedElements(modelId);
+        for (Long viewId : schemaMapInfo.getMatchedViewInfos()) {
+            List<SchemaElementMatch> schemaElementMatchList = schemaMapInfo.getMatchedElements(viewId);
             if (CollectionUtils.isEmpty(schemaElementMatchList)) {
                 continue;
             }
-            SchemaElement entity = getEntity(modelId);
+            SchemaElement entity = getEntity(viewId, queryContext);
             if (entity == null || entity.getId() == null) {
                 continue;
             }
@@ -66,9 +65,9 @@ public class EntityMapper extends BaseMapper {
         return false;
     }
 
-    private SchemaElement getEntity(Long modelId) {
-        SemanticService semanticService = ContextUtils.getBean(SemanticService.class);
-        ModelSchema modelSchema = semanticService.getModelSchema(modelId);
+    private SchemaElement getEntity(Long viewId, QueryContext queryContext) {
+        SemanticSchema semanticSchema = queryContext.getSemanticSchema();
+        ViewSchema modelSchema = semanticSchema.getViewSchemaMap().get(viewId);
         if (modelSchema != null && modelSchema.getEntity() != null) {
             return modelSchema.getEntity();
         }

chat/core/src/main/java/com/tencent/supersonic/chat/core/mapper/HanlpDictMatchStrategy.java

@@ -1,12 +1,11 @@
-package com.tencent.supersonic.chat.mapper;
+package com.tencent.supersonic.chat.core.mapper;
 
-import com.hankcs.hanlp.seg.common.Term;
-import com.tencent.supersonic.chat.api.pojo.QueryContext;
-import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
-import com.tencent.supersonic.chat.config.OptimizationConfig;
+import com.tencent.supersonic.chat.core.config.OptimizationConfig;
+import com.tencent.supersonic.chat.core.pojo.QueryContext;
 import com.tencent.supersonic.common.pojo.Constants;
-import com.tencent.supersonic.knowledge.dictionary.HanlpMapResult;
-import com.tencent.supersonic.knowledge.service.SearchService;
+import com.tencent.supersonic.headless.api.pojo.response.S2Term;
+import com.tencent.supersonic.headless.core.knowledge.HanlpMapResult;
+import com.tencent.supersonic.headless.server.service.KnowledgeService;
 import java.util.HashMap;
 import java.util.LinkedHashSet;
 import java.util.List;
@@ -21,7 +20,9 @@ import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.stereotype.Service;
 
 /**
- * match strategy implement
+ * HanlpDictMatchStrategy uses <a href="https://www.hanlp.com/">HanLP</a> to
+ * match schema elements. It currently supports prefix and suffix matching
+ * against names, values and aliases.
  */
 @Service
 @Slf4j
@@ -33,18 +34,20 @@ public class HanlpDictMatchStrategy extends BaseMatchStrategy<HanlpMapResult> {
     @Autowired
     private OptimizationConfig optimizationConfig;
 
+    @Autowired
+    private KnowledgeService knowledgeService;
+
     @Override
-    public Map<MatchText, List<HanlpMapResult>> match(QueryContext queryContext, List<Term> terms,
-            Set<Long> detectModelIds) {
-        QueryReq queryReq = queryContext.getRequest();
-        String text = queryReq.getQueryText();
+    public Map<MatchText, List<HanlpMapResult>> match(QueryContext queryContext, List<S2Term> terms,
+            Set<Long> detectViewIds) {
+        String text = queryContext.getQueryText();
         if (Objects.isNull(terms) || StringUtils.isEmpty(text)) {
             return null;
         }
 
-        log.debug("retryCount:{},terms:{},,detectModelIds:{}", terms, detectModelIds);
+        log.debug("retryCount:{},terms:{},,detectModelIds:{}", terms, detectViewIds);
 
-        List<HanlpMapResult> detects = detect(queryContext, terms, detectModelIds);
+        List<HanlpMapResult> detects = detect(queryContext, terms, detectViewIds);
         Map<MatchText, List<HanlpMapResult>> result = new HashMap<>();
 
         result.put(MatchText.builder().regText(text).detectSegment(text).build(), detects);
@@ -57,22 +60,15 @@ public class HanlpDictMatchStrategy extends BaseMatchStrategy<HanlpMapResult> {
                 && existResult.getDetectWord().length() < oneRoundResult.getDetectWord().length();
     }
 
-    public void detectByStep(QueryContext queryContext, Set<HanlpMapResult> existResults, Set<Long> detectModelIds,
-            Integer startIndex, Integer index, int offset) {
-        QueryReq queryReq = queryContext.getRequest();
-        String text = queryReq.getQueryText();
-        Integer agentId = queryReq.getAgentId();
-        String detectSegment = text.substring(startIndex, index);
-
+    public void detectByStep(QueryContext queryContext, Set<HanlpMapResult> existResults, Set<Long> detectViewIds,
+            String detectSegment, int offset) {
         // step1. pre search
         Integer oneDetectionMaxSize = optimizationConfig.getOneDetectionMaxSize();
-        LinkedHashSet<HanlpMapResult> hanlpMapResults = SearchService.prefixSearch(detectSegment, oneDetectionMaxSize,
-                agentId,
-                detectModelIds).stream().collect(Collectors.toCollection(LinkedHashSet::new));
+        LinkedHashSet<HanlpMapResult> hanlpMapResults = knowledgeService.prefixSearch(detectSegment,
+                oneDetectionMaxSize, detectViewIds).stream().collect(Collectors.toCollection(LinkedHashSet::new));
         // step2. suffix search
-        LinkedHashSet<HanlpMapResult> suffixHanlpMapResults = SearchService.suffixSearch(detectSegment,
-                        oneDetectionMaxSize, agentId, detectModelIds).stream()
-                .collect(Collectors.toCollection(LinkedHashSet::new));
+        LinkedHashSet<HanlpMapResult> suffixHanlpMapResults = knowledgeService.suffixSearch(detectSegment,
+                oneDetectionMaxSize, detectViewIds).stream().collect(Collectors.toCollection(LinkedHashSet::new));
 
         hanlpMapResults.addAll(suffixHanlpMapResults);
 

chat/core/src/main/java/com/tencent/supersonic/chat/core/mapper/KeywordMapper.java

@@ -0,0 +1,125 @@
+package com.tencent.supersonic.chat.core.mapper;
+
+import com.tencent.supersonic.headless.api.pojo.SchemaElement;
+import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
+import com.tencent.supersonic.headless.api.pojo.SchemaElementType;
+import com.tencent.supersonic.chat.api.pojo.SchemaMapInfo;
+import com.tencent.supersonic.headless.api.pojo.response.S2Term;
+import com.tencent.supersonic.headless.core.knowledge.DatabaseMapResult;
+import com.tencent.supersonic.headless.core.knowledge.HanlpMapResult;
+import com.tencent.supersonic.chat.core.pojo.QueryContext;
+import com.tencent.supersonic.headless.server.service.KnowledgeService;
+import com.tencent.supersonic.headless.core.knowledge.helper.HanlpHelper;
+import com.tencent.supersonic.headless.core.knowledge.helper.NatureHelper;
+import com.tencent.supersonic.common.util.ContextUtils;
+import lombok.extern.slf4j.Slf4j;
+import org.springframework.util.CollectionUtils;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/***
+ * A mapper that recognizes schema elements with keyword.
+ * It leverages two matching strategies: HanlpDictMatchStrategy and DatabaseMatchStrategy.
+ */
+@Slf4j
+public class KeywordMapper extends BaseMapper {
+
+    @Override
+    public void doMap(QueryContext queryContext) {
+        String queryText = queryContext.getQueryText();
+        //1.hanlpDict Match
+        KnowledgeService knowledgeService = ContextUtils.getBean(KnowledgeService.class);
+        List<S2Term> terms = knowledgeService.getTerms(queryText);
+        HanlpDictMatchStrategy hanlpMatchStrategy = ContextUtils.getBean(HanlpDictMatchStrategy.class);
+
+        List<HanlpMapResult> hanlpMapResults = hanlpMatchStrategy.getMatches(queryContext, terms);
+        convertHanlpMapResultToMapInfo(hanlpMapResults, queryContext, terms);
+
+        //2.database Match
+        DatabaseMatchStrategy databaseMatchStrategy = ContextUtils.getBean(DatabaseMatchStrategy.class);
+
+        List<DatabaseMapResult> databaseResults = databaseMatchStrategy.getMatches(queryContext, terms);
+        convertDatabaseMapResultToMapInfo(queryContext, databaseResults);
+    }
+
+    private void convertHanlpMapResultToMapInfo(List<HanlpMapResult> mapResults, QueryContext queryContext,
+            List<S2Term> terms) {
+        if (CollectionUtils.isEmpty(mapResults)) {
+            return;
+        }
+        HanlpHelper.transLetterOriginal(mapResults);
+        Map<String, Long> wordNatureToFrequency = terms.stream().collect(
+                Collectors.toMap(entry -> entry.getWord() + entry.getNature(),
+                        term -> Long.valueOf(term.getFrequency()), (value1, value2) -> value2));
+
+        for (HanlpMapResult hanlpMapResult : mapResults) {
+            for (String nature : hanlpMapResult.getNatures()) {
+                Long viewId = NatureHelper.getViewId(nature);
+                if (Objects.isNull(viewId)) {
+                    continue;
+                }
+                SchemaElementType elementType = NatureHelper.convertToElementType(nature);
+                if (Objects.isNull(elementType)) {
+                    continue;
+                }
+                Long elementID = NatureHelper.getElementID(nature);
+                SchemaElement element = getSchemaElement(viewId, elementType,
+                        elementID, queryContext.getSemanticSchema());
+                if (element == null) {
+                    continue;
+                }
+                if (element.getType().equals(SchemaElementType.VALUE)) {
+                    element.setName(hanlpMapResult.getName());
+                }
+                Long frequency = wordNatureToFrequency.get(hanlpMapResult.getName() + nature);
+                SchemaElementMatch schemaElementMatch = SchemaElementMatch.builder()
+                        .element(element)
+                        .frequency(frequency)
+                        .word(hanlpMapResult.getName())
+                        .similarity(hanlpMapResult.getSimilarity())
+                        .detectWord(hanlpMapResult.getDetectWord())
+                        .build();
+
+                addToSchemaMap(queryContext.getMapInfo(), viewId, schemaElementMatch);
+            }
+        }
+    }
+
+    private void convertDatabaseMapResultToMapInfo(QueryContext queryContext, List<DatabaseMapResult> mapResults) {
+        MapperHelper mapperHelper = ContextUtils.getBean(MapperHelper.class);
+        for (DatabaseMapResult match : mapResults) {
+            SchemaElement schemaElement = match.getSchemaElement();
+            Set<Long> regElementSet = getRegElementSet(queryContext.getMapInfo(), schemaElement);
+            if (regElementSet.contains(schemaElement.getId())) {
+                continue;
+            }
+            SchemaElementMatch schemaElementMatch = SchemaElementMatch.builder()
+                    .element(schemaElement)
+                    .word(schemaElement.getName())
+                    .detectWord(match.getDetectWord())
+                    .frequency(10000L)
+                    .similarity(mapperHelper.getSimilarity(match.getDetectWord(), schemaElement.getName()))
+                    .build();
+            log.info("add to schema, elementMatch {}", schemaElementMatch);
+            addToSchemaMap(queryContext.getMapInfo(), schemaElement.getView(), schemaElementMatch);
+        }
+    }
+
+    private Set<Long> getRegElementSet(SchemaMapInfo schemaMap, SchemaElement schemaElement) {
+        List<SchemaElementMatch> elements = schemaMap.getMatchedElements(schemaElement.getView());
+        if (CollectionUtils.isEmpty(elements)) {
+            return new HashSet<>();
+        }
+        return elements.stream()
+                .filter(elementMatch ->
+                        SchemaElementType.METRIC.equals(elementMatch.getElement().getType())
+                                || SchemaElementType.DIMENSION.equals(elementMatch.getElement().getType()))
+                .map(elementMatch -> elementMatch.getElement().getId())
+                .collect(Collectors.toSet());
+    }
+}

chat/core/src/main/java/com/tencent/supersonic/chat/core/mapper/MapperHelper.java

@@ -1,12 +1,15 @@
-package com.tencent.supersonic.chat.mapper;
+package com.tencent.supersonic.chat.core.mapper;
 
 import com.hankcs.hanlp.algorithm.EditDistance;
-import com.hankcs.hanlp.seg.common.Term;
-import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
-import com.tencent.supersonic.chat.config.OptimizationConfig;
-import com.tencent.supersonic.chat.service.AgentService;
-import com.tencent.supersonic.common.util.ContextUtils;
-import com.tencent.supersonic.knowledge.utils.NatureHelper;
+import com.tencent.supersonic.chat.core.agent.Agent;
+import com.tencent.supersonic.chat.core.config.OptimizationConfig;
+import com.tencent.supersonic.headless.api.pojo.response.S2Term;
+import com.tencent.supersonic.headless.core.knowledge.helper.NatureHelper;
+import lombok.Data;
+import lombok.extern.slf4j.Slf4j;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Service;
+
 import java.util.Comparator;
 import java.util.HashSet;
 import java.util.List;
@@ -14,14 +17,6 @@ import java.util.Map;
 import java.util.Objects;
 import java.util.Set;
 import java.util.stream.Collectors;
-import lombok.Data;
-import lombok.extern.slf4j.Slf4j;
-import org.springframework.beans.factory.annotation.Autowired;
-import org.springframework.stereotype.Service;
-
-/**
- * Mapper helper
- */
 
 @Data
 @Service
@@ -41,9 +36,8 @@ public class MapperHelper {
         return index;
     }
 
-
-    public Integer getStepOffset(List<Term> termList, Integer index) {
-        List<Integer> offsetList = termList.stream().sorted(Comparator.comparing(Term::getOffset))
+    public Integer getStepOffset(List<S2Term> termList, Integer index) {
+        List<Integer> offsetList = termList.stream().sorted(Comparator.comparing(S2Term::getOffset))
                 .map(term -> term.getOffset()).collect(Collectors.toList());
 
         for (int j = 0; j < termList.size() - 1; j++) {
@@ -68,7 +62,7 @@ public class MapperHelper {
      */
     public boolean existDimensionValues(List<String> natures) {
         for (String nature : natures) {
-            if (NatureHelper.isDimensionValueModelId(nature)) {
+            if (NatureHelper.isDimensionValueViewId(nature)) {
                 return true;
             }
         }
@@ -88,34 +82,33 @@ public class MapperHelper {
                 detectSegment.length());
     }
 
-    public Set<Long> getModelIds(QueryReq request) {
+    public Set<Long> getViewIds(Long viewId, Agent agent) {
 
-        Long modelId = request.getModelId();
-
-        AgentService agentService = ContextUtils.getBean(AgentService.class);
-
-        Set<Long> detectModelIds = agentService.getModelIds(request.getAgentId(), null);
+        Set<Long> detectViewIds = new HashSet<>();
+        if (Objects.nonNull(agent)) {
+            detectViewIds = agent.getViewIds(null);
+        }
         //contains all
-        if (agentService.containsAllModel(detectModelIds)) {
-            if (Objects.nonNull(modelId) && modelId > 0) {
+        if (Agent.containsAllModel(detectViewIds)) {
+            if (Objects.nonNull(viewId) && viewId > 0) {
                 Set<Long> result = new HashSet<>();
-                result.add(modelId);
+                result.add(viewId);
                 return result;
             }
             return new HashSet<>();
         }
 
-        if (Objects.nonNull(detectModelIds)) {
-            detectModelIds = detectModelIds.stream().filter(entry -> entry > 0).collect(Collectors.toSet());
+        if (Objects.nonNull(detectViewIds)) {
+            detectViewIds = detectViewIds.stream().filter(entry -> entry > 0).collect(Collectors.toSet());
         }
 
-        if (Objects.nonNull(modelId) && modelId > 0 && Objects.nonNull(detectModelIds)) {
-            if (detectModelIds.contains(modelId)) {
+        if (Objects.nonNull(viewId) && viewId > 0 && Objects.nonNull(detectViewIds)) {
+            if (detectViewIds.contains(viewId)) {
                 Set<Long> result = new HashSet<>();
-                result.add(modelId);
+                result.add(viewId);
                 return result;
             }
         }
-        return detectModelIds;
+        return detectViewIds;
     }
 }

chat/core/src/main/java/com/tencent/supersonic/chat/core/mapper/MatchStrategy.java

@@ -0,0 +1,18 @@
+package com.tencent.supersonic.chat.core.mapper;
+
+import com.tencent.supersonic.chat.core.pojo.QueryContext;
+import com.tencent.supersonic.headless.api.pojo.response.S2Term;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * MatchStrategy encapsulates a concrete matching algorithm
+ * executed during query or search process.
+ */
+public interface MatchStrategy<T> {
+
+    Map<MatchText, List<T>> match(QueryContext queryContext, List<S2Term> terms, Set<Long> detectViewIds);
+
+}

chat/core/src/main/java/com/tencent/supersonic/chat/core/mapper/MatchText.java

@@ -1,4 +1,4 @@
-package com.tencent.supersonic.chat.mapper;
+package com.tencent.supersonic.chat.core.mapper;
 
 import java.util.Objects;
 import lombok.Builder;

chat/core/src/main/java/com/tencent/supersonic/chat/core/mapper/ModelWithSemanticType.java

@@ -1,6 +1,6 @@
-package com.tencent.supersonic.chat.mapper;
+package com.tencent.supersonic.chat.core.mapper;
 
-import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
+import com.tencent.supersonic.headless.api.pojo.SchemaElementType;
 import java.io.Serializable;
 import lombok.Data;
 import lombok.ToString;
@@ -10,10 +10,10 @@ import lombok.ToString;
 public class ModelWithSemanticType implements Serializable {
 
     private Long model;
-    private SchemaElementType semanticType;
+    private SchemaElementType schemaElementType;
 
-    public ModelWithSemanticType(Long model, SchemaElementType semanticType) {
+    public ModelWithSemanticType(Long model, SchemaElementType schemaElementType) {
         this.model = model;
-        this.semanticType = semanticType;
+        this.schemaElementType = schemaElementType;
     }
 }

chat/core/src/main/java/com/tencent/supersonic/chat/core/mapper/QueryFilterMapper.java

@@ -1,18 +1,18 @@
-package com.tencent.supersonic.chat.mapper;
+package com.tencent.supersonic.chat.core.mapper;
 
 import com.google.common.collect.Lists;
-import com.tencent.supersonic.chat.api.component.SchemaMapper;
-import com.tencent.supersonic.chat.api.pojo.QueryContext;
-import com.tencent.supersonic.chat.api.pojo.SchemaMapInfo;
+import com.tencent.supersonic.headless.api.pojo.SchemaElement;
 import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
-import com.tencent.supersonic.chat.api.pojo.SchemaElement;
-import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
+import com.tencent.supersonic.headless.api.pojo.SchemaElementType;
+import com.tencent.supersonic.chat.api.pojo.SchemaMapInfo;
 import com.tencent.supersonic.chat.api.pojo.request.QueryFilter;
 import com.tencent.supersonic.chat.api.pojo.request.QueryFilters;
-import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
+import com.tencent.supersonic.headless.core.knowledge.builder.BaseWordBuilder;
+import com.tencent.supersonic.chat.core.pojo.QueryContext;
 import com.tencent.supersonic.common.pojo.Constants;
 import lombok.extern.slf4j.Slf4j;
 import org.springframework.util.CollectionUtils;
+
 import java.util.List;
 import java.util.Map;
 import java.util.stream.Collectors;
@@ -20,40 +20,39 @@ import java.util.stream.Collectors;
 @Slf4j
 public class QueryFilterMapper implements SchemaMapper {
 
-    private Long frequency = 9999999L;
     private double similarity = 1.0;
 
     @Override
     public void map(QueryContext queryContext) {
-        QueryReq queryReq = queryContext.getRequest();
-        Long modelId = queryReq.getModelId();
-        if (modelId == null || modelId <= 0) {
+        Long viewId = queryContext.getViewId();
+        if (viewId == null || viewId <= 0) {
             return;
         }
         SchemaMapInfo schemaMapInfo = queryContext.getMapInfo();
-        clearOtherSchemaElementMatch(modelId, schemaMapInfo);
-        List<SchemaElementMatch> schemaElementMatches = schemaMapInfo.getMatchedElements(modelId);
+        clearOtherSchemaElementMatch(viewId, schemaMapInfo);
+        List<SchemaElementMatch> schemaElementMatches = schemaMapInfo.getMatchedElements(viewId);
         if (schemaElementMatches == null) {
             schemaElementMatches = Lists.newArrayList();
-            schemaMapInfo.setMatchedElements(modelId, schemaElementMatches);
+            schemaMapInfo.setMatchedElements(viewId, schemaElementMatches);
         }
-        addValueSchemaElementMatch(schemaElementMatches, queryReq.getQueryFilters());
+        addValueSchemaElementMatch(queryContext, schemaElementMatches);
     }
 
     private void clearOtherSchemaElementMatch(Long modelId, SchemaMapInfo schemaMapInfo) {
-        for (Map.Entry<Long, List<SchemaElementMatch>> entry : schemaMapInfo.getModelElementMatches().entrySet()) {
+        for (Map.Entry<Long, List<SchemaElementMatch>> entry : schemaMapInfo.getViewElementMatches().entrySet()) {
             if (!entry.getKey().equals(modelId)) {
                 entry.getValue().clear();
             }
         }
     }
 
-    private List<SchemaElementMatch> addValueSchemaElementMatch(List<SchemaElementMatch> candidateElementMatches,
-                                                           QueryFilters queryFilter) {
-        if (queryFilter == null || CollectionUtils.isEmpty(queryFilter.getFilters())) {
+    private List<SchemaElementMatch> addValueSchemaElementMatch(QueryContext queryContext,
+            List<SchemaElementMatch> candidateElementMatches) {
+        QueryFilters queryFilters = queryContext.getQueryFilters();
+        if (queryFilters == null || CollectionUtils.isEmpty(queryFilters.getFilters())) {
             return candidateElementMatches;
         }
-        for (QueryFilter filter : queryFilter.getFilters()) {
+        for (QueryFilter filter : queryFilters.getFilters()) {
             if (checkExistSameValueSchemaElementMatch(filter, candidateElementMatches)) {
                 continue;
             }
@@ -62,10 +61,11 @@ public class QueryFilterMapper implements SchemaMapper {
                     .name(String.valueOf(filter.getValue()))
                     .type(SchemaElementType.VALUE)
                     .bizName(filter.getBizName())
+                    .view(queryContext.getViewId())
                     .build();
             SchemaElementMatch schemaElementMatch = SchemaElementMatch.builder()
                     .element(element)
-                    .frequency(frequency)
+                    .frequency(BaseWordBuilder.DEFAULT_FREQUENCY)
                     .word(String.valueOf(filter.getValue()))
                     .similarity(similarity)
                     .detectWord(Constants.EMPTY)

chat/core/src/main/java/com/tencent/supersonic/chat/core/mapper/SchemaMapper.java

@@ -1,6 +1,6 @@
-package com.tencent.supersonic.chat.api.component;
+package com.tencent.supersonic.chat.core.mapper;
 
-import com.tencent.supersonic.chat.api.pojo.QueryContext;
+import com.tencent.supersonic.chat.core.pojo.QueryContext;
 
 /**
  * A schema mapper identifies references to schema elements(metrics/dimensions/entities/values)

chat/core/src/main/java/com/tencent/supersonic/chat/core/mapper/SearchMatchStrategy.java

@@ -1,12 +1,12 @@
-package com.tencent.supersonic.chat.mapper;
+package com.tencent.supersonic.chat.core.mapper;
 
 import com.google.common.collect.Lists;
-import com.hankcs.hanlp.seg.common.Term;
-import com.tencent.supersonic.chat.api.pojo.QueryContext;
-import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
+import com.tencent.supersonic.headless.api.pojo.response.S2Term;
+import com.tencent.supersonic.headless.core.knowledge.HanlpMapResult;
+import com.tencent.supersonic.headless.core.knowledge.SearchService;
+import com.tencent.supersonic.chat.core.pojo.QueryContext;
 import com.tencent.supersonic.common.pojo.enums.DictWordType;
-import com.tencent.supersonic.knowledge.dictionary.HanlpMapResult;
-import com.tencent.supersonic.knowledge.service.SearchService;
+import com.tencent.supersonic.headless.server.service.KnowledgeService;
 import java.util.List;
 import java.util.Map;
 import java.util.Objects;
@@ -15,21 +15,25 @@ import java.util.concurrent.ConcurrentHashMap;
 import java.util.stream.Collectors;
 import org.apache.commons.collections.CollectionUtils;
 import org.apache.commons.lang3.StringUtils;
+import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.stereotype.Service;
 
 /**
- * match strategy implement
+ * SearchMatchStrategy encapsulates a concrete matching algorithm
+ * executed during search process.
  */
 @Service
 public class SearchMatchStrategy extends BaseMatchStrategy<HanlpMapResult> {
 
     private static final int SEARCH_SIZE = 3;
 
+    @Autowired
+    private KnowledgeService knowledgeService;
+
     @Override
-    public Map<MatchText, List<HanlpMapResult>> match(QueryContext queryContext, List<Term> originals,
-            Set<Long> detectModelIds) {
-        QueryReq queryReq = queryContext.getRequest();
-        String text = queryReq.getQueryText();
+    public Map<MatchText, List<HanlpMapResult>> match(QueryContext queryContext, List<S2Term> originals,
+            Set<Long> detectViewIds) {
+        String text = queryContext.getQueryText();
         Map<Integer, Integer> regOffsetToLength = getRegOffsetToLength(originals);
 
         List<Integer> detectIndexList = Lists.newArrayList();
@@ -52,10 +56,10 @@ public class SearchMatchStrategy extends BaseMatchStrategy<HanlpMapResult> {
                     String detectSegment = text.substring(detectIndex);
 
                     if (StringUtils.isNotEmpty(detectSegment)) {
-                        List<HanlpMapResult> hanlpMapResults = SearchService.prefixSearch(detectSegment,
-                                SearchService.SEARCH_SIZE, queryReq.getAgentId(), detectModelIds);
-                        List<HanlpMapResult> suffixHanlpMapResults = SearchService.suffixSearch(
-                                detectSegment, SEARCH_SIZE, queryReq.getAgentId(), detectModelIds);
+                        List<HanlpMapResult> hanlpMapResults = knowledgeService.prefixSearch(detectSegment,
+                                SearchService.SEARCH_SIZE, detectViewIds);
+                        List<HanlpMapResult> suffixHanlpMapResults = knowledgeService.suffixSearch(
+                                detectSegment, SEARCH_SIZE, detectViewIds);
                         hanlpMapResults.addAll(suffixHanlpMapResults);
                         // remove entity name where search
                         hanlpMapResults = hanlpMapResults.stream().filter(entry -> {
@@ -89,9 +93,9 @@ public class SearchMatchStrategy extends BaseMatchStrategy<HanlpMapResult> {
     }
 
     @Override
-    public void detectByStep(QueryContext queryContext, Set<HanlpMapResult> results, Set<Long> detectModelIds,
-            Integer startIndex,
-            Integer i, int offset) {
+    public void detectByStep(QueryContext queryContext, Set<HanlpMapResult> existResults, Set<Long> detectViewIds,
+            String detectSegment, int offset) {
 
     }
+
 }

chat/core/src/main/java/com/tencent/supersonic/chat/core/parser/JavaLLMProxy.java

@@ -0,0 +1,66 @@
+package com.tencent.supersonic.chat.core.parser;
+
+import com.tencent.supersonic.chat.core.parser.plugin.function.FunctionPromptGenerator;
+import com.tencent.supersonic.chat.core.parser.plugin.function.FunctionReq;
+import com.tencent.supersonic.chat.core.parser.plugin.function.FunctionResp;
+import com.tencent.supersonic.chat.core.parser.sql.llm.OutputFormat;
+import com.tencent.supersonic.chat.core.parser.sql.llm.SqlGeneration;
+import com.tencent.supersonic.chat.core.parser.sql.llm.SqlGenerationFactory;
+import com.tencent.supersonic.chat.core.pojo.QueryContext;
+import com.tencent.supersonic.chat.core.query.llm.s2sql.LLMReq;
+import com.tencent.supersonic.chat.core.query.llm.s2sql.LLMReq.SqlGenerationMode;
+import com.tencent.supersonic.chat.core.query.llm.s2sql.LLMResp;
+import com.tencent.supersonic.common.util.ContextUtils;
+import dev.langchain4j.model.chat.ChatLanguageModel;
+import lombok.extern.slf4j.Slf4j;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.stereotype.Component;
+
+import java.util.Objects;
+
+/**
+ * LLMProxy based on langchain4j Java version.
+ */
+@Slf4j
+@Component
+public class JavaLLMProxy implements LLMProxy {
+
+    private static final Logger keyPipelineLog = LoggerFactory.getLogger("keyPipeline");
+
+    @Override
+    public boolean isSkip(QueryContext queryContext) {
+        ChatLanguageModel chatLanguageModel = ContextUtils.getBean(ChatLanguageModel.class);
+        if (Objects.isNull(chatLanguageModel)) {
+            log.warn("chatLanguageModel is null, skip :{}", JavaLLMProxy.class.getName());
+            return true;
+        }
+        return false;
+    }
+
+    public LLMResp query2sql(LLMReq llmReq, Long viewId) {
+
+        SqlGeneration sqlGeneration = SqlGenerationFactory.get(
+                SqlGenerationMode.getMode(llmReq.getSqlGenerationMode()));
+        String modelName = llmReq.getSchema().getViewName();
+        LLMResp result = sqlGeneration.generation(llmReq, viewId);
+        result.setQuery(llmReq.getQueryText());
+        result.setModelName(modelName);
+        return result;
+    }
+
+    @Override
+    public FunctionResp requestFunction(FunctionReq functionReq) {
+
+        FunctionPromptGenerator promptGenerator = ContextUtils.getBean(FunctionPromptGenerator.class);
+
+        ChatLanguageModel chatLanguageModel = ContextUtils.getBean(ChatLanguageModel.class);
+        String functionCallPrompt = promptGenerator.generateFunctionCallPrompt(functionReq.getQueryText(),
+                functionReq.getPluginConfigs());
+        keyPipelineLog.info("functionCallPrompt:{}", functionCallPrompt);
+        String response = chatLanguageModel.generate(functionCallPrompt);
+        keyPipelineLog.info("functionCall response:{}", response);
+        return OutputFormat.functionCallParse(response);
+    }
+
+}

chat/core/src/main/java/com/tencent/supersonic/chat/core/parser/LLMProxy.java

@@ -0,0 +1,22 @@
+package com.tencent.supersonic.chat.core.parser;
+
+import com.tencent.supersonic.chat.core.pojo.QueryContext;
+import com.tencent.supersonic.chat.core.parser.plugin.function.FunctionReq;
+import com.tencent.supersonic.chat.core.parser.plugin.function.FunctionResp;
+import com.tencent.supersonic.chat.core.query.llm.s2sql.LLMReq;
+import com.tencent.supersonic.chat.core.query.llm.s2sql.LLMResp;
+
+/**
+ * LLMProxy encapsulates functions performed by LLMs so that multiple
+ * orchestration frameworks (e.g. LangChain in python, LangChain4j in java)
+ * could be used.
+ */
+public interface LLMProxy {
+
+    boolean isSkip(QueryContext queryContext);
+
+    LLMResp query2sql(LLMReq llmReq, Long viewId);
+
+    FunctionResp requestFunction(FunctionReq functionReq);
+
+}