diff --git a/CHANGELOG b/CHANGELOG
deleted file mode 100644
index 88e9053ba..000000000
--- a/CHANGELOG
+++ /dev/null
@@ -1,21 +0,0 @@
-davinci 0.3.0 change log
-1) add data portal
-2) add metric trend chart
-3) add feedback component
-4) add tab component
-5) add page setting
-6) modify permission process
-7) optimize css style
-8) optimize filter
-9) delete view module
-
-
-supersonic 0.6.0 change log
-1) add llm parser and llm api server
-2) support fuzzy mapping
-3) support query filter and domain filter in query and search
-4) support standalone mode
-5) add dsl query in semantic
-6) code architecture adjustment in semantic and chat
-7) add unit testing and integration testing
-8) support dimension and metric alias
\ No newline at end of file
diff --git a/CHANGELOG.md b/CHANGELOG.md
new file mode 100644
index 000000000..5ad4db1e5
--- /dev/null
+++ b/CHANGELOG.md
@@ -0,0 +1,99 @@
+# SuperSonic Changelog
+
+- All notable changes to this project will be documented in this file.
+- "Breaking Changes" describes any changes that may break existing functionality or cause
+ compatibility issues with previous versions.
+
+
+## SuperSonic [0.7.0] - 2023-07-30
+
+### Added
+
+- Add function call parser and embedding recall parser
+- Add plugin management
+- Add web page query and web service query
+- Metric filter query support querying metrics and comparing them in different dimensions
+- Support dimension value mapping
+- Support dimension/metric invisible, chat filter related data
+- Add user guide docs
+
+
+### Fixed
+
+- Fix the data problem of getDomainList interface in standalone mode
+
+## SuperSonic [0.6.0] - 2023-07-16
+
+### Added
+
+- Support llm parser and llm api server - users can query data through complex natural language.
+- Support fuzzy query dimension and metric name - users can set the 'metric.dimension.threshold'
+ parameter to control the fuzzy threshold.
+- Support query filter and domain filter in query and search - users can specify domainId and query
+ filter to filter the results in search and query.
+- Support standalone mode - users can integrate semantic and chat services in one process for easy
+ management and debugging.
+- Support dsl query in semantic - users can specify DSL language to query data in Semantic. In the
+ past, data querying was limited to struct language.
+- Add unit and integration testing - add integration tests for single-turn and multi-turn
+ conversations, to efficiently validate the code.
+- Support dimension and metric alias - users can specify one or multiple aliases to expand search
+ and query.
+- Add scheduled semantic metadata update functionality in chat.
+- Support create datasource by table name in the web page.
+- Add the ability to set permissions for domain.
+- Add a local/Remote implementation to the SemanticLayer interface.
+
+### Updated
+
+- Code architecture adjustment in chat.
+
+1) Abstracting into three modules, namely api, core, and knowledge. Providing four core interfaces:
+ SchemaMapper, SemanticLayer, SemanticParser, and SemanticQuery.
+2) Add RuleSemanticQuery and LLMSemanticQuery implement to SemanticQuery.
+3) Add all possible queries to the candidate queries, and then select the most suitable query from
+ the candidate queries.
+
+- Code architecture adjustment in semantic.
+
+1) Refactor semantic layer SQL parsing code through Calcite.
+2) Add QueryOptimizer interface.
+
+- Chat config subdivided into detailed and metric scenarios - users can set different parameters in these two scenarios.
+
+### Fixed
+
+- Resolved last word not be recognized in SchemaMapper.
+- Fix context inheritance problem.
+- Fix the error of querying H2 database by month unit.
+- Set faker user to context when authentication disable.
+
+## SuperSonic [0.5.0] - 2023-06-15
+
+### Added
+- Add the search and query feature in chat according to rules in an extensible way.
+- Add semantic/chat independent service for users.
+- Add Modeling Interface - users can visually define and maintain semantic models in the web page.
+- Add a unified semantic parsing layer - user can query data by struct language.
+
+# Davinci Changelog
+
+## Davinci [0.3.0] - 2023-06-15
+
+### Added
+
+- add data portal
+- add metric trend chart
+- add feedback component
+- add tab component
+- add page setting
+
+### Updated
+
+- modify permission process
+- optimize css style
+- optimize filter
+
+### Removed
+
+- delete view module
\ No newline at end of file
diff --git a/README.md b/README.md
index d8ce42237..ad7ed42b7 100644
--- a/README.md
+++ b/README.md
@@ -2,10 +2,9 @@ English | [中文](README_CN.md)
# SuperSonic (超音数)
-**SuperSonic is an out-of-the-box yet highly extensible framework for building a data chatbot**. SuperSonic provides a chat interface that empowers users to query data using natural language and visualize the results with suitable charts. To enable such experience, the only thing necessary is to define logical semantic models (metrics, dimensions, relationships, etc) on top of physical data models, and no data modification or copying is required. Meanwhile SuperSonic is designed to be plugable, allowing new functionalities to be added through plugins and core components to be integrated into other systems.
+**SuperSonic is an out-of-the-box yet highly extensible framework for building a data chatbot**. SuperSonic provides a chat interface that empowers users to query data using natural language and visualize the results with suitable charts. To enable such experience, the only thing necessary is to define logical semantic models (metrics, dimensions, aliases, relationships, etc) on top of physical data models, and no data modification or copying is required. Meanwhile SuperSonic is designed to be plugable, allowing new functionalities to be added through plugins and core components to be integrated into other systems.
-
-
+
## Motivation
@@ -13,9 +12,9 @@ The emergence of Large Language Models (LLMs) like ChatGPT is reshaping the way
From our perspective, the key to filling the real-world gap lies in two aspects:
1. Utilize a combination of rule-based and model-based semantic parsers to deal with different scenarios
-2. Introduce a semantic model layer to encapsulate underlying complexity thus simplify the semantic parsers
+2. Introduce a semantic model layer to encapsulate underlying complexity thus simplify the semantic parsers
-With these ideas in mind, we developed SuperSonic as a reference implementation and used it to power our real-world products. Additionally, to encourage further development of data chatbots, we decided to open source SuperSonic as an extensible framework.
+With these ideas in mind, we developed SuperSonic as a practical reference implementation and used it to power our real-world products. Additionally, to encourage further development of data chatbots, we decided to open source SuperSonic as an extensible framework.
## Out-of-the-box Features
@@ -29,9 +28,7 @@ With these ideas in mind, we developed SuperSonic as a reference implementation
SuperSonic is composed of two layers: supersonic-chat and supersonic-semantic. The chat layer is responsible for converting **natural language query** into semantic query (also known as DSL query), whereas the semantic layer is responsible for converting DSL query into **SQL query**. The high-level architecture and main process flow is shown in below diagram:
-
-
-
+
### Chat Layer
@@ -67,4 +64,12 @@ SuperSonic comes with sample semantic models as well as chat conversations that
## How to Build
-Pull the source code and run script "assembly/bin/build-standalone.sh" to build packages in the standalone mode.
+SuperSonic can be deployed in two modes: standalone (intended for quick demo) and distributed (intended for production).
+
+### Build for Standalone Mode
+
+Pull the source code and run script "assembly/bin/build-standalone.sh" to build a single packages.
+
+### Build for Distributed Mode
+
+Pull the source code and run scripts "assembly/bin/build-chat.sh" and "assembly/bin/build-semantic.sh" separately to build packages.
diff --git a/README_CN.md b/README_CN.md
index 3a4bdf4a5..f2b6e2606 100644
--- a/README_CN.md
+++ b/README_CN.md
@@ -1,8 +1,8 @@
# 超音数(SuperSonic)
-**超音数是一个开箱即用且易于扩展的数据问答对话框架**。通过超音数的问答对话界面,用户能够使用自然语言查询数据,系统会选择合适的可视化图表呈现结果。超音数不需要修改或复制数据,只需要在物理数据库之上构建逻辑语义模型(定义指标、维度、相互间关系等),即可开启数据问答体验。与此同时,超音数被设计为可插拔式框架,允许以插件形式来扩展新功能,或者将核心组件与其他系统集成。
+**超音数是一个开箱即用且易于扩展的数据问答对话框架**。通过超音数的问答对话界面,用户能够使用自然语言查询数据,系统会选择合适的可视化图表呈现结果。超音数不需要修改或复制数据,只需要在物理数据库之上构建逻辑语义模型(定义指标、维度、别名、相互间关系等),即可开启数据问答体验。与此同时,超音数被设计为可插拔式框架,允许以插件形式来扩展新功能,或者将核心组件与其他系统集成。
-
+
## 项目动机
@@ -12,7 +12,7 @@
1. 将基于规则和基于模型的语义解析器相结合,发挥各自优势,以便处理不同的场景
2. 引入语义模型层来封装数据底层的复杂性,从而简化语义解析器的问题求解空间
-为了验证上述想法,我们开发了超音数项目,并将其应用在实际的内部产品中。与此同时,我们决定将超音数作为一个可扩展的框架开源,希望能够促进数据问答对话领域的进一步发展。
+为了落地上述想法,我们开发了超音数项目,并将其应用在实际的内部产品中。与此同时,我们决定将超音数作为一个可扩展的框架开源,希望能够促进数据问答对话领域的进一步发展。
## 开箱即用的特性
@@ -26,7 +26,7 @@
超音数主要分为两层:supersonic-chat and supersonic-semantic。问答层负责将自然语言查询转换为语义查询(也称为DSL查询),而语义层负责将DSL查询转换为SQL查询。超音数的整体架构和主流程如下图所示:
-
+
### 问答层
@@ -62,4 +62,12 @@
## 如何构建
-下载源码包,运行脚本"assembly/bin/build-standalone.sh",将所有服务一起编译打包
\ No newline at end of file
+超音数可以运行在两个模式:standalone(一般用于快速演示)和distributed(一般用于生产环境)。
+
+### Standalone模式构建
+
+下载源码包,运行脚本"assembly/bin/build-standalone.sh",将所有服务一起编译打包
+
+### Distributed模式构建
+
+下载源码包,分别运行脚本"assembly/bin/build-chat.sh"、"assembly/bin/build-semantic.sh",为问答层服务和语义层服务编译打包
\ No newline at end of file
diff --git a/assembly/bin/build-ide.sh b/assembly/bin/build-ide.sh
new file mode 100755
index 000000000..b2bcd0ca3
--- /dev/null
+++ b/assembly/bin/build-ide.sh
@@ -0,0 +1,13 @@
+#!/usr/bin/env bash
+
+sbinDir=$(cd "$(dirname "$0")"; pwd)
+baseDir=$(readlink -f $sbinDir/../)
+buildDir=$baseDir/build
+
+cd $baseDir/bin
+sh build-standalone.sh
+
+cd $buildDir
+tar xvf supersonic-webapp.tar.gz
+mv supersonic-webapp webapp
+mv webapp ../../launchers/standalone/target/classes
\ No newline at end of file
diff --git a/auth/api/src/main/java/com/tencent/supersonic/auth/api/authorization/request/QueryGroupReq.java b/auth/api/src/main/java/com/tencent/supersonic/auth/api/authorization/request/QueryGroupReq.java
index e4134274f..0940fcb5b 100644
--- a/auth/api/src/main/java/com/tencent/supersonic/auth/api/authorization/request/QueryGroupReq.java
+++ b/auth/api/src/main/java/com/tencent/supersonic/auth/api/authorization/request/QueryGroupReq.java
@@ -1,7 +1,7 @@
package com.tencent.supersonic.auth.api.authorization.request;
-import com.tencent.supersonic.common.request.PageBaseReq;
+import com.tencent.supersonic.common.pojo.PageBaseReq;
import java.util.List;
import lombok.Data;
diff --git a/auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/application/UserServiceImpl.java b/auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/application/UserServiceImpl.java
index 65f01a222..b8c4d6fb1 100644
--- a/auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/application/UserServiceImpl.java
+++ b/auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/application/UserServiceImpl.java
@@ -33,14 +33,6 @@ public class UserServiceImpl implements UserService {
return userRepository.getUser(name);
}
- public boolean checkExist(UserWithPassword user) {
- UserDO userDO = getUser(user.getName());
- if (userDO == null) {
- return false;
- }
- return userDO.getPassword().equals(user.getPassword());
- }
-
@Override
public List getUserNames() {
return getUserDOList().stream().map(UserDO::getName).collect(Collectors.toList());
diff --git a/auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/domain/interceptor/AuthenticationInterceptor.java b/auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/domain/interceptor/AuthenticationInterceptor.java
index 2f0e12424..433a0ddd1 100644
--- a/auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/domain/interceptor/AuthenticationInterceptor.java
+++ b/auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/domain/interceptor/AuthenticationInterceptor.java
@@ -4,7 +4,7 @@ import com.tencent.supersonic.auth.api.authentication.config.AuthenticationConfi
import com.tencent.supersonic.auth.api.authentication.constant.UserConstants;
import com.tencent.supersonic.auth.authentication.application.UserServiceImpl;
import com.tencent.supersonic.auth.authentication.domain.utils.UserTokenUtils;
-import com.tencent.supersonic.common.util.context.S2ThreadContext;
+import com.tencent.supersonic.common.util.S2ThreadContext;
import java.lang.reflect.Field;
import java.util.Arrays;
import java.util.List;
diff --git a/auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/domain/interceptor/DefaultAuthenticationInterceptor.java b/auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/domain/interceptor/DefaultAuthenticationInterceptor.java
index c925b21bd..23a72cf49 100644
--- a/auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/domain/interceptor/DefaultAuthenticationInterceptor.java
+++ b/auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/domain/interceptor/DefaultAuthenticationInterceptor.java
@@ -6,10 +6,10 @@ import com.tencent.supersonic.auth.api.authentication.pojo.User;
import com.tencent.supersonic.auth.api.authentication.pojo.UserWithPassword;
import com.tencent.supersonic.auth.authentication.application.UserServiceImpl;
import com.tencent.supersonic.auth.authentication.domain.utils.UserTokenUtils;
-import com.tencent.supersonic.common.exception.AccessException;
-import com.tencent.supersonic.common.util.context.ContextUtils;
-import com.tencent.supersonic.common.util.context.S2ThreadContext;
-import com.tencent.supersonic.common.util.context.ThreadContext;
+import com.tencent.supersonic.common.pojo.exception.AccessException;
+import com.tencent.supersonic.common.util.ContextUtils;
+import com.tencent.supersonic.common.util.S2ThreadContext;
+import com.tencent.supersonic.common.util.ThreadContext;
import java.lang.reflect.Method;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
diff --git a/auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/domain/utils/UserTokenUtils.java b/auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/domain/utils/UserTokenUtils.java
index f25788d14..94f76b91c 100644
--- a/auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/domain/utils/UserTokenUtils.java
+++ b/auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/domain/utils/UserTokenUtils.java
@@ -13,7 +13,7 @@ import static com.tencent.supersonic.auth.api.authentication.constant.UserConsta
import com.tencent.supersonic.auth.api.authentication.config.AuthenticationConfig;
import com.tencent.supersonic.auth.api.authentication.pojo.User;
import com.tencent.supersonic.auth.api.authentication.pojo.UserWithPassword;
-import com.tencent.supersonic.common.exception.AccessException;
+import com.tencent.supersonic.common.pojo.exception.AccessException;
import io.jsonwebtoken.Claims;
import io.jsonwebtoken.Jwts;
import io.jsonwebtoken.SignatureAlgorithm;
diff --git a/auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/infrastructure/mapper/UserDOMapper.java b/auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/infrastructure/mapper/UserDOMapper.java
index 8f3cf6b58..c3b5dc43e 100644
--- a/auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/infrastructure/mapper/UserDOMapper.java
+++ b/auth/authentication/src/main/java/com/tencent/supersonic/auth/authentication/infrastructure/mapper/UserDOMapper.java
@@ -9,43 +9,14 @@ import org.apache.ibatis.annotations.Mapper;
@Mapper
public interface UserDOMapper {
- /**
- * @mbg.generated
- */
- long countByExample(UserDOExample example);
-
- /**
- * @mbg.generated
- */
- int deleteByPrimaryKey(Long id);
-
/**
* @mbg.generated
*/
int insert(UserDO record);
- /**
- * @mbg.generated
- */
- int insertSelective(UserDO record);
-
/**
* @mbg.generated
*/
List selectByExample(UserDOExample example);
- /**
- * @mbg.generated
- */
- UserDO selectByPrimaryKey(Long id);
-
- /**
- * @mbg.generated
- */
- int updateByPrimaryKeySelective(UserDO record);
-
- /**
- * @mbg.generated
- */
- int updateByPrimaryKey(UserDO record);
-}
\ No newline at end of file
+}
diff --git a/auth/authorization/src/main/java/com/tencent/supersonic/auth/authorization/application/AuthServiceImpl.java b/auth/authorization/src/main/java/com/tencent/supersonic/auth/authorization/application/AuthServiceImpl.java
index 9e7797b9f..d797b2f1e 100644
--- a/auth/authorization/src/main/java/com/tencent/supersonic/auth/authorization/application/AuthServiceImpl.java
+++ b/auth/authorization/src/main/java/com/tencent/supersonic/auth/authorization/application/AuthServiceImpl.java
@@ -124,8 +124,8 @@ public class AuthServiceImpl implements AuthService {
}
private List getAuthGroups(QueryAuthResReq req) {
- List groups = load().stream().
- filter(group -> {
+ List groups = load().stream()
+ .filter(group -> {
if (!Objects.equals(group.getDomainId(), req.getDomainId())) {
return false;
}
diff --git a/chat/api/src/main/java/com/tencent/supersonic/chat/api/component/SchemaMapper.java b/chat/api/src/main/java/com/tencent/supersonic/chat/api/component/SchemaMapper.java
index caf15515c..79c62f75f 100644
--- a/chat/api/src/main/java/com/tencent/supersonic/chat/api/component/SchemaMapper.java
+++ b/chat/api/src/main/java/com/tencent/supersonic/chat/api/component/SchemaMapper.java
@@ -1,6 +1,6 @@
package com.tencent.supersonic.chat.api.component;
-import com.tencent.supersonic.chat.api.request.QueryContextReq;
+import com.tencent.supersonic.chat.api.pojo.QueryContext;
/**
* This interface defines the contract for a schema mapper that identifies references to schema
@@ -11,5 +11,5 @@ import com.tencent.supersonic.chat.api.request.QueryContextReq;
*/
public interface SchemaMapper {
- void map(QueryContextReq queryContext);
+ void map(QueryContext queryContext);
}
diff --git a/chat/api/src/main/java/com/tencent/supersonic/chat/api/component/SemanticLayer.java b/chat/api/src/main/java/com/tencent/supersonic/chat/api/component/SemanticLayer.java
index f3c2566f1..011a7d244 100644
--- a/chat/api/src/main/java/com/tencent/supersonic/chat/api/component/SemanticLayer.java
+++ b/chat/api/src/main/java/com/tencent/supersonic/chat/api/component/SemanticLayer.java
@@ -2,10 +2,15 @@ package com.tencent.supersonic.chat.api.component;
import com.github.pagehelper.PageInfo;
import com.tencent.supersonic.auth.api.authentication.pojo.User;
-import com.tencent.supersonic.semantic.api.core.request.PageDimensionReq;
-import com.tencent.supersonic.semantic.api.core.request.PageMetricReq;
-import com.tencent.supersonic.semantic.api.core.response.*;
-import com.tencent.supersonic.semantic.api.query.request.QuerySqlReq;
+import com.tencent.supersonic.chat.api.pojo.DomainSchema;
+import com.tencent.supersonic.semantic.api.model.request.PageDimensionReq;
+import com.tencent.supersonic.semantic.api.model.request.PageMetricReq;
+import com.tencent.supersonic.semantic.api.model.response.DimensionResp;
+import com.tencent.supersonic.semantic.api.model.response.DomainResp;
+import com.tencent.supersonic.semantic.api.model.response.QueryResultWithSchemaResp;
+import com.tencent.supersonic.semantic.api.model.response.MetricResp;
+import com.tencent.supersonic.semantic.api.query.request.QueryDslReq;
+import com.tencent.supersonic.semantic.api.query.request.QueryMultiStructReq;
import com.tencent.supersonic.semantic.api.query.request.QueryStructReq;
import java.util.List;
@@ -25,27 +30,16 @@ import java.util.List;
public interface SemanticLayer {
QueryResultWithSchemaResp queryByStruct(QueryStructReq queryStructReq, User user);
+ QueryResultWithSchemaResp queryByMultiStruct(QueryMultiStructReq queryMultiStructReq, User user);
+ QueryResultWithSchemaResp queryByDsl(QueryDslReq queryDslReq, User user);
- QueryResultWithSchemaResp queryBySql(QuerySqlReq querySqlReq, User user);
-
- DomainSchemaResp getDomainSchemaInfo(Long domain, Boolean cacheEnable);
-
- List getDomainSchemaInfo(List ids);
+ List getDomainSchema();
+ List getDomainSchema(List ids);
+ DomainSchema getDomainSchema(Long domain, Boolean cacheEnable);
+ PageInfo getDimensionPage(PageDimensionReq pageDimensionCmd);
+ PageInfo getMetricPage(PageMetricReq pageMetricCmd);
List getDomainListForViewer();
-
List getDomainListForAdmin();
- PageInfo queryDimensionPage(PageDimensionReq pageDimensionCmd);
-
- PageInfo queryMetricPage(PageMetricReq pageMetricCmd);
-
-// PageInfo queryMetricPage(PageMetricReq pageMetricCmd);
-//
-// PageInfo queryDimensionPage(PageDimensionReq pageDimensionCmd);
-//
-// List getDomainListForAdmin();
-//
-// List getDomainListForViewer();
-
}
diff --git a/chat/api/src/main/java/com/tencent/supersonic/chat/api/component/SemanticParser.java b/chat/api/src/main/java/com/tencent/supersonic/chat/api/component/SemanticParser.java
index 218537502..198c9ec1b 100644
--- a/chat/api/src/main/java/com/tencent/supersonic/chat/api/component/SemanticParser.java
+++ b/chat/api/src/main/java/com/tencent/supersonic/chat/api/component/SemanticParser.java
@@ -2,7 +2,7 @@ package com.tencent.supersonic.chat.api.component;
import com.tencent.supersonic.chat.api.pojo.ChatContext;
-import com.tencent.supersonic.chat.api.request.QueryContextReq;
+import com.tencent.supersonic.chat.api.pojo.QueryContext;
/**
* This interface defines the contract for a semantic parser that can analyze natural language query
@@ -13,5 +13,5 @@ import com.tencent.supersonic.chat.api.request.QueryContextReq;
*/
public interface SemanticParser {
- void parse(QueryContextReq queryContext, ChatContext chatContext);
+ void parse(QueryContext queryContext, ChatContext chatContext);
}
diff --git a/chat/api/src/main/java/com/tencent/supersonic/chat/api/component/SemanticQuery.java b/chat/api/src/main/java/com/tencent/supersonic/chat/api/component/SemanticQuery.java
index 88429f69e..f8ddf147a 100644
--- a/chat/api/src/main/java/com/tencent/supersonic/chat/api/component/SemanticQuery.java
+++ b/chat/api/src/main/java/com/tencent/supersonic/chat/api/component/SemanticQuery.java
@@ -2,7 +2,8 @@ package com.tencent.supersonic.chat.api.component;
import com.tencent.supersonic.auth.api.authentication.pojo.User;
import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
-import com.tencent.supersonic.chat.api.response.QueryResultResp;
+import com.tencent.supersonic.chat.api.pojo.response.QueryResult;
+import org.apache.calcite.sql.parser.SqlParseException;
/**
* This class defines the contract for a semantic query that executes specific type of
@@ -12,7 +13,7 @@ public interface SemanticQuery {
String getQueryMode();
- QueryResultResp execute(User user);
+ QueryResult execute(User user) throws SqlParseException;
SemanticParseInfo getParseInfo();
}
diff --git a/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/DomainSchema.java b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/DomainSchema.java
new file mode 100644
index 000000000..d1aeb1855
--- /dev/null
+++ b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/DomainSchema.java
@@ -0,0 +1,45 @@
+package com.tencent.supersonic.chat.api.pojo;
+
+import lombok.Data;
+
+import java.util.HashSet;
+import java.util.Optional;
+import java.util.Set;
+
+@Data
+public class DomainSchema {
+
+ private SchemaElement domain;
+ private Set metrics = new HashSet<>();
+ private Set dimensions = new HashSet<>();
+ private Set dimensionValues = new HashSet<>();
+ private Set entities = new HashSet<>();
+
+ public SchemaElement getElement(SchemaElementType elementType, long elementID) {
+ Optional element = Optional.empty();
+ switch (elementType) {
+ case DOMAIN:
+ element = Optional.of(domain);
+ break;
+ case METRIC:
+ element = metrics.stream().filter(e -> e.getId() == elementID).findFirst();
+ break;
+ case DIMENSION:
+ element = dimensions.stream().filter(e -> e.getId() == elementID).findFirst();
+ break;
+ case ENTITY:
+ element = entities.stream().filter(e -> e.getId() == elementID).findFirst();
+ break;
+ case VALUE:
+ element = dimensionValues.stream().filter(e -> e.getId() == elementID).findFirst();
+ default:
+ }
+
+ if (element.isPresent()) {
+ return element.get();
+ } else {
+ return null;
+ }
+ }
+
+}
diff --git a/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/QueryContext.java b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/QueryContext.java
new file mode 100644
index 000000000..ba92177af
--- /dev/null
+++ b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/QueryContext.java
@@ -0,0 +1,20 @@
+package com.tencent.supersonic.chat.api.pojo;
+
+import com.tencent.supersonic.chat.api.component.SemanticQuery;
+import com.tencent.supersonic.chat.api.pojo.request.QueryRequest;
+import lombok.Data;
+
+import java.util.ArrayList;
+import java.util.List;
+
+@Data
+public class QueryContext {
+
+ private QueryRequest request;
+ private List candidateQueries = new ArrayList<>();
+ private SchemaMapInfo mapInfo = new SchemaMapInfo();
+
+ public QueryContext(QueryRequest request) {
+ this.request = request;
+ }
+}
diff --git a/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/QueryMatchInfo.java b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/QueryMatchInfo.java
deleted file mode 100644
index 7e866f050..000000000
--- a/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/QueryMatchInfo.java
+++ /dev/null
@@ -1,12 +0,0 @@
-package com.tencent.supersonic.chat.api.pojo;
-
-import lombok.Data;
-
-@Data
-public class QueryMatchInfo {
-
- SchemaElementType elementType;
- String detectWord;
- private Integer count = 0;
- private double maxSimilarity;
-}
diff --git a/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/SchemaElement.java b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/SchemaElement.java
new file mode 100644
index 000000000..0907c25c9
--- /dev/null
+++ b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/SchemaElement.java
@@ -0,0 +1,56 @@
+package com.tencent.supersonic.chat.api.pojo;
+
+import com.google.common.base.Objects;
+
+import java.io.Serializable;
+import java.util.List;
+
+import lombok.Builder;
+import lombok.Data;
+
+@Data
+@Builder
+public class SchemaElement implements Serializable {
+
+ private Long domain;
+ private Long id;
+ private String name;
+ private String bizName;
+ private Long useCnt;
+ private SchemaElementType type;
+ private List alias;
+
+ public SchemaElement() {
+ }
+
+ public SchemaElement(Long domain, Long id, String name, String bizName,
+ Long useCnt, SchemaElementType type, List alias) {
+ this.domain = domain;
+ this.id = id;
+ this.name = name;
+ this.bizName = bizName;
+ this.useCnt = useCnt;
+ this.type = type;
+ this.alias = alias;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ SchemaElement schemaElement = (SchemaElement) o;
+ return Objects.equal(domain, schemaElement.domain) && Objects.equal(id,
+ schemaElement.id) && Objects.equal(name, schemaElement.name)
+ && Objects.equal(bizName, schemaElement.bizName) && Objects.equal(
+ useCnt, schemaElement.useCnt) && Objects.equal(type, schemaElement.type);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hashCode(domain, id, name, bizName, useCnt, type);
+ }
+}
diff --git a/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/SchemaElementMatch.java b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/SchemaElementMatch.java
index 592ee3bbe..605023199 100644
--- a/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/SchemaElementMatch.java
+++ b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/SchemaElementMatch.java
@@ -13,15 +13,9 @@ import lombok.ToString;
@NoArgsConstructor
public class SchemaElementMatch {
- SchemaElementType elementType;
-
- int elementID;
-
+ SchemaElement element;
double similarity;
-
String detectWord;
-
String word;
-
Long frequency;
}
diff --git a/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/SchemaMapInfo.java b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/SchemaMapInfo.java
index 66f8c2ca3..588d93f02 100644
--- a/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/SchemaMapInfo.java
+++ b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/SchemaMapInfo.java
@@ -7,26 +7,21 @@ import java.util.Set;
public class SchemaMapInfo {
- private Map> domainElementMatches = new HashMap<>();
+ private Map> domainElementMatches = new HashMap<>();
- public Set getMatchedDomains() {
+ public Set getMatchedDomains() {
return domainElementMatches.keySet();
}
- public List getMatchedElements(Integer domain) {
+ public List getMatchedElements(Long domain) {
return domainElementMatches.get(domain);
}
- public Map> getDomainElementMatches() {
+ public Map> getDomainElementMatches() {
return domainElementMatches;
}
- public void setDomainElementMatches(
- Map> domainElementMatches) {
- this.domainElementMatches = domainElementMatches;
- }
-
- public void setMatchedElements(Integer domain, List elementMatches) {
+ public void setMatchedElements(Long domain, List elementMatches) {
domainElementMatches.put(domain, elementMatches);
}
}
diff --git a/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/SemanticParseInfo.java b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/SemanticParseInfo.java
index 03adf3897..b0059cfc7 100644
--- a/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/SemanticParseInfo.java
+++ b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/SemanticParseInfo.java
@@ -1,34 +1,55 @@
package com.tencent.supersonic.chat.api.pojo;
-import com.tencent.supersonic.common.enums.AggregateTypeEnum;
+import com.tencent.supersonic.chat.api.pojo.request.QueryFilter;
import com.tencent.supersonic.common.pojo.DateConf;
import com.tencent.supersonic.common.pojo.Order;
-import com.tencent.supersonic.common.pojo.SchemaItem;
-
+import com.tencent.supersonic.common.pojo.enums.AggregateTypeEnum;
import java.util.ArrayList;
import java.util.LinkedHashSet;
import java.util.List;
+import java.util.Map;
import java.util.Set;
+import java.util.stream.Collectors;
import lombok.Data;
@Data
public class SemanticParseInfo {
String queryMode;
- AggregateTypeEnum aggType = AggregateTypeEnum.NONE;
- Long domainId = 0L;
- String domainName;
+ SchemaElement domain;
+ Set metrics = new LinkedHashSet();
+ Set dimensions = new LinkedHashSet();
Long entity = 0L;
- Set metrics = new LinkedHashSet();
- Set dimensions = new LinkedHashSet();
- Set dimensionFilters = new LinkedHashSet();
- Set metricFilters = new LinkedHashSet();
+ AggregateTypeEnum aggType = AggregateTypeEnum.NONE;
+ Set dimensionFilters = new LinkedHashSet();
+ Set metricFilters = new LinkedHashSet();
private Set orders = new LinkedHashSet();
private DateConf dateInfo;
private Long limit;
private Boolean nativeQuery = false;
private Double bonus = 0d;
private List elementMatches = new ArrayList<>();
- private Object info;
+ private Map properties;
+
+ public Long getDomainId() {
+ return domain != null ? domain.getId() : 0L;
+ }
+
+ public String getDomainName() {
+ return domain != null ? domain.getName() : "null";
+ }
+
+ public Set getMetrics() {
+ this.metrics = this.metrics.stream().sorted((o1, o2) -> {
+ int len1 = o1.getName().length();
+ int len2 = o2.getName().length();
+ if (len1 != len2) {
+ return len1 - len2;
+ } else {
+ return o1.getName().compareTo(o2.getName());
+ }
+ }).collect(Collectors.toCollection(LinkedHashSet::new));
+ return this.metrics;
+ }
}
diff --git a/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/SemanticSchema.java b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/SemanticSchema.java
new file mode 100644
index 000000000..7e199f00d
--- /dev/null
+++ b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/SemanticSchema.java
@@ -0,0 +1,54 @@
+package com.tencent.supersonic.chat.api.pojo;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class SemanticSchema implements Serializable {
+ private List domainSchemaList;
+
+ public SemanticSchema(List domainSchemaList) {
+ this.domainSchemaList = domainSchemaList;
+ }
+
+ public void add(DomainSchema schema) {
+ domainSchemaList.add(schema);
+ }
+
+ public Map getDomainIdToName() {
+ return domainSchemaList.stream()
+ .collect(Collectors.toMap(a -> a.getDomain().getId(), a -> a.getDomain().getName(), (k1, k2) -> k1));
+ }
+
+ public List getDimensionValues() {
+ List dimensionValues = new ArrayList<>();
+ domainSchemaList.stream().forEach(d -> dimensionValues.addAll(d.getDimensionValues()));
+ return dimensionValues;
+ }
+
+ public List getDimensions() {
+ List dimensions = new ArrayList<>();
+ domainSchemaList.stream().forEach(d -> dimensions.addAll(d.getDimensions()));
+ return dimensions;
+ }
+
+ public List getMetrics() {
+ List metrics = new ArrayList<>();
+ domainSchemaList.stream().forEach(d -> metrics.addAll(d.getMetrics()));
+ return metrics;
+ }
+
+ public List getDomains() {
+ List domains = new ArrayList<>();
+ domainSchemaList.stream().forEach(d -> domains.add(d.getDomain()));
+ return domains;
+ }
+
+ public List getEntities() {
+ List entities = new ArrayList<>();
+ domainSchemaList.stream().forEach(d -> entities.addAll(d.getEntities()));
+ return entities;
+ }
+}
diff --git a/chat/core/src/main/java/com/tencent/supersonic/chat/domain/pojo/chat/PageQueryInfoReq.java b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/PageQueryInfoReq.java
similarity index 91%
rename from chat/core/src/main/java/com/tencent/supersonic/chat/domain/pojo/chat/PageQueryInfoReq.java
rename to chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/PageQueryInfoReq.java
index a6b092ae1..873c9cfd6 100644
--- a/chat/core/src/main/java/com/tencent/supersonic/chat/domain/pojo/chat/PageQueryInfoReq.java
+++ b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/PageQueryInfoReq.java
@@ -1,4 +1,4 @@
-package com.tencent.supersonic.chat.domain.pojo.chat;
+package com.tencent.supersonic.chat.api.pojo.request;
import lombok.Data;
diff --git a/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/PluginQueryReq.java b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/PluginQueryReq.java
new file mode 100644
index 000000000..f3ba04966
--- /dev/null
+++ b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/PluginQueryReq.java
@@ -0,0 +1,22 @@
+package com.tencent.supersonic.chat.api.pojo.request;
+
+
+import lombok.Data;
+
+@Data
+public class PluginQueryReq {
+
+
+ private String showElementId;
+
+ //DASHBOARD WIDGET
+ private String showType;
+
+ private String type;
+
+ private String domain;
+
+ private String pattern;
+
+
+}
diff --git a/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/QueryDataRequest.java b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/QueryDataRequest.java
new file mode 100644
index 000000000..73ef07130
--- /dev/null
+++ b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/QueryDataRequest.java
@@ -0,0 +1,24 @@
+package com.tencent.supersonic.chat.api.pojo.request;
+
+
+import java.util.HashSet;
+import java.util.Set;
+
+import com.tencent.supersonic.chat.api.pojo.SchemaElement;
+import com.tencent.supersonic.common.pojo.DateConf;
+import com.tencent.supersonic.common.pojo.Order;
+import lombok.Data;
+
+@Data
+public class QueryDataRequest {
+ String queryMode;
+ SchemaElement domain;
+ Set metrics = new HashSet<>();
+ Set dimensions = new HashSet<>();
+ Set dimensionFilters = new HashSet<>();
+ Set metricFilters = new HashSet<>();
+ private Set orders = new HashSet<>();
+ private DateConf dateInfo;
+ private Long limit;
+ private Boolean nativeQuery = false;
+}
diff --git a/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/Filter.java b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/QueryFilter.java
similarity index 88%
rename from chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/Filter.java
rename to chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/QueryFilter.java
index 5b5d7a8aa..8a2054b95 100644
--- a/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/Filter.java
+++ b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/QueryFilter.java
@@ -1,4 +1,4 @@
-package com.tencent.supersonic.chat.api.pojo;
+package com.tencent.supersonic.chat.api.pojo.request;
import com.tencent.supersonic.semantic.api.query.enums.FilterOperatorEnum;
import java.util.Objects;
@@ -7,7 +7,7 @@ import lombok.ToString;
@Data
@ToString(callSuper = true)
-public class Filter {
+public class QueryFilter {
private String bizName;
@@ -27,7 +27,7 @@ public class Filter {
if (o == null || getClass() != o.getClass()) {
return false;
}
- Filter filter = (Filter) o;
+ QueryFilter filter = (QueryFilter) o;
return Objects.equals(bizName, filter.bizName) && Objects.equals(name, filter.name)
&& operator == filter.operator && Objects.equals(value, filter.value) && Objects.equals(
elementID, filter.elementID);
diff --git a/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/QueryFilter.java b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/QueryFilters.java
similarity index 57%
rename from chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/QueryFilter.java
rename to chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/QueryFilters.java
index c1bb4c4ac..903288cd1 100644
--- a/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/QueryFilter.java
+++ b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/QueryFilters.java
@@ -1,4 +1,4 @@
-package com.tencent.supersonic.chat.api.pojo;
+package com.tencent.supersonic.chat.api.pojo.request;
import lombok.Data;
import java.util.ArrayList;
@@ -7,10 +7,7 @@ import java.util.List;
import java.util.Map;
@Data
-public class QueryFilter {
-
- private List filters = new ArrayList<>();
-
+public class QueryFilters {
+ private List filters = new ArrayList<>();
private Map params = new HashMap<>();
-
}
diff --git a/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/QueryRequest.java b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/QueryRequest.java
new file mode 100644
index 000000000..d51ccee2f
--- /dev/null
+++ b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/QueryRequest.java
@@ -0,0 +1,15 @@
+package com.tencent.supersonic.chat.api.pojo.request;
+
+import com.tencent.supersonic.auth.api.authentication.pojo.User;
+import lombok.Data;
+
+@Data
+public class QueryRequest {
+
+ private String queryText;
+ private Integer chatId;
+ private Long domainId = 0L;
+ private User user;
+ private QueryFilters queryFilters;
+ private boolean saveAnswer = true;
+}
diff --git a/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/RecommendedQuestion.java b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/RecommendedQuestion.java
new file mode 100644
index 000000000..0aa8a6bdf
--- /dev/null
+++ b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/request/RecommendedQuestion.java
@@ -0,0 +1,12 @@
+package com.tencent.supersonic.chat.api.pojo.request;
+
+import lombok.Data;
+import lombok.ToString;
+
+@Data
+@ToString
+public class RecommendedQuestion {
+
+ private String question;
+
+}
\ No newline at end of file
diff --git a/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/AggregateInfo.java b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/AggregateInfo.java
new file mode 100644
index 000000000..96cb05e13
--- /dev/null
+++ b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/AggregateInfo.java
@@ -0,0 +1,10 @@
+package com.tencent.supersonic.chat.api.pojo.response;
+
+import java.util.ArrayList;
+import java.util.List;
+import lombok.Data;
+
+@Data
+public class AggregateInfo {
+ private List metricInfos = new ArrayList<>();
+}
diff --git a/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/DataInfo.java b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/DataInfo.java
similarity index 74%
rename from chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/DataInfo.java
rename to chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/DataInfo.java
index 07a452f9d..7b22daf73 100644
--- a/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/DataInfo.java
+++ b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/DataInfo.java
@@ -1,4 +1,4 @@
-package com.tencent.supersonic.chat.api.pojo;
+package com.tencent.supersonic.chat.api.pojo.response;
import lombok.Data;
diff --git a/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/DomainInfo.java b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/DomainInfo.java
similarity index 80%
rename from chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/DomainInfo.java
rename to chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/DomainInfo.java
index 36073cf9c..2be895e86 100644
--- a/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/DomainInfo.java
+++ b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/DomainInfo.java
@@ -1,4 +1,4 @@
-package com.tencent.supersonic.chat.api.pojo;
+package com.tencent.supersonic.chat.api.pojo.response;
import java.io.Serializable;
import java.util.List;
diff --git a/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/EntityInfo.java b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/EntityInfo.java
similarity index 84%
rename from chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/EntityInfo.java
rename to chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/EntityInfo.java
index 329704688..0f0039380 100644
--- a/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/EntityInfo.java
+++ b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/EntityInfo.java
@@ -1,4 +1,4 @@
-package com.tencent.supersonic.chat.api.pojo;
+package com.tencent.supersonic.chat.api.pojo.response;
import java.util.ArrayList;
import java.util.List;
diff --git a/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/MetricInfo.java b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/MetricInfo.java
new file mode 100644
index 000000000..ef9264145
--- /dev/null
+++ b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/MetricInfo.java
@@ -0,0 +1,14 @@
+package com.tencent.supersonic.chat.api.pojo.response;
+
+import java.util.Map;
+import lombok.Data;
+
+@Data
+public class MetricInfo {
+
+ private String name;
+ private String value;
+ private String date;
+ private Map statistics;
+
+}
diff --git a/chat/core/src/main/java/com/tencent/supersonic/chat/domain/pojo/chat/ChatQueryVO.java b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/QueryResponse.java
similarity index 54%
rename from chat/core/src/main/java/com/tencent/supersonic/chat/domain/pojo/chat/ChatQueryVO.java
rename to chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/QueryResponse.java
index d32120313..2ab39fb93 100644
--- a/chat/core/src/main/java/com/tencent/supersonic/chat/domain/pojo/chat/ChatQueryVO.java
+++ b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/QueryResponse.java
@@ -1,11 +1,10 @@
-package com.tencent.supersonic.chat.domain.pojo.chat;
+package com.tencent.supersonic.chat.api.pojo.response;
-import com.tencent.supersonic.chat.api.response.QueryResultResp;
import java.util.Date;
import lombok.Data;
@Data
-public class ChatQueryVO {
+public class QueryResponse {
private Long questionId;
private Date createTime;
@@ -13,5 +12,5 @@ public class ChatQueryVO {
private Integer score;
private String feedback;
private String queryText;
- private QueryResultResp queryResponse;
+ private QueryResult queryResult;
}
\ No newline at end of file
diff --git a/chat/api/src/main/java/com/tencent/supersonic/chat/api/response/QueryResultResp.java b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/QueryResult.java
similarity index 61%
rename from chat/api/src/main/java/com/tencent/supersonic/chat/api/response/QueryResultResp.java
rename to chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/QueryResult.java
index 6ed1fc587..3858f64f2 100644
--- a/chat/api/src/main/java/com/tencent/supersonic/chat/api/response/QueryResultResp.java
+++ b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/QueryResult.java
@@ -1,21 +1,21 @@
-package com.tencent.supersonic.chat.api.response;
+package com.tencent.supersonic.chat.api.pojo.response;
-import com.tencent.supersonic.chat.api.pojo.EntityInfo;
import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
-import com.tencent.supersonic.semantic.api.core.pojo.QueryAuthorization;
-import com.tencent.supersonic.semantic.api.core.pojo.QueryColumn;
+import com.tencent.supersonic.common.pojo.QueryAuthorization;
+import com.tencent.supersonic.common.pojo.QueryColumn;
import java.util.List;
import java.util.Map;
import lombok.Data;
@Data
-public class QueryResultResp {
+public class QueryResult {
public EntityInfo entityInfo;
+ public AggregateInfo aggregateInfo;
private Long queryId;
private String queryMode;
private String querySql;
- private int queryState;
+ private QueryState queryState = QueryState.EMPTY;
private List queryColumns;
private QueryAuthorization queryAuthorization;
private SemanticParseInfo chatContext;
diff --git a/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/QueryState.java b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/QueryState.java
new file mode 100644
index 000000000..30ecc1e2c
--- /dev/null
+++ b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/QueryState.java
@@ -0,0 +1,8 @@
+package com.tencent.supersonic.chat.api.pojo.response;
+
+public enum QueryState {
+ SUCCESS,
+ SEARCH_EXCEPTION,
+ EMPTY,
+ INVALID;
+}
diff --git a/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/RecommendQuestion.java b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/RecommendQuestion.java
new file mode 100644
index 000000000..67cec7302
--- /dev/null
+++ b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/RecommendQuestion.java
@@ -0,0 +1,14 @@
+package com.tencent.supersonic.chat.api.pojo.response;
+
+import com.tencent.supersonic.chat.api.pojo.request.RecommendedQuestion;
+import lombok.AllArgsConstructor;
+import lombok.Data;
+
+import java.util.List;
+
+@Data
+@AllArgsConstructor
+public class RecommendQuestion {
+ private Long domainId;
+ private List recommendedQuestions;
+}
diff --git a/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/RecommendResponse.java b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/RecommendResponse.java
new file mode 100644
index 000000000..89fd38a34
--- /dev/null
+++ b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/RecommendResponse.java
@@ -0,0 +1,12 @@
+package com.tencent.supersonic.chat.api.pojo.response;
+
+import com.tencent.supersonic.chat.api.pojo.SchemaElement;
+import lombok.Data;
+
+import java.util.List;
+
+@Data
+public class RecommendResponse {
+ private List dimensions;
+ private List metrics;
+}
diff --git a/chat/core/src/main/java/com/tencent/supersonic/chat/domain/pojo/search/SearchResponse.java b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/SearchResponse.java
similarity index 79%
rename from chat/core/src/main/java/com/tencent/supersonic/chat/domain/pojo/search/SearchResponse.java
rename to chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/SearchResponse.java
index 7da6c4092..5f6b406ee 100644
--- a/chat/core/src/main/java/com/tencent/supersonic/chat/domain/pojo/search/SearchResponse.java
+++ b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/SearchResponse.java
@@ -1,4 +1,4 @@
-package com.tencent.supersonic.chat.domain.pojo.search;
+package com.tencent.supersonic.chat.api.pojo.response;
import java.util.List;
import lombok.Data;
@@ -6,8 +6,6 @@ import lombok.Getter;
import lombok.Setter;
@Data
-@Setter
-@Getter
public class SearchResponse {
private List searchResults;
diff --git a/chat/core/src/main/java/com/tencent/supersonic/chat/domain/pojo/search/SearchResult.java b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/SearchResult.java
similarity index 54%
rename from chat/core/src/main/java/com/tencent/supersonic/chat/domain/pojo/search/SearchResult.java
rename to chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/SearchResult.java
index cd0641073..3c0c6a7a1 100644
--- a/chat/core/src/main/java/com/tencent/supersonic/chat/domain/pojo/search/SearchResult.java
+++ b/chat/api/src/main/java/com/tencent/supersonic/chat/api/pojo/response/SearchResult.java
@@ -1,7 +1,8 @@
-package com.tencent.supersonic.chat.domain.pojo.search;
+package com.tencent.supersonic.chat.api.pojo.response;
import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
import java.util.Objects;
+import lombok.Builder;
import lombok.Data;
import lombok.Getter;
import lombok.Setter;
@@ -9,6 +10,7 @@ import lombok.Setter;
@Data
@Setter
@Getter
+@Builder
public class SearchResult {
private String recommend;
@@ -17,29 +19,12 @@ public class SearchResult {
private String domainName;
- private Integer domainId;
+ private Long domainId;
private SchemaElementType schemaElementType;
private boolean isComplete = true;
- public SearchResult(String recommend, String subRecommend, String className, Integer domainId,
- SchemaElementType schemaElementType) {
- this.recommend = recommend;
- this.subRecommend = subRecommend;
- this.domainName = className;
- this.domainId = domainId;
- this.schemaElementType = schemaElementType;
- }
-
- public SearchResult(String recommend, String subRecommend, String className, Integer domainId, boolean isComplete) {
- this.recommend = recommend;
- this.subRecommend = subRecommend;
- this.domainName = className;
- this.domainId = domainId;
- this.isComplete = isComplete;
- }
-
@Override
public boolean equals(Object o) {
if (this == o) {
diff --git a/chat/api/src/main/java/com/tencent/supersonic/chat/api/request/QueryContextReq.java b/chat/api/src/main/java/com/tencent/supersonic/chat/api/request/QueryContextReq.java
deleted file mode 100644
index 87622822f..000000000
--- a/chat/api/src/main/java/com/tencent/supersonic/chat/api/request/QueryContextReq.java
+++ /dev/null
@@ -1,23 +0,0 @@
-package com.tencent.supersonic.chat.api.request;
-
-import com.tencent.supersonic.auth.api.authentication.pojo.User;
-import com.tencent.supersonic.chat.api.component.SemanticQuery;
-import com.tencent.supersonic.chat.api.pojo.QueryFilter;
-import com.tencent.supersonic.chat.api.pojo.SchemaMapInfo;
-import lombok.Data;
-
-import java.util.ArrayList;
-import java.util.List;
-
-@Data
-public class QueryContextReq {
-
- private String queryText;
- private Integer chatId;
- private Integer domainId = 0;
- private User user;
- private QueryFilter queryFilter;
- private List candidateQueries = new ArrayList<>();
- private SchemaMapInfo mapInfo = new SchemaMapInfo();
- private boolean saveAnswer = true;
-}
diff --git a/chat/core/pom.xml b/chat/core/pom.xml
index 63369ba78..f30672455 100644
--- a/chat/core/pom.xml
+++ b/chat/core/pom.xml
@@ -113,12 +113,6 @@
semantic-api
${project.version}
-
-
-
-
-
-
com.tencent.supersonic
semantic-query
@@ -137,6 +131,12 @@
${project.version}
compile
+
+
+ com.github.xkzhangsan
+ xk-time
+ ${xk.time.version}
+
diff --git a/chat/core/src/main/java/com/tencent/supersonic/chat/application/DomainEntityService.java b/chat/core/src/main/java/com/tencent/supersonic/chat/application/DomainEntityService.java
deleted file mode 100644
index 41cc10d6a..000000000
--- a/chat/core/src/main/java/com/tencent/supersonic/chat/application/DomainEntityService.java
+++ /dev/null
@@ -1,213 +0,0 @@
-package com.tencent.supersonic.chat.application;
-
-
-import com.tencent.supersonic.auth.api.authentication.pojo.User;
-import com.tencent.supersonic.chat.api.component.SemanticLayer;
-import com.tencent.supersonic.chat.api.pojo.DataInfo;
-import com.tencent.supersonic.chat.api.pojo.DomainInfo;
-import com.tencent.supersonic.chat.api.pojo.EntityInfo;
-import com.tencent.supersonic.chat.api.pojo.Filter;
-import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
-import com.tencent.supersonic.chat.api.component.SemanticLayer;
-import com.tencent.supersonic.chat.domain.pojo.config.ChatConfigRichResp;
-import com.tencent.supersonic.chat.domain.pojo.config.ChatDefaultRichConfig;
-import com.tencent.supersonic.chat.domain.pojo.config.EntityRichInfo;
-import com.tencent.supersonic.chat.domain.utils.ComponentFactory;
-import com.tencent.supersonic.chat.domain.utils.SchemaInfoConverter;
-import com.tencent.supersonic.common.pojo.DateConf;
-import com.tencent.supersonic.common.pojo.SchemaItem;
-import com.tencent.supersonic.semantic.api.core.response.QueryResultWithSchemaResp;
-import com.tencent.supersonic.semantic.api.query.enums.FilterOperatorEnum;
-import java.util.ArrayList;
-import java.util.LinkedHashSet;
-import java.util.List;
-import java.util.Map;
-import java.util.Objects;
-import java.util.Set;
-import lombok.extern.slf4j.Slf4j;
-import org.springframework.beans.factory.annotation.Autowired;
-import org.springframework.stereotype.Service;
-import org.springframework.util.CollectionUtils;
-
-@Service
-@Slf4j
-public class DomainEntityService {
-
- private SemanticLayer semanticLayer = ComponentFactory.getSemanticLayer();
-
- @Autowired
- private ConfigServiceImpl configService;
-
- public EntityInfo getEntityInfo(SemanticParseInfo parseInfo, User user) {
- if (parseInfo != null && parseInfo.getDomainId() > 0) {
- EntityInfo entityInfo = getEntityInfo(parseInfo.getDomainId());
- if (parseInfo.getDimensionFilters().size() <= 0) {
- entityInfo.setMetrics(null);
- entityInfo.setDimensions(null);
- return entityInfo;
- }
- if (entityInfo.getDomainInfo() != null && entityInfo.getDomainInfo().getPrimaryEntityBizName() != null) {
- String domainInfoPrimaryName = entityInfo.getDomainInfo().getPrimaryEntityBizName();
- String domainInfoId = "";
- for (Filter chatFilter : parseInfo.getDimensionFilters()) {
- if (chatFilter != null && chatFilter.getBizName() != null && chatFilter.getBizName()
- .equals(domainInfoPrimaryName)) {
- if (chatFilter.getOperator().equals(FilterOperatorEnum.EQUALS)) {
- domainInfoId = chatFilter.getValue().toString();
- }
- if (chatFilter.getOperator().equals(FilterOperatorEnum.IN)) {
- domainInfoId = ((List) chatFilter.getValue()).get(0);
- }
- }
- }
- if (!"".equals(domainInfoId)) {
- try {
- setMainDomain(entityInfo, parseInfo.getDomainId(),
- domainInfoId, user);
-
- return entityInfo;
- } catch (Exception e) {
- log.error("setMaintDomain error {}", e);
- }
- }
- }
- }
- return null;
- }
-
- public EntityInfo getEntityInfo(Long domain) {
- ChatConfigRichResp chaConfigRichDesc = configService.getConfigRichInfo(domain);
- if (Objects.isNull(chaConfigRichDesc) || Objects.isNull(chaConfigRichDesc.getChatDetailRichConfig())) {
- return new EntityInfo();
- }
- return getEntityInfo(chaConfigRichDesc);
- }
-
- private EntityInfo getEntityInfo(ChatConfigRichResp chaConfigRichDesc) {
-
- EntityInfo entityInfo = new EntityInfo();
- EntityRichInfo entityDesc = chaConfigRichDesc.getChatDetailRichConfig().getEntity();
- if (entityDesc != null && Objects.nonNull(chaConfigRichDesc.getDomainId())) {
- DomainInfo domainInfo = new DomainInfo();
- domainInfo.setItemId(Integer.valueOf(chaConfigRichDesc.getDomainId().intValue()));
- domainInfo.setName(chaConfigRichDesc.getDomainName());
- domainInfo.setWords(entityDesc.getNames());
- domainInfo.setBizName(chaConfigRichDesc.getBizName());
- if (Objects.nonNull(entityDesc.getDimItem())) {
- domainInfo.setPrimaryEntityBizName(entityDesc.getDimItem().getBizName());
- }
-
- entityInfo.setDomainInfo(domainInfo);
- List dimensions = new ArrayList<>();
- List metrics = new ArrayList<>();
-
- if (Objects.nonNull(chaConfigRichDesc) && Objects.nonNull(chaConfigRichDesc.getChatDetailRichConfig())
- && Objects.nonNull(chaConfigRichDesc.getChatDetailRichConfig().getChatDefaultConfig())) {
- ChatDefaultRichConfig chatDefaultConfig = chaConfigRichDesc.getChatDetailRichConfig().getChatDefaultConfig();
- if(!CollectionUtils.isEmpty(chatDefaultConfig.getDimensions())){
- for (SchemaItem dimensionDesc : chatDefaultConfig.getDimensions()) {
- DataInfo mainEntityDimension = new DataInfo();
- mainEntityDimension.setItemId(dimensionDesc.getId().intValue());
- mainEntityDimension.setName(dimensionDesc.getName());
- mainEntityDimension.setBizName(dimensionDesc.getBizName());
- dimensions.add(mainEntityDimension);
- }
- entityInfo.setDimensions(dimensions);
- }
-
- if(!CollectionUtils.isEmpty(chatDefaultConfig.getMetrics())){
- for (SchemaItem metricDesc : chatDefaultConfig.getMetrics()) {
- DataInfo dataInfo = new DataInfo();
- dataInfo.setName(metricDesc.getName());
- dataInfo.setBizName(metricDesc.getBizName());
- dataInfo.setItemId(metricDesc.getId().intValue());
- metrics.add(dataInfo);
- }
- entityInfo.setMetrics(metrics);
- }
- }
- }
- return entityInfo;
- }
-
- public void setMainDomain(EntityInfo domainInfo, Long domain, String entity, User user) {
- domainInfo.setEntityId(entity);
- SemanticParseInfo semanticParseInfo = new SemanticParseInfo();
- semanticParseInfo.setDomainId(Long.valueOf(domain));
- semanticParseInfo.setNativeQuery(true);
- semanticParseInfo.setMetrics(getMetrics(domainInfo));
- semanticParseInfo.setDimensions(getDimensions(domainInfo));
- DateConf dateInfo = new DateConf();
- dateInfo.setUnit(1);
- dateInfo.setDateMode(DateConf.DateMode.RECENT_UNITS);
- semanticParseInfo.setDateInfo(dateInfo);
-
- // add filter
- Filter chatFilter = new Filter();
- chatFilter.setValue(String.valueOf(entity));
- chatFilter.setOperator(FilterOperatorEnum.EQUALS);
- chatFilter.setBizName(getEntityPrimaryName(domainInfo));
- Set chatFilters = new LinkedHashSet();
- chatFilters.add(chatFilter);
- semanticParseInfo.setDimensionFilters(chatFilters);
-
- QueryResultWithSchemaResp queryResultWithColumns = null;
- try {
- queryResultWithColumns = semanticLayer.queryByStruct(SchemaInfoConverter.convertTo(semanticParseInfo),
- user);
- } catch (Exception e) {
- log.warn("setMainDomain queryByStruct error, e:", e);
- }
-
- if (queryResultWithColumns != null) {
- if (!CollectionUtils.isEmpty(queryResultWithColumns.getResultList())
- && queryResultWithColumns.getResultList().size() > 0) {
- Map result = queryResultWithColumns.getResultList().get(0);
- for (Map.Entry entry : result.entrySet()) {
- String entryKey = getEntryKey(entry);
- if (entry.getValue() == null || entryKey == null) {
- continue;
- }
- domainInfo.getDimensions().stream().filter(i -> entryKey.equals(i.getBizName()))
- .forEach(i -> i.setValue(entry.getValue().toString()));
- domainInfo.getMetrics().stream().filter(i -> entryKey.equals(i.getBizName()))
- .forEach(i -> i.setValue(entry.getValue().toString()));
- }
- }
- }
- }
-
- private Set getDimensions(EntityInfo domainInfo) {
- Set dimensions = new LinkedHashSet();
- for (DataInfo mainEntityDimension : domainInfo.getDimensions()) {
- SchemaItem dimension = new SchemaItem();
- dimension.setBizName(mainEntityDimension.getBizName());
- dimensions.add(dimension);
- }
- return dimensions;
- }
-
- private String getEntryKey(Map.Entry entry) {
- // metric parser special handle, TODO delete
- String entryKey = entry.getKey();
- if (entryKey.contains("__")) {
- entryKey = entryKey.split("__")[1];
- }
- return entryKey;
- }
-
- private Set getMetrics(EntityInfo domainInfo) {
- Set metrics = new LinkedHashSet();
- for (DataInfo metricValue : domainInfo.getMetrics()) {
- SchemaItem metric = new SchemaItem();
- metric.setBizName(metricValue.getBizName());
- metrics.add(metric);
- }
- return metrics;
- }
-
- private String getEntityPrimaryName(EntityInfo domainInfo) {
- return domainInfo.getDomainInfo().getPrimaryEntityBizName();
- }
-
-}
diff --git a/chat/core/src/main/java/com/tencent/supersonic/chat/application/QueryServiceImpl.java b/chat/core/src/main/java/com/tencent/supersonic/chat/application/QueryServiceImpl.java
deleted file mode 100644
index 4fdde6e91..000000000
--- a/chat/core/src/main/java/com/tencent/supersonic/chat/application/QueryServiceImpl.java
+++ /dev/null
@@ -1,93 +0,0 @@
-package com.tencent.supersonic.chat.application;
-
-
-import com.tencent.supersonic.auth.api.authentication.pojo.User;
-import com.tencent.supersonic.chat.api.component.*;
-import com.tencent.supersonic.chat.api.pojo.ChatContext;
-import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
-import com.tencent.supersonic.chat.api.request.QueryContextReq;
-import com.tencent.supersonic.chat.api.response.QueryResultResp;
-import com.tencent.supersonic.chat.application.query.QuerySelector;
-import com.tencent.supersonic.chat.domain.pojo.chat.QueryData;
-import com.tencent.supersonic.chat.domain.pojo.search.QueryState;
-import com.tencent.supersonic.chat.domain.service.QueryService;
-import com.tencent.supersonic.chat.domain.service.ChatService;
-import com.tencent.supersonic.chat.domain.utils.ComponentFactory;
-import com.tencent.supersonic.chat.domain.utils.SchemaInfoConverter;
-import com.tencent.supersonic.common.util.json.JsonUtil;
-import com.tencent.supersonic.semantic.api.core.response.QueryResultWithSchemaResp;
-import java.util.List;
-import java.util.stream.Collectors;
-
-import lombok.extern.slf4j.Slf4j;
-import org.springframework.beans.BeanUtils;
-import org.springframework.beans.factory.annotation.Autowired;
-import org.springframework.context.annotation.Primary;
-import org.springframework.stereotype.Component;
-import org.springframework.stereotype.Service;
-
-@Service
-@Component("chatQueryService")
-@Primary
-@Slf4j
-public class QueryServiceImpl implements QueryService {
-
- @Autowired
- private ChatService chatService;
-
- private List schemaMappers = ComponentFactory.getSchemaMappers();
- private List semanticParsers = ComponentFactory.getSemanticParsers();
- private SemanticLayer semanticLayer = ComponentFactory.getSemanticLayer();
- private QuerySelector querySelector = ComponentFactory.getQuerySelector();
-
- @Override
- public QueryResultResp executeQuery(QueryContextReq queryCtx) throws Exception {
- schemaMappers.stream().forEach(s -> s.map(queryCtx));
-
- // in order to support multi-turn conversation, we need to consider chat context
- ChatContext chatCtx = chatService.getOrCreateContext(queryCtx.getChatId());
-
- for (SemanticParser semanticParser : semanticParsers) {
- log.info("semanticParser processing:[{}]", semanticParser.getClass().getName());
- semanticParser.parse(queryCtx, chatCtx);
- }
- if (queryCtx.getCandidateQueries().size() > 0) {
- log.info("pick before [{}]", queryCtx.getCandidateQueries().stream().collect(
- Collectors.toList()));
- SemanticQuery semanticQuery = querySelector.select(queryCtx.getCandidateQueries());
- log.info("pick after [{}]", semanticQuery);
-
- QueryResultResp queryResponse = semanticQuery.execute(queryCtx.getUser());
- if (queryResponse != null) {
- // update chat context after a successful semantic query
- if (queryCtx.isSaveAnswer() && queryResponse.getQueryState() == QueryState.NORMAL.getState()) {
- chatService.updateContext(chatCtx, queryCtx, semanticQuery.getParseInfo());
- }
- queryResponse.setChatContext(chatCtx.getParseInfo());
- chatService.addQuery(queryResponse, queryCtx, chatCtx);
- return queryResponse;
- }
- }
-
- return null;
- }
-
- @Override
- public SemanticParseInfo queryContext(QueryContextReq queryCtx) {
- ChatContext context = chatService.getOrCreateContext(queryCtx.getChatId());
- return context.getParseInfo();
- }
-
- @Override
- public QueryResultResp executeDirectQuery(QueryData queryData, User user) throws Exception {
- SemanticParseInfo semanticParseInfo = new SemanticParseInfo();
- QueryResultResp queryResponse = new QueryResultResp();
- BeanUtils.copyProperties(queryData, semanticParseInfo);
- QueryResultWithSchemaResp resultWithColumns = semanticLayer.queryByStruct(
- SchemaInfoConverter.convertTo(semanticParseInfo), user);
- queryResponse.setQueryColumns(resultWithColumns.getColumns());
- queryResponse.setQueryResults(resultWithColumns.getResultList());
- return queryResponse;
- }
-}
-
diff --git a/chat/core/src/main/java/com/tencent/supersonic/chat/application/RecommendServiceImpl.java b/chat/core/src/main/java/com/tencent/supersonic/chat/application/RecommendServiceImpl.java
deleted file mode 100644
index b88ce07e4..000000000
--- a/chat/core/src/main/java/com/tencent/supersonic/chat/application/RecommendServiceImpl.java
+++ /dev/null
@@ -1,55 +0,0 @@
-package com.tencent.supersonic.chat.application;
-
-
-import com.tencent.supersonic.chat.api.request.QueryContextReq;
-import com.tencent.supersonic.chat.api.component.SemanticLayer;
-import com.tencent.supersonic.chat.domain.utils.ComponentFactory;
-import com.tencent.supersonic.semantic.api.core.response.DomainSchemaResp;
-import com.tencent.supersonic.chat.domain.pojo.chat.RecommendResponse;
-import com.tencent.supersonic.chat.domain.service.RecommendService;
-import java.util.List;
-import java.util.Objects;
-import java.util.stream.Collectors;
-
-import org.springframework.stereotype.Service;
-
-/***
- * Recommend Service impl
- */
-@Service
-public class RecommendServiceImpl implements RecommendService {
-
- private SemanticLayer semanticLayer = ComponentFactory.getSemanticLayer();
-
- @Override
- public RecommendResponse recommend(QueryContextReq queryCtx) {
- Integer domainId = queryCtx.getDomainId();
- if (Objects.isNull(domainId)) {
- return new RecommendResponse();
- }
-
- DomainSchemaResp domainSchemaDesc = semanticLayer.getDomainSchemaInfo(
- Long.valueOf(domainId), true);
-
- List dimensions = domainSchemaDesc.getDimensions().stream().map(dimSchemaDesc -> {
- RecommendResponse.Item item = new RecommendResponse.Item();
- item.setDomain(domainId);
- item.setName(dimSchemaDesc.getName());
- item.setBizName(dimSchemaDesc.getBizName());
- return item;
- }).collect(Collectors.toList());
-
- List metrics = domainSchemaDesc.getMetrics().stream().map(metricSchemaDesc -> {
- RecommendResponse.Item item = new RecommendResponse.Item();
- item.setDomain(domainId);
- item.setName(metricSchemaDesc.getName());
- item.setBizName(metricSchemaDesc.getBizName());
- return item;
- }).collect(Collectors.toList());
-
- RecommendResponse response = new RecommendResponse();
- response.setDimensions(dimensions);
- response.setMetrics(metrics);
- return response;
- }
-}
diff --git a/chat/core/src/main/java/com/tencent/supersonic/chat/application/knowledge/ApplicationStartedInit.java b/chat/core/src/main/java/com/tencent/supersonic/chat/application/knowledge/ApplicationStartedInit.java
deleted file mode 100644
index 517851ed0..000000000
--- a/chat/core/src/main/java/com/tencent/supersonic/chat/application/knowledge/ApplicationStartedInit.java
+++ /dev/null
@@ -1,69 +0,0 @@
-package com.tencent.supersonic.chat.application.knowledge;
-
-import com.tencent.supersonic.common.nlp.WordNature;
-import com.tencent.supersonic.knowledge.domain.service.OnlineKnowledgeService;
-import lombok.extern.slf4j.Slf4j;
-import org.apache.commons.collections.CollectionUtils;
-import org.springframework.beans.factory.annotation.Autowired;
-import org.springframework.boot.context.event.ApplicationStartedEvent;
-import org.springframework.context.ApplicationListener;
-import org.springframework.scheduling.annotation.Scheduled;
-import org.springframework.stereotype.Component;
-
-import java.util.List;
-
-@Slf4j
-@Component
-public class ApplicationStartedInit implements ApplicationListener {
-
- @Autowired
- private OnlineKnowledgeService onlineKnowledgeService;
-
- @Autowired
- private WordNatureService wordNatureService;
-
-
- @Override
- public void onApplicationEvent(ApplicationStartedEvent event) {
- try {
- log.info("ApplicationStartedInit start");
-
- List wordNatures = wordNatureService.getAllWordNature();
-
- wordNatureService.setPreWordNatures(wordNatures);
-
- onlineKnowledgeService.reloadAllData(wordNatures);
-
- log.info("ApplicationStartedInit end");
- } catch (Exception e) {
- log.error("ApplicationStartedInit error", e);
- }
- }
-
- /***
- * reload knowledge task
- */
- @Scheduled(cron = "${reload.knowledge.corn:0 0/1 * * * ?}")
- public void reloadKnowledge() {
- log.info("reloadKnowledge start");
-
- try {
- List wordNatures = wordNatureService.getAllWordNature();
- List preWordNatures = wordNatureService.getPreWordNatures();
-
- if (CollectionUtils.isEqualCollection(wordNatures, preWordNatures)) {
- log.debug("wordNatures is not change, reloadKnowledge end");
- return;
- }
- log.info("wordNatures is change");
- wordNatureService.setPreWordNatures(wordNatures);
- onlineKnowledgeService.updateOnlineKnowledge(wordNatureService.getAllWordNature());
- wordNatureService.getCache().refresh("");
-
- } catch (Exception e) {
- log.error("reloadKnowledge error", e);
- }
-
- log.info("reloadKnowledge end");
- }
-}
\ No newline at end of file
diff --git a/chat/core/src/main/java/com/tencent/supersonic/chat/application/knowledge/NatureHelper.java b/chat/core/src/main/java/com/tencent/supersonic/chat/application/knowledge/NatureHelper.java
deleted file mode 100644
index c01c8a08b..000000000
--- a/chat/core/src/main/java/com/tencent/supersonic/chat/application/knowledge/NatureHelper.java
+++ /dev/null
@@ -1,144 +0,0 @@
-package com.tencent.supersonic.chat.application.knowledge;
-
-import com.hankcs.hanlp.corpus.tag.Nature;
-import com.hankcs.hanlp.seg.common.Term;
-import com.tencent.supersonic.chat.application.mapper.HanlpSchemaMapper;
-import com.tencent.supersonic.chat.domain.pojo.search.DomainInfoStat;
-import com.tencent.supersonic.common.nlp.NatureType;
-import com.tencent.supersonic.knowledge.application.online.BaseWordNature;
-import java.util.ArrayList;
-import java.util.Comparator;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-import java.util.Objects;
-import java.util.stream.Collectors;
-
-import lombok.extern.slf4j.Slf4j;
-import org.apache.commons.lang3.StringUtils;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-/**
- * nature parse helper
- */
-@Slf4j
-public class NatureHelper {
-
- private static boolean isDomainOrEntity(Term term, Integer domain) {
- return (NatureType.NATURE_SPILT + domain).equals(term.nature.toString()) || term.nature.toString()
- .endsWith(NatureType.ENTITY.getType());
- }
-
- public static Integer getDomainByNature(Nature nature) {
- if (nature.startsWith(NatureType.NATURE_SPILT)) {
- String[] dimensionValues = nature.toString().split(NatureType.NATURE_SPILT);
- if (StringUtils.isNumeric(dimensionValues[1])) {
- return Integer.valueOf(dimensionValues[1]);
- }
- }
- return 0;
- }
-
- public static Integer getDomain(String nature) {
- try {
- String[] split = nature.split(NatureType.NATURE_SPILT);
- if (split.length <= 1) {
- return null;
- }
- return Integer.valueOf(split[1]);
- } catch (NumberFormatException e) {
- log.error("", e);
- }
- return null;
- }
-
- public static boolean isDimensionValueClassId(String nature) {
- if (StringUtils.isEmpty(nature)) {
- return false;
- }
- if (!nature.startsWith(NatureType.NATURE_SPILT)) {
- return false;
- }
- String[] split = nature.split(NatureType.NATURE_SPILT);
- if (split.length <= 1) {
- return false;
- }
- return !nature.endsWith(NatureType.METRIC.getType()) && !nature.endsWith(NatureType.DIMENSION.getType())
- && StringUtils.isNumeric(split[1]);
- }
-
- public static DomainInfoStat getDomainStat(List terms) {
- DomainInfoStat stat = new DomainInfoStat();
- stat.setDimensionDomainCount(getDimensionCount(terms));
- stat.setMetricDomainCount(getMetricCount(terms));
- stat.setDomainCount(getDomainCount(terms));
- stat.setDimensionValueDomainCount(getDimensionValueCount(terms));
- return stat;
- }
-
-
- private static long getDomainCount(List terms) {
- return terms.stream().filter(term -> isDomainOrEntity(term, getDomainByNature(term.nature))).count();
- }
-
- private static long getDimensionValueCount(List terms) {
- return terms.stream().filter(term -> isDimensionValueClassId(term.nature.toString())).count();
- }
-
- private static long getDimensionCount(List terms) {
- return terms.stream().filter(term -> term.nature.startsWith(NatureType.NATURE_SPILT) && term.nature.toString()
- .endsWith(NatureType.DIMENSION.getType())).count();
- }
-
- private static long getMetricCount(List terms) {
- return terms.stream().filter(term -> term.nature.startsWith(NatureType.NATURE_SPILT) && term.nature.toString()
- .endsWith(NatureType.METRIC.getType())).count();
- }
-
- /**
- * Get the number of types of class parts of speech
- * domainId -> (nature , natureCount)
- *
- * @param terms
- * @return
- */
- public static Map> getDomainToNatureStat(List terms) {
- Map> domainToNature = new HashMap<>();
- terms.stream().filter(
- term -> term.nature.startsWith(NatureType.NATURE_SPILT)
- ).forEach(term -> {
- NatureType natureType = NatureType.getNatureType(String.valueOf(term.nature));
- Integer domain = getDomain(String.valueOf(term.nature));
-
- Map natureTypeMap = new HashMap<>();
- natureTypeMap.put(natureType, 1);
-
- Map original = domainToNature.get(domain);
- if (Objects.isNull(original)) {
- domainToNature.put(domain, natureTypeMap);
- } else {
- Integer count = original.get(natureType);
- if (Objects.isNull(count)) {
- count = 1;
- } else {
- count = count + 1;
- }
- original.put(natureType, count);
- }
- });
- return domainToNature;
- }
-
- public static List selectPossibleDomains(List terms) {
- Map> domainToNatureStat = getDomainToNatureStat(terms);
- Integer maxDomainTypeSize = domainToNatureStat.entrySet().stream()
- .max(Comparator.comparingInt(o -> o.getValue().size())).map(entry -> entry.getValue().size())
- .orElse(null);
- if (Objects.isNull(maxDomainTypeSize) || maxDomainTypeSize == 0) {
- return new ArrayList<>();
- }
- return domainToNatureStat.entrySet().stream().filter(entry -> entry.getValue().size() == maxDomainTypeSize)
- .map(entry -> entry.getKey()).collect(Collectors.toList());
- }
-}
diff --git a/chat/core/src/main/java/com/tencent/supersonic/chat/application/knowledge/WordNatureService.java b/chat/core/src/main/java/com/tencent/supersonic/chat/application/knowledge/WordNatureService.java
deleted file mode 100644
index c24805448..000000000
--- a/chat/core/src/main/java/com/tencent/supersonic/chat/application/knowledge/WordNatureService.java
+++ /dev/null
@@ -1,81 +0,0 @@
-package com.tencent.supersonic.chat.application.knowledge;
-
-import com.google.common.cache.CacheBuilder;
-import com.google.common.cache.CacheLoader;
-import com.google.common.cache.LoadingCache;
-import com.tencent.supersonic.chat.api.component.SemanticLayer;
-import com.tencent.supersonic.chat.domain.pojo.chat.DomainInfos;
-import com.tencent.supersonic.chat.domain.utils.ComponentFactory;
-import com.tencent.supersonic.chat.domain.utils.SchemaInfoConverter;
-import com.tencent.supersonic.common.nlp.ItemDO;
-import com.tencent.supersonic.common.nlp.NatureType;
-import com.tencent.supersonic.common.nlp.WordNature;
-import com.tencent.supersonic.knowledge.application.online.WordNatureStrategyFactory;
-import java.util.ArrayList;
-import java.util.List;
-import java.util.concurrent.TimeUnit;
-
-import lombok.extern.slf4j.Slf4j;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-import org.springframework.stereotype.Service;
-
-
-/**
- * word nature service
- **/
-@Service
-@Slf4j
-public class WordNatureService {
-
- private static final Integer META_CACHE_TIME = 5;
- private SemanticLayer semanticLayer = ComponentFactory.getSemanticLayer();
- private List preWordNatures = new ArrayList<>();
-
- private LoadingCache cache = CacheBuilder.newBuilder()
- .expireAfterWrite(META_CACHE_TIME, TimeUnit.MINUTES)
- .build(
- new CacheLoader() {
- @Override
- public DomainInfos load(String key) {
- log.info("load getDomainSchemaInfo cache [{}]", key);
- return SchemaInfoConverter.convert(semanticLayer.getDomainSchemaInfo(new ArrayList<>()));
- }
- }
- );
-
- public List getAllWordNature() {
- SemanticLayer semanticLayer = ComponentFactory.getSemanticLayer();
- DomainInfos domainInfos = SchemaInfoConverter.convert(semanticLayer.getDomainSchemaInfo(new ArrayList<>()));
-
- List natures = new ArrayList<>();
-
- addNatureToResult(NatureType.DIMENSION, domainInfos.getDimensions(), natures);
-
- addNatureToResult(NatureType.METRIC, domainInfos.getMetrics(), natures);
-
- addNatureToResult(NatureType.DOMAIN, domainInfos.getDomains(), natures);
-
- addNatureToResult(NatureType.ENTITY, domainInfos.getEntities(), natures);
-
- return natures;
- }
-
- private void addNatureToResult(NatureType value, List metas, List natures) {
- List natureList = WordNatureStrategyFactory.get(value).getWordNatureList(metas);
- log.debug("nature type:{} , nature size:{}", value.name(), natureList.size());
- natures.addAll(natureList);
- }
-
- public List getPreWordNatures() {
- return preWordNatures;
- }
-
- public void setPreWordNatures(List preWordNatures) {
- this.preWordNatures = preWordNatures;
- }
-
- public LoadingCache getCache() {
- return cache;
- }
-}
diff --git a/chat/core/src/main/java/com/tencent/supersonic/chat/application/mapper/DatabaseSchemaMapper.java b/chat/core/src/main/java/com/tencent/supersonic/chat/application/mapper/DatabaseSchemaMapper.java
deleted file mode 100644
index f22f1ac2c..000000000
--- a/chat/core/src/main/java/com/tencent/supersonic/chat/application/mapper/DatabaseSchemaMapper.java
+++ /dev/null
@@ -1,143 +0,0 @@
-package com.tencent.supersonic.chat.application.mapper;
-
-import com.hankcs.hanlp.seg.common.Term;
-import com.tencent.supersonic.chat.api.component.SchemaMapper;
-import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
-import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
-import com.tencent.supersonic.chat.api.pojo.SchemaMapInfo;
-import com.tencent.supersonic.chat.api.request.QueryContextReq;
-import com.tencent.supersonic.chat.application.knowledge.WordNatureService;
-import com.tencent.supersonic.chat.domain.pojo.chat.DomainInfos;
-import com.tencent.supersonic.common.nlp.ItemDO;
-import com.tencent.supersonic.common.util.context.ContextUtils;
-import com.tencent.supersonic.knowledge.infrastructure.nlp.HanlpHelper;
-import java.util.ArrayList;
-import java.util.Comparator;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Map;
-import java.util.Objects;
-import java.util.Set;
-import java.util.stream.Collectors;
-import lombok.extern.slf4j.Slf4j;
-import org.springframework.util.CollectionUtils;
-
-@Slf4j
-public class DatabaseSchemaMapper implements SchemaMapper {
-
- @Override
- public void map(QueryContextReq queryContext) {
-
- log.debug("before db mapper,mapInfo:{}", queryContext.getMapInfo());
-
- List terms = HanlpHelper.getTerms(queryContext.getQueryText());
-
- WordNatureService wordNatureService = ContextUtils.getBean(WordNatureService.class);
-
- DomainInfos domainInfos = wordNatureService.getCache().getUnchecked("");
-
- detectAndAddToSchema(queryContext, terms, domainInfos.getDimensions(),
- SchemaElementType.DIMENSION);
- detectAndAddToSchema(queryContext, terms, domainInfos.getMetrics(), SchemaElementType.METRIC);
-
- log.debug("after db mapper,mapInfo:{}", queryContext.getMapInfo());
- }
-
- private void detectAndAddToSchema(QueryContextReq queryContext, List terms, List domains,
- SchemaElementType schemaElementType) {
- try {
- String queryText = queryContext.getQueryText();
-
- Map> domainResultSet = getResultSet(queryText, terms, domains);
-
- addToSchemaMapInfo(domainResultSet, queryContext.getMapInfo(), schemaElementType);
-
- } catch (Exception e) {
- log.error("detectAndAddToSchema error", e);
- }
- }
-
- private Map> getResultSet(String queryText, List terms, List domains) {
-
- MapperHelper mapperHelper = ContextUtils.getBean(MapperHelper.class);
-
- Map> nameToItems = getNameToItems(domains);
-
- Map regOffsetToLength = terms.stream().sorted(Comparator.comparing(Term::length))
- .collect(Collectors.toMap(Term::getOffset, term -> term.word.length(), (value1, value2) -> value2));
-
- Map> domainResultSet = new HashMap<>();
- for (Integer index = 0; index <= queryText.length() - 1; ) {
- for (Integer i = index; i <= queryText.length(); ) {
- i = mapperHelper.getStepIndex(regOffsetToLength, i);
- if (i <= queryText.length()) {
- String detectSegment = queryText.substring(index, i);
- nameToItems.forEach(
- (name, newItemDOs) -> {
- if (name.contains(detectSegment)
- && mapperHelper.getSimilarity(detectSegment, name)
- >= mapperHelper.getMetricDimensionThresholdConfig()) {
- Set preItemDOS = domainResultSet.putIfAbsent(detectSegment, newItemDOs);
- if (Objects.nonNull(preItemDOS)) {
- preItemDOS.addAll(newItemDOs);
- }
- }
- }
- );
- }
- }
- index = mapperHelper.getStepIndex(regOffsetToLength, index);
- }
- return domainResultSet;
- }
-
- private Map> getNameToItems(List domains) {
- return domains.stream()
- .collect(Collectors.toMap(ItemDO::getName, a -> {
- Set result = new HashSet<>();
- result.add(a);
- return result;
- }, (k1, k2) -> {
- k1.addAll(k2);
- return k1;
- }));
- }
-
- private void addToSchemaMapInfo(Map> mapResultRowSet, SchemaMapInfo schemaMap,
- SchemaElementType schemaElementType) {
- if (Objects.isNull(mapResultRowSet) || mapResultRowSet.size() <= 0) {
- return;
- }
- MapperHelper mapperHelper = ContextUtils.getBean(MapperHelper.class);
-
- for (Map.Entry> entry : mapResultRowSet.entrySet()) {
- String detectWord = entry.getKey();
- Set itemDOS = entry.getValue();
- for (ItemDO itemDO : itemDOS) {
-
- List elements = schemaMap.getMatchedElements(itemDO.getDomain());
- if (CollectionUtils.isEmpty(elements)) {
- elements = new ArrayList<>();
- schemaMap.setMatchedElements(itemDO.getDomain(), elements);
- }
- Set regElementSet = elements.stream()
- .filter(elementMatch -> schemaElementType.equals(elementMatch.getElementType()))
- .map(elementMatch -> elementMatch.getElementID())
- .collect(Collectors.toSet());
-
- if (regElementSet.contains(itemDO.getItemId())) {
- continue;
- }
- SchemaElementMatch schemaElementMatch = SchemaElementMatch.builder()
- .elementID(itemDO.getItemId()).word(itemDO.getName()).frequency(10000L)
- .elementType(schemaElementType).detectWord(detectWord)
- .similarity(mapperHelper.getSimilarity(detectWord, itemDO.getName()))
- .build();
- log.info("schemaElementType:{},add to schema, elementMatch {}", schemaElementType, schemaElementMatch);
- elements.add(schemaElementMatch);
- }
- }
-
- }
-}
diff --git a/chat/core/src/main/java/com/tencent/supersonic/chat/application/mapper/HanlpSchemaMapper.java b/chat/core/src/main/java/com/tencent/supersonic/chat/application/mapper/HanlpSchemaMapper.java
deleted file mode 100644
index 13f1709b4..000000000
--- a/chat/core/src/main/java/com/tencent/supersonic/chat/application/mapper/HanlpSchemaMapper.java
+++ /dev/null
@@ -1,103 +0,0 @@
-package com.tencent.supersonic.chat.application.mapper;
-
-import com.hankcs.hanlp.seg.common.Term;
-import com.tencent.supersonic.chat.api.component.SchemaMapper;
-import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
-import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
-import com.tencent.supersonic.chat.api.pojo.SchemaMapInfo;
-import com.tencent.supersonic.chat.api.request.QueryContextReq;
-import com.tencent.supersonic.chat.application.knowledge.NatureHelper;
-import com.tencent.supersonic.chat.domain.pojo.search.MatchText;
-import com.tencent.supersonic.chat.domain.utils.NatureConverter;
-import com.tencent.supersonic.common.nlp.MapResult;
-import com.tencent.supersonic.common.nlp.NatureType;
-import com.tencent.supersonic.common.util.context.ContextUtils;
-import com.tencent.supersonic.knowledge.application.online.BaseWordNature;
-import com.tencent.supersonic.knowledge.application.online.WordNatureStrategyFactory;
-import com.tencent.supersonic.knowledge.infrastructure.nlp.HanlpHelper;
-import java.util.ArrayList;
-import java.util.List;
-import java.util.Map;
-import java.util.Objects;
-import java.util.Optional;
-import java.util.stream.Collectors;
-import lombok.extern.slf4j.Slf4j;
-import org.apache.commons.collections.CollectionUtils;
-
-@Slf4j
-public class HanlpSchemaMapper implements SchemaMapper {
-
- @Override
- public void map(QueryContextReq queryContext) {
-
- List terms = HanlpHelper.getTerms(queryContext.getQueryText());
-
- terms.forEach(
- item -> log.info("word:{},nature:{},frequency:{}", item.word, item.nature.toString(),
- item.getFrequency())
- );
- QueryMatchStrategy matchStrategy = ContextUtils.getBean(QueryMatchStrategy.class);
-
- Map> matchResult = matchStrategy.match(queryContext.getQueryText(), terms,
- queryContext.getDomainId());
- List matches = new ArrayList<>();
- if (Objects.nonNull(matchResult)) {
- Optional> first = matchResult.entrySet().stream()
- .filter(entry -> CollectionUtils.isNotEmpty(entry.getValue()))
- .map(entry -> entry.getValue()).findFirst();
- if (first.isPresent()) {
- matches = first.get();
- }
- }
- HanlpHelper.transLetterOriginal(matches);
- log.info("queryContext:{},matches:{}", queryContext, matches);
-
- convertTermsToSchemaMapInfo(matches, queryContext.getMapInfo(), terms);
- }
-
-
-
- private void convertTermsToSchemaMapInfo(List mapResults, SchemaMapInfo schemaMap, List terms) {
- if (CollectionUtils.isEmpty(mapResults)) {
- return;
- }
-
- Map wordNatureToFrequency = terms.stream().collect(
- Collectors.toMap(entry -> entry.getWord() + entry.getNature(),
- term -> Long.valueOf(term.getFrequency()), (value1, value2) -> value2));
-
- for (MapResult mapResult : mapResults) {
- for (String nature : mapResult.getNatures()) {
- Integer domain = NatureHelper.getDomain(nature);
- if (Objects.isNull(domain)) {
- continue;
- }
- SchemaElementType elementType = NatureConverter.convertTo(nature);
- if (Objects.isNull(elementType)) {
- continue;
- }
-
- BaseWordNature baseWordNature = WordNatureStrategyFactory.get(NatureType.getNatureType(nature));
- Integer elementID = baseWordNature.getElementID(nature);
- Long frequency = wordNatureToFrequency.get(mapResult.getName() + nature);
- SchemaElementMatch schemaElementMatch = SchemaElementMatch.builder()
- .elementType(elementType)
- .elementID(elementID)
- .frequency(frequency)
- .word(mapResult.getName())
- .similarity(mapResult.getSimilarity())
- .detectWord(mapResult.getDetectWord())
- .build();
-
- Map> domainElementMatches = schemaMap.getDomainElementMatches();
- List schemaElementMatches = domainElementMatches.putIfAbsent(domain,
- new ArrayList<>());
- if (schemaElementMatches == null) {
- schemaElementMatches = domainElementMatches.get(domain);
- }
- schemaElementMatches.add(schemaElementMatch);
- }
- }
- }
-
-}
diff --git a/chat/core/src/main/java/com/tencent/supersonic/chat/application/mapper/QueryFilterMapper.java b/chat/core/src/main/java/com/tencent/supersonic/chat/application/mapper/QueryFilterMapper.java
deleted file mode 100644
index 5d4977a65..000000000
--- a/chat/core/src/main/java/com/tencent/supersonic/chat/application/mapper/QueryFilterMapper.java
+++ /dev/null
@@ -1,66 +0,0 @@
-package com.tencent.supersonic.chat.application.mapper;
-
-import com.google.common.collect.Lists;
-import com.tencent.supersonic.chat.api.component.SchemaMapper;
-import com.tencent.supersonic.chat.api.pojo.*;
-import com.tencent.supersonic.chat.api.request.QueryContextReq;
-import com.tencent.supersonic.common.constant.Constants;
-import lombok.extern.slf4j.Slf4j;
-import org.springframework.util.CollectionUtils;
-import java.util.List;
-import java.util.Map;
-import java.util.stream.Collectors;
-
-
-@Slf4j
-public class QueryFilterMapper implements SchemaMapper {
-
- private Long FREQUENCY = 9999999L;
-
- private double SIMILARITY = 1.0;
-
- @Override
- public void map(QueryContextReq queryContext) {
- Integer domainId = queryContext.getDomainId();
- if (domainId == null || domainId <= 0 || queryContext.getQueryFilter() == null) {
- return;
- }
- QueryFilter queryFilter = queryContext.getQueryFilter();
- SchemaMapInfo schemaMapInfo = queryContext.getMapInfo();
- List schemaElementMatches = schemaMapInfo.getMatchedElements(domainId);
- clearOtherSchemaElementMatch(domainId, schemaMapInfo);
- convertFilterToSchemaMapInfo(queryFilter.getFilters(), schemaElementMatches);
- }
-
- private void convertFilterToSchemaMapInfo(List filters, List schemaElementMatches) {
- log.info("schemaElementMatches before queryFilerMapper:{}", schemaElementMatches);
- if (CollectionUtils.isEmpty(schemaElementMatches)) {
- schemaElementMatches = Lists.newArrayList();
- }
- List words = schemaElementMatches.stream().map(SchemaElementMatch::getWord).collect(Collectors.toList());
- for (Filter filter : filters) {
- SchemaElementMatch schemaElementMatch = SchemaElementMatch.builder()
- .elementType(SchemaElementType.VALUE)
- .elementID(filter.getElementID().intValue())
- .frequency(FREQUENCY)
- .word(String.valueOf(filter.getValue()))
- .similarity(SIMILARITY)
- .detectWord(Constants.EMPTY)
- .build();
- if (words.contains(schemaElementMatch.getWord())) {
- continue;
- }
- schemaElementMatches.add(schemaElementMatch);
- }
- log.info("schemaElementMatches after queryFilerMapper:{}", schemaElementMatches);
- }
-
- private void clearOtherSchemaElementMatch(Integer domainId, SchemaMapInfo schemaMapInfo) {
- for (Map.Entry> entry : schemaMapInfo.getDomainElementMatches().entrySet()) {
- if (!entry.getKey().equals(domainId)) {
- entry.getValue().clear();
- }
- }
- }
-
-}
diff --git a/chat/core/src/main/java/com/tencent/supersonic/chat/application/parser/AggregateSemanticParser.java b/chat/core/src/main/java/com/tencent/supersonic/chat/application/parser/AggregateSemanticParser.java
deleted file mode 100644
index 9a9940c45..000000000
--- a/chat/core/src/main/java/com/tencent/supersonic/chat/application/parser/AggregateSemanticParser.java
+++ /dev/null
@@ -1,102 +0,0 @@
-package com.tencent.supersonic.chat.application.parser;
-
-import com.tencent.supersonic.chat.api.component.SemanticParser;
-import com.tencent.supersonic.chat.api.component.SemanticQuery;
-import com.tencent.supersonic.chat.api.pojo.ChatContext;
-import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
-import com.tencent.supersonic.chat.api.request.QueryContextReq;
-import com.tencent.supersonic.chat.application.query.EntityListFilter;
-import com.tencent.supersonic.chat.application.query.MetricGroupBy;
-import com.tencent.supersonic.chat.application.query.MetricOrderBy;
-import com.tencent.supersonic.common.enums.AggregateTypeEnum;
-import java.util.HashMap;
-import java.util.Map;
-import java.util.Objects;
-import java.util.regex.Matcher;
-import java.util.regex.Pattern;
-import lombok.extern.slf4j.Slf4j;
-
-@Slf4j
-public class AggregateSemanticParser implements SemanticParser {
-
- public static final Integer TOPN_LIMIT = 1000;
-
- private static Map aggregateRegexMap = new HashMap<>();
-
- static {
- aggregateRegexMap.put(AggregateTypeEnum.MAX, Pattern.compile("(?i)(最大值|最大|max|峰值|最高|最多)"));
- aggregateRegexMap.put(AggregateTypeEnum.MIN, Pattern.compile("(?i)(最小值|最小|min|最低|最少)"));
- aggregateRegexMap.put(AggregateTypeEnum.SUM, Pattern.compile("(?i)(汇总|总和|sum)"));
- aggregateRegexMap.put(AggregateTypeEnum.AVG, Pattern.compile("(?i)(平均值|日均|平均|avg)"));
- aggregateRegexMap.put(AggregateTypeEnum.TOPN, Pattern.compile("(?i)(top)"));
- aggregateRegexMap.put(AggregateTypeEnum.DISTINCT, Pattern.compile("(?i)(uv)"));
- aggregateRegexMap.put(AggregateTypeEnum.COUNT, Pattern.compile("(?i)(总数|pv)"));
- aggregateRegexMap.put(AggregateTypeEnum.NONE, Pattern.compile("(?i)(明细)"));
- }
-
- public static AggregateTypeEnum resolveAggregateType(String queryText) {
-
- Map aggregateCount = new HashMap<>(aggregateRegexMap.size());
- for (Map.Entry entry : aggregateRegexMap.entrySet()) {
- Matcher matcher = entry.getValue().matcher(queryText);
- int count = 0;
- while (matcher.find()) {
- count++;
- }
- if (count > 0) {
- aggregateCount.put(entry.getKey(), count);
- }
- }
-
- return aggregateCount.entrySet().stream().max(Map.Entry.comparingByValue()).map(entry -> entry.getKey())
- .orElse(null);
- }
-
- @Override
- public void parse(QueryContextReq queryContext, ChatContext chatContext) {
- AggregateTypeEnum aggregateType = resolveAggregateType(queryContext.getQueryText());
-
- for (SemanticQuery semanticQuery : queryContext.getCandidateQueries()) {
- SemanticParseInfo semanticParse = semanticQuery.getParseInfo();
-
- semanticParse.setNativeQuery(getNativeQuery(aggregateType, semanticParse));
- semanticParse.setAggType(aggregateType);
- if (Objects.isNull(semanticParse.getLimit()) || semanticParse.getLimit() <= 0) {
- semanticParse.setLimit(Long.valueOf(TOPN_LIMIT));
- }
- resetQueryModeByAggregateType(semanticParse, aggregateType);
- }
- }
-
- /**
- * query mode reset by the AggregateType
- *
- * @param parseInfo
- * @param aggregateType
- */
- private void resetQueryModeByAggregateType(SemanticParseInfo parseInfo,
- AggregateTypeEnum aggregateType) {
-
- String queryMode = parseInfo.getQueryMode();
- if (MetricGroupBy.QUERY_MODE.equals(queryMode) || MetricGroupBy.QUERY_MODE.equals(queryMode)) {
- if (AggregateTypeEnum.MAX.equals(aggregateType) || AggregateTypeEnum.MIN.equals(aggregateType)
- || AggregateTypeEnum.TOPN.equals(aggregateType)) {
- parseInfo.setQueryMode(MetricOrderBy.QUERY_MODE);
- } else {
- parseInfo.setQueryMode(MetricGroupBy.QUERY_MODE);
- }
- log.info("queryMode mode [{}]->[{}]", queryMode, parseInfo.getQueryMode());
- }
- }
-
- private boolean getNativeQuery(AggregateTypeEnum aggregateType, SemanticParseInfo semanticParse) {
- if (AggregateTypeEnum.TOPN.equals(aggregateType)) {
- return true;
- }
- if (EntityListFilter.QUERY_MODE.equals(semanticParse.getQueryMode()) && (semanticParse.getMetrics() == null
- || semanticParse.getMetrics().isEmpty())) {
- return true;
- }
- return semanticParse.getNativeQuery();
- }
-}
diff --git a/chat/core/src/main/java/com/tencent/supersonic/chat/application/parser/DomainResolver.java b/chat/core/src/main/java/com/tencent/supersonic/chat/application/parser/DomainResolver.java
deleted file mode 100644
index b3ca4e295..000000000
--- a/chat/core/src/main/java/com/tencent/supersonic/chat/application/parser/DomainResolver.java
+++ /dev/null
@@ -1,19 +0,0 @@
-package com.tencent.supersonic.chat.application.parser;
-
-
-import com.tencent.supersonic.chat.api.pojo.ChatContext;
-import com.tencent.supersonic.chat.api.pojo.SchemaMapInfo;
-import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
-import com.tencent.supersonic.chat.api.request.QueryContextReq;
-import com.tencent.supersonic.chat.api.component.SemanticQuery;
-
-import java.util.Map;
-
-public interface DomainResolver {
-
- Integer resolve(Map domainQueryModes, QueryContextReq queryCtx, ChatContext chatCtx,
- SchemaMapInfo schemaMap);
-
- boolean isDomainSwitch(ChatContext chatCtx, SemanticParseInfo semanticParseInfo);
-
-}
\ No newline at end of file
diff --git a/chat/core/src/main/java/com/tencent/supersonic/chat/application/parser/DomainSemanticParser.java b/chat/core/src/main/java/com/tencent/supersonic/chat/application/parser/DomainSemanticParser.java
deleted file mode 100644
index b24be50fc..000000000
--- a/chat/core/src/main/java/com/tencent/supersonic/chat/application/parser/DomainSemanticParser.java
+++ /dev/null
@@ -1,250 +0,0 @@
-package com.tencent.supersonic.chat.application.parser;
-
-import com.tencent.supersonic.chat.api.component.SemanticLayer;
-import com.tencent.supersonic.chat.api.component.SemanticParser;
-import com.tencent.supersonic.chat.api.pojo.ChatContext;
-import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
-import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
-import com.tencent.supersonic.chat.api.pojo.SchemaMapInfo;
-import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
-import com.tencent.supersonic.chat.api.request.QueryContextReq;
-import com.tencent.supersonic.chat.application.query.EntitySemanticQuery;
-import com.tencent.supersonic.chat.application.query.MetricSemanticQuery;
-import com.tencent.supersonic.chat.application.query.RuleSemanticQuery;
-import com.tencent.supersonic.chat.application.query.RuleSemanticQueryManager;
-import com.tencent.supersonic.chat.domain.pojo.chat.DomainInfos;
-import com.tencent.supersonic.chat.domain.pojo.config.ChatConfigResp;
-import com.tencent.supersonic.chat.domain.service.ConfigService;
-import com.tencent.supersonic.chat.domain.utils.ComponentFactory;
-import com.tencent.supersonic.chat.domain.utils.ContextHelper;
-import com.tencent.supersonic.chat.domain.utils.DefaultMetricUtils;
-import com.tencent.supersonic.chat.domain.utils.SchemaInfoConverter;
-import com.tencent.supersonic.common.util.context.ContextUtils;
-
-import java.util.*;
-import java.util.stream.Collectors;
-
-import lombok.extern.slf4j.Slf4j;
-import org.springframework.util.CollectionUtils;
-
-@Slf4j
-public class DomainSemanticParser implements SemanticParser {
-
- private SemanticLayer semanticLayer = ComponentFactory.getSemanticLayer();
-
- @Override
- public void parse(QueryContextReq queryContext, ChatContext chatContext) {
- DomainInfos domainInfosDb = SchemaInfoConverter.convert(semanticLayer.getDomainSchemaInfo(new ArrayList<>()));
- Map domainToName = domainInfosDb.getDomainToName();
- SchemaMapInfo mapInfo = queryContext.getMapInfo();
-
- // iterate all schemaElementMatches to resolve semantic query
- for (Integer domainId : mapInfo.getMatchedDomains()) {
- List elementMatches = mapInfo.getMatchedElements(domainId);
- List queries = resolveQuery(elementMatches, queryContext);
- for (RuleSemanticQuery query : queries) {
-
- if (useBlackItem(query, domainId)) {
- log.info("useBlackItem, skip query:{}", query);
- continue;
- }
- addCandidateQuery(queryContext, chatContext, domainId.longValue(),
- domainToName.get(domainId), query);
- }
- }
-
- // if no candidates have been found yet, count in chat context and try again
- if (queryContext.getCandidateQueries().size() <= 0) {
- if (chatContext.getParseInfo() != null && chatContext.getParseInfo().getDomainId() > 0) {
- Integer chatDomainId = Integer.valueOf(chatContext.getParseInfo().getDomainId().intValue());
- if (mapInfo.getMatchedDomains().contains(chatDomainId)) {
- List elementMatches = mapInfo.getMatchedElements(chatDomainId);
-
- List queries = tryParseByContext(elementMatches, chatContext, queryContext);
- for (RuleSemanticQuery query : queries) {
- addCandidateQuery(queryContext, chatContext, chatDomainId.longValue(),
- domainToName.get(chatDomainId), query);
- }
- }
- }
- }
- }
-
- private boolean useBlackItem(RuleSemanticQuery query, Integer domainId) {
- if (Objects.isNull(domainId)) {
- return false;
- }
- ConfigService configService = ContextUtils.getBean(ConfigService.class);
- ChatConfigResp chatConfigResp = configService.fetchConfigByDomainId(domainId.longValue());
- if (Objects.nonNull(chatConfigResp) && Objects.nonNull(query) && Objects.nonNull(query.getParseInfo())) {
- List elementMatches = query.getParseInfo().getElementMatches();
- if (!CollectionUtils.isEmpty(elementMatches)) {
- return useBlackItemInternal(elementMatches, chatConfigResp, query);
-
- }
- }
- return false;
- }
-
- private boolean useBlackItemInternal(List elementMatches, ChatConfigResp chatConfigResp, RuleSemanticQuery query) {
- if (Objects.isNull(chatConfigResp)) {
- return false;
- }
- List blackDimIdList = new ArrayList<>();
- List blackMetricIdList = new ArrayList<>();
- if (query instanceof EntitySemanticQuery
- && Objects.nonNull(chatConfigResp.getChatDetailConfig())
- && Objects.nonNull(chatConfigResp.getChatDetailConfig().getVisibility())) {
- log.info("useBlackItem, handle EntitySemanticQuery blackList logic");
- blackDimIdList = chatConfigResp.getChatDetailConfig().getVisibility().getBlackDimIdList();
- blackMetricIdList = chatConfigResp.getChatDetailConfig().getVisibility().getBlackMetricIdList();
- }
-
- if (query instanceof MetricSemanticQuery
- && Objects.nonNull(chatConfigResp.getChatAggConfig())
- && Objects.nonNull(chatConfigResp.getChatAggConfig().getVisibility())) {
- log.info("useBlackItem, handle MetricSemanticQuery blackList logic");
- blackDimIdList = chatConfigResp.getChatAggConfig().getVisibility().getBlackDimIdList();
- blackMetricIdList = chatConfigResp.getChatAggConfig().getVisibility().getBlackMetricIdList();
- }
- return useBlackItemWithElementMatches(elementMatches, blackDimIdList, blackMetricIdList);
- }
-
- private boolean useBlackItemWithElementMatches(List elementMatches, List blackDimIdList, List blackMetricIdList) {
-
- Set dimIds = elementMatches.stream()
- .filter(element -> SchemaElementType.VALUE.equals(element.getElementType()) || SchemaElementType.DIMENSION.equals(element.getElementType()))
- .map(element -> Long.valueOf(element.getElementID())).collect(Collectors.toSet());
-
- Set metricIds = elementMatches.stream()
- .filter(element -> SchemaElementType.METRIC.equals(element.getElementType()))
- .map(element -> Long.valueOf(element.getElementID())).collect(Collectors.toSet());
-
-
- return useBlackItemWithIds(dimIds, metricIds, blackDimIdList, blackMetricIdList);
- }
-
- private boolean useBlackItemWithIds(Set dimIds, Set metricIds, List blackDimIdList, List blackMetricIdList) {
-
- if (!CollectionUtils.isEmpty(blackDimIdList) && !CollectionUtils.isEmpty(dimIds)) {
- if (blackDimIdList.stream().anyMatch(dimIds::contains)) {
- log.info("useBlackItem, blackDimIdList:{}", blackDimIdList.stream().filter(dimIds::contains).collect(Collectors.toList()));
- return true;
- }
- }
- if (!CollectionUtils.isEmpty(blackMetricIdList) && !CollectionUtils.isEmpty(metricIds)) {
- if (blackMetricIdList.stream().anyMatch(metricIds::contains)) {
- log.info("useBlackItem, blackMetricIdList:{}", blackMetricIdList.stream().filter(metricIds::contains).collect(Collectors.toList()));
- return true;
- }
- }
- return false;
- }
-
- private void addCandidateQuery(QueryContextReq queryContext, ChatContext chatContext,
- Long domainId, String domainName, RuleSemanticQuery semanticQuery) {
- if (semanticQuery != null) {
- DefaultMetricUtils defaultMetricUtils = ContextUtils.getBean(DefaultMetricUtils.class);
- defaultMetricUtils.fillParseInfo(semanticQuery, domainId, domainName);
- inheritContext(semanticQuery, chatContext);
- defaultMetricUtils.fillDefaultMetric(semanticQuery.getParseInfo(), queryContext, chatContext);
- queryContext.getCandidateQueries().add(semanticQuery);
- }
- }
-
- protected void inheritContext(RuleSemanticQuery semanticQuery, ChatContext chatContext) {
- // is domain switch
- SemanticParseInfo semanticParse = semanticQuery.getParseInfo();
- DomainResolver domainResolver = ComponentFactory.getDomainResolver();
- if (!domainResolver.isDomainSwitch(chatContext, semanticParse)) {
- semanticQuery.inheritContext(chatContext);
- }
- }
-
- /**
- * try to add ChatContext to SchemaMatch and look if match QueryMode
- *
- * @param elementMatches
- * @param chatCtx
- * @return
- */
- private List tryParseByContext(List elementMatches,
- ChatContext chatCtx, QueryContextReq queryCtx) {
- if (chatCtx.getParseInfo() != null && chatCtx.getParseInfo().getEntity() > 0) {
- Long entityCount = elementMatches.stream().filter(i -> SchemaElementType.ENTITY.equals(i.getElementType()))
- .count();
- Long metricCount = elementMatches.stream().filter(i -> SchemaElementType.METRIC.equals(i.getElementType()))
- .count();
- if (entityCount <= 0 && metricCount <= 0 && ContextHelper.hasEntityId(chatCtx)) {
- // try entity parse
- SchemaElementMatch entityElementMatch = SchemaElementMatch.builder()
- .elementType(SchemaElementType.ENTITY).build();
- List newSchemaMatches = new ArrayList<>();
- if (!CollectionUtils.isEmpty(elementMatches)) {
- newSchemaMatches.addAll(elementMatches);
- }
- newSchemaMatches.add(entityElementMatch);
- List queries = doParseByContext(newSchemaMatches, chatCtx, queryCtx);
- if (queries.size() > 0) {
- return queries;
- }
- }
- }
- return doParseByContext(elementMatches, chatCtx, queryCtx);
- }
-
-
- private List doParseByContext(List elementMatches,
- ChatContext chatCtx, QueryContextReq queryContext) {
- SemanticParseInfo contextSemanticParse = chatCtx.getParseInfo();
- if (contextSemanticParse != null) {
- List newElementMatches = new ArrayList<>();
- List> trySchemaElementTypes = new LinkedList<>();
- // try DIMENSION+METRIC+VALUE
- // try DIMENSION+METRIC METRIC+VALUE DIMENSION+VALUE
- // try DIMENSION METRIC VALUE single
- trySchemaElementTypes.add(new ArrayList<>(
- Arrays.asList(SchemaElementType.DIMENSION, SchemaElementType.METRIC, SchemaElementType.VALUE)));
- trySchemaElementTypes.add(
- new ArrayList<>(Arrays.asList(SchemaElementType.METRIC, SchemaElementType.VALUE)));
- trySchemaElementTypes.add(
- new ArrayList<>(Arrays.asList(SchemaElementType.DIMENSION, SchemaElementType.METRIC)));
- trySchemaElementTypes.add(
- new ArrayList<>(Arrays.asList(SchemaElementType.DIMENSION, SchemaElementType.VALUE)));
- trySchemaElementTypes.add(new ArrayList<>(Arrays.asList(SchemaElementType.METRIC)));
- trySchemaElementTypes.add(new ArrayList<>(Arrays.asList(SchemaElementType.VALUE)));
- trySchemaElementTypes.add(new ArrayList<>(Arrays.asList(SchemaElementType.DIMENSION)));
-
- for (List schemaTypes : trySchemaElementTypes) {
- newElementMatches.clear();
- if (!CollectionUtils.isEmpty(elementMatches)) {
- newElementMatches.addAll(elementMatches);
- }
- ContextHelper.mergeContextSchemaElementMatch(newElementMatches, elementMatches, schemaTypes,
- contextSemanticParse);
- List queries = resolveQuery(newElementMatches, queryContext);
- if (queries.size() > 0) {
- return queries;
- }
- }
- }
- return new ArrayList<>();
- }
-
- private List resolveQuery(List candidateElementMatches,
- QueryContextReq queryContext) {
- List matchedQueries = new ArrayList<>();
- for (RuleSemanticQuery semanticQuery : RuleSemanticQueryManager.getSemanticQueries()) {
- List matches = semanticQuery.match(candidateElementMatches, queryContext);
-
- if (matches.size() > 0) {
- log.info("resolve match [{}:{}] ", semanticQuery.getQueryMode(), matches.size());
- RuleSemanticQuery query = RuleSemanticQueryManager.create(semanticQuery.getQueryMode());
- query.getParseInfo().getElementMatches().addAll(matches);
- matchedQueries.add(query);
- }
- }
-
- return matchedQueries;
- }
-}
diff --git a/chat/core/src/main/java/com/tencent/supersonic/chat/application/parser/HeuristicDomainResolver.java b/chat/core/src/main/java/com/tencent/supersonic/chat/application/parser/HeuristicDomainResolver.java
deleted file mode 100644
index 1b781be17..000000000
--- a/chat/core/src/main/java/com/tencent/supersonic/chat/application/parser/HeuristicDomainResolver.java
+++ /dev/null
@@ -1,147 +0,0 @@
-package com.tencent.supersonic.chat.application.parser;
-
-import com.tencent.supersonic.chat.api.pojo.*;
-import com.tencent.supersonic.chat.api.request.QueryContextReq;
-import com.tencent.supersonic.chat.api.component.SemanticQuery;
-import com.tencent.supersonic.chat.domain.utils.ContextHelper;
-import lombok.extern.slf4j.Slf4j;
-
-import java.util.*;
-import java.util.stream.Collectors;
-
-@Slf4j
-public class HeuristicDomainResolver implements DomainResolver {
-
- protected static Integer selectDomainBySchemaElementCount(Map domainQueryModes,
- SchemaMapInfo schemaMap) {
- Map domainTypeMap = getDomainTypeMap(schemaMap);
- if (domainTypeMap.size() == 1) {
- Integer domainSelect = domainTypeMap.entrySet().stream().collect(Collectors.toList()).get(0).getKey();
- if (domainQueryModes.containsKey(domainSelect)) {
- log.info("selectDomain from domainTypeMap not order [{}]", domainSelect);
- return domainSelect;
- }
- } else {
- Map.Entry maxDomain = domainTypeMap.entrySet().stream()
- .filter(entry -> domainQueryModes.containsKey(entry.getKey()))
- .sorted(ContextHelper.DomainStatComparator).findFirst().orElse(null);
- if (maxDomain != null) {
- log.info("selectDomain from domainTypeMap order [{}]", maxDomain.getKey());
- return maxDomain.getKey();
- }
- }
- return 0;
- }
-
- /**
- * to check can switch domain if context exit domain
- *
- * @return false will use context domain, true will use other domain , maybe include context domain
- */
- protected static boolean isAllowSwitch(Map domainQueryModes, SchemaMapInfo schemaMap,
- ChatContext chatCtx, QueryContextReq searchCtx, Integer domainId) {
- if (!Objects.nonNull(domainId) || domainId <= 0) {
- return true;
- }
- // except content domain, calculate the number of types for each domain, if numbers<=1 will not switch
- Map domainTypeMap = getDomainTypeMap(schemaMap);
- log.info("isAllowSwitch domainTypeMap [{}]", domainTypeMap);
- long otherDomainTypeNumBigOneCount = domainTypeMap.entrySet().stream()
- .filter(entry -> domainQueryModes.containsKey(entry.getKey()) && !entry.getKey().equals(domainId))
- .filter(entry -> entry.getValue().getCount() > 1).count();
- if (otherDomainTypeNumBigOneCount >= 1) {
- return true;
- }
- // if query text only contain time , will not switch
- for (SemanticQuery semanticQuery : domainQueryModes.values()) {
- SemanticParseInfo semanticParseInfo = semanticQuery.getParseInfo();
- if (semanticParseInfo == null) {
- continue;
- }
- if (searchCtx.getQueryText() != null && semanticParseInfo.getDateInfo() != null) {
- if (semanticParseInfo.getDateInfo().getText() != null) {
- if (semanticParseInfo.getDateInfo().getText().equalsIgnoreCase(searchCtx.getQueryText())) {
- log.info("timeParseResults is not null , can not switch context , timeParseResults:{},",
- semanticParseInfo.getDateInfo());
- return false;
- }
- }
- }
- }
-
- // if context domain not in schemaMap , will switch
- if (schemaMap.getMatchedElements(domainId) == null || schemaMap.getMatchedElements(domainId).size() <= 0) {
- log.info("domainId not in schemaMap ");
- return true;
- }
- // other will not switch
- return false;
- }
-
- public static Map getDomainTypeMap(SchemaMapInfo schemaMap) {
- Map domainCount = new HashMap<>();
- for (Map.Entry> entry : schemaMap.getDomainElementMatches().entrySet()) {
- List schemaElementMatches = schemaMap.getMatchedElements(entry.getKey());
- if (schemaElementMatches != null && schemaElementMatches.size() > 0) {
- if (!domainCount.containsKey(entry.getKey())) {
- domainCount.put(entry.getKey(), new QueryMatchInfo());
- }
- QueryMatchInfo queryMatchInfo = domainCount.get(entry.getKey());
- Set schemaElementTypes = new HashSet<>();
- schemaElementMatches.stream()
- .forEach(schemaElementMatch -> schemaElementTypes.add(schemaElementMatch.getElementType()));
- SchemaElementMatch schemaElementMatchMax = schemaElementMatches.stream()
- .sorted(ContextHelper.schemaElementMatchComparatorBySimilarity).findFirst().orElse(null);
- if (schemaElementMatchMax != null) {
- queryMatchInfo.setMaxSimilarity(schemaElementMatchMax.getSimilarity());
- }
- queryMatchInfo.setCount(schemaElementTypes.size());
-
- }
- }
- return domainCount;
- }
-
- @Override
- public boolean isDomainSwitch(ChatContext chatCtx, SemanticParseInfo semanticParseInfo) {
- Long contextDomain = chatCtx.getParseInfo().getDomainId();
- Long currentDomain = semanticParseInfo.getDomainId();
- boolean noSwitch =
- currentDomain == null || contextDomain == null || contextDomain.equals(currentDomain);
- log.debug("ChatContext isDomainSwitch [{}] [{}]",
- semanticParseInfo.getQueryMode(), !noSwitch);
- return !noSwitch;
- }
-
- @Override
- public Integer resolve(Map domainQueryModes, QueryContextReq searchCtx,
- ChatContext chatCtx, SchemaMapInfo schemaMap) {
- Integer selectDomain = selectDomain(domainQueryModes, searchCtx, chatCtx, schemaMap);
- if (selectDomain > 0) {
- log.info("selectDomain {} ", selectDomain);
- return selectDomain;
- }
- // get the max SchemaElementType number
- return selectDomainBySchemaElementCount(domainQueryModes, schemaMap);
- }
-
- public Integer selectDomain(Map domainQueryModes, QueryContextReq searchCtx,
- ChatContext chatCtx,
- SchemaMapInfo schemaMap) {
- // if QueryContext has domainId and in domainQueryModes
- if (domainQueryModes.containsKey(searchCtx.getDomainId())) {
- log.info("selectDomain from QueryContext [{}]", searchCtx.getDomainId());
- return searchCtx.getDomainId();
- }
- // if ChatContext has domainId and in domainQueryModes
- if (chatCtx.getParseInfo().getDomainId() > 0) {
- Integer domainId = Integer.valueOf(chatCtx.getParseInfo().getDomainId().intValue());
- if (!isAllowSwitch(domainQueryModes, schemaMap, chatCtx, searchCtx, domainId)) {
- log.info("selectDomain from ChatContext [{}]", domainId);
- return domainId;
- }
- }
- // default 0
- return 0;
- }
-}
\ No newline at end of file
diff --git a/chat/core/src/main/java/com/tencent/supersonic/chat/application/parser/LLMSemanticParser.java b/chat/core/src/main/java/com/tencent/supersonic/chat/application/parser/LLMSemanticParser.java
deleted file mode 100644
index b3aa00471..000000000
--- a/chat/core/src/main/java/com/tencent/supersonic/chat/application/parser/LLMSemanticParser.java
+++ /dev/null
@@ -1,160 +0,0 @@
-package com.tencent.supersonic.chat.application.parser;
-
-import com.tencent.supersonic.chat.api.component.SemanticParser;
-import com.tencent.supersonic.chat.api.component.SemanticQuery;
-import com.tencent.supersonic.chat.api.pojo.ChatContext;
-import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
-import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
-import com.tencent.supersonic.chat.api.pojo.SchemaMapInfo;
-import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
-import com.tencent.supersonic.chat.api.request.QueryContextReq;
-import com.tencent.supersonic.chat.application.knowledge.WordNatureService;
-import com.tencent.supersonic.chat.application.query.LLMSemanticQuery;
-import com.tencent.supersonic.chat.domain.config.LLMConfig;
-import com.tencent.supersonic.chat.domain.pojo.chat.DomainInfos;
-import com.tencent.supersonic.chat.domain.pojo.chat.LLMReq;
-import com.tencent.supersonic.chat.domain.pojo.chat.LLMResp;
-import com.tencent.supersonic.chat.domain.pojo.chat.LLMSchema;
-import com.tencent.supersonic.chat.domain.utils.DslToSemanticInfo;
-import com.tencent.supersonic.chat.domain.utils.SemanticSatisfactionChecker;
-import com.tencent.supersonic.common.nlp.ItemDO;
-import com.tencent.supersonic.common.util.context.ContextUtils;
-import com.tencent.supersonic.common.util.json.JsonUtil;
-import java.util.ArrayList;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-import java.util.Objects;
-import java.util.Optional;
-import java.util.Set;
-import java.util.stream.Collectors;
-import lombok.extern.slf4j.Slf4j;
-import org.apache.calcite.sql.parser.SqlParseException;
-import org.apache.commons.lang3.StringUtils;
-import org.springframework.core.io.support.SpringFactoriesLoader;
-import org.springframework.http.HttpEntity;
-import org.springframework.http.HttpHeaders;
-import org.springframework.http.HttpMethod;
-import org.springframework.http.MediaType;
-import org.springframework.http.ResponseEntity;
-import org.springframework.web.client.RestTemplate;
-
-@Slf4j
-public class LLMSemanticParser implements SemanticParser {
-
- private DslToSemanticInfo dslToSemanticInfo = new DslToSemanticInfo();
-
- @Override
- public void parse(QueryContextReq queryContext, ChatContext chatCtx) {
- String queryText = queryContext.getQueryText();
-
- if (SemanticSatisfactionChecker.check(queryContext)) {
- log.info("There is no need parse by llm , queryText:{}", queryText);
- return;
- }
-
- try {
- Integer domainId = getDomainId(queryContext, chatCtx);
- LLMResp llmResp = requestLLM(queryContext, domainId);
- if (Objects.isNull(llmResp)) {
- return;
- }
- LLMSemanticQuery semanticQuery = new LLMSemanticQuery();
- SemanticParseInfo parseInfo = semanticQuery.getParseInfo();
- String sql = convertToSql(llmResp, parseInfo, domainId);
-
- parseInfo.setInfo(sql);
- parseInfo.setDomainId(Long.valueOf(domainId));
- parseInfo.setBonus(queryText.length() * 1.0);
- parseInfo.setQueryMode(LLMSemanticQuery.QUERY_MODE);
- queryContext.getCandidateQueries().add(semanticQuery);
- return;
- } catch (Exception e) {
- log.error("llm parse error , skip the parser. error:", e);
- }
- }
-
- protected String convertToSql(LLMResp llmResp, SemanticParseInfo parseInfo, Integer domainId)
- throws SqlParseException {
- return dslToSemanticInfo.convert(parseInfo, llmResp, domainId);
- }
-
- protected LLMResp requestLLM(QueryContextReq queryContext, Integer domainId) {
- final LLMConfig llmConfig = ContextUtils.getBean(LLMConfig.class);
-
- if (StringUtils.isEmpty(llmConfig.getUrl())) {
- log.warn("llmConfig url is null, skip llm parser");
- return null;
- }
-
- DomainInfos domainInfos = ContextUtils.getBean(WordNatureService.class).getCache().getUnchecked("");
-
- Map domainIdToName = domainInfos.getDomains().stream()
- .collect(Collectors.toMap(ItemDO::getDomain, a -> a.getName(), (k1, k2) -> k1));
-
- Map itemIdToName = domainInfos.getDimensions().stream()
- .filter(entry -> domainId.equals(entry.getDomain()))
- .collect(Collectors.toMap(ItemDO::getItemId, ItemDO::getName, (value1, value2) -> value2));
-
- String domainName = domainIdToName.get(domainId);
- LLMReq llmReq = new LLMReq();
- llmReq.setQueryText(queryContext.getQueryText());
-
- List matchedElements = queryContext.getMapInfo().getMatchedElements(domainId);
-
- Set fieldNameList = matchedElements.stream()
- .filter(schemaElementMatch ->
- SchemaElementType.METRIC.equals(schemaElementMatch.getElementType()) ||
- SchemaElementType.DIMENSION.equals(schemaElementMatch.getElementType()) ||
- SchemaElementType.VALUE.equals(schemaElementMatch.getElementType()))
- .map(schemaElementMatch -> {
- if (!SchemaElementType.VALUE.equals(schemaElementMatch.getElementType())) {
- return schemaElementMatch.getWord();
- }
- return itemIdToName.get(schemaElementMatch.getElementID());
- })
- .filter(name -> StringUtils.isNotEmpty(name) && !name.contains("%"))
- .collect(Collectors.toSet());
-
- LLMSchema llmSchema = new LLMSchema();
- llmSchema.setDomainName(domainName);
- llmSchema.setFieldNameList(new ArrayList<>(fieldNameList));
- llmReq.setSchema(llmSchema);
-
- log.info("requestLLM request, domainId:{},llmReq:{}", domainId, llmReq);
- String questUrl = llmConfig.getUrl() + llmConfig.getQueryToSqlPath();
-
- RestTemplate restTemplate = ContextUtils.getBean(RestTemplate.class);
-
- HttpHeaders headers = new HttpHeaders();
- headers.setContentType(MediaType.APPLICATION_JSON);
- HttpEntity entity = new HttpEntity<>(JsonUtil.toString(llmReq), headers);
-
- ResponseEntity responseEntity = restTemplate.exchange(questUrl, HttpMethod.POST, entity,
- LLMResp.class);
-
- log.info("requestLLM response, questUrl:{} \n entity:{} \n body:{}", questUrl, entity,
- responseEntity.getBody());
- return responseEntity.getBody();
- }
-
- protected Integer getDomainId(QueryContextReq queryContext, ChatContext chatCtx) {
- SchemaMapInfo mapInfo = queryContext.getMapInfo();
- Set matchedDomains = mapInfo.getMatchedDomains();
- Map domainQueryModes = new HashMap<>();
- for (Integer matchedDomain : matchedDomains) {
- domainQueryModes.put(matchedDomain, new LLMSemanticQuery());
- }
- List domainResolverList = SpringFactoriesLoader.loadFactories(DomainResolver.class,
- Thread.currentThread().getContextClassLoader());
- Optional domainId = domainResolverList.stream()
- .map(domainResolver -> domainResolver.resolve(domainQueryModes, queryContext, chatCtx,
- queryContext.getMapInfo())).filter(d -> d > 0).findFirst();
- if (domainId.isPresent()) {
- return domainId.get();
- }
- return 0;
- }
-}
-
-
diff --git a/chat/core/src/main/java/com/tencent/supersonic/chat/application/parser/TimeSemanticParser.java b/chat/core/src/main/java/com/tencent/supersonic/chat/application/parser/TimeSemanticParser.java
deleted file mode 100644
index 4190c7012..000000000
--- a/chat/core/src/main/java/com/tencent/supersonic/chat/application/parser/TimeSemanticParser.java
+++ /dev/null
@@ -1,144 +0,0 @@
-package com.tencent.supersonic.chat.application.parser;
-
-import com.tencent.supersonic.chat.api.component.SemanticParser;
-import com.tencent.supersonic.chat.api.component.SemanticQuery;
-import com.tencent.supersonic.chat.api.pojo.ChatContext;
-import com.tencent.supersonic.chat.api.request.QueryContextReq;
-import com.tencent.supersonic.chat.application.query.MetricSemanticQuery;
-import com.tencent.supersonic.chat.application.query.RuleSemanticQuery;
-import com.tencent.supersonic.chat.application.query.RuleSemanticQueryManager;
-import com.tencent.supersonic.common.constant.Constants;
-import com.tencent.supersonic.common.pojo.DateConf;
-import com.tencent.supersonic.common.pojo.SchemaItem;
-import com.tencent.supersonic.semantic.api.core.enums.TimeDimensionEnum;
-import java.time.LocalDate;
-import java.util.HashSet;
-import java.util.Objects;
-import java.util.Set;
-import java.util.Stack;
-import java.util.regex.Matcher;
-import java.util.regex.Pattern;
-import org.apache.commons.collections.CollectionUtils;
-import org.apache.logging.log4j.util.Strings;
-
-public class TimeSemanticParser implements SemanticParser {
-
- private static final Pattern recentPeriodPattern = Pattern.compile(
- ".*(?(近|过去)((?\\d+)|(?[一二三四五六七八九十百千万亿]+))个?(?[天周月年])).*");
-
- private int zhNumParse(String zhNumStr) {
- Stack stack = new Stack<>();
- String numStr = "一二三四五六七八九";
- String unitStr = "十百千万亿";
-
- String[] ssArr = zhNumStr.split("");
- for (String e : ssArr) {
- int numIndex = numStr.indexOf(e);
- int unitIndex = unitStr.indexOf(e);
- if (numIndex != -1) {
- stack.push(numIndex + 1);
- } else if (unitIndex != -1) {
- int unitNum = (int) Math.pow(10, unitIndex + 1);
- if (stack.isEmpty()) {
- stack.push(unitNum);
- } else {
- stack.push(stack.pop() * unitNum);
- }
- }
- }
-
- return stack.stream().mapToInt(s -> s).sum();
- }
-
- @Override
- public void parse(QueryContextReq queryContext, ChatContext chatContext) {
- Matcher m = recentPeriodPattern.matcher(queryContext.getQueryText());
- if (m.matches()) {
- int num = 0;
- String enNum = m.group("enNum");
- String zhNum = m.group("zhNum");
- if (enNum != null) {
- num = Integer.parseInt(enNum);
- } else if (zhNum != null) {
- num = zhNumParse(zhNum);
- }
- if (num > 0) {
- DateConf info = new DateConf();
- String zhPeriod = m.group("zhPeriod");
- int days;
- switch (zhPeriod) {
- case "周":
- days = 7;
- info.setPeriod(Constants.WEEK);
- break;
- case "月":
- days = 30;
- info.setPeriod(Constants.MONTH);
- break;
- case "年":
- days = 365;
- info.setPeriod(Constants.YEAR);
- break;
- default:
- days = 1;
- info.setPeriod(Constants.DAY);
- }
- days = days * num;
- info.setDateMode(DateConf.DateMode.RECENT_UNITS);
- String text = "近" + num + zhPeriod;
- if (Strings.isNotEmpty(m.group("periodStr"))) {
- text = m.group("periodStr");
- }
- info.setText(text);
- info.setStartDate(LocalDate.now().minusDays(days).toString());
- info.setUnit(num);
- //queryContext.getParseInfo().setDateInfo(info);
- for (SemanticQuery query : queryContext.getCandidateQueries()) {
- if (query instanceof MetricSemanticQuery) {
- query.getParseInfo().setDateInfo(info);
- }
- }
- doParseOnlyTime(queryContext, chatContext, info);
- }
- }
- }
-
- protected void doParseOnlyTime(QueryContextReq queryContext, ChatContext chatContext, DateConf info) {
- if (!queryContext.getCandidateQueries().isEmpty() || chatContext.getParseInfo() == null || Objects.isNull(
- info.getText())) {
- return;
- }
- if (info.getText().equals(queryContext.getQueryText()) && queryContext.getMapInfo().getDomainElementMatches()
- .isEmpty()
- ) {
- if (Objects.nonNull(chatContext.getParseInfo().getQueryMode()) && Objects.nonNull(
- chatContext.getParseInfo().getDomainId()) && chatContext.getParseInfo().getDomainId() > 0) {
- if (Objects.nonNull(chatContext.getParseInfo().getDateInfo()) && !chatContext.getParseInfo()
- .getDateInfo().getPeriod().equals(info.getPeriod())) {
- if (!CollectionUtils.isEmpty(chatContext.getParseInfo().getDimensions())) {
- String dateField = TimeDimensionEnum.DAY.getName();
- if (Constants.MONTH.equals(chatContext.getParseInfo().getDateInfo().getPeriod())) {
- dateField = TimeDimensionEnum.MONTH.getName();
- }
- if (Constants.WEEK.equals(chatContext.getParseInfo().getDateInfo().getPeriod())) {
- dateField = TimeDimensionEnum.WEEK.getName();
- }
- Set dimensions = new HashSet<>();
- for (SchemaItem schemaItem : chatContext.getParseInfo().getDimensions()) {
- if (schemaItem.getBizName().equals(dateField)) {
- continue;
- }
- dimensions.add(schemaItem);
- }
- chatContext.getParseInfo().setDimensions(dimensions);
- }
- }
- chatContext.getParseInfo().setDateInfo(info);
- RuleSemanticQuery semanticQuery = RuleSemanticQueryManager.create(
- chatContext.getParseInfo().getQueryMode());
- semanticQuery.setParseInfo(chatContext.getParseInfo());
- queryContext.getCandidateQueries().add(semanticQuery);
- }
- }
- }
-}
\ No newline at end of file
diff --git a/chat/core/src/main/java/com/tencent/supersonic/chat/application/query/EntityDetail.java b/chat/core/src/main/java/com/tencent/supersonic/chat/application/query/EntityDetail.java
deleted file mode 100644
index db626d7af..000000000
--- a/chat/core/src/main/java/com/tencent/supersonic/chat/application/query/EntityDetail.java
+++ /dev/null
@@ -1,33 +0,0 @@
-package com.tencent.supersonic.chat.application.query;
-
-import static com.tencent.supersonic.chat.api.pojo.SchemaElementType.*;
-import static com.tencent.supersonic.chat.application.query.QueryMatchOption.RequireNumberType.AT_LEAST;
-import static com.tencent.supersonic.chat.domain.pojo.chat.SchemaElementOption.REQUIRED;
-
-import com.tencent.supersonic.chat.api.pojo.ChatContext;
-import com.tencent.supersonic.chat.domain.utils.ContextHelper;
-import org.springframework.stereotype.Component;
-
-@Component
-public class EntityDetail extends EntitySemanticQuery {
-
- public static String QUERY_MODE = "ENTITY_DETAIL";
-
- public EntityDetail() {
- super();
- queryMatcher.addOption(DIMENSION, REQUIRED, AT_LEAST, 1)
- .addOption(VALUE, REQUIRED, AT_LEAST, 1);
- }
-
- @Override
- public String getQueryMode() {
- return QUERY_MODE;
- }
-
- @Override
- public void inheritContext(ChatContext chatContext) {
- ContextHelper.addIfEmpty(chatContext.getParseInfo().getDimensionFilters(),
- parseInfo.getDimensionFilters());
- }
-
-}
diff --git a/chat/core/src/main/java/com/tencent/supersonic/chat/application/query/EntityListFilter.java b/chat/core/src/main/java/com/tencent/supersonic/chat/application/query/EntityListFilter.java
deleted file mode 100644
index aadc64412..000000000
--- a/chat/core/src/main/java/com/tencent/supersonic/chat/application/query/EntityListFilter.java
+++ /dev/null
@@ -1,128 +0,0 @@
-package com.tencent.supersonic.chat.application.query;
-
-import static com.tencent.supersonic.chat.api.pojo.SchemaElementType.*;
-import static com.tencent.supersonic.chat.application.query.QueryMatchOption.RequireNumberType.*;
-import static com.tencent.supersonic.chat.domain.pojo.chat.SchemaElementOption.*;
-import static com.tencent.supersonic.common.constant.Constants.DAY;
-
-import com.tencent.supersonic.chat.api.pojo.ChatContext;
-import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
-import com.tencent.supersonic.chat.domain.pojo.config.ChatConfigRichResp;
-import com.tencent.supersonic.chat.domain.pojo.config.ChatDefaultRichConfig;
-import com.tencent.supersonic.chat.domain.pojo.config.EntityRichInfo;
-import com.tencent.supersonic.chat.domain.service.ConfigService;
-import com.tencent.supersonic.chat.domain.utils.ContextHelper;
-import com.tencent.supersonic.common.constant.Constants;
-import com.tencent.supersonic.common.pojo.DateConf;
-import com.tencent.supersonic.common.pojo.Order;
-import com.tencent.supersonic.common.pojo.SchemaItem;
-import com.tencent.supersonic.common.util.context.ContextUtils;
-import com.tencent.supersonic.semantic.api.core.response.DimSchemaResp;
-
-import java.util.HashSet;
-import java.util.LinkedHashSet;
-import java.util.Objects;
-import java.util.Set;
-
-import lombok.extern.slf4j.Slf4j;
-import org.springframework.beans.BeanUtils;
-import org.springframework.stereotype.Component;
-
-
-@Slf4j
-@Component
-public class EntityListFilter extends EntitySemanticQuery {
-
- public static String QUERY_MODE = "ENTITY_LIST_FILTER";
- private static Long entityListLimit = 200L;
-
-
- public EntityListFilter() {
- super();
- queryMatcher.addOption(VALUE, REQUIRED, AT_LEAST, 1)
- .addOption(ENTITY, REQUIRED, AT_LEAST, 1);
- }
-
-
- @Override
- public String getQueryMode() {
- return QUERY_MODE;
- }
-
-
- @Override
- public void inheritContext(ChatContext chatContext) {
- SemanticParseInfo chatParseInfo = chatContext.getParseInfo();
- ContextHelper.addIfEmpty(chatParseInfo.getDimensionFilters(), parseInfo.getDimensionFilters());
- parseInfo.setLimit(entityListLimit);
- this.fillDateEntityFilter(parseInfo);
- this.addEntityDetailAndOrderByMetric(parseInfo);
- this.dealNativeQuery(parseInfo, true);
- }
-
-
- private void fillDateEntityFilter(SemanticParseInfo semanticParseInfo) {
- DateConf dateInfo = new DateConf();
- dateInfo.setDateMode(DateConf.DateMode.RECENT_UNITS);
- dateInfo.setUnit(1);
- dateInfo.setPeriod(DAY);
- dateInfo.setText(String.format("近1天"));
- semanticParseInfo.setDateInfo(dateInfo);
- }
-
- private void addEntityDetailAndOrderByMetric(SemanticParseInfo semanticParseInfo) {
- if (semanticParseInfo.getDomainId() > 0L) {
- ConfigService configService = ContextUtils.getBean(ConfigService.class);
- ChatConfigRichResp chaConfigRichDesc = configService.getConfigRichInfo(
- semanticParseInfo.getDomainId());
- if (chaConfigRichDesc != null && chaConfigRichDesc.getChatDetailRichConfig() != null
- && chaConfigRichDesc.getChatDetailRichConfig().getEntity() != null) {
-// SemanticParseInfo semanticParseInfo = queryContext.getParseInfo();
-// EntityRichInfo entity = chaConfigRichDesc.getChatDetailRichConfig().getEntity();
- Set dimensions = new LinkedHashSet();
-// Set primaryDimensions = this.addPrimaryDimension(entity, dimensions);
- Set metrics = new LinkedHashSet();
- Set orders = new LinkedHashSet();
- ChatDefaultRichConfig chatDefaultConfig = chaConfigRichDesc.getChatDetailRichConfig().getChatDefaultConfig();
- if (chatDefaultConfig != null) {
- chatDefaultConfig.getMetrics().stream()
- .forEach(metric -> {
- metrics.add(metric);
- orders.add(new Order(metric.getBizName(), Constants.DESC_UPPER));
- });
- chatDefaultConfig.getDimensions().stream()
-// .filter((m) -> !primaryDimensions.contains(m.getBizName()))
- .forEach(dimension -> dimensions.add(dimension));
-
- }
-
- semanticParseInfo.setDimensions(dimensions);
- semanticParseInfo.setMetrics(metrics);
- semanticParseInfo.setOrders(orders);
- }
- }
-
- }
-
- private Set addPrimaryDimension(EntityRichInfo entity, Set dimensions) {
- Set primaryDimensions = new HashSet();
- DimSchemaResp dimItem = entity.getDimItem();
- if (Objects.nonNull(entity) && Objects.nonNull(dimItem)) {
- SchemaItem dimension = new SchemaItem();
- BeanUtils.copyProperties(dimItem, dimension);
- dimensions.add(dimension);
- primaryDimensions.add(dimItem.getBizName());
- return primaryDimensions;
- } else {
- return primaryDimensions;
- }
- }
-
- private void dealNativeQuery(SemanticParseInfo semanticParseInfo, boolean isNativeQuery) {
- if (Objects.nonNull(semanticParseInfo)) {
- semanticParseInfo.setNativeQuery(isNativeQuery);
- }
-
- }
-
-}
diff --git a/chat/core/src/main/java/com/tencent/supersonic/chat/application/query/EntityListTopN.java b/chat/core/src/main/java/com/tencent/supersonic/chat/application/query/EntityListTopN.java
deleted file mode 100644
index aba98b7a5..000000000
--- a/chat/core/src/main/java/com/tencent/supersonic/chat/application/query/EntityListTopN.java
+++ /dev/null
@@ -1,35 +0,0 @@
-package com.tencent.supersonic.chat.application.query;
-
-import static com.tencent.supersonic.chat.api.pojo.SchemaElementType.*;
-import static com.tencent.supersonic.chat.application.query.QueryMatchOption.RequireNumberType.AT_LEAST;
-import static com.tencent.supersonic.chat.domain.pojo.chat.SchemaElementOption.REQUIRED;
-
-import com.tencent.supersonic.chat.api.pojo.ChatContext;
-import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
-import com.tencent.supersonic.chat.domain.utils.ContextHelper;
-import org.springframework.stereotype.Component;
-
-@Component
-public class EntityListTopN extends EntitySemanticQuery {
-
- public static String QUERY_MODE = "ENTITY_LIST_TOPN";
-
- public EntityListTopN() {
- super();
- queryMatcher.addOption(METRIC, REQUIRED, AT_LEAST, 1)
- .setSupportOrderBy(true);
- }
-
- @Override
- public String getQueryMode() {
- return QUERY_MODE;
- }
-
- @Override
- public void inheritContext(ChatContext chatContext) {
- SemanticParseInfo chatParseInfo = chatContext.getParseInfo();
- ContextHelper.updateTimeIfEmpty(chatParseInfo, parseInfo);
- ContextHelper.addIfEmpty(chatParseInfo.getMetrics(), parseInfo.getMetrics());
- }
-
-}
diff --git a/chat/core/src/main/java/com/tencent/supersonic/chat/application/query/EntityMetricFilter.java b/chat/core/src/main/java/com/tencent/supersonic/chat/application/query/EntityMetricFilter.java
deleted file mode 100644
index 4139f83ea..000000000
--- a/chat/core/src/main/java/com/tencent/supersonic/chat/application/query/EntityMetricFilter.java
+++ /dev/null
@@ -1,35 +0,0 @@
-package com.tencent.supersonic.chat.application.query;
-
-import static com.tencent.supersonic.chat.api.pojo.SchemaElementType.METRIC;
-import static com.tencent.supersonic.chat.api.pojo.SchemaElementType.VALUE;
-import static com.tencent.supersonic.chat.application.query.QueryMatchOption.RequireNumberType.AT_LEAST;
-import static com.tencent.supersonic.chat.domain.pojo.chat.SchemaElementOption.REQUIRED;
-
-import com.tencent.supersonic.chat.api.pojo.ChatContext;
-import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
-import com.tencent.supersonic.chat.domain.utils.ContextHelper;
-import org.springframework.stereotype.Component;
-
-@Component
-public class EntityMetricFilter extends EntitySemanticQuery {
-
- public static String QUERY_MODE = "ENTITY_METRIC_FILTER";
-
- public EntityMetricFilter() {
- super();
- queryMatcher.addOption(METRIC, REQUIRED, AT_LEAST, 1)
- .addOption(VALUE, REQUIRED, AT_LEAST, 1);
- }
-
- @Override
- public String getQueryMode() {
- return QUERY_MODE;
- }
-
- @Override
- public void inheritContext(ChatContext chatContext) {
- SemanticParseInfo chatParseInfo = chatContext.getParseInfo();
- ContextHelper.addIfEmpty(chatParseInfo.getDimensionFilters(), parseInfo.getDimensionFilters());
- }
-
-}
diff --git a/chat/core/src/main/java/com/tencent/supersonic/chat/application/query/EntitySemanticQuery.java b/chat/core/src/main/java/com/tencent/supersonic/chat/application/query/EntitySemanticQuery.java
deleted file mode 100644
index 416af0244..000000000
--- a/chat/core/src/main/java/com/tencent/supersonic/chat/application/query/EntitySemanticQuery.java
+++ /dev/null
@@ -1,13 +0,0 @@
-package com.tencent.supersonic.chat.application.query;
-
-import static com.tencent.supersonic.chat.api.pojo.SchemaElementType.ENTITY;
-import static com.tencent.supersonic.chat.application.query.QueryMatchOption.RequireNumberType.AT_LEAST;
-import static com.tencent.supersonic.chat.domain.pojo.chat.SchemaElementOption.REQUIRED;
-
-public abstract class EntitySemanticQuery extends RuleSemanticQuery {
-
- public EntitySemanticQuery() {
- super();
- queryMatcher.addOption(ENTITY, REQUIRED, AT_LEAST, 1);
- }
-}
diff --git a/chat/core/src/main/java/com/tencent/supersonic/chat/application/query/LLMSemanticQuery.java b/chat/core/src/main/java/com/tencent/supersonic/chat/application/query/LLMSemanticQuery.java
deleted file mode 100644
index d7a7fbbdc..000000000
--- a/chat/core/src/main/java/com/tencent/supersonic/chat/application/query/LLMSemanticQuery.java
+++ /dev/null
@@ -1,71 +0,0 @@
-package com.tencent.supersonic.chat.application.query;
-
-import com.tencent.supersonic.auth.api.authentication.pojo.User;
-import com.tencent.supersonic.chat.api.component.SemanticLayer;
-import com.tencent.supersonic.chat.api.component.SemanticQuery;
-import com.tencent.supersonic.chat.api.pojo.EntityInfo;
-import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
-import com.tencent.supersonic.chat.api.response.QueryResultResp;
-import com.tencent.supersonic.chat.application.DomainEntityService;
-import com.tencent.supersonic.chat.domain.utils.ComponentFactory;
-import com.tencent.supersonic.chat.domain.utils.SchemaInfoConverter;
-import com.tencent.supersonic.common.util.context.ContextUtils;
-import com.tencent.supersonic.semantic.api.core.pojo.QueryColumn;
-import com.tencent.supersonic.semantic.api.core.response.QueryResultWithSchemaResp;
-import lombok.extern.slf4j.Slf4j;
-import org.apache.commons.lang3.StringUtils;
-
-import java.util.ArrayList;
-import java.util.List;
-import java.util.Map;
-
-@Slf4j
-public class LLMSemanticQuery implements SemanticQuery {
-
- public static String QUERY_MODE = "DSL";
-
- private SemanticParseInfo semanticParse = new SemanticParseInfo();
- private SemanticLayer semanticLayer = ComponentFactory.getSemanticLayer();
-
- @Override
- public String getQueryMode() {
- return QUERY_MODE;
- }
-
- @Override
- public QueryResultResp execute(User user) {
- String queryMode = semanticParse.getQueryMode();
-
- if (semanticParse.getDomainId() < 0 || StringUtils.isEmpty(queryMode)) {
- // reach here some error may happen
- log.error("not find QueryMode");
- throw new RuntimeException("not find QueryMode");
- }
- QueryResultResp queryResponse = new QueryResultResp();
- QueryResultWithSchemaResp queryResult = semanticLayer.queryBySql(
- SchemaInfoConverter.convertToQuerySqlReq(semanticParse), user);
-
- if (queryResult != null) {
- queryResponse.setQueryAuthorization(queryResult.getQueryAuthorization());
- }
- String sql = queryResult == null ? null : queryResult.getSql();
- List