mirror of
https://github.com/tencentmusic/supersonic.git
synced 2026-04-22 06:44:18 +08:00
[improvement][project] supersonic 0.7.2 version backend update (#28)
Co-authored-by: jipengli <jipengli@tencent.com>
This commit is contained in:
17
CHANGELOG.md
17
CHANGELOG.md
@@ -5,6 +5,23 @@
|
|||||||
compatibility issues with previous versions.
|
compatibility issues with previous versions.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## SuperSonic [0.7.2] - 2023-08-12
|
||||||
|
|
||||||
|
### Added
|
||||||
|
- Support asynchronous query - return parse information to user before executing result
|
||||||
|
- Add Model as the basic data structure of the semantic definitions - this will repalce the old conception of subdomain
|
||||||
|
|
||||||
|
### Updated
|
||||||
|
- improve knowledge word similarity algorithm
|
||||||
|
- improve embedding plugin chooser
|
||||||
|
- improve DSLQuery field correction and parser
|
||||||
|
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
- Fix mapper error that detectWord text is shorter than word
|
||||||
|
- Fix MetricDomainQuery inherit context
|
||||||
|
|
||||||
## SuperSonic [0.7.0] - 2023-07-30
|
## SuperSonic [0.7.0] - 2023-07-30
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
|
|||||||
29
README.md
29
README.md
@@ -2,7 +2,7 @@ English | [中文](README_CN.md)
|
|||||||
|
|
||||||
# SuperSonic (超音数)
|
# SuperSonic (超音数)
|
||||||
|
|
||||||
**SuperSonic is an out-of-the-box yet highly extensible framework for building a data chatbot**. SuperSonic provides a chat interface that empowers users to query data using natural language and visualize the results with suitable charts. To enable such experience, the only thing necessary is to build logical semantic models (definition of metrics/dimensions/entities, along with their meaning, context and relationships) on top of physical data models, and no data modification or copying is required. Meanwhile, SuperSonic is designed to be pluggable, allowing new tools to be added through plugins.
|
**SuperSonic is an out-of-the-box yet highly extensible framework for building a data chatbot**. SuperSonic provides a chat interface that empowers users to query data using natural language and visualize the results with suitable charts. To enable such experience, the only thing necessary is to build logical semantic models (definition of metrics/dimensions/entities, along with their meaning, context and relationships) on top of physical data models, and no data modification or copying is required. Meanwhile, SuperSonic is designed to be pluggable, allowing new functionalities to be added through plugins and core components to be integrated with other systems.
|
||||||
|
|
||||||
<img src="./docs/images/supersonic_demo.gif" height="100%" width="100%" align="center"/>
|
<img src="./docs/images/supersonic_demo.gif" height="100%" width="100%" align="center"/>
|
||||||
|
|
||||||
@@ -10,34 +10,35 @@ English | [中文](README_CN.md)
|
|||||||
|
|
||||||
The emergence of Large Language Model (LLM) like ChatGPT is reshaping the way information is retrieved. In the field of data analytics, both academia and industry are primarily focused on leveraging LLM to convert natural language queries into SQL queries. While some works show promising results, they are still not applicable to real-world scenarios.
|
The emergence of Large Language Model (LLM) like ChatGPT is reshaping the way information is retrieved. In the field of data analytics, both academia and industry are primarily focused on leveraging LLM to convert natural language queries into SQL queries. While some works show promising results, they are still not applicable to real-world scenarios.
|
||||||
|
|
||||||
From our perspective, the key to filling the real-world gap lies in three aspects:
|
From our perspective, the key to filling the real-world gap lies in two aspects:
|
||||||
1. Utilize a combination of rule-based and LLM-based semantic parsers to deal with different scenarios.
|
1. Utilize a combination of rule-based and model-based semantic parsers to deal with different scenarios.
|
||||||
2. Introduce a semantic model layer encapsulating the underlying data complexity(joins, formulas, etc) to simplify semantic parsing.
|
2. Introduce a semantic model layer encapsulating the underlying data complexity(joins, formulas, etc) to simplify semantic parsing.
|
||||||
3. Integrate third-party plugins to augment semantic parsing capabilities or complement custom functionalities.
|
|
||||||
|
|
||||||
With these ideas in mind, we develop SuperSonic as a practical reference implementation and use it to power our real-world products. Additionally, to facilitate further development of data chatbot, we decide to open source SuperSonic as an extensible framework.
|
With these ideas in mind, we develop SuperSonic as a practical reference implementation and use it to power our real-world products. Additionally, to facilitate further development of data chatbot, we decide to open source SuperSonic as an extensible framework.
|
||||||
|
|
||||||
## Out-of-the-box Features
|
## Out-of-the-box Features
|
||||||
|
|
||||||
- Built-in chat UI for business users to enter natural language queries and answer results with appropriate visualization charts.
|
- Built-in graphical interface for business users to enter data queries
|
||||||
- Built-in modelling UI for analytics engineers to manage semantic models. The configurations related to access permission and chat conversation can also be set on the UI.
|
- Built-in graphical interface for analytics engineers to manage semantic models
|
||||||
- Support input auto-completion as well as query recommendation.
|
- Support input auto-completion as well as query recommendation
|
||||||
- Support multi-turn conversation and history context management.
|
- Support multi-turn conversation and history context management
|
||||||
- Support four-level permission control: domain-level, model-level, column-level and row-level.
|
- Support three-level permission control: domain-level, column-level and row-level
|
||||||
|
|
||||||
## Extensible Components
|
## Extensible Components
|
||||||
|
|
||||||
The high-level architecture and main process flow is shown in below diagram:
|
The high-level architecture and main process flow is shown in below diagram:
|
||||||
|
|
||||||
<img src="./docs/images/supersonic_components.png" height="70%" width="70%" align="center"/>
|
<img src="./docs/images/supersonic_components.png" height="80%" width="80%" align="center"/>
|
||||||
|
|
||||||
|
- **Chat Interface:** accepts natural language queries and answer results with appropriate visualization charts. It supports input auto-completion as well as multi-turn conversation.
|
||||||
|
|
||||||
|
- **Modeling Interface:** empowers analytics engineers to visually define and maintain semantic models. The configurations related to access permission and chat conversation can also be set on the UI.
|
||||||
|
|
||||||
- **Schema Mapper Chain:** identifies references to schema elements(metrics/dimensions/entities/values) in user queries. It matches the query text against a knowledge base constructed from the semantic models.
|
- **Schema Mapper Chain:** identifies references to schema elements(metrics/dimensions/entities/values) in user queries. It matches the query text against a knowledge base constructed from the semantic models.
|
||||||
|
|
||||||
- **Semantic Parser Chain:** understands user queries and extract semantic information. It consists of a combination of rule-based and LLM-based parsers, each of which deals with specific scenarios.
|
- **Semantic Parser Chain:** understands user queries and extract semantic information. It consists of a combination of rule-based and model-based parsers, each of which deals with specific scenarios.
|
||||||
|
|
||||||
- **Semantic Layer:** performs execution according to extracted semantic information. It generates SQL queries and executes them against physical data models.
|
- **Semantic Query:** performs execution according to extracted semantic information. It generates SQL queries and executes them against physical data models.
|
||||||
|
|
||||||
- **Chat Plugins**: perform custom execution given the results of schema mapping and semantic parsing. It would optionally resort to semantic layer to query semantic models.
|
|
||||||
|
|
||||||
## Quick Demo
|
## Quick Demo
|
||||||
|
|
||||||
|
|||||||
29
README_CN.md
29
README_CN.md
@@ -8,42 +8,43 @@
|
|||||||
|
|
||||||
大型语言模型(LLMs)如ChatGPT的出现正在重塑信息检索的方式。在数据分析领域,学术界和工业界主要关注利用深度学习模型将自然语言查询转换为SQL查询。虽然一些工作显示出有前景的结果,但它们还并不适用于实际场景。
|
大型语言模型(LLMs)如ChatGPT的出现正在重塑信息检索的方式。在数据分析领域,学术界和工业界主要关注利用深度学习模型将自然语言查询转换为SQL查询。虽然一些工作显示出有前景的结果,但它们还并不适用于实际场景。
|
||||||
|
|
||||||
在我们看来,为了在实际场景发挥价值,有三个关键点:
|
在我们看来,为了在实际场景发挥价值,有两个关键点:
|
||||||
1. 将基于规则和基于模型的语义解析器相结合,发挥各自优势,以便处理不同的场景。
|
1. 将基于规则和基于模型的语义解析器相结合,发挥各自优势,以便处理不同的场景。
|
||||||
2. 引入语义模型层来封装数据底层的复杂性(关联、公式等),从而简化语义解析的求解空间。
|
2. 引入语义模型层来封装数据底层的复杂性(关联、公式等),从而简化语义解析的求解空间。
|
||||||
3. 整合第三方插件用于增强语义解析能力,或者扩充自定义功能。
|
|
||||||
|
|
||||||
为了验证上述想法,我们开发了超音数项目,并将其应用在实际的内部产品中。与此同时,我们将超音数作为一个可扩展的框架开源,希望能够促进数据问答对话领域的进一步发展。
|
为了验证上述想法,我们开发了超音数项目,并将其应用在实际的内部产品中。与此同时,我们将超音数作为一个可扩展的框架开源,希望能够促进数据问答对话领域的进一步发展。
|
||||||
|
|
||||||
## 开箱即用的特性
|
## 开箱即用的特性
|
||||||
|
|
||||||
- 内置的对话图形界面,使用户能够通过自然语言问询,并最终选择合适的可视化图表呈现结果。
|
- 内置图形界面以便业务用户输入数据查询。
|
||||||
- 内置的建模图形界面,使分析工程师能够通过可视化方式定义和维护语义模型,与访问权限和聊天对话相关的配置也可以在用户界面上设置。
|
- 内置图形界面以便分析工程师管理语义模型。
|
||||||
- 支持文本输入的联想和查询问题的推荐。
|
- 支持文本输入的联想和查询问题的推荐。
|
||||||
- 支持多轮对话,根据语境自动切换上下文。
|
- 支持多轮对话,根据语境自动切换上下文。
|
||||||
- 支持四级权限控制:主题域级、模型级、列级及行级。
|
- 支持三级权限控制:主题域级、列级、行级。
|
||||||
|
|
||||||
## 易于扩展的组件
|
## 易于扩展的组件
|
||||||
|
|
||||||
超音数的整体架构和主流程如下图所示:
|
超音数的整体架构和主流程如下图所示:
|
||||||
|
|
||||||
<img src="./docs/images/supersonic_components.png" height="70%" width="70%" align="center"/>
|
<img src="./docs/images/supersonic_components.png" height="80%" width="80%" align="center"/>
|
||||||
|
|
||||||
|
- **问答对话界面(chat interface)**:接受用户查询并选择合适的可视化图表呈现结果,支持输入联想和多轮对话。
|
||||||
|
|
||||||
|
- **语义建模界面(modeling interface)**:使分析工程师能够通过可视化方式定义和维护语义模型,与访问权限和聊天对话相关的配置也可以在用户界面上设置。
|
||||||
|
|
||||||
- **模式映射器(schema mapper chain)**:基于语义模型构建知识库,然后将自然语言文本在知识库中进行匹配,为后续的语义解析提供相关信息。
|
- **模式映射器(schema mapper chain)**:基于语义模型构建知识库,然后将自然语言文本在知识库中进行匹配,为后续的语义解析提供相关信息。
|
||||||
|
|
||||||
- **语义解析器(semantic parser chain)**:理解用户查询并抽取语义信息,其由一组基于规则和基于模型的解析器组成,每个解析器可应对不同的特定场景。
|
- **语义解析器(semantic parser chain)**:理解用户查询并抽取语义信息,其由一组基于规则和基于模型的解析器组成,每个解析器可应对不同的特定场景。
|
||||||
|
|
||||||
- **语义层(semantic layer)**: 根据语义信息生成物理SQL执行查询。
|
- **语义查询(semantic query)**: 根据语义信息生成物理SQL执行查询。
|
||||||
|
|
||||||
- **问答插件(chat plugins)**:基于模式映射和语义解析的结果,执行自定义的操作,同时可以选择利用语义层来查询语义模型。
|
|
||||||
|
|
||||||
## 快速体验
|
## 快速体验
|
||||||
|
|
||||||
超音数自带样例的语义模型和问答对话,只需以下三步即可快速体验:
|
超音数自带样例的语义模型和问答对话,只需以下三步即可快速体验:
|
||||||
|
|
||||||
- 从[release page](https://github.com/tencentmusic/supersonic/releases)下载预先构建好的发行包;
|
- 从[release page](https://github.com/tencentmusic/supersonic/releases)下载预先构建好的发行包
|
||||||
- 运行 "bin/start-standalone.sh"启动服务;
|
- 运行 "bin/start-standalone.sh"启动服务
|
||||||
- 在浏览器访问http://localhost:9080 开启探索。
|
- 在浏览器访问http://localhost:9080 开启探索
|
||||||
|
|
||||||
## 如何构建
|
## 如何构建
|
||||||
|
|
||||||
@@ -51,8 +52,8 @@
|
|||||||
|
|
||||||
### Standalone模式构建
|
### Standalone模式构建
|
||||||
|
|
||||||
下载源码包,运行脚本"assembly/bin/build-standalone.sh",将所有服务一起编译打包。
|
下载源码包,运行脚本"assembly/bin/build-standalone.sh",将所有服务一起编译打包
|
||||||
|
|
||||||
### Distributed模式构建
|
### Distributed模式构建
|
||||||
|
|
||||||
下载源码包,分别运行脚本"assembly/bin/build-chat.sh"、"assembly/bin/build-semantic.sh",为问答层服务和语义层服务编译打包。
|
下载源码包,分别运行脚本"assembly/bin/build-chat.sh"、"assembly/bin/build-semantic.sh",为问答层服务和语义层服务编译打包
|
||||||
@@ -3,7 +3,6 @@ package com.tencent.supersonic.auth.api.authentication.adaptor;
|
|||||||
import com.tencent.supersonic.auth.api.authentication.pojo.Organization;
|
import com.tencent.supersonic.auth.api.authentication.pojo.Organization;
|
||||||
import com.tencent.supersonic.auth.api.authentication.pojo.User;
|
import com.tencent.supersonic.auth.api.authentication.pojo.User;
|
||||||
import com.tencent.supersonic.auth.api.authentication.request.UserReq;
|
import com.tencent.supersonic.auth.api.authentication.request.UserReq;
|
||||||
|
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
import java.util.Set;
|
import java.util.Set;
|
||||||
|
|
||||||
|
|||||||
@@ -1,9 +1,8 @@
|
|||||||
package com.tencent.supersonic.auth.api.authentication.pojo;
|
package com.tencent.supersonic.auth.api.authentication.pojo;
|
||||||
|
|
||||||
import com.google.common.collect.Lists;
|
import com.google.common.collect.Lists;
|
||||||
import lombok.Data;
|
|
||||||
|
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
|
import lombok.Data;
|
||||||
|
|
||||||
@Data
|
@Data
|
||||||
public class Organization {
|
public class Organization {
|
||||||
|
|||||||
@@ -3,7 +3,6 @@ package com.tencent.supersonic.auth.api.authentication.service;
|
|||||||
import com.tencent.supersonic.auth.api.authentication.pojo.Organization;
|
import com.tencent.supersonic.auth.api.authentication.pojo.Organization;
|
||||||
import com.tencent.supersonic.auth.api.authentication.pojo.User;
|
import com.tencent.supersonic.auth.api.authentication.pojo.User;
|
||||||
import com.tencent.supersonic.auth.api.authentication.request.UserReq;
|
import com.tencent.supersonic.auth.api.authentication.request.UserReq;
|
||||||
|
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
import java.util.Set;
|
import java.util.Set;
|
||||||
|
|
||||||
|
|||||||
@@ -1,7 +1,6 @@
|
|||||||
package com.tencent.supersonic.auth.api.authentication.service;
|
package com.tencent.supersonic.auth.api.authentication.service;
|
||||||
|
|
||||||
import com.tencent.supersonic.auth.api.authentication.pojo.User;
|
import com.tencent.supersonic.auth.api.authentication.pojo.User;
|
||||||
|
|
||||||
import javax.servlet.http.HttpServletRequest;
|
import javax.servlet.http.HttpServletRequest;
|
||||||
import javax.servlet.http.HttpServletResponse;
|
import javax.servlet.http.HttpServletResponse;
|
||||||
|
|
||||||
|
|||||||
@@ -6,7 +6,7 @@ import lombok.Data;
|
|||||||
@Data
|
@Data
|
||||||
public class AuthGroup {
|
public class AuthGroup {
|
||||||
|
|
||||||
private String domainId;
|
private String modelId;
|
||||||
private String name;
|
private String name;
|
||||||
private Integer groupId;
|
private Integer groupId;
|
||||||
private List<AuthRule> authRules;
|
private List<AuthRule> authRules;
|
||||||
|
|||||||
@@ -7,14 +7,14 @@ import lombok.ToString;
|
|||||||
@ToString
|
@ToString
|
||||||
public class AuthRes {
|
public class AuthRes {
|
||||||
|
|
||||||
private String domainId;
|
private String modelId;
|
||||||
private String name;
|
private String name;
|
||||||
|
|
||||||
public AuthRes() {
|
public AuthRes() {
|
||||||
}
|
}
|
||||||
|
|
||||||
public AuthRes(String domainId, String name) {
|
public AuthRes(String modelId, String name) {
|
||||||
this.domainId = domainId;
|
this.modelId = modelId;
|
||||||
this.name = name;
|
this.name = name;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -3,7 +3,6 @@ package com.tencent.supersonic.auth.api.authorization.request;
|
|||||||
import com.tencent.supersonic.auth.api.authorization.pojo.AuthRes;
|
import com.tencent.supersonic.auth.api.authorization.pojo.AuthRes;
|
||||||
import java.util.ArrayList;
|
import java.util.ArrayList;
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
|
|
||||||
import lombok.Data;
|
import lombok.Data;
|
||||||
import lombok.ToString;
|
import lombok.ToString;
|
||||||
|
|
||||||
@@ -17,5 +16,5 @@ public class QueryAuthResReq {
|
|||||||
|
|
||||||
private List<AuthRes> resources;
|
private List<AuthRes> resources;
|
||||||
|
|
||||||
private String domainId;
|
private String modelId;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -4,7 +4,6 @@ import com.tencent.supersonic.auth.api.authorization.pojo.AuthResGrp;
|
|||||||
import com.tencent.supersonic.auth.api.authorization.pojo.DimensionFilter;
|
import com.tencent.supersonic.auth.api.authorization.pojo.DimensionFilter;
|
||||||
import java.util.ArrayList;
|
import java.util.ArrayList;
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
|
|
||||||
import lombok.Data;
|
import lombok.Data;
|
||||||
|
|
||||||
@Data
|
@Data
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ package com.tencent.supersonic.auth.api.authorization.service;
|
|||||||
import com.tencent.supersonic.auth.api.authorization.pojo.AuthGroup;
|
import com.tencent.supersonic.auth.api.authorization.pojo.AuthGroup;
|
||||||
import com.tencent.supersonic.auth.api.authorization.request.QueryAuthResReq;
|
import com.tencent.supersonic.auth.api.authorization.request.QueryAuthResReq;
|
||||||
import com.tencent.supersonic.auth.api.authorization.response.AuthorizedResourceResp;
|
import com.tencent.supersonic.auth.api.authorization.response.AuthorizedResourceResp;
|
||||||
import javax.servlet.http.HttpServletRequest;
|
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
|
import javax.servlet.http.HttpServletRequest;
|
||||||
|
|
||||||
public interface AuthService {
|
public interface AuthService {
|
||||||
|
|
||||||
|
|||||||
@@ -11,10 +11,10 @@ import com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO;
|
|||||||
import com.tencent.supersonic.auth.authentication.persistence.repository.UserRepository;
|
import com.tencent.supersonic.auth.authentication.persistence.repository.UserRepository;
|
||||||
import com.tencent.supersonic.auth.authentication.utils.UserTokenUtils;
|
import com.tencent.supersonic.auth.authentication.utils.UserTokenUtils;
|
||||||
import com.tencent.supersonic.common.util.ContextUtils;
|
import com.tencent.supersonic.common.util.ContextUtils;
|
||||||
import org.springframework.beans.BeanUtils;
|
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
import java.util.Set;
|
import java.util.Set;
|
||||||
import java.util.stream.Collectors;
|
import java.util.stream.Collectors;
|
||||||
|
import org.springframework.beans.BeanUtils;
|
||||||
|
|
||||||
public class DefaultUserAdaptor implements UserAdaptor {
|
public class DefaultUserAdaptor implements UserAdaptor {
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,21 @@
|
|||||||
|
package com.tencent.supersonic.auth.authentication.config;
|
||||||
|
|
||||||
|
|
||||||
|
import lombok.Data;
|
||||||
|
import org.springframework.beans.factory.annotation.Value;
|
||||||
|
import org.springframework.context.annotation.Configuration;
|
||||||
|
|
||||||
|
@Data
|
||||||
|
@Configuration
|
||||||
|
public class TppConfig {
|
||||||
|
|
||||||
|
@Value(value = "${auth.app.secret:}")
|
||||||
|
private String appSecret;
|
||||||
|
|
||||||
|
@Value(value = "${auth.app.key:}")
|
||||||
|
private String appKey;
|
||||||
|
|
||||||
|
@Value(value = "${auth.oa.url:}")
|
||||||
|
private String tppOaUrl;
|
||||||
|
|
||||||
|
}
|
||||||
@@ -35,11 +35,14 @@ public class DefaultAuthenticationInterceptor extends AuthenticationInterceptor
|
|||||||
setFakerUser(request);
|
setFakerUser(request);
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
HandlerMethod handlerMethod = (HandlerMethod) handler;
|
|
||||||
Method method = handlerMethod.getMethod();
|
if (handler instanceof HandlerMethod) {
|
||||||
AuthenticationIgnore ignore = method.getAnnotation(AuthenticationIgnore.class);
|
HandlerMethod handlerMethod = (HandlerMethod) handler;
|
||||||
if (ignore != null) {
|
Method method = handlerMethod.getMethod();
|
||||||
return true;
|
AuthenticationIgnore ignore = method.getAnnotation(AuthenticationIgnore.class);
|
||||||
|
if (ignore != null) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
String uri = request.getServletPath();
|
String uri = request.getServletPath();
|
||||||
|
|||||||
@@ -1,12 +1,11 @@
|
|||||||
package com.tencent.supersonic.auth.authentication.interceptor;
|
package com.tencent.supersonic.auth.authentication.interceptor;
|
||||||
|
|
||||||
|
import java.util.List;
|
||||||
import org.springframework.context.annotation.Configuration;
|
import org.springframework.context.annotation.Configuration;
|
||||||
import org.springframework.core.io.support.SpringFactoriesLoader;
|
import org.springframework.core.io.support.SpringFactoriesLoader;
|
||||||
import org.springframework.web.servlet.config.annotation.InterceptorRegistry;
|
import org.springframework.web.servlet.config.annotation.InterceptorRegistry;
|
||||||
import org.springframework.web.servlet.config.annotation.WebMvcConfigurer;
|
import org.springframework.web.servlet.config.annotation.WebMvcConfigurer;
|
||||||
|
|
||||||
import java.util.List;
|
|
||||||
|
|
||||||
@Configuration
|
@Configuration
|
||||||
public class InterceptorFactory implements WebMvcConfigurer {
|
public class InterceptorFactory implements WebMvcConfigurer {
|
||||||
|
|
||||||
|
|||||||
@@ -3,8 +3,8 @@ package com.tencent.supersonic.auth.authentication.persistence.repository.Impl;
|
|||||||
|
|
||||||
import com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO;
|
import com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO;
|
||||||
import com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDOExample;
|
import com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDOExample;
|
||||||
import com.tencent.supersonic.auth.authentication.persistence.repository.UserRepository;
|
|
||||||
import com.tencent.supersonic.auth.authentication.persistence.mapper.UserDOMapper;
|
import com.tencent.supersonic.auth.authentication.persistence.mapper.UserDOMapper;
|
||||||
|
import com.tencent.supersonic.auth.authentication.persistence.repository.UserRepository;
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
import java.util.Optional;
|
import java.util.Optional;
|
||||||
import org.springframework.stereotype.Component;
|
import org.springframework.stereotype.Component;
|
||||||
|
|||||||
@@ -11,7 +11,12 @@ import java.util.Set;
|
|||||||
import javax.servlet.http.HttpServletRequest;
|
import javax.servlet.http.HttpServletRequest;
|
||||||
import javax.servlet.http.HttpServletResponse;
|
import javax.servlet.http.HttpServletResponse;
|
||||||
import lombok.extern.slf4j.Slf4j;
|
import lombok.extern.slf4j.Slf4j;
|
||||||
import org.springframework.web.bind.annotation.*;
|
import org.springframework.web.bind.annotation.GetMapping;
|
||||||
|
import org.springframework.web.bind.annotation.PathVariable;
|
||||||
|
import org.springframework.web.bind.annotation.PostMapping;
|
||||||
|
import org.springframework.web.bind.annotation.RequestBody;
|
||||||
|
import org.springframework.web.bind.annotation.RequestMapping;
|
||||||
|
import org.springframework.web.bind.annotation.RestController;
|
||||||
|
|
||||||
@RestController
|
@RestController
|
||||||
@RequestMapping("/api/auth/user")
|
@RequestMapping("/api/auth/user")
|
||||||
|
|||||||
@@ -7,7 +7,6 @@ import com.tencent.supersonic.auth.api.authentication.service.UserService;
|
|||||||
import com.tencent.supersonic.auth.authentication.utils.ComponentFactory;
|
import com.tencent.supersonic.auth.authentication.utils.ComponentFactory;
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
import java.util.Set;
|
import java.util.Set;
|
||||||
|
|
||||||
import org.springframework.stereotype.Service;
|
import org.springframework.stereotype.Service;
|
||||||
|
|
||||||
@Service
|
@Service
|
||||||
@@ -16,7 +15,7 @@ public class UserServiceImpl implements UserService {
|
|||||||
|
|
||||||
@Override
|
@Override
|
||||||
public List<String> getUserNames() {
|
public List<String> getUserNames() {
|
||||||
return ComponentFactory.getUserAdaptor().getUserNames();
|
return ComponentFactory.getUserAdaptor().getUserNames();
|
||||||
}
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
|
|||||||
@@ -1,8 +1,8 @@
|
|||||||
package com.tencent.supersonic.auth.authentication.utils;
|
package com.tencent.supersonic.auth.authentication.utils;
|
||||||
|
|
||||||
import com.tencent.supersonic.auth.api.authentication.adaptor.UserAdaptor;
|
import com.tencent.supersonic.auth.api.authentication.adaptor.UserAdaptor;
|
||||||
import org.springframework.core.io.support.SpringFactoriesLoader;
|
|
||||||
import java.util.Objects;
|
import java.util.Objects;
|
||||||
|
import org.springframework.core.io.support.SpringFactoriesLoader;
|
||||||
|
|
||||||
public class ComponentFactory {
|
public class ComponentFactory {
|
||||||
|
|
||||||
|
|||||||
@@ -1,145 +1,159 @@
|
|||||||
<?xml version="1.0" encoding="UTF-8"?>
|
<?xml version="1.0" encoding="UTF-8"?>
|
||||||
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
|
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN"
|
||||||
|
"http://mybatis.org/dtd/mybatis-3-mapper.dtd">
|
||||||
<mapper namespace="com.tencent.supersonic.auth.authentication.persistence.mapper.UserDOMapper">
|
<mapper namespace="com.tencent.supersonic.auth.authentication.persistence.mapper.UserDOMapper">
|
||||||
<resultMap id="BaseResultMap" type="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO">
|
<resultMap id="BaseResultMap"
|
||||||
<id column="id" jdbcType="BIGINT" property="id" />
|
type="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO">
|
||||||
<result column="name" jdbcType="VARCHAR" property="name" />
|
<id column="id" jdbcType="BIGINT" property="id"/>
|
||||||
<result column="password" jdbcType="VARCHAR" property="password" />
|
<result column="name" jdbcType="VARCHAR" property="name"/>
|
||||||
<result column="display_name" jdbcType="VARCHAR" property="displayName" />
|
<result column="password" jdbcType="VARCHAR" property="password"/>
|
||||||
<result column="email" jdbcType="VARCHAR" property="email" />
|
<result column="display_name" jdbcType="VARCHAR" property="displayName"/>
|
||||||
</resultMap>
|
<result column="email" jdbcType="VARCHAR" property="email"/>
|
||||||
<sql id="Example_Where_Clause">
|
</resultMap>
|
||||||
<where>
|
<sql id="Example_Where_Clause">
|
||||||
<foreach collection="oredCriteria" item="criteria" separator="or">
|
<where>
|
||||||
<if test="criteria.valid">
|
<foreach collection="oredCriteria" item="criteria" separator="or">
|
||||||
<trim prefix="(" prefixOverrides="and" suffix=")">
|
<if test="criteria.valid">
|
||||||
<foreach collection="criteria.criteria" item="criterion">
|
<trim prefix="(" prefixOverrides="and" suffix=")">
|
||||||
<choose>
|
<foreach collection="criteria.criteria" item="criterion">
|
||||||
<when test="criterion.noValue">
|
<choose>
|
||||||
and ${criterion.condition}
|
<when test="criterion.noValue">
|
||||||
</when>
|
and ${criterion.condition}
|
||||||
<when test="criterion.singleValue">
|
</when>
|
||||||
and ${criterion.condition} #{criterion.value}
|
<when test="criterion.singleValue">
|
||||||
</when>
|
and ${criterion.condition} #{criterion.value}
|
||||||
<when test="criterion.betweenValue">
|
</when>
|
||||||
and ${criterion.condition} #{criterion.value} and #{criterion.secondValue}
|
<when test="criterion.betweenValue">
|
||||||
</when>
|
and ${criterion.condition} #{criterion.value} and
|
||||||
<when test="criterion.listValue">
|
#{criterion.secondValue}
|
||||||
and ${criterion.condition}
|
</when>
|
||||||
<foreach close=")" collection="criterion.value" item="listItem" open="(" separator=",">
|
<when test="criterion.listValue">
|
||||||
#{listItem}
|
and ${criterion.condition}
|
||||||
</foreach>
|
<foreach close=")" collection="criterion.value" item="listItem"
|
||||||
</when>
|
open="(" separator=",">
|
||||||
</choose>
|
#{listItem}
|
||||||
|
</foreach>
|
||||||
|
</when>
|
||||||
|
</choose>
|
||||||
|
</foreach>
|
||||||
|
</trim>
|
||||||
|
</if>
|
||||||
</foreach>
|
</foreach>
|
||||||
</trim>
|
</where>
|
||||||
|
</sql>
|
||||||
|
<sql id="Base_Column_List">
|
||||||
|
id
|
||||||
|
, name, password, display_name, email
|
||||||
|
</sql>
|
||||||
|
<select id="selectByExample"
|
||||||
|
parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDOExample"
|
||||||
|
resultMap="BaseResultMap">
|
||||||
|
select
|
||||||
|
<if test="distinct">
|
||||||
|
distinct
|
||||||
</if>
|
</if>
|
||||||
</foreach>
|
<include refid="Base_Column_List"/>
|
||||||
</where>
|
from s2_user
|
||||||
</sql>
|
<if test="_parameter != null">
|
||||||
<sql id="Base_Column_List">
|
<include refid="Example_Where_Clause"/>
|
||||||
id, name, password, display_name, email
|
</if>
|
||||||
</sql>
|
<if test="orderByClause != null">
|
||||||
<select id="selectByExample" parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDOExample" resultMap="BaseResultMap">
|
order by ${orderByClause}
|
||||||
select
|
</if>
|
||||||
<if test="distinct">
|
<if test="limitStart != null and limitStart>=0">
|
||||||
distinct
|
limit #{limitStart} , #{limitEnd}
|
||||||
</if>
|
</if>
|
||||||
<include refid="Base_Column_List" />
|
</select>
|
||||||
from s2_user
|
<select id="selectByPrimaryKey" parameterType="java.lang.Long" resultMap="BaseResultMap">
|
||||||
<if test="_parameter != null">
|
select
|
||||||
<include refid="Example_Where_Clause" />
|
<include refid="Base_Column_List"/>
|
||||||
</if>
|
from s2_user
|
||||||
<if test="orderByClause != null">
|
where id = #{id,jdbcType=BIGINT}
|
||||||
order by ${orderByClause}
|
</select>
|
||||||
</if>
|
<delete id="deleteByPrimaryKey" parameterType="java.lang.Long">
|
||||||
<if test="limitStart != null and limitStart>=0">
|
delete
|
||||||
limit #{limitStart} , #{limitEnd}
|
from s2_user
|
||||||
</if>
|
where id = #{id,jdbcType=BIGINT}
|
||||||
</select>
|
</delete>
|
||||||
<select id="selectByPrimaryKey" parameterType="java.lang.Long" resultMap="BaseResultMap">
|
<insert id="insert"
|
||||||
select
|
parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO">
|
||||||
<include refid="Base_Column_List" />
|
insert into s2_user (id, name, password,
|
||||||
from s2_user
|
display_name, email)
|
||||||
where id = #{id,jdbcType=BIGINT}
|
values (#{id,jdbcType=BIGINT}, #{name,jdbcType=VARCHAR}, #{password,jdbcType=VARCHAR},
|
||||||
</select>
|
#{displayName,jdbcType=VARCHAR}, #{email,jdbcType=VARCHAR})
|
||||||
<delete id="deleteByPrimaryKey" parameterType="java.lang.Long">
|
</insert>
|
||||||
delete from s2_user
|
<insert id="insertSelective"
|
||||||
where id = #{id,jdbcType=BIGINT}
|
parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO">
|
||||||
</delete>
|
insert into s2_user
|
||||||
<insert id="insert" parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO">
|
<trim prefix="(" suffix=")" suffixOverrides=",">
|
||||||
insert into s2_user (id, name, password,
|
<if test="id != null">
|
||||||
display_name, email)
|
id,
|
||||||
values (#{id,jdbcType=BIGINT}, #{name,jdbcType=VARCHAR}, #{password,jdbcType=VARCHAR},
|
</if>
|
||||||
#{displayName,jdbcType=VARCHAR}, #{email,jdbcType=VARCHAR})
|
<if test="name != null">
|
||||||
</insert>
|
name,
|
||||||
<insert id="insertSelective" parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO">
|
</if>
|
||||||
insert into s2_user
|
<if test="password != null">
|
||||||
<trim prefix="(" suffix=")" suffixOverrides=",">
|
password,
|
||||||
<if test="id != null">
|
</if>
|
||||||
id,
|
<if test="displayName != null">
|
||||||
</if>
|
display_name,
|
||||||
<if test="name != null">
|
</if>
|
||||||
name,
|
<if test="email != null">
|
||||||
</if>
|
email,
|
||||||
<if test="password != null">
|
</if>
|
||||||
password,
|
</trim>
|
||||||
</if>
|
<trim prefix="values (" suffix=")" suffixOverrides=",">
|
||||||
<if test="displayName != null">
|
<if test="id != null">
|
||||||
display_name,
|
#{id,jdbcType=BIGINT},
|
||||||
</if>
|
</if>
|
||||||
<if test="email != null">
|
<if test="name != null">
|
||||||
email,
|
#{name,jdbcType=VARCHAR},
|
||||||
</if>
|
</if>
|
||||||
</trim>
|
<if test="password != null">
|
||||||
<trim prefix="values (" suffix=")" suffixOverrides=",">
|
#{password,jdbcType=VARCHAR},
|
||||||
<if test="id != null">
|
</if>
|
||||||
#{id,jdbcType=BIGINT},
|
<if test="displayName != null">
|
||||||
</if>
|
#{displayName,jdbcType=VARCHAR},
|
||||||
<if test="name != null">
|
</if>
|
||||||
#{name,jdbcType=VARCHAR},
|
<if test="email != null">
|
||||||
</if>
|
#{email,jdbcType=VARCHAR},
|
||||||
<if test="password != null">
|
</if>
|
||||||
#{password,jdbcType=VARCHAR},
|
</trim>
|
||||||
</if>
|
</insert>
|
||||||
<if test="displayName != null">
|
<select id="countByExample"
|
||||||
#{displayName,jdbcType=VARCHAR},
|
parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDOExample"
|
||||||
</if>
|
resultType="java.lang.Long">
|
||||||
<if test="email != null">
|
select count(*) from s2_user
|
||||||
#{email,jdbcType=VARCHAR},
|
<if test="_parameter != null">
|
||||||
</if>
|
<include refid="Example_Where_Clause"/>
|
||||||
</trim>
|
</if>
|
||||||
</insert>
|
</select>
|
||||||
<select id="countByExample" parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDOExample" resultType="java.lang.Long">
|
<update id="updateByPrimaryKeySelective"
|
||||||
select count(*) from s2_user
|
parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO">
|
||||||
<if test="_parameter != null">
|
update s2_user
|
||||||
<include refid="Example_Where_Clause" />
|
<set>
|
||||||
</if>
|
<if test="name != null">
|
||||||
</select>
|
name = #{name,jdbcType=VARCHAR},
|
||||||
<update id="updateByPrimaryKeySelective" parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO">
|
</if>
|
||||||
update s2_user
|
<if test="password != null">
|
||||||
<set>
|
password = #{password,jdbcType=VARCHAR},
|
||||||
<if test="name != null">
|
</if>
|
||||||
name = #{name,jdbcType=VARCHAR},
|
<if test="displayName != null">
|
||||||
</if>
|
display_name = #{displayName,jdbcType=VARCHAR},
|
||||||
<if test="password != null">
|
</if>
|
||||||
password = #{password,jdbcType=VARCHAR},
|
<if test="email != null">
|
||||||
</if>
|
email = #{email,jdbcType=VARCHAR},
|
||||||
<if test="displayName != null">
|
</if>
|
||||||
display_name = #{displayName,jdbcType=VARCHAR},
|
</set>
|
||||||
</if>
|
where id = #{id,jdbcType=BIGINT}
|
||||||
<if test="email != null">
|
</update>
|
||||||
email = #{email,jdbcType=VARCHAR},
|
<update id="updateByPrimaryKey"
|
||||||
</if>
|
parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO">
|
||||||
</set>
|
update s2_user
|
||||||
where id = #{id,jdbcType=BIGINT}
|
set name = #{name,jdbcType=VARCHAR},
|
||||||
</update>
|
password = #{password,jdbcType=VARCHAR},
|
||||||
<update id="updateByPrimaryKey" parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO">
|
display_name = #{displayName,jdbcType=VARCHAR},
|
||||||
update s2_user
|
email = #{email,jdbcType=VARCHAR}
|
||||||
set name = #{name,jdbcType=VARCHAR},
|
where id = #{id,jdbcType=BIGINT}
|
||||||
password = #{password,jdbcType=VARCHAR},
|
</update>
|
||||||
display_name = #{displayName,jdbcType=VARCHAR},
|
|
||||||
email = #{email,jdbcType=VARCHAR}
|
|
||||||
where id = #{id,jdbcType=BIGINT}
|
|
||||||
</update>
|
|
||||||
</mapper>
|
</mapper>
|
||||||
@@ -3,25 +3,27 @@ package com.tencent.supersonic.auth.authorization.application;
|
|||||||
import com.google.common.base.Strings;
|
import com.google.common.base.Strings;
|
||||||
import com.google.gson.Gson;
|
import com.google.gson.Gson;
|
||||||
import com.tencent.supersonic.auth.api.authentication.service.UserService;
|
import com.tencent.supersonic.auth.api.authentication.service.UserService;
|
||||||
|
import com.tencent.supersonic.auth.api.authorization.pojo.AuthGroup;
|
||||||
import com.tencent.supersonic.auth.api.authorization.pojo.AuthRes;
|
import com.tencent.supersonic.auth.api.authorization.pojo.AuthRes;
|
||||||
import com.tencent.supersonic.auth.api.authorization.pojo.AuthResGrp;
|
import com.tencent.supersonic.auth.api.authorization.pojo.AuthResGrp;
|
||||||
|
import com.tencent.supersonic.auth.api.authorization.pojo.AuthRule;
|
||||||
import com.tencent.supersonic.auth.api.authorization.pojo.DimensionFilter;
|
import com.tencent.supersonic.auth.api.authorization.pojo.DimensionFilter;
|
||||||
import com.tencent.supersonic.auth.api.authorization.request.QueryAuthResReq;
|
import com.tencent.supersonic.auth.api.authorization.request.QueryAuthResReq;
|
||||||
import com.tencent.supersonic.auth.api.authorization.response.AuthorizedResourceResp;
|
import com.tencent.supersonic.auth.api.authorization.response.AuthorizedResourceResp;
|
||||||
import com.tencent.supersonic.auth.api.authorization.service.AuthService;
|
import com.tencent.supersonic.auth.api.authorization.service.AuthService;
|
||||||
|
import java.util.ArrayList;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Map;
|
||||||
|
import java.util.Objects;
|
||||||
|
import java.util.Set;
|
||||||
|
import java.util.stream.Collectors;
|
||||||
import javax.servlet.http.HttpServletRequest;
|
import javax.servlet.http.HttpServletRequest;
|
||||||
|
|
||||||
import com.tencent.supersonic.auth.api.authorization.pojo.AuthGroup;
|
|
||||||
import com.tencent.supersonic.auth.api.authorization.pojo.AuthRule;
|
|
||||||
import lombok.extern.slf4j.Slf4j;
|
import lombok.extern.slf4j.Slf4j;
|
||||||
import org.apache.commons.lang3.StringUtils;
|
import org.apache.commons.lang3.StringUtils;
|
||||||
import org.springframework.jdbc.core.JdbcTemplate;
|
import org.springframework.jdbc.core.JdbcTemplate;
|
||||||
import org.springframework.stereotype.Service;
|
import org.springframework.stereotype.Service;
|
||||||
import org.springframework.util.CollectionUtils;
|
import org.springframework.util.CollectionUtils;
|
||||||
|
|
||||||
import java.util.*;
|
|
||||||
import java.util.stream.Collectors;
|
|
||||||
|
|
||||||
@Service
|
@Service
|
||||||
@Slf4j
|
@Slf4j
|
||||||
public class AuthServiceImpl implements AuthService {
|
public class AuthServiceImpl implements AuthService {
|
||||||
@@ -31,7 +33,7 @@ public class AuthServiceImpl implements AuthService {
|
|||||||
private UserService userService;
|
private UserService userService;
|
||||||
|
|
||||||
public AuthServiceImpl(JdbcTemplate jdbcTemplate,
|
public AuthServiceImpl(JdbcTemplate jdbcTemplate,
|
||||||
UserService userService) {
|
UserService userService) {
|
||||||
this.jdbcTemplate = jdbcTemplate;
|
this.jdbcTemplate = jdbcTemplate;
|
||||||
this.userService = userService;
|
this.userService = userService;
|
||||||
}
|
}
|
||||||
@@ -43,10 +45,10 @@ public class AuthServiceImpl implements AuthService {
|
|||||||
}
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public List<AuthGroup> queryAuthGroups(String domainId, Integer groupId) {
|
public List<AuthGroup> queryAuthGroups(String modelId, Integer groupId) {
|
||||||
return load().stream()
|
return load().stream()
|
||||||
.filter(group -> (Objects.isNull(groupId) || groupId.equals(group.getGroupId()))
|
.filter(group -> (Objects.isNull(groupId) || groupId.equals(group.getGroupId()))
|
||||||
&& domainId.equals(group.getDomainId()))
|
&& modelId.equals(group.getModelId()))
|
||||||
.collect(Collectors.toList());
|
.collect(Collectors.toList());
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -83,16 +85,16 @@ public class AuthServiceImpl implements AuthService {
|
|||||||
}
|
}
|
||||||
List<AuthGroup> groups = getAuthGroups(req);
|
List<AuthGroup> groups = getAuthGroups(req);
|
||||||
AuthorizedResourceResp resource = new AuthorizedResourceResp();
|
AuthorizedResourceResp resource = new AuthorizedResourceResp();
|
||||||
Map<String, List<AuthGroup>> authGroupsByDomainId = groups.stream()
|
Map<String, List<AuthGroup>> authGroupsByModelId = groups.stream()
|
||||||
.collect(Collectors.groupingBy(AuthGroup::getDomainId));
|
.collect(Collectors.groupingBy(AuthGroup::getModelId));
|
||||||
Map<String, List<AuthRes>> reqAuthRes = req.getResources().stream()
|
Map<String, List<AuthRes>> reqAuthRes = req.getResources().stream()
|
||||||
.collect(Collectors.groupingBy(AuthRes::getDomainId));
|
.collect(Collectors.groupingBy(AuthRes::getModelId));
|
||||||
|
|
||||||
for (String domainId : reqAuthRes.keySet()) {
|
for (String modelId : reqAuthRes.keySet()) {
|
||||||
List<AuthRes> reqResourcesList = reqAuthRes.get(domainId);
|
List<AuthRes> reqResourcesList = reqAuthRes.get(modelId);
|
||||||
AuthResGrp rg = new AuthResGrp();
|
AuthResGrp rg = new AuthResGrp();
|
||||||
if (authGroupsByDomainId.containsKey(domainId)) {
|
if (authGroupsByModelId.containsKey(modelId)) {
|
||||||
List<AuthGroup> authGroups = authGroupsByDomainId.get(domainId);
|
List<AuthGroup> authGroups = authGroupsByModelId.get(modelId);
|
||||||
for (AuthRes reqRes : reqResourcesList) {
|
for (AuthRes reqRes : reqResourcesList) {
|
||||||
for (AuthGroup authRuleGroup : authGroups) {
|
for (AuthGroup authRuleGroup : authGroups) {
|
||||||
List<AuthRule> authRules = authRuleGroup.getAuthRules();
|
List<AuthRule> authRules = authRuleGroup.getAuthRules();
|
||||||
@@ -111,8 +113,8 @@ public class AuthServiceImpl implements AuthService {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if (StringUtils.isNotEmpty(req.getDomainId())) {
|
if (StringUtils.isNotEmpty(req.getModelId())) {
|
||||||
List<AuthGroup> authGroups = authGroupsByDomainId.get(req.getDomainId());
|
List<AuthGroup> authGroups = authGroupsByModelId.get(req.getModelId());
|
||||||
if (!CollectionUtils.isEmpty(authGroups)) {
|
if (!CollectionUtils.isEmpty(authGroups)) {
|
||||||
for (AuthGroup group : authGroups) {
|
for (AuthGroup group : authGroups) {
|
||||||
if (group.getDimensionFilters() != null
|
if (group.getDimensionFilters() != null
|
||||||
@@ -131,7 +133,7 @@ public class AuthServiceImpl implements AuthService {
|
|||||||
private List<AuthGroup> getAuthGroups(QueryAuthResReq req) {
|
private List<AuthGroup> getAuthGroups(QueryAuthResReq req) {
|
||||||
List<AuthGroup> groups = load().stream()
|
List<AuthGroup> groups = load().stream()
|
||||||
.filter(group -> {
|
.filter(group -> {
|
||||||
if (!Objects.equals(group.getDomainId(), req.getDomainId())) {
|
if (!Objects.equals(group.getModelId(), req.getModelId())) {
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
if (!CollectionUtils.isEmpty(group.getAuthorizedUsers()) && group.getAuthorizedUsers()
|
if (!CollectionUtils.isEmpty(group.getAuthorizedUsers()) && group.getAuthorizedUsers()
|
||||||
|
|||||||
@@ -1,9 +1,9 @@
|
|||||||
package com.tencent.supersonic.auth.authorization.rest;
|
package com.tencent.supersonic.auth.authorization.rest;
|
||||||
|
|
||||||
|
import com.tencent.supersonic.auth.api.authorization.pojo.AuthGroup;
|
||||||
import com.tencent.supersonic.auth.api.authorization.request.QueryAuthResReq;
|
import com.tencent.supersonic.auth.api.authorization.request.QueryAuthResReq;
|
||||||
import com.tencent.supersonic.auth.api.authorization.response.AuthorizedResourceResp;
|
import com.tencent.supersonic.auth.api.authorization.response.AuthorizedResourceResp;
|
||||||
import com.tencent.supersonic.auth.api.authorization.service.AuthService;
|
import com.tencent.supersonic.auth.api.authorization.service.AuthService;
|
||||||
import com.tencent.supersonic.auth.api.authorization.pojo.AuthGroup;
|
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
import javax.servlet.http.HttpServletRequest;
|
import javax.servlet.http.HttpServletRequest;
|
||||||
import lombok.extern.slf4j.Slf4j;
|
import lombok.extern.slf4j.Slf4j;
|
||||||
@@ -26,9 +26,9 @@ public class AuthController {
|
|||||||
}
|
}
|
||||||
|
|
||||||
@GetMapping("/queryGroup")
|
@GetMapping("/queryGroup")
|
||||||
public List<AuthGroup> queryAuthGroup(@RequestParam("domainId") String domainId,
|
public List<AuthGroup> queryAuthGroup(@RequestParam("modelId") String modelId,
|
||||||
@RequestParam(value = "groupId", required = false) Integer groupId) {
|
@RequestParam(value = "groupId", required = false) Integer groupId) {
|
||||||
return authService.queryAuthGroups(domainId, groupId);
|
return authService.queryAuthGroups(modelId, groupId);
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|||||||
@@ -2,17 +2,18 @@ package com.tencent.supersonic.chat.api.component;
|
|||||||
|
|
||||||
import com.github.pagehelper.PageInfo;
|
import com.github.pagehelper.PageInfo;
|
||||||
import com.tencent.supersonic.auth.api.authentication.pojo.User;
|
import com.tencent.supersonic.auth.api.authentication.pojo.User;
|
||||||
import com.tencent.supersonic.chat.api.pojo.DomainSchema;
|
import com.tencent.supersonic.chat.api.pojo.ModelSchema;
|
||||||
|
import com.tencent.supersonic.common.pojo.enums.AuthType;
|
||||||
import com.tencent.supersonic.semantic.api.model.request.PageDimensionReq;
|
import com.tencent.supersonic.semantic.api.model.request.PageDimensionReq;
|
||||||
import com.tencent.supersonic.semantic.api.model.request.PageMetricReq;
|
import com.tencent.supersonic.semantic.api.model.request.PageMetricReq;
|
||||||
import com.tencent.supersonic.semantic.api.model.response.DimensionResp;
|
import com.tencent.supersonic.semantic.api.model.response.DimensionResp;
|
||||||
import com.tencent.supersonic.semantic.api.model.response.DomainResp;
|
import com.tencent.supersonic.semantic.api.model.response.DomainResp;
|
||||||
import com.tencent.supersonic.semantic.api.model.response.QueryResultWithSchemaResp;
|
|
||||||
import com.tencent.supersonic.semantic.api.model.response.MetricResp;
|
import com.tencent.supersonic.semantic.api.model.response.MetricResp;
|
||||||
|
import com.tencent.supersonic.semantic.api.model.response.ModelResp;
|
||||||
|
import com.tencent.supersonic.semantic.api.model.response.QueryResultWithSchemaResp;
|
||||||
import com.tencent.supersonic.semantic.api.query.request.QueryDslReq;
|
import com.tencent.supersonic.semantic.api.query.request.QueryDslReq;
|
||||||
import com.tencent.supersonic.semantic.api.query.request.QueryMultiStructReq;
|
import com.tencent.supersonic.semantic.api.query.request.QueryMultiStructReq;
|
||||||
import com.tencent.supersonic.semantic.api.query.request.QueryStructReq;
|
import com.tencent.supersonic.semantic.api.query.request.QueryStructReq;
|
||||||
|
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -30,16 +31,22 @@ import java.util.List;
|
|||||||
public interface SemanticLayer {
|
public interface SemanticLayer {
|
||||||
|
|
||||||
QueryResultWithSchemaResp queryByStruct(QueryStructReq queryStructReq, User user);
|
QueryResultWithSchemaResp queryByStruct(QueryStructReq queryStructReq, User user);
|
||||||
|
|
||||||
QueryResultWithSchemaResp queryByMultiStruct(QueryMultiStructReq queryMultiStructReq, User user);
|
QueryResultWithSchemaResp queryByMultiStruct(QueryMultiStructReq queryMultiStructReq, User user);
|
||||||
|
|
||||||
QueryResultWithSchemaResp queryByDsl(QueryDslReq queryDslReq, User user);
|
QueryResultWithSchemaResp queryByDsl(QueryDslReq queryDslReq, User user);
|
||||||
|
|
||||||
List<DomainSchema> getDomainSchema();
|
List<ModelSchema> getModelSchema();
|
||||||
List<DomainSchema> getDomainSchema(List<Long> ids);
|
|
||||||
DomainSchema getDomainSchema(Long domain, Boolean cacheEnable);
|
List<ModelSchema> getModelSchema(List<Long> ids);
|
||||||
|
|
||||||
|
ModelSchema getModelSchema(Long model, Boolean cacheEnable);
|
||||||
|
|
||||||
PageInfo<DimensionResp> getDimensionPage(PageDimensionReq pageDimensionCmd);
|
PageInfo<DimensionResp> getDimensionPage(PageDimensionReq pageDimensionCmd);
|
||||||
|
|
||||||
PageInfo<MetricResp> getMetricPage(PageMetricReq pageMetricCmd);
|
PageInfo<MetricResp> getMetricPage(PageMetricReq pageMetricCmd);
|
||||||
|
|
||||||
List<DomainResp> getDomainListForViewer();
|
List<DomainResp> getDomainList(User user);
|
||||||
List<DomainResp> getDomainListForAdmin();
|
|
||||||
|
|
||||||
|
List<ModelResp> getModelList(AuthType authType, Long domainId, User user);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,15 +1,14 @@
|
|||||||
package com.tencent.supersonic.chat.api.pojo;
|
package com.tencent.supersonic.chat.api.pojo;
|
||||||
|
|
||||||
import lombok.Data;
|
|
||||||
|
|
||||||
import java.util.HashSet;
|
import java.util.HashSet;
|
||||||
import java.util.Optional;
|
import java.util.Optional;
|
||||||
import java.util.Set;
|
import java.util.Set;
|
||||||
|
import lombok.Data;
|
||||||
|
|
||||||
@Data
|
@Data
|
||||||
public class DomainSchema {
|
public class ModelSchema {
|
||||||
|
|
||||||
private SchemaElement domain;
|
private SchemaElement model;
|
||||||
private Set<SchemaElement> metrics = new HashSet<>();
|
private Set<SchemaElement> metrics = new HashSet<>();
|
||||||
private Set<SchemaElement> dimensions = new HashSet<>();
|
private Set<SchemaElement> dimensions = new HashSet<>();
|
||||||
private Set<SchemaElement> dimensionValues = new HashSet<>();
|
private Set<SchemaElement> dimensionValues = new HashSet<>();
|
||||||
@@ -22,8 +21,8 @@ public class DomainSchema {
|
|||||||
case ENTITY:
|
case ENTITY:
|
||||||
element = Optional.ofNullable(entity);
|
element = Optional.ofNullable(entity);
|
||||||
break;
|
break;
|
||||||
case DOMAIN:
|
case MODEL:
|
||||||
element = Optional.of(domain);
|
element = Optional.of(model);
|
||||||
break;
|
break;
|
||||||
case METRIC:
|
case METRIC:
|
||||||
element = metrics.stream().filter(e -> e.getId() == elementID).findFirst();
|
element = metrics.stream().filter(e -> e.getId() == elementID).findFirst();
|
||||||
@@ -2,12 +2,13 @@ package com.tencent.supersonic.chat.api.pojo;
|
|||||||
|
|
||||||
import com.tencent.supersonic.chat.api.component.SemanticQuery;
|
import com.tencent.supersonic.chat.api.component.SemanticQuery;
|
||||||
import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
|
import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
|
||||||
import lombok.Data;
|
|
||||||
|
|
||||||
import java.util.ArrayList;
|
import java.util.ArrayList;
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
|
import lombok.Data;
|
||||||
|
import lombok.NoArgsConstructor;
|
||||||
|
|
||||||
@Data
|
@Data
|
||||||
|
@NoArgsConstructor
|
||||||
public class QueryContext {
|
public class QueryContext {
|
||||||
|
|
||||||
private QueryReq request;
|
private QueryReq request;
|
||||||
|
|||||||
@@ -1,11 +1,12 @@
|
|||||||
package com.tencent.supersonic.chat.api.pojo;
|
package com.tencent.supersonic.chat.api.pojo;
|
||||||
|
|
||||||
import com.google.common.base.Objects;
|
import com.google.common.base.Objects;
|
||||||
|
|
||||||
import java.io.Serializable;
|
import java.io.Serializable;
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
|
import lombok.Builder;
|
||||||
import lombok.*;
|
import lombok.Data;
|
||||||
|
import lombok.Getter;
|
||||||
|
import lombok.NoArgsConstructor;
|
||||||
|
|
||||||
@Data
|
@Data
|
||||||
@Getter
|
@Getter
|
||||||
@@ -14,7 +15,7 @@ import lombok.*;
|
|||||||
//@AllArgsConstructor
|
//@AllArgsConstructor
|
||||||
public class SchemaElement implements Serializable {
|
public class SchemaElement implements Serializable {
|
||||||
|
|
||||||
private Long domain;
|
private Long model;
|
||||||
private Long id;
|
private Long id;
|
||||||
private String name;
|
private String name;
|
||||||
private String bizName;
|
private String bizName;
|
||||||
@@ -25,9 +26,9 @@ public class SchemaElement implements Serializable {
|
|||||||
// public SchemaElement() {
|
// public SchemaElement() {
|
||||||
// }
|
// }
|
||||||
|
|
||||||
public SchemaElement(Long domain, Long id, String name, String bizName,
|
public SchemaElement(Long model, Long id, String name, String bizName,
|
||||||
Long useCnt, SchemaElementType type, List<String> alias) {
|
Long useCnt, SchemaElementType type, List<String> alias) {
|
||||||
this.domain = domain;
|
this.model = model;
|
||||||
this.id = id;
|
this.id = id;
|
||||||
this.name = name;
|
this.name = name;
|
||||||
this.bizName = bizName;
|
this.bizName = bizName;
|
||||||
@@ -45,7 +46,7 @@ public class SchemaElement implements Serializable {
|
|||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
SchemaElement schemaElement = (SchemaElement) o;
|
SchemaElement schemaElement = (SchemaElement) o;
|
||||||
return Objects.equal(domain, schemaElement.domain) && Objects.equal(id,
|
return Objects.equal(model, schemaElement.model) && Objects.equal(id,
|
||||||
schemaElement.id) && Objects.equal(name, schemaElement.name)
|
schemaElement.id) && Objects.equal(name, schemaElement.name)
|
||||||
&& Objects.equal(bizName, schemaElement.bizName) && Objects.equal(
|
&& Objects.equal(bizName, schemaElement.bizName) && Objects.equal(
|
||||||
useCnt, schemaElement.useCnt) && Objects.equal(type, schemaElement.type);
|
useCnt, schemaElement.useCnt) && Objects.equal(type, schemaElement.type);
|
||||||
@@ -53,6 +54,6 @@ public class SchemaElement implements Serializable {
|
|||||||
|
|
||||||
@Override
|
@Override
|
||||||
public int hashCode() {
|
public int hashCode() {
|
||||||
return Objects.hashCode(domain, id, name, bizName, useCnt, type);
|
return Objects.hashCode(model, id, name, bizName, useCnt, type);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -18,10 +18,6 @@ public class SchemaElementMatch {
|
|||||||
String detectWord;
|
String detectWord;
|
||||||
String word;
|
String word;
|
||||||
Long frequency;
|
Long frequency;
|
||||||
MatchMode mode = MatchMode.CURRENT;
|
boolean isInherited;
|
||||||
|
|
||||||
public enum MatchMode {
|
|
||||||
CURRENT,
|
|
||||||
INHERIT
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
package com.tencent.supersonic.chat.api.pojo;
|
package com.tencent.supersonic.chat.api.pojo;
|
||||||
|
|
||||||
public enum SchemaElementType {
|
public enum SchemaElementType {
|
||||||
DOMAIN,
|
MODEL,
|
||||||
METRIC,
|
METRIC,
|
||||||
DIMENSION,
|
DIMENSION,
|
||||||
VALUE,
|
VALUE,
|
||||||
|
|||||||
@@ -7,21 +7,21 @@ import java.util.Set;
|
|||||||
|
|
||||||
public class SchemaMapInfo {
|
public class SchemaMapInfo {
|
||||||
|
|
||||||
private Map<Long, List<SchemaElementMatch>> domainElementMatches = new HashMap<>();
|
private Map<Long, List<SchemaElementMatch>> modelElementMatches = new HashMap<>();
|
||||||
|
|
||||||
public Set<Long> getMatchedDomains() {
|
public Set<Long> getMatchedModels() {
|
||||||
return domainElementMatches.keySet();
|
return modelElementMatches.keySet();
|
||||||
}
|
}
|
||||||
|
|
||||||
public List<SchemaElementMatch> getMatchedElements(Long domain) {
|
public List<SchemaElementMatch> getMatchedElements(Long model) {
|
||||||
return domainElementMatches.get(domain);
|
return modelElementMatches.get(model);
|
||||||
}
|
}
|
||||||
|
|
||||||
public Map<Long, List<SchemaElementMatch>> getDomainElementMatches() {
|
public Map<Long, List<SchemaElementMatch>> getModelElementMatches() {
|
||||||
return domainElementMatches;
|
return modelElementMatches;
|
||||||
}
|
}
|
||||||
|
|
||||||
public void setMatchedElements(Long domain, List<SchemaElementMatch> elementMatches) {
|
public void setMatchedElements(Long model, List<SchemaElementMatch> elementMatches) {
|
||||||
domainElementMatches.put(domain, elementMatches);
|
modelElementMatches.put(model, elementMatches);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,19 +1,25 @@
|
|||||||
package com.tencent.supersonic.chat.api.pojo;
|
package com.tencent.supersonic.chat.api.pojo;
|
||||||
|
|
||||||
|
|
||||||
import java.util.*;
|
|
||||||
|
|
||||||
import com.tencent.supersonic.chat.api.pojo.request.QueryFilter;
|
import com.tencent.supersonic.chat.api.pojo.request.QueryFilter;
|
||||||
import com.tencent.supersonic.common.pojo.DateConf;
|
import com.tencent.supersonic.common.pojo.DateConf;
|
||||||
import com.tencent.supersonic.common.pojo.Order;
|
import com.tencent.supersonic.common.pojo.Order;
|
||||||
import com.tencent.supersonic.common.pojo.enums.AggregateTypeEnum;
|
import com.tencent.supersonic.common.pojo.enums.AggregateTypeEnum;
|
||||||
|
import java.util.ArrayList;
|
||||||
|
import java.util.Comparator;
|
||||||
|
import java.util.HashMap;
|
||||||
|
import java.util.LinkedHashSet;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Map;
|
||||||
|
import java.util.Set;
|
||||||
|
import java.util.TreeSet;
|
||||||
import lombok.Data;
|
import lombok.Data;
|
||||||
|
|
||||||
@Data
|
@Data
|
||||||
public class SemanticParseInfo {
|
public class SemanticParseInfo {
|
||||||
|
|
||||||
private String queryMode;
|
private String queryMode;
|
||||||
private SchemaElement domain;
|
private SchemaElement model;
|
||||||
private Set<SchemaElement> metrics = new TreeSet<>(new SchemaNameLengthComparator());
|
private Set<SchemaElement> metrics = new TreeSet<>(new SchemaNameLengthComparator());
|
||||||
private Set<SchemaElement> dimensions = new LinkedHashSet();
|
private Set<SchemaElement> dimensions = new LinkedHashSet();
|
||||||
private SchemaElement entity;
|
private SchemaElement entity;
|
||||||
@@ -28,15 +34,16 @@ public class SemanticParseInfo {
|
|||||||
private List<SchemaElementMatch> elementMatches = new ArrayList<>();
|
private List<SchemaElementMatch> elementMatches = new ArrayList<>();
|
||||||
private Map<String, Object> properties = new HashMap<>();
|
private Map<String, Object> properties = new HashMap<>();
|
||||||
|
|
||||||
public Long getDomainId() {
|
public Long getModelId() {
|
||||||
return domain != null ? domain.getId() : 0L;
|
return model != null ? model.getId() : 0L;
|
||||||
}
|
}
|
||||||
|
|
||||||
public String getDomainName() {
|
public String getModelName() {
|
||||||
return domain != null ? domain.getName() : "null";
|
return model != null ? model.getName() : "null";
|
||||||
}
|
}
|
||||||
|
|
||||||
private static class SchemaNameLengthComparator implements Comparator<SchemaElement> {
|
private static class SchemaNameLengthComparator implements Comparator<SchemaElement> {
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public int compare(SchemaElement o1, SchemaElement o2) {
|
public int compare(SchemaElement o1, SchemaElement o2) {
|
||||||
int len1 = o1.getName().length();
|
int len1 = o1.getName().length();
|
||||||
@@ -49,4 +56,11 @@ public class SemanticParseInfo {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public Set<SchemaElement> getMetrics() {
|
||||||
|
Set<SchemaElement> metricSet = new TreeSet<>(new SchemaNameLengthComparator());
|
||||||
|
metricSet.addAll(metrics);
|
||||||
|
metrics = metricSet;
|
||||||
|
return metrics;
|
||||||
|
}
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -7,48 +7,49 @@ import java.util.Map;
|
|||||||
import java.util.stream.Collectors;
|
import java.util.stream.Collectors;
|
||||||
|
|
||||||
public class SemanticSchema implements Serializable {
|
public class SemanticSchema implements Serializable {
|
||||||
private List<DomainSchema> domainSchemaList;
|
|
||||||
|
|
||||||
public SemanticSchema(List<DomainSchema> domainSchemaList) {
|
private List<ModelSchema> modelSchemaList;
|
||||||
this.domainSchemaList = domainSchemaList;
|
|
||||||
|
public SemanticSchema(List<ModelSchema> modelSchemaList) {
|
||||||
|
this.modelSchemaList = modelSchemaList;
|
||||||
}
|
}
|
||||||
|
|
||||||
public void add(DomainSchema schema) {
|
public void add(ModelSchema schema) {
|
||||||
domainSchemaList.add(schema);
|
modelSchemaList.add(schema);
|
||||||
}
|
}
|
||||||
|
|
||||||
public Map<Long, String> getDomainIdToName() {
|
public Map<Long, String> getModelIdToName() {
|
||||||
return domainSchemaList.stream()
|
return modelSchemaList.stream()
|
||||||
.collect(Collectors.toMap(a -> a.getDomain().getId(), a -> a.getDomain().getName(), (k1, k2) -> k1));
|
.collect(Collectors.toMap(a -> a.getModel().getId(), a -> a.getModel().getName(), (k1, k2) -> k1));
|
||||||
}
|
}
|
||||||
|
|
||||||
public List<SchemaElement> getDimensionValues() {
|
public List<SchemaElement> getDimensionValues() {
|
||||||
List<SchemaElement> dimensionValues = new ArrayList<>();
|
List<SchemaElement> dimensionValues = new ArrayList<>();
|
||||||
domainSchemaList.stream().forEach(d -> dimensionValues.addAll(d.getDimensionValues()));
|
modelSchemaList.stream().forEach(d -> dimensionValues.addAll(d.getDimensionValues()));
|
||||||
return dimensionValues;
|
return dimensionValues;
|
||||||
}
|
}
|
||||||
|
|
||||||
public List<SchemaElement> getDimensions() {
|
public List<SchemaElement> getDimensions() {
|
||||||
List<SchemaElement> dimensions = new ArrayList<>();
|
List<SchemaElement> dimensions = new ArrayList<>();
|
||||||
domainSchemaList.stream().forEach(d -> dimensions.addAll(d.getDimensions()));
|
modelSchemaList.stream().forEach(d -> dimensions.addAll(d.getDimensions()));
|
||||||
return dimensions;
|
return dimensions;
|
||||||
}
|
}
|
||||||
|
|
||||||
public List<SchemaElement> getMetrics() {
|
public List<SchemaElement> getMetrics() {
|
||||||
List<SchemaElement> metrics = new ArrayList<>();
|
List<SchemaElement> metrics = new ArrayList<>();
|
||||||
domainSchemaList.stream().forEach(d -> metrics.addAll(d.getMetrics()));
|
modelSchemaList.stream().forEach(d -> metrics.addAll(d.getMetrics()));
|
||||||
return metrics;
|
return metrics;
|
||||||
}
|
}
|
||||||
|
|
||||||
public List<SchemaElement> getDomains() {
|
public List<SchemaElement> getModels() {
|
||||||
List<SchemaElement> domains = new ArrayList<>();
|
List<SchemaElement> models = new ArrayList<>();
|
||||||
domainSchemaList.stream().forEach(d -> domains.add(d.getDomain()));
|
modelSchemaList.stream().forEach(d -> models.add(d.getModel()));
|
||||||
return domains;
|
return models;
|
||||||
}
|
}
|
||||||
|
|
||||||
public List<SchemaElement> getEntities() {
|
public List<SchemaElement> getEntities() {
|
||||||
List<SchemaElement> entities = new ArrayList<>();
|
List<SchemaElement> entities = new ArrayList<>();
|
||||||
domainSchemaList.stream().forEach(d -> entities.add(d.getEntity()));
|
modelSchemaList.stream().forEach(d -> entities.add(d.getEntity()));
|
||||||
return entities;
|
return entities;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,8 +1,7 @@
|
|||||||
package com.tencent.supersonic.chat.api.pojo.request;
|
package com.tencent.supersonic.chat.api.pojo.request;
|
||||||
|
|
||||||
import lombok.Data;
|
|
||||||
|
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
|
import lombok.Data;
|
||||||
|
|
||||||
@Data
|
@Data
|
||||||
public class ChatAggConfigReq {
|
public class ChatAggConfigReq {
|
||||||
@@ -13,7 +12,7 @@ public class ChatAggConfigReq {
|
|||||||
private ItemVisibility visibility;
|
private ItemVisibility visibility;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* information about dictionary about the domain
|
* information about dictionary about the model
|
||||||
*/
|
*/
|
||||||
private List<KnowledgeInfoReq> knowledgeInfos;
|
private List<KnowledgeInfoReq> knowledgeInfos;
|
||||||
|
|
||||||
|
|||||||
@@ -1,34 +1,32 @@
|
|||||||
package com.tencent.supersonic.chat.api.pojo.request;
|
package com.tencent.supersonic.chat.api.pojo.request;
|
||||||
|
|
||||||
import com.tencent.supersonic.common.pojo.enums.StatusEnum;
|
import com.tencent.supersonic.common.pojo.enums.StatusEnum;
|
||||||
|
import java.util.List;
|
||||||
import lombok.Data;
|
import lombok.Data;
|
||||||
import lombok.ToString;
|
import lombok.ToString;
|
||||||
|
|
||||||
import java.util.List;
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* extended information command about domain
|
* extended information command about model
|
||||||
*/
|
*/
|
||||||
@Data
|
@Data
|
||||||
@ToString
|
@ToString
|
||||||
public class ChatConfigBaseReq {
|
public class ChatConfigBaseReq {
|
||||||
|
|
||||||
private Long domainId;
|
private Long modelId;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* the chatDetailConfig about the domain
|
* the chatDetailConfig about the model
|
||||||
*/
|
*/
|
||||||
private ChatDetailConfigReq chatDetailConfig;
|
private ChatDetailConfigReq chatDetailConfig;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* the chatAggConfig about the domain
|
* the chatAggConfig about the model
|
||||||
*/
|
*/
|
||||||
private ChatAggConfigReq chatAggConfig;
|
private ChatAggConfigReq chatAggConfig;
|
||||||
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* the recommended questions about the domain
|
* the recommended questions about the model
|
||||||
*/
|
*/
|
||||||
private List<RecommendedQuestionReq> recommendedQuestions;
|
private List<RecommendedQuestionReq> recommendedQuestions;
|
||||||
|
|
||||||
|
|||||||
@@ -10,6 +10,6 @@ import lombok.NoArgsConstructor;
|
|||||||
public class ChatConfigFilter {
|
public class ChatConfigFilter {
|
||||||
|
|
||||||
private Long id;
|
private Long id;
|
||||||
private Long domainId;
|
private Long modelId;
|
||||||
private StatusEnum status = StatusEnum.ONLINE;
|
private StatusEnum status = StatusEnum.ONLINE;
|
||||||
}
|
}
|
||||||
@@ -2,10 +2,9 @@ package com.tencent.supersonic.chat.api.pojo.request;
|
|||||||
|
|
||||||
|
|
||||||
import com.tencent.supersonic.common.pojo.Constants;
|
import com.tencent.supersonic.common.pojo.Constants;
|
||||||
import lombok.Data;
|
|
||||||
|
|
||||||
import java.util.ArrayList;
|
import java.util.ArrayList;
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
|
import lombok.Data;
|
||||||
|
|
||||||
@Data
|
@Data
|
||||||
public class ChatDefaultConfigReq {
|
public class ChatDefaultConfigReq {
|
||||||
|
|||||||
@@ -1,8 +1,7 @@
|
|||||||
package com.tencent.supersonic.chat.api.pojo.request;
|
package com.tencent.supersonic.chat.api.pojo.request;
|
||||||
|
|
||||||
import lombok.Data;
|
|
||||||
|
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
|
import lombok.Data;
|
||||||
|
|
||||||
@Data
|
@Data
|
||||||
public class ChatDetailConfigReq {
|
public class ChatDetailConfigReq {
|
||||||
@@ -13,7 +12,7 @@ public class ChatDetailConfigReq {
|
|||||||
private ItemVisibility visibility;
|
private ItemVisibility visibility;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* information about dictionary about the domain
|
* information about dictionary about the model
|
||||||
*/
|
*/
|
||||||
private List<KnowledgeInfoReq> knowledgeInfos;
|
private List<KnowledgeInfoReq> knowledgeInfos;
|
||||||
|
|
||||||
|
|||||||
@@ -7,7 +7,7 @@ import lombok.NoArgsConstructor;
|
|||||||
import lombok.ToString;
|
import lombok.ToString;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* the entity info about the domain
|
* the entity info about the model
|
||||||
*/
|
*/
|
||||||
@Data
|
@Data
|
||||||
@AllArgsConstructor
|
@AllArgsConstructor
|
||||||
|
|||||||
@@ -7,6 +7,7 @@ import lombok.Data;
|
|||||||
|
|
||||||
@Data
|
@Data
|
||||||
public class ExecuteQueryReq {
|
public class ExecuteQueryReq {
|
||||||
|
|
||||||
private User user;
|
private User user;
|
||||||
private Integer chatId;
|
private Integer chatId;
|
||||||
private String queryText;
|
private String queryText;
|
||||||
|
|||||||
@@ -1,9 +1,8 @@
|
|||||||
package com.tencent.supersonic.chat.api.pojo.request;
|
package com.tencent.supersonic.chat.api.pojo.request;
|
||||||
|
|
||||||
import lombok.Data;
|
|
||||||
|
|
||||||
import java.util.ArrayList;
|
import java.util.ArrayList;
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
|
import lombok.Data;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* advanced knowledge config
|
* advanced knowledge config
|
||||||
|
|||||||
@@ -1,20 +1,18 @@
|
|||||||
package com.tencent.supersonic.chat.api.pojo.request;
|
package com.tencent.supersonic.chat.api.pojo.request;
|
||||||
|
|
||||||
import com.tencent.supersonic.common.pojo.enums.TypeEnums;
|
import com.tencent.supersonic.common.pojo.enums.TypeEnums;
|
||||||
|
|
||||||
import javax.validation.constraints.NotNull;
|
import javax.validation.constraints.NotNull;
|
||||||
|
|
||||||
import lombok.Data;
|
import lombok.Data;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* information about dictionary about the domain
|
* information about dictionary about the model
|
||||||
*/
|
*/
|
||||||
|
|
||||||
@Data
|
@Data
|
||||||
public class KnowledgeInfoReq {
|
public class KnowledgeInfoReq {
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* metricId、DimensionId、domainId
|
* metricId、DimensionId、modelId
|
||||||
*/
|
*/
|
||||||
private Long itemId;
|
private Long itemId;
|
||||||
|
|
||||||
|
|||||||
@@ -13,7 +13,7 @@ public class PluginQueryReq {
|
|||||||
|
|
||||||
private String type;
|
private String type;
|
||||||
|
|
||||||
private String domain;
|
private String model;
|
||||||
|
|
||||||
private String pattern;
|
private String pattern;
|
||||||
|
|
||||||
|
|||||||
@@ -1,18 +1,18 @@
|
|||||||
package com.tencent.supersonic.chat.api.pojo.request;
|
package com.tencent.supersonic.chat.api.pojo.request;
|
||||||
|
|
||||||
|
|
||||||
import java.util.HashSet;
|
|
||||||
import java.util.Set;
|
|
||||||
|
|
||||||
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
|
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
|
||||||
import com.tencent.supersonic.common.pojo.DateConf;
|
import com.tencent.supersonic.common.pojo.DateConf;
|
||||||
import com.tencent.supersonic.common.pojo.Order;
|
import com.tencent.supersonic.common.pojo.Order;
|
||||||
|
import java.util.HashSet;
|
||||||
|
import java.util.Set;
|
||||||
import lombok.Data;
|
import lombok.Data;
|
||||||
|
|
||||||
@Data
|
@Data
|
||||||
public class QueryDataReq {
|
public class QueryDataReq {
|
||||||
|
|
||||||
String queryMode;
|
String queryMode;
|
||||||
SchemaElement domain;
|
SchemaElement model;
|
||||||
Set<SchemaElement> metrics = new HashSet<>();
|
Set<SchemaElement> metrics = new HashSet<>();
|
||||||
Set<SchemaElement> dimensions = new HashSet<>();
|
Set<SchemaElement> dimensions = new HashSet<>();
|
||||||
Set<QueryFilter> dimensionFilters = new HashSet<>();
|
Set<QueryFilter> dimensionFilters = new HashSet<>();
|
||||||
|
|||||||
@@ -1,13 +1,14 @@
|
|||||||
package com.tencent.supersonic.chat.api.pojo.request;
|
package com.tencent.supersonic.chat.api.pojo.request;
|
||||||
|
|
||||||
import lombok.Data;
|
|
||||||
import java.util.ArrayList;
|
import java.util.ArrayList;
|
||||||
import java.util.HashMap;
|
import java.util.HashMap;
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
import java.util.Map;
|
import java.util.Map;
|
||||||
|
import lombok.Data;
|
||||||
|
|
||||||
@Data
|
@Data
|
||||||
public class QueryFilters {
|
public class QueryFilters {
|
||||||
|
|
||||||
private List<QueryFilter> filters = new ArrayList<>();
|
private List<QueryFilter> filters = new ArrayList<>();
|
||||||
private Map<String, Object> params = new HashMap<>();
|
private Map<String, Object> params = new HashMap<>();
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -5,9 +5,10 @@ import lombok.Data;
|
|||||||
|
|
||||||
@Data
|
@Data
|
||||||
public class QueryReq {
|
public class QueryReq {
|
||||||
|
|
||||||
private String queryText;
|
private String queryText;
|
||||||
private Integer chatId;
|
private Integer chatId;
|
||||||
private Long domainId = 0L;
|
private Long modelId = 0L;
|
||||||
private User user;
|
private User user;
|
||||||
private QueryFilters queryFilters;
|
private QueryFilters queryFilters;
|
||||||
private boolean saveAnswer = true;
|
private boolean saveAnswer = true;
|
||||||
|
|||||||
@@ -6,5 +6,6 @@ import lombok.Data;
|
|||||||
|
|
||||||
@Data
|
@Data
|
||||||
public class AggregateInfo {
|
public class AggregateInfo {
|
||||||
private List<MetricInfo> metricInfos = new ArrayList<>();
|
|
||||||
|
private List<MetricInfo> metricInfos = new ArrayList<>();
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -2,9 +2,8 @@ package com.tencent.supersonic.chat.api.pojo.response;
|
|||||||
|
|
||||||
import com.tencent.supersonic.chat.api.pojo.request.KnowledgeAdvancedConfig;
|
import com.tencent.supersonic.chat.api.pojo.request.KnowledgeAdvancedConfig;
|
||||||
import com.tencent.supersonic.chat.api.pojo.request.KnowledgeInfoReq;
|
import com.tencent.supersonic.chat.api.pojo.request.KnowledgeInfoReq;
|
||||||
import lombok.Data;
|
|
||||||
|
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
|
import lombok.Data;
|
||||||
|
|
||||||
@Data
|
@Data
|
||||||
public class ChatAggRichConfigResp {
|
public class ChatAggRichConfigResp {
|
||||||
@@ -15,7 +14,7 @@ public class ChatAggRichConfigResp {
|
|||||||
private ItemVisibilityInfo visibility;
|
private ItemVisibilityInfo visibility;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* information about dictionary about the domain
|
* information about dictionary about the model
|
||||||
*/
|
*/
|
||||||
private List<KnowledgeInfoReq> knowledgeInfos;
|
private List<KnowledgeInfoReq> knowledgeInfos;
|
||||||
|
|
||||||
|
|||||||
@@ -4,10 +4,8 @@ import com.tencent.supersonic.chat.api.pojo.request.ChatAggConfigReq;
|
|||||||
import com.tencent.supersonic.chat.api.pojo.request.ChatDetailConfigReq;
|
import com.tencent.supersonic.chat.api.pojo.request.ChatDetailConfigReq;
|
||||||
import com.tencent.supersonic.chat.api.pojo.request.RecommendedQuestionReq;
|
import com.tencent.supersonic.chat.api.pojo.request.RecommendedQuestionReq;
|
||||||
import com.tencent.supersonic.common.pojo.enums.StatusEnum;
|
import com.tencent.supersonic.common.pojo.enums.StatusEnum;
|
||||||
|
|
||||||
import java.util.Date;
|
import java.util.Date;
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
|
|
||||||
import lombok.Data;
|
import lombok.Data;
|
||||||
|
|
||||||
@Data
|
@Data
|
||||||
@@ -15,7 +13,7 @@ public class ChatConfigResp {
|
|||||||
|
|
||||||
private Long id;
|
private Long id;
|
||||||
|
|
||||||
private Long domainId;
|
private Long modelId;
|
||||||
|
|
||||||
private ChatDetailConfigReq chatDetailConfig;
|
private ChatDetailConfigReq chatDetailConfig;
|
||||||
|
|
||||||
|
|||||||
@@ -4,7 +4,6 @@ import com.tencent.supersonic.chat.api.pojo.request.RecommendedQuestionReq;
|
|||||||
import com.tencent.supersonic.common.pojo.enums.StatusEnum;
|
import com.tencent.supersonic.common.pojo.enums.StatusEnum;
|
||||||
import java.util.Date;
|
import java.util.Date;
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
|
|
||||||
import lombok.Data;
|
import lombok.Data;
|
||||||
|
|
||||||
@Data
|
@Data
|
||||||
@@ -12,9 +11,9 @@ public class ChatConfigRichResp {
|
|||||||
|
|
||||||
private Long id;
|
private Long id;
|
||||||
|
|
||||||
private Long domainId;
|
private Long modelId;
|
||||||
|
|
||||||
private String domainName;
|
private String modelName;
|
||||||
private String bizName;
|
private String bizName;
|
||||||
|
|
||||||
private ChatAggRichConfigResp chatAggRichConfig;
|
private ChatAggRichConfigResp chatAggRichConfig;
|
||||||
|
|||||||
@@ -4,9 +4,8 @@ package com.tencent.supersonic.chat.api.pojo.response;
|
|||||||
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
|
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
|
||||||
import com.tencent.supersonic.chat.api.pojo.request.ChatDefaultConfigReq;
|
import com.tencent.supersonic.chat.api.pojo.request.ChatDefaultConfigReq;
|
||||||
import com.tencent.supersonic.common.pojo.Constants;
|
import com.tencent.supersonic.common.pojo.Constants;
|
||||||
import lombok.Data;
|
|
||||||
|
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
|
import lombok.Data;
|
||||||
|
|
||||||
@Data
|
@Data
|
||||||
public class ChatDefaultRichConfigResp {
|
public class ChatDefaultRichConfigResp {
|
||||||
|
|||||||
@@ -2,9 +2,8 @@ package com.tencent.supersonic.chat.api.pojo.response;
|
|||||||
|
|
||||||
import com.tencent.supersonic.chat.api.pojo.request.KnowledgeAdvancedConfig;
|
import com.tencent.supersonic.chat.api.pojo.request.KnowledgeAdvancedConfig;
|
||||||
import com.tencent.supersonic.chat.api.pojo.request.KnowledgeInfoReq;
|
import com.tencent.supersonic.chat.api.pojo.request.KnowledgeInfoReq;
|
||||||
import lombok.Data;
|
|
||||||
|
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
|
import lombok.Data;
|
||||||
|
|
||||||
@Data
|
@Data
|
||||||
public class ChatDetailRichConfigResp {
|
public class ChatDetailRichConfigResp {
|
||||||
@@ -15,7 +14,7 @@ public class ChatDetailRichConfigResp {
|
|||||||
private ItemVisibilityInfo visibility;
|
private ItemVisibilityInfo visibility;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* information about dictionary about the domain
|
* information about dictionary about the model
|
||||||
*/
|
*/
|
||||||
private List<KnowledgeInfoReq> knowledgeInfos;
|
private List<KnowledgeInfoReq> knowledgeInfos;
|
||||||
|
|
||||||
|
|||||||
@@ -7,7 +7,7 @@ import lombok.Data;
|
|||||||
@Data
|
@Data
|
||||||
public class EntityInfo {
|
public class EntityInfo {
|
||||||
|
|
||||||
private DomainInfo domainInfo = new DomainInfo();
|
private ModelInfo modelInfo = new ModelInfo();
|
||||||
private List<DataInfo> dimensions = new ArrayList<>();
|
private List<DataInfo> dimensions = new ArrayList<>();
|
||||||
private List<DataInfo> metrics = new ArrayList<>();
|
private List<DataInfo> metrics = new ArrayList<>();
|
||||||
private String entityId;
|
private String entityId;
|
||||||
|
|||||||
@@ -1,14 +1,14 @@
|
|||||||
package com.tencent.supersonic.chat.api.pojo.response;
|
package com.tencent.supersonic.chat.api.pojo.response;
|
||||||
|
|
||||||
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
|
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
|
||||||
|
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
import lombok.Data;
|
import lombok.Data;
|
||||||
|
|
||||||
@Data
|
@Data
|
||||||
public class EntityRichInfoResp {
|
public class EntityRichInfoResp {
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* entity alias
|
* entity alias
|
||||||
*/
|
*/
|
||||||
private List<String> names;
|
private List<String> names;
|
||||||
|
|
||||||
|
|||||||
@@ -5,7 +5,7 @@ import java.util.List;
|
|||||||
import lombok.Data;
|
import lombok.Data;
|
||||||
|
|
||||||
@Data
|
@Data
|
||||||
public class DomainInfo extends DataInfo implements Serializable {
|
public class ModelInfo extends DataInfo implements Serializable {
|
||||||
|
|
||||||
private List<String> words;
|
private List<String> words;
|
||||||
private String primaryEntityBizName;
|
private String primaryEntityBizName;
|
||||||
@@ -1,9 +1,12 @@
|
|||||||
package com.tencent.supersonic.chat.api.pojo.response;
|
package com.tencent.supersonic.chat.api.pojo.response;
|
||||||
|
|
||||||
import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
|
import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
|
||||||
import lombok.*;
|
|
||||||
|
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
|
import lombok.AllArgsConstructor;
|
||||||
|
import lombok.Builder;
|
||||||
|
import lombok.Data;
|
||||||
|
import lombok.Getter;
|
||||||
|
import lombok.NoArgsConstructor;
|
||||||
|
|
||||||
@Data
|
@Data
|
||||||
@Getter
|
@Getter
|
||||||
@@ -11,6 +14,7 @@ import java.util.List;
|
|||||||
@NoArgsConstructor
|
@NoArgsConstructor
|
||||||
@AllArgsConstructor
|
@AllArgsConstructor
|
||||||
public class ParseResp {
|
public class ParseResp {
|
||||||
|
|
||||||
private Integer chatId;
|
private Integer chatId;
|
||||||
private String queryText;
|
private String queryText;
|
||||||
private ParseState state;
|
private ParseState state;
|
||||||
|
|||||||
@@ -1,14 +1,14 @@
|
|||||||
package com.tencent.supersonic.chat.api.pojo.response;
|
package com.tencent.supersonic.chat.api.pojo.response;
|
||||||
|
|
||||||
import com.tencent.supersonic.chat.api.pojo.request.RecommendedQuestionReq;
|
import com.tencent.supersonic.chat.api.pojo.request.RecommendedQuestionReq;
|
||||||
|
import java.util.List;
|
||||||
import lombok.AllArgsConstructor;
|
import lombok.AllArgsConstructor;
|
||||||
import lombok.Data;
|
import lombok.Data;
|
||||||
|
|
||||||
import java.util.List;
|
|
||||||
|
|
||||||
@Data
|
@Data
|
||||||
@AllArgsConstructor
|
@AllArgsConstructor
|
||||||
public class RecommendQuestionResp {
|
public class RecommendQuestionResp {
|
||||||
private Long domainId;
|
|
||||||
|
private Long modelId;
|
||||||
private List<RecommendedQuestionReq> recommendedQuestions;
|
private List<RecommendedQuestionReq> recommendedQuestions;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,12 +1,12 @@
|
|||||||
package com.tencent.supersonic.chat.api.pojo.response;
|
package com.tencent.supersonic.chat.api.pojo.response;
|
||||||
|
|
||||||
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
|
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
|
||||||
import lombok.Data;
|
|
||||||
|
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
|
import lombok.Data;
|
||||||
|
|
||||||
@Data
|
@Data
|
||||||
public class RecommendResp {
|
public class RecommendResp {
|
||||||
|
|
||||||
private List<SchemaElement> dimensions;
|
private List<SchemaElement> dimensions;
|
||||||
private List<SchemaElement> metrics;
|
private List<SchemaElement> metrics;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -2,8 +2,6 @@ package com.tencent.supersonic.chat.api.pojo.response;
|
|||||||
|
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
import lombok.Data;
|
import lombok.Data;
|
||||||
import lombok.Getter;
|
|
||||||
import lombok.Setter;
|
|
||||||
|
|
||||||
@Data
|
@Data
|
||||||
public class SearchResp {
|
public class SearchResp {
|
||||||
|
|||||||
@@ -17,9 +17,9 @@ public class SearchResult {
|
|||||||
|
|
||||||
private String subRecommend;
|
private String subRecommend;
|
||||||
|
|
||||||
private String domainName;
|
private String modelName;
|
||||||
|
|
||||||
private Long domainId;
|
private Long modelId;
|
||||||
|
|
||||||
private SchemaElementType schemaElementType;
|
private SchemaElementType schemaElementType;
|
||||||
|
|
||||||
@@ -34,12 +34,12 @@ public class SearchResult {
|
|||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
SearchResult searchResult1 = (SearchResult) o;
|
SearchResult searchResult1 = (SearchResult) o;
|
||||||
return Objects.equals(recommend, searchResult1.recommend) && Objects.equals(domainName,
|
return Objects.equals(recommend, searchResult1.recommend) && Objects.equals(modelName,
|
||||||
searchResult1.domainName);
|
searchResult1.modelName);
|
||||||
}
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public int hashCode() {
|
public int hashCode() {
|
||||||
return Objects.hash(recommend, domainName);
|
return Objects.hash(recommend, modelName);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
<?xml version="1.0" encoding="UTF-8"?>
|
<?xml version="1.0" encoding="UTF-8"?>
|
||||||
<project xmlns="http://maven.apache.org/POM/4.0.0"
|
<project xmlns="http://maven.apache.org/POM/4.0.0"
|
||||||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||||
<parent>
|
<parent>
|
||||||
<artifactId>chat</artifactId>
|
<artifactId>chat</artifactId>
|
||||||
<groupId>com.tencent.supersonic</groupId>
|
<groupId>com.tencent.supersonic</groupId>
|
||||||
|
|||||||
@@ -3,11 +3,13 @@ package com.tencent.supersonic.chat.config;
|
|||||||
import lombok.Data;
|
import lombok.Data;
|
||||||
import org.springframework.beans.factory.annotation.Value;
|
import org.springframework.beans.factory.annotation.Value;
|
||||||
import org.springframework.context.annotation.Configuration;
|
import org.springframework.context.annotation.Configuration;
|
||||||
|
|
||||||
@Configuration
|
@Configuration
|
||||||
@Data
|
@Data
|
||||||
public class AggregatorConfig {
|
public class AggregatorConfig {
|
||||||
@Value("${metric.aggregator.ratio.enable:true}")
|
|
||||||
private Boolean enableRatio;
|
@Value("${metric.aggregator.ratio.enable:true}")
|
||||||
|
private Boolean enableRatio;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -3,13 +3,12 @@ package com.tencent.supersonic.chat.config;
|
|||||||
import com.tencent.supersonic.chat.api.pojo.request.ChatAggConfigReq;
|
import com.tencent.supersonic.chat.api.pojo.request.ChatAggConfigReq;
|
||||||
import com.tencent.supersonic.chat.api.pojo.request.ChatDetailConfigReq;
|
import com.tencent.supersonic.chat.api.pojo.request.ChatDetailConfigReq;
|
||||||
import com.tencent.supersonic.chat.api.pojo.request.RecommendedQuestionReq;
|
import com.tencent.supersonic.chat.api.pojo.request.RecommendedQuestionReq;
|
||||||
import com.tencent.supersonic.common.pojo.enums.StatusEnum;
|
|
||||||
import com.tencent.supersonic.common.pojo.RecordInfo;
|
import com.tencent.supersonic.common.pojo.RecordInfo;
|
||||||
|
import com.tencent.supersonic.common.pojo.enums.StatusEnum;
|
||||||
|
import java.util.List;
|
||||||
import lombok.Data;
|
import lombok.Data;
|
||||||
import lombok.ToString;
|
import lombok.ToString;
|
||||||
|
|
||||||
import java.util.List;
|
|
||||||
|
|
||||||
@Data
|
@Data
|
||||||
@ToString
|
@ToString
|
||||||
public class ChatConfig {
|
public class ChatConfig {
|
||||||
@@ -19,15 +18,15 @@ public class ChatConfig {
|
|||||||
*/
|
*/
|
||||||
private Long id;
|
private Long id;
|
||||||
|
|
||||||
private Long domainId;
|
private Long modelId;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* the chatDetailConfig about the domain
|
* the chatDetailConfig about the model
|
||||||
*/
|
*/
|
||||||
private ChatDetailConfigReq chatDetailConfig;
|
private ChatDetailConfigReq chatDetailConfig;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* the chatAggConfig about the domain
|
* the chatAggConfig about the model
|
||||||
*/
|
*/
|
||||||
private ChatAggConfigReq chatAggConfig;
|
private ChatAggConfigReq chatAggConfig;
|
||||||
|
|
||||||
|
|||||||
@@ -6,6 +6,6 @@ import lombok.Data;
|
|||||||
public class ChatConfigFilterInternal {
|
public class ChatConfigFilterInternal {
|
||||||
|
|
||||||
private Long id;
|
private Long id;
|
||||||
private Long domainId;
|
private Long modelId;
|
||||||
private Integer status;
|
private Integer status;
|
||||||
}
|
}
|
||||||
@@ -6,7 +6,7 @@ import lombok.ToString;
|
|||||||
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* default metrics about the domain
|
* default metrics about the model
|
||||||
*/
|
*/
|
||||||
|
|
||||||
@ToString
|
@ToString
|
||||||
|
|||||||
@@ -7,6 +7,7 @@ import org.springframework.context.annotation.Configuration;
|
|||||||
@Configuration
|
@Configuration
|
||||||
@Data
|
@Data
|
||||||
public class FunctionCallInfoConfig {
|
public class FunctionCallInfoConfig {
|
||||||
|
|
||||||
@Value("${functionCall.url:}")
|
@Value("${functionCall.url:}")
|
||||||
private String url;
|
private String url;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,21 +0,0 @@
|
|||||||
package com.tencent.supersonic.chat.mapper;
|
|
||||||
|
|
||||||
import java.io.Serializable;
|
|
||||||
import lombok.Builder;
|
|
||||||
import lombok.Data;
|
|
||||||
import lombok.ToString;
|
|
||||||
|
|
||||||
@Data
|
|
||||||
@ToString
|
|
||||||
@Builder
|
|
||||||
public class DomainInfoStat implements Serializable {
|
|
||||||
|
|
||||||
private long domainCount;
|
|
||||||
|
|
||||||
private long metricDomainCount;
|
|
||||||
|
|
||||||
private long dimensionDomainCount;
|
|
||||||
|
|
||||||
private long dimensionValueDomainCount;
|
|
||||||
|
|
||||||
}
|
|
||||||
@@ -2,14 +2,19 @@ package com.tencent.supersonic.chat.mapper;
|
|||||||
|
|
||||||
|
|
||||||
import com.tencent.supersonic.chat.api.component.SchemaMapper;
|
import com.tencent.supersonic.chat.api.component.SchemaMapper;
|
||||||
import com.tencent.supersonic.chat.api.pojo.*;
|
import com.tencent.supersonic.chat.api.pojo.ModelSchema;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.QueryContext;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SchemaMapInfo;
|
||||||
import com.tencent.supersonic.chat.service.SemanticService;
|
import com.tencent.supersonic.chat.service.SemanticService;
|
||||||
import com.tencent.supersonic.common.util.ContextUtils;
|
import com.tencent.supersonic.common.util.ContextUtils;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.stream.Collectors;
|
||||||
import lombok.extern.slf4j.Slf4j;
|
import lombok.extern.slf4j.Slf4j;
|
||||||
import org.springframework.beans.BeanUtils;
|
import org.springframework.beans.BeanUtils;
|
||||||
import org.springframework.util.CollectionUtils;
|
import org.springframework.util.CollectionUtils;
|
||||||
import java.util.List;
|
|
||||||
import java.util.stream.Collectors;
|
|
||||||
|
|
||||||
|
|
||||||
@Slf4j
|
@Slf4j
|
||||||
@@ -18,20 +23,20 @@ public class EntityMapper implements SchemaMapper {
|
|||||||
@Override
|
@Override
|
||||||
public void map(QueryContext queryContext) {
|
public void map(QueryContext queryContext) {
|
||||||
SchemaMapInfo schemaMapInfo = queryContext.getMapInfo();
|
SchemaMapInfo schemaMapInfo = queryContext.getMapInfo();
|
||||||
for (Long domainId : schemaMapInfo.getMatchedDomains()) {
|
for (Long modelId : schemaMapInfo.getMatchedModels()) {
|
||||||
List<SchemaElementMatch> schemaElementMatchList = schemaMapInfo.getMatchedElements(domainId);
|
List<SchemaElementMatch> schemaElementMatchList = schemaMapInfo.getMatchedElements(modelId);
|
||||||
if (CollectionUtils.isEmpty(schemaElementMatchList)) {
|
if (CollectionUtils.isEmpty(schemaElementMatchList)) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
SchemaElement entity = getEntity(domainId);
|
SchemaElement entity = getEntity(modelId);
|
||||||
if (entity == null || entity.getId() == null) {
|
if (entity == null || entity.getId() == null) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
List<SchemaElementMatch> valueSchemaElements = schemaElementMatchList.stream().filter(schemaElementMatch ->
|
List<SchemaElementMatch> valueSchemaElements = schemaElementMatchList.stream().filter(schemaElementMatch ->
|
||||||
SchemaElementType.VALUE.equals(schemaElementMatch.getElement().getType()))
|
SchemaElementType.VALUE.equals(schemaElementMatch.getElement().getType()))
|
||||||
.collect(Collectors.toList());
|
.collect(Collectors.toList());
|
||||||
for (SchemaElementMatch schemaElementMatch : valueSchemaElements) {
|
for (SchemaElementMatch schemaElementMatch : valueSchemaElements) {
|
||||||
if (!entity.getId().equals(schemaElementMatch.getElement().getId())){
|
if (!entity.getId().equals(schemaElementMatch.getElement().getId())) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
if (!checkExistSameEntitySchemaElements(schemaElementMatch, schemaElementMatchList)) {
|
if (!checkExistSameEntitySchemaElements(schemaElementMatch, schemaElementMatchList)) {
|
||||||
@@ -46,7 +51,7 @@ public class EntityMapper implements SchemaMapper {
|
|||||||
}
|
}
|
||||||
|
|
||||||
private boolean checkExistSameEntitySchemaElements(SchemaElementMatch valueSchemaElementMatch,
|
private boolean checkExistSameEntitySchemaElements(SchemaElementMatch valueSchemaElementMatch,
|
||||||
List<SchemaElementMatch> schemaElementMatchList) {
|
List<SchemaElementMatch> schemaElementMatchList) {
|
||||||
List<SchemaElementMatch> entitySchemaElements = schemaElementMatchList.stream().filter(schemaElementMatch ->
|
List<SchemaElementMatch> entitySchemaElements = schemaElementMatchList.stream().filter(schemaElementMatch ->
|
||||||
SchemaElementType.ENTITY.equals(schemaElementMatch.getElement().getType()))
|
SchemaElementType.ENTITY.equals(schemaElementMatch.getElement().getType()))
|
||||||
.collect(Collectors.toList());
|
.collect(Collectors.toList());
|
||||||
@@ -58,11 +63,11 @@ public class EntityMapper implements SchemaMapper {
|
|||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
private SchemaElement getEntity(Long domainId) {
|
private SchemaElement getEntity(Long modelId) {
|
||||||
SemanticService semanticService = ContextUtils.getBean(SemanticService.class);
|
SemanticService semanticService = ContextUtils.getBean(SemanticService.class);
|
||||||
DomainSchema domainSchema = semanticService.getDomainSchema(domainId);
|
ModelSchema modelSchema = semanticService.getModelSchema(modelId);
|
||||||
if (domainSchema != null && domainSchema.getEntity() != null) {
|
if (modelSchema != null && modelSchema.getEntity() != null) {
|
||||||
return domainSchema.getEntity();
|
return modelSchema.getEntity();
|
||||||
}
|
}
|
||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -2,10 +2,14 @@ package com.tencent.supersonic.chat.mapper;
|
|||||||
|
|
||||||
import com.hankcs.hanlp.seg.common.Term;
|
import com.hankcs.hanlp.seg.common.Term;
|
||||||
import com.tencent.supersonic.chat.api.component.SchemaMapper;
|
import com.tencent.supersonic.chat.api.component.SchemaMapper;
|
||||||
import com.tencent.supersonic.chat.api.pojo.*;
|
|
||||||
import com.tencent.supersonic.chat.api.pojo.QueryContext;
|
import com.tencent.supersonic.chat.api.pojo.QueryContext;
|
||||||
import com.tencent.supersonic.knowledge.service.SchemaService;
|
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SchemaMapInfo;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SemanticSchema;
|
||||||
import com.tencent.supersonic.common.util.ContextUtils;
|
import com.tencent.supersonic.common.util.ContextUtils;
|
||||||
|
import com.tencent.supersonic.knowledge.service.SchemaService;
|
||||||
import com.tencent.supersonic.knowledge.utils.HanlpHelper;
|
import com.tencent.supersonic.knowledge.utils.HanlpHelper;
|
||||||
import java.util.ArrayList;
|
import java.util.ArrayList;
|
||||||
import java.util.Comparator;
|
import java.util.Comparator;
|
||||||
@@ -39,13 +43,13 @@ public class FuzzyNameMapper implements SchemaMapper {
|
|||||||
log.debug("after db mapper,mapInfo:{}", queryContext.getMapInfo());
|
log.debug("after db mapper,mapInfo:{}", queryContext.getMapInfo());
|
||||||
}
|
}
|
||||||
|
|
||||||
private void detectAndAddToSchema(QueryContext queryContext, List<Term> terms, List<SchemaElement> domains,
|
private void detectAndAddToSchema(QueryContext queryContext, List<Term> terms, List<SchemaElement> Models,
|
||||||
SchemaElementType schemaElementType) {
|
SchemaElementType schemaElementType) {
|
||||||
try {
|
try {
|
||||||
|
|
||||||
Map<String, Set<SchemaElement>> domainResultSet = getResultSet(queryContext, terms, domains);
|
Map<String, Set<SchemaElement>> ModelResultSet = getResultSet(queryContext, terms, Models);
|
||||||
|
|
||||||
addToSchemaMapInfo(domainResultSet, queryContext.getMapInfo(), schemaElementType);
|
addToSchemaMapInfo(ModelResultSet, queryContext.getMapInfo(), schemaElementType);
|
||||||
|
|
||||||
} catch (Exception e) {
|
} catch (Exception e) {
|
||||||
log.error("detectAndAddToSchema error", e);
|
log.error("detectAndAddToSchema error", e);
|
||||||
@@ -53,7 +57,7 @@ public class FuzzyNameMapper implements SchemaMapper {
|
|||||||
}
|
}
|
||||||
|
|
||||||
private Map<String, Set<SchemaElement>> getResultSet(QueryContext queryContext, List<Term> terms,
|
private Map<String, Set<SchemaElement>> getResultSet(QueryContext queryContext, List<Term> terms,
|
||||||
List<SchemaElement> domains) {
|
List<SchemaElement> Models) {
|
||||||
|
|
||||||
String queryText = queryContext.getRequest().getQueryText();
|
String queryText = queryContext.getRequest().getQueryText();
|
||||||
|
|
||||||
@@ -61,12 +65,12 @@ public class FuzzyNameMapper implements SchemaMapper {
|
|||||||
|
|
||||||
Double metricDimensionThresholdConfig = getThreshold(queryContext, mapperHelper);
|
Double metricDimensionThresholdConfig = getThreshold(queryContext, mapperHelper);
|
||||||
|
|
||||||
Map<String, Set<SchemaElement>> nameToItems = getNameToItems(domains);
|
Map<String, Set<SchemaElement>> nameToItems = getNameToItems(Models);
|
||||||
|
|
||||||
Map<Integer, Integer> regOffsetToLength = terms.stream().sorted(Comparator.comparing(Term::length))
|
Map<Integer, Integer> regOffsetToLength = terms.stream().sorted(Comparator.comparing(Term::length))
|
||||||
.collect(Collectors.toMap(Term::getOffset, term -> term.word.length(), (value1, value2) -> value2));
|
.collect(Collectors.toMap(Term::getOffset, term -> term.word.length(), (value1, value2) -> value2));
|
||||||
|
|
||||||
Map<String, Set<SchemaElement>> domainResultSet = new HashMap<>();
|
Map<String, Set<SchemaElement>> ModelResultSet = new HashMap<>();
|
||||||
for (Integer startIndex = 0; startIndex <= queryText.length() - 1; ) {
|
for (Integer startIndex = 0; startIndex <= queryText.length() - 1; ) {
|
||||||
for (Integer endIndex = startIndex; endIndex <= queryText.length(); ) {
|
for (Integer endIndex = startIndex; endIndex <= queryText.length(); ) {
|
||||||
endIndex = mapperHelper.getStepIndex(regOffsetToLength, endIndex);
|
endIndex = mapperHelper.getStepIndex(regOffsetToLength, endIndex);
|
||||||
@@ -82,7 +86,7 @@ public class FuzzyNameMapper implements SchemaMapper {
|
|||||||
|| mapperHelper.getSimilarity(detectSegment, name) < metricDimensionThresholdConfig) {
|
|| mapperHelper.getSimilarity(detectSegment, name) < metricDimensionThresholdConfig) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
Set<SchemaElement> preSchemaElements = domainResultSet.putIfAbsent(detectSegment,
|
Set<SchemaElement> preSchemaElements = ModelResultSet.putIfAbsent(detectSegment,
|
||||||
schemaElements);
|
schemaElements);
|
||||||
if (Objects.nonNull(preSchemaElements)) {
|
if (Objects.nonNull(preSchemaElements)) {
|
||||||
preSchemaElements.addAll(schemaElements);
|
preSchemaElements.addAll(schemaElements);
|
||||||
@@ -91,7 +95,7 @@ public class FuzzyNameMapper implements SchemaMapper {
|
|||||||
}
|
}
|
||||||
startIndex = mapperHelper.getStepIndex(regOffsetToLength, startIndex);
|
startIndex = mapperHelper.getStepIndex(regOffsetToLength, startIndex);
|
||||||
}
|
}
|
||||||
return domainResultSet;
|
return ModelResultSet;
|
||||||
}
|
}
|
||||||
|
|
||||||
private Double getThreshold(QueryContext queryContext, MapperHelper mapperHelper) {
|
private Double getThreshold(QueryContext queryContext, MapperHelper mapperHelper) {
|
||||||
@@ -99,9 +103,9 @@ public class FuzzyNameMapper implements SchemaMapper {
|
|||||||
Double metricDimensionThresholdConfig = mapperHelper.getMetricDimensionThresholdConfig();
|
Double metricDimensionThresholdConfig = mapperHelper.getMetricDimensionThresholdConfig();
|
||||||
Double metricDimensionMinThresholdConfig = mapperHelper.getMetricDimensionMinThresholdConfig();
|
Double metricDimensionMinThresholdConfig = mapperHelper.getMetricDimensionMinThresholdConfig();
|
||||||
|
|
||||||
Map<Long, List<SchemaElementMatch>> domainElementMatches = queryContext.getMapInfo()
|
Map<Long, List<SchemaElementMatch>> ModelElementMatches = queryContext.getMapInfo()
|
||||||
.getDomainElementMatches();
|
.getModelElementMatches();
|
||||||
boolean existElement = domainElementMatches.entrySet().stream()
|
boolean existElement = ModelElementMatches.entrySet().stream()
|
||||||
.anyMatch(entry -> entry.getValue().size() >= 1);
|
.anyMatch(entry -> entry.getValue().size() >= 1);
|
||||||
|
|
||||||
if (!existElement) {
|
if (!existElement) {
|
||||||
@@ -109,14 +113,14 @@ public class FuzzyNameMapper implements SchemaMapper {
|
|||||||
|
|
||||||
metricDimensionThresholdConfig = halfThreshold >= metricDimensionMinThresholdConfig ? halfThreshold
|
metricDimensionThresholdConfig = halfThreshold >= metricDimensionMinThresholdConfig ? halfThreshold
|
||||||
: metricDimensionMinThresholdConfig;
|
: metricDimensionMinThresholdConfig;
|
||||||
log.info("domainElementMatches:{} , not exist Element metricDimensionThresholdConfig reduce by half:{}",
|
log.info("ModelElementMatches:{} , not exist Element metricDimensionThresholdConfig reduce by half:{}",
|
||||||
domainElementMatches, metricDimensionThresholdConfig);
|
ModelElementMatches, metricDimensionThresholdConfig);
|
||||||
}
|
}
|
||||||
return metricDimensionThresholdConfig;
|
return metricDimensionThresholdConfig;
|
||||||
}
|
}
|
||||||
|
|
||||||
private Map<String, Set<SchemaElement>> getNameToItems(List<SchemaElement> domains) {
|
private Map<String, Set<SchemaElement>> getNameToItems(List<SchemaElement> Models) {
|
||||||
return domains.stream().collect(
|
return Models.stream().collect(
|
||||||
Collectors.toMap(SchemaElement::getName, a -> {
|
Collectors.toMap(SchemaElement::getName, a -> {
|
||||||
Set<SchemaElement> result = new HashSet<>();
|
Set<SchemaElement> result = new HashSet<>();
|
||||||
result.add(a);
|
result.add(a);
|
||||||
@@ -139,10 +143,10 @@ public class FuzzyNameMapper implements SchemaMapper {
|
|||||||
Set<SchemaElement> schemaElements = entry.getValue();
|
Set<SchemaElement> schemaElements = entry.getValue();
|
||||||
for (SchemaElement schemaElement : schemaElements) {
|
for (SchemaElement schemaElement : schemaElements) {
|
||||||
|
|
||||||
List<SchemaElementMatch> elements = schemaMap.getMatchedElements(schemaElement.getDomain());
|
List<SchemaElementMatch> elements = schemaMap.getMatchedElements(schemaElement.getModel());
|
||||||
if (CollectionUtils.isEmpty(elements)) {
|
if (CollectionUtils.isEmpty(elements)) {
|
||||||
elements = new ArrayList<>();
|
elements = new ArrayList<>();
|
||||||
schemaMap.setMatchedElements(schemaElement.getDomain(), elements);
|
schemaMap.setMatchedElements(schemaElement.getModel(), elements);
|
||||||
}
|
}
|
||||||
Set<Long> regElementSet = elements.stream()
|
Set<Long> regElementSet = elements.stream()
|
||||||
.filter(elementMatch -> schemaElementType.equals(elementMatch.getElement().getType()))
|
.filter(elementMatch -> schemaElementType.equals(elementMatch.getElement().getType()))
|
||||||
|
|||||||
@@ -2,13 +2,18 @@ package com.tencent.supersonic.chat.mapper;
|
|||||||
|
|
||||||
import com.hankcs.hanlp.seg.common.Term;
|
import com.hankcs.hanlp.seg.common.Term;
|
||||||
import com.tencent.supersonic.chat.api.component.SchemaMapper;
|
import com.tencent.supersonic.chat.api.component.SchemaMapper;
|
||||||
import com.tencent.supersonic.chat.api.pojo.*;
|
import com.tencent.supersonic.chat.api.pojo.ModelSchema;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.QueryContext;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SchemaMapInfo;
|
||||||
import com.tencent.supersonic.chat.service.SemanticService;
|
import com.tencent.supersonic.chat.service.SemanticService;
|
||||||
import com.tencent.supersonic.chat.utils.NatureHelper;
|
import com.tencent.supersonic.chat.utils.NatureHelper;
|
||||||
import com.tencent.supersonic.knowledge.dictionary.builder.BaseWordBuilder;
|
|
||||||
import com.tencent.supersonic.knowledge.dictionary.MapResult;
|
|
||||||
import com.tencent.supersonic.knowledge.dictionary.DictWordType;
|
|
||||||
import com.tencent.supersonic.common.util.ContextUtils;
|
import com.tencent.supersonic.common.util.ContextUtils;
|
||||||
|
import com.tencent.supersonic.knowledge.dictionary.DictWordType;
|
||||||
|
import com.tencent.supersonic.knowledge.dictionary.MapResult;
|
||||||
|
import com.tencent.supersonic.knowledge.dictionary.builder.BaseWordBuilder;
|
||||||
import com.tencent.supersonic.knowledge.dictionary.builder.WordBuilderFactory;
|
import com.tencent.supersonic.knowledge.dictionary.builder.WordBuilderFactory;
|
||||||
import com.tencent.supersonic.knowledge.utils.HanlpHelper;
|
import com.tencent.supersonic.knowledge.utils.HanlpHelper;
|
||||||
import java.util.ArrayList;
|
import java.util.ArrayList;
|
||||||
@@ -32,10 +37,10 @@ public class HanlpDictMapper implements SchemaMapper {
|
|||||||
for (Term term : terms) {
|
for (Term term : terms) {
|
||||||
log.info("word:{},nature:{},frequency:{}", term.word, term.nature.toString(), term.getFrequency());
|
log.info("word:{},nature:{},frequency:{}", term.word, term.nature.toString(), term.getFrequency());
|
||||||
}
|
}
|
||||||
Long domainId = queryContext.getRequest().getDomainId();
|
Long modelId = queryContext.getRequest().getModelId();
|
||||||
|
|
||||||
QueryMatchStrategy matchStrategy = ContextUtils.getBean(QueryMatchStrategy.class);
|
QueryMatchStrategy matchStrategy = ContextUtils.getBean(QueryMatchStrategy.class);
|
||||||
Map<MatchText, List<MapResult>> matchResult = matchStrategy.match(queryText, terms, domainId);
|
Map<MatchText, List<MapResult>> matchResult = matchStrategy.match(queryText, terms, modelId);
|
||||||
|
|
||||||
List<MapResult> matches = getMatches(matchResult);
|
List<MapResult> matches = getMatches(matchResult);
|
||||||
|
|
||||||
@@ -57,8 +62,8 @@ public class HanlpDictMapper implements SchemaMapper {
|
|||||||
|
|
||||||
for (MapResult mapResult : mapResults) {
|
for (MapResult mapResult : mapResults) {
|
||||||
for (String nature : mapResult.getNatures()) {
|
for (String nature : mapResult.getNatures()) {
|
||||||
Long domainId = NatureHelper.getDomainId(nature);
|
Long modelId = NatureHelper.getModelId(nature);
|
||||||
if (Objects.isNull(domainId)) {
|
if (Objects.isNull(modelId)) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
SchemaElementType elementType = NatureHelper.convertToElementType(nature);
|
SchemaElementType elementType = NatureHelper.convertToElementType(nature);
|
||||||
@@ -67,14 +72,14 @@ public class HanlpDictMapper implements SchemaMapper {
|
|||||||
}
|
}
|
||||||
|
|
||||||
SemanticService schemaService = ContextUtils.getBean(SemanticService.class);
|
SemanticService schemaService = ContextUtils.getBean(SemanticService.class);
|
||||||
DomainSchema domainSchema = schemaService.getDomainSchema(domainId);
|
ModelSchema modelSchema = schemaService.getModelSchema(modelId);
|
||||||
|
|
||||||
BaseWordBuilder baseWordBuilder = WordBuilderFactory.get(DictWordType.getNatureType(nature));
|
BaseWordBuilder baseWordBuilder = WordBuilderFactory.get(DictWordType.getNatureType(nature));
|
||||||
Long elementID = baseWordBuilder.getElementID(nature);
|
Long elementID = baseWordBuilder.getElementID(nature);
|
||||||
Long frequency = wordNatureToFrequency.get(mapResult.getName() + nature);
|
Long frequency = wordNatureToFrequency.get(mapResult.getName() + nature);
|
||||||
|
|
||||||
SchemaElement element = domainSchema.getElement(elementType, elementID);
|
SchemaElement element = modelSchema.getElement(elementType, elementID);
|
||||||
if(Objects.isNull(element)){
|
if (Objects.isNull(element)) {
|
||||||
log.info("element is null, elementType:{},elementID:{}", elementType, elementID);
|
log.info("element is null, elementType:{},elementID:{}", elementType, elementID);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
@@ -89,11 +94,11 @@ public class HanlpDictMapper implements SchemaMapper {
|
|||||||
.detectWord(mapResult.getDetectWord())
|
.detectWord(mapResult.getDetectWord())
|
||||||
.build();
|
.build();
|
||||||
|
|
||||||
Map<Long, List<SchemaElementMatch>> domainElementMatches = schemaMap.getDomainElementMatches();
|
Map<Long, List<SchemaElementMatch>> modelElementMatches = schemaMap.getModelElementMatches();
|
||||||
List<SchemaElementMatch> schemaElementMatches = domainElementMatches.putIfAbsent(domainId,
|
List<SchemaElementMatch> schemaElementMatches = modelElementMatches.putIfAbsent(modelId,
|
||||||
new ArrayList<>());
|
new ArrayList<>());
|
||||||
if (schemaElementMatches == null) {
|
if (schemaElementMatches == null) {
|
||||||
schemaElementMatches = domainElementMatches.get(domainId);
|
schemaElementMatches = modelElementMatches.get(modelId);
|
||||||
}
|
}
|
||||||
schemaElementMatches.add(schemaElementMatch);
|
schemaElementMatches.add(schemaElementMatch);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -83,4 +83,6 @@ public class MapperHelper {
|
|||||||
return 1 - (double) EditDistance.compute(detectSegmentLower, matchNameLower) / Math.max(matchName.length(),
|
return 1 - (double) EditDistance.compute(detectSegmentLower, matchNameLower) / Math.max(matchName.length(),
|
||||||
detectSegment.length());
|
detectSegment.length());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
}
|
}
|
||||||
@@ -2,7 +2,6 @@ package com.tencent.supersonic.chat.mapper;
|
|||||||
|
|
||||||
import com.hankcs.hanlp.seg.common.Term;
|
import com.hankcs.hanlp.seg.common.Term;
|
||||||
import com.tencent.supersonic.knowledge.dictionary.MapResult;
|
import com.tencent.supersonic.knowledge.dictionary.MapResult;
|
||||||
|
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
import java.util.Map;
|
import java.util.Map;
|
||||||
|
|
||||||
@@ -11,6 +10,6 @@ import java.util.Map;
|
|||||||
*/
|
*/
|
||||||
public interface MatchStrategy {
|
public interface MatchStrategy {
|
||||||
|
|
||||||
Map<MatchText, List<MapResult>> match(String text, List<Term> terms, Long detectDomainId);
|
Map<MatchText, List<MapResult>> match(String text, List<Term> terms, Long detectModelId);
|
||||||
|
|
||||||
}
|
}
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
package com.tencent.supersonic.chat.mapper;
|
||||||
|
|
||||||
|
import java.io.Serializable;
|
||||||
|
import lombok.Builder;
|
||||||
|
import lombok.Data;
|
||||||
|
import lombok.ToString;
|
||||||
|
|
||||||
|
@Data
|
||||||
|
@ToString
|
||||||
|
@Builder
|
||||||
|
public class ModelInfoStat implements Serializable {
|
||||||
|
|
||||||
|
private long modelCount;
|
||||||
|
|
||||||
|
private long metricModelCount;
|
||||||
|
|
||||||
|
private long dimensionModelCount;
|
||||||
|
|
||||||
|
private long dimensionValueModelCount;
|
||||||
|
|
||||||
|
}
|
||||||
@@ -7,13 +7,13 @@ import lombok.ToString;
|
|||||||
|
|
||||||
@Data
|
@Data
|
||||||
@ToString
|
@ToString
|
||||||
public class DomainWithSemanticType implements Serializable {
|
public class ModelWithSemanticType implements Serializable {
|
||||||
|
|
||||||
private Long domain;
|
private Long model;
|
||||||
private SchemaElementType semanticType;
|
private SchemaElementType semanticType;
|
||||||
|
|
||||||
public DomainWithSemanticType(Long domain, SchemaElementType semanticType) {
|
public ModelWithSemanticType(Long model, SchemaElementType semanticType) {
|
||||||
this.domain = domain;
|
this.model = model;
|
||||||
this.semanticType = semanticType;
|
this.semanticType = semanticType;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -2,15 +2,20 @@ package com.tencent.supersonic.chat.mapper;
|
|||||||
|
|
||||||
import com.google.common.collect.Lists;
|
import com.google.common.collect.Lists;
|
||||||
import com.tencent.supersonic.chat.api.component.SchemaMapper;
|
import com.tencent.supersonic.chat.api.component.SchemaMapper;
|
||||||
import com.tencent.supersonic.chat.api.pojo.*;
|
import com.tencent.supersonic.chat.api.pojo.QueryContext;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SchemaMapInfo;
|
||||||
import com.tencent.supersonic.chat.api.pojo.request.QueryFilter;
|
import com.tencent.supersonic.chat.api.pojo.request.QueryFilter;
|
||||||
import com.tencent.supersonic.chat.api.pojo.request.QueryFilters;
|
import com.tencent.supersonic.chat.api.pojo.request.QueryFilters;
|
||||||
import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
|
import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
|
||||||
import lombok.extern.slf4j.Slf4j;
|
import com.tencent.supersonic.common.pojo.Constants;
|
||||||
import org.springframework.util.CollectionUtils;
|
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
import java.util.Map;
|
import java.util.Map;
|
||||||
import java.util.stream.Collectors;
|
import java.util.stream.Collectors;
|
||||||
|
import lombok.extern.slf4j.Slf4j;
|
||||||
|
import org.springframework.util.CollectionUtils;
|
||||||
|
|
||||||
@Slf4j
|
@Slf4j
|
||||||
public class QueryFilterMapper implements SchemaMapper {
|
public class QueryFilterMapper implements SchemaMapper {
|
||||||
@@ -21,35 +26,35 @@ public class QueryFilterMapper implements SchemaMapper {
|
|||||||
@Override
|
@Override
|
||||||
public void map(QueryContext queryContext) {
|
public void map(QueryContext queryContext) {
|
||||||
QueryReq queryReq = queryContext.getRequest();
|
QueryReq queryReq = queryContext.getRequest();
|
||||||
Long domainId = queryReq.getDomainId();
|
Long modelId = queryReq.getModelId();
|
||||||
if (domainId == null || domainId <= 0) {
|
if (modelId == null || modelId <= 0) {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
SchemaMapInfo schemaMapInfo = queryContext.getMapInfo();
|
SchemaMapInfo schemaMapInfo = queryContext.getMapInfo();
|
||||||
clearOtherSchemaElementMatch(domainId, schemaMapInfo);
|
clearOtherSchemaElementMatch(modelId, schemaMapInfo);
|
||||||
List<SchemaElementMatch> schemaElementMatches = schemaMapInfo.getMatchedElements(domainId);
|
List<SchemaElementMatch> schemaElementMatches = schemaMapInfo.getMatchedElements(modelId);
|
||||||
if (schemaElementMatches == null) {
|
if (schemaElementMatches == null) {
|
||||||
schemaElementMatches = Lists.newArrayList();
|
schemaElementMatches = Lists.newArrayList();
|
||||||
schemaMapInfo.setMatchedElements(domainId, schemaElementMatches);
|
schemaMapInfo.setMatchedElements(modelId, schemaElementMatches);
|
||||||
}
|
}
|
||||||
addValueSchemaElementMatch(schemaElementMatches, queryReq.getQueryFilters());
|
addValueSchemaElementMatch(schemaElementMatches, queryReq.getQueryFilters());
|
||||||
}
|
}
|
||||||
|
|
||||||
private void clearOtherSchemaElementMatch(Long domainId, SchemaMapInfo schemaMapInfo) {
|
private void clearOtherSchemaElementMatch(Long modelId, SchemaMapInfo schemaMapInfo) {
|
||||||
for (Map.Entry<Long, List<SchemaElementMatch>> entry : schemaMapInfo.getDomainElementMatches().entrySet()) {
|
for (Map.Entry<Long, List<SchemaElementMatch>> entry : schemaMapInfo.getModelElementMatches().entrySet()) {
|
||||||
if (!entry.getKey().equals(domainId)) {
|
if (!entry.getKey().equals(modelId)) {
|
||||||
entry.getValue().clear();
|
entry.getValue().clear();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
private List<SchemaElementMatch> addValueSchemaElementMatch(List<SchemaElementMatch> candidateElementMatches,
|
private List<SchemaElementMatch> addValueSchemaElementMatch(List<SchemaElementMatch> candidateElementMatches,
|
||||||
QueryFilters queryFilter) {
|
QueryFilters queryFilter) {
|
||||||
if (queryFilter == null || CollectionUtils.isEmpty(queryFilter.getFilters())) {
|
if (queryFilter == null || CollectionUtils.isEmpty(queryFilter.getFilters())) {
|
||||||
return candidateElementMatches;
|
return candidateElementMatches;
|
||||||
}
|
}
|
||||||
for (QueryFilter filter : queryFilter.getFilters()) {
|
for (QueryFilter filter : queryFilter.getFilters()) {
|
||||||
if (checkExistSameValueSchemaElementMatch(filter, candidateElementMatches)) {
|
if (checkExistSameValueSchemaElementMatch(filter, candidateElementMatches)) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
SchemaElement element = SchemaElement.builder()
|
SchemaElement element = SchemaElement.builder()
|
||||||
@@ -63,7 +68,7 @@ public class QueryFilterMapper implements SchemaMapper {
|
|||||||
.frequency(FREQUENCY)
|
.frequency(FREQUENCY)
|
||||||
.word(String.valueOf(filter.getValue()))
|
.word(String.valueOf(filter.getValue()))
|
||||||
.similarity(SIMILARITY)
|
.similarity(SIMILARITY)
|
||||||
.detectWord(filter.getName())
|
.detectWord(Constants.EMPTY)
|
||||||
.build();
|
.build();
|
||||||
candidateElementMatches.add(schemaElementMatch);
|
candidateElementMatches.add(schemaElementMatch);
|
||||||
}
|
}
|
||||||
@@ -71,13 +76,13 @@ public class QueryFilterMapper implements SchemaMapper {
|
|||||||
}
|
}
|
||||||
|
|
||||||
private boolean checkExistSameValueSchemaElementMatch(QueryFilter queryFilter,
|
private boolean checkExistSameValueSchemaElementMatch(QueryFilter queryFilter,
|
||||||
List<SchemaElementMatch> schemaElementMatches) {
|
List<SchemaElementMatch> schemaElementMatches) {
|
||||||
List<SchemaElementMatch> valueSchemaElements = schemaElementMatches.stream().filter(schemaElementMatch ->
|
List<SchemaElementMatch> valueSchemaElements = schemaElementMatches.stream().filter(schemaElementMatch ->
|
||||||
SchemaElementType.VALUE.equals(schemaElementMatch.getElement().getType()))
|
SchemaElementType.VALUE.equals(schemaElementMatch.getElement().getType()))
|
||||||
.collect(Collectors.toList());
|
.collect(Collectors.toList());
|
||||||
for (SchemaElementMatch schemaElementMatch : valueSchemaElements) {
|
for (SchemaElementMatch schemaElementMatch : valueSchemaElements) {
|
||||||
if (schemaElementMatch.getElement().getId().equals(queryFilter.getElementID())
|
if (schemaElementMatch.getElement().getId().equals(queryFilter.getElementID())
|
||||||
&& schemaElementMatch.getWord().equals(String.valueOf(queryFilter.getValue()))) {
|
&& schemaElementMatch.getWord().equals(String.valueOf(queryFilter.getValue()))) {
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,10 +1,10 @@
|
|||||||
package com.tencent.supersonic.chat.mapper;
|
package com.tencent.supersonic.chat.mapper;
|
||||||
|
|
||||||
import com.hankcs.hanlp.seg.common.Term;
|
import com.hankcs.hanlp.seg.common.Term;
|
||||||
import com.tencent.supersonic.knowledge.dictionary.MapResult;
|
import com.tencent.supersonic.common.pojo.Constants;
|
||||||
import com.tencent.supersonic.knowledge.dictionary.DictWordType;
|
import com.tencent.supersonic.knowledge.dictionary.DictWordType;
|
||||||
|
import com.tencent.supersonic.knowledge.dictionary.MapResult;
|
||||||
import com.tencent.supersonic.knowledge.service.SearchService;
|
import com.tencent.supersonic.knowledge.service.SearchService;
|
||||||
|
|
||||||
import java.util.ArrayList;
|
import java.util.ArrayList;
|
||||||
import java.util.Comparator;
|
import java.util.Comparator;
|
||||||
import java.util.HashMap;
|
import java.util.HashMap;
|
||||||
@@ -32,7 +32,7 @@ public class QueryMatchStrategy implements MatchStrategy {
|
|||||||
private MapperHelper mapperHelper;
|
private MapperHelper mapperHelper;
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public Map<MatchText, List<MapResult>> match(String text, List<Term> terms, Long detectDomainId) {
|
public Map<MatchText, List<MapResult>> match(String text, List<Term> terms, Long detectmodelId) {
|
||||||
if (Objects.isNull(terms) || StringUtils.isEmpty(text)) {
|
if (Objects.isNull(terms) || StringUtils.isEmpty(text)) {
|
||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
@@ -43,10 +43,10 @@ public class QueryMatchStrategy implements MatchStrategy {
|
|||||||
List<Integer> offsetList = terms.stream().sorted(Comparator.comparing(Term::getOffset))
|
List<Integer> offsetList = terms.stream().sorted(Comparator.comparing(Term::getOffset))
|
||||||
.map(term -> term.getOffset()).collect(Collectors.toList());
|
.map(term -> term.getOffset()).collect(Collectors.toList());
|
||||||
|
|
||||||
log.debug("retryCount:{},terms:{},regOffsetToLength:{},offsetList:{},detectDomainId:{}", terms,
|
log.debug("retryCount:{},terms:{},regOffsetToLength:{},offsetList:{},detectmodelId:{}", terms,
|
||||||
regOffsetToLength, offsetList, detectDomainId);
|
regOffsetToLength, offsetList, detectmodelId);
|
||||||
|
|
||||||
List<MapResult> detects = detect(text, regOffsetToLength, offsetList, detectDomainId);
|
List<MapResult> detects = detect(text, regOffsetToLength, offsetList, detectmodelId);
|
||||||
Map<MatchText, List<MapResult>> result = new HashMap<>();
|
Map<MatchText, List<MapResult>> result = new HashMap<>();
|
||||||
|
|
||||||
MatchText matchText = MatchText.builder()
|
MatchText matchText = MatchText.builder()
|
||||||
@@ -58,7 +58,7 @@ public class QueryMatchStrategy implements MatchStrategy {
|
|||||||
}
|
}
|
||||||
|
|
||||||
private List<MapResult> detect(String text, Map<Integer, Integer> regOffsetToLength, List<Integer> offsetList,
|
private List<MapResult> detect(String text, Map<Integer, Integer> regOffsetToLength, List<Integer> offsetList,
|
||||||
Long detectDomainId) {
|
Long detectmodelId) {
|
||||||
List<MapResult> results = Lists.newArrayList();
|
List<MapResult> results = Lists.newArrayList();
|
||||||
|
|
||||||
for (Integer index = 0; index <= text.length() - 1; ) {
|
for (Integer index = 0; index <= text.length() - 1; ) {
|
||||||
@@ -69,18 +69,44 @@ public class QueryMatchStrategy implements MatchStrategy {
|
|||||||
int offset = mapperHelper.getStepOffset(offsetList, index);
|
int offset = mapperHelper.getStepOffset(offsetList, index);
|
||||||
i = mapperHelper.getStepIndex(regOffsetToLength, i);
|
i = mapperHelper.getStepIndex(regOffsetToLength, i);
|
||||||
if (i <= text.length()) {
|
if (i <= text.length()) {
|
||||||
List<MapResult> mapResults = detectByStep(text, detectDomainId, index, i, offset);
|
List<MapResult> mapResults = detectByStep(text, detectmodelId, index, i, offset);
|
||||||
mapResultRowSet.addAll(mapResults);
|
selectMapResultInOneRound(mapResultRowSet, mapResults);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
index = mapperHelper.getStepIndex(regOffsetToLength, index);
|
index = mapperHelper.getStepIndex(regOffsetToLength, index);
|
||||||
results.addAll(mapResultRowSet);
|
results.addAll(mapResultRowSet);
|
||||||
}
|
}
|
||||||
return results;
|
return results;
|
||||||
}
|
}
|
||||||
|
|
||||||
private List<MapResult> detectByStep(String text, Long detectDomainId, Integer index, Integer i, int offset) {
|
private void selectMapResultInOneRound(Set<MapResult> mapResultRowSet, List<MapResult> mapResults) {
|
||||||
|
for (MapResult mapResult : mapResults) {
|
||||||
|
if (mapResultRowSet.contains(mapResult)) {
|
||||||
|
boolean isDeleted = mapResultRowSet.removeIf(
|
||||||
|
entry -> {
|
||||||
|
boolean deleted = getMapKey(mapResult).equals(getMapKey(entry))
|
||||||
|
&& entry.getDetectWord().length() < mapResult.getDetectWord().length();
|
||||||
|
if (deleted) {
|
||||||
|
log.info("deleted entry:{}", entry);
|
||||||
|
}
|
||||||
|
return deleted;
|
||||||
|
}
|
||||||
|
);
|
||||||
|
if (isDeleted) {
|
||||||
|
log.info("deleted, add mapResult:{}", mapResult);
|
||||||
|
mapResultRowSet.add(mapResult);
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
mapResultRowSet.add(mapResult);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private String getMapKey(MapResult a) {
|
||||||
|
return a.getName() + Constants.UNDERLINE + String.join(Constants.UNDERLINE, a.getNatures());
|
||||||
|
}
|
||||||
|
|
||||||
|
private List<MapResult> detectByStep(String text, Long detectmodelId, Integer index, Integer i, int offset) {
|
||||||
String detectSegment = text.substring(index, i);
|
String detectSegment = text.substring(index, i);
|
||||||
Integer oneDetectionSize = mapperHelper.getOneDetectionSize();
|
Integer oneDetectionSize = mapperHelper.getOneDetectionSize();
|
||||||
// step1. pre search
|
// step1. pre search
|
||||||
@@ -100,17 +126,17 @@ public class QueryMatchStrategy implements MatchStrategy {
|
|||||||
mapResults = mapResults.stream().sorted((a, b) -> -(b.getName().length() - a.getName().length()))
|
mapResults = mapResults.stream().sorted((a, b) -> -(b.getName().length() - a.getName().length()))
|
||||||
.collect(Collectors.toCollection(LinkedHashSet::new));
|
.collect(Collectors.toCollection(LinkedHashSet::new));
|
||||||
// step4. filter by classId
|
// step4. filter by classId
|
||||||
if (Objects.nonNull(detectDomainId) && detectDomainId > 0) {
|
if (Objects.nonNull(detectmodelId) && detectmodelId > 0) {
|
||||||
log.debug("detectDomainId:{}, before parseResults:{}", mapResults);
|
log.debug("detectmodelId:{}, before parseResults:{}", mapResults);
|
||||||
mapResults = mapResults.stream().map(entry -> {
|
mapResults = mapResults.stream().map(entry -> {
|
||||||
List<String> natures = entry.getNatures().stream().filter(
|
List<String> natures = entry.getNatures().stream().filter(
|
||||||
nature -> nature.startsWith(DictWordType.NATURE_SPILT + detectDomainId) || (nature.startsWith(
|
nature -> nature.startsWith(DictWordType.NATURE_SPILT + detectmodelId) || (nature.startsWith(
|
||||||
DictWordType.NATURE_SPILT))
|
DictWordType.NATURE_SPILT))
|
||||||
).collect(Collectors.toList());
|
).collect(Collectors.toList());
|
||||||
entry.setNatures(natures);
|
entry.setNatures(natures);
|
||||||
return entry;
|
return entry;
|
||||||
}).collect(Collectors.toCollection(LinkedHashSet::new));
|
}).collect(Collectors.toCollection(LinkedHashSet::new));
|
||||||
log.info("after domainId parseResults:{}", mapResults);
|
log.info("after modelId parseResults:{}", mapResults);
|
||||||
}
|
}
|
||||||
// step5. filter by similarity
|
// step5. filter by similarity
|
||||||
mapResults = mapResults.stream()
|
mapResults = mapResults.stream()
|
||||||
|
|||||||
@@ -2,10 +2,9 @@ package com.tencent.supersonic.chat.mapper;
|
|||||||
|
|
||||||
import com.google.common.collect.Lists;
|
import com.google.common.collect.Lists;
|
||||||
import com.hankcs.hanlp.seg.common.Term;
|
import com.hankcs.hanlp.seg.common.Term;
|
||||||
import com.tencent.supersonic.knowledge.dictionary.MapResult;
|
|
||||||
import com.tencent.supersonic.knowledge.dictionary.DictWordType;
|
import com.tencent.supersonic.knowledge.dictionary.DictWordType;
|
||||||
|
import com.tencent.supersonic.knowledge.dictionary.MapResult;
|
||||||
import com.tencent.supersonic.knowledge.service.SearchService;
|
import com.tencent.supersonic.knowledge.service.SearchService;
|
||||||
|
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
import java.util.Map;
|
import java.util.Map;
|
||||||
import java.util.Objects;
|
import java.util.Objects;
|
||||||
@@ -25,7 +24,7 @@ public class SearchMatchStrategy implements MatchStrategy {
|
|||||||
|
|
||||||
@Override
|
@Override
|
||||||
public Map<MatchText, List<MapResult>> match(String text, List<Term> originals,
|
public Map<MatchText, List<MapResult>> match(String text, List<Term> originals,
|
||||||
Long detectDomainId) {
|
Long detectModelId) {
|
||||||
|
|
||||||
Map<Integer, Integer> regOffsetToLength = originals.stream()
|
Map<Integer, Integer> regOffsetToLength = originals.stream()
|
||||||
.filter(entry -> !entry.nature.toString().startsWith(DictWordType.NATURE_SPILT))
|
.filter(entry -> !entry.nature.toString().startsWith(DictWordType.NATURE_SPILT))
|
||||||
@@ -60,10 +59,10 @@ public class SearchMatchStrategy implements MatchStrategy {
|
|||||||
List<String> natures = entry.getNatures().stream()
|
List<String> natures = entry.getNatures().stream()
|
||||||
.filter(nature -> !nature.endsWith(DictWordType.ENTITY.getType()))
|
.filter(nature -> !nature.endsWith(DictWordType.ENTITY.getType()))
|
||||||
.filter(nature -> {
|
.filter(nature -> {
|
||||||
if (Objects.isNull(detectDomainId) || detectDomainId <= 0) {
|
if (Objects.isNull(detectModelId) || detectModelId <= 0) {
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
if (nature.startsWith(DictWordType.NATURE_SPILT + detectDomainId)
|
if (nature.startsWith(DictWordType.NATURE_SPILT + detectModelId)
|
||||||
&& nature.startsWith(DictWordType.NATURE_SPILT)) {
|
&& nature.startsWith(DictWordType.NATURE_SPILT)) {
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -2,22 +2,9 @@ package com.tencent.supersonic.chat.parser;
|
|||||||
|
|
||||||
|
|
||||||
import com.tencent.supersonic.chat.api.component.SemanticQuery;
|
import com.tencent.supersonic.chat.api.component.SemanticQuery;
|
||||||
import com.tencent.supersonic.chat.api.pojo.*;
|
import com.tencent.supersonic.chat.api.pojo.QueryContext;
|
||||||
import com.tencent.supersonic.chat.plugin.PluginParseResult;
|
import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
|
||||||
import com.tencent.supersonic.chat.query.plugin.PluginSemanticQuery;
|
|
||||||
import com.tencent.supersonic.chat.query.rule.RuleSemanticQuery;
|
|
||||||
import com.tencent.supersonic.common.pojo.Constants;
|
|
||||||
import com.tencent.supersonic.common.pojo.DateConf;
|
|
||||||
import com.tencent.supersonic.common.util.JsonUtil;
|
|
||||||
import lombok.extern.slf4j.Slf4j;
|
import lombok.extern.slf4j.Slf4j;
|
||||||
import org.apache.commons.compress.utils.Lists;
|
|
||||||
import org.apache.commons.lang3.StringUtils;
|
|
||||||
import org.springframework.util.CollectionUtils;
|
|
||||||
import java.util.HashSet;
|
|
||||||
import java.util.List;
|
|
||||||
import java.util.Map;
|
|
||||||
import java.util.Optional;
|
|
||||||
import java.util.stream.Collectors;
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* This checker can be used by semantic parsers to check if query intent
|
* This checker can be used by semantic parsers to check if query intent
|
||||||
@@ -30,86 +17,20 @@ public class SatisfactionChecker {
|
|||||||
private static final double LONG_TEXT_THRESHOLD = 0.8;
|
private static final double LONG_TEXT_THRESHOLD = 0.8;
|
||||||
private static final double SHORT_TEXT_THRESHOLD = 0.5;
|
private static final double SHORT_TEXT_THRESHOLD = 0.5;
|
||||||
private static final int QUERY_TEXT_LENGTH_THRESHOLD = 10;
|
private static final int QUERY_TEXT_LENGTH_THRESHOLD = 10;
|
||||||
public static final double EMBEDDING_THRESHOLD = 0.2;
|
|
||||||
|
|
||||||
// check all the parse info in candidate
|
// check all the parse info in candidate
|
||||||
public static boolean check(QueryContext queryCtx) {
|
public static boolean check(QueryContext queryContext) {
|
||||||
for (SemanticQuery query : queryCtx.getCandidateQueries()) {
|
for (SemanticQuery query : queryContext.getCandidateQueries()) {
|
||||||
if (query instanceof RuleSemanticQuery) {
|
if (checkThreshold(queryContext.getRequest().getQueryText(), query.getParseInfo())) {
|
||||||
if (checkRuleThreshHold(queryCtx.getRequest().getQueryText(), query.getParseInfo())) {
|
return true;
|
||||||
return true;
|
|
||||||
}
|
|
||||||
} else if (query instanceof PluginSemanticQuery) {
|
|
||||||
if (checkEmbeddingThreshold(query.getParseInfo())) {
|
|
||||||
log.info("query mode :{} satisfy check", query.getQueryMode());
|
|
||||||
return true;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
private static boolean checkEmbeddingThreshold(SemanticParseInfo semanticParseInfo) {
|
private static boolean checkThreshold(String queryText, SemanticParseInfo semanticParseInfo) {
|
||||||
Object object = semanticParseInfo.getProperties().get(Constants.CONTEXT);
|
|
||||||
PluginParseResult pluginParseResult = JsonUtil.toObject(JsonUtil.toString(object), PluginParseResult.class);
|
|
||||||
return EMBEDDING_THRESHOLD > pluginParseResult.getDistance();
|
|
||||||
}
|
|
||||||
|
|
||||||
//check single parse info
|
|
||||||
private static boolean checkRuleThreshHold(String text, SemanticParseInfo semanticParseInfo) {
|
|
||||||
List<SchemaElementMatch> schemaElementMatches = semanticParseInfo.getElementMatches();
|
|
||||||
if (CollectionUtils.isEmpty(schemaElementMatches)) {
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
List<String> detectWords = Lists.newArrayList();
|
|
||||||
Map<Long, String> detectWordMap = schemaElementMatches.stream()
|
|
||||||
.collect(Collectors.toMap(m -> m.getElement().getId(), SchemaElementMatch::getDetectWord,
|
|
||||||
(id1, id2) -> id1));
|
|
||||||
// get detect word in text by element id in semantic layer
|
|
||||||
Long domainId = semanticParseInfo.getDomainId();
|
|
||||||
if (domainId != null && domainId > 0) {
|
|
||||||
for (SchemaElementMatch schemaElementMatch : schemaElementMatches) {
|
|
||||||
if (SchemaElementType.DOMAIN.equals(schemaElementMatch.getElement().getType())) {
|
|
||||||
detectWords.add(schemaElementMatch.getDetectWord());
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
for (SchemaElementMatch schemaElementMatch : schemaElementMatches) {
|
|
||||||
if (SchemaElementType.VALUE.equals(schemaElementMatch.getElement().getType())) {
|
|
||||||
detectWords.add(schemaElementMatch.getDetectWord());
|
|
||||||
}
|
|
||||||
}
|
|
||||||
for (SchemaElementMatch schemaElementMatch : schemaElementMatches) {
|
|
||||||
if (SchemaElementType.ID.equals(schemaElementMatch.getElement().getType())) {
|
|
||||||
detectWords.add(schemaElementMatch.getDetectWord());
|
|
||||||
}
|
|
||||||
}
|
|
||||||
for (SchemaElement schemaItem : semanticParseInfo.getMetrics()) {
|
|
||||||
detectWords.add(
|
|
||||||
detectWordMap.getOrDefault(Optional.ofNullable(schemaItem.getId()).orElse(0L), ""));
|
|
||||||
// only first metric
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
for (SchemaElement schemaItem : semanticParseInfo.getDimensions()) {
|
|
||||||
detectWords.add(
|
|
||||||
detectWordMap.getOrDefault(Optional.ofNullable(schemaItem.getId()).orElse(0L), ""));
|
|
||||||
// only first dimension
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
String dateText = Optional.ofNullable(semanticParseInfo.getDateInfo()).orElse(new DateConf()).getText();
|
|
||||||
if (StringUtils.isNotBlank(dateText) && !dateText.equalsIgnoreCase(Constants.NULL)) {
|
|
||||||
detectWords.add(dateText);
|
|
||||||
}
|
|
||||||
detectWords.removeIf(word -> !text.contains(word) && !text.contains(StringUtils.reverse(word)));
|
|
||||||
//compare the length between detect words and query text
|
|
||||||
return checkThreshold(text, detectWords, semanticParseInfo);
|
|
||||||
}
|
|
||||||
|
|
||||||
private static boolean checkThreshold(String queryText, List<String> detectWords, SemanticParseInfo semanticParseInfo) {
|
|
||||||
String detectWordsDistinct = StringUtils.join(new HashSet<>(detectWords), "");
|
|
||||||
int detectWordsLength = detectWordsDistinct.length();
|
|
||||||
int queryTextLength = queryText.length();
|
int queryTextLength = queryText.length();
|
||||||
double degree = detectWordsLength * 1.0 / queryTextLength;
|
double degree = semanticParseInfo.getScore() / queryTextLength;
|
||||||
if (queryTextLength > QUERY_TEXT_LENGTH_THRESHOLD) {
|
if (queryTextLength > QUERY_TEXT_LENGTH_THRESHOLD) {
|
||||||
if (degree < LONG_TEXT_THRESHOLD) {
|
if (degree < LONG_TEXT_THRESHOLD) {
|
||||||
return false;
|
return false;
|
||||||
@@ -117,8 +38,8 @@ public class SatisfactionChecker {
|
|||||||
} else if (degree < SHORT_TEXT_THRESHOLD) {
|
} else if (degree < SHORT_TEXT_THRESHOLD) {
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
log.info("queryMode:{}, degree:{}, detectWords:{}, parse info:{}",
|
log.info("queryMode:{}, degree:{}, parse info:{}",
|
||||||
semanticParseInfo.getQueryMode(), degree, detectWordsDistinct, semanticParseInfo);
|
semanticParseInfo.getQueryMode(), degree, semanticParseInfo);
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -1,13 +1,18 @@
|
|||||||
package com.tencent.supersonic.chat.parser.embedding;
|
package com.tencent.supersonic.chat.parser.embedding;
|
||||||
|
|
||||||
import com.google.common.collect.Lists;
|
import com.google.common.collect.Lists;
|
||||||
import com.google.common.collect.Sets;
|
|
||||||
import com.tencent.supersonic.chat.api.component.SemanticParser;
|
import com.tencent.supersonic.chat.api.component.SemanticParser;
|
||||||
import com.tencent.supersonic.chat.api.pojo.*;
|
import com.tencent.supersonic.chat.api.pojo.ChatContext;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.ModelSchema;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.QueryContext;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SchemaMapInfo;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
|
||||||
import com.tencent.supersonic.chat.api.pojo.request.QueryFilter;
|
import com.tencent.supersonic.chat.api.pojo.request.QueryFilter;
|
||||||
import com.tencent.supersonic.chat.api.pojo.request.QueryFilters;
|
|
||||||
import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
|
import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
|
||||||
import com.tencent.supersonic.chat.parser.SatisfactionChecker;
|
import com.tencent.supersonic.chat.parser.ParseMode;
|
||||||
import com.tencent.supersonic.chat.plugin.Plugin;
|
import com.tencent.supersonic.chat.plugin.Plugin;
|
||||||
import com.tencent.supersonic.chat.plugin.PluginManager;
|
import com.tencent.supersonic.chat.plugin.PluginManager;
|
||||||
import com.tencent.supersonic.chat.plugin.PluginParseResult;
|
import com.tencent.supersonic.chat.plugin.PluginParseResult;
|
||||||
@@ -17,18 +22,20 @@ import com.tencent.supersonic.chat.service.PluginService;
|
|||||||
import com.tencent.supersonic.chat.service.SemanticService;
|
import com.tencent.supersonic.chat.service.SemanticService;
|
||||||
import com.tencent.supersonic.common.pojo.Constants;
|
import com.tencent.supersonic.common.pojo.Constants;
|
||||||
import com.tencent.supersonic.common.util.ContextUtils;
|
import com.tencent.supersonic.common.util.ContextUtils;
|
||||||
import java.util.*;
|
|
||||||
import java.util.stream.Collectors;
|
|
||||||
import com.tencent.supersonic.semantic.api.query.enums.FilterOperatorEnum;
|
import com.tencent.supersonic.semantic.api.query.enums.FilterOperatorEnum;
|
||||||
|
import java.util.Comparator;
|
||||||
|
import java.util.HashMap;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Map;
|
||||||
|
import java.util.stream.Collectors;
|
||||||
import lombok.extern.slf4j.Slf4j;
|
import lombok.extern.slf4j.Slf4j;
|
||||||
import org.apache.commons.lang3.StringUtils;
|
import org.apache.commons.lang3.StringUtils;
|
||||||
|
import org.apache.commons.lang3.tuple.Pair;
|
||||||
import org.springframework.util.CollectionUtils;
|
import org.springframework.util.CollectionUtils;
|
||||||
|
|
||||||
@Slf4j
|
@Slf4j
|
||||||
public class EmbeddingBasedParser implements SemanticParser {
|
public class EmbeddingBasedParser implements SemanticParser {
|
||||||
|
|
||||||
private final static double THRESHOLD = 0.2d;
|
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public void parse(QueryContext queryContext, ChatContext chatContext) {
|
public void parse(QueryContext queryContext, ChatContext chatContext) {
|
||||||
EmbeddingConfig embeddingConfig = ContextUtils.getBean(EmbeddingConfig.class);
|
EmbeddingConfig embeddingConfig = ContextUtils.getBean(EmbeddingConfig.class);
|
||||||
@@ -36,109 +43,15 @@ public class EmbeddingBasedParser implements SemanticParser {
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
log.info("EmbeddingBasedParser parser query ctx: {}, chat ctx: {}", queryContext, chatContext);
|
log.info("EmbeddingBasedParser parser query ctx: {}, chat ctx: {}", queryContext, chatContext);
|
||||||
Set<Long> domainIds = getDomainMatched(queryContext);
|
|
||||||
String text = queryContext.getRequest().getQueryText();
|
String text = queryContext.getRequest().getQueryText();
|
||||||
if (!CollectionUtils.isEmpty(domainIds)) {
|
List<RecallRetrieval> embeddingRetrievals = recallResult(text);
|
||||||
for (Long domainId : domainIds) {
|
choosePlugin(embeddingRetrievals, queryContext);
|
||||||
List<SchemaElementMatch> schemaElementMatches = getMatchedElements(queryContext, domainId);
|
|
||||||
String textReplaced = replaceText(text, schemaElementMatches);
|
|
||||||
List<RecallRetrieval> embeddingRetrievals = recallResult(textReplaced, hasCandidateQuery(queryContext));
|
|
||||||
Optional<Plugin> pluginOptional = choosePlugin(embeddingRetrievals, domainId);
|
|
||||||
log.info("domain id :{} embedding result, text:{} embeddingResp:{} ",domainId, textReplaced, embeddingRetrievals);
|
|
||||||
pluginOptional.ifPresent(plugin -> buildQuery(plugin, embeddingRetrievals, domainId, textReplaced, queryContext, schemaElementMatches));
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
List<RecallRetrieval> embeddingRetrievals = recallResult(text, hasCandidateQuery(queryContext));
|
|
||||||
Optional<Plugin> pluginOptional = choosePlugin(embeddingRetrievals, null);
|
|
||||||
pluginOptional.ifPresent(plugin -> buildQuery(plugin, embeddingRetrievals, null, text, queryContext, Lists.newArrayList()));
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
private void buildQuery(Plugin plugin, List<RecallRetrieval> embeddingRetrievals,
|
private void choosePlugin(List<RecallRetrieval> embeddingRetrievals,
|
||||||
Long domainId, String text,
|
QueryContext queryContext) {
|
||||||
QueryContext queryContext, List<SchemaElementMatch> schemaElementMatches) {
|
|
||||||
Map<String, RecallRetrieval> embeddingRetrievalMap = embeddingRetrievals.stream()
|
|
||||||
.collect(Collectors.toMap(RecallRetrieval::getId, e -> e, (value1, value2) -> value1));
|
|
||||||
log.info("EmbeddingBasedParser text: {} domain: {} choose plugin: [{} {}]",
|
|
||||||
text, domainId, plugin.getId(), plugin.getName());
|
|
||||||
PluginSemanticQuery pluginQuery = QueryManager.createPluginQuery(plugin.getType());
|
|
||||||
SemanticParseInfo semanticParseInfo = buildSemanticParseInfo(domainId, plugin, text,
|
|
||||||
queryContext.getRequest(), embeddingRetrievalMap, schemaElementMatches);
|
|
||||||
semanticParseInfo.setQueryMode(pluginQuery.getQueryMode());
|
|
||||||
pluginQuery.setParseInfo(semanticParseInfo);
|
|
||||||
queryContext.getCandidateQueries().add(pluginQuery);
|
|
||||||
}
|
|
||||||
|
|
||||||
private Set<Long> getDomainMatched(QueryContext queryContext) {
|
|
||||||
Long queryDomainId = queryContext.getRequest().getDomainId();
|
|
||||||
if (queryDomainId != null && queryDomainId > 0) {
|
|
||||||
return Sets.newHashSet(queryDomainId);
|
|
||||||
}
|
|
||||||
return queryContext.getMapInfo().getMatchedDomains();
|
|
||||||
}
|
|
||||||
|
|
||||||
private SemanticParseInfo buildSemanticParseInfo(Long domainId, Plugin plugin, String text, QueryReq queryReq,
|
|
||||||
Map<String, RecallRetrieval> embeddingRetrievalMap,
|
|
||||||
List<SchemaElementMatch> schemaElementMatches) {
|
|
||||||
SchemaElement schemaElement = new SchemaElement();
|
|
||||||
schemaElement.setDomain(domainId);
|
|
||||||
schemaElement.setId(domainId);
|
|
||||||
SemanticParseInfo semanticParseInfo = new SemanticParseInfo();
|
|
||||||
semanticParseInfo.setElementMatches(schemaElementMatches);
|
|
||||||
semanticParseInfo.setDomain(schemaElement);
|
|
||||||
double distance = Double.parseDouble(embeddingRetrievalMap.get(plugin.getId().toString()).getDistance());
|
|
||||||
double score = text.length() * (1 - distance);
|
|
||||||
Map<String, Object> properties = new HashMap<>();
|
|
||||||
PluginParseResult pluginParseResult = new PluginParseResult();
|
|
||||||
pluginParseResult.setPlugin(plugin);
|
|
||||||
pluginParseResult.setRequest(queryReq);
|
|
||||||
pluginParseResult.setDistance(distance);
|
|
||||||
properties.put(Constants.CONTEXT, pluginParseResult);
|
|
||||||
semanticParseInfo.setProperties(properties);
|
|
||||||
semanticParseInfo.setScore(score);
|
|
||||||
fillSemanticParseInfo(semanticParseInfo);
|
|
||||||
setEntity(domainId, semanticParseInfo);
|
|
||||||
return semanticParseInfo;
|
|
||||||
}
|
|
||||||
|
|
||||||
private List<SchemaElementMatch> getMatchedElements(QueryContext queryContext, Long domainId) {
|
|
||||||
SchemaMapInfo schemaMapInfo = queryContext.getMapInfo();
|
|
||||||
List<SchemaElementMatch> schemaElementMatches = schemaMapInfo.getMatchedElements(domainId);
|
|
||||||
if (schemaElementMatches == null) {
|
|
||||||
return Lists.newArrayList();
|
|
||||||
}
|
|
||||||
QueryReq queryReq = queryContext.getRequest();
|
|
||||||
QueryFilters queryFilters = queryReq.getQueryFilters();
|
|
||||||
if (queryFilters == null || CollectionUtils.isEmpty(queryFilters.getFilters())) {
|
|
||||||
return schemaElementMatches;
|
|
||||||
}
|
|
||||||
Map<Long, Object> element = queryFilters.getFilters().stream()
|
|
||||||
.collect(Collectors.toMap(QueryFilter::getElementID, QueryFilter::getValue, (v1, v2) -> v1));
|
|
||||||
return schemaElementMatches.stream().filter(schemaElementMatch ->
|
|
||||||
SchemaElementType.VALUE.equals(schemaElementMatch.getElement().getType())
|
|
||||||
|| SchemaElementType.ID.equals(schemaElementMatch.getElement().getType())
|
|
||||||
|| SchemaElementType.ENTITY.equals(schemaElementMatch.getElement().getType()))
|
|
||||||
.filter(schemaElementMatch ->
|
|
||||||
!element.containsKey(schemaElementMatch.getElement().getId()) || (
|
|
||||||
element.containsKey(schemaElementMatch.getElement().getId()) &&
|
|
||||||
element.get(schemaElementMatch.getElement().getId()).toString()
|
|
||||||
.equalsIgnoreCase(schemaElementMatch.getWord())
|
|
||||||
))
|
|
||||||
.collect(Collectors.toList());
|
|
||||||
}
|
|
||||||
|
|
||||||
private void setEntity(Long domainId, SemanticParseInfo semanticParseInfo) {
|
|
||||||
SemanticService semanticService = ContextUtils.getBean(SemanticService.class);
|
|
||||||
DomainSchema domainSchema = semanticService.getDomainSchema(domainId);
|
|
||||||
if (domainSchema != null && domainSchema.getEntity() != null) {
|
|
||||||
semanticParseInfo.setEntity(domainSchema.getEntity());
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
private Optional<Plugin> choosePlugin(List<RecallRetrieval> embeddingRetrievals,
|
|
||||||
Long domainId) {
|
|
||||||
if (CollectionUtils.isEmpty(embeddingRetrievals)) {
|
if (CollectionUtils.isEmpty(embeddingRetrievals)) {
|
||||||
return Optional.empty();
|
return;
|
||||||
}
|
}
|
||||||
PluginService pluginService = ContextUtils.getBean(PluginService.class);
|
PluginService pluginService = ContextUtils.getBean(PluginService.class);
|
||||||
List<Plugin> plugins = pluginService.getPluginList();
|
List<Plugin> plugins = pluginService.getPluginList();
|
||||||
@@ -148,27 +61,92 @@ public class EmbeddingBasedParser implements SemanticParser {
|
|||||||
if (plugin == null) {
|
if (plugin == null) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
if (domainId == null) {
|
Pair<Boolean, List<Long>> pair = PluginManager.resolve(plugin, queryContext);
|
||||||
return Optional.of(plugin);
|
log.info("embedding plugin resolve: {}", pair);
|
||||||
}
|
if (pair.getLeft()) {
|
||||||
if (!CollectionUtils.isEmpty(plugin.getDomainList()) && plugin.getDomainList().contains(domainId)) {
|
List<Long> modelList = pair.getRight();
|
||||||
return Optional.of(plugin);
|
if (CollectionUtils.isEmpty(modelList)) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
modelList = distinctModelList(plugin, queryContext.getMapInfo(), modelList);
|
||||||
|
for (Long modelId : modelList) {
|
||||||
|
buildQuery(plugin, Double.parseDouble(embeddingRetrieval.getDistance()), modelId, queryContext,
|
||||||
|
queryContext.getMapInfo().getMatchedElements(modelId));
|
||||||
|
}
|
||||||
|
return;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return Optional.empty();
|
|
||||||
}
|
}
|
||||||
|
|
||||||
public List<RecallRetrieval> recallResult(String embeddingText, boolean hasCandidateQuery) {
|
public List<Long> distinctModelList(Plugin plugin, SchemaMapInfo schemaMapInfo, List<Long> modelList) {
|
||||||
|
if (!plugin.isContainsAllModel()) {
|
||||||
|
return modelList;
|
||||||
|
}
|
||||||
|
boolean noElementMatch = true;
|
||||||
|
for (Long model : modelList) {
|
||||||
|
List<SchemaElementMatch> schemaElementMatches = schemaMapInfo.getMatchedElements(model);
|
||||||
|
if (!CollectionUtils.isEmpty(schemaElementMatches)) {
|
||||||
|
noElementMatch = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (noElementMatch) {
|
||||||
|
return modelList.subList(0, 1);
|
||||||
|
}
|
||||||
|
return modelList;
|
||||||
|
}
|
||||||
|
|
||||||
|
private void buildQuery(Plugin plugin, double distance, Long modelId,
|
||||||
|
QueryContext queryContext, List<SchemaElementMatch> schemaElementMatches) {
|
||||||
|
log.info("EmbeddingBasedParser Model: {} choose plugin: [{} {}]", modelId, plugin.getId(), plugin.getName());
|
||||||
|
PluginSemanticQuery pluginQuery = QueryManager.createPluginQuery(plugin.getType());
|
||||||
|
plugin.setParseMode(ParseMode.EMBEDDING_RECALL);
|
||||||
|
SemanticParseInfo semanticParseInfo = buildSemanticParseInfo(modelId, plugin, queryContext.getRequest(),
|
||||||
|
schemaElementMatches, distance);
|
||||||
|
double score = queryContext.getRequest().getQueryText().length() * (1 - distance);
|
||||||
|
semanticParseInfo.setQueryMode(pluginQuery.getQueryMode());
|
||||||
|
semanticParseInfo.setScore(score);
|
||||||
|
pluginQuery.setParseInfo(semanticParseInfo);
|
||||||
|
queryContext.getCandidateQueries().add(pluginQuery);
|
||||||
|
}
|
||||||
|
|
||||||
|
private SemanticParseInfo buildSemanticParseInfo(Long modelId, Plugin plugin, QueryReq queryReq,
|
||||||
|
List<SchemaElementMatch> schemaElementMatches, double distance) {
|
||||||
|
if (modelId == null && !CollectionUtils.isEmpty(plugin.getModelList())) {
|
||||||
|
modelId = plugin.getModelList().get(0);
|
||||||
|
}
|
||||||
|
SchemaElement Model = new SchemaElement();
|
||||||
|
Model.setModel(modelId);
|
||||||
|
Model.setId(modelId);
|
||||||
|
SemanticParseInfo semanticParseInfo = new SemanticParseInfo();
|
||||||
|
semanticParseInfo.setElementMatches(schemaElementMatches);
|
||||||
|
semanticParseInfo.setModel(Model);
|
||||||
|
Map<String, Object> properties = new HashMap<>();
|
||||||
|
PluginParseResult pluginParseResult = new PluginParseResult();
|
||||||
|
pluginParseResult.setPlugin(plugin);
|
||||||
|
pluginParseResult.setRequest(queryReq);
|
||||||
|
pluginParseResult.setDistance(distance);
|
||||||
|
properties.put(Constants.CONTEXT, pluginParseResult);
|
||||||
|
semanticParseInfo.setProperties(properties);
|
||||||
|
semanticParseInfo.setScore(distance);
|
||||||
|
fillSemanticParseInfo(semanticParseInfo);
|
||||||
|
setEntity(modelId, semanticParseInfo);
|
||||||
|
return semanticParseInfo;
|
||||||
|
}
|
||||||
|
|
||||||
|
private void setEntity(Long modelId, SemanticParseInfo semanticParseInfo) {
|
||||||
|
SemanticService semanticService = ContextUtils.getBean(SemanticService.class);
|
||||||
|
ModelSchema ModelSchema = semanticService.getModelSchema(modelId);
|
||||||
|
if (ModelSchema != null && ModelSchema.getEntity() != null) {
|
||||||
|
semanticParseInfo.setEntity(ModelSchema.getEntity());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public List<RecallRetrieval> recallResult(String embeddingText) {
|
||||||
try {
|
try {
|
||||||
PluginManager pluginManager = ContextUtils.getBean(PluginManager.class);
|
PluginManager pluginManager = ContextUtils.getBean(PluginManager.class);
|
||||||
EmbeddingResp embeddingResp = pluginManager.recognize(embeddingText);
|
EmbeddingResp embeddingResp = pluginManager.recognize(embeddingText);
|
||||||
List<RecallRetrieval> embeddingRetrievals = embeddingResp.getRetrieval();
|
List<RecallRetrieval> embeddingRetrievals = embeddingResp.getRetrieval();
|
||||||
if(!CollectionUtils.isEmpty(embeddingRetrievals)){
|
if (!CollectionUtils.isEmpty(embeddingRetrievals)) {
|
||||||
if (hasCandidateQuery) {
|
|
||||||
embeddingRetrievals = embeddingRetrievals.stream()
|
|
||||||
.filter(llmRetrieval -> Double.parseDouble(llmRetrieval.getDistance())<THRESHOLD)
|
|
||||||
.collect(Collectors.toList());
|
|
||||||
}
|
|
||||||
embeddingRetrievals = embeddingRetrievals.stream().sorted(Comparator.comparingDouble(o ->
|
embeddingRetrievals = embeddingRetrievals.stream().sorted(Comparator.comparingDouble(o ->
|
||||||
Math.abs(Double.parseDouble(o.getDistance())))).collect(Collectors.toList());
|
Math.abs(Double.parseDouble(o.getDistance())))).collect(Collectors.toList());
|
||||||
embeddingResp.setRetrieval(embeddingRetrievals);
|
embeddingResp.setRetrieval(embeddingRetrievals);
|
||||||
@@ -180,45 +158,22 @@ public class EmbeddingBasedParser implements SemanticParser {
|
|||||||
return Lists.newArrayList();
|
return Lists.newArrayList();
|
||||||
}
|
}
|
||||||
|
|
||||||
private boolean hasCandidateQuery(QueryContext queryContext) {
|
|
||||||
return !CollectionUtils.isEmpty(queryContext.getCandidateQueries());
|
|
||||||
}
|
|
||||||
|
|
||||||
private void fillSemanticParseInfo(SemanticParseInfo semanticParseInfo) {
|
private void fillSemanticParseInfo(SemanticParseInfo semanticParseInfo) {
|
||||||
List<SchemaElementMatch> schemaElementMatches = semanticParseInfo.getElementMatches();
|
List<SchemaElementMatch> schemaElementMatches = semanticParseInfo.getElementMatches();
|
||||||
if (!CollectionUtils.isEmpty(schemaElementMatches)) {
|
if (!CollectionUtils.isEmpty(schemaElementMatches)) {
|
||||||
schemaElementMatches.stream().filter(schemaElementMatch ->
|
schemaElementMatches.stream().filter(schemaElementMatch ->
|
||||||
SchemaElementType.VALUE.equals(schemaElementMatch.getElement().getType())
|
SchemaElementType.VALUE.equals(schemaElementMatch.getElement().getType())
|
||||||
|| SchemaElementType.ID.equals(schemaElementMatch.getElement().getType()))
|
|| SchemaElementType.ID.equals(schemaElementMatch.getElement().getType()))
|
||||||
.forEach(schemaElementMatch -> {
|
.forEach(schemaElementMatch -> {
|
||||||
QueryFilter queryFilter = new QueryFilter();
|
QueryFilter queryFilter = new QueryFilter();
|
||||||
queryFilter.setValue(schemaElementMatch.getWord());
|
queryFilter.setValue(schemaElementMatch.getWord());
|
||||||
queryFilter.setElementID(schemaElementMatch.getElement().getId());
|
queryFilter.setElementID(schemaElementMatch.getElement().getId());
|
||||||
queryFilter.setName(schemaElementMatch.getElement().getName());
|
queryFilter.setName(schemaElementMatch.getElement().getName());
|
||||||
queryFilter.setOperator(FilterOperatorEnum.EQUALS);
|
queryFilter.setOperator(FilterOperatorEnum.EQUALS);
|
||||||
queryFilter.setBizName(schemaElementMatch.getElement().getBizName());
|
queryFilter.setBizName(schemaElementMatch.getElement().getBizName());
|
||||||
semanticParseInfo.getDimensionFilters().add(queryFilter);
|
semanticParseInfo.getDimensionFilters().add(queryFilter);
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
protected String replaceText(String text, List<SchemaElementMatch> schemaElementMatches) {
|
|
||||||
if (CollectionUtils.isEmpty(schemaElementMatches)) {
|
|
||||||
return text;
|
|
||||||
}
|
|
||||||
List<SchemaElementMatch> valueSchemaElementMatches = schemaElementMatches.stream()
|
|
||||||
.filter(schemaElementMatch ->
|
|
||||||
SchemaElementType.VALUE.equals(schemaElementMatch.getElement().getType())
|
|
||||||
|| SchemaElementType.ID.equals(schemaElementMatch.getElement().getType()))
|
|
||||||
.collect(Collectors.toList());
|
|
||||||
for (SchemaElementMatch schemaElementMatch : valueSchemaElementMatches) {
|
|
||||||
String detectWord = schemaElementMatch.getDetectWord();
|
|
||||||
if (StringUtils.isBlank(detectWord)) {
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
text = text.replace(detectWord, "");
|
|
||||||
}
|
|
||||||
return text;
|
|
||||||
}
|
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,19 +1,22 @@
|
|||||||
package com.tencent.supersonic.chat.parser.embedding;
|
package com.tencent.supersonic.chat.parser.embedding;
|
||||||
|
|
||||||
import com.alibaba.fastjson.JSONObject;
|
import com.alibaba.fastjson.JSONObject;
|
||||||
import com.tencent.supersonic.chat.api.pojo.*;
|
import com.tencent.supersonic.chat.api.pojo.ChatContext;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.QueryContext;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SchemaMapInfo;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
|
||||||
import com.tencent.supersonic.chat.api.pojo.request.QueryFilter;
|
import com.tencent.supersonic.chat.api.pojo.request.QueryFilter;
|
||||||
import com.tencent.supersonic.chat.api.pojo.request.QueryFilters;
|
import com.tencent.supersonic.chat.api.pojo.request.QueryFilters;
|
||||||
import com.tencent.supersonic.chat.service.ConfigService;
|
import com.tencent.supersonic.chat.service.ConfigService;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Objects;
|
||||||
|
import java.util.Set;
|
||||||
import lombok.extern.slf4j.Slf4j;
|
import lombok.extern.slf4j.Slf4j;
|
||||||
import org.apache.commons.collections.CollectionUtils;
|
import org.apache.commons.collections.CollectionUtils;
|
||||||
import org.apache.commons.lang3.StringUtils;
|
import org.apache.commons.lang3.StringUtils;
|
||||||
import org.springframework.stereotype.Component;
|
import org.springframework.stereotype.Component;
|
||||||
|
|
||||||
import java.util.List;
|
|
||||||
import java.util.Objects;
|
|
||||||
import java.util.Set;
|
|
||||||
|
|
||||||
@Slf4j
|
@Slf4j
|
||||||
@Component("EmbeddingEntityResolver")
|
@Component("EmbeddingEntityResolver")
|
||||||
public class EmbeddingEntityResolver {
|
public class EmbeddingEntityResolver {
|
||||||
@@ -25,18 +28,19 @@ public class EmbeddingEntityResolver {
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
private Long getEntityValue(Long domainId, Long entityElementId, QueryContext queryCtx, ChatContext chatCtx) {
|
private Long getEntityValue(Long modelId, Long entityElementId, QueryContext queryCtx, ChatContext chatCtx) {
|
||||||
Long entityId = null;
|
Long entityId = null;
|
||||||
QueryFilters queryFilters = queryCtx.getRequest().getQueryFilters();
|
QueryFilters queryFilters = queryCtx.getRequest().getQueryFilters();
|
||||||
if (queryFilters != null) {
|
if (queryFilters != null) {
|
||||||
entityId = getEntityValueFromQueryFilter(queryFilters.getFilters());
|
entityId = getEntityValueFromQueryFilter(queryFilters.getFilters());
|
||||||
if (entityId != null) {
|
if (entityId != null) {
|
||||||
log.info("get entity id:{} domain id:{} from query filter :{} ", entityId, domainId, queryFilters);
|
log.info("get entity id:{} model id:{} from query filter :{} ", entityId, modelId, queryFilters);
|
||||||
return entityId;
|
return entityId;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
entityId = getEntityValueFromSchemaMapInfo(domainId, queryCtx.getMapInfo(), entityElementId);
|
entityId = getEntityValueFromSchemaMapInfo(modelId, queryCtx.getMapInfo(), entityElementId);
|
||||||
log.info("get entity id:{} from schema map Info :{} ", entityId, JSONObject.toJSONString(queryCtx.getMapInfo()));
|
log.info("get entity id:{} from schema map Info :{} ", entityId,
|
||||||
|
JSONObject.toJSONString(queryCtx.getMapInfo()));
|
||||||
if (entityId == null || entityId == 0) {
|
if (entityId == null || entityId == 0) {
|
||||||
Long entityIdFromChat = getEntityValueFromParseInfo(chatCtx.getParseInfo(), entityElementId);
|
Long entityIdFromChat = getEntityValueFromParseInfo(chatCtx.getParseInfo(), entityElementId);
|
||||||
if (entityIdFromChat != null && entityIdFromChat > 0) {
|
if (entityIdFromChat != null && entityIdFromChat > 0) {
|
||||||
@@ -75,8 +79,8 @@ public class EmbeddingEntityResolver {
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
private Long getEntityValueFromSchemaMapInfo(Long domainId, SchemaMapInfo schemaMapInfo, Long entityElementId) {
|
private Long getEntityValueFromSchemaMapInfo(Long modelId, SchemaMapInfo schemaMapInfo, Long entityElementId) {
|
||||||
List<SchemaElementMatch> schemaElementMatchList = schemaMapInfo.getMatchedElements(domainId);
|
List<SchemaElementMatch> schemaElementMatchList = schemaMapInfo.getMatchedElements(modelId);
|
||||||
if (CollectionUtils.isEmpty(schemaElementMatchList)) {
|
if (CollectionUtils.isEmpty(schemaElementMatchList)) {
|
||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,9 +1,8 @@
|
|||||||
package com.tencent.supersonic.chat.parser.embedding;
|
package com.tencent.supersonic.chat.parser.embedding;
|
||||||
|
|
||||||
|
|
||||||
import lombok.Data;
|
|
||||||
|
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
|
import lombok.Data;
|
||||||
|
|
||||||
@Data
|
@Data
|
||||||
public class EmbeddingResp {
|
public class EmbeddingResp {
|
||||||
|
|||||||
@@ -7,18 +7,20 @@ import com.tencent.supersonic.chat.api.pojo.QueryContext;
|
|||||||
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
|
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
|
||||||
import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
|
import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
|
||||||
import com.tencent.supersonic.chat.config.FunctionCallInfoConfig;
|
import com.tencent.supersonic.chat.config.FunctionCallInfoConfig;
|
||||||
|
import com.tencent.supersonic.chat.parser.ParseMode;
|
||||||
import com.tencent.supersonic.chat.parser.SatisfactionChecker;
|
import com.tencent.supersonic.chat.parser.SatisfactionChecker;
|
||||||
import com.tencent.supersonic.chat.plugin.Plugin;
|
import com.tencent.supersonic.chat.plugin.Plugin;
|
||||||
import com.tencent.supersonic.chat.plugin.PluginManager;
|
import com.tencent.supersonic.chat.plugin.PluginManager;
|
||||||
import com.tencent.supersonic.chat.plugin.PluginParseConfig;
|
import com.tencent.supersonic.chat.plugin.PluginParseConfig;
|
||||||
import com.tencent.supersonic.chat.plugin.PluginParseResult;
|
import com.tencent.supersonic.chat.plugin.PluginParseResult;
|
||||||
import com.tencent.supersonic.chat.query.QueryManager;
|
import com.tencent.supersonic.chat.query.QueryManager;
|
||||||
|
import com.tencent.supersonic.chat.query.dsl.DSLQuery;
|
||||||
import com.tencent.supersonic.chat.query.plugin.PluginSemanticQuery;
|
import com.tencent.supersonic.chat.query.plugin.PluginSemanticQuery;
|
||||||
import com.tencent.supersonic.chat.query.plugin.dsl.DSLQuery;
|
|
||||||
import com.tencent.supersonic.chat.service.PluginService;
|
import com.tencent.supersonic.chat.service.PluginService;
|
||||||
import com.tencent.supersonic.chat.utils.ComponentFactory;
|
import com.tencent.supersonic.chat.utils.ComponentFactory;
|
||||||
import com.tencent.supersonic.common.pojo.Constants;
|
import com.tencent.supersonic.common.pojo.Constants;
|
||||||
import com.tencent.supersonic.common.util.ContextUtils;
|
import com.tencent.supersonic.common.util.ContextUtils;
|
||||||
|
import com.tencent.supersonic.common.util.JsonUtil;
|
||||||
import java.net.URI;
|
import java.net.URI;
|
||||||
import java.util.ArrayList;
|
import java.util.ArrayList;
|
||||||
import java.util.HashMap;
|
import java.util.HashMap;
|
||||||
@@ -29,10 +31,9 @@ import java.util.Objects;
|
|||||||
import java.util.Optional;
|
import java.util.Optional;
|
||||||
import java.util.Set;
|
import java.util.Set;
|
||||||
import java.util.stream.Collectors;
|
import java.util.stream.Collectors;
|
||||||
|
|
||||||
import com.tencent.supersonic.common.util.JsonUtil;
|
|
||||||
import lombok.extern.slf4j.Slf4j;
|
import lombok.extern.slf4j.Slf4j;
|
||||||
import org.apache.commons.lang3.StringUtils;
|
import org.apache.commons.lang3.StringUtils;
|
||||||
|
import org.apache.commons.lang3.tuple.Pair;
|
||||||
import org.springframework.http.HttpEntity;
|
import org.springframework.http.HttpEntity;
|
||||||
import org.springframework.http.HttpHeaders;
|
import org.springframework.http.HttpHeaders;
|
||||||
import org.springframework.http.HttpMethod;
|
import org.springframework.http.HttpMethod;
|
||||||
@@ -55,21 +56,12 @@ public class FunctionBasedParser implements SemanticParser {
|
|||||||
FunctionCallInfoConfig functionCallConfig = ContextUtils.getBean(FunctionCallInfoConfig.class);
|
FunctionCallInfoConfig functionCallConfig = ContextUtils.getBean(FunctionCallInfoConfig.class);
|
||||||
PluginService pluginService = ContextUtils.getBean(PluginService.class);
|
PluginService pluginService = ContextUtils.getBean(PluginService.class);
|
||||||
String functionUrl = functionCallConfig.getUrl();
|
String functionUrl = functionCallConfig.getUrl();
|
||||||
|
|
||||||
if (StringUtils.isBlank(functionUrl) || SatisfactionChecker.check(queryCtx)) {
|
if (StringUtils.isBlank(functionUrl) || SatisfactionChecker.check(queryCtx)) {
|
||||||
log.info("functionUrl:{}, skip function parser, queryText:{}", functionUrl,
|
log.info("functionUrl:{}, skip function parser, queryText:{}", functionUrl,
|
||||||
queryCtx.getRequest().getQueryText());
|
queryCtx.getRequest().getQueryText());
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
List<PluginParseConfig> functionDOList = getFunctionDO(queryCtx.getRequest().getModelId(), queryCtx);
|
||||||
Set<Long> matchedDomains = getMatchDomains(queryCtx);
|
|
||||||
List<String> functionNames = getFunctionNames(matchedDomains);
|
|
||||||
log.info("matchedDomains:{},functionNames:{}", matchedDomains, functionNames);
|
|
||||||
|
|
||||||
if (CollectionUtils.isEmpty(functionNames) || CollectionUtils.isEmpty(matchedDomains)) {
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
List<PluginParseConfig> functionDOList = getFunctionDO(queryCtx.getRequest().getDomainId());
|
|
||||||
FunctionReq functionReq = FunctionReq.builder()
|
FunctionReq functionReq = FunctionReq.builder()
|
||||||
.queryText(queryCtx.getRequest().getQueryText())
|
.queryText(queryCtx.getRequest().getQueryText())
|
||||||
.pluginConfigs(functionDOList).build();
|
.pluginConfigs(functionDOList).build();
|
||||||
@@ -78,7 +70,6 @@ public class FunctionBasedParser implements SemanticParser {
|
|||||||
if (skipFunction(queryCtx, functionResp)) {
|
if (skipFunction(queryCtx, functionResp)) {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
PluginParseResult functionCallParseResult = new PluginParseResult();
|
PluginParseResult functionCallParseResult = new PluginParseResult();
|
||||||
String toolSelection = functionResp.getToolSelection();
|
String toolSelection = functionResp.getToolSelection();
|
||||||
Optional<Plugin> pluginOptional = pluginService.getPluginByName(toolSelection);
|
Optional<Plugin> pluginOptional = pluginService.getPluginByName(toolSelection);
|
||||||
@@ -87,24 +78,26 @@ public class FunctionBasedParser implements SemanticParser {
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
Plugin plugin = pluginOptional.get();
|
Plugin plugin = pluginOptional.get();
|
||||||
|
plugin.setParseMode(ParseMode.FUNCTION_CALL);
|
||||||
toolSelection = plugin.getType();
|
toolSelection = plugin.getType();
|
||||||
functionCallParseResult.setPlugin(plugin);
|
functionCallParseResult.setPlugin(plugin);
|
||||||
log.info("QueryManager PluginQueryModes:{}", QueryManager.getPluginQueryModes());
|
log.info("QueryManager PluginQueryModes:{}", QueryManager.getPluginQueryModes());
|
||||||
PluginSemanticQuery semanticQuery = QueryManager.createPluginQuery(toolSelection);
|
PluginSemanticQuery semanticQuery = QueryManager.createPluginQuery(toolSelection);
|
||||||
DomainResolver domainResolver = ComponentFactory.getDomainResolver();
|
ModelResolver ModelResolver = ComponentFactory.getModelResolver();
|
||||||
|
log.info("plugin ModelList:{}", plugin.getModelList());
|
||||||
Long domainId = domainResolver.resolve(queryCtx, chatCtx, plugin.getDomainList());
|
Pair<Boolean, List<Long>> pluginResolveResult = PluginManager.resolve(plugin, queryCtx);
|
||||||
log.info("FunctionBasedParser domainId:{}",domainId);
|
Long modelId = ModelResolver.resolve(queryCtx, chatCtx, pluginResolveResult.getRight());
|
||||||
if ((Objects.isNull(domainId) || domainId <= 0) && !plugin.isContainsAllDomain()) {
|
log.info("FunctionBasedParser modelId:{}", modelId);
|
||||||
log.info("domain is null, skip the parse, select tool: {}", toolSelection);
|
if ((Objects.isNull(modelId) || modelId <= 0) && !plugin.isContainsAllModel()) {
|
||||||
|
log.info("Model is null, skip the parse, select tool: {}", toolSelection);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
if (!plugin.getDomainList().contains(domainId) && !plugin.isContainsAllDomain()) {
|
if (!plugin.getModelList().contains(modelId) && !plugin.isContainsAllModel()) {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
SemanticParseInfo parseInfo = semanticQuery.getParseInfo();
|
SemanticParseInfo parseInfo = semanticQuery.getParseInfo();
|
||||||
if (Objects.nonNull(domainId) && domainId > 0){
|
if (Objects.nonNull(modelId) && modelId > 0) {
|
||||||
parseInfo.getElementMatches().addAll(queryCtx.getMapInfo().getMatchedElements(domainId));
|
parseInfo.getElementMatches().addAll(queryCtx.getMapInfo().getMatchedElements(modelId));
|
||||||
}
|
}
|
||||||
functionCallParseResult.setRequest(queryCtx.getRequest());
|
functionCallParseResult.setRequest(queryCtx.getRequest());
|
||||||
Map<String, Object> properties = new HashMap<>();
|
Map<String, Object> properties = new HashMap<>();
|
||||||
@@ -112,21 +105,22 @@ public class FunctionBasedParser implements SemanticParser {
|
|||||||
parseInfo.setProperties(properties);
|
parseInfo.setProperties(properties);
|
||||||
parseInfo.setScore(FUNCTION_BONUS_THRESHOLD);
|
parseInfo.setScore(FUNCTION_BONUS_THRESHOLD);
|
||||||
parseInfo.setQueryMode(semanticQuery.getQueryMode());
|
parseInfo.setQueryMode(semanticQuery.getQueryMode());
|
||||||
SchemaElement domain = new SchemaElement();
|
SchemaElement Model = new SchemaElement();
|
||||||
domain.setDomain(domainId);
|
Model.setModel(modelId);
|
||||||
domain.setId(domainId);
|
Model.setId(modelId);
|
||||||
parseInfo.setDomain(domain);
|
parseInfo.setModel(Model);
|
||||||
queryCtx.getCandidateQueries().add(semanticQuery);
|
queryCtx.getCandidateQueries().add(semanticQuery);
|
||||||
}
|
}
|
||||||
|
|
||||||
private Set<Long> getMatchDomains(QueryContext queryCtx) {
|
|
||||||
|
private Set<Long> getMatchModels(QueryContext queryCtx) {
|
||||||
Set<Long> result = new HashSet<>();
|
Set<Long> result = new HashSet<>();
|
||||||
Long domainId = queryCtx.getRequest().getDomainId();
|
Long modelId = queryCtx.getRequest().getModelId();
|
||||||
if (Objects.nonNull(domainId) && domainId > 0) {
|
if (Objects.nonNull(modelId) && modelId > 0) {
|
||||||
result.add(domainId);
|
result.add(modelId);
|
||||||
return result;
|
return result;
|
||||||
}
|
}
|
||||||
return queryCtx.getMapInfo().getMatchedDomains();
|
return queryCtx.getMapInfo().getMatchedModels();
|
||||||
}
|
}
|
||||||
|
|
||||||
private boolean skipFunction(QueryContext queryCtx, FunctionResp functionResp) {
|
private boolean skipFunction(QueryContext queryCtx, FunctionResp functionResp) {
|
||||||
@@ -144,36 +138,46 @@ public class FunctionBasedParser implements SemanticParser {
|
|||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
private List<PluginParseConfig> getFunctionDO(Long domainId) {
|
private List<PluginParseConfig> getFunctionDO(Long modelId, QueryContext queryContext) {
|
||||||
log.info("user decide domain:{}", domainId);
|
log.info("user decide Model:{}", modelId);
|
||||||
List<Plugin> plugins = PluginManager.getPlugins();
|
List<Plugin> plugins = PluginManager.getPlugins();
|
||||||
List<PluginParseConfig> functionDOList = plugins.stream().filter(o -> {
|
List<PluginParseConfig> functionDOList = plugins.stream().filter(plugin -> {
|
||||||
if (o.getParseModeConfig() == null) {
|
if (DSLQuery.QUERY_MODE.equalsIgnoreCase(plugin.getType())) {
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
if (!CollectionUtils.isEmpty(o.getDomainList())) {//过滤掉没选主题域的插件
|
if (plugin.getParseModeConfig() == null) {
|
||||||
return true;
|
return false;
|
||||||
}
|
}
|
||||||
if (domainId == null || domainId <= 0L) {
|
PluginParseConfig pluginParseConfig = JsonUtil.toObject(plugin.getParseModeConfig(),
|
||||||
return true;
|
|
||||||
} else {
|
|
||||||
return o.getDomainList().contains(domainId);
|
|
||||||
}
|
|
||||||
}).map(o -> {
|
|
||||||
PluginParseConfig functionCallConfig = JsonUtil.toObject(o.getParseModeConfig(),
|
|
||||||
PluginParseConfig.class);
|
PluginParseConfig.class);
|
||||||
return functionCallConfig;
|
if (StringUtils.isBlank(pluginParseConfig.getName())) {
|
||||||
}).collect(Collectors.toList());
|
return false;
|
||||||
|
}
|
||||||
|
Pair<Boolean, List<Long>> pluginResolverResult = PluginManager.resolve(plugin, queryContext);
|
||||||
|
log.info("embedding plugin [{}-{}] resolve: {}", plugin.getId(), plugin.getName(), pluginResolverResult);
|
||||||
|
if (!pluginResolverResult.getLeft()) {
|
||||||
|
return false;
|
||||||
|
} else {
|
||||||
|
List<Long> resolveModel = pluginResolverResult.getRight();
|
||||||
|
if (modelId != null && modelId > 0) {
|
||||||
|
if (plugin.isContainsAllModel()) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
return resolveModel.contains(modelId);
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
}).map(o -> JsonUtil.toObject(o.getParseModeConfig(), PluginParseConfig.class)).collect(Collectors.toList());
|
||||||
log.info("getFunctionDO:{}", JsonUtil.toString(functionDOList));
|
log.info("getFunctionDO:{}", JsonUtil.toString(functionDOList));
|
||||||
return functionDOList;
|
return functionDOList;
|
||||||
}
|
}
|
||||||
|
|
||||||
private List<String> getFunctionNames(Set<Long> matchedDomains) {
|
private List<String> getFunctionNames(Set<Long> matchedModels) {
|
||||||
List<Plugin> plugins = PluginManager.getPlugins();
|
List<Plugin> plugins = PluginManager.getPlugins();
|
||||||
Set<String> functionNames = plugins.stream()
|
Set<String> functionNames = plugins.stream()
|
||||||
.filter(entry -> {
|
.filter(entry -> {
|
||||||
if (!CollectionUtils.isEmpty(entry.getDomainList()) && !CollectionUtils.isEmpty(matchedDomains)) {
|
if (!CollectionUtils.isEmpty(entry.getModelList()) && !CollectionUtils.isEmpty(matchedModels)) {
|
||||||
return entry.getDomainList().stream().anyMatch(matchedDomains::contains);
|
return entry.getModelList().stream().anyMatch(matchedModels::contains);
|
||||||
}
|
}
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,8 +1,7 @@
|
|||||||
package com.tencent.supersonic.chat.parser.function;
|
package com.tencent.supersonic.chat.parser.function;
|
||||||
|
|
||||||
import java.util.List;
|
|
||||||
|
|
||||||
import com.tencent.supersonic.chat.plugin.PluginParseConfig;
|
import com.tencent.supersonic.chat.plugin.PluginParseConfig;
|
||||||
|
import java.util.List;
|
||||||
import lombok.Builder;
|
import lombok.Builder;
|
||||||
import lombok.Data;
|
import lombok.Data;
|
||||||
|
|
||||||
|
|||||||
@@ -1,181 +0,0 @@
|
|||||||
package com.tencent.supersonic.chat.parser.function;
|
|
||||||
|
|
||||||
import com.tencent.supersonic.chat.api.pojo.*;
|
|
||||||
import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
|
|
||||||
import com.tencent.supersonic.chat.api.component.SemanticQuery;
|
|
||||||
import lombok.extern.slf4j.Slf4j;
|
|
||||||
|
|
||||||
import java.util.*;
|
|
||||||
import java.util.stream.Collectors;
|
|
||||||
import org.apache.commons.collections.CollectionUtils;
|
|
||||||
|
|
||||||
@Slf4j
|
|
||||||
public class HeuristicDomainResolver implements DomainResolver {
|
|
||||||
|
|
||||||
protected static Long selectDomainBySchemaElementCount(Map<Long, SemanticQuery> domainQueryModes,
|
|
||||||
SchemaMapInfo schemaMap) {
|
|
||||||
Map<Long, DomainMatchResult> domainTypeMap = getDomainTypeMap(schemaMap);
|
|
||||||
if (domainTypeMap.size() == 1) {
|
|
||||||
Long domainSelect = domainTypeMap.entrySet().stream().collect(Collectors.toList()).get(0).getKey();
|
|
||||||
if (domainQueryModes.containsKey(domainSelect)) {
|
|
||||||
log.info("selectDomain with only one domain [{}]", domainSelect);
|
|
||||||
return domainSelect;
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
Map.Entry<Long, DomainMatchResult> maxDomain = domainTypeMap.entrySet().stream()
|
|
||||||
.filter(entry -> domainQueryModes.containsKey(entry.getKey()))
|
|
||||||
.sorted((o1, o2) -> {
|
|
||||||
int difference = o2.getValue().getCount() - o1.getValue().getCount();
|
|
||||||
if (difference == 0) {
|
|
||||||
return (int) ((o2.getValue().getMaxSimilarity()
|
|
||||||
- o1.getValue().getMaxSimilarity()) * 100);
|
|
||||||
}
|
|
||||||
return difference;
|
|
||||||
}).findFirst().orElse(null);
|
|
||||||
if (maxDomain != null) {
|
|
||||||
log.info("selectDomain with multiple domains [{}]", maxDomain.getKey());
|
|
||||||
return maxDomain.getKey();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return 0L;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* to check can switch domain if context exit domain
|
|
||||||
*
|
|
||||||
* @return false will use context domain, true will use other domain , maybe include context domain
|
|
||||||
*/
|
|
||||||
protected static boolean isAllowSwitch(Map<Long, SemanticQuery> domainQueryModes, SchemaMapInfo schemaMap,
|
|
||||||
ChatContext chatCtx, QueryReq searchCtx, Long domainId, List<Long> restrictiveDomains) {
|
|
||||||
if (!Objects.nonNull(domainId) || domainId <= 0) {
|
|
||||||
return true;
|
|
||||||
}
|
|
||||||
// except content domain, calculate the number of types for each domain, if numbers<=1 will not switch
|
|
||||||
Map<Long, DomainMatchResult> domainTypeMap = getDomainTypeMap(schemaMap);
|
|
||||||
log.info("isAllowSwitch domainTypeMap [{}]", domainTypeMap);
|
|
||||||
long otherDomainTypeNumBigOneCount = domainTypeMap.entrySet().stream()
|
|
||||||
.filter(entry -> domainQueryModes.containsKey(entry.getKey()) && !entry.getKey().equals(domainId))
|
|
||||||
.filter(entry -> entry.getValue().getCount() > 1).count();
|
|
||||||
if (otherDomainTypeNumBigOneCount >= 1) {
|
|
||||||
return true;
|
|
||||||
}
|
|
||||||
// if query text only contain time , will not switch
|
|
||||||
if (!CollectionUtils.isEmpty(domainQueryModes.values())) {
|
|
||||||
for (SemanticQuery semanticQuery : domainQueryModes.values()) {
|
|
||||||
if (semanticQuery == null) {
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
SemanticParseInfo semanticParseInfo = semanticQuery.getParseInfo();
|
|
||||||
if (semanticParseInfo == null) {
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
if (searchCtx.getQueryText() != null && semanticParseInfo.getDateInfo() != null) {
|
|
||||||
if (semanticParseInfo.getDateInfo().getText() != null) {
|
|
||||||
if (semanticParseInfo.getDateInfo().getText().equalsIgnoreCase(searchCtx.getQueryText())) {
|
|
||||||
log.info("timeParseResults is not null , can not switch context , timeParseResults:{},",
|
|
||||||
semanticParseInfo.getDateInfo());
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if (CollectionUtils.isNotEmpty(restrictiveDomains) && !restrictiveDomains.contains(domainId)) {
|
|
||||||
return true;
|
|
||||||
}
|
|
||||||
// if context domain not in schemaMap , will switch
|
|
||||||
if (schemaMap.getMatchedElements(domainId) == null || schemaMap.getMatchedElements(domainId).size() <= 0) {
|
|
||||||
log.info("domainId not in schemaMap ");
|
|
||||||
return true;
|
|
||||||
}
|
|
||||||
// other will not switch
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
|
|
||||||
public static Map<Long, DomainMatchResult> getDomainTypeMap(SchemaMapInfo schemaMap) {
|
|
||||||
Map<Long, DomainMatchResult> domainCount = new HashMap<>();
|
|
||||||
for (Map.Entry<Long, List<SchemaElementMatch>> entry : schemaMap.getDomainElementMatches().entrySet()) {
|
|
||||||
List<SchemaElementMatch> schemaElementMatches = schemaMap.getMatchedElements(entry.getKey());
|
|
||||||
if (schemaElementMatches != null && schemaElementMatches.size() > 0) {
|
|
||||||
if (!domainCount.containsKey(entry.getKey())) {
|
|
||||||
domainCount.put(entry.getKey(), new DomainMatchResult());
|
|
||||||
}
|
|
||||||
DomainMatchResult domainMatchResult = domainCount.get(entry.getKey());
|
|
||||||
Set<SchemaElementType> schemaElementTypes = new HashSet<>();
|
|
||||||
schemaElementMatches.stream()
|
|
||||||
.forEach(schemaElementMatch -> schemaElementTypes.add(
|
|
||||||
schemaElementMatch.getElement().getType()));
|
|
||||||
SchemaElementMatch schemaElementMatchMax = schemaElementMatches.stream()
|
|
||||||
.sorted((o1, o2) ->
|
|
||||||
((int) ((o2.getSimilarity() - o1.getSimilarity()) * 100))
|
|
||||||
).findFirst().orElse(null);
|
|
||||||
if (schemaElementMatchMax != null) {
|
|
||||||
domainMatchResult.setMaxSimilarity(schemaElementMatchMax.getSimilarity());
|
|
||||||
}
|
|
||||||
domainMatchResult.setCount(schemaElementTypes.size());
|
|
||||||
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return domainCount;
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
public Long resolve(QueryContext queryContext, ChatContext chatCtx, List<Long> restrictiveDomains) {
|
|
||||||
Long domainId = queryContext.getRequest().getDomainId();
|
|
||||||
if (Objects.nonNull(domainId) && domainId > 0) {
|
|
||||||
if (CollectionUtils.isNotEmpty(restrictiveDomains) && restrictiveDomains.contains(domainId)) {
|
|
||||||
return domainId;
|
|
||||||
} else {
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
SchemaMapInfo mapInfo = queryContext.getMapInfo();
|
|
||||||
Set<Long> matchedDomains = mapInfo.getMatchedDomains();
|
|
||||||
if (CollectionUtils.isNotEmpty(restrictiveDomains)) {
|
|
||||||
matchedDomains = matchedDomains.stream()
|
|
||||||
.filter(matchedDomain -> restrictiveDomains.contains(matchedDomain))
|
|
||||||
.collect(Collectors.toSet());
|
|
||||||
}
|
|
||||||
Map<Long, SemanticQuery> domainQueryModes = new HashMap<>();
|
|
||||||
for (Long matchedDomain : matchedDomains) {
|
|
||||||
domainQueryModes.put(matchedDomain, null);
|
|
||||||
}
|
|
||||||
if(domainQueryModes.size()==1){
|
|
||||||
return domainQueryModes.keySet().stream().findFirst().get();
|
|
||||||
}
|
|
||||||
return resolve(domainQueryModes, queryContext, chatCtx,
|
|
||||||
queryContext.getMapInfo(),restrictiveDomains);
|
|
||||||
}
|
|
||||||
|
|
||||||
public Long resolve(Map<Long, SemanticQuery> domainQueryModes, QueryContext queryContext,
|
|
||||||
ChatContext chatCtx, SchemaMapInfo schemaMap, List<Long> restrictiveDomains) {
|
|
||||||
Long selectDomain = selectDomain(domainQueryModes, queryContext.getRequest(), chatCtx, schemaMap,restrictiveDomains);
|
|
||||||
if (selectDomain > 0) {
|
|
||||||
log.info("selectDomain {} ", selectDomain);
|
|
||||||
return selectDomain;
|
|
||||||
}
|
|
||||||
// get the max SchemaElementType number
|
|
||||||
return selectDomainBySchemaElementCount(domainQueryModes, schemaMap);
|
|
||||||
}
|
|
||||||
|
|
||||||
public Long selectDomain(Map<Long, SemanticQuery> domainQueryModes, QueryReq queryContext,
|
|
||||||
ChatContext chatCtx,
|
|
||||||
SchemaMapInfo schemaMap, List<Long> restrictiveDomains) {
|
|
||||||
// if QueryContext has domainId and in domainQueryModes
|
|
||||||
if (domainQueryModes.containsKey(queryContext.getDomainId())) {
|
|
||||||
log.info("selectDomain from QueryContext [{}]", queryContext.getDomainId());
|
|
||||||
return queryContext.getDomainId();
|
|
||||||
}
|
|
||||||
// if ChatContext has domainId and in domainQueryModes
|
|
||||||
if (chatCtx.getParseInfo().getDomainId() > 0) {
|
|
||||||
Long domainId = chatCtx.getParseInfo().getDomainId();
|
|
||||||
if (!isAllowSwitch(domainQueryModes, schemaMap, chatCtx, queryContext, domainId,restrictiveDomains)) {
|
|
||||||
log.info("selectDomain from ChatContext [{}]", domainId);
|
|
||||||
return domainId;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
// default 0
|
|
||||||
return 0L;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -0,0 +1,192 @@
|
|||||||
|
package com.tencent.supersonic.chat.parser.function;
|
||||||
|
|
||||||
|
import com.tencent.supersonic.chat.api.component.SemanticQuery;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.ChatContext;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.QueryContext;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SchemaMapInfo;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
|
||||||
|
import java.util.HashMap;
|
||||||
|
import java.util.HashSet;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Map;
|
||||||
|
import java.util.Objects;
|
||||||
|
import java.util.Set;
|
||||||
|
import java.util.stream.Collectors;
|
||||||
|
import lombok.extern.slf4j.Slf4j;
|
||||||
|
import org.apache.commons.collections.CollectionUtils;
|
||||||
|
|
||||||
|
@Slf4j
|
||||||
|
public class HeuristicModelResolver implements ModelResolver {
|
||||||
|
|
||||||
|
protected static Long selectModelBySchemaElementCount(Map<Long, SemanticQuery> ModelQueryModes,
|
||||||
|
SchemaMapInfo schemaMap) {
|
||||||
|
Map<Long, ModelMatchResult> ModelTypeMap = getModelTypeMap(schemaMap);
|
||||||
|
if (ModelTypeMap.size() == 1) {
|
||||||
|
Long ModelSelect = ModelTypeMap.entrySet().stream().collect(Collectors.toList()).get(0).getKey();
|
||||||
|
if (ModelQueryModes.containsKey(ModelSelect)) {
|
||||||
|
log.info("selectModel with only one Model [{}]", ModelSelect);
|
||||||
|
return ModelSelect;
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
Map.Entry<Long, ModelMatchResult> maxModel = ModelTypeMap.entrySet().stream()
|
||||||
|
.filter(entry -> ModelQueryModes.containsKey(entry.getKey()))
|
||||||
|
.sorted((o1, o2) -> {
|
||||||
|
int difference = o2.getValue().getCount() - o1.getValue().getCount();
|
||||||
|
if (difference == 0) {
|
||||||
|
return (int) ((o2.getValue().getMaxSimilarity()
|
||||||
|
- o1.getValue().getMaxSimilarity()) * 100);
|
||||||
|
}
|
||||||
|
return difference;
|
||||||
|
}).findFirst().orElse(null);
|
||||||
|
if (maxModel != null) {
|
||||||
|
log.info("selectModel with multiple Models [{}]", maxModel.getKey());
|
||||||
|
return maxModel.getKey();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return 0L;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* to check can switch Model if context exit Model
|
||||||
|
*
|
||||||
|
* @return false will use context Model, true will use other Model , maybe include context Model
|
||||||
|
*/
|
||||||
|
protected static boolean isAllowSwitch(Map<Long, SemanticQuery> ModelQueryModes, SchemaMapInfo schemaMap,
|
||||||
|
ChatContext chatCtx, QueryReq searchCtx, Long modelId, List<Long> restrictiveModels) {
|
||||||
|
if (!Objects.nonNull(modelId) || modelId <= 0) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
// except content Model, calculate the number of types for each Model, if numbers<=1 will not switch
|
||||||
|
Map<Long, ModelMatchResult> ModelTypeMap = getModelTypeMap(schemaMap);
|
||||||
|
log.info("isAllowSwitch ModelTypeMap [{}]", ModelTypeMap);
|
||||||
|
long otherModelTypeNumBigOneCount = ModelTypeMap.entrySet().stream()
|
||||||
|
.filter(entry -> ModelQueryModes.containsKey(entry.getKey()) && !entry.getKey().equals(modelId))
|
||||||
|
.filter(entry -> entry.getValue().getCount() > 1).count();
|
||||||
|
if (otherModelTypeNumBigOneCount >= 1) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
// if query text only contain time , will not switch
|
||||||
|
if (!CollectionUtils.isEmpty(ModelQueryModes.values())) {
|
||||||
|
for (SemanticQuery semanticQuery : ModelQueryModes.values()) {
|
||||||
|
if (semanticQuery == null) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
SemanticParseInfo semanticParseInfo = semanticQuery.getParseInfo();
|
||||||
|
if (semanticParseInfo == null) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (searchCtx.getQueryText() != null && semanticParseInfo.getDateInfo() != null) {
|
||||||
|
if (semanticParseInfo.getDateInfo().getDetectWord() != null) {
|
||||||
|
if (semanticParseInfo.getDateInfo().getDetectWord()
|
||||||
|
.equalsIgnoreCase(searchCtx.getQueryText())) {
|
||||||
|
log.info("timeParseResults is not null , can not switch context , timeParseResults:{},",
|
||||||
|
semanticParseInfo.getDateInfo());
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (CollectionUtils.isNotEmpty(restrictiveModels) && !restrictiveModels.contains(modelId)) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
// if context Model not in schemaMap , will switch
|
||||||
|
if (schemaMap.getMatchedElements(modelId) == null || schemaMap.getMatchedElements(modelId).size() <= 0) {
|
||||||
|
log.info("modelId not in schemaMap ");
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
// other will not switch
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
public static Map<Long, ModelMatchResult> getModelTypeMap(SchemaMapInfo schemaMap) {
|
||||||
|
Map<Long, ModelMatchResult> ModelCount = new HashMap<>();
|
||||||
|
for (Map.Entry<Long, List<SchemaElementMatch>> entry : schemaMap.getModelElementMatches().entrySet()) {
|
||||||
|
List<SchemaElementMatch> schemaElementMatches = schemaMap.getMatchedElements(entry.getKey());
|
||||||
|
if (schemaElementMatches != null && schemaElementMatches.size() > 0) {
|
||||||
|
if (!ModelCount.containsKey(entry.getKey())) {
|
||||||
|
ModelCount.put(entry.getKey(), new ModelMatchResult());
|
||||||
|
}
|
||||||
|
ModelMatchResult ModelMatchResult = ModelCount.get(entry.getKey());
|
||||||
|
Set<SchemaElementType> schemaElementTypes = new HashSet<>();
|
||||||
|
schemaElementMatches.stream()
|
||||||
|
.forEach(schemaElementMatch -> schemaElementTypes.add(
|
||||||
|
schemaElementMatch.getElement().getType()));
|
||||||
|
SchemaElementMatch schemaElementMatchMax = schemaElementMatches.stream()
|
||||||
|
.sorted((o1, o2) ->
|
||||||
|
((int) ((o2.getSimilarity() - o1.getSimilarity()) * 100))
|
||||||
|
).findFirst().orElse(null);
|
||||||
|
if (schemaElementMatchMax != null) {
|
||||||
|
ModelMatchResult.setMaxSimilarity(schemaElementMatchMax.getSimilarity());
|
||||||
|
}
|
||||||
|
ModelMatchResult.setCount(schemaElementTypes.size());
|
||||||
|
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return ModelCount;
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
public Long resolve(QueryContext queryContext, ChatContext chatCtx, List<Long> restrictiveModels) {
|
||||||
|
Long modelId = queryContext.getRequest().getModelId();
|
||||||
|
if (Objects.nonNull(modelId) && modelId > 0) {
|
||||||
|
if (CollectionUtils.isNotEmpty(restrictiveModels) && restrictiveModels.contains(modelId)) {
|
||||||
|
return modelId;
|
||||||
|
} else {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
SchemaMapInfo mapInfo = queryContext.getMapInfo();
|
||||||
|
Set<Long> matchedModels = mapInfo.getMatchedModels();
|
||||||
|
if (CollectionUtils.isNotEmpty(restrictiveModels)) {
|
||||||
|
matchedModels = matchedModels.stream()
|
||||||
|
.filter(restrictiveModels::contains)
|
||||||
|
.collect(Collectors.toSet());
|
||||||
|
}
|
||||||
|
Map<Long, SemanticQuery> ModelQueryModes = new HashMap<>();
|
||||||
|
for (Long matchedModel : matchedModels) {
|
||||||
|
ModelQueryModes.put(matchedModel, null);
|
||||||
|
}
|
||||||
|
if (ModelQueryModes.size() == 1) {
|
||||||
|
return ModelQueryModes.keySet().stream().findFirst().get();
|
||||||
|
}
|
||||||
|
return resolve(ModelQueryModes, queryContext, chatCtx,
|
||||||
|
queryContext.getMapInfo(), restrictiveModels);
|
||||||
|
}
|
||||||
|
|
||||||
|
public Long resolve(Map<Long, SemanticQuery> ModelQueryModes, QueryContext queryContext,
|
||||||
|
ChatContext chatCtx, SchemaMapInfo schemaMap, List<Long> restrictiveModels) {
|
||||||
|
Long selectModel = selectModel(ModelQueryModes, queryContext.getRequest(), chatCtx, schemaMap,
|
||||||
|
restrictiveModels);
|
||||||
|
if (selectModel > 0) {
|
||||||
|
log.info("selectModel {} ", selectModel);
|
||||||
|
return selectModel;
|
||||||
|
}
|
||||||
|
// get the max SchemaElementType number
|
||||||
|
return selectModelBySchemaElementCount(ModelQueryModes, schemaMap);
|
||||||
|
}
|
||||||
|
|
||||||
|
public Long selectModel(Map<Long, SemanticQuery> ModelQueryModes, QueryReq queryContext,
|
||||||
|
ChatContext chatCtx,
|
||||||
|
SchemaMapInfo schemaMap, List<Long> restrictiveModels) {
|
||||||
|
// if QueryContext has modelId and in ModelQueryModes
|
||||||
|
if (ModelQueryModes.containsKey(queryContext.getModelId())) {
|
||||||
|
log.info("selectModel from QueryContext [{}]", queryContext.getModelId());
|
||||||
|
return queryContext.getModelId();
|
||||||
|
}
|
||||||
|
// if ChatContext has modelId and in ModelQueryModes
|
||||||
|
if (chatCtx.getParseInfo().getModelId() > 0) {
|
||||||
|
Long modelId = chatCtx.getParseInfo().getModelId();
|
||||||
|
if (!isAllowSwitch(ModelQueryModes, schemaMap, chatCtx, queryContext, modelId, restrictiveModels)) {
|
||||||
|
log.info("selectModel from ChatContext [{}]", modelId);
|
||||||
|
return modelId;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// default 0
|
||||||
|
return 0L;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -3,7 +3,8 @@ package com.tencent.supersonic.chat.parser.function;
|
|||||||
import lombok.Data;
|
import lombok.Data;
|
||||||
|
|
||||||
@Data
|
@Data
|
||||||
public class DomainMatchResult {
|
public class ModelMatchResult {
|
||||||
|
|
||||||
private Integer count = 0;
|
private Integer count = 0;
|
||||||
private double maxSimilarity;
|
private double maxSimilarity;
|
||||||
}
|
}
|
||||||
@@ -5,8 +5,8 @@ import com.tencent.supersonic.chat.api.pojo.ChatContext;
|
|||||||
import com.tencent.supersonic.chat.api.pojo.QueryContext;
|
import com.tencent.supersonic.chat.api.pojo.QueryContext;
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
|
|
||||||
public interface DomainResolver {
|
public interface ModelResolver {
|
||||||
|
|
||||||
Long resolve(QueryContext queryContext, ChatContext chatCtx, List<Long> restrictiveDomains);
|
Long resolve(QueryContext queryContext, ChatContext chatCtx, List<Long> restrictiveModels);
|
||||||
|
|
||||||
}
|
}
|
||||||
@@ -1,8 +1,8 @@
|
|||||||
package com.tencent.supersonic.chat.parser.function;
|
package com.tencent.supersonic.chat.parser.function;
|
||||||
|
|
||||||
import lombok.Data;
|
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
import java.util.Map;
|
import java.util.Map;
|
||||||
|
import lombok.Data;
|
||||||
|
|
||||||
@Data
|
@Data
|
||||||
public class Parameters {
|
public class Parameters {
|
||||||
|
|||||||
@@ -0,0 +1,11 @@
|
|||||||
|
package com.tencent.supersonic.chat.parser.llm;
|
||||||
|
|
||||||
|
import com.tencent.supersonic.chat.plugin.PluginParseResult;
|
||||||
|
import com.tencent.supersonic.chat.query.dsl.LLMResp;
|
||||||
|
import lombok.Data;
|
||||||
|
|
||||||
|
@Data
|
||||||
|
public class DSLParseResult extends PluginParseResult {
|
||||||
|
|
||||||
|
private LLMResp llmResp;
|
||||||
|
}
|
||||||
@@ -0,0 +1,231 @@
|
|||||||
|
package com.tencent.supersonic.chat.parser.llm;
|
||||||
|
|
||||||
|
import com.tencent.supersonic.chat.api.component.SemanticParser;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.ChatContext;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.QueryContext;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SemanticSchema;
|
||||||
|
import com.tencent.supersonic.chat.config.LLMConfig;
|
||||||
|
import com.tencent.supersonic.chat.parser.SatisfactionChecker;
|
||||||
|
import com.tencent.supersonic.chat.parser.function.ModelResolver;
|
||||||
|
import com.tencent.supersonic.chat.plugin.Plugin;
|
||||||
|
import com.tencent.supersonic.chat.plugin.PluginManager;
|
||||||
|
import com.tencent.supersonic.chat.query.QueryManager;
|
||||||
|
import com.tencent.supersonic.chat.query.dsl.DSLBuilder;
|
||||||
|
import com.tencent.supersonic.chat.query.dsl.DSLQuery;
|
||||||
|
import com.tencent.supersonic.chat.query.dsl.LLMReq;
|
||||||
|
import com.tencent.supersonic.chat.query.dsl.LLMReq.ElementValue;
|
||||||
|
import com.tencent.supersonic.chat.query.dsl.LLMResp;
|
||||||
|
import com.tencent.supersonic.chat.query.plugin.PluginSemanticQuery;
|
||||||
|
import com.tencent.supersonic.chat.utils.ComponentFactory;
|
||||||
|
import com.tencent.supersonic.common.pojo.Constants;
|
||||||
|
import com.tencent.supersonic.common.util.ContextUtils;
|
||||||
|
import com.tencent.supersonic.common.util.DateUtils;
|
||||||
|
import com.tencent.supersonic.common.util.JsonUtil;
|
||||||
|
import com.tencent.supersonic.knowledge.service.SchemaService;
|
||||||
|
import java.util.ArrayList;
|
||||||
|
import java.util.HashMap;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Map;
|
||||||
|
import java.util.Objects;
|
||||||
|
import java.util.Set;
|
||||||
|
import java.util.stream.Collectors;
|
||||||
|
import lombok.extern.slf4j.Slf4j;
|
||||||
|
import org.apache.commons.lang3.StringUtils;
|
||||||
|
import org.springframework.http.HttpEntity;
|
||||||
|
import org.springframework.http.HttpHeaders;
|
||||||
|
import org.springframework.http.HttpMethod;
|
||||||
|
import org.springframework.http.MediaType;
|
||||||
|
import org.springframework.http.ResponseEntity;
|
||||||
|
import org.springframework.util.CollectionUtils;
|
||||||
|
import org.springframework.web.client.RestTemplate;
|
||||||
|
|
||||||
|
@Slf4j
|
||||||
|
public class LLMDSLParser implements SemanticParser {
|
||||||
|
|
||||||
|
public static final double FUNCTION_BONUS_THRESHOLD = 201;
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void parse(QueryContext queryCtx, ChatContext chatCtx) {
|
||||||
|
final LLMConfig llmConfig = ContextUtils.getBean(LLMConfig.class);
|
||||||
|
if (StringUtils.isEmpty(llmConfig.getUrl()) || SatisfactionChecker.check(queryCtx)) {
|
||||||
|
log.info("llmConfig:{}, skip function parser, queryText:{}", llmConfig,
|
||||||
|
queryCtx.getRequest().getQueryText());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
List<Plugin> dslPlugins = PluginManager.getPlugins().stream()
|
||||||
|
.filter(plugin -> DSLQuery.QUERY_MODE.equalsIgnoreCase(plugin.getType()))
|
||||||
|
.collect(Collectors.toList());
|
||||||
|
|
||||||
|
if (CollectionUtils.isEmpty(dslPlugins)) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
Plugin plugin = dslPlugins.get(0);
|
||||||
|
List<Long> dslModels = plugin.getModelList();
|
||||||
|
|
||||||
|
try {
|
||||||
|
ModelResolver modelResolver = ComponentFactory.getModelResolver();
|
||||||
|
Long modelId = modelResolver.resolve(queryCtx, chatCtx, dslModels);
|
||||||
|
log.info("resolve modelId:{},dslModels:{}", modelId, dslModels);
|
||||||
|
|
||||||
|
if (Objects.isNull(modelId)) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
LLMResp llmResp = requestLLM(queryCtx, modelId);
|
||||||
|
if (Objects.isNull(llmResp)) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
PluginSemanticQuery semanticQuery = QueryManager.createPluginQuery(DSLQuery.QUERY_MODE);
|
||||||
|
|
||||||
|
SemanticParseInfo parseInfo = semanticQuery.getParseInfo();
|
||||||
|
if (Objects.nonNull(modelId) && modelId > 0) {
|
||||||
|
parseInfo.getElementMatches().addAll(queryCtx.getMapInfo().getMatchedElements(modelId));
|
||||||
|
}
|
||||||
|
DSLParseResult dslParseResult = new DSLParseResult();
|
||||||
|
dslParseResult.setRequest(queryCtx.getRequest());
|
||||||
|
dslParseResult.setLlmResp(llmResp);
|
||||||
|
dslParseResult.setPlugin(plugin);
|
||||||
|
|
||||||
|
Map<String, Object> properties = new HashMap<>();
|
||||||
|
properties.put(Constants.CONTEXT, dslParseResult);
|
||||||
|
parseInfo.setProperties(properties);
|
||||||
|
parseInfo.setScore(FUNCTION_BONUS_THRESHOLD);
|
||||||
|
parseInfo.setQueryMode(semanticQuery.getQueryMode());
|
||||||
|
SchemaElement Model = new SchemaElement();
|
||||||
|
Model.setModel(modelId);
|
||||||
|
Model.setId(modelId);
|
||||||
|
parseInfo.setModel(Model);
|
||||||
|
queryCtx.getCandidateQueries().add(semanticQuery);
|
||||||
|
} catch (Exception e) {
|
||||||
|
log.error("LLMDSLParser error", e);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
private LLMResp requestLLM(QueryContext queryCtx, Long modelId) {
|
||||||
|
long startTime = System.currentTimeMillis();
|
||||||
|
String queryText = queryCtx.getRequest().getQueryText();
|
||||||
|
final LLMConfig llmConfig = ContextUtils.getBean(LLMConfig.class);
|
||||||
|
|
||||||
|
if (StringUtils.isEmpty(llmConfig.getUrl())) {
|
||||||
|
log.warn("llmConfig url is null, skip llm parser");
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
SemanticSchema semanticSchema = ContextUtils.getBean(SchemaService.class).getSemanticSchema();
|
||||||
|
Map<Long, String> modelIdToName = semanticSchema.getModelIdToName();
|
||||||
|
|
||||||
|
LLMReq llmReq = new LLMReq();
|
||||||
|
llmReq.setQueryText(queryText);
|
||||||
|
LLMReq.LLMSchema llmSchema = new LLMReq.LLMSchema();
|
||||||
|
llmSchema.setModelName(modelIdToName.get(modelId));
|
||||||
|
llmSchema.setDomainName(modelIdToName.get(modelId));
|
||||||
|
List<String> fieldNameList = getFieldNameList(queryCtx, modelId, semanticSchema);
|
||||||
|
fieldNameList.add(DSLBuilder.DATA_Field);
|
||||||
|
llmSchema.setFieldNameList(fieldNameList);
|
||||||
|
llmReq.setSchema(llmSchema);
|
||||||
|
List<ElementValue> linking = new ArrayList<>();
|
||||||
|
linking.addAll(getValueList(queryCtx, modelId, semanticSchema));
|
||||||
|
llmReq.setLinking(linking);
|
||||||
|
String currentDate = getCurrentDate(modelId);
|
||||||
|
llmReq.setCurrentDate(currentDate);
|
||||||
|
|
||||||
|
log.info("requestLLM request, modelId:{},llmReq:{}", modelId, llmReq);
|
||||||
|
String questUrl = llmConfig.getUrl() + llmConfig.getQueryToSqlPath();
|
||||||
|
|
||||||
|
RestTemplate restTemplate = ContextUtils.getBean(RestTemplate.class);
|
||||||
|
|
||||||
|
try {
|
||||||
|
HttpHeaders headers = new HttpHeaders();
|
||||||
|
headers.setContentType(MediaType.APPLICATION_JSON);
|
||||||
|
HttpEntity<String> entity = new HttpEntity<>(JsonUtil.toString(llmReq), headers);
|
||||||
|
ResponseEntity<LLMResp> responseEntity = restTemplate.exchange(questUrl, HttpMethod.POST, entity,
|
||||||
|
LLMResp.class);
|
||||||
|
|
||||||
|
log.info("requestLLM response,cost:{}, questUrl:{} \n entity:{} \n body:{}",
|
||||||
|
System.currentTimeMillis() - startTime, questUrl, entity, responseEntity.getBody());
|
||||||
|
return responseEntity.getBody();
|
||||||
|
} catch (Exception e) {
|
||||||
|
log.error("requestLLM error", e);
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
private String getCurrentDate(Long modelId) {
|
||||||
|
return DateUtils.getBeforeDate(4);
|
||||||
|
// ChatConfigFilter filter = new ChatConfigFilter();
|
||||||
|
// filter.setModelId(modelId);
|
||||||
|
//
|
||||||
|
// List<ChatConfigResp> configResps = ContextUtils.getBean(ConfigService.class).search(filter, null);
|
||||||
|
// if (CollectionUtils.isEmpty(configResps)) {
|
||||||
|
// return
|
||||||
|
// }
|
||||||
|
// ChatConfigResp chatConfigResp = configResps.get(0);
|
||||||
|
// chatConfigResp.getChatDetailConfig().getChatDefaultConfig().get
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
private List<ElementValue> getValueList(QueryContext queryCtx, Long modelId, SemanticSchema semanticSchema) {
|
||||||
|
Map<Long, String> itemIdToName = getItemIdToName(modelId, semanticSchema);
|
||||||
|
|
||||||
|
List<SchemaElementMatch> matchedElements = queryCtx.getMapInfo().getMatchedElements(modelId);
|
||||||
|
if (CollectionUtils.isEmpty(matchedElements)) {
|
||||||
|
return new ArrayList<>();
|
||||||
|
}
|
||||||
|
Set<ElementValue> valueMatches = matchedElements.stream()
|
||||||
|
.filter(schemaElementMatch -> {
|
||||||
|
SchemaElementType type = schemaElementMatch.getElement().getType();
|
||||||
|
return SchemaElementType.VALUE.equals(type) || SchemaElementType.ID.equals(type);
|
||||||
|
})
|
||||||
|
.map(elementMatch ->
|
||||||
|
{
|
||||||
|
ElementValue elementValue = new ElementValue();
|
||||||
|
elementValue.setFieldName(itemIdToName.get(elementMatch.getElement().getId()));
|
||||||
|
elementValue.setFieldValue(elementMatch.getWord());
|
||||||
|
return elementValue;
|
||||||
|
}
|
||||||
|
)
|
||||||
|
.collect(Collectors.toSet());
|
||||||
|
return new ArrayList<>(valueMatches);
|
||||||
|
}
|
||||||
|
|
||||||
|
private List<String> getFieldNameList(QueryContext queryCtx, Long modelId, SemanticSchema semanticSchema) {
|
||||||
|
Map<Long, String> itemIdToName = getItemIdToName(modelId, semanticSchema);
|
||||||
|
|
||||||
|
List<SchemaElementMatch> matchedElements = queryCtx.getMapInfo().getMatchedElements(modelId);
|
||||||
|
if (CollectionUtils.isEmpty(matchedElements)) {
|
||||||
|
return new ArrayList<>();
|
||||||
|
}
|
||||||
|
Set<String> fieldNameList = matchedElements.stream()
|
||||||
|
.filter(schemaElementMatch -> {
|
||||||
|
SchemaElementType elementType = schemaElementMatch.getElement().getType();
|
||||||
|
return SchemaElementType.METRIC.equals(elementType) ||
|
||||||
|
SchemaElementType.DIMENSION.equals(elementType) ||
|
||||||
|
SchemaElementType.VALUE.equals(elementType);
|
||||||
|
})
|
||||||
|
.map(schemaElementMatch -> {
|
||||||
|
SchemaElementType elementType = schemaElementMatch.getElement().getType();
|
||||||
|
|
||||||
|
if (!SchemaElementType.VALUE.equals(elementType)) {
|
||||||
|
return schemaElementMatch.getWord();
|
||||||
|
}
|
||||||
|
return itemIdToName.get(schemaElementMatch.getElement().getId());
|
||||||
|
})
|
||||||
|
.filter(name -> StringUtils.isNotEmpty(name) && !name.contains("%"))
|
||||||
|
.collect(Collectors.toSet());
|
||||||
|
return new ArrayList<>(fieldNameList);
|
||||||
|
}
|
||||||
|
|
||||||
|
private Map<Long, String> getItemIdToName(Long modelId, SemanticSchema semanticSchema) {
|
||||||
|
return semanticSchema.getDimensions().stream()
|
||||||
|
.filter(entry -> modelId.equals(entry.getModel()))
|
||||||
|
.collect(Collectors.toMap(SchemaElement::getId, SchemaElement::getName, (value1, value2) -> value2));
|
||||||
|
}
|
||||||
|
|
||||||
|
}
|
||||||
@@ -17,7 +17,7 @@ public class LLMTimeEnhancementParse implements SemanticParser {
|
|||||||
|
|
||||||
@Override
|
@Override
|
||||||
public void parse(QueryContext queryContext, ChatContext chatContext) {
|
public void parse(QueryContext queryContext, ChatContext chatContext) {
|
||||||
log.info("before queryContext:{},chatContext:{}",queryContext,chatContext);
|
log.info("before queryContext:{},chatContext:{}", queryContext, chatContext);
|
||||||
ChatGptHelper chatGptHelper = ContextUtils.getBean(ChatGptHelper.class);
|
ChatGptHelper chatGptHelper = ContextUtils.getBean(ChatGptHelper.class);
|
||||||
String inferredTime = chatGptHelper.inferredTime(queryContext.getRequest().getQueryText());
|
String inferredTime = chatGptHelper.inferredTime(queryContext.getRequest().getQueryText());
|
||||||
try {
|
try {
|
||||||
@@ -25,12 +25,12 @@ public class LLMTimeEnhancementParse implements SemanticParser {
|
|||||||
for (SemanticQuery query : queryContext.getCandidateQueries()) {
|
for (SemanticQuery query : queryContext.getCandidateQueries()) {
|
||||||
DateConf dateInfo = query.getParseInfo().getDateInfo();
|
DateConf dateInfo = query.getParseInfo().getDateInfo();
|
||||||
JSONObject jsonObject = JSON.parseObject(inferredTime);
|
JSONObject jsonObject = JSON.parseObject(inferredTime);
|
||||||
if (jsonObject.containsKey("date")){
|
if (jsonObject.containsKey("date")) {
|
||||||
dateInfo.setDateMode(DateConf.DateMode.BETWEEN);
|
dateInfo.setDateMode(DateConf.DateMode.BETWEEN);
|
||||||
dateInfo.setStartDate(jsonObject.getString("date"));
|
dateInfo.setStartDate(jsonObject.getString("date"));
|
||||||
dateInfo.setEndDate(jsonObject.getString("date"));
|
dateInfo.setEndDate(jsonObject.getString("date"));
|
||||||
query.getParseInfo().setDateInfo(dateInfo);
|
query.getParseInfo().setDateInfo(dateInfo);
|
||||||
}else if (jsonObject.containsKey("start")){
|
} else if (jsonObject.containsKey("start")) {
|
||||||
dateInfo.setDateMode(DateConf.DateMode.BETWEEN);
|
dateInfo.setDateMode(DateConf.DateMode.BETWEEN);
|
||||||
dateInfo.setStartDate(jsonObject.getString("start"));
|
dateInfo.setStartDate(jsonObject.getString("start"));
|
||||||
dateInfo.setEndDate(jsonObject.getString("end"));
|
dateInfo.setEndDate(jsonObject.getString("end"));
|
||||||
@@ -38,11 +38,12 @@ public class LLMTimeEnhancementParse implements SemanticParser {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}catch (Exception exception){
|
} catch (Exception exception) {
|
||||||
log.error("{} parse error,this reason is:{}",LLMTimeEnhancementParse.class.getSimpleName(), (Object) exception.getStackTrace());
|
log.error("{} parse error,this reason is:{}", LLMTimeEnhancementParse.class.getSimpleName(),
|
||||||
|
(Object) exception.getStackTrace());
|
||||||
}
|
}
|
||||||
|
|
||||||
log.info("after queryContext:{},chatContext:{}",queryContext,chatContext);
|
log.info("after queryContext:{},chatContext:{}", queryContext, chatContext);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -21,7 +21,9 @@ import java.util.regex.Matcher;
|
|||||||
import java.util.regex.Pattern;
|
import java.util.regex.Pattern;
|
||||||
import java.util.stream.Collectors;
|
import java.util.stream.Collectors;
|
||||||
import java.util.stream.Stream;
|
import java.util.stream.Stream;
|
||||||
|
import lombok.AllArgsConstructor;
|
||||||
import lombok.extern.slf4j.Slf4j;
|
import lombok.extern.slf4j.Slf4j;
|
||||||
|
import org.apache.commons.lang3.StringUtils;
|
||||||
|
|
||||||
@Slf4j
|
@Slf4j
|
||||||
public class AggregateTypeParser implements SemanticParser {
|
public class AggregateTypeParser implements SemanticParser {
|
||||||
@@ -35,36 +37,60 @@ public class AggregateTypeParser implements SemanticParser {
|
|||||||
new AbstractMap.SimpleEntry<>(DISTINCT, Pattern.compile("(?i)(uv)")),
|
new AbstractMap.SimpleEntry<>(DISTINCT, Pattern.compile("(?i)(uv)")),
|
||||||
new AbstractMap.SimpleEntry<>(COUNT, Pattern.compile("(?i)(总数|pv)")),
|
new AbstractMap.SimpleEntry<>(COUNT, Pattern.compile("(?i)(总数|pv)")),
|
||||||
new AbstractMap.SimpleEntry<>(NONE, Pattern.compile("(?i)(明细)"))
|
new AbstractMap.SimpleEntry<>(NONE, Pattern.compile("(?i)(明细)"))
|
||||||
).collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue,(k1,k2)->k2));
|
).collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue, (k1, k2) -> k2));
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public void parse(QueryContext queryContext, ChatContext chatContext) {
|
public void parse(QueryContext queryContext, ChatContext chatContext) {
|
||||||
|
String queryText = queryContext.getRequest().getQueryText();
|
||||||
|
AggregateConf aggregateConf = resolveAggregateConf(queryText);
|
||||||
|
|
||||||
for (SemanticQuery semanticQuery : queryContext.getCandidateQueries()) {
|
for (SemanticQuery semanticQuery : queryContext.getCandidateQueries()) {
|
||||||
if (!AggregateTypeEnum.NONE.equals(semanticQuery.getParseInfo().getAggType())) {
|
if (!AggregateTypeEnum.NONE.equals(semanticQuery.getParseInfo().getAggType())) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
semanticQuery.getParseInfo().setAggType(aggregateConf.type);
|
||||||
String queryText = queryContext.getRequest().getQueryText();
|
int detectWordLength = 0;
|
||||||
semanticQuery.getParseInfo().setAggType(resolveAggregateType(queryText));
|
if (StringUtils.isNotEmpty(aggregateConf.detectWord)) {
|
||||||
|
detectWordLength = aggregateConf.detectWord.length();
|
||||||
|
}
|
||||||
|
semanticQuery.getParseInfo().setScore(semanticQuery.getParseInfo().getScore() + detectWordLength);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
public static AggregateTypeEnum resolveAggregateType(String queryText) {
|
public AggregateTypeEnum resolveAggregateType(String queryText) {
|
||||||
|
AggregateConf aggregateConf = resolveAggregateConf(queryText);
|
||||||
|
return aggregateConf.type;
|
||||||
|
}
|
||||||
|
|
||||||
|
private AggregateConf resolveAggregateConf(String queryText) {
|
||||||
Map<AggregateTypeEnum, Integer> aggregateCount = new HashMap<>(REGX_MAP.size());
|
Map<AggregateTypeEnum, Integer> aggregateCount = new HashMap<>(REGX_MAP.size());
|
||||||
|
Map<AggregateTypeEnum, String> aggregateWord = new HashMap<>(REGX_MAP.size());
|
||||||
|
|
||||||
for (Map.Entry<AggregateTypeEnum, Pattern> entry : REGX_MAP.entrySet()) {
|
for (Map.Entry<AggregateTypeEnum, Pattern> entry : REGX_MAP.entrySet()) {
|
||||||
Matcher matcher = entry.getValue().matcher(queryText);
|
Matcher matcher = entry.getValue().matcher(queryText);
|
||||||
int count = 0;
|
int count = 0;
|
||||||
|
String detectWord = null;
|
||||||
while (matcher.find()) {
|
while (matcher.find()) {
|
||||||
count++;
|
count++;
|
||||||
|
detectWord = matcher.group();
|
||||||
}
|
}
|
||||||
if (count > 0) {
|
if (count > 0) {
|
||||||
aggregateCount.put(entry.getKey(), count);
|
aggregateCount.put(entry.getKey(), count);
|
||||||
|
aggregateWord.put(entry.getKey(), detectWord);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
return aggregateCount.entrySet().stream().max(Map.Entry.comparingByValue())
|
AggregateTypeEnum type = aggregateCount.entrySet().stream().max(Map.Entry.comparingByValue())
|
||||||
.map(entry -> entry.getKey()).orElse(NONE);
|
.map(entry -> entry.getKey()).orElse(NONE);
|
||||||
|
String detectWord = aggregateWord.get(type);
|
||||||
|
return new AggregateConf(type, detectWord);
|
||||||
|
}
|
||||||
|
|
||||||
|
@AllArgsConstructor
|
||||||
|
class AggregateConf {
|
||||||
|
|
||||||
|
public AggregateTypeEnum type;
|
||||||
|
public String detectWord;
|
||||||
}
|
}
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,5 +1,12 @@
|
|||||||
package com.tencent.supersonic.chat.parser.rule;
|
package com.tencent.supersonic.chat.parser.rule;
|
||||||
|
|
||||||
|
import static com.tencent.supersonic.chat.api.pojo.SchemaElementType.DIMENSION;
|
||||||
|
import static com.tencent.supersonic.chat.api.pojo.SchemaElementType.ENTITY;
|
||||||
|
import static com.tencent.supersonic.chat.api.pojo.SchemaElementType.ID;
|
||||||
|
import static com.tencent.supersonic.chat.api.pojo.SchemaElementType.METRIC;
|
||||||
|
import static com.tencent.supersonic.chat.api.pojo.SchemaElementType.MODEL;
|
||||||
|
import static com.tencent.supersonic.chat.api.pojo.SchemaElementType.VALUE;
|
||||||
|
|
||||||
import com.tencent.supersonic.chat.api.component.SemanticParser;
|
import com.tencent.supersonic.chat.api.component.SemanticParser;
|
||||||
import com.tencent.supersonic.chat.api.component.SemanticQuery;
|
import com.tencent.supersonic.chat.api.component.SemanticQuery;
|
||||||
import com.tencent.supersonic.chat.api.pojo.ChatContext;
|
import com.tencent.supersonic.chat.api.pojo.ChatContext;
|
||||||
@@ -8,8 +15,8 @@ import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
|
|||||||
import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
|
import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
|
||||||
import com.tencent.supersonic.chat.query.QueryManager;
|
import com.tencent.supersonic.chat.query.QueryManager;
|
||||||
import com.tencent.supersonic.chat.query.rule.RuleSemanticQuery;
|
import com.tencent.supersonic.chat.query.rule.RuleSemanticQuery;
|
||||||
import com.tencent.supersonic.chat.query.rule.metric.MetricDomainQuery;
|
|
||||||
import com.tencent.supersonic.chat.query.rule.metric.MetricEntityQuery;
|
import com.tencent.supersonic.chat.query.rule.metric.MetricEntityQuery;
|
||||||
|
import com.tencent.supersonic.chat.query.rule.metric.MetricModelQuery;
|
||||||
import com.tencent.supersonic.chat.query.rule.metric.MetricSemanticQuery;
|
import com.tencent.supersonic.chat.query.rule.metric.MetricSemanticQuery;
|
||||||
import java.util.AbstractMap;
|
import java.util.AbstractMap;
|
||||||
import java.util.ArrayList;
|
import java.util.ArrayList;
|
||||||
@@ -21,8 +28,6 @@ import java.util.stream.Collectors;
|
|||||||
import java.util.stream.Stream;
|
import java.util.stream.Stream;
|
||||||
import lombok.extern.slf4j.Slf4j;
|
import lombok.extern.slf4j.Slf4j;
|
||||||
|
|
||||||
import static com.tencent.supersonic.chat.api.pojo.SchemaElementType.*;
|
|
||||||
|
|
||||||
@Slf4j
|
@Slf4j
|
||||||
public class ContextInheritParser implements SemanticParser {
|
public class ContextInheritParser implements SemanticParser {
|
||||||
|
|
||||||
@@ -31,7 +36,7 @@ public class ContextInheritParser implements SemanticParser {
|
|||||||
new AbstractMap.SimpleEntry<>(DIMENSION, Arrays.asList(DIMENSION, VALUE)),
|
new AbstractMap.SimpleEntry<>(DIMENSION, Arrays.asList(DIMENSION, VALUE)),
|
||||||
new AbstractMap.SimpleEntry<>(VALUE, Arrays.asList(VALUE, DIMENSION)),
|
new AbstractMap.SimpleEntry<>(VALUE, Arrays.asList(VALUE, DIMENSION)),
|
||||||
new AbstractMap.SimpleEntry<>(ENTITY, Arrays.asList(ENTITY)),
|
new AbstractMap.SimpleEntry<>(ENTITY, Arrays.asList(ENTITY)),
|
||||||
new AbstractMap.SimpleEntry<>(DOMAIN, Arrays.asList(DOMAIN)),
|
new AbstractMap.SimpleEntry<>(MODEL, Arrays.asList(MODEL)),
|
||||||
new AbstractMap.SimpleEntry<>(ID, Arrays.asList(ID))
|
new AbstractMap.SimpleEntry<>(ID, Arrays.asList(ID))
|
||||||
).collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
|
).collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
|
||||||
|
|
||||||
@@ -41,9 +46,9 @@ public class ContextInheritParser implements SemanticParser {
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
Long domainId = chatContext.getParseInfo().getDomainId();
|
Long modelId = chatContext.getParseInfo().getModelId();
|
||||||
List<SchemaElementMatch> elementMatches = queryContext.getMapInfo()
|
List<SchemaElementMatch> elementMatches = queryContext.getMapInfo()
|
||||||
.getMatchedElements(domainId);
|
.getMatchedElements(modelId);
|
||||||
|
|
||||||
List<SchemaElementMatch> matchesToInherit = new ArrayList<>();
|
List<SchemaElementMatch> matchesToInherit = new ArrayList<>();
|
||||||
for (SchemaElementMatch match : chatContext.getParseInfo().getElementMatches()) {
|
for (SchemaElementMatch match : chatContext.getParseInfo().getElementMatches()) {
|
||||||
@@ -51,7 +56,7 @@ public class ContextInheritParser implements SemanticParser {
|
|||||||
// mutual exclusive element types should not be inherited
|
// mutual exclusive element types should not be inherited
|
||||||
RuleSemanticQuery ruleQuery = QueryManager.getRuleQuery(chatContext.getParseInfo().getQueryMode());
|
RuleSemanticQuery ruleQuery = QueryManager.getRuleQuery(chatContext.getParseInfo().getQueryMode());
|
||||||
if (!containsTypes(elementMatches, matchType, ruleQuery)) {
|
if (!containsTypes(elementMatches, matchType, ruleQuery)) {
|
||||||
match.setMode(SchemaElementMatch.MatchMode.INHERIT);
|
match.setInherited(true);
|
||||||
matchesToInherit.add(match);
|
matchesToInherit.add(match);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -59,11 +64,24 @@ public class ContextInheritParser implements SemanticParser {
|
|||||||
|
|
||||||
List<RuleSemanticQuery> queries = RuleSemanticQuery.resolve(elementMatches, queryContext);
|
List<RuleSemanticQuery> queries = RuleSemanticQuery.resolve(elementMatches, queryContext);
|
||||||
for (RuleSemanticQuery query : queries) {
|
for (RuleSemanticQuery query : queries) {
|
||||||
query.fillParseInfo(domainId, chatContext);
|
query.fillParseInfo(modelId, queryContext, chatContext);
|
||||||
|
if (existSameQuery(query.getParseInfo().getModelId(), query.getQueryMode(), queryContext)) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
queryContext.getCandidateQueries().add(query);
|
queryContext.getCandidateQueries().add(query);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private boolean existSameQuery(Long modelId, String queryMode, QueryContext queryContext) {
|
||||||
|
for (SemanticQuery semanticQuery : queryContext.getCandidateQueries()) {
|
||||||
|
if (semanticQuery.getQueryMode().equals(queryMode)
|
||||||
|
&& semanticQuery.getParseInfo().getModelId().equals(modelId)) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
private boolean containsTypes(List<SchemaElementMatch> matches, SchemaElementType matchType,
|
private boolean containsTypes(List<SchemaElementMatch> matches, SchemaElementType matchType,
|
||||||
RuleSemanticQuery ruleQuery) {
|
RuleSemanticQuery ruleQuery) {
|
||||||
List<SchemaElementType> types = MUTUAL_EXCLUSIVE_MAP.get(matchType);
|
List<SchemaElementType> types = MUTUAL_EXCLUSIVE_MAP.get(matchType);
|
||||||
@@ -79,16 +97,18 @@ public class ContextInheritParser implements SemanticParser {
|
|||||||
}
|
}
|
||||||
|
|
||||||
protected boolean shouldInherit(QueryContext queryContext, ChatContext chatContext) {
|
protected boolean shouldInherit(QueryContext queryContext, ChatContext chatContext) {
|
||||||
Long contextDomainId = chatContext.getParseInfo().getDomainId();
|
Long contextmodelId = chatContext.getParseInfo().getModelId();
|
||||||
if (queryContext.getMapInfo().getMatchedElements(contextDomainId) == null) {
|
// if map info doesn't contain the same Model of the context,
|
||||||
|
// no inheritance could be done
|
||||||
|
if (queryContext.getMapInfo().getMatchedElements(contextmodelId) == null) {
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
// if candidates have only one MetricDomain mode and context has value filter , count in context
|
// if candidates only have MetricModel mode, count in context
|
||||||
List<SemanticQuery> candidateQueries = queryContext.getCandidateQueries().stream()
|
List<SemanticQuery> metricModelQueries = queryContext.getCandidateQueries().stream()
|
||||||
.filter(semanticQuery -> semanticQuery.getParseInfo().getDomainId().equals(contextDomainId)).collect(
|
.filter(query -> query instanceof MetricModelQuery).collect(
|
||||||
Collectors.toList());
|
Collectors.toList());
|
||||||
if (candidateQueries.size() == 1 && (candidateQueries.get(0) instanceof MetricDomainQuery)) {
|
if (metricModelQueries.size() == queryContext.getCandidateQueries().size()) {
|
||||||
return true;
|
return true;
|
||||||
} else {
|
} else {
|
||||||
return queryContext.getCandidateQueries().size() == 0;
|
return queryContext.getCandidateQueries().size() == 0;
|
||||||
|
|||||||
@@ -1,11 +1,12 @@
|
|||||||
package com.tencent.supersonic.chat.parser.rule;
|
package com.tencent.supersonic.chat.parser.rule;
|
||||||
|
|
||||||
import com.tencent.supersonic.chat.api.component.SemanticParser;
|
import com.tencent.supersonic.chat.api.component.SemanticParser;
|
||||||
import com.tencent.supersonic.chat.api.pojo.*;
|
import com.tencent.supersonic.chat.api.pojo.ChatContext;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.QueryContext;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.SchemaMapInfo;
|
||||||
import com.tencent.supersonic.chat.query.rule.RuleSemanticQuery;
|
import com.tencent.supersonic.chat.query.rule.RuleSemanticQuery;
|
||||||
|
import java.util.List;
|
||||||
import java.util.*;
|
|
||||||
|
|
||||||
import lombok.extern.slf4j.Slf4j;
|
import lombok.extern.slf4j.Slf4j;
|
||||||
|
|
||||||
@Slf4j
|
@Slf4j
|
||||||
@@ -16,12 +17,12 @@ public class QueryModeParser implements SemanticParser {
|
|||||||
SchemaMapInfo mapInfo = queryContext.getMapInfo();
|
SchemaMapInfo mapInfo = queryContext.getMapInfo();
|
||||||
|
|
||||||
// iterate all schemaElementMatches to resolve semantic query
|
// iterate all schemaElementMatches to resolve semantic query
|
||||||
for (Long domainId : mapInfo.getMatchedDomains()) {
|
for (Long modelId : mapInfo.getMatchedModels()) {
|
||||||
List<SchemaElementMatch> elementMatches = mapInfo.getMatchedElements(domainId);
|
List<SchemaElementMatch> elementMatches = mapInfo.getMatchedElements(modelId);
|
||||||
List<RuleSemanticQuery> queries = RuleSemanticQuery.resolve(elementMatches, queryContext);
|
List<RuleSemanticQuery> queries = RuleSemanticQuery.resolve(elementMatches, queryContext);
|
||||||
|
|
||||||
for (RuleSemanticQuery query : queries) {
|
for (RuleSemanticQuery query : queries) {
|
||||||
query.fillParseInfo(domainId, chatContext);
|
query.fillParseInfo(modelId, queryContext, chatContext);
|
||||||
queryContext.getCandidateQueries().add(query);
|
queryContext.getCandidateQueries().add(query);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -4,21 +4,21 @@ import com.tencent.supersonic.chat.api.component.SemanticParser;
|
|||||||
import com.tencent.supersonic.chat.api.component.SemanticQuery;
|
import com.tencent.supersonic.chat.api.component.SemanticQuery;
|
||||||
import com.tencent.supersonic.chat.api.pojo.ChatContext;
|
import com.tencent.supersonic.chat.api.pojo.ChatContext;
|
||||||
import com.tencent.supersonic.chat.api.pojo.QueryContext;
|
import com.tencent.supersonic.chat.api.pojo.QueryContext;
|
||||||
import com.tencent.supersonic.chat.query.rule.RuleSemanticQuery;
|
|
||||||
import com.tencent.supersonic.chat.query.QueryManager;
|
import com.tencent.supersonic.chat.query.QueryManager;
|
||||||
|
import com.tencent.supersonic.chat.query.rule.RuleSemanticQuery;
|
||||||
import com.tencent.supersonic.common.pojo.Constants;
|
import com.tencent.supersonic.common.pojo.Constants;
|
||||||
import com.tencent.supersonic.common.pojo.DateConf;
|
import com.tencent.supersonic.common.pojo.DateConf;
|
||||||
|
import com.xkzhangsan.time.nlp.TimeNLP;
|
||||||
|
import com.xkzhangsan.time.nlp.TimeNLPUtil;
|
||||||
import java.text.DateFormat;
|
import java.text.DateFormat;
|
||||||
import java.text.ParseException;
|
import java.text.ParseException;
|
||||||
import java.text.SimpleDateFormat;
|
import java.text.SimpleDateFormat;
|
||||||
import java.time.LocalDate;
|
import java.time.LocalDate;
|
||||||
import java.util.*;
|
import java.util.Date;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Stack;
|
||||||
import java.util.regex.Matcher;
|
import java.util.regex.Matcher;
|
||||||
import java.util.regex.Pattern;
|
import java.util.regex.Pattern;
|
||||||
|
|
||||||
import com.xkzhangsan.time.nlp.TimeNLP;
|
|
||||||
import com.xkzhangsan.time.nlp.TimeNLPUtil;
|
|
||||||
import lombok.extern.slf4j.Slf4j;
|
import lombok.extern.slf4j.Slf4j;
|
||||||
import org.apache.logging.log4j.util.Strings;
|
import org.apache.logging.log4j.util.Strings;
|
||||||
|
|
||||||
@@ -45,12 +45,16 @@ public class TimeRangeParser implements SemanticParser {
|
|||||||
if (queryContext.getCandidateQueries().size() > 0) {
|
if (queryContext.getCandidateQueries().size() > 0) {
|
||||||
for (SemanticQuery query : queryContext.getCandidateQueries()) {
|
for (SemanticQuery query : queryContext.getCandidateQueries()) {
|
||||||
query.getParseInfo().setDateInfo(dateConf);
|
query.getParseInfo().setDateInfo(dateConf);
|
||||||
|
query.getParseInfo().setScore(query.getParseInfo().getScore()
|
||||||
|
+ dateConf.getDetectWord().length());
|
||||||
}
|
}
|
||||||
} else if (QueryManager.containsRuleQuery(chatContext.getParseInfo().getQueryMode())) {
|
} else if (QueryManager.containsRuleQuery(chatContext.getParseInfo().getQueryMode())) {
|
||||||
RuleSemanticQuery semanticQuery = QueryManager.createRuleQuery(
|
RuleSemanticQuery semanticQuery = QueryManager.createRuleQuery(
|
||||||
chatContext.getParseInfo().getQueryMode());
|
chatContext.getParseInfo().getQueryMode());
|
||||||
// inherit parse info from context
|
// inherit parse info from context
|
||||||
chatContext.getParseInfo().setDateInfo(dateConf);
|
chatContext.getParseInfo().setDateInfo(dateConf);
|
||||||
|
chatContext.getParseInfo().setScore(chatContext.getParseInfo().getScore()
|
||||||
|
+ dateConf.getDetectWord().length());
|
||||||
semanticQuery.setParseInfo(chatContext.getParseInfo());
|
semanticQuery.setParseInfo(chatContext.getParseInfo());
|
||||||
queryContext.getCandidateQueries().add(semanticQuery);
|
queryContext.getCandidateQueries().add(semanticQuery);
|
||||||
}
|
}
|
||||||
@@ -60,41 +64,48 @@ public class TimeRangeParser implements SemanticParser {
|
|||||||
private DateConf parseDateCN(String queryText) {
|
private DateConf parseDateCN(String queryText) {
|
||||||
Date startDate = null;
|
Date startDate = null;
|
||||||
Date endDate;
|
Date endDate;
|
||||||
|
String detectWord = null;
|
||||||
|
|
||||||
List<TimeNLP> times = TimeNLPUtil.parse(queryText);
|
List<TimeNLP> times = TimeNLPUtil.parse(queryText);
|
||||||
if (times.size() > 0) {
|
if (times.size() > 0) {
|
||||||
startDate = times.get(0).getTime();
|
startDate = times.get(0).getTime();
|
||||||
|
detectWord = times.get(0).getTimeExpression();
|
||||||
} else {
|
} else {
|
||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (times.size() > 1) {
|
if (times.size() > 1) {
|
||||||
endDate = times.get(1).getTime();
|
endDate = times.get(1).getTime();
|
||||||
|
detectWord += "~" + times.get(0).getTimeExpression();
|
||||||
} else {
|
} else {
|
||||||
endDate = startDate;
|
endDate = startDate;
|
||||||
}
|
}
|
||||||
|
|
||||||
return getDateConf(startDate, endDate);
|
return getDateConf(startDate, endDate, detectWord);
|
||||||
}
|
}
|
||||||
|
|
||||||
private DateConf parseDateNumber(String queryText) {
|
private DateConf parseDateNumber(String queryText) {
|
||||||
String startDate;
|
String startDate;
|
||||||
String endDate = null;
|
String endDate = null;
|
||||||
|
String detectWord = null;
|
||||||
|
|
||||||
Matcher dateMatcher = DATE_PATTERN_NUMBER.matcher(queryText);
|
Matcher dateMatcher = DATE_PATTERN_NUMBER.matcher(queryText);
|
||||||
if (dateMatcher.find()) {
|
if (dateMatcher.find()) {
|
||||||
startDate = dateMatcher.group();
|
startDate = dateMatcher.group();
|
||||||
|
detectWord = startDate;
|
||||||
} else {
|
} else {
|
||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (dateMatcher.find()) {
|
if (dateMatcher.find()) {
|
||||||
endDate = dateMatcher.group();
|
endDate = dateMatcher.group();
|
||||||
|
detectWord += "~" + endDate;
|
||||||
}
|
}
|
||||||
|
|
||||||
endDate = endDate != null ? endDate : startDate;
|
endDate = endDate != null ? endDate : startDate;
|
||||||
|
|
||||||
try {
|
try {
|
||||||
return getDateConf(DATE_FORMAT_NUMBER.parse(startDate), DATE_FORMAT_NUMBER.parse(endDate));
|
return getDateConf(DATE_FORMAT_NUMBER.parse(startDate), DATE_FORMAT_NUMBER.parse(endDate), detectWord);
|
||||||
} catch (ParseException e) {
|
} catch (ParseException e) {
|
||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
@@ -134,11 +145,11 @@ public class TimeRangeParser implements SemanticParser {
|
|||||||
}
|
}
|
||||||
days = days * num;
|
days = days * num;
|
||||||
info.setDateMode(DateConf.DateMode.RECENT);
|
info.setDateMode(DateConf.DateMode.RECENT);
|
||||||
String text = "近" + num + zhPeriod;
|
String detectWord = "近" + num + zhPeriod;
|
||||||
if (Strings.isNotEmpty(m.group("periodStr"))) {
|
if (Strings.isNotEmpty(m.group("periodStr"))) {
|
||||||
text = m.group("periodStr");
|
detectWord = m.group("periodStr");
|
||||||
}
|
}
|
||||||
info.setText(text);
|
info.setDetectWord(detectWord);
|
||||||
info.setStartDate(LocalDate.now().minusDays(days).toString());
|
info.setStartDate(LocalDate.now().minusDays(days).toString());
|
||||||
info.setUnit(num);
|
info.setUnit(num);
|
||||||
|
|
||||||
@@ -173,7 +184,7 @@ public class TimeRangeParser implements SemanticParser {
|
|||||||
return stack.stream().mapToInt(s -> s).sum();
|
return stack.stream().mapToInt(s -> s).sum();
|
||||||
}
|
}
|
||||||
|
|
||||||
private DateConf getDateConf(Date startDate, Date endDate) {
|
private DateConf getDateConf(Date startDate, Date endDate, String detectWord) {
|
||||||
if (startDate == null || endDate == null) {
|
if (startDate == null || endDate == null) {
|
||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
@@ -182,6 +193,7 @@ public class TimeRangeParser implements SemanticParser {
|
|||||||
info.setDateMode(DateConf.DateMode.BETWEEN);
|
info.setDateMode(DateConf.DateMode.BETWEEN);
|
||||||
info.setStartDate(DATE_FORMAT.format(startDate));
|
info.setStartDate(DATE_FORMAT.format(startDate));
|
||||||
info.setEndDate(DATE_FORMAT.format(endDate));
|
info.setEndDate(DATE_FORMAT.format(endDate));
|
||||||
|
info.setDetectWord(detectWord);
|
||||||
return info;
|
return info;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -1,7 +1,6 @@
|
|||||||
package com.tencent.supersonic.chat.persistence.dataobject;
|
package com.tencent.supersonic.chat.persistence.dataobject;
|
||||||
|
|
||||||
import java.util.Date;
|
import java.util.Date;
|
||||||
|
|
||||||
import lombok.Data;
|
import lombok.Data;
|
||||||
import lombok.ToString;
|
import lombok.ToString;
|
||||||
|
|
||||||
@@ -15,7 +14,7 @@ public class ChatConfigDO {
|
|||||||
*/
|
*/
|
||||||
private Long id;
|
private Long id;
|
||||||
|
|
||||||
private Long domainId;
|
private Long modelId;
|
||||||
|
|
||||||
private String chatDetailConfig;
|
private String chatDetailConfig;
|
||||||
|
|
||||||
|
|||||||
@@ -1,11 +1,10 @@
|
|||||||
package com.tencent.supersonic.chat.persistence.dataobject;
|
package com.tencent.supersonic.chat.persistence.dataobject;
|
||||||
|
|
||||||
|
|
||||||
import java.util.ArrayList;
|
|
||||||
import java.util.List;
|
|
||||||
|
|
||||||
import com.tencent.supersonic.chat.config.DefaultMetric;
|
import com.tencent.supersonic.chat.config.DefaultMetric;
|
||||||
import com.tencent.supersonic.chat.config.Dim4Dict;
|
import com.tencent.supersonic.chat.config.Dim4Dict;
|
||||||
|
import java.util.ArrayList;
|
||||||
|
import java.util.List;
|
||||||
import lombok.Data;
|
import lombok.Data;
|
||||||
import lombok.ToString;
|
import lombok.ToString;
|
||||||
|
|
||||||
@@ -14,14 +13,14 @@ import lombok.ToString;
|
|||||||
@ToString
|
@ToString
|
||||||
public class DimValueDO {
|
public class DimValueDO {
|
||||||
|
|
||||||
private Long domainId;
|
private Long modelId;
|
||||||
|
|
||||||
private List<DefaultMetric> defaultMetricDescList = new ArrayList<>();
|
private List<DefaultMetric> defaultMetricDescList = new ArrayList<>();
|
||||||
|
|
||||||
private List<Dim4Dict> dimensions = new ArrayList<>();
|
private List<Dim4Dict> dimensions = new ArrayList<>();
|
||||||
|
|
||||||
public DimValueDO setDomainId(Long domainId) {
|
public DimValueDO setModelId(Long modelId) {
|
||||||
this.domainId = domainId;
|
this.modelId = modelId;
|
||||||
return this;
|
return this;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -3,6 +3,7 @@ package com.tencent.supersonic.chat.persistence.dataobject;
|
|||||||
import java.util.Date;
|
import java.util.Date;
|
||||||
|
|
||||||
public class PluginDO {
|
public class PluginDO {
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
*
|
||||||
*/
|
*/
|
||||||
@@ -16,7 +17,7 @@ public class PluginDO {
|
|||||||
/**
|
/**
|
||||||
*
|
*
|
||||||
*/
|
*/
|
||||||
private String domain;
|
private String model;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
*
|
||||||
@@ -69,7 +70,6 @@ public class PluginDO {
|
|||||||
private String comment;
|
private String comment;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @return id
|
* @return id
|
||||||
*/
|
*/
|
||||||
public Long getId() {
|
public Long getId() {
|
||||||
@@ -77,7 +77,6 @@ public class PluginDO {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @param id
|
* @param id
|
||||||
*/
|
*/
|
||||||
public void setId(Long id) {
|
public void setId(Long id) {
|
||||||
@@ -86,6 +85,7 @@ public class PluginDO {
|
|||||||
|
|
||||||
/**
|
/**
|
||||||
* DASHBOARD,WIDGET,URL
|
* DASHBOARD,WIDGET,URL
|
||||||
|
*
|
||||||
* @return type DASHBOARD,WIDGET,URL
|
* @return type DASHBOARD,WIDGET,URL
|
||||||
*/
|
*/
|
||||||
public String getType() {
|
public String getType() {
|
||||||
@@ -94,6 +94,7 @@ public class PluginDO {
|
|||||||
|
|
||||||
/**
|
/**
|
||||||
* DASHBOARD,WIDGET,URL
|
* DASHBOARD,WIDGET,URL
|
||||||
|
*
|
||||||
* @param type DASHBOARD,WIDGET,URL
|
* @param type DASHBOARD,WIDGET,URL
|
||||||
*/
|
*/
|
||||||
public void setType(String type) {
|
public void setType(String type) {
|
||||||
@@ -101,23 +102,20 @@ public class PluginDO {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
* @return model
|
||||||
* @return domain
|
|
||||||
*/
|
*/
|
||||||
public String getDomain() {
|
public String getModel() {
|
||||||
return domain;
|
return model;
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
* @param model
|
||||||
* @param domain
|
|
||||||
*/
|
*/
|
||||||
public void setDomain(String domain) {
|
public void setModel(String model) {
|
||||||
this.domain = domain == null ? null : domain.trim();
|
this.model = model == null ? null : model.trim();
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @return pattern
|
* @return pattern
|
||||||
*/
|
*/
|
||||||
public String getPattern() {
|
public String getPattern() {
|
||||||
@@ -125,7 +123,6 @@ public class PluginDO {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @param pattern
|
* @param pattern
|
||||||
*/
|
*/
|
||||||
public void setPattern(String pattern) {
|
public void setPattern(String pattern) {
|
||||||
@@ -133,7 +130,6 @@ public class PluginDO {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @return parse_mode
|
* @return parse_mode
|
||||||
*/
|
*/
|
||||||
public String getParseMode() {
|
public String getParseMode() {
|
||||||
@@ -141,7 +137,6 @@ public class PluginDO {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @param parseMode
|
* @param parseMode
|
||||||
*/
|
*/
|
||||||
public void setParseMode(String parseMode) {
|
public void setParseMode(String parseMode) {
|
||||||
@@ -149,7 +144,6 @@ public class PluginDO {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @return name
|
* @return name
|
||||||
*/
|
*/
|
||||||
public String getName() {
|
public String getName() {
|
||||||
@@ -157,7 +151,6 @@ public class PluginDO {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @param name
|
* @param name
|
||||||
*/
|
*/
|
||||||
public void setName(String name) {
|
public void setName(String name) {
|
||||||
@@ -165,7 +158,6 @@ public class PluginDO {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @return created_at
|
* @return created_at
|
||||||
*/
|
*/
|
||||||
public Date getCreatedAt() {
|
public Date getCreatedAt() {
|
||||||
@@ -173,7 +165,6 @@ public class PluginDO {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @param createdAt
|
* @param createdAt
|
||||||
*/
|
*/
|
||||||
public void setCreatedAt(Date createdAt) {
|
public void setCreatedAt(Date createdAt) {
|
||||||
@@ -181,7 +172,6 @@ public class PluginDO {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @return created_by
|
* @return created_by
|
||||||
*/
|
*/
|
||||||
public String getCreatedBy() {
|
public String getCreatedBy() {
|
||||||
@@ -189,7 +179,6 @@ public class PluginDO {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @param createdBy
|
* @param createdBy
|
||||||
*/
|
*/
|
||||||
public void setCreatedBy(String createdBy) {
|
public void setCreatedBy(String createdBy) {
|
||||||
@@ -197,7 +186,6 @@ public class PluginDO {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @return updated_at
|
* @return updated_at
|
||||||
*/
|
*/
|
||||||
public Date getUpdatedAt() {
|
public Date getUpdatedAt() {
|
||||||
@@ -205,7 +193,6 @@ public class PluginDO {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @param updatedAt
|
* @param updatedAt
|
||||||
*/
|
*/
|
||||||
public void setUpdatedAt(Date updatedAt) {
|
public void setUpdatedAt(Date updatedAt) {
|
||||||
@@ -213,7 +200,6 @@ public class PluginDO {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @return updated_by
|
* @return updated_by
|
||||||
*/
|
*/
|
||||||
public String getUpdatedBy() {
|
public String getUpdatedBy() {
|
||||||
@@ -221,7 +207,6 @@ public class PluginDO {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @param updatedBy
|
* @param updatedBy
|
||||||
*/
|
*/
|
||||||
public void setUpdatedBy(String updatedBy) {
|
public void setUpdatedBy(String updatedBy) {
|
||||||
@@ -229,7 +214,6 @@ public class PluginDO {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @return parse_mode_config
|
* @return parse_mode_config
|
||||||
*/
|
*/
|
||||||
public String getParseModeConfig() {
|
public String getParseModeConfig() {
|
||||||
@@ -237,7 +221,6 @@ public class PluginDO {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @param parseModeConfig
|
* @param parseModeConfig
|
||||||
*/
|
*/
|
||||||
public void setParseModeConfig(String parseModeConfig) {
|
public void setParseModeConfig(String parseModeConfig) {
|
||||||
@@ -245,7 +228,6 @@ public class PluginDO {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @return config
|
* @return config
|
||||||
*/
|
*/
|
||||||
public String getConfig() {
|
public String getConfig() {
|
||||||
@@ -253,7 +235,6 @@ public class PluginDO {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @param config
|
* @param config
|
||||||
*/
|
*/
|
||||||
public void setConfig(String config) {
|
public void setConfig(String config) {
|
||||||
@@ -261,7 +242,6 @@ public class PluginDO {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @return comment
|
* @return comment
|
||||||
*/
|
*/
|
||||||
public String getComment() {
|
public String getComment() {
|
||||||
@@ -269,7 +249,6 @@ public class PluginDO {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @param comment
|
* @param comment
|
||||||
*/
|
*/
|
||||||
public void setComment(String comment) {
|
public void setComment(String comment) {
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import java.util.Date;
|
|||||||
import java.util.List;
|
import java.util.List;
|
||||||
|
|
||||||
public class PluginDOExample {
|
public class PluginDOExample {
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* s2_plugin
|
* s2_plugin
|
||||||
*/
|
*/
|
||||||
@@ -31,7 +32,6 @@ public class PluginDOExample {
|
|||||||
protected Integer limitEnd;
|
protected Integer limitEnd;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @mbg.generated
|
* @mbg.generated
|
||||||
*/
|
*/
|
||||||
public PluginDOExample() {
|
public PluginDOExample() {
|
||||||
@@ -39,7 +39,6 @@ public class PluginDOExample {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @mbg.generated
|
* @mbg.generated
|
||||||
*/
|
*/
|
||||||
public void setOrderByClause(String orderByClause) {
|
public void setOrderByClause(String orderByClause) {
|
||||||
@@ -47,7 +46,6 @@ public class PluginDOExample {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @mbg.generated
|
* @mbg.generated
|
||||||
*/
|
*/
|
||||||
public String getOrderByClause() {
|
public String getOrderByClause() {
|
||||||
@@ -55,7 +53,6 @@ public class PluginDOExample {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @mbg.generated
|
* @mbg.generated
|
||||||
*/
|
*/
|
||||||
public void setDistinct(boolean distinct) {
|
public void setDistinct(boolean distinct) {
|
||||||
@@ -63,7 +60,6 @@ public class PluginDOExample {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @mbg.generated
|
* @mbg.generated
|
||||||
*/
|
*/
|
||||||
public boolean isDistinct() {
|
public boolean isDistinct() {
|
||||||
@@ -71,7 +67,6 @@ public class PluginDOExample {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @mbg.generated
|
* @mbg.generated
|
||||||
*/
|
*/
|
||||||
public List<Criteria> getOredCriteria() {
|
public List<Criteria> getOredCriteria() {
|
||||||
@@ -79,7 +74,6 @@ public class PluginDOExample {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @mbg.generated
|
* @mbg.generated
|
||||||
*/
|
*/
|
||||||
public void or(Criteria criteria) {
|
public void or(Criteria criteria) {
|
||||||
@@ -87,7 +81,6 @@ public class PluginDOExample {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @mbg.generated
|
* @mbg.generated
|
||||||
*/
|
*/
|
||||||
public Criteria or() {
|
public Criteria or() {
|
||||||
@@ -97,7 +90,6 @@ public class PluginDOExample {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @mbg.generated
|
* @mbg.generated
|
||||||
*/
|
*/
|
||||||
public Criteria createCriteria() {
|
public Criteria createCriteria() {
|
||||||
@@ -109,7 +101,6 @@ public class PluginDOExample {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @mbg.generated
|
* @mbg.generated
|
||||||
*/
|
*/
|
||||||
protected Criteria createCriteriaInternal() {
|
protected Criteria createCriteriaInternal() {
|
||||||
@@ -118,7 +109,6 @@ public class PluginDOExample {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @mbg.generated
|
* @mbg.generated
|
||||||
*/
|
*/
|
||||||
public void clear() {
|
public void clear() {
|
||||||
@@ -128,15 +118,13 @@ public class PluginDOExample {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @mbg.generated
|
* @mbg.generated
|
||||||
*/
|
*/
|
||||||
public void setLimitStart(Integer limitStart) {
|
public void setLimitStart(Integer limitStart) {
|
||||||
this.limitStart=limitStart;
|
this.limitStart = limitStart;
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @mbg.generated
|
* @mbg.generated
|
||||||
*/
|
*/
|
||||||
public Integer getLimitStart() {
|
public Integer getLimitStart() {
|
||||||
@@ -144,15 +132,13 @@ public class PluginDOExample {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @mbg.generated
|
* @mbg.generated
|
||||||
*/
|
*/
|
||||||
public void setLimitEnd(Integer limitEnd) {
|
public void setLimitEnd(Integer limitEnd) {
|
||||||
this.limitEnd=limitEnd;
|
this.limitEnd = limitEnd;
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
*
|
|
||||||
* @mbg.generated
|
* @mbg.generated
|
||||||
*/
|
*/
|
||||||
public Integer getLimitEnd() {
|
public Integer getLimitEnd() {
|
||||||
@@ -163,6 +149,7 @@ public class PluginDOExample {
|
|||||||
* s2_plugin null
|
* s2_plugin null
|
||||||
*/
|
*/
|
||||||
protected abstract static class GeneratedCriteria {
|
protected abstract static class GeneratedCriteria {
|
||||||
|
|
||||||
protected List<Criterion> criteria;
|
protected List<Criterion> criteria;
|
||||||
|
|
||||||
protected GeneratedCriteria() {
|
protected GeneratedCriteria() {
|
||||||
@@ -333,73 +320,73 @@ public class PluginDOExample {
|
|||||||
return (Criteria) this;
|
return (Criteria) this;
|
||||||
}
|
}
|
||||||
|
|
||||||
public Criteria andDomainIsNull() {
|
public Criteria andModelIsNull() {
|
||||||
addCriterion("domain is null");
|
addCriterion("model is null");
|
||||||
return (Criteria) this;
|
return (Criteria) this;
|
||||||
}
|
}
|
||||||
|
|
||||||
public Criteria andDomainIsNotNull() {
|
public Criteria andModelIsNotNull() {
|
||||||
addCriterion("domain is not null");
|
addCriterion("model is not null");
|
||||||
return (Criteria) this;
|
return (Criteria) this;
|
||||||
}
|
}
|
||||||
|
|
||||||
public Criteria andDomainEqualTo(String value) {
|
public Criteria andModelEqualTo(String value) {
|
||||||
addCriterion("domain =", value, "domain");
|
addCriterion("model =", value, "model");
|
||||||
return (Criteria) this;
|
return (Criteria) this;
|
||||||
}
|
}
|
||||||
|
|
||||||
public Criteria andDomainNotEqualTo(String value) {
|
public Criteria andModelNotEqualTo(String value) {
|
||||||
addCriterion("domain <>", value, "domain");
|
addCriterion("model <>", value, "model");
|
||||||
return (Criteria) this;
|
return (Criteria) this;
|
||||||
}
|
}
|
||||||
|
|
||||||
public Criteria andDomainGreaterThan(String value) {
|
public Criteria andModelGreaterThan(String value) {
|
||||||
addCriterion("domain >", value, "domain");
|
addCriterion("model >", value, "model");
|
||||||
return (Criteria) this;
|
return (Criteria) this;
|
||||||
}
|
}
|
||||||
|
|
||||||
public Criteria andDomainGreaterThanOrEqualTo(String value) {
|
public Criteria andModelGreaterThanOrEqualTo(String value) {
|
||||||
addCriterion("domain >=", value, "domain");
|
addCriterion("model >=", value, "model");
|
||||||
return (Criteria) this;
|
return (Criteria) this;
|
||||||
}
|
}
|
||||||
|
|
||||||
public Criteria andDomainLessThan(String value) {
|
public Criteria andModelLessThan(String value) {
|
||||||
addCriterion("domain <", value, "domain");
|
addCriterion("model <", value, "model");
|
||||||
return (Criteria) this;
|
return (Criteria) this;
|
||||||
}
|
}
|
||||||
|
|
||||||
public Criteria andDomainLessThanOrEqualTo(String value) {
|
public Criteria andModelLessThanOrEqualTo(String value) {
|
||||||
addCriterion("domain <=", value, "domain");
|
addCriterion("model <=", value, "model");
|
||||||
return (Criteria) this;
|
return (Criteria) this;
|
||||||
}
|
}
|
||||||
|
|
||||||
public Criteria andDomainLike(String value) {
|
public Criteria andModelLike(String value) {
|
||||||
addCriterion("domain like", value, "domain");
|
addCriterion("model like", value, "model");
|
||||||
return (Criteria) this;
|
return (Criteria) this;
|
||||||
}
|
}
|
||||||
|
|
||||||
public Criteria andDomainNotLike(String value) {
|
public Criteria andModelNotLike(String value) {
|
||||||
addCriterion("domain not like", value, "domain");
|
addCriterion("model not like", value, "model");
|
||||||
return (Criteria) this;
|
return (Criteria) this;
|
||||||
}
|
}
|
||||||
|
|
||||||
public Criteria andDomainIn(List<String> values) {
|
public Criteria andModelIn(List<String> values) {
|
||||||
addCriterion("domain in", values, "domain");
|
addCriterion("model in", values, "model");
|
||||||
return (Criteria) this;
|
return (Criteria) this;
|
||||||
}
|
}
|
||||||
|
|
||||||
public Criteria andDomainNotIn(List<String> values) {
|
public Criteria andModelNotIn(List<String> values) {
|
||||||
addCriterion("domain not in", values, "domain");
|
addCriterion("model not in", values, "model");
|
||||||
return (Criteria) this;
|
return (Criteria) this;
|
||||||
}
|
}
|
||||||
|
|
||||||
public Criteria andDomainBetween(String value1, String value2) {
|
public Criteria andModelBetween(String value1, String value2) {
|
||||||
addCriterion("domain between", value1, value2, "domain");
|
addCriterion("model between", value1, value2, "model");
|
||||||
return (Criteria) this;
|
return (Criteria) this;
|
||||||
}
|
}
|
||||||
|
|
||||||
public Criteria andDomainNotBetween(String value1, String value2) {
|
public Criteria andModelNotBetween(String value1, String value2) {
|
||||||
addCriterion("domain not between", value1, value2, "domain");
|
addCriterion("model not between", value1, value2, "model");
|
||||||
return (Criteria) this;
|
return (Criteria) this;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -888,6 +875,7 @@ public class PluginDOExample {
|
|||||||
* s2_plugin null
|
* s2_plugin null
|
||||||
*/
|
*/
|
||||||
public static class Criterion {
|
public static class Criterion {
|
||||||
|
|
||||||
private String condition;
|
private String condition;
|
||||||
|
|
||||||
private Object value;
|
private Object value;
|
||||||
|
|||||||
@@ -14,5 +14,5 @@ public interface ChatConfigMapper {
|
|||||||
|
|
||||||
List<ChatConfigDO> search(ChatConfigFilterInternal filterInternal);
|
List<ChatConfigDO> search(ChatConfigFilterInternal filterInternal);
|
||||||
|
|
||||||
ChatConfigDO fetchConfigByDomainId(Long domainId);
|
ChatConfigDO fetchConfigByModelId(Long modelId);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -2,9 +2,8 @@ package com.tencent.supersonic.chat.persistence.mapper;
|
|||||||
|
|
||||||
import com.tencent.supersonic.chat.persistence.dataobject.ChatQueryDO;
|
import com.tencent.supersonic.chat.persistence.dataobject.ChatQueryDO;
|
||||||
import com.tencent.supersonic.chat.persistence.dataobject.ChatQueryDOExample;
|
import com.tencent.supersonic.chat.persistence.dataobject.ChatQueryDOExample;
|
||||||
import org.apache.ibatis.annotations.Mapper;
|
|
||||||
|
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
|
import org.apache.ibatis.annotations.Mapper;
|
||||||
|
|
||||||
@Mapper
|
@Mapper
|
||||||
public interface ChatQueryDOMapper {
|
public interface ChatQueryDOMapper {
|
||||||
|
|||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user