[improvement][project] supersonic 0.7.2 version backend update (#28)

Co-authored-by: jipengli <jipengli@tencent.com>
This commit is contained in:
jipeli
2023-08-15 08:56:18 +08:00
committed by GitHub
parent 27283001a8
commit b1952d64ab
461 changed files with 18548 additions and 11939 deletions

View File

@@ -3,7 +3,24 @@
- All notable changes to this project will be documented in this file. - All notable changes to this project will be documented in this file.
- "Breaking Changes" describes any changes that may break existing functionality or cause - "Breaking Changes" describes any changes that may break existing functionality or cause
compatibility issues with previous versions. compatibility issues with previous versions.
## SuperSonic [0.7.2] - 2023-08-12
### Added
- Support asynchronous query - return parse information to user before executing result
- Add Model as the basic data structure of the semantic definitions - this will repalce the old conception of subdomain
### Updated
- improve knowledge word similarity algorithm
- improve embedding plugin chooser
- improve DSLQuery field correction and parser
### Fixed
- Fix mapper error that detectWord text is shorter than word
- Fix MetricDomainQuery inherit context
## SuperSonic [0.7.0] - 2023-07-30 ## SuperSonic [0.7.0] - 2023-07-30
@@ -96,4 +113,4 @@
### Removed ### Removed
- delete view module - delete view module

View File

@@ -2,7 +2,7 @@ English | [中文](README_CN.md)
# SuperSonic (超音数) # SuperSonic (超音数)
**SuperSonic is an out-of-the-box yet highly extensible framework for building a data chatbot**. SuperSonic provides a chat interface that empowers users to query data using natural language and visualize the results with suitable charts. To enable such experience, the only thing necessary is to build logical semantic models (definition of metrics/dimensions/entities, along with their meaning, context and relationships) on top of physical data models, and no data modification or copying is required. Meanwhile, SuperSonic is designed to be pluggable, allowing new tools to be added through plugins. **SuperSonic is an out-of-the-box yet highly extensible framework for building a data chatbot**. SuperSonic provides a chat interface that empowers users to query data using natural language and visualize the results with suitable charts. To enable such experience, the only thing necessary is to build logical semantic models (definition of metrics/dimensions/entities, along with their meaning, context and relationships) on top of physical data models, and no data modification or copying is required. Meanwhile, SuperSonic is designed to be pluggable, allowing new functionalities to be added through plugins and core components to be integrated with other systems.
<img src="./docs/images/supersonic_demo.gif" height="100%" width="100%" align="center"/> <img src="./docs/images/supersonic_demo.gif" height="100%" width="100%" align="center"/>
@@ -10,34 +10,35 @@ English | [中文](README_CN.md)
The emergence of Large Language Model (LLM) like ChatGPT is reshaping the way information is retrieved. In the field of data analytics, both academia and industry are primarily focused on leveraging LLM to convert natural language queries into SQL queries. While some works show promising results, they are still not applicable to real-world scenarios. The emergence of Large Language Model (LLM) like ChatGPT is reshaping the way information is retrieved. In the field of data analytics, both academia and industry are primarily focused on leveraging LLM to convert natural language queries into SQL queries. While some works show promising results, they are still not applicable to real-world scenarios.
From our perspective, the key to filling the real-world gap lies in three aspects: From our perspective, the key to filling the real-world gap lies in two aspects:
1. Utilize a combination of rule-based and LLM-based semantic parsers to deal with different scenarios. 1. Utilize a combination of rule-based and model-based semantic parsers to deal with different scenarios.
2. Introduce a semantic model layer encapsulating the underlying data complexity(joins, formulas, etc) to simplify semantic parsing. 2. Introduce a semantic model layer encapsulating the underlying data complexity(joins, formulas, etc) to simplify semantic parsing.
3. Integrate third-party plugins to augment semantic parsing capabilities or complement custom functionalities.
With these ideas in mind, we develop SuperSonic as a practical reference implementation and use it to power our real-world products. Additionally, to facilitate further development of data chatbot, we decide to open source SuperSonic as an extensible framework. With these ideas in mind, we develop SuperSonic as a practical reference implementation and use it to power our real-world products. Additionally, to facilitate further development of data chatbot, we decide to open source SuperSonic as an extensible framework.
## Out-of-the-box Features ## Out-of-the-box Features
- Built-in chat UI for business users to enter natural language queries and answer results with appropriate visualization charts. - Built-in graphical interface for business users to enter data queries
- Built-in modelling UI for analytics engineers to manage semantic models. The configurations related to access permission and chat conversation can also be set on the UI. - Built-in graphical interface for analytics engineers to manage semantic models
- Support input auto-completion as well as query recommendation. - Support input auto-completion as well as query recommendation
- Support multi-turn conversation and history context management. - Support multi-turn conversation and history context management
- Support four-level permission control: domain-level, model-level, column-level and row-level. - Support three-level permission control: domain-level, column-level and row-level
## Extensible Components ## Extensible Components
The high-level architecture and main process flow is shown in below diagram: The high-level architecture and main process flow is shown in below diagram:
<img src="./docs/images/supersonic_components.png" height="70%" width="70%" align="center"/> <img src="./docs/images/supersonic_components.png" height="80%" width="80%" align="center"/>
- **Chat Interface:** accepts natural language queries and answer results with appropriate visualization charts. It supports input auto-completion as well as multi-turn conversation.
- **Modeling Interface:** empowers analytics engineers to visually define and maintain semantic models. The configurations related to access permission and chat conversation can also be set on the UI.
- **Schema Mapper Chain:** identifies references to schema elements(metrics/dimensions/entities/values) in user queries. It matches the query text against a knowledge base constructed from the semantic models. - **Schema Mapper Chain:** identifies references to schema elements(metrics/dimensions/entities/values) in user queries. It matches the query text against a knowledge base constructed from the semantic models.
- **Semantic Parser Chain:** understands user queries and extract semantic information. It consists of a combination of rule-based and LLM-based parsers, each of which deals with specific scenarios. - **Semantic Parser Chain:** understands user queries and extract semantic information. It consists of a combination of rule-based and model-based parsers, each of which deals with specific scenarios.
- **Semantic Layer:** performs execution according to extracted semantic information. It generates SQL queries and executes them against physical data models. - **Semantic Query:** performs execution according to extracted semantic information. It generates SQL queries and executes them against physical data models.
- **Chat Plugins**: perform custom execution given the results of schema mapping and semantic parsing. It would optionally resort to semantic layer to query semantic models.
## Quick Demo ## Quick Demo
@@ -57,4 +58,4 @@ Pull the source code and run script "assembly/bin/build-standalone.sh" to build
### Build for Distributed Mode ### Build for Distributed Mode
Pull the source code and run scripts "assembly/bin/build-chat.sh" and "assembly/bin/build-semantic.sh" separately to build packages. Pull the source code and run scripts "assembly/bin/build-chat.sh" and "assembly/bin/build-semantic.sh" separately to build packages.

View File

@@ -8,42 +8,43 @@
大型语言模型LLMs如ChatGPT的出现正在重塑信息检索的方式。在数据分析领域学术界和工业界主要关注利用深度学习模型将自然语言查询转换为SQL查询。虽然一些工作显示出有前景的结果但它们还并不适用于实际场景。 大型语言模型LLMs如ChatGPT的出现正在重塑信息检索的方式。在数据分析领域学术界和工业界主要关注利用深度学习模型将自然语言查询转换为SQL查询。虽然一些工作显示出有前景的结果但它们还并不适用于实际场景。
在我们看来,为了在实际场景发挥价值,有个关键点: 在我们看来,为了在实际场景发挥价值,有个关键点:
1. 将基于规则和基于模型的语义解析器相结合,发挥各自优势,以便处理不同的场景。 1. 将基于规则和基于模型的语义解析器相结合,发挥各自优势,以便处理不同的场景。
2. 引入语义模型层来封装数据底层的复杂性(关联、公式等),从而简化语义解析的求解空间。 2. 引入语义模型层来封装数据底层的复杂性(关联、公式等),从而简化语义解析的求解空间。
3. 整合第三方插件用于增强语义解析能力,或者扩充自定义功能。
为了验证上述想法,我们开发了超音数项目,并将其应用在实际的内部产品中。与此同时,我们将超音数作为一个可扩展的框架开源,希望能够促进数据问答对话领域的进一步发展。 为了验证上述想法,我们开发了超音数项目,并将其应用在实际的内部产品中。与此同时,我们将超音数作为一个可扩展的框架开源,希望能够促进数据问答对话领域的进一步发展。
## 开箱即用的特性 ## 开箱即用的特性
- 内置的对话图形界面,使用户能够通过自然语言问询,并最终选择合适的可视化图表呈现结果 - 内置图形界面以便业务用户输入数据查询
- 内置的建模图形界面,使分析工程师能够通过可视化方式定义和维护语义模型,与访问权限和聊天对话相关的配置也可以在用户界面上设置 - 内置图形界面以便分析工程师管理语义模型
- 支持文本输入的联想和查询问题的推荐。 - 支持文本输入的联想和查询问题的推荐。
- 支持多轮对话,根据语境自动切换上下文。 - 支持多轮对话,根据语境自动切换上下文。
- 支持级权限控制:主题域级、模型级、列级行级。 - 支持级权限控制:主题域级、列级行级。
## 易于扩展的组件 ## 易于扩展的组件
超音数的整体架构和主流程如下图所示: 超音数的整体架构和主流程如下图所示:
<img src="./docs/images/supersonic_components.png" height="70%" width="70%" align="center"/> <img src="./docs/images/supersonic_components.png" height="80%" width="80%" align="center"/>
- **问答对话界面(chat interface)**:接受用户查询并选择合适的可视化图表呈现结果,支持输入联想和多轮对话。
- **语义建模界面(modeling interface)**:使分析工程师能够通过可视化方式定义和维护语义模型,与访问权限和聊天对话相关的配置也可以在用户界面上设置。
- **模式映射器(schema mapper chain)**:基于语义模型构建知识库,然后将自然语言文本在知识库中进行匹配,为后续的语义解析提供相关信息。 - **模式映射器(schema mapper chain)**:基于语义模型构建知识库,然后将自然语言文本在知识库中进行匹配,为后续的语义解析提供相关信息。
- **语义解析器(semantic parser chain)**:理解用户查询并抽取语义信息,其由一组基于规则和基于模型的解析器组成,每个解析器可应对不同的特定场景。 - **语义解析器(semantic parser chain)**:理解用户查询并抽取语义信息,其由一组基于规则和基于模型的解析器组成,每个解析器可应对不同的特定场景。
- **语义(semantic layer)**: 根据语义信息生成物理SQL执行查询。 - **语义查询(semantic query)**: 根据语义信息生成物理SQL执行查询。
- **问答插件(chat plugins)**:基于模式映射和语义解析的结果,执行自定义的操作,同时可以选择利用语义层来查询语义模型。
## 快速体验 ## 快速体验
超音数自带样例的语义模型和问答对话,只需以下三步即可快速体验: 超音数自带样例的语义模型和问答对话,只需以下三步即可快速体验:
- 从[release page](https://github.com/tencentmusic/supersonic/releases)下载预先构建好的发行包 - 从[release page](https://github.com/tencentmusic/supersonic/releases)下载预先构建好的发行包
- 运行 "bin/start-standalone.sh"启动服务 - 运行 "bin/start-standalone.sh"启动服务
- 在浏览器访问http://localhost:9080 开启探索 - 在浏览器访问http://localhost:9080 开启探索
## 如何构建 ## 如何构建
@@ -51,8 +52,8 @@
### Standalone模式构建 ### Standalone模式构建
下载源码包,运行脚本"assembly/bin/build-standalone.sh",将所有服务一起编译打包 下载源码包,运行脚本"assembly/bin/build-standalone.sh",将所有服务一起编译打包
### Distributed模式构建 ### Distributed模式构建
下载源码包,分别运行脚本"assembly/bin/build-chat.sh"、"assembly/bin/build-semantic.sh",为问答层服务和语义层服务编译打包 下载源码包,分别运行脚本"assembly/bin/build-chat.sh"、"assembly/bin/build-semantic.sh",为问答层服务和语义层服务编译打包

View File

@@ -3,7 +3,6 @@ package com.tencent.supersonic.auth.api.authentication.adaptor;
import com.tencent.supersonic.auth.api.authentication.pojo.Organization; import com.tencent.supersonic.auth.api.authentication.pojo.Organization;
import com.tencent.supersonic.auth.api.authentication.pojo.User; import com.tencent.supersonic.auth.api.authentication.pojo.User;
import com.tencent.supersonic.auth.api.authentication.request.UserReq; import com.tencent.supersonic.auth.api.authentication.request.UserReq;
import java.util.List; import java.util.List;
import java.util.Set; import java.util.Set;

View File

@@ -1,9 +1,8 @@
package com.tencent.supersonic.auth.api.authentication.pojo; package com.tencent.supersonic.auth.api.authentication.pojo;
import com.google.common.collect.Lists; import com.google.common.collect.Lists;
import lombok.Data;
import java.util.List; import java.util.List;
import lombok.Data;
@Data @Data
public class Organization { public class Organization {

View File

@@ -3,7 +3,6 @@ package com.tencent.supersonic.auth.api.authentication.service;
import com.tencent.supersonic.auth.api.authentication.pojo.Organization; import com.tencent.supersonic.auth.api.authentication.pojo.Organization;
import com.tencent.supersonic.auth.api.authentication.pojo.User; import com.tencent.supersonic.auth.api.authentication.pojo.User;
import com.tencent.supersonic.auth.api.authentication.request.UserReq; import com.tencent.supersonic.auth.api.authentication.request.UserReq;
import java.util.List; import java.util.List;
import java.util.Set; import java.util.Set;

View File

@@ -1,7 +1,6 @@
package com.tencent.supersonic.auth.api.authentication.service; package com.tencent.supersonic.auth.api.authentication.service;
import com.tencent.supersonic.auth.api.authentication.pojo.User; import com.tencent.supersonic.auth.api.authentication.pojo.User;
import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse; import javax.servlet.http.HttpServletResponse;

View File

@@ -6,7 +6,7 @@ import lombok.Data;
@Data @Data
public class AuthGroup { public class AuthGroup {
private String domainId; private String modelId;
private String name; private String name;
private Integer groupId; private Integer groupId;
private List<AuthRule> authRules; private List<AuthRule> authRules;

View File

@@ -7,14 +7,14 @@ import lombok.ToString;
@ToString @ToString
public class AuthRes { public class AuthRes {
private String domainId; private String modelId;
private String name; private String name;
public AuthRes() { public AuthRes() {
} }
public AuthRes(String domainId, String name) { public AuthRes(String modelId, String name) {
this.domainId = domainId; this.modelId = modelId;
this.name = name; this.name = name;
} }
} }

View File

@@ -3,7 +3,6 @@ package com.tencent.supersonic.auth.api.authorization.request;
import com.tencent.supersonic.auth.api.authorization.pojo.AuthRes; import com.tencent.supersonic.auth.api.authorization.pojo.AuthRes;
import java.util.ArrayList; import java.util.ArrayList;
import java.util.List; import java.util.List;
import lombok.Data; import lombok.Data;
import lombok.ToString; import lombok.ToString;
@@ -17,5 +16,5 @@ public class QueryAuthResReq {
private List<AuthRes> resources; private List<AuthRes> resources;
private String domainId; private String modelId;
} }

View File

@@ -4,7 +4,6 @@ import com.tencent.supersonic.auth.api.authorization.pojo.AuthResGrp;
import com.tencent.supersonic.auth.api.authorization.pojo.DimensionFilter; import com.tencent.supersonic.auth.api.authorization.pojo.DimensionFilter;
import java.util.ArrayList; import java.util.ArrayList;
import java.util.List; import java.util.List;
import lombok.Data; import lombok.Data;
@Data @Data

View File

@@ -3,8 +3,8 @@ package com.tencent.supersonic.auth.api.authorization.service;
import com.tencent.supersonic.auth.api.authorization.pojo.AuthGroup; import com.tencent.supersonic.auth.api.authorization.pojo.AuthGroup;
import com.tencent.supersonic.auth.api.authorization.request.QueryAuthResReq; import com.tencent.supersonic.auth.api.authorization.request.QueryAuthResReq;
import com.tencent.supersonic.auth.api.authorization.response.AuthorizedResourceResp; import com.tencent.supersonic.auth.api.authorization.response.AuthorizedResourceResp;
import javax.servlet.http.HttpServletRequest;
import java.util.List; import java.util.List;
import javax.servlet.http.HttpServletRequest;
public interface AuthService { public interface AuthService {

View File

@@ -11,10 +11,10 @@ import com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO;
import com.tencent.supersonic.auth.authentication.persistence.repository.UserRepository; import com.tencent.supersonic.auth.authentication.persistence.repository.UserRepository;
import com.tencent.supersonic.auth.authentication.utils.UserTokenUtils; import com.tencent.supersonic.auth.authentication.utils.UserTokenUtils;
import com.tencent.supersonic.common.util.ContextUtils; import com.tencent.supersonic.common.util.ContextUtils;
import org.springframework.beans.BeanUtils;
import java.util.List; import java.util.List;
import java.util.Set; import java.util.Set;
import java.util.stream.Collectors; import java.util.stream.Collectors;
import org.springframework.beans.BeanUtils;
public class DefaultUserAdaptor implements UserAdaptor { public class DefaultUserAdaptor implements UserAdaptor {

View File

@@ -0,0 +1,21 @@
package com.tencent.supersonic.auth.authentication.config;
import lombok.Data;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Configuration;
@Data
@Configuration
public class TppConfig {
@Value(value = "${auth.app.secret:}")
private String appSecret;
@Value(value = "${auth.app.key:}")
private String appKey;
@Value(value = "${auth.oa.url:}")
private String tppOaUrl;
}

View File

@@ -35,11 +35,14 @@ public class DefaultAuthenticationInterceptor extends AuthenticationInterceptor
setFakerUser(request); setFakerUser(request);
return true; return true;
} }
HandlerMethod handlerMethod = (HandlerMethod) handler;
Method method = handlerMethod.getMethod(); if (handler instanceof HandlerMethod) {
AuthenticationIgnore ignore = method.getAnnotation(AuthenticationIgnore.class); HandlerMethod handlerMethod = (HandlerMethod) handler;
if (ignore != null) { Method method = handlerMethod.getMethod();
return true; AuthenticationIgnore ignore = method.getAnnotation(AuthenticationIgnore.class);
if (ignore != null) {
return true;
}
} }
String uri = request.getServletPath(); String uri = request.getServletPath();

View File

@@ -1,12 +1,11 @@
package com.tencent.supersonic.auth.authentication.interceptor; package com.tencent.supersonic.auth.authentication.interceptor;
import java.util.List;
import org.springframework.context.annotation.Configuration; import org.springframework.context.annotation.Configuration;
import org.springframework.core.io.support.SpringFactoriesLoader; import org.springframework.core.io.support.SpringFactoriesLoader;
import org.springframework.web.servlet.config.annotation.InterceptorRegistry; import org.springframework.web.servlet.config.annotation.InterceptorRegistry;
import org.springframework.web.servlet.config.annotation.WebMvcConfigurer; import org.springframework.web.servlet.config.annotation.WebMvcConfigurer;
import java.util.List;
@Configuration @Configuration
public class InterceptorFactory implements WebMvcConfigurer { public class InterceptorFactory implements WebMvcConfigurer {

View File

@@ -3,8 +3,8 @@ package com.tencent.supersonic.auth.authentication.persistence.repository.Impl;
import com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO; import com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO;
import com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDOExample; import com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDOExample;
import com.tencent.supersonic.auth.authentication.persistence.repository.UserRepository;
import com.tencent.supersonic.auth.authentication.persistence.mapper.UserDOMapper; import com.tencent.supersonic.auth.authentication.persistence.mapper.UserDOMapper;
import com.tencent.supersonic.auth.authentication.persistence.repository.UserRepository;
import java.util.List; import java.util.List;
import java.util.Optional; import java.util.Optional;
import org.springframework.stereotype.Component; import org.springframework.stereotype.Component;

View File

@@ -11,7 +11,12 @@ import java.util.Set;
import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse; import javax.servlet.http.HttpServletResponse;
import lombok.extern.slf4j.Slf4j; import lombok.extern.slf4j.Slf4j;
import org.springframework.web.bind.annotation.*; import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
@RestController @RestController
@RequestMapping("/api/auth/user") @RequestMapping("/api/auth/user")

View File

@@ -7,7 +7,6 @@ import com.tencent.supersonic.auth.api.authentication.service.UserService;
import com.tencent.supersonic.auth.authentication.utils.ComponentFactory; import com.tencent.supersonic.auth.authentication.utils.ComponentFactory;
import java.util.List; import java.util.List;
import java.util.Set; import java.util.Set;
import org.springframework.stereotype.Service; import org.springframework.stereotype.Service;
@Service @Service
@@ -16,7 +15,7 @@ public class UserServiceImpl implements UserService {
@Override @Override
public List<String> getUserNames() { public List<String> getUserNames() {
return ComponentFactory.getUserAdaptor().getUserNames(); return ComponentFactory.getUserAdaptor().getUserNames();
} }
@Override @Override

View File

@@ -1,8 +1,8 @@
package com.tencent.supersonic.auth.authentication.utils; package com.tencent.supersonic.auth.authentication.utils;
import com.tencent.supersonic.auth.api.authentication.adaptor.UserAdaptor; import com.tencent.supersonic.auth.api.authentication.adaptor.UserAdaptor;
import org.springframework.core.io.support.SpringFactoriesLoader;
import java.util.Objects; import java.util.Objects;
import org.springframework.core.io.support.SpringFactoriesLoader;
public class ComponentFactory { public class ComponentFactory {

View File

@@ -1,145 +1,159 @@
<?xml version="1.0" encoding="UTF-8"?> <?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd"> <!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN"
"http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="com.tencent.supersonic.auth.authentication.persistence.mapper.UserDOMapper"> <mapper namespace="com.tencent.supersonic.auth.authentication.persistence.mapper.UserDOMapper">
<resultMap id="BaseResultMap" type="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO"> <resultMap id="BaseResultMap"
<id column="id" jdbcType="BIGINT" property="id" /> type="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO">
<result column="name" jdbcType="VARCHAR" property="name" /> <id column="id" jdbcType="BIGINT" property="id"/>
<result column="password" jdbcType="VARCHAR" property="password" /> <result column="name" jdbcType="VARCHAR" property="name"/>
<result column="display_name" jdbcType="VARCHAR" property="displayName" /> <result column="password" jdbcType="VARCHAR" property="password"/>
<result column="email" jdbcType="VARCHAR" property="email" /> <result column="display_name" jdbcType="VARCHAR" property="displayName"/>
</resultMap> <result column="email" jdbcType="VARCHAR" property="email"/>
<sql id="Example_Where_Clause"> </resultMap>
<where> <sql id="Example_Where_Clause">
<foreach collection="oredCriteria" item="criteria" separator="or"> <where>
<if test="criteria.valid"> <foreach collection="oredCriteria" item="criteria" separator="or">
<trim prefix="(" prefixOverrides="and" suffix=")"> <if test="criteria.valid">
<foreach collection="criteria.criteria" item="criterion"> <trim prefix="(" prefixOverrides="and" suffix=")">
<choose> <foreach collection="criteria.criteria" item="criterion">
<when test="criterion.noValue"> <choose>
and ${criterion.condition} <when test="criterion.noValue">
</when> and ${criterion.condition}
<when test="criterion.singleValue"> </when>
and ${criterion.condition} #{criterion.value} <when test="criterion.singleValue">
</when> and ${criterion.condition} #{criterion.value}
<when test="criterion.betweenValue"> </when>
and ${criterion.condition} #{criterion.value} and #{criterion.secondValue} <when test="criterion.betweenValue">
</when> and ${criterion.condition} #{criterion.value} and
<when test="criterion.listValue"> #{criterion.secondValue}
and ${criterion.condition} </when>
<foreach close=")" collection="criterion.value" item="listItem" open="(" separator=","> <when test="criterion.listValue">
#{listItem} and ${criterion.condition}
</foreach> <foreach close=")" collection="criterion.value" item="listItem"
</when> open="(" separator=",">
</choose> #{listItem}
</foreach>
</when>
</choose>
</foreach>
</trim>
</if>
</foreach> </foreach>
</trim> </where>
</sql>
<sql id="Base_Column_List">
id
, name, password, display_name, email
</sql>
<select id="selectByExample"
parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDOExample"
resultMap="BaseResultMap">
select
<if test="distinct">
distinct
</if> </if>
</foreach> <include refid="Base_Column_List"/>
</where> from s2_user
</sql> <if test="_parameter != null">
<sql id="Base_Column_List"> <include refid="Example_Where_Clause"/>
id, name, password, display_name, email </if>
</sql> <if test="orderByClause != null">
<select id="selectByExample" parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDOExample" resultMap="BaseResultMap"> order by ${orderByClause}
select </if>
<if test="distinct"> <if test="limitStart != null and limitStart>=0">
distinct limit #{limitStart} , #{limitEnd}
</if> </if>
<include refid="Base_Column_List" /> </select>
from s2_user <select id="selectByPrimaryKey" parameterType="java.lang.Long" resultMap="BaseResultMap">
<if test="_parameter != null"> select
<include refid="Example_Where_Clause" /> <include refid="Base_Column_List"/>
</if> from s2_user
<if test="orderByClause != null"> where id = #{id,jdbcType=BIGINT}
order by ${orderByClause} </select>
</if> <delete id="deleteByPrimaryKey" parameterType="java.lang.Long">
<if test="limitStart != null and limitStart>=0"> delete
limit #{limitStart} , #{limitEnd} from s2_user
</if> where id = #{id,jdbcType=BIGINT}
</select> </delete>
<select id="selectByPrimaryKey" parameterType="java.lang.Long" resultMap="BaseResultMap"> <insert id="insert"
select parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO">
<include refid="Base_Column_List" /> insert into s2_user (id, name, password,
from s2_user display_name, email)
where id = #{id,jdbcType=BIGINT} values (#{id,jdbcType=BIGINT}, #{name,jdbcType=VARCHAR}, #{password,jdbcType=VARCHAR},
</select> #{displayName,jdbcType=VARCHAR}, #{email,jdbcType=VARCHAR})
<delete id="deleteByPrimaryKey" parameterType="java.lang.Long"> </insert>
delete from s2_user <insert id="insertSelective"
where id = #{id,jdbcType=BIGINT} parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO">
</delete> insert into s2_user
<insert id="insert" parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO"> <trim prefix="(" suffix=")" suffixOverrides=",">
insert into s2_user (id, name, password, <if test="id != null">
display_name, email) id,
values (#{id,jdbcType=BIGINT}, #{name,jdbcType=VARCHAR}, #{password,jdbcType=VARCHAR}, </if>
#{displayName,jdbcType=VARCHAR}, #{email,jdbcType=VARCHAR}) <if test="name != null">
</insert> name,
<insert id="insertSelective" parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO"> </if>
insert into s2_user <if test="password != null">
<trim prefix="(" suffix=")" suffixOverrides=","> password,
<if test="id != null"> </if>
id, <if test="displayName != null">
</if> display_name,
<if test="name != null"> </if>
name, <if test="email != null">
</if> email,
<if test="password != null"> </if>
password, </trim>
</if> <trim prefix="values (" suffix=")" suffixOverrides=",">
<if test="displayName != null"> <if test="id != null">
display_name, #{id,jdbcType=BIGINT},
</if> </if>
<if test="email != null"> <if test="name != null">
email, #{name,jdbcType=VARCHAR},
</if> </if>
</trim> <if test="password != null">
<trim prefix="values (" suffix=")" suffixOverrides=","> #{password,jdbcType=VARCHAR},
<if test="id != null"> </if>
#{id,jdbcType=BIGINT}, <if test="displayName != null">
</if> #{displayName,jdbcType=VARCHAR},
<if test="name != null"> </if>
#{name,jdbcType=VARCHAR}, <if test="email != null">
</if> #{email,jdbcType=VARCHAR},
<if test="password != null"> </if>
#{password,jdbcType=VARCHAR}, </trim>
</if> </insert>
<if test="displayName != null"> <select id="countByExample"
#{displayName,jdbcType=VARCHAR}, parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDOExample"
</if> resultType="java.lang.Long">
<if test="email != null"> select count(*) from s2_user
#{email,jdbcType=VARCHAR}, <if test="_parameter != null">
</if> <include refid="Example_Where_Clause"/>
</trim> </if>
</insert> </select>
<select id="countByExample" parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDOExample" resultType="java.lang.Long"> <update id="updateByPrimaryKeySelective"
select count(*) from s2_user parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO">
<if test="_parameter != null"> update s2_user
<include refid="Example_Where_Clause" /> <set>
</if> <if test="name != null">
</select> name = #{name,jdbcType=VARCHAR},
<update id="updateByPrimaryKeySelective" parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO"> </if>
update s2_user <if test="password != null">
<set> password = #{password,jdbcType=VARCHAR},
<if test="name != null"> </if>
name = #{name,jdbcType=VARCHAR}, <if test="displayName != null">
</if> display_name = #{displayName,jdbcType=VARCHAR},
<if test="password != null"> </if>
password = #{password,jdbcType=VARCHAR}, <if test="email != null">
</if> email = #{email,jdbcType=VARCHAR},
<if test="displayName != null"> </if>
display_name = #{displayName,jdbcType=VARCHAR}, </set>
</if> where id = #{id,jdbcType=BIGINT}
<if test="email != null"> </update>
email = #{email,jdbcType=VARCHAR}, <update id="updateByPrimaryKey"
</if> parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO">
</set> update s2_user
where id = #{id,jdbcType=BIGINT} set name = #{name,jdbcType=VARCHAR},
</update> password = #{password,jdbcType=VARCHAR},
<update id="updateByPrimaryKey" parameterType="com.tencent.supersonic.auth.authentication.persistence.dataobject.UserDO"> display_name = #{displayName,jdbcType=VARCHAR},
update s2_user email = #{email,jdbcType=VARCHAR}
set name = #{name,jdbcType=VARCHAR}, where id = #{id,jdbcType=BIGINT}
password = #{password,jdbcType=VARCHAR}, </update>
display_name = #{displayName,jdbcType=VARCHAR},
email = #{email,jdbcType=VARCHAR}
where id = #{id,jdbcType=BIGINT}
</update>
</mapper> </mapper>

View File

@@ -3,25 +3,27 @@ package com.tencent.supersonic.auth.authorization.application;
import com.google.common.base.Strings; import com.google.common.base.Strings;
import com.google.gson.Gson; import com.google.gson.Gson;
import com.tencent.supersonic.auth.api.authentication.service.UserService; import com.tencent.supersonic.auth.api.authentication.service.UserService;
import com.tencent.supersonic.auth.api.authorization.pojo.AuthGroup;
import com.tencent.supersonic.auth.api.authorization.pojo.AuthRes; import com.tencent.supersonic.auth.api.authorization.pojo.AuthRes;
import com.tencent.supersonic.auth.api.authorization.pojo.AuthResGrp; import com.tencent.supersonic.auth.api.authorization.pojo.AuthResGrp;
import com.tencent.supersonic.auth.api.authorization.pojo.AuthRule;
import com.tencent.supersonic.auth.api.authorization.pojo.DimensionFilter; import com.tencent.supersonic.auth.api.authorization.pojo.DimensionFilter;
import com.tencent.supersonic.auth.api.authorization.request.QueryAuthResReq; import com.tencent.supersonic.auth.api.authorization.request.QueryAuthResReq;
import com.tencent.supersonic.auth.api.authorization.response.AuthorizedResourceResp; import com.tencent.supersonic.auth.api.authorization.response.AuthorizedResourceResp;
import com.tencent.supersonic.auth.api.authorization.service.AuthService; import com.tencent.supersonic.auth.api.authorization.service.AuthService;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.stream.Collectors;
import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletRequest;
import com.tencent.supersonic.auth.api.authorization.pojo.AuthGroup;
import com.tencent.supersonic.auth.api.authorization.pojo.AuthRule;
import lombok.extern.slf4j.Slf4j; import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils; import org.apache.commons.lang3.StringUtils;
import org.springframework.jdbc.core.JdbcTemplate; import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Service; import org.springframework.stereotype.Service;
import org.springframework.util.CollectionUtils; import org.springframework.util.CollectionUtils;
import java.util.*;
import java.util.stream.Collectors;
@Service @Service
@Slf4j @Slf4j
public class AuthServiceImpl implements AuthService { public class AuthServiceImpl implements AuthService {
@@ -31,7 +33,7 @@ public class AuthServiceImpl implements AuthService {
private UserService userService; private UserService userService;
public AuthServiceImpl(JdbcTemplate jdbcTemplate, public AuthServiceImpl(JdbcTemplate jdbcTemplate,
UserService userService) { UserService userService) {
this.jdbcTemplate = jdbcTemplate; this.jdbcTemplate = jdbcTemplate;
this.userService = userService; this.userService = userService;
} }
@@ -43,10 +45,10 @@ public class AuthServiceImpl implements AuthService {
} }
@Override @Override
public List<AuthGroup> queryAuthGroups(String domainId, Integer groupId) { public List<AuthGroup> queryAuthGroups(String modelId, Integer groupId) {
return load().stream() return load().stream()
.filter(group -> (Objects.isNull(groupId) || groupId.equals(group.getGroupId())) .filter(group -> (Objects.isNull(groupId) || groupId.equals(group.getGroupId()))
&& domainId.equals(group.getDomainId())) && modelId.equals(group.getModelId()))
.collect(Collectors.toList()); .collect(Collectors.toList());
} }
@@ -83,16 +85,16 @@ public class AuthServiceImpl implements AuthService {
} }
List<AuthGroup> groups = getAuthGroups(req); List<AuthGroup> groups = getAuthGroups(req);
AuthorizedResourceResp resource = new AuthorizedResourceResp(); AuthorizedResourceResp resource = new AuthorizedResourceResp();
Map<String, List<AuthGroup>> authGroupsByDomainId = groups.stream() Map<String, List<AuthGroup>> authGroupsByModelId = groups.stream()
.collect(Collectors.groupingBy(AuthGroup::getDomainId)); .collect(Collectors.groupingBy(AuthGroup::getModelId));
Map<String, List<AuthRes>> reqAuthRes = req.getResources().stream() Map<String, List<AuthRes>> reqAuthRes = req.getResources().stream()
.collect(Collectors.groupingBy(AuthRes::getDomainId)); .collect(Collectors.groupingBy(AuthRes::getModelId));
for (String domainId : reqAuthRes.keySet()) { for (String modelId : reqAuthRes.keySet()) {
List<AuthRes> reqResourcesList = reqAuthRes.get(domainId); List<AuthRes> reqResourcesList = reqAuthRes.get(modelId);
AuthResGrp rg = new AuthResGrp(); AuthResGrp rg = new AuthResGrp();
if (authGroupsByDomainId.containsKey(domainId)) { if (authGroupsByModelId.containsKey(modelId)) {
List<AuthGroup> authGroups = authGroupsByDomainId.get(domainId); List<AuthGroup> authGroups = authGroupsByModelId.get(modelId);
for (AuthRes reqRes : reqResourcesList) { for (AuthRes reqRes : reqResourcesList) {
for (AuthGroup authRuleGroup : authGroups) { for (AuthGroup authRuleGroup : authGroups) {
List<AuthRule> authRules = authRuleGroup.getAuthRules(); List<AuthRule> authRules = authRuleGroup.getAuthRules();
@@ -111,8 +113,8 @@ public class AuthServiceImpl implements AuthService {
} }
} }
if (StringUtils.isNotEmpty(req.getDomainId())) { if (StringUtils.isNotEmpty(req.getModelId())) {
List<AuthGroup> authGroups = authGroupsByDomainId.get(req.getDomainId()); List<AuthGroup> authGroups = authGroupsByModelId.get(req.getModelId());
if (!CollectionUtils.isEmpty(authGroups)) { if (!CollectionUtils.isEmpty(authGroups)) {
for (AuthGroup group : authGroups) { for (AuthGroup group : authGroups) {
if (group.getDimensionFilters() != null if (group.getDimensionFilters() != null
@@ -131,7 +133,7 @@ public class AuthServiceImpl implements AuthService {
private List<AuthGroup> getAuthGroups(QueryAuthResReq req) { private List<AuthGroup> getAuthGroups(QueryAuthResReq req) {
List<AuthGroup> groups = load().stream() List<AuthGroup> groups = load().stream()
.filter(group -> { .filter(group -> {
if (!Objects.equals(group.getDomainId(), req.getDomainId())) { if (!Objects.equals(group.getModelId(), req.getModelId())) {
return false; return false;
} }
if (!CollectionUtils.isEmpty(group.getAuthorizedUsers()) && group.getAuthorizedUsers() if (!CollectionUtils.isEmpty(group.getAuthorizedUsers()) && group.getAuthorizedUsers()

View File

@@ -1,9 +1,9 @@
package com.tencent.supersonic.auth.authorization.rest; package com.tencent.supersonic.auth.authorization.rest;
import com.tencent.supersonic.auth.api.authorization.pojo.AuthGroup;
import com.tencent.supersonic.auth.api.authorization.request.QueryAuthResReq; import com.tencent.supersonic.auth.api.authorization.request.QueryAuthResReq;
import com.tencent.supersonic.auth.api.authorization.response.AuthorizedResourceResp; import com.tencent.supersonic.auth.api.authorization.response.AuthorizedResourceResp;
import com.tencent.supersonic.auth.api.authorization.service.AuthService; import com.tencent.supersonic.auth.api.authorization.service.AuthService;
import com.tencent.supersonic.auth.api.authorization.pojo.AuthGroup;
import java.util.List; import java.util.List;
import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletRequest;
import lombok.extern.slf4j.Slf4j; import lombok.extern.slf4j.Slf4j;
@@ -26,9 +26,9 @@ public class AuthController {
} }
@GetMapping("/queryGroup") @GetMapping("/queryGroup")
public List<AuthGroup> queryAuthGroup(@RequestParam("domainId") String domainId, public List<AuthGroup> queryAuthGroup(@RequestParam("modelId") String modelId,
@RequestParam(value = "groupId", required = false) Integer groupId) { @RequestParam(value = "groupId", required = false) Integer groupId) {
return authService.queryAuthGroups(domainId, groupId); return authService.queryAuthGroups(modelId, groupId);
} }
/** /**

View File

@@ -2,17 +2,18 @@ package com.tencent.supersonic.chat.api.component;
import com.github.pagehelper.PageInfo; import com.github.pagehelper.PageInfo;
import com.tencent.supersonic.auth.api.authentication.pojo.User; import com.tencent.supersonic.auth.api.authentication.pojo.User;
import com.tencent.supersonic.chat.api.pojo.DomainSchema; import com.tencent.supersonic.chat.api.pojo.ModelSchema;
import com.tencent.supersonic.common.pojo.enums.AuthType;
import com.tencent.supersonic.semantic.api.model.request.PageDimensionReq; import com.tencent.supersonic.semantic.api.model.request.PageDimensionReq;
import com.tencent.supersonic.semantic.api.model.request.PageMetricReq; import com.tencent.supersonic.semantic.api.model.request.PageMetricReq;
import com.tencent.supersonic.semantic.api.model.response.DimensionResp; import com.tencent.supersonic.semantic.api.model.response.DimensionResp;
import com.tencent.supersonic.semantic.api.model.response.DomainResp; import com.tencent.supersonic.semantic.api.model.response.DomainResp;
import com.tencent.supersonic.semantic.api.model.response.QueryResultWithSchemaResp;
import com.tencent.supersonic.semantic.api.model.response.MetricResp; import com.tencent.supersonic.semantic.api.model.response.MetricResp;
import com.tencent.supersonic.semantic.api.model.response.ModelResp;
import com.tencent.supersonic.semantic.api.model.response.QueryResultWithSchemaResp;
import com.tencent.supersonic.semantic.api.query.request.QueryDslReq; import com.tencent.supersonic.semantic.api.query.request.QueryDslReq;
import com.tencent.supersonic.semantic.api.query.request.QueryMultiStructReq; import com.tencent.supersonic.semantic.api.query.request.QueryMultiStructReq;
import com.tencent.supersonic.semantic.api.query.request.QueryStructReq; import com.tencent.supersonic.semantic.api.query.request.QueryStructReq;
import java.util.List; import java.util.List;
/** /**
@@ -30,16 +31,22 @@ import java.util.List;
public interface SemanticLayer { public interface SemanticLayer {
QueryResultWithSchemaResp queryByStruct(QueryStructReq queryStructReq, User user); QueryResultWithSchemaResp queryByStruct(QueryStructReq queryStructReq, User user);
QueryResultWithSchemaResp queryByMultiStruct(QueryMultiStructReq queryMultiStructReq, User user); QueryResultWithSchemaResp queryByMultiStruct(QueryMultiStructReq queryMultiStructReq, User user);
QueryResultWithSchemaResp queryByDsl(QueryDslReq queryDslReq, User user); QueryResultWithSchemaResp queryByDsl(QueryDslReq queryDslReq, User user);
List<DomainSchema> getDomainSchema(); List<ModelSchema> getModelSchema();
List<DomainSchema> getDomainSchema(List<Long> ids);
DomainSchema getDomainSchema(Long domain, Boolean cacheEnable); List<ModelSchema> getModelSchema(List<Long> ids);
ModelSchema getModelSchema(Long model, Boolean cacheEnable);
PageInfo<DimensionResp> getDimensionPage(PageDimensionReq pageDimensionCmd); PageInfo<DimensionResp> getDimensionPage(PageDimensionReq pageDimensionCmd);
PageInfo<MetricResp> getMetricPage(PageMetricReq pageMetricCmd); PageInfo<MetricResp> getMetricPage(PageMetricReq pageMetricCmd);
List<DomainResp> getDomainListForViewer(); List<DomainResp> getDomainList(User user);
List<DomainResp> getDomainListForAdmin();
List<ModelResp> getModelList(AuthType authType, Long domainId, User user);
} }

View File

@@ -1,15 +1,14 @@
package com.tencent.supersonic.chat.api.pojo; package com.tencent.supersonic.chat.api.pojo;
import lombok.Data;
import java.util.HashSet; import java.util.HashSet;
import java.util.Optional; import java.util.Optional;
import java.util.Set; import java.util.Set;
import lombok.Data;
@Data @Data
public class DomainSchema { public class ModelSchema {
private SchemaElement domain; private SchemaElement model;
private Set<SchemaElement> metrics = new HashSet<>(); private Set<SchemaElement> metrics = new HashSet<>();
private Set<SchemaElement> dimensions = new HashSet<>(); private Set<SchemaElement> dimensions = new HashSet<>();
private Set<SchemaElement> dimensionValues = new HashSet<>(); private Set<SchemaElement> dimensionValues = new HashSet<>();
@@ -22,8 +21,8 @@ public class DomainSchema {
case ENTITY: case ENTITY:
element = Optional.ofNullable(entity); element = Optional.ofNullable(entity);
break; break;
case DOMAIN: case MODEL:
element = Optional.of(domain); element = Optional.of(model);
break; break;
case METRIC: case METRIC:
element = metrics.stream().filter(e -> e.getId() == elementID).findFirst(); element = metrics.stream().filter(e -> e.getId() == elementID).findFirst();

View File

@@ -2,12 +2,13 @@ package com.tencent.supersonic.chat.api.pojo;
import com.tencent.supersonic.chat.api.component.SemanticQuery; import com.tencent.supersonic.chat.api.component.SemanticQuery;
import com.tencent.supersonic.chat.api.pojo.request.QueryReq; import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
import lombok.Data;
import java.util.ArrayList; import java.util.ArrayList;
import java.util.List; import java.util.List;
import lombok.Data;
import lombok.NoArgsConstructor;
@Data @Data
@NoArgsConstructor
public class QueryContext { public class QueryContext {
private QueryReq request; private QueryReq request;

View File

@@ -1,11 +1,12 @@
package com.tencent.supersonic.chat.api.pojo; package com.tencent.supersonic.chat.api.pojo;
import com.google.common.base.Objects; import com.google.common.base.Objects;
import java.io.Serializable; import java.io.Serializable;
import java.util.List; import java.util.List;
import lombok.Builder;
import lombok.*; import lombok.Data;
import lombok.Getter;
import lombok.NoArgsConstructor;
@Data @Data
@Getter @Getter
@@ -14,7 +15,7 @@ import lombok.*;
//@AllArgsConstructor //@AllArgsConstructor
public class SchemaElement implements Serializable { public class SchemaElement implements Serializable {
private Long domain; private Long model;
private Long id; private Long id;
private String name; private String name;
private String bizName; private String bizName;
@@ -25,9 +26,9 @@ public class SchemaElement implements Serializable {
// public SchemaElement() { // public SchemaElement() {
// } // }
public SchemaElement(Long domain, Long id, String name, String bizName, public SchemaElement(Long model, Long id, String name, String bizName,
Long useCnt, SchemaElementType type, List<String> alias) { Long useCnt, SchemaElementType type, List<String> alias) {
this.domain = domain; this.model = model;
this.id = id; this.id = id;
this.name = name; this.name = name;
this.bizName = bizName; this.bizName = bizName;
@@ -45,7 +46,7 @@ public class SchemaElement implements Serializable {
return false; return false;
} }
SchemaElement schemaElement = (SchemaElement) o; SchemaElement schemaElement = (SchemaElement) o;
return Objects.equal(domain, schemaElement.domain) && Objects.equal(id, return Objects.equal(model, schemaElement.model) && Objects.equal(id,
schemaElement.id) && Objects.equal(name, schemaElement.name) schemaElement.id) && Objects.equal(name, schemaElement.name)
&& Objects.equal(bizName, schemaElement.bizName) && Objects.equal( && Objects.equal(bizName, schemaElement.bizName) && Objects.equal(
useCnt, schemaElement.useCnt) && Objects.equal(type, schemaElement.type); useCnt, schemaElement.useCnt) && Objects.equal(type, schemaElement.type);
@@ -53,6 +54,6 @@ public class SchemaElement implements Serializable {
@Override @Override
public int hashCode() { public int hashCode() {
return Objects.hashCode(domain, id, name, bizName, useCnt, type); return Objects.hashCode(model, id, name, bizName, useCnt, type);
} }
} }

View File

@@ -18,10 +18,6 @@ public class SchemaElementMatch {
String detectWord; String detectWord;
String word; String word;
Long frequency; Long frequency;
MatchMode mode = MatchMode.CURRENT; boolean isInherited;
public enum MatchMode {
CURRENT,
INHERIT
}
} }

View File

@@ -1,7 +1,7 @@
package com.tencent.supersonic.chat.api.pojo; package com.tencent.supersonic.chat.api.pojo;
public enum SchemaElementType { public enum SchemaElementType {
DOMAIN, MODEL,
METRIC, METRIC,
DIMENSION, DIMENSION,
VALUE, VALUE,

View File

@@ -7,21 +7,21 @@ import java.util.Set;
public class SchemaMapInfo { public class SchemaMapInfo {
private Map<Long, List<SchemaElementMatch>> domainElementMatches = new HashMap<>(); private Map<Long, List<SchemaElementMatch>> modelElementMatches = new HashMap<>();
public Set<Long> getMatchedDomains() { public Set<Long> getMatchedModels() {
return domainElementMatches.keySet(); return modelElementMatches.keySet();
} }
public List<SchemaElementMatch> getMatchedElements(Long domain) { public List<SchemaElementMatch> getMatchedElements(Long model) {
return domainElementMatches.get(domain); return modelElementMatches.get(model);
} }
public Map<Long, List<SchemaElementMatch>> getDomainElementMatches() { public Map<Long, List<SchemaElementMatch>> getModelElementMatches() {
return domainElementMatches; return modelElementMatches;
} }
public void setMatchedElements(Long domain, List<SchemaElementMatch> elementMatches) { public void setMatchedElements(Long model, List<SchemaElementMatch> elementMatches) {
domainElementMatches.put(domain, elementMatches); modelElementMatches.put(model, elementMatches);
} }
} }

View File

@@ -1,19 +1,25 @@
package com.tencent.supersonic.chat.api.pojo; package com.tencent.supersonic.chat.api.pojo;
import java.util.*;
import com.tencent.supersonic.chat.api.pojo.request.QueryFilter; import com.tencent.supersonic.chat.api.pojo.request.QueryFilter;
import com.tencent.supersonic.common.pojo.DateConf; import com.tencent.supersonic.common.pojo.DateConf;
import com.tencent.supersonic.common.pojo.Order; import com.tencent.supersonic.common.pojo.Order;
import com.tencent.supersonic.common.pojo.enums.AggregateTypeEnum; import com.tencent.supersonic.common.pojo.enums.AggregateTypeEnum;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.HashMap;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.TreeSet;
import lombok.Data; import lombok.Data;
@Data @Data
public class SemanticParseInfo { public class SemanticParseInfo {
private String queryMode; private String queryMode;
private SchemaElement domain; private SchemaElement model;
private Set<SchemaElement> metrics = new TreeSet<>(new SchemaNameLengthComparator()); private Set<SchemaElement> metrics = new TreeSet<>(new SchemaNameLengthComparator());
private Set<SchemaElement> dimensions = new LinkedHashSet(); private Set<SchemaElement> dimensions = new LinkedHashSet();
private SchemaElement entity; private SchemaElement entity;
@@ -28,15 +34,16 @@ public class SemanticParseInfo {
private List<SchemaElementMatch> elementMatches = new ArrayList<>(); private List<SchemaElementMatch> elementMatches = new ArrayList<>();
private Map<String, Object> properties = new HashMap<>(); private Map<String, Object> properties = new HashMap<>();
public Long getDomainId() { public Long getModelId() {
return domain != null ? domain.getId() : 0L; return model != null ? model.getId() : 0L;
} }
public String getDomainName() { public String getModelName() {
return domain != null ? domain.getName() : "null"; return model != null ? model.getName() : "null";
} }
private static class SchemaNameLengthComparator implements Comparator<SchemaElement> { private static class SchemaNameLengthComparator implements Comparator<SchemaElement> {
@Override @Override
public int compare(SchemaElement o1, SchemaElement o2) { public int compare(SchemaElement o1, SchemaElement o2) {
int len1 = o1.getName().length(); int len1 = o1.getName().length();
@@ -49,4 +56,11 @@ public class SemanticParseInfo {
} }
} }
public Set<SchemaElement> getMetrics() {
Set<SchemaElement> metricSet = new TreeSet<>(new SchemaNameLengthComparator());
metricSet.addAll(metrics);
metrics = metricSet;
return metrics;
}
} }

View File

@@ -7,48 +7,49 @@ import java.util.Map;
import java.util.stream.Collectors; import java.util.stream.Collectors;
public class SemanticSchema implements Serializable { public class SemanticSchema implements Serializable {
private List<DomainSchema> domainSchemaList;
public SemanticSchema(List<DomainSchema> domainSchemaList) { private List<ModelSchema> modelSchemaList;
this.domainSchemaList = domainSchemaList;
public SemanticSchema(List<ModelSchema> modelSchemaList) {
this.modelSchemaList = modelSchemaList;
} }
public void add(DomainSchema schema) { public void add(ModelSchema schema) {
domainSchemaList.add(schema); modelSchemaList.add(schema);
} }
public Map<Long, String> getDomainIdToName() { public Map<Long, String> getModelIdToName() {
return domainSchemaList.stream() return modelSchemaList.stream()
.collect(Collectors.toMap(a -> a.getDomain().getId(), a -> a.getDomain().getName(), (k1, k2) -> k1)); .collect(Collectors.toMap(a -> a.getModel().getId(), a -> a.getModel().getName(), (k1, k2) -> k1));
} }
public List<SchemaElement> getDimensionValues() { public List<SchemaElement> getDimensionValues() {
List<SchemaElement> dimensionValues = new ArrayList<>(); List<SchemaElement> dimensionValues = new ArrayList<>();
domainSchemaList.stream().forEach(d -> dimensionValues.addAll(d.getDimensionValues())); modelSchemaList.stream().forEach(d -> dimensionValues.addAll(d.getDimensionValues()));
return dimensionValues; return dimensionValues;
} }
public List<SchemaElement> getDimensions() { public List<SchemaElement> getDimensions() {
List<SchemaElement> dimensions = new ArrayList<>(); List<SchemaElement> dimensions = new ArrayList<>();
domainSchemaList.stream().forEach(d -> dimensions.addAll(d.getDimensions())); modelSchemaList.stream().forEach(d -> dimensions.addAll(d.getDimensions()));
return dimensions; return dimensions;
} }
public List<SchemaElement> getMetrics() { public List<SchemaElement> getMetrics() {
List<SchemaElement> metrics = new ArrayList<>(); List<SchemaElement> metrics = new ArrayList<>();
domainSchemaList.stream().forEach(d -> metrics.addAll(d.getMetrics())); modelSchemaList.stream().forEach(d -> metrics.addAll(d.getMetrics()));
return metrics; return metrics;
} }
public List<SchemaElement> getDomains() { public List<SchemaElement> getModels() {
List<SchemaElement> domains = new ArrayList<>(); List<SchemaElement> models = new ArrayList<>();
domainSchemaList.stream().forEach(d -> domains.add(d.getDomain())); modelSchemaList.stream().forEach(d -> models.add(d.getModel()));
return domains; return models;
} }
public List<SchemaElement> getEntities() { public List<SchemaElement> getEntities() {
List<SchemaElement> entities = new ArrayList<>(); List<SchemaElement> entities = new ArrayList<>();
domainSchemaList.stream().forEach(d -> entities.add(d.getEntity())); modelSchemaList.stream().forEach(d -> entities.add(d.getEntity()));
return entities; return entities;
} }
} }

View File

@@ -1,8 +1,7 @@
package com.tencent.supersonic.chat.api.pojo.request; package com.tencent.supersonic.chat.api.pojo.request;
import lombok.Data;
import java.util.List; import java.util.List;
import lombok.Data;
@Data @Data
public class ChatAggConfigReq { public class ChatAggConfigReq {
@@ -13,7 +12,7 @@ public class ChatAggConfigReq {
private ItemVisibility visibility; private ItemVisibility visibility;
/** /**
* information about dictionary about the domain * information about dictionary about the model
*/ */
private List<KnowledgeInfoReq> knowledgeInfos; private List<KnowledgeInfoReq> knowledgeInfos;

View File

@@ -1,34 +1,32 @@
package com.tencent.supersonic.chat.api.pojo.request; package com.tencent.supersonic.chat.api.pojo.request;
import com.tencent.supersonic.common.pojo.enums.StatusEnum; import com.tencent.supersonic.common.pojo.enums.StatusEnum;
import java.util.List;
import lombok.Data; import lombok.Data;
import lombok.ToString; import lombok.ToString;
import java.util.List;
/** /**
* extended information command about domain * extended information command about model
*/ */
@Data @Data
@ToString @ToString
public class ChatConfigBaseReq { public class ChatConfigBaseReq {
private Long domainId; private Long modelId;
/** /**
* the chatDetailConfig about the domain * the chatDetailConfig about the model
*/ */
private ChatDetailConfigReq chatDetailConfig; private ChatDetailConfigReq chatDetailConfig;
/** /**
* the chatAggConfig about the domain * the chatAggConfig about the model
*/ */
private ChatAggConfigReq chatAggConfig; private ChatAggConfigReq chatAggConfig;
/** /**
* the recommended questions about the domain * the recommended questions about the model
*/ */
private List<RecommendedQuestionReq> recommendedQuestions; private List<RecommendedQuestionReq> recommendedQuestions;

View File

@@ -10,6 +10,6 @@ import lombok.NoArgsConstructor;
public class ChatConfigFilter { public class ChatConfigFilter {
private Long id; private Long id;
private Long domainId; private Long modelId;
private StatusEnum status = StatusEnum.ONLINE; private StatusEnum status = StatusEnum.ONLINE;
} }

View File

@@ -2,10 +2,9 @@ package com.tencent.supersonic.chat.api.pojo.request;
import com.tencent.supersonic.common.pojo.Constants; import com.tencent.supersonic.common.pojo.Constants;
import lombok.Data;
import java.util.ArrayList; import java.util.ArrayList;
import java.util.List; import java.util.List;
import lombok.Data;
@Data @Data
public class ChatDefaultConfigReq { public class ChatDefaultConfigReq {

View File

@@ -1,8 +1,7 @@
package com.tencent.supersonic.chat.api.pojo.request; package com.tencent.supersonic.chat.api.pojo.request;
import lombok.Data;
import java.util.List; import java.util.List;
import lombok.Data;
@Data @Data
public class ChatDetailConfigReq { public class ChatDetailConfigReq {
@@ -13,7 +12,7 @@ public class ChatDetailConfigReq {
private ItemVisibility visibility; private ItemVisibility visibility;
/** /**
* information about dictionary about the domain * information about dictionary about the model
*/ */
private List<KnowledgeInfoReq> knowledgeInfos; private List<KnowledgeInfoReq> knowledgeInfos;

View File

@@ -7,7 +7,7 @@ import lombok.NoArgsConstructor;
import lombok.ToString; import lombok.ToString;
/** /**
* the entity info about the domain * the entity info about the model
*/ */
@Data @Data
@AllArgsConstructor @AllArgsConstructor

View File

@@ -7,6 +7,7 @@ import lombok.Data;
@Data @Data
public class ExecuteQueryReq { public class ExecuteQueryReq {
private User user; private User user;
private Integer chatId; private Integer chatId;
private String queryText; private String queryText;

View File

@@ -1,9 +1,8 @@
package com.tencent.supersonic.chat.api.pojo.request; package com.tencent.supersonic.chat.api.pojo.request;
import lombok.Data;
import java.util.ArrayList; import java.util.ArrayList;
import java.util.List; import java.util.List;
import lombok.Data;
/** /**
* advanced knowledge config * advanced knowledge config

View File

@@ -1,20 +1,18 @@
package com.tencent.supersonic.chat.api.pojo.request; package com.tencent.supersonic.chat.api.pojo.request;
import com.tencent.supersonic.common.pojo.enums.TypeEnums; import com.tencent.supersonic.common.pojo.enums.TypeEnums;
import javax.validation.constraints.NotNull; import javax.validation.constraints.NotNull;
import lombok.Data; import lombok.Data;
/** /**
* information about dictionary about the domain * information about dictionary about the model
*/ */
@Data @Data
public class KnowledgeInfoReq { public class KnowledgeInfoReq {
/** /**
* metricId、DimensionId、domainId * metricId、DimensionId、modelId
*/ */
private Long itemId; private Long itemId;

View File

@@ -13,7 +13,7 @@ public class PluginQueryReq {
private String type; private String type;
private String domain; private String model;
private String pattern; private String pattern;

View File

@@ -1,18 +1,18 @@
package com.tencent.supersonic.chat.api.pojo.request; package com.tencent.supersonic.chat.api.pojo.request;
import java.util.HashSet;
import java.util.Set;
import com.tencent.supersonic.chat.api.pojo.SchemaElement; import com.tencent.supersonic.chat.api.pojo.SchemaElement;
import com.tencent.supersonic.common.pojo.DateConf; import com.tencent.supersonic.common.pojo.DateConf;
import com.tencent.supersonic.common.pojo.Order; import com.tencent.supersonic.common.pojo.Order;
import java.util.HashSet;
import java.util.Set;
import lombok.Data; import lombok.Data;
@Data @Data
public class QueryDataReq { public class QueryDataReq {
String queryMode; String queryMode;
SchemaElement domain; SchemaElement model;
Set<SchemaElement> metrics = new HashSet<>(); Set<SchemaElement> metrics = new HashSet<>();
Set<SchemaElement> dimensions = new HashSet<>(); Set<SchemaElement> dimensions = new HashSet<>();
Set<QueryFilter> dimensionFilters = new HashSet<>(); Set<QueryFilter> dimensionFilters = new HashSet<>();

View File

@@ -1,13 +1,14 @@
package com.tencent.supersonic.chat.api.pojo.request; package com.tencent.supersonic.chat.api.pojo.request;
import lombok.Data;
import java.util.ArrayList; import java.util.ArrayList;
import java.util.HashMap; import java.util.HashMap;
import java.util.List; import java.util.List;
import java.util.Map; import java.util.Map;
import lombok.Data;
@Data @Data
public class QueryFilters { public class QueryFilters {
private List<QueryFilter> filters = new ArrayList<>(); private List<QueryFilter> filters = new ArrayList<>();
private Map<String, Object> params = new HashMap<>(); private Map<String, Object> params = new HashMap<>();
} }

View File

@@ -5,9 +5,10 @@ import lombok.Data;
@Data @Data
public class QueryReq { public class QueryReq {
private String queryText; private String queryText;
private Integer chatId; private Integer chatId;
private Long domainId = 0L; private Long modelId = 0L;
private User user; private User user;
private QueryFilters queryFilters; private QueryFilters queryFilters;
private boolean saveAnswer = true; private boolean saveAnswer = true;

View File

@@ -6,5 +6,6 @@ import lombok.Data;
@Data @Data
public class AggregateInfo { public class AggregateInfo {
private List<MetricInfo> metricInfos = new ArrayList<>();
private List<MetricInfo> metricInfos = new ArrayList<>();
} }

View File

@@ -2,9 +2,8 @@ package com.tencent.supersonic.chat.api.pojo.response;
import com.tencent.supersonic.chat.api.pojo.request.KnowledgeAdvancedConfig; import com.tencent.supersonic.chat.api.pojo.request.KnowledgeAdvancedConfig;
import com.tencent.supersonic.chat.api.pojo.request.KnowledgeInfoReq; import com.tencent.supersonic.chat.api.pojo.request.KnowledgeInfoReq;
import lombok.Data;
import java.util.List; import java.util.List;
import lombok.Data;
@Data @Data
public class ChatAggRichConfigResp { public class ChatAggRichConfigResp {
@@ -15,7 +14,7 @@ public class ChatAggRichConfigResp {
private ItemVisibilityInfo visibility; private ItemVisibilityInfo visibility;
/** /**
* information about dictionary about the domain * information about dictionary about the model
*/ */
private List<KnowledgeInfoReq> knowledgeInfos; private List<KnowledgeInfoReq> knowledgeInfos;

View File

@@ -4,10 +4,8 @@ import com.tencent.supersonic.chat.api.pojo.request.ChatAggConfigReq;
import com.tencent.supersonic.chat.api.pojo.request.ChatDetailConfigReq; import com.tencent.supersonic.chat.api.pojo.request.ChatDetailConfigReq;
import com.tencent.supersonic.chat.api.pojo.request.RecommendedQuestionReq; import com.tencent.supersonic.chat.api.pojo.request.RecommendedQuestionReq;
import com.tencent.supersonic.common.pojo.enums.StatusEnum; import com.tencent.supersonic.common.pojo.enums.StatusEnum;
import java.util.Date; import java.util.Date;
import java.util.List; import java.util.List;
import lombok.Data; import lombok.Data;
@Data @Data
@@ -15,7 +13,7 @@ public class ChatConfigResp {
private Long id; private Long id;
private Long domainId; private Long modelId;
private ChatDetailConfigReq chatDetailConfig; private ChatDetailConfigReq chatDetailConfig;

View File

@@ -4,7 +4,6 @@ import com.tencent.supersonic.chat.api.pojo.request.RecommendedQuestionReq;
import com.tencent.supersonic.common.pojo.enums.StatusEnum; import com.tencent.supersonic.common.pojo.enums.StatusEnum;
import java.util.Date; import java.util.Date;
import java.util.List; import java.util.List;
import lombok.Data; import lombok.Data;
@Data @Data
@@ -12,9 +11,9 @@ public class ChatConfigRichResp {
private Long id; private Long id;
private Long domainId; private Long modelId;
private String domainName; private String modelName;
private String bizName; private String bizName;
private ChatAggRichConfigResp chatAggRichConfig; private ChatAggRichConfigResp chatAggRichConfig;

View File

@@ -4,9 +4,8 @@ package com.tencent.supersonic.chat.api.pojo.response;
import com.tencent.supersonic.chat.api.pojo.SchemaElement; import com.tencent.supersonic.chat.api.pojo.SchemaElement;
import com.tencent.supersonic.chat.api.pojo.request.ChatDefaultConfigReq; import com.tencent.supersonic.chat.api.pojo.request.ChatDefaultConfigReq;
import com.tencent.supersonic.common.pojo.Constants; import com.tencent.supersonic.common.pojo.Constants;
import lombok.Data;
import java.util.List; import java.util.List;
import lombok.Data;
@Data @Data
public class ChatDefaultRichConfigResp { public class ChatDefaultRichConfigResp {

View File

@@ -2,9 +2,8 @@ package com.tencent.supersonic.chat.api.pojo.response;
import com.tencent.supersonic.chat.api.pojo.request.KnowledgeAdvancedConfig; import com.tencent.supersonic.chat.api.pojo.request.KnowledgeAdvancedConfig;
import com.tencent.supersonic.chat.api.pojo.request.KnowledgeInfoReq; import com.tencent.supersonic.chat.api.pojo.request.KnowledgeInfoReq;
import lombok.Data;
import java.util.List; import java.util.List;
import lombok.Data;
@Data @Data
public class ChatDetailRichConfigResp { public class ChatDetailRichConfigResp {
@@ -15,7 +14,7 @@ public class ChatDetailRichConfigResp {
private ItemVisibilityInfo visibility; private ItemVisibilityInfo visibility;
/** /**
* information about dictionary about the domain * information about dictionary about the model
*/ */
private List<KnowledgeInfoReq> knowledgeInfos; private List<KnowledgeInfoReq> knowledgeInfos;

View File

@@ -7,7 +7,7 @@ import lombok.Data;
@Data @Data
public class EntityInfo { public class EntityInfo {
private DomainInfo domainInfo = new DomainInfo(); private ModelInfo modelInfo = new ModelInfo();
private List<DataInfo> dimensions = new ArrayList<>(); private List<DataInfo> dimensions = new ArrayList<>();
private List<DataInfo> metrics = new ArrayList<>(); private List<DataInfo> metrics = new ArrayList<>();
private String entityId; private String entityId;

View File

@@ -1,14 +1,14 @@
package com.tencent.supersonic.chat.api.pojo.response; package com.tencent.supersonic.chat.api.pojo.response;
import com.tencent.supersonic.chat.api.pojo.SchemaElement; import com.tencent.supersonic.chat.api.pojo.SchemaElement;
import java.util.List; import java.util.List;
import lombok.Data; import lombok.Data;
@Data @Data
public class EntityRichInfoResp { public class EntityRichInfoResp {
/** /**
* entity alias * entity alias
*/ */
private List<String> names; private List<String> names;

View File

@@ -5,7 +5,7 @@ import java.util.List;
import lombok.Data; import lombok.Data;
@Data @Data
public class DomainInfo extends DataInfo implements Serializable { public class ModelInfo extends DataInfo implements Serializable {
private List<String> words; private List<String> words;
private String primaryEntityBizName; private String primaryEntityBizName;

View File

@@ -1,9 +1,12 @@
package com.tencent.supersonic.chat.api.pojo.response; package com.tencent.supersonic.chat.api.pojo.response;
import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo; import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
import lombok.*;
import java.util.List; import java.util.List;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.Getter;
import lombok.NoArgsConstructor;
@Data @Data
@Getter @Getter
@@ -11,6 +14,7 @@ import java.util.List;
@NoArgsConstructor @NoArgsConstructor
@AllArgsConstructor @AllArgsConstructor
public class ParseResp { public class ParseResp {
private Integer chatId; private Integer chatId;
private String queryText; private String queryText;
private ParseState state; private ParseState state;

View File

@@ -1,14 +1,14 @@
package com.tencent.supersonic.chat.api.pojo.response; package com.tencent.supersonic.chat.api.pojo.response;
import com.tencent.supersonic.chat.api.pojo.request.RecommendedQuestionReq; import com.tencent.supersonic.chat.api.pojo.request.RecommendedQuestionReq;
import java.util.List;
import lombok.AllArgsConstructor; import lombok.AllArgsConstructor;
import lombok.Data; import lombok.Data;
import java.util.List;
@Data @Data
@AllArgsConstructor @AllArgsConstructor
public class RecommendQuestionResp { public class RecommendQuestionResp {
private Long domainId;
private Long modelId;
private List<RecommendedQuestionReq> recommendedQuestions; private List<RecommendedQuestionReq> recommendedQuestions;
} }

View File

@@ -1,12 +1,12 @@
package com.tencent.supersonic.chat.api.pojo.response; package com.tencent.supersonic.chat.api.pojo.response;
import com.tencent.supersonic.chat.api.pojo.SchemaElement; import com.tencent.supersonic.chat.api.pojo.SchemaElement;
import lombok.Data;
import java.util.List; import java.util.List;
import lombok.Data;
@Data @Data
public class RecommendResp { public class RecommendResp {
private List<SchemaElement> dimensions; private List<SchemaElement> dimensions;
private List<SchemaElement> metrics; private List<SchemaElement> metrics;
} }

View File

@@ -2,8 +2,6 @@ package com.tencent.supersonic.chat.api.pojo.response;
import java.util.List; import java.util.List;
import lombok.Data; import lombok.Data;
import lombok.Getter;
import lombok.Setter;
@Data @Data
public class SearchResp { public class SearchResp {

View File

@@ -17,9 +17,9 @@ public class SearchResult {
private String subRecommend; private String subRecommend;
private String domainName; private String modelName;
private Long domainId; private Long modelId;
private SchemaElementType schemaElementType; private SchemaElementType schemaElementType;
@@ -34,12 +34,12 @@ public class SearchResult {
return false; return false;
} }
SearchResult searchResult1 = (SearchResult) o; SearchResult searchResult1 = (SearchResult) o;
return Objects.equals(recommend, searchResult1.recommend) && Objects.equals(domainName, return Objects.equals(recommend, searchResult1.recommend) && Objects.equals(modelName,
searchResult1.domainName); searchResult1.modelName);
} }
@Override @Override
public int hashCode() { public int hashCode() {
return Objects.hash(recommend, domainName); return Objects.hash(recommend, modelName);
} }
} }

View File

@@ -1,7 +1,7 @@
<?xml version="1.0" encoding="UTF-8"?> <?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" <project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent> <parent>
<artifactId>chat</artifactId> <artifactId>chat</artifactId>
<groupId>com.tencent.supersonic</groupId> <groupId>com.tencent.supersonic</groupId>

View File

@@ -3,11 +3,13 @@ package com.tencent.supersonic.chat.config;
import lombok.Data; import lombok.Data;
import org.springframework.beans.factory.annotation.Value; import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Configuration; import org.springframework.context.annotation.Configuration;
@Configuration @Configuration
@Data @Data
public class AggregatorConfig { public class AggregatorConfig {
@Value("${metric.aggregator.ratio.enable:true}")
private Boolean enableRatio; @Value("${metric.aggregator.ratio.enable:true}")
private Boolean enableRatio;
} }

View File

@@ -3,13 +3,12 @@ package com.tencent.supersonic.chat.config;
import com.tencent.supersonic.chat.api.pojo.request.ChatAggConfigReq; import com.tencent.supersonic.chat.api.pojo.request.ChatAggConfigReq;
import com.tencent.supersonic.chat.api.pojo.request.ChatDetailConfigReq; import com.tencent.supersonic.chat.api.pojo.request.ChatDetailConfigReq;
import com.tencent.supersonic.chat.api.pojo.request.RecommendedQuestionReq; import com.tencent.supersonic.chat.api.pojo.request.RecommendedQuestionReq;
import com.tencent.supersonic.common.pojo.enums.StatusEnum;
import com.tencent.supersonic.common.pojo.RecordInfo; import com.tencent.supersonic.common.pojo.RecordInfo;
import com.tencent.supersonic.common.pojo.enums.StatusEnum;
import java.util.List;
import lombok.Data; import lombok.Data;
import lombok.ToString; import lombok.ToString;
import java.util.List;
@Data @Data
@ToString @ToString
public class ChatConfig { public class ChatConfig {
@@ -19,15 +18,15 @@ public class ChatConfig {
*/ */
private Long id; private Long id;
private Long domainId; private Long modelId;
/** /**
* the chatDetailConfig about the domain * the chatDetailConfig about the model
*/ */
private ChatDetailConfigReq chatDetailConfig; private ChatDetailConfigReq chatDetailConfig;
/** /**
* the chatAggConfig about the domain * the chatAggConfig about the model
*/ */
private ChatAggConfigReq chatAggConfig; private ChatAggConfigReq chatAggConfig;

View File

@@ -6,6 +6,6 @@ import lombok.Data;
public class ChatConfigFilterInternal { public class ChatConfigFilterInternal {
private Long id; private Long id;
private Long domainId; private Long modelId;
private Integer status; private Integer status;
} }

View File

@@ -6,7 +6,7 @@ import lombok.ToString;
/** /**
* default metrics about the domain * default metrics about the model
*/ */
@ToString @ToString

View File

@@ -7,6 +7,7 @@ import org.springframework.context.annotation.Configuration;
@Configuration @Configuration
@Data @Data
public class FunctionCallInfoConfig { public class FunctionCallInfoConfig {
@Value("${functionCall.url:}") @Value("${functionCall.url:}")
private String url; private String url;
} }

View File

@@ -1,21 +0,0 @@
package com.tencent.supersonic.chat.mapper;
import java.io.Serializable;
import lombok.Builder;
import lombok.Data;
import lombok.ToString;
@Data
@ToString
@Builder
public class DomainInfoStat implements Serializable {
private long domainCount;
private long metricDomainCount;
private long dimensionDomainCount;
private long dimensionValueDomainCount;
}

View File

@@ -2,14 +2,19 @@ package com.tencent.supersonic.chat.mapper;
import com.tencent.supersonic.chat.api.component.SchemaMapper; import com.tencent.supersonic.chat.api.component.SchemaMapper;
import com.tencent.supersonic.chat.api.pojo.*; import com.tencent.supersonic.chat.api.pojo.ModelSchema;
import com.tencent.supersonic.chat.api.pojo.QueryContext;
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
import com.tencent.supersonic.chat.api.pojo.SchemaMapInfo;
import com.tencent.supersonic.chat.service.SemanticService; import com.tencent.supersonic.chat.service.SemanticService;
import com.tencent.supersonic.common.util.ContextUtils; import com.tencent.supersonic.common.util.ContextUtils;
import java.util.List;
import java.util.stream.Collectors;
import lombok.extern.slf4j.Slf4j; import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.BeanUtils; import org.springframework.beans.BeanUtils;
import org.springframework.util.CollectionUtils; import org.springframework.util.CollectionUtils;
import java.util.List;
import java.util.stream.Collectors;
@Slf4j @Slf4j
@@ -18,20 +23,20 @@ public class EntityMapper implements SchemaMapper {
@Override @Override
public void map(QueryContext queryContext) { public void map(QueryContext queryContext) {
SchemaMapInfo schemaMapInfo = queryContext.getMapInfo(); SchemaMapInfo schemaMapInfo = queryContext.getMapInfo();
for (Long domainId : schemaMapInfo.getMatchedDomains()) { for (Long modelId : schemaMapInfo.getMatchedModels()) {
List<SchemaElementMatch> schemaElementMatchList = schemaMapInfo.getMatchedElements(domainId); List<SchemaElementMatch> schemaElementMatchList = schemaMapInfo.getMatchedElements(modelId);
if (CollectionUtils.isEmpty(schemaElementMatchList)) { if (CollectionUtils.isEmpty(schemaElementMatchList)) {
continue; continue;
} }
SchemaElement entity = getEntity(domainId); SchemaElement entity = getEntity(modelId);
if (entity == null || entity.getId() == null) { if (entity == null || entity.getId() == null) {
continue; continue;
} }
List<SchemaElementMatch> valueSchemaElements = schemaElementMatchList.stream().filter(schemaElementMatch -> List<SchemaElementMatch> valueSchemaElements = schemaElementMatchList.stream().filter(schemaElementMatch ->
SchemaElementType.VALUE.equals(schemaElementMatch.getElement().getType())) SchemaElementType.VALUE.equals(schemaElementMatch.getElement().getType()))
.collect(Collectors.toList()); .collect(Collectors.toList());
for (SchemaElementMatch schemaElementMatch : valueSchemaElements) { for (SchemaElementMatch schemaElementMatch : valueSchemaElements) {
if (!entity.getId().equals(schemaElementMatch.getElement().getId())){ if (!entity.getId().equals(schemaElementMatch.getElement().getId())) {
continue; continue;
} }
if (!checkExistSameEntitySchemaElements(schemaElementMatch, schemaElementMatchList)) { if (!checkExistSameEntitySchemaElements(schemaElementMatch, schemaElementMatchList)) {
@@ -46,7 +51,7 @@ public class EntityMapper implements SchemaMapper {
} }
private boolean checkExistSameEntitySchemaElements(SchemaElementMatch valueSchemaElementMatch, private boolean checkExistSameEntitySchemaElements(SchemaElementMatch valueSchemaElementMatch,
List<SchemaElementMatch> schemaElementMatchList) { List<SchemaElementMatch> schemaElementMatchList) {
List<SchemaElementMatch> entitySchemaElements = schemaElementMatchList.stream().filter(schemaElementMatch -> List<SchemaElementMatch> entitySchemaElements = schemaElementMatchList.stream().filter(schemaElementMatch ->
SchemaElementType.ENTITY.equals(schemaElementMatch.getElement().getType())) SchemaElementType.ENTITY.equals(schemaElementMatch.getElement().getType()))
.collect(Collectors.toList()); .collect(Collectors.toList());
@@ -58,11 +63,11 @@ public class EntityMapper implements SchemaMapper {
return false; return false;
} }
private SchemaElement getEntity(Long domainId) { private SchemaElement getEntity(Long modelId) {
SemanticService semanticService = ContextUtils.getBean(SemanticService.class); SemanticService semanticService = ContextUtils.getBean(SemanticService.class);
DomainSchema domainSchema = semanticService.getDomainSchema(domainId); ModelSchema modelSchema = semanticService.getModelSchema(modelId);
if (domainSchema != null && domainSchema.getEntity() != null) { if (modelSchema != null && modelSchema.getEntity() != null) {
return domainSchema.getEntity(); return modelSchema.getEntity();
} }
return null; return null;
} }

View File

@@ -2,10 +2,14 @@ package com.tencent.supersonic.chat.mapper;
import com.hankcs.hanlp.seg.common.Term; import com.hankcs.hanlp.seg.common.Term;
import com.tencent.supersonic.chat.api.component.SchemaMapper; import com.tencent.supersonic.chat.api.component.SchemaMapper;
import com.tencent.supersonic.chat.api.pojo.*;
import com.tencent.supersonic.chat.api.pojo.QueryContext; import com.tencent.supersonic.chat.api.pojo.QueryContext;
import com.tencent.supersonic.knowledge.service.SchemaService; import com.tencent.supersonic.chat.api.pojo.SchemaElement;
import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
import com.tencent.supersonic.chat.api.pojo.SchemaMapInfo;
import com.tencent.supersonic.chat.api.pojo.SemanticSchema;
import com.tencent.supersonic.common.util.ContextUtils; import com.tencent.supersonic.common.util.ContextUtils;
import com.tencent.supersonic.knowledge.service.SchemaService;
import com.tencent.supersonic.knowledge.utils.HanlpHelper; import com.tencent.supersonic.knowledge.utils.HanlpHelper;
import java.util.ArrayList; import java.util.ArrayList;
import java.util.Comparator; import java.util.Comparator;
@@ -39,13 +43,13 @@ public class FuzzyNameMapper implements SchemaMapper {
log.debug("after db mapper,mapInfo:{}", queryContext.getMapInfo()); log.debug("after db mapper,mapInfo:{}", queryContext.getMapInfo());
} }
private void detectAndAddToSchema(QueryContext queryContext, List<Term> terms, List<SchemaElement> domains, private void detectAndAddToSchema(QueryContext queryContext, List<Term> terms, List<SchemaElement> Models,
SchemaElementType schemaElementType) { SchemaElementType schemaElementType) {
try { try {
Map<String, Set<SchemaElement>> domainResultSet = getResultSet(queryContext, terms, domains); Map<String, Set<SchemaElement>> ModelResultSet = getResultSet(queryContext, terms, Models);
addToSchemaMapInfo(domainResultSet, queryContext.getMapInfo(), schemaElementType); addToSchemaMapInfo(ModelResultSet, queryContext.getMapInfo(), schemaElementType);
} catch (Exception e) { } catch (Exception e) {
log.error("detectAndAddToSchema error", e); log.error("detectAndAddToSchema error", e);
@@ -53,7 +57,7 @@ public class FuzzyNameMapper implements SchemaMapper {
} }
private Map<String, Set<SchemaElement>> getResultSet(QueryContext queryContext, List<Term> terms, private Map<String, Set<SchemaElement>> getResultSet(QueryContext queryContext, List<Term> terms,
List<SchemaElement> domains) { List<SchemaElement> Models) {
String queryText = queryContext.getRequest().getQueryText(); String queryText = queryContext.getRequest().getQueryText();
@@ -61,12 +65,12 @@ public class FuzzyNameMapper implements SchemaMapper {
Double metricDimensionThresholdConfig = getThreshold(queryContext, mapperHelper); Double metricDimensionThresholdConfig = getThreshold(queryContext, mapperHelper);
Map<String, Set<SchemaElement>> nameToItems = getNameToItems(domains); Map<String, Set<SchemaElement>> nameToItems = getNameToItems(Models);
Map<Integer, Integer> regOffsetToLength = terms.stream().sorted(Comparator.comparing(Term::length)) Map<Integer, Integer> regOffsetToLength = terms.stream().sorted(Comparator.comparing(Term::length))
.collect(Collectors.toMap(Term::getOffset, term -> term.word.length(), (value1, value2) -> value2)); .collect(Collectors.toMap(Term::getOffset, term -> term.word.length(), (value1, value2) -> value2));
Map<String, Set<SchemaElement>> domainResultSet = new HashMap<>(); Map<String, Set<SchemaElement>> ModelResultSet = new HashMap<>();
for (Integer startIndex = 0; startIndex <= queryText.length() - 1; ) { for (Integer startIndex = 0; startIndex <= queryText.length() - 1; ) {
for (Integer endIndex = startIndex; endIndex <= queryText.length(); ) { for (Integer endIndex = startIndex; endIndex <= queryText.length(); ) {
endIndex = mapperHelper.getStepIndex(regOffsetToLength, endIndex); endIndex = mapperHelper.getStepIndex(regOffsetToLength, endIndex);
@@ -82,7 +86,7 @@ public class FuzzyNameMapper implements SchemaMapper {
|| mapperHelper.getSimilarity(detectSegment, name) < metricDimensionThresholdConfig) { || mapperHelper.getSimilarity(detectSegment, name) < metricDimensionThresholdConfig) {
continue; continue;
} }
Set<SchemaElement> preSchemaElements = domainResultSet.putIfAbsent(detectSegment, Set<SchemaElement> preSchemaElements = ModelResultSet.putIfAbsent(detectSegment,
schemaElements); schemaElements);
if (Objects.nonNull(preSchemaElements)) { if (Objects.nonNull(preSchemaElements)) {
preSchemaElements.addAll(schemaElements); preSchemaElements.addAll(schemaElements);
@@ -91,7 +95,7 @@ public class FuzzyNameMapper implements SchemaMapper {
} }
startIndex = mapperHelper.getStepIndex(regOffsetToLength, startIndex); startIndex = mapperHelper.getStepIndex(regOffsetToLength, startIndex);
} }
return domainResultSet; return ModelResultSet;
} }
private Double getThreshold(QueryContext queryContext, MapperHelper mapperHelper) { private Double getThreshold(QueryContext queryContext, MapperHelper mapperHelper) {
@@ -99,9 +103,9 @@ public class FuzzyNameMapper implements SchemaMapper {
Double metricDimensionThresholdConfig = mapperHelper.getMetricDimensionThresholdConfig(); Double metricDimensionThresholdConfig = mapperHelper.getMetricDimensionThresholdConfig();
Double metricDimensionMinThresholdConfig = mapperHelper.getMetricDimensionMinThresholdConfig(); Double metricDimensionMinThresholdConfig = mapperHelper.getMetricDimensionMinThresholdConfig();
Map<Long, List<SchemaElementMatch>> domainElementMatches = queryContext.getMapInfo() Map<Long, List<SchemaElementMatch>> ModelElementMatches = queryContext.getMapInfo()
.getDomainElementMatches(); .getModelElementMatches();
boolean existElement = domainElementMatches.entrySet().stream() boolean existElement = ModelElementMatches.entrySet().stream()
.anyMatch(entry -> entry.getValue().size() >= 1); .anyMatch(entry -> entry.getValue().size() >= 1);
if (!existElement) { if (!existElement) {
@@ -109,14 +113,14 @@ public class FuzzyNameMapper implements SchemaMapper {
metricDimensionThresholdConfig = halfThreshold >= metricDimensionMinThresholdConfig ? halfThreshold metricDimensionThresholdConfig = halfThreshold >= metricDimensionMinThresholdConfig ? halfThreshold
: metricDimensionMinThresholdConfig; : metricDimensionMinThresholdConfig;
log.info("domainElementMatches:{} , not exist Element metricDimensionThresholdConfig reduce by half:{}", log.info("ModelElementMatches:{} , not exist Element metricDimensionThresholdConfig reduce by half:{}",
domainElementMatches, metricDimensionThresholdConfig); ModelElementMatches, metricDimensionThresholdConfig);
} }
return metricDimensionThresholdConfig; return metricDimensionThresholdConfig;
} }
private Map<String, Set<SchemaElement>> getNameToItems(List<SchemaElement> domains) { private Map<String, Set<SchemaElement>> getNameToItems(List<SchemaElement> Models) {
return domains.stream().collect( return Models.stream().collect(
Collectors.toMap(SchemaElement::getName, a -> { Collectors.toMap(SchemaElement::getName, a -> {
Set<SchemaElement> result = new HashSet<>(); Set<SchemaElement> result = new HashSet<>();
result.add(a); result.add(a);
@@ -139,10 +143,10 @@ public class FuzzyNameMapper implements SchemaMapper {
Set<SchemaElement> schemaElements = entry.getValue(); Set<SchemaElement> schemaElements = entry.getValue();
for (SchemaElement schemaElement : schemaElements) { for (SchemaElement schemaElement : schemaElements) {
List<SchemaElementMatch> elements = schemaMap.getMatchedElements(schemaElement.getDomain()); List<SchemaElementMatch> elements = schemaMap.getMatchedElements(schemaElement.getModel());
if (CollectionUtils.isEmpty(elements)) { if (CollectionUtils.isEmpty(elements)) {
elements = new ArrayList<>(); elements = new ArrayList<>();
schemaMap.setMatchedElements(schemaElement.getDomain(), elements); schemaMap.setMatchedElements(schemaElement.getModel(), elements);
} }
Set<Long> regElementSet = elements.stream() Set<Long> regElementSet = elements.stream()
.filter(elementMatch -> schemaElementType.equals(elementMatch.getElement().getType())) .filter(elementMatch -> schemaElementType.equals(elementMatch.getElement().getType()))

View File

@@ -2,13 +2,18 @@ package com.tencent.supersonic.chat.mapper;
import com.hankcs.hanlp.seg.common.Term; import com.hankcs.hanlp.seg.common.Term;
import com.tencent.supersonic.chat.api.component.SchemaMapper; import com.tencent.supersonic.chat.api.component.SchemaMapper;
import com.tencent.supersonic.chat.api.pojo.*; import com.tencent.supersonic.chat.api.pojo.ModelSchema;
import com.tencent.supersonic.chat.api.pojo.QueryContext;
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
import com.tencent.supersonic.chat.api.pojo.SchemaMapInfo;
import com.tencent.supersonic.chat.service.SemanticService; import com.tencent.supersonic.chat.service.SemanticService;
import com.tencent.supersonic.chat.utils.NatureHelper; import com.tencent.supersonic.chat.utils.NatureHelper;
import com.tencent.supersonic.knowledge.dictionary.builder.BaseWordBuilder;
import com.tencent.supersonic.knowledge.dictionary.MapResult;
import com.tencent.supersonic.knowledge.dictionary.DictWordType;
import com.tencent.supersonic.common.util.ContextUtils; import com.tencent.supersonic.common.util.ContextUtils;
import com.tencent.supersonic.knowledge.dictionary.DictWordType;
import com.tencent.supersonic.knowledge.dictionary.MapResult;
import com.tencent.supersonic.knowledge.dictionary.builder.BaseWordBuilder;
import com.tencent.supersonic.knowledge.dictionary.builder.WordBuilderFactory; import com.tencent.supersonic.knowledge.dictionary.builder.WordBuilderFactory;
import com.tencent.supersonic.knowledge.utils.HanlpHelper; import com.tencent.supersonic.knowledge.utils.HanlpHelper;
import java.util.ArrayList; import java.util.ArrayList;
@@ -32,10 +37,10 @@ public class HanlpDictMapper implements SchemaMapper {
for (Term term : terms) { for (Term term : terms) {
log.info("word:{},nature:{},frequency:{}", term.word, term.nature.toString(), term.getFrequency()); log.info("word:{},nature:{},frequency:{}", term.word, term.nature.toString(), term.getFrequency());
} }
Long domainId = queryContext.getRequest().getDomainId(); Long modelId = queryContext.getRequest().getModelId();
QueryMatchStrategy matchStrategy = ContextUtils.getBean(QueryMatchStrategy.class); QueryMatchStrategy matchStrategy = ContextUtils.getBean(QueryMatchStrategy.class);
Map<MatchText, List<MapResult>> matchResult = matchStrategy.match(queryText, terms, domainId); Map<MatchText, List<MapResult>> matchResult = matchStrategy.match(queryText, terms, modelId);
List<MapResult> matches = getMatches(matchResult); List<MapResult> matches = getMatches(matchResult);
@@ -57,8 +62,8 @@ public class HanlpDictMapper implements SchemaMapper {
for (MapResult mapResult : mapResults) { for (MapResult mapResult : mapResults) {
for (String nature : mapResult.getNatures()) { for (String nature : mapResult.getNatures()) {
Long domainId = NatureHelper.getDomainId(nature); Long modelId = NatureHelper.getModelId(nature);
if (Objects.isNull(domainId)) { if (Objects.isNull(modelId)) {
continue; continue;
} }
SchemaElementType elementType = NatureHelper.convertToElementType(nature); SchemaElementType elementType = NatureHelper.convertToElementType(nature);
@@ -67,14 +72,14 @@ public class HanlpDictMapper implements SchemaMapper {
} }
SemanticService schemaService = ContextUtils.getBean(SemanticService.class); SemanticService schemaService = ContextUtils.getBean(SemanticService.class);
DomainSchema domainSchema = schemaService.getDomainSchema(domainId); ModelSchema modelSchema = schemaService.getModelSchema(modelId);
BaseWordBuilder baseWordBuilder = WordBuilderFactory.get(DictWordType.getNatureType(nature)); BaseWordBuilder baseWordBuilder = WordBuilderFactory.get(DictWordType.getNatureType(nature));
Long elementID = baseWordBuilder.getElementID(nature); Long elementID = baseWordBuilder.getElementID(nature);
Long frequency = wordNatureToFrequency.get(mapResult.getName() + nature); Long frequency = wordNatureToFrequency.get(mapResult.getName() + nature);
SchemaElement element = domainSchema.getElement(elementType, elementID); SchemaElement element = modelSchema.getElement(elementType, elementID);
if(Objects.isNull(element)){ if (Objects.isNull(element)) {
log.info("element is null, elementType:{},elementID:{}", elementType, elementID); log.info("element is null, elementType:{},elementID:{}", elementType, elementID);
continue; continue;
} }
@@ -89,11 +94,11 @@ public class HanlpDictMapper implements SchemaMapper {
.detectWord(mapResult.getDetectWord()) .detectWord(mapResult.getDetectWord())
.build(); .build();
Map<Long, List<SchemaElementMatch>> domainElementMatches = schemaMap.getDomainElementMatches(); Map<Long, List<SchemaElementMatch>> modelElementMatches = schemaMap.getModelElementMatches();
List<SchemaElementMatch> schemaElementMatches = domainElementMatches.putIfAbsent(domainId, List<SchemaElementMatch> schemaElementMatches = modelElementMatches.putIfAbsent(modelId,
new ArrayList<>()); new ArrayList<>());
if (schemaElementMatches == null) { if (schemaElementMatches == null) {
schemaElementMatches = domainElementMatches.get(domainId); schemaElementMatches = modelElementMatches.get(modelId);
} }
schemaElementMatches.add(schemaElementMatch); schemaElementMatches.add(schemaElementMatch);
} }

View File

@@ -83,4 +83,6 @@ public class MapperHelper {
return 1 - (double) EditDistance.compute(detectSegmentLower, matchNameLower) / Math.max(matchName.length(), return 1 - (double) EditDistance.compute(detectSegmentLower, matchNameLower) / Math.max(matchName.length(),
detectSegment.length()); detectSegment.length());
} }
} }

View File

@@ -2,7 +2,6 @@ package com.tencent.supersonic.chat.mapper;
import com.hankcs.hanlp.seg.common.Term; import com.hankcs.hanlp.seg.common.Term;
import com.tencent.supersonic.knowledge.dictionary.MapResult; import com.tencent.supersonic.knowledge.dictionary.MapResult;
import java.util.List; import java.util.List;
import java.util.Map; import java.util.Map;
@@ -11,6 +10,6 @@ import java.util.Map;
*/ */
public interface MatchStrategy { public interface MatchStrategy {
Map<MatchText, List<MapResult>> match(String text, List<Term> terms, Long detectDomainId); Map<MatchText, List<MapResult>> match(String text, List<Term> terms, Long detectModelId);
} }

View File

@@ -0,0 +1,21 @@
package com.tencent.supersonic.chat.mapper;
import java.io.Serializable;
import lombok.Builder;
import lombok.Data;
import lombok.ToString;
@Data
@ToString
@Builder
public class ModelInfoStat implements Serializable {
private long modelCount;
private long metricModelCount;
private long dimensionModelCount;
private long dimensionValueModelCount;
}

View File

@@ -7,13 +7,13 @@ import lombok.ToString;
@Data @Data
@ToString @ToString
public class DomainWithSemanticType implements Serializable { public class ModelWithSemanticType implements Serializable {
private Long domain; private Long model;
private SchemaElementType semanticType; private SchemaElementType semanticType;
public DomainWithSemanticType(Long domain, SchemaElementType semanticType) { public ModelWithSemanticType(Long model, SchemaElementType semanticType) {
this.domain = domain; this.model = model;
this.semanticType = semanticType; this.semanticType = semanticType;
} }
} }

View File

@@ -2,15 +2,20 @@ package com.tencent.supersonic.chat.mapper;
import com.google.common.collect.Lists; import com.google.common.collect.Lists;
import com.tencent.supersonic.chat.api.component.SchemaMapper; import com.tencent.supersonic.chat.api.component.SchemaMapper;
import com.tencent.supersonic.chat.api.pojo.*; import com.tencent.supersonic.chat.api.pojo.QueryContext;
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
import com.tencent.supersonic.chat.api.pojo.SchemaMapInfo;
import com.tencent.supersonic.chat.api.pojo.request.QueryFilter; import com.tencent.supersonic.chat.api.pojo.request.QueryFilter;
import com.tencent.supersonic.chat.api.pojo.request.QueryFilters; import com.tencent.supersonic.chat.api.pojo.request.QueryFilters;
import com.tencent.supersonic.chat.api.pojo.request.QueryReq; import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
import lombok.extern.slf4j.Slf4j; import com.tencent.supersonic.common.pojo.Constants;
import org.springframework.util.CollectionUtils;
import java.util.List; import java.util.List;
import java.util.Map; import java.util.Map;
import java.util.stream.Collectors; import java.util.stream.Collectors;
import lombok.extern.slf4j.Slf4j;
import org.springframework.util.CollectionUtils;
@Slf4j @Slf4j
public class QueryFilterMapper implements SchemaMapper { public class QueryFilterMapper implements SchemaMapper {
@@ -21,35 +26,35 @@ public class QueryFilterMapper implements SchemaMapper {
@Override @Override
public void map(QueryContext queryContext) { public void map(QueryContext queryContext) {
QueryReq queryReq = queryContext.getRequest(); QueryReq queryReq = queryContext.getRequest();
Long domainId = queryReq.getDomainId(); Long modelId = queryReq.getModelId();
if (domainId == null || domainId <= 0) { if (modelId == null || modelId <= 0) {
return; return;
} }
SchemaMapInfo schemaMapInfo = queryContext.getMapInfo(); SchemaMapInfo schemaMapInfo = queryContext.getMapInfo();
clearOtherSchemaElementMatch(domainId, schemaMapInfo); clearOtherSchemaElementMatch(modelId, schemaMapInfo);
List<SchemaElementMatch> schemaElementMatches = schemaMapInfo.getMatchedElements(domainId); List<SchemaElementMatch> schemaElementMatches = schemaMapInfo.getMatchedElements(modelId);
if (schemaElementMatches == null) { if (schemaElementMatches == null) {
schemaElementMatches = Lists.newArrayList(); schemaElementMatches = Lists.newArrayList();
schemaMapInfo.setMatchedElements(domainId, schemaElementMatches); schemaMapInfo.setMatchedElements(modelId, schemaElementMatches);
} }
addValueSchemaElementMatch(schemaElementMatches, queryReq.getQueryFilters()); addValueSchemaElementMatch(schemaElementMatches, queryReq.getQueryFilters());
} }
private void clearOtherSchemaElementMatch(Long domainId, SchemaMapInfo schemaMapInfo) { private void clearOtherSchemaElementMatch(Long modelId, SchemaMapInfo schemaMapInfo) {
for (Map.Entry<Long, List<SchemaElementMatch>> entry : schemaMapInfo.getDomainElementMatches().entrySet()) { for (Map.Entry<Long, List<SchemaElementMatch>> entry : schemaMapInfo.getModelElementMatches().entrySet()) {
if (!entry.getKey().equals(domainId)) { if (!entry.getKey().equals(modelId)) {
entry.getValue().clear(); entry.getValue().clear();
} }
} }
} }
private List<SchemaElementMatch> addValueSchemaElementMatch(List<SchemaElementMatch> candidateElementMatches, private List<SchemaElementMatch> addValueSchemaElementMatch(List<SchemaElementMatch> candidateElementMatches,
QueryFilters queryFilter) { QueryFilters queryFilter) {
if (queryFilter == null || CollectionUtils.isEmpty(queryFilter.getFilters())) { if (queryFilter == null || CollectionUtils.isEmpty(queryFilter.getFilters())) {
return candidateElementMatches; return candidateElementMatches;
} }
for (QueryFilter filter : queryFilter.getFilters()) { for (QueryFilter filter : queryFilter.getFilters()) {
if (checkExistSameValueSchemaElementMatch(filter, candidateElementMatches)) { if (checkExistSameValueSchemaElementMatch(filter, candidateElementMatches)) {
continue; continue;
} }
SchemaElement element = SchemaElement.builder() SchemaElement element = SchemaElement.builder()
@@ -63,7 +68,7 @@ public class QueryFilterMapper implements SchemaMapper {
.frequency(FREQUENCY) .frequency(FREQUENCY)
.word(String.valueOf(filter.getValue())) .word(String.valueOf(filter.getValue()))
.similarity(SIMILARITY) .similarity(SIMILARITY)
.detectWord(filter.getName()) .detectWord(Constants.EMPTY)
.build(); .build();
candidateElementMatches.add(schemaElementMatch); candidateElementMatches.add(schemaElementMatch);
} }
@@ -71,13 +76,13 @@ public class QueryFilterMapper implements SchemaMapper {
} }
private boolean checkExistSameValueSchemaElementMatch(QueryFilter queryFilter, private boolean checkExistSameValueSchemaElementMatch(QueryFilter queryFilter,
List<SchemaElementMatch> schemaElementMatches) { List<SchemaElementMatch> schemaElementMatches) {
List<SchemaElementMatch> valueSchemaElements = schemaElementMatches.stream().filter(schemaElementMatch -> List<SchemaElementMatch> valueSchemaElements = schemaElementMatches.stream().filter(schemaElementMatch ->
SchemaElementType.VALUE.equals(schemaElementMatch.getElement().getType())) SchemaElementType.VALUE.equals(schemaElementMatch.getElement().getType()))
.collect(Collectors.toList()); .collect(Collectors.toList());
for (SchemaElementMatch schemaElementMatch : valueSchemaElements) { for (SchemaElementMatch schemaElementMatch : valueSchemaElements) {
if (schemaElementMatch.getElement().getId().equals(queryFilter.getElementID()) if (schemaElementMatch.getElement().getId().equals(queryFilter.getElementID())
&& schemaElementMatch.getWord().equals(String.valueOf(queryFilter.getValue()))) { && schemaElementMatch.getWord().equals(String.valueOf(queryFilter.getValue()))) {
return true; return true;
} }
} }

View File

@@ -1,10 +1,10 @@
package com.tencent.supersonic.chat.mapper; package com.tencent.supersonic.chat.mapper;
import com.hankcs.hanlp.seg.common.Term; import com.hankcs.hanlp.seg.common.Term;
import com.tencent.supersonic.knowledge.dictionary.MapResult; import com.tencent.supersonic.common.pojo.Constants;
import com.tencent.supersonic.knowledge.dictionary.DictWordType; import com.tencent.supersonic.knowledge.dictionary.DictWordType;
import com.tencent.supersonic.knowledge.dictionary.MapResult;
import com.tencent.supersonic.knowledge.service.SearchService; import com.tencent.supersonic.knowledge.service.SearchService;
import java.util.ArrayList; import java.util.ArrayList;
import java.util.Comparator; import java.util.Comparator;
import java.util.HashMap; import java.util.HashMap;
@@ -32,7 +32,7 @@ public class QueryMatchStrategy implements MatchStrategy {
private MapperHelper mapperHelper; private MapperHelper mapperHelper;
@Override @Override
public Map<MatchText, List<MapResult>> match(String text, List<Term> terms, Long detectDomainId) { public Map<MatchText, List<MapResult>> match(String text, List<Term> terms, Long detectmodelId) {
if (Objects.isNull(terms) || StringUtils.isEmpty(text)) { if (Objects.isNull(terms) || StringUtils.isEmpty(text)) {
return null; return null;
} }
@@ -43,10 +43,10 @@ public class QueryMatchStrategy implements MatchStrategy {
List<Integer> offsetList = terms.stream().sorted(Comparator.comparing(Term::getOffset)) List<Integer> offsetList = terms.stream().sorted(Comparator.comparing(Term::getOffset))
.map(term -> term.getOffset()).collect(Collectors.toList()); .map(term -> term.getOffset()).collect(Collectors.toList());
log.debug("retryCount:{},terms:{},regOffsetToLength:{},offsetList:{},detectDomainId:{}", terms, log.debug("retryCount:{},terms:{},regOffsetToLength:{},offsetList:{},detectmodelId:{}", terms,
regOffsetToLength, offsetList, detectDomainId); regOffsetToLength, offsetList, detectmodelId);
List<MapResult> detects = detect(text, regOffsetToLength, offsetList, detectDomainId); List<MapResult> detects = detect(text, regOffsetToLength, offsetList, detectmodelId);
Map<MatchText, List<MapResult>> result = new HashMap<>(); Map<MatchText, List<MapResult>> result = new HashMap<>();
MatchText matchText = MatchText.builder() MatchText matchText = MatchText.builder()
@@ -58,7 +58,7 @@ public class QueryMatchStrategy implements MatchStrategy {
} }
private List<MapResult> detect(String text, Map<Integer, Integer> regOffsetToLength, List<Integer> offsetList, private List<MapResult> detect(String text, Map<Integer, Integer> regOffsetToLength, List<Integer> offsetList,
Long detectDomainId) { Long detectmodelId) {
List<MapResult> results = Lists.newArrayList(); List<MapResult> results = Lists.newArrayList();
for (Integer index = 0; index <= text.length() - 1; ) { for (Integer index = 0; index <= text.length() - 1; ) {
@@ -69,18 +69,44 @@ public class QueryMatchStrategy implements MatchStrategy {
int offset = mapperHelper.getStepOffset(offsetList, index); int offset = mapperHelper.getStepOffset(offsetList, index);
i = mapperHelper.getStepIndex(regOffsetToLength, i); i = mapperHelper.getStepIndex(regOffsetToLength, i);
if (i <= text.length()) { if (i <= text.length()) {
List<MapResult> mapResults = detectByStep(text, detectDomainId, index, i, offset); List<MapResult> mapResults = detectByStep(text, detectmodelId, index, i, offset);
mapResultRowSet.addAll(mapResults); selectMapResultInOneRound(mapResultRowSet, mapResults);
} }
} }
index = mapperHelper.getStepIndex(regOffsetToLength, index); index = mapperHelper.getStepIndex(regOffsetToLength, index);
results.addAll(mapResultRowSet); results.addAll(mapResultRowSet);
} }
return results; return results;
} }
private List<MapResult> detectByStep(String text, Long detectDomainId, Integer index, Integer i, int offset) { private void selectMapResultInOneRound(Set<MapResult> mapResultRowSet, List<MapResult> mapResults) {
for (MapResult mapResult : mapResults) {
if (mapResultRowSet.contains(mapResult)) {
boolean isDeleted = mapResultRowSet.removeIf(
entry -> {
boolean deleted = getMapKey(mapResult).equals(getMapKey(entry))
&& entry.getDetectWord().length() < mapResult.getDetectWord().length();
if (deleted) {
log.info("deleted entry:{}", entry);
}
return deleted;
}
);
if (isDeleted) {
log.info("deleted, add mapResult:{}", mapResult);
mapResultRowSet.add(mapResult);
}
} else {
mapResultRowSet.add(mapResult);
}
}
}
private String getMapKey(MapResult a) {
return a.getName() + Constants.UNDERLINE + String.join(Constants.UNDERLINE, a.getNatures());
}
private List<MapResult> detectByStep(String text, Long detectmodelId, Integer index, Integer i, int offset) {
String detectSegment = text.substring(index, i); String detectSegment = text.substring(index, i);
Integer oneDetectionSize = mapperHelper.getOneDetectionSize(); Integer oneDetectionSize = mapperHelper.getOneDetectionSize();
// step1. pre search // step1. pre search
@@ -100,17 +126,17 @@ public class QueryMatchStrategy implements MatchStrategy {
mapResults = mapResults.stream().sorted((a, b) -> -(b.getName().length() - a.getName().length())) mapResults = mapResults.stream().sorted((a, b) -> -(b.getName().length() - a.getName().length()))
.collect(Collectors.toCollection(LinkedHashSet::new)); .collect(Collectors.toCollection(LinkedHashSet::new));
// step4. filter by classId // step4. filter by classId
if (Objects.nonNull(detectDomainId) && detectDomainId > 0) { if (Objects.nonNull(detectmodelId) && detectmodelId > 0) {
log.debug("detectDomainId:{}, before parseResults:{}", mapResults); log.debug("detectmodelId:{}, before parseResults:{}", mapResults);
mapResults = mapResults.stream().map(entry -> { mapResults = mapResults.stream().map(entry -> {
List<String> natures = entry.getNatures().stream().filter( List<String> natures = entry.getNatures().stream().filter(
nature -> nature.startsWith(DictWordType.NATURE_SPILT + detectDomainId) || (nature.startsWith( nature -> nature.startsWith(DictWordType.NATURE_SPILT + detectmodelId) || (nature.startsWith(
DictWordType.NATURE_SPILT)) DictWordType.NATURE_SPILT))
).collect(Collectors.toList()); ).collect(Collectors.toList());
entry.setNatures(natures); entry.setNatures(natures);
return entry; return entry;
}).collect(Collectors.toCollection(LinkedHashSet::new)); }).collect(Collectors.toCollection(LinkedHashSet::new));
log.info("after domainId parseResults:{}", mapResults); log.info("after modelId parseResults:{}", mapResults);
} }
// step5. filter by similarity // step5. filter by similarity
mapResults = mapResults.stream() mapResults = mapResults.stream()

View File

@@ -2,10 +2,9 @@ package com.tencent.supersonic.chat.mapper;
import com.google.common.collect.Lists; import com.google.common.collect.Lists;
import com.hankcs.hanlp.seg.common.Term; import com.hankcs.hanlp.seg.common.Term;
import com.tencent.supersonic.knowledge.dictionary.MapResult;
import com.tencent.supersonic.knowledge.dictionary.DictWordType; import com.tencent.supersonic.knowledge.dictionary.DictWordType;
import com.tencent.supersonic.knowledge.dictionary.MapResult;
import com.tencent.supersonic.knowledge.service.SearchService; import com.tencent.supersonic.knowledge.service.SearchService;
import java.util.List; import java.util.List;
import java.util.Map; import java.util.Map;
import java.util.Objects; import java.util.Objects;
@@ -25,7 +24,7 @@ public class SearchMatchStrategy implements MatchStrategy {
@Override @Override
public Map<MatchText, List<MapResult>> match(String text, List<Term> originals, public Map<MatchText, List<MapResult>> match(String text, List<Term> originals,
Long detectDomainId) { Long detectModelId) {
Map<Integer, Integer> regOffsetToLength = originals.stream() Map<Integer, Integer> regOffsetToLength = originals.stream()
.filter(entry -> !entry.nature.toString().startsWith(DictWordType.NATURE_SPILT)) .filter(entry -> !entry.nature.toString().startsWith(DictWordType.NATURE_SPILT))
@@ -60,10 +59,10 @@ public class SearchMatchStrategy implements MatchStrategy {
List<String> natures = entry.getNatures().stream() List<String> natures = entry.getNatures().stream()
.filter(nature -> !nature.endsWith(DictWordType.ENTITY.getType())) .filter(nature -> !nature.endsWith(DictWordType.ENTITY.getType()))
.filter(nature -> { .filter(nature -> {
if (Objects.isNull(detectDomainId) || detectDomainId <= 0) { if (Objects.isNull(detectModelId) || detectModelId <= 0) {
return true; return true;
} }
if (nature.startsWith(DictWordType.NATURE_SPILT + detectDomainId) if (nature.startsWith(DictWordType.NATURE_SPILT + detectModelId)
&& nature.startsWith(DictWordType.NATURE_SPILT)) { && nature.startsWith(DictWordType.NATURE_SPILT)) {
return true; return true;
} }

View File

@@ -2,22 +2,9 @@ package com.tencent.supersonic.chat.parser;
import com.tencent.supersonic.chat.api.component.SemanticQuery; import com.tencent.supersonic.chat.api.component.SemanticQuery;
import com.tencent.supersonic.chat.api.pojo.*; import com.tencent.supersonic.chat.api.pojo.QueryContext;
import com.tencent.supersonic.chat.plugin.PluginParseResult; import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
import com.tencent.supersonic.chat.query.plugin.PluginSemanticQuery;
import com.tencent.supersonic.chat.query.rule.RuleSemanticQuery;
import com.tencent.supersonic.common.pojo.Constants;
import com.tencent.supersonic.common.pojo.DateConf;
import com.tencent.supersonic.common.util.JsonUtil;
import lombok.extern.slf4j.Slf4j; import lombok.extern.slf4j.Slf4j;
import org.apache.commons.compress.utils.Lists;
import org.apache.commons.lang3.StringUtils;
import org.springframework.util.CollectionUtils;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.stream.Collectors;
/** /**
* This checker can be used by semantic parsers to check if query intent * This checker can be used by semantic parsers to check if query intent
@@ -30,86 +17,20 @@ public class SatisfactionChecker {
private static final double LONG_TEXT_THRESHOLD = 0.8; private static final double LONG_TEXT_THRESHOLD = 0.8;
private static final double SHORT_TEXT_THRESHOLD = 0.5; private static final double SHORT_TEXT_THRESHOLD = 0.5;
private static final int QUERY_TEXT_LENGTH_THRESHOLD = 10; private static final int QUERY_TEXT_LENGTH_THRESHOLD = 10;
public static final double EMBEDDING_THRESHOLD = 0.2;
// check all the parse info in candidate // check all the parse info in candidate
public static boolean check(QueryContext queryCtx) { public static boolean check(QueryContext queryContext) {
for (SemanticQuery query : queryCtx.getCandidateQueries()) { for (SemanticQuery query : queryContext.getCandidateQueries()) {
if (query instanceof RuleSemanticQuery) { if (checkThreshold(queryContext.getRequest().getQueryText(), query.getParseInfo())) {
if (checkRuleThreshHold(queryCtx.getRequest().getQueryText(), query.getParseInfo())) { return true;
return true;
}
} else if (query instanceof PluginSemanticQuery) {
if (checkEmbeddingThreshold(query.getParseInfo())) {
log.info("query mode :{} satisfy check", query.getQueryMode());
return true;
}
} }
} }
return false; return false;
} }
private static boolean checkEmbeddingThreshold(SemanticParseInfo semanticParseInfo) { private static boolean checkThreshold(String queryText, SemanticParseInfo semanticParseInfo) {
Object object = semanticParseInfo.getProperties().get(Constants.CONTEXT);
PluginParseResult pluginParseResult = JsonUtil.toObject(JsonUtil.toString(object), PluginParseResult.class);
return EMBEDDING_THRESHOLD > pluginParseResult.getDistance();
}
//check single parse info
private static boolean checkRuleThreshHold(String text, SemanticParseInfo semanticParseInfo) {
List<SchemaElementMatch> schemaElementMatches = semanticParseInfo.getElementMatches();
if (CollectionUtils.isEmpty(schemaElementMatches)) {
return false;
}
List<String> detectWords = Lists.newArrayList();
Map<Long, String> detectWordMap = schemaElementMatches.stream()
.collect(Collectors.toMap(m -> m.getElement().getId(), SchemaElementMatch::getDetectWord,
(id1, id2) -> id1));
// get detect word in text by element id in semantic layer
Long domainId = semanticParseInfo.getDomainId();
if (domainId != null && domainId > 0) {
for (SchemaElementMatch schemaElementMatch : schemaElementMatches) {
if (SchemaElementType.DOMAIN.equals(schemaElementMatch.getElement().getType())) {
detectWords.add(schemaElementMatch.getDetectWord());
}
}
}
for (SchemaElementMatch schemaElementMatch : schemaElementMatches) {
if (SchemaElementType.VALUE.equals(schemaElementMatch.getElement().getType())) {
detectWords.add(schemaElementMatch.getDetectWord());
}
}
for (SchemaElementMatch schemaElementMatch : schemaElementMatches) {
if (SchemaElementType.ID.equals(schemaElementMatch.getElement().getType())) {
detectWords.add(schemaElementMatch.getDetectWord());
}
}
for (SchemaElement schemaItem : semanticParseInfo.getMetrics()) {
detectWords.add(
detectWordMap.getOrDefault(Optional.ofNullable(schemaItem.getId()).orElse(0L), ""));
// only first metric
break;
}
for (SchemaElement schemaItem : semanticParseInfo.getDimensions()) {
detectWords.add(
detectWordMap.getOrDefault(Optional.ofNullable(schemaItem.getId()).orElse(0L), ""));
// only first dimension
break;
}
String dateText = Optional.ofNullable(semanticParseInfo.getDateInfo()).orElse(new DateConf()).getText();
if (StringUtils.isNotBlank(dateText) && !dateText.equalsIgnoreCase(Constants.NULL)) {
detectWords.add(dateText);
}
detectWords.removeIf(word -> !text.contains(word) && !text.contains(StringUtils.reverse(word)));
//compare the length between detect words and query text
return checkThreshold(text, detectWords, semanticParseInfo);
}
private static boolean checkThreshold(String queryText, List<String> detectWords, SemanticParseInfo semanticParseInfo) {
String detectWordsDistinct = StringUtils.join(new HashSet<>(detectWords), "");
int detectWordsLength = detectWordsDistinct.length();
int queryTextLength = queryText.length(); int queryTextLength = queryText.length();
double degree = detectWordsLength * 1.0 / queryTextLength; double degree = semanticParseInfo.getScore() / queryTextLength;
if (queryTextLength > QUERY_TEXT_LENGTH_THRESHOLD) { if (queryTextLength > QUERY_TEXT_LENGTH_THRESHOLD) {
if (degree < LONG_TEXT_THRESHOLD) { if (degree < LONG_TEXT_THRESHOLD) {
return false; return false;
@@ -117,8 +38,8 @@ public class SatisfactionChecker {
} else if (degree < SHORT_TEXT_THRESHOLD) { } else if (degree < SHORT_TEXT_THRESHOLD) {
return false; return false;
} }
log.info("queryMode:{}, degree:{}, detectWords:{}, parse info:{}", log.info("queryMode:{}, degree:{}, parse info:{}",
semanticParseInfo.getQueryMode(), degree, detectWordsDistinct, semanticParseInfo); semanticParseInfo.getQueryMode(), degree, semanticParseInfo);
return true; return true;
} }

View File

@@ -1,13 +1,18 @@
package com.tencent.supersonic.chat.parser.embedding; package com.tencent.supersonic.chat.parser.embedding;
import com.google.common.collect.Lists; import com.google.common.collect.Lists;
import com.google.common.collect.Sets;
import com.tencent.supersonic.chat.api.component.SemanticParser; import com.tencent.supersonic.chat.api.component.SemanticParser;
import com.tencent.supersonic.chat.api.pojo.*; import com.tencent.supersonic.chat.api.pojo.ChatContext;
import com.tencent.supersonic.chat.api.pojo.ModelSchema;
import com.tencent.supersonic.chat.api.pojo.QueryContext;
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
import com.tencent.supersonic.chat.api.pojo.SchemaMapInfo;
import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
import com.tencent.supersonic.chat.api.pojo.request.QueryFilter; import com.tencent.supersonic.chat.api.pojo.request.QueryFilter;
import com.tencent.supersonic.chat.api.pojo.request.QueryFilters;
import com.tencent.supersonic.chat.api.pojo.request.QueryReq; import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
import com.tencent.supersonic.chat.parser.SatisfactionChecker; import com.tencent.supersonic.chat.parser.ParseMode;
import com.tencent.supersonic.chat.plugin.Plugin; import com.tencent.supersonic.chat.plugin.Plugin;
import com.tencent.supersonic.chat.plugin.PluginManager; import com.tencent.supersonic.chat.plugin.PluginManager;
import com.tencent.supersonic.chat.plugin.PluginParseResult; import com.tencent.supersonic.chat.plugin.PluginParseResult;
@@ -17,18 +22,20 @@ import com.tencent.supersonic.chat.service.PluginService;
import com.tencent.supersonic.chat.service.SemanticService; import com.tencent.supersonic.chat.service.SemanticService;
import com.tencent.supersonic.common.pojo.Constants; import com.tencent.supersonic.common.pojo.Constants;
import com.tencent.supersonic.common.util.ContextUtils; import com.tencent.supersonic.common.util.ContextUtils;
import java.util.*;
import java.util.stream.Collectors;
import com.tencent.supersonic.semantic.api.query.enums.FilterOperatorEnum; import com.tencent.supersonic.semantic.api.query.enums.FilterOperatorEnum;
import java.util.Comparator;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;
import lombok.extern.slf4j.Slf4j; import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils; import org.apache.commons.lang3.StringUtils;
import org.apache.commons.lang3.tuple.Pair;
import org.springframework.util.CollectionUtils; import org.springframework.util.CollectionUtils;
@Slf4j @Slf4j
public class EmbeddingBasedParser implements SemanticParser { public class EmbeddingBasedParser implements SemanticParser {
private final static double THRESHOLD = 0.2d;
@Override @Override
public void parse(QueryContext queryContext, ChatContext chatContext) { public void parse(QueryContext queryContext, ChatContext chatContext) {
EmbeddingConfig embeddingConfig = ContextUtils.getBean(EmbeddingConfig.class); EmbeddingConfig embeddingConfig = ContextUtils.getBean(EmbeddingConfig.class);
@@ -36,109 +43,15 @@ public class EmbeddingBasedParser implements SemanticParser {
return; return;
} }
log.info("EmbeddingBasedParser parser query ctx: {}, chat ctx: {}", queryContext, chatContext); log.info("EmbeddingBasedParser parser query ctx: {}, chat ctx: {}", queryContext, chatContext);
Set<Long> domainIds = getDomainMatched(queryContext);
String text = queryContext.getRequest().getQueryText(); String text = queryContext.getRequest().getQueryText();
if (!CollectionUtils.isEmpty(domainIds)) { List<RecallRetrieval> embeddingRetrievals = recallResult(text);
for (Long domainId : domainIds) { choosePlugin(embeddingRetrievals, queryContext);
List<SchemaElementMatch> schemaElementMatches = getMatchedElements(queryContext, domainId);
String textReplaced = replaceText(text, schemaElementMatches);
List<RecallRetrieval> embeddingRetrievals = recallResult(textReplaced, hasCandidateQuery(queryContext));
Optional<Plugin> pluginOptional = choosePlugin(embeddingRetrievals, domainId);
log.info("domain id :{} embedding result, text:{} embeddingResp:{} ",domainId, textReplaced, embeddingRetrievals);
pluginOptional.ifPresent(plugin -> buildQuery(plugin, embeddingRetrievals, domainId, textReplaced, queryContext, schemaElementMatches));
}
} else {
List<RecallRetrieval> embeddingRetrievals = recallResult(text, hasCandidateQuery(queryContext));
Optional<Plugin> pluginOptional = choosePlugin(embeddingRetrievals, null);
pluginOptional.ifPresent(plugin -> buildQuery(plugin, embeddingRetrievals, null, text, queryContext, Lists.newArrayList()));
}
} }
private void buildQuery(Plugin plugin, List<RecallRetrieval> embeddingRetrievals, private void choosePlugin(List<RecallRetrieval> embeddingRetrievals,
Long domainId, String text, QueryContext queryContext) {
QueryContext queryContext, List<SchemaElementMatch> schemaElementMatches) {
Map<String, RecallRetrieval> embeddingRetrievalMap = embeddingRetrievals.stream()
.collect(Collectors.toMap(RecallRetrieval::getId, e -> e, (value1, value2) -> value1));
log.info("EmbeddingBasedParser text: {} domain: {} choose plugin: [{} {}]",
text, domainId, plugin.getId(), plugin.getName());
PluginSemanticQuery pluginQuery = QueryManager.createPluginQuery(plugin.getType());
SemanticParseInfo semanticParseInfo = buildSemanticParseInfo(domainId, plugin, text,
queryContext.getRequest(), embeddingRetrievalMap, schemaElementMatches);
semanticParseInfo.setQueryMode(pluginQuery.getQueryMode());
pluginQuery.setParseInfo(semanticParseInfo);
queryContext.getCandidateQueries().add(pluginQuery);
}
private Set<Long> getDomainMatched(QueryContext queryContext) {
Long queryDomainId = queryContext.getRequest().getDomainId();
if (queryDomainId != null && queryDomainId > 0) {
return Sets.newHashSet(queryDomainId);
}
return queryContext.getMapInfo().getMatchedDomains();
}
private SemanticParseInfo buildSemanticParseInfo(Long domainId, Plugin plugin, String text, QueryReq queryReq,
Map<String, RecallRetrieval> embeddingRetrievalMap,
List<SchemaElementMatch> schemaElementMatches) {
SchemaElement schemaElement = new SchemaElement();
schemaElement.setDomain(domainId);
schemaElement.setId(domainId);
SemanticParseInfo semanticParseInfo = new SemanticParseInfo();
semanticParseInfo.setElementMatches(schemaElementMatches);
semanticParseInfo.setDomain(schemaElement);
double distance = Double.parseDouble(embeddingRetrievalMap.get(plugin.getId().toString()).getDistance());
double score = text.length() * (1 - distance);
Map<String, Object> properties = new HashMap<>();
PluginParseResult pluginParseResult = new PluginParseResult();
pluginParseResult.setPlugin(plugin);
pluginParseResult.setRequest(queryReq);
pluginParseResult.setDistance(distance);
properties.put(Constants.CONTEXT, pluginParseResult);
semanticParseInfo.setProperties(properties);
semanticParseInfo.setScore(score);
fillSemanticParseInfo(semanticParseInfo);
setEntity(domainId, semanticParseInfo);
return semanticParseInfo;
}
private List<SchemaElementMatch> getMatchedElements(QueryContext queryContext, Long domainId) {
SchemaMapInfo schemaMapInfo = queryContext.getMapInfo();
List<SchemaElementMatch> schemaElementMatches = schemaMapInfo.getMatchedElements(domainId);
if (schemaElementMatches == null) {
return Lists.newArrayList();
}
QueryReq queryReq = queryContext.getRequest();
QueryFilters queryFilters = queryReq.getQueryFilters();
if (queryFilters == null || CollectionUtils.isEmpty(queryFilters.getFilters())) {
return schemaElementMatches;
}
Map<Long, Object> element = queryFilters.getFilters().stream()
.collect(Collectors.toMap(QueryFilter::getElementID, QueryFilter::getValue, (v1, v2) -> v1));
return schemaElementMatches.stream().filter(schemaElementMatch ->
SchemaElementType.VALUE.equals(schemaElementMatch.getElement().getType())
|| SchemaElementType.ID.equals(schemaElementMatch.getElement().getType())
|| SchemaElementType.ENTITY.equals(schemaElementMatch.getElement().getType()))
.filter(schemaElementMatch ->
!element.containsKey(schemaElementMatch.getElement().getId()) || (
element.containsKey(schemaElementMatch.getElement().getId()) &&
element.get(schemaElementMatch.getElement().getId()).toString()
.equalsIgnoreCase(schemaElementMatch.getWord())
))
.collect(Collectors.toList());
}
private void setEntity(Long domainId, SemanticParseInfo semanticParseInfo) {
SemanticService semanticService = ContextUtils.getBean(SemanticService.class);
DomainSchema domainSchema = semanticService.getDomainSchema(domainId);
if (domainSchema != null && domainSchema.getEntity() != null) {
semanticParseInfo.setEntity(domainSchema.getEntity());
}
}
private Optional<Plugin> choosePlugin(List<RecallRetrieval> embeddingRetrievals,
Long domainId) {
if (CollectionUtils.isEmpty(embeddingRetrievals)) { if (CollectionUtils.isEmpty(embeddingRetrievals)) {
return Optional.empty(); return;
} }
PluginService pluginService = ContextUtils.getBean(PluginService.class); PluginService pluginService = ContextUtils.getBean(PluginService.class);
List<Plugin> plugins = pluginService.getPluginList(); List<Plugin> plugins = pluginService.getPluginList();
@@ -148,27 +61,92 @@ public class EmbeddingBasedParser implements SemanticParser {
if (plugin == null) { if (plugin == null) {
continue; continue;
} }
if (domainId == null) { Pair<Boolean, List<Long>> pair = PluginManager.resolve(plugin, queryContext);
return Optional.of(plugin); log.info("embedding plugin resolve: {}", pair);
} if (pair.getLeft()) {
if (!CollectionUtils.isEmpty(plugin.getDomainList()) && plugin.getDomainList().contains(domainId)) { List<Long> modelList = pair.getRight();
return Optional.of(plugin); if (CollectionUtils.isEmpty(modelList)) {
return;
}
modelList = distinctModelList(plugin, queryContext.getMapInfo(), modelList);
for (Long modelId : modelList) {
buildQuery(plugin, Double.parseDouble(embeddingRetrieval.getDistance()), modelId, queryContext,
queryContext.getMapInfo().getMatchedElements(modelId));
}
return;
} }
} }
return Optional.empty();
} }
public List<RecallRetrieval> recallResult(String embeddingText, boolean hasCandidateQuery) { public List<Long> distinctModelList(Plugin plugin, SchemaMapInfo schemaMapInfo, List<Long> modelList) {
if (!plugin.isContainsAllModel()) {
return modelList;
}
boolean noElementMatch = true;
for (Long model : modelList) {
List<SchemaElementMatch> schemaElementMatches = schemaMapInfo.getMatchedElements(model);
if (!CollectionUtils.isEmpty(schemaElementMatches)) {
noElementMatch = false;
}
}
if (noElementMatch) {
return modelList.subList(0, 1);
}
return modelList;
}
private void buildQuery(Plugin plugin, double distance, Long modelId,
QueryContext queryContext, List<SchemaElementMatch> schemaElementMatches) {
log.info("EmbeddingBasedParser Model: {} choose plugin: [{} {}]", modelId, plugin.getId(), plugin.getName());
PluginSemanticQuery pluginQuery = QueryManager.createPluginQuery(plugin.getType());
plugin.setParseMode(ParseMode.EMBEDDING_RECALL);
SemanticParseInfo semanticParseInfo = buildSemanticParseInfo(modelId, plugin, queryContext.getRequest(),
schemaElementMatches, distance);
double score = queryContext.getRequest().getQueryText().length() * (1 - distance);
semanticParseInfo.setQueryMode(pluginQuery.getQueryMode());
semanticParseInfo.setScore(score);
pluginQuery.setParseInfo(semanticParseInfo);
queryContext.getCandidateQueries().add(pluginQuery);
}
private SemanticParseInfo buildSemanticParseInfo(Long modelId, Plugin plugin, QueryReq queryReq,
List<SchemaElementMatch> schemaElementMatches, double distance) {
if (modelId == null && !CollectionUtils.isEmpty(plugin.getModelList())) {
modelId = plugin.getModelList().get(0);
}
SchemaElement Model = new SchemaElement();
Model.setModel(modelId);
Model.setId(modelId);
SemanticParseInfo semanticParseInfo = new SemanticParseInfo();
semanticParseInfo.setElementMatches(schemaElementMatches);
semanticParseInfo.setModel(Model);
Map<String, Object> properties = new HashMap<>();
PluginParseResult pluginParseResult = new PluginParseResult();
pluginParseResult.setPlugin(plugin);
pluginParseResult.setRequest(queryReq);
pluginParseResult.setDistance(distance);
properties.put(Constants.CONTEXT, pluginParseResult);
semanticParseInfo.setProperties(properties);
semanticParseInfo.setScore(distance);
fillSemanticParseInfo(semanticParseInfo);
setEntity(modelId, semanticParseInfo);
return semanticParseInfo;
}
private void setEntity(Long modelId, SemanticParseInfo semanticParseInfo) {
SemanticService semanticService = ContextUtils.getBean(SemanticService.class);
ModelSchema ModelSchema = semanticService.getModelSchema(modelId);
if (ModelSchema != null && ModelSchema.getEntity() != null) {
semanticParseInfo.setEntity(ModelSchema.getEntity());
}
}
public List<RecallRetrieval> recallResult(String embeddingText) {
try { try {
PluginManager pluginManager = ContextUtils.getBean(PluginManager.class); PluginManager pluginManager = ContextUtils.getBean(PluginManager.class);
EmbeddingResp embeddingResp = pluginManager.recognize(embeddingText); EmbeddingResp embeddingResp = pluginManager.recognize(embeddingText);
List<RecallRetrieval> embeddingRetrievals = embeddingResp.getRetrieval(); List<RecallRetrieval> embeddingRetrievals = embeddingResp.getRetrieval();
if(!CollectionUtils.isEmpty(embeddingRetrievals)){ if (!CollectionUtils.isEmpty(embeddingRetrievals)) {
if (hasCandidateQuery) {
embeddingRetrievals = embeddingRetrievals.stream()
.filter(llmRetrieval -> Double.parseDouble(llmRetrieval.getDistance())<THRESHOLD)
.collect(Collectors.toList());
}
embeddingRetrievals = embeddingRetrievals.stream().sorted(Comparator.comparingDouble(o -> embeddingRetrievals = embeddingRetrievals.stream().sorted(Comparator.comparingDouble(o ->
Math.abs(Double.parseDouble(o.getDistance())))).collect(Collectors.toList()); Math.abs(Double.parseDouble(o.getDistance())))).collect(Collectors.toList());
embeddingResp.setRetrieval(embeddingRetrievals); embeddingResp.setRetrieval(embeddingRetrievals);
@@ -180,45 +158,22 @@ public class EmbeddingBasedParser implements SemanticParser {
return Lists.newArrayList(); return Lists.newArrayList();
} }
private boolean hasCandidateQuery(QueryContext queryContext) {
return !CollectionUtils.isEmpty(queryContext.getCandidateQueries());
}
private void fillSemanticParseInfo(SemanticParseInfo semanticParseInfo) { private void fillSemanticParseInfo(SemanticParseInfo semanticParseInfo) {
List<SchemaElementMatch> schemaElementMatches = semanticParseInfo.getElementMatches(); List<SchemaElementMatch> schemaElementMatches = semanticParseInfo.getElementMatches();
if (!CollectionUtils.isEmpty(schemaElementMatches)) { if (!CollectionUtils.isEmpty(schemaElementMatches)) {
schemaElementMatches.stream().filter(schemaElementMatch -> schemaElementMatches.stream().filter(schemaElementMatch ->
SchemaElementType.VALUE.equals(schemaElementMatch.getElement().getType()) SchemaElementType.VALUE.equals(schemaElementMatch.getElement().getType())
|| SchemaElementType.ID.equals(schemaElementMatch.getElement().getType())) || SchemaElementType.ID.equals(schemaElementMatch.getElement().getType()))
.forEach(schemaElementMatch -> { .forEach(schemaElementMatch -> {
QueryFilter queryFilter = new QueryFilter(); QueryFilter queryFilter = new QueryFilter();
queryFilter.setValue(schemaElementMatch.getWord()); queryFilter.setValue(schemaElementMatch.getWord());
queryFilter.setElementID(schemaElementMatch.getElement().getId()); queryFilter.setElementID(schemaElementMatch.getElement().getId());
queryFilter.setName(schemaElementMatch.getElement().getName()); queryFilter.setName(schemaElementMatch.getElement().getName());
queryFilter.setOperator(FilterOperatorEnum.EQUALS); queryFilter.setOperator(FilterOperatorEnum.EQUALS);
queryFilter.setBizName(schemaElementMatch.getElement().getBizName()); queryFilter.setBizName(schemaElementMatch.getElement().getBizName());
semanticParseInfo.getDimensionFilters().add(queryFilter); semanticParseInfo.getDimensionFilters().add(queryFilter);
}); });
} }
} }
protected String replaceText(String text, List<SchemaElementMatch> schemaElementMatches) {
if (CollectionUtils.isEmpty(schemaElementMatches)) {
return text;
}
List<SchemaElementMatch> valueSchemaElementMatches = schemaElementMatches.stream()
.filter(schemaElementMatch ->
SchemaElementType.VALUE.equals(schemaElementMatch.getElement().getType())
|| SchemaElementType.ID.equals(schemaElementMatch.getElement().getType()))
.collect(Collectors.toList());
for (SchemaElementMatch schemaElementMatch : valueSchemaElementMatches) {
String detectWord = schemaElementMatch.getDetectWord();
if (StringUtils.isBlank(detectWord)) {
continue;
}
text = text.replace(detectWord, "");
}
return text;
}
} }

View File

@@ -1,23 +1,26 @@
package com.tencent.supersonic.chat.parser.embedding; package com.tencent.supersonic.chat.parser.embedding;
import com.alibaba.fastjson.JSONObject; import com.alibaba.fastjson.JSONObject;
import com.tencent.supersonic.chat.api.pojo.*; import com.tencent.supersonic.chat.api.pojo.ChatContext;
import com.tencent.supersonic.chat.api.pojo.QueryContext;
import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
import com.tencent.supersonic.chat.api.pojo.SchemaMapInfo;
import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
import com.tencent.supersonic.chat.api.pojo.request.QueryFilter; import com.tencent.supersonic.chat.api.pojo.request.QueryFilter;
import com.tencent.supersonic.chat.api.pojo.request.QueryFilters; import com.tencent.supersonic.chat.api.pojo.request.QueryFilters;
import com.tencent.supersonic.chat.service.ConfigService; import com.tencent.supersonic.chat.service.ConfigService;
import java.util.List;
import java.util.Objects;
import java.util.Set;
import lombok.extern.slf4j.Slf4j; import lombok.extern.slf4j.Slf4j;
import org.apache.commons.collections.CollectionUtils; import org.apache.commons.collections.CollectionUtils;
import org.apache.commons.lang3.StringUtils; import org.apache.commons.lang3.StringUtils;
import org.springframework.stereotype.Component; import org.springframework.stereotype.Component;
import java.util.List;
import java.util.Objects;
import java.util.Set;
@Slf4j @Slf4j
@Component("EmbeddingEntityResolver") @Component("EmbeddingEntityResolver")
public class EmbeddingEntityResolver { public class EmbeddingEntityResolver {
private ConfigService configService; private ConfigService configService;
public EmbeddingEntityResolver(ConfigService configService) { public EmbeddingEntityResolver(ConfigService configService) {
@@ -25,18 +28,19 @@ public class EmbeddingEntityResolver {
} }
private Long getEntityValue(Long domainId, Long entityElementId, QueryContext queryCtx, ChatContext chatCtx) { private Long getEntityValue(Long modelId, Long entityElementId, QueryContext queryCtx, ChatContext chatCtx) {
Long entityId = null; Long entityId = null;
QueryFilters queryFilters = queryCtx.getRequest().getQueryFilters(); QueryFilters queryFilters = queryCtx.getRequest().getQueryFilters();
if (queryFilters != null) { if (queryFilters != null) {
entityId = getEntityValueFromQueryFilter(queryFilters.getFilters()); entityId = getEntityValueFromQueryFilter(queryFilters.getFilters());
if (entityId != null) { if (entityId != null) {
log.info("get entity id:{} domain id:{} from query filter :{} ", entityId, domainId, queryFilters); log.info("get entity id:{} model id:{} from query filter :{} ", entityId, modelId, queryFilters);
return entityId; return entityId;
} }
} }
entityId = getEntityValueFromSchemaMapInfo(domainId, queryCtx.getMapInfo(), entityElementId); entityId = getEntityValueFromSchemaMapInfo(modelId, queryCtx.getMapInfo(), entityElementId);
log.info("get entity id:{} from schema map Info :{} ", entityId, JSONObject.toJSONString(queryCtx.getMapInfo())); log.info("get entity id:{} from schema map Info :{} ", entityId,
JSONObject.toJSONString(queryCtx.getMapInfo()));
if (entityId == null || entityId == 0) { if (entityId == null || entityId == 0) {
Long entityIdFromChat = getEntityValueFromParseInfo(chatCtx.getParseInfo(), entityElementId); Long entityIdFromChat = getEntityValueFromParseInfo(chatCtx.getParseInfo(), entityElementId);
if (entityIdFromChat != null && entityIdFromChat > 0) { if (entityIdFromChat != null && entityIdFromChat > 0) {
@@ -75,8 +79,8 @@ public class EmbeddingEntityResolver {
} }
private Long getEntityValueFromSchemaMapInfo(Long domainId, SchemaMapInfo schemaMapInfo, Long entityElementId) { private Long getEntityValueFromSchemaMapInfo(Long modelId, SchemaMapInfo schemaMapInfo, Long entityElementId) {
List<SchemaElementMatch> schemaElementMatchList = schemaMapInfo.getMatchedElements(domainId); List<SchemaElementMatch> schemaElementMatchList = schemaMapInfo.getMatchedElements(modelId);
if (CollectionUtils.isEmpty(schemaElementMatchList)) { if (CollectionUtils.isEmpty(schemaElementMatchList)) {
return null; return null;
} }

View File

@@ -1,9 +1,8 @@
package com.tencent.supersonic.chat.parser.embedding; package com.tencent.supersonic.chat.parser.embedding;
import lombok.Data;
import java.util.List; import java.util.List;
import lombok.Data;
@Data @Data
public class EmbeddingResp { public class EmbeddingResp {

View File

@@ -7,18 +7,20 @@ import com.tencent.supersonic.chat.api.pojo.QueryContext;
import com.tencent.supersonic.chat.api.pojo.SchemaElement; import com.tencent.supersonic.chat.api.pojo.SchemaElement;
import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo; import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
import com.tencent.supersonic.chat.config.FunctionCallInfoConfig; import com.tencent.supersonic.chat.config.FunctionCallInfoConfig;
import com.tencent.supersonic.chat.parser.ParseMode;
import com.tencent.supersonic.chat.parser.SatisfactionChecker; import com.tencent.supersonic.chat.parser.SatisfactionChecker;
import com.tencent.supersonic.chat.plugin.Plugin; import com.tencent.supersonic.chat.plugin.Plugin;
import com.tencent.supersonic.chat.plugin.PluginManager; import com.tencent.supersonic.chat.plugin.PluginManager;
import com.tencent.supersonic.chat.plugin.PluginParseConfig; import com.tencent.supersonic.chat.plugin.PluginParseConfig;
import com.tencent.supersonic.chat.plugin.PluginParseResult; import com.tencent.supersonic.chat.plugin.PluginParseResult;
import com.tencent.supersonic.chat.query.QueryManager; import com.tencent.supersonic.chat.query.QueryManager;
import com.tencent.supersonic.chat.query.dsl.DSLQuery;
import com.tencent.supersonic.chat.query.plugin.PluginSemanticQuery; import com.tencent.supersonic.chat.query.plugin.PluginSemanticQuery;
import com.tencent.supersonic.chat.query.plugin.dsl.DSLQuery;
import com.tencent.supersonic.chat.service.PluginService; import com.tencent.supersonic.chat.service.PluginService;
import com.tencent.supersonic.chat.utils.ComponentFactory; import com.tencent.supersonic.chat.utils.ComponentFactory;
import com.tencent.supersonic.common.pojo.Constants; import com.tencent.supersonic.common.pojo.Constants;
import com.tencent.supersonic.common.util.ContextUtils; import com.tencent.supersonic.common.util.ContextUtils;
import com.tencent.supersonic.common.util.JsonUtil;
import java.net.URI; import java.net.URI;
import java.util.ArrayList; import java.util.ArrayList;
import java.util.HashMap; import java.util.HashMap;
@@ -29,10 +31,9 @@ import java.util.Objects;
import java.util.Optional; import java.util.Optional;
import java.util.Set; import java.util.Set;
import java.util.stream.Collectors; import java.util.stream.Collectors;
import com.tencent.supersonic.common.util.JsonUtil;
import lombok.extern.slf4j.Slf4j; import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils; import org.apache.commons.lang3.StringUtils;
import org.apache.commons.lang3.tuple.Pair;
import org.springframework.http.HttpEntity; import org.springframework.http.HttpEntity;
import org.springframework.http.HttpHeaders; import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpMethod; import org.springframework.http.HttpMethod;
@@ -55,21 +56,12 @@ public class FunctionBasedParser implements SemanticParser {
FunctionCallInfoConfig functionCallConfig = ContextUtils.getBean(FunctionCallInfoConfig.class); FunctionCallInfoConfig functionCallConfig = ContextUtils.getBean(FunctionCallInfoConfig.class);
PluginService pluginService = ContextUtils.getBean(PluginService.class); PluginService pluginService = ContextUtils.getBean(PluginService.class);
String functionUrl = functionCallConfig.getUrl(); String functionUrl = functionCallConfig.getUrl();
if (StringUtils.isBlank(functionUrl) || SatisfactionChecker.check(queryCtx)) { if (StringUtils.isBlank(functionUrl) || SatisfactionChecker.check(queryCtx)) {
log.info("functionUrl:{}, skip function parser, queryText:{}", functionUrl, log.info("functionUrl:{}, skip function parser, queryText:{}", functionUrl,
queryCtx.getRequest().getQueryText()); queryCtx.getRequest().getQueryText());
return; return;
} }
List<PluginParseConfig> functionDOList = getFunctionDO(queryCtx.getRequest().getModelId(), queryCtx);
Set<Long> matchedDomains = getMatchDomains(queryCtx);
List<String> functionNames = getFunctionNames(matchedDomains);
log.info("matchedDomains:{},functionNames:{}", matchedDomains, functionNames);
if (CollectionUtils.isEmpty(functionNames) || CollectionUtils.isEmpty(matchedDomains)) {
return;
}
List<PluginParseConfig> functionDOList = getFunctionDO(queryCtx.getRequest().getDomainId());
FunctionReq functionReq = FunctionReq.builder() FunctionReq functionReq = FunctionReq.builder()
.queryText(queryCtx.getRequest().getQueryText()) .queryText(queryCtx.getRequest().getQueryText())
.pluginConfigs(functionDOList).build(); .pluginConfigs(functionDOList).build();
@@ -78,7 +70,6 @@ public class FunctionBasedParser implements SemanticParser {
if (skipFunction(queryCtx, functionResp)) { if (skipFunction(queryCtx, functionResp)) {
return; return;
} }
PluginParseResult functionCallParseResult = new PluginParseResult(); PluginParseResult functionCallParseResult = new PluginParseResult();
String toolSelection = functionResp.getToolSelection(); String toolSelection = functionResp.getToolSelection();
Optional<Plugin> pluginOptional = pluginService.getPluginByName(toolSelection); Optional<Plugin> pluginOptional = pluginService.getPluginByName(toolSelection);
@@ -87,24 +78,26 @@ public class FunctionBasedParser implements SemanticParser {
return; return;
} }
Plugin plugin = pluginOptional.get(); Plugin plugin = pluginOptional.get();
plugin.setParseMode(ParseMode.FUNCTION_CALL);
toolSelection = plugin.getType(); toolSelection = plugin.getType();
functionCallParseResult.setPlugin(plugin); functionCallParseResult.setPlugin(plugin);
log.info("QueryManager PluginQueryModes:{}", QueryManager.getPluginQueryModes()); log.info("QueryManager PluginQueryModes:{}", QueryManager.getPluginQueryModes());
PluginSemanticQuery semanticQuery = QueryManager.createPluginQuery(toolSelection); PluginSemanticQuery semanticQuery = QueryManager.createPluginQuery(toolSelection);
DomainResolver domainResolver = ComponentFactory.getDomainResolver(); ModelResolver ModelResolver = ComponentFactory.getModelResolver();
log.info("plugin ModelList:{}", plugin.getModelList());
Long domainId = domainResolver.resolve(queryCtx, chatCtx, plugin.getDomainList()); Pair<Boolean, List<Long>> pluginResolveResult = PluginManager.resolve(plugin, queryCtx);
log.info("FunctionBasedParser domainId:{}",domainId); Long modelId = ModelResolver.resolve(queryCtx, chatCtx, pluginResolveResult.getRight());
if ((Objects.isNull(domainId) || domainId <= 0) && !plugin.isContainsAllDomain()) { log.info("FunctionBasedParser modelId:{}", modelId);
log.info("domain is null, skip the parse, select tool: {}", toolSelection); if ((Objects.isNull(modelId) || modelId <= 0) && !plugin.isContainsAllModel()) {
log.info("Model is null, skip the parse, select tool: {}", toolSelection);
return; return;
} }
if (!plugin.getDomainList().contains(domainId) && !plugin.isContainsAllDomain()) { if (!plugin.getModelList().contains(modelId) && !plugin.isContainsAllModel()) {
return; return;
} }
SemanticParseInfo parseInfo = semanticQuery.getParseInfo(); SemanticParseInfo parseInfo = semanticQuery.getParseInfo();
if (Objects.nonNull(domainId) && domainId > 0){ if (Objects.nonNull(modelId) && modelId > 0) {
parseInfo.getElementMatches().addAll(queryCtx.getMapInfo().getMatchedElements(domainId)); parseInfo.getElementMatches().addAll(queryCtx.getMapInfo().getMatchedElements(modelId));
} }
functionCallParseResult.setRequest(queryCtx.getRequest()); functionCallParseResult.setRequest(queryCtx.getRequest());
Map<String, Object> properties = new HashMap<>(); Map<String, Object> properties = new HashMap<>();
@@ -112,21 +105,22 @@ public class FunctionBasedParser implements SemanticParser {
parseInfo.setProperties(properties); parseInfo.setProperties(properties);
parseInfo.setScore(FUNCTION_BONUS_THRESHOLD); parseInfo.setScore(FUNCTION_BONUS_THRESHOLD);
parseInfo.setQueryMode(semanticQuery.getQueryMode()); parseInfo.setQueryMode(semanticQuery.getQueryMode());
SchemaElement domain = new SchemaElement(); SchemaElement Model = new SchemaElement();
domain.setDomain(domainId); Model.setModel(modelId);
domain.setId(domainId); Model.setId(modelId);
parseInfo.setDomain(domain); parseInfo.setModel(Model);
queryCtx.getCandidateQueries().add(semanticQuery); queryCtx.getCandidateQueries().add(semanticQuery);
} }
private Set<Long> getMatchDomains(QueryContext queryCtx) {
private Set<Long> getMatchModels(QueryContext queryCtx) {
Set<Long> result = new HashSet<>(); Set<Long> result = new HashSet<>();
Long domainId = queryCtx.getRequest().getDomainId(); Long modelId = queryCtx.getRequest().getModelId();
if (Objects.nonNull(domainId) && domainId > 0) { if (Objects.nonNull(modelId) && modelId > 0) {
result.add(domainId); result.add(modelId);
return result; return result;
} }
return queryCtx.getMapInfo().getMatchedDomains(); return queryCtx.getMapInfo().getMatchedModels();
} }
private boolean skipFunction(QueryContext queryCtx, FunctionResp functionResp) { private boolean skipFunction(QueryContext queryCtx, FunctionResp functionResp) {
@@ -144,36 +138,46 @@ public class FunctionBasedParser implements SemanticParser {
return false; return false;
} }
private List<PluginParseConfig> getFunctionDO(Long domainId) { private List<PluginParseConfig> getFunctionDO(Long modelId, QueryContext queryContext) {
log.info("user decide domain:{}", domainId); log.info("user decide Model:{}", modelId);
List<Plugin> plugins = PluginManager.getPlugins(); List<Plugin> plugins = PluginManager.getPlugins();
List<PluginParseConfig> functionDOList = plugins.stream().filter(o -> { List<PluginParseConfig> functionDOList = plugins.stream().filter(plugin -> {
if (o.getParseModeConfig() == null) { if (DSLQuery.QUERY_MODE.equalsIgnoreCase(plugin.getType())) {
return false; return false;
} }
if (!CollectionUtils.isEmpty(o.getDomainList())) {//过滤掉没选主题域的插件 if (plugin.getParseModeConfig() == null) {
return true; return false;
} }
if (domainId == null || domainId <= 0L) { PluginParseConfig pluginParseConfig = JsonUtil.toObject(plugin.getParseModeConfig(),
return true;
} else {
return o.getDomainList().contains(domainId);
}
}).map(o -> {
PluginParseConfig functionCallConfig = JsonUtil.toObject(o.getParseModeConfig(),
PluginParseConfig.class); PluginParseConfig.class);
return functionCallConfig; if (StringUtils.isBlank(pluginParseConfig.getName())) {
}).collect(Collectors.toList()); return false;
}
Pair<Boolean, List<Long>> pluginResolverResult = PluginManager.resolve(plugin, queryContext);
log.info("embedding plugin [{}-{}] resolve: {}", plugin.getId(), plugin.getName(), pluginResolverResult);
if (!pluginResolverResult.getLeft()) {
return false;
} else {
List<Long> resolveModel = pluginResolverResult.getRight();
if (modelId != null && modelId > 0) {
if (plugin.isContainsAllModel()) {
return true;
}
return resolveModel.contains(modelId);
}
return true;
}
}).map(o -> JsonUtil.toObject(o.getParseModeConfig(), PluginParseConfig.class)).collect(Collectors.toList());
log.info("getFunctionDO:{}", JsonUtil.toString(functionDOList)); log.info("getFunctionDO:{}", JsonUtil.toString(functionDOList));
return functionDOList; return functionDOList;
} }
private List<String> getFunctionNames(Set<Long> matchedDomains) { private List<String> getFunctionNames(Set<Long> matchedModels) {
List<Plugin> plugins = PluginManager.getPlugins(); List<Plugin> plugins = PluginManager.getPlugins();
Set<String> functionNames = plugins.stream() Set<String> functionNames = plugins.stream()
.filter(entry -> { .filter(entry -> {
if (!CollectionUtils.isEmpty(entry.getDomainList()) && !CollectionUtils.isEmpty(matchedDomains)) { if (!CollectionUtils.isEmpty(entry.getModelList()) && !CollectionUtils.isEmpty(matchedModels)) {
return entry.getDomainList().stream().anyMatch(matchedDomains::contains); return entry.getModelList().stream().anyMatch(matchedModels::contains);
} }
return true; return true;
} }

View File

@@ -1,8 +1,7 @@
package com.tencent.supersonic.chat.parser.function; package com.tencent.supersonic.chat.parser.function;
import java.util.List;
import com.tencent.supersonic.chat.plugin.PluginParseConfig; import com.tencent.supersonic.chat.plugin.PluginParseConfig;
import java.util.List;
import lombok.Builder; import lombok.Builder;
import lombok.Data; import lombok.Data;

View File

@@ -1,181 +0,0 @@
package com.tencent.supersonic.chat.parser.function;
import com.tencent.supersonic.chat.api.pojo.*;
import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
import com.tencent.supersonic.chat.api.component.SemanticQuery;
import lombok.extern.slf4j.Slf4j;
import java.util.*;
import java.util.stream.Collectors;
import org.apache.commons.collections.CollectionUtils;
@Slf4j
public class HeuristicDomainResolver implements DomainResolver {
protected static Long selectDomainBySchemaElementCount(Map<Long, SemanticQuery> domainQueryModes,
SchemaMapInfo schemaMap) {
Map<Long, DomainMatchResult> domainTypeMap = getDomainTypeMap(schemaMap);
if (domainTypeMap.size() == 1) {
Long domainSelect = domainTypeMap.entrySet().stream().collect(Collectors.toList()).get(0).getKey();
if (domainQueryModes.containsKey(domainSelect)) {
log.info("selectDomain with only one domain [{}]", domainSelect);
return domainSelect;
}
} else {
Map.Entry<Long, DomainMatchResult> maxDomain = domainTypeMap.entrySet().stream()
.filter(entry -> domainQueryModes.containsKey(entry.getKey()))
.sorted((o1, o2) -> {
int difference = o2.getValue().getCount() - o1.getValue().getCount();
if (difference == 0) {
return (int) ((o2.getValue().getMaxSimilarity()
- o1.getValue().getMaxSimilarity()) * 100);
}
return difference;
}).findFirst().orElse(null);
if (maxDomain != null) {
log.info("selectDomain with multiple domains [{}]", maxDomain.getKey());
return maxDomain.getKey();
}
}
return 0L;
}
/**
* to check can switch domain if context exit domain
*
* @return false will use context domain, true will use other domain , maybe include context domain
*/
protected static boolean isAllowSwitch(Map<Long, SemanticQuery> domainQueryModes, SchemaMapInfo schemaMap,
ChatContext chatCtx, QueryReq searchCtx, Long domainId, List<Long> restrictiveDomains) {
if (!Objects.nonNull(domainId) || domainId <= 0) {
return true;
}
// except content domain, calculate the number of types for each domain, if numbers<=1 will not switch
Map<Long, DomainMatchResult> domainTypeMap = getDomainTypeMap(schemaMap);
log.info("isAllowSwitch domainTypeMap [{}]", domainTypeMap);
long otherDomainTypeNumBigOneCount = domainTypeMap.entrySet().stream()
.filter(entry -> domainQueryModes.containsKey(entry.getKey()) && !entry.getKey().equals(domainId))
.filter(entry -> entry.getValue().getCount() > 1).count();
if (otherDomainTypeNumBigOneCount >= 1) {
return true;
}
// if query text only contain time , will not switch
if (!CollectionUtils.isEmpty(domainQueryModes.values())) {
for (SemanticQuery semanticQuery : domainQueryModes.values()) {
if (semanticQuery == null) {
continue;
}
SemanticParseInfo semanticParseInfo = semanticQuery.getParseInfo();
if (semanticParseInfo == null) {
continue;
}
if (searchCtx.getQueryText() != null && semanticParseInfo.getDateInfo() != null) {
if (semanticParseInfo.getDateInfo().getText() != null) {
if (semanticParseInfo.getDateInfo().getText().equalsIgnoreCase(searchCtx.getQueryText())) {
log.info("timeParseResults is not null , can not switch context , timeParseResults:{},",
semanticParseInfo.getDateInfo());
return false;
}
}
}
}
}
if (CollectionUtils.isNotEmpty(restrictiveDomains) && !restrictiveDomains.contains(domainId)) {
return true;
}
// if context domain not in schemaMap , will switch
if (schemaMap.getMatchedElements(domainId) == null || schemaMap.getMatchedElements(domainId).size() <= 0) {
log.info("domainId not in schemaMap ");
return true;
}
// other will not switch
return false;
}
public static Map<Long, DomainMatchResult> getDomainTypeMap(SchemaMapInfo schemaMap) {
Map<Long, DomainMatchResult> domainCount = new HashMap<>();
for (Map.Entry<Long, List<SchemaElementMatch>> entry : schemaMap.getDomainElementMatches().entrySet()) {
List<SchemaElementMatch> schemaElementMatches = schemaMap.getMatchedElements(entry.getKey());
if (schemaElementMatches != null && schemaElementMatches.size() > 0) {
if (!domainCount.containsKey(entry.getKey())) {
domainCount.put(entry.getKey(), new DomainMatchResult());
}
DomainMatchResult domainMatchResult = domainCount.get(entry.getKey());
Set<SchemaElementType> schemaElementTypes = new HashSet<>();
schemaElementMatches.stream()
.forEach(schemaElementMatch -> schemaElementTypes.add(
schemaElementMatch.getElement().getType()));
SchemaElementMatch schemaElementMatchMax = schemaElementMatches.stream()
.sorted((o1, o2) ->
((int) ((o2.getSimilarity() - o1.getSimilarity()) * 100))
).findFirst().orElse(null);
if (schemaElementMatchMax != null) {
domainMatchResult.setMaxSimilarity(schemaElementMatchMax.getSimilarity());
}
domainMatchResult.setCount(schemaElementTypes.size());
}
}
return domainCount;
}
public Long resolve(QueryContext queryContext, ChatContext chatCtx, List<Long> restrictiveDomains) {
Long domainId = queryContext.getRequest().getDomainId();
if (Objects.nonNull(domainId) && domainId > 0) {
if (CollectionUtils.isNotEmpty(restrictiveDomains) && restrictiveDomains.contains(domainId)) {
return domainId;
} else {
return null;
}
}
SchemaMapInfo mapInfo = queryContext.getMapInfo();
Set<Long> matchedDomains = mapInfo.getMatchedDomains();
if (CollectionUtils.isNotEmpty(restrictiveDomains)) {
matchedDomains = matchedDomains.stream()
.filter(matchedDomain -> restrictiveDomains.contains(matchedDomain))
.collect(Collectors.toSet());
}
Map<Long, SemanticQuery> domainQueryModes = new HashMap<>();
for (Long matchedDomain : matchedDomains) {
domainQueryModes.put(matchedDomain, null);
}
if(domainQueryModes.size()==1){
return domainQueryModes.keySet().stream().findFirst().get();
}
return resolve(domainQueryModes, queryContext, chatCtx,
queryContext.getMapInfo(),restrictiveDomains);
}
public Long resolve(Map<Long, SemanticQuery> domainQueryModes, QueryContext queryContext,
ChatContext chatCtx, SchemaMapInfo schemaMap, List<Long> restrictiveDomains) {
Long selectDomain = selectDomain(domainQueryModes, queryContext.getRequest(), chatCtx, schemaMap,restrictiveDomains);
if (selectDomain > 0) {
log.info("selectDomain {} ", selectDomain);
return selectDomain;
}
// get the max SchemaElementType number
return selectDomainBySchemaElementCount(domainQueryModes, schemaMap);
}
public Long selectDomain(Map<Long, SemanticQuery> domainQueryModes, QueryReq queryContext,
ChatContext chatCtx,
SchemaMapInfo schemaMap, List<Long> restrictiveDomains) {
// if QueryContext has domainId and in domainQueryModes
if (domainQueryModes.containsKey(queryContext.getDomainId())) {
log.info("selectDomain from QueryContext [{}]", queryContext.getDomainId());
return queryContext.getDomainId();
}
// if ChatContext has domainId and in domainQueryModes
if (chatCtx.getParseInfo().getDomainId() > 0) {
Long domainId = chatCtx.getParseInfo().getDomainId();
if (!isAllowSwitch(domainQueryModes, schemaMap, chatCtx, queryContext, domainId,restrictiveDomains)) {
log.info("selectDomain from ChatContext [{}]", domainId);
return domainId;
}
}
// default 0
return 0L;
}
}

View File

@@ -0,0 +1,192 @@
package com.tencent.supersonic.chat.parser.function;
import com.tencent.supersonic.chat.api.component.SemanticQuery;
import com.tencent.supersonic.chat.api.pojo.ChatContext;
import com.tencent.supersonic.chat.api.pojo.QueryContext;
import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
import com.tencent.supersonic.chat.api.pojo.SchemaMapInfo;
import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.stream.Collectors;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.collections.CollectionUtils;
@Slf4j
public class HeuristicModelResolver implements ModelResolver {
protected static Long selectModelBySchemaElementCount(Map<Long, SemanticQuery> ModelQueryModes,
SchemaMapInfo schemaMap) {
Map<Long, ModelMatchResult> ModelTypeMap = getModelTypeMap(schemaMap);
if (ModelTypeMap.size() == 1) {
Long ModelSelect = ModelTypeMap.entrySet().stream().collect(Collectors.toList()).get(0).getKey();
if (ModelQueryModes.containsKey(ModelSelect)) {
log.info("selectModel with only one Model [{}]", ModelSelect);
return ModelSelect;
}
} else {
Map.Entry<Long, ModelMatchResult> maxModel = ModelTypeMap.entrySet().stream()
.filter(entry -> ModelQueryModes.containsKey(entry.getKey()))
.sorted((o1, o2) -> {
int difference = o2.getValue().getCount() - o1.getValue().getCount();
if (difference == 0) {
return (int) ((o2.getValue().getMaxSimilarity()
- o1.getValue().getMaxSimilarity()) * 100);
}
return difference;
}).findFirst().orElse(null);
if (maxModel != null) {
log.info("selectModel with multiple Models [{}]", maxModel.getKey());
return maxModel.getKey();
}
}
return 0L;
}
/**
* to check can switch Model if context exit Model
*
* @return false will use context Model, true will use other Model , maybe include context Model
*/
protected static boolean isAllowSwitch(Map<Long, SemanticQuery> ModelQueryModes, SchemaMapInfo schemaMap,
ChatContext chatCtx, QueryReq searchCtx, Long modelId, List<Long> restrictiveModels) {
if (!Objects.nonNull(modelId) || modelId <= 0) {
return true;
}
// except content Model, calculate the number of types for each Model, if numbers<=1 will not switch
Map<Long, ModelMatchResult> ModelTypeMap = getModelTypeMap(schemaMap);
log.info("isAllowSwitch ModelTypeMap [{}]", ModelTypeMap);
long otherModelTypeNumBigOneCount = ModelTypeMap.entrySet().stream()
.filter(entry -> ModelQueryModes.containsKey(entry.getKey()) && !entry.getKey().equals(modelId))
.filter(entry -> entry.getValue().getCount() > 1).count();
if (otherModelTypeNumBigOneCount >= 1) {
return true;
}
// if query text only contain time , will not switch
if (!CollectionUtils.isEmpty(ModelQueryModes.values())) {
for (SemanticQuery semanticQuery : ModelQueryModes.values()) {
if (semanticQuery == null) {
continue;
}
SemanticParseInfo semanticParseInfo = semanticQuery.getParseInfo();
if (semanticParseInfo == null) {
continue;
}
if (searchCtx.getQueryText() != null && semanticParseInfo.getDateInfo() != null) {
if (semanticParseInfo.getDateInfo().getDetectWord() != null) {
if (semanticParseInfo.getDateInfo().getDetectWord()
.equalsIgnoreCase(searchCtx.getQueryText())) {
log.info("timeParseResults is not null , can not switch context , timeParseResults:{},",
semanticParseInfo.getDateInfo());
return false;
}
}
}
}
}
if (CollectionUtils.isNotEmpty(restrictiveModels) && !restrictiveModels.contains(modelId)) {
return true;
}
// if context Model not in schemaMap , will switch
if (schemaMap.getMatchedElements(modelId) == null || schemaMap.getMatchedElements(modelId).size() <= 0) {
log.info("modelId not in schemaMap ");
return true;
}
// other will not switch
return false;
}
public static Map<Long, ModelMatchResult> getModelTypeMap(SchemaMapInfo schemaMap) {
Map<Long, ModelMatchResult> ModelCount = new HashMap<>();
for (Map.Entry<Long, List<SchemaElementMatch>> entry : schemaMap.getModelElementMatches().entrySet()) {
List<SchemaElementMatch> schemaElementMatches = schemaMap.getMatchedElements(entry.getKey());
if (schemaElementMatches != null && schemaElementMatches.size() > 0) {
if (!ModelCount.containsKey(entry.getKey())) {
ModelCount.put(entry.getKey(), new ModelMatchResult());
}
ModelMatchResult ModelMatchResult = ModelCount.get(entry.getKey());
Set<SchemaElementType> schemaElementTypes = new HashSet<>();
schemaElementMatches.stream()
.forEach(schemaElementMatch -> schemaElementTypes.add(
schemaElementMatch.getElement().getType()));
SchemaElementMatch schemaElementMatchMax = schemaElementMatches.stream()
.sorted((o1, o2) ->
((int) ((o2.getSimilarity() - o1.getSimilarity()) * 100))
).findFirst().orElse(null);
if (schemaElementMatchMax != null) {
ModelMatchResult.setMaxSimilarity(schemaElementMatchMax.getSimilarity());
}
ModelMatchResult.setCount(schemaElementTypes.size());
}
}
return ModelCount;
}
public Long resolve(QueryContext queryContext, ChatContext chatCtx, List<Long> restrictiveModels) {
Long modelId = queryContext.getRequest().getModelId();
if (Objects.nonNull(modelId) && modelId > 0) {
if (CollectionUtils.isNotEmpty(restrictiveModels) && restrictiveModels.contains(modelId)) {
return modelId;
} else {
return null;
}
}
SchemaMapInfo mapInfo = queryContext.getMapInfo();
Set<Long> matchedModels = mapInfo.getMatchedModels();
if (CollectionUtils.isNotEmpty(restrictiveModels)) {
matchedModels = matchedModels.stream()
.filter(restrictiveModels::contains)
.collect(Collectors.toSet());
}
Map<Long, SemanticQuery> ModelQueryModes = new HashMap<>();
for (Long matchedModel : matchedModels) {
ModelQueryModes.put(matchedModel, null);
}
if (ModelQueryModes.size() == 1) {
return ModelQueryModes.keySet().stream().findFirst().get();
}
return resolve(ModelQueryModes, queryContext, chatCtx,
queryContext.getMapInfo(), restrictiveModels);
}
public Long resolve(Map<Long, SemanticQuery> ModelQueryModes, QueryContext queryContext,
ChatContext chatCtx, SchemaMapInfo schemaMap, List<Long> restrictiveModels) {
Long selectModel = selectModel(ModelQueryModes, queryContext.getRequest(), chatCtx, schemaMap,
restrictiveModels);
if (selectModel > 0) {
log.info("selectModel {} ", selectModel);
return selectModel;
}
// get the max SchemaElementType number
return selectModelBySchemaElementCount(ModelQueryModes, schemaMap);
}
public Long selectModel(Map<Long, SemanticQuery> ModelQueryModes, QueryReq queryContext,
ChatContext chatCtx,
SchemaMapInfo schemaMap, List<Long> restrictiveModels) {
// if QueryContext has modelId and in ModelQueryModes
if (ModelQueryModes.containsKey(queryContext.getModelId())) {
log.info("selectModel from QueryContext [{}]", queryContext.getModelId());
return queryContext.getModelId();
}
// if ChatContext has modelId and in ModelQueryModes
if (chatCtx.getParseInfo().getModelId() > 0) {
Long modelId = chatCtx.getParseInfo().getModelId();
if (!isAllowSwitch(ModelQueryModes, schemaMap, chatCtx, queryContext, modelId, restrictiveModels)) {
log.info("selectModel from ChatContext [{}]", modelId);
return modelId;
}
}
// default 0
return 0L;
}
}

View File

@@ -3,7 +3,8 @@ package com.tencent.supersonic.chat.parser.function;
import lombok.Data; import lombok.Data;
@Data @Data
public class DomainMatchResult { public class ModelMatchResult {
private Integer count = 0; private Integer count = 0;
private double maxSimilarity; private double maxSimilarity;
} }

View File

@@ -5,8 +5,8 @@ import com.tencent.supersonic.chat.api.pojo.ChatContext;
import com.tencent.supersonic.chat.api.pojo.QueryContext; import com.tencent.supersonic.chat.api.pojo.QueryContext;
import java.util.List; import java.util.List;
public interface DomainResolver { public interface ModelResolver {
Long resolve(QueryContext queryContext, ChatContext chatCtx, List<Long> restrictiveDomains); Long resolve(QueryContext queryContext, ChatContext chatCtx, List<Long> restrictiveModels);
} }

View File

@@ -1,8 +1,8 @@
package com.tencent.supersonic.chat.parser.function; package com.tencent.supersonic.chat.parser.function;
import lombok.Data;
import java.util.List; import java.util.List;
import java.util.Map; import java.util.Map;
import lombok.Data;
@Data @Data
public class Parameters { public class Parameters {

View File

@@ -0,0 +1,11 @@
package com.tencent.supersonic.chat.parser.llm;
import com.tencent.supersonic.chat.plugin.PluginParseResult;
import com.tencent.supersonic.chat.query.dsl.LLMResp;
import lombok.Data;
@Data
public class DSLParseResult extends PluginParseResult {
private LLMResp llmResp;
}

View File

@@ -0,0 +1,231 @@
package com.tencent.supersonic.chat.parser.llm;
import com.tencent.supersonic.chat.api.component.SemanticParser;
import com.tencent.supersonic.chat.api.pojo.ChatContext;
import com.tencent.supersonic.chat.api.pojo.QueryContext;
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
import com.tencent.supersonic.chat.api.pojo.SemanticSchema;
import com.tencent.supersonic.chat.config.LLMConfig;
import com.tencent.supersonic.chat.parser.SatisfactionChecker;
import com.tencent.supersonic.chat.parser.function.ModelResolver;
import com.tencent.supersonic.chat.plugin.Plugin;
import com.tencent.supersonic.chat.plugin.PluginManager;
import com.tencent.supersonic.chat.query.QueryManager;
import com.tencent.supersonic.chat.query.dsl.DSLBuilder;
import com.tencent.supersonic.chat.query.dsl.DSLQuery;
import com.tencent.supersonic.chat.query.dsl.LLMReq;
import com.tencent.supersonic.chat.query.dsl.LLMReq.ElementValue;
import com.tencent.supersonic.chat.query.dsl.LLMResp;
import com.tencent.supersonic.chat.query.plugin.PluginSemanticQuery;
import com.tencent.supersonic.chat.utils.ComponentFactory;
import com.tencent.supersonic.common.pojo.Constants;
import com.tencent.supersonic.common.util.ContextUtils;
import com.tencent.supersonic.common.util.DateUtils;
import com.tencent.supersonic.common.util.JsonUtil;
import com.tencent.supersonic.knowledge.service.SchemaService;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.stream.Collectors;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.springframework.http.HttpEntity;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpMethod;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.util.CollectionUtils;
import org.springframework.web.client.RestTemplate;
@Slf4j
public class LLMDSLParser implements SemanticParser {
public static final double FUNCTION_BONUS_THRESHOLD = 201;
@Override
public void parse(QueryContext queryCtx, ChatContext chatCtx) {
final LLMConfig llmConfig = ContextUtils.getBean(LLMConfig.class);
if (StringUtils.isEmpty(llmConfig.getUrl()) || SatisfactionChecker.check(queryCtx)) {
log.info("llmConfig:{}, skip function parser, queryText:{}", llmConfig,
queryCtx.getRequest().getQueryText());
return;
}
List<Plugin> dslPlugins = PluginManager.getPlugins().stream()
.filter(plugin -> DSLQuery.QUERY_MODE.equalsIgnoreCase(plugin.getType()))
.collect(Collectors.toList());
if (CollectionUtils.isEmpty(dslPlugins)) {
return;
}
Plugin plugin = dslPlugins.get(0);
List<Long> dslModels = plugin.getModelList();
try {
ModelResolver modelResolver = ComponentFactory.getModelResolver();
Long modelId = modelResolver.resolve(queryCtx, chatCtx, dslModels);
log.info("resolve modelId:{},dslModels:{}", modelId, dslModels);
if (Objects.isNull(modelId)) {
return;
}
LLMResp llmResp = requestLLM(queryCtx, modelId);
if (Objects.isNull(llmResp)) {
return;
}
PluginSemanticQuery semanticQuery = QueryManager.createPluginQuery(DSLQuery.QUERY_MODE);
SemanticParseInfo parseInfo = semanticQuery.getParseInfo();
if (Objects.nonNull(modelId) && modelId > 0) {
parseInfo.getElementMatches().addAll(queryCtx.getMapInfo().getMatchedElements(modelId));
}
DSLParseResult dslParseResult = new DSLParseResult();
dslParseResult.setRequest(queryCtx.getRequest());
dslParseResult.setLlmResp(llmResp);
dslParseResult.setPlugin(plugin);
Map<String, Object> properties = new HashMap<>();
properties.put(Constants.CONTEXT, dslParseResult);
parseInfo.setProperties(properties);
parseInfo.setScore(FUNCTION_BONUS_THRESHOLD);
parseInfo.setQueryMode(semanticQuery.getQueryMode());
SchemaElement Model = new SchemaElement();
Model.setModel(modelId);
Model.setId(modelId);
parseInfo.setModel(Model);
queryCtx.getCandidateQueries().add(semanticQuery);
} catch (Exception e) {
log.error("LLMDSLParser error", e);
}
}
private LLMResp requestLLM(QueryContext queryCtx, Long modelId) {
long startTime = System.currentTimeMillis();
String queryText = queryCtx.getRequest().getQueryText();
final LLMConfig llmConfig = ContextUtils.getBean(LLMConfig.class);
if (StringUtils.isEmpty(llmConfig.getUrl())) {
log.warn("llmConfig url is null, skip llm parser");
return null;
}
SemanticSchema semanticSchema = ContextUtils.getBean(SchemaService.class).getSemanticSchema();
Map<Long, String> modelIdToName = semanticSchema.getModelIdToName();
LLMReq llmReq = new LLMReq();
llmReq.setQueryText(queryText);
LLMReq.LLMSchema llmSchema = new LLMReq.LLMSchema();
llmSchema.setModelName(modelIdToName.get(modelId));
llmSchema.setDomainName(modelIdToName.get(modelId));
List<String> fieldNameList = getFieldNameList(queryCtx, modelId, semanticSchema);
fieldNameList.add(DSLBuilder.DATA_Field);
llmSchema.setFieldNameList(fieldNameList);
llmReq.setSchema(llmSchema);
List<ElementValue> linking = new ArrayList<>();
linking.addAll(getValueList(queryCtx, modelId, semanticSchema));
llmReq.setLinking(linking);
String currentDate = getCurrentDate(modelId);
llmReq.setCurrentDate(currentDate);
log.info("requestLLM request, modelId:{},llmReq:{}", modelId, llmReq);
String questUrl = llmConfig.getUrl() + llmConfig.getQueryToSqlPath();
RestTemplate restTemplate = ContextUtils.getBean(RestTemplate.class);
try {
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.APPLICATION_JSON);
HttpEntity<String> entity = new HttpEntity<>(JsonUtil.toString(llmReq), headers);
ResponseEntity<LLMResp> responseEntity = restTemplate.exchange(questUrl, HttpMethod.POST, entity,
LLMResp.class);
log.info("requestLLM response,cost:{}, questUrl:{} \n entity:{} \n body:{}",
System.currentTimeMillis() - startTime, questUrl, entity, responseEntity.getBody());
return responseEntity.getBody();
} catch (Exception e) {
log.error("requestLLM error", e);
}
return null;
}
private String getCurrentDate(Long modelId) {
return DateUtils.getBeforeDate(4);
// ChatConfigFilter filter = new ChatConfigFilter();
// filter.setModelId(modelId);
//
// List<ChatConfigResp> configResps = ContextUtils.getBean(ConfigService.class).search(filter, null);
// if (CollectionUtils.isEmpty(configResps)) {
// return
// }
// ChatConfigResp chatConfigResp = configResps.get(0);
// chatConfigResp.getChatDetailConfig().getChatDefaultConfig().get
}
private List<ElementValue> getValueList(QueryContext queryCtx, Long modelId, SemanticSchema semanticSchema) {
Map<Long, String> itemIdToName = getItemIdToName(modelId, semanticSchema);
List<SchemaElementMatch> matchedElements = queryCtx.getMapInfo().getMatchedElements(modelId);
if (CollectionUtils.isEmpty(matchedElements)) {
return new ArrayList<>();
}
Set<ElementValue> valueMatches = matchedElements.stream()
.filter(schemaElementMatch -> {
SchemaElementType type = schemaElementMatch.getElement().getType();
return SchemaElementType.VALUE.equals(type) || SchemaElementType.ID.equals(type);
})
.map(elementMatch ->
{
ElementValue elementValue = new ElementValue();
elementValue.setFieldName(itemIdToName.get(elementMatch.getElement().getId()));
elementValue.setFieldValue(elementMatch.getWord());
return elementValue;
}
)
.collect(Collectors.toSet());
return new ArrayList<>(valueMatches);
}
private List<String> getFieldNameList(QueryContext queryCtx, Long modelId, SemanticSchema semanticSchema) {
Map<Long, String> itemIdToName = getItemIdToName(modelId, semanticSchema);
List<SchemaElementMatch> matchedElements = queryCtx.getMapInfo().getMatchedElements(modelId);
if (CollectionUtils.isEmpty(matchedElements)) {
return new ArrayList<>();
}
Set<String> fieldNameList = matchedElements.stream()
.filter(schemaElementMatch -> {
SchemaElementType elementType = schemaElementMatch.getElement().getType();
return SchemaElementType.METRIC.equals(elementType) ||
SchemaElementType.DIMENSION.equals(elementType) ||
SchemaElementType.VALUE.equals(elementType);
})
.map(schemaElementMatch -> {
SchemaElementType elementType = schemaElementMatch.getElement().getType();
if (!SchemaElementType.VALUE.equals(elementType)) {
return schemaElementMatch.getWord();
}
return itemIdToName.get(schemaElementMatch.getElement().getId());
})
.filter(name -> StringUtils.isNotEmpty(name) && !name.contains("%"))
.collect(Collectors.toSet());
return new ArrayList<>(fieldNameList);
}
private Map<Long, String> getItemIdToName(Long modelId, SemanticSchema semanticSchema) {
return semanticSchema.getDimensions().stream()
.filter(entry -> modelId.equals(entry.getModel()))
.collect(Collectors.toMap(SchemaElement::getId, SchemaElement::getName, (value1, value2) -> value2));
}
}

View File

@@ -17,7 +17,7 @@ public class LLMTimeEnhancementParse implements SemanticParser {
@Override @Override
public void parse(QueryContext queryContext, ChatContext chatContext) { public void parse(QueryContext queryContext, ChatContext chatContext) {
log.info("before queryContext:{},chatContext:{}",queryContext,chatContext); log.info("before queryContext:{},chatContext:{}", queryContext, chatContext);
ChatGptHelper chatGptHelper = ContextUtils.getBean(ChatGptHelper.class); ChatGptHelper chatGptHelper = ContextUtils.getBean(ChatGptHelper.class);
String inferredTime = chatGptHelper.inferredTime(queryContext.getRequest().getQueryText()); String inferredTime = chatGptHelper.inferredTime(queryContext.getRequest().getQueryText());
try { try {
@@ -25,12 +25,12 @@ public class LLMTimeEnhancementParse implements SemanticParser {
for (SemanticQuery query : queryContext.getCandidateQueries()) { for (SemanticQuery query : queryContext.getCandidateQueries()) {
DateConf dateInfo = query.getParseInfo().getDateInfo(); DateConf dateInfo = query.getParseInfo().getDateInfo();
JSONObject jsonObject = JSON.parseObject(inferredTime); JSONObject jsonObject = JSON.parseObject(inferredTime);
if (jsonObject.containsKey("date")){ if (jsonObject.containsKey("date")) {
dateInfo.setDateMode(DateConf.DateMode.BETWEEN); dateInfo.setDateMode(DateConf.DateMode.BETWEEN);
dateInfo.setStartDate(jsonObject.getString("date")); dateInfo.setStartDate(jsonObject.getString("date"));
dateInfo.setEndDate(jsonObject.getString("date")); dateInfo.setEndDate(jsonObject.getString("date"));
query.getParseInfo().setDateInfo(dateInfo); query.getParseInfo().setDateInfo(dateInfo);
}else if (jsonObject.containsKey("start")){ } else if (jsonObject.containsKey("start")) {
dateInfo.setDateMode(DateConf.DateMode.BETWEEN); dateInfo.setDateMode(DateConf.DateMode.BETWEEN);
dateInfo.setStartDate(jsonObject.getString("start")); dateInfo.setStartDate(jsonObject.getString("start"));
dateInfo.setEndDate(jsonObject.getString("end")); dateInfo.setEndDate(jsonObject.getString("end"));
@@ -38,11 +38,12 @@ public class LLMTimeEnhancementParse implements SemanticParser {
} }
} }
} }
}catch (Exception exception){ } catch (Exception exception) {
log.error("{} parse error,this reason is:{}",LLMTimeEnhancementParse.class.getSimpleName(), (Object) exception.getStackTrace()); log.error("{} parse error,this reason is:{}", LLMTimeEnhancementParse.class.getSimpleName(),
(Object) exception.getStackTrace());
} }
log.info("after queryContext:{},chatContext:{}",queryContext,chatContext); log.info("after queryContext:{},chatContext:{}", queryContext, chatContext);
} }

View File

@@ -21,7 +21,9 @@ import java.util.regex.Matcher;
import java.util.regex.Pattern; import java.util.regex.Pattern;
import java.util.stream.Collectors; import java.util.stream.Collectors;
import java.util.stream.Stream; import java.util.stream.Stream;
import lombok.AllArgsConstructor;
import lombok.extern.slf4j.Slf4j; import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
@Slf4j @Slf4j
public class AggregateTypeParser implements SemanticParser { public class AggregateTypeParser implements SemanticParser {
@@ -35,36 +37,60 @@ public class AggregateTypeParser implements SemanticParser {
new AbstractMap.SimpleEntry<>(DISTINCT, Pattern.compile("(?i)(uv)")), new AbstractMap.SimpleEntry<>(DISTINCT, Pattern.compile("(?i)(uv)")),
new AbstractMap.SimpleEntry<>(COUNT, Pattern.compile("(?i)(总数|pv)")), new AbstractMap.SimpleEntry<>(COUNT, Pattern.compile("(?i)(总数|pv)")),
new AbstractMap.SimpleEntry<>(NONE, Pattern.compile("(?i)(明细)")) new AbstractMap.SimpleEntry<>(NONE, Pattern.compile("(?i)(明细)"))
).collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue,(k1,k2)->k2)); ).collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue, (k1, k2) -> k2));
@Override @Override
public void parse(QueryContext queryContext, ChatContext chatContext) { public void parse(QueryContext queryContext, ChatContext chatContext) {
String queryText = queryContext.getRequest().getQueryText();
AggregateConf aggregateConf = resolveAggregateConf(queryText);
for (SemanticQuery semanticQuery : queryContext.getCandidateQueries()) { for (SemanticQuery semanticQuery : queryContext.getCandidateQueries()) {
if (!AggregateTypeEnum.NONE.equals(semanticQuery.getParseInfo().getAggType())) { if (!AggregateTypeEnum.NONE.equals(semanticQuery.getParseInfo().getAggType())) {
continue; continue;
} }
semanticQuery.getParseInfo().setAggType(aggregateConf.type);
String queryText = queryContext.getRequest().getQueryText(); int detectWordLength = 0;
semanticQuery.getParseInfo().setAggType(resolveAggregateType(queryText)); if (StringUtils.isNotEmpty(aggregateConf.detectWord)) {
detectWordLength = aggregateConf.detectWord.length();
}
semanticQuery.getParseInfo().setScore(semanticQuery.getParseInfo().getScore() + detectWordLength);
} }
} }
public static AggregateTypeEnum resolveAggregateType(String queryText) { public AggregateTypeEnum resolveAggregateType(String queryText) {
AggregateConf aggregateConf = resolveAggregateConf(queryText);
return aggregateConf.type;
}
private AggregateConf resolveAggregateConf(String queryText) {
Map<AggregateTypeEnum, Integer> aggregateCount = new HashMap<>(REGX_MAP.size()); Map<AggregateTypeEnum, Integer> aggregateCount = new HashMap<>(REGX_MAP.size());
Map<AggregateTypeEnum, String> aggregateWord = new HashMap<>(REGX_MAP.size());
for (Map.Entry<AggregateTypeEnum, Pattern> entry : REGX_MAP.entrySet()) { for (Map.Entry<AggregateTypeEnum, Pattern> entry : REGX_MAP.entrySet()) {
Matcher matcher = entry.getValue().matcher(queryText); Matcher matcher = entry.getValue().matcher(queryText);
int count = 0; int count = 0;
String detectWord = null;
while (matcher.find()) { while (matcher.find()) {
count++; count++;
detectWord = matcher.group();
} }
if (count > 0) { if (count > 0) {
aggregateCount.put(entry.getKey(), count); aggregateCount.put(entry.getKey(), count);
aggregateWord.put(entry.getKey(), detectWord);
} }
} }
return aggregateCount.entrySet().stream().max(Map.Entry.comparingByValue()) AggregateTypeEnum type = aggregateCount.entrySet().stream().max(Map.Entry.comparingByValue())
.map(entry -> entry.getKey()).orElse(NONE); .map(entry -> entry.getKey()).orElse(NONE);
String detectWord = aggregateWord.get(type);
return new AggregateConf(type, detectWord);
}
@AllArgsConstructor
class AggregateConf {
public AggregateTypeEnum type;
public String detectWord;
} }
} }

View File

@@ -1,5 +1,12 @@
package com.tencent.supersonic.chat.parser.rule; package com.tencent.supersonic.chat.parser.rule;
import static com.tencent.supersonic.chat.api.pojo.SchemaElementType.DIMENSION;
import static com.tencent.supersonic.chat.api.pojo.SchemaElementType.ENTITY;
import static com.tencent.supersonic.chat.api.pojo.SchemaElementType.ID;
import static com.tencent.supersonic.chat.api.pojo.SchemaElementType.METRIC;
import static com.tencent.supersonic.chat.api.pojo.SchemaElementType.MODEL;
import static com.tencent.supersonic.chat.api.pojo.SchemaElementType.VALUE;
import com.tencent.supersonic.chat.api.component.SemanticParser; import com.tencent.supersonic.chat.api.component.SemanticParser;
import com.tencent.supersonic.chat.api.component.SemanticQuery; import com.tencent.supersonic.chat.api.component.SemanticQuery;
import com.tencent.supersonic.chat.api.pojo.ChatContext; import com.tencent.supersonic.chat.api.pojo.ChatContext;
@@ -8,8 +15,8 @@ import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
import com.tencent.supersonic.chat.api.pojo.SchemaElementType; import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
import com.tencent.supersonic.chat.query.QueryManager; import com.tencent.supersonic.chat.query.QueryManager;
import com.tencent.supersonic.chat.query.rule.RuleSemanticQuery; import com.tencent.supersonic.chat.query.rule.RuleSemanticQuery;
import com.tencent.supersonic.chat.query.rule.metric.MetricDomainQuery;
import com.tencent.supersonic.chat.query.rule.metric.MetricEntityQuery; import com.tencent.supersonic.chat.query.rule.metric.MetricEntityQuery;
import com.tencent.supersonic.chat.query.rule.metric.MetricModelQuery;
import com.tencent.supersonic.chat.query.rule.metric.MetricSemanticQuery; import com.tencent.supersonic.chat.query.rule.metric.MetricSemanticQuery;
import java.util.AbstractMap; import java.util.AbstractMap;
import java.util.ArrayList; import java.util.ArrayList;
@@ -21,8 +28,6 @@ import java.util.stream.Collectors;
import java.util.stream.Stream; import java.util.stream.Stream;
import lombok.extern.slf4j.Slf4j; import lombok.extern.slf4j.Slf4j;
import static com.tencent.supersonic.chat.api.pojo.SchemaElementType.*;
@Slf4j @Slf4j
public class ContextInheritParser implements SemanticParser { public class ContextInheritParser implements SemanticParser {
@@ -31,7 +36,7 @@ public class ContextInheritParser implements SemanticParser {
new AbstractMap.SimpleEntry<>(DIMENSION, Arrays.asList(DIMENSION, VALUE)), new AbstractMap.SimpleEntry<>(DIMENSION, Arrays.asList(DIMENSION, VALUE)),
new AbstractMap.SimpleEntry<>(VALUE, Arrays.asList(VALUE, DIMENSION)), new AbstractMap.SimpleEntry<>(VALUE, Arrays.asList(VALUE, DIMENSION)),
new AbstractMap.SimpleEntry<>(ENTITY, Arrays.asList(ENTITY)), new AbstractMap.SimpleEntry<>(ENTITY, Arrays.asList(ENTITY)),
new AbstractMap.SimpleEntry<>(DOMAIN, Arrays.asList(DOMAIN)), new AbstractMap.SimpleEntry<>(MODEL, Arrays.asList(MODEL)),
new AbstractMap.SimpleEntry<>(ID, Arrays.asList(ID)) new AbstractMap.SimpleEntry<>(ID, Arrays.asList(ID))
).collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue)); ).collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
@@ -41,9 +46,9 @@ public class ContextInheritParser implements SemanticParser {
return; return;
} }
Long domainId = chatContext.getParseInfo().getDomainId(); Long modelId = chatContext.getParseInfo().getModelId();
List<SchemaElementMatch> elementMatches = queryContext.getMapInfo() List<SchemaElementMatch> elementMatches = queryContext.getMapInfo()
.getMatchedElements(domainId); .getMatchedElements(modelId);
List<SchemaElementMatch> matchesToInherit = new ArrayList<>(); List<SchemaElementMatch> matchesToInherit = new ArrayList<>();
for (SchemaElementMatch match : chatContext.getParseInfo().getElementMatches()) { for (SchemaElementMatch match : chatContext.getParseInfo().getElementMatches()) {
@@ -51,7 +56,7 @@ public class ContextInheritParser implements SemanticParser {
// mutual exclusive element types should not be inherited // mutual exclusive element types should not be inherited
RuleSemanticQuery ruleQuery = QueryManager.getRuleQuery(chatContext.getParseInfo().getQueryMode()); RuleSemanticQuery ruleQuery = QueryManager.getRuleQuery(chatContext.getParseInfo().getQueryMode());
if (!containsTypes(elementMatches, matchType, ruleQuery)) { if (!containsTypes(elementMatches, matchType, ruleQuery)) {
match.setMode(SchemaElementMatch.MatchMode.INHERIT); match.setInherited(true);
matchesToInherit.add(match); matchesToInherit.add(match);
} }
} }
@@ -59,11 +64,24 @@ public class ContextInheritParser implements SemanticParser {
List<RuleSemanticQuery> queries = RuleSemanticQuery.resolve(elementMatches, queryContext); List<RuleSemanticQuery> queries = RuleSemanticQuery.resolve(elementMatches, queryContext);
for (RuleSemanticQuery query : queries) { for (RuleSemanticQuery query : queries) {
query.fillParseInfo(domainId, chatContext); query.fillParseInfo(modelId, queryContext, chatContext);
if (existSameQuery(query.getParseInfo().getModelId(), query.getQueryMode(), queryContext)) {
continue;
}
queryContext.getCandidateQueries().add(query); queryContext.getCandidateQueries().add(query);
} }
} }
private boolean existSameQuery(Long modelId, String queryMode, QueryContext queryContext) {
for (SemanticQuery semanticQuery : queryContext.getCandidateQueries()) {
if (semanticQuery.getQueryMode().equals(queryMode)
&& semanticQuery.getParseInfo().getModelId().equals(modelId)) {
return true;
}
}
return false;
}
private boolean containsTypes(List<SchemaElementMatch> matches, SchemaElementType matchType, private boolean containsTypes(List<SchemaElementMatch> matches, SchemaElementType matchType,
RuleSemanticQuery ruleQuery) { RuleSemanticQuery ruleQuery) {
List<SchemaElementType> types = MUTUAL_EXCLUSIVE_MAP.get(matchType); List<SchemaElementType> types = MUTUAL_EXCLUSIVE_MAP.get(matchType);
@@ -79,16 +97,18 @@ public class ContextInheritParser implements SemanticParser {
} }
protected boolean shouldInherit(QueryContext queryContext, ChatContext chatContext) { protected boolean shouldInherit(QueryContext queryContext, ChatContext chatContext) {
Long contextDomainId = chatContext.getParseInfo().getDomainId(); Long contextmodelId = chatContext.getParseInfo().getModelId();
if (queryContext.getMapInfo().getMatchedElements(contextDomainId) == null) { // if map info doesn't contain the same Model of the context,
// no inheritance could be done
if (queryContext.getMapInfo().getMatchedElements(contextmodelId) == null) {
return false; return false;
} }
// if candidates have only one MetricDomain mode and context has value filter , count in context // if candidates only have MetricModel mode, count in context
List<SemanticQuery> candidateQueries = queryContext.getCandidateQueries().stream() List<SemanticQuery> metricModelQueries = queryContext.getCandidateQueries().stream()
.filter(semanticQuery -> semanticQuery.getParseInfo().getDomainId().equals(contextDomainId)).collect( .filter(query -> query instanceof MetricModelQuery).collect(
Collectors.toList()); Collectors.toList());
if (candidateQueries.size() == 1 && (candidateQueries.get(0) instanceof MetricDomainQuery)) { if (metricModelQueries.size() == queryContext.getCandidateQueries().size()) {
return true; return true;
} else { } else {
return queryContext.getCandidateQueries().size() == 0; return queryContext.getCandidateQueries().size() == 0;

View File

@@ -1,11 +1,12 @@
package com.tencent.supersonic.chat.parser.rule; package com.tencent.supersonic.chat.parser.rule;
import com.tencent.supersonic.chat.api.component.SemanticParser; import com.tencent.supersonic.chat.api.component.SemanticParser;
import com.tencent.supersonic.chat.api.pojo.*; import com.tencent.supersonic.chat.api.pojo.ChatContext;
import com.tencent.supersonic.chat.api.pojo.QueryContext;
import com.tencent.supersonic.chat.api.pojo.SchemaElementMatch;
import com.tencent.supersonic.chat.api.pojo.SchemaMapInfo;
import com.tencent.supersonic.chat.query.rule.RuleSemanticQuery; import com.tencent.supersonic.chat.query.rule.RuleSemanticQuery;
import java.util.List;
import java.util.*;
import lombok.extern.slf4j.Slf4j; import lombok.extern.slf4j.Slf4j;
@Slf4j @Slf4j
@@ -16,12 +17,12 @@ public class QueryModeParser implements SemanticParser {
SchemaMapInfo mapInfo = queryContext.getMapInfo(); SchemaMapInfo mapInfo = queryContext.getMapInfo();
// iterate all schemaElementMatches to resolve semantic query // iterate all schemaElementMatches to resolve semantic query
for (Long domainId : mapInfo.getMatchedDomains()) { for (Long modelId : mapInfo.getMatchedModels()) {
List<SchemaElementMatch> elementMatches = mapInfo.getMatchedElements(domainId); List<SchemaElementMatch> elementMatches = mapInfo.getMatchedElements(modelId);
List<RuleSemanticQuery> queries = RuleSemanticQuery.resolve(elementMatches, queryContext); List<RuleSemanticQuery> queries = RuleSemanticQuery.resolve(elementMatches, queryContext);
for (RuleSemanticQuery query : queries) { for (RuleSemanticQuery query : queries) {
query.fillParseInfo(domainId, chatContext); query.fillParseInfo(modelId, queryContext, chatContext);
queryContext.getCandidateQueries().add(query); queryContext.getCandidateQueries().add(query);
} }
} }

View File

@@ -4,21 +4,21 @@ import com.tencent.supersonic.chat.api.component.SemanticParser;
import com.tencent.supersonic.chat.api.component.SemanticQuery; import com.tencent.supersonic.chat.api.component.SemanticQuery;
import com.tencent.supersonic.chat.api.pojo.ChatContext; import com.tencent.supersonic.chat.api.pojo.ChatContext;
import com.tencent.supersonic.chat.api.pojo.QueryContext; import com.tencent.supersonic.chat.api.pojo.QueryContext;
import com.tencent.supersonic.chat.query.rule.RuleSemanticQuery;
import com.tencent.supersonic.chat.query.QueryManager; import com.tencent.supersonic.chat.query.QueryManager;
import com.tencent.supersonic.chat.query.rule.RuleSemanticQuery;
import com.tencent.supersonic.common.pojo.Constants; import com.tencent.supersonic.common.pojo.Constants;
import com.tencent.supersonic.common.pojo.DateConf; import com.tencent.supersonic.common.pojo.DateConf;
import com.xkzhangsan.time.nlp.TimeNLP;
import com.xkzhangsan.time.nlp.TimeNLPUtil;
import java.text.DateFormat; import java.text.DateFormat;
import java.text.ParseException; import java.text.ParseException;
import java.text.SimpleDateFormat; import java.text.SimpleDateFormat;
import java.time.LocalDate; import java.time.LocalDate;
import java.util.*; import java.util.Date;
import java.util.List;
import java.util.Stack;
import java.util.regex.Matcher; import java.util.regex.Matcher;
import java.util.regex.Pattern; import java.util.regex.Pattern;
import com.xkzhangsan.time.nlp.TimeNLP;
import com.xkzhangsan.time.nlp.TimeNLPUtil;
import lombok.extern.slf4j.Slf4j; import lombok.extern.slf4j.Slf4j;
import org.apache.logging.log4j.util.Strings; import org.apache.logging.log4j.util.Strings;
@@ -45,12 +45,16 @@ public class TimeRangeParser implements SemanticParser {
if (queryContext.getCandidateQueries().size() > 0) { if (queryContext.getCandidateQueries().size() > 0) {
for (SemanticQuery query : queryContext.getCandidateQueries()) { for (SemanticQuery query : queryContext.getCandidateQueries()) {
query.getParseInfo().setDateInfo(dateConf); query.getParseInfo().setDateInfo(dateConf);
query.getParseInfo().setScore(query.getParseInfo().getScore()
+ dateConf.getDetectWord().length());
} }
} else if (QueryManager.containsRuleQuery(chatContext.getParseInfo().getQueryMode())) { } else if (QueryManager.containsRuleQuery(chatContext.getParseInfo().getQueryMode())) {
RuleSemanticQuery semanticQuery = QueryManager.createRuleQuery( RuleSemanticQuery semanticQuery = QueryManager.createRuleQuery(
chatContext.getParseInfo().getQueryMode()); chatContext.getParseInfo().getQueryMode());
// inherit parse info from context // inherit parse info from context
chatContext.getParseInfo().setDateInfo(dateConf); chatContext.getParseInfo().setDateInfo(dateConf);
chatContext.getParseInfo().setScore(chatContext.getParseInfo().getScore()
+ dateConf.getDetectWord().length());
semanticQuery.setParseInfo(chatContext.getParseInfo()); semanticQuery.setParseInfo(chatContext.getParseInfo());
queryContext.getCandidateQueries().add(semanticQuery); queryContext.getCandidateQueries().add(semanticQuery);
} }
@@ -60,41 +64,48 @@ public class TimeRangeParser implements SemanticParser {
private DateConf parseDateCN(String queryText) { private DateConf parseDateCN(String queryText) {
Date startDate = null; Date startDate = null;
Date endDate; Date endDate;
String detectWord = null;
List<TimeNLP> times = TimeNLPUtil.parse(queryText); List<TimeNLP> times = TimeNLPUtil.parse(queryText);
if (times.size() > 0) { if (times.size() > 0) {
startDate = times.get(0).getTime(); startDate = times.get(0).getTime();
detectWord = times.get(0).getTimeExpression();
} else { } else {
return null; return null;
} }
if (times.size() > 1) { if (times.size() > 1) {
endDate = times.get(1).getTime(); endDate = times.get(1).getTime();
detectWord += "~" + times.get(0).getTimeExpression();
} else { } else {
endDate = startDate; endDate = startDate;
} }
return getDateConf(startDate, endDate); return getDateConf(startDate, endDate, detectWord);
} }
private DateConf parseDateNumber(String queryText) { private DateConf parseDateNumber(String queryText) {
String startDate; String startDate;
String endDate = null; String endDate = null;
String detectWord = null;
Matcher dateMatcher = DATE_PATTERN_NUMBER.matcher(queryText); Matcher dateMatcher = DATE_PATTERN_NUMBER.matcher(queryText);
if (dateMatcher.find()) { if (dateMatcher.find()) {
startDate = dateMatcher.group(); startDate = dateMatcher.group();
detectWord = startDate;
} else { } else {
return null; return null;
} }
if (dateMatcher.find()) { if (dateMatcher.find()) {
endDate = dateMatcher.group(); endDate = dateMatcher.group();
detectWord += "~" + endDate;
} }
endDate = endDate != null ? endDate : startDate; endDate = endDate != null ? endDate : startDate;
try { try {
return getDateConf(DATE_FORMAT_NUMBER.parse(startDate), DATE_FORMAT_NUMBER.parse(endDate)); return getDateConf(DATE_FORMAT_NUMBER.parse(startDate), DATE_FORMAT_NUMBER.parse(endDate), detectWord);
} catch (ParseException e) { } catch (ParseException e) {
return null; return null;
} }
@@ -134,11 +145,11 @@ public class TimeRangeParser implements SemanticParser {
} }
days = days * num; days = days * num;
info.setDateMode(DateConf.DateMode.RECENT); info.setDateMode(DateConf.DateMode.RECENT);
String text = "" + num + zhPeriod; String detectWord = "" + num + zhPeriod;
if (Strings.isNotEmpty(m.group("periodStr"))) { if (Strings.isNotEmpty(m.group("periodStr"))) {
text = m.group("periodStr"); detectWord = m.group("periodStr");
} }
info.setText(text); info.setDetectWord(detectWord);
info.setStartDate(LocalDate.now().minusDays(days).toString()); info.setStartDate(LocalDate.now().minusDays(days).toString());
info.setUnit(num); info.setUnit(num);
@@ -173,7 +184,7 @@ public class TimeRangeParser implements SemanticParser {
return stack.stream().mapToInt(s -> s).sum(); return stack.stream().mapToInt(s -> s).sum();
} }
private DateConf getDateConf(Date startDate, Date endDate) { private DateConf getDateConf(Date startDate, Date endDate, String detectWord) {
if (startDate == null || endDate == null) { if (startDate == null || endDate == null) {
return null; return null;
} }
@@ -182,6 +193,7 @@ public class TimeRangeParser implements SemanticParser {
info.setDateMode(DateConf.DateMode.BETWEEN); info.setDateMode(DateConf.DateMode.BETWEEN);
info.setStartDate(DATE_FORMAT.format(startDate)); info.setStartDate(DATE_FORMAT.format(startDate));
info.setEndDate(DATE_FORMAT.format(endDate)); info.setEndDate(DATE_FORMAT.format(endDate));
info.setDetectWord(detectWord);
return info; return info;
} }

View File

@@ -1,7 +1,6 @@
package com.tencent.supersonic.chat.persistence.dataobject; package com.tencent.supersonic.chat.persistence.dataobject;
import java.util.Date; import java.util.Date;
import lombok.Data; import lombok.Data;
import lombok.ToString; import lombok.ToString;
@@ -15,7 +14,7 @@ public class ChatConfigDO {
*/ */
private Long id; private Long id;
private Long domainId; private Long modelId;
private String chatDetailConfig; private String chatDetailConfig;

View File

@@ -1,11 +1,10 @@
package com.tencent.supersonic.chat.persistence.dataobject; package com.tencent.supersonic.chat.persistence.dataobject;
import java.util.ArrayList;
import java.util.List;
import com.tencent.supersonic.chat.config.DefaultMetric; import com.tencent.supersonic.chat.config.DefaultMetric;
import com.tencent.supersonic.chat.config.Dim4Dict; import com.tencent.supersonic.chat.config.Dim4Dict;
import java.util.ArrayList;
import java.util.List;
import lombok.Data; import lombok.Data;
import lombok.ToString; import lombok.ToString;
@@ -14,14 +13,14 @@ import lombok.ToString;
@ToString @ToString
public class DimValueDO { public class DimValueDO {
private Long domainId; private Long modelId;
private List<DefaultMetric> defaultMetricDescList = new ArrayList<>(); private List<DefaultMetric> defaultMetricDescList = new ArrayList<>();
private List<Dim4Dict> dimensions = new ArrayList<>(); private List<Dim4Dict> dimensions = new ArrayList<>();
public DimValueDO setDomainId(Long domainId) { public DimValueDO setModelId(Long modelId) {
this.domainId = domainId; this.modelId = modelId;
return this; return this;
} }

View File

@@ -3,8 +3,9 @@ package com.tencent.supersonic.chat.persistence.dataobject;
import java.util.Date; import java.util.Date;
public class PluginDO { public class PluginDO {
/** /**
* *
*/ */
private Long id; private Long id;
@@ -14,71 +15,69 @@ public class PluginDO {
private String type; private String type;
/** /**
* *
*/ */
private String domain; private String model;
/** /**
* *
*/ */
private String pattern; private String pattern;
/** /**
* *
*/ */
private String parseMode; private String parseMode;
/** /**
* *
*/ */
private String name; private String name;
/** /**
* *
*/ */
private Date createdAt; private Date createdAt;
/** /**
* *
*/ */
private String createdBy; private String createdBy;
/** /**
* *
*/ */
private Date updatedAt; private Date updatedAt;
/** /**
* *
*/ */
private String updatedBy; private String updatedBy;
/** /**
* *
*/ */
private String parseModeConfig; private String parseModeConfig;
/** /**
* *
*/ */
private String config; private String config;
/** /**
* *
*/ */
private String comment; private String comment;
/** /**
* * @return id
* @return id
*/ */
public Long getId() { public Long getId() {
return id; return id;
} }
/** /**
* * @param id
* @param id
*/ */
public void setId(Long id) { public void setId(Long id) {
this.id = id; this.id = id;
@@ -86,6 +85,7 @@ public class PluginDO {
/** /**
* DASHBOARD,WIDGET,URL * DASHBOARD,WIDGET,URL
*
* @return type DASHBOARD,WIDGET,URL * @return type DASHBOARD,WIDGET,URL
*/ */
public String getType() { public String getType() {
@@ -94,6 +94,7 @@ public class PluginDO {
/** /**
* DASHBOARD,WIDGET,URL * DASHBOARD,WIDGET,URL
*
* @param type DASHBOARD,WIDGET,URL * @param type DASHBOARD,WIDGET,URL
*/ */
public void setType(String type) { public void setType(String type) {
@@ -101,176 +102,154 @@ public class PluginDO {
} }
/** /**
* * @return model
* @return domain
*/ */
public String getDomain() { public String getModel() {
return domain; return model;
} }
/** /**
* * @param model
* @param domain
*/ */
public void setDomain(String domain) { public void setModel(String model) {
this.domain = domain == null ? null : domain.trim(); this.model = model == null ? null : model.trim();
} }
/** /**
* * @return pattern
* @return pattern
*/ */
public String getPattern() { public String getPattern() {
return pattern; return pattern;
} }
/** /**
* * @param pattern
* @param pattern
*/ */
public void setPattern(String pattern) { public void setPattern(String pattern) {
this.pattern = pattern == null ? null : pattern.trim(); this.pattern = pattern == null ? null : pattern.trim();
} }
/** /**
* * @return parse_mode
* @return parse_mode
*/ */
public String getParseMode() { public String getParseMode() {
return parseMode; return parseMode;
} }
/** /**
* * @param parseMode
* @param parseMode
*/ */
public void setParseMode(String parseMode) { public void setParseMode(String parseMode) {
this.parseMode = parseMode == null ? null : parseMode.trim(); this.parseMode = parseMode == null ? null : parseMode.trim();
} }
/** /**
* * @return name
* @return name
*/ */
public String getName() { public String getName() {
return name; return name;
} }
/** /**
* * @param name
* @param name
*/ */
public void setName(String name) { public void setName(String name) {
this.name = name == null ? null : name.trim(); this.name = name == null ? null : name.trim();
} }
/** /**
* * @return created_at
* @return created_at
*/ */
public Date getCreatedAt() { public Date getCreatedAt() {
return createdAt; return createdAt;
} }
/** /**
* * @param createdAt
* @param createdAt
*/ */
public void setCreatedAt(Date createdAt) { public void setCreatedAt(Date createdAt) {
this.createdAt = createdAt; this.createdAt = createdAt;
} }
/** /**
* * @return created_by
* @return created_by
*/ */
public String getCreatedBy() { public String getCreatedBy() {
return createdBy; return createdBy;
} }
/** /**
* * @param createdBy
* @param createdBy
*/ */
public void setCreatedBy(String createdBy) { public void setCreatedBy(String createdBy) {
this.createdBy = createdBy == null ? null : createdBy.trim(); this.createdBy = createdBy == null ? null : createdBy.trim();
} }
/** /**
* * @return updated_at
* @return updated_at
*/ */
public Date getUpdatedAt() { public Date getUpdatedAt() {
return updatedAt; return updatedAt;
} }
/** /**
* * @param updatedAt
* @param updatedAt
*/ */
public void setUpdatedAt(Date updatedAt) { public void setUpdatedAt(Date updatedAt) {
this.updatedAt = updatedAt; this.updatedAt = updatedAt;
} }
/** /**
* * @return updated_by
* @return updated_by
*/ */
public String getUpdatedBy() { public String getUpdatedBy() {
return updatedBy; return updatedBy;
} }
/** /**
* * @param updatedBy
* @param updatedBy
*/ */
public void setUpdatedBy(String updatedBy) { public void setUpdatedBy(String updatedBy) {
this.updatedBy = updatedBy == null ? null : updatedBy.trim(); this.updatedBy = updatedBy == null ? null : updatedBy.trim();
} }
/** /**
* * @return parse_mode_config
* @return parse_mode_config
*/ */
public String getParseModeConfig() { public String getParseModeConfig() {
return parseModeConfig; return parseModeConfig;
} }
/** /**
* * @param parseModeConfig
* @param parseModeConfig
*/ */
public void setParseModeConfig(String parseModeConfig) { public void setParseModeConfig(String parseModeConfig) {
this.parseModeConfig = parseModeConfig == null ? null : parseModeConfig.trim(); this.parseModeConfig = parseModeConfig == null ? null : parseModeConfig.trim();
} }
/** /**
* * @return config
* @return config
*/ */
public String getConfig() { public String getConfig() {
return config; return config;
} }
/** /**
* * @param config
* @param config
*/ */
public void setConfig(String config) { public void setConfig(String config) {
this.config = config == null ? null : config.trim(); this.config = config == null ? null : config.trim();
} }
/** /**
* * @return comment
* @return comment
*/ */
public String getComment() { public String getComment() {
return comment; return comment;
} }
/** /**
* * @param comment
* @param comment
*/ */
public void setComment(String comment) { public void setComment(String comment) {
this.comment = comment == null ? null : comment.trim(); this.comment = comment == null ? null : comment.trim();

View File

@@ -5,6 +5,7 @@ import java.util.Date;
import java.util.List; import java.util.List;
public class PluginDOExample { public class PluginDOExample {
/** /**
* s2_plugin * s2_plugin
*/ */
@@ -31,7 +32,6 @@ public class PluginDOExample {
protected Integer limitEnd; protected Integer limitEnd;
/** /**
*
* @mbg.generated * @mbg.generated
*/ */
public PluginDOExample() { public PluginDOExample() {
@@ -39,7 +39,6 @@ public class PluginDOExample {
} }
/** /**
*
* @mbg.generated * @mbg.generated
*/ */
public void setOrderByClause(String orderByClause) { public void setOrderByClause(String orderByClause) {
@@ -47,7 +46,6 @@ public class PluginDOExample {
} }
/** /**
*
* @mbg.generated * @mbg.generated
*/ */
public String getOrderByClause() { public String getOrderByClause() {
@@ -55,7 +53,6 @@ public class PluginDOExample {
} }
/** /**
*
* @mbg.generated * @mbg.generated
*/ */
public void setDistinct(boolean distinct) { public void setDistinct(boolean distinct) {
@@ -63,7 +60,6 @@ public class PluginDOExample {
} }
/** /**
*
* @mbg.generated * @mbg.generated
*/ */
public boolean isDistinct() { public boolean isDistinct() {
@@ -71,7 +67,6 @@ public class PluginDOExample {
} }
/** /**
*
* @mbg.generated * @mbg.generated
*/ */
public List<Criteria> getOredCriteria() { public List<Criteria> getOredCriteria() {
@@ -79,7 +74,6 @@ public class PluginDOExample {
} }
/** /**
*
* @mbg.generated * @mbg.generated
*/ */
public void or(Criteria criteria) { public void or(Criteria criteria) {
@@ -87,7 +81,6 @@ public class PluginDOExample {
} }
/** /**
*
* @mbg.generated * @mbg.generated
*/ */
public Criteria or() { public Criteria or() {
@@ -97,7 +90,6 @@ public class PluginDOExample {
} }
/** /**
*
* @mbg.generated * @mbg.generated
*/ */
public Criteria createCriteria() { public Criteria createCriteria() {
@@ -109,7 +101,6 @@ public class PluginDOExample {
} }
/** /**
*
* @mbg.generated * @mbg.generated
*/ */
protected Criteria createCriteriaInternal() { protected Criteria createCriteriaInternal() {
@@ -118,7 +109,6 @@ public class PluginDOExample {
} }
/** /**
*
* @mbg.generated * @mbg.generated
*/ */
public void clear() { public void clear() {
@@ -128,15 +118,13 @@ public class PluginDOExample {
} }
/** /**
*
* @mbg.generated * @mbg.generated
*/ */
public void setLimitStart(Integer limitStart) { public void setLimitStart(Integer limitStart) {
this.limitStart=limitStart; this.limitStart = limitStart;
} }
/** /**
*
* @mbg.generated * @mbg.generated
*/ */
public Integer getLimitStart() { public Integer getLimitStart() {
@@ -144,15 +132,13 @@ public class PluginDOExample {
} }
/** /**
*
* @mbg.generated * @mbg.generated
*/ */
public void setLimitEnd(Integer limitEnd) { public void setLimitEnd(Integer limitEnd) {
this.limitEnd=limitEnd; this.limitEnd = limitEnd;
} }
/** /**
*
* @mbg.generated * @mbg.generated
*/ */
public Integer getLimitEnd() { public Integer getLimitEnd() {
@@ -163,6 +149,7 @@ public class PluginDOExample {
* s2_plugin null * s2_plugin null
*/ */
protected abstract static class GeneratedCriteria { protected abstract static class GeneratedCriteria {
protected List<Criterion> criteria; protected List<Criterion> criteria;
protected GeneratedCriteria() { protected GeneratedCriteria() {
@@ -333,73 +320,73 @@ public class PluginDOExample {
return (Criteria) this; return (Criteria) this;
} }
public Criteria andDomainIsNull() { public Criteria andModelIsNull() {
addCriterion("domain is null"); addCriterion("model is null");
return (Criteria) this; return (Criteria) this;
} }
public Criteria andDomainIsNotNull() { public Criteria andModelIsNotNull() {
addCriterion("domain is not null"); addCriterion("model is not null");
return (Criteria) this; return (Criteria) this;
} }
public Criteria andDomainEqualTo(String value) { public Criteria andModelEqualTo(String value) {
addCriterion("domain =", value, "domain"); addCriterion("model =", value, "model");
return (Criteria) this; return (Criteria) this;
} }
public Criteria andDomainNotEqualTo(String value) { public Criteria andModelNotEqualTo(String value) {
addCriterion("domain <>", value, "domain"); addCriterion("model <>", value, "model");
return (Criteria) this; return (Criteria) this;
} }
public Criteria andDomainGreaterThan(String value) { public Criteria andModelGreaterThan(String value) {
addCriterion("domain >", value, "domain"); addCriterion("model >", value, "model");
return (Criteria) this; return (Criteria) this;
} }
public Criteria andDomainGreaterThanOrEqualTo(String value) { public Criteria andModelGreaterThanOrEqualTo(String value) {
addCriterion("domain >=", value, "domain"); addCriterion("model >=", value, "model");
return (Criteria) this; return (Criteria) this;
} }
public Criteria andDomainLessThan(String value) { public Criteria andModelLessThan(String value) {
addCriterion("domain <", value, "domain"); addCriterion("model <", value, "model");
return (Criteria) this; return (Criteria) this;
} }
public Criteria andDomainLessThanOrEqualTo(String value) { public Criteria andModelLessThanOrEqualTo(String value) {
addCriterion("domain <=", value, "domain"); addCriterion("model <=", value, "model");
return (Criteria) this; return (Criteria) this;
} }
public Criteria andDomainLike(String value) { public Criteria andModelLike(String value) {
addCriterion("domain like", value, "domain"); addCriterion("model like", value, "model");
return (Criteria) this; return (Criteria) this;
} }
public Criteria andDomainNotLike(String value) { public Criteria andModelNotLike(String value) {
addCriterion("domain not like", value, "domain"); addCriterion("model not like", value, "model");
return (Criteria) this; return (Criteria) this;
} }
public Criteria andDomainIn(List<String> values) { public Criteria andModelIn(List<String> values) {
addCriterion("domain in", values, "domain"); addCriterion("model in", values, "model");
return (Criteria) this; return (Criteria) this;
} }
public Criteria andDomainNotIn(List<String> values) { public Criteria andModelNotIn(List<String> values) {
addCriterion("domain not in", values, "domain"); addCriterion("model not in", values, "model");
return (Criteria) this; return (Criteria) this;
} }
public Criteria andDomainBetween(String value1, String value2) { public Criteria andModelBetween(String value1, String value2) {
addCriterion("domain between", value1, value2, "domain"); addCriterion("model between", value1, value2, "model");
return (Criteria) this; return (Criteria) this;
} }
public Criteria andDomainNotBetween(String value1, String value2) { public Criteria andModelNotBetween(String value1, String value2) {
addCriterion("domain not between", value1, value2, "domain"); addCriterion("model not between", value1, value2, "model");
return (Criteria) this; return (Criteria) this;
} }
@@ -888,6 +875,7 @@ public class PluginDOExample {
* s2_plugin null * s2_plugin null
*/ */
public static class Criterion { public static class Criterion {
private String condition; private String condition;
private Object value; private Object value;

View File

@@ -14,5 +14,5 @@ public interface ChatConfigMapper {
List<ChatConfigDO> search(ChatConfigFilterInternal filterInternal); List<ChatConfigDO> search(ChatConfigFilterInternal filterInternal);
ChatConfigDO fetchConfigByDomainId(Long domainId); ChatConfigDO fetchConfigByModelId(Long modelId);
} }

View File

@@ -2,9 +2,8 @@ package com.tencent.supersonic.chat.persistence.mapper;
import com.tencent.supersonic.chat.persistence.dataobject.ChatQueryDO; import com.tencent.supersonic.chat.persistence.dataobject.ChatQueryDO;
import com.tencent.supersonic.chat.persistence.dataobject.ChatQueryDOExample; import com.tencent.supersonic.chat.persistence.dataobject.ChatQueryDOExample;
import org.apache.ibatis.annotations.Mapper;
import java.util.List; import java.util.List;
import org.apache.ibatis.annotations.Mapper;
@Mapper @Mapper
public interface ChatQueryDOMapper { public interface ChatQueryDOMapper {

Some files were not shown because too many files have changed in this diff Show More