mirror of
https://github.com/tencentmusic/supersonic.git
synced 2025-12-10 19:51:00 +00:00
Compare commits
25 Commits
1f6d217b26
...
master
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
593d26a072 | ||
|
|
9162b922c4 | ||
|
|
1d9324f689 | ||
|
|
6c5f8fce40 | ||
|
|
04b1edb2e2 | ||
|
|
9857256488 | ||
|
|
d695bed75d | ||
|
|
7490dabdc3 | ||
|
|
fad28ee5ac | ||
|
|
353c8d8b16 | ||
|
|
3dd53bad89 | ||
|
|
2d39ebf38b | ||
|
|
6c472e1c76 | ||
|
|
431aa60e4d | ||
|
|
25df22758a | ||
|
|
9af6499491 | ||
|
|
c992e57b13 | ||
|
|
80aaabe58b | ||
|
|
5a4fd2b888 | ||
|
|
5df0b87da9 | ||
|
|
ab24b1777a | ||
|
|
ff76f8edbd | ||
|
|
76745f38a4 | ||
|
|
ce4cdb62ab | ||
|
|
c2ce3a75b7 |
199
CHANGELOG.md
199
CHANGELOG.md
@@ -3,6 +3,205 @@
|
|||||||
- All notable changes to this project will be documented in this file.
|
- All notable changes to this project will be documented in this file.
|
||||||
- "Breaking Changes" describes any changes that may break existing functionality or cause
|
- "Breaking Changes" describes any changes that may break existing functionality or cause
|
||||||
compatibility issues with previous versions.
|
compatibility issues with previous versions.
|
||||||
|
## SuperSonic [1.0.0] - 2025-08-05
|
||||||
|
|
||||||
|
### 重大特性变更 / Major Features
|
||||||
|
|
||||||
|
#### 多数据库支持扩展 / Multi-Database Support
|
||||||
|
- **Oracle数据库支持**: 新增Oracle数据库引擎类型及适配器 ([8eeed87ba](https://github.com/tencentmusic/supersonic/commit/8eeed87ba) by supersonicbi)
|
||||||
|
- **StarRocks支持**: 支持StarRocks和多catalog功能 ([33268bf3d](https://github.com/tencentmusic/supersonic/commit/33268bf3d) by zyclove)
|
||||||
|
- **SAP HANA支持**: 新增SAP HANA数据库适配支持 ([2e28a4c7a](https://github.com/tencentmusic/supersonic/commit/2e28a4c7a) by wwsheng009)
|
||||||
|
- **DuckDB支持**: 支持DuckDB数据库 ([a058dc8b6](https://github.com/tencentmusic/supersonic/commit/a058dc8b6) by jerryjzhang)
|
||||||
|
- **Kyuubi支持**: 支持Kyuubi Presto Trino ([5e3bafb95](https://github.com/tencentmusic/supersonic/commit/5e3bafb95) by zyclove)
|
||||||
|
- **OpenSearch支持**: 新增OpenSearch支持 ([d942d35c9](https://github.com/tencentmusic/supersonic/commit/d942d35c9) by zyclove)
|
||||||
|
|
||||||
|
#### 智能问答增强 / AI-Enhanced Query Processing
|
||||||
|
- **LLM纠错器**: 新增LLM物理SQL纠错器 ([f899d23b6](https://github.com/tencentmusic/supersonic/commit/f899d23b6) by 柯慕灵)
|
||||||
|
- **记忆管理**: Agent记忆管理启用few-shot优先机制 ([fae9118c2](https://github.com/tencentmusic/supersonic/commit/fae9118c2) by feelshana)
|
||||||
|
- **结构化查询**: 支持struct查询中的offset子句 ([d2a43a99c](https://github.com/tencentmusic/supersonic/commit/d2a43a99c) by jerryjzhang)
|
||||||
|
- **向量召回优化**: 优化嵌入向量召回机制 ([8c6ae6252](https://github.com/tencentmusic/supersonic/commit/8c6ae6252) by lexluo09)
|
||||||
|
|
||||||
|
#### 权限管理系统 / Permission Management
|
||||||
|
- **Agent权限**: 支持agent级别的权限管理 ([b5aa6e046](https://github.com/tencentmusic/supersonic/commit/b5aa6e046) by jerryjzhang)
|
||||||
|
- **用户管理**: 支持用户删除功能 ([1c9cf788c](https://github.com/tencentmusic/supersonic/commit/1c9cf788c) by supersonicbi)
|
||||||
|
- **鉴权优化**: 全面优化鉴权与召回机制 ([1faf84e37](https://github.com/tencentmusic/supersonic/commit/1faf84e37), [7e6639df8](https://github.com/tencentmusic/supersonic/commit/7e6639df8) by guilinlewis)
|
||||||
|
|
||||||
|
### 架构升级 / Architecture Upgrades
|
||||||
|
|
||||||
|
#### 核心框架升级 / Core Framework Upgrades
|
||||||
|
- **SpringBoot 3升级**: 完成SpringBoot 3.x升级 ([07f6be51c](https://github.com/tencentmusic/supersonic/commit/07f6be51c) by mislayming)
|
||||||
|
- **依赖升级**: 升级依赖包并修复安全漏洞 ([232a20227](https://github.com/tencentmusic/supersonic/commit/232a20227) by beat4ocean)
|
||||||
|
- **LangChain4j更新**: 替换已废弃的LangChain4j APIs ([acffc03c7](https://github.com/tencentmusic/supersonic/commit/acffc03c7) by beat4ocean)
|
||||||
|
- **Swagger升级**: 使用SpringDoc支持Swagger在Spring 3.x ([758d170bb](https://github.com/tencentmusic/supersonic/commit/758d170bb) by jerryjzhang)
|
||||||
|
|
||||||
|
#### 许可证变更 / License Changes
|
||||||
|
- **Apache 2.0**: 从MIT更改为Apache 2.0许可证 ([0aa002882](https://github.com/tencentmusic/supersonic/commit/0aa002882) by jerryjzhang)
|
||||||
|
|
||||||
|
### 性能优化 / Performance Improvements
|
||||||
|
|
||||||
|
#### 系统性能 / System Performance
|
||||||
|
- **GC优化**: 实现Generational ZGC ([3fc1ec42b](https://github.com/tencentmusic/supersonic/commit/3fc1ec42b) by beat4ocean)
|
||||||
|
- **Docker优化**: 减少Docker镜像体积 ([614917ba7](https://github.com/tencentmusic/supersonic/commit/614917ba7) by kino)
|
||||||
|
- **并行处理**: 嵌入向量并行执行优化 ([8c6ae6252](https://github.com/tencentmusic/supersonic/commit/8c6ae6252) by lexluo09)
|
||||||
|
- **记忆评估**: 记忆评估性能优化 ([524ec38ed](https://github.com/tencentmusic/supersonic/commit/524ec38ed) by yudong)
|
||||||
|
- **多平台构建**: 支持Docker多平台构建 ([da6d28c18](https://github.com/tencentmusic/supersonic/commit/da6d28c18) by jerryjzhang)
|
||||||
|
|
||||||
|
#### 数据处理优化 / Data Processing Optimization
|
||||||
|
- **日期格式**: 支持更多日期字符串格式 ([2b13866c0](https://github.com/tencentmusic/supersonic/commit/2b13866c0) by supersonicbi)
|
||||||
|
- **SQL优化**: 优化SQL生成和执行性能 ([0ab764329](https://github.com/tencentmusic/supersonic/commit/0ab764329) by jerryjzhang)
|
||||||
|
- **模型关联**: 优化模型关联查询性能 ([47c2595fb](https://github.com/tencentmusic/supersonic/commit/47c2595fb) by Willy-J)
|
||||||
|
|
||||||
|
### 功能增强 / Feature Enhancements
|
||||||
|
|
||||||
|
#### 前端界面优化 / Frontend Improvements
|
||||||
|
- **图表导出**: 消息支持导出图表图片 ([ce9ae1c0c](https://github.com/tencentmusic/supersonic/commit/ce9ae1c0c) by pisces)
|
||||||
|
- **路由重构**: 重构语义建模路由交互 ([82c63a7f2](https://github.com/tencentmusic/supersonic/commit/82c63a7f2) by tristanliu)
|
||||||
|
- **权限界面**: 统一助理权限设置交互界面 ([46d64d78f](https://github.com/tencentmusic/supersonic/commit/46d64d78f) by tristanliu)
|
||||||
|
- **图表优化**: 优化ChatMsg图表条件 ([06fb6ba74](https://github.com/tencentmusic/supersonic/commit/06fb6ba74) by FredTsang)
|
||||||
|
- **数据格式**: 提取formatByDataFormatType()方法 ([9ffdba956](https://github.com/tencentmusic/supersonic/commit/9ffdba956) by FredTsang)
|
||||||
|
|
||||||
|
#### 开发体验 / Developer Experience
|
||||||
|
- **构建脚本**: 优化Web应用构建脚本 ([baae7f74b](https://github.com/tencentmusic/supersonic/commit/baae7f74b) by zyclove)
|
||||||
|
- **GitHub Actions**: 优化GitHub Actions镜像推送 ([6a4458a57](https://github.com/tencentmusic/supersonic/commit/6a4458a57) by lexluo09)
|
||||||
|
- **基准测试**: 改进基准测试,增加解析结果分析 ([97710a90c](https://github.com/tencentmusic/supersonic/commit/97710a90c) by Antgeek)
|
||||||
|
|
||||||
|
### Bug修复 / Bug Fixes
|
||||||
|
|
||||||
|
#### 核心功能修复 / Core Function Fixes
|
||||||
|
- **插件功能**: 修复插件功能无法调用/结果被NL2SQL覆盖问题 ([c75233e37](https://github.com/tencentmusic/supersonic/commit/c75233e37) by QJ_wonder)
|
||||||
|
- **维度别名**: 修复映射阶段维度值别名不生效问题 ([785bda6cd](https://github.com/tencentmusic/supersonic/commit/785bda6cd) by feelshana)
|
||||||
|
- **模型字段**: 修复模型字段更新问题 ([6bd897084](https://github.com/tencentmusic/supersonic/commit/6bd897084) by WDEP)
|
||||||
|
- **多轮对话**: 修复headless中字段查询及多轮对话使用问题 ([be0447ae1](https://github.com/tencentmusic/supersonic/commit/be0447ae1) by QJ_wonder)
|
||||||
|
|
||||||
|
#### NPE异常修复 / NPE Exception Fixes
|
||||||
|
- **聊天查询**: 修复EmbeddingMatchStrategy.detectByBatch() NPE异常 ([6d907b6ad](https://github.com/tencentmusic/supersonic/commit/6d907b6ad) by wangyong)
|
||||||
|
- **文件处理**: 修复FileHandlerImpl.convert2Resp() 维度值数据行首字符为空格异常 ([da172a030](https://github.com/tencentmusic/supersonic/commit/da172a030) by wangyong)
|
||||||
|
- **头部服务**: 修复多处headless NPE问题 ([79a44b27e](https://github.com/tencentmusic/supersonic/commit/79a44b27e) by jerryjzhang)
|
||||||
|
- **解析信息**: 修复getParseInfo中的NPE ([dce9a8a58](https://github.com/tencentmusic/supersonic/commit/dce9a8a58) by supersonicbi)
|
||||||
|
|
||||||
|
#### SQL兼容性修复 / SQL Compatibility Fixes
|
||||||
|
- **SQL处理**: 修复SQL前后换行符导致的语句结尾";"删除问题 ([55ac3d1aa](https://github.com/tencentmusic/supersonic/commit/55ac3d1aa) by wangyong)
|
||||||
|
- **查询别名**: DictUtils.constructQuerySqlReq针对sql query增加别名 ([042791762](https://github.com/tencentmusic/supersonic/commit/042791762) by andybj0228)
|
||||||
|
- **SQL变量**: 支持SQL脚本变量替换 ([0709575cd](https://github.com/tencentmusic/supersonic/commit/0709575cd) by wanglongqiang)
|
||||||
|
|
||||||
|
#### 前端Bug修复 / Frontend Bug Fixes
|
||||||
|
- **UI样式**: 修复问答对话右侧历史对话模块样式异常 ([c33a85b58](https://github.com/tencentmusic/supersonic/commit/c33a85b58) by wangyong)
|
||||||
|
- **推荐维度**: 修复页面不显示推荐下钻维度问题 ([62b9db679](https://github.com/tencentmusic/supersonic/commit/62b9db679) by WDEP)
|
||||||
|
- **图表显示**: 修复饼图显示条件问题 ([1b8cd7f0d](https://github.com/tencentmusic/supersonic/commit/1b8cd7f0d) by WDEP)
|
||||||
|
- **负数支持**: 支持负数显示 ([2552e2ae4](https://github.com/tencentmusic/supersonic/commit/2552e2ae4) by FredTsang)
|
||||||
|
- **百分比显示**: 支持bar图needMultiply100显示正确百分比值 ([8abfc923a](https://github.com/tencentmusic/supersonic/commit/8abfc923a) by coosir)
|
||||||
|
- **TypeScript错误**: 修复前端TypeScript错误 ([5585b9e22](https://github.com/tencentmusic/supersonic/commit/5585b9e22) by poncheen)
|
||||||
|
|
||||||
|
#### 系统兼容性修复 / System Compatibility Fixes
|
||||||
|
- **Windows脚本**: 修复Windows daemon.bat路径配置问题 ([e5a41765b](https://github.com/tencentmusic/supersonic/commit/e5a41765b) by 柯慕灵)
|
||||||
|
- **字符编码**: 将utf8编码修改为utf8mb4,解决字符问题 ([2e81b190a](https://github.com/tencentmusic/supersonic/commit/2e81b190a) by Kun Gu)
|
||||||
|
- **记忆缓存**: 修复记忆管理中因缓存无法存储的问题 ([81cd60d2d](https://github.com/tencentmusic/supersonic/commit/81cd60d2d) by guilinlewis)
|
||||||
|
- **Mac兼容**: 降级djl库以支持Mac Intel机器 ([bf3213e8f](https://github.com/tencentmusic/supersonic/commit/bf3213e8f) by jerryjzhang)
|
||||||
|
|
||||||
|
### 数据管理优化 / Data Management Improvements
|
||||||
|
|
||||||
|
#### 维度指标管理 / Dimension & Metric Management
|
||||||
|
- **维度检索**: 修复维度和指标检索及百分比显示问题 ([d8fe2ed2b](https://github.com/tencentmusic/supersonic/commit/d8fe2ed2b) by 木鱼和尚)
|
||||||
|
- **查询导出**: 基于queryColumns导出数据 ([11d1264d3](https://github.com/tencentmusic/supersonic/commit/11d1264d3) by FredTsang)
|
||||||
|
- **表格排序**: 移除表格defaultSortOrder ([32675387d](https://github.com/tencentmusic/supersonic/commit/32675387d) by FredTsang)
|
||||||
|
- **维度搜索**: 修复维度搜索带key查询范围超出问题 ([269f146c1](https://github.com/tencentmusic/supersonic/commit/269f146c1) by wangyong)
|
||||||
|
|
||||||
|
### 测试和质量保证 / Testing & Quality Assurance
|
||||||
|
|
||||||
|
#### 单元测试 / Unit Testing
|
||||||
|
- **测试修复**: 修复单元测试用例 ([91e4b51ef](https://github.com/tencentmusic/supersonic/commit/91e4b51ef) by jerryjzhang)
|
||||||
|
- **模型测试**: 修复ModelCreateForm.tsx错误 ([d2aa73b85](https://github.com/tencentmusic/supersonic/commit/d2aa73b85) by Antgeek)
|
||||||
|
|
||||||
|
### 重要变更说明 / Breaking Changes
|
||||||
|
|
||||||
|
#### 升级注意事项 / Upgrade Notes
|
||||||
|
1. **SpringBoot 3升级**: 可能需要更新依赖配置和代码适配
|
||||||
|
2. **许可证变更**: 从MIT变更为Apache 2.0,请注意法律合规
|
||||||
|
3. **API接口调整**: 部分API接口为支持新功能进行了调整
|
||||||
|
4. **数据库兼容**: 新增多种数据库支持,配置方式有所变化
|
||||||
|
|
||||||
|
### 完整提交统计 / Commit Statistics
|
||||||
|
- **总提交数**: 419个提交
|
||||||
|
- **主要贡献者**:
|
||||||
|
- jerryjzhang: 158次提交
|
||||||
|
- supersonicbi: 22次提交
|
||||||
|
- zyclove: 20次提交
|
||||||
|
- beat4ocean: 15次提交
|
||||||
|
- guilinlewis: 11次提交
|
||||||
|
- wangyong: 11次提交
|
||||||
|
- 其他贡献者: 182次提交
|
||||||
|
- **涉及模块**: headless, chat, auth, common, webapp, launcher, docker
|
||||||
|
- **时间跨度**: 2024年11月1日 - 2025年8月5日
|
||||||
|
|
||||||
|
### 致谢 / Acknowledgments
|
||||||
|
|
||||||
|
感谢所有为SuperSonic 1.0.0版本贡献代码、文档、测试和建议的开发者们!🎉
|
||||||
|
|
||||||
|
#### 核心贡献者 / Core Contributors
|
||||||
|
- **jerryjzhang** - 项目维护者,核心架构设计与实现
|
||||||
|
- **supersonicbi** - 核心功能开发,多数据库支持
|
||||||
|
- **beat4ocean** - 架构升级,依赖管理,安全优化
|
||||||
|
- **zyclove** - 数据库适配,构建优化
|
||||||
|
- **guilinlewis** - 鉴权系统,召回优化
|
||||||
|
- **wangyong** - Bug修复,NPE异常处理
|
||||||
|
|
||||||
|
#### 活跃贡献者 / Active Contributors
|
||||||
|
- **WDEP** - 前端优化,图表功能
|
||||||
|
- **FredTsang** - Chat SDK优化,数据导出
|
||||||
|
- **feelshana** - 记忆管理,向量召回
|
||||||
|
- **QJ_wonder** - 插件功能,多轮对话
|
||||||
|
- **Willy-J** - 模型关联,数据库兼容
|
||||||
|
- **iridescentpeo** - 查询优化,模型管理
|
||||||
|
- **tristanliu** - 前端路由,权限界面
|
||||||
|
- **mislayming** - SpringBoot 3升级
|
||||||
|
- **Antgeek** - 基准测试,模型修复
|
||||||
|
- **柯慕灵** - LLM纠错器,Windows脚本
|
||||||
|
- **superhero** - 项目管理,代码审查
|
||||||
|
|
||||||
|
#### 其他重要贡献者 / Other Important Contributors
|
||||||
|
- **木鱼和尚** - 维度指标检索优化
|
||||||
|
- **pisces** - 图表导出功能
|
||||||
|
- **lexluo09** - 并行处理,GitHub Actions
|
||||||
|
- **andybj0228** - SQL查询优化
|
||||||
|
- **wanglongqiang** - SQL变量支持
|
||||||
|
- **Hyman_bz** - StarRocks支持
|
||||||
|
- **wwsheng009** - SAP HANA适配
|
||||||
|
- **poncheen** - TypeScript错误修复
|
||||||
|
- **kino** - Docker镜像优化
|
||||||
|
- **coosir** - 前端百分比显示
|
||||||
|
- **Kun Gu** - 字符编码优化
|
||||||
|
- **chixiaopao** - NPE异常修复
|
||||||
|
- **naimehao** - 核心功能修复
|
||||||
|
- **yudong** - 记忆评估优化
|
||||||
|
- **mroldx** - 数据库脚本更新
|
||||||
|
- **ChPi** - 解析器性能优化
|
||||||
|
- **Hwting** - Docker配置优化
|
||||||
|
|
||||||
|
#### 特别感谢 / Special Thanks
|
||||||
|
感谢所有提交Issue、参与讨论、提供反馈的社区用户,你们的每一个建议都让SuperSonic变得更好!
|
||||||
|
|
||||||
|
#### 社区支持 / Community Support
|
||||||
|
SuperSonic是一个开源项目,我们欢迎更多开发者加入:
|
||||||
|
- 🔗 **GitHub**: https://github.com/tencentmusic/supersonic
|
||||||
|
- 📖 **文档**: 详见项目README和Wiki
|
||||||
|
- 🐛 **Issue报告**: 欢迎提交Bug和功能请求
|
||||||
|
- 🚀 **贡献代码**: 欢迎提交Pull Request
|
||||||
|
- 💬 **社区讨论**: 加入我们的技术交流群
|
||||||
|
|
||||||
|
#### 未来展望 / Future Vision
|
||||||
|
SuperSonic 1.0.0是一个重要的里程碑,但这只是开始。我们将继续:
|
||||||
|
- 🌟 **持续优化性能和稳定性**
|
||||||
|
- 🔧 **扩展更多数据库和AI模型支持**
|
||||||
|
- 🎨 **改善用户体验和界面设计**
|
||||||
|
- 📚 **完善文档和最佳实践**
|
||||||
|
- 🤝 **建设更活跃的开源社区**
|
||||||
|
|
||||||
|
**让我们一起把SuperSonic做得更好!** ✨
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
*如果您在使用过程中遇到问题或有改进建议,欢迎随时与我们交流。每一份贡献都让SuperSonic更加强大!*
|
||||||
|
|
||||||
|
|
||||||
## SuperSonic [0.9.8] - 2024-11-01
|
## SuperSonic [0.9.8] - 2024-11-01
|
||||||
- Add LLM management module to reuse connection across agents.
|
- Add LLM management module to reuse connection across agents.
|
||||||
|
|||||||
@@ -15,4 +15,6 @@ public class UserReq {
|
|||||||
|
|
||||||
@NotBlank(message = "password can not be null")
|
@NotBlank(message = "password can not be null")
|
||||||
private String newPassword;
|
private String newPassword;
|
||||||
|
|
||||||
|
private String role;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -18,4 +18,5 @@ public class ChatExecuteReq {
|
|||||||
private int parseId;
|
private int parseId;
|
||||||
private String queryText;
|
private String queryText;
|
||||||
private boolean saveAnswer;
|
private boolean saveAnswer;
|
||||||
|
private boolean streamingResult;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,19 @@
|
|||||||
|
package com.tencent.supersonic.chat.api.pojo.request;
|
||||||
|
|
||||||
|
import lombok.AllArgsConstructor;
|
||||||
|
import lombok.Builder;
|
||||||
|
import lombok.Data;
|
||||||
|
import lombok.NoArgsConstructor;
|
||||||
|
|
||||||
|
import java.util.List;
|
||||||
|
|
||||||
|
@Data
|
||||||
|
@Builder
|
||||||
|
@NoArgsConstructor
|
||||||
|
@AllArgsConstructor
|
||||||
|
public class ChatMemoryDeleteReq {
|
||||||
|
|
||||||
|
private List<Long> ids;
|
||||||
|
|
||||||
|
private Integer agentId;
|
||||||
|
}
|
||||||
@@ -32,6 +32,7 @@ import dev.langchain4j.model.input.PromptTemplate;
|
|||||||
import dev.langchain4j.model.output.Response;
|
import dev.langchain4j.model.output.Response;
|
||||||
import dev.langchain4j.provider.ModelProvider;
|
import dev.langchain4j.provider.ModelProvider;
|
||||||
import lombok.extern.slf4j.Slf4j;
|
import lombok.extern.slf4j.Slf4j;
|
||||||
|
import org.apache.commons.lang3.StringUtils;
|
||||||
import org.slf4j.Logger;
|
import org.slf4j.Logger;
|
||||||
import org.slf4j.LoggerFactory;
|
import org.slf4j.LoggerFactory;
|
||||||
|
|
||||||
@@ -171,10 +172,6 @@ public class NL2SQLParser implements ChatQueryParser {
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
// derive mapping result of current question and parsing result of last question.
|
|
||||||
ChatLayerService chatLayerService = ContextUtils.getBean(ChatLayerService.class);
|
|
||||||
MapResp currentMapResult = chatLayerService.map(queryNLReq);
|
|
||||||
|
|
||||||
List<QueryResp> historyQueries =
|
List<QueryResp> historyQueries =
|
||||||
getHistoryQueries(parseContext.getRequest().getChatId(), 1);
|
getHistoryQueries(parseContext.getRequest().getChatId(), 1);
|
||||||
if (historyQueries.isEmpty()) {
|
if (historyQueries.isEmpty()) {
|
||||||
@@ -182,12 +179,18 @@ public class NL2SQLParser implements ChatQueryParser {
|
|||||||
}
|
}
|
||||||
QueryResp lastQuery = historyQueries.get(0);
|
QueryResp lastQuery = historyQueries.get(0);
|
||||||
SemanticParseInfo lastParseInfo = lastQuery.getParseInfos().get(0);
|
SemanticParseInfo lastParseInfo = lastQuery.getParseInfos().get(0);
|
||||||
Long dataId = lastParseInfo.getDataSetId();
|
String histSQL = lastParseInfo.getSqlInfo().getCorrectedS2SQL();
|
||||||
|
if (StringUtils.isBlank(histSQL)) // 优化性能,如果问答不是chat bi 则无需重写,因为数据都不全
|
||||||
|
return;
|
||||||
|
|
||||||
|
// derive mapping result of current question and parsing result of last question.
|
||||||
|
ChatLayerService chatLayerService = ContextUtils.getBean(ChatLayerService.class);
|
||||||
|
MapResp currentMapResult = chatLayerService.map(queryNLReq); // 优化性能 ,只有满足条件才mapping
|
||||||
|
|
||||||
|
Long dataId = lastParseInfo.getDataSetId();
|
||||||
String curtMapStr =
|
String curtMapStr =
|
||||||
generateSchemaPrompt(currentMapResult.getMapInfo().getMatchedElements(dataId));
|
generateSchemaPrompt(currentMapResult.getMapInfo().getMatchedElements(dataId));
|
||||||
String histMapStr = generateSchemaPrompt(lastParseInfo.getElementMatches());
|
String histMapStr = generateSchemaPrompt(lastParseInfo.getElementMatches());
|
||||||
String histSQL = lastParseInfo.getSqlInfo().getCorrectedS2SQL();
|
|
||||||
|
|
||||||
Map<String, Object> variables = new HashMap<>();
|
Map<String, Object> variables = new HashMap<>();
|
||||||
variables.put("current_question", currentMapResult.getQueryText());
|
variables.put("current_question", currentMapResult.getQueryText());
|
||||||
|
|||||||
@@ -35,9 +35,7 @@ public class ChatMemoryRepositoryImpl implements ChatMemoryRepository {
|
|||||||
if (CollectionUtils.isEmpty(ids)) {
|
if (CollectionUtils.isEmpty(ids)) {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
for (Long id : ids) {
|
chatMemoryMapper.deleteByIds(ids);
|
||||||
chatMemoryMapper.deleteById(id);
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
|
|||||||
@@ -1,13 +1,20 @@
|
|||||||
package com.tencent.supersonic.chat.server.processor.execute;
|
package com.tencent.supersonic.chat.server.processor.execute;
|
||||||
|
|
||||||
|
import com.alibaba.fastjson.JSON;
|
||||||
|
import com.alibaba.fastjson.JSONObject;
|
||||||
import com.tencent.supersonic.chat.api.pojo.response.QueryResult;
|
import com.tencent.supersonic.chat.api.pojo.response.QueryResult;
|
||||||
import com.tencent.supersonic.chat.server.agent.Agent;
|
import com.tencent.supersonic.chat.server.agent.Agent;
|
||||||
|
import com.tencent.supersonic.chat.server.persistence.dataobject.ChatQueryDO;
|
||||||
|
import com.tencent.supersonic.chat.server.persistence.repository.ChatQueryRepository;
|
||||||
import com.tencent.supersonic.chat.server.pojo.ExecuteContext;
|
import com.tencent.supersonic.chat.server.pojo.ExecuteContext;
|
||||||
import com.tencent.supersonic.common.pojo.ChatApp;
|
import com.tencent.supersonic.common.pojo.ChatApp;
|
||||||
import com.tencent.supersonic.common.pojo.enums.AppModule;
|
import com.tencent.supersonic.common.pojo.enums.AppModule;
|
||||||
import com.tencent.supersonic.common.util.ChatAppManager;
|
import com.tencent.supersonic.common.util.ChatAppManager;
|
||||||
|
import com.tencent.supersonic.common.util.ContextUtils;
|
||||||
import dev.langchain4j.data.message.AiMessage;
|
import dev.langchain4j.data.message.AiMessage;
|
||||||
|
import dev.langchain4j.model.StreamingResponseHandler;
|
||||||
import dev.langchain4j.model.chat.ChatLanguageModel;
|
import dev.langchain4j.model.chat.ChatLanguageModel;
|
||||||
|
import dev.langchain4j.model.chat.StreamingChatLanguageModel;
|
||||||
import dev.langchain4j.model.input.Prompt;
|
import dev.langchain4j.model.input.Prompt;
|
||||||
import dev.langchain4j.model.input.PromptTemplate;
|
import dev.langchain4j.model.input.PromptTemplate;
|
||||||
import dev.langchain4j.model.output.Response;
|
import dev.langchain4j.model.output.Response;
|
||||||
@@ -24,9 +31,11 @@ import java.util.Objects;
|
|||||||
* DataInterpretProcessor interprets query result to make it more readable to the users.
|
* DataInterpretProcessor interprets query result to make it more readable to the users.
|
||||||
*/
|
*/
|
||||||
public class DataInterpretProcessor implements ExecuteResultProcessor {
|
public class DataInterpretProcessor implements ExecuteResultProcessor {
|
||||||
|
public static String tip = "AI 回答中...\r\n";
|
||||||
private static final Logger keyPipelineLog = LoggerFactory.getLogger("keyPipeline");
|
private static final Logger keyPipelineLog = LoggerFactory.getLogger("keyPipeline");
|
||||||
|
|
||||||
|
private static Map<Long, StringBuffer> resultCache = new HashMap<>();
|
||||||
|
|
||||||
public static final String APP_KEY = "DATA_INTERPRETER";
|
public static final String APP_KEY = "DATA_INTERPRETER";
|
||||||
private static final String INSTRUCTION = ""
|
private static final String INSTRUCTION = ""
|
||||||
+ "#Role: You are a data expert who communicates with business users everyday."
|
+ "#Role: You are a data expert who communicates with business users everyday."
|
||||||
@@ -41,6 +50,16 @@ public class DataInterpretProcessor implements ExecuteResultProcessor {
|
|||||||
.appModule(AppModule.CHAT).description("通过大模型对结果数据做提炼总结").enable(false).build());
|
.appModule(AppModule.CHAT).description("通过大模型对结果数据做提炼总结").enable(false).build());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public static String getTextSummary(Long queryId) {
|
||||||
|
if (resultCache.get(queryId) != null) {
|
||||||
|
return resultCache.get(queryId).toString();
|
||||||
|
}
|
||||||
|
return "";
|
||||||
|
}
|
||||||
|
|
||||||
|
public static Map<Long, StringBuffer> getResultCache() {
|
||||||
|
return resultCache;
|
||||||
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public boolean accept(ExecuteContext executeContext) {
|
public boolean accept(ExecuteContext executeContext) {
|
||||||
@@ -71,14 +90,49 @@ public class DataInterpretProcessor implements ExecuteResultProcessor {
|
|||||||
variable.put("data", queryResult.getTextResult());
|
variable.put("data", queryResult.getTextResult());
|
||||||
|
|
||||||
Prompt prompt = PromptTemplate.from(chatApp.getPrompt()).apply(variable);
|
Prompt prompt = PromptTemplate.from(chatApp.getPrompt()).apply(variable);
|
||||||
ChatLanguageModel chatLanguageModel =
|
if (executeContext.getRequest().isStreamingResult()) {
|
||||||
ModelProvider.getChatModel(chatApp.getChatModelConfig());
|
StreamingChatLanguageModel chatLanguageModel =
|
||||||
Response<AiMessage> response = chatLanguageModel.generate(prompt.toUserMessage());
|
ModelProvider.getChatStreamingModel(chatApp.getChatModelConfig());
|
||||||
String anwser = response.content().text();
|
final Long queryId = executeContext.getRequest().getQueryId();
|
||||||
keyPipelineLog.info("DataInterpretProcessor modelReq:\n{} \nmodelResp:\n{}", prompt.text(),
|
resultCache.put(queryId, new StringBuffer(tip));
|
||||||
anwser);
|
chatLanguageModel.generate(prompt.toUserMessage(),
|
||||||
if (StringUtils.isNotBlank(anwser)) {
|
new StreamingResponseHandler<AiMessage>() {
|
||||||
queryResult.setTextSummary(anwser);
|
@Override
|
||||||
|
public void onNext(String token) {
|
||||||
|
resultCache.get(queryId).append(token);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void onComplete(Response<AiMessage> response) {
|
||||||
|
ChatQueryRepository chatQueryRepository =
|
||||||
|
ContextUtils.getBean(ChatQueryRepository.class);
|
||||||
|
ChatQueryDO chatQueryDO = chatQueryRepository.getChatQueryDO(queryId);
|
||||||
|
JSONObject queryResult = JSON.parseObject(chatQueryDO.getQueryResult());
|
||||||
|
queryResult.put("textSummary",
|
||||||
|
resultCache.get(queryId).toString().substring(tip.length()));
|
||||||
|
chatQueryDO.setQueryResult(queryResult.toJSONString());
|
||||||
|
chatQueryRepository.updateChatQuery(chatQueryDO);
|
||||||
|
resultCache.remove(queryId);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void onError(Throwable error) {
|
||||||
|
error.printStackTrace();
|
||||||
|
resultCache.remove(queryId);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
} else {
|
||||||
|
ChatLanguageModel chatLanguageModel =
|
||||||
|
ModelProvider.getChatModel(chatApp.getChatModelConfig());
|
||||||
|
Response<AiMessage> response = chatLanguageModel.generate(prompt.toUserMessage());
|
||||||
|
String anwser = response.content().text();
|
||||||
|
keyPipelineLog.info("DataInterpretProcessor modelReq:\n{} \nmodelResp:\n{}",
|
||||||
|
prompt.text(), anwser);
|
||||||
|
if (StringUtils.isNotBlank(anwser)) {
|
||||||
|
queryResult.setTextSummary(anwser);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import com.tencent.supersonic.chat.api.pojo.request.ChatExecuteReq;
|
|||||||
import com.tencent.supersonic.chat.api.pojo.request.ChatParseReq;
|
import com.tencent.supersonic.chat.api.pojo.request.ChatParseReq;
|
||||||
import com.tencent.supersonic.chat.api.pojo.request.ChatQueryDataReq;
|
import com.tencent.supersonic.chat.api.pojo.request.ChatQueryDataReq;
|
||||||
import com.tencent.supersonic.chat.api.pojo.response.ChatParseResp;
|
import com.tencent.supersonic.chat.api.pojo.response.ChatParseResp;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.response.QueryResult;
|
||||||
import com.tencent.supersonic.chat.server.service.ChatQueryService;
|
import com.tencent.supersonic.chat.server.service.ChatQueryService;
|
||||||
import com.tencent.supersonic.common.pojo.User;
|
import com.tencent.supersonic.common.pojo.User;
|
||||||
import com.tencent.supersonic.common.pojo.exception.InvalidArgumentException;
|
import com.tencent.supersonic.common.pojo.exception.InvalidArgumentException;
|
||||||
@@ -50,6 +51,14 @@ public class ChatQueryController {
|
|||||||
return chatQueryService.execute(chatExecuteReq);
|
return chatQueryService.execute(chatExecuteReq);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@PostMapping("getExecuteSummary")
|
||||||
|
public Object getExecuteSummary(@RequestBody ChatExecuteReq chatExecuteReq,
|
||||||
|
HttpServletRequest request, HttpServletResponse response) {
|
||||||
|
chatExecuteReq.setUser(UserHolder.findUser(request, response));
|
||||||
|
QueryResult res = chatQueryService.getTextSummary(chatExecuteReq);
|
||||||
|
return res;
|
||||||
|
}
|
||||||
|
|
||||||
@PostMapping("/")
|
@PostMapping("/")
|
||||||
public Object query(@RequestBody ChatParseReq chatParseReq, HttpServletRequest request,
|
public Object query(@RequestBody ChatParseReq chatParseReq, HttpServletRequest request,
|
||||||
HttpServletResponse response) throws Exception {
|
HttpServletResponse response) throws Exception {
|
||||||
|
|||||||
@@ -4,12 +4,12 @@ import com.github.pagehelper.PageInfo;
|
|||||||
import com.tencent.supersonic.auth.api.authentication.utils.UserHolder;
|
import com.tencent.supersonic.auth.api.authentication.utils.UserHolder;
|
||||||
import com.tencent.supersonic.chat.api.pojo.enums.MemoryReviewResult;
|
import com.tencent.supersonic.chat.api.pojo.enums.MemoryReviewResult;
|
||||||
import com.tencent.supersonic.chat.api.pojo.request.ChatMemoryCreateReq;
|
import com.tencent.supersonic.chat.api.pojo.request.ChatMemoryCreateReq;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.request.ChatMemoryDeleteReq;
|
||||||
import com.tencent.supersonic.chat.api.pojo.request.ChatMemoryUpdateReq;
|
import com.tencent.supersonic.chat.api.pojo.request.ChatMemoryUpdateReq;
|
||||||
import com.tencent.supersonic.chat.api.pojo.request.PageMemoryReq;
|
import com.tencent.supersonic.chat.api.pojo.request.PageMemoryReq;
|
||||||
import com.tencent.supersonic.chat.server.pojo.ChatMemory;
|
import com.tencent.supersonic.chat.server.pojo.ChatMemory;
|
||||||
import com.tencent.supersonic.chat.server.service.MemoryService;
|
import com.tencent.supersonic.chat.server.service.MemoryService;
|
||||||
import com.tencent.supersonic.common.pojo.User;
|
import com.tencent.supersonic.common.pojo.User;
|
||||||
import com.tencent.supersonic.headless.api.pojo.request.MetaBatchReq;
|
|
||||||
import jakarta.servlet.http.HttpServletRequest;
|
import jakarta.servlet.http.HttpServletRequest;
|
||||||
import jakarta.servlet.http.HttpServletResponse;
|
import jakarta.servlet.http.HttpServletResponse;
|
||||||
import org.springframework.beans.factory.annotation.Autowired;
|
import org.springframework.beans.factory.annotation.Autowired;
|
||||||
@@ -53,8 +53,10 @@ public class MemoryController {
|
|||||||
}
|
}
|
||||||
|
|
||||||
@PostMapping("batchDelete")
|
@PostMapping("batchDelete")
|
||||||
public Boolean batchDelete(@RequestBody MetaBatchReq metaBatchReq) {
|
public Boolean deleteMemory(@RequestBody ChatMemoryDeleteReq chatMemoryDeleteReq,
|
||||||
memoryService.batchDelete(metaBatchReq.getIds());
|
HttpServletRequest request, HttpServletResponse response) {
|
||||||
|
User user = UserHolder.findUser(request, response);
|
||||||
|
memoryService.batchDelete(chatMemoryDeleteReq, user);
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -35,6 +35,8 @@ public interface ChatManageService {
|
|||||||
|
|
||||||
QueryResp getChatQuery(Long queryId);
|
QueryResp getChatQuery(Long queryId);
|
||||||
|
|
||||||
|
ChatQueryDO getChatQueryDO(Long queryId);
|
||||||
|
|
||||||
List<QueryResp> getChatQueries(Integer chatId);
|
List<QueryResp> getChatQueries(Integer chatId);
|
||||||
|
|
||||||
ShowCaseResp queryShowCase(PageQueryInfoReq pageQueryInfoReq, int agentId);
|
ShowCaseResp queryShowCase(PageQueryInfoReq pageQueryInfoReq, int agentId);
|
||||||
|
|||||||
@@ -19,6 +19,8 @@ public interface ChatQueryService {
|
|||||||
|
|
||||||
QueryResult execute(ChatExecuteReq chatExecuteReq) throws Exception;
|
QueryResult execute(ChatExecuteReq chatExecuteReq) throws Exception;
|
||||||
|
|
||||||
|
QueryResult getTextSummary(ChatExecuteReq chatExecuteReq);
|
||||||
|
|
||||||
QueryResult parseAndExecute(ChatParseReq chatParseReq);
|
QueryResult parseAndExecute(ChatParseReq chatParseReq);
|
||||||
|
|
||||||
Object queryData(ChatQueryDataReq chatQueryDataReq, User user) throws Exception;
|
Object queryData(ChatQueryDataReq chatQueryDataReq, User user) throws Exception;
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
package com.tencent.supersonic.chat.server.service;
|
package com.tencent.supersonic.chat.server.service;
|
||||||
|
|
||||||
import com.github.pagehelper.PageInfo;
|
import com.github.pagehelper.PageInfo;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.request.ChatMemoryDeleteReq;
|
||||||
import com.tencent.supersonic.chat.api.pojo.request.ChatMemoryFilter;
|
import com.tencent.supersonic.chat.api.pojo.request.ChatMemoryFilter;
|
||||||
import com.tencent.supersonic.chat.api.pojo.request.ChatMemoryUpdateReq;
|
import com.tencent.supersonic.chat.api.pojo.request.ChatMemoryUpdateReq;
|
||||||
import com.tencent.supersonic.chat.api.pojo.request.PageMemoryReq;
|
import com.tencent.supersonic.chat.api.pojo.request.PageMemoryReq;
|
||||||
@@ -14,7 +15,7 @@ public interface MemoryService {
|
|||||||
|
|
||||||
void updateMemory(ChatMemoryUpdateReq chatMemoryUpdateReq, User user);
|
void updateMemory(ChatMemoryUpdateReq chatMemoryUpdateReq, User user);
|
||||||
|
|
||||||
void batchDelete(List<Long> ids);
|
void batchDelete(ChatMemoryDeleteReq chatMemoryDeleteReq, User user);
|
||||||
|
|
||||||
PageInfo<ChatMemory> pageMemories(PageMemoryReq pageMemoryReq);
|
PageInfo<ChatMemory> pageMemories(PageMemoryReq pageMemoryReq);
|
||||||
|
|
||||||
|
|||||||
@@ -123,6 +123,11 @@ public class ChatManageServiceImpl implements ChatManageService {
|
|||||||
return chatQueryRepository.getChatQuery(queryId);
|
return chatQueryRepository.getChatQuery(queryId);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public ChatQueryDO getChatQueryDO(Long queryId) {
|
||||||
|
return chatQueryRepository.getChatQueryDO(queryId);
|
||||||
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public List<QueryResp> getChatQueries(Integer chatId) {
|
public List<QueryResp> getChatQueries(Integer chatId) {
|
||||||
List<QueryResp> queries = chatQueryRepository.getChatQueries(chatId);
|
List<QueryResp> queries = chatQueryRepository.getChatQueries(chatId);
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
package com.tencent.supersonic.chat.server.service.impl;
|
package com.tencent.supersonic.chat.server.service.impl;
|
||||||
|
|
||||||
|
import com.alibaba.fastjson2.JSON;
|
||||||
import com.google.common.collect.Lists;
|
import com.google.common.collect.Lists;
|
||||||
import com.tencent.supersonic.chat.api.pojo.request.ChatExecuteReq;
|
import com.tencent.supersonic.chat.api.pojo.request.ChatExecuteReq;
|
||||||
import com.tencent.supersonic.chat.api.pojo.request.ChatParseReq;
|
import com.tencent.supersonic.chat.api.pojo.request.ChatParseReq;
|
||||||
@@ -9,8 +10,10 @@ import com.tencent.supersonic.chat.api.pojo.response.QueryResult;
|
|||||||
import com.tencent.supersonic.chat.server.agent.Agent;
|
import com.tencent.supersonic.chat.server.agent.Agent;
|
||||||
import com.tencent.supersonic.chat.server.executor.ChatQueryExecutor;
|
import com.tencent.supersonic.chat.server.executor.ChatQueryExecutor;
|
||||||
import com.tencent.supersonic.chat.server.parser.ChatQueryParser;
|
import com.tencent.supersonic.chat.server.parser.ChatQueryParser;
|
||||||
|
import com.tencent.supersonic.chat.server.persistence.dataobject.ChatQueryDO;
|
||||||
import com.tencent.supersonic.chat.server.pojo.ExecuteContext;
|
import com.tencent.supersonic.chat.server.pojo.ExecuteContext;
|
||||||
import com.tencent.supersonic.chat.server.pojo.ParseContext;
|
import com.tencent.supersonic.chat.server.pojo.ParseContext;
|
||||||
|
import com.tencent.supersonic.chat.server.processor.execute.DataInterpretProcessor;
|
||||||
import com.tencent.supersonic.chat.server.processor.execute.ExecuteResultProcessor;
|
import com.tencent.supersonic.chat.server.processor.execute.ExecuteResultProcessor;
|
||||||
import com.tencent.supersonic.chat.server.processor.parse.ParseResultProcessor;
|
import com.tencent.supersonic.chat.server.processor.parse.ParseResultProcessor;
|
||||||
import com.tencent.supersonic.chat.server.service.AgentService;
|
import com.tencent.supersonic.chat.server.service.AgentService;
|
||||||
@@ -18,7 +21,11 @@ import com.tencent.supersonic.chat.server.service.ChatManageService;
|
|||||||
import com.tencent.supersonic.chat.server.service.ChatQueryService;
|
import com.tencent.supersonic.chat.server.service.ChatQueryService;
|
||||||
import com.tencent.supersonic.chat.server.util.ComponentFactory;
|
import com.tencent.supersonic.chat.server.util.ComponentFactory;
|
||||||
import com.tencent.supersonic.chat.server.util.QueryReqConverter;
|
import com.tencent.supersonic.chat.server.util.QueryReqConverter;
|
||||||
import com.tencent.supersonic.common.jsqlparser.*;
|
import com.tencent.supersonic.common.jsqlparser.FieldExpression;
|
||||||
|
import com.tencent.supersonic.common.jsqlparser.SqlAddHelper;
|
||||||
|
import com.tencent.supersonic.common.jsqlparser.SqlRemoveHelper;
|
||||||
|
import com.tencent.supersonic.common.jsqlparser.SqlReplaceHelper;
|
||||||
|
import com.tencent.supersonic.common.jsqlparser.SqlSelectHelper;
|
||||||
import com.tencent.supersonic.common.pojo.User;
|
import com.tencent.supersonic.common.pojo.User;
|
||||||
import com.tencent.supersonic.common.pojo.enums.FilterOperatorEnum;
|
import com.tencent.supersonic.common.pojo.enums.FilterOperatorEnum;
|
||||||
import com.tencent.supersonic.common.util.DateUtils;
|
import com.tencent.supersonic.common.util.DateUtils;
|
||||||
@@ -44,7 +51,11 @@ import lombok.extern.slf4j.Slf4j;
|
|||||||
import net.sf.jsqlparser.expression.Expression;
|
import net.sf.jsqlparser.expression.Expression;
|
||||||
import net.sf.jsqlparser.expression.LongValue;
|
import net.sf.jsqlparser.expression.LongValue;
|
||||||
import net.sf.jsqlparser.expression.StringValue;
|
import net.sf.jsqlparser.expression.StringValue;
|
||||||
import net.sf.jsqlparser.expression.operators.relational.*;
|
import net.sf.jsqlparser.expression.operators.relational.ComparisonOperator;
|
||||||
|
import net.sf.jsqlparser.expression.operators.relational.GreaterThanEquals;
|
||||||
|
import net.sf.jsqlparser.expression.operators.relational.InExpression;
|
||||||
|
import net.sf.jsqlparser.expression.operators.relational.MinorThanEquals;
|
||||||
|
import net.sf.jsqlparser.expression.operators.relational.ParenthesedExpressionList;
|
||||||
import net.sf.jsqlparser.schema.Column;
|
import net.sf.jsqlparser.schema.Column;
|
||||||
import org.apache.commons.lang3.StringUtils;
|
import org.apache.commons.lang3.StringUtils;
|
||||||
import org.apache.commons.lang3.tuple.Pair;
|
import org.apache.commons.lang3.tuple.Pair;
|
||||||
@@ -53,7 +64,14 @@ import org.springframework.context.annotation.Lazy;
|
|||||||
import org.springframework.stereotype.Service;
|
import org.springframework.stereotype.Service;
|
||||||
import org.springframework.util.CollectionUtils;
|
import org.springframework.util.CollectionUtils;
|
||||||
|
|
||||||
import java.util.*;
|
import java.util.ArrayList;
|
||||||
|
import java.util.HashMap;
|
||||||
|
import java.util.HashSet;
|
||||||
|
import java.util.Iterator;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Map;
|
||||||
|
import java.util.Objects;
|
||||||
|
import java.util.Set;
|
||||||
import java.util.stream.Collectors;
|
import java.util.stream.Collectors;
|
||||||
|
|
||||||
@Slf4j
|
@Slf4j
|
||||||
@@ -110,6 +128,8 @@ public class ChatQueryServiceImpl implements ChatQueryService {
|
|||||||
}
|
}
|
||||||
|
|
||||||
if (!parseContext.needFeedback()) {
|
if (!parseContext.needFeedback()) {
|
||||||
|
parseContext.getResponse().getParseTimeCost().setParseTime(System.currentTimeMillis()
|
||||||
|
- parseContext.getResponse().getParseTimeCost().getParseStartTime());
|
||||||
chatManageService.batchAddParse(chatParseReq, parseContext.getResponse());
|
chatManageService.batchAddParse(chatParseReq, parseContext.getResponse());
|
||||||
chatManageService.updateParseCostTime(parseContext.getResponse());
|
chatManageService.updateParseCostTime(parseContext.getResponse());
|
||||||
}
|
}
|
||||||
@@ -143,6 +163,21 @@ public class ChatQueryServiceImpl implements ChatQueryService {
|
|||||||
return queryResult;
|
return queryResult;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public QueryResult getTextSummary(ChatExecuteReq chatExecuteReq) {
|
||||||
|
String text = DataInterpretProcessor.getTextSummary(chatExecuteReq.getQueryId());
|
||||||
|
if (StringUtils.isNotBlank(text)) {
|
||||||
|
QueryResult res = new QueryResult();
|
||||||
|
res.setTextSummary(text);
|
||||||
|
res.setQueryId(chatExecuteReq.getQueryId());
|
||||||
|
return res;
|
||||||
|
} else {
|
||||||
|
ChatQueryDO chatQueryDo = chatManageService.getChatQueryDO(chatExecuteReq.getQueryId());
|
||||||
|
QueryResult res = JSON.parseObject(chatQueryDo.getQueryResult(), QueryResult.class);
|
||||||
|
return res;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public QueryResult parseAndExecute(ChatParseReq chatParseReq) {
|
public QueryResult parseAndExecute(ChatParseReq chatParseReq) {
|
||||||
ChatParseResp parseResp = parse(chatParseReq);
|
ChatParseResp parseResp = parse(chatParseReq);
|
||||||
|
|||||||
@@ -6,6 +6,7 @@ import com.github.pagehelper.PageHelper;
|
|||||||
import com.github.pagehelper.PageInfo;
|
import com.github.pagehelper.PageInfo;
|
||||||
import com.tencent.supersonic.chat.api.pojo.enums.MemoryReviewResult;
|
import com.tencent.supersonic.chat.api.pojo.enums.MemoryReviewResult;
|
||||||
import com.tencent.supersonic.chat.api.pojo.enums.MemoryStatus;
|
import com.tencent.supersonic.chat.api.pojo.enums.MemoryStatus;
|
||||||
|
import com.tencent.supersonic.chat.api.pojo.request.ChatMemoryDeleteReq;
|
||||||
import com.tencent.supersonic.chat.api.pojo.request.ChatMemoryFilter;
|
import com.tencent.supersonic.chat.api.pojo.request.ChatMemoryFilter;
|
||||||
import com.tencent.supersonic.chat.api.pojo.request.ChatMemoryUpdateReq;
|
import com.tencent.supersonic.chat.api.pojo.request.ChatMemoryUpdateReq;
|
||||||
import com.tencent.supersonic.chat.api.pojo.request.PageMemoryReq;
|
import com.tencent.supersonic.chat.api.pojo.request.PageMemoryReq;
|
||||||
@@ -26,6 +27,7 @@ import org.springframework.boot.CommandLineRunner;
|
|||||||
import org.springframework.stereotype.Service;
|
import org.springframework.stereotype.Service;
|
||||||
import org.springframework.util.CollectionUtils;
|
import org.springframework.util.CollectionUtils;
|
||||||
|
|
||||||
|
import java.util.ArrayList;
|
||||||
import java.util.Date;
|
import java.util.Date;
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
import java.util.Objects;
|
import java.util.Objects;
|
||||||
@@ -108,14 +110,21 @@ public class MemoryServiceImpl implements MemoryService, CommandLineRunner {
|
|||||||
}
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public void batchDelete(List<Long> ids) {
|
public void batchDelete(ChatMemoryDeleteReq chatMemoryDeleteReq, User user) {
|
||||||
QueryWrapper<ChatMemoryDO> queryWrapper = new QueryWrapper<>();
|
QueryWrapper<ChatMemoryDO> queryWrapper = new QueryWrapper<>();
|
||||||
queryWrapper.lambda().in(ChatMemoryDO::getId, ids);
|
if (!CollectionUtils.isEmpty(chatMemoryDeleteReq.getIds())) {
|
||||||
|
queryWrapper.lambda().in(ChatMemoryDO::getId, chatMemoryDeleteReq.getIds());
|
||||||
|
}
|
||||||
|
if (chatMemoryDeleteReq.getAgentId() != null) {
|
||||||
|
queryWrapper.lambda().eq(ChatMemoryDO::getAgentId, chatMemoryDeleteReq.getAgentId());
|
||||||
|
}
|
||||||
List<ChatMemoryDO> chatMemoryDOS = chatMemoryRepository.getMemories(queryWrapper);
|
List<ChatMemoryDO> chatMemoryDOS = chatMemoryRepository.getMemories(queryWrapper);
|
||||||
|
List<Long> ids = new ArrayList<>();
|
||||||
chatMemoryDOS.forEach(chatMemoryDO -> {
|
chatMemoryDOS.forEach(chatMemoryDO -> {
|
||||||
if (MemoryStatus.ENABLED.toString().equals(chatMemoryDO.getStatus().trim())) {
|
if (MemoryStatus.ENABLED.toString().equals(chatMemoryDO.getStatus().trim())) {
|
||||||
disableMemory(chatMemoryDO);
|
disableMemory(chatMemoryDO);
|
||||||
}
|
}
|
||||||
|
ids.add(chatMemoryDO.getId());
|
||||||
});
|
});
|
||||||
chatMemoryRepository.batchDelete(ids);
|
chatMemoryRepository.batchDelete(ids);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -108,6 +108,7 @@ public class PluginServiceImpl implements PluginService {
|
|||||||
if (StringUtils.isNotBlank(pluginQueryReq.getCreatedBy())) {
|
if (StringUtils.isNotBlank(pluginQueryReq.getCreatedBy())) {
|
||||||
queryWrapper.lambda().eq(PluginDO::getCreatedBy, pluginQueryReq.getCreatedBy());
|
queryWrapper.lambda().eq(PluginDO::getCreatedBy, pluginQueryReq.getCreatedBy());
|
||||||
}
|
}
|
||||||
|
queryWrapper.orderByAsc("name");
|
||||||
List<PluginDO> pluginDOS = pluginRepository.query(queryWrapper);
|
List<PluginDO> pluginDOS = pluginRepository.query(queryWrapper);
|
||||||
if (StringUtils.isNotBlank(pluginQueryReq.getPattern())) {
|
if (StringUtils.isNotBlank(pluginQueryReq.getPattern())) {
|
||||||
pluginDOS = pluginDOS.stream()
|
pluginDOS = pluginDOS.stream()
|
||||||
|
|||||||
@@ -46,6 +46,62 @@ public class FieldValueReplaceVisitor extends ExpressionVisitorAdapter {
|
|||||||
replaceComparisonExpression(expr);
|
replaceComparisonExpression(expr);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public void visit(LikeExpression expr) {
|
||||||
|
Expression leftExpression = expr.getLeftExpression();
|
||||||
|
Expression rightExpression = expr.getRightExpression();
|
||||||
|
|
||||||
|
if (!(leftExpression instanceof Column)) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (CollectionUtils.isEmpty(filedNameToValueMap)) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (Objects.isNull(rightExpression) || Objects.isNull(leftExpression)) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
Column column = (Column) leftExpression;
|
||||||
|
String columnName = column.getColumnName();
|
||||||
|
if (StringUtils.isEmpty(columnName)) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
Map<String, String> valueMap = filedNameToValueMap.get(columnName);
|
||||||
|
if (Objects.isNull(valueMap) || valueMap.isEmpty()) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (rightExpression instanceof StringValue) {
|
||||||
|
StringValue rightStringValue = (StringValue) rightExpression;
|
||||||
|
String value = rightStringValue.getValue();
|
||||||
|
|
||||||
|
// 使用split处理方式,按通配符分割字符串,对每个片段进行转换
|
||||||
|
String[] parts = value.split("%", -1);
|
||||||
|
boolean changed = false;
|
||||||
|
|
||||||
|
// 处理每个部分
|
||||||
|
for (int i = 0; i < parts.length; i++) {
|
||||||
|
if (!parts[i].isEmpty()) {
|
||||||
|
String replaceValue = getReplaceValue(valueMap, parts[i]);
|
||||||
|
if (StringUtils.isNotEmpty(replaceValue) && !parts[i].equals(replaceValue)) {
|
||||||
|
parts[i] = replaceValue;
|
||||||
|
changed = true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 如果有任何部分发生变化,则重新构建字符串
|
||||||
|
if (changed) {
|
||||||
|
StringBuilder sb = new StringBuilder();
|
||||||
|
for (int i = 0; i < parts.length; i++) {
|
||||||
|
sb.append(parts[i]);
|
||||||
|
// 除了最后一个部分,其他部分后面都需要加上"%"
|
||||||
|
if (i < parts.length - 1) {
|
||||||
|
sb.append("%");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
rightStringValue.setValue(sb.toString());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
public void visit(InExpression inExpression) {
|
public void visit(InExpression inExpression) {
|
||||||
if (!(inExpression.getLeftExpression() instanceof Column)) {
|
if (!(inExpression.getLeftExpression() instanceof Column)) {
|
||||||
return;
|
return;
|
||||||
|
|||||||
@@ -16,16 +16,21 @@ import net.sf.jsqlparser.parser.CCJSqlParserUtil;
|
|||||||
import net.sf.jsqlparser.schema.Column;
|
import net.sf.jsqlparser.schema.Column;
|
||||||
import org.apache.commons.collections.CollectionUtils;
|
import org.apache.commons.collections.CollectionUtils;
|
||||||
|
|
||||||
import java.util.ArrayList;
|
import java.util.*;
|
||||||
import java.util.List;
|
|
||||||
import java.util.Objects;
|
|
||||||
import java.util.Set;
|
|
||||||
|
|
||||||
@Slf4j
|
@Slf4j
|
||||||
public class FiledFilterReplaceVisitor extends ExpressionVisitorAdapter {
|
public class FiledFilterReplaceVisitor extends ExpressionVisitorAdapter {
|
||||||
|
|
||||||
private List<Expression> waitingForAdds = new ArrayList<>();
|
private List<Expression> waitingForAdds = new ArrayList<>();
|
||||||
private Set<String> fieldNames;
|
private Set<String> fieldNames;
|
||||||
|
private Map<String, String> fieldNameMap = new HashMap<>();
|
||||||
|
|
||||||
|
private static Set<String> HAVING_AGG_TYPES = Set.of("SUM", "AVG", "MAX", "MIN", "COUNT");
|
||||||
|
|
||||||
|
public FiledFilterReplaceVisitor(Map<String, String> fieldNameMap) {
|
||||||
|
this.fieldNameMap = fieldNameMap;
|
||||||
|
this.fieldNames = fieldNameMap.keySet();
|
||||||
|
}
|
||||||
|
|
||||||
public FiledFilterReplaceVisitor(Set<String> fieldNames) {
|
public FiledFilterReplaceVisitor(Set<String> fieldNames) {
|
||||||
this.fieldNames = fieldNames;
|
this.fieldNames = fieldNames;
|
||||||
@@ -82,7 +87,22 @@ public class FiledFilterReplaceVisitor extends ExpressionVisitorAdapter {
|
|||||||
Expression leftExpression = comparisonOperator.getLeftExpression();
|
Expression leftExpression = comparisonOperator.getLeftExpression();
|
||||||
|
|
||||||
if (!(leftExpression instanceof Function)) {
|
if (!(leftExpression instanceof Function)) {
|
||||||
return result;
|
if (leftExpression instanceof Column) {
|
||||||
|
Column leftColumn = (Column) leftExpression;
|
||||||
|
String agg = fieldNameMap.get(leftColumn.getColumnName());
|
||||||
|
if (agg != null && HAVING_AGG_TYPES.contains(agg.toUpperCase())) {
|
||||||
|
Expression expression = parseCondExpression(comparisonOperator, condExpr);
|
||||||
|
if (Objects.nonNull(expression)) {
|
||||||
|
result.add(expression);
|
||||||
|
return result;
|
||||||
|
} else {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return result;
|
||||||
|
} else {
|
||||||
|
return result;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
Function leftFunction = (Function) leftExpression;
|
Function leftFunction = (Function) leftExpression;
|
||||||
@@ -102,14 +122,24 @@ public class FiledFilterReplaceVisitor extends ExpressionVisitorAdapter {
|
|||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
Expression expression = parseCondExpression(comparisonOperator, condExpr);
|
||||||
|
if (Objects.nonNull(expression)) {
|
||||||
|
result.add(expression);
|
||||||
|
return result;
|
||||||
|
} else {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private Expression parseCondExpression(ComparisonOperator comparisonOperator, String condExpr) {
|
||||||
try {
|
try {
|
||||||
|
String comparisonOperatorStr = comparisonOperator.toString();
|
||||||
ComparisonOperator parsedExpression =
|
ComparisonOperator parsedExpression =
|
||||||
(ComparisonOperator) CCJSqlParserUtil.parseCondExpression(condExpr);
|
(ComparisonOperator) CCJSqlParserUtil.parseCondExpression(condExpr);
|
||||||
comparisonOperator.setLeftExpression(parsedExpression.getLeftExpression());
|
comparisonOperator.setLeftExpression(parsedExpression.getLeftExpression());
|
||||||
comparisonOperator.setRightExpression(parsedExpression.getRightExpression());
|
comparisonOperator.setRightExpression(parsedExpression.getRightExpression());
|
||||||
comparisonOperator.setASTNode(parsedExpression.getASTNode());
|
comparisonOperator.setASTNode(parsedExpression.getASTNode());
|
||||||
result.add(CCJSqlParserUtil.parseCondExpression(comparisonOperatorStr));
|
return CCJSqlParserUtil.parseCondExpression(comparisonOperatorStr);
|
||||||
return result;
|
|
||||||
} catch (JSQLParserException e) {
|
} catch (JSQLParserException e) {
|
||||||
log.error("JSQLParserException", e);
|
log.error("JSQLParserException", e);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -309,7 +309,7 @@ public class SqlAddHelper {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
public static String addHaving(String sql, Set<String> fieldNames) {
|
public static String addHaving(String sql, Map<String, String> fieldNames) {
|
||||||
Select selectStatement = SqlSelectHelper.getSelect(sql);
|
Select selectStatement = SqlSelectHelper.getSelect(sql);
|
||||||
|
|
||||||
if (!(selectStatement instanceof PlainSelect)) {
|
if (!(selectStatement instanceof PlainSelect)) {
|
||||||
|
|||||||
@@ -1,27 +1,26 @@
|
|||||||
package com.tencent.supersonic.common.pojo;
|
package com.tencent.supersonic.common.pojo;
|
||||||
|
|
||||||
import com.tencent.supersonic.common.pojo.enums.EventType;
|
import com.tencent.supersonic.common.pojo.enums.EventType;
|
||||||
|
import lombok.Getter;
|
||||||
import org.springframework.context.ApplicationEvent;
|
import org.springframework.context.ApplicationEvent;
|
||||||
|
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
|
|
||||||
|
@Getter
|
||||||
public class DataEvent extends ApplicationEvent {
|
public class DataEvent extends ApplicationEvent {
|
||||||
|
|
||||||
private List<DataItem> dataItems;
|
private final List<DataItem> dataItems;
|
||||||
|
|
||||||
private EventType eventType;
|
private final EventType eventType;
|
||||||
|
|
||||||
public DataEvent(Object source, List<DataItem> dataItems, EventType eventType) {
|
private final String userName;
|
||||||
|
|
||||||
|
public DataEvent(Object source, List<DataItem> dataItems, EventType eventType,
|
||||||
|
String userName) {
|
||||||
super(source);
|
super(source);
|
||||||
this.dataItems = dataItems;
|
this.dataItems = dataItems;
|
||||||
this.eventType = eventType;
|
this.eventType = eventType;
|
||||||
|
this.userName = userName;
|
||||||
}
|
}
|
||||||
|
|
||||||
public List<DataItem> getDataItems() {
|
|
||||||
return dataItems;
|
|
||||||
}
|
|
||||||
|
|
||||||
public EventType getEventType() {
|
|
||||||
return eventType;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -5,4 +5,6 @@ public class DimensionConstants {
|
|||||||
public static final String DIMENSION_TIME_FORMAT = "time_format";
|
public static final String DIMENSION_TIME_FORMAT = "time_format";
|
||||||
|
|
||||||
public static final String DIMENSION_TYPE = "dimension_type";
|
public static final String DIMENSION_TYPE = "dimension_type";
|
||||||
|
|
||||||
|
public static final String DIMENSION_DATA_TYPE = "dimension_data_type";
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
package com.tencent.supersonic.common.pojo.enums;
|
package com.tencent.supersonic.common.pojo.enums;
|
||||||
|
|
||||||
public enum TypeEnums {
|
public enum TypeEnums {
|
||||||
METRIC, DIMENSION, TAG, DOMAIN, DATASET, MODEL, UNKNOWN
|
METRIC, DIMENSION, VALUE, TAG, DOMAIN, DATASET, MODEL, UNKNOWN
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -15,5 +15,5 @@ public interface ChatModelService {
|
|||||||
|
|
||||||
ChatModel updateChatModel(ChatModel chatModel, User user);
|
ChatModel updateChatModel(ChatModel chatModel, User user);
|
||||||
|
|
||||||
void deleteChatModel(Integer id);
|
void deleteChatModel(Integer id, User user);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -79,7 +79,12 @@ public class ChatModelServiceImpl extends ServiceImpl<ChatModelMapper, ChatModel
|
|||||||
}
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public void deleteChatModel(Integer id) {
|
public void deleteChatModel(Integer id, User user) {
|
||||||
|
ChatModel chatModel = getChatModel(id);
|
||||||
|
if (!checkAdminPermission(user, chatModel)) {
|
||||||
|
throw new RuntimeException("没有权限删除该大模型");
|
||||||
|
}
|
||||||
|
|
||||||
removeById(id);
|
removeById(id);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -103,4 +108,13 @@ public class ChatModelServiceImpl extends ServiceImpl<ChatModelMapper, ChatModel
|
|||||||
chatModelDO.setConfig(JsonUtil.toString(chatModel.getConfig()));
|
chatModelDO.setConfig(JsonUtil.toString(chatModel.getConfig()));
|
||||||
return chatModelDO;
|
return chatModelDO;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private boolean checkAdminPermission(User user, ChatModel chatModel) {
|
||||||
|
String admin = chatModel.getAdmin();
|
||||||
|
if (user.isSuperAdmin()) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
return admin != null && admin.equals(user.getName())
|
||||||
|
|| chatModel.getCreatedBy().equals(user.getName());
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -16,6 +16,7 @@ import org.springframework.util.CollectionUtils;
|
|||||||
|
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
import java.util.concurrent.atomic.AtomicReference;
|
import java.util.concurrent.atomic.AtomicReference;
|
||||||
|
import java.util.stream.Collectors;
|
||||||
|
|
||||||
@Service
|
@Service
|
||||||
public class SystemConfigServiceImpl extends ServiceImpl<SystemConfigMapper, SystemConfigDO>
|
public class SystemConfigServiceImpl extends ServiceImpl<SystemConfigMapper, SystemConfigDO>
|
||||||
@@ -38,8 +39,8 @@ public class SystemConfigServiceImpl extends ServiceImpl<SystemConfigMapper, Sys
|
|||||||
return systemConfigDb;
|
return systemConfigDb;
|
||||||
}
|
}
|
||||||
|
|
||||||
private SystemConfig getSystemConfigFromDB() {
|
private SystemConfig getSystemConfigFromDB() { // 加上id ,如果有多条记录,会出错
|
||||||
List<SystemConfigDO> list = list();
|
List<SystemConfigDO> list = this.lambdaQuery().eq(SystemConfigDO::getId, 1).list();
|
||||||
if (CollectionUtils.isEmpty(list)) {
|
if (CollectionUtils.isEmpty(list)) {
|
||||||
SystemConfig systemConfig = new SystemConfig();
|
SystemConfig systemConfig = new SystemConfig();
|
||||||
systemConfig.setId(1);
|
systemConfig.setId(1);
|
||||||
|
|||||||
@@ -7,6 +7,7 @@ import dev.langchain4j.model.chat.ChatLanguageModel;
|
|||||||
import dev.langchain4j.model.dify.DifyAiChatModel;
|
import dev.langchain4j.model.dify.DifyAiChatModel;
|
||||||
import dev.langchain4j.model.embedding.EmbeddingModel;
|
import dev.langchain4j.model.embedding.EmbeddingModel;
|
||||||
import dev.langchain4j.model.openai.OpenAiEmbeddingModel;
|
import dev.langchain4j.model.openai.OpenAiEmbeddingModel;
|
||||||
|
import dev.langchain4j.model.openai.OpenAiStreamingChatModel;
|
||||||
import org.springframework.beans.factory.InitializingBean;
|
import org.springframework.beans.factory.InitializingBean;
|
||||||
import org.springframework.stereotype.Service;
|
import org.springframework.stereotype.Service;
|
||||||
|
|
||||||
@@ -25,6 +26,11 @@ public class DifyModelFactory implements ModelFactory, InitializingBean {
|
|||||||
.modelName(modelConfig.getModelName()).timeOut(modelConfig.getTimeOut()).build();
|
.modelName(modelConfig.getModelName()).timeOut(modelConfig.getTimeOut()).build();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public OpenAiStreamingChatModel createChatStreamingModel(ChatModelConfig modelConfig) {
|
||||||
|
throw new RuntimeException("待开发");
|
||||||
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public EmbeddingModel createEmbeddingModel(EmbeddingModelConfig embeddingModelConfig) {
|
public EmbeddingModel createEmbeddingModel(EmbeddingModelConfig embeddingModelConfig) {
|
||||||
return OpenAiEmbeddingModel.builder().baseUrl(embeddingModelConfig.getBaseUrl())
|
return OpenAiEmbeddingModel.builder().baseUrl(embeddingModelConfig.getBaseUrl())
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import com.tencent.supersonic.common.pojo.EmbeddingModelConfig;
|
|||||||
import dev.langchain4j.model.chat.ChatLanguageModel;
|
import dev.langchain4j.model.chat.ChatLanguageModel;
|
||||||
import dev.langchain4j.model.embedding.EmbeddingModel;
|
import dev.langchain4j.model.embedding.EmbeddingModel;
|
||||||
import dev.langchain4j.model.embedding.S2OnnxEmbeddingModel;
|
import dev.langchain4j.model.embedding.S2OnnxEmbeddingModel;
|
||||||
|
import dev.langchain4j.model.openai.OpenAiStreamingChatModel;
|
||||||
import org.apache.commons.lang3.StringUtils;
|
import org.apache.commons.lang3.StringUtils;
|
||||||
import org.springframework.beans.factory.InitializingBean;
|
import org.springframework.beans.factory.InitializingBean;
|
||||||
import org.springframework.stereotype.Service;
|
import org.springframework.stereotype.Service;
|
||||||
@@ -35,6 +36,11 @@ public class InMemoryModelFactory implements ModelFactory, InitializingBean {
|
|||||||
return EmbeddingModelConstant.BGE_SMALL_ZH_MODEL;
|
return EmbeddingModelConstant.BGE_SMALL_ZH_MODEL;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public OpenAiStreamingChatModel createChatStreamingModel(ChatModelConfig modelConfig) {
|
||||||
|
throw new RuntimeException("待开发");
|
||||||
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public void afterPropertiesSet() {
|
public void afterPropertiesSet() {
|
||||||
ModelProvider.add(PROVIDER, this);
|
ModelProvider.add(PROVIDER, this);
|
||||||
|
|||||||
@@ -6,6 +6,7 @@ import dev.langchain4j.model.chat.ChatLanguageModel;
|
|||||||
import dev.langchain4j.model.embedding.EmbeddingModel;
|
import dev.langchain4j.model.embedding.EmbeddingModel;
|
||||||
import dev.langchain4j.model.localai.LocalAiChatModel;
|
import dev.langchain4j.model.localai.LocalAiChatModel;
|
||||||
import dev.langchain4j.model.localai.LocalAiEmbeddingModel;
|
import dev.langchain4j.model.localai.LocalAiEmbeddingModel;
|
||||||
|
import dev.langchain4j.model.openai.OpenAiStreamingChatModel;
|
||||||
import org.springframework.beans.factory.InitializingBean;
|
import org.springframework.beans.factory.InitializingBean;
|
||||||
import org.springframework.stereotype.Service;
|
import org.springframework.stereotype.Service;
|
||||||
|
|
||||||
@@ -27,6 +28,11 @@ public class LocalAiModelFactory implements ModelFactory, InitializingBean {
|
|||||||
.build();
|
.build();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public OpenAiStreamingChatModel createChatStreamingModel(ChatModelConfig modelConfig) {
|
||||||
|
throw new RuntimeException("待开发");
|
||||||
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public EmbeddingModel createEmbeddingModel(EmbeddingModelConfig embeddingModel) {
|
public EmbeddingModel createEmbeddingModel(EmbeddingModelConfig embeddingModel) {
|
||||||
return LocalAiEmbeddingModel.builder().baseUrl(embeddingModel.getBaseUrl())
|
return LocalAiEmbeddingModel.builder().baseUrl(embeddingModel.getBaseUrl())
|
||||||
|
|||||||
@@ -4,9 +4,12 @@ import com.tencent.supersonic.common.pojo.ChatModelConfig;
|
|||||||
import com.tencent.supersonic.common.pojo.EmbeddingModelConfig;
|
import com.tencent.supersonic.common.pojo.EmbeddingModelConfig;
|
||||||
import dev.langchain4j.model.chat.ChatLanguageModel;
|
import dev.langchain4j.model.chat.ChatLanguageModel;
|
||||||
import dev.langchain4j.model.embedding.EmbeddingModel;
|
import dev.langchain4j.model.embedding.EmbeddingModel;
|
||||||
|
import dev.langchain4j.model.openai.OpenAiStreamingChatModel;
|
||||||
|
|
||||||
public interface ModelFactory {
|
public interface ModelFactory {
|
||||||
ChatLanguageModel createChatModel(ChatModelConfig modelConfig);
|
ChatLanguageModel createChatModel(ChatModelConfig modelConfig);
|
||||||
|
|
||||||
|
OpenAiStreamingChatModel createChatStreamingModel(ChatModelConfig modelConfig);
|
||||||
|
|
||||||
EmbeddingModel createEmbeddingModel(EmbeddingModelConfig embeddingModel);
|
EmbeddingModel createEmbeddingModel(EmbeddingModelConfig embeddingModel);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -5,7 +5,9 @@ import com.tencent.supersonic.common.pojo.ChatModelConfig;
|
|||||||
import com.tencent.supersonic.common.pojo.EmbeddingModelConfig;
|
import com.tencent.supersonic.common.pojo.EmbeddingModelConfig;
|
||||||
import com.tencent.supersonic.common.util.ContextUtils;
|
import com.tencent.supersonic.common.util.ContextUtils;
|
||||||
import dev.langchain4j.model.chat.ChatLanguageModel;
|
import dev.langchain4j.model.chat.ChatLanguageModel;
|
||||||
|
import dev.langchain4j.model.chat.StreamingChatLanguageModel;
|
||||||
import dev.langchain4j.model.embedding.EmbeddingModel;
|
import dev.langchain4j.model.embedding.EmbeddingModel;
|
||||||
|
import dev.langchain4j.model.openai.OpenAiStreamingChatModel;
|
||||||
import org.apache.commons.lang3.StringUtils;
|
import org.apache.commons.lang3.StringUtils;
|
||||||
|
|
||||||
import java.util.HashMap;
|
import java.util.HashMap;
|
||||||
@@ -41,6 +43,20 @@ public class ModelProvider {
|
|||||||
"Unsupported ChatLanguageModel provider: " + modelConfig.getProvider());
|
"Unsupported ChatLanguageModel provider: " + modelConfig.getProvider());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public static StreamingChatLanguageModel getChatStreamingModel(ChatModelConfig modelConfig) {
|
||||||
|
if (modelConfig == null || StringUtils.isBlank(modelConfig.getProvider())
|
||||||
|
|| StringUtils.isBlank(modelConfig.getBaseUrl())) {
|
||||||
|
modelConfig = DEMO_CHAT_MODEL;
|
||||||
|
}
|
||||||
|
ModelFactory modelFactory = factories.get(modelConfig.getProvider().toUpperCase());
|
||||||
|
if (modelFactory != null) {
|
||||||
|
return modelFactory.createChatStreamingModel(modelConfig);
|
||||||
|
}
|
||||||
|
|
||||||
|
throw new RuntimeException(
|
||||||
|
"Unsupported ChatLanguageModel provider: " + modelConfig.getProvider());
|
||||||
|
}
|
||||||
|
|
||||||
public static EmbeddingModel getEmbeddingModel() {
|
public static EmbeddingModel getEmbeddingModel() {
|
||||||
return getEmbeddingModel(null);
|
return getEmbeddingModel(null);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -6,6 +6,7 @@ import dev.langchain4j.model.chat.ChatLanguageModel;
|
|||||||
import dev.langchain4j.model.embedding.EmbeddingModel;
|
import dev.langchain4j.model.embedding.EmbeddingModel;
|
||||||
import dev.langchain4j.model.ollama.OllamaChatModel;
|
import dev.langchain4j.model.ollama.OllamaChatModel;
|
||||||
import dev.langchain4j.model.ollama.OllamaEmbeddingModel;
|
import dev.langchain4j.model.ollama.OllamaEmbeddingModel;
|
||||||
|
import dev.langchain4j.model.openai.OpenAiStreamingChatModel;
|
||||||
import org.springframework.beans.factory.InitializingBean;
|
import org.springframework.beans.factory.InitializingBean;
|
||||||
import org.springframework.stereotype.Service;
|
import org.springframework.stereotype.Service;
|
||||||
|
|
||||||
@@ -28,6 +29,11 @@ public class OllamaModelFactory implements ModelFactory, InitializingBean {
|
|||||||
.logResponses(modelConfig.getLogResponses()).build();
|
.logResponses(modelConfig.getLogResponses()).build();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public OpenAiStreamingChatModel createChatStreamingModel(ChatModelConfig modelConfig) {
|
||||||
|
throw new RuntimeException("待开发");
|
||||||
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public EmbeddingModel createEmbeddingModel(EmbeddingModelConfig embeddingModelConfig) {
|
public EmbeddingModel createEmbeddingModel(EmbeddingModelConfig embeddingModelConfig) {
|
||||||
return OllamaEmbeddingModel.builder().baseUrl(embeddingModelConfig.getBaseUrl())
|
return OllamaEmbeddingModel.builder().baseUrl(embeddingModelConfig.getBaseUrl())
|
||||||
|
|||||||
@@ -6,6 +6,7 @@ import dev.langchain4j.model.chat.ChatLanguageModel;
|
|||||||
import dev.langchain4j.model.embedding.EmbeddingModel;
|
import dev.langchain4j.model.embedding.EmbeddingModel;
|
||||||
import dev.langchain4j.model.openai.OpenAiChatModel;
|
import dev.langchain4j.model.openai.OpenAiChatModel;
|
||||||
import dev.langchain4j.model.openai.OpenAiEmbeddingModel;
|
import dev.langchain4j.model.openai.OpenAiEmbeddingModel;
|
||||||
|
import dev.langchain4j.model.openai.OpenAiStreamingChatModel;
|
||||||
import org.springframework.beans.factory.InitializingBean;
|
import org.springframework.beans.factory.InitializingBean;
|
||||||
import org.springframework.stereotype.Service;
|
import org.springframework.stereotype.Service;
|
||||||
|
|
||||||
@@ -30,13 +31,23 @@ public class OpenAiModelFactory implements ModelFactory, InitializingBean {
|
|||||||
.timeout(Duration.ofSeconds(modelConfig.getTimeOut()))
|
.timeout(Duration.ofSeconds(modelConfig.getTimeOut()))
|
||||||
.logRequests(modelConfig.getLogRequests())
|
.logRequests(modelConfig.getLogRequests())
|
||||||
.logResponses(modelConfig.getLogResponses());
|
.logResponses(modelConfig.getLogResponses());
|
||||||
if (modelConfig.getJsonFormat()) {
|
if (modelConfig.getJsonFormat() != null && modelConfig.getJsonFormat()) {
|
||||||
openAiChatModelBuilder.strictJsonSchema(true)
|
openAiChatModelBuilder.strictJsonSchema(true)
|
||||||
.responseFormat(modelConfig.getJsonFormatType());
|
.responseFormat(modelConfig.getJsonFormatType());
|
||||||
}
|
}
|
||||||
return openAiChatModelBuilder.build();
|
return openAiChatModelBuilder.build();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public OpenAiStreamingChatModel createChatStreamingModel(ChatModelConfig modelConfig) {
|
||||||
|
return OpenAiStreamingChatModel.builder().baseUrl(modelConfig.getBaseUrl())
|
||||||
|
.modelName(modelConfig.getModelName()).apiKey(modelConfig.keyDecrypt())
|
||||||
|
.temperature(modelConfig.getTemperature()).topP(modelConfig.getTopP())
|
||||||
|
.timeout(Duration.ofSeconds(modelConfig.getTimeOut()))
|
||||||
|
.logRequests(modelConfig.getLogRequests())
|
||||||
|
.logResponses(modelConfig.getLogResponses()).build();
|
||||||
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public EmbeddingModel createEmbeddingModel(EmbeddingModelConfig embeddingModel) {
|
public EmbeddingModel createEmbeddingModel(EmbeddingModelConfig embeddingModel) {
|
||||||
return OpenAiEmbeddingModel.builder().baseUrl(embeddingModel.getBaseUrl())
|
return OpenAiEmbeddingModel.builder().baseUrl(embeddingModel.getBaseUrl())
|
||||||
|
|||||||
@@ -42,6 +42,6 @@ public class TextSegmentConvert {
|
|||||||
if (Objects.isNull(textSegment) || Objects.isNull(textSegment.metadata())) {
|
if (Objects.isNull(textSegment) || Objects.isNull(textSegment.metadata())) {
|
||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
return textSegment.metadata().get(QUERY_ID);
|
return textSegment.metadata().getString(QUERY_ID);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -338,8 +338,8 @@ class SqlAddHelperTest {
|
|||||||
List<String> groupByFields = new ArrayList<>();
|
List<String> groupByFields = new ArrayList<>();
|
||||||
groupByFields.add("department");
|
groupByFields.add("department");
|
||||||
|
|
||||||
Set<String> fieldNames = new HashSet<>();
|
Map<String, String> fieldNames = new HashMap<>();
|
||||||
fieldNames.add("pv");
|
fieldNames.put("pv", "sum");
|
||||||
|
|
||||||
String replaceSql = SqlAddHelper.addHaving(sql, fieldNames);
|
String replaceSql = SqlAddHelper.addHaving(sql, fieldNames);
|
||||||
|
|
||||||
@@ -355,6 +355,14 @@ class SqlAddHelperTest {
|
|||||||
Assert.assertEquals("SELECT department, sum(pv) FROM t_1 WHERE sys_imp_date = '2023-09-11' "
|
Assert.assertEquals("SELECT department, sum(pv) FROM t_1 WHERE sys_imp_date = '2023-09-11' "
|
||||||
+ "GROUP BY department HAVING sum(pv) > 2000 ORDER BY sum(pv) DESC LIMIT 10",
|
+ "GROUP BY department HAVING sum(pv) > 2000 ORDER BY sum(pv) DESC LIMIT 10",
|
||||||
replaceSql);
|
replaceSql);
|
||||||
|
|
||||||
|
sql = "SELECT 数据日期,访问用户数 FROM 超音数数据集 WHERE 访问次数 > 10 GROUP BY 数据日期";
|
||||||
|
|
||||||
|
fieldNames.put("访问次数", "sum");
|
||||||
|
replaceSql = SqlAddHelper.addHaving(sql, fieldNames);
|
||||||
|
|
||||||
|
Assert.assertEquals("SELECT 数据日期, 访问用户数 FROM 超音数数据集 GROUP BY 数据日期 HAVING 访问次数 > 10",
|
||||||
|
replaceSql);
|
||||||
}
|
}
|
||||||
|
|
||||||
@Test
|
@Test
|
||||||
|
|||||||
@@ -3,6 +3,7 @@ package com.tencent.supersonic.headless.chat.corrector;
|
|||||||
import com.tencent.supersonic.common.jsqlparser.SqlAddHelper;
|
import com.tencent.supersonic.common.jsqlparser.SqlAddHelper;
|
||||||
import com.tencent.supersonic.common.jsqlparser.SqlSelectFunctionHelper;
|
import com.tencent.supersonic.common.jsqlparser.SqlSelectFunctionHelper;
|
||||||
import com.tencent.supersonic.common.jsqlparser.SqlSelectHelper;
|
import com.tencent.supersonic.common.jsqlparser.SqlSelectHelper;
|
||||||
|
import com.tencent.supersonic.headless.api.pojo.SchemaElement;
|
||||||
import com.tencent.supersonic.headless.api.pojo.SemanticParseInfo;
|
import com.tencent.supersonic.headless.api.pojo.SemanticParseInfo;
|
||||||
import com.tencent.supersonic.headless.api.pojo.SemanticSchema;
|
import com.tencent.supersonic.headless.api.pojo.SemanticSchema;
|
||||||
import com.tencent.supersonic.headless.chat.ChatQueryContext;
|
import com.tencent.supersonic.headless.chat.ChatQueryContext;
|
||||||
@@ -11,7 +12,8 @@ import net.sf.jsqlparser.expression.Expression;
|
|||||||
import org.springframework.util.CollectionUtils;
|
import org.springframework.util.CollectionUtils;
|
||||||
|
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
import java.util.Set;
|
import java.util.Map;
|
||||||
|
import java.util.Optional;
|
||||||
import java.util.stream.Collectors;
|
import java.util.stream.Collectors;
|
||||||
|
|
||||||
/** Perform SQL corrections on the "Having" section in S2SQL. */
|
/** Perform SQL corrections on the "Having" section in S2SQL. */
|
||||||
@@ -29,8 +31,9 @@ public class HavingCorrector extends BaseSemanticCorrector {
|
|||||||
|
|
||||||
SemanticSchema semanticSchema = chatQueryContext.getSemanticSchema();
|
SemanticSchema semanticSchema = chatQueryContext.getSemanticSchema();
|
||||||
|
|
||||||
Set<String> metrics = semanticSchema.getMetrics(dataSet).stream()
|
Map<String, String> metrics = semanticSchema.getMetrics(dataSet).stream()
|
||||||
.map(schemaElement -> schemaElement.getName()).collect(Collectors.toSet());
|
.collect(Collectors.toMap(SchemaElement::getName,
|
||||||
|
e -> Optional.ofNullable(e.getDefaultAgg()).orElse("")));
|
||||||
|
|
||||||
if (CollectionUtils.isEmpty(metrics)) {
|
if (CollectionUtils.isEmpty(metrics)) {
|
||||||
return;
|
return;
|
||||||
|
|||||||
@@ -33,6 +33,7 @@ import java.util.Objects;
|
|||||||
import java.util.PriorityQueue;
|
import java.util.PriorityQueue;
|
||||||
import java.util.TreeMap;
|
import java.util.TreeMap;
|
||||||
import java.util.concurrent.ConcurrentHashMap;
|
import java.util.concurrent.ConcurrentHashMap;
|
||||||
|
import java.util.concurrent.PriorityBlockingQueue;
|
||||||
|
|
||||||
import static com.hankcs.hanlp.utility.Predefine.logger;
|
import static com.hankcs.hanlp.utility.Predefine.logger;
|
||||||
|
|
||||||
@@ -40,7 +41,7 @@ public class MultiCustomDictionary extends DynamicCustomDictionary {
|
|||||||
|
|
||||||
public static int MAX_SIZE = 10;
|
public static int MAX_SIZE = 10;
|
||||||
public static Boolean removeDuplicates = true;
|
public static Boolean removeDuplicates = true;
|
||||||
public static ConcurrentHashMap<String, PriorityQueue<Term>> NATURE_TO_VALUES =
|
public static ConcurrentHashMap<String, PriorityBlockingQueue<Term>> NATURE_TO_VALUES =
|
||||||
new ConcurrentHashMap<>();
|
new ConcurrentHashMap<>();
|
||||||
private static boolean addToSuggesterTrie = true;
|
private static boolean addToSuggesterTrie = true;
|
||||||
|
|
||||||
@@ -116,9 +117,17 @@ public class MultiCustomDictionary extends DynamicCustomDictionary {
|
|||||||
dictWord.setAlias(word.toLowerCase());
|
dictWord.setAlias(word.toLowerCase());
|
||||||
String[] split = nature.split(DictWordType.NATURE_SPILT);
|
String[] split = nature.split(DictWordType.NATURE_SPILT);
|
||||||
if (split.length >= 2) {
|
if (split.length >= 2) {
|
||||||
Long dimId = Long.parseLong(
|
try {
|
||||||
nature.split(DictWordType.NATURE_SPILT)[split.length - 1]);
|
Long dimId = Long.parseLong(
|
||||||
KnowledgeBaseService.addDimValueAlias(dimId, Arrays.asList(dictWord));
|
nature.split(DictWordType.NATURE_SPILT)[split.length - 1]);
|
||||||
|
KnowledgeBaseService.addDimValueAlias(dimId,
|
||||||
|
Arrays.asList(dictWord));
|
||||||
|
} catch (NumberFormatException e) {
|
||||||
|
logger.warning(path + " : 非标准文件,不存入KnowledgeBaseService");
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -146,9 +155,10 @@ public class MultiCustomDictionary extends DynamicCustomDictionary {
|
|||||||
}
|
}
|
||||||
for (int i = 0; i < attribute.nature.length; i++) {
|
for (int i = 0; i < attribute.nature.length; i++) {
|
||||||
Nature nature = attribute.nature[i];
|
Nature nature = attribute.nature[i];
|
||||||
PriorityQueue<Term> priorityQueue = NATURE_TO_VALUES.get(nature.toString());
|
PriorityBlockingQueue<Term> priorityQueue =
|
||||||
|
NATURE_TO_VALUES.get(nature.toString());
|
||||||
if (Objects.isNull(priorityQueue)) {
|
if (Objects.isNull(priorityQueue)) {
|
||||||
priorityQueue = new PriorityQueue<>(MAX_SIZE,
|
priorityQueue = new PriorityBlockingQueue<>(MAX_SIZE,
|
||||||
Comparator.comparingInt(Term::getFrequency).reversed());
|
Comparator.comparingInt(Term::getFrequency).reversed());
|
||||||
NATURE_TO_VALUES.put(nature.toString(), priorityQueue);
|
NATURE_TO_VALUES.put(nature.toString(), priorityQueue);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -24,6 +24,7 @@ import java.util.PriorityQueue;
|
|||||||
import java.util.Set;
|
import java.util.Set;
|
||||||
import java.util.TreeMap;
|
import java.util.TreeMap;
|
||||||
import java.util.TreeSet;
|
import java.util.TreeSet;
|
||||||
|
import java.util.concurrent.PriorityBlockingQueue;
|
||||||
import java.util.stream.Collectors;
|
import java.util.stream.Collectors;
|
||||||
|
|
||||||
@Slf4j
|
@Slf4j
|
||||||
@@ -200,7 +201,7 @@ public class SearchService {
|
|||||||
public static List<String> getDimensionValue(DimensionValueReq dimensionValueReq) {
|
public static List<String> getDimensionValue(DimensionValueReq dimensionValueReq) {
|
||||||
String nature = DictWordType.NATURE_SPILT + dimensionValueReq.getModelId()
|
String nature = DictWordType.NATURE_SPILT + dimensionValueReq.getModelId()
|
||||||
+ DictWordType.NATURE_SPILT + dimensionValueReq.getElementID();
|
+ DictWordType.NATURE_SPILT + dimensionValueReq.getElementID();
|
||||||
PriorityQueue<Term> terms = MultiCustomDictionary.NATURE_TO_VALUES.get(nature);
|
PriorityBlockingQueue<Term> terms = MultiCustomDictionary.NATURE_TO_VALUES.get(nature);
|
||||||
if (CollectionUtils.isEmpty(terms)) {
|
if (CollectionUtils.isEmpty(terms)) {
|
||||||
return new ArrayList<>();
|
return new ArrayList<>();
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -12,12 +12,17 @@ import org.springframework.beans.factory.annotation.Qualifier;
|
|||||||
import org.springframework.stereotype.Service;
|
import org.springframework.stereotype.Service;
|
||||||
|
|
||||||
import java.util.HashMap;
|
import java.util.HashMap;
|
||||||
|
import java.util.HashSet;
|
||||||
import java.util.List;
|
import java.util.List;
|
||||||
import java.util.Map;
|
import java.util.Map;
|
||||||
import java.util.Objects;
|
import java.util.Objects;
|
||||||
import java.util.Set;
|
import java.util.Set;
|
||||||
import java.util.concurrent.Callable;
|
import java.util.concurrent.CompletableFuture;
|
||||||
|
import java.util.concurrent.ExecutionException;
|
||||||
import java.util.concurrent.ThreadPoolExecutor;
|
import java.util.concurrent.ThreadPoolExecutor;
|
||||||
|
import java.util.function.Function;
|
||||||
|
import java.util.function.Supplier;
|
||||||
|
import java.util.stream.Collectors;
|
||||||
|
|
||||||
@Service
|
@Service
|
||||||
@Slf4j
|
@Slf4j
|
||||||
@@ -72,18 +77,39 @@ public abstract class BaseMatchStrategy<T extends MapResult> implements MatchStr
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
protected void executeTasks(List<Callable<Void>> tasks) {
|
protected Set<T> executeTasks(List<Supplier<List<T>>> tasks) {
|
||||||
|
|
||||||
|
Function<Supplier<List<T>>, Supplier<List<T>>> decorator = taskDecorator();
|
||||||
|
List<CompletableFuture<List<T>>> futures;
|
||||||
|
if (decorator == null) {
|
||||||
|
futures = tasks.stream().map(t -> CompletableFuture.supplyAsync(t, executor)).toList();
|
||||||
|
} else {
|
||||||
|
futures = tasks.stream()
|
||||||
|
.map(t -> CompletableFuture.supplyAsync(decorator.apply(t), executor)).toList();
|
||||||
|
}
|
||||||
|
|
||||||
|
CompletableFuture<List<T>> listCompletableFuture =
|
||||||
|
CompletableFuture.allOf(futures.toArray(new CompletableFuture<?>[0]))
|
||||||
|
.thenApply(v -> futures.stream()
|
||||||
|
.flatMap(listFuture -> listFuture.join().stream())
|
||||||
|
.collect(Collectors.toList()));
|
||||||
try {
|
try {
|
||||||
executor.invokeAll(tasks);
|
List<T> ts = listCompletableFuture.get();
|
||||||
for (Callable<Void> future : tasks) {
|
Set<T> results = new HashSet<>();
|
||||||
future.call();
|
selectResultInOneRound(results, ts);
|
||||||
}
|
return results;
|
||||||
} catch (Exception e) {
|
} catch (InterruptedException e) {
|
||||||
Thread.currentThread().interrupt();
|
Thread.currentThread().interrupt();
|
||||||
throw new RuntimeException("Task execution interrupted", e);
|
throw new RuntimeException("Task execution interrupted", e);
|
||||||
|
} catch (ExecutionException e) {
|
||||||
|
throw new RuntimeException(e);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public Function<Supplier<List<T>>, Supplier<List<T>>> taskDecorator() {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
public double getThreshold(Double threshold, Double minThreshold, MapModeEnum mapModeEnum) {
|
public double getThreshold(Double threshold, Double minThreshold, MapModeEnum mapModeEnum) {
|
||||||
if (MapModeEnum.STRICT.equals(mapModeEnum)) {
|
if (MapModeEnum.STRICT.equals(mapModeEnum)) {
|
||||||
return 1.0d;
|
return 1.0d;
|
||||||
|
|||||||
@@ -17,6 +17,8 @@ import java.util.List;
|
|||||||
import java.util.Map;
|
import java.util.Map;
|
||||||
import java.util.Map.Entry;
|
import java.util.Map.Entry;
|
||||||
import java.util.Set;
|
import java.util.Set;
|
||||||
|
import java.util.function.Function;
|
||||||
|
import java.util.function.Supplier;
|
||||||
import java.util.stream.Collectors;
|
import java.util.stream.Collectors;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -76,6 +78,22 @@ public class DatabaseMatchStrategy extends SingleMatchStrategy<DatabaseMapResult
|
|||||||
return allElements;
|
return allElements;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public Function<Supplier<List<DatabaseMapResult>>, Supplier<List<DatabaseMapResult>>> taskDecorator() {
|
||||||
|
List<SchemaElement> schemaElements = allElements.get();
|
||||||
|
if (CollectionUtils.isEmpty(schemaElements)) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
return (t) -> (Supplier<List<DatabaseMapResult>>) () -> {
|
||||||
|
try {
|
||||||
|
allElements.set(schemaElements);
|
||||||
|
return t.get();
|
||||||
|
} finally {
|
||||||
|
allElements.remove();
|
||||||
|
}
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
private Double getThreshold(ChatQueryContext chatQueryContext) {
|
private Double getThreshold(ChatQueryContext chatQueryContext) {
|
||||||
Double threshold =
|
Double threshold =
|
||||||
Double.valueOf(mapperConfig.getParameterValue(MapperConfig.MAPPER_NAME_THRESHOLD));
|
Double.valueOf(mapperConfig.getParameterValue(MapperConfig.MAPPER_NAME_THRESHOLD));
|
||||||
|
|||||||
@@ -24,8 +24,7 @@ import org.springframework.beans.factory.annotation.Autowired;
|
|||||||
import org.springframework.stereotype.Service;
|
import org.springframework.stereotype.Service;
|
||||||
|
|
||||||
import java.util.*;
|
import java.util.*;
|
||||||
import java.util.concurrent.Callable;
|
import java.util.function.Supplier;
|
||||||
import java.util.concurrent.ConcurrentHashMap;
|
|
||||||
import java.util.stream.Collectors;
|
import java.util.stream.Collectors;
|
||||||
|
|
||||||
import static com.tencent.supersonic.headless.chat.mapper.MapperConfig.*;
|
import static com.tencent.supersonic.headless.chat.mapper.MapperConfig.*;
|
||||||
@@ -141,7 +140,6 @@ public class EmbeddingMatchStrategy extends BatchMatchStrategy<EmbeddingResult>
|
|||||||
*/
|
*/
|
||||||
public List<EmbeddingResult> detectByBatch(ChatQueryContext chatQueryContext,
|
public List<EmbeddingResult> detectByBatch(ChatQueryContext chatQueryContext,
|
||||||
Set<Long> detectDataSetIds, Set<String> detectSegments, boolean useLlm) {
|
Set<Long> detectDataSetIds, Set<String> detectSegments, boolean useLlm) {
|
||||||
Set<EmbeddingResult> results = ConcurrentHashMap.newKeySet();
|
|
||||||
int embeddingMapperBatch = Integer
|
int embeddingMapperBatch = Integer
|
||||||
.valueOf(mapperConfig.getParameterValue(MapperConfig.EMBEDDING_MAPPER_BATCH));
|
.valueOf(mapperConfig.getParameterValue(MapperConfig.EMBEDDING_MAPPER_BATCH));
|
||||||
|
|
||||||
@@ -154,12 +152,11 @@ public class EmbeddingMatchStrategy extends BatchMatchStrategy<EmbeddingResult>
|
|||||||
Lists.partition(queryTextsList, embeddingMapperBatch);
|
Lists.partition(queryTextsList, embeddingMapperBatch);
|
||||||
|
|
||||||
// Create and execute tasks for each batch
|
// Create and execute tasks for each batch
|
||||||
List<Callable<Void>> tasks = new ArrayList<>();
|
List<Supplier<List<EmbeddingResult>>> tasks = new ArrayList<>();
|
||||||
for (List<String> queryTextsSub : queryTextsSubList) {
|
for (List<String> queryTextsSub : queryTextsSubList) {
|
||||||
tasks.add(
|
tasks.add(createTask(chatQueryContext, detectDataSetIds, queryTextsSub, useLlm));
|
||||||
createTask(chatQueryContext, detectDataSetIds, queryTextsSub, results, useLlm));
|
|
||||||
}
|
}
|
||||||
executeTasks(tasks);
|
Set<EmbeddingResult> results = executeTasks(tasks);
|
||||||
|
|
||||||
// Apply LLM filtering if enabled
|
// Apply LLM filtering if enabled
|
||||||
if (useLlm) {
|
if (useLlm) {
|
||||||
@@ -196,20 +193,13 @@ public class EmbeddingMatchStrategy extends BatchMatchStrategy<EmbeddingResult>
|
|||||||
* @param chatQueryContext The context of the chat query
|
* @param chatQueryContext The context of the chat query
|
||||||
* @param detectDataSetIds Target dataset IDs
|
* @param detectDataSetIds Target dataset IDs
|
||||||
* @param queryTextsSub Sub-list of query texts to process
|
* @param queryTextsSub Sub-list of query texts to process
|
||||||
* @param results Shared result set for collecting results
|
|
||||||
* @param useLlm Whether to use LLM
|
* @param useLlm Whether to use LLM
|
||||||
* @return Callable task
|
* @return Supplier task
|
||||||
*/
|
*/
|
||||||
private Callable<Void> createTask(ChatQueryContext chatQueryContext, Set<Long> detectDataSetIds,
|
private Supplier<List<EmbeddingResult>> createTask(ChatQueryContext chatQueryContext,
|
||||||
List<String> queryTextsSub, Set<EmbeddingResult> results, boolean useLlm) {
|
Set<Long> detectDataSetIds, List<String> queryTextsSub, boolean useLlm) {
|
||||||
return () -> {
|
return () -> detectByQueryTextsSub(detectDataSetIds, queryTextsSub, chatQueryContext,
|
||||||
List<EmbeddingResult> oneRoundResults = detectByQueryTextsSub(detectDataSetIds,
|
useLlm);
|
||||||
queryTextsSub, chatQueryContext, useLlm);
|
|
||||||
synchronized (results) {
|
|
||||||
selectResultInOneRound(results, oneRoundResults);
|
|
||||||
}
|
|
||||||
return null;
|
|
||||||
};
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|||||||
@@ -11,8 +11,7 @@ import java.util.ArrayList;
|
|||||||
import java.util.List;
|
import java.util.List;
|
||||||
import java.util.Map;
|
import java.util.Map;
|
||||||
import java.util.Set;
|
import java.util.Set;
|
||||||
import java.util.concurrent.Callable;
|
import java.util.function.Supplier;
|
||||||
import java.util.concurrent.ConcurrentHashMap;
|
|
||||||
|
|
||||||
@Service
|
@Service
|
||||||
@Slf4j
|
@Slf4j
|
||||||
@@ -26,8 +25,7 @@ public abstract class SingleMatchStrategy<T extends MapResult> extends BaseMatch
|
|||||||
Set<Long> detectDataSetIds) {
|
Set<Long> detectDataSetIds) {
|
||||||
Map<Integer, Integer> regOffsetToLength = mapperHelper.getRegOffsetToLength(terms);
|
Map<Integer, Integer> regOffsetToLength = mapperHelper.getRegOffsetToLength(terms);
|
||||||
String text = chatQueryContext.getRequest().getQueryText();
|
String text = chatQueryContext.getRequest().getQueryText();
|
||||||
Set<T> results = ConcurrentHashMap.newKeySet();
|
List<Supplier<List<T>>> tasks = new ArrayList<>();
|
||||||
List<Callable<Void>> tasks = new ArrayList<>();
|
|
||||||
|
|
||||||
for (int startIndex = 0; startIndex <= text.length() - 1;) {
|
for (int startIndex = 0; startIndex <= text.length() - 1;) {
|
||||||
for (int index = startIndex; index <= text.length();) {
|
for (int index = startIndex; index <= text.length();) {
|
||||||
@@ -35,27 +33,20 @@ public abstract class SingleMatchStrategy<T extends MapResult> extends BaseMatch
|
|||||||
index = mapperHelper.getStepIndex(regOffsetToLength, index);
|
index = mapperHelper.getStepIndex(regOffsetToLength, index);
|
||||||
if (index <= text.length()) {
|
if (index <= text.length()) {
|
||||||
String detectSegment = text.substring(startIndex, index).trim();
|
String detectSegment = text.substring(startIndex, index).trim();
|
||||||
Callable<Void> task = createTask(chatQueryContext, detectDataSetIds,
|
Supplier<List<T>> task =
|
||||||
detectSegment, offset, results);
|
createTask(chatQueryContext, detectDataSetIds, detectSegment, offset);
|
||||||
tasks.add(task);
|
tasks.add(task);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
startIndex = mapperHelper.getStepIndex(regOffsetToLength, startIndex);
|
startIndex = mapperHelper.getStepIndex(regOffsetToLength, startIndex);
|
||||||
}
|
}
|
||||||
executeTasks(tasks);
|
Set<T> results = executeTasks(tasks);
|
||||||
return new ArrayList<>(results);
|
return new ArrayList<>(results);
|
||||||
}
|
}
|
||||||
|
|
||||||
private Callable<Void> createTask(ChatQueryContext chatQueryContext, Set<Long> detectDataSetIds,
|
private Supplier<List<T>> createTask(ChatQueryContext chatQueryContext,
|
||||||
String detectSegment, int offset, Set<T> results) {
|
Set<Long> detectDataSetIds, String detectSegment, int offset) {
|
||||||
return () -> {
|
return () -> detectByStep(chatQueryContext, detectDataSetIds, detectSegment, offset);
|
||||||
List<T> oneRoundResults =
|
|
||||||
detectByStep(chatQueryContext, detectDataSetIds, detectSegment, offset);
|
|
||||||
synchronized (results) {
|
|
||||||
selectResultInOneRound(results, oneRoundResults);
|
|
||||||
}
|
|
||||||
return null;
|
|
||||||
};
|
|
||||||
}
|
}
|
||||||
|
|
||||||
public abstract List<T> detectByStep(ChatQueryContext chatQueryContext,
|
public abstract List<T> detectByStep(ChatQueryContext chatQueryContext,
|
||||||
|
|||||||
@@ -17,6 +17,7 @@ import org.springframework.util.CollectionUtils;
|
|||||||
import java.util.*;
|
import java.util.*;
|
||||||
import java.util.stream.Collectors;
|
import java.util.stream.Collectors;
|
||||||
|
|
||||||
|
import static com.tencent.supersonic.common.pojo.DimensionConstants.*;
|
||||||
import static com.tencent.supersonic.headless.chat.parser.ParserConfig.*;
|
import static com.tencent.supersonic.headless.chat.parser.ParserConfig.*;
|
||||||
|
|
||||||
@Component
|
@Component
|
||||||
@@ -142,6 +143,10 @@ public class PromptHelper {
|
|||||||
dimension.getAlias().forEach(a -> alias.append(a).append(";"));
|
dimension.getAlias().forEach(a -> alias.append(a).append(";"));
|
||||||
dimensionStr.append(" ALIAS '").append(alias).append("'");
|
dimensionStr.append(" ALIAS '").append(alias).append("'");
|
||||||
}
|
}
|
||||||
|
if (Objects.nonNull(dimension.getExtInfo().get(DIMENSION_DATA_TYPE))) {
|
||||||
|
dimensionStr.append(" DATATYPE '")
|
||||||
|
.append(dimension.getExtInfo().get(DIMENSION_DATA_TYPE)).append("'");
|
||||||
|
}
|
||||||
if (StringUtils.isNotEmpty(dimension.getTimeFormat())) {
|
if (StringUtils.isNotEmpty(dimension.getTimeFormat())) {
|
||||||
dimensionStr.append(" FORMAT '").append(dimension.getTimeFormat()).append("'");
|
dimensionStr.append(" FORMAT '").append(dimension.getTimeFormat()).append("'");
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -32,7 +32,7 @@ public class TimeRangeParser implements SemanticParser {
|
|||||||
|
|
||||||
private static final Pattern RECENT_PATTERN_CN = Pattern.compile(
|
private static final Pattern RECENT_PATTERN_CN = Pattern.compile(
|
||||||
".*(?<periodStr>(近|过去)((?<enNum>\\d+)|(?<zhNum>[一二三四五六七八九十百千万亿]+))个?(?<zhPeriod>[天周月年])).*");
|
".*(?<periodStr>(近|过去)((?<enNum>\\d+)|(?<zhNum>[一二三四五六七八九十百千万亿]+))个?(?<zhPeriod>[天周月年])).*");
|
||||||
private static final Pattern DATE_PATTERN_NUMBER = Pattern.compile("(\\d{8})");
|
private static final Pattern DATE_PATTERN_NUMBER = Pattern.compile("\\b(\\d{8})\\b");
|
||||||
private static final DateFormat DATE_FORMAT_NUMBER = new SimpleDateFormat("yyyyMMdd");
|
private static final DateFormat DATE_FORMAT_NUMBER = new SimpleDateFormat("yyyyMMdd");
|
||||||
private static final DateFormat DATE_FORMAT = new SimpleDateFormat("yyyy-MM-dd");
|
private static final DateFormat DATE_FORMAT = new SimpleDateFormat("yyyy-MM-dd");
|
||||||
|
|
||||||
|
|||||||
@@ -47,24 +47,20 @@ public class SqlQueryParser implements QueryParser {
|
|||||||
SqlQuery sqlQuery = queryStatement.getSqlQuery();
|
SqlQuery sqlQuery = queryStatement.getSqlQuery();
|
||||||
List<String> queryFields = SqlSelectHelper.getAllSelectFields(sqlQuery.getSql());
|
List<String> queryFields = SqlSelectHelper.getAllSelectFields(sqlQuery.getSql());
|
||||||
Set<String> queryAliases = SqlSelectHelper.getAliasFields(sqlQuery.getSql());
|
Set<String> queryAliases = SqlSelectHelper.getAliasFields(sqlQuery.getSql());
|
||||||
Set<String> ontologyMetricsDimensions = Collections.synchronizedSet(new HashSet<String>());
|
List<Pair<String, String>> ontologyMetricsDimensionsAndBizName =
|
||||||
Set<String> ontologyBizNameMetricsDimensions = Collections.synchronizedSet(new HashSet<>());
|
Collections.synchronizedList(new ArrayList<>());
|
||||||
queryFields.removeAll(queryAliases);
|
queryFields.removeAll(queryAliases);
|
||||||
Ontology ontology = queryStatement.getOntology();
|
Ontology ontology = queryStatement.getOntology();
|
||||||
OntologyQuery ontologyQuery = buildOntologyQuery(ontology, queryFields);
|
OntologyQuery ontologyQuery = buildOntologyQuery(ontology, queryFields);
|
||||||
Set<String> queryFieldsSet = new HashSet<>(queryFields);
|
Set<String> queryFieldsSet = new HashSet<>(queryFields);
|
||||||
ontologyQuery.getMetrics().forEach(m -> {
|
ontologyQuery.getMetrics().forEach(m -> {
|
||||||
ontologyMetricsDimensions.add(m.getName());
|
ontologyMetricsDimensionsAndBizName.add(Pair.of(m.getName(), m.getBizName()));
|
||||||
ontologyBizNameMetricsDimensions.add(m.getBizName());
|
|
||||||
});
|
});
|
||||||
ontologyQuery.getDimensions().forEach(d -> {
|
ontologyQuery.getDimensions().forEach(d -> {
|
||||||
ontologyMetricsDimensions.add(d.getName());
|
ontologyMetricsDimensionsAndBizName.add(Pair.of(d.getName(), d.getBizName()));
|
||||||
ontologyBizNameMetricsDimensions.add(d.getBizName());
|
|
||||||
});
|
});
|
||||||
// check if there are fields not matched with any metric or dimension
|
// check if there are fields not matched with any metric or dimension
|
||||||
|
if (!allFieldMatched(queryFieldsSet, ontologyMetricsDimensionsAndBizName)) {
|
||||||
if (!(queryFieldsSet.containsAll(ontologyMetricsDimensions)
|
|
||||||
|| queryFieldsSet.containsAll(ontologyBizNameMetricsDimensions))) {
|
|
||||||
List<String> semanticFields = Lists.newArrayList();
|
List<String> semanticFields = Lists.newArrayList();
|
||||||
ontologyQuery.getMetrics().forEach(m -> semanticFields.add(m.getName()));
|
ontologyQuery.getMetrics().forEach(m -> semanticFields.add(m.getName()));
|
||||||
ontologyQuery.getDimensions().forEach(d -> semanticFields.add(d.getName()));
|
ontologyQuery.getDimensions().forEach(d -> semanticFields.add(d.getName()));
|
||||||
@@ -103,6 +99,16 @@ public class SqlQueryParser implements QueryParser {
|
|||||||
log.info("parse sqlQuery [{}] ", sqlQuery);
|
log.info("parse sqlQuery [{}] ", sqlQuery);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private boolean allFieldMatched(Set<String> queryFields,
|
||||||
|
List<Pair<String, String>> ontologyMetricsDimensionsAndBizName) {
|
||||||
|
for (Pair<String, String> pair : ontologyMetricsDimensionsAndBizName) {
|
||||||
|
if (!(queryFields.contains(pair.getLeft()) || queryFields.contains(pair.getRight()))) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
private void aliasesWithBackticks(QueryStatement queryStatement) {
|
private void aliasesWithBackticks(QueryStatement queryStatement) {
|
||||||
String sql = queryStatement.getSqlQuery().getSql();
|
String sql = queryStatement.getSqlQuery().getSql();
|
||||||
sql = SqlReplaceHelper.replaceAliasWithBackticks(sql);
|
sql = SqlReplaceHelper.replaceAliasWithBackticks(sql);
|
||||||
|
|||||||
@@ -46,8 +46,10 @@ public class ChatModelController {
|
|||||||
}
|
}
|
||||||
|
|
||||||
@DeleteMapping("/{id}")
|
@DeleteMapping("/{id}")
|
||||||
public boolean deleteModel(@PathVariable("id") Integer id) {
|
public boolean deleteModel(@PathVariable("id") Integer id,
|
||||||
chatModelService.deleteChatModel(id);
|
HttpServletRequest httpServletRequest, HttpServletResponse httpServletResponse) {
|
||||||
|
User user = UserHolder.findUser(httpServletRequest, httpServletResponse);
|
||||||
|
chatModelService.deleteChatModel(id, user);
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -64,8 +64,10 @@ public class DatabaseController {
|
|||||||
}
|
}
|
||||||
|
|
||||||
@DeleteMapping("/{id}")
|
@DeleteMapping("/{id}")
|
||||||
public boolean deleteDatabase(@PathVariable("id") Long id) {
|
public boolean deleteDatabase(@PathVariable("id") Long id, HttpServletRequest request,
|
||||||
databaseService.deleteDatabase(id);
|
HttpServletResponse response) {
|
||||||
|
User user = UserHolder.findUser(request, response);
|
||||||
|
databaseService.deleteDatabase(id, user);
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -34,7 +34,7 @@ public interface DatabaseService {
|
|||||||
|
|
||||||
List<DatabaseResp> getDatabaseList(User user);
|
List<DatabaseResp> getDatabaseList(User user);
|
||||||
|
|
||||||
void deleteDatabase(Long databaseId);
|
void deleteDatabase(Long databaseId, User user);
|
||||||
|
|
||||||
List<String> getCatalogs(Long id) throws SQLException;
|
List<String> getCatalogs(Long id) throws SQLException;
|
||||||
|
|
||||||
|
|||||||
@@ -52,7 +52,7 @@ public interface DimensionService {
|
|||||||
|
|
||||||
List<DimValueMap> mockDimensionValueAlias(DimensionReq dimensionReq, User user);
|
List<DimValueMap> mockDimensionValueAlias(DimensionReq dimensionReq, User user);
|
||||||
|
|
||||||
void sendDimensionEventBatch(List<Long> modelIds, EventType eventType);
|
void sendDimensionEventBatch(List<Long> modelIds, EventType eventType, User user);
|
||||||
|
|
||||||
DataEvent getAllDataEvents();
|
DataEvent getAllDataEvents();
|
||||||
|
|
||||||
|
|||||||
@@ -70,7 +70,7 @@ public interface MetricService {
|
|||||||
|
|
||||||
MetricQueryDefaultConfig getMetricQueryDefaultConfig(Long metricId, User user);
|
MetricQueryDefaultConfig getMetricQueryDefaultConfig(Long metricId, User user);
|
||||||
|
|
||||||
void sendMetricEventBatch(List<Long> modelIds, EventType eventType);
|
void sendMetricEventBatch(List<Long> modelIds, EventType eventType, User user);
|
||||||
|
|
||||||
List<MetricResp> queryMetrics(MetricsFilter metricsFilter);
|
List<MetricResp> queryMetrics(MetricsFilter metricsFilter);
|
||||||
|
|
||||||
|
|||||||
@@ -138,7 +138,12 @@ public class DatabaseServiceImpl extends ServiceImpl<DatabaseDOMapper, DatabaseD
|
|||||||
}
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public void deleteDatabase(Long databaseId) {
|
public void deleteDatabase(Long databaseId, User user) {
|
||||||
|
DatabaseResp databaseResp = getDatabase(databaseId);
|
||||||
|
if (!checkAdminPermission(user, databaseResp)) {
|
||||||
|
throw new RuntimeException("没有权限删除该数据库");
|
||||||
|
}
|
||||||
|
|
||||||
ModelFilter modelFilter = new ModelFilter();
|
ModelFilter modelFilter = new ModelFilter();
|
||||||
modelFilter.setDatabaseId(databaseId);
|
modelFilter.setDatabaseId(databaseId);
|
||||||
modelFilter.setIncludesDetail(false);
|
modelFilter.setIncludesDetail(false);
|
||||||
|
|||||||
@@ -48,7 +48,7 @@ public class DictWordService {
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
setPreDictWords(dictWords);
|
setPreDictWords(dictWords);
|
||||||
knowledgeBaseService.reloadAllData(getAllDictWords());
|
knowledgeBaseService.reloadAllData(dictWords);
|
||||||
long duration = System.currentTimeMillis() - startTime;
|
long duration = System.currentTimeMillis() - startTime;
|
||||||
log.info("Dictionary has been regularly reloaded in {} milliseconds", duration);
|
log.info("Dictionary has been regularly reloaded in {} milliseconds", duration);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -8,7 +8,10 @@ import com.github.pagehelper.PageHelper;
|
|||||||
import com.github.pagehelper.PageInfo;
|
import com.github.pagehelper.PageInfo;
|
||||||
import com.google.common.collect.ImmutableMap;
|
import com.google.common.collect.ImmutableMap;
|
||||||
import com.google.common.collect.Lists;
|
import com.google.common.collect.Lists;
|
||||||
import com.tencent.supersonic.common.pojo.*;
|
import com.tencent.supersonic.common.pojo.DataEvent;
|
||||||
|
import com.tencent.supersonic.common.pojo.DataItem;
|
||||||
|
import com.tencent.supersonic.common.pojo.ModelRela;
|
||||||
|
import com.tencent.supersonic.common.pojo.User;
|
||||||
import com.tencent.supersonic.common.pojo.enums.EventType;
|
import com.tencent.supersonic.common.pojo.enums.EventType;
|
||||||
import com.tencent.supersonic.common.pojo.enums.StatusEnum;
|
import com.tencent.supersonic.common.pojo.enums.StatusEnum;
|
||||||
import com.tencent.supersonic.common.pojo.enums.TypeEnums;
|
import com.tencent.supersonic.common.pojo.enums.TypeEnums;
|
||||||
@@ -22,14 +25,22 @@ import com.tencent.supersonic.headless.api.pojo.request.DimValueAliasReq;
|
|||||||
import com.tencent.supersonic.headless.api.pojo.request.DimensionReq;
|
import com.tencent.supersonic.headless.api.pojo.request.DimensionReq;
|
||||||
import com.tencent.supersonic.headless.api.pojo.request.MetaBatchReq;
|
import com.tencent.supersonic.headless.api.pojo.request.MetaBatchReq;
|
||||||
import com.tencent.supersonic.headless.api.pojo.request.PageDimensionReq;
|
import com.tencent.supersonic.headless.api.pojo.request.PageDimensionReq;
|
||||||
import com.tencent.supersonic.headless.api.pojo.response.*;
|
import com.tencent.supersonic.headless.api.pojo.response.DataSetResp;
|
||||||
|
import com.tencent.supersonic.headless.api.pojo.response.DatabaseResp;
|
||||||
|
import com.tencent.supersonic.headless.api.pojo.response.DimensionResp;
|
||||||
|
import com.tencent.supersonic.headless.api.pojo.response.ModelResp;
|
||||||
|
import com.tencent.supersonic.headless.api.pojo.response.SemanticQueryResp;
|
||||||
import com.tencent.supersonic.headless.server.persistence.dataobject.DimensionDO;
|
import com.tencent.supersonic.headless.server.persistence.dataobject.DimensionDO;
|
||||||
import com.tencent.supersonic.headless.server.persistence.mapper.DimensionDOMapper;
|
import com.tencent.supersonic.headless.server.persistence.mapper.DimensionDOMapper;
|
||||||
import com.tencent.supersonic.headless.server.persistence.repository.DimensionRepository;
|
import com.tencent.supersonic.headless.server.persistence.repository.DimensionRepository;
|
||||||
import com.tencent.supersonic.headless.server.pojo.DimensionFilter;
|
import com.tencent.supersonic.headless.server.pojo.DimensionFilter;
|
||||||
import com.tencent.supersonic.headless.server.pojo.DimensionsFilter;
|
import com.tencent.supersonic.headless.server.pojo.DimensionsFilter;
|
||||||
import com.tencent.supersonic.headless.server.pojo.ModelFilter;
|
import com.tencent.supersonic.headless.server.pojo.ModelFilter;
|
||||||
import com.tencent.supersonic.headless.server.service.*;
|
import com.tencent.supersonic.headless.server.service.DataSetService;
|
||||||
|
import com.tencent.supersonic.headless.server.service.DatabaseService;
|
||||||
|
import com.tencent.supersonic.headless.server.service.DimensionService;
|
||||||
|
import com.tencent.supersonic.headless.server.service.ModelRelaService;
|
||||||
|
import com.tencent.supersonic.headless.server.service.ModelService;
|
||||||
import com.tencent.supersonic.headless.server.utils.AliasGenerateHelper;
|
import com.tencent.supersonic.headless.server.utils.AliasGenerateHelper;
|
||||||
import com.tencent.supersonic.headless.server.utils.DimensionConverter;
|
import com.tencent.supersonic.headless.server.utils.DimensionConverter;
|
||||||
import com.tencent.supersonic.headless.server.utils.NameCheckUtils;
|
import com.tencent.supersonic.headless.server.utils.NameCheckUtils;
|
||||||
@@ -37,12 +48,16 @@ import lombok.extern.slf4j.Slf4j;
|
|||||||
import org.apache.commons.lang3.StringUtils;
|
import org.apache.commons.lang3.StringUtils;
|
||||||
import org.apache.commons.lang3.builder.EqualsBuilder;
|
import org.apache.commons.lang3.builder.EqualsBuilder;
|
||||||
import org.springframework.beans.BeanUtils;
|
import org.springframework.beans.BeanUtils;
|
||||||
import org.springframework.beans.factory.annotation.Autowired;
|
|
||||||
import org.springframework.context.ApplicationEventPublisher;
|
import org.springframework.context.ApplicationEventPublisher;
|
||||||
import org.springframework.stereotype.Service;
|
import org.springframework.stereotype.Service;
|
||||||
import org.springframework.util.CollectionUtils;
|
import org.springframework.util.CollectionUtils;
|
||||||
|
|
||||||
import java.util.*;
|
import java.util.ArrayList;
|
||||||
|
import java.util.Date;
|
||||||
|
import java.util.HashMap;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Map;
|
||||||
|
import java.util.Objects;
|
||||||
import java.util.stream.Collectors;
|
import java.util.stream.Collectors;
|
||||||
|
|
||||||
@Service
|
@Service
|
||||||
@@ -50,31 +65,31 @@ import java.util.stream.Collectors;
|
|||||||
public class DimensionServiceImpl extends ServiceImpl<DimensionDOMapper, DimensionDO>
|
public class DimensionServiceImpl extends ServiceImpl<DimensionDOMapper, DimensionDO>
|
||||||
implements DimensionService {
|
implements DimensionService {
|
||||||
|
|
||||||
private DimensionRepository dimensionRepository;
|
private final DimensionRepository dimensionRepository;
|
||||||
|
|
||||||
private ModelService modelService;
|
private final ModelService modelService;
|
||||||
|
|
||||||
private AliasGenerateHelper aliasGenerateHelper;
|
private final AliasGenerateHelper aliasGenerateHelper;
|
||||||
|
|
||||||
private DatabaseService databaseService;
|
private final DatabaseService databaseService;
|
||||||
|
|
||||||
private ModelRelaService modelRelaService;
|
private final ModelRelaService modelRelaService;
|
||||||
|
|
||||||
private DataSetService dataSetService;
|
private final DataSetService dataSetService;
|
||||||
|
|
||||||
|
private final ApplicationEventPublisher eventPublisher;
|
||||||
@Autowired
|
|
||||||
private ApplicationEventPublisher eventPublisher;
|
|
||||||
|
|
||||||
public DimensionServiceImpl(DimensionRepository dimensionRepository, ModelService modelService,
|
public DimensionServiceImpl(DimensionRepository dimensionRepository, ModelService modelService,
|
||||||
AliasGenerateHelper aliasGenerateHelper, DatabaseService databaseService,
|
AliasGenerateHelper aliasGenerateHelper, DatabaseService databaseService,
|
||||||
ModelRelaService modelRelaService, DataSetService dataSetService) {
|
ModelRelaService modelRelaService, DataSetService dataSetService,
|
||||||
|
ApplicationEventPublisher eventPublisher) {
|
||||||
this.modelService = modelService;
|
this.modelService = modelService;
|
||||||
this.dimensionRepository = dimensionRepository;
|
this.dimensionRepository = dimensionRepository;
|
||||||
this.aliasGenerateHelper = aliasGenerateHelper;
|
this.aliasGenerateHelper = aliasGenerateHelper;
|
||||||
this.databaseService = databaseService;
|
this.databaseService = databaseService;
|
||||||
this.modelRelaService = modelRelaService;
|
this.modelRelaService = modelRelaService;
|
||||||
this.dataSetService = dataSetService;
|
this.dataSetService = dataSetService;
|
||||||
|
this.eventPublisher = eventPublisher;
|
||||||
}
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
@@ -83,7 +98,7 @@ public class DimensionServiceImpl extends ServiceImpl<DimensionDOMapper, Dimensi
|
|||||||
dimensionReq.createdBy(user.getName());
|
dimensionReq.createdBy(user.getName());
|
||||||
DimensionDO dimensionDO = DimensionConverter.convert2DimensionDO(dimensionReq);
|
DimensionDO dimensionDO = DimensionConverter.convert2DimensionDO(dimensionReq);
|
||||||
dimensionRepository.createDimension(dimensionDO);
|
dimensionRepository.createDimension(dimensionDO);
|
||||||
sendEventBatch(Lists.newArrayList(dimensionDO), EventType.ADD);
|
sendEventBatch(Lists.newArrayList(dimensionDO), EventType.ADD, user);
|
||||||
|
|
||||||
// should update modelDetail
|
// should update modelDetail
|
||||||
modelService.updateModelByDimAndMetric(dimensionReq.getModelId(),
|
modelService.updateModelByDimAndMetric(dimensionReq.getModelId(),
|
||||||
@@ -101,29 +116,11 @@ public class DimensionServiceImpl extends ServiceImpl<DimensionDOMapper, Dimensi
|
|||||||
.collect(Collectors.toMap(DimensionResp::getBizName, a -> a, (k1, k2) -> k1));
|
.collect(Collectors.toMap(DimensionResp::getBizName, a -> a, (k1, k2) -> k1));
|
||||||
|
|
||||||
List<DimensionReq> dimensionToInsert = Lists.newArrayList();
|
List<DimensionReq> dimensionToInsert = Lists.newArrayList();
|
||||||
List<DimensionReq> dimensionToUpdate = Lists.newArrayList();
|
|
||||||
List<Long> dimensionToDelete = Lists.newArrayList();
|
|
||||||
|
|
||||||
// look for which dimension need to insert, update, delete
|
// look for which dimension need to insert, update, delete
|
||||||
dimensionReqs.stream().forEach(dimension -> {
|
dimensionReqs.forEach(dimension -> {
|
||||||
if (!bizNameMap.containsKey(dimension.getBizName())) {
|
if (!bizNameMap.containsKey(dimension.getBizName())) {
|
||||||
dimensionToInsert.add(dimension);
|
dimensionToInsert.add(dimension);
|
||||||
} else {
|
|
||||||
DimensionResp dimensionRespByBizName = bizNameMap.get(dimension.getBizName());
|
|
||||||
if (null != dimensionRespByBizName && isChange(dimension, dimensionRespByBizName)) {
|
|
||||||
dimension.setId(dimensionRespByBizName.getId());
|
|
||||||
dimension.updatedBy(user.getName());
|
|
||||||
dimensionToUpdate.add(dimension);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
});
|
|
||||||
|
|
||||||
// the bizNames from alter dimensions
|
|
||||||
List<String> bizNames =
|
|
||||||
dimensionReqs.stream().map(DimensionReq::getBizName).collect(Collectors.toList());
|
|
||||||
bizNameMap.keySet().forEach(bizNameInDb -> {
|
|
||||||
if (!bizNames.contains(bizNameInDb)) {
|
|
||||||
dimensionToDelete.add(bizNameMap.get(bizNameInDb).getId());
|
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -132,16 +129,6 @@ public class DimensionServiceImpl extends ServiceImpl<DimensionDOMapper, Dimensi
|
|||||||
createDimensionBatch(dimensionToInsert, user);
|
createDimensionBatch(dimensionToInsert, user);
|
||||||
}
|
}
|
||||||
|
|
||||||
// update
|
|
||||||
if (!CollectionUtils.isEmpty(dimensionToUpdate)) {
|
|
||||||
updateDimensionBatch(dimensionToUpdate, user);
|
|
||||||
}
|
|
||||||
|
|
||||||
// delete
|
|
||||||
if (!CollectionUtils.isEmpty(dimensionToDelete)) {
|
|
||||||
deleteDimensionBatch(dimensionToDelete, user);
|
|
||||||
}
|
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
@@ -154,7 +141,7 @@ public class DimensionServiceImpl extends ServiceImpl<DimensionDOMapper, Dimensi
|
|||||||
modelService.updateModelByDimAndMetric(dimensionReqs.get(0).getModelId(), dimensionReqs,
|
modelService.updateModelByDimAndMetric(dimensionReqs.get(0).getModelId(), dimensionReqs,
|
||||||
null, user);
|
null, user);
|
||||||
|
|
||||||
sendEventBatch(dimensionDOS, EventType.ADD);
|
sendEventBatch(dimensionDOS, EventType.ADD, user);
|
||||||
}
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
@@ -170,7 +157,7 @@ public class DimensionServiceImpl extends ServiceImpl<DimensionDOMapper, Dimensi
|
|||||||
Lists.newArrayList(dimensionReq), null, user);
|
Lists.newArrayList(dimensionReq), null, user);
|
||||||
|
|
||||||
if (!oldName.equals(dimensionDO.getName())) {
|
if (!oldName.equals(dimensionDO.getName())) {
|
||||||
sendEvent(getDataItem(dimensionDO), EventType.UPDATE);
|
sendEvent(getDataItem(dimensionDO), EventType.UPDATE, user.getName());
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -183,7 +170,7 @@ public class DimensionServiceImpl extends ServiceImpl<DimensionDOMapper, Dimensi
|
|||||||
// should update modelDetail as well
|
// should update modelDetail as well
|
||||||
modelService.updateModelByDimAndMetric(dimensionReqList.get(0).getModelId(),
|
modelService.updateModelByDimAndMetric(dimensionReqList.get(0).getModelId(),
|
||||||
dimensionReqList, null, user);
|
dimensionReqList, null, user);
|
||||||
sendEventBatch(dimensionDOS, EventType.UPDATE);
|
sendEventBatch(dimensionDOS, EventType.UPDATE, user);
|
||||||
}
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
@@ -205,9 +192,9 @@ public class DimensionServiceImpl extends ServiceImpl<DimensionDOMapper, Dimensi
|
|||||||
dimensionRepository.batchUpdateStatus(dimensionDOS);
|
dimensionRepository.batchUpdateStatus(dimensionDOS);
|
||||||
if (StatusEnum.OFFLINE.getCode().equals(metaBatchReq.getStatus())
|
if (StatusEnum.OFFLINE.getCode().equals(metaBatchReq.getStatus())
|
||||||
|| StatusEnum.DELETED.getCode().equals(metaBatchReq.getStatus())) {
|
|| StatusEnum.DELETED.getCode().equals(metaBatchReq.getStatus())) {
|
||||||
sendEventBatch(dimensionDOS, EventType.DELETE);
|
sendEventBatch(dimensionDOS, EventType.DELETE, user);
|
||||||
} else if (StatusEnum.ONLINE.getCode().equals(metaBatchReq.getStatus())) {
|
} else if (StatusEnum.ONLINE.getCode().equals(metaBatchReq.getStatus())) {
|
||||||
sendEventBatch(dimensionDOS, EventType.ADD);
|
sendEventBatch(dimensionDOS, EventType.ADD, user);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -240,7 +227,7 @@ public class DimensionServiceImpl extends ServiceImpl<DimensionDOMapper, Dimensi
|
|||||||
// should update modelDetail
|
// should update modelDetail
|
||||||
modelService.deleteModelDetailByDimAndMetric(dimensionDO.getModelId(),
|
modelService.deleteModelDetailByDimAndMetric(dimensionDO.getModelId(),
|
||||||
Lists.newArrayList(dimensionDO), null);
|
Lists.newArrayList(dimensionDO), null);
|
||||||
sendEventBatch(Lists.newArrayList(dimensionDO), EventType.DELETE);
|
sendEventBatch(Lists.newArrayList(dimensionDO), EventType.DELETE, user);
|
||||||
}
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
@@ -261,7 +248,7 @@ public class DimensionServiceImpl extends ServiceImpl<DimensionDOMapper, Dimensi
|
|||||||
// should update modelDetail
|
// should update modelDetail
|
||||||
modelService.deleteModelDetailByDimAndMetric(dimensionDOList.get(0).getModelId(),
|
modelService.deleteModelDetailByDimAndMetric(dimensionDOList.get(0).getModelId(),
|
||||||
dimensionDOList, null);
|
dimensionDOList, null);
|
||||||
sendEventBatch(dimensionDOList, EventType.DELETE);
|
sendEventBatch(dimensionDOList, EventType.DELETE, user);
|
||||||
}
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
@@ -462,22 +449,22 @@ public class DimensionServiceImpl extends ServiceImpl<DimensionDOMapper, Dimensi
|
|||||||
}
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public void sendDimensionEventBatch(List<Long> modelIds, EventType eventType) {
|
public void sendDimensionEventBatch(List<Long> modelIds, EventType eventType, User user) {
|
||||||
DimensionFilter dimensionFilter = new DimensionFilter();
|
DimensionFilter dimensionFilter = new DimensionFilter();
|
||||||
dimensionFilter.setModelIds(modelIds);
|
dimensionFilter.setModelIds(modelIds);
|
||||||
List<DimensionDO> dimensionDOS = queryDimension(dimensionFilter);
|
List<DimensionDO> dimensionDOS = queryDimension(dimensionFilter);
|
||||||
sendEventBatch(dimensionDOS, eventType);
|
sendEventBatch(dimensionDOS, eventType, user);
|
||||||
}
|
}
|
||||||
|
|
||||||
private void sendEventBatch(List<DimensionDO> dimensionDOS, EventType eventType) {
|
private void sendEventBatch(List<DimensionDO> dimensionDOS, EventType eventType, User user) {
|
||||||
DataEvent dataEvent = getDataEvent(dimensionDOS, eventType);
|
DataEvent dataEvent = getDataEvent(dimensionDOS, eventType, user.getName());
|
||||||
eventPublisher.publishEvent(dataEvent);
|
eventPublisher.publishEvent(dataEvent);
|
||||||
}
|
}
|
||||||
|
|
||||||
public DataEvent getAllDataEvents() {
|
public DataEvent getAllDataEvents() {
|
||||||
DimensionFilter dimensionFilter = new DimensionFilter();
|
DimensionFilter dimensionFilter = new DimensionFilter();
|
||||||
List<DimensionDO> dimensionDOS = queryDimension(dimensionFilter);
|
List<DimensionDO> dimensionDOS = queryDimension(dimensionFilter);
|
||||||
return getDataEvent(dimensionDOS, EventType.ADD);
|
return getDataEvent(dimensionDOS, EventType.ADD, "");
|
||||||
}
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
@@ -519,6 +506,9 @@ public class DimensionServiceImpl extends ServiceImpl<DimensionDOMapper, Dimensi
|
|||||||
|
|
||||||
private DataItem getDataItem(DimensionDO dimensionDO) {
|
private DataItem getDataItem(DimensionDO dimensionDO) {
|
||||||
ModelResp modelResp = modelService.getModel(dimensionDO.getModelId());
|
ModelResp modelResp = modelService.getModel(dimensionDO.getModelId());
|
||||||
|
if (modelResp == null) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
DimensionResp dimensionResp = DimensionConverter.convert2DimensionResp(dimensionDO,
|
DimensionResp dimensionResp = DimensionConverter.convert2DimensionResp(dimensionDO,
|
||||||
ImmutableMap.of(modelResp.getId(), modelResp));
|
ImmutableMap.of(modelResp.getId(), modelResp));
|
||||||
return DataItem.builder().id(dimensionResp.getId().toString()).name(dimensionResp.getName())
|
return DataItem.builder().id(dimensionResp.getId().toString()).name(dimensionResp.getName())
|
||||||
@@ -526,14 +516,16 @@ public class DimensionServiceImpl extends ServiceImpl<DimensionDOMapper, Dimensi
|
|||||||
.domainId(dimensionResp.getDomainId().toString()).type(TypeEnums.DIMENSION).build();
|
.domainId(dimensionResp.getDomainId().toString()).type(TypeEnums.DIMENSION).build();
|
||||||
}
|
}
|
||||||
|
|
||||||
private DataEvent getDataEvent(List<DimensionDO> dimensionDOS, EventType eventType) {
|
private DataEvent getDataEvent(List<DimensionDO> dimensionDOS, EventType eventType,
|
||||||
List<DataItem> dataItems =
|
String userName) {
|
||||||
dimensionDOS.stream().map(this::getDataItem).collect(Collectors.toList());
|
List<DataItem> dataItems = dimensionDOS.stream().map(this::getDataItem)
|
||||||
return new DataEvent(this, dataItems, eventType);
|
.filter(Objects::nonNull).collect(Collectors.toList());
|
||||||
|
return new DataEvent(this, dataItems, eventType, userName);
|
||||||
}
|
}
|
||||||
|
|
||||||
private void sendEvent(DataItem dataItem, EventType eventType) {
|
private void sendEvent(DataItem dataItem, EventType eventType, String userName) {
|
||||||
eventPublisher.publishEvent(new DataEvent(this, Lists.newArrayList(dataItem), eventType));
|
eventPublisher.publishEvent(
|
||||||
|
new DataEvent(this, Lists.newArrayList(dataItem), eventType, userName));
|
||||||
}
|
}
|
||||||
|
|
||||||
private boolean isChange(DimensionReq dimensionReq, DimensionResp dimensionResp) {
|
private boolean isChange(DimensionReq dimensionReq, DimensionResp dimensionResp) {
|
||||||
|
|||||||
@@ -8,23 +8,61 @@ import com.github.pagehelper.PageInfo;
|
|||||||
import com.google.common.collect.ImmutableMap;
|
import com.google.common.collect.ImmutableMap;
|
||||||
import com.google.common.collect.Lists;
|
import com.google.common.collect.Lists;
|
||||||
import com.google.common.collect.Sets;
|
import com.google.common.collect.Sets;
|
||||||
import com.tencent.supersonic.common.pojo.*;
|
import com.tencent.supersonic.common.pojo.Aggregator;
|
||||||
import com.tencent.supersonic.common.pojo.enums.*;
|
import com.tencent.supersonic.common.pojo.DataEvent;
|
||||||
|
import com.tencent.supersonic.common.pojo.DataItem;
|
||||||
|
import com.tencent.supersonic.common.pojo.DateConf;
|
||||||
|
import com.tencent.supersonic.common.pojo.Filter;
|
||||||
|
import com.tencent.supersonic.common.pojo.User;
|
||||||
|
import com.tencent.supersonic.common.pojo.enums.AuthType;
|
||||||
|
import com.tencent.supersonic.common.pojo.enums.EventType;
|
||||||
|
import com.tencent.supersonic.common.pojo.enums.QueryType;
|
||||||
|
import com.tencent.supersonic.common.pojo.enums.StatusEnum;
|
||||||
|
import com.tencent.supersonic.common.pojo.enums.TypeEnums;
|
||||||
import com.tencent.supersonic.common.util.BeanMapper;
|
import com.tencent.supersonic.common.util.BeanMapper;
|
||||||
import com.tencent.supersonic.common.util.JsonUtil;
|
import com.tencent.supersonic.headless.api.pojo.DrillDownDimension;
|
||||||
import com.tencent.supersonic.headless.api.pojo.*;
|
import com.tencent.supersonic.headless.api.pojo.Measure;
|
||||||
import com.tencent.supersonic.headless.api.pojo.enums.DimensionType;
|
import com.tencent.supersonic.headless.api.pojo.MetaFilter;
|
||||||
|
import com.tencent.supersonic.headless.api.pojo.MetricParam;
|
||||||
|
import com.tencent.supersonic.headless.api.pojo.MetricQueryDefaultConfig;
|
||||||
|
import com.tencent.supersonic.headless.api.pojo.SchemaElementMatch;
|
||||||
|
import com.tencent.supersonic.headless.api.pojo.SchemaElementType;
|
||||||
|
import com.tencent.supersonic.headless.api.pojo.SchemaItem;
|
||||||
import com.tencent.supersonic.headless.api.pojo.enums.MapModeEnum;
|
import com.tencent.supersonic.headless.api.pojo.enums.MapModeEnum;
|
||||||
import com.tencent.supersonic.headless.api.pojo.enums.MetricDefineType;
|
import com.tencent.supersonic.headless.api.pojo.enums.MetricDefineType;
|
||||||
import com.tencent.supersonic.headless.api.pojo.request.*;
|
import com.tencent.supersonic.headless.api.pojo.request.MetaBatchReq;
|
||||||
import com.tencent.supersonic.headless.api.pojo.response.*;
|
import com.tencent.supersonic.headless.api.pojo.request.MetricBaseReq;
|
||||||
|
import com.tencent.supersonic.headless.api.pojo.request.MetricReq;
|
||||||
|
import com.tencent.supersonic.headless.api.pojo.request.PageMetricReq;
|
||||||
|
import com.tencent.supersonic.headless.api.pojo.request.QueryMapReq;
|
||||||
|
import com.tencent.supersonic.headless.api.pojo.request.QueryMetricReq;
|
||||||
|
import com.tencent.supersonic.headless.api.pojo.request.QueryStructReq;
|
||||||
|
import com.tencent.supersonic.headless.api.pojo.response.DataSetMapInfo;
|
||||||
|
import com.tencent.supersonic.headless.api.pojo.response.DataSetResp;
|
||||||
|
import com.tencent.supersonic.headless.api.pojo.response.DimensionResp;
|
||||||
|
import com.tencent.supersonic.headless.api.pojo.response.MapInfoResp;
|
||||||
|
import com.tencent.supersonic.headless.api.pojo.response.MetricResp;
|
||||||
|
import com.tencent.supersonic.headless.api.pojo.response.ModelResp;
|
||||||
import com.tencent.supersonic.headless.server.facade.service.ChatLayerService;
|
import com.tencent.supersonic.headless.server.facade.service.ChatLayerService;
|
||||||
import com.tencent.supersonic.headless.server.persistence.dataobject.*;
|
import com.tencent.supersonic.headless.server.persistence.dataobject.CollectDO;
|
||||||
|
import com.tencent.supersonic.headless.server.persistence.dataobject.MetricDO;
|
||||||
|
import com.tencent.supersonic.headless.server.persistence.dataobject.MetricQueryDefaultConfigDO;
|
||||||
import com.tencent.supersonic.headless.server.persistence.mapper.MetricDOMapper;
|
import com.tencent.supersonic.headless.server.persistence.mapper.MetricDOMapper;
|
||||||
import com.tencent.supersonic.headless.server.persistence.repository.MetricRepository;
|
import com.tencent.supersonic.headless.server.persistence.repository.MetricRepository;
|
||||||
import com.tencent.supersonic.headless.server.pojo.*;
|
import com.tencent.supersonic.headless.server.pojo.DimensionsFilter;
|
||||||
import com.tencent.supersonic.headless.server.service.*;
|
import com.tencent.supersonic.headless.server.pojo.MetricFilter;
|
||||||
import com.tencent.supersonic.headless.server.utils.*;
|
import com.tencent.supersonic.headless.server.pojo.MetricsFilter;
|
||||||
|
import com.tencent.supersonic.headless.server.pojo.ModelCluster;
|
||||||
|
import com.tencent.supersonic.headless.server.pojo.ModelFilter;
|
||||||
|
import com.tencent.supersonic.headless.server.service.CollectService;
|
||||||
|
import com.tencent.supersonic.headless.server.service.DataSetService;
|
||||||
|
import com.tencent.supersonic.headless.server.service.DimensionService;
|
||||||
|
import com.tencent.supersonic.headless.server.service.MetricService;
|
||||||
|
import com.tencent.supersonic.headless.server.service.ModelService;
|
||||||
|
import com.tencent.supersonic.headless.server.utils.AliasGenerateHelper;
|
||||||
|
import com.tencent.supersonic.headless.server.utils.MetricCheckUtils;
|
||||||
|
import com.tencent.supersonic.headless.server.utils.MetricConverter;
|
||||||
|
import com.tencent.supersonic.headless.server.utils.ModelClusterBuilder;
|
||||||
import lombok.extern.slf4j.Slf4j;
|
import lombok.extern.slf4j.Slf4j;
|
||||||
import org.apache.commons.lang3.StringUtils;
|
import org.apache.commons.lang3.StringUtils;
|
||||||
import org.springframework.beans.BeanUtils;
|
import org.springframework.beans.BeanUtils;
|
||||||
@@ -33,7 +71,18 @@ import org.springframework.context.annotation.Lazy;
|
|||||||
import org.springframework.stereotype.Service;
|
import org.springframework.stereotype.Service;
|
||||||
import org.springframework.util.CollectionUtils;
|
import org.springframework.util.CollectionUtils;
|
||||||
|
|
||||||
import java.util.*;
|
import java.util.ArrayList;
|
||||||
|
import java.util.Arrays;
|
||||||
|
import java.util.Collection;
|
||||||
|
import java.util.Collections;
|
||||||
|
import java.util.Comparator;
|
||||||
|
import java.util.Date;
|
||||||
|
import java.util.HashMap;
|
||||||
|
import java.util.HashSet;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Map;
|
||||||
|
import java.util.Objects;
|
||||||
|
import java.util.Set;
|
||||||
import java.util.stream.Collectors;
|
import java.util.stream.Collectors;
|
||||||
|
|
||||||
@Service
|
@Service
|
||||||
@@ -41,21 +90,21 @@ import java.util.stream.Collectors;
|
|||||||
public class MetricServiceImpl extends ServiceImpl<MetricDOMapper, MetricDO>
|
public class MetricServiceImpl extends ServiceImpl<MetricDOMapper, MetricDO>
|
||||||
implements MetricService {
|
implements MetricService {
|
||||||
|
|
||||||
private MetricRepository metricRepository;
|
private final MetricRepository metricRepository;
|
||||||
|
|
||||||
private ModelService modelService;
|
private final ModelService modelService;
|
||||||
|
|
||||||
private DimensionService dimensionService;
|
private final DimensionService dimensionService;
|
||||||
|
|
||||||
private AliasGenerateHelper aliasGenerateHelper;
|
private final AliasGenerateHelper aliasGenerateHelper;
|
||||||
|
|
||||||
private CollectService collectService;
|
private final CollectService collectService;
|
||||||
|
|
||||||
private DataSetService dataSetService;
|
private final DataSetService dataSetService;
|
||||||
|
|
||||||
private ApplicationEventPublisher eventPublisher;
|
private final ApplicationEventPublisher eventPublisher;
|
||||||
|
|
||||||
private ChatLayerService chatLayerService;
|
private final ChatLayerService chatLayerService;
|
||||||
|
|
||||||
public MetricServiceImpl(MetricRepository metricRepository, ModelService modelService,
|
public MetricServiceImpl(MetricRepository metricRepository, ModelService modelService,
|
||||||
AliasGenerateHelper aliasGenerateHelper, CollectService collectService,
|
AliasGenerateHelper aliasGenerateHelper, CollectService collectService,
|
||||||
@@ -78,7 +127,7 @@ public class MetricServiceImpl extends ServiceImpl<MetricDOMapper, MetricDO>
|
|||||||
metricReq.createdBy(user.getName());
|
metricReq.createdBy(user.getName());
|
||||||
MetricDO metricDO = MetricConverter.convert2MetricDO(metricReq);
|
MetricDO metricDO = MetricConverter.convert2MetricDO(metricReq);
|
||||||
metricRepository.createMetric(metricDO);
|
metricRepository.createMetric(metricDO);
|
||||||
sendEventBatch(Lists.newArrayList(metricDO), EventType.ADD);
|
sendEventBatch(Lists.newArrayList(metricDO), EventType.ADD, user);
|
||||||
// should update modelDetail as well
|
// should update modelDetail as well
|
||||||
modelService.updateModelByDimAndMetric(metricReq.getModelId(), null,
|
modelService.updateModelByDimAndMetric(metricReq.getModelId(), null,
|
||||||
Lists.newArrayList(metricReq), user);
|
Lists.newArrayList(metricReq), user);
|
||||||
@@ -97,7 +146,7 @@ public class MetricServiceImpl extends ServiceImpl<MetricDOMapper, MetricDO>
|
|||||||
modelService.updateModelByDimAndMetric(metricReqs.get(0).getModelId(), null, metricReqs,
|
modelService.updateModelByDimAndMetric(metricReqs.get(0).getModelId(), null, metricReqs,
|
||||||
user);
|
user);
|
||||||
|
|
||||||
sendEventBatch(metricDOS, EventType.ADD);
|
sendEventBatch(metricDOS, EventType.ADD, user);
|
||||||
}
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
@@ -108,28 +157,9 @@ public class MetricServiceImpl extends ServiceImpl<MetricDOMapper, MetricDO>
|
|||||||
.collect(Collectors.toMap(MetricResp::getBizName, a -> a, (k1, k2) -> k1));
|
.collect(Collectors.toMap(MetricResp::getBizName, a -> a, (k1, k2) -> k1));
|
||||||
|
|
||||||
List<MetricReq> metricToInsert = Lists.newArrayList();
|
List<MetricReq> metricToInsert = Lists.newArrayList();
|
||||||
List<MetricReq> metricToUpdate = Lists.newArrayList();
|
metricReqs.forEach(metric -> {
|
||||||
List<Long> metricToDelete = Lists.newArrayList();
|
|
||||||
|
|
||||||
metricReqs.stream().forEach(metric -> {
|
|
||||||
if (!bizNameMap.containsKey(metric.getBizName())) {
|
if (!bizNameMap.containsKey(metric.getBizName())) {
|
||||||
metricToInsert.add(metric);
|
metricToInsert.add(metric);
|
||||||
} else {
|
|
||||||
MetricResp metricRespByBizName = bizNameMap.get(metric.getBizName());
|
|
||||||
if (null != metricRespByBizName && isChange(metric, metricRespByBizName)) {
|
|
||||||
metric.setId(metricRespByBizName.getId());
|
|
||||||
metric.updatedBy(user.getName());
|
|
||||||
metricToUpdate.add(metric);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
});
|
|
||||||
|
|
||||||
// the bizNames from alter dimensions
|
|
||||||
List<String> bizNames =
|
|
||||||
metricReqs.stream().map(MetricReq::getBizName).collect(Collectors.toList());
|
|
||||||
bizNameMap.keySet().forEach(bizNameInDb -> {
|
|
||||||
if (!bizNames.contains(bizNameInDb)) {
|
|
||||||
metricToDelete.add(bizNameMap.get(bizNameInDb).getId());
|
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -138,16 +168,6 @@ public class MetricServiceImpl extends ServiceImpl<MetricDOMapper, MetricDO>
|
|||||||
createMetricBatch(metricToInsert, user);
|
createMetricBatch(metricToInsert, user);
|
||||||
}
|
}
|
||||||
|
|
||||||
// update
|
|
||||||
if (!CollectionUtils.isEmpty(metricToUpdate)) {
|
|
||||||
updateMetricBatch(metricToUpdate, user);
|
|
||||||
}
|
|
||||||
|
|
||||||
// delete
|
|
||||||
if (!CollectionUtils.isEmpty(metricToDelete)) {
|
|
||||||
deleteMetricBatch(metricToDelete, user);
|
|
||||||
}
|
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
@@ -163,7 +183,7 @@ public class MetricServiceImpl extends ServiceImpl<MetricDOMapper, MetricDO>
|
|||||||
DataItem dataItem = getDataItem(metricDO);
|
DataItem dataItem = getDataItem(metricDO);
|
||||||
dataItem.setName(oldName);
|
dataItem.setName(oldName);
|
||||||
dataItem.setNewName(metricDO.getName());
|
dataItem.setNewName(metricDO.getName());
|
||||||
sendEvent(dataItem, EventType.UPDATE);
|
sendEvent(dataItem, EventType.UPDATE, user);
|
||||||
}
|
}
|
||||||
// should update modelDetail as well
|
// should update modelDetail as well
|
||||||
modelService.updateModelByDimAndMetric(metricReq.getModelId(), null,
|
modelService.updateModelByDimAndMetric(metricReq.getModelId(), null,
|
||||||
@@ -181,7 +201,7 @@ public class MetricServiceImpl extends ServiceImpl<MetricDOMapper, MetricDO>
|
|||||||
// should update modelDetail as well
|
// should update modelDetail as well
|
||||||
modelService.updateModelByDimAndMetric(metricReqs.get(0).getModelId(), null, metricReqs,
|
modelService.updateModelByDimAndMetric(metricReqs.get(0).getModelId(), null, metricReqs,
|
||||||
user);
|
user);
|
||||||
sendEventBatch(metricDOS, EventType.UPDATE);
|
sendEventBatch(metricDOS, EventType.UPDATE, user);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@@ -202,9 +222,9 @@ public class MetricServiceImpl extends ServiceImpl<MetricDOMapper, MetricDO>
|
|||||||
metricRepository.batchUpdateStatus(metricDOS);
|
metricRepository.batchUpdateStatus(metricDOS);
|
||||||
if (StatusEnum.OFFLINE.getCode().equals(metaBatchReq.getStatus())
|
if (StatusEnum.OFFLINE.getCode().equals(metaBatchReq.getStatus())
|
||||||
|| StatusEnum.DELETED.getCode().equals(metaBatchReq.getStatus())) {
|
|| StatusEnum.DELETED.getCode().equals(metaBatchReq.getStatus())) {
|
||||||
sendEventBatch(metricDOS, EventType.DELETE);
|
sendEventBatch(metricDOS, EventType.DELETE, user);
|
||||||
} else if (StatusEnum.ONLINE.getCode().equals(metaBatchReq.getStatus())) {
|
} else if (StatusEnum.ONLINE.getCode().equals(metaBatchReq.getStatus())) {
|
||||||
sendEventBatch(metricDOS, EventType.ADD);
|
sendEventBatch(metricDOS, EventType.ADD, user);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -285,7 +305,7 @@ public class MetricServiceImpl extends ServiceImpl<MetricDOMapper, MetricDO>
|
|||||||
// should update modelDetail
|
// should update modelDetail
|
||||||
modelService.deleteModelDetailByDimAndMetric(metricDO.getModelId(), null,
|
modelService.deleteModelDetailByDimAndMetric(metricDO.getModelId(), null,
|
||||||
Lists.newArrayList(metricDO));
|
Lists.newArrayList(metricDO));
|
||||||
sendEventBatch(Lists.newArrayList(metricDO), EventType.DELETE);
|
sendEventBatch(Lists.newArrayList(metricDO), EventType.DELETE, user);
|
||||||
}
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
@@ -306,7 +326,7 @@ public class MetricServiceImpl extends ServiceImpl<MetricDOMapper, MetricDO>
|
|||||||
// should update modelDetail
|
// should update modelDetail
|
||||||
modelService.deleteModelDetailByDimAndMetric(metricDOList.get(0).getModelId(), null,
|
modelService.deleteModelDetailByDimAndMetric(metricDOList.get(0).getModelId(), null,
|
||||||
metricDOList);
|
metricDOList);
|
||||||
sendEventBatch(metricDOList, EventType.DELETE);
|
sendEventBatch(metricDOList, EventType.DELETE, user);
|
||||||
}
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
@@ -661,11 +681,11 @@ public class MetricServiceImpl extends ServiceImpl<MetricDOMapper, MetricDO>
|
|||||||
}
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
public void sendMetricEventBatch(List<Long> modelIds, EventType eventType) {
|
public void sendMetricEventBatch(List<Long> modelIds, EventType eventType, User user) {
|
||||||
MetricFilter metricFilter = new MetricFilter();
|
MetricFilter metricFilter = new MetricFilter();
|
||||||
metricFilter.setModelIds(modelIds);
|
metricFilter.setModelIds(modelIds);
|
||||||
List<MetricDO> metricDOS = queryMetric(metricFilter);
|
List<MetricDO> metricDOS = queryMetric(metricFilter);
|
||||||
sendEventBatch(metricDOS, eventType);
|
sendEventBatch(metricDOS, eventType, user);
|
||||||
}
|
}
|
||||||
|
|
||||||
@Override
|
@Override
|
||||||
@@ -678,22 +698,23 @@ public class MetricServiceImpl extends ServiceImpl<MetricDOMapper, MetricDO>
|
|||||||
public DataEvent getDataEvent() {
|
public DataEvent getDataEvent() {
|
||||||
MetricsFilter metricsFilter = new MetricsFilter();
|
MetricsFilter metricsFilter = new MetricsFilter();
|
||||||
List<MetricDO> metricDOS = metricRepository.getMetrics(metricsFilter);
|
List<MetricDO> metricDOS = metricRepository.getMetrics(metricsFilter);
|
||||||
return getDataEvent(metricDOS, EventType.ADD);
|
return getDataEvent(metricDOS, EventType.ADD, User.getDefaultUser());
|
||||||
}
|
}
|
||||||
|
|
||||||
private DataEvent getDataEvent(List<MetricDO> metricDOS, EventType eventType) {
|
private DataEvent getDataEvent(List<MetricDO> metricDOS, EventType eventType, User user) {
|
||||||
List<DataItem> dataItems =
|
List<DataItem> dataItems = metricDOS.stream().map(this::getDataItem)
|
||||||
metricDOS.stream().map(this::getDataItem).collect(Collectors.toList());
|
.filter(Objects::nonNull).collect(Collectors.toList());
|
||||||
return new DataEvent(this, dataItems, eventType);
|
return new DataEvent(this, dataItems, eventType, user.getName());
|
||||||
}
|
}
|
||||||
|
|
||||||
private void sendEventBatch(List<MetricDO> metricDOS, EventType eventType) {
|
private void sendEventBatch(List<MetricDO> metricDOS, EventType eventType, User user) {
|
||||||
DataEvent dataEvent = getDataEvent(metricDOS, eventType);
|
DataEvent dataEvent = getDataEvent(metricDOS, eventType, user);
|
||||||
eventPublisher.publishEvent(dataEvent);
|
eventPublisher.publishEvent(dataEvent);
|
||||||
}
|
}
|
||||||
|
|
||||||
private void sendEvent(DataItem dataItem, EventType eventType) {
|
private void sendEvent(DataItem dataItem, EventType eventType, User user) {
|
||||||
eventPublisher.publishEvent(new DataEvent(this, Lists.newArrayList(dataItem), eventType));
|
eventPublisher.publishEvent(
|
||||||
|
new DataEvent(this, Lists.newArrayList(dataItem), eventType, user.getName()));
|
||||||
}
|
}
|
||||||
|
|
||||||
private DataItem getDataItem(MetricDO metricDO) {
|
private DataItem getDataItem(MetricDO metricDO) {
|
||||||
|
|||||||
@@ -95,7 +95,7 @@ public class ModelServiceImpl implements ModelService {
|
|||||||
// create or update metric
|
// create or update metric
|
||||||
List<MetricReq> metricReqs = ModelConverter.convertMetricList(modelDO);
|
List<MetricReq> metricReqs = ModelConverter.convertMetricList(modelDO);
|
||||||
metricService.alterMetricBatch(metricReqs, modelDO.getId(), user);
|
metricService.alterMetricBatch(metricReqs, modelDO.getId(), user);
|
||||||
sendEvent(modelDO, EventType.ADD);
|
sendEvent(modelDO, EventType.ADD, user);
|
||||||
return ModelConverter.convert(modelDO);
|
return ModelConverter.convert(modelDO);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -129,7 +129,7 @@ public class ModelServiceImpl implements ModelService {
|
|||||||
// create or update metric
|
// create or update metric
|
||||||
List<MetricReq> metricReqs = ModelConverter.convertMetricList(modelDO);
|
List<MetricReq> metricReqs = ModelConverter.convertMetricList(modelDO);
|
||||||
metricService.alterMetricBatch(metricReqs, modelDO.getId(), user);
|
metricService.alterMetricBatch(metricReqs, modelDO.getId(), user);
|
||||||
sendEvent(modelDO, EventType.UPDATE);
|
sendEvent(modelDO, EventType.UPDATE, user);
|
||||||
return ModelConverter.convert(modelDO);
|
return ModelConverter.convert(modelDO);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -507,14 +507,14 @@ public class ModelServiceImpl implements ModelService {
|
|||||||
if (StatusEnum.OFFLINE.getCode().equals(metaBatchReq.getStatus())
|
if (StatusEnum.OFFLINE.getCode().equals(metaBatchReq.getStatus())
|
||||||
|| StatusEnum.DELETED.getCode().equals(metaBatchReq.getStatus())) {
|
|| StatusEnum.DELETED.getCode().equals(metaBatchReq.getStatus())) {
|
||||||
metricService.sendMetricEventBatch(Lists.newArrayList(modelDO.getId()),
|
metricService.sendMetricEventBatch(Lists.newArrayList(modelDO.getId()),
|
||||||
EventType.DELETE);
|
EventType.DELETE, user);
|
||||||
dimensionService.sendDimensionEventBatch(Lists.newArrayList(modelDO.getId()),
|
dimensionService.sendDimensionEventBatch(Lists.newArrayList(modelDO.getId()),
|
||||||
EventType.DELETE);
|
EventType.DELETE, user);
|
||||||
} else if (StatusEnum.ONLINE.getCode().equals(metaBatchReq.getStatus())) {
|
} else if (StatusEnum.ONLINE.getCode().equals(metaBatchReq.getStatus())) {
|
||||||
metricService.sendMetricEventBatch(Lists.newArrayList(modelDO.getId()),
|
metricService.sendMetricEventBatch(Lists.newArrayList(modelDO.getId()),
|
||||||
EventType.ADD);
|
EventType.ADD, user);
|
||||||
dimensionService.sendDimensionEventBatch(Lists.newArrayList(modelDO.getId()),
|
dimensionService.sendDimensionEventBatch(Lists.newArrayList(modelDO.getId()),
|
||||||
EventType.ADD);
|
EventType.ADD, user);
|
||||||
}
|
}
|
||||||
}).collect(Collectors.toList());
|
}).collect(Collectors.toList());
|
||||||
modelRepository.batchUpdate(modelDOS);
|
modelRepository.batchUpdate(modelDOS);
|
||||||
@@ -670,9 +670,10 @@ public class ModelServiceImpl implements ModelService {
|
|||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
private void sendEvent(ModelDO modelDO, EventType eventType) {
|
private void sendEvent(ModelDO modelDO, EventType eventType, User user) {
|
||||||
DataItem dataItem = getDataItem(modelDO);
|
DataItem dataItem = getDataItem(modelDO);
|
||||||
eventPublisher.publishEvent(new DataEvent(this, Lists.newArrayList(dataItem), eventType));
|
eventPublisher.publishEvent(
|
||||||
|
new DataEvent(this, Lists.newArrayList(dataItem), eventType, user.getName()));
|
||||||
}
|
}
|
||||||
|
|
||||||
private DataItem getDataItem(ModelDO modelDO) {
|
private DataItem getDataItem(ModelDO modelDO) {
|
||||||
|
|||||||
@@ -79,7 +79,7 @@ public class RetrieveServiceImpl implements RetrieveService {
|
|||||||
Set<Long> dataSetIds = queryNLReq.getDataSetIds();
|
Set<Long> dataSetIds = queryNLReq.getDataSetIds();
|
||||||
|
|
||||||
ChatQueryContext chatQueryContext = new ChatQueryContext(queryNLReq);
|
ChatQueryContext chatQueryContext = new ChatQueryContext(queryNLReq);
|
||||||
chatQueryContext.setModelIdToDataSetIds(dataSetService.getModelIdToDataSetIds());
|
chatQueryContext.setModelIdToDataSetIds(modelIdToDataSetIds);
|
||||||
|
|
||||||
Map<MatchText, List<HanlpMapResult>> regTextMap =
|
Map<MatchText, List<HanlpMapResult>> regTextMap =
|
||||||
searchMatchStrategy.match(chatQueryContext, originals, dataSetIds);
|
searchMatchStrategy.match(chatQueryContext, originals, dataSetIds);
|
||||||
|
|||||||
@@ -1,11 +1,13 @@
|
|||||||
package com.tencent.supersonic.headless.server.utils;
|
package com.tencent.supersonic.headless.server.utils;
|
||||||
|
|
||||||
import com.google.common.collect.Lists;
|
import com.google.common.collect.Lists;
|
||||||
|
import com.google.common.collect.Maps;
|
||||||
import com.tencent.supersonic.common.pojo.DimensionConstants;
|
import com.tencent.supersonic.common.pojo.DimensionConstants;
|
||||||
import com.tencent.supersonic.headless.api.pojo.*;
|
import com.tencent.supersonic.headless.api.pojo.*;
|
||||||
import com.tencent.supersonic.headless.api.pojo.response.DataSetSchemaResp;
|
import com.tencent.supersonic.headless.api.pojo.response.DataSetSchemaResp;
|
||||||
import com.tencent.supersonic.headless.api.pojo.response.DimSchemaResp;
|
import com.tencent.supersonic.headless.api.pojo.response.DimSchemaResp;
|
||||||
import com.tencent.supersonic.headless.api.pojo.response.MetricSchemaResp;
|
import com.tencent.supersonic.headless.api.pojo.response.MetricSchemaResp;
|
||||||
|
import com.tencent.supersonic.headless.api.pojo.response.ModelResp;
|
||||||
import com.tencent.supersonic.headless.api.pojo.response.TermResp;
|
import com.tencent.supersonic.headless.api.pojo.response.TermResp;
|
||||||
import org.apache.commons.lang3.StringUtils;
|
import org.apache.commons.lang3.StringUtils;
|
||||||
import org.springframework.beans.BeanUtils;
|
import org.springframework.beans.BeanUtils;
|
||||||
@@ -92,6 +94,13 @@ public class DataSetSchemaBuilder {
|
|||||||
|
|
||||||
private static Set<SchemaElement> getDimensions(DataSetSchemaResp resp) {
|
private static Set<SchemaElement> getDimensions(DataSetSchemaResp resp) {
|
||||||
Set<SchemaElement> dimensions = new HashSet<>();
|
Set<SchemaElement> dimensions = new HashSet<>();
|
||||||
|
Map<Long, Map<String, String>> dataTypeMap = Maps.newHashMap();
|
||||||
|
for (ModelResp modelResp : resp.getModelResps()) {
|
||||||
|
dataTypeMap.put(modelResp.getId(),
|
||||||
|
modelResp.getModelDetail().getFields().stream().collect(Collectors
|
||||||
|
.toMap(Field::getFieldName, Field::getDataType, (k1, k2) -> k2)));
|
||||||
|
}
|
||||||
|
|
||||||
for (DimSchemaResp dim : resp.getDimensions()) {
|
for (DimSchemaResp dim : resp.getDimensions()) {
|
||||||
List<String> alias = SchemaItem.getAliasList(dim.getAlias());
|
List<String> alias = SchemaItem.getAliasList(dim.getAlias());
|
||||||
List<DimValueMap> dimValueMaps = dim.getDimValueMaps();
|
List<DimValueMap> dimValueMaps = dim.getDimValueMaps();
|
||||||
@@ -109,7 +118,14 @@ public class DataSetSchemaBuilder {
|
|||||||
.alias(alias).schemaValueMaps(schemaValueMaps).isTag(dim.getIsTag())
|
.alias(alias).schemaValueMaps(schemaValueMaps).isTag(dim.getIsTag())
|
||||||
.description(dim.getDescription()).type(SchemaElementType.DIMENSION).build();
|
.description(dim.getDescription()).type(SchemaElementType.DIMENSION).build();
|
||||||
dimToAdd.getExtInfo().put(DimensionConstants.DIMENSION_TYPE, dim.getType());
|
dimToAdd.getExtInfo().put(DimensionConstants.DIMENSION_TYPE, dim.getType());
|
||||||
|
// data type
|
||||||
|
if (dim.getDataType() != null) {
|
||||||
|
dimToAdd.getExtInfo().put(DimensionConstants.DIMENSION_DATA_TYPE,
|
||||||
|
dim.getDataType());
|
||||||
|
} else {
|
||||||
|
dimToAdd.getExtInfo().put(DimensionConstants.DIMENSION_DATA_TYPE,
|
||||||
|
dataTypeMap.get(dim.getModelId()).get(dim.getBizName()));
|
||||||
|
}
|
||||||
if (dim.isTimeDimension()) {
|
if (dim.isTimeDimension()) {
|
||||||
String timeFormat =
|
String timeFormat =
|
||||||
String.valueOf(dim.getExt().get(DimensionConstants.DIMENSION_TIME_FORMAT));
|
String.valueOf(dim.getExt().get(DimensionConstants.DIMENSION_TIME_FORMAT));
|
||||||
|
|||||||
@@ -422,33 +422,25 @@ public class DictUtils {
|
|||||||
return joiner.toString();
|
return joiner.toString();
|
||||||
}
|
}
|
||||||
|
|
||||||
public String defaultDateFilter(DateConf dateConf) {
|
|
||||||
String format = itemValueDateFormat;
|
|
||||||
String start = LocalDate.now().minusDays(itemValueDateStart)
|
|
||||||
.format(DateTimeFormatter.ofPattern(format));
|
|
||||||
String end = LocalDate.now().minusDays(itemValueDateEnd)
|
|
||||||
.format(DateTimeFormatter.ofPattern(format));
|
|
||||||
if (Objects.nonNull(dateConf)) {
|
|
||||||
return String.format("( %s >= '%s' and %s <= '%s' )", dateConf.getDateField(), start,
|
|
||||||
dateConf.getDateField(), end);
|
|
||||||
} else {
|
|
||||||
return String.format("( %s >= '%s' and %s <= '%s' )", "dt", start, "dt", end);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
private String generateDictDateFilter(DictItemResp dictItemResp) {
|
private String generateDictDateFilter(DictItemResp dictItemResp) {
|
||||||
ItemValueConfig config = dictItemResp.getConfig();
|
Dimension partitionTimeDimension = getPartitionTimeDimension(dictItemResp.getModelId());
|
||||||
if (config == null) {
|
// 如果没有设置数据时间维度,则无法做时间分区过滤
|
||||||
|
if (partitionTimeDimension == null) {
|
||||||
return "";
|
return "";
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!partitionedModel(dictItemResp.getModelId())) {
|
ItemValueConfig config = dictItemResp.getConfig();
|
||||||
return "";
|
// 默认使用数据时间维度进行时间分区过滤
|
||||||
}
|
|
||||||
// 未进行设置
|
|
||||||
if (Objects.isNull(config) || Objects.isNull(config.getDateConf())) {
|
if (Objects.isNull(config) || Objects.isNull(config.getDateConf())) {
|
||||||
return defaultDateFilter(null);
|
String startDate = LocalDate.now().minusDays(itemValueDateStart)
|
||||||
|
.format(DateTimeFormatter.ofPattern(partitionTimeDimension.getDateFormat()));
|
||||||
|
String endDate = LocalDate.now().minusDays(itemValueDateEnd)
|
||||||
|
.format(DateTimeFormatter.ofPattern(partitionTimeDimension.getDateFormat()));
|
||||||
|
return String.format("( %s >= '%s' and %s <= '%s' )",
|
||||||
|
partitionTimeDimension.getBizName(), startDate,
|
||||||
|
partitionTimeDimension.getBizName(), endDate);
|
||||||
}
|
}
|
||||||
|
|
||||||
// 全表扫描
|
// 全表扫描
|
||||||
if (DateConf.DateMode.ALL.equals(config.getDateConf().getDateMode())) {
|
if (DateConf.DateMode.ALL.equals(config.getDateConf().getDateMode())) {
|
||||||
return "";
|
return "";
|
||||||
@@ -467,15 +459,15 @@ public class DictUtils {
|
|||||||
return "";
|
return "";
|
||||||
}
|
}
|
||||||
|
|
||||||
private boolean partitionedModel(Long modelId) {
|
private Dimension getPartitionTimeDimension(Long modelId) {
|
||||||
ModelResp model = modelService.getModel(modelId);
|
ModelResp model = modelService.getModel(modelId);
|
||||||
if (Objects.nonNull(model)) {
|
if (Objects.nonNull(model)) {
|
||||||
List<Dimension> timeDims = model.getTimeDimension();
|
List<Dimension> timeDims = model.getTimeDimension();
|
||||||
if (!CollectionUtils.isEmpty(timeDims)) {
|
if (!CollectionUtils.isEmpty(timeDims)) {
|
||||||
return true;
|
return timeDims.get(0);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return false;
|
return null;
|
||||||
}
|
}
|
||||||
|
|
||||||
private String generateDictDateFilterRecent(DictItemResp dictItemResp) {
|
private String generateDictDateFilterRecent(DictItemResp dictItemResp) {
|
||||||
|
|||||||
@@ -10,7 +10,14 @@ import {
|
|||||||
SimilarQuestionType,
|
SimilarQuestionType,
|
||||||
} from '../../common/type';
|
} from '../../common/type';
|
||||||
import { createContext, useEffect, useRef, useState } from 'react';
|
import { createContext, useEffect, useRef, useState } from 'react';
|
||||||
import { chatExecute, chatParse, queryData, deleteQuery, switchEntity } from '../../service';
|
import {
|
||||||
|
chatExecute,
|
||||||
|
chatParse,
|
||||||
|
queryData,
|
||||||
|
deleteQuery,
|
||||||
|
switchEntity,
|
||||||
|
getExecuteSummary,
|
||||||
|
} from '../../service';
|
||||||
import { PARSE_ERROR_TIP, PREFIX_CLS, SEARCH_EXCEPTION_TIP } from '../../common/constants';
|
import { PARSE_ERROR_TIP, PREFIX_CLS, SEARCH_EXCEPTION_TIP } from '../../common/constants';
|
||||||
import { message, Spin } from 'antd';
|
import { message, Spin } from 'antd';
|
||||||
import IconFont from '../IconFont';
|
import IconFont from '../IconFont';
|
||||||
@@ -169,7 +176,7 @@ const ChatItem: React.FC<Props> = ({
|
|||||||
setExecuteLoading(true);
|
setExecuteLoading(true);
|
||||||
}
|
}
|
||||||
try {
|
try {
|
||||||
const res: any = await chatExecute(msg, conversationId!, parseInfoValue, agentId);
|
const res: any = await chatExecute(msg, conversationId!, parseInfoValue, agentId, true);
|
||||||
const valid = updateData(res);
|
const valid = updateData(res);
|
||||||
onMsgDataLoaded?.(
|
onMsgDataLoaded?.(
|
||||||
{
|
{
|
||||||
@@ -180,6 +187,20 @@ const ChatItem: React.FC<Props> = ({
|
|||||||
valid,
|
valid,
|
||||||
isRefresh
|
isRefresh
|
||||||
);
|
);
|
||||||
|
const queryId = parseInfoValue.queryId; // 伪流式 大模型输出
|
||||||
|
if (queryId != undefined && res.data.queryState != 'INVALID') {
|
||||||
|
const getSummary = async (data: any, queryId: number) => {
|
||||||
|
const res2: any = await getExecuteSummary(queryId);
|
||||||
|
if (res2.data.queryMode == null) {
|
||||||
|
res2.data = { ...data, textSummary: res2.data.textSummary };
|
||||||
|
setData(res2.data);
|
||||||
|
setTimeout(() => getSummary(data, queryId), 500);
|
||||||
|
} else {
|
||||||
|
setData(res2.data);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
setTimeout(() => getSummary(res.data, queryId), 500);
|
||||||
|
}
|
||||||
} catch (e) {
|
} catch (e) {
|
||||||
const tip = SEARCH_EXCEPTION_TIP;
|
const tip = SEARCH_EXCEPTION_TIP;
|
||||||
setExecuteTip(SEARCH_EXCEPTION_TIP);
|
setExecuteTip(SEARCH_EXCEPTION_TIP);
|
||||||
@@ -423,6 +444,10 @@ const ChatItem: React.FC<Props> = ({
|
|||||||
return result;
|
return result;
|
||||||
}, {});
|
}, {});
|
||||||
});
|
});
|
||||||
|
if (exportData.length === 0) {
|
||||||
|
message.error('该条消息暂不支持该操作');
|
||||||
|
return;
|
||||||
|
}
|
||||||
exportCsvFile(exportData);
|
exportCsvFile(exportData);
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|||||||
@@ -79,7 +79,8 @@ export function chatExecute(
|
|||||||
queryText: string,
|
queryText: string,
|
||||||
chatId: number,
|
chatId: number,
|
||||||
parseInfo: ChatContextType,
|
parseInfo: ChatContextType,
|
||||||
agentId?: number
|
agentId?: number,
|
||||||
|
streamingResult?:boolean
|
||||||
) {
|
) {
|
||||||
return axios.post<MsgDataType>(`${prefix}/chat/query/execute`, {
|
return axios.post<MsgDataType>(`${prefix}/chat/query/execute`, {
|
||||||
queryText,
|
queryText,
|
||||||
@@ -87,6 +88,15 @@ export function chatExecute(
|
|||||||
chatId: chatId || DEFAULT_CHAT_ID,
|
chatId: chatId || DEFAULT_CHAT_ID,
|
||||||
queryId: parseInfo.queryId,
|
queryId: parseInfo.queryId,
|
||||||
parseId: parseInfo.id,
|
parseId: parseInfo.id,
|
||||||
|
streamingResult:streamingResult
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export function getExecuteSummary(
|
||||||
|
queryId: number
|
||||||
|
) {
|
||||||
|
return axios.post<MsgDataType>(`${prefix}/chat/query/getExecuteSummary`, {
|
||||||
|
queryId: queryId,
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user