Commit Graph

196 Commits

Author SHA1 Message Date
Jun Zhang
a8868ba9ab Merge branch 'tencentmusic:master' into master 2024-12-11 22:48:14 +08:00
jerryjzhang
8c44c9f42f [improvement][headless]Remove fixed TimeEnums fields. 2024-12-11 21:33:43 +08:00
mislayming
07f6be51c7 SpringBoot3升级 (#1947)
Some checks failed
supersonic CentOS CI / build (21) (push) Has been cancelled
supersonic mac CI / build (21) (push) Has been cancelled
supersonic ubuntu CI / build (21) (push) Has been cancelled
supersonic windows CI / build (21) (push) Has been cancelled
2024-12-11 17:09:20 +08:00
Jun Zhang
e55f43c737 [improvement]Use QueryWrapper in place of hard-coded SQLs (#1944)
Some checks are pending
supersonic CentOS CI / build (11) (push) Waiting to run
supersonic CentOS CI / build (21) (push) Waiting to run
supersonic CentOS CI / build (8) (push) Waiting to run
supersonic mac CI / build (11) (push) Waiting to run
supersonic mac CI / build (21) (push) Waiting to run
supersonic mac CI / build (8) (push) Waiting to run
supersonic ubuntu CI / build (11) (push) Waiting to run
supersonic ubuntu CI / build (21) (push) Waiting to run
supersonic ubuntu CI / build (8) (push) Waiting to run
supersonic windows CI / build (11) (push) Waiting to run
supersonic windows CI / build (21) (push) Waiting to run
supersonic windows CI / build (8) (push) Waiting to run
* [improvement][launcher]Use API to get element ID avoiding hard-code.

* [fix][launcher]Fix mysql scripts.

* [improvement][launcher]Support DuckDB database and refactor translator code structure.

* [improvement][headless-fe] Revamped the interaction for semantic modeling routing and successfully implemented the switching between dimension and dataset management.

* [improvement][Headless] Add table ddl in Dbschema

* [improvement][Headless] Add get database by type

* [improvement][Headless] Supports automatic batch creation of models based on db table names.

* [improvement][Headless] Supports getting domain by bizName

* [improvement][launcher]Refactor unit tests and demo data.

* [fix][launcher]Change default vector dimension to 512.

* [improvement](Dict) add dimValueAliasMap info for KnowledgeBaseService

* [improvement][headless]Use QueryWrapper to replace hard-code SQL in mapper xml.

* [improvement][chat]Introduce ChatMemory to delegate ChatMemoryDO.

* [fix][common]Fix embedding store sys configs.

* [fix][common]Fix postgres schema, using varchar instead of char.

* [improvement][launcher]Change supersonic docker deployment from mysql to postgres.

* [Fix][launcher]Fix a number of issues related to semantic modeling.

* [Fix][headless]Fix the evaluation logic of agg type.

* [fix][assembly]Fix Dockerfile and add docker compose run script.

* [fix][chat]Fix "multiple assignments to same column "similar_queries".

* [improvement][headless]Use LamdaQueryWrapper to avoid hard-coded column names.

* [improvement][headless]Refactor headless infra to support advanced semantic modelling.

* [improvement][headless]Change class name `Dim` to `Dimension`.

* [improvement][chat]Introduce `TimeFieldMapper` to always map time field.

* [fix][headless]Remove unnecessary dimension existence check.

* [fix][chat]Fix adjusted filters don't take effect.

---------
2024-12-08 13:32:29 +08:00
jerryjzhang
aa400176a5 [improvement][chat]Introduce TimeFieldMapper to always map time field. 2024-12-04 23:48:43 +08:00
daikon12
efac1efdb0 [improvement](Dict) add dimValueAliasMap info for KnowledgeBaseService 2024-12-01 23:48:51 +08:00
Jun Zhang
593597fe26 Refactor translator module (#1932)
* [improvement][Chat] Support agent permission management #1143

* [improvement][chat]Iterate LLM prompts of parsing and correction.

* [improvement][headless]Clean code logic of headless core.

* (fix) (chat) 记忆管理更新不生效 (#1912)

* [improvement][headless-fe] Added null-check conditions to the data formatting function.

* [improvement][headless]Clean code logic of headless translator.

* [improvement][headless-fe] Added permissions management for agents.

* [improvement][headless-fe] Unified the assistant's permission settings interaction to match the system style.

* [improvement](Dict)Support returns dict task list of dimensions by page

* [improvement][headless-fe] Revised the interaction for semantic modeling routing and implemented the initial version of metric management switching.

* [improvement][launcher]Set system property `s2.test` in junit tests in order to facilitate conditional breakpoints.

* [improvement][headless] add validateAndQuery interface in SqlQueryApiController

* [improvement][launcher]Use API to get element ID avoiding hard-code.

* [improvement][launcher]Support DuckDB database and refactor translator code structure.

---------

Co-authored-by: lxwcodemonkey <jolunoluo@tencent.com>
Co-authored-by: tristanliu <tristanliu@tencent.com>
Co-authored-by: daikon12 <1059907724@qq.com>
Co-authored-by: lexluo09 <39718951+lexluo09@users.noreply.github.com>
2024-11-30 00:27:33 +08:00
jerryjzhang
c80794e8fc [fix][launcher]Fix mysql scripts. 2024-11-28 09:44:49 +08:00
Jun Zhang
cb183b7ac8 [improvement][Chat] Support agent permission management (#1923)
* [improvement][Chat] Support agent permission management #1143

* [improvement][chat]Iterate LLM prompts of parsing and correction.

* [improvement][headless-fe] Added null-check conditions to the data formatting function.

* [improvement][headless]Clean code logic of headless translator.

---------

Co-authored-by: lxwcodemonkey <jolunoluo@tencent.com>
Co-authored-by: tristanliu <tristanliu@tencent.com>
2024-11-23 09:09:04 +08:00
jerryjzhang
d7586a5d3b [improvement][headless]Clean code logic of headless translator. 2024-11-20 22:23:53 +08:00
jerryjzhang
a0f53359ef [improvement][chat]Iterate LLM prompts of parsing and correction. 2024-11-17 09:06:42 +08:00
Jun Zhang
ba1938f04b Merge fixes and improvements (#1910)
Co-authored-by: tristanliu <tristanliu@tencent.com>
2024-11-16 13:57:54 +08:00
zhaodongsheng
5e22b412c6 (fix) CoreDictionaryPath Incorrect Assignment (#1903) 2024-11-14 22:34:59 +08:00
jerryjzhang
14a19a901f [improvement][project]Simplify code logic in multiple modules. 2024-11-10 14:31:12 +08:00
lexluo09
ca4545bb15 [improvement][chat] Only vector retrieval is enabled in loose mode (#1899) 2024-11-10 10:39:17 +08:00
Jun Zhang
e0e167fd40 [improvement][chat]Refactor code logic in rule-based parsing. 2024-11-09 15:49:08 +08:00
lexluo09
d4a9d5a7e6 [fix][chat] The terms in the descriptor of the term no longer undergo descriptor mapping (#1897) 2024-11-09 14:16:55 +08:00
Jun Zhang
c9c6dc4e44 Merge a number of fixes and improvements (#1896) 2024-11-09 00:23:02 +08:00
jerryjzhang
6f2af79756 [improvement][chat]Move processor related logic from headless to chat. 2024-11-04 16:27:26 +08:00
jerryjzhang
9a05b5cce6 [improvement][headless]Deprecate and remove entity-related abstraction and logic.#1876 2024-11-04 10:17:30 +08:00
lexluo09
3b65b1c80b [improvement][headless] Remove entities from the dictionary and search interface (#1878) 2024-11-04 10:06:10 +08:00
jerryjzhang
1e5bf7909e [improvement][headless]Deprecate and remove entity-related abstraction and logic.#1876 2024-11-04 09:53:18 +08:00
lexluo09
cf359f3e2f [improvement][chat] Optimize and modify the mapper method for terminology (#1866) 2024-10-31 11:18:35 +08:00
jerryjzhang
adde7bbd45 [fix][chat]Fix TERM schema element not mapped. 2024-10-30 22:34:46 +08:00
jerryjzhang
9644fc4207 [improvement][chat]Remove deprecated SatisfactoryChecker. 2024-10-30 10:36:19 +08:00
jerryjzhang
53ddc67262 [improvement][chat]Sort parses in NL2SQLParser right after rule-based parsing. 2024-10-29 23:31:02 +08:00
jerryjzhang
847505b293 [fix][chat]Deprecate and remove ContextInheritParser. 2024-10-29 22:34:11 +08:00
lexluo09
d9cf874536 [improvement][chat] In STRICT mode, embedingMapper does not perform mapping (#1858) 2024-10-29 20:54:37 +08:00
jerryjzhang
cbb76550c7 [improvement][chat]Optimize NL2SQL parsing logic. 2024-10-29 20:33:32 +08:00
jerryjzhang
996cb3df56 [improvement][chat]Change query type resolution logic. 2024-10-29 19:03:10 +08:00
jerryjzhang
400b9f86f0 [improvement][chat]Modify core workflow of NL2SQLParser, always invoking rule-based parsers first.#1729 2024-10-29 13:12:56 +08:00
jerryjzhang
414aaaa0b6 [improvement][chat]Modify core workflow of NL2SQLParser, always invoking rule-based parsers first.#1729 2024-10-29 00:41:44 +08:00
lexluo09
5d9b1b917e [improvement][chat] The parser interface supports using the dataSetId provided by the frontend as the reference (#1852) 2024-10-28 21:51:11 +08:00
jerryjzhang
e046a55567 [improvement][chat]Only return the parse info of the highest score for one dataset pass.#1847 2024-10-28 20:00:34 +08:00
lexluo09
de4384062c [improvement][chat] The similarity of MapModeEnum.STRICT is 1 (#1851) 2024-10-28 19:58:13 +08:00
zhaodongsheng
c07b64d33c [fix]自定义词典:xxxx 读取错误,#1826 (#1840) 2024-10-28 13:23:35 +08:00
jerryjzhang
eb28d832bc [improvement][chat]Support user feedback to semantic parse info.#1729 2024-10-28 02:07:54 +08:00
jerryjzhang
b69ee81d58 [improvement][chat]Move generation of semantic text info and rewrite of error message to dedicated ResultProcessor. 2024-10-27 22:34:16 +08:00
jerryjzhang
bb363a0286 [improvement][headless-chat]Incorporate Request into Context objects, removing unnecessary copy. 2024-10-27 18:47:34 +08:00
jerryjzhang
397b527bc6 [improvement][project]Remove unnecessary SchemaMapInfo from ParseContext. 2024-10-27 15:16:42 +08:00
jerryjzhang
1e3daffade [improvement][project]Opt some code structures. 2024-10-27 10:38:30 +08:00
lexluo09
473329d398 [improvement][chat] Filter at the lowest level in the Map based on the dataSetId (#1834) 2024-10-20 21:51:57 +08:00
LXW
1f208ffcb7 (improvement)(Common) Move ChatModel management to common modules and add chatApp for data-model builder (#1831)
Co-authored-by: lxwcodemonkey
2024-10-20 18:42:08 +08:00
lexluo09
2250289c94 (fix)(chat) Fix the issue of mapping in the map (#1825) 2024-10-19 13:22:39 +08:00
jerryjzhang
bb07fd77cf [improvement][headless-chat]Optimize text2sql prompt. 2024-10-18 22:53:30 +08:00
jerryjzhang
5c8b9a0ee1 [improvement][project]Optimize log files and outputs. 2024-10-18 14:17:47 +08:00
zhaodongsheng
55a4d5a198 (fix) NoSuchFileException when reading dimension file (#1813) 2024-10-17 20:12:37 +08:00
lexluo09
ee4b218a1e [improvement][chat] Fix the issue of HDFS path concatenation (#1820) 2024-10-17 14:12:12 +08:00
jerryjzhang
d064393253 [fix][chat]Fix NPE problem. 2024-10-16 13:08:22 +08:00
lexluo09
9302d1f6e4 [improvement][chat] The two characters need to be a perfect match (#1805) 2024-10-15 12:21:14 +08:00