206 Commits

Author SHA1 Message Date
jipeli
be158a1776 [improvement][headless] fix derived metric aggOption error (#679) 2024-01-22 19:13:55 +08:00
jipeli
c12f5d23f0 [improvement][headless] fix derived metric case parse error (#678) 2024-01-22 12:54:45 +08:00
tristanliu
7ec77c7d23 [improvement][semantic-fe] fix update the metric list. (#677)
* [improvement][semantic-fe] Add model alias setting & Add view permission restrictions to the model permission management tab.
[improvement][semantic-fe] Add permission control to the action buttons for the main domain; apply high sensitivity filtering to the authorization of metrics/dimensions.
[improvement][semantic-fe] Optimize the editing mode in the dimension/metric/datasource components to use the modelId stored in the database for data, instead of relying on the data from the state manager.

* [improvement][semantic-fe] Add time granularity setting in the data source configuration.

* [improvement][semantic-fe] Dictionary import for dimension values supported in Q&A visibility

* [improvement][semantic-fe] Modification of data source creation prompt wording"

* [improvement][semantic-fe] metric market experience optimization

* [improvement][semantic-fe] enhance the analysis of metric trends

* [improvement][semantic-fe] optimize the presentation of metric trend permissions

* [improvement][semantic-fe] add metric trend download functionality

* [improvement][semantic-fe] fix the dimension initialization issue in metric correlation

* [improvement][semantic-fe] Fix the issue of database changes not taking effect when creating based on an SQL data source.

* [improvement][semantic-fe] Optimizing pagination logic and some CSS styles

* [improvement][semantic-fe] Fixing the API for the indicator list by changing "current" to "pageNum"

* [improvement][semantic-fe] Fixing the default value setting for the indicator list

* [improvement][semantic-fe] Adding batch operations for indicators/dimensions/models

* [improvement][semantic-fe] Replacing the single status update API for indicators/dimensions with a batch update API

* [improvement][semantic-fe] Redesigning the indicator homepage to incorporate trend charts and table functionality for indicators

* [improvement][semantic-fe] Optimizing the logic for setting dimension values and editing data sources, and adding system settings functionality

* [improvement][semantic-fe] Upgrading antd version to 5.x, extracting the batch operation button component, optimizing the interaction for system settings, and expanding the configuration generation types for list-to-select component.

* [improvement][semantic-fe] Adding the ability to filter dimensions based on whether they are tags or not.

* [improvement][semantic-fe] Adding the ability to edit relationships between models in the canvas.

* [improvement][semantic-fe] Updating the datePicker component to use dayjs instead.

* [improvement][semantic-fe] Fixing the issue with passing the model ID for dimensions in the indicator market.

* [improvement][semantic-fe] Fixing the abnormal state of the popup when creating a model.

* [improvement][semantic-fe] Adding permission logic for bulk operations in the indicator market.

* [improvement][semantic-fe] Adding the ability to download and transpose data.

* [improvement][semantic-fe] Fixing the initialization issue with the date selection component in the indicator details page when switching time granularity.

* [improvement][semantic-fe] Fixing the logic error in the dimension value setting.

* [improvement][semantic-fe] Fixing the synchronization issue with the question and answer settings information.

* [improvement][semantic-fe] Optimizing the canvas functionality for better performance and user experience.

* [improvement][semantic-fe] Optimizing the update process for drawing model relationship edges in the canvas.

* [improvement][semantic-fe] Changing the line type for canvas connections.

* [improvement][semantic-fe] Replacing the initialization variable from "semantic" to "headless".

* [improvement][semantic-fe] Fixing the missing migration issue for default drill-down dimension configuration in model editing. Additionally, optimizing the data retrieval method for initializing fields in the model.

* [improvement][semantic-fe] Updating the logic for the fieldName.

* [improvement][semantic-fe] Adjusting the position of the metrics tab.

* [improvement][semantic-fe] Changing the 字段名称 to 英文名称.

* [improvement][semantic-fe] Fix metric measurement deletion.

* [improvement][semantic-fe] UI optimization for metric details page.

* [improvement][semantic-fe] UI optimization for metric details page.

* [improvement][semantic-fe] UI adjustment for metric details page.

* [improvement][semantic-fe] The granularity field in the time type of model editing now supports setting it as empty.

* [improvement][semantic-fe] Added field type and metric type to the metric creation options.

* [improvement][semantic-fe] The organization structure selection feature has been added to the permission management.

* [improvement][semantic-fe] Improved user experience for the metric list.

* [improvement][semantic-fe] fix update the metric list.
2024-01-22 12:21:49 +08:00
lexluo09
f04cc28f25 (improvement)(project) Upgrade Maven version to 0.8.4-SNAPSHOT and add changelog. (#674) 2024-01-22 10:47:54 +08:00
LXW
b28eb637c8 (improvement)(chat) Put queryText to PluginParseResult (#673)
Co-authored-by: jolunoluo
2024-01-21 16:26:49 +08:00
Jun Zhang
97c767a45b Make some refactoring in Headless (#672)
* [improvement][headless]Move QueryCache from server to core and move yaml-related classes from core to server

* [improvement][headless]Declare QueryParser and QueryPlanner as interface instead of class.

---------

Co-authored-by: LXW
2024-01-21 16:14:39 +08:00
lexluo09
7afa42b4bc (improvement)(Headless) Support dropdown for MySQL version in the data source. (#671) 2024-01-19 23:16:57 +08:00
LXW
a375a922c2 (improvement)(chat) Add showcase demo (#670)
Co-authored-by: jolunoluo
2024-01-19 23:14:16 +08:00
jipeli
add74b9589 (improvement)(Headless) fix field parse error (#669) 2024-01-19 23:06:11 +08:00
LXW
566321895e (improvement)(headless) QueryStructReq covert to s2Sql adding a switch to determine whether it is bizName (#668)
Co-authored-by: jolunoluo
2024-01-19 22:58:17 +08:00
lexluo09
f154c2a2b4 (improvement)(Headless) Fix displaying the final executed SQL. (#667) 2024-01-19 22:52:20 +08:00
lexluo09
9c6bd7cf19 (improvement)(Headless) Support optimizing and generating SQL based on the engine type. (#666) 2024-01-19 22:32:28 +08:00
LXW
20c8456705 (improvement)(auth) Add organization demo (#664)
Co-authored-by: jolunoluo
2024-01-19 21:31:49 +08:00
jipeli
c154f476cb (improvement)(headless) fix derived metric parse error (#665) 2024-01-19 21:30:27 +08:00
LXW
1ef1aa53a3 (improvement)(headless) Remove redundant user dimensions in demo data (#663)
Co-authored-by: jolunoluo
2024-01-19 21:10:32 +08:00
lexluo09
08c184c7b0 (improvement)(test) Fix Checkstyle error (#662) 2024-01-19 18:24:26 +08:00
lexluo09
36edc0c1b4 (improvement)(test) Fix Python exemplar file name error and fix chat unit tests. (#661) 2024-01-19 18:12:01 +08:00
tristanliu
026cf2056d [improvement][semantic-fe] Improved user experience for the metric list. (#660)
* [improvement][semantic-fe] Add model alias setting & Add view permission restrictions to the model permission management tab.
[improvement][semantic-fe] Add permission control to the action buttons for the main domain; apply high sensitivity filtering to the authorization of metrics/dimensions.
[improvement][semantic-fe] Optimize the editing mode in the dimension/metric/datasource components to use the modelId stored in the database for data, instead of relying on the data from the state manager.

* [improvement][semantic-fe] Add time granularity setting in the data source configuration.

* [improvement][semantic-fe] Dictionary import for dimension values supported in Q&A visibility

* [improvement][semantic-fe] Modification of data source creation prompt wording"

* [improvement][semantic-fe] metric market experience optimization

* [improvement][semantic-fe] enhance the analysis of metric trends

* [improvement][semantic-fe] optimize the presentation of metric trend permissions

* [improvement][semantic-fe] add metric trend download functionality

* [improvement][semantic-fe] fix the dimension initialization issue in metric correlation

* [improvement][semantic-fe] Fix the issue of database changes not taking effect when creating based on an SQL data source.

* [improvement][semantic-fe] Optimizing pagination logic and some CSS styles

* [improvement][semantic-fe] Fixing the API for the indicator list by changing "current" to "pageNum"

* [improvement][semantic-fe] Fixing the default value setting for the indicator list

* [improvement][semantic-fe] Adding batch operations for indicators/dimensions/models

* [improvement][semantic-fe] Replacing the single status update API for indicators/dimensions with a batch update API

* [improvement][semantic-fe] Redesigning the indicator homepage to incorporate trend charts and table functionality for indicators

* [improvement][semantic-fe] Optimizing the logic for setting dimension values and editing data sources, and adding system settings functionality

* [improvement][semantic-fe] Upgrading antd version to 5.x, extracting the batch operation button component, optimizing the interaction for system settings, and expanding the configuration generation types for list-to-select component.

* [improvement][semantic-fe] Adding the ability to filter dimensions based on whether they are tags or not.

* [improvement][semantic-fe] Adding the ability to edit relationships between models in the canvas.

* [improvement][semantic-fe] Updating the datePicker component to use dayjs instead.

* [improvement][semantic-fe] Fixing the issue with passing the model ID for dimensions in the indicator market.

* [improvement][semantic-fe] Fixing the abnormal state of the popup when creating a model.

* [improvement][semantic-fe] Adding permission logic for bulk operations in the indicator market.

* [improvement][semantic-fe] Adding the ability to download and transpose data.

* [improvement][semantic-fe] Fixing the initialization issue with the date selection component in the indicator details page when switching time granularity.

* [improvement][semantic-fe] Fixing the logic error in the dimension value setting.

* [improvement][semantic-fe] Fixing the synchronization issue with the question and answer settings information.

* [improvement][semantic-fe] Optimizing the canvas functionality for better performance and user experience.

* [improvement][semantic-fe] Optimizing the update process for drawing model relationship edges in the canvas.

* [improvement][semantic-fe] Changing the line type for canvas connections.

* [improvement][semantic-fe] Replacing the initialization variable from "semantic" to "headless".

* [improvement][semantic-fe] Fixing the missing migration issue for default drill-down dimension configuration in model editing. Additionally, optimizing the data retrieval method for initializing fields in the model.

* [improvement][semantic-fe] Updating the logic for the fieldName.

* [improvement][semantic-fe] Adjusting the position of the metrics tab.

* [improvement][semantic-fe] Changing the 字段名称 to 英文名称.

* [improvement][semantic-fe] Fix metric measurement deletion.

* [improvement][semantic-fe] UI optimization for metric details page.

* [improvement][semantic-fe] UI optimization for metric details page.

* [improvement][semantic-fe] UI adjustment for metric details page.

* [improvement][semantic-fe] The granularity field in the time type of model editing now supports setting it as empty.

* [improvement][semantic-fe] Added field type and metric type to the metric creation options.

* [improvement][semantic-fe] The organization structure selection feature has been added to the permission management.

* [improvement][semantic-fe] Improved user experience for the metric list.
2024-01-19 17:50:17 +08:00
LXW
fc82350af5 (improvement)(headless) Update data demo (#659)
Co-authored-by: jolunoluo
2024-01-19 17:39:37 +08:00
jipeli
6b5d84a13f (improvement)(headless) refator query code (#658) 2024-01-19 17:11:18 +08:00
tristanliu
3ba9073a1b [improvement][semantic-fe] The organization structure selection feature has been added to the permission management. (#657)
* [improvement][semantic-fe] Add model alias setting & Add view permission restrictions to the model permission management tab.
[improvement][semantic-fe] Add permission control to the action buttons for the main domain; apply high sensitivity filtering to the authorization of metrics/dimensions.
[improvement][semantic-fe] Optimize the editing mode in the dimension/metric/datasource components to use the modelId stored in the database for data, instead of relying on the data from the state manager.

* [improvement][semantic-fe] Add time granularity setting in the data source configuration.

* [improvement][semantic-fe] Dictionary import for dimension values supported in Q&A visibility

* [improvement][semantic-fe] Modification of data source creation prompt wording"

* [improvement][semantic-fe] metric market experience optimization

* [improvement][semantic-fe] enhance the analysis of metric trends

* [improvement][semantic-fe] optimize the presentation of metric trend permissions

* [improvement][semantic-fe] add metric trend download functionality

* [improvement][semantic-fe] fix the dimension initialization issue in metric correlation

* [improvement][semantic-fe] Fix the issue of database changes not taking effect when creating based on an SQL data source.

* [improvement][semantic-fe] Optimizing pagination logic and some CSS styles

* [improvement][semantic-fe] Fixing the API for the indicator list by changing "current" to "pageNum"

* [improvement][semantic-fe] Fixing the default value setting for the indicator list

* [improvement][semantic-fe] Adding batch operations for indicators/dimensions/models

* [improvement][semantic-fe] Replacing the single status update API for indicators/dimensions with a batch update API

* [improvement][semantic-fe] Redesigning the indicator homepage to incorporate trend charts and table functionality for indicators

* [improvement][semantic-fe] Optimizing the logic for setting dimension values and editing data sources, and adding system settings functionality

* [improvement][semantic-fe] Upgrading antd version to 5.x, extracting the batch operation button component, optimizing the interaction for system settings, and expanding the configuration generation types for list-to-select component.

* [improvement][semantic-fe] Adding the ability to filter dimensions based on whether they are tags or not.

* [improvement][semantic-fe] Adding the ability to edit relationships between models in the canvas.

* [improvement][semantic-fe] Updating the datePicker component to use dayjs instead.

* [improvement][semantic-fe] Fixing the issue with passing the model ID for dimensions in the indicator market.

* [improvement][semantic-fe] Fixing the abnormal state of the popup when creating a model.

* [improvement][semantic-fe] Adding permission logic for bulk operations in the indicator market.

* [improvement][semantic-fe] Adding the ability to download and transpose data.

* [improvement][semantic-fe] Fixing the initialization issue with the date selection component in the indicator details page when switching time granularity.

* [improvement][semantic-fe] Fixing the logic error in the dimension value setting.

* [improvement][semantic-fe] Fixing the synchronization issue with the question and answer settings information.

* [improvement][semantic-fe] Optimizing the canvas functionality for better performance and user experience.

* [improvement][semantic-fe] Optimizing the update process for drawing model relationship edges in the canvas.

* [improvement][semantic-fe] Changing the line type for canvas connections.

* [improvement][semantic-fe] Replacing the initialization variable from "semantic" to "headless".

* [improvement][semantic-fe] Fixing the missing migration issue for default drill-down dimension configuration in model editing. Additionally, optimizing the data retrieval method for initializing fields in the model.

* [improvement][semantic-fe] Updating the logic for the fieldName.

* [improvement][semantic-fe] Adjusting the position of the metrics tab.

* [improvement][semantic-fe] Changing the 字段名称 to 英文名称.

* [improvement][semantic-fe] Fix metric measurement deletion.

* [improvement][semantic-fe] UI optimization for metric details page.

* [improvement][semantic-fe] UI optimization for metric details page.

* [improvement][semantic-fe] UI adjustment for metric details page.

* [improvement][semantic-fe] The granularity field in the time type of model editing now supports setting it as empty.

* [improvement][semantic-fe] Added field type and metric type to the metric creation options.

* [improvement][semantic-fe] The organization structure selection feature has been added to the permission management.
2024-01-19 16:26:35 +08:00
lexluo09
cbf38ed785 (improvement)(chat) Remove QueryReq parameter from QueryContext. (#656) 2024-01-19 16:17:31 +08:00
Scott
f017f41201 [chore]add ci and ci badge configure (#653) 2024-01-19 16:14:48 +08:00
LXW
b40670b0e3 (improvement)(headless) (improvement)(headless) Supports creating new metric by fields and metrics and convert struct to sql (#654)
Co-authored-by: jolunoluo
2024-01-19 14:51:13 +08:00
tristanliu
7af5afc3eb [improvement][semantic-fe] Added field type and metric type to the metric creation options. (#655)
* [improvement][semantic-fe] Add model alias setting & Add view permission restrictions to the model permission management tab.
[improvement][semantic-fe] Add permission control to the action buttons for the main domain; apply high sensitivity filtering to the authorization of metrics/dimensions.
[improvement][semantic-fe] Optimize the editing mode in the dimension/metric/datasource components to use the modelId stored in the database for data, instead of relying on the data from the state manager.

* [improvement][semantic-fe] Add time granularity setting in the data source configuration.

* [improvement][semantic-fe] Dictionary import for dimension values supported in Q&A visibility

* [improvement][semantic-fe] Modification of data source creation prompt wording"

* [improvement][semantic-fe] metric market experience optimization

* [improvement][semantic-fe] enhance the analysis of metric trends

* [improvement][semantic-fe] optimize the presentation of metric trend permissions

* [improvement][semantic-fe] add metric trend download functionality

* [improvement][semantic-fe] fix the dimension initialization issue in metric correlation

* [improvement][semantic-fe] Fix the issue of database changes not taking effect when creating based on an SQL data source.

* [improvement][semantic-fe] Optimizing pagination logic and some CSS styles

* [improvement][semantic-fe] Fixing the API for the indicator list by changing "current" to "pageNum"

* [improvement][semantic-fe] Fixing the default value setting for the indicator list

* [improvement][semantic-fe] Adding batch operations for indicators/dimensions/models

* [improvement][semantic-fe] Replacing the single status update API for indicators/dimensions with a batch update API

* [improvement][semantic-fe] Redesigning the indicator homepage to incorporate trend charts and table functionality for indicators

* [improvement][semantic-fe] Optimizing the logic for setting dimension values and editing data sources, and adding system settings functionality

* [improvement][semantic-fe] Upgrading antd version to 5.x, extracting the batch operation button component, optimizing the interaction for system settings, and expanding the configuration generation types for list-to-select component.

* [improvement][semantic-fe] Adding the ability to filter dimensions based on whether they are tags or not.

* [improvement][semantic-fe] Adding the ability to edit relationships between models in the canvas.

* [improvement][semantic-fe] Updating the datePicker component to use dayjs instead.

* [improvement][semantic-fe] Fixing the issue with passing the model ID for dimensions in the indicator market.

* [improvement][semantic-fe] Fixing the abnormal state of the popup when creating a model.

* [improvement][semantic-fe] Adding permission logic for bulk operations in the indicator market.

* [improvement][semantic-fe] Adding the ability to download and transpose data.

* [improvement][semantic-fe] Fixing the initialization issue with the date selection component in the indicator details page when switching time granularity.

* [improvement][semantic-fe] Fixing the logic error in the dimension value setting.

* [improvement][semantic-fe] Fixing the synchronization issue with the question and answer settings information.

* [improvement][semantic-fe] Optimizing the canvas functionality for better performance and user experience.

* [improvement][semantic-fe] Optimizing the update process for drawing model relationship edges in the canvas.

* [improvement][semantic-fe] Changing the line type for canvas connections.

* [improvement][semantic-fe] Replacing the initialization variable from "semantic" to "headless".

* [improvement][semantic-fe] Fixing the missing migration issue for default drill-down dimension configuration in model editing. Additionally, optimizing the data retrieval method for initializing fields in the model.

* [improvement][semantic-fe] Updating the logic for the fieldName.

* [improvement][semantic-fe] Adjusting the position of the metrics tab.

* [improvement][semantic-fe] Changing the 字段名称 to 英文名称.

* [improvement][semantic-fe] Fix metric measurement deletion.

* [improvement][semantic-fe] UI optimization for metric details page.

* [improvement][semantic-fe] UI optimization for metric details page.

* [improvement][semantic-fe] UI adjustment for metric details page.

* [improvement][semantic-fe] The granularity field in the time type of model editing now supports setting it as empty.

* [improvement][semantic-fe] Added field type and metric type to the metric creation options.
2024-01-19 14:46:41 +08:00
lexluo09
0abbd83f51 (improvement)(headless) Fix checkstyle issues and optimize unused code in QueryServiceImpl (#652) 2024-01-19 11:23:57 +08:00
lexluo09
3e77fc3069 (improvement)(headless) Headless has preliminarily completed the abstraction of QueryCache, QueryParser, QueryPlanner, and QueryExecutor. (#651) 2024-01-18 22:39:58 +08:00
jipeli
b019f4d9bb (improvement)(headless) parser add model field (#650) 2024-01-18 21:52:36 +08:00
Scott
90f9da162e (fix)(chat):skip null values to avoid NPE (#645) 2024-01-18 19:56:03 +08:00
lexluo09
a06a1fa898 (improvement)(headless) Remove ItemDateResp and databaseResp from SemanticModel. (#648) 2024-01-18 16:30:18 +08:00
lexluo09
dfb8e3a427 (improvement)(headless) Refactor HeadlessModel-related code, remove modelSchemaResps. (#647) 2024-01-18 11:41:27 +08:00
jipeli
d4eecc1bf8 (improvement)(headless) adapter for derived metrics (#646) 2024-01-18 11:11:56 +08:00
lexluo09
71c491a80d (improvement)(headless) Extend data source parameters (placeholder and require) (#643) 2024-01-17 21:23:03 +08:00
williamhliu
93c3ce1631 (feature)(chat-sdk) modify sql tip (#642) 2024-01-17 20:14:40 +08:00
LXW
c181ce6945 (improvement)(headless) fix compile error in metric test (#641)
Co-authored-by: jolunoluo
2024-01-17 16:48:14 +08:00
LXW
07e6924cfd (improvement)(headless) Supports creating new metric by fields and metrics (#639)
Co-authored-by: jolunoluo
2024-01-17 14:21:48 +08:00
williamhliu
b2beecb5b8 (fix)(chat-sdk) fix error when showcase return null (#638) 2024-01-17 13:59:57 +08:00
williamhliu
eb08667d90 (fix)(chat-sdk) fix error when showcase return null (#636) 2024-01-17 13:56:01 +08:00
lexluo09
74b89a9430 (improvement)(chat) Extend support for PostgreSQL data source. (#635) 2024-01-17 12:55:28 +08:00
jipeli
7707179faa [fix][headless] fix struct parse error (#633) 2024-01-16 23:04:00 +08:00
Scott
af103f3aa3 (common)(fix):add quarter date period (#632) 2024-01-16 19:21:35 +08:00
LXW
7f65057a0f (fix)(headless) fix filter for measure (#626)
Co-authored-by: jolunoluo
2024-01-15 16:10:54 +08:00
Scott
a6818fb6ff (common)(fix):add recent date calculate if period is year (#625) 2024-01-15 15:56:09 +08:00
tristanliu
e7d654f150 [improvement][semantic-fe] The granularity field in the time type of model editing now supports setting it as empty. (#623)
* [improvement][semantic-fe] Add model alias setting & Add view permission restrictions to the model permission management tab.
[improvement][semantic-fe] Add permission control to the action buttons for the main domain; apply high sensitivity filtering to the authorization of metrics/dimensions.
[improvement][semantic-fe] Optimize the editing mode in the dimension/metric/datasource components to use the modelId stored in the database for data, instead of relying on the data from the state manager.

* [improvement][semantic-fe] Add time granularity setting in the data source configuration.

* [improvement][semantic-fe] Dictionary import for dimension values supported in Q&A visibility

* [improvement][semantic-fe] Modification of data source creation prompt wording"

* [improvement][semantic-fe] metric market experience optimization

* [improvement][semantic-fe] enhance the analysis of metric trends

* [improvement][semantic-fe] optimize the presentation of metric trend permissions

* [improvement][semantic-fe] add metric trend download functionality

* [improvement][semantic-fe] fix the dimension initialization issue in metric correlation

* [improvement][semantic-fe] Fix the issue of database changes not taking effect when creating based on an SQL data source.

* [improvement][semantic-fe] Optimizing pagination logic and some CSS styles

* [improvement][semantic-fe] Fixing the API for the indicator list by changing "current" to "pageNum"

* [improvement][semantic-fe] Fixing the default value setting for the indicator list

* [improvement][semantic-fe] Adding batch operations for indicators/dimensions/models

* [improvement][semantic-fe] Replacing the single status update API for indicators/dimensions with a batch update API

* [improvement][semantic-fe] Redesigning the indicator homepage to incorporate trend charts and table functionality for indicators

* [improvement][semantic-fe] Optimizing the logic for setting dimension values and editing data sources, and adding system settings functionality

* [improvement][semantic-fe] Upgrading antd version to 5.x, extracting the batch operation button component, optimizing the interaction for system settings, and expanding the configuration generation types for list-to-select component.

* [improvement][semantic-fe] Adding the ability to filter dimensions based on whether they are tags or not.

* [improvement][semantic-fe] Adding the ability to edit relationships between models in the canvas.

* [improvement][semantic-fe] Updating the datePicker component to use dayjs instead.

* [improvement][semantic-fe] Fixing the issue with passing the model ID for dimensions in the indicator market.

* [improvement][semantic-fe] Fixing the abnormal state of the popup when creating a model.

* [improvement][semantic-fe] Adding permission logic for bulk operations in the indicator market.

* [improvement][semantic-fe] Adding the ability to download and transpose data.

* [improvement][semantic-fe] Fixing the initialization issue with the date selection component in the indicator details page when switching time granularity.

* [improvement][semantic-fe] Fixing the logic error in the dimension value setting.

* [improvement][semantic-fe] Fixing the synchronization issue with the question and answer settings information.

* [improvement][semantic-fe] Optimizing the canvas functionality for better performance and user experience.

* [improvement][semantic-fe] Optimizing the update process for drawing model relationship edges in the canvas.

* [improvement][semantic-fe] Changing the line type for canvas connections.

* [improvement][semantic-fe] Replacing the initialization variable from "semantic" to "headless".

* [improvement][semantic-fe] Fixing the missing migration issue for default drill-down dimension configuration in model editing. Additionally, optimizing the data retrieval method for initializing fields in the model.

* [improvement][semantic-fe] Updating the logic for the fieldName.

* [improvement][semantic-fe] Adjusting the position of the metrics tab.

* [improvement][semantic-fe] Changing the 字段名称 to 英文名称.

* [improvement][semantic-fe] Fix metric measurement deletion.

* [improvement][semantic-fe] UI optimization for metric details page.

* [improvement][semantic-fe] UI optimization for metric details page.

* [improvement][semantic-fe] UI adjustment for metric details page.

* [improvement][semantic-fe] The granularity field in the time type of model editing now supports setting it as empty.
2024-01-15 11:25:32 +08:00
jipeli
e29ecec0c9 (improvement)(headless) remove null collation config (#622) 2024-01-12 18:14:47 +08:00
Scott
dcc1f26542 (fix)(chat):queryByStruct use default metrics period (#621) 2024-01-12 17:00:53 +08:00
jipeli
3436b36552 (improvement)(headless) add simplify optimizer enable conf (#620) 2024-01-11 21:37:35 +08:00
jipeli
4322ae42ac (improvement)(headless) simplify optimizer enable conf (#619) 2024-01-11 20:14:03 +08:00
jipeli
e9c7237794 (improvement)(headless) add simplify optimizer (#618) 2024-01-11 17:11:06 +08:00
tristanliu
3a5349c916 [improvement][semantic-fe] UI adjustment for metric details page. (#617)
* [improvement][semantic-fe] Add model alias setting & Add view permission restrictions to the model permission management tab.
[improvement][semantic-fe] Add permission control to the action buttons for the main domain; apply high sensitivity filtering to the authorization of metrics/dimensions.
[improvement][semantic-fe] Optimize the editing mode in the dimension/metric/datasource components to use the modelId stored in the database for data, instead of relying on the data from the state manager.

* [improvement][semantic-fe] Add time granularity setting in the data source configuration.

* [improvement][semantic-fe] Dictionary import for dimension values supported in Q&A visibility

* [improvement][semantic-fe] Modification of data source creation prompt wording"

* [improvement][semantic-fe] metric market experience optimization

* [improvement][semantic-fe] enhance the analysis of metric trends

* [improvement][semantic-fe] optimize the presentation of metric trend permissions

* [improvement][semantic-fe] add metric trend download functionality

* [improvement][semantic-fe] fix the dimension initialization issue in metric correlation

* [improvement][semantic-fe] Fix the issue of database changes not taking effect when creating based on an SQL data source.

* [improvement][semantic-fe] Optimizing pagination logic and some CSS styles

* [improvement][semantic-fe] Fixing the API for the indicator list by changing "current" to "pageNum"

* [improvement][semantic-fe] Fixing the default value setting for the indicator list

* [improvement][semantic-fe] Adding batch operations for indicators/dimensions/models

* [improvement][semantic-fe] Replacing the single status update API for indicators/dimensions with a batch update API

* [improvement][semantic-fe] Redesigning the indicator homepage to incorporate trend charts and table functionality for indicators

* [improvement][semantic-fe] Optimizing the logic for setting dimension values and editing data sources, and adding system settings functionality

* [improvement][semantic-fe] Upgrading antd version to 5.x, extracting the batch operation button component, optimizing the interaction for system settings, and expanding the configuration generation types for list-to-select component.

* [improvement][semantic-fe] Adding the ability to filter dimensions based on whether they are tags or not.

* [improvement][semantic-fe] Adding the ability to edit relationships between models in the canvas.

* [improvement][semantic-fe] Updating the datePicker component to use dayjs instead.

* [improvement][semantic-fe] Fixing the issue with passing the model ID for dimensions in the indicator market.

* [improvement][semantic-fe] Fixing the abnormal state of the popup when creating a model.

* [improvement][semantic-fe] Adding permission logic for bulk operations in the indicator market.

* [improvement][semantic-fe] Adding the ability to download and transpose data.

* [improvement][semantic-fe] Fixing the initialization issue with the date selection component in the indicator details page when switching time granularity.

* [improvement][semantic-fe] Fixing the logic error in the dimension value setting.

* [improvement][semantic-fe] Fixing the synchronization issue with the question and answer settings information.

* [improvement][semantic-fe] Optimizing the canvas functionality for better performance and user experience.

* [improvement][semantic-fe] Optimizing the update process for drawing model relationship edges in the canvas.

* [improvement][semantic-fe] Changing the line type for canvas connections.

* [improvement][semantic-fe] Replacing the initialization variable from "semantic" to "headless".

* [improvement][semantic-fe] Fixing the missing migration issue for default drill-down dimension configuration in model editing. Additionally, optimizing the data retrieval method for initializing fields in the model.

* [improvement][semantic-fe] Updating the logic for the fieldName.

* [improvement][semantic-fe] Adjusting the position of the metrics tab.

* [improvement][semantic-fe] Changing the 字段名称 to 英文名称.

* [improvement][semantic-fe] Fix metric measurement deletion.

* [improvement][semantic-fe] UI optimization for metric details page.

* [improvement][semantic-fe] UI optimization for metric details page.

* [improvement][semantic-fe] UI adjustment for metric details page.
2024-01-11 15:20:38 +08:00
mainmain
1e93282c9f [improvement] llm supports all models (#615) 2024-01-10 22:01:21 +08:00
LXW
9c8039c499 (improvement)(headless) metric list support filter by hasCollect (#613)
Co-authored-by: jolunoluo
2024-01-10 14:12:39 +08:00
LXW
61da52650c (improvement)(headless) Supplement the model name when obtaining metric information (#612)
Co-authored-by: jolunoluo
2024-01-10 10:13:45 +08:00
tristanliu
87a60eeba2 [improvement][semantic-fe] UI optimization for metric details page. (#611)
* [improvement][semantic-fe] Add model alias setting & Add view permission restrictions to the model permission management tab.
[improvement][semantic-fe] Add permission control to the action buttons for the main domain; apply high sensitivity filtering to the authorization of metrics/dimensions.
[improvement][semantic-fe] Optimize the editing mode in the dimension/metric/datasource components to use the modelId stored in the database for data, instead of relying on the data from the state manager.

* [improvement][semantic-fe] Add time granularity setting in the data source configuration.

* [improvement][semantic-fe] Dictionary import for dimension values supported in Q&A visibility

* [improvement][semantic-fe] Modification of data source creation prompt wording"

* [improvement][semantic-fe] metric market experience optimization

* [improvement][semantic-fe] enhance the analysis of metric trends

* [improvement][semantic-fe] optimize the presentation of metric trend permissions

* [improvement][semantic-fe] add metric trend download functionality

* [improvement][semantic-fe] fix the dimension initialization issue in metric correlation

* [improvement][semantic-fe] Fix the issue of database changes not taking effect when creating based on an SQL data source.

* [improvement][semantic-fe] Optimizing pagination logic and some CSS styles

* [improvement][semantic-fe] Fixing the API for the indicator list by changing "current" to "pageNum"

* [improvement][semantic-fe] Fixing the default value setting for the indicator list

* [improvement][semantic-fe] Adding batch operations for indicators/dimensions/models

* [improvement][semantic-fe] Replacing the single status update API for indicators/dimensions with a batch update API

* [improvement][semantic-fe] Redesigning the indicator homepage to incorporate trend charts and table functionality for indicators

* [improvement][semantic-fe] Optimizing the logic for setting dimension values and editing data sources, and adding system settings functionality

* [improvement][semantic-fe] Upgrading antd version to 5.x, extracting the batch operation button component, optimizing the interaction for system settings, and expanding the configuration generation types for list-to-select component.

* [improvement][semantic-fe] Adding the ability to filter dimensions based on whether they are tags or not.

* [improvement][semantic-fe] Adding the ability to edit relationships between models in the canvas.

* [improvement][semantic-fe] Updating the datePicker component to use dayjs instead.

* [improvement][semantic-fe] Fixing the issue with passing the model ID for dimensions in the indicator market.

* [improvement][semantic-fe] Fixing the abnormal state of the popup when creating a model.

* [improvement][semantic-fe] Adding permission logic for bulk operations in the indicator market.

* [improvement][semantic-fe] Adding the ability to download and transpose data.

* [improvement][semantic-fe] Fixing the initialization issue with the date selection component in the indicator details page when switching time granularity.

* [improvement][semantic-fe] Fixing the logic error in the dimension value setting.

* [improvement][semantic-fe] Fixing the synchronization issue with the question and answer settings information.

* [improvement][semantic-fe] Optimizing the canvas functionality for better performance and user experience.

* [improvement][semantic-fe] Optimizing the update process for drawing model relationship edges in the canvas.

* [improvement][semantic-fe] Changing the line type for canvas connections.

* [improvement][semantic-fe] Replacing the initialization variable from "semantic" to "headless".

* [improvement][semantic-fe] Fixing the missing migration issue for default drill-down dimension configuration in model editing. Additionally, optimizing the data retrieval method for initializing fields in the model.

* [improvement][semantic-fe] Updating the logic for the fieldName.

* [improvement][semantic-fe] Adjusting the position of the metrics tab.

* [improvement][semantic-fe] Changing the 字段名称 to 英文名称.

* [improvement][semantic-fe] Fix metric measurement deletion.

* [improvement][semantic-fe] UI optimization for metric details page.
2024-01-09 20:31:21 +08:00
tristanliu
052e217c8c [improvement][semantic-fe] UI optimization for metric details page. (#610)
* [improvement][semantic-fe] Add model alias setting & Add view permission restrictions to the model permission management tab.
[improvement][semantic-fe] Add permission control to the action buttons for the main domain; apply high sensitivity filtering to the authorization of metrics/dimensions.
[improvement][semantic-fe] Optimize the editing mode in the dimension/metric/datasource components to use the modelId stored in the database for data, instead of relying on the data from the state manager.

* [improvement][semantic-fe] Add time granularity setting in the data source configuration.

* [improvement][semantic-fe] Dictionary import for dimension values supported in Q&A visibility

* [improvement][semantic-fe] Modification of data source creation prompt wording"

* [improvement][semantic-fe] metric market experience optimization

* [improvement][semantic-fe] enhance the analysis of metric trends

* [improvement][semantic-fe] optimize the presentation of metric trend permissions

* [improvement][semantic-fe] add metric trend download functionality

* [improvement][semantic-fe] fix the dimension initialization issue in metric correlation

* [improvement][semantic-fe] Fix the issue of database changes not taking effect when creating based on an SQL data source.

* [improvement][semantic-fe] Optimizing pagination logic and some CSS styles

* [improvement][semantic-fe] Fixing the API for the indicator list by changing "current" to "pageNum"

* [improvement][semantic-fe] Fixing the default value setting for the indicator list

* [improvement][semantic-fe] Adding batch operations for indicators/dimensions/models

* [improvement][semantic-fe] Replacing the single status update API for indicators/dimensions with a batch update API

* [improvement][semantic-fe] Redesigning the indicator homepage to incorporate trend charts and table functionality for indicators

* [improvement][semantic-fe] Optimizing the logic for setting dimension values and editing data sources, and adding system settings functionality

* [improvement][semantic-fe] Upgrading antd version to 5.x, extracting the batch operation button component, optimizing the interaction for system settings, and expanding the configuration generation types for list-to-select component.

* [improvement][semantic-fe] Adding the ability to filter dimensions based on whether they are tags or not.

* [improvement][semantic-fe] Adding the ability to edit relationships between models in the canvas.

* [improvement][semantic-fe] Updating the datePicker component to use dayjs instead.

* [improvement][semantic-fe] Fixing the issue with passing the model ID for dimensions in the indicator market.

* [improvement][semantic-fe] Fixing the abnormal state of the popup when creating a model.

* [improvement][semantic-fe] Adding permission logic for bulk operations in the indicator market.

* [improvement][semantic-fe] Adding the ability to download and transpose data.

* [improvement][semantic-fe] Fixing the initialization issue with the date selection component in the indicator details page when switching time granularity.

* [improvement][semantic-fe] Fixing the logic error in the dimension value setting.

* [improvement][semantic-fe] Fixing the synchronization issue with the question and answer settings information.

* [improvement][semantic-fe] Optimizing the canvas functionality for better performance and user experience.

* [improvement][semantic-fe] Optimizing the update process for drawing model relationship edges in the canvas.

* [improvement][semantic-fe] Changing the line type for canvas connections.

* [improvement][semantic-fe] Replacing the initialization variable from "semantic" to "headless".

* [improvement][semantic-fe] Fixing the missing migration issue for default drill-down dimension configuration in model editing. Additionally, optimizing the data retrieval method for initializing fields in the model.

* [improvement][semantic-fe] Updating the logic for the fieldName.

* [improvement][semantic-fe] Adjusting the position of the metrics tab.

* [improvement][semantic-fe] Changing the 字段名称 to 英文名称.

* [improvement][semantic-fe] Fix metric measurement deletion.

* [improvement][semantic-fe] UI optimization for metric details page.
2024-01-09 20:28:58 +08:00
lexluo09
bbad302efd (improvement)(chat) Add the pluginList parameter to QueryContext. (#609) 2024-01-09 15:23:07 +08:00
LXW
ed54d7bae3 (improvement)(headless) Query metric/dimension by id in SchemaController and update api header check (#608)
* (improvement)(headless) Query metric/dimension by id in SchemaController and update api header check

* (improvement)(headless) Fix the issue where metrics were not recognized when the result field contained an aggregate function.

---------

Co-authored-by: jolunoluo
2024-01-09 10:26:10 +08:00
tristanliu
0a6160272b [improvement][semantic-fe] Fix metric measurement deletion. (#607)
* [improvement][semantic-fe] Add model alias setting & Add view permission restrictions to the model permission management tab.
[improvement][semantic-fe] Add permission control to the action buttons for the main domain; apply high sensitivity filtering to the authorization of metrics/dimensions.
[improvement][semantic-fe] Optimize the editing mode in the dimension/metric/datasource components to use the modelId stored in the database for data, instead of relying on the data from the state manager.

* [improvement][semantic-fe] Add time granularity setting in the data source configuration.

* [improvement][semantic-fe] Dictionary import for dimension values supported in Q&A visibility

* [improvement][semantic-fe] Modification of data source creation prompt wording"

* [improvement][semantic-fe] metric market experience optimization

* [improvement][semantic-fe] enhance the analysis of metric trends

* [improvement][semantic-fe] optimize the presentation of metric trend permissions

* [improvement][semantic-fe] add metric trend download functionality

* [improvement][semantic-fe] fix the dimension initialization issue in metric correlation

* [improvement][semantic-fe] Fix the issue of database changes not taking effect when creating based on an SQL data source.

* [improvement][semantic-fe] Optimizing pagination logic and some CSS styles

* [improvement][semantic-fe] Fixing the API for the indicator list by changing "current" to "pageNum"

* [improvement][semantic-fe] Fixing the default value setting for the indicator list

* [improvement][semantic-fe] Adding batch operations for indicators/dimensions/models

* [improvement][semantic-fe] Replacing the single status update API for indicators/dimensions with a batch update API

* [improvement][semantic-fe] Redesigning the indicator homepage to incorporate trend charts and table functionality for indicators

* [improvement][semantic-fe] Optimizing the logic for setting dimension values and editing data sources, and adding system settings functionality

* [improvement][semantic-fe] Upgrading antd version to 5.x, extracting the batch operation button component, optimizing the interaction for system settings, and expanding the configuration generation types for list-to-select component.

* [improvement][semantic-fe] Adding the ability to filter dimensions based on whether they are tags or not.

* [improvement][semantic-fe] Adding the ability to edit relationships between models in the canvas.

* [improvement][semantic-fe] Updating the datePicker component to use dayjs instead.

* [improvement][semantic-fe] Fixing the issue with passing the model ID for dimensions in the indicator market.

* [improvement][semantic-fe] Fixing the abnormal state of the popup when creating a model.

* [improvement][semantic-fe] Adding permission logic for bulk operations in the indicator market.

* [improvement][semantic-fe] Adding the ability to download and transpose data.

* [improvement][semantic-fe] Fixing the initialization issue with the date selection component in the indicator details page when switching time granularity.

* [improvement][semantic-fe] Fixing the logic error in the dimension value setting.

* [improvement][semantic-fe] Fixing the synchronization issue with the question and answer settings information.

* [improvement][semantic-fe] Optimizing the canvas functionality for better performance and user experience.

* [improvement][semantic-fe] Optimizing the update process for drawing model relationship edges in the canvas.

* [improvement][semantic-fe] Changing the line type for canvas connections.

* [improvement][semantic-fe] Replacing the initialization variable from "semantic" to "headless".

* [improvement][semantic-fe] Fixing the missing migration issue for default drill-down dimension configuration in model editing. Additionally, optimizing the data retrieval method for initializing fields in the model.

* [improvement][semantic-fe] Updating the logic for the fieldName.

* [improvement][semantic-fe] Adjusting the position of the metrics tab.

* [improvement][semantic-fe] Changing the 字段名称 to 英文名称.

* [improvement][semantic-fe] Fix metric measurement deletion.
2024-01-08 20:36:41 +08:00
lexluo09
ad8079e058 (improvement)(chat) The system settings module supports setting the number of returned parses. (#606) 2024-01-08 19:35:28 +08:00
lexluo09
8eef11f342 (improvement)(chat) Fix the thread safety issue in parallel insertion of nature. (#603) 2024-01-08 17:08:38 +08:00
lexluo09
b55b4c130e (improvement)(chat) Add queryFilters setting to queryContext. (#601) 2024-01-05 12:04:55 +08:00
tristanliu
0408f0fe9a [improvement][semantic-fe] Changing the 字段名称 to 英文名称. (#600)
* [improvement][semantic-fe] Add model alias setting & Add view permission restrictions to the model permission management tab.
[improvement][semantic-fe] Add permission control to the action buttons for the main domain; apply high sensitivity filtering to the authorization of metrics/dimensions.
[improvement][semantic-fe] Optimize the editing mode in the dimension/metric/datasource components to use the modelId stored in the database for data, instead of relying on the data from the state manager.

* [improvement][semantic-fe] Add time granularity setting in the data source configuration.

* [improvement][semantic-fe] Dictionary import for dimension values supported in Q&A visibility

* [improvement][semantic-fe] Modification of data source creation prompt wording"

* [improvement][semantic-fe] metric market experience optimization

* [improvement][semantic-fe] enhance the analysis of metric trends

* [improvement][semantic-fe] optimize the presentation of metric trend permissions

* [improvement][semantic-fe] add metric trend download functionality

* [improvement][semantic-fe] fix the dimension initialization issue in metric correlation

* [improvement][semantic-fe] Fix the issue of database changes not taking effect when creating based on an SQL data source.

* [improvement][semantic-fe] Optimizing pagination logic and some CSS styles

* [improvement][semantic-fe] Fixing the API for the indicator list by changing "current" to "pageNum"

* [improvement][semantic-fe] Fixing the default value setting for the indicator list

* [improvement][semantic-fe] Adding batch operations for indicators/dimensions/models

* [improvement][semantic-fe] Replacing the single status update API for indicators/dimensions with a batch update API

* [improvement][semantic-fe] Redesigning the indicator homepage to incorporate trend charts and table functionality for indicators

* [improvement][semantic-fe] Optimizing the logic for setting dimension values and editing data sources, and adding system settings functionality

* [improvement][semantic-fe] Upgrading antd version to 5.x, extracting the batch operation button component, optimizing the interaction for system settings, and expanding the configuration generation types for list-to-select component.

* [improvement][semantic-fe] Adding the ability to filter dimensions based on whether they are tags or not.

* [improvement][semantic-fe] Adding the ability to edit relationships between models in the canvas.

* [improvement][semantic-fe] Updating the datePicker component to use dayjs instead.

* [improvement][semantic-fe] Fixing the issue with passing the model ID for dimensions in the indicator market.

* [improvement][semantic-fe] Fixing the abnormal state of the popup when creating a model.

* [improvement][semantic-fe] Adding permission logic for bulk operations in the indicator market.

* [improvement][semantic-fe] Adding the ability to download and transpose data.

* [improvement][semantic-fe] Fixing the initialization issue with the date selection component in the indicator details page when switching time granularity.

* [improvement][semantic-fe] Fixing the logic error in the dimension value setting.

* [improvement][semantic-fe] Fixing the synchronization issue with the question and answer settings information.

* [improvement][semantic-fe] Optimizing the canvas functionality for better performance and user experience.

* [improvement][semantic-fe] Optimizing the update process for drawing model relationship edges in the canvas.

* [improvement][semantic-fe] Changing the line type for canvas connections.

* [improvement][semantic-fe] Replacing the initialization variable from "semantic" to "headless".

* [improvement][semantic-fe] Fixing the missing migration issue for default drill-down dimension configuration in model editing. Additionally, optimizing the data retrieval method for initializing fields in the model.

* [improvement][semantic-fe] Updating the logic for the fieldName.

* [improvement][semantic-fe] Adjusting the position of the metrics tab.

* [improvement][semantic-fe] Changing the 字段名称 to 英文名称.
2024-01-05 11:16:21 +08:00
williamhliu
062f7340e5 (feature)(chat-sdk) add Few-shot examples (#599) 2024-01-05 10:17:17 +08:00
jipeli
72bd79fe73 Squashed commit of the following: (#598) 2024-01-04 22:00:55 +08:00
lexluo09
602b9547b8 (improvement)(chat) Make it compatible with the case when the agent is empty. (#596) 2024-01-04 21:39:56 +08:00
lexluo09
ade96c3adc (improvement)(chat) Clean up unused code and optimize some code and make the tests pass (#595) 2024-01-04 18:22:40 +08:00
lexluo09
023e84c420 (improvement)(chat) Split chat into three modules: server, api, and core. (#594) 2024-01-04 16:56:49 +08:00
LXW
0858c13365 (improvement)(headless) Update server and core, server calls core one-way (#592)
* (improvement)(headless) Update server and core, server calls core one-way
* (improvement)(Auth) When obtaining the user information, determine whether the user is a system admin.
---------

Co-authored-by: jolunoluo
2024-01-04 12:08:12 +08:00
tristanliu
7acb48da0e [improvement][semantic-fe] Adjusting the position of the metrics tab. (#591)
* [improvement][semantic-fe] Add model alias setting & Add view permission restrictions to the model permission management tab.
[improvement][semantic-fe] Add permission control to the action buttons for the main domain; apply high sensitivity filtering to the authorization of metrics/dimensions.
[improvement][semantic-fe] Optimize the editing mode in the dimension/metric/datasource components to use the modelId stored in the database for data, instead of relying on the data from the state manager.

* [improvement][semantic-fe] Add time granularity setting in the data source configuration.

* [improvement][semantic-fe] Dictionary import for dimension values supported in Q&A visibility

* [improvement][semantic-fe] Modification of data source creation prompt wording"

* [improvement][semantic-fe] metric market experience optimization

* [improvement][semantic-fe] enhance the analysis of metric trends

* [improvement][semantic-fe] optimize the presentation of metric trend permissions

* [improvement][semantic-fe] add metric trend download functionality

* [improvement][semantic-fe] fix the dimension initialization issue in metric correlation

* [improvement][semantic-fe] Fix the issue of database changes not taking effect when creating based on an SQL data source.

* [improvement][semantic-fe] Optimizing pagination logic and some CSS styles

* [improvement][semantic-fe] Fixing the API for the indicator list by changing "current" to "pageNum"

* [improvement][semantic-fe] Fixing the default value setting for the indicator list

* [improvement][semantic-fe] Adding batch operations for indicators/dimensions/models

* [improvement][semantic-fe] Replacing the single status update API for indicators/dimensions with a batch update API

* [improvement][semantic-fe] Redesigning the indicator homepage to incorporate trend charts and table functionality for indicators

* [improvement][semantic-fe] Optimizing the logic for setting dimension values and editing data sources, and adding system settings functionality

* [improvement][semantic-fe] Upgrading antd version to 5.x, extracting the batch operation button component, optimizing the interaction for system settings, and expanding the configuration generation types for list-to-select component.

* [improvement][semantic-fe] Adding the ability to filter dimensions based on whether they are tags or not.

* [improvement][semantic-fe] Adding the ability to edit relationships between models in the canvas.

* [improvement][semantic-fe] Updating the datePicker component to use dayjs instead.

* [improvement][semantic-fe] Fixing the issue with passing the model ID for dimensions in the indicator market.

* [improvement][semantic-fe] Fixing the abnormal state of the popup when creating a model.

* [improvement][semantic-fe] Adding permission logic for bulk operations in the indicator market.

* [improvement][semantic-fe] Adding the ability to download and transpose data.

* [improvement][semantic-fe] Fixing the initialization issue with the date selection component in the indicator details page when switching time granularity.

* [improvement][semantic-fe] Fixing the logic error in the dimension value setting.

* [improvement][semantic-fe] Fixing the synchronization issue with the question and answer settings information.

* [improvement][semantic-fe] Optimizing the canvas functionality for better performance and user experience.

* [improvement][semantic-fe] Optimizing the update process for drawing model relationship edges in the canvas.

* [improvement][semantic-fe] Changing the line type for canvas connections.

* [improvement][semantic-fe] Replacing the initialization variable from "semantic" to "headless".

* [improvement][semantic-fe] Fixing the missing migration issue for default drill-down dimension configuration in model editing. Additionally, optimizing the data retrieval method for initializing fields in the model.

* [improvement][semantic-fe] Updating the logic for the fieldName.

* [improvement][semantic-fe] Adjusting the position of the metrics tab.
2024-01-03 19:56:59 +08:00
LXW
52fea5311d (improvement)(headless) Merge server and core in headless-api (#590)
(improvement)(headless) Merge server and core in headless-api

---------

Co-authored-by: jolunoluo
2024-01-02 20:08:23 +08:00
lexluo09
d72166944c (improvement)(chat) Add 'Few-shot Examples' display to the Chat chart. (#589) 2024-01-02 18:07:15 +08:00
LXW
e7f13572d7 (improvement)(headless) Modify the headless module structure to api, core and server (#588)
Co-authored-by: jolunoluo
2024-01-02 16:43:28 +08:00
codescracker
af1c560cc4 (fix)(chat):fix typo in s2sql and add prompt to output. (#581)
* 1.refactor the retrieval module. 2.refactor the http service module. 3.upgrade text2sql output format the parse for absolute time related expression in query.

* fix bug.

* upgrade the config module, now support config llm suppoted by langchain.

* fix conflicts.

* update text2sql config reload to be compitable with new config format.

* modify default config.

* 1.add self-consistency feature for text2sql. 2.upgrade llm api call from sync to async. 3.refactor text2sql module. 4. refactor semantical retriever modules.

* merege with upstream master

* add general retrieve service.

* add api service for sql_agent for crud opereations of few-shots examples.

* modify requirements

* add auto-cot feature

* 1. output log to a fixed log file.  2.allow few-shots examples tied to data model, and add strategy that extend examples when retrieved examples tied to a data model is not enough. 3. fix misformat in s2ql args.

* add prior_ext to output.

* fix type in in s2sql

* add prompt to output.

---------

Co-authored-by: shaoweigong <shaoweigong@tencent.com>
2024-01-02 16:35:33 +08:00
tristanliu
49f0a4dc1d [improvement][semantic-fe] Updating the logic for the fieldName. (#587)
* [improvement][semantic-fe] Add model alias setting & Add view permission restrictions to the model permission management tab.
[improvement][semantic-fe] Add permission control to the action buttons for the main domain; apply high sensitivity filtering to the authorization of metrics/dimensions.
[improvement][semantic-fe] Optimize the editing mode in the dimension/metric/datasource components to use the modelId stored in the database for data, instead of relying on the data from the state manager.

* [improvement][semantic-fe] Add time granularity setting in the data source configuration.

* [improvement][semantic-fe] Dictionary import for dimension values supported in Q&A visibility

* [improvement][semantic-fe] Modification of data source creation prompt wording"

* [improvement][semantic-fe] metric market experience optimization

* [improvement][semantic-fe] enhance the analysis of metric trends

* [improvement][semantic-fe] optimize the presentation of metric trend permissions

* [improvement][semantic-fe] add metric trend download functionality

* [improvement][semantic-fe] fix the dimension initialization issue in metric correlation

* [improvement][semantic-fe] Fix the issue of database changes not taking effect when creating based on an SQL data source.

* [improvement][semantic-fe] Optimizing pagination logic and some CSS styles

* [improvement][semantic-fe] Fixing the API for the indicator list by changing "current" to "pageNum"

* [improvement][semantic-fe] Fixing the default value setting for the indicator list

* [improvement][semantic-fe] Adding batch operations for indicators/dimensions/models

* [improvement][semantic-fe] Replacing the single status update API for indicators/dimensions with a batch update API

* [improvement][semantic-fe] Redesigning the indicator homepage to incorporate trend charts and table functionality for indicators

* [improvement][semantic-fe] Optimizing the logic for setting dimension values and editing data sources, and adding system settings functionality

* [improvement][semantic-fe] Upgrading antd version to 5.x, extracting the batch operation button component, optimizing the interaction for system settings, and expanding the configuration generation types for list-to-select component.

* [improvement][semantic-fe] Adding the ability to filter dimensions based on whether they are tags or not.

* [improvement][semantic-fe] Adding the ability to edit relationships between models in the canvas.

* [improvement][semantic-fe] Updating the datePicker component to use dayjs instead.

* [improvement][semantic-fe] Fixing the issue with passing the model ID for dimensions in the indicator market.

* [improvement][semantic-fe] Fixing the abnormal state of the popup when creating a model.

* [improvement][semantic-fe] Adding permission logic for bulk operations in the indicator market.

* [improvement][semantic-fe] Adding the ability to download and transpose data.

* [improvement][semantic-fe] Fixing the initialization issue with the date selection component in the indicator details page when switching time granularity.

* [improvement][semantic-fe] Fixing the logic error in the dimension value setting.

* [improvement][semantic-fe] Fixing the synchronization issue with the question and answer settings information.

* [improvement][semantic-fe] Optimizing the canvas functionality for better performance and user experience.

* [improvement][semantic-fe] Optimizing the update process for drawing model relationship edges in the canvas.

* [improvement][semantic-fe] Changing the line type for canvas connections.

* [improvement][semantic-fe] Replacing the initialization variable from "semantic" to "headless".

* [improvement][semantic-fe] Fixing the missing migration issue for default drill-down dimension configuration in model editing. Additionally, optimizing the data retrieval method for initializing fields in the model.

* [improvement][semantic-fe] Updating the logic for the fieldName.
2024-01-02 10:14:58 +08:00
daikon
31f1fc315d (improvement)(headless) Remove materialization logic (#586)
Co-authored-by: kanedai <kanedai@tencent.com>
2023-12-29 23:10:54 +08:00
LXW
afa8fd74a6 (improvement)(headless) Update headless version in pom.xml (#585)
Co-authored-by: jolunoluo
2023-12-29 17:57:51 +08:00
jipeli
56b0f35250 (improvement)(headless) parseSql add option (#584) 2023-12-29 16:47:29 +08:00
LXW
af6c8cdbda (feature)(headless) Add a new headless-api module and Change old headless-api module to headless-common (#582)
* (improvement)(headless) Add headless-api module

* (improvement)(headless) Change old headless-api module to headless-common

---------

Co-authored-by: jolunoluo <jolunoluo@tencent.com>
2023-12-28 22:14:57 +08:00
jipeli
9e69002d70 [improvement][headless] fixed sql optimizer utf8 error (#580) 2023-12-28 18:32:21 +08:00
tristanliu
cd727663a5 [improvement][semantic-fe] Fixing the missing migration issue for default drill-down dimension configuration in model editing. Additionally, optimizing the data retrieval method for initializing fields in the model. (#579)
* [improvement][semantic-fe] Add model alias setting & Add view permission restrictions to the model permission management tab.
[improvement][semantic-fe] Add permission control to the action buttons for the main domain; apply high sensitivity filtering to the authorization of metrics/dimensions.
[improvement][semantic-fe] Optimize the editing mode in the dimension/metric/datasource components to use the modelId stored in the database for data, instead of relying on the data from the state manager.

* [improvement][semantic-fe] Add time granularity setting in the data source configuration.

* [improvement][semantic-fe] Dictionary import for dimension values supported in Q&A visibility

* [improvement][semantic-fe] Modification of data source creation prompt wording"

* [improvement][semantic-fe] metric market experience optimization

* [improvement][semantic-fe] enhance the analysis of metric trends

* [improvement][semantic-fe] optimize the presentation of metric trend permissions

* [improvement][semantic-fe] add metric trend download functionality

* [improvement][semantic-fe] fix the dimension initialization issue in metric correlation

* [improvement][semantic-fe] Fix the issue of database changes not taking effect when creating based on an SQL data source.

* [improvement][semantic-fe] Optimizing pagination logic and some CSS styles

* [improvement][semantic-fe] Fixing the API for the indicator list by changing "current" to "pageNum"

* [improvement][semantic-fe] Fixing the default value setting for the indicator list

* [improvement][semantic-fe] Adding batch operations for indicators/dimensions/models

* [improvement][semantic-fe] Replacing the single status update API for indicators/dimensions with a batch update API

* [improvement][semantic-fe] Redesigning the indicator homepage to incorporate trend charts and table functionality for indicators

* [improvement][semantic-fe] Optimizing the logic for setting dimension values and editing data sources, and adding system settings functionality

* [improvement][semantic-fe] Upgrading antd version to 5.x, extracting the batch operation button component, optimizing the interaction for system settings, and expanding the configuration generation types for list-to-select component.

* [improvement][semantic-fe] Adding the ability to filter dimensions based on whether they are tags or not.

* [improvement][semantic-fe] Adding the ability to edit relationships between models in the canvas.

* [improvement][semantic-fe] Updating the datePicker component to use dayjs instead.

* [improvement][semantic-fe] Fixing the issue with passing the model ID for dimensions in the indicator market.

* [improvement][semantic-fe] Fixing the abnormal state of the popup when creating a model.

* [improvement][semantic-fe] Adding permission logic for bulk operations in the indicator market.

* [improvement][semantic-fe] Adding the ability to download and transpose data.

* [improvement][semantic-fe] Fixing the initialization issue with the date selection component in the indicator details page when switching time granularity.

* [improvement][semantic-fe] Fixing the logic error in the dimension value setting.

* [improvement][semantic-fe] Fixing the synchronization issue with the question and answer settings information.

* [improvement][semantic-fe] Optimizing the canvas functionality for better performance and user experience.

* [improvement][semantic-fe] Optimizing the update process for drawing model relationship edges in the canvas.

* [improvement][semantic-fe] Changing the line type for canvas connections.

* [improvement][semantic-fe] Replacing the initialization variable from "semantic" to "headless".

* [improvement][semantic-fe] Fixing the missing migration issue for default drill-down dimension configuration in model editing. Additionally, optimizing the data retrieval method for initializing fields in the model.
2023-12-28 17:01:43 +08:00
LXW
d0289a3243 (improvement)(headless) Update schema-mysql.sql and sql-update.sql (#578)
Co-authored-by: jolunoluo
2023-12-28 10:05:56 +08:00
jipeli
327bab015e (improvement)(headless) fixed package loss error (#577) 2023-12-27 22:32:00 +08:00
lexluo09
f788249b1a (improvement)(chat) Modify getFilterExpression to support the datediff function. (#576) 2023-12-27 21:20:05 +08:00
lexluo09
6c8ebdfe1a (improvement)(chat) Modify the default configuration related to text2sql numbers. (#575) 2023-12-27 20:39:08 +08:00
codescracker
b706c4efb4 Add feature to s2sql that allow few-shots example tied to data model. (#571)
* 1.refactor the retrieval module. 2.refactor the http service module. 3.upgrade text2sql output format the parse for absolute time related expression in query.

* fix bug.

* upgrade the config module, now support config llm suppoted by langchain.

* fix conflicts.

* update text2sql config reload to be compitable with new config format.

* modify default config.

* 1.add self-consistency feature for text2sql. 2.upgrade llm api call from sync to async. 3.refactor text2sql module. 4. refactor semantical retriever modules.

* merege with upstream master

* add general retrieve service.

* add api service for sql_agent for crud opereations of few-shots examples.

* modify requirements

* add auto-cot feature

* 1. output log to a fixed log file.  2.allow few-shots examples tied to data model, and add strategy that extend examples when retrieved examples tied to a data model is not enough. 3. fix misformat in s2ql args.

* add prior_ext to output.

---------

Co-authored-by: shaoweigong <shaoweigong@tencent.com>
2023-12-27 19:39:50 +08:00
Chx
cf2b4bfb5c (fix)(project):modify the s2_app table DDL in sql-update.sql (#574)
Co-authored-by: caihaixin
2023-12-27 18:56:11 +08:00
LXW
40c86810bb (improvement)(chat) Modify query type rules in QueryTypeParser (#570)
Co-authored-by: jolunoluo
2023-12-25 20:54:04 +08:00
jipeli
5ab1cade0a (improvement)(headless) add sql optimizer and push minMax date filter down: (#567)
Co-authored-by: jipengli <jipengli@tencent.com>
2023-12-22 22:26:06 +08:00
LXW
e0955c0618 (improvement)(chat) Replace the table name in correct sql with a string concatenated by the model name in FromCorrector (#566)
Co-authored-by: jolunoluo
2023-12-22 19:24:57 +08:00
williamhliu
125598bd6e (improvement)(chat-sdk) fix the issue of incorrect parameters in query similar questions (#565)
Co-authored-by: williamhliu <williamhliu@tencent.com>
2023-12-22 18:25:41 +08:00
lexluo09
0cbfe473dd (improvement)(chat) Change the storage of InMemoryEmbeddingStore entity to a Set for deduplication. (#564) 2023-12-22 14:49:53 +08:00
Scott
090abbceed (fix)(common):check embedding exist before add to store (#563) 2023-12-22 14:41:30 +08:00
lexluo09
0a1f446fb8 (improvement)(chat) After a failed query, the absence of a returned queryId causes the recommendation question call to fail. (#562) 2023-12-21 22:10:48 +08:00
LXW
fa38e37be3 (improvement)(headless) Add API interface to provide data services to other applications (#561)
Co-authored-by: jolunoluo
2023-12-21 22:10:29 +08:00
lexluo09
7b580b7c94 (improvement)(chat) Introduce a separate log file to record key outputs in the pipeline. (#559) 2023-12-20 22:45:20 +08:00
LXW
2631352c30 (improvement)(common) Add JsonUtil configuration to avoid deserialization failure when enumeration values change (#558)
Co-authored-by: jolunoluo
2023-12-20 22:11:36 +08:00
Scott
f7914ff6f4 (fix):fill values into param options in webservice query, some refactor (#557) 2023-12-20 19:56:06 +08:00
lexluo09
ab077df36d (improvement)(project) Modify the log name for 'headless' and add 'headless' and 'chat' configurations to support separate deployment. (#556) 2023-12-20 17:35:17 +08:00
lexluo09
150d67f903 (improvement)(semantic) Change 'semantic' to 'headless' (#555) 2023-12-20 17:08:16 +08:00
tristanliu
187dcacbe7 [improvement][semantic-fe] Replacing the initialization variable from "semantic" to "headless". (#554)
* [improvement][semantic-fe] Add model alias setting & Add view permission restrictions to the model permission management tab.
[improvement][semantic-fe] Add permission control to the action buttons for the main domain; apply high sensitivity filtering to the authorization of metrics/dimensions.
[improvement][semantic-fe] Optimize the editing mode in the dimension/metric/datasource components to use the modelId stored in the database for data, instead of relying on the data from the state manager.

* [improvement][semantic-fe] Add time granularity setting in the data source configuration.

* [improvement][semantic-fe] Dictionary import for dimension values supported in Q&A visibility

* [improvement][semantic-fe] Modification of data source creation prompt wording"

* [improvement][semantic-fe] metric market experience optimization

* [improvement][semantic-fe] enhance the analysis of metric trends

* [improvement][semantic-fe] optimize the presentation of metric trend permissions

* [improvement][semantic-fe] add metric trend download functionality

* [improvement][semantic-fe] fix the dimension initialization issue in metric correlation

* [improvement][semantic-fe] Fix the issue of database changes not taking effect when creating based on an SQL data source.

* [improvement][semantic-fe] Optimizing pagination logic and some CSS styles

* [improvement][semantic-fe] Fixing the API for the indicator list by changing "current" to "pageNum"

* [improvement][semantic-fe] Fixing the default value setting for the indicator list

* [improvement][semantic-fe] Adding batch operations for indicators/dimensions/models

* [improvement][semantic-fe] Replacing the single status update API for indicators/dimensions with a batch update API

* [improvement][semantic-fe] Redesigning the indicator homepage to incorporate trend charts and table functionality for indicators

* [improvement][semantic-fe] Optimizing the logic for setting dimension values and editing data sources, and adding system settings functionality

* [improvement][semantic-fe] Upgrading antd version to 5.x, extracting the batch operation button component, optimizing the interaction for system settings, and expanding the configuration generation types for list-to-select component.

* [improvement][semantic-fe] Adding the ability to filter dimensions based on whether they are tags or not.

* [improvement][semantic-fe] Adding the ability to edit relationships between models in the canvas.

* [improvement][semantic-fe] Updating the datePicker component to use dayjs instead.

* [improvement][semantic-fe] Fixing the issue with passing the model ID for dimensions in the indicator market.

* [improvement][semantic-fe] Fixing the abnormal state of the popup when creating a model.

* [improvement][semantic-fe] Adding permission logic for bulk operations in the indicator market.

* [improvement][semantic-fe] Adding the ability to download and transpose data.

* [improvement][semantic-fe] Fixing the initialization issue with the date selection component in the indicator details page when switching time granularity.

* [improvement][semantic-fe] Fixing the logic error in the dimension value setting.

* [improvement][semantic-fe] Fixing the synchronization issue with the question and answer settings information.

* [improvement][semantic-fe] Optimizing the canvas functionality for better performance and user experience.

* [improvement][semantic-fe] Optimizing the update process for drawing model relationship edges in the canvas.

* [improvement][semantic-fe] Changing the line type for canvas connections.

* [improvement][semantic-fe] Replacing the initialization variable from "semantic" to "headless".
2023-12-20 16:49:26 +08:00
Scott
a194822cdd (fix):fix proxy ip default value (#552) 2023-12-19 20:14:10 +08:00
LXW
6bbc0a2cb4 (improvement)(chat) remove model.name and api.key in sys parameter configuration (#551)
Co-authored-by: jolunoluo
2023-12-19 17:02:18 +08:00
lexluo09
f60c1675cd [improvement](chat) udpate EmbeddingConfig embeddingStorePersistentPath and if the persistentPath directory does not exist, create the directory. (#550) 2023-12-19 16:12:11 +08:00
lexluo09
1d9b6d6877 [improvement](chat) optimize functionCall result format (#549) 2023-12-19 15:46:17 +08:00
LXW
d8930e8906 (improvement)(chat) Optimize the conditions for chatDemo loading (#548)
Co-authored-by: jolunoluo
2023-12-19 14:57:30 +08:00
lexluo09
c68df24375 [improvement](chat) Fix get metrics error in queryReqConverter (#547) 2023-12-19 14:29:18 +08:00
lexluo09
bb1001677d [improvement](chat) Fix the issue of having empty elements in addFieldsToSelect. (#545) 2023-12-19 11:54:28 +08:00
lexluo09
7a1cfbcef8 (improvement)(build) moveToRuntime checks if the tar.gz file exists, and if it doesn't, no operation is performed. (#544) 2023-12-19 11:02:35 +08:00
lexluo09
67b9c4bf79 (improvement)(chat) update description in s2SQLParameter (#543) 2023-12-18 23:18:05 +08:00
LXW
7cb7697353 (improvement)(chat) Modify the order of demo loaders (#542)
Co-authored-by: jolunoluo
2023-12-18 23:15:23 +08:00
lexluo09
3e18655c69 (improvement)(chat) add description to sqlGenerationMode (#541) 2023-12-18 22:51:21 +08:00
LXW
e7d52f87f0 (improvement)(python) run.py read files in utf-8 format (#540)
Co-authored-by: jolunoluo
2023-12-18 22:07:27 +08:00
lexluo09
2cd8f8022b [improvement](chat) fix SqlGeneration format issues (#539) 2023-12-18 22:05:43 +08:00
lexluo09
e08435902a [improvement](chat) fix SqlGeneration format issues (#538) 2023-12-18 22:04:40 +08:00
lexluo09
b44fa2bf3c [improvement](chat) fix unit test not right (#537) 2023-12-18 21:27:12 +08:00
LXW
d7f1f06daf (improvement)(chat) Remove candidateParses in ParseResp (#536)
Co-authored-by: jolunoluo
2023-12-18 21:10:53 +08:00
lexluo09
4c26e0c972 [improvement](project) update mvn version to 0.8.2 and update changelog (#535) 2023-12-18 20:51:55 +08:00
LXW
d7fafa361d (improvement)(chat) Switching metric supports default aggregation method of metric (#534)
Co-authored-by: jolunoluo
2023-12-18 18:51:21 +08:00
Scott
0c69651ef3 (fix):remove parantheses in generated sql (#532) 2023-12-18 17:57:06 +08:00
tristanliu
b5fdbfbbf6 [improvement][semantic-fe] Changing the line type for canvas connections. (#531)
* [improvement][semantic-fe] Add model alias setting & Add view permission restrictions to the model permission management tab.
[improvement][semantic-fe] Add permission control to the action buttons for the main domain; apply high sensitivity filtering to the authorization of metrics/dimensions.
[improvement][semantic-fe] Optimize the editing mode in the dimension/metric/datasource components to use the modelId stored in the database for data, instead of relying on the data from the state manager.

* [improvement][semantic-fe] Add time granularity setting in the data source configuration.

* [improvement][semantic-fe] Dictionary import for dimension values supported in Q&A visibility

* [improvement][semantic-fe] Modification of data source creation prompt wording"

* [improvement][semantic-fe] metric market experience optimization

* [improvement][semantic-fe] enhance the analysis of metric trends

* [improvement][semantic-fe] optimize the presentation of metric trend permissions

* [improvement][semantic-fe] add metric trend download functionality

* [improvement][semantic-fe] fix the dimension initialization issue in metric correlation

* [improvement][semantic-fe] Fix the issue of database changes not taking effect when creating based on an SQL data source.

* [improvement][semantic-fe] Optimizing pagination logic and some CSS styles

* [improvement][semantic-fe] Fixing the API for the indicator list by changing "current" to "pageNum"

* [improvement][semantic-fe] Fixing the default value setting for the indicator list

* [improvement][semantic-fe] Adding batch operations for indicators/dimensions/models

* [improvement][semantic-fe] Replacing the single status update API for indicators/dimensions with a batch update API

* [improvement][semantic-fe] Redesigning the indicator homepage to incorporate trend charts and table functionality for indicators

* [improvement][semantic-fe] Optimizing the logic for setting dimension values and editing data sources, and adding system settings functionality

* [improvement][semantic-fe] Upgrading antd version to 5.x, extracting the batch operation button component, optimizing the interaction for system settings, and expanding the configuration generation types for list-to-select component.

* [improvement][semantic-fe] Adding the ability to filter dimensions based on whether they are tags or not.

* [improvement][semantic-fe] Adding the ability to edit relationships between models in the canvas.

* [improvement][semantic-fe] Updating the datePicker component to use dayjs instead.

* [improvement][semantic-fe] Fixing the issue with passing the model ID for dimensions in the indicator market.

* [improvement][semantic-fe] Fixing the abnormal state of the popup when creating a model.

* [improvement][semantic-fe] Adding permission logic for bulk operations in the indicator market.

* [improvement][semantic-fe] Adding the ability to download and transpose data.

* [improvement][semantic-fe] Fixing the initialization issue with the date selection component in the indicator details page when switching time granularity.

* [improvement][semantic-fe] Fixing the logic error in the dimension value setting.

* [improvement][semantic-fe] Fixing the synchronization issue with the question and answer settings information.

* [improvement][semantic-fe] Optimizing the canvas functionality for better performance and user experience.

* [improvement][semantic-fe] Optimizing the update process for drawing model relationship edges in the canvas.

* [improvement][semantic-fe] Changing the line type for canvas connections.
2023-12-18 16:37:53 +08:00
LXW
33a2688e77 (improvement)(semantic) Add default canvas display data (#530)
Co-authored-by: jolunoluo
2023-12-18 15:45:16 +08:00
LXW
6bd97cd8af (improvement)(script) Update bat script (#529)
Co-authored-by: jolunoluo
2023-12-18 14:07:30 +08:00
LXW
64615cbef9 (improvement)(semantic) Querying dimension values supports models without time dimensions (#528)
Co-authored-by: jolunoluo
2023-12-18 13:18:18 +08:00
tristanliu
dfb3b59984 [improvement][semantic-fe] Optimizing the update process for drawing model relationship edges in the canvas. (#527)
* [improvement][semantic-fe] Add model alias setting & Add view permission restrictions to the model permission management tab.
[improvement][semantic-fe] Add permission control to the action buttons for the main domain; apply high sensitivity filtering to the authorization of metrics/dimensions.
[improvement][semantic-fe] Optimize the editing mode in the dimension/metric/datasource components to use the modelId stored in the database for data, instead of relying on the data from the state manager.

* [improvement][semantic-fe] Add time granularity setting in the data source configuration.

* [improvement][semantic-fe] Dictionary import for dimension values supported in Q&A visibility

* [improvement][semantic-fe] Modification of data source creation prompt wording"

* [improvement][semantic-fe] metric market experience optimization

* [improvement][semantic-fe] enhance the analysis of metric trends

* [improvement][semantic-fe] optimize the presentation of metric trend permissions

* [improvement][semantic-fe] add metric trend download functionality

* [improvement][semantic-fe] fix the dimension initialization issue in metric correlation

* [improvement][semantic-fe] Fix the issue of database changes not taking effect when creating based on an SQL data source.

* [improvement][semantic-fe] Optimizing pagination logic and some CSS styles

* [improvement][semantic-fe] Fixing the API for the indicator list by changing "current" to "pageNum"

* [improvement][semantic-fe] Fixing the default value setting for the indicator list

* [improvement][semantic-fe] Adding batch operations for indicators/dimensions/models

* [improvement][semantic-fe] Replacing the single status update API for indicators/dimensions with a batch update API

* [improvement][semantic-fe] Redesigning the indicator homepage to incorporate trend charts and table functionality for indicators

* [improvement][semantic-fe] Optimizing the logic for setting dimension values and editing data sources, and adding system settings functionality

* [improvement][semantic-fe] Upgrading antd version to 5.x, extracting the batch operation button component, optimizing the interaction for system settings, and expanding the configuration generation types for list-to-select component.

* [improvement][semantic-fe] Adding the ability to filter dimensions based on whether they are tags or not.

* [improvement][semantic-fe] Adding the ability to edit relationships between models in the canvas.

* [improvement][semantic-fe] Updating the datePicker component to use dayjs instead.

* [improvement][semantic-fe] Fixing the issue with passing the model ID for dimensions in the indicator market.

* [improvement][semantic-fe] Fixing the abnormal state of the popup when creating a model.

* [improvement][semantic-fe] Adding permission logic for bulk operations in the indicator market.

* [improvement][semantic-fe] Adding the ability to download and transpose data.

* [improvement][semantic-fe] Fixing the initialization issue with the date selection component in the indicator details page when switching time granularity.

* [improvement][semantic-fe] Fixing the logic error in the dimension value setting.

* [improvement][semantic-fe] Fixing the synchronization issue with the question and answer settings information.

* [improvement][semantic-fe] Optimizing the canvas functionality for better performance and user experience.

* [improvement][semantic-fe] Optimizing the update process for drawing model relationship edges in the canvas.
2023-12-18 10:53:42 +08:00
LXW
61641ecb00 (improvement)(chat) The history record only retains the query result of the first parse (#526)
Co-authored-by: jolunoluo
2023-12-18 10:20:50 +08:00
lexluo09
5016881ce3 [improvement](chat) Add EmbeddingPersistentTask to Persist and reload InMemoryS2EmbeddingStore (#525) 2023-12-17 21:53:21 +08:00
lexluo09
fe75b3e393 [improvement](chat) Add persistence to InMemoryEmbeddingStore, fix the issue of PythonServiceS2EmbeddingStore being empty. (#524) 2023-12-17 11:04:29 +08:00
jerryjzhang
3db443f9b1 [improvement][docs]Revise certain descriptions in README 2023-12-16 22:12:36 +08:00
lexluo09
59c21ea19a [improvement](chat) support filterCondition in InMemoryS2EmbeddingStore (#523) 2023-12-16 20:04:01 +08:00
lexluo09
95334441b1 [improvement](build) Support building pyllm separately. (#522) 2023-12-16 18:44:34 +08:00
lexluo09
276b224c13 [improvement](chat) Support sqlGenerationMode in python interface (#521) 2023-12-16 17:24:24 +08:00
lexluo09
f03da53d6f [improvement](chat) Support frontend configuration for S2SQL generation method. (#520) 2023-12-16 17:10:34 +08:00
lexluo09
9201550027 [improvement](chat) Support new four methods of generating SQL using SqlGeneration large models. (#519) 2023-12-16 14:44:56 +08:00
jipeli
c86cd9f901 [improvement][semantic] refactor zipper source type (#518)
Co-authored-by: jipengli <jipengli@tencent.com>
2023-12-15 21:47:35 +08:00
tristanliu
ef8caea9d2 [improvement][semantic-fe] Optimizing the canvas functionality for better performance and user experience. (#517)
* [improvement][semantic-fe] Add model alias setting & Add view permission restrictions to the model permission management tab.
[improvement][semantic-fe] Add permission control to the action buttons for the main domain; apply high sensitivity filtering to the authorization of metrics/dimensions.
[improvement][semantic-fe] Optimize the editing mode in the dimension/metric/datasource components to use the modelId stored in the database for data, instead of relying on the data from the state manager.

* [improvement][semantic-fe] Add time granularity setting in the data source configuration.

* [improvement][semantic-fe] Dictionary import for dimension values supported in Q&A visibility

* [improvement][semantic-fe] Modification of data source creation prompt wording"

* [improvement][semantic-fe] metric market experience optimization

* [improvement][semantic-fe] enhance the analysis of metric trends

* [improvement][semantic-fe] optimize the presentation of metric trend permissions

* [improvement][semantic-fe] add metric trend download functionality

* [improvement][semantic-fe] fix the dimension initialization issue in metric correlation

* [improvement][semantic-fe] Fix the issue of database changes not taking effect when creating based on an SQL data source.

* [improvement][semantic-fe] Optimizing pagination logic and some CSS styles

* [improvement][semantic-fe] Fixing the API for the indicator list by changing "current" to "pageNum"

* [improvement][semantic-fe] Fixing the default value setting for the indicator list

* [improvement][semantic-fe] Adding batch operations for indicators/dimensions/models

* [improvement][semantic-fe] Replacing the single status update API for indicators/dimensions with a batch update API

* [improvement][semantic-fe] Redesigning the indicator homepage to incorporate trend charts and table functionality for indicators

* [improvement][semantic-fe] Optimizing the logic for setting dimension values and editing data sources, and adding system settings functionality

* [improvement][semantic-fe] Upgrading antd version to 5.x, extracting the batch operation button component, optimizing the interaction for system settings, and expanding the configuration generation types for list-to-select component.

* [improvement][semantic-fe] Adding the ability to filter dimensions based on whether they are tags or not.

* [improvement][semantic-fe] Adding the ability to edit relationships between models in the canvas.

* [improvement][semantic-fe] Updating the datePicker component to use dayjs instead.

* [improvement][semantic-fe] Fixing the issue with passing the model ID for dimensions in the indicator market.

* [improvement][semantic-fe] Fixing the abnormal state of the popup when creating a model.

* [improvement][semantic-fe] Adding permission logic for bulk operations in the indicator market.

* [improvement][semantic-fe] Adding the ability to download and transpose data.

* [improvement][semantic-fe] Fixing the initialization issue with the date selection component in the indicator details page when switching time granularity.

* [improvement][semantic-fe] Fixing the logic error in the dimension value setting.

* [improvement][semantic-fe] Fixing the synchronization issue with the question and answer settings information.

* [improvement][semantic-fe] Optimizing the canvas functionality for better performance and user experience.
2023-12-15 19:12:34 +08:00
williamhliu
6daaff8c30 (feature)(chat-sdk) show showcase button in mobile mode (#516) 2023-12-15 18:19:57 +08:00
williamhliu
4b00c16eb7 (feature)(chat-sdk) modify the method for obtaining similar questions and recommended drill-down dimensions; do not display assistant button when there is only one assistant (#514) 2023-12-15 17:41:35 +08:00
LXW
4dae84034e (improvement)(chat) Rename SimilarMetricProcessor, SimilarQueryProcessor to MetricRecommendProcessor, QueryRecommendProcessor (#513)
Co-authored-by: jolunoluo
2023-12-15 14:56:44 +08:00
LXW
e6eac03ec6 (improvement)(chat) Rename DrillDownDimensionProcessor to DimensionRecommendProcessor (#512)
Co-authored-by: jolunoluo
2023-12-15 11:48:41 +08:00
LXW
e9a479e2df (improvement)(chat) Add DrillDownDimensionProcessor and SimilarQueryProcessor to obtain recommended dimensions and similar queries (#511)
Co-authored-by: jolunoluo
2023-12-15 10:27:33 +08:00
lexluo09
7db1cc270e [improvement](chat) fix InProcess config (#509) 2023-12-14 17:10:27 +08:00
tristanliu
3bf5b86535 [improvement][semantic-fe] Fixing the synchronization issue with the question and answer settings information. (#508)
* [improvement][semantic-fe] Add model alias setting & Add view permission restrictions to the model permission management tab.
[improvement][semantic-fe] Add permission control to the action buttons for the main domain; apply high sensitivity filtering to the authorization of metrics/dimensions.
[improvement][semantic-fe] Optimize the editing mode in the dimension/metric/datasource components to use the modelId stored in the database for data, instead of relying on the data from the state manager.

* [improvement][semantic-fe] Add time granularity setting in the data source configuration.

* [improvement][semantic-fe] Dictionary import for dimension values supported in Q&A visibility

* [improvement][semantic-fe] Modification of data source creation prompt wording"

* [improvement][semantic-fe] metric market experience optimization

* [improvement][semantic-fe] enhance the analysis of metric trends

* [improvement][semantic-fe] optimize the presentation of metric trend permissions

* [improvement][semantic-fe] add metric trend download functionality

* [improvement][semantic-fe] fix the dimension initialization issue in metric correlation

* [improvement][semantic-fe] Fix the issue of database changes not taking effect when creating based on an SQL data source.

* [improvement][semantic-fe] Optimizing pagination logic and some CSS styles

* [improvement][semantic-fe] Fixing the API for the indicator list by changing "current" to "pageNum"

* [improvement][semantic-fe] Fixing the default value setting for the indicator list

* [improvement][semantic-fe] Adding batch operations for indicators/dimensions/models

* [improvement][semantic-fe] Replacing the single status update API for indicators/dimensions with a batch update API

* [improvement][semantic-fe] Redesigning the indicator homepage to incorporate trend charts and table functionality for indicators

* [improvement][semantic-fe] Optimizing the logic for setting dimension values and editing data sources, and adding system settings functionality

* [improvement][semantic-fe] Upgrading antd version to 5.x, extracting the batch operation button component, optimizing the interaction for system settings, and expanding the configuration generation types for list-to-select component.

* [improvement][semantic-fe] Adding the ability to filter dimensions based on whether they are tags or not.

* [improvement][semantic-fe] Adding the ability to edit relationships between models in the canvas.

* [improvement][semantic-fe] Updating the datePicker component to use dayjs instead.

* [improvement][semantic-fe] Fixing the issue with passing the model ID for dimensions in the indicator market.

* [improvement][semantic-fe] Fixing the abnormal state of the popup when creating a model.

* [improvement][semantic-fe] Adding permission logic for bulk operations in the indicator market.

* [improvement][semantic-fe] Adding the ability to download and transpose data.

* [improvement][semantic-fe] Fixing the initialization issue with the date selection component in the indicator details page when switching time granularity.

* [improvement][semantic-fe] Fixing the logic error in the dimension value setting.

* [improvement][semantic-fe] Fixing the synchronization issue with the question and answer settings information.
2023-12-14 16:37:19 +08:00
lexluo09
3d30632b41 [improvement](python) Remove unused Python-related files (#507) 2023-12-14 15:19:52 +08:00
lexluo09
287a6561ff [improvement](chat) Add an in_process provider and support offline loading of local embedding models. (#505) 2023-12-14 14:16:03 +08:00
jerryjzhang
169262cc62 [improvement][doc]Compile new project description 2023-12-14 13:11:39 +08:00
LXW
fda5a577d6 (improvement)(chat) rename QueryResponder to ExecuteResultProcessor and add ResultProcessor (#504)
Co-authored-by: jolunoluo
2023-12-13 22:16:42 +08:00
LXW
f89be48e98 (improvement)(semantic) save metric default query config (#503)
Co-authored-by: jolunoluo
2023-12-13 21:29:15 +08:00
LXW
2c7afd0d55 (improvement)(common) add sys parameter description (#502)
Co-authored-by: jolunoluo
2023-12-13 21:02:56 +08:00
lexluo09
2ad0553f6c [improvement](chat) fix concurrentModificationException in a multi-threaded environment. (#501) 2023-12-13 13:26:01 +08:00
lexluo09
340cb2c835 [improvement](chat) Resolve the issue of an empty result in schemaLinkParse. (#500) 2023-12-13 11:18:15 +08:00
tristanliu
caefa501f2 [improvement][semantic-fe] Fixing the logic error in the dimension value setting. (#499)
* [improvement][semantic-fe] Add model alias setting & Add view permission restrictions to the model permission management tab.
[improvement][semantic-fe] Add permission control to the action buttons for the main domain; apply high sensitivity filtering to the authorization of metrics/dimensions.
[improvement][semantic-fe] Optimize the editing mode in the dimension/metric/datasource components to use the modelId stored in the database for data, instead of relying on the data from the state manager.

* [improvement][semantic-fe] Add time granularity setting in the data source configuration.

* [improvement][semantic-fe] Dictionary import for dimension values supported in Q&A visibility

* [improvement][semantic-fe] Modification of data source creation prompt wording"

* [improvement][semantic-fe] metric market experience optimization

* [improvement][semantic-fe] enhance the analysis of metric trends

* [improvement][semantic-fe] optimize the presentation of metric trend permissions

* [improvement][semantic-fe] add metric trend download functionality

* [improvement][semantic-fe] fix the dimension initialization issue in metric correlation

* [improvement][semantic-fe] Fix the issue of database changes not taking effect when creating based on an SQL data source.

* [improvement][semantic-fe] Optimizing pagination logic and some CSS styles

* [improvement][semantic-fe] Fixing the API for the indicator list by changing "current" to "pageNum"

* [improvement][semantic-fe] Fixing the default value setting for the indicator list

* [improvement][semantic-fe] Adding batch operations for indicators/dimensions/models

* [improvement][semantic-fe] Replacing the single status update API for indicators/dimensions with a batch update API

* [improvement][semantic-fe] Redesigning the indicator homepage to incorporate trend charts and table functionality for indicators

* [improvement][semantic-fe] Optimizing the logic for setting dimension values and editing data sources, and adding system settings functionality

* [improvement][semantic-fe] Upgrading antd version to 5.x, extracting the batch operation button component, optimizing the interaction for system settings, and expanding the configuration generation types for list-to-select component.

* [improvement][semantic-fe] Adding the ability to filter dimensions based on whether they are tags or not.

* [improvement][semantic-fe] Adding the ability to edit relationships between models in the canvas.

* [improvement][semantic-fe] Updating the datePicker component to use dayjs instead.

* [improvement][semantic-fe] Fixing the issue with passing the model ID for dimensions in the indicator market.

* [improvement][semantic-fe] Fixing the abnormal state of the popup when creating a model.

* [improvement][semantic-fe] Adding permission logic for bulk operations in the indicator market.

* [improvement][semantic-fe] Adding the ability to download and transpose data.

* [improvement][semantic-fe] Fixing the initialization issue with the date selection component in the indicator details page when switching time granularity.

* [improvement][semantic-fe] Fixing the logic error in the dimension value setting.
2023-12-12 19:40:24 +08:00
lexluo09
5c96d75d39 [improvement](project) If there are any failed operations during the build process, stop subsequent operations and issue a unified 'stop' command. (#498) 2023-12-12 17:13:57 +08:00
jerryjzhang
86c2f96942 [improvement][fix]Fix codestyle 2023-12-12 15:57:37 +08:00
lexluo09
73899e3174 [improvement](chat) Change llmparser to pyllm, retrieve LLMProxy from environment variables, defaulting to JavaLLMProxy. (#497) 2023-12-12 14:29:35 +08:00
Scott
49bb2c6d8b fix:use correct sql generated mode in java code (#496) 2023-12-11 21:46:57 +08:00
williamhliu
9223a4f856 (feature)(webapp) change sql text (#495) 2023-12-11 19:56:53 +08:00
Scott
f3f60af231 (fix):fix python service interact (#494) 2023-12-11 19:31:36 +08:00
jerryjzhang
3cdfcae01c [improvement][chat]Add javadoc to Mapper related classes 2023-12-11 18:07:26 +08:00
lexluo09
0c6efada43 [improvement](chat) Merge HanlpDictMapper and FuzzyNameMapper into KeywordMapper. (#493)
Co-authored-by: lexluo <lexluo@tencent.com>
2023-12-11 17:22:57 +08:00
codescracker
d79f73eab6 add auto-CoT feature (#483)
* 1.refactor the retrieval module. 2.refactor the http service module. 3.upgrade text2sql output format the parse for absolute time related expression in query.

* fix bug.

* upgrade the config module, now support config llm suppoted by langchain.

* fix conflicts.

* update text2sql config reload to be compitable with new config format.

* modify default config.

* 1.add self-consistency feature for text2sql. 2.upgrade llm api call from sync to async. 3.refactor text2sql module. 4. refactor semantical retriever modules.

* merege with upstream master

* add general retrieve service.

* add api service for sql_agent for crud opereations of few-shots examples.

* modify requirements

* add auto-cot feature

---------

Co-authored-by: shaoweigong <shaoweigong@tencent.com>
2023-12-11 16:07:49 +08:00
lexluo09
3ae720ef30 [improvement](chat) add log4j-api in pom (#492) 2023-12-11 12:26:32 +08:00
jerryjzhang
221e88de0f [improvement][common]Replace QueryType.OTHER with ID 2023-12-11 11:47:50 +08:00
lexluo09
23d926f195 [improvement](chat) remove ModelProvider and in_memory (#491) 2023-12-11 11:17:30 +08:00
lexluo09
97b11ec244 [improvement](chat) default use AllMiniLmL6V2EmbeddingModel (#490) 2023-12-11 11:01:47 +08:00
lexluo09
899047dbd1 [improvement](chat) Adjust related packages, add other SqlGeneration implementations, and rename EmbedLLMProxy to JavaLLMProxy. (#489) 2023-12-11 10:24:22 +08:00
Scott
cb4b91878f fix:add base url (#488) 2023-12-10 20:55:47 +08:00
lexluo09
6af661459c Add SqlGeneration abstraction and implementation, optimize LLMSqlParser skip. (#487) 2023-12-10 11:34:48 +08:00
lexluo09
0e0ba51750 [improvement](chat) Resolving jtokkit JAR package conflicts. (#486) 2023-12-09 00:43:30 +08:00
lexluo09
a5c32ac064 [improvement](chat) Remove unnecessary pom imports, optimize code, and adapt the test project. (#485) 2023-12-08 20:03:42 +08:00
lexluo09
abbe8c84a1 [improvement](python) LLM related services support Java service invocation (#484) 2023-12-08 19:24:58 +08:00
tristanliu
6c0f88d8b5 [improvement][semantic-fe] Fixing the initialization issue with the date selection component in the indicator details page when switching time granularity. (#482)
* [improvement][semantic-fe] Add model alias setting & Add view permission restrictions to the model permission management tab.
[improvement][semantic-fe] Add permission control to the action buttons for the main domain; apply high sensitivity filtering to the authorization of metrics/dimensions.
[improvement][semantic-fe] Optimize the editing mode in the dimension/metric/datasource components to use the modelId stored in the database for data, instead of relying on the data from the state manager.

* [improvement][semantic-fe] Add time granularity setting in the data source configuration.

* [improvement][semantic-fe] Dictionary import for dimension values supported in Q&A visibility

* [improvement][semantic-fe] Modification of data source creation prompt wording"

* [improvement][semantic-fe] metric market experience optimization

* [improvement][semantic-fe] enhance the analysis of metric trends

* [improvement][semantic-fe] optimize the presentation of metric trend permissions

* [improvement][semantic-fe] add metric trend download functionality

* [improvement][semantic-fe] fix the dimension initialization issue in metric correlation

* [improvement][semantic-fe] Fix the issue of database changes not taking effect when creating based on an SQL data source.

* [improvement][semantic-fe] Optimizing pagination logic and some CSS styles

* [improvement][semantic-fe] Fixing the API for the indicator list by changing "current" to "pageNum"

* [improvement][semantic-fe] Fixing the default value setting for the indicator list

* [improvement][semantic-fe] Adding batch operations for indicators/dimensions/models

* [improvement][semantic-fe] Replacing the single status update API for indicators/dimensions with a batch update API

* [improvement][semantic-fe] Redesigning the indicator homepage to incorporate trend charts and table functionality for indicators

* [improvement][semantic-fe] Optimizing the logic for setting dimension values and editing data sources, and adding system settings functionality

* [improvement][semantic-fe] Upgrading antd version to 5.x, extracting the batch operation button component, optimizing the interaction for system settings, and expanding the configuration generation types for list-to-select component.

* [improvement][semantic-fe] Adding the ability to filter dimensions based on whether they are tags or not.

* [improvement][semantic-fe] Adding the ability to edit relationships between models in the canvas.

* [improvement][semantic-fe] Updating the datePicker component to use dayjs instead.

* [improvement][semantic-fe] Fixing the issue with passing the model ID for dimensions in the indicator market.

* [improvement][semantic-fe] Fixing the abnormal state of the popup when creating a model.

* [improvement][semantic-fe] Adding permission logic for bulk operations in the indicator market.

* [improvement][semantic-fe] Adding the ability to download and transpose data.

* [improvement][semantic-fe] Fixing the initialization issue with the date selection component in the indicator details page when switching time granularity.
2023-12-08 15:08:12 +08:00
LXW
68ada561ac (fix)(test) fix test compile error (#481)
Co-authored-by: jolunoluo
2023-12-08 13:34:12 +08:00
LXW
18b52ec742 (improvement)(semantic) Metric homepage download supports choosing whether to transform (#480)
Co-authored-by: jolunoluo
2023-12-08 13:20:27 +08:00
lexluo09
ca8d7d89c1 (improvement)(build) When remove runtimeDir in the build script, add prefix deletion handling. (#479) 2023-12-07 10:27:02 +08:00
williamhliu
e6ab7cb5ff (fix)(chat-sdk) fix date change bug and hide recommend metrics tip when no metrics (#478) 2023-12-06 20:54:31 +08:00
LXW
9679169e6f (improvement)(semantic) Metric homepage download supports choosing whether to transform (#477)
Co-authored-by: jolunoluo
2023-12-06 18:02:20 +08:00
lexluo09
ed0f856438 [improvement](chat) Unified vector-related interfaces to go through EmbeddingUtils. (#476) 2023-12-06 14:50:57 +08:00
Scott
9aa5c93d9d fix:consider alias of metric when correct group sql (#475) 2023-12-06 14:10:37 +08:00
tristanliu
b45592c009 [improvement][semantic-fe] Adding the ability to download and transpose data. (#474)
* [improvement][semantic-fe] Add model alias setting & Add view permission restrictions to the model permission management tab.
[improvement][semantic-fe] Add permission control to the action buttons for the main domain; apply high sensitivity filtering to the authorization of metrics/dimensions.
[improvement][semantic-fe] Optimize the editing mode in the dimension/metric/datasource components to use the modelId stored in the database for data, instead of relying on the data from the state manager.

* [improvement][semantic-fe] Add time granularity setting in the data source configuration.

* [improvement][semantic-fe] Dictionary import for dimension values supported in Q&A visibility

* [improvement][semantic-fe] Modification of data source creation prompt wording"

* [improvement][semantic-fe] metric market experience optimization

* [improvement][semantic-fe] enhance the analysis of metric trends

* [improvement][semantic-fe] optimize the presentation of metric trend permissions

* [improvement][semantic-fe] add metric trend download functionality

* [improvement][semantic-fe] fix the dimension initialization issue in metric correlation

* [improvement][semantic-fe] Fix the issue of database changes not taking effect when creating based on an SQL data source.

* [improvement][semantic-fe] Optimizing pagination logic and some CSS styles

* [improvement][semantic-fe] Fixing the API for the indicator list by changing "current" to "pageNum"

* [improvement][semantic-fe] Fixing the default value setting for the indicator list

* [improvement][semantic-fe] Adding batch operations for indicators/dimensions/models

* [improvement][semantic-fe] Replacing the single status update API for indicators/dimensions with a batch update API

* [improvement][semantic-fe] Redesigning the indicator homepage to incorporate trend charts and table functionality for indicators

* [improvement][semantic-fe] Optimizing the logic for setting dimension values and editing data sources, and adding system settings functionality

* [improvement][semantic-fe] Upgrading antd version to 5.x, extracting the batch operation button component, optimizing the interaction for system settings, and expanding the configuration generation types for list-to-select component.

* [improvement][semantic-fe] Adding the ability to filter dimensions based on whether they are tags or not.

* [improvement][semantic-fe] Adding the ability to edit relationships between models in the canvas.

* [improvement][semantic-fe] Updating the datePicker component to use dayjs instead.

* [improvement][semantic-fe] Fixing the issue with passing the model ID for dimensions in the indicator market.

* [improvement][semantic-fe] Fixing the abnormal state of the popup when creating a model.

* [improvement][semantic-fe] Adding permission logic for bulk operations in the indicator market.

* [improvement][semantic-fe] Adding the ability to download and transpose data.
2023-12-06 11:34:51 +08:00
tristanliu
6e0fa95e6f [improvement][semantic-fe] Adding permission logic for bulk operations in the indicator market. (#471)
* [improvement][semantic-fe] Add model alias setting & Add view permission restrictions to the model permission management tab.
[improvement][semantic-fe] Add permission control to the action buttons for the main domain; apply high sensitivity filtering to the authorization of metrics/dimensions.
[improvement][semantic-fe] Optimize the editing mode in the dimension/metric/datasource components to use the modelId stored in the database for data, instead of relying on the data from the state manager.

* [improvement][semantic-fe] Add time granularity setting in the data source configuration.

* [improvement][semantic-fe] Dictionary import for dimension values supported in Q&A visibility

* [improvement][semantic-fe] Modification of data source creation prompt wording"

* [improvement][semantic-fe] metric market experience optimization

* [improvement][semantic-fe] enhance the analysis of metric trends

* [improvement][semantic-fe] optimize the presentation of metric trend permissions

* [improvement][semantic-fe] add metric trend download functionality

* [improvement][semantic-fe] fix the dimension initialization issue in metric correlation

* [improvement][semantic-fe] Fix the issue of database changes not taking effect when creating based on an SQL data source.

* [improvement][semantic-fe] Optimizing pagination logic and some CSS styles

* [improvement][semantic-fe] Fixing the API for the indicator list by changing "current" to "pageNum"

* [improvement][semantic-fe] Fixing the default value setting for the indicator list

* [improvement][semantic-fe] Adding batch operations for indicators/dimensions/models

* [improvement][semantic-fe] Replacing the single status update API for indicators/dimensions with a batch update API

* [improvement][semantic-fe] Redesigning the indicator homepage to incorporate trend charts and table functionality for indicators

* [improvement][semantic-fe] Optimizing the logic for setting dimension values and editing data sources, and adding system settings functionality

* [improvement][semantic-fe] Upgrading antd version to 5.x, extracting the batch operation button component, optimizing the interaction for system settings, and expanding the configuration generation types for list-to-select component.

* [improvement][semantic-fe] Adding the ability to filter dimensions based on whether they are tags or not.

* [improvement][semantic-fe] Adding the ability to edit relationships between models in the canvas.

* [improvement][semantic-fe] Updating the datePicker component to use dayjs instead.

* [improvement][semantic-fe] Fixing the issue with passing the model ID for dimensions in the indicator market.

* [improvement][semantic-fe] Fixing the abnormal state of the popup when creating a model.

* [improvement][semantic-fe] Adding permission logic for bulk operations in the indicator market.
2023-12-04 20:18:36 +08:00
lexluo09
94f310d17f (improvement)(chat) fix compilation style issues. (#467) 2023-12-04 10:59:31 +08:00
mainmain
2bc29d64a4 (improvement)(chat) rewrite datediff logic and perfect sql style (#461) 2023-12-04 10:41:41 +08:00
Scott
c220ca69c2 fix:convert name or alias to schema element (#465) 2023-12-03 20:22:02 +08:00
LXW
4280aad0a7 (improvement)(semantic) Supply and compatible with mysql demo data (#464)
Co-authored-by: jolunoluo
2023-12-02 23:17:27 +08:00
lexluo09
c98d15059b [improvement][chat]Consider Chinese date fields in MetricCheckProcessor for data (#460) 2023-12-01 11:30:23 +08:00
LXW
a862a83272 (improvement)(chat) SemanticParseInfo removes elements whose model is empty (#459)
* (improvement)(chat) Update spi in test profile

* (improvement)(chat) SemanticParseInfo removes elements whose model is empty

* (improvement)(chat) MetricCheckProcessor compatible time dimension chName

---------

Co-authored-by: jolunoluo
2023-12-01 11:27:07 +08:00
williamhliu
c6d59701db (feature)(webapp) change model name to model names tag (#455) 2023-11-30 21:36:09 +08:00
williamhliu
39a85dc4ed (feature)(webapp) change agent tool enum value (#454) 2023-11-30 21:14:07 +08:00
LXW
507c02a8fd (improvement)(chat) Remove redundant maven dependencies and classes (#453)
Co-authored-by: jolunoluo
2023-11-30 19:36:18 +08:00
tristanliu
380597f0c3 [improvement][semantic-fe] Fixing the abnormal state of the popup when creating a model. (#452)
* [improvement][semantic-fe] Add model alias setting & Add view permission restrictions to the model permission management tab.
[improvement][semantic-fe] Add permission control to the action buttons for the main domain; apply high sensitivity filtering to the authorization of metrics/dimensions.
[improvement][semantic-fe] Optimize the editing mode in the dimension/metric/datasource components to use the modelId stored in the database for data, instead of relying on the data from the state manager.

* [improvement][semantic-fe] Add time granularity setting in the data source configuration.

* [improvement][semantic-fe] Dictionary import for dimension values supported in Q&A visibility

* [improvement][semantic-fe] Modification of data source creation prompt wording"

* [improvement][semantic-fe] metric market experience optimization

* [improvement][semantic-fe] enhance the analysis of metric trends

* [improvement][semantic-fe] optimize the presentation of metric trend permissions

* [improvement][semantic-fe] add metric trend download functionality

* [improvement][semantic-fe] fix the dimension initialization issue in metric correlation

* [improvement][semantic-fe] Fix the issue of database changes not taking effect when creating based on an SQL data source.

* [improvement][semantic-fe] Optimizing pagination logic and some CSS styles

* [improvement][semantic-fe] Fixing the API for the indicator list by changing "current" to "pageNum"

* [improvement][semantic-fe] Fixing the default value setting for the indicator list

* [improvement][semantic-fe] Adding batch operations for indicators/dimensions/models

* [improvement][semantic-fe] Replacing the single status update API for indicators/dimensions with a batch update API

* [improvement][semantic-fe] Redesigning the indicator homepage to incorporate trend charts and table functionality for indicators

* [improvement][semantic-fe] Optimizing the logic for setting dimension values and editing data sources, and adding system settings functionality

* [improvement][semantic-fe] Upgrading antd version to 5.x, extracting the batch operation button component, optimizing the interaction for system settings, and expanding the configuration generation types for list-to-select component.

* [improvement][semantic-fe] Adding the ability to filter dimensions based on whether they are tags or not.

* [improvement][semantic-fe] Adding the ability to edit relationships between models in the canvas.

* [improvement][semantic-fe] Updating the datePicker component to use dayjs instead.

* [improvement][semantic-fe] Fixing the issue with passing the model ID for dimensions in the indicator market.

* [improvement][semantic-fe] Fixing the abnormal state of the popup when creating a model.
2023-11-30 18:48:41 +08:00
mainmain
e469c449b4 [improvement] supersonic adapt checkstyle (#451) 2023-11-30 11:41:55 +08:00
jerryjzhang
f8bdb8a4b4 [improvement][checkstyle]Add multiple empty lines checks 2023-11-30 11:03:22 +08:00
jerryjzhang
d76216a2ec [improvement][chat]Refactor certain packages to make code cleaner 2023-11-30 10:05:45 +08:00
jerryjzhang
82cfb3050d [improvement][chat]Rename LLMInterpreter as LLMProxy 2023-11-29 16:57:48 +08:00
jerryjzhang
57f7d0c67d [improvement][chat]Restructure Agent&Tool package 2023-11-29 16:34:52 +08:00
LXW
c11a242f34 (improvement)(chat) Add model name set in ModelCluster (#449)
Co-authored-by: jolunoluo
2023-11-29 12:13:26 +08:00
jerryjzhang
576fad5fb1 Merge remote-tracking branch 'origin/master' 2023-11-29 10:29:05 +08:00
jerryjzhang
8171d754e0 [improvement][chat]Rename PostProcessor to ResponseProcessor 2023-11-29 10:28:49 +08:00
mainmain
6be0f02c75 [improvement] add database config (#447)
Co-authored-by: zuopengge
2023-11-29 10:14:52 +08:00
LXW
95e3138ab2 (improvement)(chat) Merge ParseResponder into PostProcessor, and change ExecuteResponder to QueryResponder (#448)
Co-authored-by: jolunoluo
2023-11-29 10:08:25 +08:00
lexluo09
3a30a1a317 [improvement](project) Support cspider multi-table association, fix dictionary loading, and repair InternalMetric exceptions. (#446) 2023-11-28 21:16:02 +08:00
LXW
46733d1728 (improvement)(semantic) Batch download compatible with empty data (#445)
Co-authored-by: jolunoluo
2023-11-28 16:38:30 +08:00
lexluo09
b6734d99e1 [improvement](chat) add DemoChatConfig in cspider (#444) 2023-11-28 15:42:13 +08:00
lexluo09
9cb01149f8 [improvement](chat) Optimize the configuration of cspider-related models, and fix the issue of different models having the same dimensionName (#443) 2023-11-28 15:08:46 +08:00
LXW
db88127da9 (fix)(semantic) Modify data type of s2_collect in schema-mysql.sql and fixed startup problem when database data is empty (#442)
Co-authored-by: jolunoluo
2023-11-28 13:55:19 +08:00
Scott
0e492ef402 (fix):fix startup crash (#441)
* (fix):fix startup crash
2023-11-28 13:29:58 +08:00
1071 changed files with 24323 additions and 23059 deletions

31
.github/workflows/ci.yml vendored Normal file
View File

@@ -0,0 +1,31 @@
name: supersonic CI
on:
push:
branches:
- master
pull_request:
branches:
- master
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Set up JDK 8
uses: actions/setup-java@v2
with:
java-version: '8'
distribution: 'adopt'
- name: Cache Maven packages
uses: actions/cache@v2
with:
path: ~/.m2
key: ${{ runner.os }}-m2-${{ hashFiles('**/pom.xml') }}
restore-keys: ${{ runner.os }}-m2
- name: Build with Maven
run: mvn -B package --file pom.xml
- name: Test with Maven
run: mvn test

View File

@@ -4,6 +4,23 @@
- "Breaking Changes" describes any changes that may break existing functionality or cause
compatibility issues with previous versions.
## SuperSonic [0.8.4] - 2024-01-19
### Added
- support creating derived metrics.
- Support creating metrics using three methods: by measure, metric, and field expressions.
- added support for postgresql data source.
- code adjustment and abstract optimization for chat and headless.
## SuperSonic [0.8.2] - 2023-12-18
### Added
- rewrite Python service with Java project, default to Java implementation.
- support setting the SQL generation method for large models in the interface.
- optimization of metric market experience.
- optimization of semantic modeling canvas experience.
- code structure adjustment and abstraction optimization for chat.
## SuperSonic [0.7.5] - 2023-10-13
### Added

View File

@@ -1,26 +1,28 @@
[中文介绍](README_CN.md) | [文档中心](https://github.com/tencentmusic/supersonic/wiki)
![Java CI](https://github.com/tencentmusic/supersonic/workflows/supersonic%20CI/badge.svg)
# SuperSonic (超音数)
**SuperSonic is an out-of-the-box yet highly extensible framework for building ChatBI**. SuperSonic provides a chat interface that empowers users to query data using natural language and visualize the results with suitable charts. To enable such experience, the only thing necessary is to build logical semantic models (definition of metrics/dimensions/entities, along with their meaning, context and relationships) on top of physical data models, and no data modification or copying is required. Meanwhile, SuperSonic is designed to be pluggable, allowing new functionalities to be added through plugins and core components to be integrated with other systems.
**SuperSonic is the next-generation LLM-powered data analytics platform that integrates ChatBI and HeadlessBI**. SuperSonic provides a chat interface that empowers users to query data using natural language and visualize the results with suitable charts. To enable such experience, the only thing necessary is to build logical semantic models (definition of entities/metrics/dimensions/tags, along with their meaning, context and relationships) on top of physical data models, and **no data modification or copying** is required. Meanwhile, SuperSonic is designed to be **highly extensible**, allowing custom functionalities to be added and configured with Java SPI.
<img src="./docs/images/supersonic_demo.gif" height="100%" width="100%" align="center"/>
## Motivation
The emergence of Large Language Model (LLM) like ChatGPT is reshaping the way information is retrieved. In the field of data analytics, both academia and industry are primarily focused on leveraging LLM to convert natural language into SQL (so called text2sql or nl2sql). While some works exhibit promising results, their **reliability** is inadequate for real-world applications.
The emergence of Large Language Model (LLM) like ChatGPT is reshaping the way information is retrieved. In the field of data analytics, both academia and industry are primarily focused on leveraging LLM to convert natural language into SQL (so called Text2SQL or NL2SQL). While some approaches exhibit promising results, their **reliability** and **efficiency** are insufficient for real-world applications.
From our perspective, the key to filling the real-world gap lies in three aspects:
1. Introduce a semantic layer encapsulating underlying data context(joins, formulas, etc) to reduce **complexity**.
2. Augment the LLM with schema mappers(as a kind of preprocessor) and semantic correctors(as a kind of postprocessor) to mitigate **hallucination**.
3. Utilize heuristic rules when necessary to improve **efficiency**(in terms of latency and cost).
1. Integrate ChatBI with HeadlessBI encapsulating underlying data context (joins, keys, formulas, etc) to **reduce complexity**.
2. Augment the LLM with schema mappers(as a kind of preprocessor) and semantic correctors(as a kind of postprocessor) to **mitigate hallucination**.
3. Utilize rule-based schema parsers when necessary to **improve efficiency**(in terms of latency and cost).
With these ideas in mind, we develop SuperSonic as a practical reference implementation and use it to power our real-world products. Additionally, to facilitate further development of ChatBI, we decide to open source SuperSonic as an extensible framework.
## Out-of-the-box Features
- Built-in CUI(Chat User Interface) for *business users* to enter data queries
- Built-in GUI(Graphical User Interface) for *analytics engineers* to build semantic models
- Built-in ChatBI interface for *business users* to enter natural language queries
- Built-in HeadlessBI interface for *analytics engineers* to build semantic models
- Built-in GUI for *system administrators* to manage chat agents and third-party plugins
- Support input auto-completion as well as query recommendation
- Support multi-turn conversation and history context management
@@ -49,7 +51,7 @@ The high-level architecture and main process flow is as follows:
SuperSonic comes with sample semantic models as well as chat conversations that can be used as a starting point. Please follow the steps:
- Download the latest prebuilt binary from the [release page](https://github.com/tencentmusic/supersonic/releases)
- Run script "bin/supersonic-daemon.sh" to start services (one java process and one python process)
- Run script "assembly/bin/supersonic-daemon.sh start" to start a standalone Java service
- Visit http://localhost:9080 in the browser to start exploration
## Build and Development

View File

@@ -1,6 +1,6 @@
# 超音数(SuperSonic
# SuperSonic (超音数)
**超音数是一个开箱即用且易于扩展的数据问答对话框架**。通过超音数的问答对话界面,用户能够使用自然语言查询数据,系统会选择合适的可视化图表呈现结果。超音数不需要修改或复制数据,只需要在物理数据模型之上构建逻辑语义模型(指标/维度/实体的定义,以及他们的业务含义、相互间关系等),即可开启数据问答体验。与此同时,超音数被设计为可插拔的框架,允许以插件形式来扩展新功能,或者将核心组件与其他系统集成
**SuperSonic融合ChatBI和HeadlessBI打造新一代的数据分析平台**。通过SuperSonic的问答对话界面,用户能够使用自然语言查询数据,系统会选择合适的可视化图表呈现结果。SuperSonic不需要修改或复制数据,只需要在物理数据模型之上构建逻辑语义模型(指标/维度/实体的定义,以及他们的业务含义、相互间关系等),即可开启数据问答体验。与此同时,SuperSonic被设计为可插拔的框架,采用Java SPI机制来扩展定制功能
<img src="./docs/images/supersonic_demo.gif" height="100%" width="100%" align="center"/>
@@ -9,24 +9,24 @@
大型语言模型LLMs如ChatGPT的出现正在重塑信息检索的方式。在数据分析领域学术界和工业界主要关注利用深度学习模型将自然语言查询转换为SQL查询。虽然一些工作显示出有前景的结果但它们的可靠性还达不到生产可用的要求。
在我们看来,为了在实际场景发挥价值,有三个关键点:
1. 引入语义模型层,封装底层数据的上下文(关联、公式等降低SQL生成的**复杂度**。
1. 融合HeadlessBI通过统一语义层封装底层数据细节关联、键值、公式等降低SQL生成的**复杂度**。
2. 通过一前一后的模式映射器和语义修正器来缓解LLM常见的**幻觉**现象。
3. 设计启发式的规则,在一些特定场景提升语义解析的**效率**。
为了验证上述想法,我们开发了超音数项目,并将其应用在实际的内部产品中。与此同时,我们将超音数作为一个可扩展的框架开源,希望能够促进数据问答对话领域的进一步发展。
为了验证上述想法,我们开发了SuperSonic项目,并将其应用在实际的内部产品中。与此同时,我们将SuperSonic作为一个可扩展的框架开源,希望能够促进数据问答对话领域的进一步发展。
## 开箱即用的特性
- 内置对话界面以便*业务用户*输入数据查询。
- 内置图形界面以便*分析工程师*构建语义模型。
- 内置图形界面以便*系统管理员*管理第三方插件和对话助理。
- 内置ChatBI界面以便*业务用户*输入数据查询。
- 内置HeadlessBI界面以便*分析工程师*构建语义模型。
- 内置图形用户界面以便*系统管理员*管理第三方插件和对话助理。
- 支持文本输入的联想和查询问题的推荐。
- 支持多轮对话,根据语境自动切换上下文。
- 支持四级权限控制:主题域级、模型级、列级、行级。
## 易于扩展的组件
超音数的整体架构和主流程如下图所示:
SuperSonic的整体架构和主流程如下图所示:
<img src="./docs/images/supersonic_components.png" height="65%" width="65%" align="center"/>
@@ -44,10 +44,10 @@
## 快速体验
超音数自带样例的语义模型和问答对话,只需以下三步即可快速体验:
SuperSonic自带样例的语义模型和问答对话,只需以下三步即可快速体验:
- 从[release page](https://github.com/tencentmusic/supersonic/releases)下载预先构建好的发行包
- 运行 "bin/supersonic-daemon.sh"启动服务一个Java进程和一个Python进程
- 运行 "assembly/bin/supersonic-daemon.sh start"启动standalone模式的Java服务
- 在浏览器访问http://localhost:9080 开启探索
## 如何构建和部署

View File

@@ -6,11 +6,18 @@ set "baseDir=%~dp0.."
set "buildDir=%baseDir%\build"
set "runtimeDir=%baseDir%\..\runtime"
set "pip_path=pip3"
set "service=%~1"
rem 1. build backend java modules
del /q "%buildDir%\*.tar.gz" 2>NUL
call mvn -f "%baseDir%\..\pom.xml" clean package -DskipTests
IF ERRORLEVEL 1 (
ECHO Failed to build backend Java modules.
EXIT /B 1
)
rem 2. move package to build
echo f|xcopy "%baseDir%\..\launchers\standalone\target\*.tar.gz" "%buildDir%\supersonic-standalone.tar.gz"
@@ -19,6 +26,11 @@ cd "%baseDir%\..\webapp"
call start-fe-prod.bat
copy /y "%baseDir%\..\webapp\supersonic-webapp.tar.gz" "%buildDir%\"
IF ERRORLEVEL 1 (
ECHO Failed to build frontend webapp.
EXIT /B 1
)
rem 4. copy webapp to java classpath
cd "%buildDir%"
tar -zxvf supersonic-webapp.tar.gz
@@ -26,16 +38,23 @@ move supersonic-webapp webapp
move webapp ..\..\launchers\standalone\target\classes
rem 5. build backend python modules
echo "start installing python modules with pip: ${pip_path}"
set requirementPath="%baseDir%/../chat/python/requirements.txt"
%pip_path% install -r %requirementPath%
echo "install python modules success"
if "%service%"=="pyllm" (
echo "start installing python modules with pip: ${pip_path}"
set requirementPath="%baseDir%/../chat/python/requirements.txt"
%pip_path% install -r %requirementPath%
echo "install python modules success"
)
call :BUILD_RUNTIME
:BUILD_RUNTIME
rem 6. reset runtime
rd /s /q "%runtimeDir%"
IF EXIST "%runtimeDir%" (
echo begin to delete dir : %runtimeDir%
rd /s /q "%runtimeDir%"
) ELSE (
echo %runtimeDir% does not exist, create directly
)
mkdir "%runtimeDir%"
tar -zxvf "%buildDir%\supersonic-standalone.tar.gz" -C "%runtimeDir%"
for /d %%f in ("%runtimeDir%\launchers-standalone-*") do (

View File

@@ -4,19 +4,22 @@ set -x
sbinDir=$(cd "$(dirname "$0")"; pwd)
chmod +x $sbinDir/supersonic-common.sh
source $sbinDir/supersonic-common.sh
cd $baseDir
service=$1
#1. build backend java modules
rm -fr ${buildDir}/*.tar.gz
rm -fr dist
set +x
mvn -f $baseDir/../ clean package -DskipTests
# check build result
if [ $? -ne 0 ]; then
echo "Failed to build backend Java modules."
exit 1
fi
#2. move package to build
cp $baseDir/../launchers/semantic/target/*.tar.gz ${buildDir}/supersonic-semantic.tar.gz
cp $baseDir/../launchers/headless/target/*.tar.gz ${buildDir}/supersonic-headless.tar.gz
cp $baseDir/../launchers/chat/target/*.tar.gz ${buildDir}/supersonic-chat.tar.gz
cp $baseDir/../launchers/standalone/target/*.tar.gz ${buildDir}/supersonic-standalone.tar.gz
@@ -26,23 +29,30 @@ cd ../webapp
sh ./start-fe-prod.sh
cp -fr ./supersonic-webapp.tar.gz ${buildDir}/
# check build result
if [ $? -ne 0 ]; then
echo "Failed to build frontend webapp."
exit 1
fi
#4. copy webapp to java classpath
cd $buildDir
tar xvf supersonic-webapp.tar.gz
mv supersonic-webapp webapp
cp -fr webapp ../../launchers/semantic/target/classes
cp -fr webapp ../../launchers/headless/target/classes
cp -fr webapp ../../launchers/chat/target/classes
cp -fr webapp ../../launchers/standalone/target/classes
rm -fr ${buildDir}/webapp
#5. build backend python modules
echo "start installing python modules with pip: ${pip_path}"
requirementPath=$baseDir/../chat/python/requirements.txt
${pip_path} install -r ${requirementPath}
echo "install python modules success"
if [ "$service" == "pyllm" ]; then
echo "start installing python modules with pip: ${pip_path}"
requirementPath=$baseDir/../chat/python/requirements.txt
${pip_path} install -r ${requirementPath}
echo "install python modules success"
fi
#6. reset runtime
rm -fr $runtimeDir/*
rm -fr $runtimeDir/supersonic*
moveAllToRuntime
setEnvToWeb chat
setEnvToWeb semantic
setEnvToWeb headless

View File

@@ -10,15 +10,15 @@ runtimeDir=$baseDir/../runtime
buildDir=$baseDir/build
readonly CHAT_APP_NAME="supersonic_chat"
readonly SEMANTIC_APP_NAME="supersonic_semantic"
readonly LLMPARSER_APP_NAME="supersonic_llmparser"
readonly HEADLESS_APP_NAME="supersonic_headless"
readonly PYLLM_APP_NAME="supersonic_pyllm"
readonly STANDALONE_APP_NAME="supersonic_standalone"
readonly CHAT_SERVICE="chat"
readonly SEMANTIC_SERVICE="semantic"
readonly LLMPARSER_SERVICE="llmparser"
readonly HEADLESS_SERVICE="headless"
readonly PYLLM_SERVICE="pyllm"
readonly STANDALONE_SERVICE="standalone"
readonly LLMPARSER_HOST="127.0.0.1"
readonly LLMPARSER_PORT="9092"
readonly PYLLM_HOST="127.0.0.1"
readonly PYLLM_PORT="9092"
function setEnvToWeb {
model_name=$1
@@ -29,11 +29,15 @@ function setEnvToWeb {
function moveToRuntime {
model_name=$1
tar -zxvf ${buildDir}/supersonic-${model_name}.tar.gz -C ${runtimeDir}
mv ${runtimeDir}/launchers-${model_name}-* ${runtimeDir}/supersonic-${model_name}
mkdir -p ${runtimeDir}/supersonic-${model_name}/webapp
cp -fr ${buildDir}/webapp/* ${runtimeDir}/supersonic-${model_name}/webapp
file="${buildDir}/supersonic-${model_name}.tar.gz"
if [ -f "$file" ]; then
tar -zxvf "$file" -C ${runtimeDir}
mv ${runtimeDir}/launchers-${model_name}-* ${runtimeDir}/supersonic-${model_name}
mkdir -p ${runtimeDir}/supersonic-${model_name}/webapp
cp -fr ${buildDir}/webapp/* ${runtimeDir}/supersonic-${model_name}/webapp
else
echo "File $file does not exist. Skipping the move to runtime."
fi
}
function moveAllToRuntime {
@@ -42,7 +46,7 @@ function moveAllToRuntime {
mv ${buildDir}/supersonic-webapp ${buildDir}/webapp
moveToRuntime chat
moveToRuntime semantic
moveToRuntime headless
moveToRuntime standalone
rm -fr ${buildDir}/webapp
}
@@ -81,23 +85,23 @@ function runJavaService {
# run python service
function runPythonService {
pythonRunDir=${runtimeDir}/supersonic-${model_name}/llmparser
pythonRunDir=${runtimeDir}/supersonic-${model_name}/pyllm
cd $pythonRunDir
nohup ${python_path} supersonic_llmparser.py > $pythonRunDir/llmparser.log 2>&1 &
nohup ${python_path} supersonic_pyllm.py > $pythonRunDir/pyllm.log 2>&1 &
# add health check
for i in {1..10}
do
echo "llmparser health check attempt $i..."
response=$(curl -s http://${LLMPARSER_HOST}:${LLMPARSER_PORT}/health)
echo "llmparser health check response: $response"
echo "pyllm health check attempt $i..."
response=$(curl -s http://${PYLLM_HOST}:${PYLLM_PORT}/health)
echo "pyllm health check response: $response"
status_ok="Healthy"
if [[ $response == *$status_ok* ]] ; then
echo "llmparser Health check passed."
echo "pyllm Health check passed."
break
else
if [ "$i" -eq 10 ]; then
echo "llmparser Health check failed after 10 attempts."
echo "May still downloading model files. Please check llmparser.log in runtime directory."
echo "pyllm Health check failed after 10 attempts."
echo "May still downloading model files. Please check pyllm.log in runtime directory."
fi
echo "Retrying after 5 seconds..."
sleep 5

View File

@@ -9,10 +9,10 @@ set "main_class=com.tencent.supersonic.StandaloneLauncher"
set "python_path=python"
set "pip_path=pip3"
set "standalone_service=standalone"
set "llmparser_service=llmparser"
set "pyllm_service=pyllm"
set "javaRunDir=%runtimeDir%\supersonic-standalone"
set "pythonRunDir=%runtimeDir%\supersonic-standalone\llmparser"
set "pythonRunDir=%runtimeDir%\supersonic-standalone\pyllm"
set "command=%~1"
set "service=%~2"
@@ -21,6 +21,10 @@ if "%service%"=="" (
set "service=%standalone_service%"
)
IF "%service%"=="pyllm" (
SET "llmProxy=PythonLLMProxy"
)
call :BUILD_RUNTIME
if "%command%"=="restart" (
@@ -42,27 +46,23 @@ if "%command%"=="restart" (
)
:START
if "%service%"=="%llmparser_service%" (
if "%service%"=="%pyllm_service%" (
call :START_PYTHON
call :START_JAVA
goto :EOF
)
call :START_PYTHON
call :START_JAVA
goto :EOF
:STOP
if "%service%"=="%llmparser_service%" (
call :STOP_PYTHON
goto :EOF
)
call :STOP_PYTHON
call :STOP_JAVA
goto :EOF
:START_PYTHON
echo 'python service starting, see logs in llmparser/llmparser.log'
echo 'python service starting, see logs in pyllm/pyllm.log'
cd "%pythonRunDir%"
start /B %python_path% supersonic_llmparser.py > %pythonRunDir%\llmparser.log 2>&1
start /B %python_path% supersonic_pyllm.py > %pythonRunDir%\pyllm.log 2>&1
timeout /t 10 >nul
echo 'python service started'
goto :EOF
@@ -71,9 +71,9 @@ if "%command%"=="restart" (
echo 'java service starting, see logs in logs/'
cd "%javaRunDir%"
if not exist "%runtimeDir%\supersonic-standalone\logs" mkdir "%runtimeDir%\supersonic-standalone\logs"
set "libDir=%runtimeDir%\supersonic-%service%\lib"
set "confDir=%runtimeDir%\supersonic-%service%\conf"
set "webDir=%runtimeDir%\supersonic-%service%\webapp"
set "libDir=%runtimeDir%\supersonic-standalone\lib"
set "confDir=%runtimeDir%\supersonic-standalone\conf"
set "webDir=%runtimeDir%\supersonic-standalone\webapp"
set "classpath=%confDir%;%webDir%;%libDir%\*"
set "java-command=-Dfile.encoding=UTF-8 -Duser.language=Zh -Duser.region=CN -Duser.timezone=GMT+08 -Xms1024m -Xmx2048m -cp %CLASSPATH% %MAIN_CLASS%"
start /B java %java-command% >nul 2>&1
@@ -96,7 +96,7 @@ if "%command%"=="restart" (
goto :EOF
:RELOAD_EXAMPLE
cd "%runtimeDir%\supersonic-standalone\llmparser\sql"
cd "%runtimeDir%\supersonic-standalone\pyllm\sql"
start %python_path% examples_reload_run.py
goto :EOF

View File

@@ -22,8 +22,9 @@ app_name=$STANDALONE_APP_NAME
main_class="com.tencent.supersonic.StandaloneLauncher"
model_name=$service
if [ "$service" == "llmparser" ]; then
if [ "$service" == "pyllm" ]; then
model_name=${STANDALONE_SERVICE}
export llmProxy=PythonLLMProxy
fi
cd $baseDir
@@ -32,8 +33,8 @@ cd $baseDir
function setMainClass {
if [ "$service" == $CHAT_SERVICE ]; then
main_class="com.tencent.supersonic.ChatLauncher"
elif [ "$service" == $SEMANTIC_SERVICE ]; then
main_class="com.tencent.supersonic.SemanticLauncher"
elif [ "$service" == $HEADLESS_SERVICE ]; then
main_class="com.tencent.supersonic.HeadlessLauncher"
fi
}
setMainClass
@@ -41,16 +42,16 @@ setMainClass
function setAppName {
if [ "$service" == $CHAT_SERVICE ]; then
app_name=$CHAT_APP_NAME
elif [ "$service" == $SEMANTIC_SERVICE ]; then
app_name=$SEMANTIC_APP_NAME
elif [ "$service" == $LLMPARSER_SERVICE ]; then
app_name=$LLMPARSER_APP_NAME
elif [ "$service" == $HEADLESS_SERVICE ]; then
app_name=$HEADLESS_APP_NAME
elif [ "$service" == $PYLLM_SERVICE ]; then
app_name=$PYLLM_APP_NAME
fi
}
setAppName
function reloadExamples {
pythonRunDir=${runtimeDir}/supersonic-${model_name}/llmparser
pythonRunDir=${runtimeDir}/supersonic-${model_name}/pyllm
cd $pythonRunDir/sql
${python_path} examples_reload_run.py
}
@@ -61,7 +62,7 @@ function start()
local_app_name=$1
pid=$(ps aux |grep ${local_app_name} | grep -v grep | awk '{print $2}')
if [[ "$pid" == "" ]]; then
if [[ ${local_app_name} == $LLMPARSER_APP_NAME ]]; then
if [[ ${local_app_name} == $PYLLM_APP_NAME ]]; then
runPythonService ${local_app_name}
else
runJavaService ${local_app_name}
@@ -87,7 +88,7 @@ function stop()
function reload()
{
if [[ $1 == $LLMPARSER_APP_NAME ]]; then
if [[ $1 == $PYLLM_APP_NAME ]]; then
reloadExamples
fi
}
@@ -95,11 +96,11 @@ function reload()
# 4. execute command operation
case "$command" in
start)
if [ "$service" == $STANDALONE_SERVICE ]; then
echo "Starting $LLMPARSER_APP_NAME"
start $LLMPARSER_APP_NAME
if [ "$service" == $PYLLM_SERVICE ]; then
echo "Starting $app_name"
start $app_name
echo "Starting $STANDALONE_APP_NAME"
start $STANDALONE_APP_NAME
else
echo "Starting $app_name"
start $app_name
@@ -107,15 +108,10 @@ case "$command" in
echo "Start success"
;;
stop)
if [ "$service" == $STANDALONE_SERVICE ]; then
echo "Stopping $LLMPARSER_APP_NAME"
stop $LLMPARSER_APP_NAME
echo "Stopping $app_name"
stop $app_name
else
echo "Stopping $app_name"
stop ${app_name}
fi
echo "Stopping $app_name"
stop $app_name
echo "Stopping $PYLLM_APP_NAME"
stop $PYLLM_APP_NAME
echo "Stop success"
;;
reload)
@@ -124,15 +120,15 @@ case "$command" in
echo "Reload success"
;;
restart)
if [ "$service" == $STANDALONE_SERVICE ]; then
if [ "$service" == $PYLLM_SERVICE ]; then
echo "Stopping ${app_name}"
stop ${app_name}
echo "Stopping ${LLMPARSER_APP_NAME}"
stop $LLMPARSER_APP_NAME
echo "Starting ${LLMPARSER_APP_NAME}"
start $LLMPARSER_APP_NAME
echo "Stopping ${STANDALONE_APP_NAME}"
stop $STANDALONE_APP_NAME
echo "Starting ${app_name}"
start ${app_name}
echo "Starting ${STANDALONE_APP_NAME}"
start $STANDALONE_APP_NAME
else
echo "Stopping ${app_name}"
stop ${app_name}

View File

@@ -22,7 +22,7 @@
</fileSet>
<fileSet>
<directory>${project.basedir}/../../chat/python</directory>
<outputDirectory>llmparser</outputDirectory>
<outputDirectory>pyllm</outputDirectory>
<fileMode>0777</fileMode>
<directoryMode>0755</directoryMode>
</fileSet>

View File

@@ -7,6 +7,9 @@ import com.tencent.supersonic.auth.api.authentication.request.UserReq;
import java.util.List;
import java.util.Set;
/**
* UserAdaptor defines some interfaces for obtaining user and organization information
*/
public interface UserAdaptor {
List<String> getUserNames();

View File

@@ -24,4 +24,13 @@ public class AuthenticationConfig {
@Value("${authentication.token.http.header.key:Authorization}")
private String tokenHttpHeaderKey;
@Value("${authentication.app.appId:appId}")
private String appId;
@Value("${authentication.app.timestamp:timestamp}")
private String timestamp;
@Value("${authentication.app.signature:signature}")
private String signature;
}

View File

@@ -1,11 +1,15 @@
package com.tencent.supersonic.auth.api.authentication.pojo;
import com.google.common.collect.Lists;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import java.util.List;
@Data
@AllArgsConstructor
@NoArgsConstructor
public class Organization {
private String id;

View File

@@ -28,6 +28,11 @@ public class User {
return new User(1L, "admin", "admin", "admin@email", 1);
}
public static User getAppUser(int appId) {
String name = String.format("app_%s", appId);
return new User(1L, name, name, "", 1);
}
public String getDisplayName() {
return StringUtils.isBlank(displayName) ? name : displayName;
}

View File

@@ -4,11 +4,15 @@ import com.tencent.supersonic.auth.api.authentication.pojo.Organization;
import com.tencent.supersonic.auth.api.authentication.pojo.User;
import com.tencent.supersonic.auth.api.authentication.request.UserReq;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.util.List;
import java.util.Set;
public interface UserService {
User getCurrentUser(HttpServletRequest httpServletRequest, HttpServletResponse httpServletResponse);
List<String> getUserNames();
List<User> getUserList();

View File

@@ -2,6 +2,11 @@ package com.tencent.supersonic.auth.api.authentication.utils;
import com.tencent.supersonic.auth.api.authentication.pojo.User;
import com.tencent.supersonic.auth.api.authentication.service.UserStrategy;
import com.tencent.supersonic.common.pojo.SysParameter;
import com.tencent.supersonic.common.service.SysParameterService;
import com.tencent.supersonic.common.util.ContextUtils;
import org.springframework.util.CollectionUtils;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
@@ -14,7 +19,14 @@ public final class UserHolder {
}
public static User findUser(HttpServletRequest request, HttpServletResponse response) {
return REPO.findUser(request, response);
User user = REPO.findUser(request, response);
SysParameterService sysParameterService = ContextUtils.getBean(SysParameterService.class);
SysParameter sysParameter = sysParameterService.getSysParameter();
if (!CollectionUtils.isEmpty(sysParameter.getAdmins())
&& sysParameter.getAdmins().contains(user.getName())) {
user.setIsAdmin(1);
}
return user;
}
}

View File

@@ -16,6 +16,9 @@ import java.util.List;
import java.util.Set;
import java.util.stream.Collectors;
/**
* DefaultUserAdaptor provides a default method to obtain user and organization information
*/
public class DefaultUserAdaptor implements UserAdaptor {
private List<UserDO> getUserDOList() {
@@ -41,7 +44,17 @@ public class DefaultUserAdaptor implements UserAdaptor {
@Override
public List<Organization> getOrganizationTree() {
return Lists.newArrayList();
Organization superSonic = new Organization("1", "0",
"SuperSonic", "SuperSonic", Lists.newArrayList(), true);
Organization hr = new Organization("2", "1",
"Hr", "SuperSonic/Hr", Lists.newArrayList(), false);
Organization sales = new Organization("3", "1",
"Sales", "SuperSonic/Sales", Lists.newArrayList(), false);
Organization marketing = new Organization("4", "1",
"Marketing", "SuperSonic/Marketing", Lists.newArrayList(), false);
List<Organization> subOrganization = Lists.newArrayList(hr, sales, marketing);
superSonic.setSubOrganizations(subOrganization);
return Lists.newArrayList(superSonic);
}
private User convert(UserDO userDO) {

View File

@@ -5,16 +5,17 @@ import com.tencent.supersonic.auth.api.authentication.constant.UserConstants;
import com.tencent.supersonic.auth.authentication.service.UserServiceImpl;
import com.tencent.supersonic.auth.authentication.utils.UserTokenUtils;
import com.tencent.supersonic.common.util.S2ThreadContext;
import java.lang.reflect.Field;
import java.util.Arrays;
import java.util.List;
import javax.servlet.http.HttpServletRequest;
import org.apache.catalina.connector.RequestFacade;
import org.apache.commons.lang3.StringUtils;
import org.apache.logging.log4j.util.Strings;
import org.apache.tomcat.util.http.MimeHeaders;
import org.springframework.util.CollectionUtils;
import org.springframework.web.multipart.support.StandardMultipartHttpServletRequest;
import org.springframework.web.servlet.HandlerInterceptor;
import javax.servlet.http.HttpServletRequest;
import java.lang.reflect.Field;
import java.util.Arrays;
import java.util.List;
public abstract class AuthenticationInterceptor implements HandlerInterceptor {
@@ -29,7 +30,6 @@ public abstract class AuthenticationInterceptor implements HandlerInterceptor {
protected S2ThreadContext s2ThreadContext;
protected boolean isExcludedUri(String uri) {
String excludePathStr = authenticationConfig.getExcludePath();
if (Strings.isEmpty(excludePathStr)) {
@@ -59,6 +59,10 @@ public abstract class AuthenticationInterceptor implements HandlerInterceptor {
return "true".equalsIgnoreCase(internal);
}
protected boolean isAppRequest(HttpServletRequest request) {
String appId = request.getHeader(authenticationConfig.getAppId());
return StringUtils.isNotBlank(appId);
}
protected void reflectSetparam(HttpServletRequest request, String key, String value) {
try {

View File

@@ -10,12 +10,12 @@ import com.tencent.supersonic.common.pojo.exception.AccessException;
import com.tencent.supersonic.common.util.ContextUtils;
import com.tencent.supersonic.common.util.S2ThreadContext;
import com.tencent.supersonic.common.util.ThreadContext;
import java.lang.reflect.Method;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.springframework.web.method.HandlerMethod;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.lang.reflect.Method;
@Slf4j
public class DefaultAuthenticationInterceptor extends AuthenticationInterceptor {
@@ -35,7 +35,10 @@ public class DefaultAuthenticationInterceptor extends AuthenticationInterceptor
setFakerUser(request);
return true;
}
if (isAppRequest(request)) {
setFakerUser(request);
return true;
}
if (handler instanceof HandlerMethod) {
HandlerMethod handlerMethod = (HandlerMethod) handler;
Method method = handlerMethod.getMethod();
@@ -76,5 +79,4 @@ public class DefaultAuthenticationInterceptor extends AuthenticationInterceptor
s2ThreadContext.set(threadContext);
}
}

View File

@@ -20,7 +20,6 @@ public class UserRepositoryImpl implements UserRepository {
this.userDOMapper = userDOMapper;
}
@Override
public List<UserDO> getUserList() {
return userDOMapper.selectByExample(new UserDOExample());
@@ -40,5 +39,4 @@ public class UserRepositoryImpl implements UserRepository {
return userDOOptional.orElse(null);
}
}

View File

@@ -5,18 +5,18 @@ import com.tencent.supersonic.auth.api.authentication.pojo.Organization;
import com.tencent.supersonic.auth.api.authentication.pojo.User;
import com.tencent.supersonic.auth.api.authentication.request.UserReq;
import com.tencent.supersonic.auth.api.authentication.service.UserService;
import com.tencent.supersonic.auth.api.authentication.utils.UserHolder;
import java.util.List;
import java.util.Set;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import lombok.extern.slf4j.Slf4j;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.util.List;
import java.util.Set;
@RestController
@RequestMapping("/api/auth/user")
@@ -29,10 +29,9 @@ public class UserController {
this.userService = userService;
}
@GetMapping("/getCurrentUser")
public User getCurrentUser(HttpServletRequest httpServletRequest, HttpServletResponse httpServletResponse) {
return UserHolder.findUser(httpServletRequest, httpServletResponse);
return userService.getCurrentUser(httpServletRequest, httpServletResponse);
}
@GetMapping("/getUserNames")
@@ -70,5 +69,4 @@ public class UserController {
return userService.login(userCmd);
}
}

View File

@@ -4,15 +4,38 @@ import com.tencent.supersonic.auth.api.authentication.pojo.Organization;
import com.tencent.supersonic.auth.api.authentication.pojo.User;
import com.tencent.supersonic.auth.api.authentication.request.UserReq;
import com.tencent.supersonic.auth.api.authentication.service.UserService;
import com.tencent.supersonic.auth.api.authentication.utils.UserHolder;
import com.tencent.supersonic.auth.authentication.utils.ComponentFactory;
import com.tencent.supersonic.common.pojo.SysParameter;
import com.tencent.supersonic.common.service.SysParameterService;
import org.springframework.stereotype.Service;
import org.springframework.util.CollectionUtils;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.util.List;
import java.util.Set;
import org.springframework.stereotype.Service;
@Service
public class UserServiceImpl implements UserService {
private SysParameterService sysParameterService;
public UserServiceImpl(SysParameterService sysParameterService) {
this.sysParameterService = sysParameterService;
}
@Override
public User getCurrentUser(HttpServletRequest httpServletRequest, HttpServletResponse httpServletResponse) {
User user = UserHolder.findUser(httpServletRequest, httpServletResponse);
if (user != null) {
SysParameter sysParameter = sysParameterService.getSysParameter();
if (!CollectionUtils.isEmpty(sysParameter.getAdmins())
&& sysParameter.getAdmins().contains(user.getName())) {
user.setIsAdmin(1);
}
}
return user;
}
@Override
public List<String> getUserNames() {

View File

@@ -20,5 +20,4 @@ public class FakeUserStrategy implements UserStrategy {
return User.getFakeUser();
}
}

View File

@@ -58,7 +58,6 @@ public class UserTokenUtils {
return generate(claims);
}
public User getUser(HttpServletRequest request) {
String token = request.getHeader(authenticationConfig.getTokenHttpHeaderKey());
final Claims claims = getClaims(token);
@@ -120,5 +119,4 @@ public class UserTokenUtils {
.compact();
}
}

View File

@@ -1,6 +1,7 @@
package com.tencent.supersonic.auth.authorization.application;
package com.tencent.supersonic.auth.authorization.service;
import com.google.common.base.Strings;
import com.google.common.collect.Lists;
import com.google.gson.Gson;
import com.tencent.supersonic.auth.api.authentication.pojo.User;
import com.tencent.supersonic.auth.api.authentication.service.UserService;
@@ -75,7 +76,6 @@ public class AuthServiceImpl implements AuthService {
jdbcTemplate.update("delete from s2_auth_groups where group_id = ?", group.getGroupId());
}
@Override
public AuthorizedResourceResp queryAuthorizedResources(QueryAuthResReq req, User user) {
Set<String> userOrgIds = userService.getUserAllOrgId(user.getName());
@@ -109,8 +109,11 @@ public class AuthServiceImpl implements AuthService {
}
}
if (req.getModelId() != null) {
List<AuthGroup> authGroups = authGroupsByModelId.get(req.getModelId());
if (!CollectionUtils.isEmpty(req.getModelIds())) {
List<AuthGroup> authGroups = Lists.newArrayList();
for (Long modelId : authGroupsByModelId.keySet()) {
authGroups.addAll(authGroupsByModelId.getOrDefault(modelId, Lists.newArrayList()));
}
if (!CollectionUtils.isEmpty(authGroups)) {
for (AuthGroup group : authGroups) {
if (group.getDimensionFilters() != null

View File

@@ -23,7 +23,7 @@
</dependency>
<dependency>
<groupId>com.tencent.supersonic</groupId>
<artifactId>semantic-api</artifactId>
<artifactId>headless-api</artifactId>
<version>${project.version}</version>
<scope>compile</scope>
</dependency>

View File

@@ -1,23 +0,0 @@
package com.tencent.supersonic.chat.api.pojo;
import com.tencent.supersonic.chat.api.component.SemanticQuery;
import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
import lombok.Data;
import lombok.NoArgsConstructor;
import java.util.ArrayList;
import java.util.List;
@Data
@NoArgsConstructor
public class QueryContext {
private QueryReq request;
private List<SemanticQuery> candidateQueries = new ArrayList<>();
private SchemaMapInfo mapInfo = new SchemaMapInfo();
private SchemaModelClusterMapInfo modelClusterMapInfo = new SchemaModelClusterMapInfo();
public QueryContext(QueryReq request) {
this.request = request;
}
}

View File

@@ -1,13 +1,13 @@
package com.tencent.supersonic.chat.api.pojo;
import com.tencent.supersonic.chat.api.pojo.response.SqlInfo;
import com.tencent.supersonic.chat.api.pojo.request.QueryFilter;
import com.tencent.supersonic.chat.api.pojo.response.EntityInfo;
import com.tencent.supersonic.chat.api.pojo.response.SqlInfo;
import com.tencent.supersonic.common.pojo.DateConf;
import com.tencent.supersonic.common.pojo.ModelCluster;
import com.tencent.supersonic.common.pojo.Order;
import com.tencent.supersonic.common.pojo.QueryType;
import com.tencent.supersonic.common.pojo.enums.QueryType;
import com.tencent.supersonic.common.pojo.enums.AggregateTypeEnum;
import com.tencent.supersonic.common.pojo.enums.FilterType;
import lombok.Data;
@@ -42,7 +42,7 @@ public class SemanticParseInfo {
private Map<String, Object> properties = new HashMap<>();
private EntityInfo entityInfo;
private SqlInfo sqlInfo = new SqlInfo();
private QueryType queryType = QueryType.OTHER;
private QueryType queryType = QueryType.ID;
public String getModelClusterKey() {
if (model == null) {
@@ -88,10 +88,11 @@ public class SemanticParseInfo {
private Map<Long, Integer> getModelElementCountMap() {
Map<Long, Integer> elementCountMap = new HashMap<>();
elementMatches.forEach(element -> {
int count = elementCountMap.getOrDefault(element.getElement().getModel(), 0);
elementCountMap.put(element.getElement().getModel(), count + 1);
});
elementMatches.stream().filter(element -> element.getElement().getModel() != null)
.forEach(element -> {
int count = elementCountMap.getOrDefault(element.getElement().getModel(), 0);
elementCountMap.put(element.getElement().getModel(), count + 1);
});
return elementCountMap;
}

View File

@@ -1,15 +1,15 @@
package com.tencent.supersonic.chat.api.pojo;
import org.springframework.util.CollectionUtils;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import java.util.Set;
import java.util.stream.Collectors;
import org.springframework.util.CollectionUtils;
public class SemanticSchema implements Serializable {
@@ -57,19 +57,19 @@ public class SemanticSchema implements Serializable {
switch (elementType) {
case ENTITY:
element = getElementsByName(name, getEntities());
element = getElementsByNameOrAlias(name, getEntities());
break;
case MODEL:
element = getElementsByName(name, getModels());
element = getElementsByNameOrAlias(name, getModels());
break;
case METRIC:
element = getElementsByName(name, getMetrics());
element = getElementsByNameOrAlias(name, getMetrics());
break;
case DIMENSION:
element = getElementsByName(name, getDimensions());
element = getElementsByNameOrAlias(name, getDimensions());
break;
case VALUE:
element = getElementsByName(name, getDimensionValues());
element = getElementsByNameOrAlias(name, getDimensionValues());
break;
default:
}
@@ -117,7 +117,8 @@ public class SemanticSchema implements Serializable {
public List<SchemaElement> getTags(Set<Long> modelIds) {
List<SchemaElement> tags = new ArrayList<>();
modelSchemaList.stream().filter(schemaElement -> modelIds.contains(schemaElement.getModel()))
modelSchemaList.stream().filter(schemaElement ->
modelIds.contains(schemaElement.getModel().getModel()))
.forEach(d -> tags.addAll(d.getTags()));
return tags;
}
@@ -139,6 +140,11 @@ public class SemanticSchema implements Serializable {
return entities;
}
public List<SchemaElement> getEntities(Set<Long> modelIds) {
List<SchemaElement> entities = getEntities();
return getElementsByModelId(modelIds, entities);
}
private List<SchemaElement> getElementsByModelId(Set<Long> modelIds, List<SchemaElement> elements) {
return elements.stream()
.filter(schemaElement -> modelIds.contains(schemaElement.getModel()))
@@ -151,10 +157,12 @@ public class SemanticSchema implements Serializable {
.findFirst();
}
private Optional<SchemaElement> getElementsByName(String name, List<SchemaElement> elements) {
private Optional<SchemaElement> getElementsByNameOrAlias(String name, List<SchemaElement> elements) {
return elements.stream()
.filter(schemaElement -> name.equals(schemaElement.getName()))
.findFirst();
.filter(schemaElement ->
name.equals(schemaElement.getName()) || (Objects.nonNull(schemaElement.getAlias())
&& schemaElement.getAlias().contains(name))
).findFirst();
}
public List<SchemaElement> getModels() {

View File

@@ -10,7 +10,7 @@ import lombok.NoArgsConstructor;
@NoArgsConstructor
@AllArgsConstructor
@Builder
public class SolvedQueryReq {
public class SimilarQueryReq {
private Long queryId;

View File

@@ -2,7 +2,7 @@ package com.tencent.supersonic.chat.api.pojo.response;
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
import com.tencent.supersonic.chat.api.pojo.request.ChatDefaultConfigReq;
import com.tencent.supersonic.chat.api.pojo.request.ChatDefaultConfigReq.TimeMode;
import com.tencent.supersonic.common.pojo.Constants;
import lombok.Data;
@@ -26,6 +26,6 @@ public class ChatDefaultRichConfigResp {
*/
private String period = Constants.DAY;
private ChatDefaultConfigReq.TimeMode timeMode;
private TimeMode timeMode;
}

View File

@@ -12,7 +12,6 @@ public class ParseResp {
private Long queryId;
private ParseState state;
private List<SemanticParseInfo> selectedParses = Lists.newArrayList();
private List<SemanticParseInfo> candidateParses = Lists.newArrayList();
private ParseTimeCostDO parseTimeCost = new ParseTimeCostDO();
public enum ParseState {

View File

@@ -6,6 +6,6 @@ import java.util.List;
@Data
public class QueryRecallResp {
private List<SolvedQueryRecallResp> solvedQueryRecallRespList;
private List<SimilarQueryRecallResp> solvedQueryRecallRespList;
private Long queryTimeCost;
}

View File

@@ -1,9 +1,9 @@
package com.tencent.supersonic.chat.api.pojo.response;
import java.util.Date;
import java.util.List;
import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
import lombok.Data;
import java.util.Date;
import java.util.List;
@Data
public class QueryResp {
@@ -16,4 +16,7 @@ public class QueryResp {
private String queryText;
private QueryResult queryResult;
private List<SemanticParseInfo> parseInfos;
private List<SimilarQueryRecallResp> similarQueries;
}

View File

@@ -1,11 +1,12 @@
package com.tencent.supersonic.chat.api.pojo.response;
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
import com.tencent.supersonic.common.pojo.QueryAuthorization;
import com.tencent.supersonic.common.pojo.QueryColumn;
import lombok.Data;
import java.util.List;
import java.util.Map;
import lombok.Data;
@Data
public class QueryResult {
@@ -22,4 +23,5 @@ public class QueryResult {
private Object response;
private List<Map<String, Object>> queryResults;
private Long queryTimeCost;
private List<SchemaElement> recommendedDimensions;
}

View File

@@ -6,7 +6,7 @@ import lombok.Data;
@Data
@Builder
public class SolvedQueryRecallResp {
public class SimilarQueryRecallResp {
private Long queryId;

View File

@@ -22,6 +22,69 @@
<artifactId>spring-context</artifactId>
</dependency>
<dependency>
<groupId>com.hankcs</groupId>
<artifactId>hanlp</artifactId>
<version>${hanlp.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${hadoop.version}</version>
<exclusions>
<exclusion>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
</exclusion>
<exclusion>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>${hadoop.version}</version>
<exclusions>
<exclusion>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
</exclusion>
<exclusion>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${hadoop.version}</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.curator</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
@@ -40,11 +103,6 @@
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
@@ -77,32 +135,20 @@
</dependency>
<dependency>
<groupId>com.tencent.supersonic</groupId>
<artifactId>chat-knowledge</artifactId>
<artifactId>headless-api</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>com.tencent.supersonic</groupId>
<artifactId>semantic-api</artifactId>
<artifactId>headless-core</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>com.tencent.supersonic</groupId>
<artifactId>semantic-query</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.tencent.supersonic</groupId>
<artifactId>chat-api</artifactId>
<version>${project.version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.tencent.supersonic</groupId>
<artifactId>semantic-query</artifactId>
<version>${project.version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.github.xkzhangsan</groupId>
@@ -116,6 +162,12 @@
<version>${mockito-inline.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.tencent.supersonic</groupId>
<artifactId>headless-server</artifactId>
<version>${project.version}</version>
<scope>compile</scope>
</dependency>
</dependencies>

View File

@@ -1,8 +1,8 @@
package com.hankcs.hanlp.collection.trie.bintrie;
import com.hankcs.hanlp.corpus.io.ByteArray;
import com.tencent.supersonic.chat.core.knowledge.LoadRemoveService;
import com.tencent.supersonic.common.util.ContextUtils;
import com.tencent.supersonic.knowledge.service.LoadRemoveService;
import java.io.DataOutputStream;
import java.io.IOException;
import java.io.ObjectInput;

View File

@@ -3,7 +3,7 @@ package com.hankcs.hanlp.seg.common;
import com.hankcs.hanlp.corpus.tag.Nature;
import com.hankcs.hanlp.dictionary.CoreDictionary;
import com.hankcs.hanlp.dictionary.CustomDictionary;
import com.tencent.supersonic.knowledge.utils.HanlpHelper;
import com.tencent.supersonic.chat.core.utils.HanlpHelper;
import lombok.Data;
import lombok.ToString;

View File

@@ -1,8 +0,0 @@
package com.tencent.supersonic.chat.agent.tool;
public enum AgentToolType {
RULE,
LLM_S2SQL,
PLUGIN,
INTERPRET
}

View File

@@ -1,12 +0,0 @@
package com.tencent.supersonic.chat.agent.tool;
import lombok.Data;
import java.util.List;
@Data
public class LLMParserTool extends CommonAgentTool {
private List<String> exampleQuestions;
}

View File

@@ -1,16 +0,0 @@
package com.tencent.supersonic.chat.agent.tool;
import com.tencent.supersonic.chat.parser.llm.interpret.MetricOption;
import lombok.Data;
import java.util.List;
@Data
public class MetricInterpretTool extends AgentTool {
private Long modelId;
private List<MetricOption> metricOptions;
}

View File

@@ -1,14 +0,0 @@
package com.tencent.supersonic.chat.config;
import com.tencent.supersonic.semantic.api.model.response.DimSchemaResp;
import com.tencent.supersonic.semantic.api.model.response.MetricSchemaResp;
import java.util.List;
import lombok.Data;
@Data
public class EntityInternalDetail {
List<DimSchemaResp> dimensionList;
List<MetricSchemaResp> metricList;
}

View File

@@ -1,16 +1,18 @@
package com.tencent.supersonic.chat.agent;
package com.tencent.supersonic.chat.core.agent;
import com.alibaba.fastjson.JSONObject;
import com.google.common.collect.Lists;
import com.tencent.supersonic.chat.agent.tool.AgentToolType;
import com.tencent.supersonic.common.pojo.RecordInfo;
import java.util.Objects;
import lombok.Data;
import org.springframework.util.CollectionUtils;
import java.util.Collection;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.stream.Collectors;
import lombok.Data;
import org.springframework.util.CollectionUtils;
@Data
public class Agent extends RecordInfo {
@@ -20,10 +22,13 @@ public class Agent extends RecordInfo {
private String name;
private String description;
//0 offline, 1 online
/**
* 0 offline, 1 online
*/
private Integer status;
private List<String> examples;
private String agentConfig;
public List<String> getTools(AgentToolType type) {
Map map = JSONObject.parseObject(agentConfig, Map.class);
if (CollectionUtils.isEmpty(map) || map.get("tools") == null) {
@@ -46,4 +51,27 @@ public class Agent extends RecordInfo {
return enableSearch != null && enableSearch == 1;
}
public static boolean containsAllModel(Set<Long> detectModelIds) {
return !CollectionUtils.isEmpty(detectModelIds) && detectModelIds.contains(-1L);
}
public List<NL2SQLTool> getParserTools(AgentToolType agentToolType) {
List<String> tools = this.getTools(agentToolType);
if (CollectionUtils.isEmpty(tools)) {
return Lists.newArrayList();
}
return tools.stream().map(tool -> JSONObject.parseObject(tool, NL2SQLTool.class))
.collect(Collectors.toList());
}
public Set<Long> getModelIds(AgentToolType agentToolType) {
List<NL2SQLTool> commonAgentTools = getParserTools(agentToolType);
if (CollectionUtils.isEmpty(commonAgentTools)) {
return new HashSet<>();
}
return commonAgentTools.stream().map(NL2SQLTool::getModelIds)
.filter(modelIds -> !CollectionUtils.isEmpty(modelIds))
.flatMap(Collection::stream)
.collect(Collectors.toSet());
}
}

View File

@@ -1,7 +1,6 @@
package com.tencent.supersonic.chat.agent;
package com.tencent.supersonic.chat.core.agent;
import com.google.common.collect.Lists;
import com.tencent.supersonic.chat.agent.tool.AgentTool;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;

View File

@@ -1,4 +1,4 @@
package com.tencent.supersonic.chat.agent.tool;
package com.tencent.supersonic.chat.core.agent;
import lombok.AllArgsConstructor;
import lombok.Data;

View File

@@ -0,0 +1,8 @@
package com.tencent.supersonic.chat.core.agent;
public enum AgentToolType {
NL2SQL_RULE,
NL2SQL_LLM,
PLUGIN,
ANALYTICS
}

View File

@@ -0,0 +1,11 @@
package com.tencent.supersonic.chat.core.agent;
import lombok.Data;
@Data
public class DataAnalyticsTool extends AgentTool {
private Long modelId;
}

View File

@@ -0,0 +1,12 @@
package com.tencent.supersonic.chat.core.agent;
import lombok.Data;
import java.util.List;
@Data
public class LLMParserTool extends NL2SQLTool {
private List<String> exampleQuestions;
}

View File

@@ -1,4 +1,4 @@
package com.tencent.supersonic.chat.agent.tool;
package com.tencent.supersonic.chat.core.agent;
import java.util.List;
@@ -9,7 +9,7 @@ import lombok.NoArgsConstructor;
@Data
@NoArgsConstructor
@AllArgsConstructor
public class CommonAgentTool extends AgentTool {
public class NL2SQLTool extends AgentTool {
protected List<Long> modelIds;

View File

@@ -1,4 +1,4 @@
package com.tencent.supersonic.chat.agent.tool;
package com.tencent.supersonic.chat.core.agent;
import lombok.Data;

View File

@@ -1,4 +1,4 @@
package com.tencent.supersonic.chat.agent.tool;
package com.tencent.supersonic.chat.core.agent;
import lombok.Data;
@@ -7,7 +7,7 @@ import org.apache.commons.collections.CollectionUtils;
import java.util.List;
@Data
public class RuleQueryTool extends CommonAgentTool {
public class RuleParserTool extends NL2SQLTool {
private List<String> queryModes;

View File

@@ -1,4 +1,4 @@
package com.tencent.supersonic.chat.config;
package com.tencent.supersonic.chat.core.config;
import lombok.Data;
import org.springframework.beans.factory.annotation.Value;

View File

@@ -1,4 +1,4 @@
package com.tencent.supersonic.chat.config;
package com.tencent.supersonic.chat.core.config;
import lombok.AllArgsConstructor;
import lombok.Data;

View File

@@ -1,4 +1,4 @@
package com.tencent.supersonic.chat.config;
package com.tencent.supersonic.chat.core.config;
import com.tencent.supersonic.common.pojo.Constants;
import lombok.Data;

View File

@@ -1,4 +1,4 @@
package com.tencent.supersonic.knowledge.semantic;
package com.tencent.supersonic.chat.core.config;
import lombok.Data;
import org.springframework.beans.factory.annotation.Value;

View File

@@ -1,4 +1,4 @@
package com.tencent.supersonic.chat.config;
package com.tencent.supersonic.chat.core.config;
import java.util.List;
import lombok.AllArgsConstructor;

View File

@@ -1,4 +1,4 @@
package com.tencent.supersonic.chat.config;
package com.tencent.supersonic.chat.core.config;
import java.util.List;
import lombok.Data;

View File

@@ -0,0 +1,14 @@
package com.tencent.supersonic.chat.core.config;
import com.tencent.supersonic.headless.api.response.DimSchemaResp;
import com.tencent.supersonic.headless.api.response.MetricSchemaResp;
import java.util.List;
import lombok.Data;
@Data
public class EntityInternalDetail {
List<DimSchemaResp> dimensionList;
List<MetricSchemaResp> metricList;
}

View File

@@ -1,4 +1,4 @@
package com.tencent.supersonic.chat.config;
package com.tencent.supersonic.chat.core.config;
import lombok.Data;
@@ -16,10 +16,12 @@ public class LLMParserConfig {
@Value("${query2sql.path:/query2sql}")
private String queryToSqlPath;
@Value("${dimension.topn:5}")
@Value("${dimension.topn:10}")
private Integer dimensionTopN;
@Value("${metric.topn:5}")
@Value("${metric.topn:10}")
private Integer metricTopN;
@Value("${all.model:false}")
private Boolean allModel;
}

View File

@@ -1,6 +1,6 @@
package com.tencent.supersonic.knowledge.dictionary;
package com.tencent.supersonic.chat.core.config;
import com.tencent.supersonic.knowledge.utils.HanlpHelper;
import com.tencent.supersonic.chat.core.utils.HanlpHelper;
import java.io.FileNotFoundException;
import lombok.Data;
import lombok.extern.slf4j.Slf4j;

View File

@@ -1,5 +1,6 @@
package com.tencent.supersonic.chat.config;
package com.tencent.supersonic.chat.core.config;
import com.tencent.supersonic.chat.core.query.llm.s2sql.LLMReq.SqlGenerationMode;
import com.tencent.supersonic.common.service.SysParameterService;
import lombok.Data;
import lombok.extern.slf4j.Slf4j;
@@ -57,10 +58,13 @@ public class OptimizationConfig {
@Value("${s2SQL.linking.value.switch:true}")
private boolean useLinkingValueSwitch;
@Value("${s2SQL.generation:TWO_PASS_AUTO_COT}")
private SqlGenerationMode sqlGenerationMode;
@Value("${s2SQL.use.switch:true}")
private boolean useS2SqlSwitch;
@Value("${text2sql.example.num:10}")
@Value("${text2sql.example.num:15}")
private int text2sqlExampleNum;
@Value("${text2sql.fewShots.num:10}")
@@ -72,6 +76,9 @@ public class OptimizationConfig {
@Value("${text2sql.collection.name:text2dsl_agent_collection}")
private String text2sqlCollectionName;
@Value("${parse.show.count:3}")
private Integer parseShowCount;
@Autowired
private SysParameterService sysParameterService;
@@ -139,6 +146,14 @@ public class OptimizationConfig {
return convertValue("s2SQL.linking.value.switch", Boolean.class, useLinkingValueSwitch);
}
public SqlGenerationMode getSqlGenerationMode() {
return convertValue("s2SQL.generation", SqlGenerationMode.class, sqlGenerationMode);
}
public Integer getParseShowCount() {
return convertValue("parse.show.count", Integer.class, parseShowCount);
}
public <T> T convertValue(String paramName, Class<T> targetType, T defaultValue) {
try {
String value = sysParameterService.getSysParameter().getParameterByName(paramName);
@@ -151,6 +166,8 @@ public class OptimizationConfig {
return targetType.cast(Integer.parseInt(value));
} else if (targetType == Boolean.class) {
return targetType.cast(Boolean.parseBoolean(value));
} else if (targetType == SqlGenerationMode.class) {
return targetType.cast(SqlGenerationMode.valueOf(value));
}
} catch (Exception e) {
log.error("convertValue", e);

View File

@@ -1,21 +1,14 @@
package com.tencent.supersonic.chat.corrector;
package com.tencent.supersonic.chat.core.corrector;
import com.tencent.supersonic.chat.api.component.SemanticCorrector;
import com.tencent.supersonic.chat.core.pojo.QueryContext;
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
import com.tencent.supersonic.chat.api.pojo.SemanticSchema;
import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
import com.tencent.supersonic.common.pojo.enums.AggregateTypeEnum;
import com.tencent.supersonic.common.pojo.enums.TimeDimensionEnum;
import com.tencent.supersonic.common.util.ContextUtils;
import com.tencent.supersonic.common.util.jsqlparser.SqlParserAddHelper;
import com.tencent.supersonic.common.util.jsqlparser.SqlParserSelectFunctionHelper;
import com.tencent.supersonic.common.util.jsqlparser.SqlParserSelectHelper;
import com.tencent.supersonic.knowledge.service.SchemaService;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.springframework.util.CollectionUtils;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
@@ -23,6 +16,10 @@ import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.stream.Collectors;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.apache.commons.lang3.tuple.Pair;
import org.springframework.util.CollectionUtils;
/**
* basic semantic correction functionality, offering common methods and an
@@ -31,24 +28,23 @@ import java.util.stream.Collectors;
@Slf4j
public abstract class BaseSemanticCorrector implements SemanticCorrector {
public void correct(QueryReq queryReq, SemanticParseInfo semanticParseInfo) {
public void correct(QueryContext queryContext, SemanticParseInfo semanticParseInfo) {
try {
if (StringUtils.isBlank(semanticParseInfo.getSqlInfo().getCorrectS2SQL())) {
return;
}
doCorrect(queryReq, semanticParseInfo);
doCorrect(queryContext, semanticParseInfo);
log.info("sqlCorrection:{} sql:{}", this.getClass().getSimpleName(), semanticParseInfo.getSqlInfo());
} catch (Exception e) {
log.error(String.format("correct error,sqlInfo:%s", semanticParseInfo.getSqlInfo()), e);
}
}
public abstract void doCorrect(QueryContext queryContext, SemanticParseInfo semanticParseInfo);
public abstract void doCorrect(QueryReq queryReq, SemanticParseInfo semanticParseInfo);
protected Map<String, String> getFieldNameMap(QueryContext queryContext, Set<Long> modelIds) {
protected Map<String, String> getFieldNameMap(Set<Long> modelIds) {
SemanticSchema semanticSchema = ContextUtils.getBean(SchemaService.class).getSemanticSchema();
SemanticSchema semanticSchema = queryContext.getSemanticSchema();
List<SchemaElement> dbAllFields = new ArrayList<>();
dbAllFields.addAll(semanticSchema.getMetrics());
@@ -101,12 +97,12 @@ public abstract class BaseSemanticCorrector implements SemanticCorrector {
semanticParseInfo.getSqlInfo().setCorrectS2SQL(replaceFields);
}
protected void addAggregateToMetric(SemanticParseInfo semanticParseInfo) {
protected void addAggregateToMetric(QueryContext queryContext, SemanticParseInfo semanticParseInfo) {
//add aggregate to all metric
String correctS2SQL = semanticParseInfo.getSqlInfo().getCorrectS2SQL();
Set<Long> modelIds = semanticParseInfo.getModel().getModelIds();
List<SchemaElement> metrics = getMetricElements(modelIds);
List<SchemaElement> metrics = getMetricElements(queryContext, modelIds);
Map<String, String> metricToAggregate = metrics.stream()
.map(schemaElement -> {
@@ -114,7 +110,15 @@ public abstract class BaseSemanticCorrector implements SemanticCorrector {
schemaElement.setDefaultAgg(AggregateTypeEnum.SUM.name());
}
return schemaElement;
}).collect(Collectors.toMap(a -> a.getName(), a -> a.getDefaultAgg(), (k1, k2) -> k1));
}).flatMap(schemaElement -> {
Set<String> elements = new HashSet<>();
elements.add(schemaElement.getName());
if (!CollectionUtils.isEmpty(schemaElement.getAlias())) {
elements.addAll(schemaElement.getAlias());
}
return elements.stream().map(element -> Pair.of(element, schemaElement.getDefaultAgg())
);
}).collect(Collectors.toMap(Pair::getLeft, Pair::getRight, (k1, k2) -> k1));
if (CollectionUtils.isEmpty(metricToAggregate)) {
return;
@@ -123,8 +127,8 @@ public abstract class BaseSemanticCorrector implements SemanticCorrector {
semanticParseInfo.getSqlInfo().setCorrectS2SQL(aggregateSql);
}
protected List<SchemaElement> getMetricElements(Set<Long> modelIds) {
SemanticSchema semanticSchema = ContextUtils.getBean(SchemaService.class).getSemanticSchema();
protected List<SchemaElement> getMetricElements(QueryContext queryContext, Set<Long> modelIds) {
SemanticSchema semanticSchema = queryContext.getSemanticSchema();
return semanticSchema.getMetrics(modelIds);
}

View File

@@ -0,0 +1,21 @@
package com.tencent.supersonic.chat.core.corrector;
import com.tencent.supersonic.chat.core.pojo.QueryContext;
import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
import com.tencent.supersonic.common.util.jsqlparser.SqlParserReplaceHelper;
import lombok.extern.slf4j.Slf4j;
/**
* Perform SQL corrections on the "From" section in S2SQL.
*/
@Slf4j
public class FromCorrector extends BaseSemanticCorrector {
@Override
public void doCorrect(QueryContext queryContext, SemanticParseInfo semanticParseInfo) {
String modelName = semanticParseInfo.getModel().getName();
String correctSql = SqlParserReplaceHelper
.replaceTable(semanticParseInfo.getSqlInfo().getCorrectS2SQL(), modelName);
semanticParseInfo.getSqlInfo().setCorrectS2SQL(correctSql);
}
}

View File

@@ -1,42 +1,39 @@
package com.tencent.supersonic.chat.corrector;
package com.tencent.supersonic.chat.core.corrector;
import com.tencent.supersonic.chat.core.pojo.QueryContext;
import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
import com.tencent.supersonic.chat.api.pojo.SemanticSchema;
import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
import com.tencent.supersonic.chat.api.pojo.response.SqlInfo;
import com.tencent.supersonic.common.pojo.enums.TimeDimensionEnum;
import com.tencent.supersonic.common.util.ContextUtils;
import com.tencent.supersonic.common.util.jsqlparser.SqlParserAddHelper;
import com.tencent.supersonic.common.util.jsqlparser.SqlParserSelectHelper;
import com.tencent.supersonic.knowledge.service.SchemaService;
import lombok.extern.slf4j.Slf4j;
import org.springframework.util.CollectionUtils;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
import java.util.stream.Collectors;
import lombok.extern.slf4j.Slf4j;
import org.springframework.util.CollectionUtils;
/**
* Perform SQL corrections on the "group by" section in S2SQL.
* Perform SQL corrections on the "Group by" section in S2SQL.
*/
@Slf4j
public class GroupByCorrector extends BaseSemanticCorrector {
@Override
public void doCorrect(QueryReq queryReq, SemanticParseInfo semanticParseInfo) {
public void doCorrect(QueryContext queryContext, SemanticParseInfo semanticParseInfo) {
addGroupByFields(semanticParseInfo);
addGroupByFields(queryContext, semanticParseInfo);
}
private void addGroupByFields(SemanticParseInfo semanticParseInfo) {
private void addGroupByFields(QueryContext queryContext, SemanticParseInfo semanticParseInfo) {
Set<Long> modelIds = semanticParseInfo.getModel().getModelIds();
//add dimension group by
SqlInfo sqlInfo = semanticParseInfo.getSqlInfo();
String correctS2SQL = sqlInfo.getCorrectS2SQL();
SemanticSchema semanticSchema = ContextUtils.getBean(SchemaService.class).getSemanticSchema();
SemanticSchema semanticSchema = queryContext.getSemanticSchema();
//add alias field name
Set<String> dimensions = semanticSchema.getDimensions(modelIds).stream()
.flatMap(
@@ -77,15 +74,15 @@ public class GroupByCorrector extends BaseSemanticCorrector {
.collect(Collectors.toSet());
semanticParseInfo.getSqlInfo().setCorrectS2SQL(SqlParserAddHelper.addGroupBy(correctS2SQL, groupByFields));
addAggregate(semanticParseInfo);
addAggregate(queryContext, semanticParseInfo);
}
private void addAggregate(SemanticParseInfo semanticParseInfo) {
private void addAggregate(QueryContext queryContext, SemanticParseInfo semanticParseInfo) {
List<String> sqlGroupByFields = SqlParserSelectHelper.getGroupByFields(
semanticParseInfo.getSqlInfo().getCorrectS2SQL());
if (CollectionUtils.isEmpty(sqlGroupByFields)) {
return;
}
addAggregateToMetric(semanticParseInfo);
addAggregateToMetric(queryContext, semanticParseInfo);
}
}

View File

@@ -1,18 +1,14 @@
package com.tencent.supersonic.chat.corrector;
package com.tencent.supersonic.chat.core.corrector;
import com.tencent.supersonic.chat.core.pojo.QueryContext;
import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
import com.tencent.supersonic.chat.api.pojo.SemanticSchema;
import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
import com.tencent.supersonic.common.util.ContextUtils;
import com.tencent.supersonic.common.util.jsqlparser.SqlParserAddHelper;
import com.tencent.supersonic.common.util.jsqlparser.SqlParserSelectFunctionHelper;
import com.tencent.supersonic.common.util.jsqlparser.SqlParserSelectHelper;
import com.tencent.supersonic.knowledge.service.SchemaService;
import java.util.Set;
import java.util.List;
import java.util.Set;
import java.util.stream.Collectors;
import lombok.extern.slf4j.Slf4j;
import net.sf.jsqlparser.expression.Expression;
import org.springframework.util.CollectionUtils;
@@ -24,20 +20,20 @@ import org.springframework.util.CollectionUtils;
public class HavingCorrector extends BaseSemanticCorrector {
@Override
public void doCorrect(QueryReq queryReq, SemanticParseInfo semanticParseInfo) {
public void doCorrect(QueryContext queryContext, SemanticParseInfo semanticParseInfo) {
//add aggregate to all metric
addHaving(semanticParseInfo);
addHaving(queryContext, semanticParseInfo);
//add having expression filed to select
addHavingToSelect(semanticParseInfo);
}
private void addHaving(SemanticParseInfo semanticParseInfo) {
private void addHaving(QueryContext queryContext, SemanticParseInfo semanticParseInfo) {
Set<Long> modelIds = semanticParseInfo.getModel().getModelIds();
SemanticSchema semanticSchema = ContextUtils.getBean(SchemaService.class).getSemanticSchema();
SemanticSchema semanticSchema = queryContext.getSemanticSchema();
Set<String> metrics = semanticSchema.getMetrics(modelIds).stream()
.map(schemaElement -> schemaElement.getName()).collect(Collectors.toSet());
@@ -62,5 +58,4 @@ public class HavingCorrector extends BaseSemanticCorrector {
return;
}
}

View File

@@ -1,10 +1,10 @@
package com.tencent.supersonic.chat.corrector;
package com.tencent.supersonic.chat.core.corrector;
import com.tencent.supersonic.chat.core.pojo.QueryContext;
import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
import com.tencent.supersonic.chat.api.pojo.response.SqlInfo;
import com.tencent.supersonic.chat.parser.llm.s2sql.ParseResult;
import com.tencent.supersonic.chat.query.llm.s2sql.LLMReq.ElementValue;
import com.tencent.supersonic.chat.core.parser.sql.llm.ParseResult;
import com.tencent.supersonic.chat.core.query.llm.s2sql.LLMReq.ElementValue;
import com.tencent.supersonic.common.pojo.Constants;
import com.tencent.supersonic.common.util.JsonUtil;
import com.tencent.supersonic.common.util.jsqlparser.AggregateEnum;
@@ -24,7 +24,7 @@ import org.springframework.util.CollectionUtils;
public class SchemaCorrector extends BaseSemanticCorrector {
@Override
public void doCorrect(QueryReq queryReq, SemanticParseInfo semanticParseInfo) {
public void doCorrect(QueryContext queryContext, SemanticParseInfo semanticParseInfo) {
correctAggFunction(semanticParseInfo);
@@ -34,7 +34,7 @@ public class SchemaCorrector extends BaseSemanticCorrector {
updateFieldValueByLinkingValue(semanticParseInfo);
correctFieldName(semanticParseInfo);
correctFieldName(queryContext, semanticParseInfo);
}
private void correctAggFunction(SemanticParseInfo semanticParseInfo) {
@@ -50,8 +50,8 @@ public class SchemaCorrector extends BaseSemanticCorrector {
sqlInfo.setCorrectS2SQL(replaceAlias);
}
private void correctFieldName(SemanticParseInfo semanticParseInfo) {
Map<String, String> fieldNameMap = getFieldNameMap(semanticParseInfo.getModel().getModelIds());
private void correctFieldName(QueryContext queryContext, SemanticParseInfo semanticParseInfo) {
Map<String, String> fieldNameMap = getFieldNameMap(queryContext, semanticParseInfo.getModel().getModelIds());
SqlInfo sqlInfo = semanticParseInfo.getSqlInfo();
String sql = SqlParserReplaceHelper.replaceFields(sqlInfo.getCorrectS2SQL(), fieldNameMap);
sqlInfo.setCorrectS2SQL(sql);
@@ -86,7 +86,6 @@ public class SchemaCorrector extends BaseSemanticCorrector {
return parseResult.getLinkingValues();
}
private void updateFieldValueByLinkingValue(SemanticParseInfo semanticParseInfo) {
List<ElementValue> linking = getLinkingValues(semanticParseInfo);
if (CollectionUtils.isEmpty(linking)) {

View File

@@ -1,7 +1,7 @@
package com.tencent.supersonic.chat.corrector;
package com.tencent.supersonic.chat.core.corrector;
import com.tencent.supersonic.chat.core.pojo.QueryContext;
import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
import com.tencent.supersonic.common.util.jsqlparser.SqlParserSelectHelper;
import java.util.List;
import lombok.extern.slf4j.Slf4j;
@@ -14,7 +14,7 @@ import org.springframework.util.CollectionUtils;
public class SelectCorrector extends BaseSemanticCorrector {
@Override
public void doCorrect(QueryReq queryReq, SemanticParseInfo semanticParseInfo) {
public void doCorrect(QueryContext queryContext, SemanticParseInfo semanticParseInfo) {
String correctS2SQL = semanticParseInfo.getSqlInfo().getCorrectS2SQL();
List<String> aggregateFields = SqlParserSelectHelper.getAggregateFields(correctS2SQL);
List<String> selectFields = SqlParserSelectHelper.getSelectFields(correctS2SQL);

View File

@@ -1,7 +1,7 @@
package com.tencent.supersonic.chat.api.component;
package com.tencent.supersonic.chat.core.corrector;
import com.tencent.supersonic.chat.core.pojo.QueryContext;
import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
/**
* A semantic corrector checks validity of extracted semantic information and
@@ -9,5 +9,5 @@ import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
*/
public interface SemanticCorrector {
void correct(QueryReq queryReq, SemanticParseInfo semanticParseInfo);
void correct(QueryContext queryContext, SemanticParseInfo semanticParseInfo);
}

View File

@@ -1,20 +1,24 @@
package com.tencent.supersonic.chat.corrector;
package com.tencent.supersonic.chat.core.corrector;
import com.tencent.supersonic.chat.core.pojo.QueryContext;
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
import com.tencent.supersonic.chat.api.pojo.SchemaValueMap;
import com.tencent.supersonic.chat.api.pojo.SemanticParseInfo;
import com.tencent.supersonic.chat.api.pojo.SemanticSchema;
import com.tencent.supersonic.chat.api.pojo.request.QueryFilters;
import com.tencent.supersonic.chat.api.pojo.request.QueryReq;
import com.tencent.supersonic.chat.parser.llm.s2sql.S2SQLDateHelper;
import com.tencent.supersonic.chat.core.parser.sql.llm.S2SqlDateHelper;
import com.tencent.supersonic.common.pojo.Constants;
import com.tencent.supersonic.common.pojo.enums.TimeDimensionEnum;
import com.tencent.supersonic.common.util.ContextUtils;
import com.tencent.supersonic.common.util.StringUtil;
import com.tencent.supersonic.common.util.jsqlparser.SqlParserAddHelper;
import com.tencent.supersonic.common.util.jsqlparser.SqlParserReplaceHelper;
import com.tencent.supersonic.common.util.jsqlparser.SqlParserSelectHelper;
import com.tencent.supersonic.knowledge.service.SchemaService;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.stream.Collectors;
import lombok.extern.slf4j.Slf4j;
import net.sf.jsqlparser.JSQLParserException;
import net.sf.jsqlparser.expression.Expression;
@@ -23,13 +27,6 @@ import org.apache.commons.lang3.StringUtils;
import org.apache.logging.log4j.util.Strings;
import org.springframework.util.CollectionUtils;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.stream.Collectors;
/**
* Perform SQL corrections on the "Where" section in S2SQL.
*/
@@ -37,19 +34,19 @@ import java.util.stream.Collectors;
public class WhereCorrector extends BaseSemanticCorrector {
@Override
public void doCorrect(QueryReq queryReq, SemanticParseInfo semanticParseInfo) {
public void doCorrect(QueryContext queryContext, SemanticParseInfo semanticParseInfo) {
addDateIfNotExist(semanticParseInfo);
addDateIfNotExist(queryContext, semanticParseInfo);
parserDateDiffFunction(semanticParseInfo);
addQueryFilter(queryReq, semanticParseInfo);
addQueryFilter(queryContext, semanticParseInfo);
updateFieldValueByTechName(semanticParseInfo);
updateFieldValueByTechName(queryContext, semanticParseInfo);
}
private void addQueryFilter(QueryReq queryReq, SemanticParseInfo semanticParseInfo) {
String queryFilter = getQueryFilter(queryReq.getQueryFilters());
private void addQueryFilter(QueryContext queryContext, SemanticParseInfo semanticParseInfo) {
String queryFilter = getQueryFilter(queryContext.getQueryFilters());
String correctS2SQL = semanticParseInfo.getSqlInfo().getCorrectS2SQL();
@@ -72,11 +69,11 @@ public class WhereCorrector extends BaseSemanticCorrector {
semanticParseInfo.getSqlInfo().setCorrectS2SQL(correctS2SQL);
}
private void addDateIfNotExist(SemanticParseInfo semanticParseInfo) {
private void addDateIfNotExist(QueryContext queryContext, SemanticParseInfo semanticParseInfo) {
String correctS2SQL = semanticParseInfo.getSqlInfo().getCorrectS2SQL();
List<String> whereFields = SqlParserSelectHelper.getWhereFields(correctS2SQL);
if (CollectionUtils.isEmpty(whereFields) || !TimeDimensionEnum.containsZhTimeDimension(whereFields)) {
String currentDate = S2SQLDateHelper.getReferenceDate(semanticParseInfo.getModelId());
String currentDate = S2SqlDateHelper.getReferenceDate(queryContext, semanticParseInfo.getModelId());
if (StringUtils.isNotBlank(currentDate)) {
correctS2SQL = SqlParserAddHelper.addParenthesisToWhere(correctS2SQL);
correctS2SQL = SqlParserAddHelper.addWhere(
@@ -100,8 +97,8 @@ public class WhereCorrector extends BaseSemanticCorrector {
.collect(Collectors.joining(Constants.AND_UPPER));
}
private void updateFieldValueByTechName(SemanticParseInfo semanticParseInfo) {
SemanticSchema semanticSchema = ContextUtils.getBean(SchemaService.class).getSemanticSchema();
private void updateFieldValueByTechName(QueryContext queryContext, SemanticParseInfo semanticParseInfo) {
SemanticSchema semanticSchema = queryContext.getSemanticSchema();
Set<Long> modelIds = semanticParseInfo.getModel().getModelIds();
List<SchemaElement> dimensions = semanticSchema.getDimensions(modelIds);

View File

@@ -1,4 +1,4 @@
package com.tencent.supersonic.knowledge.dictionary;
package com.tencent.supersonic.chat.core.knowledge;
import com.google.common.base.Objects;
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
@@ -7,7 +7,7 @@ import lombok.ToString;
@Data
@ToString
public class FuzzyResult extends MapResult {
public class DatabaseMapResult extends MapResult {
private SchemaElement schemaElement;
@@ -19,7 +19,7 @@ public class FuzzyResult extends MapResult {
if (o == null || getClass() != o.getClass()) {
return false;
}
FuzzyResult that = (FuzzyResult) o;
DatabaseMapResult that = (DatabaseMapResult) o;
return Objects.equal(name, that.name) && Objects.equal(schemaElement, that.schemaElement);
}

View File

@@ -1,7 +1,6 @@
package com.tencent.supersonic.knowledge.dictionary;
package com.tencent.supersonic.chat.core.knowledge;
import java.util.List;
import lombok.Data;

View File

@@ -1,4 +1,5 @@
package com.tencent.supersonic.knowledge.dictionary;
package com.tencent.supersonic.chat.core.knowledge;
public enum DictUpdateMode {

View File

@@ -1,4 +1,4 @@
package com.tencent.supersonic.knowledge.dictionary;
package com.tencent.supersonic.chat.core.knowledge;
import java.util.Objects;
import lombok.Data;

View File

@@ -1,4 +1,4 @@
package com.tencent.supersonic.knowledge.dictionary;
package com.tencent.supersonic.chat.core.knowledge.dictionary;
import com.hankcs.hanlp.corpus.tag.Nature;
import com.hankcs.hanlp.dictionary.CoreDictionary;

View File

@@ -1,4 +1,4 @@
package com.tencent.supersonic.knowledge.dictionary;
package com.tencent.supersonic.chat.core.knowledge;
import java.util.HashMap;

View File

@@ -1,4 +1,4 @@
package com.tencent.supersonic.knowledge.dictionary;
package com.tencent.supersonic.chat.core.knowledge;
import com.tencent.supersonic.common.pojo.enums.TaskStatusEnum;

View File

@@ -1,4 +1,4 @@
package com.tencent.supersonic.knowledge.dictionary;
package com.tencent.supersonic.chat.core.knowledge;
import com.tencent.supersonic.common.pojo.enums.TypeEnums;

View File

@@ -1,4 +1,4 @@
package com.tencent.supersonic.knowledge.dictionary;
package com.tencent.supersonic.chat.core.knowledge;
import com.google.common.base.Objects;
import java.util.Map;

View File

@@ -1,4 +1,4 @@
package com.tencent.supersonic.knowledge.dictionary;
package com.tencent.supersonic.chat.core.knowledge;
import java.util.List;
@@ -19,7 +19,6 @@ public interface FileHandler {
*/
void createDir(String path);
Boolean existPath(String path);
/**

View File

@@ -1,4 +1,4 @@
package com.tencent.supersonic.knowledge.dictionary;
package com.tencent.supersonic.chat.core.knowledge;
import com.hankcs.hanlp.corpus.io.IIOAdapter;
import java.io.IOException;

View File

@@ -1,4 +1,4 @@
package com.tencent.supersonic.knowledge.dictionary;
package com.tencent.supersonic.chat.core.knowledge;
import com.google.common.base.Objects;
import java.util.List;

View File

@@ -1,6 +1,6 @@
package com.tencent.supersonic.knowledge.service;
package com.tencent.supersonic.chat.core.knowledge;
import com.tencent.supersonic.knowledge.utils.NatureHelper;
import com.tencent.supersonic.chat.core.utils.NatureHelper;
import java.util.ArrayList;
import java.util.List;
import java.util.Objects;
@@ -21,7 +21,6 @@ public class LoadRemoveService {
@Value("${mapper.remove.nature.prefix:}")
private String mapperRemoveNaturePrefix;
public List removeNatures(List value, Integer agentId, Set<Long> detectModelIds) {
if (CollectionUtils.isEmpty(value)) {
return value;
@@ -52,4 +51,4 @@ public class LoadRemoveService {
return resultList;
}
}
}

View File

@@ -1,6 +1,7 @@
package com.tencent.supersonic.knowledge.dictionary;
package com.tencent.supersonic.chat.core.knowledge;
import com.tencent.supersonic.chat.core.config.LocalFileConfig;
import java.io.BufferedWriter;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
@@ -10,7 +11,6 @@ import java.nio.file.Paths;
import java.nio.file.StandardCopyOption;
import java.nio.file.StandardOpenOption;
import java.util.List;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Component;
import org.springframework.util.CollectionUtils;
@@ -106,7 +106,6 @@ public class LocalFileHandler implements FileHandler {
return Paths.get(path).toAbsolutePath().toString();
}
@Override
public String getDictRootPath() {
return Paths.get(localFileConfig.getDictDirectoryLatest()).toAbsolutePath().toString();

View File

@@ -1,4 +1,4 @@
package com.tencent.supersonic.knowledge.dictionary;
package com.tencent.supersonic.chat.core.knowledge;
import java.io.Serializable;
import lombok.Data;

View File

@@ -1,4 +1,4 @@
package com.tencent.supersonic.knowledge.dictionary;
package com.tencent.supersonic.chat.core.knowledge;
import java.io.Serializable;
import lombok.Builder;

View File

@@ -1,4 +1,4 @@
package com.tencent.supersonic.knowledge.dictionary;
package com.tencent.supersonic.chat.core.knowledge;
import static com.hankcs.hanlp.utility.Predefine.logger;
@@ -15,8 +15,8 @@ import com.hankcs.hanlp.seg.common.Term;
import com.hankcs.hanlp.utility.LexiconUtility;
import com.hankcs.hanlp.utility.Predefine;
import com.hankcs.hanlp.utility.TextUtility;
import com.tencent.supersonic.knowledge.service.SearchService;
import com.tencent.supersonic.knowledge.utils.HanlpHelper;
import com.tencent.supersonic.chat.core.knowledge.dictionary.DictionaryAttributeUtil;
import com.tencent.supersonic.chat.core.utils.HanlpHelper;
import java.io.BufferedOutputStream;
import java.io.BufferedReader;
import java.io.DataOutputStream;
@@ -222,7 +222,10 @@ public class MultiCustomDictionary extends DynamicCustomDictionary {
new BufferedOutputStream(IOUtil.newOutputStream(mainPath + ".bin")));
if (customNatureCollector.isEmpty()) {
for (int i = Nature.begin.ordinal() + 1; i < Nature.values().length; ++i) {
customNatureCollector.add(Nature.values()[i]);
Nature nature = Nature.values()[i];
if (Objects.nonNull(nature)) {
customNatureCollector.add(nature);
}
}
}
@@ -350,7 +353,7 @@ public class MultiCustomDictionary extends DynamicCustomDictionary {
}
public boolean insert(String word, String natureWithFrequency) {
public synchronized boolean insert(String word, String natureWithFrequency) {
if (word == null) {
return false;
} else {

View File

@@ -1,27 +1,27 @@
package com.tencent.supersonic.knowledge.service;
package com.tencent.supersonic.chat.core.knowledge;
import com.hankcs.hanlp.collection.trie.bintrie.BaseNode;
import com.hankcs.hanlp.collection.trie.bintrie.BinTrie;
import com.hankcs.hanlp.corpus.tag.Nature;
import com.hankcs.hanlp.dictionary.CoreDictionary;
import com.tencent.supersonic.knowledge.dictionary.DictWord;
import com.hankcs.hanlp.seg.common.Term;
import com.tencent.supersonic.chat.api.pojo.request.DimensionValueReq;
import com.tencent.supersonic.chat.core.knowledge.dictionary.DictionaryAttributeUtil;
import com.tencent.supersonic.common.pojo.enums.DictWordType;
import com.tencent.supersonic.knowledge.dictionary.DictionaryAttributeUtil;
import com.tencent.supersonic.knowledge.dictionary.HanlpMapResult;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.PriorityQueue;
import java.util.Set;
import java.util.TreeMap;
import java.util.TreeSet;
import java.util.stream.Collectors;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.springframework.stereotype.Service;
import org.springframework.util.CollectionUtils;
@Service
@Slf4j
public class SearchService {
@@ -118,7 +118,6 @@ public class SearchService {
trie.put(key, getValue(attribute.nature));
}
public static void loadSuffix(List<DictWord> suffixes) {
if (CollectionUtils.isEmpty(suffixes)) {
return;
@@ -157,5 +156,15 @@ public class SearchService {
suffixTrie.remove(dictWord.getWord());
}
}
public static List<String> getDimensionValue(DimensionValueReq dimensionValueReq) {
String nature = DictWordType.NATURE_SPILT + dimensionValueReq.getModelId() + DictWordType.NATURE_SPILT
+ dimensionValueReq.getElementID();
PriorityQueue<Term> terms = MultiCustomDictionary.NATURE_TO_VALUES.get(nature);
if (org.apache.commons.collections.CollectionUtils.isEmpty(terms)) {
return new ArrayList<>();
}
return terms.stream().map(term -> term.getWord()).collect(Collectors.toList());
}
}

View File

@@ -1,7 +1,7 @@
package com.tencent.supersonic.knowledge.dictionary.builder;
package com.tencent.supersonic.chat.core.knowledge.builder;
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
import com.tencent.supersonic.knowledge.dictionary.DictWord;
import com.tencent.supersonic.chat.core.knowledge.DictWord;
import java.util.ArrayList;
import java.util.List;
import lombok.extern.slf4j.Slf4j;

View File

@@ -1,13 +1,11 @@
package com.tencent.supersonic.knowledge.dictionary.builder;
package com.tencent.supersonic.chat.core.knowledge.builder;
import com.google.common.collect.Lists;
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
import com.tencent.supersonic.chat.core.knowledge.DictWord;
import com.tencent.supersonic.common.pojo.enums.DictWordType;
import java.util.ArrayList;
import java.util.List;
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
import com.tencent.supersonic.knowledge.dictionary.DictWord;
import com.tencent.supersonic.common.pojo.enums.DictWordType;
import org.apache.commons.lang3.StringUtils;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
@@ -22,7 +20,6 @@ public class DimensionWordBuilder extends BaseWordBuilder {
@Value("${nlp.dimension.use.suffix:true}")
private boolean nlpDimensionUseSuffix = true;
@Override
public List<DictWord> doGet(String word, SchemaElement schemaElement) {
List<DictWord> result = Lists.newArrayList();

View File

@@ -1,13 +1,12 @@
package com.tencent.supersonic.knowledge.dictionary.builder;
package com.tencent.supersonic.chat.core.knowledge.builder;
import com.google.common.collect.Lists;
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
import com.tencent.supersonic.chat.core.knowledge.DictWord;
import com.tencent.supersonic.common.pojo.enums.DictWordType;
import java.util.List;
import java.util.Objects;
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
import com.tencent.supersonic.knowledge.dictionary.DictWord;
import com.tencent.supersonic.common.pojo.enums.DictWordType;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import org.springframework.util.CollectionUtils;

View File

@@ -1,13 +1,11 @@
package com.tencent.supersonic.knowledge.dictionary.builder;
package com.tencent.supersonic.chat.core.knowledge.builder;
import com.google.common.collect.Lists;
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
import com.tencent.supersonic.chat.core.knowledge.DictWord;
import com.tencent.supersonic.common.pojo.enums.DictWordType;
import java.util.ArrayList;
import java.util.List;
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
import com.tencent.supersonic.knowledge.dictionary.DictWord;
import com.tencent.supersonic.common.pojo.enums.DictWordType;
import org.apache.commons.lang3.StringUtils;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;

View File

@@ -1,9 +1,9 @@
package com.tencent.supersonic.knowledge.dictionary.builder;
package com.tencent.supersonic.chat.core.knowledge.builder;
import com.google.common.collect.Lists;
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
import com.tencent.supersonic.chat.core.knowledge.DictWord;
import com.tencent.supersonic.common.pojo.enums.DictWordType;
import com.tencent.supersonic.knowledge.dictionary.DictWord;
import java.util.List;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.collections.CollectionUtils;

View File

@@ -1,13 +1,12 @@
package com.tencent.supersonic.knowledge.dictionary.builder;
package com.tencent.supersonic.chat.core.knowledge.builder;
import com.google.common.collect.Lists;
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
import com.tencent.supersonic.chat.core.knowledge.DictWord;
import com.tencent.supersonic.common.pojo.enums.DictWordType;
import java.util.List;
import java.util.Objects;
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
import com.tencent.supersonic.knowledge.dictionary.DictWord;
import com.tencent.supersonic.common.pojo.enums.DictWordType;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import org.springframework.util.CollectionUtils;

View File

@@ -1,4 +1,4 @@
package com.tencent.supersonic.knowledge.dictionary.builder;
package com.tencent.supersonic.chat.core.knowledge.builder;
import com.tencent.supersonic.common.pojo.enums.DictWordType;
@@ -23,4 +23,4 @@ public class WordBuilderFactory {
public static BaseWordBuilder get(DictWordType strategyType) {
return wordNatures.get(strategyType);
}
}
}

View File

@@ -1,10 +1,9 @@
package com.tencent.supersonic.knowledge.semantic;
package com.tencent.supersonic.chat.core.knowledge.semantic;
import com.google.common.cache.Cache;
import com.google.common.cache.CacheBuilder;
import com.tencent.supersonic.chat.api.component.SemanticInterpreter;
import com.tencent.supersonic.chat.api.pojo.ModelSchema;
import com.tencent.supersonic.semantic.api.model.response.ModelSchemaResp;
import com.tencent.supersonic.headless.api.response.ModelSchemaResp;
import java.util.ArrayList;
import java.util.List;
import java.util.Optional;

View File

@@ -1,36 +1,35 @@
package com.tencent.supersonic.knowledge.semantic;
package com.tencent.supersonic.chat.core.knowledge.semantic;
import com.github.pagehelper.PageInfo;
import com.tencent.supersonic.auth.api.authentication.pojo.User;
import com.tencent.supersonic.common.pojo.enums.AuthType;
import com.tencent.supersonic.common.util.ContextUtils;
import com.tencent.supersonic.common.util.JsonUtil;
import com.tencent.supersonic.semantic.api.model.request.ModelSchemaFilterReq;
import com.tencent.supersonic.semantic.api.model.request.PageDimensionReq;
import com.tencent.supersonic.semantic.api.model.request.PageMetricReq;
import com.tencent.supersonic.semantic.api.model.response.DimensionResp;
import com.tencent.supersonic.semantic.api.model.response.DomainResp;
import com.tencent.supersonic.semantic.api.model.response.ExplainResp;
import com.tencent.supersonic.semantic.api.model.response.MetricResp;
import com.tencent.supersonic.semantic.api.model.response.ModelResp;
import com.tencent.supersonic.semantic.api.model.response.ModelSchemaResp;
import com.tencent.supersonic.semantic.api.model.response.QueryResultWithSchemaResp;
import com.tencent.supersonic.semantic.api.query.request.ExplainSqlReq;
import com.tencent.supersonic.semantic.api.query.request.QueryDimValueReq;
import com.tencent.supersonic.semantic.api.query.request.QueryMultiStructReq;
import com.tencent.supersonic.semantic.api.query.request.QueryS2SQLReq;
import com.tencent.supersonic.semantic.api.query.request.QueryStructReq;
import com.tencent.supersonic.semantic.model.domain.DimensionService;
import com.tencent.supersonic.semantic.model.domain.MetricService;
import com.tencent.supersonic.semantic.query.service.QueryService;
import com.tencent.supersonic.semantic.query.service.SchemaService;
import com.tencent.supersonic.headless.api.request.ExplainSqlReq;
import com.tencent.supersonic.headless.api.request.ModelSchemaFilterReq;
import com.tencent.supersonic.headless.api.request.PageDimensionReq;
import com.tencent.supersonic.headless.api.request.PageMetricReq;
import com.tencent.supersonic.headless.api.request.QueryDimValueReq;
import com.tencent.supersonic.headless.api.request.QueryMultiStructReq;
import com.tencent.supersonic.headless.api.request.QuerySqlReq;
import com.tencent.supersonic.headless.api.request.QueryStructReq;
import com.tencent.supersonic.headless.api.response.DimensionResp;
import com.tencent.supersonic.headless.api.response.DomainResp;
import com.tencent.supersonic.headless.api.response.ExplainResp;
import com.tencent.supersonic.headless.api.response.MetricResp;
import com.tencent.supersonic.headless.api.response.ModelResp;
import com.tencent.supersonic.headless.api.response.ModelSchemaResp;
import com.tencent.supersonic.headless.api.response.SemanticQueryResp;
import com.tencent.supersonic.headless.server.service.DimensionService;
import com.tencent.supersonic.headless.server.service.MetricService;
import com.tencent.supersonic.headless.server.service.QueryService;
import com.tencent.supersonic.headless.server.service.SchemaService;
import java.util.ArrayList;
import java.util.List;
import lombok.SneakyThrows;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import java.util.HashMap;
import java.util.List;
@Slf4j
public class LocalSemanticInterpreter extends BaseSemanticInterpreter {
@@ -41,20 +40,20 @@ public class LocalSemanticInterpreter extends BaseSemanticInterpreter {
@SneakyThrows
@Override
public QueryResultWithSchemaResp queryByStruct(QueryStructReq queryStructReq, User user) {
public SemanticQueryResp queryByStruct(QueryStructReq queryStructReq, User user) {
if (StringUtils.isNotBlank(queryStructReq.getCorrectS2SQL())) {
QueryS2SQLReq queryS2SQLReq = new QueryS2SQLReq();
queryS2SQLReq.setSql(queryStructReq.getCorrectS2SQL());
queryS2SQLReq.setModelIds(queryStructReq.getModelIdSet());
queryS2SQLReq.setVariables(new HashMap<>());
return queryByS2SQL(queryS2SQLReq, user);
QuerySqlReq querySQLReq = new QuerySqlReq();
querySQLReq.setSql(queryStructReq.getCorrectS2SQL());
querySQLReq.setModelIds(queryStructReq.getModelIdSet());
querySQLReq.setParams(new ArrayList<>());
return queryByS2SQL(querySQLReq, user);
}
queryService = ContextUtils.getBean(QueryService.class);
return queryService.queryByStructWithAuth(queryStructReq, user);
}
@Override
public QueryResultWithSchemaResp queryByMultiStruct(QueryMultiStructReq queryMultiStructReq, User user) {
public SemanticQueryResp queryByMultiStruct(QueryMultiStructReq queryMultiStructReq, User user) {
try {
queryService = ContextUtils.getBean(QueryService.class);
return queryService.queryByMultiStruct(queryMultiStructReq, user);
@@ -66,15 +65,15 @@ public class LocalSemanticInterpreter extends BaseSemanticInterpreter {
@Override
@SneakyThrows
public QueryResultWithSchemaResp queryByS2SQL(QueryS2SQLReq queryS2SQLReq, User user) {
public SemanticQueryResp queryByS2SQL(QuerySqlReq querySQLReq, User user) {
queryService = ContextUtils.getBean(QueryService.class);
Object object = queryService.queryBySql(queryS2SQLReq, user);
return JsonUtil.toObject(JsonUtil.toString(object), QueryResultWithSchemaResp.class);
SemanticQueryResp object = queryService.queryBySql(querySQLReq, user);
return JsonUtil.toObject(JsonUtil.toString(object), SemanticQueryResp.class);
}
@Override
@SneakyThrows
public QueryResultWithSchemaResp queryDimValue(QueryDimValueReq queryDimValueReq, User user) {
public SemanticQueryResp queryDimValue(QueryDimValueReq queryDimValueReq, User user) {
queryService = ContextUtils.getBean(QueryService.class);
return queryService.queryDimValue(queryDimValueReq, user);
}

View File

@@ -1,4 +1,4 @@
package com.tencent.supersonic.knowledge.semantic;
package com.tencent.supersonic.chat.core.knowledge.semantic;
import com.google.common.collect.Lists;
import com.tencent.supersonic.chat.api.pojo.ModelSchema;
@@ -6,12 +6,12 @@ import com.tencent.supersonic.chat.api.pojo.RelatedSchemaElement;
import com.tencent.supersonic.chat.api.pojo.SchemaElement;
import com.tencent.supersonic.chat.api.pojo.SchemaElementType;
import com.tencent.supersonic.chat.api.pojo.SchemaValueMap;
import com.tencent.supersonic.semantic.api.model.pojo.DimValueMap;
import com.tencent.supersonic.semantic.api.model.pojo.RelateDimension;
import com.tencent.supersonic.semantic.api.model.pojo.SchemaItem;
import com.tencent.supersonic.semantic.api.model.response.DimSchemaResp;
import com.tencent.supersonic.semantic.api.model.response.MetricSchemaResp;
import com.tencent.supersonic.semantic.api.model.response.ModelSchemaResp;
import com.tencent.supersonic.headless.api.pojo.DimValueMap;
import com.tencent.supersonic.headless.api.pojo.RelateDimension;
import com.tencent.supersonic.headless.api.pojo.SchemaItem;
import com.tencent.supersonic.headless.api.response.DimSchemaResp;
import com.tencent.supersonic.headless.api.response.MetricSchemaResp;
import com.tencent.supersonic.headless.api.response.ModelSchemaResp;
import org.apache.logging.log4j.util.Strings;
import org.springframework.beans.BeanUtils;
import org.springframework.util.CollectionUtils;
@@ -26,7 +26,7 @@ public class ModelSchemaBuilder {
public static ModelSchema build(ModelSchemaResp resp) {
ModelSchema modelSchema = new ModelSchema();
SchemaElement domain = SchemaElement.builder()
SchemaElement model = SchemaElement.builder()
.model(resp.getId())
.id(resp.getId())
.name(resp.getName())
@@ -34,7 +34,7 @@ public class ModelSchemaBuilder {
.type(SchemaElementType.MODEL)
.alias(SchemaItem.getAliasList(resp.getAlias()))
.build();
modelSchema.setModel(domain);
modelSchema.setModel(model);
modelSchema.setModelRelas(resp.getModelRelas());
Set<SchemaElement> metrics = new HashSet<>();

Some files were not shown because too many files have changed in this diff Show More