[improvement][headless-fe] Added support for data sets with non-partitioned time fields (#1560)

* [improvement][semantic-fe] fix the dimension initialization issue in metric correlation

* [improvement][semantic-fe] Fix the issue of database changes not taking effect when creating based on an SQL data source.

* [improvement][semantic-fe] Optimizing pagination logic and some CSS styles

* [improvement][semantic-fe] Fixing the API for the indicator list by changing "current" to "pageNum"

* [improvement][semantic-fe] Fixing the default value setting for the indicator list

* [improvement][semantic-fe] Adding batch operations for indicators/dimensions/models

* [improvement][semantic-fe] Replacing the single status update API for indicators/dimensions with a batch update API

* [improvement][semantic-fe] Redesigning the indicator homepage to incorporate trend charts and table functionality for indicators

* [improvement][semantic-fe] Optimizing the logic for setting dimension values and editing data sources, and adding system settings functionality

* [improvement][semantic-fe] Upgrading antd version to 5.x, extracting the batch operation button component, optimizing the interaction for system settings, and expanding the configuration generation types for list-to-select component.

* [improvement][semantic-fe] Adding the ability to filter dimensions based on whether they are tags or not.

* [improvement][semantic-fe] Adding the ability to edit relationships between models in the canvas.

* [improvement][semantic-fe] Updating the datePicker component to use dayjs instead.

* [improvement][semantic-fe] Fixing the issue with passing the model ID for dimensions in the indicator market.

* [improvement][semantic-fe] Fixing the abnormal state of the popup when creating a model.

* [improvement][semantic-fe] Adding permission logic for bulk operations in the indicator market.

* [improvement][semantic-fe] Adding the ability to download and transpose data.

* [improvement][semantic-fe] Fixing the initialization issue with the date selection component in the indicator details page when switching time granularity.

* [improvement][semantic-fe] Fixing the logic error in the dimension value setting.

* [improvement][semantic-fe] Fixing the synchronization issue with the question and answer settings information.

* [improvement][semantic-fe] Optimizing the canvas functionality for better performance and user experience.

* [improvement][semantic-fe] Optimizing the update process for drawing model relationship edges in the canvas.

* [improvement][semantic-fe] Changing the line type for canvas connections.

* [improvement][semantic-fe] Replacing the initialization variable from "semantic" to "headless".

* [improvement][semantic-fe] Fixing the missing migration issue for default drill-down dimension configuration in model editing. Additionally, optimizing the data retrieval method for initializing fields in the model.

* [improvement][semantic-fe] Updating the logic for the fieldName.

* [improvement][semantic-fe] Adjusting the position of the metrics tab.

* [improvement][semantic-fe] Changing the 字段名称 to 英文名称.

* [improvement][semantic-fe] Fix metric measurement deletion.

* [improvement][semantic-fe] UI optimization for metric details page.

* [improvement][semantic-fe] UI optimization for metric details page.

* [improvement][semantic-fe] UI adjustment for metric details page.

* [improvement][semantic-fe] The granularity field in the time type of model editing now supports setting it as empty.

* [improvement][semantic-fe] Added field type and metric type to the metric creation options.

* [improvement][semantic-fe] The organization structure selection feature has been added to the permission management.

* [improvement][semantic-fe] Improved user experience for the metric list.

* [improvement][semantic-fe] fix update the metric list.

* [improvement][headless-fe] Added view management functionality.

* [improvement][headless-fe] The view management functionality has been added. This feature allows users to create, edit, and manage different views within the system.

* [improvement][headless-fe] Added model editing side effect detection.

* [improvement][headless-fe] Fixed the logic error in view editing.

* [improvement][headless-fe] Fixed the issue with initializing dimension associations in metric settings.

* [improvement][headless-fe] Added the ability to hide the Q&A settings entry point.

* [improvement][headless-fe] Fixed the issue with selecting search results in metric field creation.

* [improvement][headless-fe] Added search functionality to the field list in model editing.

* [improvement][headless-fe] fix the field list in model editing

* [improvement][headless-fe] Restructured the data for the dimension value settings interface.

* [improvement][headless-fe] Added dynamic variable functionality to model creation based on SQL scripts.

* [improvement][headless-fe] Added support for passing dynamic variables as parameters in the executeSql function.

* [improvement][headless-fe] Resolved the issue where users were unable to select all options for dimensions, metrics, and fields in the metric generation process.

* [improvement][headless-fe] Replaced the term "view" with "dataset"

* [improvement][headless-fe] Added the ability to export metrics and dimensions to a specific target.

* [improvement][headless-fe] Enhanced dataset creation to support the tag mode.

* [improvement][headless-fe] Added tag value setting.

* [improvement][headless-fe] Optimized the tag setting system.

* [improvement][headless-fe] Optimized the tag setting system.

* [improvement][headless-fe] Updated the data initialization for model editing to use API requests instead.

* [improvement][headless-fe] Added search functionality to model management.

* [improvement][headless-fe] Removed field null validation during model editing.

* [improvement][headless-fe] Updated the batch operation button component.

* [improvement][headless-fe] Optimized the logic for initializing indicators in dimension value settings.

* [improvement][headless-fe] Adjusted the length of the input field for model editing names.

* [improvement][headless-fe]  Lock the version of the @ant-design/pro-table component and replace it with @ant-design/pro-components.

* [improvement][headless-fe] Optimized the style of the metrics market and tags market.

* [improvement][headless-fe] The quick creation of model fields now defaults to using the "comment" field for filling.

* [improvement][headless-fe] The quick creation of model fields now defaults to using the "comment" field for filling

* [improvement][headless-fe] The quick creation of model fields now defaults to using the "comment" field for filling.

* [improvement][headless-fe] Fixed the issue where the conditions for metric measurement creation were not being saved correctly.

* [improvement][headless-fe] Default value setting for hiding dimensions.

* [improvement][headless-fe] Updated the file imports in the project.

* [improvement][headless-fe] Adjusted the logic for displaying the tab in the theme domain.

* [improvement][headless-fe] Added term management functionality.

* [improvement][headless-fe] When creating a model, the current metric operator now allows for clearance.

* [improvement][headless-fe] Term management interface transformation

* [improvement][headless-fe] Migrating scaffold version to @umi/max

* [improvement][headless-fe] remove modle column

* [improvement][headless-fe] 1.Added configuration for the large language model in the agent; 2.upgraded React version from 17 to 18; 3.modified some UI effects.

* [improvement][headless-fe] Added a simplified mode to the question-answering system.

* [improvement][headless-fe] remove pnpm-lock

* [improvement][headless-fe] add pnpm-lock

* [improvement][headless-fe] Fixed the issue with passing the modelId during initialization.

* [improvement][headless-fe] Fixed the issue with abnormal comments during model creation.

* [improvement][headless-fe] fix  build bugs

* [improvement][headless-fe]  change build config

* [improvement][headless-fe] route config change

* [improvement][headless-fe] Optimized data updating when switching domains.

* [improvement][headless-fe] css change

* [improvement][semantic-fe] logo css change

* [improvement][semantic-fe] system config add defaultValue support

* [improvement][semantic-fe] tag mode wording change

* [improvement][semantic-fe] fix metric edit page init error

* [improvement][supersonic-fe] Updated the way chat projects are imported in supersonic-fe.

* [improvement][chat-engine] Added a background silent mode for watching chat projects.

* [improvement][supersonic-fe] fix proxy url

* [improvement][headless-fe] agent setting update

* [improvement][headless-fe] Agent configuration: Added connectivity testing for large models.

* [improvement][headless-fe] Chat: Enabled integration with agent configuration.

* [improvement][headless-fe] SQL formatter: Updated the import method.

* [improvement][headless-fe] login fixed

* [improvement][headless-fe] Agent: Optimized the logic for saving.

* [improvement][headless-fe] Model management: Integrated into the main theme domain.

* [improvement][headless-fe] Sensitivity: Added bulk modification functionality.

* [improvement][headless-fe] wording change

* [improvement][headless-fe] Prompt configuration: Added the ability to configure prompts.

* [improvement][headless-fe] Added the ability to configure embedding models.

* [improvement][headless-fe] hidden configure embedding models.

* [improvement][headless-fe] Connection test prompt update for large language model settings.

* [improvement][headless-fe]  add memory review config

* [improvement][headless-fe] Rollback of data structure for large language model configuration.

* [improvement][headless-fe] Added dependency relationships between various items in system configuration.

* [improvement][headless-fe] Added password parsing to the generation rules of system settings.

* [improvement][headless-fe] Added slider parsing to the generation rules of system settings.

* [improvement][headless-fe] Optimized the logic for initializing values in system settings.

* [improvement][headless-fe] Optimized the time format

* [improvement][headless-fe] Unified the SQL generation method for data sets to ensure consistency and improve efficiency.

* [improvement][headless-fe] Added support for data sets with non-partitioned time fields

---------

Co-authored-by: tristanliu <tristanliu@tencent.com>
This commit is contained in:
tristanliu
2024-08-12 19:59:11 +08:00
committed by GitHub
parent 4d02bb7068
commit 4d65b8b93a
7 changed files with 450 additions and 288 deletions

View File

@@ -3,7 +3,7 @@ import { Form, Button, Modal, Steps, message } from 'antd';
import ModelBasicForm from './ModelBasicForm';
import ModelFieldForm from './ModelFieldForm';
import { formLayout } from '@/components/FormHelper/utils';
import { EnumDataSourceType } from '../constants';
import { EnumDataSourceType, EnumModelDataType } from '../constants';
import styles from '../style.less';
import {
updateModel,
@@ -203,6 +203,19 @@ const ModelCreateForm: React.FC<CreateFormProps> = ({
},
});
break;
case EnumDataSourceType.PARTITION_TIME:
fieldsClassify.dimensions.push({
bizName: fieldName,
type,
isCreateDimension,
name,
dateFormat,
typeParams: {
isPrimary: true,
timeGranularity: timeGranularity || '',
},
});
break;
case EnumDataSourceType.FOREIGN:
case EnumDataSourceType.PRIMARY:
fieldsClassify.identifiers.push({
@@ -303,11 +316,21 @@ const ModelCreateForm: React.FC<CreateFormProps> = ({
setFields(columnFields || []);
};
const formatterIdentifiers = (identifiersList: any[] = []) => {
return identifiersList.map((identifiers: any) => {
return {
...identifiers,
classType: EnumModelDataType.IDENTIFIERS,
};
});
};
const formatterMeasures = (measuresList: any[] = []) => {
return measuresList.map((measures: any) => {
return {
...measures,
type: EnumDataSourceType.MEASURES,
classType: EnumModelDataType.MEASURES,
};
});
};
@@ -317,6 +340,7 @@ const ModelCreateForm: React.FC<CreateFormProps> = ({
return {
...dimension,
timeGranularity: typeParams?.timeGranularity || '',
classType: EnumModelDataType.DIMENSION,
};
});
};
@@ -358,7 +382,7 @@ const ModelCreateForm: React.FC<CreateFormProps> = ({
form.setFieldsValue(initValue);
const formatFields = [
...formatterDimensions(dimensions || []),
...(identifiers || []),
...formatterIdentifiers(identifiers || []),
...formatterMeasures(measures || []),
];
initFields(formatFields, columns);
@@ -517,7 +541,7 @@ const ModelCreateForm: React.FC<CreateFormProps> = ({
return (
<Modal
forceRender
width={currentStep ? 1300 : 800}
width={currentStep ? 1400 : 800}
destroyOnClose
title={`${isEdit ? '编辑' : '新建'}模型`}
maskClosable={false}

View File

@@ -11,6 +11,9 @@ import {
AGG_OPTIONS,
EnumDataSourceType,
DATE_OPTIONS,
DIM_OPTIONS,
EnumModelDataType,
PARTITION_TIME_FORMATTER,
} from '../constants';
import styles from '../style.less';
@@ -20,6 +23,7 @@ type FieldItem = {
dataType: string;
name: string;
type: EnumDataSourceType;
classType: EnumModelDataType;
comment?: string;
agg?: string;
checked?: number;
@@ -43,9 +47,11 @@ type Props = {
const { Option } = Select;
const getCreateFieldName = (type: EnumDataSourceType) => {
const isCreateName = [EnumDataSourceType.CATEGORICAL, EnumDataSourceType.TIME].includes(
type as EnumDataSourceType,
)
const isCreateName = [
EnumDataSourceType.CATEGORICAL,
EnumDataSourceType.TIME,
EnumDataSourceType.PARTITION_TIME,
].includes(type as EnumDataSourceType)
? 'isCreateDimension'
: 'isCreateMetric';
return isCreateName;
@@ -72,68 +78,134 @@ const ModelFieldForm: React.FC<Props> = ({
{
title: '英文名称',
dataIndex: 'fieldName',
width: 100,
width: 250,
},
{
title: '字段类型',
dataIndex: 'dataType',
width: 80,
width: 150,
},
{
title: '语义类型',
dataIndex: 'type',
width: 100,
width: 250,
render: (_: any, record: FieldItem) => {
const type = fields.find((field) => field.bizName === record.bizName)?.type;
const classType = fields.find((field) => field.bizName === record.bizName)?.classType;
const selectTypeValue = [EnumModelDataType.DIMENSION].includes(classType)
? classType
: type;
return (
<Select
placeholder="字段类型"
value={type}
allowClear
onChange={(value) => {
let defaultParams = {};
if (value === EnumDataSourceType.MEASURES) {
defaultParams = {
agg: AGG_OPTIONS[0].value,
<Space>
<Select
placeholder="字段类型"
value={selectTypeValue}
allowClear
onChange={(value) => {
let defaultParams = {};
if (value === EnumDataSourceType.MEASURES) {
defaultParams = {
agg: AGG_OPTIONS[0].value,
classType: EnumDataSourceType.MEASURES,
type: EnumDataSourceType.MEASURES,
};
}
// else if (value === EnumDataSourceType.TIME) {
// defaultParams = {
// dateFormat: DATE_FORMATTER[0],
// timeGranularity: 'day',
// };
// }
else if (value === EnumModelDataType.DIMENSION) {
defaultParams = {
type: DIM_OPTIONS[0].value,
classType: EnumModelDataType.DIMENSION,
};
} else if (value === EnumDataSourceType.PRIMARY) {
defaultParams = {
type: EnumDataSourceType.PRIMARY,
classType: EnumModelDataType.IDENTIFIERS,
};
} else {
defaultParams = {
agg: undefined,
dateFormat: undefined,
timeGranularity: undefined,
};
}
const isCreateName = getCreateFieldName(value);
const editState = !isUndefined(record[isCreateName])
? !!record[isCreateName]
: true;
const { name, comment } = record;
const params = {
...record,
type: value,
classType: value,
name: name || comment,
[isCreateName]: editState,
...defaultParams,
};
} else if (value === EnumDataSourceType.TIME) {
defaultParams = {
dateFormat: DATE_FORMATTER[0],
timeGranularity: 'day',
};
} else {
defaultParams = {
agg: undefined,
dateFormat: undefined,
timeGranularity: undefined,
};
}
const isCreateName = getCreateFieldName(value);
const editState = !isUndefined(record[isCreateName]) ? !!record[isCreateName] : true;
const { name, comment } = record;
onFieldChange(record.bizName, {
...record,
type: value,
name: name || comment,
[isCreateName]: editState,
...defaultParams,
});
}}
style={{ width: '100%' }}
>
{TYPE_OPTIONS.map((item) => (
<Option key={item.label} value={item.value}>
{item.label}
</Option>
))}
</Select>
onFieldChange(record.bizName, params);
}}
style={{ width: '100%' }}
>
{TYPE_OPTIONS.map((item) => (
<Option key={item.label} value={item.value}>
{item.label}
</Option>
))}
</Select>
{classType === EnumModelDataType.DIMENSION && (
<Select
placeholder="维度类型"
value={type}
onChange={(value) => {
// handleFieldChange(record, 'agg', value);
let defaultParams = {};
if (value === EnumDataSourceType.TIME) {
defaultParams = {
dateFormat: DATE_FORMATTER[0],
timeGranularity: 'day',
};
} else {
defaultParams = {
agg: undefined,
dateFormat: undefined,
timeGranularity: undefined,
};
}
const isCreateName = getCreateFieldName(value);
const editState = !isUndefined(record[isCreateName])
? !!record[isCreateName]
: true;
const { name, comment } = record;
onFieldChange(record.bizName, {
...record,
type: value,
name: name || comment,
[isCreateName]: editState,
...defaultParams,
});
}}
defaultValue={DIM_OPTIONS[0].value}
style={{ width: '100px' }}
>
{DIM_OPTIONS.map((item) => (
<Option key={item.value} value={item.value}>
{item.label}
</Option>
))}
</Select>
)}
</Space>
);
},
},
{
title: '扩展配置',
dataIndex: 'extender',
width: 200,
// width: 200,
render: (_: any, record: FieldItem) => {
const { type } = record;
if (type === EnumDataSourceType.PRIMARY) {
@@ -181,63 +253,73 @@ const ModelFieldForm: React.FC<Props> = ({
const isTag = fields.find((field) => field.bizName === record.bizName)?.isTag;
return (
<Space>
<span></span>
<Switch
defaultChecked
size="small"
checked={!!isTag}
onChange={(value) => {
handleFieldChange(record, 'isTag', value);
}}
/>
<Tooltip title="如果勾选,代表维度的取值都是一种“标签”,可用作对实体的圈选">
<ExclamationCircleOutlined />
</Tooltip>
<Space>
<span>:</span>
<Switch
defaultChecked
size="small"
checked={!!isTag}
onChange={(value) => {
handleFieldChange(record, 'isTag', value);
}}
/>
<Tooltip title="如果勾选,代表维度的取值都是一种“标签”,可用作对实体的圈选">
<ExclamationCircleOutlined />
</Tooltip>
</Space>
</Space>
);
}
if (type === EnumDataSourceType.TIME) {
if ([EnumDataSourceType.TIME, EnumDataSourceType.PARTITION_TIME].includes(type)) {
const dateFormat = fields.find((field) => field.bizName === record.bizName)?.dateFormat;
const timeGranularity = fields.find(
(field) => field.bizName === record.bizName,
)?.timeGranularity;
const dateFormatterOptions =
type === EnumDataSourceType.PARTITION_TIME ? PARTITION_TIME_FORMATTER : DATE_FORMATTER;
return (
<Space>
<Select
placeholder="时间格式"
value={dateFormat}
onChange={(value) => {
handleFieldChange(record, 'dateFormat', value);
}}
defaultValue={DATE_FORMATTER[0]}
style={{ minWidth: 150 }}
>
{DATE_FORMATTER.map((item) => (
<Option key={item} value={item}>
{item}
</Option>
))}
</Select>
<Tooltip title="请选择数据库中时间字段对应格式">
<ExclamationCircleOutlined />
</Tooltip>
<span>:</span>
<Select
placeholder="时间粒度"
value={timeGranularity === '' ? undefined : timeGranularity}
onChange={(value) => {
handleFieldChange(record, 'timeGranularity', value);
}}
defaultValue={timeGranularity === '' ? undefined : DATE_OPTIONS[0]}
style={{ minWidth: 80 }}
allowClear
>
{DATE_OPTIONS.map((item) => (
<Option key={item} value={item}>
{item}
</Option>
))}
</Select>
<Space direction="vertical">
<Space>
<span>:</span>
<Select
placeholder="时间格式"
value={dateFormat}
onChange={(value) => {
handleFieldChange(record, 'dateFormat', value);
}}
defaultValue={dateFormatterOptions[0]}
style={{ minWidth: 250 }}
>
{dateFormatterOptions.map((item) => (
<Option key={item} value={item}>
{item}
</Option>
))}
</Select>
<Tooltip title="请选择数据库中时间字段对应格式">
<ExclamationCircleOutlined />
</Tooltip>
</Space>
<Space>
<span>:</span>
<Select
placeholder="时间粒度"
value={timeGranularity === '' ? undefined : timeGranularity}
onChange={(value) => {
handleFieldChange(record, 'timeGranularity', value);
}}
defaultValue={timeGranularity === '' ? undefined : DATE_OPTIONS[0]}
style={{ minWidth: 180 }}
allowClear
>
{DATE_OPTIONS.map((item) => (
<Option key={item} value={item}>
{item}
</Option>
))}
</Select>
</Space>
</Space>
);
}
@@ -262,55 +344,51 @@ const ModelFieldForm: React.FC<Props> = ({
EnumDataSourceType.FOREIGN,
EnumDataSourceType.CATEGORICAL,
EnumDataSourceType.TIME,
EnumDataSourceType.PARTITION_TIME,
EnumDataSourceType.MEASURES,
].includes(type as EnumDataSourceType)
) {
const isCreateName = getCreateFieldName(type);
const editState = !isUndefined(record[isCreateName]) ? !!record[isCreateName] : true;
return (
<Row>
<Col flex="25px">
<Checkbox
style={{ width: '100%', position: 'relative', top: 5 }}
checked={editState}
onChange={(e) => {
const value = e.target.checked ? 1 : 0;
if (!value) {
onFieldChange(record.bizName, {
...record,
name: '',
checked: value,
[isCreateName]: value,
});
} else {
onFieldChange(record.bizName, {
...record,
checked: value,
[isCreateName]: value,
});
}
}}
/>
</Col>
<Col flex="auto">
<Input
className={!inputValue && styles.dataSourceFieldsName}
style={{ minHeight: 20 }}
value={inputValue}
disabled={!editState}
minLength={1}
onChange={(e) => {
const value = e.target.value;
<Space>
<Checkbox
checked={editState}
onChange={(e) => {
const value = e.target.checked ? 1 : 0;
if (!value) {
onFieldChange(record.bizName, {
...record,
name: value,
[isCreateName]: 1,
name: '',
checked: value,
[isCreateName]: value,
});
}}
placeholder="请填写名称"
/>
</Col>
</Row>
} else {
onFieldChange(record.bizName, {
...record,
checked: value,
[isCreateName]: value,
});
}
}}
/>
<Input
className={!inputValue && styles.dataSourceFieldsName}
style={{ minHeight: 20 }}
value={inputValue}
disabled={!editState}
minLength={1}
onChange={(e) => {
const value = e.target.value;
onFieldChange(record.bizName, {
...record,
name: value,
[isCreateName]: 1,
});
}}
placeholder="请填写名称"
/>
</Space>
);
}
return <></>;

View File

@@ -7,20 +7,52 @@ export const EDITOR_HEIGHT_MAP = new Map([
export enum EnumDataSourceType {
CATEGORICAL = 'categorical',
TIME = 'time',
PARTITION_TIME = 'partition_time',
MEASURES = 'measures',
PRIMARY = 'primary',
FOREIGN = 'foreign',
}
export const TYPE_OPTIONS = [
export enum EnumModelDataType {
DIMENSION = 'dimension',
IDENTIFIERS = 'identifiers',
MEASURES = 'measures',
}
export const modelDataClass = {
[EnumModelDataType.DIMENSION]: [
EnumDataSourceType.CATEGORICAL,
EnumDataSourceType.TIME,
EnumDataSourceType.PARTITION_TIME,
],
[EnumModelDataType.IDENTIFIERS]: [EnumDataSourceType.PRIMARY, EnumDataSourceType.FOREIGN],
[EnumModelDataType.MEASURES]: [EnumDataSourceType.MEASURES],
};
export const DIM_OPTIONS = [
{
label: '维度',
label: '枚举',
value: EnumDataSourceType.CATEGORICAL,
},
{
label: '日期',
label: '普通时间',
value: EnumDataSourceType.TIME,
},
{
label: '分区时间',
value: EnumDataSourceType.PARTITION_TIME,
},
];
export const TYPE_OPTIONS = [
{
label: '维度',
value: EnumModelDataType.DIMENSION,
},
// {
// label: '日期',
// value: EnumDataSourceType.TIME,
// },
{
label: '度量',
value: EnumDataSourceType.MEASURES,
@@ -64,4 +96,20 @@ export const AGG_OPTIONS = [
export const DATE_OPTIONS = ['day', 'week', 'month'];
export const DATE_FORMATTER = ['yyyy-MM-dd', 'yyyyMMdd', 'yyyy-MM', 'yyyyMM'];
export const DATE_FORMATTER = [
'yyyy-MM-dd',
'yyyy-MM-dd HH:mm:ss',
'yyyy-MM-dd HH:mm',
'yyyy-MM-dd HH',
'yyyyMMdd',
'yyyy-MM',
'yyyyMM',
];
export const PARTITION_TIME_FORMATTER = [
'yyyy-MM-dd',
'yyyy-MM-dd HH',
'yyyyMMdd',
'yyyy-MM',
'yyyyMM',
];