Migraine Pulse – duomenų schemos perkėlimas į backendą

Santrauka: Šiame straipsnyje aprašomas duomenų schemos perkėlimas iš frontend’o į backend’ą Migraine Pulse programėlėje. Sužinosite, kaip implementavome dinaminę schemą naudojant NestJS, JSON schema validaciją ir kaip tai palengvino duomenų eksporto/importo procesą.

Kaip ir rašiau įvadiniame įraše apie aplikaciją - pirmą versija buvo prototipas. React aplikacija į kurią aš importavau JSON formatą laikoma lokaliai. Jis užpildydavo aplikaciją mano migrenes įvykių, simptomų ir t.t. Aš patobulindavau tam tikrus dalykus, pakeisdavau (pridėdavau duomenų). Pabaigoje darydavau eksportą, kad turėti pačius naujausius pakeitimus.

Tuo pačiu patikrindavau ir tobulindavau reikalingus laukus kiekvienam elementui.

Paskui jau perėjau kurti backend kuris jau turėjo ryšį su duomenų baze. Bet kuriant modulius NestJS atsižvelgdavau į schemą kurią atidirbau aukščiau.

Bet iki šiol pats formatas schemos buvo aprašytas statiškai puslapyje settings/DataManagement.

Nutariau jį padaryti dinamišką, t.y. pasikeitus laukams ar pridėjus, kad nereikėtų rankomis kaskart keisti puslapyje.

Implementacija

Pirmas žingsnis gana paprastas sukurti NestJS nauja modulį data-management su servisu ir kontroleriu.

nest g module data-management 
nest g service data-management
nest g controller data-management

Servise grąžina JSON schemos objektą

import { Injectable } from '@nestjs/common';
import Ajv from 'ajv';
import addFormats from 'ajv-formats';
import {
  dtoToJsonSchema,
  ClassType,
  JsonSchema,
} from './utils/dto-to-json-schema';
import { CreateIncidentDto } from '../incidents/dto/create-incident.dto';
import { CreateTriggerDto } from '../triggers/dto/create-trigger.dto';
import { CreateSymptomDto } from '../symptoms/dto/create-symptom.dto';
import { CreateMedicationDto } from '../medications/dto/create-medication.dto';
import { CreateLocationDto } from '../locations/dto/create-locations.dto';
import {
  CreateWaterDto,
  CreateBloodPressureDto,
  CreateHeightDto,
  CreateSleepDto,
  CreateWeightDto,
} from '../health-logs/dto/create-health-logs.dto';
import { DataValidationResponse } from './interface/validation.interface';

@Injectable()
export class DataManagementService {
  private ajv: Ajv;

  constructor() {
    this.ajv = new Ajv({ allErrors: true });
    addFormats(this.ajv); 
  }

  getSchema() {
    // Incident schema
    const incidentObjectSchema = dtoToJsonSchema(
      CreateIncidentDto as ClassType<unknown>,
    );

    incidentObjectSchema.properties = incidentObjectSchema.properties ?? {};
    incidentObjectSchema.properties.id = { type: 'string' } as JsonSchema;
    incidentObjectSchema.properties.createdAt = {
      type: 'string',
      format: 'date-time',
    } as JsonSchema;

    let existingRequired = Array.isArray(incidentObjectSchema.required)
      ? incidentObjectSchema.required
      : [];
    let requiredSet = new Set<string>(existingRequired);
    ['userId', 'type', 'startTime', 'datetimeAt'].forEach((r) =>
      requiredSet.add(r),
    );
    incidentObjectSchema.required = Array.from(requiredSet);

    const incidentsSchema: JsonSchema = {
      type: 'array',
      items: incidentObjectSchema,
    };
    // Other schemas from DTOs

    const rootSchema: JsonSchema = {
      type: 'object',
      properties: {
        incidents: incidentsSchema,
        triggers: triggersSchema,
        symptoms: symptomsSchema,
        medications: medicationsSchema,
        locations: locationsSchema,
        healthLogs: healthLogsSchema,
        exportedAt: { type: 'string', format: 'date-time' },
        version: { type: 'string' },
      },
      required: [
        'incidents',
        'triggers',
        'symptoms',
        'medications',
        'locations',
        'healthLogs',
        'exportedAt',
        'version',
      ],
      additionalProperties: false,
    };

    return rootSchema;
  }
}

Kontroleryje grąžina JSON schemos objektą iš serviso. Padaromas pasiekiamas su USER role, nes tas puslapis pasiekiamas tik prisijungusiems.

import { Controller, Get } from '@nestjs/common';
import { DataManagementService } from './data-management.service';
import { Roles } from '../auth/decorators/roles.decorator';
import { Role } from '../auth/enums/roles.enum';

@Controller('data-management')
export class DataManagementController {
  constructor(private readonly dataManagementService: DataManagementService) {}

  @Roles(Role.USER)
  @Get('schema')
  getSchema(): object {
    return this.dataManagementService.getSchema();
  }
}

Integruoti naujai sukurtą į settings/DataManagement puslapį.

export const getDataSchema = async (token: string) => {
  const response = await fetch(`${env.MIGRAINE_BACKEND_API_URL}/api/v1/data-management/schema`, {
    headers: getHeaders(token),
  });

  if (!response.ok) {
    throw new Error('Failed to fetch data schema');
  }

  return response.json();
};

Ir pirmas žingsnis suteikia galimybę modifikuoti ne HTML, o JSON schemą, jau ją aprašinėti ir skaidyti į dalis. Pridėti versijavimą. Validuoti prieš importuojant bylas, kad nesulaužyti duomenų struktūros.

Validacija schemos servisas ir kontroleris

Pridedame naujus npm paketus skirtus validacijai naudojant json schemas.

pnpm add ajv ajv-formats

import { Injectable } from '@nestjs/common';
import Ajv from 'ajv';
import addFormats from 'ajv-formats';
import {
  dtoToJsonSchema,
  ClassType,
  JsonSchema,
} from './utils/dto-to-json-schema';
import { CreateIncidentDto } from '../incidents/dto/create-incident.dto';
import { CreateTriggerDto } from '../triggers/dto/create-trigger.dto';
import { CreateSymptomDto } from '../symptoms/dto/create-symptom.dto';
import { CreateMedicationDto } from '../medications/dto/create-medication.dto';
import { CreateLocationDto } from '../locations/dto/create-locations.dto';
import {
  CreateWaterDto,
  CreateBloodPressureDto,
  CreateHeightDto,
  CreateSleepDto,
  CreateWeightDto,
} from '../health-logs/dto/create-health-logs.dto';
import { DataValidationResponse } from './interface/validation.interface';

@Injectable()
export class DataManagementService {
  private ajv: Ajv;

  constructor() {
    this.ajv = new Ajv({ allErrors: true });
    addFormats(this.ajv); 
  }

  getSchema() {
    // Above content
  }

  /**
   * Validates parsed JSON data against our schema
   */
  validateImportData(data: unknown): DataValidationResponse {
    const schema: JsonSchema = this.getSchema();
    const validate = this.ajv.compile(schema);
    const isValid = validate(data);

    if (!isValid) {
      return { isValid: false, errors: validate.errors };
    }
    return { isValid: true };
  }
}

Pagalbine funkcija mapValidatorToSchema konvertuoti DTO į JSON schemas

import 'reflect-metadata';
import { getMetadataStorage } from 'class-validator';

export interface ClassType<T = unknown> {
  new (...args: unknown[]): T;
}

type PrimitiveTypeName = 'string' | 'number' | 'integer' | 'boolean' | 'null';

export interface PrimitiveSchema {
  type: PrimitiveTypeName;
  format?: string;
  enum?: (string | number | boolean | null)[];
}

export interface ArraySchema {
  type: 'array';
  items: JsonSchema;
}

export interface ObjectSchema {
  type: 'object';
  properties?: Record<string, JsonSchema>;
  required?: string[];
  additionalProperties?: boolean;
}

export type JsonSchema = PrimitiveSchema | ArraySchema | ObjectSchema;

interface ValidationMetadataLite {
  propertyName: string;
  type: string;
  constraints?: unknown[];
  each?: boolean;
}

interface MetadataStorageLite {
  getTargetValidationMetadatas(
    targetConstructor: Function, // eslint-disable-line @typescript-eslint/no-unsafe-function-type
    targetSchema?: string,
  ): ValidationMetadataLite[];
}

function asMetadataStorage(obj: unknown): MetadataStorageLite | null {
  if (obj && typeof obj === 'object' && 'getTargetValidationMetadatas' in obj) {
    const cast = obj as any; // eslint-disable-line @typescript-eslint/no-explicit-any, @typescript-eslint/no-unsafe-assignment
    // eslint-disable-next-line @typescript-eslint/no-unsafe-member-access
    if (typeof cast.getTargetValidationMetadatas === 'function') {
      return cast as MetadataStorageLite;
    }
  }
  return null;
}

function mapValidatorToSchema(
  meta: ValidationMetadataLite | undefined,
  designType?: Function, // eslint-disable-line @typescript-eslint/no-unsafe-function-type
): JsonSchema {
  if (!meta) {
    if (designType === String) return { type: 'string' };
    if (designType === Number) return { type: 'number' };
    if (designType === Boolean) return { type: 'boolean' };
    if (designType === Array)
      return { type: 'array', items: { type: 'string' } };
    return { type: 'object', additionalProperties: true };
  }

  switch (meta.type) {
    case 'isString':
      return { type: 'string' };
    case 'isNumber':
      return { type: 'number' };
    case 'isInt':
      return { type: 'integer' };
    case 'isBoolean':
      return { type: 'boolean' };
    case 'isDateString':
      return { type: 'string', format: 'date-time' };
    case 'isUUID':
      return { type: 'string', format: 'uuid' };
    case 'isEmail':
      return { type: 'string', format: 'email' };
    case 'isArray':
      return { type: 'array', items: { type: 'string' } };
    case 'isEnum': {
      const maybeEnum = Array.isArray(meta.constraints?.[0])
        ? (meta.constraints[0] as (string | number | boolean)[])
        : undefined;
      const schema: PrimitiveSchema = { type: 'string' };
      if (maybeEnum) schema.enum = maybeEnum;
      return schema;
    }
    default:
      if (designType === String) return { type: 'string' };
      if (designType === Number) return { type: 'number' };
      if (designType === Boolean) return { type: 'boolean' };
      if (designType === Array)
        return { type: 'array', items: { type: 'string' } };
      return { type: 'object', additionalProperties: true };
  }
}

/**
 * Convert a DTO class to a JSON Schema object.
 */
export function dtoToJsonSchema<T = unknown>(
  dtoClass: ClassType<T>,
): ObjectSchema {
  const rawStorage = getMetadataStorage();
  const storage = asMetadataStorage(rawStorage);
  const metadatas: ValidationMetadataLite[] = storage
    ? storage.getTargetValidationMetadatas(dtoClass as unknown as Function, '') // eslint-disable-line @typescript-eslint/no-unsafe-function-type
    : [];

  const byProp = new Map<string, ValidationMetadataLite[]>();
  for (const m of metadatas) {
    const prop = m.propertyName;
    const arr = byProp.get(prop) ?? [];
    arr.push(m);
    byProp.set(prop, arr);
  }

  const properties: Record<string, JsonSchema> = {};
  const required: string[] = [];

  for (const [prop, metas] of byProp.entries()) {
    const designType = Reflect.getMetadata(
      'design:type',
      dtoClass.prototype, // eslint-disable-line @typescript-eslint/no-unsafe-argument
      prop,
    ) as Function | undefined; // eslint-disable-line @typescript-eslint/no-unsafe-function-type

    const isArray = metas.some((m) => m.type === 'isArray');
    if (isArray) {
      const eachMeta = metas.find(
        (m) => m.each === true && m.type !== 'isArray',
      );
      const itemsSchema = mapValidatorToSchema(eachMeta, undefined);
      properties[prop] = { type: 'array', items: itemsSchema };
    } else {
      const preferredOrder = [
        'isDateString',
        'isEnum',
        'isNumber',
        'isInt',
        'isBoolean',
        'isUUID',
        'isEmail',
        'isString',
        'isObject',
      ];
      let chosen: ValidationMetadataLite | undefined;
      for (const t of preferredOrder) {
        chosen = metas.find((m) => m.type === t);
        if (chosen) break;
      }
      if (!chosen) chosen = metas[0];

      const propSchema = mapValidatorToSchema(chosen, designType);

      const hasNested = metas.some((m) => m.type === 'nestedValidation');
      if (
        hasNested &&
        designType &&
        designType !== Object &&
        designType !== Array
      ) {
        properties[prop] = dtoToJsonSchema(designType as ClassType<unknown>);
      } else {
        properties[prop] = propSchema;
      }
    }

    const isOptional = metas.some((m) => m.type === 'isOptional');
    const hasNotEmpty = metas.some((m) => m.type === 'isNotEmpty');
    if (!isOptional && hasNotEmpty) required.push(prop);
  }

  const schema: ObjectSchema = {
    type: 'object',
    properties,
    additionalProperties: false,
  };
  if (required.length) schema.required = required;
  return schema;
}

Kontrolerio paketimas

import { 
  Controller, 
  Get, 
  Post, 
  UploadedFile, 
  UseInterceptors, 
  BadRequestException 
} from '@nestjs/common';
import { FileInterceptor } from '@nestjs/platform-express';
import { DataManagementService } from './data-management.service';
import { Public } from '../auth/decorators/public.decorator';

@Controller('data-management')
export class DataManagementController {
  constructor(private readonly dataManagementService: DataManagementService) {}

  @Roles(Role.USER)
  @Get('schema')
  getSchema() {
    return this.dataManagementService.getSchema();
  }

  @Roles(Role.USER)
  @Post('validate-upload')
  @UseInterceptors(FileInterceptor('file'))
  validateFile(
    @UploadedFile() file: Express.Multer.File,
  ): DataValidationResponse {
    if (!file) {
      throw new BadRequestException('No file uploaded.');
    }

    if (file.mimetype !== 'application/json') {
      throw new BadRequestException('Uploaded file must be a JSON file.');
    }

    try {
      const fileContent: string = file.buffer.toString('utf-8');

      const jsonData: unknown = JSON.parse(fileContent);

      const validationResult: unknown =
        this.dataManagementService.validateImportData(jsonData);

      if (validationResult instanceof Error) {
        throw new BadRequestException(validationResult.message);
      }

      return validationResult as DataValidationResponse;
    } catch (error) {
      ErrorExceptionLogging(error, DataManagementController.name);
      throw error;
    }
  }
}

Panaudojimas frontende:

// Example frontend implementation
const checkUploadedFile = async (file: File) => {
  const formData = new FormData();
  formData.append('file', file);

  const response = await fetch(`${API_URL}/api/v1/data-management/validate-upload`, {
    method: 'POST',
    body: formData,
  });

  const result = await response.json();

  if (result.isValid) {
    console.log("File is perfect! Proceed with import.");
  } else {
    console.error("Validation failed:", result.errors);
  }
};

Čia pilni kodo pakeitimai - backend ir frontend