Learn how Migraine Pulse implemented a dynamic data schema in the backend. We added JSON schema validation, versioning, and reliable export/import functionality using NestJS.

Summary: This article describes moving the data schema from frontend to backend in the Migraine Pulse app. You’ll learn how we implemented a dynamic schema using NestJS, JSON schema validation, and how it simplified the data export/import process.

As I mentioned in the introductory post about the app - the first version was a prototype. A React application where I imported JSON format stored locally. It would populate the app with my migraine incidents, symptoms, etc. I would refine certain things, modify (add data). At the end, I would do an export to have the latest changes.

At the same time, I would check and refine the required fields for each element.

Later I started building a backend which already had a connection to the database. But when creating NestJS modules, I took into account the schema I worked out above.

However, until now the schema format itself was described statically on the settings/DataManagement page.

I decided to make it dynamic, i.e., when fields change or are added, so you don’t have to manually edit the page every time.

Implementation

The first step is simple - create a new NestJS data-management module with service and controller.

nest g module data-management 
nest g service data-management
nest g controller data-management

The service returns a JSON schema object.

import { Injectable } from '@nestjs/common';
import Ajv from 'ajv';
import addFormats from 'ajv-formats';
import {
  dtoToJsonSchema,
  ClassType,
  JsonSchema,
} from './utils/dto-to-json-schema';
import { CreateIncidentDto } from '../incidents/dto/create-incident.dto';
import { CreateTriggerDto } from '../triggers/dto/create-trigger.dto';
import { CreateSymptomDto } from '../symptoms/dto/create-symptom.dto';
import { CreateMedicationDto } from '../medications/dto/create-medication.dto';
import { CreateLocationDto } from '../locations/dto/create-locations.dto';
import {
  CreateWaterDto,
  CreateBloodPressureDto,
  CreateHeightDto,
  CreateSleepDto,
  CreateWeightDto,
} from '../health-logs/dto/create-health-logs.dto';
import { DataValidationResponse } from './interface/validation.interface';

@Injectable()
export class DataManagementService {
  private ajv: Ajv;

  constructor() {
    this.ajv = new Ajv({ allErrors: true });
    addFormats(this.ajv); 
  }

  getSchema() {
    const incidentObjectSchema = dtoToJsonSchema(
      CreateIncidentDto as ClassType<unknown>,
    );

    incidentObjectSchema.properties = incidentObjectSchema.properties ?? {};
    incidentObjectSchema.properties.id = { type: 'string' } as JsonSchema;
    incidentObjectSchema.properties.createdAt = {
      type: 'string',
      format: 'date-time',
    } as JsonSchema;

    let existingRequired = Array.isArray(incidentObjectSchema.required)
      ? incidentObjectSchema.required
      : [];
    let requiredSet = new Set<string>(existingRequired);
    ['userId', 'type', 'startTime', 'datetimeAt'].forEach((r) =>
      requiredSet.add(r),
    );
    incidentObjectSchema.required = Array.from(requiredSet);

    const incidentsSchema: JsonSchema = {
      type: 'array',
      items: incidentObjectSchema,
    };

    const rootSchema: JsonSchema = {
      type: 'object',
      properties: {
        incidents: incidentsSchema,
        triggers: triggersSchema,
        symptoms: symptomsSchema,
        medications: medicationsSchema,
        locations: locationsSchema,
        healthLogs: healthLogsSchema,
        exportedAt: { type: 'string', format: 'date-time' },
        version: { type: 'string' },
      },
      required: [
        'incidents',
        'triggers',
        'symptoms',
        'medications',
        'locations',
        'healthLogs',
        'exportedAt',
        'version',
      ],
      additionalProperties: false,
    };

    return rootSchema;
  }
}

In the controller, return the JSON schema object from the service. It’s accessible with USER role, since that page is only accessible to logged-in users.

import { Controller, Get } from '@nestjs/common';
import { DataManagementService } from './data-management.service';
import { Roles } from '../auth/decorators/roles.decorator';
import { Role } from '../auth/enums/roles.enum';

@Controller('data-management')
export class DataManagementController {
  constructor(private readonly dataManagementService: DataManagementService) {}

  @Roles(Role.USER)
  @Get('schema')
  getSchema(): object {
    return this.dataManagementService.getSchema();
  }
}

Integrate the newly created into the settings/DataManagement page.

export const getDataSchema = async (token: string) => {
  const response = await fetch(`${env.MIGRAINE_BACKEND_API_URL}/api/v1/data-management/schema`, {
    headers: getHeaders(token),
  });

  if (!response.ok) {
    throw new Error('Failed to fetch data schema');
  }

  return response.json();
};

And the first step provides the ability to modify not HTML, but JSON schema, to describe it and split into parts. Add versioning. Validate files before importing to not break the data structure.

Schema validation service and controller

Add new npm packages for validation using json schemas.

pnpm add ajv ajv-formats
import { Injectable } from '@nestjs/common';
import Ajv from 'ajv';
import addFormats from 'ajv-formats';
import {
  dtoToJsonSchema,
  ClassType,
  JsonSchema,
} from './utils/dto-to-json-schema';
import { CreateIncidentDto } from '../incidents/dto/create-incident.dto';
import { CreateTriggerDto } from '../triggers/dto/create-trigger.dto';
import { CreateSymptomDto } from '../symptoms/dto/create-symptom.dto';
import { CreateMedicationDto } from '../medications/dto/create-medication.dto';
import { CreateLocationDto } from '../locations/dto/create-locations.dto';
import {
  CreateWaterDto,
  CreateBloodPressureDto,
  CreateHeightDto,
  CreateSleepDto,
  CreateWeightDto,
} from '../health-logs/dto/create-health-logs.dto';
import { DataValidationResponse } from './interface/validation.interface';

@Injectable()
export class DataManagementService {
  private ajv: Ajv;

  constructor() {
    this.ajv = new Ajv({ allErrors: true });
    addFormats(this.ajv); 
  }

  getSchema() {
    // Above content
  }

  validateImportData(data: unknown): DataValidationResponse {
    const schema: JsonSchema = this.getSchema();
    const validate = this.ajv.compile(schema);
    const isValid = validate(data);

    if (!isValid) {
      return { isValid: false, errors: validate.errors };
    }
    return { isValid: true };
  }
}

Helper function mapValidatorToSchema to convert DTO to JSON schema.

import 'reflect-metadata';
import { getMetadataStorage } from 'class-validator';

export interface ClassType<T = unknown> {
  new (...args: unknown[]): T;
}

type PrimitiveTypeName = 'string' | 'number' | 'integer' | 'boolean' | 'null';

export interface PrimitiveSchema {
  type: PrimitiveTypeName;
  format?: string;
  enum?: (string | number | boolean | null)[];
}

export interface ArraySchema {
  type: 'array';
  items: JsonSchema;
}

export interface ObjectSchema {
  type: 'object';
  properties?: Record<string, JsonSchema>;
  required?: string[];
  additionalProperties?: boolean;
}

export type JsonSchema = PrimitiveSchema | ArraySchema | ObjectSchema;

interface ValidationMetadataLite {
  propertyName: string;
  type: string;
  constraints?: unknown[];
  each?: boolean;
}

interface MetadataStorageLite {
  getTargetValidationMetadatas(
    targetConstructor: Function,
    targetSchema?: string,
  ): ValidationMetadataLite[];
}

function asMetadataStorage(obj: unknown): MetadataStorageLite | null {
  if (obj && typeof obj === 'object' && 'getTargetValidationMetadatas' in obj) {
    const cast = obj as any;
    if (typeof cast.getTargetValidationMetadatas === 'function') {
      return cast as MetadataStorageLite;
    }
  }
  return null;
}

function mapValidatorToSchema(
  meta: ValidationMetadataLite | undefined,
  designType?: Function,
): JsonSchema {
  if (!meta) {
    if (designType === String) return { type: 'string' };
    if (designType === Number) return { type: 'number' };
    if (designType === Boolean) return { type: 'boolean' };
    if (designType === Array)
      return { type: 'array', items: { type: 'string' } };
    return { type: 'object', additionalProperties: true };
  }

  switch (meta.type) {
    case 'isString':
      return { type: 'string' };
    case 'isNumber':
      return { type: 'number' };
    case 'isInt':
      return { type: 'integer' };
    case 'isBoolean':
      return { type: 'boolean' };
    case 'isDateString':
      return { type: 'string', format: 'date-time' };
    case 'isUUID':
      return { type: 'string', format: 'uuid' };
    case 'isEmail':
      return { type: 'string', format: 'email' };
    case 'isArray':
      return { type: 'array', items: { type: 'string' } };
    case 'isEnum': {
      const maybeEnum = Array.isArray(meta.constraints?.[0])
        ? (meta.constraints[0] as (string | number | boolean)[])
        : undefined;
      const schema: PrimitiveSchema = { type: 'string' };
      if (maybeEnum) schema.enum = maybeEnum;
      return schema;
    }
    default:
      if (designType === String) return { type: 'string' };
      if (designType === Number) return { type: 'number' };
      if (designType === Boolean) return { type: 'boolean' };
      if (designType === Array)
        return { type: 'array', items: { type: 'string' } };
      return { type: 'object', additionalProperties: true };
  }
}

export function dtoToJsonSchema<T = unknown>(
  dtoClass: ClassType<T>,
): ObjectSchema {
  const rawStorage = getMetadataStorage();
  const storage = asMetadataStorage(rawStorage);
  const metadatas: ValidationMetadataLite[] = storage
    ? storage.getTargetValidationMetadatas(dtoClass as unknown as Function, '')
    : [];

  const byProp = new Map<string, ValidationMetadataLite[]>();
  for (const m of metadatas) {
    const prop = m.propertyName;
    const arr = byProp.get(prop) ?? [];
    arr.push(m);
    byProp.set(prop, arr);
  }

  const properties: Record<string, JsonSchema> = {};
  const required: string[] = [];

  for (const [prop, metas] of byProp.entries()) {
    const designType = Reflect.getMetadata(
      'design:type',
      dtoClass.prototype,
      prop,
    ) as Function | undefined;

    const isArray = metas.some((m) => m.type === 'isArray');
    if (isArray) {
      const eachMeta = metas.find(
        (m) => m.each === true && m.type !== 'isArray',
      );
      const itemsSchema = mapValidatorToSchema(eachMeta, undefined);
      properties[prop] = { type: 'array', items: itemsSchema };
    } else {
      const preferredOrder = [
        'isDateString',
        'isEnum',
        'isNumber',
        'isInt',
        'isBoolean',
        'isUUID',
        'isEmail',
        'isString',
        'isObject',
      ];
      let chosen: ValidationMetadataLite | undefined;
      for (const t of preferredOrder) {
        chosen = metas.find((m) => m.type === t);
        if (chosen) break;
      }
      if (!chosen) chosen = metas[0];

      const propSchema = mapValidatorToSchema(chosen, designType);

      const hasNested = metas.some((m) => m.type === 'nestedValidation');
      if (
        hasNested &&
        designType &&
        designType !== Object &&
        designType !== Array
      ) {
        properties[prop] = dtoToJsonSchema(designType as ClassType<unknown>);
      } else {
        properties[prop] = propSchema;
      }
    }

    const isOptional = metas.some((m) => m.type === 'isOptional');
    const hasNotEmpty = metas.some((m) => m.type === 'isNotEmpty');
    if (!isOptional && hasNotEmpty) required.push(prop);
  }

  const schema: ObjectSchema = {
    type: 'object',
    properties,
    additionalProperties: false,
  };
  if (required.length) schema.required = required;
  return schema;
}

Controller endpoint for file validation.

import { 
  Controller, 
  Get, 
  Post, 
  UploadedFile, 
  UseInterceptors, 
  BadRequestException 
} from '@nestjs/common';
import { FileInterceptor } from '@nestjs/platform-express';
import { DataManagementService } from './data-management.service';
import { Public } from '../auth/decorators/public.decorator';

@Controller('data-management')
export class DataManagementController {
  constructor(private readonly dataManagementService: DataManagementService) {}

  @Roles(Role.USER)
  @Get('schema')
  getSchema() {
    return this.dataManagementService.getSchema();
  }

  @Roles(Role.USER)
  @Post('validate-upload')
  @UseInterceptors(FileInterceptor('file'))
  validateFile(
    @UploadedFile() file: Express.Multer.File,
  ): DataValidationResponse {
    if (!file) {
      throw new BadRequestException('No file uploaded.');
    }

    if (file.mimetype !== 'application/json') {
      throw new BadRequestException('Uploaded file must be a JSON file.');
    }

    try {
      const fileContent: string = file.buffer.toString('utf-8');

      const jsonData: unknown = JSON.parse(fileContent);

      const validationResult: unknown =
        this.dataManagementService.validateImportData(jsonData);

      if (validationResult instanceof Error) {
        throw new BadRequestException(validationResult.message);
      }

      return validationResult as DataValidationResponse;
    } catch (error) {
      ErrorExceptionLogging(error, DataManagementController.name);
      throw error;
    }
  }
}

Frontend usage example:

const checkUploadedFile = async (file: File) => {
  const formData = new FormData();
  formData.append('file', file);

  const response = await fetch(`${API_URL}/api/v1/data-management/validate-upload`, {
    method: 'POST',
    body: formData,
  });

  const result = await response.json();

  if (result.isValid) {
    console.log("File is perfect! Proceed with import.");
  } else {
    console.error("Validation failed:", result.errors);
  }
};

Here full code changes - backend and frontend