Video Measurement

The Vitals SDK uses advanced computer vision and signal processing techniques to extract physiological wellness indicators from video streams. This section explains the complete measurement pipeline in detail.

Overview

The video measurement pipeline consists of several key stages:

  1. Video Capture: Acquire continuous video stream from camera
  2. ROI Detection: Identify optimal regions for signal extraction
  3. Signal Extraction: Extract rPPG signal using color processing
  4. Preprocessing: Clean and normalize the signal
  5. Feature Extraction: Extract physiological features from signal
  6. Quality Assessment: Evaluate signal quality and frame usability
  7. Window Aggregation: Aggregate measurements over time windows

ROI Detection

Region of Interest (ROI) detection is critical for accurate signal extraction. The SDK identifies optimal facial regions with good blood perfusion and minimal motion.

Forehead-Based ROI

The primary ROI is the forehead region, which offers several advantages:

  • High vascular density and blood flow visibility
  • Minimal expression-related movements
  • Relatively uniform skin tone
  • Less occlusion by hair or accessories
  • Optimal distance from camera in typical use cases
javascript
// ROI configuration
const roiConfig = {
  primaryRegion: 'forehead',
  padding: 20, // pixels
  fallbackRegions: ['cheeks', 'nose'],
  detectionMethod: 'landmark-based',
  minFaceSize: 100, // pixels
  maxFaceSize: 800 // pixels
};

// SDK automatically handles ROI selection
session.setROI(roiConfig);

Alternative ROIs

The SDK can also use alternative regions if the forehead is not suitable:

  • Cheeks: Good blood perfusion, larger surface area
  • Nose: Central facial feature, consistent lighting
  • Combined ROIs: Multiple regions for improved signal quality

Signal Processing Pipeline

Once the ROI is identified, the SDK extracts and processes the rPPG signal through multiple stages.

Signal Extraction - CHROM Method

The SDK uses the CHROM-based color processing method, which is robust to motion and illumination changes:

  • Extract RGB values from ROI pixels
  • Apply chrominance-based signal separation
  • Combine color channels to isolate blood pulsation signal
  • Compensate for motion artifacts
javascript
// CHROM signal extraction process
function extractCHROMSignal(rgbData) {
  // 1. Extract color channels
  const R = rgbData.r;
  const G = rgbData.g;
  const B = rgbData.b;
  
  // 2. Normalize channels
  const Xs = (3 * R - 2 * G) / 2;
  const Ys = (1.5 * R + G - 1.5 * B) / 2;
  
  // 3. Apply chrominance processing
  const S = Xs - (mean(Xs) / mean(Ys)) * Ys;
  
  // 4. Bandpass filter for cardiac frequency
  const filtered = bandpassFilter(S, 0.7, 4.0); // 0.7-4.0 Hz (42-240 BPM)
  
  return filtered;
}

Signal Preprocessing

Raw signals require preprocessing to remove noise and artifacts:

  • Detrending: Remove slow variations and baseline drift
  • Bandpass Filtering: Isolate cardiac frequency band (0.7-4.0 Hz)
  • Normalization: Scale signal to consistent amplitude
  • Outlier Removal: Reject extreme values caused by motion
javascript
// Preprocessing pipeline
function preprocessSignal(rawSignal) {
  // 1. Detrending - remove slow variations
  const detrended = detrend(rawSignal, method: 'linear');
  
  // 2. Bandpass filtering (0.7-4.0 Hz for cardiac)
  const filtered = butterworthFilter(
    detrended,
    lowCut: 0.7,
    highCut: 4.0,
    order: 4
  );
  
  // 3. Normalization (z-score)
  const normalized = zscore(filtered);
  
  // 4. Outlier removal
  const cleaned = removeOutliers(normalized, threshold: 3);
  
  return cleaned;
}

Heart Rate Estimation

Heart rate is estimated using frequency domain analysis with peak validation.

FFT-Based Estimation

  • Apply Fast Fourier Transform (FFT) to preprocessed signal
  • Identify dominant frequency in cardiac band
  • Convert frequency to BPM: BPM = frequency * 60
  • Apply SNR gating to ensure reliable detection
javascript
// Heart rate estimation
function estimateHeartRate(signal, samplingRate) {
  // 1. Apply FFT
  const fftResult = fft(signal);
  
  // 2. Find power spectrum
  const powerSpectrum = computePowerSpectrum(fftResult);
  
  // 3. Search cardiac frequency band (0.7-4.0 Hz)
  const frequencies = [];
  const powers = [];
  
  for (let i = 0; i < powerSpectrum.length; i++) {
    const freq = i * samplingRate / signal.length;
    if (freq >= 0.7 && freq <= 4.0) {
      frequencies.push(freq);
      powers.push(powerSpectrum[i]);
    }
  }
  
  // 4. Find dominant frequency
  const dominantIndex = powers.indexOf(Math.max(...powers));
  const heartRateHz = frequencies[dominantIndex];
  const heartRateBPM = heartRateHz * 60;
  
  // 5. Calculate SNR
  const snr = calculateSNR(powers, dominantIndex);
  
  return {
    heartRate: heartRateBPM,
    confidence: Math.min(snr / 10, 1.0),
    quality: snr > 10 ? 'good' : 'poor'
  };
}

Peak-Based Validation

  • Detect peaks in time-domain signal
  • Calculate inter-beat intervals (IBI)
  • Validate consistency of peak detection
  • Remove outliers from IBI sequence

Window-Based Aggregation

The SDK uses a sliding window approach to provide stable, averaged results.

Default Window Configuration

  • Window Duration: 45 seconds (default, configurable)
  • Overlap: 0-15 seconds (for continuous monitoring)
  • Update Rate: Every 1-5 seconds
javascript
// Window aggregation
class WindowAggregator {
  constructor(windowSize = 45) {
    this.windowSize = windowSize; // seconds
    this.measurements = [];
    this.lastUpdate = Date.now();
  }
  
  addMeasurement(measurement) {
    const timestamp = Date.now();
    this.measurements.push({
      ...measurement,
      timestamp
    });
    
    // Remove old measurements outside window
    const cutoff = timestamp - (this.windowSize * 1000);
    this.measurements = this.measurements.filter(
      m => m.timestamp > cutoff
    );
    
    // Calculate averaged result
    return this.calculateAverage();
  }
  
  calculateAverage() {
    if (this.measurements.length === 0) return null;
    
    const heartRates = this.measurements.map(m => m.heartRate);
    const avgHeartRate = mean(heartRates);
    const stdHeartRate = std(heartRates);
    
    return {
      averagedHeartRate: avgHeartRate,
      heartRateVariability: stdHeartRate,
      windowSize: this.measurements.length,
      confidence: this.calculateConfidence(heartRates)
    };
  }
  
  calculateConfidence(values) {
    // Confidence based on stability and window fill
    const stability = 1 - (std(values) / mean(values));
    const fill = this.measurements.length / (this.windowSize * 30); // assuming 30 FPS
    
    return Math.min(stability * fill, 1.0);
  }
}

Signal Smoothing

To prevent unrealistic jumps and provide smooth transitions, the SDK applies smoothing constraints:

  • Moving Average: Smooth short-term fluctuations
  • Rate Limiter: Limit maximum change between consecutive measurements
  • Kalman Filter: Predict and correct measurements (optional)
  • Outlier Rejection: Discard values that deviate significantly from expected range
javascript
// Signal smoothing
class Smoother {
  constructor(maxChange = 10) {
    this.maxChange = maxChange; // maximum BPM change per second
    this.previousValue = null;
    this.previousTimestamp = null;
  }
  
  smooth(value, timestamp) {
    if (this.previousValue === null) {
      this.previousValue = value;
      this.previousTimestamp = timestamp;
      return value;
    }
    
    const timeDelta = (timestamp - this.previousTimestamp) / 1000;
    const allowedChange = this.maxChange * timeDelta;
    const actualChange = Math.abs(value - this.previousValue);
    
    let smoothed;
    if (actualChange <= allowedChange) {
      // Within reasonable range
      smoothed = value;
    } else {
      // Constrain to maximum allowed change
      smoothed = this.previousValue + 
        Math.sign(value - this.previousValue) * allowedChange;
    }
    
    this.previousValue = smoothed;
    this.previousTimestamp = timestamp;
    
    return smoothed;
  }
}

Frame Quality Gating

The SDK continuously evaluates frame quality and rejects low-quality frames to ensure accurate measurements.

Quality Metrics

  • Brightness: Frame brightness within optimal range
  • Sharpness: Image clarity and focus
  • Face Detection: Successful face detection and tracking
  • Motion: Minimal subject and camera motion
  • ROI Visibility: ROI region not occluded
javascript
// Quality assessment
function assessFrameQuality(frame, faceDetection) {
  const quality = {
    brightness: calculateBrightness(frame),
    sharpness: calculateSharpness(frame),
    faceDetected: faceDetection.success,
    motion: calculateMotion(frame),
    roiVisible: checkROIVisibility(faceDetection)
  };
  
  // Apply thresholds
  const thresholds = {
    brightness: { min: 50, max: 200 },
    sharpness: { min: 0.5 },
    motion: { max: 10 }
  };
  
  const isAcceptable = 
    quality.brightness >= thresholds.brightness.min &&
    quality.brightness <= thresholds.brightness.max &&
    quality.sharpness >= thresholds.sharpness.min &&
    quality.faceDetected &&
    quality.motion <= thresholds.motion.max &&
    quality.roiVisible;
  
  return {
    ...quality,
    isAcceptable,
    score: calculateQualityScore(quality)
  };
}

function calculateQualityScore(quality) {
  let score = 0;
  
  if (quality.faceDetected) score += 0.3;
  if (quality.roiVisible) score += 0.2;
  if (quality.sharpness >= 0.5) score += 0.2;
  if (quality.brightness >= 50 && quality.brightness <= 200) score += 0.2;
  if (quality.motion <= 10) score += 0.1;
  
  return score;
}

Usable Frame Ratio

The SDK tracks the ratio of usable frames to total frames processed:

  • Calculation: Usable Frame Ratio = Usable Frames / Total Frames
  • Target: > 0.7 (70%) for reliable results
  • Minimum: 0.5 (50%) for any results
Important: Results are AI-estimated wellness indicators and are not a medical diagnosis. The accuracy of measurements depends heavily on proper lighting, camera quality, and subject cooperation.

Signal Confidence Scoring

Each measurement includes a confidence score based on multiple factors:

  • Signal-to-Noise Ratio (SNR)
  • Usable Frame Ratio
  • Window Fill Percentage
  • Measurement Stability
  • Quality Metrics Consistency
javascript
// Confidence scoring
function calculateConfidence(measurements) {
  const factors = {
    snr: calculateSNR(measurements),
    usableFrameRatio: calculateUsableFrameRatio(measurements),
    windowFill: calculateWindowFill(measurements),
    stability: calculateStability(measurements),
    qualityConsistency: calculateQualityConsistency(measurements)
  };
  
  // Weighted average
  const weights = {
    snr: 0.3,
    usableFrameRatio: 0.25,
    windowFill: 0.2,
    stability: 0.15,
    qualityConsistency: 0.1
  };
  
  let confidence = 0;
  for (const [factor, weight] of Object.entries(weights)) {
    confidence += factors[factor] * weight;
  }
  
  return Math.min(Math.max(confidence, 0), 1);
}

Next Steps