This document provides a detailed technical and functional specification for designing and developing an automated inspection tunnel system. Inspired by solutions like UVeye, this system utilizes multiple Luxonis OAK cameras, the DepthAI Python library, and a pre-trained custom YOLO model to capture synchronized, high-resolution data and perform real-time defect or object detection. Special emphasis is placed on the software-based synchronization of data streams from multiple cameras ('n' cameras), offering a robust framework for development, particularly when using AI-assisted coding tools like Cursor.

Key System Highlights

• Multi-Camera Synchronization: Employs a robust software-based strategy using DepthAI timestamps and buffering to align data streams from multiple OAK cameras for cohesive analysis.
• Integrated AI Detection: Seamlessly integrates a custom-trained YOLO model onto the OAK cameras' Myriad X VPU or host machine for efficient, real-time object or defect identification using captured visual data.
• High-Resolution Data Capture: Leverages Luxonis OAK cameras (e.g., OAK-D series) to capture synchronized color (RGB), depth, and potentially IMU data, providing rich information for detailed inspection.

---

System Architecture Overview

Conceptual Framework

The system operates on a distributed architecture. Each Luxonis OAK camera acts as an intelligent sensor, capturing data and potentially running the YOLO model on its integrated VPU. A central computing device manages these cameras, orchestrates the DepthAI pipelines, implements the crucial software synchronization across cameras, aggregates data, performs post-processing, and handles data storage or transmission.

Component Layers

1. Hardware Layer: Consists of 'n' Luxonis OAK cameras (e.g., OAK-D, OAK-D Pro recommended for depth capabilities), a central host computer (PC or embedded system like NVIDIA Jetson), reliable power supplies, networking infrastructure (USB or Ethernet), and mounting hardware for camera placement within the tunnel. Optional components include controlled lighting and motion sensors.
2. Software Layer: Built upon Python 3.x, utilizing the depthai library for camera control and pipeline management, opencv-python for image processing, numpy for numerical operations, the custom YOLO model (in .blob format for Myriad X or ONNX/PyTorch for host processing), and potentially libraries like Redis for inter-process communication if using a distributed processing architecture.
3. Data Flow Layer: Raw data streams (color, depth, IMU) originate from each camera. Data is timestamped by the OAK hardware. The central host receives these streams, applies software synchronization logic, feeds synchronized data to the YOLO model (if processed on host), associates detections with depth information, and outputs structured results.

Architectural Mindmap

This mindmap illustrates the core components and their relationships within the automated inspection tunnel system, highlighting the flow from data capture to final analysis.

---

Functional and Technical Requirements

Functional Needs

• Multi-Sensor Data Acquisition: The system must simultaneously capture synchronized color video, depth maps, and optionally IMU data from all 'n' connected Luxonis OAK cameras.
• Custom Object Detection: Utilize the provided custom YOLO model to detect specific objects or defects (e.g., cracks, dents, anomalies, specific components) within the captured color frames in real-time or near-real-time.
• Data Association: Accurately associate the detections made by the YOLO model with corresponding depth information from the synchronized depth maps to enable spatial understanding (e.g., size, location in 3D).
• Scalable Camera Management: The system architecture must support a variable number ('n') of cameras, allowing for flexible configuration based on tunnel size and inspection coverage requirements.
• Configuration Flexibility: Provide mechanisms (e.g., configuration files) to set parameters like camera resolution, frame rate, synchronization tolerance, and YOLO model specifics.
• Data Handling: Store or transmit the synchronized data streams (video, depth, IMU) along with the YOLO detection results (bounding boxes, classes, confidence scores) for analysis, logging, or visualization.
• Operational Environment: Function effectively within a tunnel environment, considering potential factors like varying lighting conditions (mitigated by camera settings and potentially external lighting) and object motion.

Technical Specifications

• Cameras: Luxonis OAK-D, OAK-D Lite, or OAK-D Pro models are recommended due to their integrated stereo depth capabilities and Myriad X VPU for on-device AI. The exact number 'n' depends on the required field of view and overlap for comprehensive inspection.
• Host System: A capable computing device (PC with sufficient CPU/RAM, or an embedded platform like NVIDIA Jetson or Raspberry Pi 4/5, depending on processing load) running a Linux-based OS or Windows.
• Software Dependencies: Python (3.8+ recommended), depthai library, opencv-python, numpy. If running YOLO on the host, requires relevant libraries (e.g., torch, onnxruntime).
• YOLO Model Format: The custom YOLO model must be available in a compatible format: .blob file (generated using Luxonis tools/OpenVINO) for running on the OAK camera's Myriad X VPU, or standard formats like ONNX or PyTorch (.pt) for host-based inference.
• Connectivity: USB 3 or Ethernet (for PoE models) connections between cameras and the host.
• Synchronization Precision: Software synchronization tolerance should be configurable (e.g., 10-50 milliseconds) to balance alignment accuracy with system responsiveness.

---

Detailed System Design

Camera Arrangement Strategy

The physical placement of the 'n' cameras is critical for achieving full coverage of the inspection target (e.g., tunnel walls, ceiling, floor, or objects passing through). A typical setup might involve:

• Overhead cameras pointing downwards.
• Side-mounted cameras pointing inwards or slightly angled.
• Potential cameras at entrance/exit points for longitudinal views.

Cameras should be positioned with sufficient overlap in their fields of view if 3D reconstruction or seamless stitching of views is desired. Precise calibration of each camera's intrinsic (focal length, distortion) and extrinsic (position, orientation relative to a common coordinate system) parameters is essential for accurate depth mapping and multi-view analysis.

DepthAI Pipeline Design (Per Camera)

Each OAK camera will run an independent DepthAI pipeline configured via Python. A standard pipeline for this system includes:

• ColorCamera Node: Configured for desired resolution (e.g., 1080p, 4K) and FPS. Captures RGB video frames.
• StereoDepth Node: Utilizes the camera's stereo pair (left/right mono cameras) to compute depth maps. Configuration includes setting disparity parameters, confidence thresholds, and potentially enabling filters (e.g., WLS) for improved depth quality.
• IMU Node (Optional): If motion tracking or orientation data is needed, this node captures data from the inertial measurement unit.
• NeuralNetwork Node: Loads and executes the custom YOLO model (.blob file) on the Myriad X VPU. Its input is typically linked from the ColorCamera node's preview or video output.
• Sync Node: Crucial for *intra-camera* synchronization. It receives inputs from multiple nodes (e.g., ColorCamera.video, StereoDepth.depth, NeuralNetwork.out, IMU.out) and outputs synchronized packets containing data from all linked sources, aligned by hardware timestamps.
• XLinkOut Nodes: Used to send the output of specific nodes (especially the Sync node's output) from the camera to the host computer over USB/Ethernet.

Pipeline Configuration Example (Conceptual Python):

import depthai as dai
import time

def create_pipeline(yolo_blob_path):
    pipeline = dai.Pipeline()

    # --- Input Nodes ---
    cam_rgb = pipeline.create(dai.node.ColorCamera)
    cam_rgb.setPreviewSize(640, 640) # Example size for NN
    cam_rgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
    cam_rgb.setInterleaved(False)
    cam_rgb.setColorOrder(dai.ColorCameraProperties.ColorOrder.BGR)
    cam_rgb.setFps(30)

    left = pipeline.create(dai.node.MonoCamera)
    left.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
    left.setBoardSocket(dai.CameraBoardSocket.LEFT)

    right = pipeline.create(dai.node.MonoCamera)
    right.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
    right.setBoardSocket(dai.CameraBoardSocket.RIGHT)

    stereo = pipeline.create(dai.node.StereoDepth)
    stereo.setDefaultProfilePreset(dai.node.StereoDepth.PresetMode.HIGH_DENSITY)
    # Options: MEDIAN_OFF, KERNEL_7x7, CONFIDENCE_THRESHOLD > 200
    stereo.setMedianFilter(dai.MedianFilter.KERNEL_7x7)
    stereo.setLeftRightCheck(True)
    stereo.setExtendedDisparity(False)
    stereo.setSubpixel(False)
    left.out.link(stereo.left)
    right.out.link(stereo.right)

    # --- AI Node ---
    detection_nn = pipeline.create(dai.node.YoloDetectionNetwork)
    # Configure YOLO input (blob path, confidence, NMS, etc.)
    detection_nn.setBlobPath(yolo_blob_path)
    detection_nn.setConfidenceThreshold(0.5)
    detection_nn.setNumClasses(80) # Set to your model's class count
    detection_nn.setCoordinateSize(4)
    detection_nn.setAnchors([]) # Set anchors if needed
    detection_nn.setAnchorMasks({}) # Set anchor masks if needed
    detection_nn.setIouThreshold(0.5)
    detection_nn.setNumInferenceThreads(2)
    detection_nn.input.setBlocking(False)
    cam_rgb.preview.link(detection_nn.input) # Link RGB preview to NN input

    # --- Sync Node ---
    sync = pipeline.create(dai.node.Sync)
    # Specify names for inputs for easier retrieval later
    sync.setSyncThreshold(33 * 1000 * 1000) # 33ms threshold in nanoseconds (approx 1 frame at 30fps)

    cam_rgb.video.link(sync.inputs["rgb"])
    stereo.depth.link(sync.inputs["depth"])
    detection_nn.out.link(sync.inputs["yolo"])

    # --- Output Nodes ---
    sync_out = pipeline.create(dai.node.XLinkOut)
    sync_out.setStreamName("synced_data")
    sync.out.link(sync_out.input)

    return pipeline

Software Synchronization Logic (Across 'n' Cameras)

While the Sync node handles synchronization *within* a single OAK camera, aligning data streams *between* multiple independent OAK cameras requires custom software logic on the host computer. This is the cornerstone of the multi-camera setup.

Strategy: Timestamp-Based Alignment

1. Accurate Timestamps: Each synchronized packet arriving from an OAK camera's Sync node output queue (XLinkOut) contains data (RGB frame, depth map, YOLO results) tagged with a precise device timestamp (getTimestampDevice()).
2. Per-Camera Buffering: The host maintains a separate buffer (e.g., a Python collections.deque) for incoming synchronized packets from each connected camera.
3. Synchronization Window (Tolerance): Define a time window (sync_tolerance_ms, e.g., 30-50ms). This determines the maximum allowable time difference between packets from different cameras to be considered 'synchronized'.
4. Matching Algorithm: Implement logic that continuously monitors the buffers. It attempts to find a set of packets (one from each camera's buffer) whose device timestamps fall within the defined synchronization window relative to each other. A common approach is to use the timestamp of the packet from a reference camera (e.g., the one with the oldest packet waiting) and search for matching packets in other buffers.
5. Grouping Synced Packets: When a complete set of synchronized packets (one per camera) is identified, group them together. This group represents a multi-camera snapshot of the inspection scene at roughly the same moment.
6. Processing the Group: Pass the synchronized group of packets to the next stage for combined analysis (e.g., fusing YOLO results, 3D reconstruction).
7. Buffer Management: Remove the processed packets from their respective buffers. Implement logic to discard packets that remain in the buffer for too long without finding a match (i.e., they are too old) to prevent buffer overflow and keep the system processing recent data.

Conceptual Synchronization Logic (Python Pseudocode):

from collections import deque
import time
import threading

camera_queues = {} # Dictionary to store output queues from DepthAI devices {device_id: queue}
buffers = {} # Dictionary to store incoming packets {device_id: deque()}
synced_data_queue = deque(maxlen=10) # Queue to hold grouped synchronized packets
camera_ids = [] # List of active camera device IDs
sync_tolerance_sec = 0.050 # 50ms tolerance

# --- Data Reception Thread (One per camera) ---
def receive_data(device_id, queue):
    while True:
        packet = queue.tryGet()
        if packet is not None:
            buffers[device_id].append(packet)
        else:
            time.sleep(0.001) # Avoid busy-waiting

# --- Synchronization Thread ---
def synchronize_streams():
    while True:
        if not all(buffers[cam_id] for cam_id in camera_ids):
            time.sleep(0.005) # Wait if any buffer is empty
            continue

        # Find the minimum 'latest' timestamp across all buffers
        # This ensures we don't try to sync too far ahead
        min_latest_ts = min(buf[-1].getTimestampDevice() for buf in buffers.values() if buf)

        # Use the oldest timestamp from the first buffer as a reference point
        # Ensure the reference packet isn't newer than the slowest stream
        if buffers[camera_ids[0]][0].getTimestampDevice() > min_latest_ts:
            time.sleep(0.005)
            continue # Wait for other streams to catch up

        ref_packet = buffers[camera_ids[0]][0]
        ref_ts = ref_packet.getTimestampDevice()

        synced_group = {camera_ids[0]: ref_packet}
        potential_match = True

        # Try to find matching packets in other buffers
        for i in range(1, len(camera_ids)):
            cam_id = camera_ids[i]
            found_match = False
            best_match_packet = None
            min_diff = float('inf')

            for packet in buffers[cam_id]:
                packet_ts = packet.getTimestampDevice()
                diff = abs((packet_ts - ref_ts).total_seconds())

                if diff <= sync_tolerance_sec:
                     # Prefer the closest match within tolerance
                    if diff < min_diff:
                        min_diff = diff
                        best_match_packet = packet
                        found_match = True
                elif packet_ts > ref_ts + timedelta(seconds=sync_tolerance_sec):
                    # Packets are ordered by timestamp, so we can stop searching early
                    break

            if found_match:
                synced_group[cam_id] = best_match_packet
            else:
                potential_match = False
                break # Cannot form a complete group for this ref_packet

        if potential_match:
            # Found a complete synced group
            synced_data_queue.append(synced_group)
            # Remove matched packets from buffers
            for cam_id, packet_to_remove in synced_group.items():
                 # Be careful removing! Need efficient way if not always the first item
                 buffers[cam_id].remove(packet_to_remove) # Note: deque.remove is O(n)
                 # Consider alternative buffer structures or popping if always the oldest.
        else:
            # The reference packet is too old or couldn't be matched
            # Discard the reference packet to allow sync attempts with newer data
            buffers[camera_ids[0].popleft()

        # Add logic to discard very old packets from all buffers periodically

# --- Main Processing Logic ---
def process_synced_data():
    while True:
        if synced_data_queue:
            group = synced_data_queue.popleft()
            # --- Combine YOLO results, use depth, etc. ---
            print(f"Processing synced group with ref timestamp: {group[camera_ids[0]].getTimestampDevice()}")
            # E.g., aggregate YOLO detections from all cameras in the 'group'
            # Access data like: group[cam_id].getData("rgb"), group[cam_id].getData("depth"), group[cam_id].getData("yolo")
            pass
        else:
            time.sleep(0.001)

# --- Initialization ---
# 1. Discover and initialize all OAK devices
# 2. Create and start pipeline on each device
# 3. Get output queues (e.g., "synced_data") into camera_queues
# 4. Initialize buffers dict
# 5. Start receiver threads for each camera
# 6. Start the synchronize_streams thread
# 7. Start the process_synced_data thread

Custom YOLO Model Integration

1. Model Preparation: Convert your custom-trained YOLO model to the .blob format using Luxonis' online converter or the OpenVINO toolkit if you intend to run inference on the OAK camera's VPU. If running on the host, ensure the model (.pt, ONNX) and necessary inference libraries (PyTorch, ONNX Runtime) are available.
2. Pipeline Integration (Edge): If using .blob, specify the path in the NeuralNetwork node configuration within the DepthAI pipeline script (as shown in the create_pipeline example). Link the ColorCamera's preview output to the NeuralNetwork node's input. Link the NeuralNetwork node's output to the Sync node.
3. Host Inference Integration: If running on the host, the synchronized packets retrieved from the cameras will contain the raw RGB frames. Your host-side Python script will: a. Extract the RGB frame for each camera from the synchronized group. b. Preprocess the frame (resize, normalize) as required by your YOLO model. c. Feed the preprocessed frame into your loaded YOLO model (PyTorch/ONNX Runtime). d. Post-process the model's output to get bounding boxes, class labels, and confidence scores.
4. Processing Results: Whether inference happens on edge or host, the results (detections) associated with each camera's view need to be parsed and utilized.

Integrating YOLO Detections with Depth Data

Once you have synchronized packets containing both YOLO detections and corresponding depth maps, you can add spatial context:

1. Retrieve Data: From a synchronized packet for a given camera, extract the YOLO detection results (list of bounding boxes [x_min, y_min, x_max, y_max], class IDs, confidence scores) and the depth map.
2. Map Detection to Depth: For each detected bounding box, determine the relevant area or central point within the depth map. Note that the YOLO detection coordinates might be relative to the input size of the neural network (e.g., 640x640), while the depth map might have a different resolution (e.g., 640x400). You'll need to scale/map the coordinates appropriately.
3. Extract Depth Value(s): Query the depth map at the point(s) corresponding to the detection (e.g., the center of the bounding box). The value retrieved represents the distance (usually in millimeters) from the camera's optical center to that point in the scene.
4. 3D Localization: Using the depth value(s) and the camera's intrinsic calibration parameters, you can project the 2D detection coordinates into the camera's 3D coordinate system.
5. Multi-View Fusion (Advanced): If the same object/defect is detected by multiple cameras in the synchronized group, you can fuse the 3D position estimates from each view (using extrinsic calibration) to obtain a more robust and accurate 3D location and potentially estimate its size.
6. Application: This 3D information is valuable for measuring defect dimensions (size, depth), pinpointing exact locations on the inspected surface, and filtering detections based on distance.

---

System Performance and Configuration Trade-offs

Designing an effective inspection tunnel involves balancing various factors. The radar chart below visualizes hypothetical trade-offs between different system configurations or design choices. For instance, increasing the number of cameras enhances coverage but also increases cost, complexity, and processing load. Running AI on the edge (OAK VPU) reduces host load but might limit model complexity compared to a powerful host GPU.

Optimization Considerations

• Inference Location: Decide whether to run YOLO on the OAK VPU (lower host load, potentially limited model size/speed) or on the host CPU/GPU (higher host load, more model flexibility).
• Frame Rate vs. Resolution: Balance the need for high resolution (detail) with high frame rate (capturing motion) and the resulting data throughput and processing demands.
• Synchronization Tolerance: A smaller tolerance improves temporal accuracy but increases the chance of dropping packets if network/processing latency varies. A larger tolerance is more robust but less precise.
• Data Throughput: High resolutions and frame rates from multiple cameras generate significant data. Ensure USB/network bandwidth and host processing capabilities are sufficient. Consider strategies like Region of Interest (ROI) processing if only specific areas need analysis.
• Parallel Processing: Utilize multi-threading or asynchronous programming (asyncio) on the host to handle data reception, synchronization, and processing concurrently.

---

Implementation Plan and Workflow

Suggested Software Modules

A modular approach facilitates development and maintenance:

• `config.py`: Stores system configurations (camera IDs, model paths, sync tolerance, resolution, FPS).
• `camera_manager.py`: Handles discovery, initialization, and management of multiple OAK devices. Retrieves device info and IDs.
• `pipeline_creator.py`: Defines the function(s) to create the DepthAI pipeline for a single OAK camera (including ColorCamera, StereoDepth, NN, Sync nodes).
• `synchronizer.py`: Implements the core multi-camera software synchronization logic (buffering, timestamp matching, grouping).
• `inference_engine.py`: Handles loading and running the YOLO model (either interacting with NN results from OAK or running inference on host).
• `data_processor.py`: Takes synchronized data groups, runs inference (if on host), associates depth, aggregates results.
• `output_handler.py`: Manages saving results (images, videos, JSON reports) or transmitting them.
• `main.py`: The main script orchestrating the entire system: initializes modules, starts camera streams, manages threads/processes, and handles graceful shutdown.

Development Steps

1. Environment Setup: Install Python, DepthAI SDK, OpenCV, NumPy, and YOLO-related libraries. 2. Single Camera Test: Connect one OAK camera. Write a script to initialize it, create a basic pipeline (e.g., ColorCamera + StereoDepth), and retrieve/display streams. 3. Single Camera YOLO: Integrate the custom YOLO model into the single-camera pipeline (either edge `.blob` or host inference) and verify detection results. 4. Intra-Camera Sync: Add the `Sync` node to the single-camera pipeline to synchronize color, depth, and YOLO results. Verify timestamp alignment in the output packets. 5. Multi-Camera Management: Develop `camera_manager.py` to detect and initialize all connected OAK cameras. 6. Pipeline Instantiation: Adapt `pipeline_creator.py` to create and start the defined pipeline on each connected camera instance. 7. Inter-Camera Synchronization: Implement the core logic in `synchronizer.py` using timestamp matching and buffering. Test rigorously with data streams from multiple cameras. 8. Data Processing Integration: Develop `data_processor.py` to consume the synchronized data groups, perform necessary processing (including host inference if applicable), and associate depth with detections. 9. Output/Storage: Implement `output_handler.py` to save or display the inspection results. 10. Orchestration: Build `main.py` to tie all modules together, manage execution flow (threading/asyncio), and handle configurations from `config.py`. 11. Calibration: Perform intrinsic and extrinsic camera calibration for accurate measurements and multi-view geometry. 12. Testing & Refinement: Test the complete system in a simulated or real tunnel environment, tune parameters (sync tolerance, YOLO confidence), and optimize performance.

---

Relevant Tunnel Inspection Example

Automated inspection systems are increasingly used in infrastructure maintenance. While this project focuses on Luxonis cameras and YOLO, understanding existing approaches provides context. The following video shows a robotic system for tunnel inspection, illustrating the type of application where this technology can be applied.

---

Error Handling and System Robustness

A production system requires robust error handling:

• Camera Disconnection: Detect camera disconnections and attempt automatic reconnection or notify the operator.
• Pipeline Errors: Catch exceptions during pipeline creation or runtime on the OAK device.
• Synchronization Failures: Monitor buffer sizes and sync success rates. Implement strategies to handle persistent sync failures (e.g., dropping old frames, alerting operator, temporary single-camera fallback).
• Inference Issues: Handle errors during model loading or inference (e.g., timeout, invalid input).
• Logging: Implement comprehensive logging for events, warnings, and errors to aid debugging and monitoring.

Common Issues and Recovery Strategies

The following table outlines potential issues and suggested recovery actions:

Error Type	Detection Method	Recovery Action
Camera Disconnect	DepthAI `isDeviceFound()` or communication timeout	Log error, attempt periodic re-initialization, notify user.
Pipeline Crash (OAK)	Exception handling around `device.getOutputQueue()` calls or watchdog timer	Log error, attempt to close and reopen the device/pipeline, notify user.
Synchronization Drift (> Tolerance)	Monitoring timestamp differences in `synchronizer.py`	Log warning, discard excessively old/unmatchable packets from buffers, potentially adjust tolerance dynamically (with caution).
Buffer Overflow (Host)	Checking `deque.maxlen` usage in `synchronizer.py`	Indicates processing bottleneck. Log error, potentially skip frames/groups, investigate host performance or optimize processing logic.
YOLO Inference Timeout/Error (Host)	Try-except blocks around model inference calls	Log error, skip inference for that frame/group, potentially reload model or check input data.
Inconsistent Timestamps	Observing erratic timestamp jumps	Check system clock synchronization (if using multiple hosts), investigate hardware/driver issues.

---

Frequently Asked Questions (FAQ)

---

Recommended Next Steps & Further Reading

• Explore detailed guides on performing multi-camera extrinsic calibration using OpenCV with Luxonis cameras.
• Investigate techniques for optimizing DepthAI pipeline performance when streaming from multiple cameras simultaneously.
• Learn about advanced methods for fusing YOLO detections and depth information from multiple camera perspectives for robust 3D analysis.
• Consider if integrating with ROS (Robot Operating System) could simplify the management and communication in a complex multi-camera system.

---

References