FrameSource 📷🖼️

FrameSource is a flexible, extensible Python framework for acquiring frames from a wide variety of sources such as webcams, industrial cameras, IP cameras, video files, and even folders of images—using a unified interface. It was created to support my many projects that require switching between different frame providers without changing the downstream frame processing code.

Note: This project was mostly written with the help of GitHub Copilot 🤖, making development fast, fun, and consistent! Even this README was largely generated by Copilot.

Supported Sources

Camera Sources

• 🖥️ Webcam (OpenCV) - Standard USB webcams, built-in laptop cameras
• 🌐 IP Camera (RTSP/HTTP) - Network cameras, security cameras
• 🏭 Industrial Cameras:
  • Basler cameras (via pypylon SDK) - High-performance industrial imaging
  • Ximea cameras - Scientific and machine vision cameras
  • Huateng cameras - Cost-effective industrial cameras
• 🔍 Intel RealSense - RGB-D cameras with depth sensing (tested with D456)

Media Sources

• 🎥 Video File (MP4, AVI, etc.) - Playback with looping and controls
• 🗂️ Folder of Images - Sorted by name or creation time with configurable FPS
• 🖼️ Screen Capture - Live region capture from desktop
• 🎵 Audio Spectrogram - Real-time audio visualization from microphone or files

Demo

Interactive 360° Camera Demo

The 360° camera example (examples/camera_360_example.py) provides an intuitive interface for exploring equirectangular footage:

• Click & Drag: Click anywhere on the 360° image and drag to smoothly pan the view
• Mouse Wheel: Scroll to zoom in/out by adjusting the field of view
• Keyboard Controls: Fine-tune pitch, yaw, roll, and FOV with precise keyboard shortcuts
• Real-time Processing: Live conversion from equirectangular to pinhole projection

Why FrameSource?

When I work on computer vision, robotics, or video analytics projects, I often need to swap between different sources of frames: a webcam for quick tests, a folder of images for batch processing, a video file for reproducibility, or a specialized camera for deployment. FrameSource lets me do this with minimal code changes—just swap the provider!

Features ✨

• Unified interface for all frame sources (cameras, video files, image folders, screen capture, audio spectrograms)
• Built-in frame processors for specialized transformations (360° equirectangular to pinhole projection)
• Easily extensible with new capture types and processing modules
• Threaded/background capture support for smooth frame acquisition
• Control over exposure, gain, resolution, FPS (where supported by the source)
• Real-time playback and looping for video and image folders
• Simple factory pattern for instantiating sources

Installation

Install Directly from GitHub

You can install FrameSource directly from GitHub without cloning:

# Latest version from main branch
pip install git+https://github.com/olkham/FrameSource.git

# Specific branch
pip install git+https://github.com/olkham/FrameSource.git@branch-name

# Specific tag or commit
pip install git+https://github.com/olkham/FrameSource.git@v1.0.0

Install from Local Clone

Clone the repository and install with pip:

git clone https://github.com/olkham/FrameSource.git
cd framesource
pip install .

Or for development (editable) install:

pip install -e .

Installation Options

FrameSource supports optional dependencies for additional features:

From GitHub:

# Basic installation (core frame sources only)
pip install git+https://github.com/olkham/FrameSource.git

# With audio spectrogram support
pip install "git+https://github.com/olkham/FrameSource.git#egg=framesource[audio]"

# With Basler camera support
pip install "git+https://github.com/olkham/FrameSource.git#egg=framesource[basler]"

# With RealSense camera support
pip install "git+https://github.com/olkham/FrameSource.git#egg=framesource[realsense]"

# With all optional features
pip install "git+https://github.com/olkham/FrameSource.git#egg=framesource[full]"

# Multiple extras at once
pip install "git+https://github.com/olkham/FrameSource.git#egg=framesource[audio,basler,realsense]"

From local installation:

# Basic installation (core frame sources only)
pip install .

# With audio spectrogram support
pip install .[audio]

# With Basler camera support
pip install .[basler]

# With RealSense camera support
pip install .[realsense]

# With all optional features
pip install .[full]

# Multiple extras at once
pip install .[audio,basler,realsense]

Manual Dependency Installation

Alternatively, you can install dependencies manually:

# Audio processing
pip install librosa soundfile pyaudio

# Basler cameras
pip install pypylon

# RealSense cameras
pip install pyrealsense2

Example Usage

💡 Tip: For comprehensive examples of each capture type, see the examples/ directory. Run python examples/run_examples.py for an interactive demo menu.

1. Using the Factory

from frame_source import FrameSourceFactory

# Webcam
cap = FrameSourceFactory.create('webcam', source=0)
cap.connect()
ret, frame = cap.read()
cap.disconnect()


# Video file (see demo in media/demo.mp4)
cap = FrameSourceFactory.create('video_file', source='media/demo.mp4', loop=True)
cap.connect()
while cap.is_connected:
    ret, frame = cap.read()
    if not ret:
        break
cap.disconnect()

# Folder of images
cap = FrameSourceFactory.create('folder', source='media/image_seq', sort_by='date', fps=10, loop=True)
cap.connect()
cap.start_async()  # For background capture
while cap.is_connected:
    ret, frame = cap.read()
    if not ret:
        break
cap.disconnect()

2. Direct Use

Intel RealSense Camera

from frame_source.realsense_capture import RealsenseCapture
from frame_processors import RealsenseDepthProcessor
from frame_processors.realsense_depth_processor import RealsenseProcessingOutput

# Tested with Intel RealSense D456 camera
cap = RealsenseCapture(width=640, height=480)
processor = RealsenseDepthProcessor(output_format=RealsenseProcessingOutput.ALIGNED_SIDE_BY_SIDE)
cap.attach_processor(processor)
cap.connect()
while cap.is_connected:
    ret, frame = cap.read()
    if not ret:
        break
    # Frame contains RGB and depth side-by-side or other configured format
cap.disconnect()

Folder of Images

from frame_source.folder_capture import FolderCapture
cap = FolderCapture('media/image_seq', sort_by='name', width=640, height=480, fps=15, real_time=True, loop=True)
cap.connect()
while cap.is_connected:
    ret, frame = cap.read()
    if not ret:
        break
cap.disconnect()

Screen Capture

from frame_source.screen_capture import ScreenCapture
cap = ScreenCapture(x=100, y=100, w=800, h=600, fps=30)
cap.connect()
cap.start_async()  # For background capture (optional)
while cap.is_connected:
    ret, frame = cap.read()
    if not ret:
        break
    # process or display frame
cap.disconnect()

Audio Spectrogram Capture

# Audio spectrogram from microphone (real-time)
cap = FrameSourceFactory.create('audio_spectrogram', 
                              source=None,  # None = default microphone
                              n_mels=128, 
                              window_duration=2.0,
                              freq_range=(20, 8000),
                              colormap=cv2.COLORMAP_VIRIDIS)
cap.connect()
cap.start_async()  # Start background audio processing
while cap.is_connected:
    ret, frame = cap.read()
    if not ret:
        break
    # frame is now a visual spectrogram that can be processed like any other image
cap.disconnect()

# Audio spectrogram from file
cap = FrameSourceFactory.create('audio_spectrogram', 
                              source='path/to/audio.wav',
                              n_mels=64,
                              frame_rate=30)
cap.connect()
while cap.is_connected:
    ret, frame = cap.read()
    if not ret:
        break
cap.disconnect()

Frame Processors 🔄

FrameSource includes powerful frame processors for specialized transformations:

Equirectangular 360° to Pinhole Projection

Convert 360° equirectangular footage to normal pinhole camera views with interactive controls:

from frame_source import FrameSourceFactory
from frame_processors.equirectangular360_processor import Equirectangular2PinholeProcessor

# Load 360° video or connect to 360° webcam
cap = FrameSourceFactory.create('video_file', source='360_video.mp4')
# Or for live 360° camera: cap = FrameSourceFactory.create('webcam', source=0)
cap.connect()

# Create processor for 90° FOV pinhole view
processor = Equirectangular2PinholeProcessor(fov=90.0, output_width=1920, output_height=1080)

# Set viewing angles (in degrees)
processor.set_parameter('yaw', 45.0)    # Look right
processor.set_parameter('pitch', 0.0)   # Look straight ahead
processor.set_parameter('roll', 0.0)    # No rotation

# Attach processor to the frame source for automatic processing
cap.attach_processor(processor)

while cap.is_connected:
    ret, frame = cap.read()  # Frame is automatically processed by attached processor
    if not ret:
        break
    
    # The frame is now the processed pinhole projection
    cv2.imshow('360° to Pinhole', frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.disconnect()

💡 Interactive Demo: Try python examples/camera_360_example.py for a fully interactive 360° viewer with mouse controls! Click and drag on the equirectangular image to look around, use the mouse wheel to zoom, and keyboard shortcuts for fine adjustments.

You can also manually process frames without attaching:

# Manual processing (without attach)
while cap.is_connected:
    ret, frame = cap.read()
    if not ret:
        break
    
    # Manually process the frame
    pinhole_frame = processor.process(frame)
    cv2.imshow('360° to Pinhole', pinhole_frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

Creating Custom Frame Processors

Extend the FrameProcessor base class for your own transformations:

from frame_processors.frame_processor import FrameProcessor
import cv2

class GrayscaleProcessor(FrameProcessor):
    def process(self, frame):
        return cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

# Use your custom processor
processor = GrayscaleProcessor()

# Option 1: Attach to frame source for automatic processing
cap = FrameSourceFactory.create('webcam', source=0)
cap.connect()
cap.attach_processor(processor)

while cap.is_connected:
    ret, frame = cap.read()  # Frame is automatically converted to grayscale
    if not ret:
        break
    cv2.imshow('Grayscale', frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

# Option 2: Manual processing
processed_frame = processor.process(original_frame)

Extending FrameSource

Adding New Frame Sources

Want to add a new camera or source? Just subclass VideoCaptureBase and register it:

from frame_source import FrameSourceFactory
FrameSourceFactory.register_capture_type('my_camera', MyCameraCapture)

Adding New Frame Processors

Create custom frame processors by extending FrameProcessor:

from frame_processors.frame_processor import FrameProcessor

class MyCustomProcessor(FrameProcessor):
    def __init__(self, custom_param=1.0):
        super().__init__()
        self.set_parameter('custom_param', custom_param)
    
    def process(self, frame):
        # Your custom processing logic here
        custom_param = self.get_parameter('custom_param')
        # ... apply transformation ...
        return processed_frame

Credits

• Written by me, with lots of help from GitHub Copilot 🤖
• OpenCV and other camera SDKs for backend support
• public RTSP URL from grigory-lobkov/rtsp-camera-view#3

---

Happy frame grabbing! 🚀