LangCache Router

A Redis-based routing system that provides LangCache client connections based on the active blue-green deployment instance. This package enables seamless switching between different LangCache deployments without downtime.

Features

• Blue-Green Deployment Support: Seamlessly switch between LangCache instances
• Redis-Based Configuration: Centralized configuration management with client-side caching
• Automatic Client Management: Smart caching of LangCache clients with automatic invalidation
• Health Monitoring: Built-in status monitoring and health checks
• CLI Tools: Command-line utilities for deployment management and testing
• Async Support: Full async/await support for modern Python applications

Installation

From Source (GitHub)

# Using uv (recommended)
uv add git+https://github.com/yourusername/langcache-router.git

# Using pip
pip install git+https://github.com/yourusername/langcache-router.git

Development Installation

# Clone the repository
git clone https://github.com/yourusername/langcache-router.git
cd langcache-router

# Install with uv
uv sync --all-extras

Examples

Basic Example

from langcache_router import LangCacheRouter
import asyncio

async def main():
    # Initialize router
    router = LangCacheRouter("redis://localhost:6379")
    
    # Set active instance
    router.set_active_instance(
        cache_url="https://api.langcache.com",
        cache_id="my-cache",
        cache_api_key_env_var="LANGCACHE_API_KEY"
    )
    
    # Use the client
    client = router.get_client()
    result = await client.search_async(prompt="Hello, world!")
    print(f"Cache result: {result}")

asyncio.run(main())

Blue-Green Deployment Example

from langcache_router import LangCacheRouter
import time

def deploy_new_version():
    router = LangCacheRouter("redis://localhost:6379")
    
    # Get current deployment
    status = router.get_status()
    current = status.get('cache_id')
    
    # Switch to the other deployment
    if current == "blue-cache":
        # Switch to green
        router.set_active_instance(
            cache_url="https://green.langcache.com",
            cache_id="green-cache",
            cache_api_key_env_var="GREEN_API_KEY"
        )
        print("Switched to Green deployment")
    else:
        # Switch to blue
        router.set_active_instance(
            cache_url="https://blue.langcache.com", 
            cache_id="blue-cache",
            cache_api_key_env_var="BLUE_API_KEY"
        )
        print("Switched to Blue deployment")

Scripts

The scripts/ directory contains utility scripts for managing and testing LangCache deployments:

query_loop.py

Continuously queries LangCache with controlled concurrency until stopped, cycling through fixed sample queries.

uv run python scripts/query_loop.py --max-concurrency 10

load_entries.py

Loads a specified number of unique entries to the inactive LangCache deployment for blue-green testing.

uv run python scripts/load_entries.py 100 --max-concurrency 24

toggle_deployment.py

Switches between blue and green LangCache deployments.

uv run python scripts/toggle_deployment.py

Environment Variables

Create a .env file in your project root:

# Redis Configuration
REDIS_URL=redis://localhost:6379

# Blue Instance
LANGCACHE_URL_BLUE=https://blue.langcache.com
LANGCACHE_CACHE_ID_BLUE=blue-cache-id
LANGCACHE_API_KEY_BLUE=your-blue-api-key

# Green Instance  
LANGCACHE_URL_GREEN=https://green.langcache.com
LANGCACHE_CACHE_ID_GREEN=green-cache-id
LANGCACHE_API_KEY_GREEN=your-green-api-key

Testing

Run the test suite:

# Using pytest
uv run pytest

Test Requirements

• Redis server running on localhost:6379 (or configure REDIS_URL)

Documentation

Architecture

The LangCache Router uses Redis as a centralized configuration store to manage blue-green deployments:

1. Configuration Storage: Active instance configuration is stored in Redis
2. Client Caching: LangCache clients are cached and automatically invalidated when configuration changes
3. Health Monitoring: Built-in status checks and error handling
4. Atomic Switching: Configuration updates are atomic, ensuring consistency

Performance Considerations

• Client Caching: Clients are cached to avoid repeated initialization overhead
• Redis Client-Side Caching: Reduces latency for configuration reads from Redis
• Automatic Invalidation: Cached clients are invalidated when configuration changes
• Connection Pooling: Redis connections are pooled for efficiency

Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Make your changes
4. Add tests for new functionality
5. Run the test suite (pytest)
6. Commit your changes (git commit -m 'Add amazing feature')
7. Push to the branch (git push origin feature/amazing-feature)
8. Open a Pull Request

Development Setup

# Clone and setup
git clone https://github.com/yourusername/langcache-router.git
cd langcache-router

# Install development dependencies
uv sync --all-extras

# Run tests
uv run pytest

# Run linting
uv run ruff check .
uv run black --check .
uv run mypy src/

License

This project is licensed under the MIT License - see the LICENSE file for details.