LangCache Observable

⚠️ Work in Progress This project is currently a work in progress and is intended for temporary debugging and metrics visibility rather than long-term production use. We welcome feedback and improvements and will work with the core LangCache team to find a robust way to continue providing these observability metrics (and soon-to-come OpenTelemetry metrics) from the LangCache client going forward.

A Python wrapper for LangCache that adds observability through Redis-based metrics tracking.

Features

• Operation tracking (attempted, successful, failed)
• Cache hit/miss metrics
• Latency percentiles (p50, p95, p99) using Redis T-Digest
• Optional error logging to Redis streams
• Time-series metrics organized by hour
• Full async/await support

Installation

pip install git+https://github.com/redis-applied-ai/langcache-observable-wrapper.git

Requirements

• Python >= 3.9
• Redis 8+ with Redis Bloom module (included by default)
• LangCache account with API credentials

Quick Start

Set Up Redis

docker run -d -p 6379:6379 redis:8

Basic Usage

import asyncio
import httpx
from redis.asyncio import Redis
from langcache import LangCache
from langcache_observable import ObservableLangCache

async def main():
    redis_client = Redis.from_url("redis://localhost:6379", decode_responses=False)

    async with httpx.AsyncClient() as httpx_client:
        langcache = LangCache(
            server_url="https://api.langcache.com",
            cache_id="your-cache-id",
            api_key="your-api-key",
            async_client=httpx_client
        )

        async with langcache as lc:
            observable_cache = ObservableLangCache(
                langcache=lc,
                redis_client=redis_client,
                cache_id="your-cache-id",
                metrics_ttl_seconds=21600,  # 6 hours
                enable_error_logs=True
            )

            # Use it just like LangCache
            await observable_cache.set_async(
                prompt="What is Python?",
                response="Python is a high-level programming language."
            )

            result = await observable_cache.search_async(
                prompt="Tell me about Python"
            )

            # View metrics
            metrics = await observable_cache.get_metrics()
            print(f"Cache hits: {metrics.get('search_async_cache_hit', 0)}")
            print(f"Cache misses: {metrics.get('search_async_cache_miss', 0)}")

    await redis_client.close()

asyncio.run(main())

API Reference

Constructor

ObservableLangCache(
    langcache: LangCache,
    redis_client: Redis,
    cache_id: str,
    metrics_ttl_seconds: int = 21600,  # 6 hours default
    enable_error_logs: bool = False,
    max_error_stream_size: int = 500
)

Methods

All methods match the LangCache async API:

• search_async() - Search for cached entries
• set_async() - Add entry to cache
• delete_by_id_async() - Delete by ID
• delete_query_async() - Delete by query
• get_metrics() - Get operation metrics
• get_latency_percentiles() - Get latency percentiles
• get_error_logs() - Get error logs (if enabled)

Metrics

Metrics are stored in Redis with keys organized by hour:

langcache_metrics:{cache_id}:{YYYYMMDDHH}:counters  # Operation counters
langcache_metrics:{cache_id}:{YYYYMMDDHH}:latency   # T-Digest latency tracking
langcache_metrics:{cache_id}:{YYYYMMDDHH}:error_log # Error logs (if enabled)

For each operation, the following counters are tracked:

• {operation}_attempted - Total attempts
• {operation}_success - Successful operations
• {operation}_failure - Failed operations
• {operation}_cache_hit - Cache hits (search_async only)
• {operation}_cache_miss - Cache misses (search_async only)

Example:

metrics = await observable_cache.get_metrics()
# {
#     'search_async_attempted': 100,
#     'search_async_success': 98,
#     'search_async_cache_hit': 75,
#     'search_async_cache_miss': 23,
#     ...
# }

Troubleshooting

T-Digest Not Available

Ensure you're using Redis 8+ (includes Redis Bloom module by default):

docker run -d -p 6379:6379 redis:8

Metrics Not Appearing

1. Verify Redis connection: await redis_client.ping()
2. Ensure operations are awaited: await observable_cache.search_async(...)
3. Check TTL hasn't expired (default: 6 hours)

License

MIT License - see LICENSE file for details.

Contributing

Contributions welcome! Please open an issue first to discuss major changes.

• Issues: GitHub Issues