Troubleshooting Guide - rapida.ai documentation

Installation Failed

Problem: Installation script or make command fails Solutions:

Check prerequisites are installed:

go version
docker --version
docker compose version
psql --version
redis-cli --version

Ensure proper permissions:

# macOS/Linux
mkdir -p ~/rapida-data/assets
chmod -R 755 ~/rapida-data

# Linux with Docker group
sudo usermod -aG docker $USER
newgrp docker

Run setup command:
```
make setup-local
```

Database Connection Failed

Problem: Services can’t connect to PostgreSQL Symptoms:

Error messages mentioning connection refused
Services failing to start
Logs showing database errors

Solutions:

Verify PostgreSQL is running:

# macOS
brew services list | grep postgres

# Linux
sudo systemctl status postgresql

# Or check directly
psql -h localhost -U rapida_user -d web_db -c "SELECT 1;"

Check database credentials:

# Verify user exists
psql postgres -c "\du"

# Verify databases exist
psql postgres -c "\l"

Check connection settings in .env:

# Docker
grep POSTGRES docker/web-api/.web.env

# Manual setup
grep POSTGRES env/.web.env

Restart PostgreSQL:

# macOS
brew services restart postgresql

# Linux
sudo systemctl restart postgresql

# Docker
make down-db
make up-db

Check PostgreSQL logs:

# Docker
docker logs postgres

# macOS
tail -f /usr/local/var/log/postgres.log

# Linux
sudo tail -f /var/log/postgresql/postgresql.log

Redis Connection Issues

Problem: Redis connection timeout or refused Symptoms:

Cache operations failing
Session errors
Redis connection refused messages

Solutions:

Verify Redis is running:
```
redis-cli ping
# Should return: PONG
```

Start Redis:

# macOS
brew services start redis

# Linux
sudo systemctl start redis-server

# Docker
make up-redis

Check Redis logs:

# Docker
docker logs redis

# Linux
sudo tail -f /var/log/redis/redis-server.log

Test Redis connection:

redis-cli -h localhost -p 6379
> ping
PONG
> exit

OpenSearch Won’t Start

Problem: OpenSearch container crashes or won’t connect Symptoms:

Container immediately exits
curl: (7) Failed to connect to localhost port 9200
Out of memory errors

Solutions:

Check resource limits:

# OpenSearch requires 512MB minimum
# Increase Docker memory allocation
# Docker Desktop -> Preferences -> Resources -> Memory: 8GB

# Or reduce allocation in docker-compose.yml:
OPENSEARCH_JAVA_OPTS: "-Xms256m -Xmx256m"

Check logs:
```
docker logs opensearch
```

Reset OpenSearch:

# Docker
make down-opensearch
docker volume rm $(docker volume ls | grep opensearch | awk '{print $2}')
make up-opensearch

# Or manually
rm -rf ~/rapida-data/assets/opensearch/*
make up-opensearch

Verify health:

# Check cluster health
curl -s http://localhost:9200/_cluster/health | jq .

# Expected: "status": "green" or "yellow"

Service Issues

Service Won’t Start

Problem: Service container exits immediately with error Symptoms:

Container exits after a few seconds
Status shows “Exited (1)”
Error logs in docker logs

Solutions:

Check service logs:

# Docker
docker logs web-api

# Manual setup
# Check terminal where service was running

Check port availability:

# Find process using port
lsof -i :9001

# Kill process or change port
kill -9 <PID>
# Or edit docker-compose.yml/config

Verify database is ready:

# PostgreSQL must be running first
psql -h localhost -U rapida_user -d web_db -c "SELECT 1;"

# If not ready, wait and retry
make down-web
sleep 10
make up-web

Check configuration:

# Verify environment variables
docker exec web-api env | grep POSTGRES

# Check config syntax
cat docker/web-api/.web.env

Services Crashing After Startup

Problem: Services start but crash after a few minutes Symptoms:

Services in “Exited” state
Intermittent crashes
Error logs showing timeouts or connection issues

Solutions:

Check resource constraints:

# Monitor resource usage
docker stats

# If 100% CPU or memory, increase limits or optimize queries

Check database connections:

# Too many connections?
psql -h localhost -U rapida_user -d web_db
SELECT count(*) FROM pg_stat_activity;

# Increase max connections
ALTER SYSTEM SET max_connections = 200;

Review logs for patterns:

docker logs web-api --tail=100 -f

# Look for patterns in error messages

High Memory Usage

Problem: Services consuming excessive memory Solutions:

Check which service is consuming memory:
```
docker stats
```

For OpenSearch (common culprit):

# Reduce memory allocation
# Edit docker-compose.yml
OPENSEARCH_JAVA_OPTS: "-Xms256m -Xmx256m"

# Restart
docker compose restart opensearch

For PostgreSQL:

# Check for slow queries
psql -h localhost -U rapida_user -d web_db
SELECT query, calls, total_time FROM pg_stat_statements
ORDER BY total_time DESC LIMIT 10;

General troubleshooting:

# Restart affected service
docker compose restart web-api

# Clean up unused data
docker system prune -a

Network Issues

Cannot Access Services

Problem: Services are running but not responding to requests Symptoms:

curl: (7) Failed to connect
Connection refused
Services ping but don’t respond

Solutions:

Verify services are running:

docker ps | grep rapida
# All services should show "Up"

Check port bindings:

# Verify ports are mapped
docker port web-api
# Should show: 9001/tcp -> 0.0.0.0:9001

Test network connectivity:

# From host
curl http://localhost:9001/readiness/

# If fails, check if service is responsive
docker exec web-api curl http://localhost:9001/readiness/

Check firewall:

# macOS
sudo lsof -i :9001

# Linux
sudo netstat -tlnp | grep 9001

Verify DNS resolution:

# Check service name resolution in Docker
docker exec web-api ping postgres
docker exec web-api ping redis
docker exec web-api ping opensearch

Services Can’t Communicate

Problem: Services running but can’t reach each other Symptoms:

Service logs showing “host not found”
Connection timeouts between services
Integration failures

Solutions:

Verify network:

# Check if services on same network
docker network ls
docker network inspect api-network

Test service-to-service connectivity:

# From web-api container
docker exec web-api curl http://assistant-api:9007/readiness/

# Check DNS resolution
docker exec web-api nslookup assistant-api

Check Docker Compose:

# Ensure services are on api-network
grep "networks:" docker-compose.yml
grep "api-network" docker-compose.yml

Recreate network:

docker compose down
docker network rm api-network
docker compose up -d

Data Issues

Lost Data After Restart

Problem: Data disappeared after stopping/restarting containers Solutions:

Verify volumes are properly mounted:

# Check volume mounts
docker inspect postgres | grep -A 5 Mounts

# Should show volume mount to ~/rapida-data/assets/db

Check Docker Compose volume configuration:

# Verify volumes section
grep -A 20 "volumes:" docker-compose.yml | head -20

Ensure data directory exists:

ls -la ~/rapida-data/assets/
# Should show: db/, redis/, opensearch/ directories

Backup and restore data:

# Backup PostgreSQL
docker compose exec postgres pg_dump -U rapida_user web_db > backup.sql

# Restore
docker compose exec -T postgres psql -U rapida_user web_db < backup.sql

Database Corruption

Problem: Database reports corruption or inconsistency Symptoms:

Strange SQL errors
Data appearing corrupted
Ability to read but not write

Solutions:

Check database integrity:

psql -h localhost -U rapida_user -d web_db
SELECT pg_database.datname, pg_database_size(pg_database.datname)
FROM pg_database;

VACUUM and ANALYZE:

psql -h localhost -U rapida_user -d web_db
VACUUM FULL;
ANALYZE;

Rebuild indexes:

psql -h localhost -U rapida_user -d web_db
REINDEX DATABASE web_db;

Full restoration:

# Stop services
make down-all

# Backup current data
cp -r ~/rapida-data/assets/db ~/rapida-data/assets/db.backup

# Reset database
rm -rf ~/rapida-data/assets/db/*

# Start and restore
make up-db
# Wait for initialization
docker compose exec -T postgres psql -U rapida_user web_db < backup.sql

Performance Issues

Slow API Responses

Problem: API endpoints responding slowly Solutions:

Check database performance:

psql -h localhost -U rapida_user -d web_db
SELECT query, calls, total_time FROM pg_stat_statements
ORDER BY total_time DESC LIMIT 10;

Monitor resource usage:

docker stats

# If CPU/memory at limit, need to optimize or increase

Check query plans:

psql -h localhost -U rapida_user -d web_db
EXPLAIN ANALYZE SELECT * FROM conversations WHERE id = '123';

Create missing indexes:

# Common queries to index:
CREATE INDEX idx_conversations_user_id ON conversations(user_id);
CREATE INDEX idx_messages_conversation_id ON messages(conversation_id);

Reduce logging verbosity:

# Change LOG_LEVEL from debug to info
docker compose restart web-api

High Database Connection Usage

Problem: Too many database connections consuming resources Solutions:

Monitor connections:

psql -h localhost -U rapida_user -d web_db
SELECT datname, count(*) FROM pg_stat_activity GROUP BY datname;

Increase connection limit:

POSTGRES__MAX_OPEN_CONNECTION=20
POSTGRES__MAX_IDEAL_CONNECTION=10

Kill idle connections:

psql -h localhost -U rapida_user -d web_db
SELECT pg_terminate_backend(pid) FROM pg_stat_activity
WHERE state = 'idle' AND query_start < now() - interval '1 hour';

Slow Search/Analytics

Problem: OpenSearch searches timing out or slow Solutions:

Check OpenSearch status:

curl http://localhost:9200/_cluster/health

Monitor task queue:

curl http://localhost:9200/_tasks?detailed

Optimize index settings:

# Reduce refresh frequency
curl -X PUT "localhost:9200/assistant-conversations/_settings" \
  -H 'Content-Type: application/json' \
  -d '{"settings": {"refresh_interval": "30s"}}'

Delete old indices:

# Remove indices older than 30 days
curl -X DELETE "localhost:9200/logs-2024-01-*"

Debugging

Enable Debug Logging

# Update .env file
LOG_LEVEL=debug

# Restart service
docker compose restart web-api

# Monitor logs
docker logs -f web-api

Check Service Configuration

# Verify service has correct config
docker exec web-api env | grep POSTGRES
docker exec web-api env | grep REDIS

# Check if service can reach dependencies
docker exec web-api curl http://postgres:5432 2>&1
docker exec web-api curl http://redis:6379 2>&1

View Network Details

# List all networks
docker network ls

# Inspect api-network
docker network inspect api-network

# View network traffic (if netcat installed)
docker exec web-api netcat -zv redis 6379

Health Checks

# Readiness check (service ready to accept traffic)
curl -s http://localhost:9001/readiness/ | jq .

# Liveness check
curl -s http://localhost:9001/healthz/ | jq .

Getting Help

Collect Debug Information

# System info
uname -a
docker --version
docker compose version

# Service status
docker ps -a
docker network ls

# Recent logs
docker logs web-api --tail=50 > web-api.log
docker logs postgres --tail=50 > postgres.log
docker logs redis --tail=50 > redis.log

# Resource usage
docker stats --no-stream > stats.log

# Configuration
cat env/.web.env > config-web.log

Report Issues

When reporting an issue, include:

Steps to reproduce
Error message/logs
System information
What you tried to fix it
Output of debug commands above

Contact Support

GitHub Issues: https://github.com/rapidaai/voice-ai/issues
Documentation: https://rapida.ai/docs
Community: https://github.com/rapidaai/voice-ai/discussions
Email: prashant@rapida.ai

Quick Reference

Common Commands

# Status
make ps-all
docker ps

# Logs
make logs-all
docker logs -f web-api

# Restart services
make restart-all
docker compose restart

# Clean up
make clean
docker compose down -v

# Database
make shell-db
psql -h localhost -U rapida_user -d web_db

# Redis
redis-cli -h localhost -p 6379

# Verify services
curl http://localhost:9001/readiness/

Next Steps

Installation - Installation guide
Configuration - Config-related issues

Documentation Index

​Installation Failed

​Database Connection Failed

​Redis Connection Issues

​OpenSearch Won’t Start

​Service Issues

​Service Won’t Start

​Services Crashing After Startup

​High Memory Usage

​Network Issues

​Cannot Access Services

​Services Can’t Communicate

​Data Issues

​Lost Data After Restart

​Database Corruption

​Performance Issues

​Slow API Responses

​High Database Connection Usage

​Slow Search/Analytics

​Debugging

​Enable Debug Logging

​Check Service Configuration

​View Network Details

​Health Checks

​Getting Help

​Collect Debug Information

​Report Issues

​Contact Support

​Quick Reference

​Common Commands

​Next Steps

Installation Failed

Database Connection Failed

Redis Connection Issues

OpenSearch Won’t Start

Service Issues

Service Won’t Start

Services Crashing After Startup

High Memory Usage

Network Issues

Cannot Access Services

Services Can’t Communicate

Data Issues

Lost Data After Restart

Database Corruption

Performance Issues

Slow API Responses

High Database Connection Usage

Slow Search/Analytics

Debugging

Enable Debug Logging

Check Service Configuration

View Network Details

Health Checks

Getting Help

Collect Debug Information

Report Issues

Contact Support

Quick Reference

Common Commands

Next Steps