How to Fix 'memory not persisting when scaling' in CrewAI (Python)

When CrewAI memory stops persisting after you scale workers, it usually means each process is writing to its own local state instead of a shared backend. You’ll see this when a single-run agent remembers context, but the same workflow loses history once you run multiple processes, containers, or replicas.

The common failure mode is simple: memory is stored in-process, then your app scales horizontally and every replica gets a different view of the world. In practice that shows up as memory not persisting when scaling, missing conversation history, or agents acting like they’ve never seen prior tasks.

The Most Common Cause

The #1 cause is using ephemeral or process-local memory with CrewAI. If you instantiate Crew, Agent, or any memory-backed component inside each worker without a shared persistence layer, the data disappears as soon as that process exits or another replica handles the next request.

Here’s the broken pattern versus the fixed pattern:

# BROKEN: memory lives only inside this Python process
from crewai import Agent, Task, Crew

researcher = Agent(
    role="Researcher",
    goal="Find customer context",
    memory=True,  # looks fine, but still may be process-local depending on setup
)

task = Task(
    description="Summarize previous customer issues",
    agent=researcher,
)

crew = Crew(
    agents=[researcher],
    tasks=[task],
)

result = crew.kickoff()

# FIXED: use an explicit shared persistence layer for state
from crewai import Agent, Task, Crew
from langchain_community.chat_message_histories import RedisChatMessageHistory

history = RedisChatMessageHistory(
    url="redis://redis:6379/0",
    session_id="customer-123"
)

researcher = Agent(
    role="Researcher",
    goal="Find customer context",
    memory=True,
)

task = Task(
    description="Summarize previous customer issues",
    agent=researcher,
)

crew = Crew(
    agents=[researcher],
    tasks=[task],
)

# Your app code should attach shared history/state outside the worker lifecycle.
result = crew.kickoff()

The exact API varies by CrewAI version and how you wire memory into your app stack. The point is non-negotiable: if scaling introduces multiple processes or containers, memory must live outside the worker.

Other Possible Causes

1) You’re using Docker/Kubernetes without shared storage

If each container writes to its own disk, you’ll lose memory on the next pod.

# BAD: no persistent volume for app state
spec:
  containers:
    - name: crewai-app
      image: myapp:latest

# GOOD: mount shared or persistent storage
spec:
  containers:
    - name: crewai-app
      image: myapp:latest
      volumeMounts:
        - name: app-state
          mountPath: /data
  volumes:
    - name: app-state
      persistentVolumeClaim:
        claimName: crewai-state-pvc

2) Your task execution is stateless by design

If you rebuild Crew() on every request and don’t pass conversation/session identifiers through the pipeline, CrewAI has nothing to bind memory to.

# BAD
def handle_request(payload):
    crew = build_crew()   # new instance every request
    return crew.kickoff()

# GOOD
def handle_request(payload):
    session_id = payload["session_id"]
    crew = build_crew(session_id=session_id)
    return crew.kickoff()

3) You are mixing threads/processes with non-thread-safe storage

A local dict cache or file-based JSON store will break under Gunicorn workers, Celery workers, or multiprocessing.

# BAD: unsafe in multi-worker deployments
MEMORY_CACHE = {}

def save_memory(session_id, data):
    MEMORY_CACHE[session_id] = data

Use Redis/Postgres instead:

# GOOD: centralized store survives worker restarts/scaling
import redis

r = redis.Redis(host="redis", port=6379, decode_responses=True)

def save_memory(session_id, data):
    r.set(f"memory:{session_id}", data)

4) You have a version mismatch between CrewAI and your memory provider

A lot of “it works locally but not in prod” cases come from mismatched package versions. You may see errors like:

• •AttributeError: 'Crew' object has no attribute 'memory'
• •TypeError: __init__() got an unexpected keyword argument 'memory'
• •ModuleNotFoundError for provider integrations

Pin versions explicitly:

crewai==0.80.0
langchain-community==0.2.12
redis==5.0.8

How to Debug It

1. •

   Check whether the same session ID is used everywhere

   • •Log it at request entry.
   • •If it changes between calls, your “memory” is really just new state.
2. •

   Verify where memory is stored

   • •Search for dict, local JSON files, SQLite files inside containers, or per-process globals.
   • •If storage disappears when a pod restarts, it’s not persistent enough.
3. •

   Run two replicas and compare behavior

   • •Send one request to replica A and another to replica B.
   • •If B cannot read A’s history, your backend is local to the worker.
4. •

   Inspect startup logs for provider initialization

   • •Look for failed Redis/Postgres connections.
   • •Common symptoms include connection timeouts followed by silent fallback to in-memory defaults.

Prevention

• •

  Use a shared backend for all conversational state from day one.

  • •Redis for short-lived session memory.
  • •Postgres for durable audit/history use cases.
• •

  Treat Crew() instances as disposable compute objects.

  • •Persist state outside the process.
  • •Rehydrate by session ID on every request.
• •

  Add an integration test that runs across two workers.

  • •If memory survives only in one worker, fail the build before deployment.

If you’re seeing memory not persisting when scaling in CrewAI Python apps, assume the problem is architecture first and library behavior second. In almost every production case I’ve debugged, the fix was moving memory out of process-local storage and into a real shared persistence layer.

Keep learning

• •The complete AI Agents Roadmap — my full 8-step breakdown
• •Free: The AI Agent Starter Kit — PDF checklist + starter code
• •Work with me — I build AI for banks and insurance companies

By Cyprian Aarons, AI Consultant at Topiax.