8000 Pipeline Auto-Advancing Through Questions Without Waiting for User Input · Issue #1691 · pipecat-ai/pipecat · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Pipeline Auto-Advancing Through Questions Without Waiting for User Input #1691

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
ebfjunior opened this issue Apr 29, 2025 · 3 comments
Closed

Pipeline Auto-Advancing Through Questions Without Waiting for User Input #1691

ebfjunior opened this issue Apr 29, 2025 · 3 comments

Comments

@ebfjunior
Copy link
ebfjunior commented Apr 29, 2025 •
edited
Loading
8000

pipecat version

0.0.65

pipecat-flows version

0.0.16

Python version

3.12.3

Operating System

macOS 14.1.1

Issue description

When running a conversation with Pipecat Flows, the pipeline automatically advances through all questions without waiting for user input. The LLM asks a question, then immediately advances to the next question without giving the user an opportunity to respond.

It's important to note that the LLM is answering its own questions - either by providing made-up answers or by simply repeating the question as the answer. This results in a completely automated conversation where the user cannot participate.

Note: While I'm not 100% sure about it, I think I started noticing this issue a bit after migrating my agent from an on-prem API to Pipecat Cloud.

Pipeline Configuration

The pipeline is configured as follows:

# Daily Transport Configuration
transport = DailyTransport(
    room_url,
    token,
    "Interview Bot",
    DailyParams(
        audio_out_enabled=True,
        video_out_enabled=True,
        camera_out_enabled=True,
        camera_out_width=512,
        camera_out_height=512,
        vad_enabled=True,
        vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.2)),
        vad_audio_passthrough=False,  # Issue persists even when set to True
        transcription_enabled=True,
        turn_analyzer=FalSmartTurnAnalyzer(
            api_key=os.getenv("FAL_API_KEY"), aiohttp_session=aiohttp_session
        ),
    ),
)

# Pipeline Configuration
context = OpenAILLMContext()
context_aggregator = llm.create_context_aggregator(context)
rtvi = RTVIProcessor(config=RTVIConfig(config=[]))

stt_mute_processor = STTMuteFilter(
    config=STTMuteConfig(
        strategies={
            STTMuteStrategy.MUTE_UNTIL_FIRST_BOT_COMPLETE,
            STTMuteStrategy.FUNCTION_CALL,
        }
    ),
)

smart_turn_metrics_processor = SmartTurnMetricsProcessor()

pipeline = Pipeline([
    transport.input(),
    rtvi,
    stt_mute_processor,
    smart_turn_metrics_processor,
    stt,
    context_aggregator.user(),
    llm,
    tts,
    transport.output(),
    context_aggregator.assistant(),
])

Note: Even with vad_audio_passthrough=True in the DailyParams configuration, the issue persists. This suggests the problem is not related to the Voice Activity Detection system failing to pass audio through.

Note 2: If I take the MuteFilter out and then say something like "Are you there?" while the bot is "stuck", the bot is able to say something and start the conversation from where it stopped faking answers (i.e. the middle of the flow config)

Note 3: While this issue is happening ~80% of the time, it's still intermittent, and sometimes I can see the expected behavior happening

Suspected Cause

The issue appears to be with how the flow transitions are configured. The logs show a transition occurring immediately after a question is handled.

These transitions are occurring immediately after the LLM has completed generating its response

The function calls related to question handling may be triggering immediate transitions to the next question node

The system is treating the LLM's outputs as user responses, creating a closed loop

Reproduction steps

  • Set up a Pipecat Flow with multiple question nodes using a static flow
  • Start the conversation
  • Observe the bot stuck without saying anything.
  • Look into the logs and see that the bot is actually asking and answering all questions in sequence without waiting for responses

Expected behavior

After asking a question, the bot should wait for the user to respond before moving to the next question.

Actual behavior

The bot asks a question, then immediately executes a function call to handle the next question, and transitions to the next question in the flow, without giving the user time to respond. The bot is also answering its own questions, resulting in a self-contained conversation loop without user input.

Logs

2025-04-29 12:22:53 2025-04-29 12:22:53.548 | DEBUG    | pipecat.processors.metrics.frame_processor_metrics:start_llm_usage_metrics:73 | 3d536ff7-6763-408f-985d-3bec3d4be351 - OpenAILLMService#0 prompt tokens: 703, completion tokens: 30
2025-04-29 12:22:53 2025-04-29 12:22:53.556 | DEBUG    | pipecat.processors.metrics.frame_processor_metrics:stop_processing_metrics:65 | 3d536ff7-6763-408f-985d-3bec3d4be351 - OpenAILLMService#0 processing time: 1.172114372253418
2025-04-29 12:22:53 2025-04-29 12:22:53.557 | DEBUG    | pipecat.services.llm_service:_run_function_call:176 | 3d536ff7-6763-408f-985d-3bec3d4be351 - OpenAILLMService#0 Calling function [handle_next_question_eb2db3e0:call_J61DmijEDDi4qEdCt3J6ptYH] with arguments {'response': 'Are there any genres you actively avoid?'}
2025-04-29 12:22:53 2025-04-29 12:22:53.557 | DEBUG    | pipecat.processors.aggregators.llm_response:_handle_function_call_in_progress:520 | 3d536ff7-6763-408f-985d-3bec3d4be351 - OpenAIAssistantContextAggregator#0 FunctionCallInProgressFrame: [handle_next_question_eb2db3e0:call_J61DmijEDDi4qEdCt3J6ptYH]
2025-04-29 12:22:53 2025-04-29 12:22:53.557 | DEBUG    | pipecat_flows.manager:transition_func:356 | 3d536ff7-6763-408f-985d-3bec3d4be351 - Function call pending: handle_next_question_eb2db3e0 (total: 1)
2025-04-29 12:22:53 2025-04-29 12:22:53.557 | DEBUG    | handlers.question_handler:handle_response:126 | 3d536ff7-6763-408f-985d-3bec3d4be351 - Handling response for question ce5f1c41-1187-45f9-ba62-784821daae3e
2025-04-29 12:22:53 2025-04-29 12:22:53.557 | DEBUG    | handlers.question_handler:_collect_response:233 | 3d536ff7-6763-408f-985d-3bec3d4be351 - Collecting response: Are there any genres you actively avoid?
2025-04-29 12:22:53 2025-04-29 12:22:53.557 | DEBUG    | handlers.question_handler:handle_response:133 | 3d536ff7-6763-408f-985d-3bec3d4be351 - Returning result for question ce5f1c41-1187-45f9-ba62-784821daae3e to trigger transition
2025-04-29 12:22:53 2025-04-29 12:22:53.557 | DEBUG    | pipecat_flows.manager:transition_func:363 | 3d536ff7-6763-408f-985d-3bec3d4be351 - Handler completed for handle_next_question_eb2db3e0
2025-04-29 12:22:53 2025-04-29 12:22:53.557 | DEBUG    | pipecat.processors.aggregators.llm_response:_handle_function_call_result:527 | 3d536ff7-6763-408f-985d-3bec3d4be351 - OpenAIAssistantContextAggregator#0 FunctionCallResultFrame: [handle_next_question_eb2db3e0:call_J61DmijEDDi4qEdCt3J6ptYH]
2025-04-29 12:22:53 2025-04-29 12:22:53.558 | DEBUG    | handlers.question_handler:_send_response_webhook:342 | 3d536ff7-6763-408f-985d-3bec3d4be351 - >>>>>> Sending response to webhook: {'question_id': 'ce5f1c41-1187-45f9-ba62-784821daae3e', 'question_prompt': 'Ask the candidate if there are any genres they actively avoid.', 'answer': 'Are there any genres you actively avoid?'}
2025-04-29 12:22:53 2025-04-29 12:22:53.558 | DEBUG    | pipecat_flows.manager:decrease_pending_function_calls:298 | 3d536ff7-6763-408f-985d-3bec3d4be351 - Function call completed: handle_next_question_eb2db3e0 (remaining: 0)
2025-04-29 12:22:53 2025-04-29 12:22:53.558 | DEBUG    | pipecat_flows.manager:on_context_updated_edge:312 | 3d536ff7-6763-408f-985d-3bec3d4be351 - Static transition to: question_5
2025-04-29 12:22:53 2025-04-29 12:22:53.558 | DEBUG    | pipecat_flows.manager:set_node:487 | 3d536ff7-6763-408f-985d-3bec3d4be351 - Setting node: question_5
2025-04-29 12:22:53 2025-04-29 12:22:53.558 | DEBUG    | pipecat_flows.manager:_register_function:457 | 3d536ff7-6763-408f-985d-3bec3d4be351 - Registered function: handle_next_question_6a5497bb
2025-04-29 12:22:53 2025-04-29 12:22:53.559 | DEBUG    | pipecat_flows.manager:_update_llm_context:646 | 3d536ff7-6763-408f-985d-3bec3d4be351 - Updated LLM context using LLMMessagesAppendFrame with strategy ContextStrategy.APPEND
2025-04-29 12:22:53 2025-04-29 12:22:53.559 | DEBUG    | pipecat_flows.manager:set_node:558 | 3d536ff7-6763-408f-985d-3bec3d4be351 - Updated LLM context
2025-04-29 12:22:53 2025-04-29 12:22:53.559 | DEBUG    | pipecat_flows.manager:set_node:572 | 3d536ff7-6763-408f-985d-3bec3d4be351 - Successfully set node: question_5
2025-04-29 12:22:53 2025-04-29 12:22:53.559 | DEBUG    | pipecat_flows.manager:on_context_updated_edge:326 | 3d536ff7-6763-408f-985d-3bec3d4be351 - Reset pending function calls counter

The logs clearly show that:

  • The LLM asks a question ("Are there any genres you actively avoid?")
  • The question handler treats this as the user's response: Collecting response: Are there any genres you actively avoid?
  • This triggers an immediate transition to the next question
  • No actual user input is ever processed
@markbackman
Copy link
Contributor

(In the future, please file Flows issues against the Flows repo.)

I've only seen this when the prompting is insufficiently specific. Check out the flows examples for recommendations on how to ensure that the LLM will call exactly one function at a time.

@ebfjunior
Copy link
Author

Thanks @markbackman It was indeed an issue with the function descriptions. It's been fixed now!

@markbackman
Copy link
Contributor
markbackman commented May 1, 2025 •
edited
Loading

Great to hear that fixed the issue!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ebfjunior @markbackman
0