Author(s): Neel Shah

Originally published on Towards AI.

As an AI engineer, designing systems that interact with Large Language Models (LLMs) like Google’s Gemini is a daily challenge. LLM API calls are inherently I/O-bound — waiting for responses from remote servers — but they can also involve CPU-intensive post-processing, such as parsing outputs or chaining responses. Terms like “concurrent” and “parallel” execution are key to optimizing these interactions for speed, scalability, and efficiency. In this post, we’ll dissect concurrent vs. parallel execution, explore their hybrid form, and tie it all to LLM API calls using Gemini as our example.

We’ll also discuss which approach suits specific scenarios, including multi-agent setups, and compare strategies like simple API calls, AI workflows, and agents with search/reasoning. Finally, we’ll touch on scaling to thousands of users and provide a practical hybrid example.

Understanding Concurrency, Parallelism, and Their Hybrid

Concurrency: Managing Multiple Tasks with Interleaving

Concurrency allows your system to handle multiple tasks by switching between them, often on a single core. It’s perfect for I/O waits, like LLM API responses, where the CPU can juggle other tasks during downtime.

Analogy: A single barista taking orders, brewing coffee, and serving — switching rapidly.

In AI: Use for batching prompts to Gemini without blocking the main thread.

Parallelism: True Simultaneous Execution

Parallelism leverages multiple cores or processes to run tasks at the same time. It’s ideal for CPU-bound work, like analyzing Gemini outputs in parallel.

Analogy: Multiple baristas each handling a customer simultaneously.

In AI: Process responses from Gemini across cores if they require heavy computation (e.g., sentiment analysis on large texts).

Parallel Concurrent Hybrid: Combining Strengths

This blends interleaving (concurrency) with simultaneous execution (parallelism). Fetch LLM responses concurrently (async I/O), then parallelize CPU-heavy processing.

Analogy: Baristas switching between tasks while multiple work in parallel.

In AI: Essential for complex pipelines where API calls are concurrent, but downstream tasks (e.g., data aggregation) benefit from parallelism.

Key Differences and When to Use Each

• Concurrent is better for: High-volume LLM calls where latency is dominated by network waits. E.g., generating personalized responses for users without CPU bottlenecks.
• Parallel is better for: When LLM outputs need intensive processing, like running simulations or ML inferences on responses.
• Hybrid is better for: End-to-end AI pipelines, such as querying Gemini concurrently and then parallelizing evaluation or chaining.

A deep understanding between them can be made by referring to the attached image below from the ByteByteGo article.

Linking to LLM API Calls: Why It Matters with Gemini

LLM APIs like Gemini (via Google’s Generative AI SDK) involve sending prompts and awaiting generated content. These calls can take seconds, and with rate limits (e.g., queries per minute), inefficient execution leads to bottlenecks. Concurrency minimizes wait times; parallelism accelerates post-call work. At scale (1000s of users), poor design causes timeouts, high costs, or service denials. Gemini’s SDK supports both sync and async calls, making it ideal for demos.

Assume you’ve set up: pip install google-generativeai and configured genai.configure(api_key=”YOUR_API_KEY”).

Scenario 1: Sequential Execution (Baseline)

Call Gemini one prompt at a time — simple but slow.

import google.generativeai as genai
import time

genai.configure(api_key="API_KEY")
def generate_text(prompt):
 model = genai.GenerativeModel('gemini-1.5-flash')
 response = model.generate_content(prompt)
 return response.text

start_time = time.time()
prompts = ["Explain AI in 50 words", "Summarize quantum computing", "Write a haiku about robots", "Describe neural networks", "What is reinforcement learning?"]
results = [generate_text(prompt) for prompt in prompts]
print(results)
print(f"Sequential took {time.time() - start_time:.2f} seconds")

• Time: Total time taken to complete the entire process: 15.10 seconds.
• Use when prototyping or strict ordering is needed.

Scenario 2: Concurrent Execution (Async)

Use Gemini’s async support to interleave calls.

import google.generativeai as genai
import asyncio
import time

genai.configure(api_key="YOUR_API_KEY_HERE")

async def generate_text_async(prompt):
 model = genai.GenerativeModel('gemini-1.5-flash')
 response = await model.generate_content_async(prompt)
 return response.text

async def main():
 prompts = [
 "Explain AI in 50 words",
 "Summarize quantum computing",
 "Write a haiku about robots",
 "Describe neural networks",
 "What is reinforcement learning?"
 ]
 tasks = [generate_text_async(prompt) for prompt in prompts]
 return await asyncio.gather(*tasks)

if __name__ == "__main__":
 start_time = time.time()
 results = asyncio.run(main())
 print(results)
 print(f"Concurrent took {time.time() - start_time:.2f} seconds")

• Time: Total time taken to complete the entire process: 7.38 seconds.
• Use when: Handling user queries in a web app.

Scenario 3: Parallel Execution (Multiprocessing)

Spawn processes for simultaneous calls (useful if mixed with CPU work).

from multiprocessing import Pool
import google.generativeai as genai
import time

genai.configure(api_key="API_KEY")

def generate_text(prompt):
 model = genai.GenerativeModel('gemini-1.5-flash')
 response = model.generate_content(prompt)
 return response.text

start_time = time.time()
prompts = ["Explain AI in 50 words", "Summarize quantum computing", "Write a haiku about robots", "Describe neural networks", "What is reinforcement learning?"]
with Pool(processes=5) as pool:
 results = pool.map(generate_text, prompts)

print(results)
print(f"Parallel took {time.time() - start_time:.2f} seconds")

• Time: Total time taken to complete the entire process: 7.68 seconds.
• Use when: API calls + heavy local processing.

Scenario 4: Parallel Concurrent Hybrid

Fetch Gemini responses concurrently (async I/O), then parallelize CPU-bound analysis (e.g., word count on outputs).

import google.generativeai as genai
import asyncio
from multiprocessing import Pool
import time

genai.configure(api_key="YOUR_API_KEY_HERE")

# Async Gemini text generation
async def generate_text_async(prompt):
 model = genai.GenerativeModel('gemini-1.5-flash')
 response = await model.generate_content_async(prompt)
 return response.text

# CPU-bound analysis (runs in multiprocessing pool)
def analyze_text(text):
 # Example: word count (replace with complex CPU logic if needed)
 return len(text.split())

async def main():
 prompts = [
 "Explain AI in 50 words",
 "Summarize quantum computing",
 "Write a haiku about robots",
 "Describe neural networks",
 "What is reinforcement learning?"
 ]

 # Run all Gemini requests concurrently (I/O-bound)
 texts = await asyncio.gather(*[generate_text_async(p) for p in prompts])

 # Use multiprocessing for CPU-bound analysis
 with Pool(processes=3) as pool:
 results = pool.map(analyze_text, texts)

 return results

if __name__ == "__main__":
 start_time = time.time()
 results = asyncio.run(main())
 print(results) # Example: [50, 20, 7, 30, 25]
 print(f"Hybrid took {time.time() - start_time:.2f} seconds")

• Time: Total time taken to complete the entire process: 7.61 seconds.
• Hybrid scenario’s parallel phase has a limited impact due to the lightweight analyze_text task.

Why hybrid? Async handles API waits efficiently; parallelism speeds up analysis.

Note: The execution times mentioned may vary depending on factors such as the user’s location, underlying hardware, network conditions, and other parameters. However, the relative comparison of timings remains consistent.

Agent Scenarios: Mapping to Execution Models

In AI systems, “agents” are LLM-powered entities (e.g., using Gemini) that reason, act, or collaborate. Here’s how concurrency/parallelism fits:

1. Multiple Agents Working Together on Different Data: Parallel best — each agent processes independent data simultaneously (e.g., agents analyzing separate user queries). Use multiprocessing for isolation.
2. Multiple Agents Working Together on Single Data: Concurrent or hybrid — agents collaborate by interleaving (e.g., one generates ideas, another critiques). Async for coordination without blocking.
3. Single Agent Working on Single Data: Sequential or concurrent — simple, no need for parallelism unless the task has I/O sub-steps.
4. Single Agent Working on Multiple Data: Concurrent — agent processes data in batches async, like a Gemini agent summarizing multiple docs.

Approaches: AI Workflow vs. Simple API Call vs. AI Agent with Search/Reasoning

• Simple API Call: Direct Gemini generate_content. Best for one-off tasks (e.g., chat completion). Use concurrent for batching. Scalable but lacks complexity.
• AI Workflow: Multi-step pipelines (e.g., prompt chaining with Gemini). Hybrid execution — concurrent for calls, parallel for branches. Ideal for orchestration (e.g., via LangChain).
• AI Agent with Search/Reasoning: Autonomous agents (e.g., ReAct pattern with Gemini + tools). Concurrent for real-time reasoning/search; parallel if multiple sub-agents. Best for dynamic tasks like research.

Choose based on needs: Simple for speed, Workflow for structure, Agent for autonomy.

Scaling to 1000s of Users

At scale, LLM costs and rate limits (Gemini: ~60 QPM for free tier) dominate.

• Concurrent: Handles bursts efficiently, queues requests without overwhelming APIs. Use rate limiters (e.g., asyncio Semaphore).
• Parallel: Risks hitting limits faster; throttle with pools. Good for offline batch processing.
• Hybrid: Optimal — concurrent APIs minimize latency, and parallel local computing maximizes throughput. Monitor with tools like Prometheus; distribute via cloud (e.g., Google Cloud Run).
• Tips: Cache responses, use cheaper models for non-critical tasks, and implement retries. For 1000s users, expect the hybrid to reduce response times by 5–10x vs. sequential.

Conclusion

From an AI engineer’s view, mastering concurrent, parallel, and hybrid execution transforms LLM apps from sluggish to scalable. With Gemini, start concurrent for most API-heavy work, layer parallelism for compute, and hybrid for production. Experiment, profile, and scale wisely — what’s your go-to approach? Share below!

Citations

1. https://bytebytego.com/guides/concurrency-is-not-parallelism/
2. https://medium.com/@itIsMadhavan/concurrency-vs-parallelism-a-brief-review-b337c8dac350

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI