Author here. The problem: every thread-based job queue (Sidekiq, GoodJob, etc.) has a max_threads limit. Set it to 25? That's your hard ceiling for concurrent LLM conversations. The 26th user waits, even though your server is 99% idle.
Switched to async-job in 30 minutes. No code changes needed. No max_threads = our AI app went from barely handling 25 users to thousands on the same hardware.
The beautiful part: existing libraries like RubyLLM automatically get async performance because Net::HTTP yields to fibers. No special async versions needed. No async/await keywords polluting your codebase.
After a decade in Python's asyncio world, this feels like how async should have been done. You get massive concurrency without the complexity tax.
Happy to discuss the technical details or migration experience. The future of Ruby AI apps is here, and it's surprisingly simple.
Author here. The problem: every thread-based job queue (Sidekiq, GoodJob, etc.) has a max_threads limit. Set it to 25? That's your hard ceiling for concurrent LLM conversations. The 26th user waits, even though your server is 99% idle.
Switched to async-job in 30 minutes. No code changes needed. No max_threads = our AI app went from barely handling 25 users to thousands on the same hardware.
The beautiful part: existing libraries like RubyLLM automatically get async performance because Net::HTTP yields to fibers. No special async versions needed. No async/await keywords polluting your codebase.
After a decade in Python's asyncio world, this feels like how async should have been done. You get massive concurrency without the complexity tax.
Happy to discuss the technical details or migration experience. The future of Ruby AI apps is here, and it's surprisingly simple.