Summary

Gearman is an open-source distributed job server framework for farming out work across machines or processes. It provides a client-worker-jobserver architecture for parallel processing, load balancing, and cross-language function calls. Used by Craigslist, Tumblr, Yelp, and Etsy.

Gearman 是一個開源分散式任務伺服器框架,用於將工作分發到各機器或進程。它提供客戶端-工作端-任務伺服器架構,支援並行處理、負載均衡和跨語言函數調用。

Key Points

  • Architecture: three parts — client (creates jobs), job server (routes jobs), worker (executes jobs)
  • Communication via TCP sockets; client and worker APIs abstract away networking
  • Multi-language: clients and workers can be written in different languages (PHP client + Python worker, etc.)
  • Job types: foreground (synchronous), background (async), and prioritized jobs
  • No single point of failure: run multiple job servers; clients/workers auto-failover
  • No message size limits: supports messages up to 4GB; chunking for larger
  • Classic use case: offload image conversion from web servers to dedicated worker machines
  • Workers use natural load balancing: job server only sends new jobs to idle workers

Insights

Gearman’s key architectural insight is separating job routing from job execution — the job server is just a traffic coordinator, not a processing node. This makes scale-out trivially simple: need more capacity? Boot more workers and connect them to the existing job server. The cross-language capability is practically underused but architecturally important: you can have a Ruby web app trigger C++ workers for compute-intensive tasks without shared-memory complexity. Gearman predates modern alternatives (Celery, Sidekiq, BullMQ) but the architectural patterns it embodies remain the foundations those systems build on.

Connections

Raw Excerpt

Gearman provides a generic application framework to farm out work to other machines or processes that are better suited to do the work. It allows you to do work in parallel, to load balance processing, and to call functions between languages.