Saturday, 7 September 2013

node.js worker farm and queue management

node.js worker farm and queue management

I'm looking for a solution to create a "worker farm" using node.js.
Basically we have an app in node and we need to send off "jobs" to be run
across n number of worker servers. For example, let's say we have 5
servers that all run certain jobs, the jobs need to be distributed or
'queued' until a worker has CPU available to process the jobs.
One way to do this would be to have a worker server that is run on every
separate machine. Each worker would pull from a queue based on it's CPU
utilization or queue availability. The application itself, would add items
to a queue (probably handled by Redis). There would be no direct
communication between the individual worker servers and the application
itself. One problem I could see with this is if multiple workers grab the
same queue at the same time. The other method would be to somehow
communicate with the worker servers from the application, which would get
worker with the least resources and 'assign' the job to that particular
worker or queue it up.
Does anyone know of a good solution for handling this?
Thank you!

No comments:

Post a Comment