210 lines
9.8 KiB
Plaintext
210 lines
9.8 KiB
Plaintext
<!DOCTYPE html>
|
|
<html>
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<title>Go by Example: Worker Pools</title>
|
|
<link rel=stylesheet href="site.css">
|
|
</head>
|
|
<body>
|
|
<div class="example" id="worker-pools">
|
|
<h2><a href="./">Go by Example</a>: Worker Pools</h2>
|
|
|
|
<table>
|
|
|
|
<tr>
|
|
<td class="docs">
|
|
<p>In this example we’ll look at how to implement
|
|
a <em>worker pool</em> using goroutines and channels.</p>
|
|
|
|
</td>
|
|
<td class="code empty leading">
|
|
|
|
|
|
</td>
|
|
</tr>
|
|
|
|
<tr>
|
|
<td class="docs">
|
|
|
|
</td>
|
|
<td class="code leading">
|
|
<a href="http://play.golang.org/p/yuHsGf712D1"><img title="Run code" class="run" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAPCAYAAADtc08vAAAABGdBTUEAANbY1E9YMgAAABl0RVh0U29mdHdhcmUAQWRvYmUgSW1hZ2VSZWFkeXHJZTwAAAJ1SURBVCjPY/j//z8DMu7o6GAAgpQgD9tLqcmJH4KDg14aaik/MtdXe2ZjY6OCrh6Fs2jRYmZ9Pd05M9uL/u9dPfU/CLS0dfxvKIz/X5Dg/z8pKdkGqwGpqakMUdExDHJSYqt37tjxf+qUSf9rc2P+79298/+RA3v+H1zV///o6r7/DrbWFQkJiQwxMTGoBjAxMTpKiQmuqMuP/f/xw/v/J0+f/W9tbvTfxVLn/8rJVf+v757z/96hRf8TQtxuCQmLMjk4OKAawMfDVWVvrvd85eTq/7tXTP6/e/XM/22lif9LCnL+b13Q/v/Kzln/L++c/X/7/Jb/VpYWuZFRUagGAAErEBtlxvi+vn944f9L26cDNcz6v21R9/8zm6aC2SBDbu+f/78kK+4/L79AO7oBYCAqxD/57JZp/y/tmPX/wrYZ/6+CbAayD6zs/78daBjIgPayFJAGG6wGAIFAcpjH/dv7F4ANABuya/b/Od3l/ye2V/+/tnv2/7ldxSANmrgMYGBhZg7fuagD7GyYIeeBrrqwdRrQgLn/l02sBGkwwWkAEAjV5EZ/vQV0LswAGAYZsLC3DKTBAJ8BzCkRni/uHFyIYcAtoNc6ypL/ANVIohigrKwMxqqqqgxMzKzM6VHeL+6iGQAKzDtAV5XlJv3n5uFLRTHgzZs3YPzz50+GwqJiPitD9Y8Pjy4BB+CNvfP+3wUmIpAhhckhr3X19LodHZ28UQxQU1MDYw0NDQYBAQEeoBOTK7JjP2xf3Pt/bkfB/4KkoDcKMmIL5OXlFerq6hhu3rzJgC8MwMDYxGSfm5vbVn9/f0cgVxAkpqioyFBfX49iAACbTAK+xT3CzgAAAABJRU5ErkJggg==" /></a>
|
|
<div class="highlight"><pre><span class="kn">package</span> <span class="nx">main</span>
|
|
</pre></div>
|
|
|
|
</td>
|
|
</tr>
|
|
|
|
<tr>
|
|
<td class="docs">
|
|
|
|
</td>
|
|
<td class="code leading">
|
|
|
|
<div class="highlight"><pre><span class="kn">import</span> <span class="s">"fmt"</span>
|
|
<span class="kn">import</span> <span class="s">"time"</span>
|
|
</pre></div>
|
|
|
|
</td>
|
|
</tr>
|
|
|
|
<tr>
|
|
<td class="docs">
|
|
<p>Here’s the worker, of which we’ll run several
|
|
concurrent instances. These workers will receive
|
|
work on the <code>jobs</code> channel and send the corresponding
|
|
results on <code>results</code>. We’ll sleep a second per job to
|
|
simulate an expensive task.</p>
|
|
|
|
</td>
|
|
<td class="code leading">
|
|
|
|
<div class="highlight"><pre><span class="kd">func</span> <span class="nx">worker</span><span class="p">(</span><span class="nx">id</span> <span class="kt">int</span><span class="p">,</span> <span class="nx">jobs</span> <span class="o"><-</span><span class="kd">chan</span> <span class="kt">int</span><span class="p">,</span> <span class="nx">results</span> <span class="kd">chan</span><span class="o"><-</span> <span class="kt">int</span><span class="p">)</span> <span class="p">{</span>
|
|
<span class="k">for</span> <span class="nx">j</span> <span class="o">:=</span> <span class="k">range</span> <span class="nx">jobs</span> <span class="p">{</span>
|
|
<span class="nx">fmt</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">"worker"</span><span class="p">,</span> <span class="nx">id</span><span class="p">,</span> <span class="s">"started job"</span><span class="p">,</span> <span class="nx">j</span><span class="p">)</span>
|
|
<span class="nx">time</span><span class="p">.</span><span class="nx">Sleep</span><span class="p">(</span><span class="nx">time</span><span class="p">.</span><span class="nx">Second</span><span class="p">)</span>
|
|
<span class="nx">fmt</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">"worker"</span><span class="p">,</span> <span class="nx">id</span><span class="p">,</span> <span class="s">"finished job"</span><span class="p">,</span> <span class="nx">j</span><span class="p">)</span>
|
|
<span class="nx">results</span> <span class="o"><-</span> <span class="nx">j</span> <span class="o">*</span> <span class="mi">2</span>
|
|
<span class="p">}</span>
|
|
<span class="p">}</span>
|
|
</pre></div>
|
|
|
|
</td>
|
|
</tr>
|
|
|
|
<tr>
|
|
<td class="docs">
|
|
|
|
</td>
|
|
<td class="code leading">
|
|
|
|
<div class="highlight"><pre><span class="kd">func</span> <span class="nx">main</span><span class="p">()</span> <span class="p">{</span>
|
|
</pre></div>
|
|
|
|
</td>
|
|
</tr>
|
|
|
|
<tr>
|
|
<td class="docs">
|
|
<p>In order to use our pool of workers we need to send
|
|
them work and collect their results. We make 2
|
|
channels for this.</p>
|
|
|
|
</td>
|
|
<td class="code leading">
|
|
|
|
<div class="highlight"><pre> <span class="nx">jobs</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">(</span><span class="kd">chan</span> <span class="kt">int</span><span class="p">,</span> <span class="mi">100</span><span class="p">)</span>
|
|
<span class="nx">results</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">(</span><span class="kd">chan</span> <span class="kt">int</span><span class="p">,</span> <span class="mi">100</span><span class="p">)</span>
|
|
</pre></div>
|
|
|
|
</td>
|
|
</tr>
|
|
|
|
<tr>
|
|
<td class="docs">
|
|
<p>This starts up 3 workers, initially blocked
|
|
because there are no jobs yet.</p>
|
|
|
|
</td>
|
|
<td class="code leading">
|
|
|
|
<div class="highlight"><pre> <span class="k">for</span> <span class="nx">w</span> <span class="o">:=</span> <span class="mi">1</span><span class="p">;</span> <span class="nx">w</span> <span class="o"><=</span> <span class="mi">3</span><span class="p">;</span> <span class="nx">w</span><span class="o">++</span> <span class="p">{</span>
|
|
<span class="k">go</span> <span class="nx">worker</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span> <span class="nx">jobs</span><span class="p">,</span> <span class="nx">results</span><span class="p">)</span>
|
|
<span class="p">}</span>
|
|
</pre></div>
|
|
|
|
</td>
|
|
</tr>
|
|
|
|
<tr>
|
|
<td class="docs">
|
|
<p>Here we send 5 <code>jobs</code> and then <code>close</code> that
|
|
channel to indicate that’s all the work we have.</p>
|
|
|
|
</td>
|
|
<td class="code leading">
|
|
|
|
<div class="highlight"><pre> <span class="k">for</span> <span class="nx">j</span> <span class="o">:=</span> <span class="mi">1</span><span class="p">;</span> <span class="nx">j</span> <span class="o"><=</span> <span class="mi">5</span><span class="p">;</span> <span class="nx">j</span><span class="o">++</span> <span class="p">{</span>
|
|
<span class="nx">jobs</span> <span class="o"><-</span> <span class="nx">j</span>
|
|
<span class="p">}</span>
|
|
<span class="nb">close</span><span class="p">(</span><span class="nx">jobs</span><span class="p">)</span>
|
|
</pre></div>
|
|
|
|
</td>
|
|
</tr>
|
|
|
|
<tr>
|
|
<td class="docs">
|
|
<p>Finally we collect all the results of the work.
|
|
This also ensures that the worker goroutines have
|
|
finished. An alternative way to wait for multiple
|
|
goroutines is to use a <a href="waitgroups">WaitGroup</a>.</p>
|
|
|
|
</td>
|
|
<td class="code">
|
|
|
|
<div class="highlight"><pre> <span class="k">for</span> <span class="nx">a</span> <span class="o">:=</span> <span class="mi">1</span><span class="p">;</span> <span class="nx">a</span> <span class="o"><=</span> <span class="mi">5</span><span class="p">;</span> <span class="nx">a</span><span class="o">++</span> <span class="p">{</span>
|
|
<span class="o"><-</span><span class="nx">results</span>
|
|
<span class="p">}</span>
|
|
<span class="p">}</span>
|
|
</pre></div>
|
|
|
|
</td>
|
|
</tr>
|
|
|
|
</table>
|
|
|
|
<table>
|
|
|
|
<tr>
|
|
<td class="docs">
|
|
<p>Our running program shows the 5 jobs being executed by
|
|
various workers. The program only takes about 2 seconds
|
|
despite doing about 5 seconds of total work because
|
|
there are 3 workers operating concurrently.</p>
|
|
|
|
</td>
|
|
<td class="code leading">
|
|
|
|
<div class="highlight"><pre><span class="gp">$</span> <span class="nb">time</span> go run worker-pools.go
|
|
<span class="go">worker 1 started job 1</span>
|
|
<span class="go">worker 2 started job 2</span>
|
|
<span class="go">worker 3 started job 3</span>
|
|
<span class="go">worker 1 finished job 1</span>
|
|
<span class="go">worker 1 started job 4</span>
|
|
<span class="go">worker 2 finished job 2</span>
|
|
<span class="go">worker 2 started job 5</span>
|
|
<span class="go">worker 3 finished job 3</span>
|
|
<span class="go">worker 1 finished job 4</span>
|
|
<span class="go">worker 2 finished job 5</span>
|
|
</pre></div>
|
|
|
|
</td>
|
|
</tr>
|
|
|
|
<tr>
|
|
<td class="docs">
|
|
|
|
</td>
|
|
<td class="code">
|
|
|
|
<div class="highlight"><pre><span class="go">real 0m2.358s</span>
|
|
</pre></div>
|
|
|
|
</td>
|
|
</tr>
|
|
|
|
</table>
|
|
|
|
|
|
<p class="next">
|
|
Next example: <a href="waitgroups">WaitGroups</a>.
|
|
</p>
|
|
|
|
<p class="footer">
|
|
by <a href="https://markmcgranaghan.com">Mark McGranaghan</a> | <a href="https://github.com/mmcgrana/gobyexample/blob/master/examples/worker-pools">source</a> | <a href="https://github.com/mmcgrana/gobyexample#license">license</a>
|
|
</p>
|
|
</div>
|
|
</body>
|
|
</html>
|