yeray/gobyexample
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
gobyexample/public/worker-pools
Hana 9e216da9ef go.mod: add go.mod and move pygments to third_party 
After go1.16, go will use module mode by default,
even when the repository is checked out under GOPATH
or in a one-off directory. Add go.mod, go.sum to keep
this repo buildable without opting out of the module
mode.

> go mod init github.com/mmcgrana/gobyexample
> go mod tidy
> go mod vendor

In module mode, the 'vendor' directory is special
and its contents will be actively maintained by the
go command. pygments aren't the dependency the go will
know about, so it will delete the contents from vendor
directory. Move it to `third_party` directory now.

And, vendor the blackfriday package.

Note: the tutorial contents are not affected by the
change in go1.16 because all the examples in this
tutorial ask users to run the go command with the
explicit list of files to be compiled (e.g.
`go run hello-world.go` or `go build command-line-arguments.go`).
When the source list is provided, the go command does
not have to compute the build list and whether it's
running in GOPATH mode or module mode becomes irrelevant.
2021-02-15 16:45:26 -05:00

232 lines
10 KiB
Plaintext
Generated
Raw Blame History

<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>Go by Example: Worker Pools</title>
<link rel=stylesheet href="site.css">
</head>
<script>
onkeydown = (e) => {
if (e.key == "ArrowLeft") {
window.location.href = 'tickers';
}
if (e.key == "ArrowRight") {
window.location.href = 'waitgroups';
}
}
</script>
<body>
<div class="example" id="worker-pools">
<h2><a href="./">Go by Example</a>: Worker Pools</h2>
<table>
<tr>
<td class="docs">
<p>In this example we&rsquo;ll look at how to implement
a <em>worker pool</em> using goroutines and channels.</p>
</td>
<td class="code empty leading">
</td>
</tr>
<tr>
<td class="docs">
</td>
<td class="code leading">
<a href="http://play.golang.org/p/hiSJJsYZJKL"><img title="Run code" src="play.png" class="run" /></a><img title="Copy code" src="clipboard.png" class="copy" />
<div class="highlight"><pre><span class="kn">package</span> <span class="nx">main</span>
</pre></div>
</td>
</tr>
<tr>
<td class="docs">
</td>
<td class="code leading">
<div class="highlight"><pre><span class="kn">import</span> <span class="p">(</span>
<span class="s">&quot;fmt&quot;</span>
<span class="s">&quot;time&quot;</span>
<span class="p">)</span>
</pre></div>
</td>
</tr>
<tr>
<td class="docs">
<p>Here&rsquo;s the worker, of which we&rsquo;ll run several
concurrent instances. These workers will receive
work on the <code>jobs</code> channel and send the corresponding
results on <code>results</code>. We&rsquo;ll sleep a second per job to
simulate an expensive task.</p>
</td>
<td class="code leading">
<div class="highlight"><pre><span class="kd">func</span> <span class="nx">worker</span><span class="p">(</span><span class="nx">id</span> <span class="kt">int</span><span class="p">,</span> <span class="nx">jobs</span> <span class="o">&lt;-</span><span class="kd">chan</span> <span class="kt">int</span><span class="p">,</span> <span class="nx">results</span> <span class="kd">chan</span><span class="o">&lt;-</span> <span class="kt">int</span><span class="p">)</span> <span class="p">{</span>
<span class="k">for</span> <span class="nx">j</span> <span class="o">:=</span> <span class="k">range</span> <span class="nx">jobs</span> <span class="p">{</span>
<span class="nx">fmt</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">&quot;worker&quot;</span><span class="p">,</span> <span class="nx">id</span><span class="p">,</span> <span class="s">&quot;started job&quot;</span><span class="p">,</span> <span class="nx">j</span><span class="p">)</span>
<span class="nx">time</span><span class="p">.</span><span class="nx">Sleep</span><span class="p">(</span><span class="nx">time</span><span class="p">.</span><span class="nx">Second</span><span class="p">)</span>
<span class="nx">fmt</span><span class="p">.</span><span class="nx">Println</span><span class="p">(</span><span class="s">&quot;worker&quot;</span><span class="p">,</span> <span class="nx">id</span><span class="p">,</span> <span class="s">&quot;finished job&quot;</span><span class="p">,</span> <span class="nx">j</span><span class="p">)</span>
<span class="nx">results</span> <span class="o">&lt;-</span> <span class="nx">j</span> <span class="o">*</span> <span class="mi">2</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
</td>
</tr>
<tr>
<td class="docs">
</td>
<td class="code leading">
<div class="highlight"><pre><span class="kd">func</span> <span class="nx">main</span><span class="p">()</span> <span class="p">{</span>
</pre></div>
</td>
</tr>
<tr>
<td class="docs">
<p>In order to use our pool of workers we need to send
them work and collect their results. We make 2
channels for this.</p>
</td>
<td class="code leading">
<div class="highlight"><pre> <span class="kd">const</span> <span class="nx">numJobs</span> <span class="p">=</span> <span class="mi">5</span>
<span class="nx">jobs</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">(</span><span class="kd">chan</span> <span class="kt">int</span><span class="p">,</span> <span class="nx">numJobs</span><span class="p">)</span>
<span class="nx">results</span> <span class="o">:=</span> <span class="nb">make</span><span class="p">(</span><span class="kd">chan</span> <span class="kt">int</span><span class="p">,</span> <span class="nx">numJobs</span><span class="p">)</span>
</pre></div>
</td>
</tr>
<tr>
<td class="docs">
<p>This starts up 3 workers, initially blocked
because there are no jobs yet.</p>
</td>
<td class="code leading">
<div class="highlight"><pre> <span class="k">for</span> <span class="nx">w</span> <span class="o">:=</span> <span class="mi">1</span><span class="p">;</span> <span class="nx">w</span> <span class="o">&lt;=</span> <span class="mi">3</span><span class="p">;</span> <span class="nx">w</span><span class="o">++</span> <span class="p">{</span>
<span class="k">go</span> <span class="nx">worker</span><span class="p">(</span><span class="nx">w</span><span class="p">,</span> <span class="nx">jobs</span><span class="p">,</span> <span class="nx">results</span><span class="p">)</span>
<span class="p">}</span>
</pre></div>
</td>
</tr>
<tr>
<td class="docs">
<p>Here we send 5 <code>jobs</code> and then <code>close</code> that
channel to indicate that&rsquo;s all the work we have.</p>
</td>
<td class="code leading">
<div class="highlight"><pre> <span class="k">for</span> <span class="nx">j</span> <span class="o">:=</span> <span class="mi">1</span><span class="p">;</span> <span class="nx">j</span> <span class="o">&lt;=</span> <span class="nx">numJobs</span><span class="p">;</span> <span class="nx">j</span><span class="o">++</span> <span class="p">{</span>
<span class="nx">jobs</span> <span class="o">&lt;-</span> <span class="nx">j</span>
<span class="p">}</span>
<span class="nb">close</span><span class="p">(</span><span class="nx">jobs</span><span class="p">)</span>
</pre></div>
</td>
</tr>
<tr>
<td class="docs">
<p>Finally we collect all the results of the work.
This also ensures that the worker goroutines have
finished. An alternative way to wait for multiple
goroutines is to use a <a href="waitgroups">WaitGroup</a>.</p>
</td>
<td class="code">
<div class="highlight"><pre> <span class="k">for</span> <span class="nx">a</span> <span class="o">:=</span> <span class="mi">1</span><span class="p">;</span> <span class="nx">a</span> <span class="o">&lt;=</span> <span class="nx">numJobs</span><span class="p">;</span> <span class="nx">a</span><span class="o">++</span> <span class="p">{</span>
<span class="o">&lt;-</span><span class="nx">results</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
</td>
</tr>
</table>
<table>
<tr>
<td class="docs">
<p>Our running program shows the 5 jobs being executed by
various workers. The program only takes about 2 seconds
despite doing about 5 seconds of total work because
there are 3 workers operating concurrently.</p>
</td>
<td class="code leading">
<div class="highlight"><pre><span class="gp">$</span> <span class="nb">time</span> go run worker-pools.go
<span class="go">worker 1 started job 1</span>
<span class="go">worker 2 started job 2</span>
<span class="go">worker 3 started job 3</span>
<span class="go">worker 1 finished job 1</span>
<span class="go">worker 1 started job 4</span>
<span class="go">worker 2 finished job 2</span>
<span class="go">worker 2 started job 5</span>
<span class="go">worker 3 finished job 3</span>
<span class="go">worker 1 finished job 4</span>
<span class="go">worker 2 finished job 5</span>
</pre></div>
</td>
</tr>
<tr>
<td class="docs">
</td>
<td class="code">
<div class="highlight"><pre><span class="go">real 0m2.358s</span>
</pre></div>
</td>
</tr>
</table>
<p class="next">
Next example: <a href="waitgroups">WaitGroups</a>.
</p>
<p class="footer">
by <a href="https://markmcgranaghan.com">Mark McGranaghan</a> | <a href="https://github.com/mmcgrana/gobyexample/blob/master/examples/worker-pools">source</a> | <a href="https://github.com/mmcgrana/gobyexample#license">license</a>
</p>
</div>
<script>
var codeLines = [];
codeLines.push('');codeLines.push('package main\u000A');codeLines.push('import (\u000A \"fmt\"\u000A \"time\"\u000A)\u000A');codeLines.push('func worker(id int, jobs \u003C-chan int, results chan\u003C- int) {\u000A for j :\u003D range jobs {\u000A fmt.Println(\"worker\", id, \"started job\", j)\u000A time.Sleep(time.Second)\u000A fmt.Println(\"worker\", id, \"finished job\", j)\u000A results \u003C- j * 2\u000A }\u000A}\u000A');codeLines.push('func main() {\u000A');codeLines.push(' const numJobs \u003D 5\u000A jobs :\u003D make(chan int, numJobs)\u000A results :\u003D make(chan int, numJobs)\u000A');codeLines.push(' for w :\u003D 1; w \u003C\u003D 3; w++ {\u000A go worker(w, jobs, results)\u000A }\u000A');codeLines.push(' for j :\u003D 1; j \u003C\u003D numJobs; j++ {\u000A jobs \u003C- j\u000A }\u000A close(jobs)\u000A');codeLines.push(' for a :\u003D 1; a \u003C\u003D numJobs; a++ {\u000A \u003C-results\u000A }\u000A}\u000A');codeLines.push('');codeLines.push('');
</script>
<script src="site.js" async></script>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink