Richard Kallos: Posts tagged 'graphs'urn:https-www-rkallos-com:-tags-graphs-html2018-04-22 15:15:00ZIntroducing wrek, a miniature Erlang graph engineurn:https-rkallos-com:blog-2018-01-26-introducing-wrek-a-miniature-erlang-graph-engine-html2018-04-22T15:15:00Z2018-04-22T15:15:00ZRichard Kallos<p>
This is a copy of <a href="https://codesync.global/media/introducing-wrek-a-miniature-erlang-graph-engine/" title="">this article</a>, kept for posterity on my blog.</p>
<p>
I gave a talk about Wrek at Code BEAM SF last month. You can view it
<a href="https://www.youtube.com/watch?v=dYDdnZoH0uk" title="">here</a>.</p>
<hr>
<p>
<a href="https://github.com/rkallos/wrek" title="">Wrek</a> is an Erlang library I wrote for concurrently executing task
dependency graphs. It’s intended purpose is to run a set of pre-defined tasks
that have a partial ordering between them. In this post, I explain why I wrote
wrek, what it can be used for, and how to use it.</p>
<!-- more -->
<h1>
Motivation</h1>
<p>
Wrek emerged as the result of two distinct forces. First, my amateurish
enjoyment of graph theory made me try to see graphs wherever I could, laying the
conceptual foundation for this library. Second, I realized that a project I was
working on at <a href="https://adgear.com/" title="">Adgear</a> would benefit from such a library, causing me to finally
start writing Wrek.</p>
<h2>
Conceptual</h2>
<p>
The graph is a pervasive data structure in computing. When I learned about graph
algorithms in school, I was amazed by their broad applications. Graphs play
essential roles in compilers and build systems. Various graph algorithms form
the backbone of how we communicate with each other over the internet.</p>
<p>
Our everyday lives tend to be filled with lists. We have to-do lists, shopping
lists, recipes, checklists, instructions, and much, much more. One day, I
realized that some of these lists are deceiving. Some of these lists are
actually graphs hiding in list’s clothing; these lists are <a href="https://en.wikipedia.org/wiki/Topological_sorting" title="">topological
orderings</a> of <a href="https://en.wikipedia.org/wiki/Directed_acyclic_graph" title="">directed acyclic graphs</a>. Two of the more obvious
cases of this are to-do lists and recipes. You can’t send a letter that you
haven’t written yet, nor can you cook spaghetti if you haven’t boiled a pot of
water.</p>
<p>
Once the sneaky graphs were discovered, I spent some time thinking about how to
benefit from treating various lists like the DAGs that they secretly are. The
clearest analogies between these lists and DAGs were to-do lists and
recipes. These very closely resembled <a href="https://en.wikipedia.org/wiki/Dependency_graph" title="">dependency graphs</a>. Unlike in their
list representation, these dependency graphs show which vertices (individual
tasks) can be executed concurrently. Sometimes this is obvious (e.g: I can chop
vegetables while I wait for the pot of water to boil! I can begin preparing a
second baking sheet of cookies while the first batch is in the oven!), but I had
my hopes that the act of reformulating lists as dependency graphs could expose
more opportunities for concurrency.</p>
<pre><code> Boil water -- Add pasta -- Cook pasta --.
\
Purée tomatoes --. \
\ \
Chop vegetables -- Combine in saucepan -- Simmer sauce -- Combine pasta and sauce -- Serve
/
Add spices --'</code></pre>
<p>
Edges between vertices in the above graph express a partial ordering. Vertices
with no path between them can be executed concurrently. The relationship between
vertices in the graph reflects the truth in the kitchen. We can cook the pasta
while our sauce is simmering. It doesn’t matter whether we chop vegetables or
purée tomatoes first; they both need to be done if we’re going to make any
sauce.</p>
<p>
Despite my best efforts, I am still a disaster in the kitchen. As a means of
changing this inconvenient fact, I thought it would be fun to try representing
recipes as directed graphs, with extra data like how long each step can be
expected to take. This sat in my list of ‘someday projects’ for a long time, and
I was only reminded of it when I begin to think about a project I was working on
at $JOB.</p>
<h2>
Practical</h2>
<p>
One of the projects I accomplished last year at Adgear was removing an expensive
computation that was taking place on each of our already maxed-out edge servers,
and replaced it with a system that performed the expensive computation on a
single machine, then distributed the result. The system worked when nobody
touched it, but it was the programming equivalent of twigs and duct tape; cron
jobs and shell scripts. This system continued to be painful to use, and more
examples of expensive computations being done on CPU-starved machines were
cropping up. At this point, it made sense start writing a more robust system.</p>
<p>
These expensive computations decomposed nicely into lists of steps to be
performed. Fetch this data, transform it, transform it some more, send some of
the data to this group of servers, send this other data to some other group of
servers. Shell scripts are pretty good at encoding these pipelines, so it was an
acceptable choice at the time of implementation. After a while, it dawned on me
that these lists of steps were not really lists; they were dependency graphs. I
tried to expose the latent concurrency in the shell scripts by spicing them up
with some background jobs and <code class="inline">waitpid</code>, but it was decided that it would make
more sense to switch to Erlang and benefit from all that OTP has to offer.</p>
<h1>
Let’s get <code class="inline">wrek</code>ed</h1>
<p>
(I’m really sorry. I couldn’t resist.)</p>
<p>
Wrek accepts dependency graphs like the above spaghetti recipe as input, and
executes each vertex. The nature of concurrently executing actions as soon as
they are able to execute is <em>generic</em> to any problem that can be represented by
a dependency graph. The structure of a dependency graph, as well as the tasks
involved in each of the graph’s vertices are <em>specific</em> to the user’s
wishes. Following the same generic/specific split as OTP; the generic portion of
this class of problems is solved by <code class="inline">wrek</code> and <code class="inline">wrek_vert</code> modules. The specific
portion is provided by the user in the form of an Erlang map describing each
vertex in the graph, and a set of callback modules that implement the
<code class="inline">wrek_vert</code> behaviour.</p>
<p>
The <code class="inline">wrek_vert</code> behaviour consists of a single callback <code class="inline">run/2</code>, where the first
argument is a list of arguments to be sent to the callback function, and the
second argument is the ID of a process which can provide information generated
by other vertices. The expected result of this callback function is either <code class="inline">{ok, Any :: any()}</code> or <code class="inline">{error, Reason :: any()}</code>. If the callback function succeeds,
<code class="inline">Any</code> will be taken by wrek and made available to other <code class="inline">wrek_vert</code>
processes. If the callback function crashes or returns an error, the entire
graph will be shut down.</p>
<h2>
Making Erlang Pasta</h2>
<p>
Following the contrived example above, let’s set about using wrek to make
pasta. Of course, our program won’t really make pasta, but it’s output ought to
fool somebody.</p>
<p>
Looking at the above graph, each step seems to do one of three things:</p>
<ol>
<li>
Add an ingredient </li>
<li>
Combine ingredients in a vessel </li>
<li>
Do something to ingredients in a vessel </li>
</ol>
<p>
Let’s go ahead write some <code class="inline">wrek_vert</code>s for each of those actions. If you don’t
feel like following along in a text editor, the full code can be found
<a href="https://github.com/rkallos/wrek_examples" title="">here</a>.</p>
<pre><code class="makeup erlang"><span class="p">-</span><span class="na">module</span><span class="p" data-group-id="8180388820-1">(</span><span class="ss">cook_add</span><span class="p" data-group-id="8180388820-1">)</span><span class="p">.</span><span class="w">
</span><span class="w">
</span><span class="p">-</span><span class="na">behaviour</span><span class="p" data-group-id="8180388820-2">(</span><span class="ss">wrek_vert</span><span class="p" data-group-id="8180388820-2">)</span><span class="p">.</span><span class="w">
</span><span class="p">-</span><span class="na">export</span><span class="p" data-group-id="8180388820-3">(</span><span class="p" data-group-id="8180388820-4">[</span><span class="ss">run</span><span class="p">/</span><span class="mi">2</span><span class="p" data-group-id="8180388820-4">]</span><span class="p" data-group-id="8180388820-3">)</span><span class="p">.</span><span class="w">
</span><span class="nf">run</span><span class="p" data-group-id="8180388820-5">(</span><span class="p" data-group-id="8180388820-6">[</span><span class="n">Ingredient</span><span class="p">,</span><span class="w"> </span><span class="n">Quantity</span><span class="p" data-group-id="8180388820-6">]</span><span class="p">,</span><span class="w"> </span><span class="p">_</span><span class="n">Pid</span><span class="p" data-group-id="8180388820-5">)</span><span class="w"> </span><span class="p">-></span><span class="w">
</span><span class="nc">io</span><span class="p">:</span><span class="nf">format</span><span class="p" data-group-id="8180388820-7">(</span><span class="s">"adding </span><span class="si">~s</span><span class="s">. amount: </span><span class="si">~s</span><span class="s">.</span><span class="si">~n</span><span class="s">"</span><span class="p">,</span><span class="w"> </span><span class="p" data-group-id="8180388820-8">[</span><span class="n">Ingredient</span><span class="p">,</span><span class="w"> </span><span class="n">Quantity</span><span class="p" data-group-id="8180388820-8">]</span><span class="p" data-group-id="8180388820-7">)</span><span class="p">,</span><span class="w">
</span><span class="p" data-group-id="8180388820-9">{</span><span class="ss">ok</span><span class="p">,</span><span class="w"> </span><span class="p" data-group-id="8180388820-10">#{</span><span class="ss">added</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p" data-group-id="8180388820-11">[</span><span class="p" data-group-id="8180388820-12">{</span><span class="n">Ingredient</span><span class="p">,</span><span class="w"> </span><span class="n">Quantity</span><span class="p" data-group-id="8180388820-12">}</span><span class="p" data-group-id="8180388820-11">]</span><span class="p" data-group-id="8180388820-10">}</span><span class="p" data-group-id="8180388820-9">}</span><span class="p">.</span></code></pre>
<p>
That’s all for <code class="inline">cook_add</code>. It prints a message, then produces a
map with a key <code class="inline">added</code> whose value is a proplist with a single pair.</p>
<pre><code class="makeup erlang"><span class="p">-</span><span class="na">module</span><span class="p" data-group-id="8994662717-1">(</span><span class="ss">cook_heat</span><span class="p" data-group-id="8994662717-1">)</span><span class="p">.</span><span class="w">
</span><span class="w">
</span><span class="p">-</span><span class="na">behaviour</span><span class="p" data-group-id="8994662717-2">(</span><span class="ss">wrek_vert</span><span class="p" data-group-id="8994662717-2">)</span><span class="p">.</span><span class="w">
</span><span class="p">-</span><span class="na">export</span><span class="p" data-group-id="8994662717-3">(</span><span class="p" data-group-id="8994662717-4">[</span><span class="ss">run</span><span class="p">/</span><span class="mi">2</span><span class="p" data-group-id="8994662717-4">]</span><span class="p" data-group-id="8994662717-3">)</span><span class="p">.</span><span class="w">
</span><span class="nf">run</span><span class="p" data-group-id="8994662717-5">(</span><span class="p" data-group-id="8994662717-6">[</span><span class="n">Verb</span><span class="p">,</span><span class="w"> </span><span class="n">Noun</span><span class="p" data-group-id="8994662717-6">]</span><span class="p">,</span><span class="w"> </span><span class="p">_</span><span class="n">Pid</span><span class="p" data-group-id="8994662717-5">)</span><span class="w"> </span><span class="p">-></span><span class="w">
</span><span class="nc">io</span><span class="p">:</span><span class="nf">format</span><span class="p" data-group-id="8994662717-7">(</span><span class="s">"</span><span class="si">~p</span><span class="s">ing </span><span class="si">~p</span><span class="s">.</span><span class="si">~n</span><span class="s">"</span><span class="p">,</span><span class="w"> </span><span class="p" data-group-id="8994662717-8">[</span><span class="n">Verb</span><span class="p">,</span><span class="w"> </span><span class="n">Noun</span><span class="p" data-group-id="8994662717-8">]</span><span class="p" data-group-id="8994662717-7">)</span><span class="p">,</span><span class="w">
</span><span class="p" data-group-id="8994662717-9">{</span><span class="ss">ok</span><span class="p">,</span><span class="w"> </span><span class="p" data-group-id="8994662717-10">#{</span><span class="p" data-group-id="8994662717-10">}</span><span class="p" data-group-id="8994662717-9">}</span><span class="p">.</span></code></pre>
<p>
<code class="inline">cook_heat</code> is also pretty short. It’s also very abstract. It could be used to
print a message about <code class="inline">Verb</code>ing any <code class="inline">Noun</code>, not just cooking ingredients!</p>
<p>
Our final callback module is a little longer, because it does a little bit more
than printing a message.</p>
<pre><code class="makeup erlang"><span class="p">-</span><span class="na">module</span><span class="p" data-group-id="4216963176-1">(</span><span class="ss">cook_combine</span><span class="p" data-group-id="4216963176-1">)</span><span class="p">.</span><span class="w">
</span><span class="w">
</span><span class="p">-</span><span class="na">behaviour</span><span class="p" data-group-id="4216963176-2">(</span><span class="ss">wrek_vert</span><span class="p" data-group-id="4216963176-2">)</span><span class="p">.</span><span class="w">
</span><span class="p">-</span><span class="na">export</span><span class="p" data-group-id="4216963176-3">(</span><span class="p" data-group-id="4216963176-4">[</span><span class="ss">run</span><span class="p">/</span><span class="mi">2</span><span class="p" data-group-id="4216963176-4">]</span><span class="p" data-group-id="4216963176-3">)</span><span class="p">.</span><span class="w">
</span><span class="nf">run</span><span class="p" data-group-id="4216963176-5">(</span><span class="p" data-group-id="4216963176-6">[</span><span class="n">Ingredients</span><span class="p">,</span><span class="w"> </span><span class="n">Vessel</span><span class="p" data-group-id="4216963176-6">]</span><span class="p">,</span><span class="w"> </span><span class="n">Pid</span><span class="p" data-group-id="4216963176-5">)</span><span class="w"> </span><span class="p">-></span><span class="w">
</span><span class="n">Fun</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nf">fun</span><span class="p" data-group-id="4216963176-7">(</span><span class="n">Step</span><span class="p">,</span><span class="w"> </span><span class="n">Acc</span><span class="p" data-group-id="4216963176-7">)</span><span class="w"> </span><span class="p">-></span><span class="w">
</span><span class="n">Stuff</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nc">wrek_vert</span><span class="p">:</span><span class="nf">get</span><span class="p" data-group-id="4216963176-8">(</span><span class="n">Pid</span><span class="p">,</span><span class="w"> </span><span class="n">Step</span><span class="p">,</span><span class="w"> </span><span class="ss">added</span><span class="p" data-group-id="4216963176-8">)</span><span class="p">,</span><span class="w">
</span><span class="nc">io</span><span class="p">:</span><span class="nf">format</span><span class="p" data-group-id="4216963176-9">(</span><span class="s">"combining </span><span class="si">~p</span><span class="s"> with </span><span class="si">~p</span><span class="s"> in </span><span class="si">~p</span><span class="s">.</span><span class="si">~n</span><span class="s">"</span><span class="p">,</span><span class="w"> </span><span class="p" data-group-id="4216963176-10">[</span><span class="n">Stuff</span><span class="p">,</span><span class="w"> </span><span class="n">Acc</span><span class="p">,</span><span class="w"> </span><span class="n">Vessel</span><span class="p" data-group-id="4216963176-10">]</span><span class="p" data-group-id="4216963176-9">)</span><span class="p">,</span><span class="w">
</span><span class="n">Stuff</span><span class="w"> </span><span class="o">++</span><span class="w"> </span><span class="n">Acc</span><span class="w">
</span><span class="k">end</span><span class="p">,</span><span class="w">
</span><span class="n">Stuff</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nc">lists</span><span class="p">:</span><span class="nf">foldl</span><span class="p" data-group-id="4216963176-11">(</span><span class="n">Fun</span><span class="p">,</span><span class="w"> </span><span class="p" data-group-id="4216963176-12">[</span><span class="p" data-group-id="4216963176-12">]</span><span class="p">,</span><span class="w"> </span><span class="n">Ingredients</span><span class="p" data-group-id="4216963176-11">)</span><span class="p">,</span><span class="w">
</span><span class="nc">io</span><span class="p">:</span><span class="nf">format</span><span class="p" data-group-id="4216963176-13">(</span><span class="s">"</span><span class="si">~p</span><span class="s"> now contains: </span><span class="si">~p</span><span class="s">.</span><span class="si">~n</span><span class="s">"</span><span class="p">,</span><span class="w"> </span><span class="p" data-group-id="4216963176-14">[</span><span class="n">Vessel</span><span class="p">,</span><span class="w"> </span><span class="n">Stuff</span><span class="p" data-group-id="4216963176-14">]</span><span class="p" data-group-id="4216963176-13">)</span><span class="p">,</span><span class="w">
</span><span class="p" data-group-id="4216963176-15">{</span><span class="ss">ok</span><span class="p">,</span><span class="w"> </span><span class="p" data-group-id="4216963176-16">#{</span><span class="ss">added</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="n">Stuff</span><span class="p" data-group-id="4216963176-16">}</span><span class="p" data-group-id="4216963176-15">}</span><span class="p">.</span></code></pre>
<p>
<code class="inline">Ingredients</code> is expected to be a list of vertex names. We finally use our
parent process’s <code class="inline">Pid</code> as an argument to <code class="inline">wrek_vert:get/3</code>. This lets us consume
the data produced by the <code class="inline">cook_add</code> callback module. After combining everything,
we returns a new collection of ingredients.</p>
<p>
Alright! We’re nearly done describing the specific portion of our problem! The
last step is to represent our dependency graph in terms of these callback
modules and the arguments we want to pass to them.</p>
<pre><code class="makeup erlang"><span class="p">-</span><span class="na">module</span><span class="p" data-group-id="8436274812-1">(</span><span class="ss">wrek_example</span><span class="p" data-group-id="8436274812-1">)</span><span class="p">.</span><span class="w">
</span><span class="w">
</span><span class="p">-</span><span class="na">export</span><span class="p" data-group-id="8436274812-2">(</span><span class="p" data-group-id="8436274812-3">[</span><span class="ss">make_pasta</span><span class="p">/</span><span class="mi">0</span><span class="p" data-group-id="8436274812-3">]</span><span class="p" data-group-id="8436274812-2">)</span><span class="p">.</span><span class="w">
</span><span class="nf">make_pasta</span><span class="p" data-group-id="8436274812-4">(</span><span class="p" data-group-id="8436274812-4">)</span><span class="w"> </span><span class="p">-></span><span class="w">
</span><span class="nc">application</span><span class="p">:</span><span class="nf">ensure_all_started</span><span class="p" data-group-id="8436274812-5">(</span><span class="ss">wrek</span><span class="p" data-group-id="8436274812-5">)</span><span class="p">,</span><span class="w">
</span><span class="n">Graph</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p" data-group-id="8436274812-6">#{</span><span class="w">
</span><span class="ss">tomatoes</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p" data-group-id="8436274812-7">#{</span><span class="w">
</span><span class="ss">module</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="ss">cook_add</span><span class="p">,</span><span class="w">
</span><span class="ss">args</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p" data-group-id="8436274812-8">[</span><span class="s">"pureed tomatoes"</span><span class="p">,</span><span class="w"> </span><span class="s">"1 can"</span><span class="p" data-group-id="8436274812-8">]</span><span class="p">,</span><span class="w">
</span><span class="ss">deps</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p" data-group-id="8436274812-9">[</span><span class="p" data-group-id="8436274812-9">]</span><span class="w">
</span><span class="p" data-group-id="8436274812-7">}</span><span class="p">,</span><span class="w">
</span><span class="ss">vegetables</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p" data-group-id="8436274812-10">#{</span><span class="w">
</span><span class="ss">module</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="ss">cook_add</span><span class="p">,</span><span class="w">
</span><span class="ss">args</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p" data-group-id="8436274812-11">[</span><span class="s">"chopped vegetables"</span><span class="p">,</span><span class="w"> </span><span class="s">"lots"</span><span class="p" data-group-id="8436274812-11">]</span><span class="p">,</span><span class="w">
</span><span class="ss">deps</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p" data-group-id="8436274812-12">[</span><span class="p" data-group-id="8436274812-12">]</span><span class="w">
</span><span class="p" data-group-id="8436274812-10">}</span><span class="p">,</span><span class="w">
</span><span class="ss">spices</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p" data-group-id="8436274812-13">#{</span><span class="w">
</span><span class="ss">module</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="ss">cook_add</span><span class="p">,</span><span class="w">
</span><span class="ss">args</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p" data-group-id="8436274812-14">[</span><span class="s">"spices"</span><span class="p">,</span><span class="w"> </span><span class="s">"to taste"</span><span class="p" data-group-id="8436274812-14">]</span><span class="p">,</span><span class="w">
</span><span class="ss">deps</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p" data-group-id="8436274812-15">[</span><span class="p" data-group-id="8436274812-15">]</span><span class="w">
</span><span class="p" data-group-id="8436274812-13">}</span><span class="p">,</span><span class="w">
</span><span class="ss">saucepan</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p" data-group-id="8436274812-16">#{</span><span class="w">
</span><span class="ss">module</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="ss">cook_combine</span><span class="p">,</span><span class="w">
</span><span class="ss">args</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p" data-group-id="8436274812-17">[</span><span class="p" data-group-id="8436274812-18">[</span><span class="ss">tomatoes</span><span class="p">,</span><span class="w"> </span><span class="ss">vegetables</span><span class="p">,</span><span class="w"> </span><span class="ss">spices</span><span class="p" data-group-id="8436274812-18">]</span><span class="p">,</span><span class="w"> </span><span class="ss">saucepan</span><span class="p" data-group-id="8436274812-17">]</span><span class="p">,</span><span class="w">
</span><span class="ss">deps</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p" data-group-id="8436274812-19">[</span><span class="ss">tomatoes</span><span class="p">,</span><span class="w"> </span><span class="ss">vegetables</span><span class="p">,</span><span class="w"> </span><span class="ss">spices</span><span class="p" data-group-id="8436274812-19">]</span><span class="w">
</span><span class="p" data-group-id="8436274812-16">}</span><span class="p">,</span><span class="w">
</span><span class="ss">simmer_sauce</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p" data-group-id="8436274812-20">#{</span><span class="w">
</span><span class="ss">module</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="ss">cook_heat</span><span class="p">,</span><span class="w">
</span><span class="ss">args</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p" data-group-id="8436274812-21">[</span><span class="ss">simmer</span><span class="p">,</span><span class="w"> </span><span class="ss">sauce</span><span class="p" data-group-id="8436274812-21">]</span><span class="p">,</span><span class="w">
</span><span class="ss">deps</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p" data-group-id="8436274812-22">[</span><span class="ss">saucepan</span><span class="p" data-group-id="8436274812-22">]</span><span class="w">
</span><span class="p" data-group-id="8436274812-20">}</span><span class="p">,</span><span class="w">
</span><span class="ss">boil_water</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p" data-group-id="8436274812-23">#{</span><span class="w">
</span><span class="ss">module</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="ss">cook_heat</span><span class="p">,</span><span class="w">
</span><span class="ss">args</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p" data-group-id="8436274812-24">[</span><span class="ss">boil</span><span class="p">,</span><span class="w"> </span><span class="ss">water</span><span class="p" data-group-id="8436274812-24">]</span><span class="p">,</span><span class="w">
</span><span class="ss">deps</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p" data-group-id="8436274812-25">[</span><span class="p" data-group-id="8436274812-25">]</span><span class="w">
</span><span class="p" data-group-id="8436274812-23">}</span><span class="p">,</span><span class="w">
</span><span class="ss">add_pasta</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p" data-group-id="8436274812-26">#{</span><span class="w">
</span><span class="ss">module</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="ss">cook_add</span><span class="p">,</span><span class="w">
</span><span class="ss">args</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p" data-group-id="8436274812-27">[</span><span class="s">"pasta"</span><span class="p">,</span><span class="w"> </span><span class="s">"1 handful"</span><span class="p" data-group-id="8436274812-27">]</span><span class="p">,</span><span class="w">
</span><span class="ss">deps</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p" data-group-id="8436274812-28">[</span><span class="ss">boil_water</span><span class="p" data-group-id="8436274812-28">]</span><span class="w">
</span><span class="p" data-group-id="8436274812-26">}</span><span class="p">,</span><span class="w">
</span><span class="ss">cook_pasta</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p" data-group-id="8436274812-29">#{</span><span class="w">
</span><span class="ss">module</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="ss">cook_heat</span><span class="p">,</span><span class="w">
</span><span class="ss">args</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p" data-group-id="8436274812-30">[</span><span class="ss">cook</span><span class="p">,</span><span class="w"> </span><span class="ss">pasta</span><span class="p" data-group-id="8436274812-30">]</span><span class="p">,</span><span class="w">
</span><span class="ss">deps</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p" data-group-id="8436274812-31">[</span><span class="ss">add_pasta</span><span class="p" data-group-id="8436274812-31">]</span><span class="w">
</span><span class="p" data-group-id="8436274812-29">}</span><span class="p">,</span><span class="w">
</span><span class="ss">mix_pasta_with_sauce</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p" data-group-id="8436274812-32">#{</span><span class="w">
</span><span class="ss">module</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="ss">cook_combine</span><span class="p">,</span><span class="w">
</span><span class="ss">args</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p" data-group-id="8436274812-33">[</span><span class="p" data-group-id="8436274812-34">[</span><span class="ss">saucepan</span><span class="p">,</span><span class="w"> </span><span class="ss">add_pasta</span><span class="p" data-group-id="8436274812-34">]</span><span class="p">,</span><span class="w"> </span><span class="ss">saucepan</span><span class="p" data-group-id="8436274812-33">]</span><span class="p">,</span><span class="w">
</span><span class="ss">deps</span><span class="w"> </span><span class="p">=></span><span class="w"> </span><span class="p" data-group-id="8436274812-35">[</span><span class="ss">simmer_sauce</span><span class="p">,</span><span class="w"> </span><span class="ss">cook_pasta</span><span class="p" data-group-id="8436274812-35">]</span><span class="w">
</span><span class="p" data-group-id="8436274812-32">}</span><span class="w">
</span><span class="p" data-group-id="8436274812-6">}</span><span class="p">,</span><span class="w">
</span><span class="nc">wrek</span><span class="p">:</span><span class="nf">start</span><span class="p" data-group-id="8436274812-36">(</span><span class="n">Graph</span><span class="p" data-group-id="8436274812-36">)</span><span class="p">.</span></code></pre>
<p>
That’s an eyeful! What we’ve done here is created an Erlang map whose keys
represent the names of vertices in our original dependency graph, and whose
values are maps that specify a callback module, arguments to pass to the
callback module, as well as any dependencies the vertex may have. I congratulate
those of you who noticed that we are never straining the pasta; I did confess to
being a disaster in the kitchen. I promise I learned my lesson.</p>
<p>
Alright, we’re done coding! Let’s start up a shell and make some pasta!</p>
<pre><code>$ rebar3 shell
[...]
1> wrek_example:make_pasta().
adding spices. amount: to taste.
adding puréed tomatoes. amount: 1 can.
adding chopped vegetables. amount: lots.
adding pasta. amount: 1 handful.
{ok,<0.133.0>}
2> boiling water.
cooking pasta.
combining [{"puréed tomatoes","1 can"}] with [] in saucepan.
combining [{"chopped vegetables","lots"}] with [{"puréed tomatoes","1 can"}] in saucepan.
combining [{"spices","to taste"}] with [{"chopped vegetables","lots"},
{"puréed tomatoes","1 can"}] in saucepan.
saucepan now contains: [{"spices","to taste"},
{"chopped vegetables","lots"},
{"puréed tomatoes","1 can"}].
simmering sauce.
combining [{"spices","to taste"},
{"chopped vegetables","lots"},
{"puréed tomatoes","1 can"}] with [] in saucepan.
combining [{"pasta","1 handful"}] with [{"spices","to taste"},
{"chopped vegetables","lots"},
{"puréed tomatoes","1 can"}] in saucepan.
saucepan now contains: [{"pasta","1 handful"},
{"spices","to taste"},
{"chopped vegetables","lots"},
{"puréed tomatoes","1 can"}].</code></pre>
<p>
It isn’t pretty, but neither is most of my cooking. However, we’ve just used
<code class="inline">wrek</code> to teach Erlang how to concurrently prepare pasta! If you run
<code class="inline">wrek_example:make_pasta()</code> over and over, you may notice that the order of the
output changes, but never enough that the steps followed in order don’t produce
valid pasta.</p>
<h1>
Uses for wrek, additional features, and plans for the future</h1>
<p>
I believe that Wrek is useful for more than pretending to make pasta. I used
Wrek at $JOB to concurrently execute various small shell scripts, collect their
output, and pass up to Erlang through and down to other small shell scripts.</p>
<p>
Wrek also contains features that I haven’t covered in this article. For
instance, you can pass a <code class="inline">gen_event</code> process to <code class="inline">wrek:start/2</code> to get Wrek to
call <code class="inline">gen_event:notify/2</code> when various events take place. This gives users a way
to monitor the execution of their graphs. Combined with Sergey Aleynikov’s
<code class="inline">erlexec</code> library, you can collect <code class="inline">stdout</code> and <code class="inline">stderr</code> from any shell command
you run in a vertex using <code class="inline">wrek_exec:exec/4</code>. I invite interested readers to
take a peek at the <a href="https://github.com/rkallos/wrek/tree/master/src" title="">source code for Wrek</a>.</p>
<p>
Wrek is still a very young project, and I am still inexperienced at writing
libraries. I am virtually certain that the API is going to go through
significant changes in the near future. I believe this is for the best, because
the whole process of specifying callback modules and passing data between
vertices can be improved dramatically. Here are some of the ideas I have for
improving Wrek:</p>
<ul>
<li>
Add support for limiting the number of active wrek_vert processes </li>
<li>
Avoid forced namespacing by vertex names, perhaps through the use of ETS
tables as a <a href="https://en.wikipedia.org/wiki/Tuple_space" title="">tuple space</a> </li>
<li>
(experimental) Describing dependency graphs by annotating vertices by the
data they produce and consume, rather than by explicitly naming other vertices
as dependencies. </li>
</ul>
<h1>
Conclusion</h1>
<p>
I had a really nice time writing Wrek, and I’m excited to be using it in
production at Adgear. I am also eagerly looking forward to presenting Wrek at
<a href="https://codesync.global/conferences/code-beam-sf-2018/" title="">Code BEAM SF in March 2018</a>.</p>
<p>
Thank you very much for reading!</p>
<p>
P.S: As I mentioned above, this is a copy of a post that was published earlier
in the year. As a result, the above paragraph doesn’t make much sense.</p>