Jekyll2023-03-31T10:58:13+00:00https://awesome.red-badger.com//feed.xmlAwesome BadgerAssorted thinking (mostly) about technology written by the Badgers.Red BadgerHieroglyphy: Taking JavaScript Type Coercion to its Illogical Conclusion2023-03-17T12:00:00+00:002023-03-17T12:00:00+00:00https://awesome.red-badger.com//chriswhealy/hieroglyphy<h1 id="table-of-contents">Table of Contents</h1>
<ul>
<li><a href="/chriswhealy/hieroglyphy/but-why/">But Why?</a></li>
<li><a href="/chriswhealy/hieroglyphy/bootstraps/">Pulling Ourselves Up By Our Bootstraps</a></li>
<li><a href="/chriswhealy/hieroglyphy/strings/">Pulling Some Strings</a></li>
<li><a href="/chriswhealy/hieroglyphy/checkpoint1/">What Have We Achieved So Far?</a></li>
<li><a href="/chriswhealy/hieroglyphy/keywords/">Extracting Characters From Keywords</a></li>
<li><a href="/chriswhealy/hieroglyphy/numbers/">Tricks With Big Numbers</a></li>
<li><a href="/chriswhealy/hieroglyphy/checkpoint2/">So Where Are We Now?</a></li>
<li><a href="/chriswhealy/hieroglyphy/functions/">Tricks With Functions</a></li>
</ul>
<h1 id="introduction">Introduction</h1>
<p>The functionality described in this blog is neither new nor is it unique.
In this case, it is an extensive rewrite and optimisation of the original <a href="https://github.com/alcuadrado/hieroglyphy">Hieroglyphy</a> by <a href="https://github.com/alcuadrado/">Patricio Palladino</a>.</p>
<p>Here is my version of <a href="https://github.com/ChrisWhealy/hieroglyphy">Hieroglyphy</a>.</p>
<p><a href="https://github.com/aemkei/jsfuck">Other variations</a> of this style of app exist that use a minimal alphabet, but in this particular case, a close-to-minimal alphabet has been chosen.</p>
<blockquote>
<p><strong><em>WARNING</em></strong><br />
I can think of no practical reason why you would ever want to use this library in a real life situation…</p>
<p>🤪</p>
<p>But that said, the process by which it works is interesting if you really want to understand the inner workings of JavaScript’s type coercion behaviour</p>
</blockquote>
<h1 id="overview">Overview</h1>
<p>There has been some investigation into encoding the source code of a JavaScript program such that it uses a reduced alphabet, but remains syntactically valid and executable.</p>
<p>The object of the exercise here is not to create a program that remains human readable, but one that can be <code class="language-plaintext highlighter-rouge">eval</code>ed and executed.</p>
<p>For example:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">$</span> <span class="nx">node</span>
<span class="nx">Welcome</span> <span class="nx">to</span> <span class="nx">Node</span><span class="p">.</span><span class="nx">js</span> <span class="nx">v16</span><span class="p">.</span><span class="mf">12.0</span><span class="p">.</span>
<span class="nx">Type</span> <span class="dl">"</span><span class="s2">.help</span><span class="dl">"</span> <span class="k">for</span> <span class="nx">more</span> <span class="nx">information</span><span class="p">.</span>
<span class="o">></span> <span class="nb">eval</span><span class="p">(</span><span class="dl">'</span><span class="s1">(+((+!![]+[])+(!![]+!![]+!![]+!![]+!![]+!![]+!![]+[])))[(!![]+[])[+[]]+([]+{})[+!![]]+([]+([]+{})[([]+{})[!![]+!![]+!![]+!![]+!![]]+([]+{})[+!![]]+([][+[]]+[])[+!![]]+(![]+[])[!![]+!![]+!![]]+(!![]+[])[+[]]+(!![]+[])[+!![]]+([][+[]]+[])[+[]]+([]+{})[!![]+!![]+!![]+!![]+!![]]+(!![]+[])[+[]]+([]+{})[+!![]]+(!![]+[])[+!![]]])[!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![]]+(!![]+[])[+[]]+(!![]+[])[+!![]]+([][+[]]+[])[!![]+!![]+!![]+!![]+!![]]+([][+[]]+[])[+!![]]+([]+([]+{})[([]+{})[!![]+!![]+!![]+!![]+!![]]+([]+{})[+!![]]+([][+[]]+[])[+!![]]+(![]+[])[!![]+!![]+!![]]+(!![]+[])[+[]]+(!![]+[])[+!![]]+([][+[]]+[])[+[]]+([]+{})[!![]+!![]+!![]+!![]+!![]]+(!![]+[])[+[]]+([]+{})[+!![]]+(!![]+[])[+!![]]])[+((+!![]+[])+(!![]+!![]+!![]+!![]+[]))]](+((!![]+!![]+!![]+[])+(!![]+!![]+!![]+!![]+!![]+!![]+[])))+(!![]+[])[!![]+!![]+!![]]+(![]+[])[!![]+!![]]+(![]+[])[!![]+!![]]+([]+{})[+!![]]</span><span class="dl">'</span><span class="p">)</span>
<span class="dl">'</span><span class="s1">hello</span><span class="dl">'</span>
<span class="o">></span>
</code></pre></div></div>
<h1 id="room-from-improvement">Room from improvement</h1>
<p>This library is just a proof of concept; there is plenty of room for improvement:</p>
<ul>
<li>Ensure that all characters have been encoded using the shortest possible representation</li>
<li>Allow some flexibility in the alphabet size to account for different target runtime environments</li>
<li>Knowing the specific runtime would allow us to take advantage of features unique to that environment, which in turn, may yield shorter character encodings</li>
</ul>Chris WhealyJavaScript is (in)famous for being a highly dynamic language that allows a developer to write very "flexible" code. One language feature that makes a significant contribution to this flexibility is the idea of type coercion; that is, JavaScript will automatically (and silently) transform a value of one type into a value of a different type.As you can imagine however, the more you explore the language's flexibility, the higher a price you pay in terms of code legibility.Largely for the sake of amusement, this blog takes JavaScript's flexibility to the most extreme (and illogical) conclusion by providing you with an encoding library that takes a regular JavaScript program as input and returns a very long character string that is executable, functionally identical, and completely unreadable!Implementing the SHA256 Hash Algorithm in WebAssembly Text2023-02-20T12:00:00+00:002023-02-20T12:00:00+00:00https://awesome.red-badger.com//chriswhealy/sha256-webassembly<h2 id="table-of-contents">Table of Contents</h2>
<ul>
<li><a href="/chriswhealy/sha256/algorithm-overview/">SHA256 Algorithm Overview</a></li>
<li><a href="/chriswhealy/sha256/endianness/">WebAssembly Does Not Have A “raw binary” Data Type</a></li>
<li><a href="/chriswhealy/sha256/architecture/">WebAssembly Program Architecture</a></li>
<li><a href="/chriswhealy/sha256/implementation/">WebAssembly Implementation</a></li>
<li><a href="/chriswhealy/sha256/testing/">Unit Testing WebAssembly Functions</a></li>
<li><a href="/chriswhealy/sha256/host-environment/">JavaScript Host Environment</a></li>
<li><a href="/chriswhealy/sha256/summary/">Summary</a></li>
</ul>
<h2 id="development-objectives">Development Objectives</h2>
<ol>
<li>See how small a binary can be produced when the SHA256 digest algorithm is implemented directly in WebAssembly Text</li>
<li>Compare the runtime performance of the WebAssembly module with the native <code class="language-plaintext highlighter-rouge">sha256sum</code> program supplied with macOS</li>
</ol>
<p>The Git repo containing the working software can be <a href="https://github.com/ChrisWhealy/wasm_sha256">found here</a></p>
<h2 id="development-challenges">Development Challenges</h2>
<p>Two challenges had to be overcome during development:</p>
<ol>
<li>The SHA256 algorithm expects to handle data in network byte order, but WebAssembly only has numeric data types that automatically rearrange a value’s byte order according to the CPU’s endianness.</li>
<li>Unit testing WASM functions within a module is an entirely manual process.
This presented an interesting challenge - especially when writing unit tests for private WASM functions</li>
</ol>Chris WhealyWebAssembly Text (WAT) is ideally suited for implementing CPU intensive algorithms such as calculating a file's SHA256 hash. This blog describes not only how I got this algorithm working in WebAssembly Text, but takes a wider view and looks at the areas where improvements could be made in the overall developer experience of working with WAT.Quick notes on how to be a Principal Dev2022-12-16T12:00:00+00:002022-12-16T12:00:00+00:00https://awesome.red-badger.com//timlee/principal-dev-conference<h1 id="quick-notes-on-how-to-be-a-principal-dev">Quick notes on how to be a Principal Dev</h1>
<p><em><a href="../">Tim Lee</a> — 16 December 2022</em></p>
<h2 id="introduction">Introduction</h2>
<p>Short notes from a 2-day <a href="https://principal.dev/">Principal Dev</a> training I attended, lead by Eduards Sizovs.</p>
<p>Full course slides can be found <a href="https://sizovs.net/principal/#1">here</a> (arrow to navigate).</p>
<h2 id="short-notes">Short notes</h2>
<h3 id="agile">Agile</h3>
<ul>
<li>Agile manifesto says “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software”. But customers don’t <em>need</em> software, they need cost-efficient solutions to their problems. So perhaps it should be reworded to: “Our highest priority is to find cost-efficient solutions to customers’ problems”</li>
<li>Process of finding the cost-efficient solution from a list of options is called Cost-Benefit Analysis (CBA). Parameters to consider - speed, cost, benefit, risk</li>
<li>Disney brainstorming method good for sourcing ideas as a group
<ol>
<li>Phase 1: gather all ideas, acting as “Dreamers”</li>
<li>Phase 2: assess ideas as “Realists” looking for limitations</li>
<li>Phase 3: analyze ideas as “Critics”, addressing possible risks</li>
</ol>
</li>
<li>Don’t do what a customer says they want, do what a customer needs (cost-efficiently). Should solve the problem, not the want.</li>
<li>Important not to shield developers from the business, because software is the implementation <strong>of</strong> the business. Shielding leads to low trust, low motivation and mediocre software solutions.
<ul>
<li>To find cost-efficient solutions, you need a whole-team approach.</li>
<li>To write great code, you need a deep understanding of the underlying problem. (software is the implementation of the problem domain. Weak problem understanding -> weak solutions)</li>
<li>Understanding the business domain turns engineers into business partners</li>
<li>The team owns the product, not product owners. Product ~owners~ leaders promote product understanding.</li>
<li>Make impacts, not software</li>
</ul>
</li>
</ul>
<h3 id="leading">Leading</h3>
<ul>
<li>As a leader your job depends on seeing things others don’t see. Can’t make yourself always busy - need head space to observe.</li>
<li>Should switch focus to high-leverage activities (HLAs) - the activities with the highest return on your time investment. Leverage = impact / effort</li>
<li>Your output is the output of your team. Being busy is not the same as being productive. So focus on HLAs (Pareto principle - 20% of activities give 80% of result). To notice HLAs, don’t overwhelm yourself with Low Level Activities (LLAs).</li>
<li>To get rid of LLAs: Delete -> Defer -> Delegate -> Do</li>
<li>Removing impediments is good, but empowering the team to see, prioritise and remove it’s own impediments is better.</li>
</ul>
<h3 id="measuring-dev-teams-performance">Measuring dev team’s performance</h3>
<ul>
<li>Lead time - time taken to solve a problem, or time a problem stays “in progress”
<ul>
<li>Can be reduced with cost efficient solutions, good architecture, full-cycle teams (autonomous, cross-functional, self-organising)</li>
</ul>
</li>
<li>Defects</li>
<li>DDP (due date performance) - ability to deliver on time</li>
<li>Reliability
<ul>
<li>MTBF (mean time between failures), MTTD (mean time to detect), MTTR (mean time to remediate)</li>
</ul>
</li>
<li>ROI - what can you offer that the market can’t? Cost-efficient solutions, cost reduction, innovation, leadership, mentoring, visibility, hiring</li>
<li>Satisfaction - of customers and business
<ul>
<li>Ask 3 questions:
<ol>
<li>Are you happy with my work?</li>
<li>What do I suck at?</li>
<li>What can I do to improve?</li>
</ol>
</li>
</ul>
</li>
</ul>
<h3 id="throughput">Throughput</h3>
<ul>
<li>Little’s law - lead time = work in progress / throughput</li>
<li>Reducing WIP leads to work being delivered more quickly</li>
<li>Focus as many people as possible on as few projects/stories/tasks as possible - swarming.</li>
<li>4 ways to increase throughput:
<ol>
<li>Hire more people - though law of diminishing returns (adding more people beyond certain threshold is less efficient). Also Brooks law - adding manpower to late project makes it even later. Small teams tend to do better than large ones - more flexible, aligned, involved, engaged, better relationships, low management overhead. If need more people, continuously refactor company into a network of small, independent, full-cycle teams</li>
<li>High-performance teams - aligned, small, diverse, full-cycle, stable (to achieve Tuckman’s stages of group development - forming, storming, norming, performing, takes about a year to get to last stage), skilled</li>
<li>Technical excellence - engineers learn by example from seniors, so choose seniors and leaders carefully. Make sure they promote software craftmanship, clean code, extreme programming, TDD. Make mentoring a prerequisite for promotion.</li>
<li>Reduce waste, which includes
<ol>
<li>Partially done work</li>
<li>Overproduction</li>
<li>Waiting/handoffs</li>
<li>Rework</li>
<li>Non-value added activities</li>
</ol>
</li>
</ol>
</li>
</ul>
<h3 id="tips-for-delivering-on-promises-namely-within-scrum">Tips for delivering on promises (namely, within Scrum)</h3>
<ul>
<li>Know your velocity (work delivered without cutting corners)</li>
<li>Even under pressure, don’t take more work than allowed by your velocity</li>
<li>Remove planned work when unplanned work appears</li>
<li>Minimise unplanned work</li>
<li>Estimate with Fibonnaci numbers to gain speed at cost of accuracy</li>
<li>Play planning poker until all team members estimates match to reduce HIPPO pressure</li>
<li>When estimation is difficult or the problem is too big - split or spike to reduce unknowns</li>
<li>Estimates are innacruate - comfortable velocity must include a buffer</li>
<li>Turn on swarming for speed and continuous flow, monitor flow to notice issues</li>
<li>Keep tech debt under control</li>
<li>Process of estimation is more important than estimates - learn about business, understand problems better, cut scope, slice work for better flow, raise inconvenient questions like why estimates are inflated…</li>
</ul>
<h3 id="tech-debt">Tech debt</h3>
<ul>
<li>There is a strong correlation between the quality of the codebase and throughput.</li>
<li>A delivery team tries to maintain it’s velocity over time.</li>
<li>Degredation of the codebase accelerates with time due to reinforcing loops - broken window theory, Brook’s law, Lehman’s law (entropy), Gresham’s law (bad code drives out good - you want to touch nice code, you don’t want to touch bad code so they stay longer and sometimes have abstractions written around them).</li>
<li>Way to achieve sustainable pace/continuous improvement - every time you touch code try and make it slightly better, never worse than before (the Boyscout rule)</li>
<li>Currency of technical debt is throughput - you borrow throughput and pay it back. It compounds, so the longer it takes to pay back, the worse it gets. Things get built on top of it. So want to pay technical debt early.</li>
<li>Tech debt is a tool for short term gain.</li>
<li>Explain the cost of tech debt when making the trade off so there’s an understanding that it will either slow down future delivery or be paid back soon.</li>
<li>Eliminating the cause of technical debt is more important than eliminating the technical debt.</li>
<li>4 main causes of tech debt:
<ol>
<li>Knowledge or skills problem - strong technical leadership/mentoring needed</li>
<li>Attitude problem - “I don’t have time for quality”</li>
<li>Borrowing</li>
<li>Learning - knowing more at a later date, finding a better way to do it. So important to attract, develop, retain tech/domain knowledge, along with KISS, YAGNI, validate before building</li>
</ol>
</li>
</ul>
<h3 id="mentoring">Mentoring</h3>
<ul>
<li>The learning pyramid goes from least effective to most effective - watching lectures -> reading -> audio/visual -> demonstration -> group discussion -> practicing by doing -> teaching others</li>
<li>Pyramid moves from passive learning method to active learning methods</li>
<li>Teaching others is most effective, notably even more than by doing, which is why mentoring is crucial</li>
<li>4 stages of learning:
<ol>
<li>Accumulate knowledge (read, hear, practice)</li>
<li>Teach, explain (code review, pairing, writing, presenting)</li>
<li>Get feedback</li>
<li>Deepen understanding (and improve how well you can articulate an idea)</li>
</ol>
</li>
<li>If you can’t explain it, you don’t understand it</li>
</ul>
<h3 id="high-performing-teams">High performing teams</h3>
<ul>
<li>Bus factor - make yourself replaceable. The more influential your role, the more you have to care about succession planning. Leaders grow leaders.</li>
<li>Team performance = sum of performance of each individual x teamwork / wastes</li>
<li>The wider the skill gap between you and your teammates, the more you have to mentor</li>
<li>A team needs T-shape people, ensure people have some capability in all areas</li>
<li>Retention more important than hiring - retain domain knowledge, benefit from growing them, don’t have cost of hiring</li>
<li>2 types of motivation
<ol>
<li>Intrinsic - driven because you find it rewarding, performing an activity for its own sake, the behaviour itself is its own reward</li>
<li>Extrinsic motivation - driven to earn a reward, get praise or avoid punishment. Do something because expect to get something in exchange</li>
</ol>
</li>
<li>Intrinsic motivation is the optimum, more sustainable driver.</li>
<li>Primary motivators:
<ol>
<li>Autonomy - freedom to make decisions, create workspace, choose what to work on. Not complete freedom, especially more junior members - not earned trust. More senior = more autonomy - more trust, more expertise.</li>
<li>Mastery - becoming more capable, learning new skills</li>
<li>Purpose - aligning work with personal goals</li>
</ol>
</li>
</ul>
<h3 id="recruiting">Recruiting</h3>
<ul>
<li>Important to improve quality of inbound traffic</li>
<li>Be visible, otherwise people will come to the interview just to find out who you are, waste of time</li>
<li>Write ads that attract good candidates (with right attitude but lacking certain skills, or with imposter syndrome) and drives away bad candidates (right skills but lacking attitude)</li>
<li>Perceived ability vs Actual ability graph. Imposter syndrome - high actual ability, low perceived ability. Dunning-Kruger - high perceived ability, low actual ability. A lot of job ads attracts Dunning-Kruger’s. Instead of “deep understanding”, “good knowledge of”, “+3 years of experience”, describe the environment, technology and what the person will be doing, what you tend to favour (pair programming, etc) and that it doesn’t matter if you don’t know something because they offer mentoring. Show job ads to as many colleageus as possible, not just the hiring manager/recruiter having written. textio.com - accessibility language checker</li>
<li>Beat expectations
<ul>
<li>People should walk out of a conversation smarter, happier, energized (even if they don’t make the hire)</li>
<li>Invite for a conversation, not an . Run as a conversation instead of a conversation.</li>
<li>Start with a presentation</li>
<li>Inspire with gift of books in the interview that are relevant - e.g. Clean Code</li>
</ul>
</li>
<li>Let candidates code in their comfortable environment
<ul>
<li>The Hawthorne effect (Thinking, fast and slow) - when in stressful situation System 1 dominates and can’t think</li>
<li>Homework is most inclusive way to assess tech skills</li>
<li>Compact but representative</li>
<li>Owe the candidate a code review (good parts, bad parts, suggestions, books)</li>
</ul>
</li>
</ul>
<h3 id="reciprocity">Reciprocity</h3>
<ul>
<li>Reciprocity is a social norm of responding to a positive action with another positive action, rewarding kind actions. If you invest in someone, they’ll invest in you - invest wisely</li>
<li>Important for:
<ol>
<li>Authority - brings influence</li>
<li>Consistency - basic social contract, do what you say</li>
<li>Liking - if people don’t like you, they’ll object to your ideas even if they’re rational</li>
<li>Scarcity - things that are available for everyone have lower perceived value. Things that are not available for everyone have higher perceived value. If you have skills that are more in demand, they’ll be more valuable.</li>
<li>Social proof - shortcut to make decisions based on things like ratings, reviews, recommendations, etc</li>
</ol>
</li>
</ul>
<h3 id="management-toolbox">Management toolbox</h3>
<ul>
<li>Encourage - help people fight self-doubts
<ul>
<li>#1 reason people leave their role is lack of appreciation - just say thanks is not good, say thank you, “name” for doing “x”</li>
<li>Support initiative</li>
<li>Behind every request is an unfulfilled desire or need - ask why if don’t understand, try to get to bottom of need</li>
</ul>
</li>
<li>Challenge - delegate technical and non-technical work
<ul>
<li>Delegate tasks that maximize learning (trade-off with speed)</li>
</ul>
</li>
<li>Answer “how can I do X or Y” with “what options do you see?”</li>
<li>Share:
<ul>
<li>Knowledge</li>
<li>Energy - should know the source of your energy, understand what energises you</li>
</ul>
</li>
<li>Set rules
<ul>
<li>Less control, more systems</li>
<li>e.g. WIP limits, YBIYRI (You Build It, You Run It), CI with daily push into the , QA should find nothing (build quality in), every time new pair</li>
</ul>
</li>
<li>What skills do my team-mates need the most?
<ul>
<li>At the intersection of personal career goals and team needs</li>
</ul>
</li>
<li>Trust - product of:
<ul>
<li>Your knowledge</li>
<li>Your consistency</li>
<li>Your authority</li>
<li>Your confidence</li>
<li>Your charisma - energy, body language, voice, words</li>
<li>Read: Nonviolent communication</li>
<li>Your Emotional Intelligence (EQ) - managing thoughts, emotions, mind, ability to connect with others</li>
</ul>
</li>
</ul>
<h3 id="answering-the-question---am-i-a-good-leader">Answering the question - am I a good leader?</h3>
<ul>
<li>Is your team successful?</li>
<li>Is each individual in your team successful?</li>
<li>Is your work appreciated by others?</li>
<li>Is your team reporting high scores in weekly health checks (Spotify strategy - R/G/B scored against categories)?</li>
<li>Is employee turnover low?</li>
<li>Are people lining up to work with you?</li>
</ul>Tim LeeDo you want to be a principal dev? Look no further. These notes will sort you right out.WebAssembly Memory Growth and the Detached ArrayBuffer Problem2022-10-05T12:00:00+00:002022-10-05T12:00:00+00:00https://awesome.red-badger.com//chriswhealy/memory-grow-and-arraybuffers<h2 id="context">Context</h2>
<ul>
<li>A WebAssembly module and its host environment can share a block of linear memory.</li>
<li>If JavaScript acts as the host environment, then shared memory appears as a JavaScript <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/ArrayBuffer"><code class="language-plaintext highlighter-rouge">ArrayBuffer</code></a>.</li>
<li>JavaScript cannot directly manipulate the contents of an <code class="language-plaintext highlighter-rouge">ArrayBuffer</code>.
Instead, it must use some sort of overlay or mask such as a <code class="language-plaintext highlighter-rouge">Uint8Array</code> or a <code class="language-plaintext highlighter-rouge">Uint32Array</code>.
Then the data in the <code class="language-plaintext highlighter-rouge">ArrayBuffer</code> can be accessed using the overlaid structure’s semantics.</li>
</ul>
<h2 id="problem-summary">Problem Summary</h2>
<p>A collision between these two facts creates the “Detached ArrayBuffer” problem:</p>
<ul>
<li>JavaScript <code class="language-plaintext highlighter-rouge">ArrayBuffer</code>s are of fixed-length and once allocated, cannot be extended.</li>
<li>WebAssembly linear memory can be extended by calling <a href="https://webassembly.github.io/spec/core/syntax/instructions.html#syntax-instr-memory"><code class="language-plaintext highlighter-rouge">memory.grow</code></a>.</li>
</ul>
<p>If WebAssembly memory grows,<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup> then the old JavaScript <code class="language-plaintext highlighter-rouge">ArrayBuffer</code> must be thrown away and a new one created.
Consequently, any JavaScript objects that used to overlay the old <code class="language-plaintext highlighter-rouge">ArrayBuffer</code> are immediately invalidated because they now point to nothing.
The floor has literally been pulled out from underneath these objects and they must all be redefined over top of the new <code class="language-plaintext highlighter-rouge">ArrayBuffer</code>.</p>
<p>There is a <a href="https://www.proposals.es/proposals/Resizable%20and%20growable%20ArrayBuffers">proposal</a> to allow a JavaScript <code class="language-plaintext highlighter-rouge">ArrayBuffer</code> to grow, and as soon as this functionality appears, this problem will disappear.</p>
<p>Meanwhile, back in Gotham City…</p>
<h2 id="what-consequences-do-these-facts-create-when-writing-in-rust">What Consequences Do These Facts Create When Writing In Rust?</h2>
<p>When writing a Rust program that you intend to distribute as a WebAssembly module, <code class="language-plaintext highlighter-rouge">cargo</code> knows that memory growth might be required; therefore, it builds the necessary functions into the WebAssembly module for calling <code class="language-plaintext highlighter-rouge">memory.grow</code>.
Should it be necessary, memory growth will now happen automatically (and silently!)</p>
<p>The consequences for JavaScript are that its shared memory <code class="language-plaintext highlighter-rouge">ArrayBuffer</code> now points to a completely new block of memory and all the overlay objects that gave access to the “pre-growth” shared memory are no longer usable (I.E. they are said to have become “detached”).</p>
<p>If you then attempt to access shared memory using one of these “pre-growth” objects, you will see an error such as this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>TypeError: Cannot perform %TypedArray%.prototype.slice on a detached ArrayBuffer
</code></pre></div></div>
<blockquote>
<h3 id="aside">Aside</h3>
<p>Before you can compile a Rust program to WebAssembly, you must first install the <code class="language-plaintext highlighter-rouge">wasm32</code> compilation target:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rustup target add wasm32-unknown-unknown
</code></pre></div> </div>
</blockquote>
<h2 id="local-execution">Local Execution</h2>
<p>The following trivial application demonstrates this problem.</p>
<p>A WebAssembly program shares a block of memory with its host for the pourposes of data exchange.
The host writes data to known locations in memory, then the WebAssembly program processes it and writes its response back at another known location.</p>
<blockquote>
<h3 id="source-code">Source Code</h3>
<p>All the source code referenced by this blog can be found in the Github repository <a href="https://github.com/ChrisWhealy/detached_arraybuffer"><code class="language-plaintext highlighter-rouge">detached_arraybuffer</code></a>.</p>
<p>If you wish to run these tests locally, first clone this repo:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone git@github.com:ChrisWhealy/detached_arraybuffer.git
</code></pre></div> </div>
</blockquote>
<h3 id="first-generate-the-webassembly-module">First, Generate the WebAssembly Module</h3>
<p>Testing can be performed using different versions of the Wasm module. One version will work because it does not perform memory growth, and the other will break because it does:</p>
<ol>
<li>
<p>Compile a <a href="https://github.com/ChrisWhealy/detached_arraybuffer/blob/master/memoryguest.wat">working version</a> from source code written in WebAssembly Text.</p>
<p>This version works simply because the WebAssembly Text source code was hand-written, and no such calls to <code class="language-plaintext highlighter-rouge">memory.grow</code> were implemented. To use this version, run</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wat2wasm memoryguest.wat
</code></pre></div> </div>
</li>
<li>
<p>Compile a <a href="https://github.com/ChrisWhealy/detached_arraybuffer/blob/master/src/lib_growth.rs">broken version</a> from source code written in Rust</p>
<p>To use this version:</p>
<ul>
<li>Rename <code class="language-plaintext highlighter-rouge">./src/lib_growth.rs</code> to <code class="language-plaintext highlighter-rouge">./src/lib.rs</code></li>
<li>Run <code class="language-plaintext highlighter-rouge">cargo build --target=wasm32-unknown-unknown</code></li>
</ul>
</li>
<li>
<p>Compile a <a href="https://github.com/ChrisWhealy/detached_arraybuffer/blob/master/src/lib_no_growth.rs">working version</a> from source code written in Rust that explicitly avoids the need for memory growth</p>
<p>To use this version:</p>
<ul>
<li>Rename <code class="language-plaintext highlighter-rouge">./src/lib_no_growth.rs</code> to <code class="language-plaintext highlighter-rouge">./src/lib.rs</code></li>
<li>Run <code class="language-plaintext highlighter-rouge">cargo build --target=wasm32-unknown-unknown</code></li>
</ul>
</li>
</ol>
<h3 id="test-the-wasm-module-by-calling-it-from-javascript">Test The Wasm Module By Calling It From JavaScript</h3>
<p>The effects of WebAssembly memory growth on JavaScript’s shared memory <code class="language-plaintext highlighter-rouge">ArrayBuffer</code> can be demonstrated as follows:</p>
<ol>
<li>In both <a href="https://github.com/ChrisWhealy/detached_arraybuffer/blob/master/server.js"><code class="language-plaintext highlighter-rouge">server.js</code></a> and <a href="https://github.com/ChrisWhealy/detached_arraybuffer/blob/master/client.js"><code class="language-plaintext highlighter-rouge">client.js</code></a>, ensure that the variable <code class="language-plaintext highlighter-rouge">wasmFilePath</code> points to the particular Wasm module you wish to test.</li>
<li>
<p>To test the Wasm module server side, run</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>node server.js
</code></pre></div> </div>
</li>
<li>
<p>To test the Wasm module in a browser:</p>
<ul>
<li>
<p>Start a temporary Web Server</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> python3 <span class="nt">-m</span> http.server 8080
</code></pre></div> </div>
</li>
<li>Point your browser to <a href="http://localhost:8080">http://localhost:8080</a></li>
<li>Open the developer console</li>
</ul>
</li>
</ol>
<p>When the test succeeds, the console will display</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Ahoy there, Testy McTestface!
</code></pre></div></div>
<p>When the test fails, the console will show the Type Error shown above.</p>
<h2 id="implementation">Implementation</h2>
<p>The map of shared memory looks like this:</p>
<table>
<thead>
<tr>
<th style="text-align: right">Offset</th>
<th>Contains</th>
<th>Offset returned by Wasm function</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: right">0</td>
<td>Salutation</td>
<td><code class="language-plaintext highlighter-rouge">get_salutation_ptr</code></td>
</tr>
<tr>
<td style="text-align: right">16</td>
<td>Name</td>
<td><code class="language-plaintext highlighter-rouge">get_name_ptr</code></td>
</tr>
<tr>
<td style="text-align: right">32</td>
<td>Formatted greeting</td>
<td><code class="language-plaintext highlighter-rouge">get_msg_ptr</code></td>
</tr>
</tbody>
</table>
<p>The JavaScript program must first obtain the values of the memory locations shown above.
Once it has these, it writes the appropriate strings to those locations.</p>
<p>Next, it calls the Wasm function <code class="language-plaintext highlighter-rouge">set_name</code> which does the following:</p>
<ul>
<li>Combines the salutation and name into a greeting</li>
<li>Writes that greeting to another known memory location</li>
<li>Returns the length of the formatted greeting</li>
</ul>
<p>Finally, the JavaScript program reads the greeting from shared memory and writes it to the console.</p>
<h2 id="but-what-caused-memory-growth">But What Caused Memory Growth?</h2>
<p>Look at the Rust coding in <a href="https://github.com/ChrisWhealy/detached_arraybuffer/blob/master/src/lib_growth.rs">./src/lib_growth.rs</a>.
Within function <code class="language-plaintext highlighter-rouge">set_name</code>, the <code class="language-plaintext highlighter-rouge">format!()</code> macro is used to assemble the result, which is then stored in an intermediate <code class="language-plaintext highlighter-rouge">String</code> called <code class="language-plaintext highlighter-rouge">greeting</code>.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[no_mangle]</span>
<span class="k">pub</span> <span class="k">unsafe</span> <span class="k">extern</span> <span class="s">"C"</span> <span class="k">fn</span> <span class="nf">set_name</span><span class="p">(</span><span class="n">sal_len</span><span class="p">:</span> <span class="nb">i32</span><span class="p">,</span> <span class="n">name_len</span><span class="p">:</span> <span class="nb">i32</span><span class="p">)</span> <span class="k">-></span> <span class="nb">i32</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">sal</span><span class="p">:</span> <span class="o">&</span><span class="nb">str</span> <span class="o">=</span> <span class="nf">str_from_buffer</span><span class="p">(</span><span class="n">SALUT_OFFSET</span><span class="p">,</span> <span class="n">sal_len</span> <span class="k">as</span> <span class="nb">usize</span><span class="p">);</span>
<span class="k">let</span> <span class="n">name</span><span class="p">:</span> <span class="o">&</span><span class="nb">str</span> <span class="o">=</span> <span class="nf">str_from_buffer</span><span class="p">(</span><span class="n">NAME_OFFSET</span><span class="p">,</span> <span class="n">name_len</span> <span class="k">as</span> <span class="nb">usize</span><span class="p">);</span>
<span class="k">let</span> <span class="n">greeting</span><span class="p">:</span> <span class="nb">String</span> <span class="o">=</span> <span class="nd">format!</span><span class="p">(</span><span class="s">"{}, {}!"</span><span class="p">,</span> <span class="n">sal</span><span class="p">,</span> <span class="n">name</span><span class="p">);</span>
<span class="c">// snip...</span>
</code></pre></div></div>
<p>Well that looks harmless enough…</p>
<p>However, the declaration of the new <code class="language-plaintext highlighter-rouge">String</code> requires more memory than is currently available; so, using the extra functions generated by <code class="language-plaintext highlighter-rouge">cargo</code>, shared memory is automatically and silently extended.</p>
<p>As far as Rust (WebAssembly) is concerned, everything is fine; however, the JavaScript host environment sees that shared memory has changed size, so it throws away the old <code class="language-plaintext highlighter-rouge">ArrayBuffer</code> and helpfully creates you a new one.</p>
<p>And now all your “pre-growth” JavaScript references into WebAssembly’s shared memory are broken…</p>
<h2 id="calling-the-broken-code-from-javascript">Calling The Broken Code From JavaScript</h2>
<p>Look at <a href="https://github.com/ChrisWhealy/detached_arraybuffer/blob/master/server.js">./server.js</a> to see the full context of this coding.</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">salutation</span> <span class="o">=</span> <span class="dl">"</span><span class="s2">Ahoy there</span><span class="dl">"</span>
<span class="kd">const</span> <span class="nx">name</span> <span class="o">=</span> <span class="dl">"</span><span class="s2">Testy McTestface</span><span class="dl">"</span>
<span class="c1">// Treat shared memory as an array of unsigned bytes</span>
<span class="kd">const</span> <span class="nx">mem8</span> <span class="o">=</span> <span class="k">new</span> <span class="nb">Uint8Array</span><span class="p">(</span><span class="nx">wasmExports</span><span class="p">.</span><span class="nx">memory</span><span class="p">.</span><span class="nx">buffer</span><span class="p">)</span>
<span class="c1">// Fetch long-lived pointers</span>
<span class="kd">const</span> <span class="nx">sal_ptr</span> <span class="o">=</span> <span class="nx">wasmExports</span><span class="p">.</span><span class="nx">get_salutation_ptr</span><span class="p">()</span>
<span class="kd">const</span> <span class="nx">name_ptr</span> <span class="o">=</span> <span class="nx">wasmExports</span><span class="p">.</span><span class="nx">get_name_ptr</span><span class="p">()</span>
<span class="kd">const</span> <span class="nx">msg_ptr</span> <span class="o">=</span> <span class="nx">wasmExports</span><span class="p">.</span><span class="nx">get_msg_ptr</span><span class="p">()</span>
<span class="c1">// Store salutation and name at the expected locations</span>
<span class="nx">mem8</span><span class="p">.</span><span class="kd">set</span><span class="p">(</span><span class="nx">stringToAsciiArray</span><span class="p">(</span><span class="nx">salutation</span><span class="p">),</span> <span class="nx">sal_ptr</span><span class="p">)</span>
<span class="nx">mem8</span><span class="p">.</span><span class="kd">set</span><span class="p">(</span><span class="nx">stringToAsciiArray</span><span class="p">(</span><span class="nx">name</span><span class="p">),</span> <span class="nx">name_ptr</span><span class="p">)</span>
<span class="c1">// Tell Wasm to write the formatted greeting to the known memory location then return its length</span>
<span class="kd">let</span> <span class="nx">msg_len</span> <span class="o">=</span> <span class="nx">wasmExports</span><span class="p">.</span><span class="nx">set_name</span><span class="p">(</span><span class="nx">salutation</span><span class="p">.</span><span class="nx">length</span><span class="p">,</span> <span class="nx">name</span><span class="p">.</span><span class="nx">length</span><span class="p">)</span>
<span class="c1">// Read greeting from shared memory</span>
<span class="kd">let</span> <span class="nx">msg_text</span> <span class="o">=</span> <span class="nx">asciiArrayToString</span><span class="p">(</span><span class="nx">mem8</span><span class="p">.</span><span class="nx">slice</span><span class="p">(</span><span class="nx">msg_ptr</span><span class="p">,</span> <span class="nx">msg_ptr</span> <span class="o">+</span> <span class="nx">msg_len</span><span class="p">))</span>
<span class="c1">// ^^^^^^^^^^ mem8 will point to nothing if memory growth occurs!</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">msg_text</span><span class="p">)</span>
</code></pre></div></div>
<p>So let’s run this.</p>
<p>If you’re using the working WebAssembly module, you’ll see:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>node server.js
Ahoy there, Testy McTestface!
</code></pre></div></div>
<p>and if you’re using the WebAssembly module that breaks JavaScript’s shared memory references, you’ll see:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>node server.js
/Users/chris/Developer/WebAssembly/detached_arraybuffer/server.js:60
<span class="nb">let </span>msg_text <span class="o">=</span> asciiArrayToString<span class="o">(</span>mem8.slice<span class="o">(</span>msg_ptr, msg_ptr + msg_len<span class="o">))</span>
^
TypeError: Cannot perform %TypedArray%.prototype.slice on a detached ArrayBuffer
at Uint8Array.slice <span class="o">(</span><anonymous><span class="o">)</span>
at /Users/chris/Developer/WebAssembly/detached_arraybuffer/server.js:60:44
</code></pre></div></div>
<h1 id="two-solutions">Two Solutions</h1>
<p>Until JavaScript’s <code class="language-plaintext highlighter-rouge">ArrayBuffer</code> is able to perform in-place growth, we must adopt one of two possible approaches to solving this problem.
Either:</p>
<ol>
<li>We monitor the size of the WebAssembly memory looking for growth; or</li>
<li>We adjust the Rust coding so that memory growth does not occur.</li>
</ol>
<h2 id="1-a-javascript-workaround">1. A JavaScript Workaround</h2>
<p>If it’s going to change, WebAssembly memory will only every increase in size.
So a simple way to workaround this problem is to monitor the size of the WebAssembly’s memory.</p>
<p>If it gets bigger, then you know you need to redefine any shared memory overlay objects.</p>
<blockquote>
<p>This is just a workaround; it does not change the underlying problem.</p>
<p>Anyone else calling the same WebAssembly function will need to implement the same workaround.</p>
</blockquote>
<p>The code does not require much modification to avoid using a possibly detached <code class="language-plaintext highlighter-rouge">ArrayBuffer</code>:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">salutation</span> <span class="o">=</span> <span class="dl">"</span><span class="s2">Ahoy there</span><span class="dl">"</span>
<span class="kd">const</span> <span class="nx">name</span> <span class="o">=</span> <span class="dl">"</span><span class="s2">Testy McTestface</span><span class="dl">"</span>
<span class="c1">// Keep track of Wasm's shared memory size</span>
<span class="kd">let</span> <span class="nx">memLength</span> <span class="o">=</span> <span class="nx">wasmExports</span><span class="p">.</span><span class="nx">memory</span><span class="p">.</span><span class="nx">buffer</span><span class="p">.</span><span class="nx">byteLength</span>
<span class="c1">// Snip</span>
<span class="c1">// Tell Wasm to write the formatted greeting to the known memory location then return its length</span>
<span class="kd">let</span> <span class="nx">msg_len</span> <span class="o">=</span> <span class="nx">wasmExports</span><span class="p">.</span><span class="nx">set_name</span><span class="p">(</span><span class="nx">salutation</span><span class="p">.</span><span class="nx">length</span><span class="p">,</span> <span class="nx">name</span><span class="p">.</span><span class="nx">length</span><span class="p">)</span>
<span class="c1">// Before allowing shared memory access, check if memory growth has occurred</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">wasmExports</span><span class="p">.</span><span class="nx">memory</span><span class="p">.</span><span class="nx">buffer</span><span class="p">.</span><span class="nx">byteLength</span> <span class="o">></span> <span class="nx">memLength</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">memLength</span> <span class="o">=</span> <span class="nx">wasmExports</span><span class="p">.</span><span class="nx">memory</span><span class="p">.</span><span class="nx">buffer</span><span class="p">.</span><span class="nx">byteLength</span>
<span class="nx">mem8</span> <span class="o">=</span> <span class="k">new</span> <span class="nb">Uint8Array</span><span class="p">(</span><span class="nx">wasmExports</span><span class="p">.</span><span class="nx">memory</span><span class="p">.</span><span class="nx">buffer</span><span class="p">)</span>
<span class="p">}</span>
<span class="c1">// Read greeting from shared memory</span>
<span class="kd">let</span> <span class="nx">msg_text</span> <span class="o">=</span> <span class="nx">asciiArrayToString</span><span class="p">(</span><span class="nx">mem8</span><span class="p">.</span><span class="nx">slice</span><span class="p">(</span><span class="nx">msg_ptr</span><span class="p">,</span> <span class="nx">msg_ptr</span> <span class="o">+</span> <span class="nx">msg_len</span><span class="p">))</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">msg_text</span><span class="p">)</span>
</code></pre></div></div>
<p>Now everything works because we’re on the lookout for memory growth and then “reattach” the <code class="language-plaintext highlighter-rouge">mem8</code> array to the new shared memory <code class="language-plaintext highlighter-rouge">ArrayBuffer</code>.</p>
<h2 id="2-solve-the-problem-in-rust">2. Solve the Problem in Rust</h2>
<p>However, to avoid causing inadvertent memory growth, the Rust coding needs to avoid invoking any instructions that might require extra memory.
In this case, it means that instead of using an intermediate <code class="language-plaintext highlighter-rouge">String</code> object, we write the bytes of the character strings directly to the <code class="language-plaintext highlighter-rouge">[u8]</code> buffer.</p>
<p>The full solution can be seen in <a href="https://github.com/ChrisWhealy/detached_arraybuffer/blob/master/src/lib_no_growth.rs">./src/lib_no_growth.rs</a>, but the important change is shown below:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">pub</span> <span class="k">unsafe</span> <span class="k">extern</span> <span class="s">"C"</span> <span class="k">fn</span> <span class="nf">set_name</span><span class="p">(</span><span class="n">sal_len</span><span class="p">:</span> <span class="nb">i32</span><span class="p">,</span> <span class="n">name_len</span><span class="p">:</span> <span class="nb">i32</span><span class="p">)</span> <span class="k">-></span> <span class="nb">i32</span> <span class="p">{</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">idx</span><span class="p">:</span> <span class="nb">usize</span><span class="p">;</span>
<span class="c">// Write salutation directly to the buffer</span>
<span class="nf">copy_bytes</span><span class="p">(</span><span class="n">MSG_OFFSET</span><span class="p">,</span> <span class="n">SALUT_OFFSET</span><span class="p">,</span> <span class="n">sal_len</span><span class="p">);</span>
<span class="n">idx</span> <span class="o">=</span> <span class="n">MSG_OFFSET</span> <span class="o">+</span> <span class="n">sal_len</span> <span class="k">as</span> <span class="nb">usize</span><span class="p">;</span>
<span class="c">// Write separator ", "</span>
<span class="n">BUFFER</span><span class="p">[</span><span class="n">idx</span><span class="p">]</span> <span class="o">=</span> <span class="n">COMMA</span><span class="p">;</span>
<span class="n">idx</span> <span class="o">+=</span> <span class="mi">1</span><span class="p">;</span>
<span class="n">BUFFER</span><span class="p">[</span><span class="n">idx</span><span class="p">]</span> <span class="o">=</span> <span class="n">SPACE</span><span class="p">;</span>
<span class="n">idx</span> <span class="o">+=</span> <span class="mi">1</span><span class="p">;</span>
<span class="c">// Write name</span>
<span class="nf">copy_bytes</span><span class="p">(</span><span class="n">idx</span><span class="p">,</span> <span class="n">NAME_OFFSET</span><span class="p">,</span> <span class="n">name_len</span><span class="p">);</span>
<span class="n">idx</span> <span class="o">+=</span> <span class="n">name_len</span> <span class="k">as</span> <span class="nb">usize</span><span class="p">;</span>
<span class="c">// Write bang character</span>
<span class="n">BUFFER</span><span class="p">[</span><span class="n">idx</span><span class="p">]</span> <span class="o">=</span> <span class="n">BANG</span><span class="p">;</span>
<span class="n">idx</span> <span class="o">+=</span> <span class="mi">1</span><span class="p">;</span>
<span class="p">(</span><span class="n">idx</span> <span class="o">-</span> <span class="n">MSG_OFFSET</span><span class="p">)</span> <span class="k">as</span> <span class="nb">i32</span>
<span class="p">}</span>
</code></pre></div></div>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>memory growth could be invoked either from WebAssembly or the host environment <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Chris WhealyWhen JavaScript acts as the host environment for WebAssembly, shared memory is visible to JavaScript as an ArrayBuffer. WebAssembly memory is allowed to grow, but JavaScript ArrayBuffers are not; so what happens when your Rust program (compiled to WebAssembly) asks for more memory?Human-centered Code Reviews2022-06-22T12:00:00+00:002022-06-22T12:00:00+00:00https://awesome.red-badger.com//niall-rb/human-centered-code-reviews<p>For the uninitiated a code review is basically when a developer presents their code to another for comments and feedback. If the code passes the review it is typically merged or if it doesn’t then the submitter has some changes to make before submitting it for review again. They can come at the end when the feature is complete or more commonly will have multiple reviews along the way.</p>
<p>In all but the smallest of teams code reviews are a commonplace practice. The primary purpose of them is to ensure that the code is of good quality and meets the team’s technical standards of fitness.</p>
<p>Some of the things a reviewer could look for could include:</p>
<ul>
<li>Proper grammar and syntax</li>
<li>Efficient algorithms</li>
<li>No duplicate code</li>
<li>Adherence to naming conventions</li>
<li>Test coverage</li>
</ul>
<p>… and so forth.</p>
<p>I think everything I’ve described above is pretty uncontroversial yet In spite of all of this I think code reviews today are (with some justification) a somewhat maligned practice.</p>
<p>Code reviews and their efficacy rely heavily on the existing relationships between the team members. Unhappiness with technical direction and personal animosities can spill over into the review process and end up harming team performance.</p>
<p>Conversely, teams that get along and have a solid and agreed vision of what good code looks like never seem to suffer these pitfalls and the process is a routine and effective guarantor of code quality.</p>
<p>To explore some of the complexities around code reviews we first need to examine its origins and recall how software used to be shipped.</p>
<h2 id="how-things-were">How things were</h2>
<p>In the past releasing new software was a chore. Preparing the release candidate, notifying internal stakeholders and users about it, weeks of manual regression testing and warnings of downtime on the day of release were all part and parcel of the process. Fear of what could go wrong loomed large over the whole process.</p>
<p>Since the cost of change was so high, <em>getting things right the first time</em> was of cardinal importance. A botched deployment meant kicking off the same lengthy process all over again to revert the failed change.</p>
<p>Developers were constrained by the tools and methodologies of the time - being totally confident in the success of a deployment was near impossible. Test driven development, a now widely accepted practice for writing resilient code wasn’t developed till the late 90’s by Kent Beck yet code was already written for devices large and small long before this.</p>
<p>In these times code reviews were a vital tool to ensure code quality. Seasoned engineers would pour over the proposed changes looking for syntax errors, potential compilation issues or any other gaps in code quality that could derail a release and result in a bunch of upset customers or users. It was manual, laborious and the stakes were high if something slipped through.</p>
<h2 id="how-things-are-now">How things are now</h2>
<p>The tools and methodologies used to ensure quality code have gone from sparse to abundant.</p>
<p>Developers have their choice of rich IDE’s that offer access to a vast plugin ecosystem for most any active programming language, providing convenience features like syntax highlighting and code formatting and more advanced features like code completion.</p>
<p>Testing frameworks are faster and more feature rich. With a solid testing strategy, developers can have a high degree of confidence that their new code has the intended effect without new bugs or regressions cropping up.</p>
<p>Today’s devs can even provision production-like services on their development machines through techniques like containerisation, providing lean standalone services - the kind of thing that would have been unimaginable for complicated subsystem teams of the past.</p>
<p>For companies that have leaned into continuous delivery and continuous integration as practices, code deployments that once took hours or days could be achieved in mere minutes. Tools like Github Actions can run automated checks against our bundled software before deploying it to production. Some of these checks can include automated tests (unit, e2e, contract etc..), static analysis tools, vulnerability scans just to name a few.</p>
<p>So what does this all mean?</p>
<p>The code review has gone from being one of a handful of practices to ensure code quality to just one amongst many. Verifying that our code can be safely merged is now also possible through another rival practice - pair programming.</p>
<h2 id="shifting-left">Shifting left</h2>
<p>The traditional way in engineering teams would solve coding problems was pretty straight forward - analyse the problem, write code to fix it then test the code. This is an approach which fits neatly into the roles of business analyst, developer and QA. Actual testing only came at the end.</p>
<p>The problem with this approach was that oftentimes requirements were poorly understood or missed entirely and too often devs would end up building the wrong thing. This left QA’s in an unenviable position of either sending time sensitive work back to developers to fix, potentially risking a deliverable, or letting it slide and creating a separate bug to track whatever got missed.</p>
<p>In 2001 Larry Smith proposed a different approach. He coined the term “shift left testing” to describe testing in smaller iterations. Instead of writing code and only testing it once it was done, he advocated for testing during development, in smaller iterations, so together the dev and QA could both be more confident that what’s being built would match what was asked for.</p>
<p>Around the same time, the Extreme Programming (XP) philosophy was gaining popularity. It challenged teams to take ownership of the software they ship, and to not view their role in the team as being siloed just to their job title. Far from just being on the shoulders of QAs and testers, ensuring the quality of the final product was becoming a whole team concern.</p>
<p>Baking in quality from the start would prove to be a powerful idiom and expanded to other areas of software delivery, particularly around non functional requirements. Testing, accessibility, observability, security - all of these that were traditionally regarded as something that was considered “once development was done” but are now seen as critical requirements that have to be considered up front.</p>
<p>Pair programming was conceived in the same spirit.</p>
<p>Having two developers working at the same terminal has existed for quite a while but true pairing - where the design, problem analysis and writing of code is conducted by peers that discuss approaches and alternate roles - would take time to grow in acceptance.</p>
<p>While controversial to some, pairing represents a shift left in the approval process.</p>
<p>Instead of having discrete write-code, review-code blocks - the two phases are merged into a single process where code is co-authored by two devs. This satisfies the “two pairs of eyes” principle and so any code they write could go to production - which is why it’s often used in conjunction with trunk based development.</p>
<p>In spite of the difficulty introducing pairing and trunk based development to engineering teams that haven’t used them before the benefits can be huge - with automated testing and CI/CD pipelines, teams can ship quality code many times a day with high confidence and none of the waiting around and context switching that characterises the humble code review.</p>
<h2 id="whats-wrong-with-code-reviews">What’s wrong with code reviews?</h2>
<p>Before exploring how we can do code reviews better we need some appreciation of what are the practical drawbacks using code reviews as a means of safeguarding code quality in the first place.</p>
<p>I’ll group my critique of code reviews into two broad categories:</p>
<ul>
<li>The process itself</li>
<li>Developer attitudes towards them</li>
</ul>
<h3 id="code-reviews-as-a-process">Code reviews as a process</h3>
<p>Utilising code reviews means developer workflows are habitually broken.</p>
<p>If you’re writing code then you need to put out a request for a review and await the outcome of it before resuming work. If you’re the reviewer then your work is potentially being interrupted by being asked to conduct a review. These shifting and exchanging responsibilities are a form of context switching - dropping what you were doing before and focusing on something else.</p>
<p>Proponents of code reviews would argue the original submitter could just do something else while the review is ongoing but this too isn’t ideal. Having devs split their attention between multiple streams of work is inefficient and risks the quality of one or both being compromised.</p>
<p>Code reviews also contain within them an innate antagonism, forcing teams to choose between one undesirable consequence and another - and that is what is the ideal size of a pull request?</p>
<p>Small commits are generally considered to be good practice in coding. They encourage developers to be deliberate in their choices and result in more focused changes. Having pull requests with fewer commits mean they reveal their intention more clearly too, which makes reviewing them far more straightforward.</p>
<p>However, pull requests consisting of only one or two commits don’t gel very well with the manual nature of code reviews. If I produce multiple pull requests in a day then I’ll require a lot of reviews, which means lots of context switching for those on my team who have to review my work (and maybe some desensitisation towards my repeated entreaties for a review too)</p>
<p>But what about batching together many commits so instead of many small ones you just have one big one? You can do this but it has a cost attached.</p>
<p>Blending many commits into a single pull request means that the surface area of change is much greater, leading to a proportionally higher review time. While good test names and commit messages can mitigate this, the overall goal of what’s being achieved can become blurred, mandating a slower line-by-line approach to reviewing code to see what’s changed and where. Overly stuffed pull requests do not clearly reveal their intention like their smaller counterparts.</p>
<p>Or even worse if the surface area of change is massive then something could just slip by entirely e.g. renaming a file while also making business logic changes that could be lost in the revision comparison tool. Being “too big to review” is a risk too.</p>
<p>Ultimately teams need to come to their own conclusions about how big or small code reviews should be but either way there’s a cost involved regardless.</p>
<h3 id="developer-attitudes-to-code-reviews">Developer attitudes to code reviews</h3>
<p>Prevailing attitudes within engineering teams towards the review process itself can help or hinder the process as a whole. We understand that code reviews are primarily a technical assessment to assess the fitness of some code though I would argue it has a psychosocial facet to it too.</p>
<p>What does good code look like? Who is in a position to judge good from bad? What technologies or technical practices are desirable or not?</p>
<p>These are all subjective questions that don’t have hard answers. In the absence of some socialised view of what “good” looks like, developers will bring their experience to bear on the review - they also bring their biases, subjectivities and sometimes their egos too.</p>
<p>Commercial software development is a time consuming and expensive process. Meeting milestones on product roadmaps can take a long time to hit and forecasting can be like crystal ball gazing at best of times. For businesses continuous feature delivery is incredibly important, if not essential to their survival.</p>
<p>However this urgency is not always reflected in the spirit with which code is reviewed. Spurious or ineffectual review comments go beyond simply being annoying and if unchecked can be harmful and erode team trust.</p>
<h2 id="is-there-a-better-way">Is there a better way?</h2>
<p>Despite all these drawbacks I do think code reviews can be made to work.</p>
<p>However I think doing them effectively means engineering teams re-evaluating governance of the review process. Useful new tools can also help as previous tedious manual checks and be neatly automated away.</p>
<p>We also need to evaluate the human impact of the review process so we’ll touch on how giving effective feedback can guide us to a better, more empathetic review process.</p>
<h2 id="team-owned-quality">Team owned quality</h2>
<p>Some of the drawbacks of code reviews can seem quite apparent but others are much more subtle. One such subtlety is who owns code quality? The natural answer seems to be the senior most developers and technical leaders with the engineering teams. Their experience makes them natural arbiters of code quality and so it’s natural to assume that they should play an oversized role in making sure the code is up to scratch.</p>
<p>While this all seems quite reasonable I think this is a backward and self-defeating approach.</p>
<p>It’s somewhat uncommon for engineering teams to be staffed entirely with experienced developers. The commercial demands placed on organisations mean that there is always more work to be done than developers to do the work.</p>
<p>For this reason we see the emergence of leveraged teams - that is to say teams with one or two highly experienced developers, some with moderate experience and then some more junior developers to round out the team.</p>
<p>I would propose that having code quality owned by a handful of experienced developers is harmful to both the experienced devs and the mid/junior developers they are seeing to grow and teach:</p>
<p>Tech leads and principal developers often have responsibilities that lie outside the immediate concerns of the delivery team. Some include: delivery assurance discussions, product discovery, line management and cultivation concerns, tech analysis sessions, engineering working groups, tech visioning discussions and so forth. In short, they have a lot on their plate outside of coding (if indeed they have time to code at all).</p>
<p>On top of all this, if they alone are required to approve the code their team generates then it’s easy for them to become a single point of failure. Code reviews take time to effectively review and delay is expensive. Equally as someone that isn’t writing much code they might actually have less context on the codebase should be doing than other engineers that do nothing but write code all day.</p>
<p>This format is harmful for less experienced developers too. Being unable to influence coding standards risks them becoming disengaged and seeing quality as something that is the responsibility of QA’s and senior developers, not themselves. Why care about standards you have no agency to change? This is also harmful to their growth and could prompt talented developers to actively look for opportunities somewhere else where they can have more autonomy and influence.</p>
<p>The solution to these problems is team owned quality.</p>
<p>Engineering teams should get together early after forming and talk about coding standards in a collaborative and inclusive way. This should continue for the life of the team. The meeting should be a discussion where everyone has a chance to contribute and debate what makes for good code.</p>
<p>Senior developers should help guide the conversation but not act as gatekeepers of what is good and what isn’t and should approach discussions with an open mind. Though junior developers are new to their career, they can produce useful insights and fresh perspectives.</p>
<p>These sessions should be repeated weekly or fortnightly, with discussion points brought in eg. “can we adopt this new technology?”, “does this approach seem off to anyone else”, “Is it just me or are our pipeline tests flaky?”. As with all well run meetings, notes should be taken and actions assigned to individuals to follow up on before the next meeting where they recap their progress.</p>
<p>The offshoot of this is that everyone has an equal responsibility to ensure the quality of the software the team writes. If indeed a bug is released, it can be included as a discussion point in the next meeting (in a candid but blame-free way) and the team can come up with actions on how to prevent a similar event from happening in the future.</p>
<p>This approach to communally owned coding standards means anyone in the team can be an approver. This frees up experienced developers from being gatekeepers of quality while giving more challenging and engaging opportunities for growth to less experienced developers as they’re forced to contemplate broader sets of criteria their code must adhere to beyond simply “does it work?”</p>
<h2 id="automate-where-possible">Automate where possible</h2>
<p>When approaching a code review there is usually some kind of coding standard to which the team adheres to. Critically assessing code can be time consuming and laborious so any way we can reduce cognitive load should be explored.</p>
<p>Simply put, if we can automate part of the review process effectively then we should. This could include things like maximum line length, indentation size, spaces after function name etc… Abstracting away these minor aspects of code style means reviewers can focus their attention on higher level concerns.</p>
<p>This can be accomplished using linters like eslint. Code formatters can be implemented as a pre-commit hook or even integrated with the developers IDE so that when they save a file the code is automatically formatted - this means that we can have linter config files as part of the codebase so that the same formatting rules are always applied regardless of who has checked out the code.</p>
<p>Automation isn’t just for linting though, we can use static analysis tools to catch missing code coverage or any security vulnerabilities that our changes to the project dependencies might miss out on.</p>
<p>Automation is one of the simplest ways to bake quality into our development process for a relatively modest initial investment.</p>
<h2 id="empathetic-feedback">Empathetic feedback</h2>
<p>The process of giving feedback, let alone technical feedback is at times poorly understood.</p>
<p>When giving feedback of any kind there are a couple of things to keep in mind:</p>
<ul>
<li>Am I acknowledging the subjectivity of my feedback? Eg. making conclusions without room for disagreement</li>
<li>Have I left my own biases and preferences at the door?</li>
<li>Have I thought about how the person who asked might receive my feedback?</li>
<li>Are my critiques actionable?</li>
</ul>
<p>This stuff all seems basic but too often it’s missing completely. Ineffective developers may seize on code reviews as a chance to exert undue influence over their peers, requiring changes that have no material impact on the outcome.</p>
<p>The best developers use code reviews as teachable moments, where the problem domain is explored in partnership with the person who asked for the review, if indeed something needs to be fixed. The experience or title gap between them is immaterial as they both explore what solution fits best. One-to-one interaction is preferred over review comments where some of the nuance of the conversation may be lost.</p>
<p>Finally, savvy reviewers will also understand that not all code quality issues need to be addressed at the code review stage. If something is unusual and perhaps a little contentious, then it can be addressed asynchronously the next time devs get together to talk about code. Good devs appreciate the cost involved in delay and will never seek to hold up code from being merged any more than strictly necessary.</p>
<p>This neatly leads us to the final point in how we can have more effective code reviews.</p>
<h2 id="technical-feedback">Technical feedback</h2>
<p>One of the most common memes when it comes to developers is that it’s hard to get a straight answer from them. Every decision seems to prompt a lot of discussions about trade offs and opportunity costs.</p>
<p>Unfortunately I think there’s an element of truth to this. Most decisions, especially technical ones, are rarely straightforward and clear cut and usually there is an element of compromise to them. In spite of this ambiguity I think there is some general guidance we can apply when reviewing code for frequently deployed systems that have a low cost of change.</p>
<p>Let’s look at some valid justifications for rejecting a pull request and asking the dev that submitted it to look at it again (for brevity we’re going to assume the project builds successfully and unit tests are passing)</p>
<h3 id="it-doesnt-adhere-to-the-teams-coding-standards">It doesn’t adhere to the teams coding standards</h3>
<p>This is pretty straightforward. I won’t go into detail as we’ve already covered this but a team should agree what good looks like and agree to follow those standards. This should be the least contentious type of review comment as everyone had a chance to give their input in shaping these standards.</p>
<h3 id="the-code-has-quantitative-deficiencies">The code has quantitative deficiencies</h3>
<p>This is where code quality is compromised in ways that, while not explicitly caught by coding standards, have a clear and identifiable negative impact.</p>
<p>For example, say a dev introduces a new dependency to the project to perform some functionality that is already present in language used. In the Javascript programming language (and others) concise dependency trees lessens potential version conflicts when upgrading. It also creates a smaller vector of attack as fewer dependencies means fewer packages that could, in the course of time, become outdated and insecure.</p>
<p>In this instance, rejecting this code change upholds a facet of code quality that can’t be strongly argued against, especially since the introduced functionality is already present within our chosen language.</p>
<p>Unjustified code duplication, using deprecated language features or indeed any change that meaningfully compromises the non functional requirements of our system are all examples of quantitative deficiencies.</p>
<p>Beyond these two cases I think review comments fall into the realm of subjectivity. Responsible reviewers can and should acknowledge this and frame their critiques accordingly. Teams that can compromise where appropriate show maturity, even if there are some minor disagreements initially.</p>
<h2 id="conclusion">Conclusion:</h2>
<p>Doing code reviews well requires trust, openness and empathy within the teams that practice them.</p>
<p>Far from being a mundane technical practice, code reviews have the chance to energise and engage software delivery teams by fully placing ownship of the software they write in their hands.</p>
<p>So let’s democratise code quality and build a vision of how our system could work that includes everyone.</p>Niall BamburyAn examination of the practices, values and ways of working that underpin effective code reviewsIn defence of the testing pyramid2022-06-13T20:00:00+00:002022-06-13T20:00:00+00:00https://awesome.red-badger.com//charypar/in-defence-of-the-testing-pyramid<p>I like simplified models to guide decisions on how to build systems. There’s enough complexity in any software project, that without some guiding principles that at least seem theoretically correct, we’re likely to make suboptimal choices based on a coin toss. Simplified models neatly encapsulate the experience from previous attempts at doing something and help inform the future attempts.</p>
<p>So long as they are not too simplified. To actually work, the models need to be clear, reasonably complete, and actionable. If I have a concrete question, even as simple as a decision between two alternative approaches, the model needs to give me clear guidance on how to proceed. Case in point, the testing pyramid:</p>
<p><img src="https://miro.medium.com/max/1400/1*Tcj3OsK8Kou7tCMQgeeCuw.png" alt="The testig pyramid" />
<em>Image source: <a href="https://betterprogramming.pub/the-test-pyramid-80d77535573">https://betterprogramming.pub/the-test-pyramid-80d77535573</a></em></p>
<p>We’ve all seen this picture. It is a seemingly helpful guide: To minimise cost of testing and maximise reliability and resulting quality, put more effort into, and rely more on tests closer to the implementation than the tests that are covering a wider scope. Seems intuitively correct, but fails my criteria: It’s unclear what integration tests and even unit tests actually are, specifically. It also doesn’t help answer questions like “Should I run end to end tests on every pull request?”. And where does static analysis fit in? – it’s unclear, incomplete, and it isn’t actionable.</p>
<p>As a consequence, people don’t seem to find it particularly helpful, and even propose various modifications which resonate with their experience better, like the <a href="https://kentcdodds.com/blog/write-tests">testing trophy, proposed by Kent C. Dodds</a>. I think these variants come down to the misalignment about what the individual layers of the pyramid are, what value they bring, and how costly they are to execute. And I’ve seen one too many testing strategies in which the pyramid is completely upside down, but the engineering team just doesn’t know how to do their testing any better. The pyramid is supposed to guide them, alas it stays quiet.</p>
<p>I don’t think there’s anything wrong with the pyramid. It’s just not quite detailed enough to be useful. Good first approximation, but I think we can do better.</p>
<h2 id="some-assumptions">Some Assumptions</h2>
<p>To make a better version of the pyramid, I will need to make some assumptions about the kind of system we’re testing, and the high-level ways of working of the team, and roles involved in the building of the product. At a high level, I will assume that:</p>
<ul>
<li>We’re building a distributed system, composed of independently deployed services and applications. Running the entire system requires an environment.</li>
<li>Our system has external dependencies which we don’t control - services delivered by separate organisations or other, independent teams in our organisation</li>
<li>We’re doing continuous delivery, even continuous deployment to production, and therefore heavily rely on a Continuous Integration (CI) service</li>
<li>Our branching workflow resembles Github flow - we have a single main branch, into which contributions are made in the form of small pull requests which get reviewed before merging</li>
</ul>
<p>This feels like a pretty typical situation most digital product teams find themselves in. The situation is probably different for game developers, desktop software developers, machine learning engineers and data scientists, and others. So, as always, your mileage may vary.</p>
<h2 id="the-improved-pyramid">The improved pyramid</h2>
<p>With that out of the way, here’s a (hopefully) more complete and useful pyramid:</p>
<p><img src="/assets/charypar/testing-pyramid.png" alt="Revised testig pyramid" /></p>
<p>This one has five layers, which are defined by what gets tested, where, when, why and how. The layers still form a pyramid, because in order for the upper layers of the pyramid to pass (or even run), the lower layers have to have passed first. Upper layer tests require more setup and more infrastructure, which makes them much more expensive to run than lower layer ones, so the overall volume of testing on the upper layers needs to be lower to maintain a stable cost/benefit ratio across layers. Or, from a more outcome oriented perspective:</p>
<blockquote>
<p>The goal of each layer is to more cheaply and more regularly gain a reasonably high confidence that the layer above will pass, so that it does not need to be run as often</p>
</blockquote>
<p>In other words, the lower layers are an optimisation of executing the upper layers, in order to reduce their cost while maintaining a high enough overall confidence in the system.</p>
<h3 id="changing-tests-with-implementation-is-fine-top-heavy-pyramids-are-not-fine">Changing tests with implementation is fine, top-heavy pyramids are not fine</h3>
<p>Some will argue that pushing tests lower, closer to the implementation will lead to us having to regularly change tests when we change implementation, and that the execution time saved will instead be spent on keeping the lower layer tests up to date. To that, I will say that the idea that it should be possible to change implementation without changing tests is just… nonsense.</p>
<p>If it were true, it would mean that everything can be exhaustively tested through the interfaces it is consumed through - through the UI or public API, or something very close to it. That thinking obviously completely ignores the “physics” of software, and all the reasons we are building systems in a decomposed, modular way, which encapsulates and reuses logic. The sheer number of test cases necessary to capture all the nuances of all our business logic all at once is just not practical. And it’s not difficult to show that.</p>
<p>Let’s say each logical “unit” of our software requires, on average, 10 scenarios to test it thoroughly. Then, if these units are in any way dependent on one another (either one uses the other, or it uses the resulting state the other created), the number of scenarios multiply, and there are a 100 scenarios to cover for just two units, a 1000 for three, … This clearly doesn’t scale even a little bit! Let alone if our system is a complex sequential user journey, like an onboarding flow or a checkout in an online shop. That’s why we decompose software into functions and modules in the first place.</p>
<p>No other discipline building even remotely complicated things approaches testing in that way. Nobody sane would argue that a car should be tested only through driving it, on an actual public road, because otherwise we’d need to change the tooling when we change components. Of course the engine is tested independently from the tyres and the chassis. Therefore, the only remaining question is - how big is a unit we will test. And the bigger it is, the more tests are needed to cover it. And the more tests will need to be written again, if the unit is changed substantially. This is an inevitable reality driven purely by the complexity of the system, and doesn’t mean we shouldn’t build instrumentation for its constituent parts, just because we might replace them.</p>
<h3 id="external-dependencies-are-costly">External dependencies are costly</h3>
<p>You might also argue that layers 2 and 3 seem like an artificial split. We should be able to test everything on layer 3, surely. And in an ideal world, we could. The problem with testing against non-production environments of external dependencies is purely practical:</p>
<ul>
<li>They are not always available</li>
<li>They are often slow or rate limited</li>
<li>They lack test data or require creating it for every run</li>
<li>They are stateful</li>
</ul>
<p>All these reasons make repeatable, reliable testing solely against external systems difficult. But fully relying on mocks is not a reliable strategy either. We need to be sure that the mocks still behave like the actual system. We need to do both.</p>
<h2 id="guiding-principles">Guiding principles</h2>
<p>So, we now have a more complete version of the pyramid. To make it an actionable model, we just need some overall guiding principles:</p>
<ol>
<li>All layers of the pyramid are necessary in order to achieve a high confidence in your system.</li>
<li>Always prefer testing functionality on lowest possible layer.</li>
<li>When a layer of testing starts slowing you down, reduce the volume of that layer and replace it by coverage on the lower layers. Introduce new forms of testing on lower layers.</li>
<li>Do not give in to the temptation of moving layers “left” in the development process, executing the more expensive ones earlier or more often.</li>
</ol>
<p>The last principle is worth talking about a little more. It suggests, for example, that running automated, system level, end to end test on each pull requests, as tempting as it sounds, is not worth the effort and complexity. End to end tests require an environment to run in (see assumptions), which would need to be created and destroyed (or allocated and cleaned up) for each pull request. In my experience, this is really complex, slow and highly unlikely to pay off in additional confidence gained, compared to running the end to end test(s) on the main branch, after merging the pull request.</p>
<p>With these principles, I hope the model is now actually useful - clear, complete and actionable. It should be possible to take every form of testing you can think of, decide what layer it belongs to, and therefore where and when it should execute.</p>
<p>Building up this pyramid will take time, and at first, expanding the higher layers will pay off much more than the lower layers. But these are diminishing returns. That’s why so many teams come up with an upside down pyramid that takes hours to run and still lets many bugs through. It’s so tempting to just add another end to end test. Don’t fall into that trap. Follow the above principles, and turn the pyramid the right way up.</p>
<h2 id="built-to-be-testable">Built to be testable</h2>
<p>I have one final observation about all this: If, despite your best effort, you can’t seem to push test coverage down the pyramid, you may need to revisit your architecture, or even technology choices. Testability entirely depends of how your system is built.</p>
<p>In a system without stubs of external dependencies, achieving reasonably high coverage with automated end to end tests will be difficult and slow. In a system with services sharing state (a database for example), pushing tests from layer 2 down to layer 1 will be difficult. In a codebase with no concept of dependency injection (even a really simple one), moving coverage from functional tests to unit tests will be difficult (and require extensive, messy mocking and stubbing). In a system without contract testing or shared API types ensuring client/server alignment on layer 0, a lot more layer 3 tests will be required to gain the same level of confidence. In a dynamic language with few static guarantees, layer 1 will almost certainly be much larger than layer 0.</p>
<p>I could keep going, but you see the point. Fast reliable testing begins with smart engineering. Hopefully this more specific pyramid will help you make smarter choices.</p>Viktor CharyparI don’t think there’s anything wrong with the testing pyramid. It’s just not quite detailed enough to be useful. Good first approximation, but we can do better.Creating a Miro Clone in the browser2022-05-20T12:00:00+00:002022-05-20T12:00:00+00:00https://awesome.red-badger.com//ceddlyburge/visual-sort<h2 id="introduction">Introduction</h2>
<p>We recently worked with a client who asked us to create a freestyle web canvas, for adding various types of ‘Card’. We love a challenge so got to work on the requirements:</p>
<ul>
<li>Panning and Zooming</li>
<li>Add Cards from a pop up tray</li>
<li>Cards should not overlap</li>
<li>Drag Cards on the canvas</li>
</ul>
<h2 id="too-long-didnt-read-tldr">Too long didn’t read (TLDR)</h2>
<p>This was a complex epic, and there was definitely some head scratching. However we finished it on time and the end result is beautiful, responsive and fully covered by end to end tests. We used <a href="https://dndkit.com/">DndKit</a> for dragging and dropping, <a href="https://github.com/d3/d3-zoom">D3 Zoom</a> for panning and zooming and <a href="https://www.cypress.io/">Cypress</a> for the tests, which were all a pleasure to work with, and we would do so again.</p>
<h2 id="panning-and-zooming">Panning and Zooming</h2>
<p><img src="/assets/ceddlyburge/visual-sort/zoom-pan.gif" alt="Zooming and panning" /></p>
<p>A very stripped down version of the Canvas component is shown below, with the code required to hook up to D3 Zoom, and to apply the tranform.</p>
<p>The transform is applied to the canvas element, and the browser will automatically apply the transform to anything under it in the DOM tree, so nothing on the canvas needs to do anything.</p>
<div class="language-tsx highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">export</span> <span class="kd">const</span> <span class="nx">Canvas</span> <span class="o">=</span> <span class="p">({</span> <span class="nx">children</span> <span class="p">}:</span> <span class="nx">CanvasProps</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">canvasRef</span> <span class="o">=</span> <span class="nx">useRef</span><span class="o"><</span><span class="nx">HTMLDivElement</span> <span class="o">|</span> <span class="kc">null</span><span class="o">></span><span class="p">(</span><span class="kc">null</span><span class="p">);</span>
<span class="c1">// store the current transform from d3</span>
<span class="kd">const</span> <span class="p">[</span><span class="nx">transform</span><span class="p">,</span> <span class="nx">setTransform</span><span class="p">]</span> <span class="o">=</span> <span class="nx">useState</span><span class="p">(</span><span class="nx">d3</span><span class="p">.</span><span class="nx">zoomIdentity</span><span class="p">);</span>
<span class="c1">// update the transform when d3 zoom notifies of a change</span>
<span class="kd">const</span> <span class="nx">updateTransform</span> <span class="o">=</span> <span class="p">({</span> <span class="nx">transform</span> <span class="p">}:</span> <span class="p">{</span> <span class="na">transform</span><span class="p">:</span> <span class="nx">d3</span><span class="p">.</span><span class="nx">ZoomTransform</span> <span class="p">})</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">setTransform</span><span class="p">(</span><span class="nx">transform</span><span class="p">);</span>
<span class="p">};</span>
<span class="c1">// create the d3 zoom object, and useMemo to retain it for rerenders</span>
<span class="kd">const</span> <span class="nx">zoomBehavior</span> <span class="o">=</span> <span class="nx">useMemo</span><span class="p">(()</span> <span class="o">=></span> <span class="nx">d3</span><span class="p">.</span><span class="nx">zoom</span><span class="o"><</span><span class="nx">HTMLDivElement</span><span class="p">,</span> <span class="nx">unknown</span><span class="o">></span><span class="p">(),</span> <span class="p">[]);</span>
<span class="nx">useLayoutEffect</span><span class="p">(()</span> <span class="o">=></span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nx">canvasRef</span><span class="p">.</span><span class="nx">current</span><span class="p">)</span> <span class="k">return</span><span class="p">;</span>
<span class="c1">// get transform changed notifications from d3 zoom</span>
<span class="nx">zoomBehavior</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="dl">"</span><span class="s2">zoom</span><span class="dl">"</span><span class="p">,</span> <span class="nx">updateTransform</span><span class="p">);</span>
<span class="c1">// attach d3 zoom to the canvas div element, which will handle</span>
<span class="c1">// mousewheel and drag events automatically for pan / zoom</span>
<span class="k">return</span> <span class="nx">d3</span>
<span class="p">.</span><span class="nx">select</span><span class="o"><</span><span class="nx">HTMLDivElement</span><span class="p">,</span> <span class="nx">unknown</span><span class="o">></span><span class="p">(</span><span class="nx">canvasRef</span><span class="p">.</span><span class="nx">current</span><span class="p">)</span>
<span class="p">.</span><span class="nx">call</span><span class="p">(</span><span class="nx">zoomBehavior</span><span class="p">);</span>
<span class="p">},</span> <span class="p">[</span><span class="nx">zoomBehavior</span><span class="p">,</span> <span class="nx">canvasRef</span><span class="p">]);</span>
<span class="c1">// animated Zoom In, which can be called from a button event (not shown in this example)</span>
<span class="kd">const</span> <span class="nx">zoomIn</span> <span class="o">=</span> <span class="p">()</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">d3</span><span class="p">.</span><span class="nx">transition</span><span class="p">()?.</span><span class="nx">call</span><span class="p">(</span><span class="nx">zoomBehavior</span><span class="p">.</span><span class="nx">scaleBy</span><span class="p">,</span> <span class="mf">1.5</span><span class="p">);</span>
<span class="p">};</span>
<span class="k">return</span> <span class="p">(</span>
<span class="p"><</span><span class="nt">div</span> <span class="na">ref</span><span class="p">=</span><span class="si">{</span><span class="nx">canvasRef</span><span class="si">}</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span>
<span class="na">style</span><span class="p">=</span><span class="si">{</span>
<span class="c1">// apply the transform from d3</span>
<span class="nx">transformOrigin</span><span class="p">:</span> <span class="dl">"</span><span class="s2">top left</span><span class="dl">"</span><span class="p">,</span>
<span class="nx">transform</span><span class="p">:</span> <span class="s2">`translate3d(</span><span class="p">${</span><span class="nx">transform</span><span class="p">.</span><span class="nx">x</span><span class="p">}</span><span class="s2">, </span><span class="p">${</span><span class="nx">transform</span><span class="p">.</span><span class="nx">y</span><span class="p">}</span><span class="s2">, </span><span class="p">${</span><span class="nx">transform</span><span class="p">.</span><span class="nx">k</span><span class="p">}</span><span class="s2">)`</span><span class="p">,</span>
<span class="si">}</span>
<span class="p">></span>
<span class="si">{</span><span class="nx">children</span><span class="si">}</span>
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p">);</span>
<span class="p">};</span>
</code></pre></div></div>
<h2 id="drag-drop-from-tray">Drag Drop from Tray</h2>
<p>In order to create insightful visual arrangements, users wanted to be able to see all of their cards in a tray, and to drag them from there on to the canvas. None of the DndKit examples were that close to what we wanted, so we had to strike out on our own (although the documentation is excellent, which made things easier.)</p>
<p><img src="/assets/ceddlyburge/visual-sort/drag-drop-from-tray.png" alt="Drag drop from tray" /></p>
<h3 id="display-cards-on-the-canvas">Display cards on the canvas</h3>
<p>When an item is drag / dropped from the tray it needs to appear on the canvas, so firstly we simply hard coded a list of cards to display. These had an <code class="language-plaintext highlighter-rouge">x, y</code> position on the canvas, and all the information they needed to render. We sized the cards to match the grid size, and used the code below to position the cards on the canvas.</p>
<div class="language-tsx highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p"><</span><span class="nt">div</span>
<span class="na">css</span><span class="p">=</span><span class="si">{</span>
<span class="nx">position</span><span class="p">:</span> <span class="dl">"</span><span class="s2">absolute</span><span class="dl">"</span><span class="p">,</span>
<span class="nx">origin</span><span class="p">:</span> <span class="dl">"</span><span class="s2">top left</span><span class="dl">"</span><span class="p">,</span>
<span class="nx">top</span><span class="p">:</span> <span class="s2">`</span><span class="p">${</span><span class="nx">pixelCoordinates</span><span class="p">.</span><span class="nx">y</span><span class="p">}</span><span class="s2">px`</span><span class="p">,</span>
<span class="nx">left</span><span class="p">:</span> <span class="s2">`</span><span class="p">${</span><span class="nx">pixelCoordinates</span><span class="p">.</span><span class="nx">x</span><span class="p">}</span><span class="s2">px`</span><span class="p">,</span>
<span class="si">}</span>
<span class="p">></span>
<span class="si">{</span><span class="nx">children</span><span class="si">}</span>
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
</code></pre></div></div>
<h3 id="allow-dropping-when-dragged-over-the-canvas">Allow dropping when dragged over the canvas</h3>
<p>In our UI, the tray popped up over the canvas, and being as the canvas was a drop target, it was initially possible to drop a card on to the canvas without having first dragged it off the tray.</p>
<p>To fix this we created a custom strategy to work out the drop target by composing existing DndKit strategies (<a href="https://docs.dndkit.com/api-documentation/context-provider/collision-detection-algorithms#building-custom-collision-detection-algorithms">as recommended by the documentation</a>).</p>
<p>We first check to see if the current drag position is intersecting with the tray, and if so we return that. If not we fallback to the standard DndKit behaviour. This requires us to set up the tray as a drop target, and for the drop event to check what drop target was found (and to ignore the tray if this is the target).</p>
<p>The code for the custom strategy is like this</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">customCollisionDetectionStrategy</span> <span class="o">=</span> <span class="p">()</span> <span class="o">=></span> <span class="p">{</span>
<span class="k">return</span> <span class="p">(</span><span class="na">args</span><span class="p">:</span> <span class="p">{</span>
<span class="na">active</span><span class="p">:</span> <span class="nx">Active</span><span class="p">;</span>
<span class="nl">collisionRect</span><span class="p">:</span> <span class="nx">ViewRect</span><span class="p">;</span>
<span class="nl">droppableContainers</span><span class="p">:</span> <span class="nx">DroppableContainer</span><span class="p">[];</span>
<span class="p">})</span> <span class="o">=></span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">args</span><span class="p">.</span><span class="nx">active</span><span class="p">.</span><span class="nx">rect</span><span class="p">.</span><span class="nx">current</span><span class="p">.</span><span class="nx">translated</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">const</span> <span class="na">targetScaled</span><span class="p">:</span> <span class="nx">ViewRect</span> <span class="o">=</span> <span class="p">{</span>
<span class="p">...</span><span class="nx">args</span><span class="p">.</span><span class="nx">active</span><span class="p">.</span><span class="nx">rect</span><span class="p">.</span><span class="nx">current</span><span class="p">.</span><span class="nx">translated</span><span class="p">,</span>
<span class="p">};</span>
<span class="kd">const</span> <span class="nx">trayRect</span> <span class="o">=</span> <span class="nx">args</span><span class="p">.</span><span class="nx">droppableContainers</span><span class="p">.</span><span class="nx">filter</span><span class="p">(</span>
<span class="p">(</span><span class="nx">droppableContainer</span><span class="p">)</span> <span class="o">=></span> <span class="nx">droppableContainer</span><span class="p">.</span><span class="nx">id</span> <span class="o">===</span> <span class="dl">"</span><span class="s2">tray</span><span class="dl">"</span>
<span class="p">);</span>
<span class="kd">const</span> <span class="nx">intersectingTrayRect</span> <span class="o">=</span> <span class="nx">rectIntersection</span><span class="p">({</span>
<span class="na">active</span><span class="p">:</span> <span class="nx">args</span><span class="p">.</span><span class="nx">active</span><span class="p">,</span>
<span class="na">collisionRect</span><span class="p">:</span> <span class="nx">targetScaled</span><span class="p">,</span>
<span class="na">droppableContainers</span><span class="p">:</span> <span class="nx">trayRect</span><span class="p">,</span>
<span class="p">});</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">intersectingTrayRect</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">intersectingTrayRect</span><span class="p">;</span>
<span class="p">}</span>
<span class="kd">const</span> <span class="nx">otherRects</span> <span class="o">=</span> <span class="nx">args</span><span class="p">.</span><span class="nx">droppableContainers</span><span class="p">.</span><span class="nx">filter</span><span class="p">(</span>
<span class="p">(</span><span class="nx">droppableContainer</span><span class="p">)</span> <span class="o">=></span> <span class="nx">droppableContainer</span><span class="p">.</span><span class="nx">id</span> <span class="o">!==</span> <span class="dl">"</span><span class="s2">tray</span><span class="dl">"</span>
<span class="p">);</span>
<span class="k">return</span> <span class="nx">rectIntersection</span><span class="p">({</span>
<span class="na">active</span><span class="p">:</span> <span class="nx">args</span><span class="p">.</span><span class="nx">active</span><span class="p">,</span>
<span class="na">collisionRect</span><span class="p">:</span> <span class="nx">targetScaled</span><span class="p">,</span>
<span class="na">droppableContainers</span><span class="p">:</span> <span class="nx">otherRects</span><span class="p">,</span>
<span class="p">});</span>
<span class="p">}</span>
<span class="k">return</span> <span class="dl">""</span><span class="p">;</span>
<span class="p">};</span>
<span class="p">};</span>
</code></pre></div></div>
<h3 id="the-drag-overlay--mouse-cursor-should-size-based-on-the-currrent-zoom-level-of-the-canvas-so-that-it-displays-as-it-will-appear-on-the-canvas">The drag overlay / mouse cursor should size based on the currrent zoom level of the canvas (so that it displays as it will appear on the canvas)</h3>
<p>The Canvas has a <a href="https://github.com/d3/d3-zoom#zoom-transforms">transform property (from D3)</a>, which has an <code class="language-plaintext highlighter-rouge">x</code> and a <code class="language-plaintext highlighter-rouge">y</code> property to define the panning, and a <code class="language-plaintext highlighter-rouge">k</code> property to define the zoom.</p>
<p>We already had canvas card components from earlier, but the zoom transform was being applied to the parent canvas component, so we still needed to size the drag overlay correctly.</p>
<p>This was achieved using the same scale transform that was applied to the canvas:</p>
<div class="language-tsx highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p"><</span><span class="nt">div</span>
<span class="na">style</span><span class="si">{</span>
<span class="nx">transformOrigin</span><span class="p">:</span> <span class="dl">'</span><span class="s1">top left</span><span class="dl">'</span><span class="p">,</span>
<span class="nx">transform</span><span class="p">:</span> <span class="s2">`scale(</span><span class="p">${</span><span class="nx">transform</span><span class="p">.</span><span class="nx">k</span><span class="p">}</span><span class="s2">)`</span><span class="p">,</span>
<span class="si">}</span>
<span class="p">></span>
</code></pre></div></div>
<h3 id="calculate-canvas-position-of-dropped-cards">Calculate canvas position of dropped cards</h3>
<p>To work out the position on the canvas, we need to know a few things:</p>
<ul>
<li>The zoom level of the canvas</li>
<li>The panning position of the canvas (relative to the window / viewport)</li>
<li>The drop position (relative to the window / viewport)</li>
</ul>
<p>The canvas position is then <code class="language-plaintext highlighter-rouge">(panning position - drop position) / zoom</code></p>
<p>We already store and have access to the zoom level and the panning position of the canvas, but the drop position is a bit trickier.</p>
<p>The DndKit drop event gives us the delta of the drag operation, but sadly doesn’t give us the initial position of the drag. It does however allow us to attach some custom data via a <code class="language-plaintext highlighter-rouge">ref</code> in <code class="language-plaintext highlighter-rouge">useDraggable</code>, so we store <code class="language-plaintext highlighter-rouge">getBoundingClientRect()</code> as <code class="language-plaintext highlighter-rouge">initialRect</code>, and can access it in the drop event with <code class="language-plaintext highlighter-rouge">active.data.current.initialRect</code>. This allows us to calculate the window / viewport drop position, which then allows us to calculate the drop position on the canvas.</p>
<p>The full code looks like this. <code class="language-plaintext highlighter-rouge">transform</code> controls the pan (<code class="language-plaintext highlighter-rouge">x, y</code>) and zoom (<code class="language-plaintext highlighter-rouge">k</code>) of the canvas.</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">calculateCanvasPosition</span> <span class="o">=</span> <span class="p">(</span>
<span class="nx">initialRect</span><span class="p">:</span> <span class="nx">DOMRect</span><span class="p">,</span>
<span class="nx">over</span><span class="p">:</span> <span class="nx">Over</span><span class="p">,</span>
<span class="nx">delta</span><span class="p">:</span> <span class="nx">Translate</span>
<span class="p">)</span> <span class="o">=></span>
<span class="nx">scaleCoordinates</span><span class="p">(</span>
<span class="p">{</span>
<span class="na">x</span><span class="p">:</span> <span class="nx">initialRect</span><span class="p">.</span><span class="nx">x</span> <span class="o">+</span> <span class="nx">delta</span><span class="p">.</span><span class="nx">x</span> <span class="o">-</span> <span class="p">(</span><span class="nx">over</span><span class="p">?.</span><span class="nx">rect</span><span class="p">?.</span><span class="nx">offsetLeft</span> <span class="o">??</span> <span class="mi">0</span><span class="p">)</span> <span class="o">-</span> <span class="nx">transform</span><span class="p">.</span><span class="nx">x</span><span class="p">,</span>
<span class="na">y</span><span class="p">:</span> <span class="nx">initialRect</span><span class="p">.</span><span class="nx">y</span> <span class="o">+</span> <span class="nx">delta</span><span class="p">.</span><span class="nx">y</span> <span class="o">-</span> <span class="p">(</span><span class="nx">over</span><span class="p">?.</span><span class="nx">rect</span><span class="p">?.</span><span class="nx">offsetTop</span> <span class="o">??</span> <span class="mi">0</span><span class="p">)</span> <span class="o">-</span> <span class="nx">transform</span><span class="p">.</span><span class="nx">y</span><span class="p">,</span>
<span class="p">},</span>
<span class="nx">transform</span><span class="p">.</span><span class="nx">k</span>
<span class="p">);</span>
<span class="kd">const</span> <span class="nx">scaleCoordinates</span> <span class="o">=</span> <span class="p">(</span><span class="nx">coords</span><span class="p">:</span> <span class="nx">Coordinates</span><span class="p">,</span> <span class="nx">scale</span><span class="p">:</span> <span class="kr">number</span><span class="p">):</span> <span class="nx">Coordinates</span> <span class="o">=></span>
<span class="p">{</span>
<span class="na">x</span><span class="p">:</span> <span class="nx">coords</span><span class="p">.</span><span class="nx">x</span> <span class="o">/</span> <span class="nx">scale</span><span class="p">,</span>
<span class="na">y</span><span class="p">:</span> <span class="nx">coords</span><span class="p">.</span><span class="nx">y</span> <span class="o">/</span> <span class="nx">scale</span><span class="p">,</span>
<span class="p">};</span>
</code></pre></div></div>
<h3 id="snap-to-grid">Snap to grid</h3>
<p>Once we have worked out a position on the grid, snapping to a grid is trivial! We just need to decide on the grid size, and then round the coordinates to it. There is even a <a href="https://docs.dndkit.com/api-documentation/modifiers#snap-to-grid">nice example in the DnDKit docs</a>.</p>
<p>Our code looked like this</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">export</span> <span class="kd">const</span> <span class="nx">snapCoordinates</span> <span class="o">=</span> <span class="p">({</span> <span class="nx">x</span><span class="p">,</span> <span class="nx">y</span> <span class="p">}:</span> <span class="nx">Coordinates</span><span class="p">):</span> <span class="nx">Coordinates</span> <span class="o">=></span> <span class="p">({</span>
<span class="na">x</span><span class="p">:</span> <span class="nx">snapCoordinate</span><span class="p">(</span><span class="nx">x</span><span class="p">,</span> <span class="nx">gridSize</span><span class="p">),</span>
<span class="na">y</span><span class="p">:</span> <span class="nx">snapCoordinate</span><span class="p">(</span><span class="nx">y</span><span class="p">,</span> <span class="nx">gridSize</span><span class="p">),</span>
<span class="p">});</span>
<span class="kd">const</span> <span class="nx">snapCoordinate</span> <span class="o">=</span> <span class="p">(</span><span class="nx">value</span><span class="p">:</span> <span class="kr">number</span><span class="p">,</span> <span class="nx">gridSize</span><span class="p">:</span> <span class="kr">number</span><span class="p">)</span> <span class="o">=></span>
<span class="nb">Math</span><span class="p">.</span><span class="nx">round</span><span class="p">(</span><span class="nx">value</span> <span class="o">/</span> <span class="nx">gridSize</span><span class="p">)</span> <span class="o">*</span> <span class="nx">gridSize</span><span class="p">;</span>
</code></pre></div></div>
<h3 id="drag-and-drop">Drag and drop</h3>
<p>Once we have all these items in place, we can integrate with DndKit.</p>
<p>There is a <code class="language-plaintext highlighter-rouge">DndContext</code>, that DndKit uses to store all the state:</p>
<div class="language-tsx highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p"><</span><span class="nc">DndContext</span>
<span class="na">sensors</span><span class="p">=</span><span class="si">{</span><span class="nx">sensors</span><span class="si">}</span>
<span class="na">onDragStart</span><span class="p">=</span><span class="si">{</span><span class="nx">handleDragStart</span><span class="si">}</span> <span class="c1">// stores the activeCard</span>
<span class="na">onDragMove</span><span class="p">=</span><span class="si">{</span><span class="nx">handleDragMove</span><span class="si">}</span> <span class="c1">// uses doCardsCollide (see "Cards should not overlap" later)</span>
<span class="na">onDragEnd</span><span class="p">=</span><span class="si">{</span><span class="nx">handleDragEnd</span><span class="si">}</span> <span class="c1">// uses calculateCanvasPosition, adds activeCard to children</span>
<span class="na">collisionDetection</span><span class="p">=</span><span class="si">{</span><span class="nx">customCollisionDetectionStrategy</span><span class="p">()</span><span class="si">}</span>
<span class="p">></span>
<span class="si">{</span><span class="nx">children</span><span class="si">}</span>
<span class="p"></</span><span class="nc">DndContext</span><span class="p">></span>
</code></pre></div></div>
<p>Then each component on the tray can <code class="language-plaintext highlighter-rouge">useDraggable</code> to enable drag and drop.</p>
<div class="language-tsx highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">export</span> <span class="kd">const</span> <span class="nx">Addable</span> <span class="o">=</span> <span class="p">({</span> <span class="nx">id</span><span class="p">,</span> <span class="nx">children</span> <span class="p">}:</span> <span class="nx">Props</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="kd">const</span> <span class="p">[</span><span class="nx">ref</span><span class="p">,</span> <span class="nx">setRef</span><span class="p">]</span> <span class="o">=</span> <span class="nx">useState</span><span class="o"><</span><span class="nx">Element</span> <span class="o">|</span> <span class="kc">null</span><span class="o">></span><span class="p">(</span><span class="kc">null</span><span class="p">);</span>
<span class="kd">const</span> <span class="p">{</span> <span class="nx">attributes</span><span class="p">,</span> <span class="nx">listeners</span><span class="p">,</span> <span class="nx">setNodeRef</span> <span class="p">}</span> <span class="o">=</span> <span class="nx">useDraggable</span><span class="p">({</span>
<span class="nx">id</span><span class="p">,</span>
<span class="na">data</span><span class="p">:</span> <span class="p">{</span> <span class="na">initialRect</span><span class="p">:</span> <span class="nx">ref</span><span class="p">?.</span><span class="nx">getBoundingClientRect</span><span class="p">()</span> <span class="p">},</span>
<span class="p">});</span>
<span class="kd">const</span> <span class="nx">updateInitialRectAndForwardRef</span> <span class="o">=</span> <span class="p">(</span><span class="na">element</span><span class="p">:</span> <span class="nx">HTMLDivElement</span> <span class="o">|</span> <span class="kc">null</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">setRef</span><span class="p">(</span><span class="nx">element</span><span class="p">);</span>
<span class="nx">setNodeRef</span><span class="p">(</span><span class="nx">element</span><span class="p">);</span>
<span class="p">};</span>
<span class="k">return</span> <span class="p">(</span>
<span class="p"><</span><span class="nt">div</span> <span class="na">ref</span><span class="p">=</span><span class="si">{</span><span class="nx">updateInitialRectAndForwardRef</span><span class="si">}</span> <span class="si">{</span><span class="p">...</span><span class="nx">listeners</span><span class="si">}</span> <span class="si">{</span><span class="p">...</span><span class="nx">attributes</span><span class="si">}</span><span class="p">></span>
<span class="si">{</span><span class="nx">children</span><span class="si">}</span>
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p">);</span>
<span class="p">};</span>
</code></pre></div></div>
<h2 id="drag-cards-on-the-canvas">Drag Cards on the canvas</h2>
<p>Once the cards are on the canvas, we can use DndKit again to make them draggable. This is a bit different to dragging / dropping from the tray, as nothing new gets added to the canvas, and instead an existing item changes position.</p>
<p>The <code class="language-plaintext highlighter-rouge">DndContext</code> is much the same as before</p>
<div class="language-tsx highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p"><</span><span class="nc">DndContext</span>
<span class="na">sensors</span><span class="p">=</span><span class="si">{</span><span class="nx">sensors</span><span class="si">}</span>
<span class="na">onDragStart</span><span class="p">=</span><span class="si">{</span><span class="nx">handleDragStart</span><span class="si">}</span> <span class="c1">// stores the activeCard</span>
<span class="na">onDragMove</span><span class="p">=</span><span class="si">{</span><span class="nx">handleDragMove</span><span class="si">}</span> <span class="c1">// uses doCardsCollide (see "Cards should not overlap" later), updates pixelCoordinates</span>
<span class="na">onDragEnd</span><span class="p">=</span><span class="si">{</span><span class="nx">handleDragEnd</span><span class="si">}</span> <span class="c1">// updates position of activeCard</span>
<span class="p">></span>
<span class="si">{</span><span class="nx">children</span><span class="si">}</span>
<span class="p"></</span><span class="nc">DndContext</span><span class="p">></span>
</code></pre></div></div>
<p>The cards on the canvas are slightly more complex, as they have to position themselves on the canvas, and update their position temporarily while they are being dragged.</p>
<div class="language-tsx highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">export</span> <span class="kd">const</span> <span class="nx">Draggable</span> <span class="o">=</span> <span class="p">({</span>
<span class="nx">id</span><span class="p">,</span>
<span class="nx">pixelCoordinates</span><span class="p">,</span>
<span class="nx">k</span><span class="p">,</span>
<span class="nx">children</span><span class="p">,</span>
<span class="p">}:</span> <span class="nx">DraggableProps</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="kd">const</span> <span class="p">{</span> <span class="nx">attributes</span><span class="p">,</span> <span class="nx">listeners</span><span class="p">,</span> <span class="nx">setNodeRef</span><span class="p">,</span> <span class="nx">transform</span> <span class="p">}</span> <span class="o">=</span> <span class="nx">useDraggable</span><span class="p">({</span>
<span class="nx">id</span><span class="p">,</span>
<span class="na">data</span><span class="p">:</span> <span class="p">{</span> <span class="nx">pixelCoordinates</span><span class="p">,</span> <span class="nx">id</span><span class="p">,</span> <span class="nx">ownId</span> <span class="p">},</span>
<span class="p">});</span>
<span class="k">return</span> <span class="p">(</span>
<span class="p"><</span><span class="nt">div</span>
<span class="c1">// position of card on canvas</span>
<span class="na">css</span><span class="p">=</span><span class="si">{</span>
<span class="nx">position</span><span class="p">:</span> <span class="dl">"</span><span class="s2">absolute</span><span class="dl">"</span><span class="p">,</span>
<span class="nx">origin</span><span class="p">:</span> <span class="dl">"</span><span class="s2">top left</span><span class="dl">"</span><span class="p">,</span>
<span class="nx">top</span><span class="p">:</span> <span class="s2">`</span><span class="p">${</span><span class="nx">pixelCoordinates</span><span class="p">.</span><span class="nx">y</span><span class="p">}</span><span class="s2">px`</span><span class="p">,</span>
<span class="nx">left</span><span class="p">:</span> <span class="s2">`</span><span class="p">${</span><span class="nx">pixelCoordinates</span><span class="p">.</span><span class="nx">x</span><span class="p">}</span><span class="s2">px`</span><span class="p">,</span>
<span class="si">}</span>
<span class="c1">// temporary change to this position when dragging</span>
<span class="na">style</span><span class="p">=</span><span class="si">{</span>
<span class="nx">transform</span>
<span class="p">?</span> <span class="p">{</span> <span class="na">transform</span><span class="p">:</span> <span class="s2">`translate3d(</span><span class="p">${</span><span class="nx">transform</span><span class="p">.</span><span class="nx">x</span><span class="p">}</span><span class="s2">, </span><span class="p">${</span><span class="nx">transform</span><span class="p">.</span><span class="nx">y</span><span class="p">}</span><span class="s2">, 0)`</span> <span class="p">}</span>
<span class="p">:</span> <span class="p">{}</span>
<span class="si">}</span>
<span class="na">ref</span><span class="p">=</span><span class="si">{</span><span class="nx">setNodeRef</span><span class="si">}</span>
<span class="si">{</span><span class="p">...</span><span class="nx">listeners</span><span class="si">}</span>
<span class="si">{</span><span class="p">...</span><span class="nx">attributes</span><span class="si">}</span>
<span class="p">></span>
<span class="si">{</span><span class="nx">children</span><span class="si">}</span>
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p">);</span>
<span class="p">};</span>
</code></pre></div></div>
<h2 id="cards-should-not-overlap">Cards should not overlap</h2>
<p>One of the requirements was that cards should not overlap on the canvas, so we needed to detect when collissions would occur and prevent them.</p>
<p>There are two collision detection scenarios, when drag dropping from the tray, and when dragging around the canvas. The 2 situations are very similar, the main differences being that the calculation of the canvas position is different when dropping from the tray, and a card being dragged around the canvas doesn’t need to worry about colliding with itself.</p>
<p>The cards themselves are square, so the code to detect whether two cards collide is trivial. The one minor complication is that the collission detection has to take place after the coordinates are snapped to the grid.</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">doCardsCollide</span> <span class="o">=</span> <span class="p">(</span><span class="nx">card1</span><span class="p">:</span> <span class="nx">Coordinates</span><span class="p">,</span> <span class="nx">card2</span><span class="p">:</span> <span class="nx">Coordinates</span><span class="p">)</span> <span class="o">=></span>
<span class="nb">Math</span><span class="p">.</span><span class="nx">abs</span><span class="p">(</span><span class="nx">card1</span><span class="p">.</span><span class="nx">x</span> <span class="o">-</span> <span class="nx">card2</span><span class="p">.</span><span class="nx">x</span><span class="p">)</span> <span class="o"><</span> <span class="nx">cardSize</span> <span class="o">&&</span>
<span class="nb">Math</span><span class="p">.</span><span class="nx">abs</span><span class="p">(</span><span class="nx">card1</span><span class="p">.</span><span class="nx">y</span> <span class="o">-</span> <span class="nx">card2</span><span class="p">.</span><span class="nx">y</span><span class="p">)</span> <span class="o"><</span> <span class="nx">cardSize</span><span class="p">;</span>
</code></pre></div></div>
<p>When dragging, if a card on the canvas would collide, we add a red overlay to it. When dragging around the canvas, we show the last known good position of a card with a dashed outline, which is where the card will go if it is dropped. This updates as a card is dragged, and snaps to the grid. Where a card on the canvas would cause a collission, the last known position simply stays where it is, until the dragged card is moved in to a collission free space.</p>
<p><img src="/assets/ceddlyburge/visual-sort/collision-detection.png" alt="Collision detection" /></p>
<h2 id="testing">Testing</h2>
<p>We added <a href="https://docs.cypress.io/api/cypress-api/custom-commands">Cypress Custom Commands</a>, like the one below to make it easy to write end to end tests.</p>
<p>The <a href="https://docs.cypress.io/api/commands/wrap">wrap</a> command turns a jquery object in to a cypress object (that you can then chain other cypress comannds off), and the <a href="https://docs.cypress.io/api/commands/trigger">trigger</a> command creates simulated events. There is a slight annoyance in that there are quite a few events triggered in response to various mouse operations, but it is all encapsulated in the custom command so writing the tests is still easy. <code class="language-plaintext highlighter-rouge">{ prevSubject: 'element' }</code> specifies that the <code class="language-plaintext highlighter-rouge">dragOntoCanvas</code> command can only be chained off cypress commands that yield <code class="language-plaintext highlighter-rouge">element</code>’s.</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">Cypress</span><span class="p">.</span><span class="nx">Commands</span><span class="p">.</span><span class="nx">add</span><span class="p">(</span>
<span class="dl">"</span><span class="s2">dragOntoCanvas</span><span class="dl">"</span><span class="p">,</span>
<span class="p">{</span> <span class="na">prevSubject</span><span class="p">:</span> <span class="dl">"</span><span class="s2">element</span><span class="dl">"</span> <span class="p">},</span>
<span class="p">(</span>
<span class="nx">item</span><span class="p">:</span> <span class="nx">JQuery</span><span class="o"><</span><span class="nx">HTMLElement</span><span class="o">></span><span class="p">,</span>
<span class="p">{</span> <span class="nx">startCoordinates</span><span class="p">,</span> <span class="nx">endCoordinates</span> <span class="p">}:</span> <span class="nx">DragOntoCanvasOptions</span>
<span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="c1">// `force: true` shouldn't be needed, but the tests think that</span>
<span class="c1">// the drag overlay is covering the canvas (which is true) and</span>
<span class="c1">// that this prevents mouse operations (which is false)</span>
<span class="kd">const</span> <span class="nx">force</span> <span class="o">=</span> <span class="p">{</span> <span class="na">force</span><span class="p">:</span> <span class="kc">true</span> <span class="p">};</span>
<span class="kd">const</span> <span class="nx">leftButton</span> <span class="o">=</span> <span class="p">{</span> <span class="na">button</span><span class="p">:</span> <span class="mi">0</span> <span class="p">};</span>
<span class="kd">const</span> <span class="nx">dragStart</span> <span class="o">=</span> <span class="p">{</span> <span class="p">...</span><span class="nx">leftButton</span><span class="p">,</span> <span class="p">...</span><span class="nx">startCoordinates</span> <span class="p">};</span>
<span class="kd">const</span> <span class="nx">dragOver</span> <span class="o">=</span> <span class="p">{</span> <span class="p">...</span><span class="nx">endCoordinates</span><span class="p">,</span> <span class="p">...</span><span class="nx">force</span> <span class="p">};</span>
<span class="kd">const</span> <span class="nx">drop</span> <span class="o">=</span> <span class="p">{</span> <span class="p">...</span><span class="nx">leftButton</span><span class="p">,</span> <span class="p">...</span><span class="nx">endCoordinates</span><span class="p">,</span> <span class="p">...</span><span class="nx">force</span> <span class="p">};</span>
<span class="kd">const</span> <span class="nx">pointerEvent</span> <span class="o">=</span> <span class="p">{</span> <span class="na">eventConstructor</span><span class="p">:</span> <span class="dl">"</span><span class="s2">PointerEvent</span><span class="dl">"</span> <span class="p">};</span>
<span class="kd">const</span> <span class="nx">mouseEvent</span> <span class="o">=</span> <span class="p">{</span> <span class="na">eventConstructor</span><span class="p">:</span> <span class="dl">"</span><span class="s2">MouseEvent</span><span class="dl">"</span> <span class="p">};</span>
<span class="kd">const</span> <span class="nx">dragEvent</span> <span class="o">=</span> <span class="p">{</span> <span class="na">eventConstructor</span><span class="p">:</span> <span class="dl">"</span><span class="s2">DragEvent</span><span class="dl">"</span> <span class="p">};</span>
<span class="nx">cy</span><span class="p">.</span><span class="nx">wrap</span><span class="p">(</span><span class="nx">item</span><span class="p">)</span>
<span class="p">.</span><span class="nx">trigger</span><span class="p">(</span><span class="dl">"</span><span class="s2">pointerdown</span><span class="dl">"</span><span class="p">,</span> <span class="p">{</span> <span class="p">...</span><span class="nx">pointerEvent</span><span class="p">,</span> <span class="p">...</span><span class="nx">dragStart</span> <span class="p">})</span>
<span class="p">.</span><span class="nx">trigger</span><span class="p">(</span><span class="dl">"</span><span class="s2">mousedown</span><span class="dl">"</span><span class="p">,</span> <span class="p">{</span> <span class="p">...</span><span class="nx">mouseEvent</span><span class="p">,</span> <span class="p">...</span><span class="nx">dragStart</span> <span class="p">})</span>
<span class="p">.</span><span class="nx">trigger</span><span class="p">(</span><span class="dl">"</span><span class="s2">dragstart</span><span class="dl">"</span><span class="p">,</span> <span class="p">{</span> <span class="p">...</span><span class="nx">dragEvent</span><span class="p">,</span> <span class="p">...</span><span class="nx">force</span> <span class="p">});</span>
<span class="nx">cy</span><span class="p">.</span><span class="nx">findByTestId</span><span class="p">(</span><span class="dl">"</span><span class="s2">canvas</span><span class="dl">"</span><span class="p">)</span>
<span class="p">.</span><span class="nx">trigger</span><span class="p">(</span><span class="dl">"</span><span class="s2">dragover</span><span class="dl">"</span><span class="p">,</span> <span class="p">{</span> <span class="p">...</span><span class="nx">dragEvent</span><span class="p">,</span> <span class="p">...</span><span class="nx">force</span> <span class="p">})</span>
<span class="p">.</span><span class="nx">trigger</span><span class="p">(</span><span class="dl">"</span><span class="s2">mousemove</span><span class="dl">"</span><span class="p">,</span> <span class="p">{</span> <span class="p">...</span><span class="nx">mouseEvent</span><span class="p">,</span> <span class="p">...</span><span class="nx">dragOver</span> <span class="p">})</span>
<span class="p">.</span><span class="nx">trigger</span><span class="p">(</span><span class="dl">"</span><span class="s2">pointermove</span><span class="dl">"</span><span class="p">,</span> <span class="p">{</span> <span class="p">...</span><span class="nx">pointerEvent</span><span class="p">,</span> <span class="p">...</span><span class="nx">dragOver</span> <span class="p">});</span>
<span class="nx">cy</span><span class="p">.</span><span class="nx">findByTestId</span><span class="p">(</span><span class="dl">"</span><span class="s2">canvas</span><span class="dl">"</span><span class="p">)</span>
<span class="p">.</span><span class="nx">trigger</span><span class="p">(</span><span class="dl">"</span><span class="s2">drop</span><span class="dl">"</span><span class="p">,</span> <span class="p">{</span> <span class="p">...</span><span class="nx">dragEvent</span><span class="p">,</span> <span class="p">...</span><span class="nx">force</span> <span class="p">})</span>
<span class="p">.</span><span class="nx">trigger</span><span class="p">(</span><span class="dl">"</span><span class="s2">mouseup</span><span class="dl">"</span><span class="p">,</span> <span class="p">{</span> <span class="p">...</span><span class="nx">mouseEvent</span><span class="p">,</span> <span class="p">...</span><span class="nx">drop</span> <span class="p">})</span>
<span class="p">.</span><span class="nx">trigger</span><span class="p">(</span><span class="dl">"</span><span class="s2">pointerup</span><span class="dl">"</span><span class="p">,</span> <span class="p">{</span> <span class="p">...</span><span class="nx">pointerEvent</span><span class="p">,</span> <span class="p">...</span><span class="nx">drop</span> <span class="p">});</span>
<span class="p">}</span>
<span class="p">);</span>
</code></pre></div></div>
<p>We can then use the custom command in tests like this.</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">cy</span><span class="p">.</span><span class="nx">findAllByText</span><span class="p">(</span><span class="dl">"</span><span class="s2">cb894</span><span class="dl">"</span><span class="p">).</span><span class="nx">dragOntoCanvas</span><span class="p">({</span>
<span class="na">start</span><span class="p">:</span> <span class="p">{</span> <span class="na">clientX</span><span class="p">:</span> <span class="mi">50</span><span class="p">,</span> <span class="na">clientY</span><span class="p">:</span> <span class="mi">50</span> <span class="p">},</span>
<span class="na">end</span><span class="p">:</span> <span class="p">{</span> <span class="na">clientX</span><span class="p">:</span> <span class="mi">700</span><span class="p">,</span> <span class="na">clientY</span><span class="p">:</span> <span class="mi">200</span> <span class="p">},</span>
<span class="p">});</span>
</code></pre></div></div>
<h2 id="wrapping-up">Wrapping up</h2>
<p>So there with have it! A fully tested custom web canvas that you can drag cards to, and then rearrange.</p>
<p>It took around one sprint to spike, and then another to get to v1, and there have been subsequent iterations to add features and improve performance, which may become topics for future posts!</p>
<p>If this sounds like the sort of work you would like to do then <a href="https://red-badger.com/jobs/">come join us</a>, or if you are tackling a similar problem at your company please <a href="mailto:hello@red-badger.com">get in touch</a>.</p>Cedd BurgeWe recently worked with a client who asked us to create a freestyle web canvas, for adding various types of 'Card'.Plotting Fractals in WebAssembly2021-12-07T12:00:00+00:002021-12-07T12:00:00+00:00https://awesome.red-badger.com//chriswhealy/plotting-fractals-in-webassembly<h2 id="introduction">Introduction</h2>
<p>This tutorial is a continuation of the earlier <a href="/chriswhealy/introduction-to-web-assembly-text">Introduction to WebAssembly Text</a></p>
<p>If you are not familiar with WebAssembly Text (WAT), then please read the above introductory tutorial first because from this point on, I will assume that you are at least able to read and understand a WebAssembly Text program.</p>
<p>In the tutorials that follow, we will take a detailed look at how to implement an application in WebAssembly Text that plots the Mandelbrot and Julia Sets</p>
<h2 id="but-why-not-just-write-the-solution-in-rust">But Why Not Just Write the Solution in Rust?</h2>
<p>I did <a href="https://github.com/chriswhealy/fractal_explorer">here</a>!</p>
<p>But here’s the thing…</p>
<p>When I wrote the above solution in Rust, I enjoyed all the advantages of using a language with much richer programming constructs and a compiler that turns out almost bullet-proof code. However, when I used <a href="https://rustwasm.github.io/wasm-pack/installer/"><code class="language-plaintext highlighter-rouge">wasm-pack</code></a> to transform the Rust executable into a <code class="language-plaintext highlighter-rouge">.wasm</code> module, the resulting file was 74Kb in size.</p>
<p>This is certainly not large, but it was much larger than I expected given the simplicity of the task being performed.</p>
<p>So as a matter of both curiosity and education, I set about re-implementing this program in WebAssembly Text (WAT) to see just how small I could get it.</p>
<p>The results are encouraging because the hand-crafted <code class="language-plaintext highlighter-rouge">.wasm</code> file is now about 150 times smaller - just 493 bytes…</p>
<h2 id="live-demo">Live Demo</h2>
<p><a href="https://raw-wasm.pages.dev/">Plotting Fractals Using WebAssembly Threads and Web Workers</a></p>
<h1 id="table-of-contents">Table of Contents</h1>
<ol>
<li><a href="/chriswhealy/FractalWASM/01%20Plotting%20Fractals/">Plotting Fractals</a></li>
<li><a href="/chriswhealy/FractalWASM//02%20Initial%20Implementation/">Initial Implementation</a></li>
<li><a href="/chriswhealy/FractalWASM//03%20WAT%20Basic%20Implementation/">Basic WAT Implementation</a></li>
<li><a href="/chriswhealy/FractalWASM//04%20WAT%20Optimised%20Implementation/">Optimised WAT Implementation</a></li>
<li><a href="/chriswhealy/FractalWASM//05%20MB%20Julia%20Set/">Plotting a Julia Set</a></li>
<li><a href="/chriswhealy/FractalWASM//06%20Zoom%20Image/">Zooming In</a></li>
<li><a href="/chriswhealy/FractalWASM//07%20Web%20Workers/">WebAssembly and Web Workers</a></li>
</ol>Chris WhealyThis set of blogs builds a progressively more optimised set of WebAssembly Text programs that plot the Mandelbrot and Julia Sets.Introduction to WebAssembly Text2021-11-24T12:00:00+00:002021-11-24T12:00:00+00:00https://awesome.red-badger.com//chriswhealy/introduction-to-web-assembly-text<h2 id="what-is-webassembly-text-wat">What is WebAssembly Text (WAT)?</h2>
<p>Any time you write a program in a high level language such as C, Python or Rust, then instruct the compiler to generate a WebAssembly<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup> file, the result will be a <code class="language-plaintext highlighter-rouge">.wasm</code> file containing the binary <a href="https://webassembly.github.io/spec/core/binary/instructions.html">op codes</a> for your program. Since this is an executable file, it is intended only to be machine-readable, not human-readable. Most of the time, this is not a problem; however, as part of your development process, you may need to look inside this machine-readable file: and at this point you need a set of tools that can translate the WebAssembly op codes back into some sort of human-readable format.</p>
<h3 id="working-with-webassembly-files">Working With WebAssembly Files</h3>
<p>When you need to directly manipulate a <code class="language-plaintext highlighter-rouge">.wasm</code> file in some way, the <a href="https://github.com/WebAssembly/wabt">WebAssembly Binary Toolkit</a> (WABT) becomes a vital resource in your development tool bag. WABT provides you with a variety of tools for working directly with WebAssembly files, in both their binary and human-readable forms.</p>
<p>Using the WABT disassembler called <code class="language-plaintext highlighter-rouge">wasm2wat</code>, you can transform a compiled WebAssembly module (a <code class="language-plaintext highlighter-rouge">.wasm</code> file) back into a file containing the plain text op codes used by that program (a <code class="language-plaintext highlighter-rouge">.wat</code> file). This plain text representation is known as WebAssembly Text (or WAT).</p>
<p>In addition to disassembling <code class="language-plaintext highlighter-rouge">.wasm</code> files to plain text <code class="language-plaintext highlighter-rouge">.wat</code> files, WABT also provides you with a tool called <code class="language-plaintext highlighter-rouge">wat2wasm</code> that does the reverse. It takes a plain text WebAssembly Text file and assembles into an executable <code class="language-plaintext highlighter-rouge">.wasm</code> file.</p>
<p>The object of this tutorial is to give an introduction to writing programs directly in WebAssembly Text format.</p>
<h2 id="but-webassembly-is-just-a-compilation-target-so-why-bother">But WebAssembly is Just a Compilation Target, So Why Bother?</h2>
<p>WebAssembly certainly is a compilation target, and it is certainly true that the majority of WebAssembly programs will be generated by compilers, not humans. However, it is a weak line of reasoning to take this fact, and from it, conclude that there are therefore no cases in which it would be beneficial to write a WebAssembly Text program by hand.</p>
<p>Although this development process requires you to write the very low-level WebAssembly Text instructions by hand, it brings with it the benefits of being able to build an extremely small, highly efficient program ideally suited for performing CPU-bound tasks.</p>
<p>This introduction to WebAssembly Text serves as the starting point for a subsequent tutorial that describes how write a browser-based WebAssembly Text program that plots the <a href="./plotting-fractals-in-webassembly">Mandelbrot and Julia Sets</a>.</p>
<h2 id="table-of-contents">Table of Contents</h2>
<ul>
<li><a href="/chriswhealy/Introduction%20to%20WebAssembly%20Text/00/">Prerequisites</a></li>
<li><a href="/chriswhealy/Introduction%20to%20WebAssembly%20Text/01/">Benefits of WebAssembly</a></li>
<li><a href="/chriswhealy/Introduction%20to%20WebAssembly%20Text/02/">Creating a WebAssembly Module</a></li>
<li><a href="/chriswhealy/Introduction%20to%20WebAssembly%20Text/03/">Calling WebAssembly from a Host Environment</a></li>
<li><a href="/chriswhealy/Introduction%20to%20WebAssembly%20Text/04/">WAT Datatypes</a></li>
<li><a href="/chriswhealy/Introduction%20to%20WebAssembly%20Text/05/">Local Variables</a></li>
<li><a href="/chriswhealy/Introduction%20to%20WebAssembly%20Text/06/">Arrangement of WAT Instructions</a></li>
<li><a href="/chriswhealy/Introduction%20to%20WebAssembly%20Text/07/">Conditions</a></li>
<li><a href="/chriswhealy/Introduction%20to%20WebAssembly%20Text/08/">Loops</a></li>
<li><a href="/chriswhealy/Introduction%20to%20WebAssembly%20Text/09/">More About Functions</a></li>
<li><a href="/chriswhealy/Introduction%20to%20WebAssembly%20Text/10/">WASM and Shared Memory</a></li>
</ul>
<hr />
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>Please note: there is no space between the words “Web” and “Assembly” <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Chris WhealyAn introduction to programming directly in WebAssembly TextWrite Better Build Scripts2021-11-11T12:00:00+00:002021-11-11T12:00:00+00:00https://awesome.red-badger.com//stuartharris/write-better-build-scripts<blockquote>
<p>We need simple and efficient build automation both locally, when developing, and in CI pipelines. The use case is slightly different but the goals are the same — scripts need to be simple to write, simple to read and understand, and they need to be repeatable, but we don’t want to repeat unnecessary work.</p>
</blockquote>
<p>It’s a challenging problem, with solutions that date back decades.</p>
<p>One of the oldest tools is <a href="https://www.gnu.org/software/make/"><code class="language-plaintext highlighter-rouge">make</code></a>, which is still extremely popular (and for good reason). However, writing a <code class="language-plaintext highlighter-rouge">Makefile</code> can get complex quite quickly. Its arcane syntax is challenging, making it hard to write, and, in my opinion, even harder to read. And often, the best feature of <code class="language-plaintext highlighter-rouge">make</code> is not even used — the ability to only rebuild a target if it’s inputs have changed (i.e. have a later modification time). Most people add <code class="language-plaintext highlighter-rouge">.PHONY</code> to targets and just use <code class="language-plaintext highlighter-rouge">make</code> as a task runner. This is because it has a built in mechanism for building a Directed Acyclic Graph of dependencies, which is useful, and because the original use case (building C or C++ projects) is not what most people are doing these days (the enlightened are using <a href="https://www.rust-lang.org/">Rust</a> instead). :-)</p>
<p>There are now many tools that have been inspired by <code class="language-plaintext highlighter-rouge">make</code>, including <a href="https://github.com/casey/just"><code class="language-plaintext highlighter-rouge">just</code></a>, and <a href="https://github.com/sagiegurari/cargo-make"><code class="language-plaintext highlighter-rouge">cargo-make</code></a>. They attempt to improve on the syntax of <code class="language-plaintext highlighter-rouge">make</code> and are useful if you want a powerful task runner (and don’t care about the change detection).</p>
<p>At the other end of the spectrum, we have tools like <a href="https://bazel.build/">Bazel</a> and <a href="https://buck.build/">Buck</a>. These are great tools, that do exactly what we want (although they can be quite complex to configure). So why don’t we use them instead? I think it’s because they need you to go all in. If you want to use either tool, you have to use it everywhere — every dependency needs to be tracked and controlled by the tool. This can work well in some ecosystems, e.g. Java. Not so well for others, e.g. Rust. This is because modern languages typically have their own highly capable toolchains (Rust has <a href="https://doc.rust-lang.org/cargo/">Cargo</a>, for instance), and using Bazel, for example, on a Rust project means that you need to stop using Cargo (which is simple and mainstream) and use Bazel instead (which is complex and less widely used).</p>
<p>So is there something in the middle?</p>
<p>I really just need 3 things:</p>
<ol>
<li>a powerful scripting language that everyone knows (i.e. not an esoteric <a href="https://en.wikipedia.org/wiki/Domain-specific_language">DSL</a>), but one that makes it easy to orchestrate shell commands</li>
<li>for local development, the ability to describe inputs and detect if they have changed since I last built</li>
<li>for CI pipelines, a way to track newly changed dependencies in a monorepo, and create a build schedule</li>
</ol>
<p>That’s really it.</p>
<p>The <a href="https://en.wikipedia.org/wiki/Unix_philosophy">UNIX philosophy</a> is all about small, sharp tools that do one thing and do it well. So, instead of trying to find a single tool that does all three of those things, why not choose 3 tools and combine them in a flexible way to achieve the overall goal?</p>
<p>Reluctant as I am to suggest it, the answer to the first need is probably JavaScript. It’s by far the most widely known scripting language. Arguably, you shouldn’t use a Turing-complete language for this job, instead preferring a declarative, rules-based, DSL. But I think JavaScript’s universality is too important to ignore, and it’s really useful to be able to easily manipulate configuration data when building software. But what about it being easy to orchestrate shell commands? This is where <a href="https://github.com/google/zx"><code class="language-plaintext highlighter-rouge">zx</code></a> comes into play, making it incredibly easy to orchestrate shell commands from within JavaScript. We’ll dig into it shortly, but in a nutshell, I’ve been blown away by how simple, yet powerful, it is for this job.</p>
<p>Secondly, if I’m rebuilding over and over again on my laptop, I need a tool to describe my inputs and detect if they have changed. This is really quite simple. Everyone uses git now (thankfully) and so describing inputs should be as simple as “every file, under these directories, that is tracked by git”. Detecting changes is as simple as “compute a hash of all these inputs, which I can compare with the hash from the last successful build”. Fortunately, there is a tool that does precisely that — it’s called <a href="https://github.com/christian-blades-cb/dirsh"><code class="language-plaintext highlighter-rouge">dirsh</code></a> — and it’s written in Rust! (Have you worked out that I love Rust, yet?). Anyway, ignoring its unfortunate name, <code class="language-plaintext highlighter-rouge">dirsh</code> is fast and simple. We’ll look at how to use this shortly.</p>
<p>Finally, we need a way for my CI pipeline to know what it has to build and in which order. It’s becoming more and more popular to use a <a href="https://blog.red-badger.com/why-dont-you-have-a-monorepo">monorepo</a>, and I highly recommend doing so as a way to make it easier to build reliable software. In a monorepo, most of your immediate dependencies are in the same repository. You can commit changes, atomically, across your whole codebase. Versioning problems evaporate. Stability and reliability become much easier to achieve. If you have a monorepo, and you probably should, <a href="https://github.com/charypar/monobuild">monobuild</a>, written by my esteemed colleague, <a href="https://twitter.com/charypar">Viktor Charypar</a>, is is an incredibly useful tool that allows you to graph your dependencies and schedule builds based on what has changed in the current branch (it’s also being <a href="https://github.com/charypar/monobuild/tree/master/rs">rewritten in rust</a>, so will become even better). Monobuild can help us create efficient build schedules and simple pipelines — we’ll look at how to do that below.</p>
<h2 id="using-zx">Using <code class="language-plaintext highlighter-rouge">zx</code></h2>
<p>First install <a href="https://github.com/google/zx"><code class="language-plaintext highlighter-rouge">zx</code></a>:</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>npm <span class="nb">install</span> <span class="nt">--global</span> zx
</code></pre></div></div>
<p>Then create a file (e.g. <code class="language-plaintext highlighter-rouge">touch make.mjs</code>), make it executable (e.g. <code class="language-plaintext highlighter-rouge">chmod +x make.mjs</code>), and add a <a href="https://en.wikipedia.org/wiki/Shebang_(Unix)">shebang</a> at the top:</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/usr/bin/env zx</span>
</code></pre></div></div>
<p>Calling into the shell is as simple as this:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">await</span> <span class="nx">$</span><span class="s2">`pwd`</span><span class="p">;</span>
</code></pre></div></div>
<p>Check this out:</p>
<p><img src="/assets/stuartharris/zx.png" alt="zx" /></p>
<p>This is the first run (because the hash changed, we run the build):</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./make.mjs <span class="nt">--build</span>
<span class="nt">----</span>
Building actor...
<span class="o">{</span>
previous: <span class="s1">'P5FPQUJTDYU4EWWQMA7MXHBCMM'</span>,
current: <span class="s1">'J62CYREXEFFEM45UKTTJ5Q4XQY'</span>
<span class="o">}</span>
<span class="nv">$ </span>cargo build <span class="nt">--release</span>
Compiling wasmcloud-graphql-interface v0.1.0 <span class="o">(</span>/Users/stuartharris/src/wasmCloud/wasmcloud-graphql-provider/interface/rust<span class="o">)</span>
Compiling pass_through v0.1.0 <span class="o">(</span>/Users/stuartharris/src/wasmCloud/wasmcloud-graphql-provider/actor<span class="o">)</span>
Finished release <span class="o">[</span>optimized] target<span class="o">(</span>s<span class="o">)</span> <span class="k">in </span>4.87s
<span class="nv">$ </span>wash claims sign target/wasm32-unknown-unknown/release/pass_through.wasm <span class="nt">--cap</span> <span class="s1">$'stuart-harris:graphql-provider'</span> <span class="nt">--cap</span> <span class="s1">$'wasmcloud:builtin:logging'</span> <span class="nt">--cap</span> <span class="s1">$'wasmcloud:httpserver'</span> <span class="nt">--name</span> pass_through <span class="nt">--ver</span> 0.1.0 <span class="nt">--rev</span> 0 <span class="nt">--destination</span> build/pass_through_s.wasm
Successfully signed build/pass_through_s.wasm with capabilities: stuart-harris:graphql-provider,wasmcloud:builtin:logging,wasmcloud:httpserver
<span class="nv">$ </span>wash claims inspect build/pass_through_s.wasm
pass_through - Module
Account ADHSOZVDL2ZLVX5UXBSGKNLN5UOMU5MPDHN3UQTQ6DYT5TFZ7HGLGIUP
Module MA5PVZ6QNJK5TELQHPQGICJJ2EFVH7YDVXKF2NCUTYGSVVHUCEOL5UW6
Expires never
Can Be Used immediately
Version 0.1.0 <span class="o">(</span>0<span class="o">)</span>
Call Alias <span class="o">(</span>Not <span class="nb">set</span><span class="o">)</span>
Capabilities
stuart-harris:graphql-provider
wasmcloud:builtin:logging
HTTP Server
Tags
None
</code></pre></div></div>
<p>This is the second run (because the hash is the same, we can skip the build):</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./make.mjs <span class="nt">--build</span>
<span class="nt">----</span>
Building actor...
<span class="o">{</span>
previous: <span class="s1">'J62CYREXEFFEM45UKTTJ5Q4XQY'</span>,
current: <span class="s1">'J62CYREXEFFEM45UKTTJ5Q4XQY'</span>
<span class="o">}</span>
<span class="nv">$ </span>wash claims inspect build/pass_through_s.wasm
pass_through - Module
Account ADHSOZVDL2ZLVX5UXBSGKNLN5UOMU5MPDHN3UQTQ6DYT5TFZ7HGLGIUP
Module MA5PVZ6QNJK5TELQHPQGICJJ2EFVH7YDVXKF2NCUTYGSVVHUCEOL5UW6
Expires never
Can Be Used immediately
Version 0.1.0 <span class="o">(</span>0<span class="o">)</span>
Call Alias <span class="o">(</span>Not <span class="nb">set</span><span class="o">)</span>
Capabilities
stuart-harris:graphql-provider
wasmcloud:builtin:logging
HTTP Server
Tags
None
</code></pre></div></div>
<p>Also, parsing JSON is a doddle:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">metadata</span> <span class="o">=</span> <span class="nx">JSON</span><span class="p">.</span><span class="nx">parse</span><span class="p">(</span>
<span class="k">await</span> <span class="nx">$</span><span class="s2">`cargo metadata --no-deps --format-version 1`</span>
<span class="p">);</span>
<span class="kd">const</span> <span class="nx">projectName</span> <span class="o">=</span> <span class="nx">metadata</span><span class="p">.</span><span class="nx">packages</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="nx">name</span><span class="p">;</span>
</code></pre></div></div>
<p>Anyway, you get the idea. Very powerful.</p>
<h2 id="using-dirsh">Using <code class="language-plaintext highlighter-rouge">dirsh</code></h2>
<p>First install <a href="https://github.com/christian-blades-cb/dirsh"><code class="language-plaintext highlighter-rouge">dirsh</code></a>:</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cargo <span class="nb">install </span>dirsh
</code></pre></div></div>
<p>Calling <code class="language-plaintext highlighter-rouge">dirsh</code>, on its own, will cause it to walk down recursively from the current directory (honouring your <code class="language-plaintext highlighter-rouge">.gitignore</code> files, and its own <code class="language-plaintext highlighter-rouge">.hashignore</code> if you need it), feeding file contents (with their modification times and modes) into the digest, and then write the digest to stdout:</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dirsh
</code></pre></div></div>
<p>produces something like this:</p>
<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>JLAU7VF3L5IXQ5L66AXEILCHE4
</code></pre></div></div>
<p>This is how I call it from <code class="language-plaintext highlighter-rouge">zx</code> (passing it an array on input directories):</p>
<p><img src="/assets/stuartharris/dirsh.png" alt="dirsh" /></p>
<p>Pretty cool.</p>
<h2 id="using-monobuild">Using <code class="language-plaintext highlighter-rouge">monobuild</code></h2>
<p>First install <a href="https://github.com/charypar/monobuild"><code class="language-plaintext highlighter-rouge">monobuild</code></a>:</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cargo <span class="nb">install</span> <span class="nt">--git</span> https://github.com/charypar/monobuild
</code></pre></div></div>
<p>Add some files, named <code class="language-plaintext highlighter-rouge">Dependencies</code>, into your monorepo. I’ve made <code class="language-plaintext highlighter-rouge">interface</code> a <em>strong</em> dependency of both <code class="language-plaintext highlighter-rouge">actor</code> and <code class="language-plaintext highlighter-rouge">provider</code> — that’s the <code class="language-plaintext highlighter-rouge">!</code> — because it has some codegen that needs to run:</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bat <span class="k">*</span>/Dependencies
───────┬──────────────────────────────────────────
│ File: actor/Dependencies
───────┼──────────────────────────────────────────
1 │ <span class="o">!</span>interface
───────┴──────────────────────────────────────────
───────┬──────────────────────────────────────────
│ File: interface/Dependencies <EMPTY>
───────┴──────────────────────────────────────────
───────┬──────────────────────────────────────────
│ File: provider/Dependencies
───────┼──────────────────────────────────────────
1 │ <span class="o">!</span>interface
───────┴──────────────────────────────────────────
</code></pre></div></div>
<p>Then get a build schedule based on what has changed (in git) since you cut your branch (or since the <code class="language-plaintext highlighter-rouge">HEAD^1</code> commit, if you’re on <code class="language-plaintext highlighter-rouge">main</code>):</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>monobuild diff <span class="nt">--dot</span>
</code></pre></div></div>
<p>If only <code class="language-plaintext highlighter-rouge">interface</code> had changed, this is what the schedule would look like. Note that because <code class="language-plaintext highlighter-rouge">actor</code> and <code class="language-plaintext highlighter-rouge">provider</code> both depend on <code class="language-plaintext highlighter-rouge">interface</code>, and <code class="language-plaintext highlighter-rouge">interface</code> has changed, we need to rebuild both:</p>
<div class="language-dot highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">digraph</span> <span class="nv">schedule</span> <span class="p">{</span>
<span class="n">randir</span><span class="p">=</span><span class="s2">"LR"</span>
<span class="k">node</span> <span class="o">[</span><span class="n">shape</span><span class="p">=</span><span class="nv">box</span><span class="o">]</span>
<span class="s2">"actor"</span> <span class="o">-></span> <span class="s2">"interface"</span>
<span class="s2">"interface"</span>
<span class="s2">"provider"</span> <span class="o">-></span> <span class="s2">"interface"</span>
<span class="p">}</span>
</code></pre></div></div>
<p>If only <code class="language-plaintext highlighter-rouge">actor</code> had changed, this would be the schedule:</p>
<div class="language-dot highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">digraph</span> <span class="nv">schedule</span> <span class="p">{</span>
<span class="n">randir</span><span class="p">=</span><span class="s2">"LR"</span>
<span class="k">node</span> <span class="o">[</span><span class="n">shape</span><span class="p">=</span><span class="nv">box</span><span class="o">]</span>
<span class="s2">"actor"</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Then all we have to do in our CI pipeline is call the relevant <code class="language-plaintext highlighter-rouge">zx</code> scripts according to the supplied graph. This is easier to parse if you don’t specify <code class="language-plaintext highlighter-rouge">--dot</code>, for example:</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>monobuild diff
</code></pre></div></div>
<p>produces an adjacency list:</p>
<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>actor: interface
interface:
provider: interface, another-dep
</code></pre></div></div>
<p>We can use this graph to decide in the CI pipeline what can be done in parallel and what needs to be done in series. At it’s simplest, creating an ordered list of dependencies to build would require parsing the adjacency list, and using a depth-first algorithm on the graph. Here’s an example that calls <code class="language-plaintext highlighter-rouge">./make.mjs</code> in each of the dependencies in the correct order (but in series):</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">#!/usr/bin/env zx
</span><span class="nx">$</span><span class="p">.</span><span class="nx">verbose</span> <span class="o">=</span> <span class="kc">false</span><span class="p">;</span>
<span class="kd">const</span> <span class="nx">schedule</span> <span class="o">=</span> <span class="nx">getSchedule</span><span class="p">(</span><span class="nx">parse</span><span class="p">((</span><span class="k">await</span> <span class="nx">$</span><span class="s2">`monobuild diff`</span><span class="p">).</span><span class="nx">stdout</span><span class="p">));</span>
<span class="nx">$</span><span class="p">.</span><span class="nx">verbose</span> <span class="o">=</span> <span class="kc">true</span><span class="p">;</span>
<span class="k">for</span> <span class="p">(</span><span class="kd">const</span> <span class="nx">dep</span> <span class="k">of</span> <span class="nx">schedule</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">cd</span><span class="p">(</span><span class="nx">dep</span><span class="p">);</span>
<span class="k">await</span> <span class="nx">$</span><span class="s2">`./make.mjs </span><span class="p">${</span><span class="nx">getArgs</span><span class="p">()}</span><span class="s2">`</span><span class="p">;</span>
<span class="p">}</span>
<span class="kd">function</span> <span class="nx">parse</span><span class="p">(</span><span class="nx">diff</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">diff</span><span class="p">.</span><span class="nx">split</span><span class="p">(</span><span class="dl">"</span><span class="se">\n</span><span class="dl">"</span><span class="p">).</span><span class="nx">reduce</span><span class="p">((</span><span class="nx">acc</span><span class="p">,</span> <span class="nx">line</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="kd">const</span> <span class="p">[</span><span class="nx">vertex</span><span class="p">,</span> <span class="nx">adjacents</span><span class="p">]</span> <span class="o">=</span> <span class="nx">line</span><span class="p">.</span><span class="nx">split</span><span class="p">(</span><span class="dl">"</span><span class="s2">:</span><span class="dl">"</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">vertex</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">acc</span><span class="p">[</span><span class="nx">vertex</span><span class="p">]</span> <span class="o">=</span> <span class="nx">adjacents</span>
<span class="p">.</span><span class="nx">split</span><span class="p">(</span><span class="dl">"</span><span class="s2">,</span><span class="dl">"</span><span class="p">)</span>
<span class="p">.</span><span class="nx">map</span><span class="p">((</span><span class="nx">a</span><span class="p">)</span> <span class="o">=></span> <span class="nx">a</span><span class="p">.</span><span class="nx">trim</span><span class="p">())</span>
<span class="p">.</span><span class="nx">filter</span><span class="p">((</span><span class="nx">a</span><span class="p">)</span> <span class="o">=></span> <span class="nx">a</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">return</span> <span class="nx">acc</span><span class="p">;</span>
<span class="p">},</span> <span class="p">{});</span>
<span class="p">}</span>
<span class="kd">function</span> <span class="nx">getSchedule</span><span class="p">(</span><span class="nx">adjacencyList</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nb">Object</span><span class="p">.</span><span class="nx">keys</span><span class="p">(</span><span class="nx">adjacencyList</span><span class="p">)</span>
<span class="p">.</span><span class="nx">map</span><span class="p">((</span><span class="nx">entryPoint</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">result</span> <span class="o">=</span> <span class="p">[];</span>
<span class="kd">const</span> <span class="nx">visited</span> <span class="o">=</span> <span class="p">{};</span>
<span class="p">(</span><span class="kd">function</span> <span class="nx">dfs</span><span class="p">(</span><span class="nx">vertex</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nx">vertex</span><span class="p">)</span> <span class="k">return</span> <span class="kc">null</span><span class="p">;</span>
<span class="nx">visited</span><span class="p">[</span><span class="nx">vertex</span><span class="p">]</span> <span class="o">=</span> <span class="kc">true</span><span class="p">;</span>
<span class="nx">result</span><span class="p">.</span><span class="nx">push</span><span class="p">(</span><span class="nx">vertex</span><span class="p">);</span>
<span class="nx">adjacencyList</span><span class="p">[</span><span class="nx">vertex</span><span class="p">].</span><span class="nx">forEach</span><span class="p">((</span><span class="nx">neighbour</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nx">visited</span><span class="p">[</span><span class="nx">neighbour</span><span class="p">])</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">dfs</span><span class="p">(</span><span class="nx">neighbour</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">});</span>
<span class="p">})(</span><span class="nx">entryPoint</span><span class="p">);</span>
<span class="k">return</span> <span class="nx">result</span><span class="p">.</span><span class="nx">reverse</span><span class="p">();</span>
<span class="p">})</span> <span class="c1">// depth-first</span>
<span class="p">.</span><span class="nx">flatMap</span><span class="p">((</span><span class="nx">list</span><span class="p">)</span> <span class="o">=></span> <span class="nx">list</span><span class="p">)</span> <span class="c1">// flatten</span>
<span class="p">.</span><span class="nx">filter</span><span class="p">((</span><span class="nx">vertex</span><span class="p">,</span> <span class="nx">index</span><span class="p">,</span> <span class="nb">self</span><span class="p">)</span> <span class="o">=></span> <span class="nb">self</span><span class="p">.</span><span class="nx">indexOf</span><span class="p">(</span><span class="nx">vertex</span><span class="p">)</span> <span class="o">===</span> <span class="nx">index</span><span class="p">);</span> <span class="c1">// unique</span>
<span class="p">}</span>
<span class="kd">function</span> <span class="nx">getArgs</span><span class="p">()</span> <span class="p">{</span>
<span class="k">return</span> <span class="nb">Object</span><span class="p">.</span><span class="nx">keys</span><span class="p">(</span><span class="nx">argv</span><span class="p">)</span>
<span class="p">.</span><span class="nx">filter</span><span class="p">((</span><span class="nx">k</span><span class="p">)</span> <span class="o">=></span> <span class="nx">k</span> <span class="o">!==</span> <span class="dl">"</span><span class="s2">_</span><span class="dl">"</span><span class="p">)</span>
<span class="p">.</span><span class="nx">flatMap</span><span class="p">((</span><span class="nx">a</span><span class="p">)</span> <span class="o">=></span> <span class="p">[</span><span class="s2">`--</span><span class="p">${</span><span class="nx">a</span><span class="p">}</span><span class="s2">`</span><span class="p">,</span> <span class="nx">argv</span><span class="p">[</span><span class="nx">a</span><span class="p">]]);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Wow, that’s more code than we thought. But it’s good that we can even do it — imagine doing something like that in a <code class="language-plaintext highlighter-rouge">Makefile</code>.</p>
<p>Interestingly, even if we don’t flatten our paths, and remove duplicates, we still wouldn’t be doing extra work because of <code class="language-plaintext highlighter-rouge">dirsh</code>.</p>
<p>In a real world, we’d wan’t to project the graph onto a set of CI tasks that run in parallel where they can and series where they can’t. Watch this space :-).</p>
<h2 id="tldr">TLDR</h2>
<p>Build scripts need more flexibility than most declarative build configurations allow for, and, whilst I would always opt for declarative over imperative, I think the flexibility and widespread use of JavaScript gives us superpowers when building software. Especially when we hash our inputs, so that we don’t repeat unnecessary work. Coupled with git-based change detection and dependency graphing in our CI pipelines, we have everything we need for simple, easy-to-grok, repeatable builds.</p>
<p>All the code examples can be found in <a href="https://github.com/StuartHarris/wasmcloud-graphql-provider">this repo</a>, which is a GraphQL provider for <a href="https://wasmcloud.dev">wasmCloud</a> that exposes a postgres database as a GraphQL API.</p>
<p><a href="https://twitter.com/stuartharris">LMK</a> what you think!</p>Stuart HarrisWe need simple and efficient build automation both locally, when developing, and in CI pipelines. The use case is slightly different but the goals are the same — scripts need to be simple to write, simple to read and understand, and they need to be repeatable, but we don’t want to repeat unnecessary work.