Nicky Marino2023-09-13T01:38:46+00:00https://nickymarino.comNicky Marinonicky.s.marino@gmail.comShorty Diary #10: Enable non-dev deployments
2023-06-15T00:00:00+00:00https://nickymarino.com/2023/06/15/shorty-10-enable-non-dev-deploys<p>To sanity check my work so far, I tried deploying a new non-development <a href="https://sst.dev/chapters/stages-in-serverless-framework.html">stage</a>, but I hit a few errors:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>npx sst deploy <span class="nt">--stage</span> <span class="nb">test</span>
</code></pre></div></div>
<p>Luckily, these errors were pretty small! I just need to JSON-ify the output values for a few lambdas I had missed:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// functions/link/get.ts</span>
<span class="kd">const</span> <span class="nx">result</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">Link</span><span class="p">.</span><span class="kd">get</span><span class="p">(</span><span class="nx">uid</span><span class="o">!</span><span class="p">);</span>
<span class="k">return</span> <span class="p">{</span>
<span class="na">body</span><span class="p">:</span> <span class="nx">JSON</span><span class="p">.</span><span class="nx">stringify</span><span class="p">({</span>
<span class="na">link</span><span class="p">:</span> <span class="nx">result</span><span class="p">,</span>
<span class="p">},</span>
<span class="p">})</span>
</code></pre></div></div>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// functions/link/list.ts</span>
<span class="kd">const</span> <span class="nx">result</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">Link</span><span class="p">.</span><span class="nx">list</span><span class="p">();</span>
<span class="k">return</span> <span class="p">{</span>
<span class="na">body</span><span class="p">:</span> <span class="nx">JSON</span><span class="p">.</span><span class="nx">stringify</span><span class="p">({</span>
<span class="na">links</span><span class="p">:</span> <span class="nx">result</span><span class="p">,</span>
<span class="p">},</span>
<span class="p">})</span>
</code></pre></div></div>
<p>For good measure, I also added <code class="language-plaintext highlighter-rouge">eslint-config-next</code> to <code class="language-plaintext highlighter-rouge">package.json</code> and upgraded SST to <code class="language-plaintext highlighter-rouge">2.11</code>. These changes fixed my build issues, and I was able to deploy a prod-ish version of my code to AWS!</p>
<p>You can view these changes in the <a href="https://github.com/nickymarino/shorty/pull/20/files"><strong>Enable non-dev deployments</strong></a> PR.</p>
Dev Journal #9: From Many, a Mono Package
2023-06-14T00:00:00+00:00https://nickymarino.com/2023/06/14/shorty-09-from-many-a-monopackage<p>I hit a snag in my dev experience: for some reason, Visual Studio Code isn’t able to understand my submodules under the <code class="language-plaintext highlighter-rouge">packages/</code> folder anymore, so any third party libraries in <code class="language-plaintext highlighter-rouge">packages/core</code> or <code class="language-plaintext highlighter-rouge">packages/functions</code> are unknown:</p>
<p><img src="https://nickymarino.com/public/assets/2023/url-shortener/09-vs-code-error-1.png" alt="VSCode complier error" class="center" /></p>
<p>Also concerning: VSCode doesn’t appear to understand the Typescript import in <code class="language-plaintext highlighter-rouge">tsconfig.json</code> files in <code class="language-plaintext highlighter-rouge">packages/</code>:</p>
<p><img src="https://nickymarino.com/public/assets/2023/url-shortener/09-vs-code-error-2.png" alt="VSCode complier error" class="center" /></p>
<p>Research didn’t turn up much; the solutions were <a href="https://github.com/microsoft/vscode-eslint/issues/722">beyond</a> <a href="https://stackoverflow.com/questions/38268776/why-does-typescript-ignore-my-tsconfig-json-inside-visual-studio-code">me</a> at best. Searching the SST Discord didn’t turn up anything either, so I decided to re-examine the <a href="https://github.com/serverless-stack/sst/tree/master/examples/quickstart-nextjs">SST NextJS example</a>.</p>
<h2 id="the-setup-change">The Setup Change</h2>
<p>Previously, borrowing from the <a href="https://github.com/serverless-stack/sst/tree/master/examples/rest-api-dynamodb">SST DynamoDB REST API example</a>, I had a <code class="language-plaintext highlighter-rouge">packages/</code> folder that included submodules for <code class="language-plaintext highlighter-rouge">core</code> and <code class="language-plaintext highlighter-rouge">links</code>:</p>
<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>├── packages
│ ├── core
│ │ ├── src
│ │ │ ├── dynamo.ts
│ │ │ └── link.ts
│ │ ├── package.json
│ │ └── tsconfig.json
│ └── functions
│ ├── src
│ │ ├── link
│ │ │ ├── create.ts
│ │ │ ├── get.ts
│ │ │ ├── list.ts
│ │ │ └── patch.ts
│ │ └── redirect.ts
│ ├── package.json
│ └── tsconfig.json
├── src
# NextJS files
├── stacks
│ ├── API.ts
│ ├── Database.ts
│ └── Site.ts
├── package.json
├── sst.config.ts
└── tsconfig.json
</code></pre></div></div>
<p>Both <code class="language-plaintext highlighter-rouge">packages/core</code> and <code class="language-plaintext highlighter-rouge">packages/functions</code> had separate <code class="language-plaintext highlighter-rouge">tsconfig.json</code> and <code class="language-plaintext highlighter-rouge">package.json</code> files, so they were independent submodules. However, SST’s <a href="https://github.com/serverless-stack/sst/tree/master/examples/quickstart-nextjs">NextJS example</a> used a “mono package” setup, with a <code class="language-plaintext highlighter-rouge">functions/</code> folder at the top level that did not have its own <code class="language-plaintext highlighter-rouge">tsconfig.json</code> or <code class="language-plaintext highlighter-rouge">package.json</code>. I updated my project to use that pattern instead:</p>
<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>.
├── functions
│ ├── core
│ │ ├── dynamo.ts
│ │ └── link.ts
│ ├── link
│ │ ├── create.ts
│ │ ├── get.ts
│ │ ├── list.ts
│ │ └── patch.ts
│ └── redirect.ts
├── src
# NextJS files
├── stacks
│ ├── API.ts
│ ├── Database.ts
│ └── Site.ts
├── package.json
├── sst.config.ts
└── tsconfig.json
</code></pre></div></div>
<p>I’m not 100% sure what the issue was here. My guess is that the way Next projects are complied is different than how the submodule pattern expects to be compiled, and while SST can deploy that without issue, VSCode doesn’t expect that setup and got confused. Neverless, I’m glad this was a relatively straightforward fix that lets me move forward.</p>
<h2 id="a-minor-rename">A Minor Rename</h2>
<p><a href="https://www.youtube.com/watch?v=5SHMDMJPuwM">One more thing</a>: I missed one rename earlier for <code class="language-plaintext highlighter-rouge">LinkEntity::service</code> name. It’s a quick fix under (the renamed) <code class="language-plaintext highlighter-rouge">functions/core/link</code>:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">export</span> <span class="kd">const</span> <span class="nx">LinkEntity</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">Entity</span><span class="p">(</span>
<span class="p">{</span>
<span class="na">model</span><span class="p">:</span> <span class="p">{</span>
<span class="na">entity</span><span class="p">:</span> <span class="dl">"</span><span class="s2">links</span><span class="dl">"</span><span class="p">,</span>
<span class="na">version</span><span class="p">:</span> <span class="dl">"</span><span class="s2">1</span><span class="dl">"</span><span class="p">,</span>
<span class="na">service</span><span class="p">:</span> <span class="dl">"</span><span class="s2">shorty</span><span class="dl">"</span><span class="p">,</span>
<span class="p">},</span>
<span class="c1">// ...</span>
<span class="p">}</span>
<span class="p">)</span>
</code></pre></div></div>
<p>These changes are in the PRs <a href="https://github.com/nickymarino/shorty/pull/16"><strong>Switch from subpackages to one package</strong></a> and <a href="https://github.com/nickymarino/shorty/pull/17"><strong>Switch LinkEntity service name to shorty</strong></a>.</p>
Dev Journal #8: I dub thee shorty
2023-06-13T00:00:00+00:00https://nickymarino.com/2023/06/13/url-shortener-08-dub-thee-shorty<p>So far, I’ve been referring to this project as just <code class="language-plaintext highlighter-rouge">URL Shortener</code>, which doesn’t do it justice. If I’m creating the Next Web Scale Startup™, I need a cool and flashy name for this project.</p>
<p>I spent a few hours poking at possible domain names in <a href="https://aws.amazon.com/route53/">AWS Route 53</a>, but it was tough to use. Every search would take a few seconds, none of my ideas were able to be purchased, and Route 53 even started recommending me domains that I had <em>already searched for</em> that were previously listed as unavailable.</p>
<p>I realized that since my plan is to host on <code class="language-plaintext highlighter-rouge">link.nickymarino.com</code> eventually, I don’t need a custom domain. I can just use Route 53 as the <a href="https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/creating-migrating.html">DNS service for the subdomain</a> and <a href="https://docs.sst.dev/custom-domains">import the custom domain in SST</a> without worrying about migrating the parent domain <code class="language-plaintext highlighter-rouge">nickymarino.com</code> away from Hover<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup>.</p>
<p>So! Unconstrained by the fickle beast of whatever domains are up for grabs and don’t cost $300/month, <a href="https://www.youtube.com/watch?v=Rayuw_gSJ-s">without further ado</a>, we’ll be doing business here as <code class="language-plaintext highlighter-rouge">shorty</code>.</p>
<p>Updating the code with the new name is pretty trivial. We’ll change the name formally in <code class="language-plaintext highlighter-rouge">package.json</code>:</p>
<div class="language-jsonc highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"shorty"</span><span class="p">,</span><span class="w">
</span><span class="c1">// ...</span><span class="w">
</span></code></pre></div></div>
<p>As well as in <code class="language-plaintext highlighter-rouge">sst.config.ts</code>:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">export</span> <span class="k">default</span> <span class="p">{</span>
<span class="nx">config</span><span class="p">(</span><span class="nx">_input</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="p">{</span>
<span class="na">name</span><span class="p">:</span> <span class="dl">"</span><span class="s2">shorty</span><span class="dl">"</span><span class="p">,</span>
<span class="na">region</span><span class="p">:</span> <span class="dl">"</span><span class="s2">us-east-1</span><span class="dl">"</span><span class="p">,</span>
<span class="p">};</span>
<span class="p">},</span>
<span class="nx">stacks</span><span class="p">(</span><span class="nx">app</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">app</span><span class="p">.</span><span class="nx">stack</span><span class="p">(</span><span class="nx">Database</span><span class="p">);</span>
<span class="nx">app</span><span class="p">.</span><span class="nx">stack</span><span class="p">(</span><span class="nx">API</span><span class="p">);</span>
<span class="nx">app</span><span class="p">.</span><span class="nx">stack</span><span class="p">(</span><span class="nx">Site</span><span class="p">);</span>
<span class="p">},</span>
<span class="p">}</span> <span class="nx">satisfies</span> <span class="nx">SSTConfig</span><span class="p">;</span>
</code></pre></div></div>
<p>And then we’ll need to update each of the sub-module <code class="language-plaintext highlighter-rouge">package.json</code> files, such as in <code class="language-plaintext highlighter-rouge">packages/core/package.json</code>:</p>
<div class="language-jsonc highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"@shorty/core"</span><span class="p">,</span><span class="w">
</span><span class="c1">// ...</span><span class="w">
</span></code></pre></div></div>
<p>Each import statement needs to be updated too, such as in <code class="language-plaintext highlighter-rouge">packages/functions/src/redirect.ts</code>:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="p">{</span> <span class="nx">Link</span> <span class="p">}</span> <span class="k">from</span> <span class="dl">"</span><span class="s2">@shorty/core/link</span><span class="dl">"</span><span class="p">;</span>
<span class="c1">// ...</span>
</code></pre></div></div>
<p>This was super straightforward using VSCode’s <a href="https://www.roboleary.net/vscode/2022/12/28/global-find-and-replace-all-text-in-vscode.html#:~:text=Alternatively%2C%20you%20can%20press%20Ctrl,Hurrah!">project-wide find and replace</a>. However, because the name of the app changed, SST deploys my stack as <code class="language-plaintext highlighter-rouge">test-shorty</code> (in <code class="language-plaintext highlighter-rouge">STAGE-APPNAME</code> format) instead of <code class="language-plaintext highlighter-rouge">test-cow-link</code><sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote" rel="footnote">2</a></sup>. To sanity check, let’s re-populate the DB with some links and load the UI to make sure it’s all connected:</p>
<p><img src="https://nickymarino.com/public/assets/2023/url-shortener/08-link-list.png" alt="UI showing a list of links" class="center" /></p>
<p>And it works! You can view the code on the <a href="https://github.com/nickymarino/url-shortener/pull/13"><strong>Rename project to shorty</strong></a> PR.</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p><a href="https://www.hover.com/">Hover</a> is fantastic, by the way! I’ve used them for years and years now, and I’ve had a great experience. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>And I had to manually delete each of the stacks for the old deploy. Grumble grumble. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
How to Fix Liquid Syntax Errors in React Code Snippets2023-06-13T00:00:00+00:00https://nickymarino.com/2023/06/13/fix-liquid-syntax-errors<p>While publishing my <a href="https://nickymarino.com/2023/06/05/building-a-url-shortener/">series of dev diaries</a> for my URL shortener project, I encountered some strange errors from <a href="https://jekyllrb.com/">Jekyll</a>. Below is example code to create a table using MUI in React:</p>
<!--
Disable liquid parsing on this codeblock to prevent errors reading '{{'
See: https://talk.jekyllrb.com/t/code-block-is-improperly-handled-and-generates-liquid-syntax-error/7599/2
-->
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">return</span> <span class="p">(</span>
<span class="o"><</span><span class="nx">Table</span> <span class="nx">sx</span><span class="o">=</span><span class="p">{{</span> <span class="na">minWidth</span><span class="p">:</span> <span class="mi">650</span> <span class="p">}}</span> <span class="nx">aria</span><span class="o">-</span><span class="nx">label</span><span class="o">=</span><span class="dl">"</span><span class="s2">simple table</span><span class="dl">"</span><span class="o">></span>
<span class="o"><</span><span class="nx">TableHead</span><span class="o">></span>
<span class="o"><</span><span class="nx">TableRow</span><span class="o">></span>
<span class="o"><</span><span class="nx">TableCell</span><span class="o">></span><span class="nx">Short</span> <span class="nx">Path</span><span class="o"><</span><span class="sr">/TableCell</span><span class="err">>
</span> <span class="o"><</span><span class="nx">TableCell</span><span class="o">></span><span class="nx">URL</span><span class="o"><</span><span class="sr">/TableCell</span><span class="err">>
</span> <span class="o"><</span><span class="nx">TableCell</span><span class="o">></span><span class="nx">Created</span><span class="o"><</span><span class="sr">/TableCell</span><span class="err">>
</span> <span class="o"><</span><span class="nx">TableCell</span><span class="o">></span><span class="nx">Updated</span><span class="o"><</span><span class="sr">/TableCell</span><span class="err">>
</span> <span class="o"><</span><span class="sr">/TableRow</span><span class="err">>
</span> <span class="o"><</span><span class="sr">/TableHead</span><span class="err">>
</span> <span class="o"><</span><span class="nx">TableBody</span><span class="o">></span>
<span class="p">{</span><span class="nx">links</span><span class="p">.</span><span class="nx">map</span><span class="p">((</span><span class="nx">row</span><span class="p">)</span> <span class="o">=></span> <span class="p">(</span>
<span class="o"><</span><span class="nx">TableRow</span>
<span class="nx">key</span><span class="o">=</span><span class="p">{</span><span class="nx">row</span><span class="p">.</span><span class="nx">uid</span><span class="p">}</span>
<span class="nx">sx</span><span class="o">=</span><span class="p">{{</span> <span class="dl">"</span><span class="s2">&:last-child td, &:last-child th</span><span class="dl">"</span><span class="p">:</span> <span class="p">{</span> <span class="na">border</span><span class="p">:</span> <span class="mi">0</span> <span class="p">}</span> <span class="p">}}</span>
<span class="o">></span>
<span class="o"><</span><span class="nx">TableCell</span> <span class="nx">component</span><span class="o">=</span><span class="dl">"</span><span class="s2">th</span><span class="dl">"</span> <span class="nx">scope</span><span class="o">=</span><span class="dl">"</span><span class="s2">row</span><span class="dl">"</span><span class="o">></span>
<span class="p">{</span><span class="nx">row</span><span class="p">.</span><span class="nx">shortPath</span><span class="p">}</span>
<span class="o"><</span><span class="sr">/TableCell</span><span class="err">>
</span> <span class="o"><</span><span class="nx">TableCell</span><span class="o">></span><span class="p">{</span><span class="nx">row</span><span class="p">.</span><span class="nx">url</span><span class="p">}</span><span class="o"><</span><span class="sr">/TableCell</span><span class="err">>
</span> <span class="o"><</span><span class="nx">TableCell</span><span class="o">></span><span class="p">{</span><span class="nx">toTimestamp</span><span class="p">(</span><span class="nx">row</span><span class="p">.</span><span class="nx">createdAt</span><span class="p">)}</span><span class="o"><</span><span class="sr">/TableCell</span><span class="err">>
</span> <span class="o"><</span><span class="nx">TableCell</span><span class="o">></span><span class="p">{</span><span class="nx">toTimestamp</span><span class="p">(</span><span class="nx">row</span><span class="p">.</span><span class="nx">updatedAt</span><span class="p">)}</span><span class="o"><</span><span class="sr">/TableCell</span><span class="err">>
</span> <span class="o"><</span><span class="sr">/TableRow</span><span class="err">>
</span> <span class="p">))}</span>
<span class="o"><</span><span class="sr">/TableBody</span><span class="err">>
</span> <span class="o"><</span><span class="sr">/Table</span><span class="err">>
</span> <span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>When I save this codeblock to a markdown file, I get a strange syntax error from liquid out of nowhere:</p>
<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Regenerating: 2 file(s) changed at 2023-06-05 19:31:13
_posts/2023-06-13-yyyy.md
.history/_posts/2023-06-13-yyyy_20230605193112.md
Liquid Exception: Liquid syntax error (line 70): Variable '{{ "&:last-child td, &:last-child th": { border: 0 }' was not properly terminated with regexp: /\}\}/ in /Users/nickymarino/Dropbox/Mac (3)/Documents/Developer/nickymarino.github.io/_posts/2023-06-13-yyyy.md
Error: Liquid syntax error (line 70): Variable '{{ "&:last-child td, &:last-child th": { border: 0 }' was not properly terminated with regexp: /\}\}/
Error: Run jekyll build --trace for more information.
</code></pre></div></div>
<p>This was really strange to me. At first I couldn’t figure out what <a href="https://shopify.github.io/liquid/">liquid</a> even was–I had forgotten it’s a tool Jekyll uses to pre-process markdown files–and I wasn’t sure why liquid was looking inside the codeblock.</p>
<p>Thanks to <a href="https://talk.jekyllrb.com/t/code-block-is-improperly-handled-and-generates-liquid-syntax-error/7599">chuckhoupt’s comment</a> on the Jekyll help forum though, I learned I can wrap my code in a <a href="https://shopify.github.io/liquid/tags/template/#raw"><code class="language-plaintext highlighter-rouge">raw</code> tag</a> to prevent processing:</p>
<!-- -->
<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><!--
{% raw %}
Disable liquid parsing on this codeblock to prevent errors reading '{{'
See: https://talk.jekyllrb.com/t/code-block-is-improperly-handled-and-generates-liquid-syntax-error/7599/2
-->
```ts
return (
<Table sx={{ minWidth: 650 }} aria-label="simple table">
<TableHead>
<TableRow>
<TableCell>Short Path</TableCell>
<TableCell>URL</TableCell>
<TableCell>Created</TableCell>
<TableCell>Updated</TableCell>
</TableRow>
</TableHead>
<TableBody>
{links.map((row) => (
<TableRow
key={row.uid}
sx={{ "&:last-child td, &:last-child th": { border: 0 } }}
>
<TableCell component="th" scope="row">
{row.shortPath}
</TableCell>
<TableCell>{row.url}</TableCell>
<TableCell>{toTimestamp(row.createdAt)}</TableCell>
<TableCell>{toTimestamp(row.updatedAt)}</TableCell>
</TableRow>
))}
</TableBody>
</Table>
);
}
```
<!-- {% endraw %} -->
</code></pre></div></div>
<p>While writing this post, I also discovered <a href="https://joshtronic.com/2020/05/24/how-to-escape-curly-brackets-in-liquid-templates/">this trick by Josh</a> to wrap the double quotes in a variable:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{{</span>
</code></pre></div></div>
<p>The downside is that it leads to some strange looking code. For example, to escape the above <code class="language-plaintext highlighter-rouge">raw</code> and <code class="language-plaintext highlighter-rouge">endraw</code> tags, I used:</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"><!--
{% raw %}
...
--></span>
<span class="c"><!-- {% endraw %} --></span>
</code></pre></div></div>
<p>Outside of code snippets, you can use the HTML entities as well: { and }. And to write this post, I had to use <a href="https://stackoverflow.com/a/31834381/2597913">this trick by Waylan</a> to use more backticks to mark the start and end of a snippet:</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
````text
<span class="c"><!-- ... --></span>
```ts
return ( ... )
```
<span class="c"><!-- ... --></span>
````
</code></pre></div></div>
Dev Journal #7: Project Management via GitHub Projects
2023-06-12T00:00:00+00:00https://nickymarino.com/2023/06/12/url-shortener-07-project-management<p>I’m pretty big into the productivity space; for a long while I bounced back and forth between <a href="https://todoist.com/">Todoist</a>, <a href="https://www.omnigroup.com/omnifocus/">OmniFocus</a>, <a href="https://support.apple.com/en-us/HT205890">Apple Reminders</a>, and even a few <a href="https://bulletjournal.com/">bullet journals</a>. These days, I’m heavily invested in Todoist, which begs the question: what’s the best way to keep track of features, ideas, and tasks for this project?</p>
<p>While Todoist was an obvious option, I initially started with <a href="https://github.com/features/issues">GitHub Issues</a> for a few reasons:</p>
<ul>
<li>Issues are on the public repo, so they force me to <a href="https://gabygoldberg.medium.com/the-building-in-public-how-to-guide-219d417f00c1">build in public</a>.</li>
<li>I can create new PRs from issues easily.</li>
<li>Most open source projects use issues to track items, and this is a good sandbox to test out developer flows.</li>
</ul>
<p>At first, I was throwing things into issues and gave them “labels” using suffixes like <code class="language-plaintext highlighter-rouge">Feature</code> and <code class="language-plaintext highlighter-rouge">DevOps</code>:</p>
<p><img src="https://nickymarino.com/public/assets/2023/url-shortener/07-issues.png" alt="List of GitHub issues for this repo" class="center" /></p>
<p>I hit a few snags quickly though. For one, it takes a <em>lot</em> of clicking to rename titles or add new labels to each issue. Most importantly, I don’t have a way of sorting issues by priority like a todo list.</p>
<p>GitHub projects was recently re-released and refreshed, so I decided to give it a go. It’s still pretty easy to create a new project, and I honestly don’t see much of a difference other than the list view is now the default.</p>
<p><img src="https://nickymarino.com/public/assets/2023/url-shortener/07-new-project.png" alt="GitHub's "Create project" UI" class="center" /></p>
<p>After some tweaks, the <a href="https://github.com/users/nickymarino/projects/2">project</a> is in a place I’m happy with. It’s a <a href="https://www.atlassian.com/agile/kanban/boards">kanban board</a> with three states:</p>
<ol>
<li><strong>No Status:</strong> These are ideas I’d like to do at some point in the future, but they’re not critical to an MVP, such as improved error handling and a polished index page.</li>
<li><strong>Prioritized:</strong> A ranked list of tasks that are critical to get the app working, where the top item is the one I want to do next.</li>
<li><strong>Done:</strong> These are items that are complete and merged via PR.</li>
</ol>
<p>I’m really happy with this setup; I can put all good ideas under <strong>No Status</strong>, I move the critical ones to <strong>Prioritized</strong>, and I only need to create issues for those that are actively on my plate to complete in the short term. Here’s a snapshot of <a href="https://github.com/users/nickymarino/projects/2">the board</a>:</p>
<p><img src="https://nickymarino.com/public/assets/2023/url-shortener/07-kanban-board.png" alt="Current kanban board" class="center" /></p>
<p>You can view the <a href="https://github.com/nickymarino/url-shortener/issues">issues for this repo here</a> and the <a href="https://github.com/users/nickymarino/projects/2">kanban project here</a>.</p>
Dev Journal #6: Add a MUI table component
2023-06-11T00:00:00+00:00https://nickymarino.com/2023/06/11/url-shortener-06-use-a-mui-table<p>Next, let’s use a proper MUI table component to display the list of links. At first, I toyed with using <a href="https://mui.com/x/react-data-grid/">MUI X’s Data Grid component</a>, but because it was an additional package with a freemium model, I decided to use a more basic display for the first crack.</p>
<p>MUI’s <a href="https://mui.com/material-ui/react-table/#basic-table">basic table example</a> is pretty good, so we’ll more or less lift that and modify it slightly to show our link-specific columns in <code class="language-plaintext highlighter-rouge">src/pages/index.tsx</code>:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">function</span> <span class="nx">LinkTable</span><span class="p">({</span> <span class="nx">links</span> <span class="p">}:</span> <span class="p">{</span> <span class="nl">links</span><span class="p">:</span> <span class="nx">Link</span><span class="p">[]</span> <span class="p">})</span> <span class="p">{</span>
<span class="kd">function</span> <span class="nx">toTimestamp</span><span class="p">(</span><span class="nx">epoch</span><span class="p">:</span> <span class="kr">number</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="k">new</span> <span class="nb">Date</span><span class="p">(</span><span class="nx">epoch</span><span class="p">).</span><span class="nx">toString</span><span class="p">();</span>
<span class="p">}</span>
<span class="k">return</span> <span class="p">(</span>
<span class="o"><</span><span class="nx">Table</span> <span class="nx">sx</span><span class="o">=</span><span class="p">{{</span> <span class="na">minWidth</span><span class="p">:</span> <span class="mi">650</span> <span class="p">}}</span> <span class="nx">aria</span><span class="o">-</span><span class="nx">label</span><span class="o">=</span><span class="dl">"</span><span class="s2">simple table</span><span class="dl">"</span><span class="o">></span>
<span class="o"><</span><span class="nx">TableHead</span><span class="o">></span>
<span class="o"><</span><span class="nx">TableRow</span><span class="o">></span>
<span class="o"><</span><span class="nx">TableCell</span><span class="o">></span><span class="nx">Short</span> <span class="nx">Path</span><span class="o"><</span><span class="sr">/TableCell</span><span class="err">>
</span> <span class="o"><</span><span class="nx">TableCell</span><span class="o">></span><span class="nx">URL</span><span class="o"><</span><span class="sr">/TableCell</span><span class="err">>
</span> <span class="o"><</span><span class="nx">TableCell</span><span class="o">></span><span class="nx">Created</span><span class="o"><</span><span class="sr">/TableCell</span><span class="err">>
</span> <span class="o"><</span><span class="nx">TableCell</span><span class="o">></span><span class="nx">Updated</span><span class="o"><</span><span class="sr">/TableCell</span><span class="err">>
</span> <span class="o"><</span><span class="sr">/TableRow</span><span class="err">>
</span> <span class="o"><</span><span class="sr">/TableHead</span><span class="err">>
</span> <span class="o"><</span><span class="nx">TableBody</span><span class="o">></span>
<span class="p">{</span><span class="nx">links</span><span class="p">.</span><span class="nx">map</span><span class="p">((</span><span class="nx">row</span><span class="p">)</span> <span class="o">=></span> <span class="p">(</span>
<span class="o"><</span><span class="nx">TableRow</span>
<span class="nx">key</span><span class="o">=</span><span class="p">{</span><span class="nx">row</span><span class="p">.</span><span class="nx">uid</span><span class="p">}</span>
<span class="nx">sx</span><span class="o">=</span><span class="p">{{</span> <span class="dl">"</span><span class="s2">&:last-child td, &:last-child th</span><span class="dl">"</span><span class="p">:</span> <span class="p">{</span> <span class="na">border</span><span class="p">:</span> <span class="mi">0</span> <span class="p">}</span> <span class="p">}}</span>
<span class="o">></span>
<span class="o"><</span><span class="nx">TableCell</span> <span class="nx">component</span><span class="o">=</span><span class="dl">"</span><span class="s2">th</span><span class="dl">"</span> <span class="nx">scope</span><span class="o">=</span><span class="dl">"</span><span class="s2">row</span><span class="dl">"</span><span class="o">></span>
<span class="p">{</span><span class="nx">row</span><span class="p">.</span><span class="nx">shortPath</span><span class="p">}</span>
<span class="o"><</span><span class="sr">/TableCell</span><span class="err">>
</span> <span class="o"><</span><span class="nx">TableCell</span><span class="o">></span><span class="p">{</span><span class="nx">row</span><span class="p">.</span><span class="nx">url</span><span class="p">}</span><span class="o"><</span><span class="sr">/TableCell</span><span class="err">>
</span> <span class="o"><</span><span class="nx">TableCell</span><span class="o">></span><span class="p">{</span><span class="nx">toTimestamp</span><span class="p">(</span><span class="nx">row</span><span class="p">.</span><span class="nx">createdAt</span><span class="p">)}</span><span class="o"><</span><span class="sr">/TableCell</span><span class="err">>
</span> <span class="o"><</span><span class="nx">TableCell</span><span class="o">></span><span class="p">{</span><span class="nx">toTimestamp</span><span class="p">(</span><span class="nx">row</span><span class="p">.</span><span class="nx">updatedAt</span><span class="p">)}</span><span class="o"><</span><span class="sr">/TableCell</span><span class="err">>
</span> <span class="o"><</span><span class="sr">/TableRow</span><span class="err">>
</span> <span class="p">))}</span>
<span class="o"><</span><span class="sr">/TableBody</span><span class="err">>
</span> <span class="o"><</span><span class="sr">/Table</span><span class="err">>
</span> <span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>We’ll use the new <code class="language-plaintext highlighter-rouge">LinkTable</code> component on the <code class="language-plaintext highlighter-rouge">Home</code> page in the same file:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">export</span> <span class="k">default</span> <span class="kd">function</span> <span class="nx">Home</span><span class="p">({</span> <span class="nx">links</span> <span class="p">}:</span> <span class="p">{</span> <span class="nl">links</span><span class="p">?:</span> <span class="nx">Link</span><span class="p">[]</span> <span class="p">})</span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">({</span> <span class="nx">links</span> <span class="p">});</span>
<span class="k">return</span> <span class="p">(</span>
<span class="o"><</span><span class="nx">Container</span> <span class="nx">maxWidth</span><span class="o">=</span><span class="dl">"</span><span class="s2">lg</span><span class="dl">"</span><span class="o">></span>
<span class="p">{</span><span class="cm">/* ... */</span><span class="p">}</span>
<span class="p">{</span><span class="nx">links</span> <span class="o">&&</span> <span class="nx">LinkTable</span><span class="p">({</span> <span class="nx">links</span> <span class="p">})}</span>
</code></pre></div></div>
<p>And then our links show up in a neat table! You can view this stage of the project on the <a href="https://github.com/nickymarino/url-shortener/pull/11"><strong>Use table for link list</strong></a> PR. Here’s what the UI looks like now:</p>
<p><img src="https://nickymarino.com/public/assets/2023/url-shortener/06-mui-table-for-links.png" alt="Links show in a table on the UI" class="center" /></p>
Dev Journal #5: A UI to list links
2023-06-10T00:00:00+00:00https://nickymarino.com/2023/06/10/url-shortener-05-basic-ui<p>The next step is to create the first UI page to display the list of existing links. Even though I followed <a href="https://docs.sst.dev/start/nextjs">SST’s docs</a> for creating a NextJS project, linking NextJS, SST, and MUI together was fraught.</p>
<h2 id="bring-nextjs-and-mui-online">Bring NextJS and MUI Online</h2>
<p>To get it all in sync, I ended up cloning the <a href="https://github.com/serverless-stack/sst/tree/master/examples/quickstart-nextjs">SST NextJS quickstart example</a> and copying over files until I was able to see Next’s <code class="language-plaintext highlighter-rouge">Hello World</code> UI:</p>
<p><img src="https://nickymarino.com/public/assets/2023/url-shortener/next-js-hello-world.png" alt="NextJS hello world screen" class="center" /></p>
<p>Then, I did the same with <a href="https://github.com/mui/material-ui/tree/master/examples/material-next-ts">MUI’s NextJS (Typescript) example</a> until I could <em>that</em> <code class="language-plaintext highlighter-rouge">Hello World</code> UI:</p>
<p><img src="https://nickymarino.com/public/assets/2023/url-shortener/mui-example-ui.png" alt="MUI hello world screen" class="center" /></p>
<p>You can view these changes in the <a href="https://github.com/nickymarino/url-shortener/pull/4"><strong>Next UI works in dev</strong></a> and <a href="https://github.com/nickymarino/url-shortener/pull/8"><strong>Add MUI</strong></a> PRs.</p>
<p>I’m eternally grateful to the devs that have written countless examples for the <a href="https://github.com/mui/material-ui/tree/master/examples">MUI</a> and <a href="https://github.com/serverless-stack/sst/tree/master/examples">SST</a> projects; without those examples, I don’t think I would’ve been able to move forward on this project with these tools.</p>
<h2 id="poll-the-api-for-links">Poll the API for Links</h2>
<p>Finally, let’s connect the UI to the API data! First, we’ll expose the <code class="language-plaintext highlighter-rouge">api</code> construct in <code class="language-plaintext highlighter-rouge">stacks/API.ts</code> so that we can <a href="https://docs.sst.dev/resource-binding">bind</a> it to the UI:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="c1">// ...</span>
<span class="nx">stack</span><span class="p">.</span><span class="nx">addOutputs</span><span class="p">({</span>
<span class="na">ApiEndpoint</span><span class="p">:</span> <span class="nx">api</span><span class="p">.</span><span class="nx">url</span><span class="p">,</span>
<span class="p">});</span>
<span class="k">return</span> <span class="p">{</span> <span class="nx">api</span> <span class="p">};</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Then bind the <code class="language-plaintext highlighter-rouge">api</code> to the <code class="language-plaintext highlighter-rouge">site</code> under <code class="language-plaintext highlighter-rouge">stacks/Site.ts</code>:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">export</span> <span class="kd">function</span> <span class="nx">Site</span><span class="p">({</span> <span class="nx">stack</span> <span class="p">}:</span> <span class="nx">StackContext</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">const</span> <span class="p">{</span> <span class="nx">api</span> <span class="p">}</span> <span class="o">=</span> <span class="nx">use</span><span class="p">(</span><span class="nx">API</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">site</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">NextjsSite</span><span class="p">(</span><span class="nx">stack</span><span class="p">,</span> <span class="dl">"</span><span class="s2">site</span><span class="dl">"</span><span class="p">,</span> <span class="p">{</span>
<span class="na">bind</span><span class="p">:</span> <span class="p">[</span><span class="nx">api</span><span class="p">],</span>
<span class="p">});</span>
</code></pre></div></div>
<p>On the index page (<code class="language-plaintext highlighter-rouge">src/pages/index.tsx</code>), we can create a <code class="language-plaintext highlighter-rouge">Link</code> type and use <code class="language-plaintext highlighter-rouge">getServerSideProps</code> to fetch data from the API:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">type</span> <span class="nx">Link</span> <span class="o">=</span> <span class="p">{</span>
<span class="na">shortPath</span><span class="p">:</span> <span class="kr">string</span><span class="p">;</span>
<span class="nl">url</span><span class="p">:</span> <span class="kr">string</span><span class="p">;</span>
<span class="nl">uid</span><span class="p">:</span> <span class="kr">string</span><span class="p">;</span>
<span class="p">};</span>
<span class="k">export</span> <span class="kd">const</span> <span class="nx">getServerSideProps</span><span class="p">:</span> <span class="nx">GetServerSideProps</span><span class="o"><</span><span class="p">{</span>
<span class="na">links</span><span class="p">:</span> <span class="nx">Link</span><span class="p">[];</span>
<span class="p">}</span><span class="o">></span> <span class="o">=</span> <span class="k">async</span> <span class="p">(</span><span class="nx">context</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">endpoint</span> <span class="o">=</span> <span class="s2">`</span><span class="p">${</span><span class="nx">Api</span><span class="p">.</span><span class="nx">api</span><span class="p">.</span><span class="nx">url</span><span class="p">}</span><span class="s2">/links`</span><span class="p">;</span>
<span class="kd">const</span> <span class="nx">res</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">fetch</span><span class="p">(</span><span class="nx">endpoint</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">data</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">res</span><span class="p">.</span><span class="nx">json</span><span class="p">();</span>
<span class="k">return</span> <span class="p">{</span> <span class="na">props</span><span class="p">:</span> <span class="nx">data</span><span class="p">.</span><span class="nx">body</span> <span class="p">};</span>
<span class="p">};</span>
</code></pre></div></div>
<p>There’s a possible breakdown between the UI and API here if the <code class="language-plaintext highlighter-rouge">Link</code> type skews from what the API returns, so I created an <a href="https://github.com/nickymarino/url-shortener/issues/2">issue</a> to revisit this in the future. In that same file, we’ll add some basic <code class="language-plaintext highlighter-rouge">p</code> tags for each link we receive:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">export</span> <span class="k">default</span> <span class="kd">function</span> <span class="nx">Home</span><span class="p">({</span> <span class="nx">links</span> <span class="p">}:</span> <span class="p">{</span> <span class="nl">links</span><span class="p">:</span> <span class="nx">Link</span><span class="p">[]</span> <span class="p">})</span> <span class="p">{</span>
<span class="k">return</span> <span class="p">(</span>
<span class="o"><</span><span class="nx">Container</span> <span class="nx">maxWidth</span><span class="o">=</span><span class="dl">"</span><span class="s2">lg</span><span class="dl">"</span><span class="o">></span>
<span class="p">{</span><span class="cm">/* ... */</span><span class="p">}</span>
<span class="p">{</span><span class="nx">links</span> <span class="o">&&</span>
<span class="nx">links</span><span class="p">.</span><span class="nx">map</span><span class="p">((</span><span class="nx">link</span><span class="p">)</span> <span class="o">=></span> <span class="p">(</span>
<span class="o"><</span><span class="nx">Typography</span>
<span class="nx">variant</span><span class="o">=</span><span class="dl">"</span><span class="s2">body1</span><span class="dl">"</span>
<span class="nx">component</span><span class="o">=</span><span class="dl">"</span><span class="s2">p</span><span class="dl">"</span>
<span class="nx">gutterBottom</span>
<span class="nx">key</span><span class="o">=</span><span class="p">{</span><span class="nx">link</span><span class="p">.</span><span class="nx">uid</span><span class="p">}</span>
<span class="o">></span>
<span class="p">{</span><span class="nx">link</span><span class="p">.</span><span class="nx">shortPath</span><span class="p">}</span> <span class="o">-</span> <span class="p">{</span><span class="nx">link</span><span class="p">.</span><span class="nx">url</span><span class="p">}</span>
<span class="o"><</span><span class="sr">/Typography</span><span class="err">>
</span> <span class="p">))}</span>
</code></pre></div></div>
<p>One last thing: NextJS requires an experimental flag in <code class="language-plaintext highlighter-rouge">next.config.js</code> to make this work. I think it’s to allow <code class="language-plaintext highlighter-rouge">async</code> calls inside <code class="language-plaintext highlighter-rouge">getServerProps</code>:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">nextConfig</span> <span class="o">=</span> <span class="p">{</span>
<span class="na">reactStrictMode</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="nx">webpack</span><span class="p">(</span><span class="nx">config</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">config</span><span class="p">.</span><span class="nx">experiments</span> <span class="o">=</span> <span class="p">{</span> <span class="p">...</span><span class="nx">config</span><span class="p">.</span><span class="nx">experiments</span><span class="p">,</span> <span class="na">topLevelAwait</span><span class="p">:</span> <span class="kc">true</span> <span class="p">};</span>
<span class="k">return</span> <span class="nx">config</span><span class="p">;</span>
<span class="p">},</span>
<span class="p">};</span>
</code></pre></div></div>
<p>And there we go, links! You can view this code on the <a href="https://github.com/nickymarino/url-shortener/pull/9"><strong>Index shows list of links</strong></a> PR.</p>
<p><img src="https://nickymarino.com/public/assets/2023/url-shortener/mui-example-ui-with-links.png" alt="NextJS hello world screen" class="center" /></p>
Dev Journal #4: API patch links and redirect
2023-06-09T00:00:00+00:00https://nickymarino.com/2023/06/09/url-shortener-04-patch-and-redirect<p>Today we’ll round out the first version of the API by:</p>
<ul>
<li>Adding create and update timestamps to <code class="language-plaintext highlighter-rouge">Link</code> entities</li>
<li>Un-hardcoding the link create function</li>
<li>Adding a way to edit links</li>
<li>Adding a way to redirect to a URL from a short path</li>
</ul>
<h2 id="add-timestamps-to-links">Add Timestamps to Links</h2>
<p>ElectroDB lets us add <code class="language-plaintext highlighter-rouge">createdAt</code> and <code class="language-plaintext highlighter-rouge">updatedAt</code> timestamps very easily, thanks to the ability to set <code class="language-plaintext highlighter-rouge">readOnly: true</code> and/or a <code class="language-plaintext highlighter-rouge">watch</code> property to modify attributes when the entity is already being edited. We’ll lift straight from <a href="https://electrodb.dev/en/recipes/created-at-updated-at/">their example</a> by editing the <code class="language-plaintext highlighter-rouge">LinkEntity</code> in <code class="language-plaintext highlighter-rouge">packages/core/src/link.ts</code>:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">export</span> <span class="kd">const</span> <span class="nx">LinkEntity</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">Entity</span><span class="p">(</span>
<span class="p">{</span>
<span class="na">model</span><span class="p">:</span> <span class="p">{</span>
<span class="na">entity</span><span class="p">:</span> <span class="dl">"</span><span class="s2">links</span><span class="dl">"</span><span class="p">,</span>
<span class="na">version</span><span class="p">:</span> <span class="dl">"</span><span class="s2">1</span><span class="dl">"</span><span class="p">,</span>
<span class="na">service</span><span class="p">:</span> <span class="dl">"</span><span class="s2">cowlinks</span><span class="dl">"</span><span class="p">,</span>
<span class="p">},</span>
<span class="na">attributes</span><span class="p">:</span> <span class="p">{</span>
<span class="c1">// ...</span>
<span class="na">createdAt</span><span class="p">:</span> <span class="p">{</span>
<span class="na">type</span><span class="p">:</span> <span class="dl">"</span><span class="s2">number</span><span class="dl">"</span><span class="p">,</span>
<span class="na">readOnly</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="na">set</span><span class="p">:</span> <span class="p">()</span> <span class="o">=></span> <span class="nb">Date</span><span class="p">.</span><span class="nx">now</span><span class="p">(),</span>
<span class="p">},</span>
<span class="na">updatedAt</span><span class="p">:</span> <span class="p">{</span>
<span class="na">type</span><span class="p">:</span> <span class="dl">"</span><span class="s2">number</span><span class="dl">"</span><span class="p">,</span>
<span class="na">readOnly</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="na">watch</span><span class="p">:</span> <span class="dl">"</span><span class="s2">*</span><span class="dl">"</span><span class="p">,</span>
<span class="na">set</span><span class="p">:</span> <span class="p">()</span> <span class="o">=></span> <span class="nb">Date</span><span class="p">.</span><span class="nx">now</span><span class="p">(),</span>
<span class="p">},</span>
</code></pre></div></div>
<p>Now, ElectroDB will automatically set <code class="language-plaintext highlighter-rouge">createdAt</code> whenever a new entity is created (and only the model can set this, not the user), and it will update <code class="language-plaintext highlighter-rouge">updatedAt</code> whenever any other property on the entity changes as well.</p>
<h2 id="un-hardcode-the-link-create-endpoint">Un-hardcode the link create endpoint</h2>
<p>Earlier, we hardcoded each link that was created to redirect to Google. Let’s update <code class="language-plaintext highlighter-rouge">packages/functions/src/link/create.ts</code> to use the <code class="language-plaintext highlighter-rouge">url</code> and <code class="language-plaintext highlighter-rouge">shortPath</code> parameters from the POST body:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">export</span> <span class="kd">const</span> <span class="nx">handler</span> <span class="o">=</span> <span class="nx">ApiHandler</span><span class="p">(</span><span class="k">async</span> <span class="p">(</span><span class="nx">_evt</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="kd">const</span> <span class="p">{</span> <span class="nx">url</span><span class="p">,</span> <span class="nx">shortPath</span> <span class="p">}</span> <span class="o">=</span> <span class="nx">useJsonBody</span><span class="p">();</span>
<span class="kd">const</span> <span class="nx">newLink</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">Link</span><span class="p">.</span><span class="nx">create</span><span class="p">(</span><span class="nx">shortPath</span><span class="p">,</span> <span class="nx">url</span><span class="p">);</span>
<span class="k">return</span> <span class="p">{</span>
<span class="na">body</span><span class="p">:</span> <span class="nx">JSON</span><span class="p">.</span><span class="nx">stringify</span><span class="p">({</span>
<span class="na">link</span><span class="p">:</span> <span class="nx">newLink</span><span class="p">,</span>
<span class="p">},</span>
<span class="p">}),</span>
<span class="p">};</span>
<span class="p">});</span>
</code></pre></div></div>
<p>Here, <code class="language-plaintext highlighter-rouge">useBody</code> is an SST builtin from the <a href="https://docs.sst.dev/clients/api#usejsonbody">api client</a>.</p>
<h2 id="add-link-patch-and-list">Add link PATCH and list</h2>
<p>Originally, I was going to use the verb PUT instead of PATCH to edit links; however, <a href="https://electrodb.dev/en/mutations/put/">ElectroDB’s <code class="language-plaintext highlighter-rouge">put</code></a> makes you provide all required attributes, and I prefer to only pass in the properties I wish to change. (Or at least, I like having the option to only provide the changed values.)</p>
<p>So I went with <a href="https://electrodb.dev/en/mutations/patch/"><code class="language-plaintext highlighter-rouge">patch</code></a> instead, especially since it ensures you’re not creating a new link accidentally:</p>
<blockquote>
<p>In DynamoDB, <code class="language-plaintext highlighter-rouge">update</code> operations by default will create an item if record being updated does not exist. Alternatively, the <code class="language-plaintext highlighter-rouge">patch</code> method will utilize the <code class="language-plaintext highlighter-rouge">attribute_exists()</code> parameter dynamically to ensure records are only “patched” and not created when updating items in your table.</p>
</blockquote>
<p>Adding the patch and list API endpoints is pretty straightforward. We’ll update <code class="language-plaintext highlighter-rouge">stacks/API.ts</code> to add two new functions:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="kd">const</span> <span class="nx">api</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">Api</span><span class="p">(</span><span class="nx">stack</span><span class="p">,</span> <span class="dl">"</span><span class="s2">api</span><span class="dl">"</span><span class="p">,</span> <span class="p">{</span>
<span class="c1">// ...</span>
<span class="na">routes</span><span class="p">:</span> <span class="p">{</span>
<span class="c1">// ...</span>
<span class="dl">"</span><span class="s2">PATCH /link/{uid}</span><span class="dl">"</span><span class="p">:</span> <span class="p">{</span>
<span class="na">function</span><span class="p">:</span> <span class="p">{</span>
<span class="na">functionName</span><span class="p">:</span> <span class="nx">nameFor</span><span class="p">(</span><span class="dl">"</span><span class="s2">LinkPatch</span><span class="dl">"</span><span class="p">),</span>
<span class="na">handler</span><span class="p">:</span> <span class="dl">"</span><span class="s2">packages/functions/src/link/patch.handler</span><span class="dl">"</span><span class="p">,</span>
<span class="p">},</span>
<span class="p">},</span>
<span class="dl">"</span><span class="s2">GET /links</span><span class="dl">"</span><span class="p">:</span> <span class="p">{</span>
<span class="na">function</span><span class="p">:</span> <span class="p">{</span>
<span class="na">functionName</span><span class="p">:</span> <span class="nx">nameFor</span><span class="p">(</span><span class="dl">"</span><span class="s2">LinkList</span><span class="dl">"</span><span class="p">),</span>
<span class="na">handler</span><span class="p">:</span> <span class="dl">"</span><span class="s2">packages/functions/src/link/list.handler</span><span class="dl">"</span><span class="p">,</span>
<span class="p">},</span>
<span class="p">},</span>
</code></pre></div></div>
<p>Expose methods in <code class="language-plaintext highlighter-rouge">packages/core/src/link.ts</code> to patch and list <code class="language-plaintext highlighter-rouge">LinkEntity</code>s:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">export</span> <span class="k">async</span> <span class="kd">function</span> <span class="nx">getByShortPath</span><span class="p">(</span><span class="nx">shortPath</span><span class="p">:</span> <span class="kr">string</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">result</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">LinkEntity</span><span class="p">.</span><span class="nx">query</span><span class="p">.</span><span class="nx">byShortPath</span><span class="p">({</span> <span class="nx">shortPath</span> <span class="p">}).</span><span class="nx">go</span><span class="p">();</span>
<span class="k">return</span> <span class="nx">result</span><span class="p">.</span><span class="nx">data</span><span class="p">;</span>
<span class="p">}</span>
<span class="kd">type</span> <span class="nx">PatchAttributes</span> <span class="o">=</span> <span class="nx">UpdateEntityItem</span><span class="o"><</span><span class="k">typeof</span> <span class="nx">LinkEntity</span><span class="o">></span><span class="p">;</span>
<span class="k">export</span> <span class="k">async</span> <span class="kd">function</span> <span class="nx">patch</span><span class="p">(</span><span class="nx">uid</span><span class="p">:</span> <span class="kr">string</span><span class="p">,</span> <span class="nx">newAttributes</span><span class="p">:</span> <span class="nx">PatchAttributes</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">result</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">LinkEntity</span><span class="p">.</span><span class="nx">patch</span><span class="p">({</span> <span class="nx">uid</span> <span class="p">}).</span><span class="kd">set</span><span class="p">(</span><span class="nx">newAttributes</span><span class="p">).</span><span class="nx">go</span><span class="p">({</span>
<span class="na">response</span><span class="p">:</span> <span class="dl">"</span><span class="s2">all_new</span><span class="dl">"</span><span class="p">,</span>
<span class="p">});</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">({</span> <span class="nx">result</span> <span class="p">});</span>
<span class="k">return</span> <span class="nx">result</span><span class="p">.</span><span class="nx">data</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">export</span> <span class="k">async</span> <span class="kd">function</span> <span class="nx">list</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">result</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">LinkEntity</span><span class="p">.</span><span class="nx">query</span><span class="p">.</span><span class="nx">byUid</span><span class="p">({}).</span><span class="nx">go</span><span class="p">();</span>
<span class="k">return</span> <span class="nx">result</span><span class="p">.</span><span class="nx">data</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>We use <code class="language-plaintext highlighter-rouge">response: "all_new"</code> so that Dynamo returns the whole updated object in the response instead of an empty object. In a heavy-scale production system, I’d more strongly consider using the default response to use eventual consistency over strong consistency. However, I expect to save a lot more debugging pain by grabbing the entire object in the short term.</p>
<p>I found the <code class="language-plaintext highlighter-rouge">UpdateEntityItem</code> party trick from <a href="https://electrodb.dev/en/reference/typescript">ElectroDB’s Typescript guide</a>. It has a lot of nice helpers for the static types, although I had to hunt around for a while to find it, and it was unclear which type I should use for <code class="language-plaintext highlighter-rouge">.patch()</code> without some experimentation.</p>
<p>Write the patch handler at <code class="language-plaintext highlighter-rouge">packages/functions/src/link/patch.ts</code>:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">export</span> <span class="kd">const</span> <span class="nx">handler</span> <span class="o">=</span> <span class="nx">ApiHandler</span><span class="p">(</span><span class="k">async</span> <span class="p">(</span><span class="nx">_evt</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">linkUid</span> <span class="o">=</span> <span class="nx">usePathParam</span><span class="p">(</span><span class="dl">"</span><span class="s2">uid</span><span class="dl">"</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">newAttributes</span> <span class="o">=</span> <span class="nx">useJsonBody</span><span class="p">();</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nx">linkUid</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="p">{</span>
<span class="na">statusCode</span><span class="p">:</span> <span class="mi">400</span><span class="p">,</span>
<span class="na">body</span><span class="p">:</span> <span class="dl">"</span><span class="s2">link uid is required</span><span class="dl">"</span><span class="p">,</span>
<span class="p">};</span>
<span class="p">}</span>
<span class="kd">const</span> <span class="nx">foundLink</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">Link</span><span class="p">.</span><span class="kd">get</span><span class="p">(</span><span class="nx">linkUid</span><span class="p">);</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">({</span> <span class="nx">foundLink</span> <span class="p">});</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nx">foundLink</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="p">{</span>
<span class="na">statusCode</span><span class="p">:</span> <span class="mi">404</span><span class="p">,</span>
<span class="na">body</span><span class="p">:</span> <span class="dl">"</span><span class="s2">not found</span><span class="dl">"</span><span class="p">,</span>
<span class="p">};</span>
<span class="p">}</span>
<span class="kd">const</span> <span class="nx">updatedLink</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">Link</span><span class="p">.</span><span class="nx">patch</span><span class="p">(</span><span class="nx">foundLink</span><span class="p">.</span><span class="nx">uid</span><span class="p">,</span> <span class="nx">newAttributes</span><span class="p">);</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">({</span> <span class="nx">updatedLink</span> <span class="p">});</span>
<span class="k">return</span> <span class="p">{</span>
<span class="na">statusCode</span><span class="p">:</span> <span class="mi">200</span><span class="p">,</span>
<span class="na">body</span><span class="p">:</span> <span class="nx">JSON</span><span class="p">.</span><span class="nx">stringify</span><span class="p">({</span>
<span class="na">link</span><span class="p">:</span> <span class="nx">updatedLink</span><span class="p">,</span>
<span class="p">}),</span>
<span class="p">};</span>
<span class="p">});</span>
</code></pre></div></div>
<p>And the list handler at <code class="language-plaintext highlighter-rouge">packages/functions/src/link/list.ts</code>:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">export</span> <span class="kd">const</span> <span class="nx">handler</span> <span class="o">=</span> <span class="nx">ApiHandler</span><span class="p">(</span><span class="k">async</span> <span class="p">(</span><span class="nx">_evt</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">result</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">Link</span><span class="p">.</span><span class="nx">list</span><span class="p">();</span>
<span class="k">return</span> <span class="p">{</span>
<span class="na">body</span><span class="p">:</span> <span class="p">{</span>
<span class="na">links</span><span class="p">:</span> <span class="nx">result</span><span class="p">,</span>
<span class="p">},</span>
<span class="p">};</span>
<span class="p">});</span>
</code></pre></div></div>
<h2 id="the-redirect-endpoint">The Redirect Endpoint</h2>
<p>Finally, the last and most important piece of the URL shortener: the redirect. I decided to use <code class="language-plaintext highlighter-rouge">/s</code> to designate shortened URLs (<code class="language-plaintext highlighter-rouge">s</code> for shorten), so <code class="language-plaintext highlighter-rouge">API_URL/s/foo</code> will look up a <code class="language-plaintext highlighter-rouge">LinkEntity</code> with a <code class="language-plaintext highlighter-rouge">shortPath</code> of <code class="language-plaintext highlighter-rouge">foo</code> and redirect to the entity’s <code class="language-plaintext highlighter-rouge">url</code>.</p>
<p>Let’s fix a bug in the <code class="language-plaintext highlighter-rouge">LinkEntity</code> under <code class="language-plaintext highlighter-rouge">packages/core/src/link.ts</code> so that we can find links by their <code class="language-plaintext highlighter-rouge">shortPath</code>:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">export</span> <span class="kd">const</span> <span class="nx">LinkEntity</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">Entity</span><span class="p">(</span>
<span class="p">{</span>
<span class="c1">// ...</span>
<span class="na">indexes</span><span class="p">:</span> <span class="p">{</span>
<span class="c1">// ...</span>
<span class="na">byShortPath</span><span class="p">:</span> <span class="p">{</span>
<span class="na">index</span><span class="p">:</span> <span class="dl">"</span><span class="s2">gsi1</span><span class="dl">"</span><span class="p">,</span>
</code></pre></div></div>
<p>Then we’ll add a new route in <code class="language-plaintext highlighter-rouge">stacks/API.ts</code>:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="dl">"</span><span class="s2">GET /s/{shortPath}</span><span class="dl">"</span><span class="p">:</span> <span class="p">{</span>
<span class="nl">function</span><span class="p">:</span> <span class="p">{</span>
<span class="na">functionName</span><span class="p">:</span> <span class="nx">nameFor</span><span class="p">(</span><span class="dl">"</span><span class="s2">Redirect</span><span class="dl">"</span><span class="p">),</span>
<span class="na">handler</span><span class="p">:</span> <span class="dl">"</span><span class="s2">packages/functions/src/redirect.handler</span><span class="dl">"</span><span class="p">,</span>
<span class="p">},</span>
<span class="p">},</span>
</code></pre></div></div>
<p>And the corresponding handler in <code class="language-plaintext highlighter-rouge">packages/functions/src/redirect.ts</code>:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">export</span> <span class="kd">const</span> <span class="nx">handler</span> <span class="o">=</span> <span class="nx">ApiHandler</span><span class="p">(</span><span class="k">async</span> <span class="p">(</span><span class="nx">_evt</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">shortPath</span> <span class="o">=</span> <span class="nx">usePathParam</span><span class="p">(</span><span class="dl">"</span><span class="s2">shortPath</span><span class="dl">"</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nx">shortPath</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="p">{</span>
<span class="na">statusCode</span><span class="p">:</span> <span class="mi">400</span><span class="p">,</span>
<span class="na">body</span><span class="p">:</span> <span class="dl">"</span><span class="s2">shortPath is required</span><span class="dl">"</span><span class="p">,</span>
<span class="p">};</span>
<span class="p">}</span>
<span class="kd">const</span> <span class="nx">foundLinks</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">Link</span><span class="p">.</span><span class="nx">getByShortPath</span><span class="p">(</span><span class="nx">shortPath</span><span class="p">);</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">({</span> <span class="nx">foundLinks</span> <span class="p">});</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">foundLinks</span><span class="p">.</span><span class="nx">length</span> <span class="o">===</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="p">{</span>
<span class="na">statusCode</span><span class="p">:</span> <span class="mi">404</span><span class="p">,</span>
<span class="na">body</span><span class="p">:</span> <span class="dl">"</span><span class="s2">not found</span><span class="dl">"</span><span class="p">,</span>
<span class="p">};</span>
<span class="p">}</span>
<span class="k">return</span> <span class="p">{</span>
<span class="na">statusCode</span><span class="p">:</span> <span class="mi">301</span><span class="p">,</span>
<span class="na">headers</span><span class="p">:</span> <span class="p">{</span>
<span class="na">Location</span><span class="p">:</span> <span class="nx">foundLinks</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="nx">url</span><span class="p">,</span>
<span class="p">},</span>
<span class="p">};</span>
<span class="p">});</span>
</code></pre></div></div>
<p>And there we go! A finished v0 API. You can view this code in this <a href="https://github.com/nickymarino/url-shortener/compare/e8a5faf2acf35a8528e425d447edf4ef7c118358..5df2c4e821414b077ee4ba8348cc3f42deeb4b19">git compare</a>.</p>
Dev Journal #3: API list links
2023-06-08T00:00:00+00:00https://nickymarino.com/2023/06/08/url-shortener-03-list-links<p>Let’s add an endpoint to get a list of all links that we’ve created. Start with adding a <code class="language-plaintext highlighter-rouge">list</code> function to <code class="language-plaintext highlighter-rouge">packages/core/src/link.ts</code>:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">export</span> <span class="k">async</span> <span class="kd">function</span> <span class="nx">list</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">result</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">LinkEntity</span><span class="p">.</span><span class="nx">query</span><span class="p">.</span><span class="nx">byUid</span><span class="p">({}).</span><span class="nx">go</span><span class="p">();</span>
<span class="k">return</span> <span class="nx">result</span><span class="p">.</span><span class="nx">data</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Add the <code class="language-plaintext highlighter-rouge">LinkList</code> lambda function to the <code class="language-plaintext highlighter-rouge">API</code> stack in <code class="language-plaintext highlighter-rouge">stacks/API.ts</code> as well:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="kd">const</span> <span class="nx">api</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">Api</span><span class="p">(</span><span class="nx">stack</span><span class="p">,</span> <span class="dl">"</span><span class="s2">api</span><span class="dl">"</span><span class="p">,</span> <span class="p">{</span>
<span class="na">routes</span><span class="p">:</span> <span class="p">{</span>
<span class="dl">"</span><span class="s2">GET /links</span><span class="dl">"</span><span class="p">:</span> <span class="p">{</span>
<span class="na">function</span><span class="p">:</span> <span class="p">{</span>
<span class="na">functionName</span><span class="p">:</span> <span class="nx">nameFor</span><span class="p">(</span><span class="dl">"</span><span class="s2">LinkList</span><span class="dl">"</span><span class="p">),</span>
<span class="na">handler</span><span class="p">:</span> <span class="dl">"</span><span class="s2">packages/functions/src/link/list.handler</span><span class="dl">"</span><span class="p">,</span>
<span class="p">},</span>
<span class="c1">// ...</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>And there we go! You can view this code in the <a href="https://github.com/nickymarino/url-shortener/commit/e8a5faf2acf35a8528e425d447edf4ef7c118358"><strong>Add link list</strong></a> commit.</p>
Dev Journal #2: API get links
2023-06-07T00:00:00+00:00https://nickymarino.com/2023/06/07/url-shortener-02-get-links<p>Let’s add an endpoint to get links that we’ve created. Start with adding a <code class="language-plaintext highlighter-rouge">get</code> function to <code class="language-plaintext highlighter-rouge">packages/core/src/link.ts</code>:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">export</span> <span class="k">async</span> <span class="kd">function</span> <span class="kd">get</span><span class="p">(</span><span class="nx">uid</span><span class="p">:</span> <span class="kr">string</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">result</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">LinkEntity</span><span class="p">.</span><span class="kd">get</span><span class="p">({</span> <span class="nx">uid</span> <span class="p">}).</span><span class="nx">go</span><span class="p">();</span>
<span class="k">return</span> <span class="nx">result</span><span class="p">.</span><span class="nx">data</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Add the <code class="language-plaintext highlighter-rouge">LinkGet</code> lambda function to the <code class="language-plaintext highlighter-rouge">API</code> stack in <code class="language-plaintext highlighter-rouge">stacks/API.ts</code> as well:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="kd">const</span> <span class="nx">api</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">Api</span><span class="p">(</span><span class="nx">stack</span><span class="p">,</span> <span class="dl">"</span><span class="s2">api</span><span class="dl">"</span><span class="p">,</span> <span class="p">{</span>
<span class="na">routes</span><span class="p">:</span> <span class="p">{</span>
<span class="dl">"</span><span class="s2">GET /link/{id}</span><span class="dl">"</span><span class="p">:</span> <span class="p">{</span>
<span class="na">function</span><span class="p">:</span> <span class="p">{</span>
<span class="na">functionName</span><span class="p">:</span> <span class="s2">`</span><span class="p">${</span><span class="nx">stack</span><span class="p">.</span><span class="nx">stackName</span><span class="p">}</span><span class="s2">-LinkGet`</span><span class="p">,</span>
<span class="na">handler</span><span class="p">:</span> <span class="dl">"</span><span class="s2">packages/functions/src/link/get.handler</span><span class="dl">"</span><span class="p">,</span>
<span class="na">bind</span><span class="p">:</span> <span class="p">[</span><span class="nx">table</span><span class="p">],</span>
<span class="p">},</span>
<span class="p">},</span>
<span class="c1">// ...</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>You can view this code in the <a href="https://github.com/nickymarino/url-shortener/commit/e80548eb53241b31d7403277b6a5a2558cc9e3c0"><strong>Add link get</strong></a> commit.</p>
Dev Journal #1: API create links
2023-06-06T00:00:00+00:00https://nickymarino.com/2023/06/06/url-shortener-01-create-links<p>Let’s set up some backend API functions, but first we need a project we can code against. <a href="https://sst.dev/examples/how-to-create-a-nextjs-app-with-serverless.html">SST’s docs</a> recommend starting with a clean NextJS app then add sst on top by running these scripts in the root folder:</p>
<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>npx create-next-app@latest
npx create-sst@latest
npm install
</code></pre></div></div>
<p>We’ll use SST to create a new DynamoDb table in <code class="language-plaintext highlighter-rouge">stacks/Database.ts</code>. Because we’re going to have two primary access patterns on the table immediately–GET by URL and GET by unique id–we’ll add one global index to handle the second access pattern:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">export</span> <span class="kd">function</span> <span class="nx">Database</span><span class="p">({</span> <span class="nx">stack</span> <span class="p">}:</span> <span class="nx">StackContext</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">table</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">Table</span><span class="p">(</span><span class="nx">stack</span><span class="p">,</span> <span class="dl">"</span><span class="s2">table</span><span class="dl">"</span><span class="p">,</span> <span class="p">{</span>
<span class="na">fields</span><span class="p">:</span> <span class="p">{</span>
<span class="na">pk</span><span class="p">:</span> <span class="dl">"</span><span class="s2">string</span><span class="dl">"</span><span class="p">,</span>
<span class="na">sk</span><span class="p">:</span> <span class="dl">"</span><span class="s2">string</span><span class="dl">"</span><span class="p">,</span>
<span class="na">gsi1pk</span><span class="p">:</span> <span class="dl">"</span><span class="s2">string</span><span class="dl">"</span><span class="p">,</span>
<span class="na">gsi1sk</span><span class="p">:</span> <span class="dl">"</span><span class="s2">string</span><span class="dl">"</span><span class="p">,</span>
<span class="p">},</span>
<span class="na">primaryIndex</span><span class="p">:</span> <span class="p">{</span>
<span class="na">partitionKey</span><span class="p">:</span> <span class="dl">"</span><span class="s2">pk</span><span class="dl">"</span><span class="p">,</span>
<span class="na">sortKey</span><span class="p">:</span> <span class="dl">"</span><span class="s2">sk</span><span class="dl">"</span><span class="p">,</span>
<span class="p">},</span>
<span class="na">globalIndexes</span><span class="p">:</span> <span class="p">{</span>
<span class="na">gsi1</span><span class="p">:</span> <span class="p">{</span>
<span class="na">partitionKey</span><span class="p">:</span> <span class="dl">"</span><span class="s2">gsi1pk</span><span class="dl">"</span><span class="p">,</span>
<span class="na">sortKey</span><span class="p">:</span> <span class="dl">"</span><span class="s2">gsi1sk</span><span class="dl">"</span><span class="p">,</span>
<span class="p">},</span>
<span class="p">},</span>
<span class="p">});</span>
<span class="k">return</span> <span class="p">{</span> <span class="nx">table</span> <span class="p">};</span>
<span class="p">}</span>
</code></pre></div></div>
<p>And we’ll similarly create an API in <code class="language-plaintext highlighter-rouge">stacks/API.ts</code> to create a link as <code class="language-plaintext highlighter-rouge">POST /link</code> backed by a lambda function:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">function</span> <span class="nx">nameFor</span><span class="p">(</span><span class="nx">shortName</span><span class="p">:</span> <span class="kr">string</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">nameGenerator</span> <span class="o">=</span> <span class="p">(</span><span class="nx">props</span><span class="p">:</span> <span class="nx">FunctionNameProps</span><span class="p">):</span> <span class="kr">string</span> <span class="o">=></span> <span class="p">{</span>
<span class="k">return</span> <span class="s2">`</span><span class="p">${</span><span class="nx">props</span><span class="p">.</span><span class="nx">stack</span><span class="p">.</span><span class="nx">stackName</span><span class="p">}</span><span class="s2">-</span><span class="p">${</span><span class="nx">shortName</span><span class="p">}</span><span class="s2">`</span><span class="p">;</span>
<span class="p">};</span>
<span class="k">return</span> <span class="nx">nameGenerator</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">export</span> <span class="kd">function</span> <span class="nx">API</span><span class="p">({</span> <span class="nx">stack</span> <span class="p">}:</span> <span class="nx">StackContext</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">const</span> <span class="p">{</span> <span class="nx">table</span> <span class="p">}</span> <span class="o">=</span> <span class="nx">use</span><span class="p">(</span><span class="nx">Database</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">api</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">Api</span><span class="p">(</span><span class="nx">stack</span><span class="p">,</span> <span class="dl">"</span><span class="s2">api</span><span class="dl">"</span><span class="p">,</span> <span class="p">{</span>
<span class="na">defaults</span><span class="p">:</span> <span class="p">{</span>
<span class="na">function</span><span class="p">:</span> <span class="p">{</span>
<span class="na">bind</span><span class="p">:</span> <span class="p">[</span><span class="nx">table</span><span class="p">],</span>
<span class="p">},</span>
<span class="p">},</span>
<span class="na">routes</span><span class="p">:</span> <span class="p">{</span>
<span class="dl">"</span><span class="s2">POST /link</span><span class="dl">"</span><span class="p">:</span> <span class="p">{</span>
<span class="na">function</span><span class="p">:</span> <span class="p">{</span>
<span class="na">functionName</span><span class="p">:</span> <span class="nx">nameFor</span><span class="p">(</span><span class="dl">"</span><span class="s2">LinkCreate</span><span class="dl">"</span><span class="p">),</span>
<span class="na">handler</span><span class="p">:</span> <span class="dl">"</span><span class="s2">packages/functions/src/link/create.handler</span><span class="dl">"</span><span class="p">,</span>
<span class="p">},</span>
<span class="p">},</span>
<span class="p">},</span>
<span class="p">});</span>
<span class="nx">stack</span><span class="p">.</span><span class="nx">addOutputs</span><span class="p">({</span>
<span class="na">ApiEndpoint</span><span class="p">:</span> <span class="nx">api</span><span class="p">.</span><span class="nx">url</span><span class="p">,</span>
<span class="p">});</span>
<span class="k">return</span> <span class="p">{</span> <span class="nx">api</span> <span class="p">};</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Here, <code class="language-plaintext highlighter-rouge">nameFor</code> is my shorthand to create nicer function names. By default, SST will let CloudFormation auto-generate the function names, and they’re pretty unreadable. <code class="language-plaintext highlighter-rouge">nameFor</code> will pass a name generator into SST as it creates functions so that the function names are human readable.</p>
<p>Let’s also update <code class="language-plaintext highlighter-rouge">sst.config.ts</code> with the <code class="language-plaintext highlighter-rouge">Database</code> and <code class="language-plaintext highlighter-rouge">API</code> stacks:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">export</span> <span class="k">default</span> <span class="p">{</span>
<span class="nx">config</span><span class="p">(</span><span class="nx">_input</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="p">{</span>
<span class="na">name</span><span class="p">:</span> <span class="dl">"</span><span class="s2">cow-link</span><span class="dl">"</span><span class="p">,</span>
<span class="na">region</span><span class="p">:</span> <span class="dl">"</span><span class="s2">us-east-1</span><span class="dl">"</span><span class="p">,</span>
<span class="p">};</span>
<span class="p">},</span>
<span class="nx">stacks</span><span class="p">(</span><span class="nx">app</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">app</span><span class="p">.</span><span class="nx">stack</span><span class="p">(</span><span class="nx">Database</span><span class="p">);</span>
<span class="nx">app</span><span class="p">.</span><span class="nx">stack</span><span class="p">(</span><span class="nx">API</span><span class="p">);</span>
<span class="nx">app</span><span class="p">.</span><span class="nx">stack</span><span class="p">(</span><span class="nx">Site</span><span class="p">);</span>
<span class="p">},</span>
<span class="p">}</span> <span class="nx">satisfies</span> <span class="nx">SSTConfig</span><span class="p">;</span>
</code></pre></div></div>
<p>We’re following <a href="https://docs.sst.dev/learn/write-to-dynamodb">recommendations in SST’s documentation</a> to use <a href="https://github.com/tywalch/electrodb">ElectroDB</a> as our DynamoDB interface, so we add <code class="language-plaintext highlighter-rouge">ulid</code>, <code class="language-plaintext highlighter-rouge">electrodb</code>, and the AWS DynamoDB client to our dependencies in <code class="language-plaintext highlighter-rouge">package.json</code>:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="w"> </span><span class="nl">"dependencies"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"ulid"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^2.3.0"</span><span class="p">,</span><span class="w">
</span><span class="nl">"electrodb"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^2.5.1"</span><span class="p">,</span><span class="w">
</span><span class="nl">"@aws-sdk/client-dynamodb"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^3.332.0"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>We’ll need to configure ElectroDB with the details about the DynamoDB table, so let’s add that in <code class="language-plaintext highlighter-rouge">packages/core/src/dynamo.ts</code>:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">export</span> <span class="kd">const</span> <span class="nx">Client</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">DynamoDBClient</span><span class="p">({});</span>
<span class="k">export</span> <span class="kd">const</span> <span class="nx">Configuration</span><span class="p">:</span> <span class="nx">EntityConfiguration</span> <span class="o">=</span> <span class="p">{</span>
<span class="na">table</span><span class="p">:</span> <span class="nx">Table</span><span class="p">.</span><span class="nx">table</span><span class="p">.</span><span class="nx">tableName</span><span class="p">,</span>
<span class="na">client</span><span class="p">:</span> <span class="nx">Client</span><span class="p">,</span>
<span class="p">};</span>
</code></pre></div></div>
<p>And we’ll need a <code class="language-plaintext highlighter-rouge">LinkEntity</code> with a corresponding <code class="language-plaintext highlighter-rouge">create</code> function in <code class="language-plaintext highlighter-rouge">packages/core/src/link.ts</code>:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">export</span> <span class="kd">const</span> <span class="nx">LinkEntity</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">Entity</span><span class="p">(</span>
<span class="p">{</span>
<span class="na">model</span><span class="p">:</span> <span class="p">{</span>
<span class="na">entity</span><span class="p">:</span> <span class="dl">"</span><span class="s2">links</span><span class="dl">"</span><span class="p">,</span>
<span class="na">version</span><span class="p">:</span> <span class="dl">"</span><span class="s2">1</span><span class="dl">"</span><span class="p">,</span>
<span class="na">service</span><span class="p">:</span> <span class="dl">"</span><span class="s2">cowlinks</span><span class="dl">"</span><span class="p">,</span>
<span class="p">},</span>
<span class="na">attributes</span><span class="p">:</span> <span class="p">{</span>
<span class="na">uid</span><span class="p">:</span> <span class="p">{</span>
<span class="na">type</span><span class="p">:</span> <span class="dl">"</span><span class="s2">string</span><span class="dl">"</span><span class="p">,</span>
<span class="na">required</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="p">},</span>
<span class="na">shortPath</span><span class="p">:</span> <span class="p">{</span>
<span class="na">type</span><span class="p">:</span> <span class="dl">"</span><span class="s2">string</span><span class="dl">"</span><span class="p">,</span>
<span class="na">required</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="p">},</span>
<span class="na">url</span><span class="p">:</span> <span class="p">{</span>
<span class="na">type</span><span class="p">:</span> <span class="dl">"</span><span class="s2">string</span><span class="dl">"</span><span class="p">,</span>
<span class="na">required</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="p">},</span>
<span class="p">},</span>
<span class="na">indexes</span><span class="p">:</span> <span class="p">{</span>
<span class="na">byUid</span><span class="p">:</span> <span class="p">{</span>
<span class="na">pk</span><span class="p">:</span> <span class="p">{</span>
<span class="na">field</span><span class="p">:</span> <span class="dl">"</span><span class="s2">pk</span><span class="dl">"</span><span class="p">,</span>
<span class="na">composite</span><span class="p">:</span> <span class="p">[],</span>
<span class="p">},</span>
<span class="na">sk</span><span class="p">:</span> <span class="p">{</span>
<span class="na">field</span><span class="p">:</span> <span class="dl">"</span><span class="s2">sk</span><span class="dl">"</span><span class="p">,</span>
<span class="na">composite</span><span class="p">:</span> <span class="p">[</span><span class="dl">"</span><span class="s2">uid</span><span class="dl">"</span><span class="p">],</span>
<span class="p">},</span>
<span class="p">},</span>
<span class="na">byShortPath</span><span class="p">:</span> <span class="p">{</span>
<span class="na">index</span><span class="p">:</span> <span class="dl">"</span><span class="s2">gsi1pk-gsi1sk-index</span><span class="dl">"</span><span class="p">,</span>
<span class="na">pk</span><span class="p">:</span> <span class="p">{</span>
<span class="na">field</span><span class="p">:</span> <span class="dl">"</span><span class="s2">gsi1pk</span><span class="dl">"</span><span class="p">,</span>
<span class="na">composite</span><span class="p">:</span> <span class="p">[</span><span class="dl">"</span><span class="s2">shortPath</span><span class="dl">"</span><span class="p">],</span>
<span class="p">},</span>
<span class="na">sk</span><span class="p">:</span> <span class="p">{</span>
<span class="na">field</span><span class="p">:</span> <span class="dl">"</span><span class="s2">gsi1sk</span><span class="dl">"</span><span class="p">,</span>
<span class="na">composite</span><span class="p">:</span> <span class="p">[],</span>
<span class="p">},</span>
<span class="p">},</span>
<span class="p">},</span>
<span class="p">},</span>
<span class="nx">Dynamo</span><span class="p">.</span><span class="nx">Configuration</span>
<span class="p">);</span>
<span class="k">export</span> <span class="k">async</span> <span class="kd">function</span> <span class="nx">create</span><span class="p">(</span><span class="nx">shortPath</span><span class="p">:</span> <span class="kr">string</span><span class="p">,</span> <span class="nx">url</span><span class="p">:</span> <span class="kr">string</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">result</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">LinkEntity</span><span class="p">.</span><span class="nx">create</span><span class="p">({</span>
<span class="na">uid</span><span class="p">:</span> <span class="nx">ulid</span><span class="p">(),</span>
<span class="nx">shortPath</span><span class="p">,</span>
<span class="nx">url</span><span class="p">,</span>
<span class="p">}).</span><span class="nx">go</span><span class="p">();</span>
<span class="k">return</span> <span class="nx">result</span><span class="p">.</span><span class="nx">data</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Last but certainly not least, we’ll write the code backing the lambda function to actually create links. To start, let’s create a hardcoded link to test under <code class="language-plaintext highlighter-rouge">packages/functions/link/create.ts</code>:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">export</span> <span class="kd">const</span> <span class="nx">handler</span> <span class="o">=</span> <span class="nx">ApiHandler</span><span class="p">(</span><span class="k">async</span> <span class="p">(</span><span class="nx">_evt</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">newLink</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">Link</span><span class="p">.</span><span class="nx">create</span><span class="p">(</span><span class="dl">"</span><span class="s2">test</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">https://google.com</span><span class="dl">"</span><span class="p">);</span>
<span class="k">return</span> <span class="p">{</span>
<span class="na">body</span><span class="p">:</span> <span class="p">{</span>
<span class="na">link</span><span class="p">:</span> <span class="nx">newLink</span><span class="p">,</span>
<span class="p">},</span>
<span class="p">};</span>
<span class="p">});</span>
</code></pre></div></div>
<p>You can view this code in the <a href="https://github.com/nickymarino/url-shortener/commit/7f55e809bd70d791daddb52aa22efd39bb582a91"><strong>Add link create</strong></a> commit.</p>
Building a Full-Stack Serverless URL Shortener2023-06-05T00:00:00+00:00https://nickymarino.com/2023/06/05/building-a-url-shortener<p><strong>I’m going to strengthen my fullstack serverless developer skills by building a clone of <a href="https://bitly.com/">bitly</a>.</strong></p>
<p>This idea came from a conversation I had with a friend of mine, who hosts parties often and manage the invite lists with a Telegram chat. He wanted to create fancy invitations by embossing a link to the chat as a QR code, and to do that he needed to create a rubber stamp for the QR code.</p>
<p>Custom rubber stamps are pretty expensive, and so I suggested that the QR code point to a shortened URL. That way, he can change the URL later without creating a new stamp. To which he replied:</p>
<blockquote>
<p>“I’ll use bitly to create the invite link, of course.”</p>
</blockquote>
<p>Now don’t get me wrong, bitly is great for non-technical folks to create shortened URLs. However, especially after <a href="https://blog.heroku.com/next-chapter">Heroku ended their free tier</a>, I’ve learned that there’s no such thing as a free lunch. I’m immediately leery of any free service that services a <em>lot</em> of people without a very obvious market strategy behind it.</p>
<p>So, it’s about time I build my own URL shortener! I have plenty of use cases for it, it’s a great side project to learn how to write fullstack apps, and I can always open it up to my friends to use as well.</p>
<h2 id="a-foray-into-developer-journals">A Foray into Developer Journals</h2>
<p>Inspired by <a href="https://jurajmajerik.com/blog/start-here/">Juraj Majerik’s diary of an Uber clone</a> and <a href="https://www.david-smith.org/dnd/">David Smith’s design diary</a>, my plan is to write short blog posts on the days I work on this project. In a perfect world, I’d like to devote at least an hour daily to this, but I’m aiming to make a post at least once a week until completion.</p>
<h2 id="architecture-overview">Architecture Overview</h2>
<p>Before I get started, I drafted what I expect my tech stack to look like in broad strokes. Additional context I used:</p>
<ul>
<li>Unlike most side projects, this is one I hope to use personally going forward.</li>
<li>At work, our stack is <a href="https://aws.amazon.com/what-is-aws/">AWS</a> and <a href="https://docs.sst.dev/">SST</a> for infrastructure, and <a href="https://react.dev/">React</a> and <a href="https://mui.com/">MUI</a> for the frontend.</li>
</ul>
<p>Here’s my plan for the architecture:</p>
<p><img src="https://nickymarino.com/public/assets/2023/url-shortener/01-arch-diagram.png" alt="Architecture diagram of the URL shortener" class="center" /></p>
<p>I’m going to use <a href="https://docs.sst.dev/">SST</a> to manage the infrastructure and stitching all the pieces together. For the API backend, the data store will be a <a href="https://aws.amazon.com/dynamodb/">DynamnoDB</a> <a href="https://aws.amazon.com/blogs/compute/creating-a-single-table-design-with-amazon-dynamodb/">mono-table</a>, and the compute will be <a href="https://aws.amazon.com/lambda/">AWS Lambda</a> functions behind an <a href="https://aws.amazon.com/api-gateway/">API Gateway</a>. <a href="https://aws.amazon.com/cognito/">AWS Cognito</a> will manage the user authentication and authorization.</p>
<p>I’m really excited to embark on this journey. By documenting my progress in short blog posts, I hope to share my experiences and insights with others who may be interested in similar projects. Stay tuned for future updates as I delve into the development process and bring this project to life. Let’s build something amazing together!</p>
Use Pyenv to Streamline your Python Projects2022-05-30T00:00:00+00:00https://nickymarino.com/2022/05/30/pyenv-and-virtualenv<blockquote>
<p><em>There are two hard problems to solve in a Python project: installing packages, and installing packages in other projects.</em></p>
</blockquote>
<p>Over the years, the challenges I’ve come across in Python projects has narrowed down to two topics <sup id="fnref:python2" role="doc-noteref"><a href="#fn:python2" class="footnote" rel="footnote">1</a></sup> :</p>
<ol>
<li>Installing packages for <em>just</em> my current project – not for all projects using that version of Python</li>
<li>Using different Python versions for different projects</li>
</ol>
<p>Problem (1) can be solved with Python’s built in <a href="https://docs.python.org/3/library/venv.html">virtual environments</a>, but they don’t automatically activate when you enter a project folder. As a result, I often forget to run <code class="language-plaintext highlighter-rouge">source venv/activate</code> and then accidentally install a bunch of packages to my system Python.</p>
<p>I highly recommend <a href="https://github.com/pyenv/pyenv">Pyenv</a> to solve both problems. Pyenv is a fantastic tool for managing Python environments simply by changing your current folder. Pyenv also integrates with <code class="language-plaintext highlighter-rouge">virtualenv</code> so that you can create virtual environments for any Python version you’d like to use.</p>
<h2 id="overview">Overview</h2>
<p>By the end of this post you will know how to:</p>
<ul>
<li>Install <code class="language-plaintext highlighter-rouge">pyenv</code> on your computer</li>
<li>Build a version of Python with <code class="language-plaintext highlighter-rouge">pyenv</code></li>
<li>Change your global and local folder settings to run the new version of Python</li>
<li>Connect <code class="language-plaintext highlighter-rouge">pyenv</code> to <code class="language-plaintext highlighter-rouge">virtualenv</code> to create a new virtual environment</li>
</ul>
<h2 id="install-pyenv">Install Pyenv</h2>
<p>First, install Xcode tools and a few development libraries that the Python build step will need:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Note: These are macOS specific commands</span>
<span class="c"># Update XCode and Homebrew</span>
xcode-select <span class="nt">--install</span>
brew update
<span class="c"># Install libraries you'll need to build Python from source</span>
brew <span class="nb">install </span>openssl readline sqlite3 xz zlib
<span class="c"># Install pyenv</span>
brew <span class="nb">install </span>pyenv
</code></pre></div></div>
<p><strong><em>Note:</em></strong> <em>The installation instructions for <code class="language-plaintext highlighter-rouge">pyenv</code> change frequently. These are the steps for <code class="language-plaintext highlighter-rouge">pyenv==2.3.1</code> on macOS. If you are using a different version or OS, see the <code class="language-plaintext highlighter-rouge">pyenv</code> <a href="https://github.com/pyenv/pyenv#installation">installation guide</a>.</em></p>
<p>If you’re using a zsh shell, run these commands to add <code class="language-plaintext highlighter-rouge">pyenv</code> to your dot files. If you’re using another shell, follow the installation steps for your shell under the <a href="https://github.com/pyenv/pyenv#set-up-your-shell-environment-for-pyenv">Shell environment section of the pyenv README</a>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">echo</span> <span class="s1">'export PYENV_ROOT="$HOME/.pyenv"'</span> <span class="o">>></span> ~/.zshrc
<span class="nb">echo</span> <span class="s1">'command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"'</span> <span class="o">>></span> ~/.zshrc
<span class="nb">echo</span> <span class="s1">'eval "$(pyenv init -)"'</span> <span class="o">>></span> ~/.zshrc
</code></pre></div></div>
<p>Restart your terminal for the changes to take effect, and verify your installation by entering the command <code class="language-plaintext highlighter-rouge">which python3</code> and confirming that <code class="language-plaintext highlighter-rouge">pyenv/.shims</code> is somewhere in the output:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ which python3
/Users/nicky/.pyenv/shims/python3
</code></pre></div></div>
<h2 id="install-python-382">Install Python 3.8.2</h2>
<p>Next let’s install Python 3.8.2. I chose this particular version of Python because I use it in a few of my side projects, so you can use any other version of Python if you’d like.</p>
<p>Recently, I came across a bug in macOS Big Sur that prevents libraries like <code class="language-plaintext highlighter-rouge">numpy</code> and <code class="language-plaintext highlighter-rouge">pandas</code> to use the <code class="language-plaintext highlighter-rouge">lzma</code> package. Credit to the thread for <a href="https://github.com/pyenv/pyenv/issues/1737#issuecomment-738080459">pyenv Issue #1737</a> for the below instructions to correct the 3.8.2 build.</p>
<p>To install Python 3.8.2, first reinstall the <code class="language-plaintext highlighter-rouge">zlib</code> and <code class="language-plaintext highlighter-rouge">bzip2</code> libraries via Homebrew:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>brew reinstall zlib bzip2
</code></pre></div></div>
<p>In your <code class="language-plaintext highlighter-rouge">~/.zshrc</code> (or <code class="language-plaintext highlighter-rouge">~/.bashrc</code> if you’re using bash), add the below flags for the Python compiler to read:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">export </span><span class="nv">LDFLAGS</span><span class="o">=</span><span class="s2">"-L/usr/local/opt/zlib/lib -L/usr/local/opt/bzip2/lib"</span>
<span class="nb">export </span><span class="nv">CPPFLAGS</span><span class="o">=</span><span class="s2">"-I/usr/local/opt/zlib/include -I/usr/local/opt/bzip2/include"</span>
</code></pre></div></div>
<p>Then install Python 3.8.2 using the below command. This may take some time to compile.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">CFLAGS</span><span class="o">=</span><span class="s2">"-I</span><span class="si">$(</span>brew <span class="nt">--prefix</span> openssl<span class="si">)</span><span class="s2">/include -I</span><span class="si">$(</span>brew <span class="nt">--prefix</span> bzip2<span class="si">)</span><span class="s2">/include -I</span><span class="si">$(</span>brew <span class="nt">--prefix</span> readline<span class="si">)</span><span class="s2">/include -I</span><span class="si">$(</span>xcrun <span class="nt">--show-sdk-path</span><span class="si">)</span><span class="s2">/usr/include"</span> <span class="se">\</span>
<span class="nv">LDFLAGS</span><span class="o">=</span><span class="s2">"-L</span><span class="si">$(</span>brew <span class="nt">--prefix</span> openssl<span class="si">)</span><span class="s2">/lib -L</span><span class="si">$(</span>brew <span class="nt">--prefix</span> readline<span class="si">)</span><span class="s2">/lib -L</span><span class="si">$(</span>brew <span class="nt">--prefix</span> zlib<span class="si">)</span><span class="s2">/lib -L</span><span class="si">$(</span>brew <span class="nt">--prefix</span> bzip2<span class="si">)</span><span class="s2">/lib"</span> <span class="se">\</span>
pyenv <span class="nb">install</span> <span class="nt">--patch</span> 3.8.2 < <<span class="o">(</span>curl <span class="nt">-sSL</span> https://github.com/python/cpython/commit/8ea6353.patch<span class="se">\?</span>full_index<span class="se">\=</span>1<span class="o">)</span>
</code></pre></div></div>
<p>Verify that you built and installed Python 3.8.2 by checking what versions of Python <code class="language-plaintext highlighter-rouge">pyenv</code> has:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ pyenv versions
system
3.8.2
</code></pre></div></div>
<h2 id="manage-pyenv-versions">Manage Pyenv Versions</h2>
<p>Pyenv has two types of versions:</p>
<ol>
<li><code class="language-plaintext highlighter-rouge">local</code> – the Python version for your current directory</li>
<li><code class="language-plaintext highlighter-rouge">global</code> – the default Python version to use if no <code class="language-plaintext highlighter-rouge">local</code> version is set.</li>
</ol>
<p>When you switch directories in your terminal, Pyenv checks for a <code class="language-plaintext highlighter-rouge">.python-version</code> file in the root of that directory. If there’s no <code class="language-plaintext highlighter-rouge">.python-version</code>, Pyenv will search each parent folder until one is found.<sup id="fnref:global" role="doc-noteref"><a href="#fn:global" class="footnote" rel="footnote">2</a></sup></p>
<p>You can set what version of Python <code class="language-plaintext highlighter-rouge">pyenv</code> will use in your current folder with <code class="language-plaintext highlighter-rouge">pyenv local</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ pyenv local 3.8.2
</code></pre></div></div>
<p>You can check <code class="language-plaintext highlighter-rouge">.python-version</code> to see what version is used:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ cat .python-version
3.8.2
</code></pre></div></div>
<p>And <code class="language-plaintext highlighter-rouge">pyenv version</code> will also tell you what version of Python you’re currently using as well as what config file set it:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ pyenv version
3.8.2 (set by /Users/nicky/Developer/my-project/.python-version)
</code></pre></div></div>
<p>If you move to a different folder without a <code class="language-plaintext highlighter-rouge">.python-version</code>, Pyenv will use your <code class="language-plaintext highlighter-rouge">global</code> setting, which is <code class="language-plaintext highlighter-rouge">system</code> by default:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ cd /path/to/other/folder
$ pyenv version
system (set by /Users/nicky/.python-version)
</code></pre></div></div>
<h2 id="how-to-use-pyenv-virtual-environments">How to Use Pyenv Virtual Environments</h2>
<p>Give <code class="language-plaintext highlighter-rouge">pyenv</code> superpowers by installing <a href="https://github.com/pyenv/pyenv-virtualenv">pyenv-virtualenv</a>. With this plugin, <code class="language-plaintext highlighter-rouge">pyenv</code> can manage virtual environments like <code class="language-plaintext highlighter-rouge">venv</code> and Conda environments.</p>
<p>Install the plugin with Homebrew:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>brew install pyenv-virtualenv
</code></pre></div></div>
<p>And add the following to your shell’s <code class="language-plaintext highlighter-rouge">.rc</code> file (such as <code class="language-plaintext highlighter-rouge">.zshrc</code> or <code class="language-plaintext highlighter-rouge">.bashrc</code>):</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">eval</span> <span class="s2">"</span><span class="si">$(</span>pyenv virtualenv-init -<span class="si">)</span><span class="s2">"</span>
</code></pre></div></div>
<p>To create a virtual environment for the Python version you’re using with <code class="language-plaintext highlighter-rouge">pyenv</code>, run <code class="language-plaintext highlighter-rouge">pyenv virtualenv [version] [new environment name]</code>. For example, to create a new <code class="language-plaintext highlighter-rouge">venv</code> for a sample project using 3.8.2:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pyenv virtualenv 3.8.2 my-project-3.8.2
</code></pre></div></div>
<p>While not required, I recommend adding the Python version to your environment names to manage them easier.</p>
<p>You can also create a new virtual environment using the current <code class="language-plaintext highlighter-rouge">pyenv</code> Python version:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pyenv virtualenv my-default-venv
</code></pre></div></div>
<p>Just like <code class="language-plaintext highlighter-rouge">pyenv</code>, your virtual environment will automatically activate whenever you move into that folder!</p>
<h2 id="helpful-commands">Helpful Commands</h2>
<p>Below is a list of commands for <code class="language-plaintext highlighter-rouge">pyenv</code> that I use often.</p>
<p>To show all of your Python versions and all virtual environments:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pyenv versions
</code></pre></div></div>
<p>To set a different version for the <code class="language-plaintext highlighter-rouge">python</code> and <code class="language-plaintext highlighter-rouge">python3</code> commands for your <code class="language-plaintext highlighter-rouge">global</code> settings:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># (This works the same with `local` as well)
pyenv global [python version] [python3 version]
</code></pre></div></div>
<p>To create a new virtual environment and use it locally:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pyenv virtualenv [python version] [new environment name]
pyenv local [new environment name]
</code></pre></div></div>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:python2" role="doc-endnote">
<p>The rest are <code class="language-plaintext highlighter-rouge">python</code> vs <code class="language-plaintext highlighter-rouge">python3</code> compatibility problems. <a href="#fnref:python2" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:global" role="doc-endnote">
<p>The <code class="language-plaintext highlighter-rouge">.python-version</code> at the root of your home folder sets your <code class="language-plaintext highlighter-rouge">global</code> Python version. <a href="#fnref:global" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
Add Your Vimrc to Obsidian2021-12-21T00:00:00+00:00https://nickymarino.com/2021/12/21/add-vimrc-to-obsidian<p>I’ve migrated my note taking system from <a href="https://www.craft.do">Craft</a> to <a href="https://www.craft.do">Obsidian</a> over the past few weeks, and it’s been great so far. One major advantage of Obsidian is its amazing developer community and wide, wide universe of <a href="https://obsidian.md/plugins">community plugins</a>. I use quite a few community plugins:</p>
<p><img src="https://nickymarino.com/public/assets/2021/add-vimrc-to-obsidian/img1.png" alt="My community plugins" class="center" /></p>
<p>One of my favorite features of Obsidian is its support for vim bindings. I use vim and VS Code’s vim bindings all the time, and I’m able to context switch much faster between coding and note writing when all of my editors use vim keys.<sup id="fnref:google-docs" role="doc-noteref"><a href="#fn:google-docs" class="footnote" rel="footnote">1</a></sup></p>
<p>You can turn on vim bindings in your own Obsidian vault by going to “Settings” > “Editor” and toggling “Vim key bindings”:</p>
<p><img src="https://nickymarino.com/public/assets/2021/add-vimrc-to-obsidian/img2.png" alt="Enable vim bindings" class="center" /></p>
<p>The only downside to Obsidian’s default vim bindings is the lack of <code class="language-plaintext highlighter-rouge">.vimrc</code> support out of the box. My personal <code class="language-plaintext highlighter-rouge">.vimrc</code> has a few shortcuts that I rely on:</p>
<ul>
<li>I map <code class="language-plaintext highlighter-rouge">;</code> to <code class="language-plaintext highlighter-rouge">:</code> in normal mode so that I don’t rely on the shift key</li>
<li>I map <code class="language-plaintext highlighter-rouge">j</code> to <code class="language-plaintext highlighter-rouge">gj</code> (and <code class="language-plaintext highlighter-rouge">k</code> to <code class="language-plaintext highlighter-rouge">gk</code>) to jump by visual lines instead of logical lines by default</li>
</ul>
<p>The <code class="language-plaintext highlighter-rouge">;</code> mapping isn’t a deal breaker in Obsidian; I’m not using vim commands that often. However, the <code class="language-plaintext highlighter-rouge">j</code> and <code class="language-plaintext highlighter-rouge">k</code> remaps are critical! Obsidian wraps text much more than a terminal thanks to its generous padding. Even on this “simple” post with just a lot of text, one line turns into four in Obsidian:</p>
<p><img src="https://nickymarino.com/public/assets/2021/add-vimrc-to-obsidian/img3.png" alt="Long text line example" class="center" /></p>
<p>Luckily, there’s plugin for Obsidian <code class="language-plaintext highlighter-rouge">.vimrc</code> files: <a href="https://github.com/esm7/obsidian-vimrc-support">Obsidian Vimrc Support Plugin</a>. To add this to your Obsidian vault, go to “Settings” > “Community plugins” > “Browse” and search for “vimrc”:</p>
<p><img src="https://nickymarino.com/public/assets/2021/add-vimrc-to-obsidian/img4.png" alt="Searching for vimrc plugin" class="center" /></p>
<p>Then click “Vimrc Support”, “Install” and “Enable”:</p>
<p><img src="https://nickymarino.com/public/assets/2021/add-vimrc-to-obsidian/img5.png" alt="Enabling vimrc support plugin" class="center" /></p>
<p>At the root of your Obsidian vault (the root folder for all of your <code class="language-plaintext highlighter-rouge">.md</code> files), create a new file named <code class="language-plaintext highlighter-rouge">.obsidian.vimrc</code>. You can paste any of your <code class="language-plaintext highlighter-rouge">~/.vimrc</code> into your <code class="language-plaintext highlighter-rouge">.obsidian.vimrc</code>.</p>
<p>Here’s my <code class="language-plaintext highlighter-rouge">.obsidian.vimrc</code> that remaps <code class="language-plaintext highlighter-rouge">;</code>, <code class="language-plaintext highlighter-rouge">j</code>, and <code class="language-plaintext highlighter-rouge">k</code> in normal mode:</p>
<div class="language-vim highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">" .obsidian.vimrc</span>
<span class="c">"</span>
<span class="c">" A small .vimrc for Obsidian vim bindings</span>
<span class="c">"</span>
<span class="c">" To enable this file, you must install the Vimrc Support plugin for Obsidian:</span>
<span class="c">" https://github.com/esm7/obsidian-vimrc-support</span>
<span class="c">"_________________________________________________________________________</span>
<span class="c">" ; (semicolon) - same as : (colon)</span>
nmap ; <span class="p">:</span>
<span class="c">" (space) - same as : (colon)</span>
nmap <span class="p"><</span>SPACE<span class="p">></span> <span class="p">:</span>
<span class="c">" j and k navigate visual lines rather than logical ones</span>
nmap <span class="k">j</span> gj
nmap <span class="k">k</span> gk
</code></pre></div></div>
<p>Once you write your <code class="language-plaintext highlighter-rouge">.obsidian.vimrc</code>, reload Obsidian and your new vim bindings will load!</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:google-docs" role="doc-endnote">
<p>The only downside is that I start typing gibberish whenever I open Google Docs or Word, but that’s a sacrifice I’m willing to make. <a href="#fnref:google-docs" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
How to Install Ruby 2.7.3 on M1 Mac2021-12-17T00:00:00+00:00https://nickymarino.com/2021/12/17/install-ruby-273-on-m1<p>Installing Ruby or Python on M1 Macs is a nightmare. I’ve lost so many hours fighting compilers and Rosetta to these issues, so I’m documenting my installation steps to spare you<sup id="fnref:and-me" role="doc-noteref"><a href="#fn:and-me" class="footnote" rel="footnote">1</a></sup> a lot of headache.</p>
<h2 id="why-273">Why 2.7.3?</h2>
<p>First, why Ruby 2.7.3 specifically? Ruby 3.0.0+ works great on M1 using <a href="https://nickymarino.com/2020/06/13/jekyll-server-rvm-macos/">my RVM install instructions post</a>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rvm install 3.0.0
</code></pre></div></div>
<p>Unfortunately, my website’s host <a href="https://pages.github.com">GitHub Pages</a> uses Ruby 2.7.3 according to their <a href="https://pages.github.com/versions/">dependency documentation</a>. And a bare-bones install of 2.7.3 blows up on my M1 machine:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rvm install 2.7.3
</code></pre></div></div>
<p>So I needed to find a way to install 2.7.3 on my Mac to build and run my website locally to preview new posts.</p>
<h2 id="the-solution">The Solution</h2>
<p>I’ll put the winning command here at the top to keep things simple. Many thanks to <a href="https://github.com/d-lebed">@d-lebed</a> for documenting their solution on this <a href="https://github.com/rvm/rvm/issues/5033#issuecomment-991949115">GitHub issue</a>. I lightly modified their code to use <code class="language-plaintext highlighter-rouge">$(brew --prefix)</code> instead of hardcoding where <code class="language-plaintext highlighter-rouge">homebrew</code> downloads <code class="language-plaintext highlighter-rouge">openssl@1.1</code>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Winning script!</span>
brew <span class="nb">install </span>openssl@1.1
<span class="nb">export </span><span class="nv">PATH</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span>brew <span class="nt">--prefix</span><span class="si">)</span><span class="s2">/opt/openssl@1.1/bin:</span><span class="nv">$PATH</span><span class="s2">"</span>
<span class="nb">export </span><span class="nv">LDFLAGS</span><span class="o">=</span><span class="s2">"-L</span><span class="si">$(</span>brew <span class="nt">--prefix</span><span class="si">)</span><span class="s2">/opt/openssl@1.1/lib"</span>
<span class="nb">export </span><span class="nv">CPPFLAGS</span><span class="o">=</span><span class="s2">"-I</span><span class="si">$(</span>brew <span class="nt">--prefix</span><span class="si">)</span><span class="s2">/opt/openssl@1.1/include"</span>
<span class="nb">export </span><span class="nv">PKG_CONFIG_PATH</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span>brew <span class="nt">--prefix</span><span class="si">)</span><span class="s2">/opt/openssl@1.1/lib/pkgconfig"</span>
rvm autolibs disable
<span class="nb">export </span><span class="nv">RUBY_CFLAGS</span><span class="o">=</span><span class="nt">-DUSE_FFI_CLOSURE_ALLOC</span>
<span class="nb">export </span><span class="nv">optflags</span><span class="o">=</span><span class="s2">"-Wno-error=implicit-function-declaration"</span>
rvm <span class="nb">install </span>2.7.3 <span class="nt">--with-openssl-dir</span><span class="o">=</span><span class="si">$(</span>brew <span class="nt">--prefix</span><span class="si">)</span>/opt/openssl@1.1
</code></pre></div></div>
<p>This successfully installed Ruby 2.7.3 for me, and I was then able to run <code class="language-plaintext highlighter-rouge">bundle install</code> at the root of my website’s repo!</p>
<h2 id="the-issue">The Issue</h2>
<p>I originally documented all of my problem solving steps and install attempts, but it grew too long and rambly for my liking. Instead, I’ll give a brief summary of a few errors I came across.</p>
<h3 id="flags">Flags</h3>
<p>At one point, <code class="language-plaintext highlighter-rouge">rvm</code> was complaining about my <code class="language-plaintext highlighter-rouge">LDFLAGS</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>checking whether LDFLAGS is valid... no
configure: error: something wrong with LDFLAGS="-L/usr/local/opt/zlib/lib -L/usr/local/opt/bzip2/lib"
</code></pre></div></div>
<p>And setting <code class="language-plaintext highlighter-rouge">LDFLAGS=""</code> in front of <code class="language-plaintext highlighter-rouge">rvm install 2.7.3</code> only resulted in errors with the compiler missing <code class="language-plaintext highlighter-rouge">openssl</code> libraries. So I needed <code class="language-plaintext highlighter-rouge">LDFLAGS</code> set somehow, but not the way that works for Python and <code class="language-plaintext highlighter-rouge">pyenv</code>.</p>
<h3 id="rosetta">Rosetta</h3>
<p>In a few GitHub issues, people recommended opening the Terminal app via Rosetta and running commands that way such as in <a href="https://github.com/rvm/rvm/issues/5146#issuecomment-967048308">this issue comment</a>. However, I saw no difference in error outputs between Rosetta Terminal and iTerm. In the end, I used iTerm without Rosetta to successfully install Ruby 2.7.3.</p>
<h3 id="usename-macro-error">Usename Macro Error</h3>
<p>I got <em>close</em> to a correct install when I added the openssl library to <code class="language-plaintext highlighter-rouge">LDFLAGS</code>, <code class="language-plaintext highlighter-rouge">CPPFLAGS</code>, and <code class="language-plaintext highlighter-rouge">PKG_CONFIG_PATH</code>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">PATH</span><span class="o">=</span><span class="s2">"/usr/local/opt/openssl@1.1/bin:</span><span class="nv">$PATH</span><span class="s2">"</span> <span class="se">\</span>
<span class="nv">LDFLAGS</span><span class="o">=</span><span class="s2">"-L</span><span class="si">$(</span>brew <span class="nt">--prefix</span><span class="si">)</span><span class="s2">/opt/openssl@1.1/lib"</span> <span class="se">\</span>
<span class="nv">CPPFLAGS</span><span class="o">=</span><span class="s2">"-I</span><span class="si">$(</span>brew <span class="nt">--prefix</span><span class="si">)</span><span class="s2">/opt/openssl@1.1/include"</span> <span class="se">\</span>
<span class="nv">PKG_CONFIG_PATH</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span>brew <span class="nt">--prefix</span><span class="si">)</span><span class="s2">/opt/openssl@1.1/lib/pkgconfig"</span> <span class="se">\</span>
<span class="nb">arch</span> <span class="nt">-x86_64</span> rvm <span class="nb">install </span>2.7.3 <span class="nt">-j</span> 1
</code></pre></div></div>
<p>But I then started seeing errors about some missing <code class="language-plaintext highlighter-rouge">rl_username_completion_function</code> macro:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>214 warnings generated.
linking shared-object date_core.bundle
installing default date_core libraries
compiling readline.c
readline.c:1904:37: error: use of undeclared identifier 'username_completion_function'; did you mean 'rl_username_completion_function'?
rl_username_completion_function);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
rl_username_completion_function
readline.c:79:42: note: expanded from macro 'rl_username_completion_function'
# define rl_username_completion_function username_completion_function
^
/opt/homebrew/opt/readline/include/readline/readline.h:485:14: note: 'rl_username_completion_function' declared here
extern char *rl_username_completion_function PARAMS((const char *, int));
^
1 error generated.
make[2]: *** [readline.o] Error 1
make[1]: *** [ext/readline/all] Error 2
make: *** [build-ext] Error 2
+__rvm_make:0> return 2
</code></pre></div></div>
<p>I then found the winning command (above) by googling for <code class="language-plaintext highlighter-rouge">rvm install 2.7.3 error extern char *rl_username_completion_function PARAMS((const char *, int));</code> and trying a few commands recommended on some GitHub issues.</p>
<h2 id="next-steps">Next Steps</h2>
<p>I recently figured out how to install Python 3.7, 3.8, and 3.9 on M1 Macs via <code class="language-plaintext highlighter-rouge">pyenv</code>, so keep an eye out for my post on how to install those.</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:and-me" role="doc-endnote">
<p>And future me <a href="#fnref:and-me" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
Create Python Web Apps with Flask and Replit2021-04-13T00:00:00+00:00https://nickymarino.com/2021/04/13/create-python-web-apps-with-flask-and-replit<p><em>I wrote this tutorial for <a href="https://www.codeyourdreams.org">Code Your Dreams</a>, an incubator of youth voice, tech skills, and social justice. Their project-based and student-centered programs enable youth to be the change makers we know they are through code. To find out how to get involved, visit their website <a href="https://www.codeyourdreams.org">here</a>.</em></p>
<p><a href="https://repl.it">Replit</a> is a free, collaborative, in-browser IDE for creating new projects without setting up any environments on your computer. With Replit, you don’t need to “deploy” your projects to any service; they’ll be instantly available to you as soon as you start typing. In this post, we’ll review how to create a Flask app, set up folders for HTML and CSS templates, and learn how to navigate your Flask app.</p>
<p>Before following these steps, you must first create an account on <a href="replit.com">replit</a>.</p>
<h1 id="creating-a-flask-project">Creating a Flask Project</h1>
<p>First, let’s create a blank Python project. On your <a href="replit.com">replit</a> homepage, create new project by clicking “Python” under the “Create” heading:</p>
<p><img src="https://nickymarino.com/public/assets/2021/flask-and-replit/Image%20(9).png" alt="Replit homepage with the "Python" button highlighted" class="center" /></p>
<p>For the project name, type <code class="language-plaintext highlighter-rouge">my-first-flask-site</code> and click “Create repl”:</p>
<p><img src="https://nickymarino.com/public/assets/2021/flask-and-replit/Image.jpeg" alt="Example showing creating a new Python repl" class="center" /></p>
<p>Your new project will automatically create a file named <code class="language-plaintext highlighter-rouge">main.py</code> and open a Python IDE for you, but we need to install Flask before we can start writing our app. On the left sidebar, click the “Packages” icon, which looks like a hexagonal box:</p>
<p><img src="https://nickymarino.com/public/assets/2021/flask-and-replit/Image%20(8).png" alt="Repl editor with the "Packages" icon highlighted" class="center" /></p>
<p>From here, we can install any Python packages that you want to import in your app. Install the Flask package by typing “flask” and selecting the first item from the list named “Flask”:</p>
<p><img src="https://nickymarino.com/public/assets/2021/flask-and-replit/Image%20(7).png" alt="Repl editor with the "Flask" item highlighted" class="center" /></p>
<p>Then, click the “Files” icon on the left sidebar to go back to the files list. You should see <code class="language-plaintext highlighter-rouge">main.py</code>, which was already created for you.</p>
<h1 id="hello-world">Hello, World!</h1>
<p>Our first Flask app will have one page—the index page—that says <code class="language-plaintext highlighter-rouge">Hello World!</code> when we go to the home page. Copy the below code into the file named <code class="language-plaintext highlighter-rouge">main.py</code>:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">flask</span> <span class="kn">import</span> <span class="n">Flask</span><span class="p">,</span> <span class="n">render_template</span>
<span class="c1"># Create a flask app
</span><span class="n">app</span> <span class="o">=</span> <span class="n">Flask</span><span class="p">(</span>
<span class="n">__name__</span><span class="p">,</span>
<span class="n">template_folder</span><span class="o">=</span><span class="s">'templates'</span><span class="p">,</span>
<span class="n">static_folder</span><span class="o">=</span><span class="s">'static'</span>
<span class="p">)</span>
<span class="c1"># Index page
</span><span class="o">@</span><span class="n">app</span><span class="p">.</span><span class="n">route</span><span class="p">(</span><span class="s">'/'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">hello</span><span class="p">():</span>
<span class="k">return</span> <span class="s">"Hello World!"</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="c1"># Run the Flask app
</span> <span class="n">app</span><span class="p">.</span><span class="n">run</span><span class="p">(</span>
<span class="n">host</span><span class="o">=</span><span class="s">'0.0.0.0'</span><span class="p">,</span>
<span class="n">debug</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
<span class="n">port</span><span class="o">=</span><span class="mi">8080</span>
<span class="p">)</span>
</code></pre></div></div>
<p>In this code, we have one page that’s controlled by the <code class="language-plaintext highlighter-rouge">hello()</code> function. It’s route is <code class="language-plaintext highlighter-rouge">’/‘</code>, which means that it is at the home page of our app.</p>
<p>For flask projects, Replit looks for a web server at the local URL <code class="language-plaintext highlighter-rouge">http://0.0.0.0:8080</code>, so we need set the <code class="language-plaintext highlighter-rouge">host</code> to <code class="language-plaintext highlighter-rouge">’0.0.0.0'</code> and the <code class="language-plaintext highlighter-rouge">port</code> to <code class="language-plaintext highlighter-rouge">8080</code> in <code class="language-plaintext highlighter-rouge">app.run(…)</code>. We also set <code class="language-plaintext highlighter-rouge">debug=True</code> so that any changes you make to files will be automatically updated when you refresh a page.</p>
<p>We’ll use <code class="language-plaintext highlighter-rouge">render_template</code>, <code class="language-plaintext highlighter-rouge">template_folder</code>, and <code class="language-plaintext highlighter-rouge">static_folder</code> later in this tutorial, so don’t worry about those just yet.</p>
<p>Now, click the green “Run” button at the top of the page. Replit should install Flask, then open a browser with your first Flask app!</p>
<p><img src="https://nickymarino.com/public/assets/2021/flask-and-replit/Image%20(6).png" alt="Repl editor showing the Flask app running" class="center" /></p>
<p>The bottom right window is the Python console, and will show any error messages or logs that are printed.</p>
<p>At any time, you can click the “Stop” button at the top and click “Run” again to restart your Flask app.</p>
<p>Sometimes, your app might have multiple pages. To go to a different URL (or <code class="language-plaintext highlighter-rouge">@app.route</code>) in your app, click the icon “Open in a new tab” on the browser window. It will be to the right of the address bar:</p>
<p><img src="https://nickymarino.com/public/assets/2021/flask-and-replit/Image%20(5).png" alt="Repl editor with the "Open in a new tab" button highlighted" class="center" /></p>
<p>You can type in a new path in the address bar of the new tab, such as <a href="https://my-first-flask-site.songsblade.repl.co/my-second-page"><code class="language-plaintext highlighter-rouge">https://my-first-flask-site.YOUR_USERNAME.repl.co/my-second-page</code></a>. Right now, your server will return a 404 for that page because it doesn’t exist.</p>
<h1 id="html-assets-and-css-styles">HTML Assets and CSS Styles</h1>
<p>Next, let’s add an HTML file and a CSS file to our Flask app. HTML files are commonly put in a <code class="language-plaintext highlighter-rouge">templates</code> folder in a Flask project, because they are usually templates to show information. Our Flask code will supply values to HTML templates via variables, so that our app can change via Python code.</p>
<h2 id="adding-an-index-page">Adding an Index Page</h2>
<p>In <code class="language-plaintext highlighter-rouge">main.py</code>, we already set up our Flask app to look in the <code class="language-plaintext highlighter-rouge">templates</code> folder for HTML files:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Create a flask app
</span><span class="n">app</span> <span class="o">=</span> <span class="n">Flask</span><span class="p">(</span>
<span class="c1"># ...
</span> <span class="n">template_folder</span><span class="o">=</span><span class="s">'templates'</span><span class="p">,</span>
<span class="c1"># ...
</span><span class="p">)</span>
</code></pre></div></div>
<p>Now let’s create the <code class="language-plaintext highlighter-rouge">templates</code> folder and create an <code class="language-plaintext highlighter-rouge">index.html</code> file. Next to the “Files” header on the top left, click the “Add folder” button and name the new folder <code class="language-plaintext highlighter-rouge">templates</code>. Then, click the three dot icon on the <code class="language-plaintext highlighter-rouge">templates</code> folder and click “Add file”. Name the new file <code class="language-plaintext highlighter-rouge">index.html</code>. To have both <code class="language-plaintext highlighter-rouge">index.html</code> and <code class="language-plaintext highlighter-rouge">main.py</code> open at the same time in your editor, right click <code class="language-plaintext highlighter-rouge">main.py</code> on the files list and click “Open tab”.</p>
<p>Your editor should now look like this (note that there’s two tabs in the editor now: one for <code class="language-plaintext highlighter-rouge">templates/index.html</code> and one for <code class="language-plaintext highlighter-rouge">main.py</code>):</p>
<p><img src="https://nickymarino.com/public/assets/2021/flask-and-replit/Image%20(4).png" alt="Repl editor with the index and main tabs both open" class="center" /></p>
<p>Copy the below code into the <code class="language-plaintext highlighter-rouge">templates/index.html</code> file:</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><!doctype html></span>
<span class="nt"><head></span>
<span class="nt"><title></span>My First Flask Website<span class="nt"></title></span>
<span class="nt"><link</span> <span class="na">href=</span><span class="s">"/static/style.css"</span> <span class="na">rel=</span><span class="s">"stylesheet"</span> <span class="na">type=</span><span class="s">"text/css"</span><span class="nt">></span>
<span class="nt"></head></span>
<span class="nt"><body></span>
<span class="nt"><h1></span>Hello, World!<span class="nt"></h1></span>
<span class="nt"><p></span>
Welcome to your first Flask website
<span class="nt"></p></span>
<span class="nt"></body></span>
</code></pre></div></div>
<p>And replace your contents in <code class="language-plaintext highlighter-rouge">main.py</code> with the below code. This new version updates the <code class="language-plaintext highlighter-rouge">hello()</code> function to <code class="language-plaintext highlighter-rouge">index()</code>, and it returns the contents of <code class="language-plaintext highlighter-rouge">index.html</code> to the user:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">flask</span> <span class="kn">import</span> <span class="n">Flask</span><span class="p">,</span> <span class="n">render_template</span>
<span class="c1"># Create a flask app
</span><span class="n">app</span> <span class="o">=</span> <span class="n">Flask</span><span class="p">(</span>
<span class="n">__name__</span><span class="p">,</span>
<span class="n">template_folder</span><span class="o">=</span><span class="s">'templates'</span><span class="p">,</span>
<span class="n">static_folder</span><span class="o">=</span><span class="s">'static'</span>
<span class="p">)</span>
<span class="c1"># Index page (now using the index.html file)
</span><span class="o">@</span><span class="n">app</span><span class="p">.</span><span class="n">route</span><span class="p">(</span><span class="s">'/'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">index</span><span class="p">():</span>
<span class="k">return</span> <span class="n">render_template</span><span class="p">(</span><span class="s">'index.html'</span><span class="p">)</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="c1"># Run the Flask app
</span> <span class="n">app</span><span class="p">.</span><span class="n">run</span><span class="p">(</span>
<span class="n">host</span><span class="o">=</span><span class="s">'0.0.0.0'</span><span class="p">,</span>
<span class="n">debug</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
<span class="n">port</span><span class="o">=</span><span class="mi">8080</span>
<span class="p">)</span>
</code></pre></div></div>
<p>Click the Refresh button in the browser window of the project (<strong>not</strong> the refresh button in Chrome or Firefox, but the refresh button for the smaller window in your project), and you should see the contents of your index page with a large header:</p>
<p><img src="https://nickymarino.com/public/assets/2021/flask-and-replit/Image%20(3).png" alt="Repl editor showing the web app's new large header" class="center" /></p>
<h2 id="adding-css-styling">Adding CSS Styling</h2>
<p>Now let’s add a CSS file to change the color of the text of our app. The Flask app is set up to look inside a <code class="language-plaintext highlighter-rouge">static</code> folder for CSS and JS assets:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Create a flask app
</span><span class="n">app</span> <span class="o">=</span> <span class="n">Flask</span><span class="p">(</span>
<span class="c1"># ...
</span> <span class="n">static_folder</span><span class="o">=</span><span class="s">'static'</span>
<span class="p">)</span>
</code></pre></div></div>
<p>And the index page is set up to look for a file named <code class="language-plaintext highlighter-rouge">style.css</code> inside the <code class="language-plaintext highlighter-rouge">static</code> folder:</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><!doctype html></span>
<span class="nt"><head></span>
<span class="c"><!-- ... --></span>
<span class="nt"><link</span> <span class="na">href=</span><span class="s">"/static/style.css"</span> <span class="na">rel=</span><span class="s">"stylesheet"</span> <span class="na">type=</span><span class="s">"text/css"</span><span class="nt">></span>
<span class="c"><!-- ... --></span>
</code></pre></div></div>
<p>Click <code class="language-plaintext highlighter-rouge">main.py</code> on the Files list, then click the “Add a folder” icon to the left of the “Files” header. Name the new folder <code class="language-plaintext highlighter-rouge">static</code>. Next, click the three dot icon next to your new <code class="language-plaintext highlighter-rouge">static</code> folder and click “Add file”. Name the file <code class="language-plaintext highlighter-rouge">style.css</code>, and open it by right clicking the file and selecting “Open in tab”.</p>
<p>Your project should now look like this:</p>
<p><img src="https://nickymarino.com/public/assets/2021/flask-and-replit/Image%20(2).png" alt="Repl editor showing current state of the project" class="center" /></p>
<p>Let’s write some CSS to change the color of the “Welcome” message to red. Add the following code to your <code class="language-plaintext highlighter-rouge">static/style.css</code> file:</p>
<div class="language-css highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">p</span> <span class="p">{</span>
<span class="nl">color</span><span class="p">:</span> <span class="no">red</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Click the Refresh button in the project browser window (<strong>not</strong> the refresh button in Chrome or Firefox, but the refresh button for the smaller window in your project), and the “Welcome” screen will turn red:</p>
<p><img src="https://nickymarino.com/public/assets/2021/flask-and-replit/Image.png" alt="Repl editor showing the web app's new red text" class="center" /></p>
<p>Congratulations, you’ve written your very first Flask app!</p>
How to Draft Blog Posts in Notion2020-11-20T00:00:00+00:00https://nickymarino.com/2020/11/20/how-to-draft-blog-posts-in-notion<p>Maintaining a streamlined process to draft, edit, and publish technical articles can be tricky. For example, I write across multiple topics such as <a href="https://nickymarino.com/2020/09/29/python-list-comprehensions/">Python tutorials</a>, <a href="https://nickymarino.com/2020/09/06/multiple-aws-profiles/">bash commands</a>, and <a href="https://nickymarino.com/2020/01/21/optimizing-virgo/">Ruby optimizations</a>. I also publish to three different locations—my <a href="https://nickymarino.com">website</a>, <a href="https://dev.to/nickymarino">dev.to</a>, and <a href="https://nickymarino.hashnode.dev">Hashnode</a>, so I write my posts in Markdown-friendly tools for cross compatibility.</p>
<p><a href="https://www.notion.so/personal">Notion</a> is a great tool for drafting blog posts, technical or otherwise. Notion has an entire suite of tools to make drafting easy: Markdown exports, page properties and filters, a beautiful interface, and many more. With Notion, I quickly draft articles, publish each one across multiple sites, and monitor each article’s progress at a glance. Here’s what my setup looks like:</p>
<p><img src="https://nickymarino.com/public/assets/2020/notion-drafts-how-to/Untitled%202.png" alt="Gallery View of My Blog Posts" class="center" /></p>
<p>In this article, we will walk through creating an Articles page in Notion. We’ll create:</p>
<ul>
<li>A page for hold articles</li>
<li>An articles database</li>
<li>A new draft template, and</li>
<li>A Kanban board to display each draft’s status</li>
</ul>
<h2 id="markdown-jekyll-and-drafts-oh-my">Markdown, Jekyll, and Drafts! Oh My!</h2>
<p>For a while, I kept my drafts in a <code class="language-plaintext highlighter-rouge">_drafts</code> folder in my website’s <a href="https://github.com/nickymarino/nickymarino.github.io">repo</a>. <a href="https://jekyllrb.com">Jekyll</a>, my static site generator of choice, ignores any Markdown files in the <code class="language-plaintext highlighter-rouge">_drafts</code> folder. However, I’ve been hesitant to heavily draft and edit posts in a publicly available location, so I haven’t been drafting many posts lately. I love that my website is open source for others to use as an example for static site layouts, but I would like to keep the messiness of article writing to myself.</p>
<p>Then, I tried drafting my articles in the app <a href="https://getdrafts.com">Drafts</a>. Using Drafts, I could write posts in Markdown and preview them privately, but the only way to mark the status of a post (draft, edit, published, abandoned, etc.) was via tags. I used tags like <code class="language-plaintext highlighter-rouge">blog</code> and <code class="language-plaintext highlighter-rouge">draft</code> for quick filtering:</p>
<p><img src="https://nickymarino.com/public/assets/2020/notion-drafts-how-to/Untitled.png" alt="Article Drafts in the Drafts App" class="center" /></p>
<p>Unfortunately, tags didn’t scale for me. The only way to view each draft is from the small sidebar on the left, and I had to remember what combination of tags I used to mark blog drafts. Often, I’d forget to tag posts, or I’d forget to update the tags after I had drafted it. I also wanted to have a space devoted only to my articles so that I could get a bird’s eye view of what I had in the pipeline. That’s where <a href="http://www.notion.so/fd8cd2d212f74c50954c11086d85997e#345a826adfdb4d32a17ddcb9799386b4">Notion databases</a> come to the rescue.</p>
<h2 id="the-notion-database">The Notion Database</h2>
<p>Databases in Notion let me organize and structure related pages. Now that my articles are in a database, I can label, search, and create views of articles. For example, I can look at all of my articles as a list:</p>
<p><img src="https://nickymarino.com/public/assets/2020/notion-drafts-how-to/Untitled%201.png" alt="My Blog Posts Main Page" class="center" /></p>
<p>I can also view article drafts as a gallery, with a preview of each article:</p>
<p><img src="https://nickymarino.com/public/assets/2020/notion-drafts-how-to/Untitled%202.png" alt="Gallery View of My Blog Posts" class="center" /></p>
<p>I can even view my articles as a Kanban board based on their status. I use this birds eye view to see what articles are currently being drafted, which are in review, and which have been posted or abandoned:</p>
<p><img src="https://nickymarino.com/public/assets/2020/notion-drafts-how-to/Untitled%203.png" alt="Kanban View of My Blog Posts" class="center" /></p>
<h2 id="creating-the-table">Creating the Table</h2>
<p>Let’s start by creating a page and a database to hold the articles. First, click “+ Add a page” on the left sidebar. Give the page any title, icon, and header that you want. I titled mine “My Articles”:</p>
<p><img src="https://nickymarino.com/public/assets/2020/notion-drafts-how-to/blank-articles-page.png" alt="Blank Articles Page" class="center" /></p>
<p>Next, we need to create an article database. Click the text that says “Type ‘/’ for commands”, then type <code class="language-plaintext highlighter-rouge">/list inline</code>. Choose the option labelled “List - Inline” under “Database”:</p>
<p><img src="https://nickymarino.com/public/assets/2020/notion-drafts-how-to/Untitled%204.png" alt="Creating a New List" class="center" /></p>
<p>Notion will create a new list for you and populate a few example pages:</p>
<p><img src="https://nickymarino.com/public/assets/2020/notion-drafts-how-to/Untitled%205.png" alt="New List View on the Articles Page" class="center" /></p>
<p>Add a title to your table by clicking on “Untitled” and adding a title, such as “Articles”:</p>
<p><img src="https://nickymarino.com/public/assets/2020/notion-drafts-how-to/Untitled%206.png" alt="Articles List with Toolbar" class="center" /></p>
<h3 id="table-properties">Table Properties</h3>
<p>Let’s add some article properties for our pages. In this database, each article will be a page in Notion. Any properties we add to the Articles database will be shown in each page. Each article needs a title, creation date, last edited time, and status. For articles, the title property will be the the title of the Notion page, but we need to add the rest of the properties to the table.</p>
<p>To add properties, hover over the Articles table, click on the three dots on the right, and select “Properties”. Add a status property by clicking “+ Add a property”. Then, for “Property Type”, pick “Select”. To show the property in the table view, enable the toggle next to Select. Similarly, add a “Created” property by selecting the type “Created time”, and add an “Edited” property by selecting the type “Last edited time”. Delete the “Tags” property by selecting the dots on the left, and then select “Delete”. The Articles table properties now look like this:</p>
<p><img src="https://nickymarino.com/public/assets/2020/notion-drafts-how-to/Untitled%207.png" alt="List View Properties Panel" class="center" /></p>
<p>Now that our table is set up for articles, we can create a “New Draft” page template!</p>
<h2 id="the-page-is-as-mighty-as-the-table">The Page is as Mighty as the Table</h2>
<p>Most likely, you’ll add new pages to your Articles table when you’re creating a new draft. With a “New Draft” page template, it’s easy to start drafting and not need to worry about setting any page properties. To create a new draft, click the dropdown arrow next to “New” on the Articles table, and then click “+ New template”.</p>
<p>Name the template “New Draft”. We’ll want any new draft to have a Draft status, so we need to create that status. Click the text “Empty” next to the Status property, type in “Draft”. This will create the Draft status for any Article page, and assign the Draft status to any page that is created with the New Draft template. I also added a ✏️ icon to the New Draft template and the Draft that they’re easy to find in lists:</p>
<p><img src="https://nickymarino.com/public/assets/2020/notion-drafts-how-to/Untitled%208.png" alt="New Draft Template" class="center" /></p>
<h3 id="hello-world">Hello World</h3>
<p>Let’s write our first article draft! Click the “Back” button to go back to the Articles table, then select “New Draft” to begin writing. Add a title and some example text for your first draft:</p>
<p><img src="https://nickymarino.com/public/assets/2020/notion-drafts-how-to/Untitled%209.png" alt="Your First Draft Page" class="center" /></p>
<p>Click “Back” again to go to the Articles table, and delete the old example pages. Practice using Notion page templates by adding another draft:</p>
<p><img src="https://nickymarino.com/public/assets/2020/notion-drafts-how-to/Untitled%2010.png" alt="Our Two Drafts in the Articles View" class="center" /></p>
<h3 id="you-are-your-own-publisher">You are Your Own Publisher</h3>
<p>One way to publish your articles is to make the page publicly accessible in Notion. You can also use Notion’s Markdown exporter to convert a page into Markdown and then import the content into sites such as <a href="https://dev.to/nickymarino">dev.to</a> or <a href="https://nickymarino.hashnode.dev">Hashnode</a>. To export a page to Markdown, click the three dots in the top right of a page, select “Export”, then select “Markdown & CSV”. The page contents will be converted into Markdown for cross compatibility:</p>
<p><img src="https://nickymarino.com/public/assets/2020/notion-drafts-how-to/Untitled%2011.png" alt="Markdown Export of the Draft" class="center" /></p>
<h3 id="great-now-theres-two-of-them">Great, Now There’s Two of Them</h3>
<p>You’ve finished editing and publishing your “Hello, World!” article. Now it’s time to change the status of that article to Posted, but we don’t have an easy way to do that yet. Let’s create a new Kanban board view of our Articles table. With a Kanban board, you can move articles as cards from one status to the next.</p>
<p>To create a new table view, click “+ Add a view” at the top of the Articles table. Select the “Board” option and then click “Create”. Notion will automatically set up the Board view to sort articles by the Status property:</p>
<p><img src="https://nickymarino.com/public/assets/2020/notion-drafts-how-to/Untitled%2012.png" alt="Articles in a Kanban View" class="center" /></p>
<p>To add the Posted status, click “+ Add a group” and enter “Posted”. No pages have the Posted status yet, so the column will be empty:</p>
<p><img src="https://nickymarino.com/public/assets/2020/notion-drafts-how-to/Untitled%2013.png" alt="Kanban View with Posted Status" class="center" /></p>
<p>To mark the “Hello, World!” article as posted, simply drag that article’s card from Draft to Posted, and it’s done!</p>
<p><img src="https://nickymarino.com/public/assets/2020/notion-drafts-how-to/Untitled%2014.png" alt="Draft moved to Posted Column" class="center" /></p>
<p>With Notion’s databases, page properties, and custom views, you can create your own system to keep track of your blog posts and articles. I’m looking forward to using Notion more to structure my data in different aspects of my life.</p>
Hash Maps from Scratch Part 1: What is a Hash Map?2020-11-17T00:00:00+00:00https://nickymarino.com/2020/11/17/hash-maps-part-1<p>If you were designing a contacts app, how would you quickly access a phone number for a person given their name? You could keep an array of <code class="language-plaintext highlighter-rouge">Contact</code> objects and check each name:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">contacts</span> <span class="o">=</span> <span class="p">[</span> <span class="p">...</span> <span class="p">]</span>
<span class="k">def</span> <span class="nf">find_contact</span><span class="p">(</span><span class="n">name</span><span class="p">):</span>
<span class="k">for</span> <span class="n">person</span> <span class="ow">in</span> <span class="n">contacts</span><span class="p">:</span>
<span class="k">if</span> <span class="n">person</span><span class="p">.</span><span class="n">full_name</span> <span class="o">==</span> <span class="n">name</span><span class="p">:</span>
<span class="k">return</span> <span class="n">person</span>
<span class="k">return</span> <span class="bp">None</span>
</code></pre></div></div>
<p>This won’t scale very well if you have a lot of contacts though. What if your user has thousands of contacts on their phone? Instead, you could create a database table, index by name, and use SQL to find the contact:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">db_connection</span> <span class="o">=</span> <span class="n">connect_to_db</span><span class="p">(...)</span>
<span class="k">def</span> <span class="nf">find_contact</span><span class="p">(</span><span class="n">name</span><span class="p">):</span>
<span class="n">query</span> <span class="o">=</span> <span class="n">db_connection</span><span class="p">.</span><span class="n">sql</span><span class="p">(</span><span class="sa">f</span><span class="s">'select * from contacts where name like </span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s">'</span><span class="p">)</span>
<span class="k">return</span> <span class="n">query</span><span class="p">.</span><span class="n">first_or_none</span><span class="p">()</span>
</code></pre></div></div>
<p>Databases are great for this! But what if you’re keeping a list of contacts locally for a short period of time? Maybe you don’t want the hassle of maintaining a database for a small project, or you’re designing an algorithm designed to solve a problem <em>really</em> fast. You can use a hash map!</p>
<h3 id="what-is-a-hash-map">What is a Hash Map?</h3>
<p>A hash map associates a <code class="language-plaintext highlighter-rouge">key</code> with a <code class="language-plaintext highlighter-rouge">value</code>. Think of a phone book, or, if you’ve never seen a phone book, think of the Contacts app on your phone. If you want to know someone’s number, you look through your contact names, select the name you want, and that person’s phone number is shown next to their name:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Johnson, Sally 317-111-1111
Whittler, Mica 431-222-2222
Rodrigo, John 228-333-3333
</code></pre></div></div>
<p>For a hash map to hold a list of contacts, the <code class="language-plaintext highlighter-rouge">key</code> could be a name, and the <code class="language-plaintext highlighter-rouge">value</code> could be a phone number. A <code class="language-plaintext highlighter-rouge">key</code> and a <code class="language-plaintext highlighter-rouge">value</code> together are called a <em>key value pair</em>.</p>
<p><img src="https://nickymarino.com/public/assets/2020/hash-maps-part-1/Untitled_Artwork_2.png" alt="A key-value pair" class="center" /></p>
<p>Typically, a key is a primitive data type like an integer, float, or string. Values can be any type, such as a string, number, list, or object. Hash maps are called <code class="language-plaintext highlighter-rouge">dictionaries</code> (or <code class="language-plaintext highlighter-rouge">dicts</code>) in Python, and can be used in many contexts. Here’s what a contacts <code class="language-plaintext highlighter-rouge">dict</code> might look like:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">contacts</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">"Sally Johnson"</span><span class="p">:</span> <span class="s">"317-111-1111"</span><span class="p">,</span>
<span class="s">"Mica Whittler"</span><span class="p">:</span> <span class="s">"317-222-2222"</span><span class="p">,</span>
<span class="s">"John Rodrigo"</span><span class="p">:</span> <span class="s">"318-333-3333"</span>
<span class="p">}</span>
<span class="c1"># Prints "317-111-1111"
</span><span class="k">print</span><span class="p">(</span><span class="n">contacts</span><span class="p">[</span><span class="s">"Sally Johnson"</span><span class="p">])</span>
</code></pre></div></div>
<p>A hash map has three main parts:</p>
<ol>
<li>An <em>array</em> to hold items</li>
<li>A <em>hash function</em> to calculate the index of an item for the array, and</li>
<li>A method to handle hash collisions in the array (more on this later)</li>
</ol>
<p>Let’s walk through an example: Cave Johnson’s phone number is 888-724-3623 (888-SCIENCE). In our contacts hash map, we want to insert the value <code class="language-plaintext highlighter-rouge">888-724-3623</code> with the key <code class="language-plaintext highlighter-rouge">Cave Johnson</code>. Here’s what the Python code might look like:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Create an empty dictionary (hash map)
</span><span class="n">contacts</span> <span class="o">=</span> <span class="p">{}</span>
<span class="c1"># Add the new contact
</span><span class="n">contacts</span><span class="p">[</span><span class="s">"Cave Johnson"</span><span class="p">]</span> <span class="o">=</span> <span class="s">"888-724-3623"</span>
<span class="c1"># Prints '{"Cave Johnson": "888-724-3623"}'
</span><span class="k">print</span><span class="p">(</span><span class="n">contacts</span><span class="p">)</span>
<span class="c1"># Prints "888-724-3623"
</span><span class="k">print</span><span class="p">(</span><span class="n">contacts</span><span class="p">[</span><span class="s">"Cave Johnson"</span><span class="p">])</span>
</code></pre></div></div>
<p>Behind the scenes, the <code class="language-plaintext highlighter-rouge">contacts</code> dictionary will compute the hash, or index, of the key <code class="language-plaintext highlighter-rouge">Cave Johnson</code>, and insert the <em>key value pair</em> <code class="language-plaintext highlighter-rouge">("Cave Johnson", "888-724-3623")</code> at that index:</p>
<p><img src="https://nickymarino.com/public/assets/2020/hash-maps-part-1/Untitled_Artwork_5.png" alt="A key-value pair is inserted into a bucket index" class="center" /></p>
<h3 id="how-can-i-build-a-hash-map">How can I build a Hash Map?</h3>
<p>Let’s review the three main pieces we need to build a hash map:</p>
<ol>
<li>An array to hold key value pairs</li>
<li>A hash function to generate indices for each key value pair, and</li>
<li>A data structure inside the array to handle hash collisions</li>
</ol>
<p>Generally, a hash map’s hash function will generate a unique index for any item inserted into the hash map. However, a hash function might return the same index for two different values, which is called a <em>hash collision</em> or <em>index collision</em>.</p>
<p><img src="https://nickymarino.com/public/assets/2020/hash-maps-part-1/Untitled_Artwork_7.png" alt="Two values have the same hash value; a hash collision!" class="center" /></p>
<p>Hash maps must account for multiple key value pairs pointing to the same index. One way to account for index collisions is to create a linked list for each for each item in the main array. If two key value pairs are hashed to the same array index, the two pairs will be added to a linked list at that index:</p>
<p><img src="https://nickymarino.com/public/assets/2020/hash-maps-part-1/Untitled_Artwork_9.png" alt="A hash map array, where each array item is a linked list" class="center" /></p>
<h3 id="wrap-up">Wrap Up</h3>
<p>Hash maps are efficient data structures to associate <code class="language-plaintext highlighter-rouge">keys</code> with <code class="language-plaintext highlighter-rouge">values</code>. Hash maps use an array to hold key value pairs, a hash function to calculate the array index for each key value pair, and some data structure to account for index collisions.</p>
<p>To account for hash collisions in this series, each value of the hash map’s array will be a linked list. If any new key value pair has the same index as a previous pair, then the new pair will be added to the end of that index’s list. In the <a href="https://nickymarino.com/2020/11/18/hash-maps-part-2">next post</a>, we’ll learn how to build the linked list that we’ll use in our hash map class.</p>
How to Write List Comprehensions with Python2020-09-29T00:00:00+00:00https://nickymarino.com/2020/09/29/python-list-comprehensions<p>One of the most common blocks you will write in Python scripts is a <em>for loop</em>. With for loops, you can repeat the same set of instructions in a block over and over. Python’s for loops are really <em>foreach</em> loops, where you repeat the instructions for every item in a collection. These collections are called <em>iterators</em>, which is something that a Python loop is able to <em>iterate</em> over, and the most common iterator is <code class="language-plaintext highlighter-rouge">list</code>.</p>
<h2 id="for-loops">For Loops</h2>
<p>Let’s look at an example of a for loop. Write a function that prints the square of each number from one to <code class="language-plaintext highlighter-rouge">n</code>:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">write_squares</span><span class="p">(</span><span class="n">n</span><span class="p">):</span>
<span class="k">for</span> <span class="n">number</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n</span><span class="p">):</span>
<span class="n">square</span> <span class="o">=</span> <span class="n">number</span> <span class="o">**</span> <span class="mi">2</span>
<span class="k">print</span><span class="p">(</span><span class="n">square</span><span class="p">)</span>
<span class="n">write_squares</span><span class="p">(</span><span class="mi">6</span><span class="p">)</span>
</code></pre></div></div>
<p>This is the output you will get from the above example:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>0
1
4
9
16
25
</code></pre></div></div>
<p>In <code class="language-plaintext highlighter-rouge">write_squares</code>, we calculate the variable <code class="language-plaintext highlighter-rouge">square</code> and print it <em>for each</em> value in <code class="language-plaintext highlighter-rouge">range(n)</code>, which is all of the numbers from <code class="language-plaintext highlighter-rouge">0</code> to <code class="language-plaintext highlighter-rouge">n</code> inclusive.</p>
<p><img src="https://nickymarino.com/public/assets/2020/python-list-comprehension/boxes-arrows.png" alt="Boxes and Arrows" class="center" /></p>
<p>Small for loops like this are very common in Python scripts. For example, we can read in lines from a file and strip any spaces from each line:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">lines_from</span><span class="p">(</span><span class="n">filename</span><span class="p">):</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="s">"r"</span><span class="p">)</span> <span class="k">as</span> <span class="n">file_obj</span><span class="p">:</span>
<span class="n">original_lines</span> <span class="o">=</span> <span class="n">file_obj</span><span class="p">.</span><span class="n">readlines</span><span class="p">()</span>
<span class="n">stripped_lines</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">original_lines</span><span class="p">:</span>
<span class="n">new_line</span> <span class="o">=</span> <span class="n">line</span><span class="p">.</span><span class="n">strip</span><span class="p">()</span>
<span class="n">stripped_lines</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">new_line</span><span class="p">)</span>
</code></pre></div></div>
<p>Or, we might have a list of IDs and want to grab a piece of data for each ID:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">email_addresses</span><span class="p">(</span><span class="n">ids</span><span class="p">):</span>
<span class="n">addresses</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">identifier</span> <span class="ow">in</span> <span class="n">ids</span><span class="p">:</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">api_request</span><span class="p">(</span><span class="n">identifier</span><span class="p">)</span>
<span class="n">email_address</span> <span class="o">=</span> <span class="n">data</span><span class="p">[</span><span class="s">"email"</span><span class="p">]</span>
<span class="n">addresses</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">email_addresses</span><span class="p">)</span>
</code></pre></div></div>
<p>With each of these for loops, we’re running the same piece of code on each item in the list. While the repeated piece of code isn’t too complicated, you still need to write (or read!) multiple lines to understand what the code block is doing. <em>List comprehensions</em> are an elegant way to create lists from existing lists.</p>
<h2 id="list-comprehension">List Comprehension</h2>
<p>First, let’s look at a quick example:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">chars</span> <span class="o">=</span> <span class="p">[</span><span class="n">letter</span> <span class="k">for</span> <span class="n">letter</span> <span class="ow">in</span> <span class="s">"Hello, world!"</span><span class="p">]</span>
<span class="k">print</span><span class="p">(</span><span class="n">chars</span><span class="p">)</span>
</code></pre></div></div>
<p>When you run this Python code, the output will be:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>['H', 'e', 'l', 'l', 'o', ',', ' ', 'w', 'o', 'r', 'l', 'd', '!']
</code></pre></div></div>
<p>In this example, a new list is created, named <code class="language-plaintext highlighter-rouge">chars</code>, that contains each of the items in the string <code class="language-plaintext highlighter-rouge">"Hello, world!"</code>. Because items in a string are characters, <code class="language-plaintext highlighter-rouge">chars</code> is a list of every character in the string.</p>
<h3 id="list-comprehension-syntax">List Comprehension Syntax</h3>
<p>List comprehensions are written as:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">new_list</span> <span class="o">=</span> <span class="p">[</span><span class="n">expression</span> <span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">old_list</span><span class="p">]</span>
</code></pre></div></div>
<p>This is what a list comprehension “unwrapped” into a traditional list would look like:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">new_list</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">old_list</span><span class="p">:</span>
<span class="n">new_item</span> <span class="o">=</span> <span class="n">expression</span>
<span class="n">new_list</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">new_item</span><span class="p">)</span>
</code></pre></div></div>
<p>Used correctly, list comprehensions can reduce for loops into a more readable, one line expression.</p>
<p><img src="https://nickymarino.com/public/assets/2020/python-list-comprehension/pretty-syntax.png" alt="Syntax for List Comprehension" class="center" /></p>
<h3 id="examples">Examples</h3>
<p>Let’s re-write our earlier examples using list comprehensions. Instead of writing an whole new function, our <code class="language-plaintext highlighter-rouge">write_squares</code> example can be reduced to one line:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">squares</span> <span class="o">=</span> <span class="p">[</span><span class="n">number</span> <span class="o">**</span> <span class="mi">2</span> <span class="k">for</span> <span class="n">number</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">6</span><span class="p">)]</span>
<span class="k">print</span><span class="p">(</span><span class="n">squares</span><span class="p">)</span>
</code></pre></div></div>
<p>We get the same results when we run this code:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>0
1
4
9
16
25
</code></pre></div></div>
<p>Now let’s look at the line stripping function. We need to keep the first few lines to read the file contents, but the for loop that appended each stripped line to a new variable has been updated to use a list comprehension instead:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">lines_from</span><span class="p">(</span><span class="n">filename</span><span class="p">):</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="s">"r"</span><span class="p">)</span> <span class="k">as</span> <span class="n">file_obj</span><span class="p">:</span>
<span class="n">original_lines</span> <span class="o">=</span> <span class="n">file_obj</span><span class="p">.</span><span class="n">readlines</span><span class="p">()</span>
<span class="k">return</span> <span class="p">[</span><span class="n">line</span><span class="p">.</span><span class="n">strip</span><span class="p">()</span> <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">original_lines</span><span class="p">]</span>
</code></pre></div></div>
<p>The email address fetcher can be reduced to one line, and it can be directly placed in another block of code:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">email_addresses</span><span class="p">(</span><span class="n">ids</span><span class="p">):</span>
<span class="k">return</span> <span class="p">[</span><span class="n">api_request</span><span class="p">(</span><span class="nb">id</span><span class="p">)[</span><span class="s">"email"</span><span class="p">]</span> <span class="k">for</span> <span class="nb">id</span> <span class="ow">in</span> <span class="n">ids</span><span class="p">]</span>
</code></pre></div></div>
<h3 id="conditionals">Conditionals</h3>
<p>List comprehensions can also use conditional statements to filter or modify the values that are added to a new list. For example, if we wanted a list of only even integers from 1 to 20, we could add a conditional to the end of a list comprehension:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="n">evens</span> <span class="o">=</span> <span class="p">[</span><span class="n">num</span> <span class="k">for</span> <span class="n">num</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">20</span><span class="p">)</span> <span class="k">if</span> <span class="n">num</span> <span class="o">%</span> <span class="mi">2</span> <span class="o">==</span> <span class="mi">0</span><span class="p">]</span>
<span class="o">>>></span> <span class="k">print</span><span class="p">(</span><span class="n">evens</span><span class="p">)</span>
<span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="mi">8</span><span class="p">,</span> <span class="mi">10</span><span class="p">,</span> <span class="mi">12</span><span class="p">,</span> <span class="mi">14</span><span class="p">,</span> <span class="mi">16</span><span class="p">,</span> <span class="mi">18</span><span class="p">]</span>
</code></pre></div></div>
<p>List comprehensions can use any number of <code class="language-plaintext highlighter-rouge">and</code> and <code class="language-plaintext highlighter-rouge">or</code> operators in conditionals. For example, we can use the conditional <code class="language-plaintext highlighter-rouge">(num % 2 == 0) and (num % 3 == 0)</code> to keep only numbers that are divisible by both 2 and 3:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="n">my_nums</span> <span class="o">=</span> <span class="p">[</span><span class="n">num</span> <span class="k">for</span> <span class="n">num</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">20</span><span class="p">)</span> <span class="k">if</span> <span class="p">(</span><span class="n">num</span> <span class="o">%</span> <span class="mi">2</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span> <span class="ow">and</span> <span class="p">(</span><span class="n">num</span> <span class="o">%</span> <span class="mi">3</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)]</span>
<span class="o">>>></span> <span class="k">print</span><span class="p">(</span><span class="n">my_nums</span><span class="p">)</span>
<span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="mi">12</span><span class="p">,</span> <span class="mi">18</span><span class="p">]</span>
</code></pre></div></div>
<h2 id="key-points">Key Points</h2>
<p>List comprehension is an elegant way to create new lists from existing lists. List comprehensions can reduce multiple-line code blocks to just one line! However, avoid writing large list comprehensions, as that may reduce legibility for your code readers.</p>
<p>Once you’re comfortable with list comprehensions, I recommend learning <a href="https://www.python.org/dev/peps/pep-0274/">dictionary comprehensions</a>, which are very similar to list comprehensions except that they operate on dictionaries.</p>
Python's Walrus Operator2020-09-23T00:00:00+00:00https://nickymarino.com/2020/09/23/python-walrus-operator<blockquote>
<p>Beautiful is better than ugly.</p>
<p>— <a href="https://www.python.org/dev/peps/pep-0020/">The Zen of Python</a></p>
</blockquote>
<p>Python introduced a brand new way to assign values to variables in version 3.8.0. The new syntax is <code class="language-plaintext highlighter-rouge">:=</code>, and it’s called a “walrus operator” because it looks like a pair of eyes and a set of tusks. The walrus operator assigns values as part of a larger expression, and it can significantly increase legibility in many areas.</p>
<h2 id="named-expressions">Named Expressions</h2>
<p>You can create <em>named expressions</em> with the walrus operator. Named expressions have the format <code class="language-plaintext highlighter-rouge">NAME := expression</code>, such as <code class="language-plaintext highlighter-rouge">x := 34</code> or <code class="language-plaintext highlighter-rouge">numbers := list(range(10))</code>. Python code can use the <code class="language-plaintext highlighter-rouge">expression</code> to evaluate a larger expression (such as an <code class="language-plaintext highlighter-rouge">if</code> statement), and the variable <code class="language-plaintext highlighter-rouge">NAME</code> is assigned the value of the expression.</p>
<p>If you’ve written Swift code before, Python’s walrus operator is similar to Swift’s <a href="https://docs.swift.org/swift-book/LanguageGuide/OptionalChaining.html">Optional Chaining</a>. With optional chaining, you assign a value to a variable inside the conditional of an <code class="language-plaintext highlighter-rouge">if</code> statement. If the new variable’s value is not <code class="language-plaintext highlighter-rouge">nil</code> (like Python’s <code class="language-plaintext highlighter-rouge">None</code>), the <code class="language-plaintext highlighter-rouge">if</code> block is executed. If the variable’s value is <code class="language-plaintext highlighter-rouge">nil</code>, then the block is ignored:</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="nv">responseMessages</span> <span class="o">=</span> <span class="p">[</span>
<span class="mi">200</span><span class="p">:</span> <span class="s">"OK"</span><span class="p">,</span>
<span class="mi">403</span><span class="p">:</span> <span class="s">"Access forbidden"</span><span class="p">,</span>
<span class="mi">404</span><span class="p">:</span> <span class="s">"File not found"</span><span class="p">,</span>
<span class="mi">500</span><span class="p">:</span> <span class="s">"Internal server error"</span>
<span class="p">]</span>
<span class="k">let</span> <span class="nv">response</span> <span class="o">=</span> <span class="mi">444</span>
<span class="k">if</span> <span class="k">let</span> <span class="nv">message</span> <span class="o">=</span> <span class="n">responseMessages</span><span class="p">[</span><span class="n">response</span><span class="p">]</span> <span class="p">{</span>
<span class="c1">// This statement won't be run because message is nil</span>
<span class="nf">print</span><span class="p">(</span><span class="s">"Message: "</span> <span class="o">+</span> <span class="n">message</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<h2 id="benefits">Benefits</h2>
<p>There are a lot of benefits to using the walrus operator in your code. But don’t take my word for it! Here’s what the authors of the idea said in their proposal:</p>
<blockquote>
<p>Naming the result of an expression is an important part of programming, allowing a descriptive name to be used in place of a longer expression, and permitting reuse.</p>
<p>— <a href="https://www.python.org/dev/peps/pep-0572/#rationale">PEP 572 – Assignment Expressions</a></p>
</blockquote>
<p>Let’s take a look at some examples.</p>
<h3 id="dont-repeat-yourself">Don’t Repeat Yourself</h3>
<p>With the walrus operator, you can more easily stick to the <a href="https://en.wikipedia.org/wiki/Don%27t_repeat_yourself">DRY principle</a> and reduce how often you repeat yourself in code. For example, if you want to print an error message if a list is too long, you might accidentally get the length of the list twice:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">my_long_list</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="mi">1000</span><span class="p">))</span>
<span class="c1"># You get the length twice!
</span><span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">my_long_list</span><span class="p">)</span> <span class="o">></span> <span class="mi">10</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">"List is too long to consume (length=</span><span class="si">{</span><span class="nb">len</span><span class="p">(</span><span class="n">my_long_list</span><span class="p">)</span><span class="si">}</span><span class="s">, max=10)"</span><span class="p">)</span>
</code></pre></div></div>
<p>Let’s use the walrus operator to only find the length of the list once <em>and</em> keep that length inside the scope of the <code class="language-plaintext highlighter-rouge">if</code> statement:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">my_long_list</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="mi">1000</span><span class="p">))</span>
<span class="c1"># Much better :)
</span><span class="k">if</span> <span class="p">(</span><span class="n">count</span> <span class="p">:</span><span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">my_long_list</span><span class="p">))</span> <span class="o">></span> <span class="mi">10</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">"List is too long to consume (length=</span><span class="si">{</span><span class="n">count</span><span class="si">}</span><span class="s">, max=10)"</span><span class="p">)</span>
</code></pre></div></div>
<p>In the code block above, <code class="language-plaintext highlighter-rouge">count := len(my_long_list)</code> assigns the value <code class="language-plaintext highlighter-rouge">1000</code> to <code class="language-plaintext highlighter-rouge">count</code>. Then, the <code class="language-plaintext highlighter-rouge">if</code> statement is evaluated as <code class="language-plaintext highlighter-rouge">if len(my_long_list) > 10</code>. The walrus operator has two benefits here:</p>
<ol>
<li>We don’t calculate the length of a (possibly large) list more than once</li>
<li>We clearly show a reader of our program that we’re going to use the <code class="language-plaintext highlighter-rouge">count</code> variable inside the scope of the <code class="language-plaintext highlighter-rouge">if</code> statement.</li>
</ol>
<h3 id="reuse-variables">Reuse Variables</h3>
<p>Another common example is using Python’s regular expression library, <code class="language-plaintext highlighter-rouge">re</code>. We want to look at a list of phone numbers and print their area codes if they have one. With a walrus operator, we can check whether the area code exists and assign it to a variable with one line:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">re</span>
<span class="n">phone_numbers</span> <span class="o">=</span> <span class="p">[</span>
<span class="s">"(317) 555-5555"</span><span class="p">,</span>
<span class="s">"431-2973"</span><span class="p">,</span>
<span class="s">"(111) 222-3344"</span><span class="p">,</span>
<span class="s">"(710) 982-3811"</span><span class="p">,</span>
<span class="s">"290-2918"</span><span class="p">,</span>
<span class="s">"711-7712"</span><span class="p">,</span>
<span class="p">]</span>
<span class="k">for</span> <span class="n">number</span> <span class="ow">in</span> <span class="n">phone_numbers</span><span class="p">:</span>
<span class="c1"># The regular expression "\(([0-9]{3})\)" checks for a substring
</span> <span class="c1"># with the pattern "(###)", where # is a 0-9 digit
</span> <span class="k">if</span> <span class="n">match</span> <span class="p">:</span><span class="o">=</span> <span class="n">re</span><span class="p">.</span><span class="n">match</span><span class="p">(</span><span class="s">"\(([0-9]{3})\)"</span><span class="p">,</span> <span class="n">number</span><span class="p">):</span>
<span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">"Area code: </span><span class="si">{</span><span class="n">match</span><span class="p">.</span><span class="n">group</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="si">}</span><span class="s">"</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="s">"No area code"</span><span class="p">)</span>
</code></pre></div></div>
<h3 id="legible-code-blocks">Legible Code Blocks</h3>
<p>A common programming pattern is performing an action, assigning the result to a variable, and then checking the result:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">result</span> <span class="o">=</span> <span class="n">parse_field_from</span><span class="p">(</span><span class="n">my_data</span><span class="p">)</span>
<span class="k">if</span> <span class="n">result</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="s">"Success"</span><span class="p">)</span>
</code></pre></div></div>
<p>In many cases, these types of blocks can be cleaned up with a walrus operator to become one indented code block:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="n">result</span> <span class="p">:</span><span class="o">=</span> <span class="n">parse_field_from</span><span class="p">(</span><span class="n">my_data</span><span class="p">):</span>
<span class="k">print</span><span class="p">(</span><span class="s">"Success"</span><span class="p">)</span>
</code></pre></div></div>
<p>These blocks can be chained together to convert a nested check statements into one line of if/elif/else statements. For example, let’s look at some students in a dictionary. We need to print each student’s graduation date if it exists, or their student id if available:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">sample_data</span> <span class="o">=</span> <span class="p">[</span>
<span class="p">{</span><span class="s">"student_id"</span><span class="p">:</span> <span class="mi">200</span><span class="p">,</span> <span class="s">"name"</span><span class="p">:</span> <span class="s">"Sally West"</span><span class="p">,</span> <span class="s">"graduation_date"</span><span class="p">:</span> <span class="s">"2019-05-01"</span><span class="p">},</span>
<span class="p">{</span><span class="s">"student_id"</span><span class="p">:</span> <span class="mi">404</span><span class="p">,</span> <span class="s">"name"</span><span class="p">:</span> <span class="s">"Zahara Durham"</span><span class="p">,</span> <span class="s">"graduation_date"</span><span class="p">:</span> <span class="bp">None</span><span class="p">},</span>
<span class="p">{</span><span class="s">"student_id"</span><span class="p">:</span> <span class="mi">555</span><span class="p">,</span> <span class="s">"name"</span><span class="p">:</span> <span class="s">"Connie Coles"</span><span class="p">,</span> <span class="s">"graduation_date"</span><span class="p">:</span> <span class="s">"2020-01-15"</span><span class="p">},</span>
<span class="p">{</span><span class="s">"student_id"</span><span class="p">:</span> <span class="bp">None</span><span class="p">,</span> <span class="s">"name"</span><span class="p">:</span> <span class="s">"Jared Hampton"</span><span class="p">,</span> <span class="s">"graduation_date"</span><span class="p">:</span> <span class="bp">None</span><span class="p">},</span>
<span class="p">]</span>
<span class="k">for</span> <span class="n">student</span> <span class="ow">in</span> <span class="n">sample_data</span><span class="p">:</span>
<span class="n">graduation_date</span> <span class="o">=</span> <span class="n">student</span><span class="p">[</span><span class="s">"graduation_date"</span><span class="p">]</span>
<span class="k">if</span> <span class="n">graduation_date</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">'</span><span class="si">{</span><span class="n">student</span><span class="p">[</span><span class="s">"name"</span><span class="p">]</span><span class="si">}</span><span class="s"> graduated on </span><span class="si">{</span><span class="n">graduation_date</span><span class="si">}</span><span class="s">'</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="c1"># This nesting can be confusing!
</span> <span class="n">student_id</span> <span class="o">=</span> <span class="n">student</span><span class="p">[</span><span class="s">"student_id"</span><span class="p">]</span>
<span class="k">if</span> <span class="n">student_id</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">'</span><span class="si">{</span><span class="n">student</span><span class="p">[</span><span class="s">"name"</span><span class="p">]</span><span class="si">}</span><span class="s"> is currently enrolled with ID </span><span class="si">{</span><span class="n">student_id</span><span class="si">}</span><span class="s">'</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">'</span><span class="si">{</span><span class="n">student</span><span class="p">[</span><span class="s">"name"</span><span class="p">]</span><span class="si">}</span><span class="s"> has no data")
</span></code></pre></div></div>
<p>With walrus operators, we can put the graduation date and student id checks next to each other, and better show that we’re checking for one or the other for each student:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">sample_data</span> <span class="o">=</span> <span class="p">[</span>
<span class="p">{</span><span class="s">"student_id"</span><span class="p">:</span> <span class="mi">200</span><span class="p">,</span> <span class="s">"name"</span><span class="p">:</span> <span class="s">"Sally West"</span><span class="p">,</span> <span class="s">"graduation_date"</span><span class="p">:</span> <span class="s">"2019-05-01"</span><span class="p">},</span>
<span class="p">{</span><span class="s">"student_id"</span><span class="p">:</span> <span class="mi">404</span><span class="p">,</span> <span class="s">"name"</span><span class="p">:</span> <span class="s">"Zahara Durham"</span><span class="p">,</span> <span class="s">"graduation_date"</span><span class="p">:</span> <span class="bp">None</span><span class="p">},</span>
<span class="p">{</span><span class="s">"student_id"</span><span class="p">:</span> <span class="mi">555</span><span class="p">,</span> <span class="s">"name"</span><span class="p">:</span> <span class="s">"Connie Coles"</span><span class="p">,</span> <span class="s">"graduation_date"</span><span class="p">:</span> <span class="s">"2020-01-15"</span><span class="p">},</span>
<span class="p">{</span><span class="s">"student_id"</span><span class="p">:</span> <span class="bp">None</span><span class="p">,</span> <span class="s">"name"</span><span class="p">:</span> <span class="s">"Jared Hampton"</span><span class="p">,</span> <span class="s">"graduation_date"</span><span class="p">:</span> <span class="bp">None</span><span class="p">},</span>
<span class="p">]</span>
<span class="k">for</span> <span class="n">student</span> <span class="ow">in</span> <span class="n">sample_data</span><span class="p">:</span>
<span class="c1"># Much cleaner
</span> <span class="k">if</span> <span class="n">graduation_date</span> <span class="p">:</span><span class="o">=</span> <span class="n">student</span><span class="p">[</span><span class="s">"graduation_date"</span><span class="p">]:</span>
<span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">'</span><span class="si">{</span><span class="n">student</span><span class="p">[</span><span class="s">"name"</span><span class="p">]</span><span class="si">}</span><span class="s"> graduated on </span><span class="si">{</span><span class="n">graduation_date</span><span class="si">}</span><span class="s">'</span><span class="p">)</span>
<span class="k">elif</span> <span class="n">student_id</span> <span class="p">:</span><span class="o">=</span> <span class="n">student</span><span class="p">[</span><span class="s">"student_id"</span><span class="p">]:</span>
<span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">'</span><span class="si">{</span><span class="n">student</span><span class="p">[</span><span class="s">"name"</span><span class="p">]</span><span class="si">}</span><span class="s"> is currently enrolled with ID </span><span class="si">{</span><span class="n">student_id</span><span class="si">}</span><span class="s">'</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">'</span><span class="si">{</span><span class="n">student</span><span class="p">[</span><span class="s">"name"</span><span class="p">]</span><span class="si">}</span><span class="s"> has no data")
</span></code></pre></div></div>
<h2 id="wrap-up">Wrap Up</h2>
<p>With walrus operators and named expressions, we can dramatically increase the legibility of our code by simplifying statements, reusing variables, and reducing indentation. For more great examples, check out the <a href="https://www.python.org/dev/peps/pep-0572/#examples">original proposal</a> and the Python 3.8 <a href="https://docs.python.org/3/whatsnew/3.8.html#what-s-new-in-python-3-8">release notes</a>.</p>
AWS CLI: Multiple Named Profiles2020-09-06T00:00:00+00:00https://nickymarino.com/2020/09/06/multiple-aws-profiles<p>The AWS CLI supports named profiles so that you can quickly switch between different AWS instances, accounts, and credential sets. Let’s assume you have two AWS accounts, each with an access key id and a secret access key. The first account is your default profile, and the second account is used less often.</p>
<h2 id="adding-a-named-profile">Adding a Named Profile</h2>
<p>First, open <code class="language-plaintext highlighter-rouge">~/.aws/credentials</code> (on Linux & Mac) or <code class="language-plaintext highlighter-rouge">%USERPROFILE%\.aws\credentials</code> (on Windows) and add your credentials:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[default]
aws_access_key_id=AKIAIOSFODNN7EXAMPLE1
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY1
[user2]
aws_access_key_id=AKIAI44QH8DHBEXAMPLE2
aws_secret_access_key=je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY2
</code></pre></div></div>
<p>If your two profiles use different regions, or output formats, you can specify them in <code class="language-plaintext highlighter-rouge">~/.aws/config</code> (on Linux & Mac) or <code class="language-plaintext highlighter-rouge">%USERPROFILE%\.aws\config</code> (on Windows):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[default]
region=us-west-2
output=json
[profile user2]
region=us-east-1
output=text
</code></pre></div></div>
<p>Note: do <strong>not</strong> add <code class="language-plaintext highlighter-rouge">profile</code> in front of the profile names in the <code class="language-plaintext highlighter-rouge">credentials</code> file, like we do above in the <code class="language-plaintext highlighter-rouge">config</code> file.</p>
<p>Most AWS CLI commands support the named profile option <code class="language-plaintext highlighter-rouge">--profile</code>. For example, verify that both of your accounts are set up properly with <code class="language-plaintext highlighter-rouge">sts get-caller-identify</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># Verify your default identity
$ aws sts get-caller-identity
# Verify your second identity
$ aws sts get-caller-identity --profile user2
</code></pre></div></div>
<p>EKS and EC2 commands also support the <code class="language-plaintext highlighter-rouge">--profile</code> option. For example, let’s list our EC2 instances for the <code class="language-plaintext highlighter-rouge">user2</code> account:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ aws ec2 describe-instances --profile user2
</code></pre></div></div>
<h2 id="setting-a-profile-for-kubeconfig">Setting a Profile for Kubeconfig</h2>
<p>The AWS CLI <code class="language-plaintext highlighter-rouge">--profile</code> option can be used to add new clusters to your <code class="language-plaintext highlighter-rouge">~/.kubeconfig</code>. By adding named profiles, you can switch between Kubernetes contexts without needing to export new AWS environment variables.</p>
<p>If your EKS instance is authenticated with only your AWS access key id and access key secret, add your cluster with <code class="language-plaintext highlighter-rouge">eks update-kubeconfig</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ aws eks update-kubeconfig --name EKS_CLUSTER_NAME --profile PROFILE
</code></pre></div></div>
<p>If your EKS instance uses an IAM Role ARN for authentication, first copy the role ARN from the AWS Console: Go to the EKS service page, then Clusters, then select your cluster name, and find the IAM Role ARN at the bottom of the page. The format of the role ARN is typically <code class="language-plaintext highlighter-rouge">arn:aws:iam::XXXXXXXXXXXX:role/role_name</code>. Then, use <code class="language-plaintext highlighter-rouge">eks update-kubeconfig</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>aws eks update-kubeconfig --name EKS_CLUSTER_NAME --role-arn ROLE_ARN --profile PROFILE
</code></pre></div></div>
<p>To verify that your <code class="language-plaintext highlighter-rouge">kubeconfig</code> is set properly, use <a href="https://github.com/ahmetb/kubectx">kubectx</a> to switch to one of your new clusters and try to list out its services:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ kubectx EKS_CLUSTER_NAME
Switched to context "EKS_CLUSTER_NAME".
$ kubectl get services
...
</code></pre></div></div>
How to Use Jekyll on macOS Catalina with RVM2020-06-13T00:00:00+00:00https://nickymarino.com/2020/06/13/jekyll-server-rvm-macos<p>Apple bundles a system version of the Ruby programming language on macOS. Because system Ruby is used by the inner workings of the operating system, this version is not meant to be upgraded or modified by a user. With the Ruby Version Manager <a href="https://rvm.io/">RVM</a>, you can install an additional Ruby version for personal use.</p>
<p>Similar to <a href="https://github.com/pyenv/pyenv">pyenv</a>, you can install multiple versions of Ruby with RVM and change the version you’re using on the fly. You can also install gems without <code class="language-plaintext highlighter-rouge">sudo</code>.</p>
<h2 id="installing-rvm-and-ruby">Installing RVM and Ruby</h2>
<p>Before downloading RVM, first install <a href="http://en.wikipedia.org/wiki/GNU_Privacy_Guard">gpg</a> and the mpapis public key:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ brew install gnupg
$ gpg --keyserver hkp://ipv4.pool.sks-keyservers.net --recv-keys xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
</code></pre></div></div>
<p>The keys (<code class="language-plaintext highlighter-rouge">xxxx...</code>) change often, so you will need to copy the most recent ones from the <a href="https://rvm.io/rvm/install#install-gpg-keys">RVM install page</a>.</p>
<p>Next, download the most recent stable version of RVM:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ \curl -sSL https://get.rvm.io | bash -s stable --ruby
</code></pre></div></div>
<p>After installation, RVM will tell you to either open a new terminal or source <code class="language-plaintext highlighter-rouge">rvm</code>, so run the command it prints:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ source ~/.rvm/scripts/rvm
</code></pre></div></div>
<p>You will also want to add <code class="language-plaintext highlighter-rouge">rvm</code> to your <code class="language-plaintext highlighter-rouge">~/.zshrc</code> or <code class="language-plaintext highlighter-rouge">~/.bashrc</code> to load when you open a terminal:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># Add this to your ~/.zshrc or ~/.bashrc
[[ -s "$HOME/.rvm/scripts/rvm" ]] && . "$HOME/.rvm/scripts/rvm"
</code></pre></div></div>
<p>Use <code class="language-plaintext highlighter-rouge">rvm list</code> to find a Ruby version you want to install, then tell RVM which version to use:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ rvm list
$ rvm use 2.7.0
</code></pre></div></div>
<p>You can then verify that you’re using an RVM-managed version of Ruby:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ which ruby
~/.rvm/rubies/ruby-2.7.0/bin/ruby
</code></pre></div></div>
<h2 id="installing-jekyll">Installing Jekyll</h2>
<p>First, verify you’re using a Ruby version managed by RVM in the above step. Then, install the <a href="https://jekyllrb.com">Jekyll</a> gem:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ gem install jekyll bundler
</code></pre></div></div>
<p>If you’re already in a Jekyll website repo (or any folder with a <code class="language-plaintext highlighter-rouge">Rakefile</code>), you can use <code class="language-plaintext highlighter-rouge">bundle</code> to install your remaining requirements:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ bundle install
</code></pre></div></div>
<p>You may then need to update Jekyll for your <code class="language-plaintext highlighter-rouge">Rakefile</code> requirements:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ bundle update jekyll
</code></pre></div></div>
<p>Now you can up the Jekyll server:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ bundle exec jekyll serve
</code></pre></div></div>
<p>Now check out your site at <code class="language-plaintext highlighter-rouge">http://localhost:4000</code>! See the <a href="https://jekyllrb.com/docs/">Jekyll Quickstart</a> for more details on starting a Jekyll blog.</p>
Optimizing Virgo Using NArray2020-01-21T00:00:00+00:00https://nickymarino.com/2020/01/21/optimizing-virgo<p><a href="/2020/01/19/virgo-wallpaper-generator/">Earlier this week</a>, I had released <a href="https://github.com/nickymarino/virgo">Virgo</a>, a Ruby CLI to generate wallpapers (including the one above). My goal was to be able to create beautiful OLED wallpapers for my phone, but unfortunately, the first version of Virgo would take about 15 seconds to generate a wallpaper the size of an iPhone 11. The first version of Virgo used <a href="https://www.rubydoc.info/github/wvanbergen/chunky_png/ChunkyPNG/Image"><code class="language-plaintext highlighter-rouge">ChunkyPNG::Image</code></a> to place pixels on the background, and the author of ChunkyPNG alludes to this possible problem <a href="https://github.com/wvanbergen/chunky_png">in his README</a>:</p>
<blockquote>
<p>Also, have a look at <a href="http://github.com/wvanbergen/oily_png">OilyPNG</a> which is a mixin module that implements some of the ChunkyPNG algorithms in C, which provides a massive speed boost to encoding and decoding.</p>
</blockquote>
<h2 id="improvement-goal">Improvement Goal</h2>
<p>My goal was to reduce the time to generate an iPhone-sized wallpaper (about 1200 by 2200 pixels) in under one second. I chose this constraint so that I can eventually write a web frontend in <a href="http://sinatrarb.com">Sinatra</a>. Users won’t wait 15 seconds for an image to be generated, especially if the file is being provided by a server without any loading indication on the page.</p>
<h2 id="thinking-of-a-solution">Thinking of a Solution</h2>
<p>When thinking of optimization ideas, two options stood out to me from Willem’s <a href="https://github.com/wvanbergen/chunky_png">README</a>:</p>
<ol>
<li>Can I adapt Virgo to use OilyPNG instead of ChunkyPNG?</li>
<li>Is there another library that implements array manipulation in C?</li>
</ol>
<p>After some research into OilyPNG, I found that the functions I used with ChunkyPNG weren’t implemented in OilyPNG<sup id="fnref:not-really-implemented" role="doc-noteref"><a href="#fn:not-really-implemented" class="footnote" rel="footnote">1</a></sup>, so I was left to find another library that enabled a faster manipulation of integer arrays. I knew that in the Python world, <a href="https://numpy.org">NumPy</a> would be the immediate answer. After some research for a Ruby alternative to NumPy, I came across <a href="https://github.com/masa16/narray">NArray</a>, which <a href="https://stackoverflow.com/questions/5653994/ruby-equivalent-of-numpy">appeared</a> to be a solution <a href="https://dev.to/kojix2/narray-ruby-equivalent-of-numpy-30ji">others</a> had relied on in the past. I alluded to this possible approach in my <a href="/2020/01/19/virgo-wallpaper-generator/">original post</a>:</p>
<blockquote>
<p><a href="https://chunkypng.com">ChunkyPNG</a> was used to manage and save the wallpaper images, and <a href="https://github.com/commander-rb/commander">Commander</a> enabled terminal argument parsing and help documentation. For optimization, I plan to use <a href="https://masa16.github.io/narray/">NArray</a> for creating the pixel locations in the wallpaper and writing a custom PNG encoder.</p>
</blockquote>
<h1 id="fast-pixel-placement">Fast Pixel Placement</h1>
<p>After some profiling, the section of code that could be improved the most was overlaying pixels on the background image:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">place_pixel</span>
<span class="c1"># Create a new canvas</span>
<span class="n">pixel</span> <span class="o">=</span> <span class="no">Image</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="vi">@pixel_diameter</span><span class="p">,</span> <span class="vi">@pixel_diameter</span><span class="p">,</span> <span class="vi">@theme</span><span class="p">.</span><span class="nf">foreground</span><span class="p">)</span>
<span class="c1"># Replace the old image with the new canvas at the pixel coordinate</span>
<span class="vi">@image</span> <span class="o">=</span> <span class="vi">@image</span><span class="p">.</span><span class="nf">replace</span><span class="p">(</span><span class="n">pixel</span><span class="p">,</span> <span class="n">pixel_coordinate</span><span class="p">.</span><span class="nf">x</span><span class="p">,</span> <span class="n">pixel_coordinate</span><span class="p">.</span><span class="nf">y</span><span class="p">)</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Rather than creating new <code class="language-plaintext highlighter-rouge">ChunkyPNG::Images</code> and overlaying them on a master <code class="language-plaintext highlighter-rouge">Image</code>, I decided to use a new data structure for the image. Instead, Virgo now uses an <code class="language-plaintext highlighter-rouge">NArray</code> to represent the image, where each item is the <code class="language-plaintext highlighter-rouge">Integer</code> representation of the pixel <code class="language-plaintext highlighter-rouge">ChunkyPNG::Color</code>. For every pixel, a portion of the array is replaced with the integer color:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">create_map</span>
<span class="c1"># Start with each pixel in the image as the background color</span>
<span class="n">map</span> <span class="o">=</span> <span class="no">NArray</span><span class="p">.</span><span class="nf">int</span><span class="p">(</span><span class="vi">@width</span><span class="p">,</span> <span class="vi">@height</span><span class="p">).</span><span class="nf">fill!</span><span class="p">(</span><span class="vi">@theme</span><span class="p">.</span><span class="nf">background</span><span class="p">)</span>
<span class="c1"># Place each pixel in the map</span>
<span class="n">count</span> <span class="o">=</span> <span class="n">number_pixels_to_place</span>
<span class="p">(</span><span class="mi">1</span><span class="o">..</span><span class="n">count</span><span class="p">).</span><span class="nf">each</span> <span class="k">do</span>
<span class="c1"># Determine pixel location</span>
<span class="n">x</span> <span class="o">=</span> <span class="vi">@x_distribution</span><span class="p">.</span><span class="nf">random_point</span>
<span class="n">x_max</span> <span class="o">=</span> <span class="n">x</span> <span class="o">+</span> <span class="vi">@pixel_diameter</span>
<span class="n">y</span> <span class="o">=</span> <span class="vi">@y_distribution</span><span class="p">.</span><span class="nf">random_point</span>
<span class="n">y_max</span> <span class="o">=</span> <span class="n">y</span> <span class="o">+</span> <span class="vi">@pixel_diameter</span>
<span class="c1"># Replace the pixel in the map with a new (random) color</span>
<span class="n">map</span><span class="p">[</span><span class="n">x</span><span class="o">..</span><span class="n">x_max</span><span class="p">,</span> <span class="n">y</span><span class="o">..</span><span class="n">y_max</span><span class="p">]</span> <span class="o">=</span> <span class="vi">@theme</span><span class="p">.</span><span class="nf">random_foreground</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Then, to create save a <code class="language-plaintext highlighter-rouge">Wallpaper</code> instance, a <code class="language-plaintext highlighter-rouge">ChunkyPNG::Image</code> is created by inserting rows of the <code class="language-plaintext highlighter-rouge">NArray</code> into the <code class="language-plaintext highlighter-rouge">Image</code>:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">image</span>
<span class="n">img</span> <span class="o">=</span> <span class="no">Image</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="vi">@width</span><span class="p">,</span> <span class="vi">@height</span><span class="p">,</span> <span class="no">Color</span><span class="o">::</span><span class="no">TRANSPARENT</span><span class="p">)</span>
<span class="c1"># Put each row of @map into the image</span>
<span class="n">num_rows</span> <span class="o">=</span> <span class="vi">@map</span><span class="p">.</span><span class="nf">shape</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="p">(</span><span class="mi">0</span><span class="o">...</span><span class="n">num_rows</span><span class="p">).</span><span class="nf">each</span> <span class="k">do</span> <span class="o">|</span><span class="n">row_idx</span><span class="o">|</span>
<span class="c1"># Replace the row in the image with the new colors</span>
<span class="n">img</span><span class="p">.</span><span class="nf">replace_row!</span><span class="p">(</span><span class="n">row_idx</span><span class="p">,</span> <span class="vi">@map</span><span class="p">[</span><span class="kp">true</span><span class="p">,</span> <span class="n">row_idx</span><span class="p">])</span>
<span class="k">end</span>
<span class="n">img</span>
<span class="k">end</span>
</code></pre></div></div>
<h2 id="int-vs-integer"><code class="language-plaintext highlighter-rouge">int</code> vs <code class="language-plaintext highlighter-rouge">Integer</code></h2>
<p>There was only one problem with this solution: the range of values for <code class="language-plaintext highlighter-rouge">ChunkyPNG::Image</code> would often exceed the range of the <code class="language-plaintext highlighter-rouge">int</code> type used by <code class="language-plaintext highlighter-rouge">NArray</code>. Therefore, most of the colors in the predefined themes could not be placed into the color map as-is.</p>
<p>I decided to implement a color hash (enum) for a <code class="language-plaintext highlighter-rouge">Theme</code> instance, where every key is a unique (low <code class="language-plaintext highlighter-rouge">Integer</code> value identifier), and each value is the (large <code class="language-plaintext highlighter-rouge">Integer</code>) <code class="language-plaintext highlighter-rouge">ChunkyPNG::Image</code> value. The background color (<code class="language-plaintext highlighter-rouge">@background</code>) will always have the identifier <code class="language-plaintext highlighter-rouge">0</code>, and the foreground colors (<code class="language-plaintext highlighter-rouge">@foregrounds</code>) have identifiers from <code class="language-plaintext highlighter-rouge">1</code> to <code class="language-plaintext highlighter-rouge">n</code>. The color hash is created in <code class="language-plaintext highlighter-rouge">Theme.initialize</code>:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="n">background</span> <span class="o">=</span> <span class="no">BACKGROUNDS</span><span class="p">[</span><span class="ss">:black</span><span class="p">],</span>
<span class="n">foregrounds</span> <span class="o">=</span> <span class="no">FOREGROUNDS</span><span class="p">[</span><span class="ss">:ruby</span><span class="p">])</span>
<span class="vi">@background</span> <span class="o">=</span> <span class="no">Color</span><span class="p">.</span><span class="nf">from_hex</span><span class="p">(</span><span class="n">background</span><span class="p">)</span>
<span class="vi">@foregrounds</span> <span class="o">=</span> <span class="n">foregrounds</span><span class="p">.</span><span class="nf">map</span> <span class="p">{</span> <span class="o">|</span><span class="n">x</span><span class="o">|</span> <span class="no">Color</span><span class="p">.</span><span class="nf">from_hex</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="p">}</span>
<span class="c1"># Because NArray can't handle the size of some ChunkyPNG::Color</span>
<span class="c1"># values, create a Hash of the background and foreground colors,</span>
<span class="c1"># where the key of the hash is an Integer, and the</span>
<span class="c1"># value is the value of the color</span>
<span class="c1"># Background has a key of 0, foregrounds have keys from 1..n</span>
<span class="n">colors</span> <span class="o">=</span> <span class="p">[</span><span class="vi">@background</span><span class="p">]</span> <span class="o">+</span> <span class="vi">@foregrounds</span>
<span class="n">colors_with_indices</span> <span class="o">=</span> <span class="n">colors</span><span class="p">.</span><span class="nf">each_with_index</span><span class="p">.</span><span class="nf">map</span> <span class="k">do</span> <span class="o">|</span><span class="n">color</span><span class="p">,</span> <span class="n">idx</span><span class="o">|</span>
<span class="p">[</span><span class="n">idx</span><span class="p">,</span> <span class="n">color</span><span class="p">]</span>
<span class="k">end</span>
<span class="vi">@color_hash</span> <span class="o">=</span> <span class="no">Hash</span><span class="p">[</span><span class="n">colors_with_indices</span><span class="p">]</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Let’s look at an example:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>2.6.3 :001 > # Construct a theme using predefined color names
2.6.3 :002 > t = Theme.from_syms(:black, :ruby)
=> #<Theme:0x00007f861f8c2128
@background=255,
@foregrounds=[
2367954943,
2720605439,
3073321215,
3425971711,
3561307903,
3646510079,
3731712255],
@color_hash={
0=>255,
1=>2367954943,
2=>2720605439,
3=>3073321215,
4=>3425971711,
5=>3561307903,
6=>3646510079,
7=>3731712255}>
</code></pre></div></div>
<p>Note that the <code class="language-plaintext highlighter-rouge">@background</code> color <code class="language-plaintext highlighter-rouge">255</code> is in <code class="language-plaintext highlighter-rouge">@color_hash</code> as <code class="language-plaintext highlighter-rouge">0=>255</code>, which means it has an identifier of <code class="language-plaintext highlighter-rouge">0</code>. The second color in <code class="language-plaintext highlighter-rouge">@foregrounds</code>, <code class="language-plaintext highlighter-rouge">2720605439</code>, is in <code class="language-plaintext highlighter-rouge">@color_hash</code> as <code class="language-plaintext highlighter-rouge">2=>2720605439</code>, meaning that color has an identifier of <code class="language-plaintext highlighter-rouge">2</code>.</p>
<p>Therefore, if the <code class="language-plaintext highlighter-rouge">Wallpaper</code> map has the following values:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[ [0, 0, 0],
[0, 2, 0],
[0, 0, 0] ]
</code></pre></div></div>
<p>Then the middle pixel is red (a value of <code class="language-plaintext highlighter-rouge">2720605439</code>), and the border pixels are black (a value of <code class="language-plaintext highlighter-rouge">255</code>).</p>
<p>Next, we add a few helper functions to <code class="language-plaintext highlighter-rouge">Theme</code> for retrieving a random foreground (pixel) color and color keys/values:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Returns the key for the background color</span>
<span class="k">def</span> <span class="nf">background_key</span>
<span class="c1"># The background always has a key of 0</span>
<span class="mi">0</span>
<span class="k">end</span>
<span class="c1"># Returns a random @color_hash foreground key</span>
<span class="k">def</span> <span class="nf">random_foreground_key</span>
<span class="c1"># (Slightly) speed up getting a foreground by returning the first</span>
<span class="c1"># item if only one exists</span>
<span class="n">color</span> <span class="o">=</span> <span class="k">if</span> <span class="vi">@foregrounds</span><span class="p">.</span><span class="nf">length</span> <span class="o">==</span> <span class="mi">1</span>
<span class="vi">@foregrounds</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="k">else</span>
<span class="vi">@foregrounds</span><span class="p">.</span><span class="nf">sample</span>
<span class="k">end</span>
<span class="n">key_from_color</span><span class="p">(</span><span class="n">color</span><span class="p">)</span>
<span class="k">end</span>
<span class="c1"># Returns the ChunkyPNG::Color value for a color key</span>
<span class="k">def</span> <span class="nf">color_from_key</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
<span class="vi">@color_hash</span><span class="p">[</span><span class="n">key</span><span class="p">]</span>
<span class="k">end</span>
<span class="c1"># Returns the key (in @color_hash) for a color value</span>
<span class="k">def</span> <span class="nf">key_from_color</span><span class="p">(</span><span class="n">color</span><span class="p">)</span>
<span class="vi">@color_hash</span><span class="p">.</span><span class="nf">key</span><span class="p">(</span><span class="n">color</span><span class="p">)</span>
<span class="k">end</span>
</code></pre></div></div>
<p>And <code class="language-plaintext highlighter-rouge">create_map</code> is updated to use the color keys rather than the large values:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">create_map</span>
<span class="c1"># Start with each pixel in the image as the background color</span>
<span class="n">map</span> <span class="o">=</span> <span class="no">NArray</span><span class="p">.</span><span class="nf">int</span><span class="p">(</span><span class="vi">@width</span><span class="p">,</span> <span class="vi">@height</span><span class="p">).</span><span class="nf">fill!</span><span class="p">(</span><span class="vi">@theme</span><span class="p">.</span><span class="nf">background_key</span><span class="p">)</span>
<span class="c1"># Place each pixel in the map</span>
<span class="n">count</span> <span class="o">=</span> <span class="n">number_pixels_to_place</span>
<span class="p">(</span><span class="mi">1</span><span class="o">..</span><span class="n">count</span><span class="p">).</span><span class="nf">each</span> <span class="k">do</span>
<span class="c1"># Determine pixel location</span>
<span class="n">x</span> <span class="o">=</span> <span class="vi">@x_distribution</span><span class="p">.</span><span class="nf">random_point</span>
<span class="n">x_max</span> <span class="o">=</span> <span class="n">x</span> <span class="o">+</span> <span class="vi">@pixel_diameter</span>
<span class="n">y</span> <span class="o">=</span> <span class="vi">@y_distribution</span><span class="p">.</span><span class="nf">random_point</span>
<span class="n">y_max</span> <span class="o">=</span> <span class="n">y</span> <span class="o">+</span> <span class="vi">@pixel_diameter</span>
<span class="c1"># Replace the pixel in the map with a new (random) color</span>
<span class="n">map</span><span class="p">[</span><span class="n">x</span><span class="o">..</span><span class="n">x_max</span><span class="p">,</span> <span class="n">y</span><span class="o">..</span><span class="n">y_max</span><span class="p">]</span> <span class="o">=</span> <span class="vi">@theme</span><span class="p">.</span><span class="nf">random_foreground_key</span>
<span class="k">end</span>
<span class="n">map</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Now we can test whether using <code class="language-plaintext highlighter-rouge">NArray</code> vs <code class="language-plaintext highlighter-rouge">ChunkyPNG::Image</code> reduced the time to generate large wallpapers.</p>
<h2 id="results">Results</h2>
<p>Using <code class="language-plaintext highlighter-rouge">NArray</code> <em>significantly</em> improved Virgo’s speed:</p>
<p><img src="https://nickymarino.com/public/assets/2020/optimizing-virgo/speed-chart.png" alt="Virgo Example Output" class="center" /></p>
<p>Note the exponential time complexity of the original implementation, and the linear time complexity.<sup id="fnref:linear-time" role="doc-noteref"><a href="#fn:linear-time" class="footnote" rel="footnote">2</a></sup> With 10 trials, <code class="language-plaintext highlighter-rouge">NArray</code> Virgo took 0.5 seconds on average to generate a 1000x1000 wallpaper, while the original implementation of Virgo takes 15.1 seconds on average. <strong>That’s a a 30x improvement!</strong> Even better, these updates will allow me to make a responsive Sinatra web frontend, without worrying about user retention or delays.</p>
<h2 id="future-improvements">Future Improvements</h2>
<p>I believe more improvements could be made to Virgo, especially since I plan on writing a web frontend. In particular, I’ve added a <code class="language-plaintext highlighter-rouge">Distribution</code> class to enable both normal and uniform pixel distributions, but I have not added it as a feature to the CLI. Further, there is a new version of <code class="language-plaintext highlighter-rouge">NArray</code> called <a href="https://github.com/ruby-numo/numo-narray"><code class="language-plaintext highlighter-rouge">Numo::NArray</code></a> that supports <code class="language-plaintext highlighter-rouge">UInt64</code> values, so there will no longer be a need to map each color in a <code class="language-plaintext highlighter-rouge">Theme</code> to unique identifiers.</p>
<p><a href="https://github.com/nickymarino/virgo">Virgo</a> is available on my <a href="https://github.com/nickymarino">GitHub profile</a>, and it’s open to pull requests!</p>
<p><img src="https://nickymarino.com/public/assets/2020/optimizing-virgo/example_13.png" alt="Virgo Example Output" class="center" /></p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:not-really-implemented" role="doc-endnote">
<p>In fact, the <a href="https://github.com/wvanbergen/oily_png">library</a> didn’t seem to have implemented much, if any, of the ChunkyPNG functionality. <a href="#fnref:not-really-implemented" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:linear-time" role="doc-endnote">
<p>Well, <code class="language-plaintext highlighter-rouge">O(n)</code> at least comparative to the <code class="language-plaintext highlighter-rouge">ChunkyPNG</code> implementation and tests I conducted up to an image size of 5000x5000. <a href="#fnref:linear-time" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
Virgo: Ruby Wallpaper Generator2020-01-19T00:00:00+00:00https://nickymarino.com/2020/01/19/virgo-wallpaper-generator<p>There are many galaxies with strange, funny, or mysterious names, such as the <a href="https://en.wikipedia.org/wiki/Sombrero_Galaxy">Sombrero Galaxy</a> or the <a href="https://en.wikipedia.org/wiki/Cartwheel_Galaxy">Cartwheel Galaxy</a>. Inspired by these galaxies I wanted to be able to make space-like wallpapers for my devices.</p>
<p><img src="https://nickymarino.com/public/assets/2020/virgo-wallpaper-generator/galaxy.jpg" alt="The Thousand Ruby Galaxy" class="center" /></p>
<p>As a result, I’ve written <a href="https://github.com/nickymarino/virgo">Virgo</a>, a wallpaper generator CLI written in Ruby. Virgo is great for creating PNG OLED phone wallpapers with a true black background. Let’s create an iPhone 11 size wallpaper:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ cd ~/virgo
$ ./virgo.rb --width 828 --height 1792
</code></pre></div></div>
<p><img src="https://nickymarino.com/public/assets/2020/virgo-wallpaper-generator/1.png" alt="Virgo Example Output" width="50%" class="center" /></p>
<p>Virgo has many options for creating wallpapers, including the image dimension, colors, and pixel properties:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./virgo.rb save PATH [options]
OPTIONS:
--background BACKGROUND
Wallpaper background as a hexcode. Use list-backgrounds to list predefined background names
--foregrounds FOREGROUNDS
Wallpaper foregrounds as hexcodes separated by commans. Use list-foregrounds to list predefined foreground names
--width PIXELS
Width of the wallpaper
--height PIXELS
Height of the wallpaper
--density RATIO
Ratio of pixels to size of the image, as a percent integer
--diameter PIXELS
Diameter of each pixel drawn on the wallpaper
</code></pre></div></div>
<p>There are also many predefined backgrounds and foregrounds for you to explore! Use the following commands to list some options:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ ./virgo.rb list_backgrounds
black: #000000
white: #ffffff
dark_blue: #355c7d
chalkboard: #2a363b
peach: #ff8c94
gray: #363636
teal: #2f9599
orange: ff4e50
brown: #594f4f
gray_green: #83af9b
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ ./virgo.rb list_foregrounds
white: ["#ffffff"]
ruby: ["#8d241f", "#a22924", "#b72f28", "#cc342d", "#d4453e", "#d95953", "#de6d68"]
sunset: ["#f8b195", "#f67280", "#c06c84", "#6c5b7b"]
primaries: ["#99b898", "#feceab", "#ff847c", "#e84a5f"]
primaries_light: ["#a8e6ce", "#bcedc2", "#ffd3b5", "#ffaaa6"]
gothic: ["#a8a7a7", "#cc527a", "#e8175d", "#474747"]
solar: ["#a7226e", "#ec2049", "#f26b38", "#9dedad"]
yellows: ["#e1f5c4", "#ede574", "#f9d423", "#fc913a"]
earth: ["#e5fcc2", "#9de0ad", "#45ada8", "#547980"]
faded: ["#fe4365", "#fc9d9a", "#f9cdad", "#c8c8a9"]
</code></pre></div></div>
<p><a href="https://chunkypng.com">ChunkyPNG</a> was used to manage and save the wallpaper images, and <a href="https://github.com/commander-rb/commander">Commander</a> enabled terminal argument parsing and help documentation. For optimization, I plan to use <a href="https://masa16.github.io/narray/">NArray</a> for creating the pixel locations in the wallpaper and writing a custom PNG encoder. Here are a few example wallpapers created by Virgo:</p>
<p><img src="https://nickymarino.com/public/assets/2020/virgo-wallpaper-generator/2.png" alt="Virgo Example Output" class="center" />
<img src="https://nickymarino.com/public/assets/2020/virgo-wallpaper-generator/4.png" alt="Virgo Example Output" class="center" />
<img src="https://nickymarino.com/public/assets/2020/virgo-wallpaper-generator/1.png" alt="Virgo Example Output" class="center" />
<img src="https://nickymarino.com/public/assets/2020/virgo-wallpaper-generator/5.png" alt="Virgo Example Output" class="center" />
<img src="https://nickymarino.com/public/assets/2020/virgo-wallpaper-generator/3.png" alt="Virgo Example Output" class="center" /></p>
<p>You can find Virgo on GitHub <a href="https://github.com/nickymarino/virgo">here</a>.</p>
Fast Introduction to Node APIs2019-11-22T00:00:00+00:00https://nickymarino.com/2019/11/22/fast-intro-to-node-apis<p>By the end of this post, we will have created an API using Node, <a href="https://expressjs.com">express</a> and <a href="node body parser json">body-parser</a>. Our API will have two endpoints: <code class="language-plaintext highlighter-rouge">/magic-8-ball</code> will return a random <a href="https://en.wikipedia.org/wiki/Magic_8-Ball">Magic 8-Ball</a> response, and <code class="language-plaintext highlighter-rouge">/to-zalgo</code> will convert given text into <a href="https://stackoverflow.com/questions/6579844/how-does-zalgo-text-work">Zalgo</a> text.</p>
<h1 id="setup">Setup</h1>
<p>First, create a new folder named <code class="language-plaintext highlighter-rouge">node-api</code> and navigate to it. We need to create a new npm package that will hold our API app. Run the following command and fill out the information. Each part can be left the default, except the entry point should be <code class="language-plaintext highlighter-rouge">app.js</code>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>npm init
</code></pre></div></div>
<p>Next, let’s install <code class="language-plaintext highlighter-rouge">express</code> and <code class="language-plaintext highlighter-rouge">body-parser</code>, as we’ll need both later:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>npm <span class="nb">install </span>express body-parser
</code></pre></div></div>
<p>In order to run our app, we’ll add a command inside <code class="language-plaintext highlighter-rouge">package.json</code> for <code class="language-plaintext highlighter-rouge">npm start</code>. Add this item to the <code class="language-plaintext highlighter-rouge">"scripts"</code> array:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="dl">"</span><span class="s2">scripts</span><span class="dl">"</span><span class="p">:</span> <span class="p">{</span>
<span class="p">...</span>
<span class="dl">"</span><span class="s2">start</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">node app.js</span><span class="dl">"</span>
<span class="p">},</span>
</code></pre></div></div>
<h1 id="express-hello-world">Express Hello World</h1>
<p>Now that we have our package set up, we can begin writing the web app. Let’s return “Hello world!” at the root of our app (<code class="language-plaintext highlighter-rouge">/</code>, or <code class="language-plaintext highlighter-rouge">http://localhost:3200/</code>):</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Load the modules we installed</span>
<span class="kd">const</span> <span class="nx">express</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">express</span><span class="dl">'</span><span class="p">)</span>
<span class="kd">const</span> <span class="nx">bodyparser</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">body-parser</span><span class="dl">'</span><span class="p">)</span>
<span class="c1">// Tell express to run the webserver on port 3200</span>
<span class="kd">const</span> <span class="nx">app</span> <span class="o">=</span> <span class="nx">express</span><span class="p">();</span>
<span class="kd">const</span> <span class="nx">port</span> <span class="o">=</span> <span class="nx">process</span><span class="p">.</span><span class="nx">env</span><span class="p">.</span><span class="nx">port</span> <span class="o">||</span> <span class="mi">3200</span>
<span class="c1">// Use body-parser for unencoding API request bodies - more on this later</span>
<span class="nx">app</span><span class="p">.</span><span class="nx">use</span><span class="p">(</span><span class="nx">bodyparser</span><span class="p">.</span><span class="nx">json</span><span class="p">())</span>
<span class="nx">app</span><span class="p">.</span><span class="nx">use</span><span class="p">(</span><span class="nx">bodyparser</span><span class="p">.</span><span class="nx">urlencoded</span><span class="p">({</span> <span class="na">extended</span><span class="p">:</span> <span class="kc">false</span> <span class="p">}))</span>
<span class="nx">app</span><span class="p">.</span><span class="nx">listen</span><span class="p">(</span><span class="nx">port</span><span class="p">,</span> <span class="p">()</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s2">`running on port </span><span class="p">${</span><span class="nx">port</span><span class="p">}</span><span class="s2">`</span><span class="p">)</span>
<span class="p">})</span>
<span class="c1">// Return "Hello world" when you go to http://localhost:3200</span>
<span class="nx">app</span><span class="p">.</span><span class="kd">get</span><span class="p">(</span><span class="dl">'</span><span class="s1">/</span><span class="dl">'</span><span class="p">,</span> <span class="p">(</span><span class="nx">req</span><span class="p">,</span> <span class="nx">res</span><span class="p">)</span> <span class="o">=></span> <span class="nx">res</span><span class="p">.</span><span class="nx">send</span><span class="p">(</span><span class="dl">'</span><span class="s1">Hello world!</span><span class="dl">'</span><span class="p">))</span>
</code></pre></div></div>
<p>To test our app, run <code class="language-plaintext highlighter-rouge">npm start</code> in one terminal window, then use <code class="language-plaintext highlighter-rouge">curl</code> in the other:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>curl http://localhost:3200
Hello world!
</code></pre></div></div>
<h1 id="magic-8-ball-responses">Magic 8-Ball Responses</h1>
<p>Our first API endpoint, <code class="language-plaintext highlighter-rouge">/magic-8-ball</code>, will return a JSON result in the form of <code class="language-plaintext highlighter-rouge">{"prediction": "<8-ball response>"}</code>. I wrote a helper function to return a random item from an array:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">function</span> <span class="nx">randomItemFromArray</span><span class="p">(</span><span class="nx">arr</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">arr</span><span class="p">[</span><span class="nb">Math</span><span class="p">.</span><span class="nx">floor</span><span class="p">(</span><span class="nb">Math</span><span class="p">.</span><span class="nx">random</span><span class="p">()</span> <span class="o">*</span> <span class="nx">arr</span><span class="p">.</span><span class="nx">length</span><span class="p">)]</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Then all we need to do is have our server keep an array of possible responses, pick a random one, and return the response in JSON format:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Return a random response for http://localhost:3200/magic-8-ball</span>
<span class="c1">// {"prediction": "<random_prediction>"}</span>
<span class="nx">app</span><span class="p">.</span><span class="kd">get</span><span class="p">(</span><span class="dl">'</span><span class="s1">/magic-8-ball</span><span class="dl">'</span><span class="p">,</span> <span class="p">(</span><span class="nx">req</span><span class="p">,</span> <span class="nx">res</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">responses</span> <span class="o">=</span> <span class="p">[</span>
<span class="dl">'</span><span class="s1">It is certain.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">It is decidedly so.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">Without a doubt.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">Yes - definitely.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">You may rely on it.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">As I see it, yes.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">Most likely.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">Outlook good.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">Yes.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">Signs point to yes.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">Reply hazy, try again.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">Ask again later.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">Better not tell you now.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">Cannot predict now.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">Concentrate and ask again.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">Don't count on it.</span><span class="dl">"</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">My reply is no.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">My sources say no.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">Outlook not so good.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">Very doubtful.</span><span class="dl">'</span>
<span class="p">]</span>
<span class="nx">res</span><span class="p">.</span><span class="nx">status</span><span class="p">(</span><span class="mi">200</span><span class="p">).</span><span class="nx">json</span><span class="p">({</span>
<span class="na">prediction</span><span class="p">:</span> <span class="nx">randomItemFromArray</span><span class="p">(</span><span class="nx">responses</span><span class="p">)</span>
<span class="p">})</span>
<span class="p">})</span>
</code></pre></div></div>
<p>Run <code class="language-plaintext highlighter-rouge">npm start</code>, and we can test it a few times using <code class="language-plaintext highlighter-rouge">curl</code>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>curl http://localhost:3200/magic-8-ball
<span class="o">{</span><span class="s2">"prediction"</span>:<span class="s2">"Without a doubt."</span><span class="o">}</span>
<span class="nv">$ </span>curl http://localhost:3200/magic-8-ball
<span class="o">{</span><span class="s2">"prediction"</span>:<span class="s2">"Yes - definitely."</span><span class="o">}</span>
<span class="nv">$ </span>curl http://localhost:3200/magic-8-ball
<span class="o">{</span><span class="s2">"prediction"</span>:<span class="s2">"Signs point to yes."</span><span class="o">}</span>
</code></pre></div></div>
<h1 id="zalgo-text">Zalgo Text</h1>
<p>Our Zalgo endpoint (<code class="language-plaintext highlighter-rouge">/to-zalgo</code>) is a little more advanced. A user will send a POST request including a message in the form <code class="language-plaintext highlighter-rouge">{"text": "your text here"}</code>, and the endpoint will return a response in the form <code class="language-plaintext highlighter-rouge">{"code": 200, "original": "your text here", "zalgo": "zalgo-ified text"}</code>. The endpoint will also return a 400 HTTP Status Code error if the input data is incorrect:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Return Zalgo-ified text for http://localhost:3200/to-zalgo</span>
<span class="c1">// Input: {"text": "your text here"}</span>
<span class="c1">// Returns: {"code": 200, "original": "your text here", "zalgo": "zalgo-ified text"}</span>
<span class="nx">app</span><span class="p">.</span><span class="nx">post</span><span class="p">(</span><span class="dl">'</span><span class="s1">/to-zalgo</span><span class="dl">'</span><span class="p">,</span> <span class="p">(</span><span class="nx">req</span><span class="p">,</span> <span class="nx">res</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="c1">// Return 400 if the input doesn't contain a "text" element</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">req</span><span class="p">.</span><span class="nx">body</span><span class="p">.</span><span class="nx">text</span> <span class="o">===</span> <span class="kc">undefined</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">res</span><span class="p">.</span><span class="nx">status</span><span class="p">(</span><span class="mi">400</span><span class="p">).</span><span class="nx">json</span><span class="p">({</span>
<span class="dl">"</span><span class="s2">code</span><span class="dl">"</span><span class="p">:</span> <span class="mi">400</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">message</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Missing 'text' argument</span><span class="dl">"</span>
<span class="p">})</span>
<span class="k">return</span>
<span class="p">}</span>
<span class="nx">original</span> <span class="o">=</span> <span class="nx">req</span><span class="p">.</span><span class="nx">body</span><span class="p">.</span><span class="nx">text</span>
<span class="nx">zalgo</span> <span class="o">=</span> <span class="nx">toZalgo</span><span class="p">(</span><span class="nx">original</span><span class="p">)</span>
<span class="nx">res</span><span class="p">.</span><span class="nx">status</span><span class="p">(</span><span class="mi">200</span><span class="p">).</span><span class="nx">json</span><span class="p">({</span>
<span class="dl">"</span><span class="s2">code</span><span class="dl">"</span><span class="p">:</span> <span class="mi">200</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">original</span><span class="dl">"</span><span class="p">:</span> <span class="nx">original</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">zalgo</span><span class="dl">"</span><span class="p">:</span> <span class="nx">zalgo</span>
<span class="p">})</span>
<span class="p">})</span>
</code></pre></div></div>
<p>Let’s test it a few times with <code class="language-plaintext highlighter-rouge">curl</code>. To send data in a POST request, like our text in JSON format, use <code class="language-plaintext highlighter-rouge">-d "data"</code>. Because we’re sending data in a JSON format, our requests via <code class="language-plaintext highlighter-rouge">curl</code> will need to include <code class="language-plaintext highlighter-rouge">-H "Content-Type: application/json"</code> as well.</p>
<p>(If you’re wondering why the text looks odd, I’d recommend checking out another <a href="https://lingojam.com/ZalgoText">Zalgo converter</a> first)</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>curl <span class="nt">-d</span> <span class="s1">'{"text":"Sphinx of black quartz, judge my vow"}'</span> <span class="nt">-H</span> <span class="s2">"Content-Type: application/json"</span> http://localhost:3200/to-zalgo
<span class="o">{</span><span class="s2">"code"</span>:200,<span class="s2">"original"</span>:<span class="s2">"Sphinx of black quartz, judge my vow"</span>,<span class="s2">"zalgo"</span>:<span class="s2">"S̡̲̳͔̻ͤ̏ͦ̾̀͒̀p̰̯̐̃͒͂ͪͨͤ͘͠h̷̖̰̩̍ͯi̸̩̜͇̪͍͉̭ͨ͐̆͞ͅn̡̧̤̭͚̤̯̼̹ͪͫ́̌ͫ̇̑̋ͅx̧̻̤̄ͩ͋ͣ͂ ̥̤̩̳̠͖ͧ͡ͅö͍̮̅ͯ̋ͣf̠͎̗͕̯̈́̀͑̐͌͊̍͒́ͅ ̦̬̱͉̫͍̞ͤͯͦ͂͜b̡̼̱̊ͅl̵̻̹͇̘̒̌̊̄aͩ̏͛̋̇̅̇ͩ̀͏̘̳̲̫͕ͅc̢̛̗̱͗́̓̆̌k̡͉͉̼̾̍̒͌̀ ̡̳͈͓̞̦̞̱̥̒̌ͦ̅̃q̰̪̟̥̿̀͝ȕ̗a͓̟͍͐̓̂ͣ̀͜r̞̭̪̦̩̹̂̒̐͗̕t̺͎͛̿̽͒̑̓̆ͧz̸͖̟͓̪̻͓̝̦ͨ̕,̻͔͙̲̓̈ͮ̍ ͍̘̟̖̩͊̀̈́ͩͯ̑j̷͕̱̖̔ͧ͌u̗̱͈ͨ̄ͩͬd̜̖̖̦̺̟́̇͐͛̒̆͊ͦ͜g̎ͩͅe̪̟̦͓̥̘͙ͭ̊ͨ̓ ͔̳̒̔̈̈́̈͠ͅm̧̪̊̌y̧͂̑ͯͤ͏͔͔͓͕̮ ̸̛͎͚͇͔̭̱̱͐ͮ̐ͪ͐̊͌v̘̬̘͋̅̽̅̄̐̀o̵̤̝̯̞̪̍͞ͅw̶̠̝̦̹͔̍ͪ͐̂ͮͭ̌͟"</span><span class="o">}</span>
<span class="o">{</span><span class="s2">"code"</span>:200,<span class="s2">"original"</span>:<span class="s2">"the blood of the ancients resides within me"</span>,<span class="s2">"zalgo"</span>:<span class="s2">"t͍̗͖͚͙͖͖̿ͪ̍h͍̘̩̤̼̞̫̜̒͟ȩ̛̺̫̖̝̰̥͋͛̎̎̈̈ ̢̼̫͈͓ͦ̿ͯb̺̖͚̤͓̲͓ͬ͊ͬ͑̅l̼̪̞̮͖̩̥͕̎ͧ̓̋̐̒ͧͯö̱̹͔̫͇́͌ͭͩ̉̆ͬ͆͠ͅô̸̶̲̫̞͔̻̝̰͓͋d̹̫̠͚͉͎ͨ͑ͯ̀ ̨̫͍̹̺̰̑͛̂̾͗ͪ̓ͅô͙̰͍͓̯͍̼̟ͭ́̽̑́͐̓f̯̥͙͈̺̮̙̙̅̌͂̓ͦ ̸͚̝̥̮̅̾t̨̟̗̟̼͔̑ͥ̊̾ͧͮ̿̿h̜̉͋ͮ͐e̪̳ͧ̾̏ ͬͤ̄̽̾̈̓͊͏̖̗̪͖͚a̢̩̖̯̹͗̊̽͢n̴̔ͥ̓͐͏̙̞̙̭̞͉c̖͕̘̗͉̠̬͂ͤͦ͋ì͕̥̱͍̗̐̅̆̓ͫe̮̩̩̮̬͕͈̾͂͒ͪ͛̇͞n̸̳̹̗͊ͦ̋ͅt͎̯̖̟̫ͯͪs͔̮͋ͧͩ͋̏ͯ̆͢ ̺̤̘̫̗̻̂r̡͚̮͇̘̻͔̉ͅĕ͔̪͖͓̯̙͙͗̂ͯ͛ͭs̵̝̘̺̠̘ͬͮi̴͖̤̟̭͚̞ͪͣd̶̛̪͈̉e͉̺̖̫ͥ̔̽̂̄͒́ͬ́́ͅṡ̵͕͟ͅ ̷̜̤̝̹̦̼͖̅ͭ̈͌͐̍ͦ͗ͅw̧̠͍̻̜͆̔ͣ͗͜i̵̶̙͉̺̦̲̅͋t̗̽͑͐ͣ̇ͣ͛ͧh̢̗͍͎̪̪̹̳̎͗̑̔̎̏͛͜i̶̱̪̺̖̻͓ͥ̿ͨ̇̅̔͗̎ͅņ̪ͬ̇ͭ̉ͬͩ͢ ̶̨̲̩̙ͦ̔̈́̄m̡̳̬̟͐e̱̩̠̙ͨ̓̇̽͑̋"</span><span class="o">}</span>
</code></pre></div></div>
<h1 id="conclusion">Conclusion</h1>
<p>Now our API has two endpoints, <code class="language-plaintext highlighter-rouge">/magic-8-ball</code> and <code class="language-plaintext highlighter-rouge">/to-zalgo</code> for you to use as a starting point for your own web app.</p>
<p>Here’s the full version of our <code class="language-plaintext highlighter-rouge">app.js</code>:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Load the modules we installed</span>
<span class="kd">const</span> <span class="nx">express</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">express</span><span class="dl">'</span><span class="p">)</span>
<span class="kd">const</span> <span class="nx">bodyparser</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">body-parser</span><span class="dl">'</span><span class="p">)</span>
<span class="kd">var</span> <span class="nx">toZalgo</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">to-zalgo</span><span class="dl">'</span><span class="p">)</span>
<span class="c1">// Tell express to run the webserver on port 3200</span>
<span class="kd">const</span> <span class="nx">app</span> <span class="o">=</span> <span class="nx">express</span><span class="p">();</span>
<span class="kd">const</span> <span class="nx">port</span> <span class="o">=</span> <span class="nx">process</span><span class="p">.</span><span class="nx">env</span><span class="p">.</span><span class="nx">port</span> <span class="o">||</span> <span class="mi">3200</span>
<span class="c1">// Use body-parser for unencoding API request bodies - more on this later</span>
<span class="nx">app</span><span class="p">.</span><span class="nx">use</span><span class="p">(</span><span class="nx">bodyparser</span><span class="p">.</span><span class="nx">json</span><span class="p">())</span>
<span class="nx">app</span><span class="p">.</span><span class="nx">use</span><span class="p">(</span><span class="nx">bodyparser</span><span class="p">.</span><span class="nx">urlencoded</span><span class="p">({</span> <span class="na">extended</span><span class="p">:</span> <span class="kc">false</span> <span class="p">}))</span>
<span class="nx">app</span><span class="p">.</span><span class="nx">listen</span><span class="p">(</span><span class="nx">port</span><span class="p">,</span> <span class="p">()</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s2">`running on port </span><span class="p">${</span><span class="nx">port</span><span class="p">}</span><span class="s2">`</span><span class="p">)</span>
<span class="p">})</span>
<span class="c1">// Return "Hello world" when you go to http://localhost:3200</span>
<span class="nx">app</span><span class="p">.</span><span class="kd">get</span><span class="p">(</span><span class="dl">'</span><span class="s1">/</span><span class="dl">'</span><span class="p">,</span> <span class="p">(</span><span class="nx">req</span><span class="p">,</span> <span class="nx">res</span><span class="p">)</span> <span class="o">=></span> <span class="nx">res</span><span class="p">.</span><span class="nx">send</span><span class="p">(</span><span class="dl">'</span><span class="s1">Hello world!</span><span class="dl">'</span><span class="p">))</span>
<span class="c1">// Return a random response for http://localhost:3200/magic-8-ball</span>
<span class="c1">// Returns: {"prediction": "<random_prediction>"}</span>
<span class="nx">app</span><span class="p">.</span><span class="kd">get</span><span class="p">(</span><span class="dl">'</span><span class="s1">/magic-8-ball</span><span class="dl">'</span><span class="p">,</span> <span class="p">(</span><span class="nx">req</span><span class="p">,</span> <span class="nx">res</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">responses</span> <span class="o">=</span> <span class="p">[</span>
<span class="dl">'</span><span class="s1">It is certain.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">It is decidedly so.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">Without a doubt.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">Yes - definitely.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">You may rely on it.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">As I see it, yes.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">Most likely.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">Outlook good.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">Yes.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">Signs point to yes.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">Reply hazy, try again.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">Ask again later.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">Better not tell you now.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">Cannot predict now.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">Concentrate and ask again.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">Don't count on it.</span><span class="dl">"</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">My reply is no.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">My sources say no.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">Outlook not so good.</span><span class="dl">'</span><span class="p">,</span>
<span class="dl">'</span><span class="s1">Very doubtful.</span><span class="dl">'</span>
<span class="p">]</span>
<span class="nx">res</span><span class="p">.</span><span class="nx">status</span><span class="p">(</span><span class="mi">200</span><span class="p">).</span><span class="nx">json</span><span class="p">({</span>
<span class="na">prediction</span><span class="p">:</span> <span class="nx">randomItemFromArray</span><span class="p">(</span><span class="nx">responses</span><span class="p">)</span>
<span class="p">})</span>
<span class="p">})</span>
<span class="c1">// Return Zalgo-ified text for http://localhost:3200/to-zalgo</span>
<span class="c1">// Input: {"text": "your text here"}</span>
<span class="c1">// Returns: {"code": 200, "original": "your text here", "zalgo": "zalgo-ified text"}</span>
<span class="nx">app</span><span class="p">.</span><span class="nx">post</span><span class="p">(</span><span class="dl">'</span><span class="s1">/to-zalgo</span><span class="dl">'</span><span class="p">,</span> <span class="p">(</span><span class="nx">req</span><span class="p">,</span> <span class="nx">res</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="c1">// Return 400 if the input doesn't contain a "text" element</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">req</span><span class="p">.</span><span class="nx">body</span><span class="p">.</span><span class="nx">text</span> <span class="o">===</span> <span class="kc">undefined</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">res</span><span class="p">.</span><span class="nx">status</span><span class="p">(</span><span class="mi">400</span><span class="p">).</span><span class="nx">json</span><span class="p">({</span>
<span class="dl">"</span><span class="s2">code</span><span class="dl">"</span><span class="p">:</span> <span class="mi">400</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">message</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Missing 'text' argument</span><span class="dl">"</span>
<span class="p">})</span>
<span class="k">return</span>
<span class="p">}</span>
<span class="nx">original</span> <span class="o">=</span> <span class="nx">req</span><span class="p">.</span><span class="nx">body</span><span class="p">.</span><span class="nx">text</span>
<span class="nx">zalgo</span> <span class="o">=</span> <span class="nx">toZalgo</span><span class="p">(</span><span class="nx">original</span><span class="p">)</span>
<span class="nx">res</span><span class="p">.</span><span class="nx">status</span><span class="p">(</span><span class="mi">200</span><span class="p">).</span><span class="nx">json</span><span class="p">({</span>
<span class="dl">"</span><span class="s2">code</span><span class="dl">"</span><span class="p">:</span> <span class="mi">200</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">original</span><span class="dl">"</span><span class="p">:</span> <span class="nx">original</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">zalgo</span><span class="dl">"</span><span class="p">:</span> <span class="nx">zalgo</span>
<span class="p">})</span>
<span class="p">})</span>
<span class="kd">function</span> <span class="nx">randomItemFromArray</span><span class="p">(</span><span class="nx">arr</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">arr</span><span class="p">[</span><span class="nb">Math</span><span class="p">.</span><span class="nx">floor</span><span class="p">(</span><span class="nb">Math</span><span class="p">.</span><span class="nx">random</span><span class="p">()</span> <span class="o">*</span> <span class="nx">arr</span><span class="p">.</span><span class="nx">length</span><span class="p">)]</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The entire example app can be found as a <a href="https://github.com/nickymarino/node-text-api">GitHub repo</a> as well.</p>
Calculating a Levenshtein Distance2019-10-23T00:00:00+00:00https://nickymarino.com/2019/10/23/calculating-levenshtein-distance<p>Imagine you’re writing a mobile app, and your user searches for the word <code class="language-plaintext highlighter-rouge">kitten</code>. Unfortunately, the only search terms you expected them to enter were from the following:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>smitten
mitten
kitty
fitting
written
</code></pre></div></div>
<p>How do we figure out which word the user meant to type?</p>
<h2 id="levenshtein-distances">Levenshtein Distances</h2>
<p>A <a href="https://en.wikipedia.org/wiki/Levenshtein_distance">Levenshtein distance</a> is a distance between two sequences <code class="language-plaintext highlighter-rouge">a</code> and <code class="language-plaintext highlighter-rouge">b</code>. If <code class="language-plaintext highlighter-rouge">a</code> and <code class="language-plaintext highlighter-rouge">b</code> are strings, the Levenshtein distance is the minimum amount of character edits needed to change one of the strings into the other. There are three types of edits allowed:</p>
<ul>
<li>Insertion: a character is added to <code class="language-plaintext highlighter-rouge">a</code></li>
<li>Deletion: a character is removed from <code class="language-plaintext highlighter-rouge">b</code></li>
<li>Substitution: a character is replaced in <code class="language-plaintext highlighter-rouge">a</code> or <code class="language-plaintext highlighter-rouge">b</code></li>
</ul>
<p>For example, if the first string <code class="language-plaintext highlighter-rouge">a = 'abc'</code> and the second string is <code class="language-plaintext highlighter-rouge">b = 'abc'</code>, the Levenshtein distance between the two strings is <code class="language-plaintext highlighter-rouge">0</code> because <code class="language-plaintext highlighter-rouge">a</code> and <code class="language-plaintext highlighter-rouge">b</code> are equal. If <code class="language-plaintext highlighter-rouge">a = 'abcd'</code> and <code class="language-plaintext highlighter-rouge">b = 'a'</code>, the distance is <code class="language-plaintext highlighter-rouge">3</code>. If <code class="language-plaintext highlighter-rouge">a = 'abcd'</code> and <code class="language-plaintext highlighter-rouge">b = 'aacc'</code>, the distance is <code class="language-plaintext highlighter-rouge">2</code>.</p>
<p>The definition of the Levenshtein distance for a string <code class="language-plaintext highlighter-rouge">a</code> with a length <code class="language-plaintext highlighter-rouge">i</code> and a string <code class="language-plaintext highlighter-rouge">b</code> with a length <code class="language-plaintext highlighter-rouge">j</code> is:</p>
<p><img src="https://wikimedia.org/api/rest_v1/media/math/render/svg/4520f5376b54613a5b0e6c6db46083989f901821" alt="" /></p>
<p>This definition is a recursive function. The first portion, <code class="language-plaintext highlighter-rouge">max(i, j) if min(i, j) = 0</code>, is the base cases where either the first string or the second string is empty.</p>
<p>The function <code class="language-plaintext highlighter-rouge">1_(ai != bi)</code> at the end of the third minimum element is the cost. If a[i] != b[i], the cost is 1, otherwise the cost is 0. The first minimum element is a deletion from <code class="language-plaintext highlighter-rouge">a</code>, the second is an insertion, and the third is a substitution.</p>
<h2 id="a-naive-implementation">A Naive Implementation</h2>
<p>First, let’s implement a straightforward implementation in Swift. We’ll create a function named <code class="language-plaintext highlighter-rouge">levenshtein_distance</code> and write the base cases to check whether either of the strings are empty:</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">func</span> <span class="nf">levenshtein_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="kt">String</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="kt">String</span><span class="p">)</span> <span class="o">-></span> <span class="kt">Int</span> <span class="p">{</span>
<span class="c1">// If either array is empty, return the length of the other array</span>
<span class="k">if</span> <span class="p">(</span><span class="n">a</span><span class="o">.</span><span class="n">count</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">b</span><span class="o">.</span><span class="n">count</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span><span class="n">b</span><span class="o">.</span><span class="n">count</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">a</span><span class="o">.</span><span class="n">count</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Then we add the recursive portion. We calculate the cost for the substitution, then find the minimum distance between the three different possible edits (deletion, insertion, or substitution):</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">func</span> <span class="nf">levenshtein_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="kt">String</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="kt">String</span><span class="p">)</span> <span class="o">-></span> <span class="kt">Int</span> <span class="p">{</span>
<span class="c1">// ...</span>
<span class="c1">// Check whether the last items are the same before testing the other items</span>
<span class="k">let</span> <span class="nv">cost</span> <span class="o">=</span> <span class="p">(</span><span class="n">a</span><span class="o">.</span><span class="n">last</span> <span class="o">==</span> <span class="n">b</span><span class="o">.</span><span class="n">last</span><span class="p">)</span> <span class="p">?</span> <span class="mi">0</span> <span class="p">:</span> <span class="mi">1</span>
<span class="k">let</span> <span class="nv">a_dropped</span> <span class="o">=</span> <span class="kt">String</span><span class="p">(</span><span class="n">a</span><span class="o">.</span><span class="nf">dropLast</span><span class="p">())</span>
<span class="k">let</span> <span class="nv">b_dropped</span> <span class="o">=</span> <span class="kt">String</span><span class="p">(</span><span class="n">b</span><span class="o">.</span><span class="nf">dropLast</span><span class="p">())</span>
<span class="k">return</span> <span class="nf">min</span><span class="p">(</span>
<span class="c1">// Find the distance if an item in a is removed</span>
<span class="nf">levenshtein_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="n">a_dropped</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="n">b</span><span class="p">)</span> <span class="o">+</span> <span class="mi">1</span><span class="p">,</span>
<span class="c1">// Find the distance if an item is removed from b (i.e. added to a)</span>
<span class="nf">levenshtein_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="n">a</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="n">b_dropped</span><span class="p">)</span> <span class="o">+</span> <span class="mi">1</span><span class="p">,</span>
<span class="c1">// Find the distance if an item is removed from a and b (i.e. substituted)</span>
<span class="nf">levenshtein_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="n">a_dropped</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="n">b_dropped</span><span class="p">)</span> <span class="o">+</span> <span class="n">cost</span>
<span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Let’s test our distance function with a simple test case:</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nf">print</span><span class="p">(</span><span class="nf">opti_leven_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="s">"123"</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="s">"12"</span><span class="p">))</span>
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1
</code></pre></div></div>
<p>More example test cases can be found below in the final files. And now we can compare the distances of our words to the string <code class="language-plaintext highlighter-rouge">kitten</code> to figure out which word the user probably meant to type:</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Print out the distances for our test case</span>
<span class="k">let</span> <span class="nv">first_word</span> <span class="o">=</span> <span class="s">"kitten"</span>
<span class="k">let</span> <span class="nv">test_words</span> <span class="o">=</span> <span class="p">[</span><span class="s">"smitten"</span><span class="p">,</span> <span class="s">"mitten"</span><span class="p">,</span> <span class="s">"kitty"</span><span class="p">,</span> <span class="s">"fitting"</span><span class="p">,</span> <span class="s">"written"</span><span class="p">]</span>
<span class="k">for</span> <span class="n">word</span> <span class="k">in</span> <span class="n">test_words</span> <span class="p">{</span>
<span class="k">let</span> <span class="nv">dist</span> <span class="o">=</span> <span class="nf">opti_leven_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="n">first_word</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="n">word</span><span class="p">)</span>
<span class="nf">print</span><span class="p">(</span><span class="s">"Distance between </span><span class="se">\(</span><span class="n">first_word</span><span class="se">)</span><span class="s"> and </span><span class="se">\(</span><span class="n">word</span><span class="se">)</span><span class="s">: </span><span class="se">\(</span><span class="n">dist</span><span class="se">)</span><span class="s">"</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Distance between kitten and smitten: 2
Distance between kitten and mitten: 1
Distance between kitten and kitty: 2
Distance between kitten and fitting: 3
Distance between kitten and written: 2
</code></pre></div></div>
<p>The user probably meant to type mitten instead of kitten!</p>
<h2 id="an-improved-implementation">An Improved Implementation</h2>
<p>The recursive implementation of the Levenshtein distance above won’t scale very well for larger strings. What if we needed to find the distance between a thousand strings, each with hundreds of characters?</p>
<p>One improved way to calculate a Levenshtein distance is to use a matrix of distances to “remember” previously calculated distances. First, the distance function should check for empty strings. Then, we’ll create a matrix to hold the distance calculations:</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">func</span> <span class="nf">opti_leven_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="kt">String</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="kt">String</span><span class="p">)</span> <span class="o">-></span> <span class="kt">Int</span> <span class="p">{</span>
<span class="c1">// Check for empty strings first</span>
<span class="k">if</span> <span class="p">(</span><span class="n">a</span><span class="o">.</span><span class="n">count</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">b</span><span class="o">.</span><span class="n">count</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span><span class="n">b</span><span class="o">.</span><span class="n">count</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">a</span><span class="o">.</span><span class="n">count</span>
<span class="p">}</span>
<span class="c1">// Create an empty distance matrix with dimensions len(a)+1 x len(b)+1</span>
<span class="k">var</span> <span class="nv">dists</span> <span class="o">=</span> <span class="kt">Array</span><span class="p">(</span><span class="nv">repeating</span><span class="p">:</span> <span class="kt">Array</span><span class="p">(</span><span class="nv">repeating</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span> <span class="nv">count</span><span class="p">:</span> <span class="n">b</span><span class="o">.</span><span class="n">count</span><span class="o">+</span><span class="mi">1</span><span class="p">),</span> <span class="nv">count</span><span class="p">:</span> <span class="n">a</span><span class="o">.</span><span class="n">count</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The first column and first row of the distance matrix are zeros as an initialization step. The next column goes from 1 to the length of <code class="language-plaintext highlighter-rouge">a</code> to represent removing each character to get to an empty string, and the next row goes from 1 to the length of <code class="language-plaintext highlighter-rouge">b</code> to represent adding (or inserting) each character to get to the value of <code class="language-plaintext highlighter-rouge">b</code>:</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">func</span> <span class="nf">opti_leven_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="kt">String</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="kt">String</span><span class="p">)</span> <span class="o">-></span> <span class="kt">Int</span> <span class="p">{</span>
<span class="c1">//...</span>
<span class="c1">// a's default distances are calculated by removing each character</span>
<span class="k">for</span> <span class="n">i</span> <span class="k">in</span> <span class="mi">1</span><span class="o">...</span><span class="p">(</span><span class="n">a</span><span class="o">.</span><span class="n">count</span><span class="p">)</span> <span class="p">{</span>
<span class="n">dists</span><span class="p">[</span><span class="n">i</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="n">i</span>
<span class="p">}</span>
<span class="c1">// b's default distances are calulated by adding each character</span>
<span class="k">for</span> <span class="n">j</span> <span class="k">in</span> <span class="mi">1</span><span class="o">...</span><span class="p">(</span><span class="n">b</span><span class="o">.</span><span class="n">count</span><span class="p">)</span> <span class="p">{</span>
<span class="n">dists</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="n">j</span><span class="p">]</span> <span class="o">=</span> <span class="n">j</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Similar to our naive implementation, we’ll check the remaining indices in the distance matrix. This time, however, we’ll use the previous values stored in the matrix to calculate the minimum distance rather than recursively calling the distance function. The final distance is the last element in the distance matrix (at the bottom right):</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">func</span> <span class="nf">opti_leven_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="kt">String</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="kt">String</span><span class="p">)</span> <span class="o">-></span> <span class="kt">Int</span> <span class="p">{</span>
<span class="c1">//...</span>
<span class="c1">// Find the remaining distances using previous distances</span>
<span class="k">for</span> <span class="n">i</span> <span class="k">in</span> <span class="mi">1</span><span class="o">...</span><span class="p">(</span><span class="n">a</span><span class="o">.</span><span class="n">count</span><span class="p">)</span> <span class="p">{</span>
<span class="k">for</span> <span class="n">j</span> <span class="k">in</span> <span class="mi">1</span><span class="o">...</span><span class="p">(</span><span class="n">b</span><span class="o">.</span><span class="n">count</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// Calculate the substitution cost</span>
<span class="k">let</span> <span class="nv">cost</span> <span class="o">=</span> <span class="p">(</span><span class="n">a</span><span class="p">[</span><span class="n">i</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">==</span> <span class="n">b</span><span class="p">[</span><span class="n">j</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span> <span class="p">?</span> <span class="mi">0</span> <span class="p">:</span> <span class="mi">1</span>
<span class="n">dists</span><span class="p">[</span><span class="n">i</span><span class="p">][</span><span class="n">j</span><span class="p">]</span> <span class="o">=</span> <span class="nf">min</span><span class="p">(</span>
<span class="c1">// Removing a character from a</span>
<span class="n">dists</span><span class="p">[</span><span class="n">i</span><span class="o">-</span><span class="mi">1</span><span class="p">][</span><span class="n">j</span><span class="p">]</span> <span class="o">+</span> <span class="mi">1</span><span class="p">,</span>
<span class="c1">// Adding a character to b</span>
<span class="n">dists</span><span class="p">[</span><span class="n">i</span><span class="p">][</span><span class="n">j</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">+</span> <span class="mi">1</span><span class="p">,</span>
<span class="c1">// Substituting a character from a to b</span>
<span class="n">dists</span><span class="p">[</span><span class="n">i</span><span class="o">-</span><span class="mi">1</span><span class="p">][</span><span class="n">j</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">+</span> <span class="n">cost</span>
<span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">dists</span><span class="o">.</span><span class="n">last</span><span class="o">!.</span><span class="n">last</span><span class="o">!</span>
<span class="p">}</span>
</code></pre></div></div>
<p>We can use our test cases again to verify that our improved implementation is correct:</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nf">print</span><span class="p">(</span><span class="nf">opti_leven_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="s">"123"</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="s">"12"</span><span class="p">))</span>
<span class="c1">// Print out the distances for our test case</span>
<span class="k">let</span> <span class="nv">first_word</span> <span class="o">=</span> <span class="s">"kitten"</span>
<span class="k">let</span> <span class="nv">test_words</span> <span class="o">=</span> <span class="p">[</span><span class="s">"smitten"</span><span class="p">,</span> <span class="s">"mitten"</span><span class="p">,</span> <span class="s">"kitty"</span><span class="p">,</span> <span class="s">"fitting"</span><span class="p">,</span> <span class="s">"written"</span><span class="p">]</span>
<span class="k">for</span> <span class="n">word</span> <span class="k">in</span> <span class="n">test_words</span> <span class="p">{</span>
<span class="k">let</span> <span class="nv">dist</span> <span class="o">=</span> <span class="nf">opti_leven_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="n">first_word</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="n">word</span><span class="p">)</span>
<span class="nf">print</span><span class="p">(</span><span class="s">"Distance between </span><span class="se">\(</span><span class="n">first_word</span><span class="se">)</span><span class="s"> and </span><span class="se">\(</span><span class="n">word</span><span class="se">)</span><span class="s">: </span><span class="se">\(</span><span class="n">dist</span><span class="se">)</span><span class="s">"</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1
Distance between kitten and smitten: 2
Distance between kitten and mitten: 1
Distance between kitten and kitty: 2
Distance between kitten and fitting: 3
Distance between kitten and written: 2
</code></pre></div></div>
<h2 id="swift-and-python-implementations">Swift and Python Implementations</h2>
<p><a href="/public/assets/2019/calculating-levenshtein-distance/Distance.playground.zip">Distance.playground</a>:</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="kd">import</span> <span class="kt">Foundation</span>
<span class="c1">/// Calculates the Levenshtein distance between two strings</span>
<span class="c1">/// - Parameter a: The first string</span>
<span class="c1">/// - Parameter b: The second string</span>
<span class="kd">func</span> <span class="nf">levenshtein_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="kt">String</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="kt">String</span><span class="p">)</span> <span class="o">-></span> <span class="kt">Int</span> <span class="p">{</span>
<span class="c1">// If either array is empty, return the length of the other array</span>
<span class="k">if</span> <span class="p">(</span><span class="n">a</span><span class="o">.</span><span class="n">count</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">b</span><span class="o">.</span><span class="n">count</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span><span class="n">b</span><span class="o">.</span><span class="n">count</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">a</span><span class="o">.</span><span class="n">count</span>
<span class="p">}</span>
<span class="c1">// Check whether the last items are the same before testing the other items</span>
<span class="k">let</span> <span class="nv">cost</span> <span class="o">=</span> <span class="p">(</span><span class="n">a</span><span class="o">.</span><span class="n">last</span> <span class="o">==</span> <span class="n">b</span><span class="o">.</span><span class="n">last</span><span class="p">)</span> <span class="p">?</span> <span class="mi">0</span> <span class="p">:</span> <span class="mi">1</span>
<span class="k">let</span> <span class="nv">a_dropped</span> <span class="o">=</span> <span class="kt">String</span><span class="p">(</span><span class="n">a</span><span class="o">.</span><span class="nf">dropLast</span><span class="p">())</span>
<span class="k">let</span> <span class="nv">b_dropped</span> <span class="o">=</span> <span class="kt">String</span><span class="p">(</span><span class="n">b</span><span class="o">.</span><span class="nf">dropLast</span><span class="p">())</span>
<span class="k">return</span> <span class="nf">min</span><span class="p">(</span>
<span class="c1">// Find the distance if an item in a is removed</span>
<span class="nf">levenshtein_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="n">a_dropped</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="n">b</span><span class="p">)</span> <span class="o">+</span> <span class="mi">1</span><span class="p">,</span>
<span class="c1">// Find the distance if an item is removed from b (i.e. added to a)</span>
<span class="nf">levenshtein_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="n">a</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="n">b_dropped</span><span class="p">)</span> <span class="o">+</span> <span class="mi">1</span><span class="p">,</span>
<span class="c1">// Find the distance if an item is removed from a and b (i.e. substituted)</span>
<span class="nf">levenshtein_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="n">a_dropped</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="n">b_dropped</span><span class="p">)</span> <span class="o">+</span> <span class="n">cost</span>
<span class="p">)</span>
<span class="p">}</span>
<span class="c1">/// String extension to add substring by Int (such as a[i-1])</span>
<span class="kd">extension</span> <span class="kt">String</span> <span class="p">{</span>
<span class="nf">subscript</span> <span class="p">(</span><span class="nv">i</span><span class="p">:</span> <span class="kt">Int</span><span class="p">)</span> <span class="o">-></span> <span class="kt">Character</span> <span class="p">{</span>
<span class="k">return</span> <span class="k">self</span><span class="p">[</span><span class="nf">index</span><span class="p">(</span><span class="n">startIndex</span><span class="p">,</span> <span class="nv">offsetBy</span><span class="p">:</span> <span class="n">i</span><span class="p">)]</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="c1">/// A more optimized version of the Levenshtein distance function using an array of previously calculated distances</span>
<span class="c1">/// - Parameter a: The first string</span>
<span class="c1">/// - Parameter b: The second string</span>
<span class="kd">func</span> <span class="nf">opti_leven_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="kt">String</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="kt">String</span><span class="p">)</span> <span class="o">-></span> <span class="kt">Int</span> <span class="p">{</span>
<span class="c1">// Check for empty strings first</span>
<span class="k">if</span> <span class="p">(</span><span class="n">a</span><span class="o">.</span><span class="n">count</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">b</span><span class="o">.</span><span class="n">count</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span><span class="n">b</span><span class="o">.</span><span class="n">count</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">a</span><span class="o">.</span><span class="n">count</span>
<span class="p">}</span>
<span class="c1">// Create an empty distance matrix with dimensions len(a)+1 x len(b)+1</span>
<span class="k">var</span> <span class="nv">dists</span> <span class="o">=</span> <span class="kt">Array</span><span class="p">(</span><span class="nv">repeating</span><span class="p">:</span> <span class="kt">Array</span><span class="p">(</span><span class="nv">repeating</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span> <span class="nv">count</span><span class="p">:</span> <span class="n">b</span><span class="o">.</span><span class="n">count</span><span class="o">+</span><span class="mi">1</span><span class="p">),</span> <span class="nv">count</span><span class="p">:</span> <span class="n">a</span><span class="o">.</span><span class="n">count</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span>
<span class="c1">// a's default distances are calculated by removing each character</span>
<span class="k">for</span> <span class="n">i</span> <span class="k">in</span> <span class="mi">1</span><span class="o">...</span><span class="p">(</span><span class="n">a</span><span class="o">.</span><span class="n">count</span><span class="p">)</span> <span class="p">{</span>
<span class="n">dists</span><span class="p">[</span><span class="n">i</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="n">i</span>
<span class="p">}</span>
<span class="c1">// b's default distances are calulated by adding each character</span>
<span class="k">for</span> <span class="n">j</span> <span class="k">in</span> <span class="mi">1</span><span class="o">...</span><span class="p">(</span><span class="n">b</span><span class="o">.</span><span class="n">count</span><span class="p">)</span> <span class="p">{</span>
<span class="n">dists</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="n">j</span><span class="p">]</span> <span class="o">=</span> <span class="n">j</span>
<span class="p">}</span>
<span class="c1">// Find the remaining distances using previous distances</span>
<span class="k">for</span> <span class="n">i</span> <span class="k">in</span> <span class="mi">1</span><span class="o">...</span><span class="p">(</span><span class="n">a</span><span class="o">.</span><span class="n">count</span><span class="p">)</span> <span class="p">{</span>
<span class="k">for</span> <span class="n">j</span> <span class="k">in</span> <span class="mi">1</span><span class="o">...</span><span class="p">(</span><span class="n">b</span><span class="o">.</span><span class="n">count</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// Calculate the substitution cost</span>
<span class="k">let</span> <span class="nv">cost</span> <span class="o">=</span> <span class="p">(</span><span class="n">a</span><span class="p">[</span><span class="n">i</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">==</span> <span class="n">b</span><span class="p">[</span><span class="n">j</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span> <span class="p">?</span> <span class="mi">0</span> <span class="p">:</span> <span class="mi">1</span>
<span class="n">dists</span><span class="p">[</span><span class="n">i</span><span class="p">][</span><span class="n">j</span><span class="p">]</span> <span class="o">=</span> <span class="nf">min</span><span class="p">(</span>
<span class="c1">// Removing a character from a</span>
<span class="n">dists</span><span class="p">[</span><span class="n">i</span><span class="o">-</span><span class="mi">1</span><span class="p">][</span><span class="n">j</span><span class="p">]</span> <span class="o">+</span> <span class="mi">1</span><span class="p">,</span>
<span class="c1">// Adding a character to b</span>
<span class="n">dists</span><span class="p">[</span><span class="n">i</span><span class="p">][</span><span class="n">j</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">+</span> <span class="mi">1</span><span class="p">,</span>
<span class="c1">// Substituting a character from a to b</span>
<span class="n">dists</span><span class="p">[</span><span class="n">i</span><span class="o">-</span><span class="mi">1</span><span class="p">][</span><span class="n">j</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">+</span> <span class="n">cost</span>
<span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">dists</span><span class="o">.</span><span class="n">last</span><span class="o">!.</span><span class="n">last</span><span class="o">!</span>
<span class="p">}</span>
<span class="c1">/// Function to test whether the distance function is working correctly</span>
<span class="c1">/// - Parameter a: The first test string</span>
<span class="c1">/// - Parameter b: The second test string</span>
<span class="c1">/// - Parameter answer: The expected answer to be returned by the distance function</span>
<span class="kd">func</span> <span class="nf">test_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="kt">String</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="kt">String</span><span class="p">,</span> <span class="nv">answer</span><span class="p">:</span> <span class="kt">Int</span><span class="p">)</span> <span class="o">-></span> <span class="kt">Bool</span> <span class="p">{</span>
<span class="k">let</span> <span class="nv">d</span> <span class="o">=</span> <span class="nf">opti_leven_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="n">a</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="n">b</span><span class="p">)</span>
<span class="k">if</span> <span class="p">(</span><span class="n">d</span> <span class="o">!=</span> <span class="n">answer</span><span class="p">)</span> <span class="p">{</span>
<span class="nf">print</span><span class="p">(</span><span class="s">"a: </span><span class="se">\(</span><span class="n">a</span><span class="se">)</span><span class="s">"</span><span class="p">)</span>
<span class="nf">print</span><span class="p">(</span><span class="s">"b: </span><span class="se">\(</span><span class="n">b</span><span class="se">)</span><span class="s">"</span><span class="p">)</span>
<span class="nf">print</span><span class="p">(</span><span class="s">"expected: </span><span class="se">\(</span><span class="n">answer</span><span class="se">)</span><span class="s">"</span><span class="p">)</span>
<span class="nf">print</span><span class="p">(</span><span class="s">"distance: </span><span class="se">\(</span><span class="n">d</span><span class="se">)</span><span class="s">"</span><span class="p">)</span>
<span class="k">return</span> <span class="kc">false</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="k">return</span> <span class="kc">true</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="c1">// Test the distance function with many different examples</span>
<span class="nf">test_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="s">""</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="s">""</span><span class="p">,</span> <span class="nv">answer</span><span class="p">:</span> <span class="mi">0</span><span class="p">)</span>
<span class="nf">test_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="s">"1"</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="s">"1"</span><span class="p">,</span> <span class="nv">answer</span><span class="p">:</span> <span class="mi">0</span><span class="p">)</span>
<span class="nf">test_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="s">"1"</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="s">"2"</span><span class="p">,</span> <span class="nv">answer</span><span class="p">:</span> <span class="mi">1</span><span class="p">)</span>
<span class="nf">test_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="s">"12"</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="s">"12"</span><span class="p">,</span> <span class="nv">answer</span><span class="p">:</span> <span class="mi">0</span><span class="p">)</span>
<span class="nf">test_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="s">"123"</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="s">"12"</span><span class="p">,</span> <span class="nv">answer</span><span class="p">:</span> <span class="mi">1</span><span class="p">)</span>
<span class="nf">test_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="s">"1234"</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="s">"1"</span><span class="p">,</span> <span class="nv">answer</span><span class="p">:</span> <span class="mi">3</span><span class="p">)</span>
<span class="nf">test_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="s">"1234"</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="s">"1233"</span><span class="p">,</span> <span class="nv">answer</span><span class="p">:</span> <span class="mi">1</span><span class="p">)</span>
<span class="nf">test_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="s">"1248"</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="s">"1349"</span><span class="p">,</span> <span class="nv">answer</span><span class="p">:</span> <span class="mi">2</span><span class="p">)</span>
<span class="nf">test_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="s">""</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="s">"12345"</span><span class="p">,</span> <span class="nv">answer</span><span class="p">:</span> <span class="mi">5</span><span class="p">)</span>
<span class="nf">test_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="s">"5677"</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="s">"1234"</span><span class="p">,</span> <span class="nv">answer</span><span class="p">:</span> <span class="mi">4</span><span class="p">)</span>
<span class="nf">test_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="s">"123456"</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="s">"12345"</span><span class="p">,</span> <span class="nv">answer</span><span class="p">:</span> <span class="mi">1</span><span class="p">)</span>
<span class="nf">test_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="s">"13579"</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="s">"12345"</span><span class="p">,</span> <span class="nv">answer</span><span class="p">:</span> <span class="mi">4</span><span class="p">)</span>
<span class="nf">test_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="s">"123"</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="s">""</span><span class="p">,</span> <span class="nv">answer</span><span class="p">:</span> <span class="mi">3</span><span class="p">)</span>
<span class="nf">test_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="s">"kitten"</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="s">"mittens"</span><span class="p">,</span> <span class="nv">answer</span><span class="p">:</span> <span class="mi">2</span><span class="p">)</span>
<span class="nf">print</span><span class="p">(</span><span class="nf">opti_leven_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="s">"123"</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="s">"12"</span><span class="p">))</span>
<span class="c1">// Print out the distances for our test case</span>
<span class="k">let</span> <span class="nv">first_word</span> <span class="o">=</span> <span class="s">"kitten"</span>
<span class="k">let</span> <span class="nv">test_words</span> <span class="o">=</span> <span class="p">[</span><span class="s">"smitten"</span><span class="p">,</span> <span class="s">"mitten"</span><span class="p">,</span> <span class="s">"kitty"</span><span class="p">,</span> <span class="s">"fitting"</span><span class="p">,</span> <span class="s">"written"</span><span class="p">]</span>
<span class="k">for</span> <span class="n">word</span> <span class="k">in</span> <span class="n">test_words</span> <span class="p">{</span>
<span class="k">let</span> <span class="nv">dist</span> <span class="o">=</span> <span class="nf">opti_leven_distance</span><span class="p">(</span><span class="nv">a</span><span class="p">:</span> <span class="n">first_word</span><span class="p">,</span> <span class="nv">b</span><span class="p">:</span> <span class="n">word</span><span class="p">)</span>
<span class="nf">print</span><span class="p">(</span><span class="s">"Distance between </span><span class="se">\(</span><span class="n">first_word</span><span class="se">)</span><span class="s"> and </span><span class="se">\(</span><span class="n">word</span><span class="se">)</span><span class="s">: </span><span class="se">\(</span><span class="n">dist</span><span class="se">)</span><span class="s">"</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Here’s a Python implementation of the Swift code above as <a href="/public/assets/2019/calculating-levenshtein-distance/distance.py">distance.py</a>. The Python version also can handle any <code class="language-plaintext highlighter-rouge">list</code> as well as any <code class="language-plaintext highlighter-rouge">str</code></p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Calculates the Levenshtein distance between two strings
</span><span class="k">def</span> <span class="nf">levenshtein_distance</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">):</span>
<span class="c1"># If either array is empty, return the length of the other array
</span> <span class="k">if</span> <span class="ow">not</span> <span class="nb">len</span><span class="p">(</span><span class="n">a</span><span class="p">):</span>
<span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="n">b</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="nb">len</span><span class="p">(</span><span class="n">b</span><span class="p">):</span>
<span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="n">a</span><span class="p">)</span>
<span class="c1"># Check whether the last items are the same before testing the other items
</span> <span class="k">if</span> <span class="n">a</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">==</span> <span class="n">b</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]:</span>
<span class="n">cost</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">cost</span> <span class="o">=</span> <span class="mi">1</span>
<span class="k">return</span> <span class="nb">min</span><span class="p">(</span>
<span class="c1"># Find the distance if an item in a is removed
</span> <span class="n">levenshtein_distance</span><span class="p">(</span><span class="n">a</span><span class="p">[:</span><span class="o">-</span><span class="mi">1</span><span class="p">],</span> <span class="n">b</span><span class="p">)</span> <span class="o">+</span> <span class="mi">1</span><span class="p">,</span>
<span class="c1"># Find the distance if an item is removed from b (i.e. added to a)
</span> <span class="n">levenshtein_distance</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">[:</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span> <span class="o">+</span> <span class="mi">1</span><span class="p">,</span>
<span class="c1"># Find the distance if an item is removed from a and b (i.e. substituted)
</span> <span class="n">levenshtein_distance</span><span class="p">(</span><span class="n">a</span><span class="p">[:</span><span class="o">-</span><span class="mi">1</span><span class="p">],</span> <span class="n">b</span><span class="p">[:</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span> <span class="o">+</span> <span class="n">cost</span>
<span class="p">)</span>
<span class="c1"># A more optimized version of the Levenshtein distance function using an array of previously calculated distances
</span><span class="k">def</span> <span class="nf">opti_leven_distance</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">):</span>
<span class="c1"># Create an empty distance matrix with dimensions len(a)+1 x len(b)+1
</span> <span class="n">dists</span> <span class="o">=</span> <span class="p">[</span> <span class="p">[</span><span class="mi">0</span> <span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">b</span><span class="p">)</span><span class="o">+</span><span class="mi">1</span><span class="p">)]</span> <span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">a</span><span class="p">)</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span> <span class="p">]</span>
<span class="c1"># a's default distances are calculated by removing each character
</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">a</span><span class="p">)</span><span class="o">+</span><span class="mi">1</span><span class="p">):</span>
<span class="n">dists</span><span class="p">[</span><span class="n">i</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="n">i</span>
<span class="c1"># b's default distances are calulated by adding each character
</span> <span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">b</span><span class="p">)</span><span class="o">+</span><span class="mi">1</span><span class="p">):</span>
<span class="n">dists</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="n">j</span><span class="p">]</span> <span class="o">=</span> <span class="n">j</span>
<span class="c1"># Find the remaining distances using previous distances
</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">a</span><span class="p">)</span><span class="o">+</span><span class="mi">1</span><span class="p">):</span>
<span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">b</span><span class="p">)</span><span class="o">+</span><span class="mi">1</span><span class="p">):</span>
<span class="c1"># Calculate the substitution cost
</span> <span class="k">if</span> <span class="n">a</span><span class="p">[</span><span class="n">i</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">==</span> <span class="n">b</span><span class="p">[</span><span class="n">j</span><span class="o">-</span><span class="mi">1</span><span class="p">]:</span>
<span class="n">cost</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">cost</span> <span class="o">=</span> <span class="mi">1</span>
<span class="n">dists</span><span class="p">[</span><span class="n">i</span><span class="p">][</span><span class="n">j</span><span class="p">]</span> <span class="o">=</span> <span class="nb">min</span><span class="p">(</span>
<span class="c1"># Removing a character from a
</span> <span class="n">dists</span><span class="p">[</span><span class="n">i</span><span class="o">-</span><span class="mi">1</span><span class="p">][</span><span class="n">j</span><span class="p">]</span> <span class="o">+</span> <span class="mi">1</span><span class="p">,</span>
<span class="c1"># Adding a character to b
</span> <span class="n">dists</span><span class="p">[</span><span class="n">i</span><span class="p">][</span><span class="n">j</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">+</span> <span class="mi">1</span><span class="p">,</span>
<span class="c1"># Substituting a character from a to b
</span> <span class="n">dists</span><span class="p">[</span><span class="n">i</span><span class="o">-</span><span class="mi">1</span><span class="p">][</span><span class="n">j</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">+</span> <span class="n">cost</span>
<span class="p">)</span>
<span class="k">return</span> <span class="n">dists</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">][</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
<span class="c1"># Function to test whether the distance function is working correctly
</span><span class="k">def</span> <span class="nf">test_distance</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">,</span> <span class="n">answer</span><span class="p">):</span>
<span class="n">dist</span> <span class="o">=</span> <span class="n">opti_leven_distance</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">)</span>
<span class="k">if</span> <span class="n">dist</span> <span class="o">!=</span> <span class="n">answer</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="s">'a:'</span><span class="p">,</span> <span class="n">a</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="s">'b:'</span><span class="p">,</span> <span class="n">b</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="s">'expected:'</span><span class="p">,</span> <span class="n">answer</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="s">'distance:'</span><span class="p">,</span> <span class="n">dist</span><span class="p">)</span>
<span class="k">print</span><span class="p">()</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="c1"># Test the distance function with many different examples
</span> <span class="n">test_distance</span><span class="p">(</span><span class="s">''</span><span class="p">,</span> <span class="s">''</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span>
<span class="n">test_distance</span><span class="p">(</span><span class="s">'1'</span><span class="p">,</span> <span class="s">'1'</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span>
<span class="n">test_distance</span><span class="p">(</span><span class="s">'1'</span><span class="p">,</span> <span class="s">'2'</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">test_distance</span><span class="p">(</span><span class="s">'12'</span><span class="p">,</span> <span class="s">'12'</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span>
<span class="n">test_distance</span><span class="p">(</span><span class="s">'123'</span><span class="p">,</span> <span class="s">'12'</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">test_distance</span><span class="p">(</span><span class="s">'1234'</span><span class="p">,</span> <span class="s">'1'</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span>
<span class="n">test_distance</span><span class="p">(</span><span class="s">'1234'</span><span class="p">,</span> <span class="s">'1233'</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">test_distance</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">8</span><span class="p">],</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">16</span><span class="p">],</span> <span class="mi">2</span><span class="p">)</span>
<span class="n">test_distance</span><span class="p">(</span><span class="s">''</span><span class="p">,</span> <span class="s">'12345'</span><span class="p">,</span> <span class="mi">5</span><span class="p">)</span>
<span class="n">test_distance</span><span class="p">([</span><span class="mi">5</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="mi">7</span><span class="p">,</span> <span class="mi">7</span><span class="p">],</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">],</span> <span class="mi">4</span><span class="p">)</span>
<span class="n">test_distance</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">6</span><span class="p">],</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">5</span><span class="p">],</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">test_distance</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">7</span><span class="p">,</span> <span class="mi">9</span><span class="p">],</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">5</span><span class="p">],</span> <span class="mi">4</span><span class="p">)</span>
<span class="n">test_distance</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">],</span> <span class="p">[],</span> <span class="mi">3</span><span class="p">)</span>
<span class="n">test_distance</span><span class="p">(</span><span class="s">'kitten'</span><span class="p">,</span> <span class="s">'mittens'</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span>
<span class="n">first_word</span> <span class="o">=</span> <span class="s">'kitten'</span>
<span class="n">test_words</span> <span class="o">=</span> <span class="p">[</span><span class="s">'smitten'</span><span class="p">,</span> <span class="s">'mitten'</span><span class="p">,</span> <span class="s">'kitty'</span><span class="p">,</span> <span class="s">'fitting'</span><span class="p">,</span> <span class="s">'written'</span><span class="p">]</span>
<span class="k">for</span> <span class="n">word</span> <span class="ow">in</span> <span class="n">test_words</span><span class="p">:</span>
<span class="n">dist</span> <span class="o">=</span> <span class="n">opti_leven_distance</span><span class="p">(</span><span class="n">first_word</span><span class="p">,</span> <span class="n">word</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">'Distance between </span><span class="si">{</span><span class="n">first_word</span><span class="si">}</span><span class="s"> and </span><span class="si">{</span><span class="n">word</span><span class="si">}</span><span class="s">: </span><span class="si">{</span><span class="n">dist</span><span class="si">}</span><span class="s">'</span><span class="p">)</span>
</code></pre></div></div>
Announcing Darkscreen - A Dark App2019-10-06T00:00:00+00:00https://nickymarino.com/2019/10/06/announcing-darkscreen
<p>I’m so excited to announce that my first iOS app, Darkscreen - A Dark App, has a <a href="https://testflight.apple.com/join/s4Sa9sEJ">public beta on Testflight</a>! Ever since I was given my first iPod (all the way back in 7th grade!) I’ve dreamed of creating something that millions of people have the ability to enjoy, and I can’t express how excited I am. Here’s the official description:</p>
<blockquote>
<p>Darkscreen allows you to use other iPad apps in Split View without any distractions, no hassle.</p>
<p>Darkscreen provides multiple themes, including:</p>
<ul>
<li>Dark</li>
<li>Light</li>
<li>80s</li>
<li>90s</li>
<li>Outrun</li>
</ul>
</blockquote>
<p>Download using <a href="https://testflight.apple.com/join/s4Sa9sEJ">Testflight</a> today!</p>
<h2 id="why-darkscreen">Why Darkscreen?</h2>
<p>I really love using <a href="https://apps.apple.com/us/app/apollo-for-reddit/id979274575">Apollo for Reddit</a> by Christian Selig, but he hasn’t gotten a chance to create a true iPad experience for his Reddit client yet. I use Darkscreen next to Apollo in Split View so that Apollo can be in an iPhone-sized container while keeping the rest of the screen black.</p>
<p>For example, posts shown in Apollo don’t quite look right when in full horizontal mode on iPad:</p>
<p><img src="https://nickymarino.com/public/assets/2019/announcing-darkscreen/1-just-apollo.png" alt="Apollo in full horizontal mode" class="center" /></p>
<p>Now with Darkscreen, I can browse Apollo in its intended view size without being distracted by other apps:</p>
<p><img src="https://nickymarino.com/public/assets/2019/announcing-darkscreen/2-split-view.png" alt="Apollo in Split View with Darkscreen" class="center" /></p>
<p>Switching to a new theme in Darkscreen automatically updates the table view as well as the root view underneath:</p>
<p><img src="https://nickymarino.com/public/assets/2019/announcing-darkscreen/3-theme-switch.gif" alt="Darkscreen switching themes" class="center" /></p>
<p>My next goal, of course, is for Darkscreen to respond to the system-wide <a href="https://developer.apple.com/documentation/xcode/supporting_dark_mode_in_your_interface">Dark Mode</a> setting.</p>
<h2 id="why-open-source">Why Open Source?</h2>
<p>I found it an interesting challenge to modify the appearance of all of all views in the app immediately after a user selects a theme in a <code class="language-plaintext highlighter-rouge">UITableView</code>, and I hope this brief example can help other developers implement their own theme system.</p>
<p>Even though iOS 13 introduces <a href="https://developer.apple.com/documentation/xcode/supporting_dark_mode_in_your_interface">system-wide Dark Mode</a>, this example app can be helpful to support any custom themes that go beyond the default dark and light styles.</p>
<h2 id="how-to-update-the-theme-for-a-view">How to Update the Theme for a View</h2>
<p>I’ve implemented the theme system using a <a href="https://developer.apple.com/library/archive/documentation/Cocoa/Conceptual/UserDefaults/Preferences/Preferences.html">Settings Bundle</a>, so the <code class="language-plaintext highlighter-rouge">BaseViewController</code> can subscribe to settings (theme) changes:</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">func</span> <span class="nf">registerForSettingsChange</span><span class="p">()</span> <span class="p">{</span>
<span class="kt">NotificationCenter</span><span class="o">.</span><span class="k">default</span><span class="o">.</span><span class="nf">addObserver</span><span class="p">(</span><span class="k">self</span><span class="p">,</span>
<span class="nv">selector</span><span class="p">:</span> <span class="kd">#selector(</span><span class="nf">BaseViewController.settingsChanged</span><span class="kd">)</span><span class="p">,</span>
<span class="nv">name</span><span class="p">:</span> <span class="kt">UserDefaults</span><span class="o">.</span><span class="n">didChangeNotification</span><span class="p">,</span>
<span class="nv">object</span><span class="p">:</span> <span class="kc">nil</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<p>A <code class="language-plaintext highlighter-rouge">Theme</code> corresponds to UI styles and colors:</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="kt">Theme</span> <span class="p">{</span>
<span class="c1">// ...</span>
<span class="nf">init</span><span class="p">(</span><span class="n">_</span> <span class="nv">name</span><span class="p">:</span> <span class="kt">String</span><span class="p">,</span> <span class="nv">statusBar</span><span class="p">:</span> <span class="kt">UIStatusBarStyle</span><span class="p">,</span> <span class="nv">background</span><span class="p">:</span> <span class="kt">UIColor</span><span class="p">,</span> <span class="nv">primary</span><span class="p">:</span> <span class="kt">UIColor</span><span class="p">,</span> <span class="nv">secondary</span><span class="p">:</span> <span class="kt">UIColor</span><span class="p">)</span> <span class="p">{</span>
<span class="k">self</span><span class="o">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">name</span>
<span class="n">statusBarStyle</span> <span class="o">=</span> <span class="n">statusBar</span>
<span class="n">backgroundColor</span> <span class="o">=</span> <span class="n">background</span>
<span class="n">primaryColor</span> <span class="o">=</span> <span class="n">primary</span>
<span class="n">secondaryColor</span> <span class="o">=</span> <span class="n">secondary</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>When a setting changes, <code class="language-plaintext highlighter-rouge">BaseViewController</code> updates its UI elements:</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">@objc</span> <span class="kd">func</span> <span class="nf">settingsChanged</span><span class="p">()</span> <span class="p">{</span>
<span class="nf">updateTheme</span><span class="p">()</span>
<span class="p">}</span>
<span class="kd">func</span> <span class="nf">updateTheme</span><span class="p">()</span> <span class="p">{</span>
<span class="c1">// Status bar</span>
<span class="nf">setNeedsStatusBarAppearanceUpdate</span><span class="p">()</span>
<span class="c1">// Background color</span>
<span class="k">self</span><span class="o">.</span><span class="n">view</span><span class="o">.</span><span class="n">backgroundColor</span> <span class="o">=</span> <span class="kt">Settings</span><span class="o">.</span><span class="n">shared</span><span class="o">.</span><span class="n">theme</span><span class="o">.</span><span class="n">backgroundColor</span>
<span class="c1">// Navigation bar</span>
<span class="k">self</span><span class="o">.</span><span class="n">navigationController</span><span class="p">?</span><span class="o">.</span><span class="n">navigationBar</span><span class="o">.</span><span class="nf">updateTheme</span><span class="p">()</span>
<span class="p">}</span>
</code></pre></div></div>
<p>And <code class="language-plaintext highlighter-rouge">UINavigationBar</code> is extended to support theme switching:</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">extension</span> <span class="kt">UINavigationBar</span> <span class="p">{</span>
<span class="kd">func</span> <span class="nf">updateTheme</span><span class="p">()</span> <span class="p">{</span>
<span class="c1">// Background color</span>
<span class="n">barTintColor</span> <span class="o">=</span> <span class="kt">Settings</span><span class="o">.</span><span class="n">shared</span><span class="o">.</span><span class="n">theme</span><span class="o">.</span><span class="n">backgroundColor</span>
<span class="c1">// Bar item color</span>
<span class="n">tintColor</span> <span class="o">=</span> <span class="kt">Settings</span><span class="o">.</span><span class="n">shared</span><span class="o">.</span><span class="n">theme</span><span class="o">.</span><span class="n">secondaryColor</span>
<span class="c1">// Title text color</span>
<span class="n">titleTextAttributes</span> <span class="o">=</span> <span class="p">[</span><span class="kt">NSAttributedString</span><span class="o">.</span><span class="kt">Key</span><span class="o">.</span><span class="nv">foregroundColor</span><span class="p">:</span> <span class="kt">Settings</span><span class="o">.</span><span class="n">shared</span><span class="o">.</span><span class="n">theme</span><span class="o">.</span><span class="n">secondaryColor</span><span class="p">]</span>
<span class="c1">// Status bar style</span>
<span class="n">barStyle</span> <span class="o">=</span> <span class="kt">Settings</span><span class="o">.</span><span class="n">shared</span><span class="o">.</span><span class="n">theme</span><span class="o">.</span><span class="n">navbarStyle</span>
<span class="c1">// Tell the system to update it</span>
<span class="nf">setNeedsDisplay</span><span class="p">()</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
How to Build a Song Recommender Using Create ML MLRecommender2019-10-02T00:00:00+00:00https://nickymarino.com/2019/10/02/using-mlrecommender<div class="alert">
<p class="alert-title"><b>Beta Warning</b></p>
<p class="alert-content">This example was written using macOS Catalina Version 10.15 Beta and Xcode Version 11.0 beta 5. Changes may have been made to the MLRecommender constructor since this article was written (October 2019).</p>
</div>
<h2 id="objective">Objective</h2>
<p>By the end of this post, we’ll learn how to use the <a href="https://developer.apple.com/documentation/createml">Create ML</a> <a href="https://developer.apple.com/documentation/createml/mlrecommender"><code class="language-plaintext highlighter-rouge">MLRecommender</code></a> to recommend a song to a user given their listening history. We’ll also learn how to parse and prepare an <a href="https://developer.apple.com/documentation/createml/mldatatable"><code class="language-plaintext highlighter-rouge">MLDataTable</code></a> using Python and data from a third party.</p>
<h2 id="introduction-to-mlrecommender">Introduction to MLRecommender</h2>
<p>A personalized recommendation system can be used in many different applications, such as a music player, video player, or social media site. A machine learning recommendation system compares a user’s past activity to a large library of activity from many other users. For example, if Spotify wanted to recommend you a new Daily Mix, their ML recommendation system might look at your listening history for the past few weeks and compare that history to your friends’ history. Our goal today is to create an <code class="language-plaintext highlighter-rouge">MLRecommender</code> to recommend songs to a user given their listening history.</p>
<p>The constructor for <code class="language-plaintext highlighter-rouge">MLRecommender</code> is:</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nf">init</span><span class="p">(</span><span class="nv">trainingData</span><span class="p">:</span> <span class="kt">MLDataTable</span><span class="p">,</span> <span class="nv">userColumn</span><span class="p">:</span> <span class="kt">String</span><span class="p">,</span> <span class="nv">itemColumn</span><span class="p">:</span> <span class="kt">String</span><span class="p">,</span> <span class="nv">ratingColumn</span><span class="p">:</span> <span class="kt">String</span><span class="p">?</span> <span class="o">=</span> <span class="kc">nil</span><span class="p">,</span> <span class="nv">parameters</span><span class="p">:</span> <span class="kt">MLRecommender</span><span class="o">.</span><span class="kt">ModelParameters</span> <span class="o">=</span> <span class="kt">ModelParameters</span><span class="p">())</span> <span class="k">throws</span>
</code></pre></div></div>
<h2 id="creating-the-data-tables">Creating the Data Tables</h2>
<p>The first step is to create the <code class="language-plaintext highlighter-rouge">trainingData</code> in the form of an <code class="language-plaintext highlighter-rouge">MLDataTable</code>. In this case, our training data is the listening history of many different users from the <a href="http://millionsongdataset.com">Million Song Dataset</a>, which holds the metadata for over a million songs and ratings provided by users.</p>
<p>We’ll use two files from the dataset. The first is <a href="https://static.turi.com/datasets/millionsong/10000.txt"><code class="language-plaintext highlighter-rouge">1000.txt</code></a>, which contains the user id, song id, and listen time for 10000 records. We’ll call that <code class="language-plaintext highlighter-rouge">history.txt</code> from now on. The second is <a href="https://static.turi.com/datasets/millionsong/song_data.csv"><code class="language-plaintext highlighter-rouge">song_data.csv</code></a>, which contains the song id, title, release date and artist name. We’ll call that <code class="language-plaintext highlighter-rouge">songs.csv</code> from now on. All of the complete files for this tutorial can be found at the end of the post.</p>
<p>Here’s what our input files look like. Notice that <code class="language-plaintext highlighter-rouge">songs.csv</code> has a header row while <code class="language-plaintext highlighter-rouge">history.txt</code> does not:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># history.txt
b80344d063b5ccb3212f76538f3d9e43d87dca9e SOAKIMP12A8C130995 1
b80344d063b5ccb3212f76538f3d9e43d87dca9e SOBBMDR12A8C13253B 2
b80344d063b5ccb3212f76538f3d9e43d87dca9e SOBXHDL12A81C204C0 1
...
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># songs.csv
song_id,title,release,artist_name,year
SOQMMHC12AB0180CB8,"Silent Night","Monster Ballads X-Mas","Faster Pussy cat",2003
SOVFVAK12A8C1350D9,"Tanssi vaan","Karkuteillä",Karkkiautomaatti,1995
SOGTUKN12AB017F4F1,"No One Could Ever",Butter,"Hudson Mohawke",2006
...
</code></pre></div></div>
<p>We’ll be using the <a href="https://pandas.pydata.org">pandas</a> Python library to handle our CSV data. First, download the files above and name them <code class="language-plaintext highlighter-rouge">history.txt</code> and <code class="language-plaintext highlighter-rouge">songs.csv</code>, and we’ll load them:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">csv</span>
<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="n">pd</span>
<span class="n">history_file</span> <span class="o">=</span> <span class="s">'history.txt'</span> <span class="c1"># 'https://static.turi.com/datasets/millionsong/10000.txt'
</span><span class="n">songs_metadata_file</span> <span class="o">=</span> <span class="s">'songs.csv'</span> <span class="c1"># 'https://static.turi.com/datasets/millionsong/song_data.csv'
</span>
<span class="c1"># Import the files
</span><span class="n">history_df</span> <span class="o">=</span> <span class="n">pd</span><span class="p">.</span><span class="n">read_table</span><span class="p">(</span><span class="n">history_file</span><span class="p">,</span> <span class="n">header</span><span class="o">=</span><span class="bp">None</span><span class="p">)</span>
<span class="n">history_df</span><span class="p">.</span><span class="n">columns</span> <span class="o">=</span> <span class="p">[</span><span class="s">'user_id'</span><span class="p">,</span> <span class="s">'song_id'</span><span class="p">,</span> <span class="s">'listen_count'</span><span class="p">]</span>
<span class="n">metadata_df</span> <span class="o">=</span> <span class="n">pd</span><span class="p">.</span><span class="n">read_csv</span><span class="p">(</span><span class="n">songs_metadata_file</span><span class="p">)</span>
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">songs.csv</code> already has the column headers in the file, so we didn’t need to add those like we did with <code class="language-plaintext highlighter-rouge">history_df</code>. This is what our dataframes now look like:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># history_df
user_id song_id listen_count
0 b80344d063b5ccb3212f76538f3d9e43d87dca9e SOAKIMP12A8C130995 1
1 b80344d063b5ccb3212f76538f3d9e43d87dca9e SOBBMDR12A8C13253B 2
2 b80344d063b5ccb3212f76538f3d9e43d87dca9e SOBXHDL12A81C204C0 1
...
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># metadata_df
# (The '\' means that the row continues onto the next lines)
song_id title release \
0 SOQMMHC12AB0180CB8 Silent Night Monster Ballads X-Mas
1 SOVFVAK12A8C1350D9 Tanssi vaan Karkuteillä
2 SOGTUKN12AB017F4F1 No One Could Ever Butter
artist_name year
0 Faster Pussy cat 2003
1 Karkkiautomaatti 1995
2 Hudson Mohawke 2006
...
</code></pre></div></div>
<p>Next, to create a single listening history for all users, we want to merge the song data in the <code class="language-plaintext highlighter-rouge">metadata_df</code> to the listening history in the <code class="language-plaintext highlighter-rouge">history_df</code> and create a CSV to use in Swift. Let’s also add a column that combines the song title with the artist name so that we can see both in our <code class="language-plaintext highlighter-rouge">MLRecommender</code>:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Merge the files into a single csv
</span><span class="n">song_df</span> <span class="o">=</span> <span class="n">pd</span><span class="p">.</span><span class="n">merge</span><span class="p">(</span><span class="n">history_df</span><span class="p">,</span> <span class="n">metadata_df</span><span class="p">.</span><span class="n">drop_duplicates</span><span class="p">([</span><span class="s">'song_id'</span><span class="p">]),</span> <span class="n">on</span><span class="o">=</span><span class="s">"song_id"</span><span class="p">,</span> <span class="n">how</span><span class="o">=</span><span class="s">"left"</span><span class="p">)</span>
<span class="n">song_df</span><span class="p">.</span><span class="n">to_csv</span><span class="p">(</span><span class="s">'merged_listen_data.csv'</span><span class="p">,</span> <span class="n">quoting</span><span class="o">=</span><span class="n">csv</span><span class="p">.</span><span class="n">QUOTE_NONNUMERIC</span><span class="p">)</span>
<span class="c1"># Add a "Title - Name" column for easier printing later
</span><span class="n">song_df</span><span class="p">[</span><span class="s">'song'</span><span class="p">]</span> <span class="o">=</span> <span class="n">song_df</span><span class="p">[</span><span class="s">'title'</span><span class="p">]</span> <span class="o">+</span> <span class="s">' - '</span> <span class="o">+</span> <span class="n">song_df</span><span class="p">[</span><span class="s">'artist_name'</span><span class="p">]</span>
</code></pre></div></div>
<p>Here’s what our combined song dataframe now looks like:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># song_df
user_id song_id listen_count \
0 b80344d063b5ccb3212f76538f3d9e43d87dca9e SOAKIMP12A8C130995 1
1 b80344d063b5ccb3212f76538f3d9e43d87dca9e SOBBMDR12A8C13253B 2
2 b80344d063b5ccb3212f76538f3d9e43d87dca9e SOBXHDL12A81C204C0 1
title release artist_name year \
0 The Cove Thicker Than Water Jack Johnson 0
1 Entre Dos Aguas Flamenco Para Niños Paco De Lucia 1976
2 Stronger Graduation Kanye West 2007
song
0 The Cove - Jack Johnson
1 Entre Dos Aguas - Paco De Lucia
2 Stronger - Kanye West
...
</code></pre></div></div>
<p>As of the time of writing, <code class="language-plaintext highlighter-rouge">MLRecommender</code> requires that the item id column in <code class="language-plaintext highlighter-rouge">trainingData</code> go from 1 to the number of items. In other words, if our <code class="language-plaintext highlighter-rouge">trainingData</code> included only three songs, <code class="language-plaintext highlighter-rouge">merged_listen_data.csv</code> would have song ids like <code class="language-plaintext highlighter-rouge">SOQMMHC12AB0180CB8</code>, <code class="language-plaintext highlighter-rouge">SOVFVAK12A8C1350D9</code>, and <code class="language-plaintext highlighter-rouge">SOGTUKN12AB017F4F1</code>, but we need to have song ids of <code class="language-plaintext highlighter-rouge">0</code>, <code class="language-plaintext highlighter-rouge">1</code>, and <code class="language-plaintext highlighter-rouge">2</code>. Let’s add a new column to the CSV that uses incremental song ids from 0 to N:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Find the unique song ids
</span><span class="n">song_ids</span> <span class="o">=</span> <span class="n">metadata_df</span><span class="p">.</span><span class="n">song_id</span><span class="p">.</span><span class="n">unique</span><span class="p">()</span>
<span class="c1"># Create a new dataframe of the unique song ids and a new incremental
# id for each one
</span><span class="n">incremental_id_df</span> <span class="o">=</span> <span class="n">pd</span><span class="p">.</span><span class="n">DataFrame</span><span class="p">({</span><span class="s">'song_id'</span><span class="p">:</span> <span class="n">song_ids</span><span class="p">})</span>
<span class="n">incremental_id_df</span><span class="p">[</span><span class="s">'incremental_song_id'</span><span class="p">]</span> <span class="o">=</span> <span class="n">incremental_id_df</span><span class="p">.</span><span class="n">index</span>
<span class="c1"># Merge the original song metadata with the incremental ids
</span><span class="n">new_song_id_df</span> <span class="o">=</span> <span class="n">pd</span><span class="p">.</span><span class="n">merge</span><span class="p">(</span><span class="n">song_id_df</span><span class="p">,</span> <span class="n">incremental_id_df</span><span class="p">,</span> <span class="n">on</span><span class="o">=</span><span class="s">'song_id'</span><span class="p">,</span> <span class="n">how</span><span class="o">=</span><span class="s">'left'</span><span class="p">)</span>
<span class="n">new_song_id_df</span><span class="p">.</span><span class="n">to_csv</span><span class="p">(</span><span class="s">'songs_incremental_id.csv'</span><span class="p">,</span> <span class="n">quoting</span><span class="o">=</span><span class="n">csv</span><span class="p">.</span><span class="n">QUOTE_NONNUMERIC</span><span class="p">)</span>
<span class="c1"># Create a new merged history and song metadata CSV with incremental ids
</span><span class="n">new_history_df</span> <span class="o">=</span> <span class="n">pd</span><span class="p">.</span><span class="n">merge</span><span class="p">(</span><span class="n">history_df</span><span class="p">,</span> <span class="n">incremental_id_df</span><span class="p">,</span> <span class="n">on</span><span class="o">=</span><span class="s">'song_id'</span><span class="p">,</span> <span class="n">how</span><span class="o">=</span><span class="s">'inner'</span><span class="p">)</span>
<span class="n">new_history_df</span><span class="p">.</span><span class="n">to_csv</span><span class="p">(</span><span class="s">'merged_listen_data_incremental_song_id.csv'</span><span class="p">,</span> <span class="n">quoting</span><span class="o">=</span><span class="n">csv</span><span class="p">.</span><span class="n">QUOTE_NONNUMERIC</span><span class="p">)</span>
</code></pre></div></div>
<p>Here’s what our new song CSV file looks like. Notice that there’s now an added column at the beginning with a song id from 0 to 999999:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># songs_incremental_id.csv
"","song_id","title","release","artist_name","year","incremental_song_id"
0,"SOQMMHC12AB0180CB8","Silent Night","Monster Ballads X-Mas","Faster Pussy cat",2003,0
1,"SOVFVAK12A8C1350D9","Tanssi vaan","Karkuteillä","Karkkiautomaatti",1995,1
2,"SOGTUKN12AB017F4F1","No One Could Ever","Butter","Hudson Mohawke",2006,2
...
</code></pre></div></div>
<p>And here’s what our final merged listening data looks like with the incremental ids, ready to be read by the <code class="language-plaintext highlighter-rouge">MLRecommender</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># merged_listen_data_incremental_song_id.csv
"","Unnamed: 0","user_id","song_id","listen_count","title","release","artist_name","year","incremental_song_id"
0,0,"b80344d063b5ccb3212f76538f3d9e43d87dca9e","SOAKIMP12A8C130995",1,"The Cove","Thicker Than Water","Jack Johnson",0,397069
1,18887,"7c86176941718984fed11b7c0674ff04c029b480","SOAKIMP12A8C130995",1,"The Cove","Thicker Than Water","Jack Johnson",0,397069
2,21627,"76235885b32c4e8c82760c340dc54f9b608d7d7e","SOAKIMP12A8C130995",3,"The Cove","Thicker Than Water","Jack Johnson",0,397069
...
</code></pre></div></div>
<p>Now we’re ready to load it into the recommender!</p>
<h2 id="using-mlrecommender">Using <code class="language-plaintext highlighter-rouge">MLRecommender</code></h2>
<p>Create a new Swift Playground, and add the two CSVs <code class="language-plaintext highlighter-rouge">merged_listen_data_incremental_song_id.csv</code> and <code class="language-plaintext highlighter-rouge">songs_incremental_id.csv</code> as resources to your Playground. For help on adding resources to a Swift Playground, check out <a href="https://medium.com/code-data/how-to-add-resources-to-your-swift-playground-d80c61ca3f9b">this post</a>. <strong>Make sure your Swift Playground is a blank macOS Playground and not an iOS Playground</strong>. Because our <code class="language-plaintext highlighter-rouge">MLRecommender</code> will only give us the user id and incremental song id when generating recommendations, we’ll use the second CSV to view the song titles.</p>
<p>First, let’s load the merged listening history with incremental ids:</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">import</span> <span class="kt">Foundation</span>
<span class="kd">import</span> <span class="kt">CreateML</span>
<span class="c1">// Create an MLDataTable from the merged CSV data</span>
<span class="k">let</span> <span class="nv">history_csv</span> <span class="o">=</span> <span class="kt">Bundle</span><span class="o">.</span><span class="n">main</span><span class="o">.</span><span class="nf">url</span><span class="p">(</span><span class="nv">forResource</span><span class="p">:</span> <span class="s">"merged_listen_data_incremental_song_id"</span><span class="p">,</span> <span class="nv">withExtension</span><span class="p">:</span> <span class="s">"csv"</span><span class="p">)</span><span class="o">!</span>
<span class="k">let</span> <span class="nv">history_table</span> <span class="o">=</span> <span class="k">try</span> <span class="kt">MLDataTable</span><span class="p">(</span><span class="nv">contentsOf</span><span class="p">:</span> <span class="n">history_csv</span><span class="p">)</span>
<span class="nf">print</span><span class="p">(</span><span class="n">history_table</span><span class="p">)</span>
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Columns:
X1 string
Unnamed: 0 integer
user_id string
song_id string
listen_count integer
title string
release string
artist_name string
year integer
incremental_song_id integer
Rows: 2000000
Data:
+----------------+----------------+----------------+----------------+----------------+
| X1 | Unnamed: 0 | user_id | song_id | listen_count |
+----------------+----------------+----------------+----------------+----------------+
| 0 | 0 | b80344d063b5...| SOAKIMP12A8C...| 1 |
| 1 | 18887 | 7c8617694171...| SOAKIMP12A8C...| 1 |
| 2 | 21627 | 76235885b32c...| SOAKIMP12A8C...| 3 |
| 3 | 27714 | 250c0fa2a77b...| SOAKIMP12A8C...| 1 |
| 4 | 34428 | 3f73f44560e8...| SOAKIMP12A8C...| 6 |
| 5 | 34715 | 7a4b8e7d2905...| SOAKIMP12A8C...| 6 |
| 6 | 55885 | b4a678fb729b...| SOAKIMP12A8C...| 2 |
| 7 | 65683 | 33280fc74b16...| SOAKIMP12A8C...| 1 |
| 8 | 75029 | be21ec120193...| SOAKIMP12A8C...| 1 |
| 9 | 105313 | 6fbb9ff93663...| SOAKIMP12A8C...| 2 |
+----------------+----------------+----------------+----------------+----------------+
+----------------+----------------+----------------+----------------+---------------------+
| title | release | artist_name | year | incremental_song_id |
+----------------+----------------+----------------+----------------+---------------------+
| The Cove | Thicker Than...| Jack Johnson | 0 | 397069 |
| The Cove | Thicker Than...| Jack Johnson | 0 | 397069 |
| The Cove | Thicker Than...| Jack Johnson | 0 | 397069 |
| The Cove | Thicker Than...| Jack Johnson | 0 | 397069 |
| The Cove | Thicker Than...| Jack Johnson | 0 | 397069 |
| The Cove | Thicker Than...| Jack Johnson | 0 | 397069 |
| The Cove | Thicker Than...| Jack Johnson | 0 | 397069 |
| The Cove | Thicker Than...| Jack Johnson | 0 | 397069 |
| The Cove | Thicker Than...| Jack Johnson | 0 | 397069 |
| The Cove | Thicker Than...| Jack Johnson | 0 | 397069 |
+----------------+----------------+----------------+----------------+---------------------+
[2000000 rows x 10 columns]
</code></pre></div></div>
<p>From there, we can create an <code class="language-plaintext highlighter-rouge">MLRecommender</code>. Our <code class="language-plaintext highlighter-rouge">trainingData</code> is the data table format of the merged listening history CSV, the <code class="language-plaintext highlighter-rouge">userColumn</code> is the <code class="language-plaintext highlighter-rouge">user_id</code> column name and the <code class="language-plaintext highlighter-rouge">itemColumn</code> is the <code class="language-plaintext highlighter-rouge">incremental_song_id</code> column name. The <code class="language-plaintext highlighter-rouge">user_id</code> of <code class="language-plaintext highlighter-rouge">b80344d063b5ccb3212f76538f3d9e43d87dca9e</code> was randomly picked from the merged CSV data:=.</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Generate recommendations</span>
<span class="k">let</span> <span class="nv">recommender</span> <span class="o">=</span> <span class="k">try</span> <span class="kt">MLRecommender</span><span class="p">(</span><span class="nv">trainingData</span><span class="p">:</span> <span class="n">history_table</span><span class="p">,</span> <span class="nv">userColumn</span><span class="p">:</span> <span class="s">"user_id"</span><span class="p">,</span> <span class="nv">itemColumn</span><span class="p">:</span> <span class="s">"incremental_song_id"</span><span class="p">)</span>
<span class="k">let</span> <span class="nv">recs</span> <span class="o">=</span> <span class="k">try</span> <span class="n">recommender</span><span class="o">.</span><span class="nf">recommendations</span><span class="p">(</span><span class="nv">fromUsers</span><span class="p">:</span> <span class="p">[</span><span class="s">"b80344d063b5ccb3212f76538f3d9e43d87dca9e"</span><span class="p">])</span>
<span class="nf">print</span><span class="p">(</span><span class="n">recs</span><span class="p">)</span>
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Columns:
user_id string
incremental_song_id integer
score float
rank integer
Rows: 10
Data:
+----------------+---------------------+----------------+----------------+
| user_id | incremental_song_id | score | rank |
+----------------+---------------------+----------------+----------------+
| b80344d063b5...| 114557 | 0.0461493 | 1 |
| b80344d063b5...| 834311 | 0.0436045 | 2 |
| b80344d063b5...| 939015 | 0.043068 | 3 |
| b80344d063b5...| 955047 | 0.0427589 | 4 |
| b80344d063b5...| 563380 | 0.0426116 | 5 |
| b80344d063b5...| 677759 | 0.0423951 | 6 |
| b80344d063b5...| 689170 | 0.0418951 | 7 |
| b80344d063b5...| 333053 | 0.041788 | 8 |
| b80344d063b5...| 381319 | 0.0403042 | 9 |
| b80344d063b5...| 117491 | 0.0400819 | 10 |
+----------------+---------------------+----------------+----------------+
[10 rows x 4 columns]
</code></pre></div></div>
<p>But we want to know the song metadata associated with each recommended <code class="language-plaintext highlighter-rouge">incremental_song_id</code>. Let’s load the song metadata table and join the recommendations with the song metadata using the incremental id:</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Use the songs data CSV to print the recommended song titles</span>
<span class="k">let</span> <span class="nv">songs_csv</span> <span class="o">=</span> <span class="kt">Bundle</span><span class="o">.</span><span class="n">main</span><span class="o">.</span><span class="nf">url</span><span class="p">(</span><span class="nv">forResource</span><span class="p">:</span> <span class="s">"songs_incremental_id"</span><span class="p">,</span> <span class="nv">withExtension</span><span class="p">:</span> <span class="s">"csv"</span><span class="p">)</span><span class="o">!</span>
<span class="k">let</span> <span class="nv">songs_table</span> <span class="o">=</span> <span class="k">try</span> <span class="kt">MLDataTable</span><span class="p">(</span><span class="nv">contentsOf</span><span class="p">:</span> <span class="n">songs_csv</span><span class="p">)</span>
<span class="nf">print</span><span class="p">(</span><span class="n">songs_table</span><span class="p">)</span>
<span class="k">let</span> <span class="nv">song_title_recs</span> <span class="o">=</span> <span class="n">recs</span><span class="o">.</span><span class="nf">join</span><span class="p">(</span><span class="nv">with</span><span class="p">:</span> <span class="n">songs_table</span><span class="p">,</span> <span class="nv">on</span><span class="p">:</span> <span class="s">"incremental_song_id"</span><span class="p">)</span>
<span class="nf">print</span><span class="p">(</span><span class="n">song_title_recs</span><span class="p">)</span>
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Columns:
X1 string
song_id string
title undefined
release string
artist_name string
year integer
incremental_song_id integer
Rows: 1000000
Data:
+----------------+----------------+----------------+----------------+----------------+
| X1 | song_id | title | release | artist_name |
+----------------+----------------+----------------+----------------+----------------+
| 0 | SOQMMHC12AB0...| Silent Night | Monster Ball...| Faster Pussy...|
| 1 | SOVFVAK12A8C...| Tanssi vaan | Karkuteillä | Karkkiautoma...|
| 2 | SOGTUKN12AB0...| No One Could...| Butter | Hudson Mohawke |
| 3 | SOBNYVR12A8C...| Si Vos Querés | De Culo | Yerba Brava |
| 4 | SOHSBXH12A8C...| Tangle Of As...| Rene Ablaze ...| Der Mystic |
| 5 | SOZVAPQ12A8C...| Symphony No....| Berwald: Sym...| David Montgo...|
| 6 | SOQVRHI12A6D...| We Have Got ...| Strictly The...| Sasha / Turb...|
| 7 | SOEYRFT12AB0...| 2 Da Beat Ch...| Da Bomb | Kris Kross |
| 8 | SOPMIYT12A6D...| Goodbye | Danny Boy | Joseph Locke |
| 9 | SOJCFMH12A8C...| Mama_ mama c...| March to cad...| The Sun Harb...|
+----------------+----------------+----------------+----------------+----------------+
+----------------+---------------------+
| year | incremental_song_id |
+----------------+---------------------+
| 2003 | 0 |
| 1995 | 1 |
| 2006 | 2 |
| 2003 | 3 |
| 0 | 4 |
| 0 | 5 |
| 0 | 6 |
| 1993 | 7 |
| 0 | 8 |
| 0 | 9 |
+----------------+---------------------+
[1000000 rows x 7 columns]
Columns:
user_id string
incremental_song_id integer
score float
rank integer
X1 string
song_id string
title undefined
release string
artist_name string
year integer
Rows: 11
Data:
+----------------+---------------------+----------------+----------------+----------------+
| user_id | incremental_song_id | score | rank | X1 |
+----------------+---------------------+----------------+----------------+----------------+
| b80344d063b5...| 114557 | 0.0461493 | 1 | 114578 |
| b80344d063b5...| 117491 | 0.0400819 | 10 | 117512 |
| b80344d063b5...| 333053 | 0.041788 | 8 | 333174 |
| b80344d063b5...| 381319 | 0.0403042 | 9 | 381465 |
| b80344d063b5...| 381319 | 0.0403042 | 9 | 444615 |
| b80344d063b5...| 563380 | 0.0426116 | 5 | 563705 |
| b80344d063b5...| 677759 | 0.0423951 | 6 | 678222 |
| b80344d063b5...| 689170 | 0.0418951 | 7 | 689654 |
| b80344d063b5...| 834311 | 0.0436045 | 2 | 834983 |
| b80344d063b5...| 939015 | 0.043068 | 3 | 939863 |
+----------------+---------------------+----------------+----------------+----------------+
+----------------+----------------+----------------+----------------+----------------+
| song_id | title | release | artist_name | year |
+----------------+----------------+----------------+----------------+----------------+
| SOHENSJ12AAF...| Great Indoors | Room For Squ...| John Mayer | 0 |
| SOOGZYY12A67...| Crying Shame | In Between D...| Jack Johnson | 2005 |
| SOGFKJE12A8C...| Sun It Rises | Fleet Foxes | Fleet Foxes | 2008 |
| SOECLAD12AAF...| St. Patrick'...| Room For Squ...| John Mayer | 0 |
| SOECLAD12AAF...| St. Patrick'...| Room For Squ...| John Mayer | 0 |
| SOAYTRA12A8C...| All At Once | Sleep Throug...| Jack Johnson | 2008 |
| SOKLVUI12A67...| If I Could | In Between D...| Jack Johnson | 2005 |
| SOYIJIL12A67...| Posters | Brushfire Fa...| Jack Johnson | 2000 |
| SORKFWO12A8C...| Quiet Houses | Fleet Foxes | Fleet Foxes | 2008 |
| SOJAMXH12A8C...| Meadowlarks | Fleet Foxes | Fleet Foxes | 2008 |
+----------------+----------------+----------------+----------------+----------------+
[11 rows x 10 columns]
</code></pre></div></div>
<p>The last table printed has our recommended songs, and the first one is “Great Indoors”! We can now use our <code class="language-plaintext highlighter-rouge">MLRecommender</code> for other user ids.</p>
<h2 id="wrap-up">Wrap Up</h2>
<p>First, we took a look at the <code class="language-plaintext highlighter-rouge">MLRecommender</code> constructor. Then, we gathered song data from the Million Song Dataset. We modified the dataset to increase legibility and added incremental ids for the song metadata. We loaded the song metadata and listening history into a Swift Playground, created an <code class="language-plaintext highlighter-rouge">MLRecommender</code> from the listening history and generated recommended songs. Then, we used the song metadata to join the recommended songs to their titles and artists.</p>
<h2 id="source-files">Source Files</h2>
<p>Each of the files mentioned in this tutorial can be found <a href="https://drive.google.com/open?id=1uONrROKv8lfMX1-4LBPo4fh9Y18ZYNA2">here</a>, including:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">songs.csv</code>: Metadata for one million songs</li>
<li><code class="language-plaintext highlighter-rouge">history.txt</code>: Song listening history for multiple users</li>
<li><code class="language-plaintext highlighter-rouge">data-parser.py</code>: Python code to manipulate the Million Song Dataset</li>
<li><code class="language-plaintext highlighter-rouge">merged_listed_data.csv</code>: Merged dataset of song metadata and listening history</li>
<li><code class="language-plaintext highlighter-rouge">merged_listed_data_incremental_song_id.csv</code>: <code class="language-plaintext highlighter-rouge">merged_listed_data.csv</code> with incremental ids added</li>
<li><code class="language-plaintext highlighter-rouge">songs_incremental_id.csv</code>: <code class="language-plaintext highlighter-rouge">songs.csv</code> with incremental ids added</li>
<li><code class="language-plaintext highlighter-rouge">MusicRecommender.playground</code>: Swift Playground for creating the MLRecommender</li>
</ul>
How to Change the Status Bar Style in iOS 122019-02-16T00:00:00+00:00https://nickymarino.com/2019/02/16/change-status-bar-ios<p>For some iOS apps, it may be helpful to change the color of the status bar at the top of the screen. For example, if I have a dark background, the default status bar style is hard to read:</p>
<p><img src="https://nickymarino.com/public/assets/2019/change-status-bar-ios/dark-dark.png" alt="Dark iPhone app" width="50%" class="center" /></p>
<p>To change the appearance of the status bar within a view controller, first add “View controller-based status bar appearance” as an item to your <code class="language-plaintext highlighter-rouge">Info.plist</code> with a value of <code class="language-plaintext highlighter-rouge">YES</code>:</p>
<p><img src="https://nickymarino.com/public/assets/2019/change-status-bar-ios/plist.png" alt="Info.plist" class="center" /></p>
<p>Then in any view controller, you override the <code class="language-plaintext highlighter-rouge">preferredStatusBarStyle</code> property:</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">override</span> <span class="k">var</span> <span class="nv">preferredStatusBarStyle</span><span class="p">:</span> <span class="kt">UIStatusBarStyle</span> <span class="p">{</span>
<span class="k">return</span> <span class="o">.</span><span class="n">lightContent</span>
<span class="p">}</span>
</code></pre></div></div>
<p>And if you ever need to update the status bar color, call <code class="language-plaintext highlighter-rouge">setNeedsStatusBarAppearanceUpdate()</code>. Now the full view controller looks like this:</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">import</span> <span class="kt">UIKit</span>
<span class="kd">class</span> <span class="kt">ViewController</span><span class="p">:</span> <span class="kt">UIViewController</span> <span class="p">{</span>
<span class="k">override</span> <span class="kd">func</span> <span class="nf">viewDidLoad</span><span class="p">()</span> <span class="p">{</span>
<span class="k">super</span><span class="o">.</span><span class="nf">viewDidLoad</span><span class="p">()</span>
<span class="c1">// Do any additional setup after loading the view,</span>
<span class="c1">// typically from a nib.</span>
<span class="nf">setNeedsStatusBarAppearanceUpdate</span><span class="p">()</span>
<span class="p">}</span>
<span class="k">override</span> <span class="k">var</span> <span class="nv">preferredStatusBarStyle</span><span class="p">:</span> <span class="kt">UIStatusBarStyle</span> <span class="p">{</span>
<span class="k">return</span> <span class="o">.</span><span class="n">lightContent</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Running this view controller, we get a light status bar!</p>
<p><img src="https://nickymarino.com/public/assets/2019/change-status-bar-ios/dark-white.png" alt="Dark iPhone app with light status bar" width="50%" class="center" /></p>
Compiling Cub for Shortcuts2018-12-27T21:00:00+00:00https://nickymarino.com/2018/12/27/compiling-cub-for-shortcuts<p>Earlier this week I heard about <a href="https://shortcuts.fun">Shortcuts JS</a>, which uses JavaScript to generate Shortcuts. Here’s an example from their homepage:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// We'll use this later to reference the output of a calculation</span>
<span class="kd">let</span> <span class="nx">calcVar</span> <span class="o">=</span> <span class="nx">actionOutput</span><span class="p">();</span>
<span class="c1">// Define a list of actions</span>
<span class="kd">const</span> <span class="nx">actions</span> <span class="o">=</span> <span class="p">[</span>
<span class="nx">comment</span><span class="p">({</span>
<span class="na">text</span><span class="p">:</span> <span class="dl">'</span><span class="s1">Hello, world!</span><span class="dl">'</span><span class="p">,</span>
<span class="p">}),</span>
<span class="nx">number</span><span class="p">({</span>
<span class="na">number</span><span class="p">:</span> <span class="mi">42</span><span class="p">,</span>
<span class="p">}),</span>
<span class="nx">calculate</span><span class="p">({</span>
<span class="na">operand</span><span class="p">:</span> <span class="mi">3</span><span class="p">,</span>
<span class="na">operation</span><span class="p">:</span> <span class="dl">'</span><span class="s1">/</span><span class="dl">'</span><span class="p">,</span>
<span class="p">},</span> <span class="nx">calcVar</span><span class="p">),</span>
<span class="nx">showResult</span><span class="p">({</span>
<span class="c1">// Use the Magic Variable</span>
<span class="na">text</span><span class="p">:</span> <span class="nx">withVariables</span><span class="s2">`Total is </span><span class="p">${</span><span class="nx">calcVar</span><span class="p">}</span><span class="s2">!`</span><span class="p">,</span>
<span class="p">}),</span>
<span class="p">];</span>
</code></pre></div></div>
<p>While this is a good first step to writing “real code” to make Shortcuts, specifying the operands and others in this fashion is clunky. I wondered how easy it would be to use the syntax tree instead to create the Shortcut, and Will Richardson has done that exact thing for <a href="https://github.com/louisdh/cub">Cub</a> in a <a href="https://willhbr.net/2018/12/26/compiling-for-shortcuts/">blog post</a>:</p>
<blockquote>
<p>All I have to do was traverse the syntax tree and generate a Shortcuts file. In terms of code this is fairly straightforward - just add an extension to each AST node that generates the corresponding code.</p>
</blockquote>
<p>I’m not familiar enough with iOS app development or Swift to do it myself, but it would be really interesting to write an app that can use something like <a href="https://github.com/yanagiba/swift-ast">swift-ast</a> to generate Shortcuts. Who knows what power iOS users could get if they could program advanced Shortcuts using Swift?</p>
State of the Apps 20182018-12-27T00:00:00+00:00https://nickymarino.com/2018/12/27/state-of-the-apps<p>Inspired by Cortex’s annual <a href="https://www.relay.fm/cortex/61">State of the Apps</a> discussion, I thought it would be fun to start documenting what I’m using the most on my phone every year. Below are the my most-used 3rd party apps of the year.</p>
<h2 id="productivity">Productivity</h2>
<p><img src="https://nickymarino.com/public/assets/2018/state-of-the-apps/1password.png" alt="1Password" width="100px" />
<a href="https://itunes.apple.com/us/app/1password-password-manager/id568903335?mt=8&uo=4">1Password</a> <br />
I moved from <a href="https://www.lastpass.com">LastPass</a> to 1Password earlier this year and I couldn’t be happier. With the <a href="https://www.macrumors.com/guide/password-autofill-ios-12-with-1password/">Password AutoFill API</a>, 1Password integrates with the iOS keyboard to fill in logins with only one tap, and then app even copies one-time authentication codes to the clipboard. 1Password’s integration with iOS 12 has even stellar and I can’t recommend it more.
<br /><br /></p>
<p><img src="https://nickymarino.com/public/assets/2018/state-of-the-apps/todoist.png" alt="Todoist" width="100" />
<a href="https://itunes.apple.com/us/app/todoist-organize-your-life/id572688855?mt=8&uo=4">Todoist</a> <br />
A cross-platform task management system, Todoist is how I keep track of all of my projects, from TA grading deadlines to senior design final deliverables. While Todoist’s UI doesn’t have a native app feel, it’s clean and consistent.
<br /><br /></p>
<p><img src="https://nickymarino.com/public/assets/2018/state-of-the-apps/slack.png" alt="Slack" width="100" />
<a href="https://itunes.apple.com/us/app/slack/id618783545?mt=8&uo=4">Slack</a> <br />
Aptly categorized by Federico Viticci as a <a href="https://www.macstories.net/stories/my-must-have-ios-apps-2018-edition/#work-essentials">barely passable iOS client designed to access a web app</a>, Slack was the app I used almost every day to communicate with several groups at my university. Slack supports some newer iOS features such as <a href="https://support.apple.com/en-us/HT201925">grouped notifications</a>, but the UI mostly resembles its desktop Electron client and often hides away features in non-obvious places, such as <a href="https://get.slack.help/hc/en-us/articles/206870317-Emoji-reactions#mobile-1">touching and holding a message</a> to add a reaction.
<br /><br /></p>
<h2 id="social--entertainment">Social & Entertainment</h2>
<p><img src="https://nickymarino.com/public/assets/2018/state-of-the-apps/instagram.png" alt="Instagram" width="100" />
<a href="https://itunes.apple.com/us/app/instagram/id389801252?mt=8&uo=4">Instagram</a> <br />
While I’m not a fan of the company behind Instagram for <a href="https://www.nytimes.com/2018/12/18/technology/facebook-privacy.html">many</a> <a href="https://www.nytimes.com/2018/04/11/technology/facebook-privacy-hearings.html">many</a> <a href="https://www.nbcnews.com/tech/social-media/timeline-facebook-s-privacy-issues-its-responses-n859651">many</a> reasons, Instagram is a place where I can keep up with my friends via Stories and posts, and I’ve found it to be much more positive overall than other social networks this year.
<br /><br /></p>
<p><img src="https://nickymarino.com/public/assets/2018/state-of-the-apps/apollo.png" alt="Apollo" width="100" />
<a href="https://itunes.apple.com/us/app/apollo-for-reddit/id979274575?mt=8&uo=4">Apollo</a> <br />
Reddit is one of my primary social networks I use every day, and Apollo is hands down my favorite way to experience Reddit on any platform. The design is intuitive and easy to personalize, and it has a fantastic night mode. I can’t stop recommending the app to my friends that use the first party client.
<br /><br /></p>
<p><img src="https://nickymarino.com/public/assets/2018/state-of-the-apps/lire.png" alt="lire" width="100" />
<a href="https://itunes.apple.com/us/app/lire-full-text-rss/id550441545?mt=8&uo=4">lire</a> <br />
A few months ago I was using <a href="https://ifttt.com">IFTTT</a> applets to monitor RSS feeds and push them to a specific project in Todoist, but it because untenable after adding too many feeds. I moved to <a href="https://www.inoreader.com">Inoreader</a> as my RSS service and lire as my RSS reader. The app has a clean, native-looking design, and it uses its own extractor to display a story’s full text. I wish the per-feed options were more straightforward and easy to access, but overall I’m very happy using it as my primary RSS reader.
<br /><br /></p>
<p><img src="https://nickymarino.com/public/assets/2018/state-of-the-apps/overcast.png" alt="Overcast" width="100" />
<a href="https://itunes.apple.com/us/app/overcast/id888422857?mt=8&uo=4">Overcast</a> <br />
Overcast has been my podcast player go-to app since I started using it over 3 years ago. Smart Speed and Voice Boost are still industry-leading features that I can’t live without, and new features are being continuously added, such as full text search in version 5.0.
<br /><br /></p>
<p><img src="https://nickymarino.com/public/assets/2018/state-of-the-apps/groupme.png" alt="GroupMe" width="100" />
<a href="https://itunes.apple.com/us/app/groupme/id392796698?mt=8&uo=4">GroupMe</a> <br />
Another app in the “barely passable” category, I use GroupMe every day at my university for group chats ranging from friend groups to club event announcements. The app is missing many basic features I expect from a communication app, including read-message syncing across devices and platforms, and I cannot wait until I can delete this app from my phone.
<br /><br /></p>
<h2 id="health--finance">Health & Finance</h2>
<p><img src="https://nickymarino.com/public/assets/2018/state-of-the-apps/autosleep.png" alt="AutoSleep" width="100" />
<a href="https://itunes.apple.com/us/app/autosleep-tracker-for-watch/id1164801111?mt=8&uo=4">AutoSleep</a> <br />
While the Apple Health app <a href="https://www.apple.com/ios/health/">tracks basic sleep information</a>, AutoSleep provides in-depth detail for sleep trends and day-to-day stats. I check it often to look at my readiness for the day, cumulative sleep debt, and overall sleep time consistency. It was especially interesting to compare one particularly-gruesome senior design week of little sleep with the rest of the semester averages.
<br /><br /></p>
<p><img src="https://nickymarino.com/public/assets/2018/state-of-the-apps/ynab.png" alt="YNAB" width="100" />
<a href="https://itunes.apple.com/us/app/ynab-you-need-a-budget/id1010865877?mt=8&uo=4">YNAB</a> <br />
While I had toyed with budgeting on <a href="https://www.mint.com">Mint</a> in the past, YNAB (or You Need a Budget) is a great way to manage your savings and expenses. The service requires a <a href="https://www.youneedabudget.com/pricing/">subscription</a>, but its <a href="https://www.youneedabudget.com/features/">features</a> such as Bank Syncing and Goal Tracking as well as it’s straightforward usage make it an excellent deal. YNAB has given me a clear way to know exactly what I’m spending every month.
<br /><br /></p>
<p><img src="https://nickymarino.com/public/assets/2018/state-of-the-apps/venmo.png" alt="Venmo" width="100" />
<a href="https://itunes.apple.com/us/app/venmo-send-receive-money/id351727428?mt=8&uo=4">Venmo</a> <br />
A digital wallet app <a href="https://techcrunch.com/2013/09/26/paypal-acquires-payments-gateway-braintree-for-800m-in-cash/">owned by PayPal</a>, Venmo allows me to send and receive payments from other people. Similar apps like <a href="https://www.apple.com/apple-pay/">Apple Pay</a> and <a href="https://cash.app">Cash App</a> are available, but Venmo is what nearly all of the people in my social circle have centralized on. Further, with the addition of a <a href="https://venmo.com/card/">Venmo card</a> option released in the summer, Venmo has made it to handle group events and easily split payments.
<br /><br /></p>
<h2 id="utilities">Utilities</h2>
<p><img src="https://nickymarino.com/public/assets/2018/state-of-the-apps/deliveries.png" alt="Deliveries" width="100" />
<a href="https://itunes.apple.com/us/app/deliveries-a-package-tracker/id290986013?mt=8&uo=4">Deliveries</a> <br />
This app has allowed me to neurotically check my packages from every online retailer. All I need to do is copy the tracking number from USPS, FedEx, or UPS and paste it into Deliveries, and then I can push updates whenever the package status has changed. Deliveries is so good it encourages me to buy more things from Amazon, if that’s even possible.
<br /><br /></p>
<p><img src="https://nickymarino.com/public/assets/2018/state-of-the-apps/carrot.png" alt="CARROT Weather" width="100" />
<a href="https://itunes.apple.com/us/app/carrot-weather/id961390574?mt=8&uo=4">CARROT Weather</a> <br />
I switched from <a href="https://itunes.apple.com/app/apple-store/id517329357?mt=8">Dark Sky</a> to CARROT Weather in February, and that was mostly because of the fun sadistic messages that managed to be both funny and relevant. At the same time, the app has seen numerous updates this year to support multiple weather locations, highly customizable Apple Watch complications, and more.
<br /><br /></p>
<p><img src="https://nickymarino.com/public/assets/2018/state-of-the-apps/1blockerx.png" alt="1Blocker X" width="100" />
<a href="https://itunes.apple.com/us/app/1blocker-x-adblock/id1365531024?mt=8&uo=4">1Blocker X</a> <br />
For any website that doesn’t respect its users and decides to use popups, newsletter signup prompts, and auto-playing video ads, 1Blocker X is great at preventing them in the background. The app does what a utility should do: work without me even remembering I have a content blocker turned on.
<br /><br /></p>
<p><img src="https://nickymarino.com/public/assets/2018/state-of-the-apps/googlemaps.png" alt="Google Maps" width="100" />
<a href="https://itunes.apple.com/us/app/google-maps-transit-food/id585027354?mt=8&uo=4">Google Maps</a> <br />
Whenever I need to travel more than 10 minutes, I turn on Google Maps for live traffic updates and possible shorter route recommendations. In addition to retailer busy times and reviews, Google Maps is a great way to gather details about locations around me.
<br /><br /></p>
<p><img src="https://nickymarino.com/public/assets/2018/state-of-the-apps/due.png" alt="Due" width="100" />
<a href="https://itunes.apple.com/us/app/due-reminders-timers/id390017969?mt=8&uo=4">Due</a> <br />
For smaller tasks that I want to be reminded of over and over, I put them in Due, which is great at spamming me with notifications until I mark the task complete. Plus, the app recently added custom snooze times from a notification in a <a href="https://www.macstories.net/reviews/due-30-adds-pure-black-theme-custom-snooze-times-and-haptic-feedback/">recent update</a>.
<br /><br /></p>
<p><img src="https://nickymarino.com/public/assets/2018/state-of-the-apps/googlephotos.png" alt="Google Photos" width="100" />
<a href="https://itunes.apple.com/us/app/google-photos/id962194608?mt=8&uo=4">Google Photos</a> <br />
I use Google Photos both as a secondary back up to <a href="https://support.apple.com/en-us/HT204264">iCloud Photos</a> and for more power photo analytics than what Apple provides. For now, I’ll gladly trade Google using my aggregated photo data to have an easy way to <a href="https://support.google.com/photos/answer/6128838?co=GENIE.Platform%3DiOS&hl=en&oco=0">search photos by person, place, or thing</a> across platforms. I also share albums and pictures through Google Photos to groups that have a mix of Android an iOS devices.
<br /><br /></p>
<p><img src="https://nickymarino.com/public/assets/2018/state-of-the-apps/tailor.png" alt="Tailor" width="100" />
<a href="https://itunes.apple.com/us/app/tailor-screenshot-stitching/id926653095?mt=8&uo=4">Tailor</a> <br />
A nifty utility app that stitches multiple screenshots into one vertical image, Tailor provides an easy way to send more readable conversations to other people, whether they’re from Messages, Slack, or other apps.
<br /><br /></p>
<h2 id="home-screen">Home Screen</h2>
<p>Finally, here’s a picture of my home screen at the end of 2018:</p>
<p><img src="https://nickymarino.com/public/assets/2018/state-of-the-apps/homescreen.png" alt="My home screen" width="500" /></p>
The Case of the 500-Mile Email2018-12-26T00:00:00+00:00https://nickymarino.com/2018/12/26/case-of-the-500-mile-email<p>Trey Harris, writing <a href="https://www.ibiblio.org/harris/500milemail.html">to sage-members</a>:</p>
<blockquote>
<p>I was working in a job running the campus email system some years ago when I got a call from the chairman of the statistics department.</p>
<p>“We’re having a problem sending email out of the department.”</p>
<p>“What’s the problem?” I asked.</p>
<p>“We can’t send mail more than 500 miles,” the chairman explained.</p>
<p>I choked on my latte. “Come again?”</p>
<p>“We can’t send mail farther than 500 miles from here,” he repeated. “A little bit more, actually. Call it 520 miles. But no farther.”</p>
</blockquote>
<p>While it’s an older story, it’s a fantastic one. And it’s a great reminder to check your software versions.</p>
Web API and Templates with Python requests and Jinja22018-12-20T00:00:00+00:00https://nickymarino.com/2018/12/20/web-apis-and-templates-python<h1 id="introduction">Introduction</h1>
<p><a href="https://developer.mozilla.org/en-US/docs/Learn/JavaScript/Client-side_web_APIs/Introduction">Web APIs</a> are becoming an <a href="https://code.tutsplus.com/articles/the-increasing-importance-of-apis-in-web-development--net-22368">increasingly popular</a> method to retrieve and store data, and they can be an extremely powerful tool for any programmer. In this walkthrough, you will use GroupMe’s <a href="https://dev.groupme.com/docs/v3">public API</a> to retrieve the last few messages in a group and display it using a custom HTML template in Python 3.7.1:</p>
<p><img src="https://nickymarino.com/public/assets/2018/web-apis-and-templates/output-site.png" alt="Output Website" /></p>
<p>While we’ll be using specific GroupMe API endpoints, this walkthrough will help you to learn the basics of working with web APIs in general.</p>
<h2 id="set-up">Set up</h2>
<p>Before we begin, you need to have the following modules installed:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">requests</code> (for connecting to the API)</li>
<li><code class="language-plaintext highlighter-rouge">jinja2</code> (for adding data to our template)</li>
</ul>
<h1 id="using-a-web-api">Using a Web API</h1>
<h2 id="working-with-requests">Working with <code class="language-plaintext highlighter-rouge">requests</code></h2>
<p>The <code class="language-plaintext highlighter-rouge">requests</code> library is great for creating HTTP requests, and it has fantastic <a href="http://docs.python-requests.org/en/master/">documentation</a>. We’ll be using <code class="language-plaintext highlighter-rouge">requests.get(url)</code> to get the information we need:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="kn">import</span> <span class="nn">requests</span>
<span class="o">>>></span> <span class="c1"># Use JSONPlaceHolder as an example
</span><span class="o">>>></span> <span class="n">req</span> <span class="o">=</span> <span class="n">requests</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">'https://jsonplaceholder.typicode.com/todos/1'</span><span class="p">)</span>
<span class="o">>>></span> <span class="k">print</span><span class="p">(</span><span class="n">req</span><span class="p">.</span><span class="n">text</span><span class="p">)</span>
<span class="p">{</span>
<span class="s">"userId"</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s">"id"</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s">"title"</span><span class="p">:</span> <span class="s">"delectus aut autem"</span><span class="p">,</span>
<span class="s">"completed"</span><span class="p">:</span> <span class="n">false</span>
<span class="p">}</span>
</code></pre></div></div>
<h2 id="create-an-application">Create an Application</h2>
<p>To use GroupMe’s API, we need to first register an application to get our API key. Log into the <a href="https://dev.groupme.com/session/new">developer application page</a> and click “Create Application”. You’ll be taken to this page:</p>
<p><img src="https://nickymarino.com/public/assets/2018/web-apis-and-templates/create-application.png" alt="Create Application Page" /></p>
<p>We won’t be using the Callback URL, so set that to any valid URL. Fill in the developer information and submit the form, and you’ll be taken to the application’s detail page:</p>
<p><img src="https://nickymarino.com/public/assets/2018/web-apis-and-templates/application-access-token.png" alt="Application Page" /></p>
<p>Copy the Access Token at the bottom of the application page. That’ll be our API key.</p>
<h2 id="find-the-group-id">Find the Group ID</h2>
<p>To get messages in a group, we’ll first need to get the group’s ID. GroupMe’s <a href="https://dev.groupme.com/docs/v3">documentation</a> says to use the base url <code class="language-plaintext highlighter-rouge">https://api.groupme.com/v3</code> and has the following example using <code class="language-plaintext highlighter-rouge">curl</code>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>curl <span class="nt">-X</span> POST <span class="nt">-H</span> <span class="s2">"Content-Type: application/json"</span> <span class="nt">-d</span> <span class="s1">'{"name": "Family"}'</span> https://api.groupme.com/v3/groups?token<span class="o">=</span>YOUR_ACCESS_TOKEN
</code></pre></div></div>
<p>From this, we know that the url we use with <code class="language-plaintext highlighter-rouge">requests.get()</code> will be in the form<sup id="fnref:url-params" role="doc-noteref"><a href="#fn:url-params" class="footnote" rel="footnote">1</a></sup></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>https://api.groupme.com/v3/...?token=YOUR_ACCESS_TOKEN
</code></pre></div></div>
<p>Looking at the <a href="https://dev.groupme.com/docs/v3#groups">groups API</a>, we can use <code class="language-plaintext highlighter-rouge">GET /groups</code> to retrieve a list of our most recent groups:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="kn">import</span> <span class="nn">requests</span>
<span class="o">>>></span> <span class="n">API_KEY</span> <span class="o">=</span> <span class="s">'YOUR_API_KEY'</span>
<span class="o">>>></span> <span class="n">base_url</span> <span class="o">=</span> <span class="s">'https://api.groupme.com/v3'</span>
<span class="o">>>></span> <span class="n">req</span> <span class="o">=</span> <span class="n">requests</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="sa">f</span><span class="s">'</span><span class="si">{</span><span class="n">base_url</span><span class="si">}</span><span class="s">/groups?token=</span><span class="si">{</span><span class="n">API_KEY</span><span class="si">}</span><span class="s">'</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">req</span><span class="p">.</span><span class="n">content</span>
<span class="sa">b</span><span class="s">'{"response":[{"id":"1111","group_id":"2222","name":"My Group Name","phone_number":"+1 5555555555","type":"private","description":"Group description goes here",
...
</span></code></pre></div></div>
<p>First, we construct the url for the API request and pass it as the argument to <code class="language-plaintext highlighter-rouge">requests.get()</code>. Then, we print the result of the API request stored as <code class="language-plaintext highlighter-rouge">req.content</code>.</p>
<p>The response we get from GroupMe is a JSON-formatted string, so we’ll move our script into its own file and parse the string using Python’s standard <code class="language-plaintext highlighter-rouge">json</code> library:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">json</span>
<span class="kn">import</span> <span class="nn">requests</span>
<span class="n">API_KEY</span> <span class="o">=</span> <span class="s">'YOUR_API_KEY'</span>
<span class="n">BASE_URL</span> <span class="o">=</span> <span class="s">'https://api.groupme.com/v3'</span>
<span class="k">def</span> <span class="nf">api_request</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
<span class="s">'''Returns the data from a request (eg /groups)'''</span>
<span class="n">url</span> <span class="o">=</span> <span class="sa">f</span><span class="s">'</span><span class="si">{</span><span class="n">BASE_URL</span><span class="si">}{</span><span class="n">request</span><span class="si">}</span><span class="s">?token=</span><span class="si">{</span><span class="n">API_KEY</span><span class="si">}</span><span class="s">'</span>
<span class="n">req</span> <span class="o">=</span> <span class="n">requests</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">url</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">req</span><span class="p">.</span><span class="n">content</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">None</span>
<span class="c1"># We only want the data associated with the "response" key
</span> <span class="k">return</span> <span class="n">json</span><span class="p">.</span><span class="n">loads</span><span class="p">(</span><span class="n">req</span><span class="p">.</span><span class="n">content</span><span class="p">)[</span><span class="s">'response'</span><span class="p">]</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">groups</span> <span class="o">=</span> <span class="n">api_request</span><span class="p">(</span><span class="s">'/groups'</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">groups</span><span class="p">),</span> <span class="s">'group(s)'</span><span class="p">)</span>
<span class="k">for</span> <span class="n">group</span> <span class="ow">in</span> <span class="n">groups</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">'ID </span><span class="si">{</span><span class="n">group</span><span class="p">[</span><span class="s">"group_id"</span><span class="p">]</span><span class="si">}</span><span class="s">: </span><span class="si">{</span><span class="n">group</span><span class="p">[</span><span class="s">"name"</span><span class="p">]</span><span class="si">}</span><span class="s">'</span><span class="p">)</span>
</code></pre></div></div>
<p>The function <code class="language-plaintext highlighter-rouge">api_request</code> does the work of creating the final URL string for us. Then, it makes the request and checks that something was returned by GroupMe’s servers. If something was sent back to us, the content is converted<sup id="fnref:json-object" role="doc-noteref"><a href="#fn:json-object" class="footnote" rel="footnote">2</a></sup> from a string into a Python object using <code class="language-plaintext highlighter-rouge">json.loads()</code>. Finally, we return the data associated with the key <code class="language-plaintext highlighter-rouge">response</code>, because the rest is metadata unimportant to us.</p>
<p>When we run the script, our most recent groups are returned (as a JSON object decoded into a Python object). The result will tell us the group names and their group IDs:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>3 group(s)
ID 11111111: Python Tips and Tricks
ID 22222222: University Friend Group
ID 33333333: GitHub Chat
</code></pre></div></div>
<h2 id="get-messages-for-a-group">Get Messages for a Group</h2>
<p>We have a list of our group IDs, so we can use the following API to get a list of recent messages for one group:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>GET /groups/<group_id>/messages
</code></pre></div></div>
<p>Let’s add this endpoint to our script as <code class="language-plaintext highlighter-rouge">get_messages_for_group(group_id)</code>:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">json</span>
<span class="kn">import</span> <span class="nn">requests</span>
<span class="n">API_KEY</span> <span class="o">=</span> <span class="s">'YOUR_API_KEY'</span>
<span class="n">BASE_URL</span> <span class="o">=</span> <span class="s">'https://api.groupme.com/v3'</span>
<span class="k">def</span> <span class="nf">api_request</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
<span class="s">'''Returns the data from a request (eg /groups)'''</span>
<span class="n">url</span> <span class="o">=</span> <span class="sa">f</span><span class="s">'</span><span class="si">{</span><span class="n">BASE_URL</span><span class="si">}{</span><span class="n">request</span><span class="si">}</span><span class="s">?token=</span><span class="si">{</span><span class="n">API_KEY</span><span class="si">}</span><span class="s">'</span>
<span class="n">req</span> <span class="o">=</span> <span class="n">requests</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">url</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">req</span><span class="p">.</span><span class="n">content</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">None</span>
<span class="c1"># We only want the data associated with the "response" key
</span> <span class="k">return</span> <span class="n">json</span><span class="p">.</span><span class="n">loads</span><span class="p">(</span><span class="n">req</span><span class="p">.</span><span class="n">content</span><span class="p">)[</span><span class="s">'response'</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">get_messages_for_group</span><span class="p">(</span><span class="n">group_id</span><span class="p">):</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">api_request</span><span class="p">(</span><span class="sa">f</span><span class="s">'/groups/</span><span class="si">{</span><span class="n">group_id</span><span class="si">}</span><span class="s">/messages'</span><span class="p">)</span>
<span class="c1"># Just return the messages (and none of the metadata)
</span> <span class="k">return</span> <span class="n">response</span><span class="p">[</span><span class="s">'messages'</span><span class="p">]</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">messages</span> <span class="o">=</span> <span class="n">get_messages_for_group</span><span class="p">(</span><span class="n">YOUR_GROUP_ID</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">messages</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
</code></pre></div></div>
<p>Our script will get the messages for a group (fill in <code class="language-plaintext highlighter-rouge">YOUR_GROUP_ID</code>) and print the most recent one. Running it will print something like:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{'attachments': [], 'avatar_url': None, 'created_at': 1544810700, 'favorited_by': [], 'group_id': '11112233', 'id': '882882828288288282', 'name': 'Johnny Test', 'sender_id': '22558899', 'sender_type': 'user', 'source_guid': 'android-11111111-3eee-4444-9999-aaaabbbbcccc', 'system': False, 'text': "Hello everyone!", 'user_id': '55551111'}
</code></pre></div></div>
<p>We can see from the message’s data that the sender’s name “Jonny Test” and the text was “Hello everyone!” Next, we should organize our API results as Python objects to be easier to expand on.</p>
<h2 id="creating-classes-for-api-objects">Creating Classes for API Objects</h2>
<p>Now that we’re ready to start processing the data from the API, it’s a good time to create objects to represent our API objects. With Python classes, we can keep only the data we need and begin to process our own information. We’ll initialize our API objects by passing them the decoded Python object from <code class="language-plaintext highlighter-rouge">api_request(request)</code>. This way, we can more easily add class properties without needing to change our request function.</p>
<p>Let’s make two classes, <code class="language-plaintext highlighter-rouge">Group</code> and <code class="language-plaintext highlighter-rouge">Message</code>:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Message</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">json</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">user_id</span> <span class="o">=</span> <span class="n">json</span><span class="p">[</span><span class="s">'user_id'</span><span class="p">]</span>
<span class="bp">self</span><span class="p">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">json</span><span class="p">[</span><span class="s">'name'</span><span class="p">]</span>
<span class="bp">self</span><span class="p">.</span><span class="n">text</span> <span class="o">=</span> <span class="n">json</span><span class="p">[</span><span class="s">'text'</span><span class="p">]</span>
<span class="k">class</span> <span class="nc">Group</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">json</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="nb">id</span> <span class="o">=</span> <span class="n">json</span><span class="p">[</span><span class="s">'group_id'</span><span class="p">]</span>
<span class="bp">self</span><span class="p">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">json</span><span class="p">[</span><span class="s">'name'</span><span class="p">]</span>
<span class="bp">self</span><span class="p">.</span><span class="n">messages</span> <span class="o">=</span> <span class="p">[]</span>
</code></pre></div></div>
<p>Then we can add a method to <code class="language-plaintext highlighter-rouge">Group</code> to fetch its recent messages:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">get_recent_messages</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">messages</span> <span class="o">=</span> <span class="n">get_messages_for_group</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="nb">id</span><span class="p">)</span>
<span class="c1"># Convert each message to our object
</span> <span class="k">for</span> <span class="n">message</span> <span class="ow">in</span> <span class="n">messages</span><span class="p">:</span>
<span class="n">new_message_object</span> <span class="o">=</span> <span class="n">Message</span><span class="p">(</span><span class="n">message</span><span class="p">)</span>
<span class="bp">self</span><span class="p">.</span><span class="n">messages</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">new_message_object</span><span class="p">)</span>
</code></pre></div></div>
<p>And then we can use our script to print out the messages for a group:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">json</span>
<span class="kn">import</span> <span class="nn">requests</span>
<span class="n">API_KEY</span> <span class="o">=</span> <span class="s">'YOUR_API_KEY'</span>
<span class="n">BASE_URL</span> <span class="o">=</span> <span class="s">'https://api.groupme.com/v3'</span>
<span class="k">def</span> <span class="nf">api_request</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
<span class="s">'''Returns the data from a request (eg /groups)'''</span>
<span class="n">url</span> <span class="o">=</span> <span class="sa">f</span><span class="s">'</span><span class="si">{</span><span class="n">BASE_URL</span><span class="si">}{</span><span class="n">request</span><span class="si">}</span><span class="s">?token=</span><span class="si">{</span><span class="n">API_KEY</span><span class="si">}</span><span class="s">'</span>
<span class="n">req</span> <span class="o">=</span> <span class="n">requests</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">url</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">req</span><span class="p">.</span><span class="n">content</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">None</span>
<span class="c1"># We only want the data associated with the "response" key
</span> <span class="k">return</span> <span class="n">json</span><span class="p">.</span><span class="n">loads</span><span class="p">(</span><span class="n">req</span><span class="p">.</span><span class="n">content</span><span class="p">)[</span><span class="s">'response'</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">get_messages_for_group</span><span class="p">(</span><span class="n">group_id</span><span class="p">):</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">api_request</span><span class="p">(</span><span class="sa">f</span><span class="s">'/groups/</span><span class="si">{</span><span class="n">group_id</span><span class="si">}</span><span class="s">/messages'</span><span class="p">)</span>
<span class="c1"># Just return the messages and none of the metadata
</span> <span class="k">return</span> <span class="n">response</span><span class="p">[</span><span class="s">'messages'</span><span class="p">]</span>
<span class="k">class</span> <span class="nc">Message</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">json</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">user_id</span> <span class="o">=</span> <span class="n">json</span><span class="p">[</span><span class="s">'user_id'</span><span class="p">]</span>
<span class="bp">self</span><span class="p">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">json</span><span class="p">[</span><span class="s">'name'</span><span class="p">]</span>
<span class="bp">self</span><span class="p">.</span><span class="n">text</span> <span class="o">=</span> <span class="n">json</span><span class="p">[</span><span class="s">'text'</span><span class="p">]</span>
<span class="k">class</span> <span class="nc">Group</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">json</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="nb">id</span> <span class="o">=</span> <span class="n">json</span><span class="p">[</span><span class="s">'group_id'</span><span class="p">]</span>
<span class="bp">self</span><span class="p">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">json</span><span class="p">[</span><span class="s">'name'</span><span class="p">]</span>
<span class="bp">self</span><span class="p">.</span><span class="n">messages</span> <span class="o">=</span> <span class="p">[]</span>
<span class="bp">self</span><span class="p">.</span><span class="n">get_recent_messages</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">get_recent_messages</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">messages</span> <span class="o">=</span> <span class="n">get_messages_for_group</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="nb">id</span><span class="p">)</span>
<span class="c1"># Convert each message to our object
</span> <span class="k">for</span> <span class="n">message</span> <span class="ow">in</span> <span class="n">messages</span><span class="p">:</span>
<span class="n">new_message_object</span> <span class="o">=</span> <span class="n">Message</span><span class="p">(</span><span class="n">message</span><span class="p">)</span>
<span class="bp">self</span><span class="p">.</span><span class="n">messages</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">new_message_object</span><span class="p">)</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">groups_json</span> <span class="o">=</span> <span class="n">api_request</span><span class="p">(</span><span class="s">'/groups'</span><span class="p">)</span>
<span class="n">my_group</span> <span class="o">=</span> <span class="n">Group</span><span class="p">(</span><span class="n">groups_json</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
<span class="k">for</span> <span class="n">message</span> <span class="ow">in</span> <span class="n">my_group</span><span class="p">.</span><span class="n">messages</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="n">message</span><span class="p">.</span><span class="n">text</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">'-- </span><span class="si">{</span><span class="n">message</span><span class="p">.</span><span class="n">name</span><span class="si">}</span><span class="s">'</span><span class="p">)</span>
<span class="k">print</span><span class="p">()</span>
</code></pre></div></div>
<p>The result is the most recent messages for our most recent group:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Hello everyone!
-- Johnny Test
Hi guys I had a question about using @classmethod
-- Alexa Jones
Wow great work!
-- Katie Alendra
</code></pre></div></div>
<p>We have the data in a manageable format, so it’s time to start formatting it in a readable form.</p>
<h1 id="using-jinja-templates">Using Jinja Templates</h1>
<p>We’ve come a long way so far! First, we learned how to make HTTP GET requests to a server. Then, we used GroupMe’s API docs to fetch data about different groups and messages, and then we created Python classes to better organize our information. Let’s create a <a href="http://jinja.pocoo.org/docs/2.10/">Jinja</a> template to print out our data.</p>
<h2 id="create-the-template">Create the Template</h2>
<p>First, I’ll make a <code class="language-plaintext highlighter-rouge">group.html</code> file that has the framework of I want the web page to look like:</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><body></span>
<span class="nt"><h1></span>GROUP NAME<span class="nt"></h1></span>
<span class="nt"><br</span> <span class="nt">/></span>
<span class="c"><!-- Repeat for every message --></span>
<span class="nt"><p><b></span>MESSAGE CONTENT<span class="nt"></b></span> <span class="ni">&mdash;</span> NAME<span class="nt"></p></span>
<span class="nt"></body></span>
</code></pre></div></div>
<p>With Jinja, variables are inserted into the template using <code class="language-plaintext highlighter-rouge">{{ variable_name }}</code>, and logic statements have a form such as:</p>
<div class="language-jinja highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="cp">{%</span> <span class="k">if</span> <span class="nv">should_display</span> <span class="cp">%}</span>
<span class="nt"><p></span>This message should be displayed<span class="nt"></p></span>
<span class="cp">{%</span> <span class="k">endif</span> <span class="cp">%}</span>
</code></pre></div></div>
<p>If we assume that we’ll pass a <code class="language-plaintext highlighter-rouge">Group()</code> instance into our Jinja template with the variable name <code class="language-plaintext highlighter-rouge">group</code>, we can rewrite <code class="language-plaintext highlighter-rouge">group.html</code> as:</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="nt"><body></span>
<span class="nt"><h1></span>{{ group.name }}<span class="nt"></h1></span>
<span class="nt"><br</span> <span class="nt">/></span>
<span class="c"><!-- Repeat for every message --></span>
{% for message in group.messages %}
<span class="nt"><p><b></span>{{ message.text }}<span class="nt"></b></span> <span class="ni">&mdash;</span>{{ message.name }}<span class="nt"></p></span>
{% endfor %}
<span class="nt"></body></span>
</code></pre></div></div>
<p>Note the <code class="language-plaintext highlighter-rouge">{% endif %}</code> and <code class="language-plaintext highlighter-rouge">{% endfor %}</code> in the above snippets; they’re required for all conditionals and loops.</p>
<h2 id="populate-the-template">Populate the Template</h2>
<p>With our template written, let’s go back to our script and add a section to import our template using <code class="language-plaintext highlighter-rouge">jinja2</code>.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s">'group.html'</span><span class="p">,</span> <span class="s">'r'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">contents</span> <span class="o">=</span> <span class="n">f</span><span class="p">.</span><span class="n">read</span><span class="p">()</span>
<span class="n">template</span> <span class="o">=</span> <span class="n">jinja2</span><span class="p">.</span><span class="n">Template</span><span class="p">(</span><span class="n">contents</span><span class="p">)</span>
<span class="n">filled_template</span> <span class="o">=</span> <span class="n">template</span><span class="p">.</span><span class="n">render</span><span class="p">(</span><span class="n">group</span><span class="o">=</span><span class="n">my_group</span><span class="p">)</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s">'output.html'</span><span class="p">,</span> <span class="s">'w'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">f</span><span class="p">.</span><span class="n">write</span><span class="p">(</span><span class="n">filled_template</span><span class="p">)</span>
</code></pre></div></div>
<p>First, we read the contents of our template file. Because we’re only going to use one file, we can just load the text of our template into <code class="language-plaintext highlighter-rouge">jinja2.Template</code>, and then we can render the template by passing our <code class="language-plaintext highlighter-rouge">my_group</code> variable (from our main script) as <code class="language-plaintext highlighter-rouge">group</code>. Finally, we write the contents to <code class="language-plaintext highlighter-rouge">output.html</code> to view it in a browser.</p>
<p>Now we have our full script:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">json</span>
<span class="kn">import</span> <span class="nn">jinja2</span>
<span class="kn">import</span> <span class="nn">requests</span>
<span class="n">API_KEY</span> <span class="o">=</span> <span class="s">'YOUR_API_KEY'</span>
<span class="n">BASE_URL</span> <span class="o">=</span> <span class="s">'https://api.groupme.com/v3'</span>
<span class="k">def</span> <span class="nf">api_request</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
<span class="s">'''Returns the data from a request (eg /groups)'''</span>
<span class="n">url</span> <span class="o">=</span> <span class="sa">f</span><span class="s">'</span><span class="si">{</span><span class="n">BASE_URL</span><span class="si">}{</span><span class="n">request</span><span class="si">}</span><span class="s">?token=</span><span class="si">{</span><span class="n">API_KEY</span><span class="si">}</span><span class="s">'</span>
<span class="n">req</span> <span class="o">=</span> <span class="n">requests</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">url</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">req</span><span class="p">.</span><span class="n">content</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">None</span>
<span class="c1"># We only want the data associated with the "response" key
</span> <span class="k">return</span> <span class="n">json</span><span class="p">.</span><span class="n">loads</span><span class="p">(</span><span class="n">req</span><span class="p">.</span><span class="n">content</span><span class="p">)[</span><span class="s">'response'</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">get_messages_for_group</span><span class="p">(</span><span class="n">group_id</span><span class="p">):</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">api_request</span><span class="p">(</span><span class="sa">f</span><span class="s">'/groups/</span><span class="si">{</span><span class="n">group_id</span><span class="si">}</span><span class="s">/messages'</span><span class="p">)</span>
<span class="c1"># Just return the messages and none of the metadata
</span> <span class="k">return</span> <span class="n">response</span><span class="p">[</span><span class="s">'messages'</span><span class="p">]</span>
<span class="k">class</span> <span class="nc">Message</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">json</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">user_id</span> <span class="o">=</span> <span class="n">json</span><span class="p">[</span><span class="s">'user_id'</span><span class="p">]</span>
<span class="bp">self</span><span class="p">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">json</span><span class="p">[</span><span class="s">'name'</span><span class="p">]</span>
<span class="bp">self</span><span class="p">.</span><span class="n">text</span> <span class="o">=</span> <span class="n">json</span><span class="p">[</span><span class="s">'text'</span><span class="p">]</span>
<span class="k">class</span> <span class="nc">Group</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">json</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="nb">id</span> <span class="o">=</span> <span class="n">json</span><span class="p">[</span><span class="s">'group_id'</span><span class="p">]</span>
<span class="bp">self</span><span class="p">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">json</span><span class="p">[</span><span class="s">'name'</span><span class="p">]</span>
<span class="bp">self</span><span class="p">.</span><span class="n">messages</span> <span class="o">=</span> <span class="p">[]</span>
<span class="bp">self</span><span class="p">.</span><span class="n">get_recent_messages</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">get_recent_messages</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">messages</span> <span class="o">=</span> <span class="n">get_messages_for_group</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="nb">id</span><span class="p">)</span>
<span class="c1"># Convert each message to our object
</span> <span class="k">for</span> <span class="n">message</span> <span class="ow">in</span> <span class="n">messages</span><span class="p">:</span>
<span class="n">new_message_object</span> <span class="o">=</span> <span class="n">Message</span><span class="p">(</span><span class="n">message</span><span class="p">)</span>
<span class="bp">self</span><span class="p">.</span><span class="n">messages</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">new_message_object</span><span class="p">)</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">groups_json</span> <span class="o">=</span> <span class="n">api_request</span><span class="p">(</span><span class="s">'/groups'</span><span class="p">)</span>
<span class="n">my_group</span> <span class="o">=</span> <span class="n">Group</span><span class="p">(</span><span class="n">groups_json</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s">'group.html'</span><span class="p">,</span> <span class="s">'r'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">contents</span> <span class="o">=</span> <span class="n">f</span><span class="p">.</span><span class="n">read</span><span class="p">()</span>
<span class="n">template</span> <span class="o">=</span> <span class="n">jinja2</span><span class="p">.</span><span class="n">Template</span><span class="p">(</span><span class="n">contents</span><span class="p">)</span>
<span class="n">filled_template</span> <span class="o">=</span> <span class="n">template</span><span class="p">.</span><span class="n">render</span><span class="p">(</span><span class="n">group</span><span class="o">=</span><span class="n">my_group</span><span class="p">)</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s">'output.html'</span><span class="p">,</span> <span class="s">'w'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">f</span><span class="p">.</span><span class="n">write</span><span class="p">(</span><span class="n">filled_template</span><span class="p">)</span>
</code></pre></div></div>
<p>Once run, we can view our <code class="language-plaintext highlighter-rouge">output.html</code> in a browser:</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><body></span>
<span class="nt"><h1></span>Python Tips and Tricks<span class="nt"></h1></span>
<span class="nt"><br</span> <span class="nt">/></span>
<span class="c"><!-- Repeat for every message --></span>
<span class="nt"><p><b></span>Hello everyone!<span class="nt"></b></span> <span class="ni">&mdash;</span>Johnny Test<span class="nt"></p></span>
<span class="nt"><p><b></span>Hi guys I had a question about using @classmethod<span class="nt"></b></span> <span class="ni">&mdash;</span>Alexa Jones<span class="nt"></p></span>
<span class="nt"><p><b></span>Wow great work!<span class="nt"></b></span> <span class="ni">&mdash;</span>Katie Alendra<span class="nt"></p></span>
<span class="nt"></body></span>
</code></pre></div></div>
<p><img src="https://nickymarino.com/public/assets/2018/web-apis-and-templates/output-site.png" alt="Output Website" /></p>
<h1 id="conclusion">Conclusion</h1>
<p>We’ve walked through how to access and parse a web API using the requests library, how to represent and organize the API data using Python classes, and how to render the information in a custom format using a Jinja template. Now go create your own cool stuff using APIs!</p>
<hr />
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:url-params" role="doc-endnote">
<p>We won’t in this walkthrough, but ff we needed to pass multiple <a href="https://en.wikipedia.org/wiki/Query_string#Web_forms">parameters</a> in the URL, it’ll look like <code class="language-plaintext highlighter-rouge">v3/...?limit=10&another_param=1000&token=YOUR_ACCESS_TOKEN</code>) <a href="#fnref:url-params" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:json-object" role="doc-endnote">
<p>Typically, <code class="language-plaintext highlighter-rouge">json.loads</code>() returns a dict that can contain more dicts, lists, and values like <code class="language-plaintext highlighter-rouge">None</code>, strings, and integers. Check out the <a href="https://docs.python.org/3/library/json.html">Python docs</a> for examples. <a href="#fnref:json-object" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
Expand Tilde Paths in Bash and Python2018-12-17T00:00:00+00:00https://nickymarino.com/2018/12/17/expand-tilde-paths<p>Sometimes it’s necessary to reference files in a script using <code class="language-plaintext highlighter-rouge">~</code>. For example, if you want to schedule a <a href="https://en.wikipedia.org/wiki/Cron">cron job</a> to run a script in a folder and place the results in the same folder, it’s helpful to use absolute referencing of the files in the script.</p>
<h2 id="bash">Bash</h2>
<p>Here’s my first attempt to append to a file:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ ./my_folder/run.sh >> "~/my_folder/output.txt"
-bash: ~/my_folder/output.txt: No such file or directory
</code></pre></div></div>
<p>The issue with the above line is that the <code class="language-plaintext highlighter-rouge">~</code> is not expanded to the home directory (such as <code class="language-plaintext highlighter-rouge">/home/username/</code>) because it is inside the quotes. To <a href="https://stackoverflow.com/questions/36623731/bash-script-echo-returns-error-no-such-file-or-directory">fix</a> this, move the path outside of the quotes, but leave the filename in single quotes (to escape the <code class="language-plaintext highlighter-rouge">.</code> in the extension):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ ./my_folder/run.sh >> ~/my_folder/'output.txt'
</code></pre></div></div>
<h2 id="python">Python</h2>
<p>I encountered a similar issue in Python:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>>>> with open('~/my_folder/output.txt', 'r') as f:
... contents = f.read()
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
FileNotFoundError: [Errno 2] No such file or directory: '~/my_folder/output.txt'
</code></pre></div></div>
<p>This can be <a href="https://stackoverflow.com/questions/2057045/pythons-os-makedirs-doesnt-understand-in-my-path">fixed</a> using <a href="https://docs.python.org/3/library/os.path.html#os.path.expanduser"><code class="language-plaintext highlighter-rouge">os.path.expanduser(path)</code></a>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>>>> import os
>>> filename = os.path.expanduser('~/my_folder/output.txt')
>>> with open(filename, 'r') as f:
... contents = f.read()
...
>>> print(contents)
10
</code></pre></div></div>
iPad Screenshots2018-11-15T00:00:00+00:00https://nickymarino.com/2018/11/15/ipad-screenshots<p>Dr. Drang:</p>
<blockquote>
<p>On the iPad, ⇧⌘3 captures the whole screen, just like the Mac (and just like capturing with the top and
volume up buttons). The ⇧⌘4 shortcut also captures the whole screen, but in a neat analogy to the Mac,
it immediately puts you into editing mode so you can crop the capture down to a smaller size.</p>
</blockquote>
<p>I don’t find these keyboard shortcuts surprising, but it <em>is</em> surprising that I never thought to try it on
an iPad. With the <a href="https://9to5mac.com/2018/07/17/macos-mojave-how-to-use-new-screenshot-and-screencast-tools-without-grab/">new screenshot tool</a>
in macOS Mojave, I wonder what other features will reach parity on macOS and iOS in the future.</p>
Removing Local Git Branches That Aren't 'master'2018-09-07T00:00:00+00:00https://nickymarino.com/2018/09/07/remove-non-master-local-branches<p>Every so often, I’ll want to delete all of my local branches for a repository that aren’t the <code class="language-plaintext highlighter-rouge">master</code> branch. An easy command to do this is:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ git branch | grep -v "master" | xargs git branch -d
</code></pre></div></div>
<p>(If you want to keep multiple branches, such as <code class="language-plaintext highlighter-rouge">master</code> and <code class="language-plaintext highlighter-rouge">develop</code>, you can chain them together using <code class="language-plaintext highlighter-rouge">grep -v "master\|develop"</code>)</p>
<p><code class="language-plaintext highlighter-rouge">git branch</code> lists all of the local branches for the repo, <code class="language-plaintext highlighter-rouge">grep -v</code> prints all of the lines from the previous command that don’t match “master”, and <code class="language-plaintext highlighter-rouge">xargs</code> takes each line from the previous output and runs <code class="language-plaintext highlighter-rouge">git branch -d <output_line></code>.</p>
<p>I recommend using <code class="language-plaintext highlighter-rouge">-d</code> rather than <code class="language-plaintext highlighter-rouge">-D</code> in case git recommends not deleting the branch.</p>
How To Change a Git Repo's Authentication Protocol2018-08-24T00:00:00+00:00https://nickymarino.com/2018/08/24/change-git-repo-auth-protocol<h2 id="https-to-ssh-key">HTTPS to SSH Key</h2>
<p>Often I need to change a git repository to use an SSH key instead of my username and password to authenticate with the remote server. In order to do so, type the following in the repository’s folder on your machine:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ git config remote.origin.url git@github.com:username/repository_name.git
</code></pre></div></div>
<p>(Make sure to include the <code class="language-plaintext highlighter-rouge">.git</code> at the end of the repository name.)</p>
<h2 id="ssh-key-to-https">SSH Key to HTTPS</h2>
<p>In order to change it to do the reverse, type:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ git config remote.origin.url https://github.com/username/repository
</code></pre></div></div>
Auto Login for PuTTY (Windows)2018-08-23T00:00:00+00:00https://nickymarino.com/2018/08/23/auto-login-for-putty-windows<p>Often I find myself wanting to have an easy way to SSH into a server on a Windows PC. Unfortunately, SSH keys on Windows can often be a challenge, but there’s an easy way to have PuTTY connect without needing to type in a password every time.</p>
<p>To create a shortcut for a PuTTY connection to automatically log in, you only need two things: the name of the profile (in PuTTY) that has the connection and appearance settings, and the password to your account (for the server). Right click on the desktop to create a new shortcut, then for the link type:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>"C:\Program Files (x86)\PuTTY\putty.exe" -load "<profileName>" -pw "<password>"
</code></pre></div></div>
<p>If you saved PuTTY to a different location other than <code class="language-plaintext highlighter-rouge">Program Files (x86)</code>, then you’ll also need to change the location of <code class="language-plaintext highlighter-rouge">putty.exe</code> in the command above.</p>
<p>Once you’ve created the shortcut, you can pin it to the taskbar or the start menu for easy access!</p>
<p><em>These instructions were inspired by the instructions for the Purdue ECE 264 <a href="https://engineering.purdue.edu/ece264/16au/setup">course page</a></em>.</p>
Pointing a Github Pages Repo to a Hover Domain2018-07-29T00:00:00+00:00https://nickymarino.com/2018/07/29/hover-github-pages<p>My blog is currently hosted using <a href="https://pages.github.com/">GitHub Pages</a>—which is a great way to host your static site or blog for free—by linking it to my custom domain that I purchased through <a href="https://www.hover.com/">Hover</a>. While both of these services are amazing, connecting the two required many open tabs and several waiting periods. This post will explain the steps needed to point a GitHub Pages repo to a custom domain on Hover.</p>
<h2 id="preflight-check">Preflight Check</h2>
<p>Before connecting GitHub Pages to a custom domain, I first updated my blog on my repository <a href="https://github.com/nickymarino/nickymarino.github.io">nickymarino.github.io</a>, and checked that it was displaying properly at its default website (normally https://nickymarino.github.io).</p>
<h2 id="link-the-repo-to-the-domain">Link the Repo to the Domain</h2>
<p>First you need to update your repository with your custom domain. In the settings for the repo, enter the domain in the “Custom domain” in the GitHub Pages section.</p>
<p><img src="https://nickymarino.com/public/assets/2018/hover-github-pages/settings.png" alt="GitHub Pages settings for the repo" /></p>
<h2 id="a-records-on-hover">A Records on Hover</h2>
<p>The next step is to configure Hover. Find <a href="https://help.github.com/articles/setting-up-an-apex-domain/#configuring-a-records-with-your-dns-provider">GitHub’s current list of IP addresses</a> to create records with. Then, go to your <a href="https://www.hover.com/">Hover</a> account, select your domain, and go to the DNS tab) to add to the DNS tab on Hover. At the time of writing, these are:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>185.199.108.153
185.199.109.153
185.199.110.153
185.199.111.153
</code></pre></div></div>
<p>Then, go to your <a href="https://www.hover.com/">Hover</a> account, select your domain, and go to the DNS tab. Delete any DNS records that have an “A” under “Records”.</p>
<p>For each IP address on GitHub’s help pages, add a DNS record. For each, the “Type” will be <code class="language-plaintext highlighter-rouge">A</code>, the “Hostname” will be <code class="language-plaintext highlighter-rouge">@</code>, and the “TTL” can be left as the default value.</p>
<p><img src="https://nickymarino.com/public/assets/2018/hover-github-pages/hover.png" alt="Hover DNS settings" /></p>
<p>It may take several hours (or up to about a day) for the changes to take effect. Take a break, get some sleep, and then come back to your domain to make sure everything’s working. Now we can enforce HTTPS!</p>
<h2 id="create-https-certificate">Create HTTPS certificate</h2>
<p>If you head back to your repo’s settings page to enforce HTTPS, you might see the following “not yet available” error:</p>
<p><img src="https://nickymarino.com/public/assets/2018/hover-github-pages/error.png" alt="GitHub Pages HTTPS error" /></p>
<p>Per <a href="https://help.github.com/articles/troubleshooting-custom-domains/#https-errors">GitHub’s troubleshooting page</a>, you need to remove and then re-add your custom domain for your repository. Wait around 24 hours for the certificate to be generated, and you should be good to go!</p>
A New Look2018-07-23T00:00:00+00:00https://nickymarino.com/2018/07/23/new-look<p>I’ve owned this domain (<a href="nickymarino.com">nickymarino.com</a>) for roughly three years now, and so far I’ve only used it as a resume/portfolio site. I’ve finally found a theme that I both appreciate and can spend time modifying to fit my needs.</p>
<p>My goal is to start writing (and podcasting!) more often, whether it’s a technical detail I found interesting, an overcome challenge I want to record for when I encounter it again, or anything else I can think of.</p>
<p>I have a few ideas up my sleeve.</p>
<p>You can stay updated via <a href="http://nickymarino.com/atom.xml">the site feed</a> or on <a href="https://twitter.com/NickyMarino_com">Twitter</a>.</p>
Let's Use LaTeX!2017-09-07T00:00:00+00:00https://nickymarino.com/2017/09/07/lets-use-latex<p><a href="https://www.latex-project.org/">LaTeX</a> is a beautiful documentation system. It’s similar to Markdown<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup>, but has many more features and is commonly used for academic papers and other publications that require a lot of equations. In this quick how to, we cover how to install LaTeX and use Visual Studio Code as an editor/previewer.</p>
<h1 id="installing-latex">Installing LaTeX</h1>
<p>I recommend downloading <a href="https://www.tug.org/texlive/">TeX Live</a> for Windows/Linux, and <a href="https://www.tug.org/mactex/">MacTex</a> for macOS.</p>
<h1 id="setting-up-our-editor">Setting up our editor</h1>
<p>If you haven’t already, install <a href="https://code.visualstudio.com/">Visual Studio Code</a> and go through a <a href="https://code.visualstudio.com/docs/introvideos/basics">tutorial</a>. Then, we need to install our extension for LaTeX itself. Head over to <a href="https://marketplace.visualstudio.com/items?itemName=James-Yu.latex-workshop">LaTeX Workshop</a> and click install.</p>
<h1 id="using-latex">Using LaTeX</h1>
<p>Now that we have our editor setup, we can write our first project. All LaTeX documents have a (non-blank) file that ends with <code class="language-plaintext highlighter-rouge">.tex</code>, which is the “main” file that has all of the text of the document. Since LaTeX usually generates more files (such as .log, etc.) while building the document, it’s recommended that every document you want to write has its own folder.</p>
<p>For starters, create a file named <code class="language-plaintext highlighter-rouge">example.tex</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>\documentclass{article}
% General document formatting
\usepackage[margin=0.7in]{geometry}
\usepackage[parfill]{parskip}
\usepackage[utf8]{inputenc}
\begin{document}
% Put your text here
This is starter text.
\end{document}
</code></pre></div></div>
<p>Press <code class="language-plaintext highlighter-rouge">Ctrl-Alt-B</code> to build your project (or use the Command Palette), then <code class="language-plaintext highlighter-rouge">Ctrl-Alt-T</code> to view the pdf in a new tab. The end result should look like this:</p>
<p><img src="https://thepracticaldev.s3.amazonaws.com/i/h7udmn31hzf8cdc3zcwb.PNG" alt="picture" /></p>
<h1 id="conclusion">Conclusion</h1>
<p>LaTeX and VSCode are a great combination that you can use to write beautiful reports and papers. Check out a <a href="https://www.latex-tutorial.com/tutorials/">tutorial</a> or <a href="http://www.rpi.edu/dept/arc/training/latex/class-slides-pc.pdf">two</a> to realize the full experience LaTeX has to offer.</p>
<p><em>Edit: <a href="https://dev.to/fannyvieira">Fanny</a> recommends another <a href="https://github.com/LewisVo/Begin-Latex-in-minutes">great tutorial</a> as well.</em></p>
<p><em>Edit 2: Fixed a tutorial link.</em></p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>Well, this depends on your definition of “similar”, but I feel it is. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>