Wednesday, March 31, 2021

Zettelkasten 3-for-1: book review, how-to summary & software survey

A few thoughts after reading How to Take Smart Notes by Sönke Ahrens. It doesn't mention it in the title, but this book is about a note-taking approach called Zettelkasten.

The book

I heard about Zettelkasten on Twitter and was intrigued - a kind of analog note taking technique that sounded like a wiki made of index cards. There are articles, blog posts and software out there that "do Zettelkasten" but nothing that made me confident I understood the idea properly. So I turned to this book, which is nice and short at ~150 pages (though it could be shorter still, see below).

The subtitle is One Simple Technique to Boost Writing, Learning and Thinking - for Students, Academics and Nonfiction Book Writers. And there is definitely a strong emphasis on the benefits to academics who spend their time researching and publishing. I don't fit that mold, or even fall into the student or nonfiction book writer categories. But I do suffer from "read an interesting nonfiction book but can't remember much about it" and would love to remedy that. So I read this book with that objective in mind.

How to Take Smart Notes is definitely worth reading it you want to understand Zettelkasten. And I think using a Zettelkasten is probably going to be valuable for me even though I am not a student/academic/nonfiction book writer.

The book explains an excellent overarching process that is necessary to work effectively. Simply installing Zettelkasten software won't cut it. In this regard the book is vastly superior to any article or blog posts I looked at when it comes to explaining this.

I do have some criticisms. Firstly, the focus is primarily on academics. This is probably a result of the author's background. Still, it would be helpful to have some material in there more oriented towards students, nonfiction book writers and more casual users. Secondly, there's too much repetition which became irritating. I don't need to hear the same claim made seven times - padding? Lastly, there are not enough examples, and what examples there are are slim. This is a shame because nothing is better than a good example for "getting" how something is done.

How Zettelkasten works

You should probably read the book, but here is my description of the mechanics of taking notes the Zettelkasten way. There are in fact two related aspects - taking notes in the first place and then the writing process that follows.

Note taking process

  • Take working notes when reading to learn. These notes can be brief and the medium used doesn’t matter too much as these are not long-lived. That said, using the same approach every time is best (see comment on decision fatigue below). These notes are basically an aide-memoire that will later allow you to produce permanent notes in your Zettelkasten.
  • Make "literature" notes. The book recommends you use (Zotero) to record the bibliographic detail of what you read (Zotero understands not just books but also articles, web pages, movies etc). Notes here should be short, related to your own lines of enquiry, things you don’t want to forget about the book etc. Zotero was new to me, although it looks pretty nice. I think it's exactly the right tool if your end product is a work that requires you to cite your sources and produce a bibliography. If not, I think you could use something simpler or maybe even skip this step and simply work with permanent notes as described in the next step.
  • Make permanent (idea/topic) notes. The book suggests (Zettelkasten) for this. (Sadly this software struck me as awful, but there are alternatives, see below). Ideally you do this within a day of making working notes so you don't forget what they meant. The recommended technique is to create notes about what interests you as you’re trying to develop your own ideas, not collect other peoples. You should write one entry (note) per discrete idea. You should make the notes high quality - full sentences, sources cited, ready to be used in a larger written piece. They need to make sense when you reread them later out of context. Avoid the temptation to copy/paste content directly from what you've read - instead really work on translating things into your own words and elaborating on the material. This deepens your understanding and helps with recall. Once done, consider the working notes "processed" and no longer needed.
  • Enrich the note. This is simply linking your new note to existing notes that seem related (and potentially linking those linked notes back to your new note). One distinct Zettelkasten concept that's a bit different to the way hypertext links work is the idea of sequences. In the original index card analog approach, one would build a sequence of notes as an idea developed. The book talks about placing your new notes "behind" an appropriate note. With a software based Zettelkasten, relationships between notes are built with links. It seems to me that sequences are just a special class of link, using some discipline to organize them that way and avoiding the temptation to link everything all together. There's also the idea of tagging your notes thematically (#note-taking, #productivity etc) but I'm not sure how much more useful that is beyond well chosen links.
  • Use the notes in your Zettelkasten to drive what you research next. Probably more relevant when research and publication is your job. The idea of this is that by looking at what is there already, identifying gaps and questions or ideas to explore, noticing clusters of notes and so on you can use the Zettelkasten to inspire and even drive your learning.
  • Project-specific notes. This is not a step in the process, but a reminder to recognize that these are a separate class of note and not to confuse them with bibliographic or permanent (idea/topic) notes. Keep them somewhere separate. These are things like to-do lists, feedback, comments, ideas, outlines and the drafts of things you are writing.

Writing process

As your Zettelkasten-based notes build, you can use the following simple approach to writing.

  • Identify a topic to write about. Eventually your Zettelkasten will have grown to a point where you have developed ideas for something interesting to write about. This may come to mind spontaneously or you can mine for it by "surfing the Zettelkasten." The beauty of this is that the idea is based on a bunch of interrelated notes you already have that come from documented sources. If you've organized your notes well, with good links between them, then most of them are likely to already be in sequences that you can turn into a paper, blog post or whatever you are writing.
  • Turn your notes into a rough draft. You probably won’t just copy and paste your notes from your Zettelkasten to your word processor. Instead you translate and elaborate upon them to make them into a cohesive piece of writing. Still, because the notes are organized and well written already, this should be much easier than starting with a blank page and having to research your topic first.
  • Edit and proofread. And you’re done.

Why this works

The book gives several convincing reasons why using a Zettelkasten and the accompanying process is effective.

For me, one of the best insights here was that there are different kinds of notes and they should be treated in different ways and kept in different places. Previously I had always been trying to figure out the right technique or the perfect tool for keeping notes. All the notes. But in fact the idea of having working notes, literature notes, permanent notes and project notes makes a lot of sense. In particular for me, separating the permanent (idea/topic) notes from project notes is something I had not thought of before and tended to muddle them up together.

Another interesting thing was how this turns the typical approach to writing on its head. The usual advice is to:

  1. come up with a topic
  2. research the topic
  3. write about the topic

In a Zettelkasten world you don't have to guess at what a good topic is. You've been actively exploring interesting stuff and organizing it in such a way that you probably have several topics you could pick. And once you do, there's a series of related notes that can form the basis of your piece.

A couple other ideas about why Zettelkasten is effective that rang true for me were:

  • the process encourages you to more critically assess what you read
  • the writing of permanent notes where you translate and elaborate upon what you read makes you gain a deeper understanding of the material
  • the one-note-per-idea structure forces a reasonable size upon each note, keeps things granular, yet you can build larger concepts through linking a series together
  • organizing notes this way should help you see relationships between disparate ideas, and the connecting of things from different disciplines is often where innovation occurs
  • having a simple defined approach to note taking reduces decision fatigue about how to take notes, where to take notes and so on
  • another thing the process helps with is getting stuff out of your short-term working memory and into a trusted system; this is like GTD but for notes rather than tasks

Software

As mentioned above, I didn't find the recommended (Zettelkasten) software all that appealing for several reasons:

  • it seems a bit buggy
  • the UI is, er, not state of the art
  • proprietary and rather limited note formatting, e.g. [f]i am bold[/f], [k]I am italic[/k], [z 2]I link to note #2[/z]
  • data is stored in a proprietary binary format
  • German-language documentation requires Google Translate
  • no recent release suggests not very active development

I had a look around for better options with an eye to the following characteristics:

  • low friction, simple solution
  • no proprietary elements (markup, file format etc.)
  • ideally open source, but would pay for a good product too if it met the other criteria
  • ideally markdown based for authoring content
  • high quality experience when reading and navigating between notes - not just an editor
  • automatic management of backlinks between notes
  • ability to sync the Zettelkasten data between different devices
  • bonus points for supporting the concept of note sequences and tagging

Here's what I found.

Zettlr

Zettlr is a Markdown editor that claims Zettelkasten support. It's nice but it is not quite right either:

  • doesn't generate backlinks (i.e. in note A if I make a link to note B, I want note B to automatically have a link back to note A))
  • links and tags are really just a search, and kind of a hokey one at that
  • more of a Markdown editor than viewer, so you have a weird hybrid of raw markdown and partially rendered content, plus the ability to browse your notes is not great

Roam

I briefly looked at Roam having heard it was the new Zettelkasten hotness, but it's paid, hosted and looks overly complicated.

StackEdit

Next I had a look at StackEdit which is a web-based Markdown editor based on the one used by Stack Overflow. It looks really polished and has some potentially nice features like integrations with Google Drive, Dropbox and GitHub. It might be promising if you hacked at it, but out of the box it has some issues:

  • no native understanding of Zettelkasten concepts, so it doesn't know about sequences and backlinks
  • most frustratingly it doesn't support navgation between documents - there has been some people asking about using it as a wiki in the past and it seems like at one point it might have been possible with some customization, but now this no longer appears straightforward
  • it's still (perhaps) the most likely candidate to fork and change into a dedicated Zettelkasten app if you want to (I didn't want to…)

Atom

Wondered if Atom could make a serviceable Zettelkasten app. It does Markdown preview and editing fairly well. However the showstopper is that links to other .md files like [Some Link](some-link.md) do not work. Like StackEdit, it might be good platform to build on to create a Zettelkasten app.

Chrome Markdown Viewer + BYOE

Tried out the Chrome Markdown Viewer plugin and pleased to discover it does let you navigate between different .md files with links such as [Some Link](some-link.md)

Obviously this is just a nice way to read markdown, so you have to BYOE (bring your own editor). I just continued to use Atom since it is my default for basic editing.

Initially I was reading content with URLs like file:///Users/jarcher/work/zettelkasten/zettelkasten.md but that felt kinda ugly so I served them up on a simple HTTP server and accessed them at http://localhost:8080/zettelkasten.md – that looked nicer and the links between files continued to work.

This works OK, but is obviously not a solution that understands Zettelkasten – you have to figure out how you want to use this toolset to deal with note sequences, links to related notes, backlinks etc. That requires some discipline and it would be possible to mess things up (especially backlinks) although perhaps no more so than a manual system based on index cards. Still, one of the benefits of using a software system should be that it helps with link maintenance. Also your note editor and reader are two separate tools here, so the UX is less pleasant.

Obsidian

For me, this is the one. Obsidian really checks a lot of boxes:

  • free for personal use
  • even the early version of the app is nicely polished
  • view/browse & edit in same app
  • content is markdown
  • supports #tags
  • supports links between documents both in the markdown style [Some Link](some-link.md) and WikiLinks style [[Some Link]] - I cannot really see why I would choose to use the WikiLinks style ever - both seem functionally equivalent but WikiLinks are not standard markdown
  • backlinks work nicely

I have this set up so that my files are in a Dropbox folder which gives me a simple device sync solution.

Wednesday, August 31, 2016

Scheduled Heroku Dyno restarts

Heroku Dynos get restarted once a day.
The cycling happens once every 24 hours (plus up to 216 random minutes, to prevent every dyno for an application from restarting at the same time). Manual restarts (heroku ps:restart) and releases (deploys or changing config vars) will reset this 24 hour period.
This is fine if you are running multiple dynos. But if you were running an app you didn't want to cycle during the day but only had one dyno (I know, I know, don't ask...) there's a quick hack you can use to instead control the time of restart so that it's done during off hours (3am or whatever works for you app's usage patterns).

You can use the Heroku API to perform a restart like below using a simple curl call. If you schedule that curl call for once-daily execution using the Heroku Scheduler add-on then you can have a reliable restart of of your app and a 24 hour runway of no unexpected cycling.

Here's the details:

# install the oauth plugin
$ heroku plugins:install heroku-cli-oauth

# create a new authorization
$ heroku authorizations:create -d "Platform API example token"
Creating OAuth Authorization... done
Client:      <none>
ID:          830d64b9-fd52-4d95-<redacted>
Description: Platform API example token
Scope:       global
Token:       1d82fc23-<redacted>

# use the new authorization token from above to interact with the API, e.g.
$ curl https://api.heroku.com/apps/ -H "Accept: application/vnd.heroku+json; version=3" -H "Authorization: Bearer 1d82fc23-<redacted>"

# so to restart dynos for an app
$ curl -X DELETE https://api.heroku.com/apps/kountdown/dynos -H "Content-Type:application/json" -H "Accept: application/vnd.heroku+json; version=3" -H "Authorization: Bearer 1d82fc23-<redacted>"

So just stick that curl command into the Heroku Scheduler for the time you want and Bob's your uncle.

More on the Heroku API can be found here.

It all pays the same (or does it?)

[I originally posted this on LinkedIn, but reposting here to have a copy on my personal blog.]

Several jobs and quite a few years ago, I was venting about some work related frustration to a friend. Despite being someone who, like me, cared a great deal about doing the right things and doing them well, he had always been more sanguine than me in the face of perceived corporate wrongheadedness.

“It all pays the same,” he said. “If you’ve told them why you think it’s the wrong thing to do and they still insist then you might as well reconcile yourself to it.” It was good advice, even if it was hard to follow.

Over the years I’ve relayed this “it all pays the same” idea to many people who’ve expressed frustration at some of the things they’ve been asked to do at work. It’s a useful little trick, just like the 10-10-10 rule for decision making. It usually gets a wry smile even if it doesn’t really fix the situation that has them all wound up.

Recently though, I’ve developed an extended interpretation. It remains true that you are going to get paid the same whether you’re working on something you believe is going to save the world or something that strikes you as an egregious waste of human talent.

The difference between the two extremes though is the price you pay psychologically to reconcile yourself to working on something your heart isn’t in. I’d say working on something you don't genuinely care about is like paying a tax — a "meaninglessness tax” if you will.

Over the years I’ve written software that’s done many different things. Here’s just a few examples of things the software I’ve worked on has done:


  • organized emergency healthcare for people on vacation
  • managed order processing for discounted menswear
  • helped conduct clinical trials for promising new cancer drugs
  • sold e-books with titles like “6-pack abs in 6 weeks!”
  • facilitated the placement of billions of ads in browsers

You probably don’t need any help picking out the ones with a higher meaninglessness tax.

Job satisfaction requires more than working on something meaningful of course. It needs to be intellectually interesting, to offer opportunities for learning and growth, appropriate amounts of autonomy, flexibility and work/life balance. It needs great colleagues and fair pay; it needs a clear vision, good leadership and the ability to get work done without bureaucracy stifling your efforts. It might even need perks like free food and free beer on a Friday afternoon. But all those things aside, what it really needs most, I think, is a sense of meaningfulness.

----

Have you ever looked in your rear view mirror only to see the vehicle behind closing the gap at an alarming pace? Worse still, the driver is oblivious to the situation, focusing solely on their phone? For several years now, distracted driving has been a worse problem than drunk driving.

At my current job with Katasi I am helping bring to market a real solution to this problem. My meaninglessness tax rate is literally 0%. And so while previous work may have paid the same, I feel much better off now that I again have the chance to work on something so necessary and meaningful.

At Katasi we plan to be hiring soon. If you’d like to work with some great people, on a really meaningful product please get in touch and I can tell you more. Or take a look at http://katasi.com/ or this piece featuring us on CNN.

Monday, January 11, 2016

Complex Systems

Borrowed from the excellent book, The Personal MBA

Gall’s Law
A complex system that works is invariably found to have evolved from a simple system that worked. The inverse proposition also appears to be true: a complex system designed from scratch never works and cannot be made to work You have to start over, beginning with a simple system.
-- John Gall, Systems Theorist

Complex systems are full of variables and interdependencies that must be arranged just right in order to function. Complex systems designed from scratch will never work in the real world, since they haven’t been subject to environmental selection forces while being designed. Uncertainty ensures that you will never be able to anticipate all of these interdependencies and variables in advance, so a complex system built from scratch will continually fail in all sorts of unexpected ways. 

Wednesday, December 10, 2014

I was on call for a week...and it didn't entirely suck!

Yup, it didn't suck. In fact it was actually pretty good -- I'd go so far as to call it kind of exciting and certainly educational. If you've been (or imagined being) on 24/7 call for a week for a high traffic internet service and think I sound insane, allow me to explain...

Earlier this year I started a new job as a software engineer at Sovrn in Boulder, CO.
“sovrn is an advocate of and partner to almost 20,000 publishers across the independent web, representing more than a million sites, who use our tools, services and analytics to grow their audience, engage their readers, and monetize their site.”
In plainer English this means we offer technology so websites can display ads and earn money. Although I personally think it would be great if more content-based websites could sustain themselves through contributions from their visitors like Wikipedia and Brainpickings do, the reality is that revenue from advertising is a more pragmatic revenue model for many website operators. In fact advertising revenue helps power much of the internet you and I use every day.

We at sovrn work at serious scale: according to Quantcast, in October of 2014 sovrn ranked as the 4th largest ad network in the world and the 3rd largest in the US.

This means we move billions of records of data all day every day from multiple datacenters to a central collection point for processing and analysis. This constant flow of data has to cope with the inevitable network and hardware issues that arise and ultimately transform data into warehouse and near-realtime reporting data stores.

Interruptions to any part of the system can cause a variety of issues, from delayed data through capacity challenges and loss of revenue for our customers. One facet of maximizing uptime and dealing with service interruptions is a sophisticated monitoring and alerting system that functions continually, necessitating on call engineers with software development, data management and enterprise IT skill sets.

My first exposure to being on call recently ended and I really did enjoy it. Although serving adverts might sound straightforward, there's a fascinating degree of sophistication involved, and when doing it at scale the problems only get more interesting.

Up until my on call week my understanding of the "big picture" of our operations was limited, having focused primarily on the needs of the team I'm part of. One of the great things about my on call experience was how it gave me a much greater appreciation for how everything fit together, and some exposure to the operational aspects of the data processing pipeline and big data toolset we employ at sovrn. (For the curious we're using Kafka, Mirrormaker, Zookeeper, Camus, Storm, Cassandra, Hadoop and more.)

Besides getting to see all of this stuff hum along in production, there's a definite air of excitement to dealing with an incident. We use a small set of tools to manage our on call duties including Icinga (for system monitoring), VictorOps (for managing on schedules and messaging on-call engineers), HipChat (we use a dedicated channel for production issues which helps keep all interested parties informed and allows multiple participants work an incident collaboratively) as well as a wiki for knowledge-base articles.

I've worked in jobs before where the software engineers didn't get anywhere near production -- primarily due to regulatory considerations necessitating a strong separation between development and operations. Although those separations may help address fraud and other similar concerns, they inhibit other very positive things besides the excitement and "big picture" comprehension I've already mentioned.

First, there's a definite camaraderie that emerges from trying to figure out what's going on when you're getting alert after alert one evening on a weekend and have to ask colleagues to help. This necessitates a level of communication and cooperation across teams that might not otherwise happen all that often and is definitely a very positive thing.

Secondly, seeing how your code responds in production is a phenomenal feedback loop for software engineers. You have a lot more skin in the game when you and your colleagues will be receiving alerts for failing systems. Suddenly great logging and debugging characteristics are first class concerns. Nothing will focus the mind on the need for writing high quality, easy to support code quite like this.

Hopefully now that explains my viewpoint and you no longer think I'm completely mad...


Wednesday, June 11, 2014

Oh CenturyLink, thou art a source of much mirth...

Had some trouble getting a connection over VPN for my new job. This resulted in much to-ing and fro-ing with their elite "Internet Escalation Team" (like the Apple store's Genius Bar with less genius, if you see what I mean...)

This exchange yielded several hilarious gems, not least of which:

Honestly Jon, you are FAR more the expert than any of us here, we have no training except to help customers find the application forwarding section in the modem under the advanced setup.

and
We don't have folks “in the know” about VPN stuff.Compare it to the car dealer not really having a way to trouble shoot an “aftermarket” NOS kit someone installed on a car.Like how a realtor can't troubleshoot the new outdoor pool construction being done on a house they sold , that didn't have one at the time of sale. Does your helpdesk have a specific item to address? I don't think we can help you with your customization of the service. VPN is beyond our knowledge.

Thank goodness I had some idea what I was doing and was able to figure out that the problem was the modem I had. After proving that my VPN connection could be established from a neighbor's house who also used CenturyLink as their ISP I managed to get them to send me a replacement and all was well.

Friday, May 30, 2014

Note to self: bash and git setup

Some notes on my bash and git setup

As much as I like IntelliJ and its git integration, it makes me feel like I'm just dealing with the tip of the git-iceberg. So I like to try and use the command line as much as possible as a way to increase my understanding of git and all the mad-clever things it can do. That's led me to want command line completion, including for git-flow, (and if you're not using git-flow, why? ;-) visibility of what branch I'm on in my prompt and a few aliases and other configuration items to make things easier. It's actually not all that much but it's made a big difference to me.

Here's what the salient parts of my ~/.bash_profile and ~/.gitconfig look like:


The git ls alias is a succinct answer to, "What's been committed lately?" at a high level. It simply lists the hash, how long ago it was committed, by whom and the commit message. I find it useful with --no-merges (which unsurprisingly omits merge commits from the output) and --author=<author> to limit things to just my work. It helps you answer the questions, "When did I commit feature X?" and "What did do lately?" The git ll variation gives a little more detail by listing the files changed along with an indication of the type and size of change. Useful when the commit message alone doesn't help me answer "What did I do lately?" ;-)

Spending more time in a console made me want to be more adept at editing commands here too; I had for years made do with Home and End and the arrow keys. I even increased my key repeat rate to speed up moving back to correct errors with the left-arrow key. Now I've added the couple things I really needed to improve things:
  • Enabling vi mode; I knew about this, but hadn't found myself on the command line quite enough to care about it until recently. (Vi is another thing like to "make" myself use just because proficiency in it just seems to really pay dividends).
  • Figuring out how to navigate back and forward one word at a time while editing a command -- crucial for crazy long commands
All that's required is a few lines in your ~/.inputrc file: