Specter, a motivational tutorial

This is a tutorial for Specter, a Clojure(script) library for transforming complex data structures.

I wrote this as an unofficial complement to the Specter documentation. It’s intended for those who’ve maybe heard of Specter, but don’t understand exactly how to use it in their Clojure(Script) projects.

The tutorial consists of a series of worked examples based on a “real-world” application, Factor. Each example presents an increasingly complex data transformation problem, introducing new Specter features as they go:

1. Getting a factory’s name
2. Getting all factory names
3. Changing a factory’s name
4. Adding a default name to factories
5. Lowercase all factory names
6. Make two changes at once
7. Delete a factory
8. Collecting values
9. Abstraction and Reusability

All the examples are based on a central data structure called app-db:

(def app-db
 {:world
  {:factories {:1 {:name "My Factory"}
               :2 {:name "My Second Factory"}}
   :items     {:1 {:name "Iron ore"}
               :2 {:name "Iron ingot"}}
   :recipes   {:1 {:name "Smelt iron ore"
                   :input  {:1 1}
                   :output {:2 1}}}}})

This is a simplified version of Factor’s actual app-db. It holds all our application state, and needs to be queried and updated in a variety of ways, as we’ll see in the examples below.

(require '[com.rpl.specter :as s])

1. Getting a factory’s name

back to top

Let’s start simple. We want to write a get-factory-name function that accepts:

• The current app-db.
• A factory’s ID (e.g. :1).

And returns:

• The name of the factory with the given ID (e.g. "My Factory").

For example:

(get-factory-name app-db :1)
; => "My Factory"

So, how would we implement this function?

Without Specter, get-factory-name can be written with a get-in call:

(defn get-factory-name [app-db id]
 (get-in app-db [:world :factories id :name]))

(get-factory-name app-db :1)
; => "My Factory"

Simple, right?

We can do the same thing with Specter’s select function:

(defn get-factory-name [app-db id]
 (first (s/select [:world :factories id :name] app-db)))

(get-factory-name app-db :1)
; => "My Factory"

select works like get-in, but instead of accepting a path of keys, it accepts a path of navigators. The navigator abstraction is the heart of what makes Specter so powerful.

(defn get-factory-name [app-db id]
 (s/select-any [:world :factories id :name] app-db))

Using navigators, we get the concise, expressive power of Clojure’s get-in, assoc-in, etc., but generalized to support “navigating through” data structures in arbitrary ways (iteration, transformation, filtering, etc.). Specter has a lot of navigators that you can use to hone in on exactly which values you want to set, even deep within a complex data structure.

In the above get-factory-name example, our navigators are simple keywords:

:world
:factories
id
:name

(You’ll see some more complex and interesting navigators later in the tutorial — we’re starting simple.)

We can compose navigators together by putting them in a vector:

[:world :factories id :name]

A vector like that is called a path. When such a path is passed to the select macro, it gets transparently compiled into optimized function calls that can sometimes beat the performance of the hand-written variant!

[:world [:factories id] :name]

(def WORLD-FACTORIES [:world :factories])
(s/select-any [WORLD-FACTORIES id :name] app-db)

(def WORLD-FACTORIES (s/path :world :factories))
(s/select-any [WORLD-FACTORIES id :name] app-db)

(s/select-any (s/path WORLD-FACTORIES id :name) app-db)

2. Getting all factory names

back to top

Next, let’s explore how Specter helps us select more than one value at once.

Suppose we want to write a get-factory-names function that accepts:

• The current app-db.

And returns:

• A sequence of the names of all the factories.

For example:

(get-factory-names app-db)
; => ("My Factory" "My Second Factory")

Without Specter, we could implement get-factory-names like so:

(defn get-factory-names [app-db]
 (-> app-db
  (get-in [:world :factories])
  (vals)
  (->> (map :name))))

(get-factory-names app-db)
; => ("My Factory" "My Second Factory")

To write the Specter equivalent of this, we need to understand how to navigate over all the values in a map, instead of honing in on a certain key. To figure out what navigator to use, we’ll check the Specter cheatsheet. It’s is organized according to the type of the currently navigated value, so we should look in the section for Map navigators.

We look through the list and notice the MAP-VALS navigator. It navigates to all the values in a map. Sounds promising!

Let’s write the Specter version of get-factory-names using MAP-VALS:

(defn get-factory-names [app-db]
 (s/select
  [:world :factories s/MAP-VALS :name]
  app-db))

(get-factory-names app-db)
; => ["My Factory" "My Second Factory"]

Same Structure, Same Intent

If you squint, the Specter and non-Specter versions are structurally similar. Both have three stages of selecting data:

1. Navigating to the :factories map.
2. Navigating to all the values of the map.
3. Navigating to the :name of each of the values.

The Specter version has an interesting property, though: It’s awfully similar to the get-factory-name function from the first example!

All we had to do is replace id with MAP-VALS, and switch the select-any to select so we return multiple values:

; example 1
(defn get-factory-name [app-db id]
 (s/select-any
  [:world :factories id :name]
  app-db))

; example 2
(defn get-factory-names [app-db]
 (s/select
  [:world :factories s/MAP-VALS :name]
  app-db))

This highlights some of the expressive power of Specter’s navigators. A small, encapsulated change in intent corresponds to an equally small, equally encapsulated change in code.

In contrast, the non-Specter version was almost completely rewritten:

; example 1
(defn get-factory-name [app-db id]
 (get-in app-db [:world :factories id :name]))

; example 2
(defn get-factory-names [app-db]
 (-> app-db
  (get-in [:world :factories])
  (vals)
  (->> (map :name))))

The logic is similar, but the actual forms are significantly different. Not only that, the non-Specter version may hide bugs more easily, especially for less experienced Clojurers. It’s easier to write bugs and harder to notice them when reading later:

; bad
(defn get-factory-names [app-db]
 (-> app-db
  (get-in [:world :factories])
  (vals)
  (map :name)))

Use Clojure enough and you notice things like that right away… most of the time.

I’m reminded of Hoare:

There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies.

Specter, used responsibly, helps us move towards the former.

(def x {:foo [-1 0 1]})

(s/select-any
 [:foo]
 x)
; => [-1 0 1]

(s/select-any
 [:foo (s/pred pos?)]
 x)
; => nil

(s/select
 [:foo (s/pred pos?)]
 x)
; => []

(s/select
 [:foo s/ALL (s/pred pos?)]
 x)
; => [1]

(s/select-any
 [:foo (s/filterer pos?)]
 x)
; => [1]

(s/select
 [:foo
  (s/filterer pos?)
  s/ALL]
 x)
; => [1]

3. Changing a factory’s name

back to top

Now let’s take a step away from select and consider one of Specter’s other functions for manipulating data structures: setval.

Suppose instead of getting a factory’s :name, we want to write an event handler that can update the :name to a new value.

We want a function that accepts:

• The current app-db.
• A factory’s ID (e.g. :1).
• A string to use as the new name (e.g. "Foo").

And returns:

• An updated app-db, where the given factory’s name is replaced with the given name.

For example:

(set-factory-name app-db :1 "Foo")
; => {:world
;     {:factories {:1 {:name "Foo"} ;; UPDATED!
;                  :2 {:name "My Second Factory"}}
;      ...}}

Without Specter, we can implement set-factory-name using assoc-in:

(defn set-factory-name [app-db id name]
 (assoc-in app-db [:world :factories id :name] name))

To solve this problem with Specter, we can use setval, which is Specter’s generalized assoc-in equivalent.

Like select, setval accepts a path of navigators. But instead of returning the navigated values, it replaces them with a new value, and returns the whole data structure with new values in place.

We can rewrite the set-factory-name function to use setval:

(defn set-factory-name [app-db id name]
 (s/setval [:world :factories id :name] name app-db))

Note the path we give to setval is exactly the same path we gave to select in exercise one:

[:world :factories id :name]

Again, this highlights the reusability of the navigator abstraction. The same navigators can be used for both selecting and transforming data.

Also like select, (and unlike assoc-in) setval has the ability to navigate to (and set) multiple values at once! Let’s see how it works in the next section:

4. Adding a default name to factories

back to top

Suppose some of our factories have no :name, and we want to give them all a name of "Unnamed Factory". But, we don’t want to overwrite the name if the factory already has one.

For example:

(def app-db 
 {:world
   {:factories {:1 {:name nil}
                :2 {:name "My Second Factory"}}}})

(add-default-name-to-factories app-db "Unnamed Factory")
; => {:world {:factories {:1 {:name "Unnamed Factory"}
;                         :2 {:name "My Second Factory"}}}}

We can use setval for this, too.

To better understand how our solution works, let’s build the solution in two stages:

1. First, we’ll solve a simpler problem: setting all factories’ names to "Unnamed Factory".
2. Then, we’ll add logic to avoid overwriting existing names.

For the first stage, we just have to write a path that navigates to the names of all the factories. Luckily, we already wrote that path back in example 2! It went like this:

[:world :factories s/MAP-VALS :name]

In example 2, we used the path to select all the names from the database. Now, we can pass the very same path to setval to replace all the names at once:

(defn add-default-name-to-factories [app-db name]
 (s/setval [:world :factories s/MAP-VALS :name]
           name
           app-db))

So far, so good. But, if we were to run this now, it would clobber all the preexisting factory names, which we don’t want:

(add-default-name-to-factories app-db "Unnamed Factory")
; => {:world {:factories {:1 {:name "Unnamed Factory"}
;                         :2 {:name "Unnamed Factory"}}}}

It’s time to fix our function so it only replaces nil values.

How?

The setval function replaces all the values that it navigates to. So, we just want to “somehow” stop navigating to non-nil names.

The simplest way is to add a nil? function to the path (remember that a plain function in the a path is used as a filter — we could also use (pred nil?) and it would have the same effect):

(defn add-default-name-to-factories [app-db name]
 (s/setval [:world :factories s/MAP-VALS :name nil?]
           name
           app-db))

Now if we run our function, it works properly! Only the nil name was replaced:

(add-default-name-to-factories app-db "Unnamed Factory")
; => {:world {:factories {:1 {:name "Unnamed Factory"}
;                         :2 {:name "My Second Factory"}}}}

5. Lowercase all factory names

back to top

So, that’s setval, which is the Specter analogue of assoc-in.

Does Specter have an analogue of update and update-in?

It does, and it’s called transform. (And it has a big brother, multi-transform, that I’ll introduce later.)

The transform function works just like setval except instead of being passed a constant value to assign, it’s passed a function called a transformer that accepts the navigated value and returns some transformed data. The navigated values are replaced with their transformed data and then the whole data structure is returned.

Suppose, for example, we want to change the names of all our factories to make them lowercase. This is a perfect use-case for transform:

(lowercase-factory-names app-db)
; => {:world {:factories {:1 {:name "my factory"}
;                         :2 {:name "my second factory"}}
;             ...}}

In Specter terms, we want to navigate to all of the factory names, and transform them with the clojure.string/lower-case function.

You could probably write this yourself at this point:

(defn lowercase-factory-names [app-db]
 (s/transform [:world :factories s/MAP-VALS :name string?]
              clojure.string/lower-case
              app-db))

We included a string? at the end of the path in order to skip transforming any non-string values that might cause lower-case to throw an exception.

The lowercase-factory-names function, defined with Specter, is concerned with expressing intent: we don’t say “how” to modify the data structure mechanically, we just say what we want to change.

Let’s compare that with a non-Specter implementation of the same function:

(defn lowercase-factory-names [app-db]
 (update-in app-db [:world :factories]
  (fn [factories]
   (->> factories
        (map (fn [[id factory]]
              [id (update factory :name
                          #(if (string? %)
                            (clojure.string/lower-case %)
                            %))]))
        (into {})))))

The non-Specter version is concerned with mechanics: threading values through a variety of different nested functions to achieve the same effect.

(defn map-vals [f m]
 (into {} (map (fn [[k v]] [k (f v)]) m)))

(defn maybe-lowercase [s]
 (if (string? s) (clojure.string/lower-case s) s))

(defn factory-name->lowercase [factory]
 (update factory :name maybe-lowercase))

(defn lowercase-factory-names [app-db]
 (update-in app-db [:world :factories]
            #(map-vals factory-name->lowercase %)))

(def MAP-FACTORIES (s/path :world :factories s/MAP-VALS))
(def FACTORY-NAME  (s/path :name string?))

(defn lowercase-factory-names [app-db]
 (s/transform
  [MAP-FACTORIES FACTORY-NAME]
  clojure.string/lower-case
  app-db))

6. Make two changes at once

back to top

Suppose we want to add two values to every factory — a :desc and an :author — but only if there are no existing values for those keys.

(add-desc-and-author app-db)
; => {:world {:factories {:1 {:name "My Factory"
;                             :desc "No description"
;                             :author "Unknown"}}
;                         :2 {:name "My Second Factory"
;                             :desc "No description"
;                             :author "Unknown"}
;                         ...}}}

There are a few ways to approach this.

One option is to thread the database through two setval calls:

(defn add-desc-and-author [app-db]
 (->> app-db
  (s/setval [:world :factories s/MAP-VALS :desc nil?]
            "No description")
  (s/setval [:world :factories s/MAP-VALS :author nil?]
            "Unknown")))

This works, but we’re iterating over all the factories twice, which is unnecessary work.

Our inner optimizer tells us, “We only need to iterate over the factories once! We can use transform — just navigate to each factory, and transform the factory by adding the :author and :desc fields at once!

(defn add-desc-and-author [app-db]
 (s/transform
  [:world :factories s/MAP-VALS]
  #(cond-> %
    (nil? (:desc %))   (assoc :desc "No description")
    (nil? (:author %)) (assoc :author "Unknown"))
  app-db))

This is nice. Our inner optimizer is happy. But it’s come at the cost of writing a somewhat-complicated transformer function. The result feels a little less declarative, a little less obviously correct.

Is there a better way?

It turns out, we can write this in a way that’s both efficient and declarative using the many-armed multi-transform function!

multi-transform allows you to perform multiple transform (or setval) operations at once. They can share navigators to avoid redundant computation, but also branch out into their own navigations using the multi-path navigator:

; Use multi-path for branching navigation!
(s/multi-path
 [(s/keypath :desc)
  ...]
 [(s/keypath :author)
  ...])

Unlike transform, multi-transform doesn’t accept a transformer function as a parameter. After all, we might want to transform each of the multi-path destinations in a different way. A single transformer function isn’t expressive enough for that.

Instead, when using multi-transform you must use the special terminal and terminal-val navigators to “terminate” the branches of your navigation and either apply a transformer or set a value.

We use the terminal navigator to apply a transformer function (a la transform):

(s/terminal clojure.string/lower-case)

Or we use the terminal-val navigator to set a value (a la setval):

(s/terminal-val "Foo")

So, to implement add-desc-and-author we can write:

(defn add-desc-and-author [app-db]
 (s/multi-transform
  [:world :factories s/MAP-VALS
   (s/multi-path
    [:desc nil? (s/terminal-val "No description")]
    [:author nil? (s/terminal-val "Unknown")])]
  app-db))

Neat! We’re only iterating over the factories once, because the MAP-VALS navigator is outside the multi-path. But we didn’t have to use a handwritten transformer, so our code is still nice and declarative.

(defn add-desc-and-author [app-db]
 (s/setval [:world :factories s/MAP-VALS
            (s/multi-path [:desc nil?]
                          [:author nil?])]
           "Unknown"
           app-db))

7. Delete a factory

back to top

Suppose we want to delete a factory from the database:

(delete-factory app-db :1)
; => {:world {:factories {:2 {:name "My Second Factory"}}
;             ...}}

Without Specter, we’d do an update-in combined with dissoc:

(defn delete-factory [app-db id]
 (update-in app-db [:world :factories] dissoc id))

At first, it’s not obvious how to do this in Specter.

We could lean on transform:

(defn delete-factory [app-db id]
 (s/transform [:world :factories] #(dissoc % id) app-db))

You may have noticed a pattern, though: Specter works best when we express our logic in navigators. Whenever we have logic in our transformer function, we should ask ourselves, “Is there some way to express this logic with navigators instead?”

So, is there a way to express the logic of “deleting” a value without using a transformer?

Turns out, there is! Specter provides a special value NONE, which represents the absence of a value. If you set a navigated value to NONE, Specter deletes it!

The NONE value can be used anywhere Specter is expecting a value: setval, transform, multi-transform, etc. all support it.

In this case, we can rewrite our function to use setval. We just have to navigate to the factory and set the value to NONE:

(defn delete-factory [app-db id]
 (s/setval [:world :factories id] s/NONE app-db))

(def edges {:a {:b "edge from a -> b"
                :c "edge from a -> c"}
            :b {:d "edge from b -> d"}})

(get-in edges [:a :b])
; => "edge from a -> b"

(get-in edges [:b :d])
; => "edge from b -> d"

(get-in edges [:a :d])
; => nil

(s/setval [:a :b] s/NONE edges)
; => {:a {:c "edge from a -> c"}
;     :b {:d "edge from b -> d"}}

(s/setval [:b :d] s/NONE edges)
; => {:a {:b "edge from a -> b"
;         :c "edge from a -> c"}
;     :b {}}

{:a {:b "edge from a -> b"
     :c "edge from a -> c"}}

(s/setval [(s/compact :b :d)] s/NONE edges)
; => {:a {:b "edge from a -> b"
;         :c "edge from a -> c"}}

(s/setval [(s/compact :a :b)]
          s/NONE
          {:a {:b "edge from a -> b"}})
; => s/NONE

(s/setval [:a (s/compact :b)]
          s/NONE
          {:a {:b "edge from a -> b"}})
; => {}

8. Collecting values

back to top

I’m going to introduce one more advanced-but-useful Specter feature: collecting values.

Collected values sorta like capture groups in a regex: any collected values are returned alongside any selected values.

Or, in the case of transform and multi-transform, the collected values are provided as additional inputs to the transformer function(s).

Suppose, for example, we want to define a function to add a new input item to a recipe. The item is referenced by its item ID.

We can do this with transform:

(defn add-recipe-input [app-db recipe-id item-id quantity]
 (s/transform [:world :recipes recipe-id :input item-id]
              (partial + quantity)
              app-db))

(add-recipe-input app-db :1 :2 123)
; => {:world
;     {:recipes {:1 {:name "Smelt iron ore"
;                    :input {:1 1
;                            :2 123} <=== added this!
;                    :output {:2 1}}}}}

So far, simple enough.

But now let’s say we don’t know the ID of the item we want to add. We just know the name of the item, e.g. the :2 item is named "Iron ingot". Well, this is a great opportunity for the collect-one navigator. Let’s use it:

(defn add-recipe-input [app-db recipe-id item-name quantity]
  (s/transform
   [:world
    (s/collect-one
     :items s/ALL
     (s/selected? s/LAST :name (s/pred= item-name))
     s/FIRST)
    :recipes recipe-id :input]
   (fn [item-id input] (update input item-id + quantity))
   app-db))

(add-recipe-input app-db :1 "Iron ingot" 123)
; => {:world
;     {:recipes {:1 {:name "Smelt iron ore"
;                    :input {:1 1
;                            :2 123} <=== added this!
;                    :output {:2 1}}}}}

This function looks much like the last one, but instead of accepting an item ID, it accepts a name. In the collect-one path, we navigate to the ID of the item with matching :name. Then, we can use the collected ID in our transformer function — the item-id parameter is the collected value.

Collecting values is an extremely “power user” feature of Specter. It might seem egregious to try and shove all this logic into navigators, but there are times when it’s super useful too!

My tip would be to start out without using Specter’s value collection. It’s complicated and can interfere with learning. Once you get more comfortable with Specter and the benefits of encapsulating logic in navigators, the value of collection will become very clear, and you can come back to collect et. al. with a much better basis for understanding.

For now, just be aware that collecting values is possible, and let’s move on to the last section!

9. Abstraction and Reusability

back to top

Hopefully by this point we have a good understanding of how Specter works.

We’ve seen that we can use navigators to express our intent declaratively instead of writing the mechanical steps to transform a data structure. And we’ve encountered a bunch of built-in Specter features that help us with that.

Now I want to delve into writing custom navigators so we can express our code’s intent even better: Instead of declaring our intent in terms of Specter’s navigators, we can declare intent using our application’s domain language.

For example, we could define a reusable (recipe id) navigator that encapsulates the idea of “navigate to a specific recipe”:

(defn recipe [id] (s/path :world :recipes id))

Remember, custom navigators are usually declared using the path macro for performance reasons, but this could also be written:

(defn recipe [id] [:world :recipes id])

The cool thing is we can use this same navigator if we want to select the value, update the value, compose with other navigators, and so on.

Get the recipe:

(s/select-any (recipe :1) db)

Transform the recipe:

(s/transform (recipe :1) #(assoc % :name "foo") db)

Custom navigators can be composed together to create more complex and useful navigations:

(def WORLD      (s/path :world))
(def RECIPE-MAP (s/path :recipes))
(def NAME       (s/path :name))

(defn recipe      [id] (s/path WORLD RECIPE-MAP id))
(defn recipe-name [id] (s/path (recipe id) NAME))

Then the recipe-name navigator can be used to update/select recipe names directly:

(s/select-any (recipe-name :1) db)

(s/setval (recipe-name :1) "foo" db)

Naming Custom Navigators

As a matter of personal taste, I like custom navigator names to give a hint as to the type of value they’re navigating to. I follow these conventions:

1. All navigator names should be nouns — where the noun is the value being navigated to.
2. If the name is plural, e.g. RECIPES, it can navigate to zero, one or more values.
3. If the name is singular, e.g. RECIPE, it must navigate to exactly zero or one values.
4. If the value being mapped to is a data structure itself, the name should include the data structure type.

So, for example:

; Navigates to the map of recipes, if it exists
; Singular because there's only one recipe map
(def RECIPE-MAP (s/path :recipes))

; navigates to each recipe in the map
; plural because it navigates to multiple values
(def RECIPES (s/path RECIPE-MAP s/MAP-VALS))

; navigates to each ID in the map
; plural because it navigates to multiple values
(def RECIPE-IDS (s/path RECIPE-MAP s/MAP-KEYS))

La fin

back to top

That’s all, folks!

If you have feedback, suggestions or corrections, send me an email.

Specter reference documentation:

• Cheatsheet: https://github.com/redplanetlabs/specter/wiki/Cheat-Sheet
• List of macros: https://github.com/redplanetlabs/specter/wiki/List-of-Macros
• List of navigators: https://github.com/redplanetlabs/specter/wiki/List-of-Navigators