Show me a picture of: names, symbols, vars, refs, namespaces, etc


#1

Please vote. Comments are encouraged!

  • :thumbsup: Yes, please teach this!
  • :thumbsdown: No, I’m not interested.

0 voters

I find that the things that are easiest to forget about a language or toolchain when you’ve been away from it for a year or so are often the things that are least clear about it. In the case of Clojure, that is the many levels of indirection and potential aliasing involved in normal referencing and evaluation.

For instances, consider some of the core functions prefixed “ns”: ns-aliases,ns-interns,ns-imports,ns-map,ns-refers,ns-resolve. All of these describe different kinds of mappings that are central to how you use namespaces to access values. But there are so many kinds! As near as I understand it, if I just, say, dereference an atom on the REPL, I’m going through at least three or four layers of conceptual dereferencing: name -> symbol -> var -> atom -> value. Right?

it does not help that every level of indirection seems to be described by a word that could just as easily have been chosen for any level of indirection. If you were just beamed down from the moon, how would you know that alias describes aliasing for a namespace, but refers describes aliasing for an object within a namespace? This terminology seems arbitrary and hence very, very confusable and hard to remember. This arbitrariness is part of the reason that the ns form is so hard to remember.

I have never seen this explained in a way that makes it permanently clear in my brain. But I think it is possible! Here is what would do it: a diagram. These are all just kinds of mappings, and it is easy to make mappings clear with arrows. Simple as that.

So what I would love to see is a mentoring session, triumphantly culminating in a single one-page diagram, that illustrated the relationship between these entities:

  • names
  • symbols
  • vars
  • namespaces
  • namespace mappings
  • namespace intern mappings
  • effects produced by refers declarations
  • effects produced by alias declarations
  • effects produced by import declaration
  • the dereferencing performed by ns-resolve
  • the dereferencing performed by normal evaluation

This may sound like a tall order but I think (hope?) it is not. There are really only about four basic graph nodes at play: names, symbols, and reference types like vars, atoms, etc… And then there are other entities that just represent different kinds of mappings along the graph, or shortcut mappings of one kind or another.

I think if this whole system could be represented visually, it could be made memorable.

Am I dreamer??


#2

Hey Alexis,

It’s interesting because I know of those ns- functions but I don’t know exactly what they do. I almost never use them!

In fact, I rarely think about symbols, vars, and namespaces in the way you’re describing. I tend to treat them like constant variables in any programming language. I’m wondering if there isn’t a specific problem you’re having?

Well, I’m still up for making a diagram and explaining it with a video. I think it could be interesting.

In the meantime, this talk about Vars might have some details you’re looking for.

Thanks
Eric


#3

Hi,

Well, it’s hard to say what specific practical problem I have. I think I am now comfortable enough with namespaces, requiring, referring, aliasing, and vars, that I have a productive way of working with them.

But I don’t feel I understand them deeply and it was a long road to get to this point. I still have moments when I’m not sure how much code needs to be reloaded so I will restart my entire REPL session rather than try to figure it out.

And I think these difficulties are probably because of how Clojure is different from most other languages in how it handles these things. Consider unusual aspects of Clojure here:

  1. Using in-ns to change into a namespace is different from using require to load code into that namespace.

    So you can change into a namespace without loading anything, and then it’s like making and changing to an empty directory.

  2. With require's :as option, you can change the name of a namespace when you require it.

    This shortened alias can be a convenience. But it also means that I have to choose the alias, so it’s arbitrary and easy to forget. This also creates potential polymorphism at the “package” level (e.g., requiring api-server.java-executor vs api-server.mock-executor).

  3. It’s called a namespace, but it doesn’t merely define names for values. They are names of Symbols associated with Vars which contain values or else reference types which contain values. Symbols and Vars are both concrete things which come up and need to be directly manipulated from time to time.

Usually I can forget these intricacies and get along fine. I treat everything I def as a constant and pretend the namespace is just a mapping of names to values, except that I can mutate the mapping by redefining things on the REPL for convenient. But when I don’t forget this complexity, and try to understand the mechanics in detail, it’s dizzying.

For instance, off the top of my head, I am very unlikely to be able to remember the forms which would help me on the REPL with questions like the following: What public defs does this namespace export? What private defs does it not export? How do I reference the private ones? What refs have I already referred into my current ns? What other ns have I required in and with what aliases? Which local unqualified refs come from which ns?

These questions come up a lot in interactive development. They are equivalent to basic command line navigation on a shell prompt, using ls -l, cd, and symlinks. So I can handle the fundamental concepts. But I find it hard to keep it straight on the Clojure REPL, and in practice I rely on cider and auto-completion to stumble through. This seems like a symptom of a more fundamental problem, either with my understanding or with the Clojure API functions I mentioned.

As I mentioned, it aggravates matters that Vars are described as one of the concurrency primitives, but in practice they are often just something that you ignore and which mysteriously makes namespaces work. I am exaggerating! — but only somewhat…

Consider also how different Clojure is from most other language. Most languages:

  • force a strict association between namespace and directory structure
  • force a strict association between namespace and package structure
  • do not allow arbitrary renaming of packages/namespaces
  • force you to load a package to interact with its namespace, rather than separating the concepts of loading a package and entering or accessing its namespace
  • do not declare (or at least do not emphasize) that every single value is contained in a reference type with special concurrency semantics.

Clojure namespaces remind me most of Common Lisp’s module system, which I found very hard to use.

Sorry to be long-winded. Here’s a way to put it that I would bet resonates with the experience of others: I can understand other languages’ package concepts just fine. I can understand file directories and symlinks just fine. So why does the REPL feel harder when I am using it for what are conceptually just packages and directories?


#4

Hi Alexis,

It sounds like you’re trying to understand things deeply, and I applaud that.

This is a very common place to be. I get there a lot. I often will flush out namespaces if I’ve been renaming a bunch of stuff. And, yes, sometimes I’ll restart the JVM. You can remove a namespace with remove-ns. Then you require it again and it’s clean. Don’t tell anyone, but I often just grep for stuff in the terminal instead of using awesome Emacs features. It’s something you get used to.

Yeah, it’s a choice you have to make. Many libraries will tell you what they recommend you alias something as. Very often there’s a de facto standard (which, unfortunately, you have to learn). Most projects develop their own conventions. It’s a bit of a pain, but it saves a lot of bigger trouble like you get without namespace (lots of clashes or naming conventions).

There’s a lot of indirection, but it’s not so bad. Symbols are the names. And Vars work so transparently, I rarely think about them. The compiler takes care of them. And I’m glad they’re there when I need them.

I’d really like to pair with you to see what kind of trouble this might be causing. I suspect you’re trying to understand what’s defined by querying the namespaces. But the namespaces are not that friendly. See below.

This reminds me of when I started with Lisp back in the day. I had read somewhere that the Lisp REPL was like having a conversation with your programming language. I was so excited. I fired up the REPL and started telling it things and asking it things. I soon found that the analogy was very stretched. For instance, I wanted it to tell me the code for a function I wrote, and let me edit it, and enter it again. Didn’t work. Well, I couldn’t find the functions to let me do that. I was very disappointed that the REPL wasn’t a better conversationalist.

I eventually settled down to using the REPL only to explore some bit of code. Basically to know what it did. I relied on features of the IDE (Emacs with Slime, now I use Emacs with CIDER) to get other stuff. And I just dealt with the slight discrepancies that develop between code in my files and what’s loaded in the VM.

Wouldn’t it be great if navigating namespaces at the REPL were as easy as a Unix dir structure? That would be awesome. I guess I’ve given up on that. I know there’s a namespace called clojure.repl that is designed to do that: https://clojure.github.io/clojure/clojure.repl-api.html

Having some experience with other Lisps and other dynamic systems, I understand the tradeoffs and think Clojure did a pretty good job.

Eric