Types, Clojure, and pain


#1

Martin Trojer’s Beyond Clojure series is really interesting reading:

On a related note, I recently watched Matthias Felleisen’s Clojure/West talk:

What do you all think? Do large production Clojure code-bases suffer from the issues that Martin describes in his Prelude (and Felleisen elaborates on his talk)?


#2

Hi @brentvukmer,

That’s a great question. The article (Prelude) is about his frustrations with Clojure at scale.

I think all languages have some scale at which things get difficult. I don’t know where to start responding to the ideas in that article. But I will just let some of the more easily expressible ideas out :slight_smile:.

I’ve worked in Haskell professionally and Haskell was quite nice (though I missed Clojure while I wrote it).

  1. When someone gripes about a language and suggests enthusiastically using a different solution, have they actually used the solution? Does the author have experience scaling a Haskell service or working on a larger Haskell team? Otherwise, this sounds like a grass-is-greener bias. When I worked in Haskell, we all felt scaling issues, too.

  2. Does scale really become a problem? We all know the feelings of speed associated with the early days of a product. But there are so many gigantic dynlang systems that seem to be doing just fine. Is the author just wishing for those early days again? Is this just rewrite fever? After a while working in the Haskell system, I daydreamed about giant refactorings to clear out some of our worst designs. And sometimes I said “this would be easier in Clojure”.

  3. Or is the author lamenting some early design decisions in their software whose problems are now becoming apparent? Maybe those design decisions are associated with some dynamic features. But who’s to say that different bad design decisions won’t be made in a typed language, and those design decisions are associated with the type system? Bad decisions are bad decisions. We made some bad decisions in our software in Haskell. Often, they were ways to deal with types, or some choice of how to structure our types. They were just suboptimal, and we paid for it every day. But there was so much code using those types, a refactor was not cost effective. It can happen anywhere.

I think typed languages are the future. We need types, or something equivalent, to help us build better software bigger and faster. We should all be exploring these options. Haskell is a great choice for learning what types are all about.

But types are not a magic bullet. Types introduce another dimension in the design space of a language and your software. There’s more room for abstraction and more rope to wander off into the weeds. The type system makes sure your code makes sense. But it can’t tell you the sense it makes is worthwhile.

Ok, those are some of my most developed thoughts on the matter.

Eric

BTW, I appreciate Elm. It is really making some hard design decisions, sometimes in direct contradiction of Haskell’s decisions. We need more of this.


#3

Thanks, @ericnormand. Two assertions from his Prelude stood out to me: “Clojure code rots quickly” and “Nulls are a problem in Clojure”.

Have you found that careful design plus automated testing (plus contracts and Schema?) address his concerns?

The Felleisen video referred to in the Prelude was really interesting because Professor Felleisen reflected on decades of research and experience with the large (500k, I think) Racket codebase.

Code rot:

After years of working on real world large-ish (20k+ lines) Clojure code-bases, I’ve come to the conclusion that Clojure (like any other dynlang) doesn’t scale. It doesn’t scale on 3 very important axis in software development; code size, team size and time elapsed. The Achilles heel is refactoring. When I say refactoring, I am not talking about huge re-writes, but the day to day tweaks and shuffles that you do to a living code base. In Clojure, like Python and Ruby, you are basically stabbing in the dark. Keyword typos, shape changes of your nested maps, nil punning etc all work against you. Maintaining any confidence that your change didn’t break the code is near impossible.

On a high level, I’ve only seen 2 mitigation strategies that really work;

Keeping a model of the entire code-base in your head (including all the subtleties). This works if you are the sole developer (which is true for many of the popular Clojure libraries for instance).

A huge corpus of unit / integration / quickcheck tests (more than 50% of the total line count)

Neither of them scale on some or all of the 3 axis; when the code grows, when the team grows or when time elapses. The horrible truth is that Clojure code rots quickly. The end result is bugs, bugs and more bugs. Most of them are really subtle as well, its a long tail of bugs that is only found in production after weeks of uptime. Its the kind of bugs that, when fixed, are accompanied with the developer saying “Ahh, yes, I didn’t think about that scenario”.

Nulls:

Another big bugbear of mine is nulls. And let me tell you, Clojure is a petri dish of them. We attach meaning and truthy-ness to nil of course, see nil punning. At first, this looks like a reasonable idea, but then you start finding the corner cases where it doesn’t work. You stop trusting it, and viola, you are back in null-checking hell. NullPointerException is a very real thing in Clojure code-bases. Its a crying shame.

I don’t think the majority of Node, Ruby, Java, Clojure developers are aware of the fact that there are languages out there without nulls. Its hard to explain how big of a deal to code quality this really is, it has to be experienced.


#4

I really appreciate your thoughtful response, and focus on design and team dynamics. Some (most?) of these problems are, I’m convinced, just part of fallible, frail mortals working together. We do not live and work in the realm of Platonic ideas, much as programmers might be tempted to think we do (or can).


#5

@ericnormand You state: ‘I think typed languages are the future.’ Although they are not the magic bullet, why are you not using Haskell all the time? Why wait for the future? :slight_smile:


#6

Hi!

This thread spurred some useful discussion on Clojure’s Slack in #other-languages, on May 6, May 7 & May 8.


#7

@tjg thanks for the links! Great discussion. I need to start hanging out on Clojure’s Slack.


#8

So I’m not going to try to dispute these statements. He’s making an absolute statement (“quickly”) when speed of rot is really relative. All code rots. What is he comparing Clojure’s rotting speed to? If he’s comparing it to Haskell, or typed languages in general, I think he still has the entire case to make ahead of him.

“Nulls are a problem in Clojure” is true. Undoubtedly. Haskell’s Maybe solves some of the problem of nulls, while carrying along a lot of the downsides. So in that sense, Maybe’s “are a problem”.

Clojure is not perfect, but neither is Haskell. I think Haskell is the most interesting typed language that is practical right now. It is definitely worth learning. Understanding typed programming has made me a better programmer. I think much more systematically. And I’m appalled by some Clojure code I see, that doesn’t do anything to mitigate the risks of nulls and other pitfalls. All languages have pitfalls and they should be understood to avoid them.

What I don’t appreciate is the rhetoric around comparing language features the way the author of the Beyond Clojure series is writing. It’s just a bunch of general statements that make one language/paradigm seem superior. It just seems like grass-is-greener syndrome without much real contribution.

The tools I use are careful design and TDD for when it gets really hairy. I use those in Clojure and I used them in Haskell. The constraints are just different. In Haskell, the design decisions are about what your types look like and whether they model the problem well. In Clojure, it is the same, except more subtle, and so more error-prone. The types are in your head, and you’re also mitigating factors like nil and whether you’ve covered all of the cases. Haskell clearly wins when the problem is simple.

But what happens when it’s complex? Your type has to get more complex. And the error messages you get from them are more difficult to understand. And you can even construct types that you can’t understand and take hours to work out how to satisfy with a program. But somehow in Clojure, those complex types have some kind of gating. I’m not really sure I have a great explanation for this. It has something to do with how you can really narrow down a type at runtime. This is how occurrence typing (the system in Typed Racket and Typed Clojure) works. It analyzes if statements and can narrow down the type of a value based on that. For instance, if I say:

(if (string? x)
  (.indexOf x)
  ...)

I can be sure that in the then branch, x is a string. But in the other branch, I still don’t know what it is (but it’s not a string). That lets me do local reasoning in that branch. In Typed Racket, the compiler helps you figure out if you’ve narrowed down your type enough.

Why am I not using Haskell? The overall experience of Clojure is better. Clojure has many design decisions that I think Haskell can learn from (for instance, basing everything on abstractions like seq instead of on concretions like List). And the huge availability of Java libraries, wrapped by Clojure or unwrapped. Haskell might catch up, but until it does, I still find Clojure a better fit for me.

To be honest, it’s something I revisit from time to time.

Eric


#9

@ericnormand thank you for your thoughtful response. This really does feeling like online mentoring, and I appreciate it.

Since I have limited time to learn a new language (and functional programming) I’m trying to be prudent and look ahead a bit. I’d love to learn Clojure and Haskell and Elixir, but I have to be very realistic about what I can get done. (I’m struggling just to find time to ramp up on Clojure development.) Based on the discussion I think I’ll continue with Clojure.

Thanks again!