How to detect and workaround dependency conflicts?


#1

Please vote. Comments are encouraged!

  • :thumbsup: Yes, please teach this!
  • :thumbsdown: No, I’m not interested.

0 voters

Perhaps because I do not have a lot of recent working experience with Java, one of the things I find most confusing in Clojure is how to detect and fix problems due to conflicting dependencies.

For instance, I was unable to run the demo project in a basic heroku walkthrough to get started. After a lot of head-scratching, I realized it worked fine when I erased my ~/.lein/profiles.clj. One of the libraries listed as a user dependency was in some way conflicting with the project. (I’m not sure which one it was, but I think it was [com.cemerick/pomegranate "0.3.0"]).

But my larger problem here is this:

  • I had no way of understanding from the error message that it was this library in my profiles that “broke” the project. This was just a dumb luck guess on my part. How should I have been able to know this?
  • I do not even conceptually understand why a library in my profiles could break a project from loading. If I am not invoking the library, when does it have an opportunity to interfere?
  • I do not understand how to watch for and prevent these kinds of conflicts. If I am adding a new dependency to my profiles, or to a project, what should I check to ensure it does not conflict with something else? Is there any way to do this even in theory? Or in practice? Right now my only method is to see if something mysteriously breaks, not understand the reason why, and keep throwing out dependencies until it works again. This is not engineering.

I would love a mentoring topic that provided me a way to understand the issues fundamentally, and advice about working practices to deal with them practically. But the understanding part is more important!


#2

Hey Alexis,

This is a really great topic.

Because this is important, I’ll address it here briefly.

Leiningen will aggregate configuration from your ~/.lein/profiles.clj and the project’s project.clj file to create the runtime configuration for any of the lein commands. This lets you add in user-level dependencies, plugins, and other settings that apply globally for your machine. When lein encounters a vector configuration (like :dependencies), it concatenates the two options to make an aggregate list of dependencies.

Unfortunately, it’s a perfect way to make your dev environment incompatible with production. I’ve had a lot of troubles that were caused by oldish things in my profiles. It’s just never the first place I look for troubles. However, dependency conflicts can come entirely from within your project.clj, as well.

If you suspect you’ve got dependency issues, run lein deps :tree at the terminal in your project. It will list all dependencies (including from profiles.clj) and their transitive dependencies. That’s great, but the really useful info is at the top, where it shows you dependency version conflicts. It also suggests how to exclude certain libraries to resolve them.

Example:

apollo $ lein deps :tree
Possibly confusing dependencies found:
[hiccup-bridge "1.0.1"] -> [org.clojure/clojure "1.6.0"]
 overrides
[lein-create-template "0.1.2"] -> [org.clojure/clojure "1.7.0"]

Consider using these exclusions:
[lein-create-template "0.1.2" :exclusions [org.clojure/clojure]]

 [clojure-complete "0.2.4" :exclusions [[org.clojure/clojure]]]
 [com.gfredericks/test.chuck "0.1.16" :scope "test"]
   [instaparse "1.3.5" :scope "test"]
 [org.clojure/clojure "1.7.0"]
 [org.clojure/core.async "0.2.374"]
 [org.clojure/core.match "0.3.0-alpha4"]
 [org.clojure/test.check "0.9.0" :scope "test"]
 [org.clojure/tools.analyzer.jvm "0.6.9"]
   [org.clojure/core.memoize "0.5.8"]
     [org.clojure/core.cache "0.6.4"]
       [org.clojure/data.priority-map "0.0.4"]
   [org.clojure/tools.analyzer "0.6.7"]
   [org.clojure/tools.reader "1.0.0-alpha1"]
   [org.ow2.asm/asm-all "4.2"]
 [org.clojure/tools.nrepl "0.2.12" :exclusions [[org.clojure/clojure]]]

I do want to make a lesson about this. I hope this helps in the meantime.
Eric


#3

Thanks, I will play with lein deps and see if that helps.

Just to provide you a (possibly pedagogically helpful) window into my confusion & ignorance, let me spell out basic questions that are not clear in my mind. Here are my thoughts.

  1. Um, okay, so this answer suggests the problem in my profile.clj was that a dependency in my profile.clj file was requiring a version of X that was different from the version of X required by a dependency in the project. So Eric’s implying the problem is a mismatch in a transitive dependency X that is never explicitly named in the project or profile file. Yikes. (Seems grotesquely anti-modular.)

  2. Presumably the heroku project did invoke code, but invoked the wrong version of its dependency, and it got that wrong version because of the dependency in my profiles. So maybe this explains why an uninvokved library (pomegranate) can break something.

  3. Eric’s comments suggest, as regards my original first question, that there is no way to know this is what caused the problem from the runtime error behavior. Double yikes.

  4. This lein deps command exposes transitive dependencies, warns about version conflicts, and provides a workaround via “exclusions”. Hmm. what are they really?

My ignorant guesses about exclusions:

  1. Exclusions seem to specify libraries by name only not name and version. So they are not a mechanism to explicitly specify the version of the transitive dependency that should be preferred. That seems sad. Maybe semver logic always decides which version is preferred? Or maybe there is another version-selection rule? I really have no idea.

  2. Or does an exclusion mean that that dependency’s preference for the version of the transitive dependency should be disregarded ("excluded’), and that we’re going to just cross our fingers and pray that it works correctly with the other version it knew nothing about?

  3. Does the Java classloader even allow us to load two versions of the same library? If Foo wants Xversion1 and Bar wants Xversion2, and they don’t re-expose X in any fashion, why can’t they each both get the version they want and not care about the fact that the version’s don’t match?

Just a window into my thoughts… :slight_smile:


#4

Hey Alexis,

Thanks for the questions. Let me take them one at a time.

Yes, this seems likely. It happens a lot. If you could restore your profile.clj and do lein deps :tree, you may confirm if this is the case.

Yeah. Java is not very smart about class files. It searches the classpath until it finds one that matches. So it’s highly dependent on the order that the dependencies are searched. A transitive dependency of one of your profile dependencies could be searched first. It’s hard to know how the JVM decides how to search them.

Well, there is a way. It seems that you did narrow it down to a problem with something in your profile.clj. You could systematically remove dependencies until you identify the one that breaks it.

No, don’t trust semver, and there’s no other selection rule. Exclusions are local to that branch of the dependency tree. When Leiningen is following transitive dependencies down that branch, it will exclude dependencies that have the name you excluded.

You can pray if you like, but I prefer testing :slight_smile:

Blame the JVM designers. Classes are a global namespace, so two libraries (or two versions of the same library) that define the same classes will conflict.

Your comments are astute. These are known problems on the JVM. Managing the classpath (which is essentially what Leiningen is doing) has always been tough in Java, and sometimes even outright broken. It is something that takes effort and human input. Sometimes you get incompatible versions and you have to hold a package behind because the new version depends on a new version of something else. It can be a pain, but lein tree :deps gives you good transparency into what’s going on.

Its recommendations seem to default to excluding the more recent version. There are also two other things that I find helpful: a plugin called lein ancient and this setting in the project file, which lets you abort or warn when there are version conflicts.

Eric