In computing performance, direct instructions are fastest, but are difficult to program. In code, indirect seems elegant, but just ruins cognitive ability. The balance should lean towards directness. Indirection may not be bad, but it's only good in small quantities.
The Core Assumption
Like so many things in software, I think as holistically about the lifecycle of my code as I can. If I'm writing a script that I will run once or twice, who cares about the composition or organization? Just bang it out, see if it works, run it for realsies, and move on. If I expect to not have to rewrite this code for years, I need to organize it so it can be understood by me 6 months from now, or whoever the person coming after me is. I promise it isn't a whole lot slower, especially as you gain experience. There's a lot of argument out there on how to do that organizing, of course. This one is about indirection.
The problem is complicated and nuanced. That's sort of the issue with programming. If the problem is obvious, we see the trend of avoiding it. If this algorithm isn't performant, it's obvious when we run it, until we add 15 orders of magnitude more compute into my watch than most computers owned by megacorporations in the 70s. Then it's less obvious and we write less performant software. Engineering discipline often needs to consider the non-obvious. This is one of those cases, and my examples are going to be contrived accordingly, because of course situations and style differences abound. My ask is that you play in different perspectives, because I want to convey the nuance that causes you to think about where the lines should be drawn, not join a religious movement, because this is trade-offs. That's why this isn't called "Indirection Bad."
Indirection is where instead of writing doThing(input) I instead create a bunch of concepts and layers, so maybe instead of doing the thing I have a task manager that takes the task and hands it to a worker and the worker verifies the input with a task schema checker. Maybe I just don't like the format of input and for my happy place I want to have it in a different format and I write a converter. The problem is 6 months from now if I forget that I introduced this conversion layer, or I forget that I decided to change names because I didn't find route descriptive enough so I changed it to path in mine. Now the error in production says route and that's not in my codebase because the library I'm using takes positional arguments and I don't discover that until I go through a bunch of code diving, when if I had just followed the standard my google search telling me what the error meant would instantly translate to a search in my code of where I'm directly interacting with that standard.
In the last week, I've had this concept in my mind during conversation about YAML, jsonnet, Typescript, OOP, double NATing, Lua, Vagrant, OCaml, and a dozen other things. If you know me, you probably have seen shades of this. If you know me well, you know it's not one is better than the other, it's all trade-offs. The core assumption I'm making here is that we use indirection for the wrong reasons.
The Things We Don't Like
I can't pick an example without being on someone's bad list, so picking the most useless zealots (as opposed to casual enjoyers), is important. OOP is an easy pick here.
I don't like OOP, of course, but I don't like functional or imperative or whatever else either. They're all rigid with sharp edges. That's why I can see the more egregious problems easier. Some people hate Java and love Python. I hate both of them and every other language, so I don't get blinded by the positive bias. Java is worse than Python unless your programmers suck. They do suck, so Java has a place. A place I don't want to work ever, but this is a tangent. The point here is it's not about what we like or don't. The point is what we do with the tools as an industry. You are your results. If you write Java, your stuff works, even with NullPointerExceptions everywhere. Your software sucks and is impossible to maintain. I don't even need to see it to know that. By their fruits shall ye know them.
So when we do OOP, regardless of language, we're talking about conceptualizing something. We like to think in objects. We like to model the system in these constructs. Blah blah blah. The reason I don't like it is because it loses. Look at Python. It's consistently at the top of coding challenges like Advent of Code because you can write it fast, getting your idea out into code quickly. You rarely see the OOP version of Python code at the top, though. Light OOP is a feature. See these trade-offs coming through? I hate OOP but it still can be useful and outcomes matter when you can share headspace with other devs. It's nice.
I still hate it because over time, you're trying to join functionality and data modeling, and that gets hard to manipulate. Adding features, fixing bugs, etc. So we created all these ways to deal with them indirectly. AbstractFactoryClasses, design pattern hell, decorators, dependency injection (the worst way to use interfaces). The list is endless. It's not always bad, but the cost is always, and I do mean always, dev time later. There's a bug. Where does the actual thing the computer is going to run happen? If you've ever been in a large Enterprise Java codebase, you know you lose endless hours to finding that, especially the more kinds if indirection you have.
Now add in config files, xml, templating engines, shell scripts that fetch values from a secret store, etc, etc.
It's not about what we do or don't like. It's about making something a joy to work in for the long haul, for all people that are going to do so. If that's just me, be it a personal project or a one-man open source project, great. The constraints are less, and my personal preferences can be more represented.
The Thing About Java
My friend RodgerTheGreat/InternetJanitor had a quip many years ago that I've often paraphrased more cynically as this: the key value of Java is that it takes a sea of mediocre developers and makes them consistently produce code that works and runs reliably. It's corporate hell, the language. I agree, and that doesn't make it bad to write that language. Ramen is calories. A boxed lasagna is a nutritous meal with little prep time. They serve functions.
The reason to not use Java is not about the value it brings or not. The reason to not use Java is that there are better languages for your purpose, for the results you want. If the result you want is "avoid Java because I hate it" you are a bad engineer. If the result you want is "write more efficient code faster, at the cost of having to limit the accessibility of the code to a developer that can understand the codebase" that's a good acceptance of a trade-off. One of many, of course.
Indirection is like that. Java people write indirection because that's how the pattern goes. I've taken 10,000 lines of Java and compressed it to 800 lines of Python, and the Python did more things. I've seen compressed Java itself and it looks a hell of a lot like C shoehorned into Java classes, and it's been fine. I prefer to write my own Java that way. I've been turned down for Java jobs because I do. Great, everyone won with that decision.
Where Indirection Shines
Indirection is useful if it's doing one of at least two things, and the benefits of doing one of those two things is worth the cost. There might be more than the two things, but I have only thought of these two.
First, if you are using indirection to have a better understanding of the purpose of the code. For example, I can make a function that abstracts out several other functions of functionality. See (deep vs shallow modules)[https://prograils.com/modular-design-deep-vs-shallow-modules] for a description of how that abstraction's value should be viewed. You want some functions that just call a bunch of other functions in order, and that's not indirection, that's just code organization. I can jump into those functions easy, and if they're appropriately named I can probably pick which one I want to dive into for a given reason.
This kind of indirection is best understood as an (API Contract)[/tnlblog/api-contracts-are-everything.html] for separating concerns. It is saying "here are the bits you should care about, and only the bits you should care about, so if you give me X, I'll do Y and give you Z." The cost is that you will always need more feature bleed and you have to manage the surface area and understanding load. A deep module will have low cognitive load. A shallow module just makes everything about this kind of indirection bad. Still, this is the most common and least problematic kind.
The second useful indirection is translation. Translation happens in all sorts of ways. Kubernetes translates yaml to json for it's API. You have to write yaml for the kubectl command, but it speaks json to the API service. That's easy to understand. It's a minor translation, thankfully. Templating is a different kind of translation. Templating is fine. Lots of people have opinions on it. Template this way or that way, have a language or not, fake declarative or not, whatever.
They all suck. All Infrastructure as Code solutions suck, for instance, so prefering the AWS CDK over Terraform is stupid, you're just moving the cheese around and now you have no hiring pool because your internal recruiter isn't finding AWS CDK on many resumes. A good recruiter will look for Terraform and understand that learning the AWS CDK is just going to be a cost the business has to pay for every hire. Templating is the same. Do the standard thing, or don't and expect a huge learning curve. Use Jinja in Python, not Chameleon. No really, you don't have an excuse. There's so many cases like this. Play with that useless stuff on your own time until it provides actual value. Don't spend (innovation tokens)[https://www.lessannoyingbusiness.com/post/innovation-tokens] on this nonsense.
Another important translation type is wrapping. Wrapping is actually good and needed, it's like the function that calls a bunch of other functions (which wasn't indirection, but this is like it). Sometimes it's providing critical functionality for the user, like a CLI that splats out a project. That's a wrapper around a clone and a template operation. I write cookie cutter templates sometimes. A CLI is a better experience, easy.
The number of layers of translation is the hard bit. You want to avoid too many, because it becomes the Enterprise Java problem, except we have editors that help you do the Java bit quickly, and you can't with layers of translation. It's like a game of telephone. You lose meaning through the layers, even if the "word" you're searching on is the same. In fact, it's worse when the name of things stays the same, because naming things is already hard and now you're changing the context of the thing through every other layer.
A translation type that is always silly is tool abuse. My favorite is when projects write Makefiles and they're all PHONY targets that just call commands. Now I have to install Make when I already have a shell. "Oh, but shell's aren't cross-platform!" you say, incorrectly. I've been running Bash on Windows, Mac and Linux for over two decades. I can install Make and get one functionality that Make wasn't intended for, or I can install bash and open up a deep module of capability. That intent thing is key. Make is there to save work in a DAG. If your targets are all PHONY you're literally just writing the worst shell script.
If you need something more capable (don't write Bash that's complicated, please), use Python. Please don't use both. It's ok to have one translation layer that's easy to follow, it's a cognitive load but it's not much. Many layers of wrappers is common, but it's always lazy. I know, I've done it too (and will do it again, yes), but it has costs and that is often speed and reliability of devs. More often than not, I'm doing it to myself, and it feels awful. Past me clearly didn't think of future him.
Still, all that automated joy in the one translation layer can be awesome, if you can maintain it well. We can't, obviously, but this is about trying to improve, not be perfect.
Just please try to minimize it, that's all I'm going for here. If you can keep things organized, you won't need much indirection, and if you don't have much indirection, it's easier to keep things organized.
If you have thoughts on more concrete lines and values, I'm all ears. This is one I'm going to keep revisiting in my own mind, as I have for many years. The lines are blurry, but that's what makes engineering valuable.