Signs of Fear Driven Development

Note this is sort of a draft/WIP post... I intend to update this sporadically as I notice or re-notice new signs and perhaps rewrite in a more organized manner one day. If you're skimming, the signs are in bold text.

I've written a bit about FDD before but it's probably worth calling out separately and compiling a list of signs you, your team, or your company may be suffering from it. And possibly some suggestions for ways out.

Today's trigger for the post was actually another software evolutionary branch that got cut off with the demise of the Lisp machines. Take a look at this tweet and its replies:

One of the coolest things about Lisp machines is that every application you write is *also* a library. Imagine if all of the programs on your computer also gave you access to every routine and subroutine used to build that program with ZERO hassle. Wouldn’t that be amazing?!?!

— '(Robert Smith) (@stylewarning) April 5, 2020

"If you were writing an MP3 database program, you could just use the MP3 parser your music player has ... No worrying about compiling a dynamic library, or finding the source code ..."

What am amazing innovation. I can think of several reasons it didn't catch on (software IP being one of them -- how dare you call my function standalone to the whole program if you don't have a license to?) but the replies suggest another fear-driven reason. "What if you want to change a function and people are calling it?"

So, here's the first sign of fear driven development. I'm deliberately making my code harder for others to use because I want maximal freedom to change it in the future in an arbitrary way (even deleting it) without worrying about upsetting anybody. Derivative signs include an obsession with static libraries over dynamic libraries.

There are a few ways to alleviate this worry/fear. The first, pointed out in the thread, is to signal public/private but don't forbid calling the private stuff. In the web world, this is similarly done by having undocumented but callable APIs ('private') and versioned, documented APIs ('public'). I work on a website builder project, we have a few public APIs for some operations, but we have a gigantic undocumented API that consists of the web service calls the front-end makes to do its operations on the back-end. (Drag-n-drop widget here, change color there, and so on.) Customers can FUBAR themselves if they start calling these, but if they complained our audit logs would quickly show what happened and there would be no promise of support. Meanwhile sometimes bugs happen anyway, and having these underlying calls available to us or support engineers makes resolving such bugs a lot simpler than otherwise.

The second is to accept Hyrum's Law: it doesn't matter if you try to cripple reuse, because over enough time with enough users every observable behavior will be depended on by somebody. For a GUI application this may indeed mean someone has gone to the trouble of autohotkey-ing a script to use your software for an unintended use-case in another pipeline. If your software breaks that use, they'll be frustrated, but you can still claim unsupported behavior and shrug about it. (Or maybe realize it's a valuable thing and open up a function call point with or without promised support...)

The third is related, it's to accept that things change/break all the time, and you're not going to do any better just because of this action. So in short, you're always going to wind up upsetting somebody, stop being so fearful of a result that's bound to happen and instead focus on ways to make things better when it does. (Yes, sometimes you can only just shrug to a complaint and say a piece of code was deleted and won't come back, but sometimes you can do better.) One genuine effort I've seen lately to describe how to do better was shown in one of Rich Hickey's talks.



(transcript)

A key point in the talk is that responsible changes (especially in distributed-to-end-user software, open source or not, where you don't have visibility into all users/callers) avoid breaking anything. And changelogs would be better structured as a list of what is required, and what is provided.

Again my point isn't to make people live up to some ideal of never breaking anything (I know the pains of long term support, and people depending on side effects of bugs you'd really like to fix but can't, and the joys of getting a go-ahead for "enough is enough") but to not let the fear of someone complaining about breaking prevent you from making your software as useful as it can be.

Second sign of FDD: I'm afraid to make this refactor because... The reason changes. I'll call out three: 1) I have no automatic IDE support for it 2) my programming language is dynamically typed 3) I have no tests.

A resolution to 1) is to re-learn what refactoring means. If you can't do it with pure reason, manually, you're not doing refactoring. Anything beyond that is nothing but a reasoning check or automation on parts of it, but as my first post on FDD detailed such extensions to manual reasons should not elevate your confidence of correctness to 100%. You should not fear because of a less than absolutely certain confidence of correctness. (If you need help with understanding what refactoring actually is, with advice that applies to both of the reasons 2 and 3 as well, check out the second edition of Refactoring which does everything with JS.) But you should understand what it is you (or your IDE) is doing, including the how and why and limitations (e.g. not even IDEs for Java can fully protect you against reflection calls to private methods, but some/some plugins can at least do the ag-equivalent of finding the method name in a string or flagging possible reflection accesses to manually review).

A resolution to 2) besides the book is to use a dynamically typed language that still supports compile-time warnings/errors about bad type usage where it's detected/annotated. Such as Common Lisp (see first post). But if you're stuck with JS or Python, you don't need to go full TypeScript or Py3 type-annotate-everything to alleviate the problem. The better approach is to change your development style.

Ask yourself what advantages dynamic types bring that can make it easier, not harder, to write correct software and not fear making refactors (or other changes). Loose coupling is encouraged, so make use of it! Even when you need to change an object's interface quite drastically, by making things loosely coupled you're less likely to have to go update its interface in a hundred places like you might in a Java program. Also understand that dynamically typed is not the same as untyped, the types still exist with the values, it's just the bindings that can change. To make things more concrete, every dynamic language has a way to program in it (often but not necessarily a REPL) with a very tight feedback loop. Exploit it! It's a great win over the traditionally slow edit, compile everything (thank goodness for languages that have incremental compiling!), run loop. Ultimately it comes down to comfort -- some people just seem to not have a problem here, while others are gripped with fear. If you're fearful, it's worth investigating why others aren't.

For 3) my suggestion is another book: Working Effectively with Legacy Code. One can answer 3) with "just write tests!" if you're worried that without tests you'll break things, but there are often obstacles to that, and the book shows how to deal with many of them. They tend to boil down to too much coupling, too much implicit dependence, and the book shows various methods (refactors?) of breaking such dependence in order to get things under test. If you need inspiration, behold SQLite. However I would suggest to not let fear drive you -- many projects have succeeded without tests at all (notably the Linux kernel, albeit for a long time there have been off-kernel tests that could be integrated at some point) -- what lets people make changes without fear here?

In summary for any objections 1, 2, and 3, the fear is misplaced, and if you're using the negative cases (IDE auto-refactoring, static types, or a full test suite) to alleviate your fear, you're using those tools incorrectly. The tools can be useful, but shouldn't be used as sedatives. (They are pretty strong sedatives though, since the routine betrayals/misses of the tools don't seem to affect their adherents' love for them whatsoever.)

Third sign of FDD: I'm afraid someone else will change this code without me noticing and break something, therefore I'll lock it down! Locking mechanisms include Perforce file locks, a git repo with a single parent gating deployment to production (thus any changes must go through pull requests, and there may be additional hooks to require certain files have certain reviewers), gold file tests that serve only as notification of change to react on, and so on.

Having this fear is a symptom of social problems, and it's better to address the social problems directly rather than trying to work around it with technical solutions. At the very least change your fear to a direct anti-social stance: the other developers I work with can't be trusted and there's nothing to be done, but at least I can protect myself. But you could work towards improving that. One of the interesting ideas from the extreme programming group (which mutated into agile+) was that of shared ownership -- with the caveat that such a thing tends to only be feasible with "100%" test coverage. It's worth considering -- can you change the social order away from siloed ownership (too common at big companies in part thanks to Conway's Law) and more towards shared ownership? There is still a technical aspect here of changing development practices (such as more tests, more code reviews) to help drive the social change, but it's not the only hammer.

---

Posted on 2020-04-08 by Jach

Tags: programming

Permalink: https://www.thejach.com/view/id/371

Trackback URL: https://www.thejach.com/view/2020/4/signs_of_fear_driven_development

Back to the top