The Problem with Scripting in Production

Using scripts in production might seem like a quick and dirty way to finish out a feature. It’s only quick and dirty though, if your team lets it be quick and dirty.

Sure, it can be as quick and dirty as you like. If it breaks the production application that it’s meant to serve however, you will wish that you had gone the normal route and jumped through the hoops of the production code delivery process.

I was once complicit in such a decision. We suggested that we could create an easy-breezy script in order to finish the second half of a semi-rushed feature. Surprisingly, the suggestion was sanctioned by the powers that be, despite its non-traditional nature.

At the time, we believed that this was a wise choice. It was an iterate to excellence kind of choice. A we-do-the-easy thing-now and we-do-the-hard-thing-later kind of choice.

The Pros

Free from the shackles of code review and the rest of the app code, I breezed through the iteration process. I think that I discovered more real issues more quickly than I would have otherwise.

The uncertainty of what my peers might think of my architectural decisions no longer haunted my every move. It was just me and the reality of the problem.nNo questions of cohesiveness or code style. Adherence to only the most reasonable best practices and my own approval was all that was necessary.

The Cons

What gets all code in the end is the fact that it “should work” and it “should not break things”. We thought that this approach would let us have our cake and eat it too on our constrained timeline. Somewhere in our fever-dream, we missed a very important detail…

It was too complicated to be a script.

In order for scripting to be reasonable, your logic had better…

Require no special setup

If it needs an educational REAMDE in order to explain the setup process, this logic probably shouldn’t come packaged as a script.

Not be destructive

If it’s potentially destructive, if it could put a production system in a bad state or lose user data, a script is a bad choice for this logic. Destructive, complicated things need testing, peer review, and as much safety tape around them as you can get. Such things shouldn’t be delivered as a script.

I wager one-million-billion dollars that when all is said and done, a potentially destructive script with complex configuration actually takes more time than it’s equivalent in-app implementation. This is because you will have to rebuild all of the test infrastructure, configuration, and devops stuff that already exists for the actual application in order to convince anyone that the script is safe and useful.

What can be written as a script?

The hallmarks of a good script are…

it’s standalone

It can be composed into a larger set of actions. It doesn’t depend on any other scripts or libraries. It doesn’t require special setup. It only relies on its inputs.

any machine can execute it

You could write a script in python, but if you write your script in anything greater than python 2.7, you will have to set that up yourself. Creating a docker container for your script could help you to get around this. But that starts to break the standalone principle, since docker is now a dependency, as well as any permissions you may need to pull images, etc.

It’s not inherently destructive

The best thing about a script is that it’s easy to use and doesn’t require any hullaballoo in order to maintain it. The only way that a piece of logic could possibly be so easy to use and safe, would be that it doesn’t do anything dangerous.

So, Scripts in Production?

For me, 99% of scripts in production is a no. The 1% of production scripts that can stay are 1. ones that only read the state of a deployment 2. utility-function type scripts that depend on the user to compose them together, which avoids building up too much complexity.