Here Today, Gone When You Exit: Proper Tempfiles in Shell Scripts

raindog308

Earlier this week I inherited a pile of utterly craptacular shell scripts (running in production, naturally) at work and spend the week essentially rewriting them so they didn't look like they'd been bought on fiverr.

I was told "weird things happen sometimes and we can't figure out a pattern...and the busier we are, the more weird things happen".

It was pretty easy to find the reason why...this identical line was in many scripts that frequently executed:

TEMPFILE=/tmp/tempfile

Every execution was a timing Russian roulette.

Although in one script they used

TEMPFILE=/tmp/tempfile.$$

...and just never rm'd it so there were tens of thousands of tempfiles in /tmp.

As I cleaned up I wrote up a tutorial on how to use tempfiles properly, and how to automatically clean them up when your script exits. It's easy to clean up one, but cleaning up multiple requires some extra thought.

I had a couple solutions in mind (UUID patterns, using an array) but also found a 2009 LJ article with a third approach.

Enjoy: https://lowendbox.com/blog/here-today-gone-when-you-exit-proper-tempfiles-in-shell-scripts/

jsg

BUMP

because that nice tutorial doesn't deserve to drop so quickly

AlwaysSkint

I like a KISS philosophy:
TEMPFILE=/tmp/tempfile.$$
.. along with a daily cron that does..
/usr/bin/find //tmp/tempfile.* -mtime +60 -exec rm {} \; > /dev/null 2>&1

This may suffice.

I enjoyed that article, which reminded me that even with a formal sysadmin training, you can never know all the nuances of bash scripting etc.
(I only remembered enough to do slightly more than basic tasks.)

Liso

Nice article

dahartigan

At first, I thought the title was an article about exit scammers. Here today, gone tomorrow.

raindog308

@AlwaysSkint said: .. along with a daily cron that does..

So handling a single tempfile cleanup with trap is easy.

But to automatically clean up tempfiles, between you and the article we've now listed:

• using a UUID pattern
• using a temp directory
• using a function to "register" tempfiles and add them to a list
• not worrying about them and cleaning up aged tempfiles via cron

Seems to me like there is a missing feature here :-)

Though now that I think about it, I don't think either perl or python offer a "guaranteed cleanup" when they exit unless you code it yourself.

MeAtExampleDotCom

@raindog308 said: I don't think either perl or python offer a "guaranteed cleanup" when they exit

Not as a built-in.

unless you code it yourself.

There are libraries out there that do that sort of thing, but they expect you to call the “clean up” function as you exit, and register a global exception handler (or whatever your language/framework supports to this effect) and/or a signal handler to fire that automatically upon uncontrolled exit. You might still have some left around to remove by other means, due to really uncontrolled exits.

Also take care when using mktemp to pick an arbitrary name to avoid conflicts – for some reason a lot of people assume such files are not created until you output something when they are created as zero-length. I've seen scripts create one initially whether they need it or not, then never remove it assuming that not having used it means it didn't really get created.

(or empty, if creating a directory) by mktemp itself. (edit: actually looked at the linked article, and it explicitly shows cleaning up in repose to bash's EXIT signal – note that this is a bashism so don't rely on it unless you explicitly specify bash as the script runner instead of relying on the OS/user's default)

__no_preserve_root

Have you tried systemd-tmpfiles? It looks pretty neat.

raindog308

@MeAtExampleDotCom said: (or empty, if creating a directory) by mktemp itself. (edit: actually looked at the linked article, and it explicitly shows cleaning up in repose to bash's EXIT signal – note that this is a bashism so don't rely on it unless you explicitly specify bash as the script runner instead of relying on the OS/user's default)

Yes...I think in other shells you have to trap each signal.

raindog308

@__no_preserve_root said: Have you tried systemd-tmpfiles? It looks pretty neat.

The only negative is that I think you either have to have root or play by root's rules (put files where they'll be cleaned up), or convince someone with root to put in new rules for you. But interesting, and apparently it's been around for quite a while. Here's an overview from 2016:

https://developers.redhat.com/blog/2016/09/20/managing-temporary-files-with-systemd-tmpfiles-on-rhel7

dane_doherty

imagine not using pastebin for temporary files

HackedServer

Nice article. If you're having to make complex bash scripts, especially in a team environment, this is a good read too:
https://google.github.io/styleguide/shellguide.html

raindog308

@HackedServer said: https://google.github.io/styleguide/shellguide.html

Indent 2 spaces. No tabs.

Today I learned why I don't want to work for Google.

Forcing others to use your preferences for indentation is evil.