taarr

Changelog

propolice@gmail.com (Ed) — Sat, 07 Sep 2024 00:00:00 +0000

### 2.4.0

Sulk Harmonics

Support no-namespace scripts rr no_namespace
No longer support funky arg:one:two arguments
Update dependencies
Fix panic related to reading from nonexistent script directory

### 2.3.0

Compacted Monastery

Prelude (script.pre) and epilogue (script.post) scripts
Update dependencies
Major refactor

### 2.2.1

Unsafe Cupcake

Fix output during parallel runs
Print README* file when no namespace:script specified
Include .lib in rrd (dump) mode

### 2.2.0

Craving Detonator

Log SIGINT
Initial support for Windows
Update dependencies

### 2.1.1

Buggy Oven

Use arguments for the LOG field
Update dependencies

### 2.1.0

Degraded Mastiff

Colorized rrl output
Changed rrl headers
Changed OP to LOG environment variable
Removed reading from OP file feature
Added .ssh/config and ssh_config as valid hosts files
Changed terminal log line color to cyan
Various code fixes and improvements

### 2.0.1

Hypnotic Antennae

Fix output in terminal mode
Fix signal handling over SSH
Update dependencies

### 2.0.0

Groggy Pauper

Removed some external dependencies to slash around 500KiB from the executable size
New extensive testing suite
New VARS file for includes and variables
Changed SSH config from rr.hosts to HOSTS
Changed log filename from rr.json to LOG
Changed rrl mode output headers and log format
Changed string for “repaired” result/status detection to __REPAIRED__
Changed __REPAIRED__ output detection from STDOUT to STDERR
Quicker copying over SSH because of one less SSH connection
Quicker remote runs, removed SSH hostname matching
More compact rrl mode output
STDOUT now streamed in terminal mode
Several code improvements and dependency upgrades

### 1.0.4

Update dependencies
Free performance improvement
Code style
Improve “make” scripts

### 1.0.3

Update dependencies
Remove spinner
Use isatty to determine console

### 1.0.2

Update dependencies
Indicate connections in the output
Fix handling of huge log files
Better ansi escape sequence for the spinner
Fix tests

### 1.0.1

Update dependencies

### 1.0.0

Add plain mode for reusing output

### 0.19.0

Remove script level task file
- Top-level TASK file is now the task field in the log
- Can be overridden by env variable TASK
Better looking table separator in report mode
Shorter start timestamps in report mode

### 0.18.0

Added Teleport mode rrt & rro
Added report mode rrl
“Idempotence” by checking ‘+++++repaired+++++’ string from STDOUT
Log to rr.json even if not in console mode
“elapsed” to “duration”
ID’s are just 8 characters now
Duration truncated at seconds

### 0.17.0

Local invocation can now use interpreter that take in STDIN
- Another interpreter besides a posix shell can’t take in arguments or use .files
- STDIN works with lua(LadyLua), lua(PUC), python, dash, bash, zsh
Overhaul Makefile
Fix local tests

### 0.16.0

Better sudo with password handling
Fix various spinner issues
Changed spinner: line for copying and dots for runs
Add spinner when checking SSH hostname
Disable ssh compression
Interpreter falls back to “sh” instead of $SHELL environment variable

### 0.15.0

Change from sftp to tar for ssh copies
Support ssh copying with sudo passwords
Also replace rsync with tar for container runs

### 0.14.0

Custom ssh_config (rr.hosts)
app=rr added to JSON log

### 0.13.0

security issue: omit environment variables from logged code

### 0.12.0

Support sudo invocations
Interpreter can be set in shell file
STDOUT and STDERR encoded to base64 in JSON log
STDOUT and STDDER also logged as DEBUG entries even without errors
Only take the first line from the task file
A breaking change; exported environment variables must prefix rr__
Code of script logged along with DEBUG entries

### 0.11.0

Script start marked as Info
Start and Finish now properly marked in json log
Contents of task file used for audit trail
Added Bash completion script
Replaced ULID with random hexadecimal string

Reference

propolice@gmail.com (Ed) — Sat, 20 Jul 2024 00:00:00 +0000

Table of Contents

Why
Multicall modes
- Console
- Silent
- Dump
- Sudo
- Log
Audit trail
Readmes
Environment variables
Idempotence
Hierarchy
VARS
Shell
Notes
Decision record

## Why

I built my own Lua-based configuration management software. A little more than thousand commits in, I realized the oneshot nature of shell scripts more convenient.

CFEngine, Puppet, Chef and others did not offer advantages over a combination of rerun/bashing + drist. Cons mostly outweight the pros. Slow and complicated are the common complaints. If you’re just templating shell commands then plain shell scripts should suffice.

According to the Lindy effect; shell scripts, openssh and tar will outlive these mentioned CM software.

Huge shell scripts can be unwieldy. taarr will help you manage your scripts.

## Multicall modes

You can use a symlink to activate the modes.

Example for sudo mode:

ln -s rr rrs

### Console

Verbose mode.

When called as rrv or a console is detected it runs in verbose mode.

$ rrv make:yyy
29 Jun 24 11:55 +0800 rr 2.2.0 “Craving Detonator”
29 Jun 24 11:55 +0800 Running make:yyy via local…
29 Jun 24 11:55 +0800 Running yyy…
 local │ -rw-r--r-- 1 root root 2439 Apr  6 10:39 /etc/passwd
 local │ stdout last
29 Jun 24 11:55 +0800 Failure running script!
 local ┌─ stderr
 local │
 local │ stderr
 local │ next
 local │ last
 local │
 local └─
 local ┌─ debug
 local │
 local │ exit status 255
 local │
 local └─
29 Jun 24 11:55 +0800 Total run time: 1s. Something went wrong.

In this mode it also logs to the file LOG in the current working directory.

NOTE: logs STDOUT and STDERR as a debug entry when no errors are detected

TIP: Example command line to extract STDOUT from the log file

grep 3EB340F6 LOG | grep stdout | cut -f28 -d\" | base64 -d

### Silent

When called as rr and a console is not detected it only shows errors as structured JSON.

{"level":"error","stdout":"ss\n","stderr":"ee\n","time":"2021-07-20T20:16:04+08:00","message":"Output"}
{"level":"error","elapsed":"1.798478ms","time":"2021-07-20T20:16:04+08:00","message":"Something went wrong."}

### Dump

When called as rrd, dumps the generated script. This is mainly for debugging and allows running scripts generated from a managed namespace without the rr executable.

### Sudo

When called as rrs, asks for a sudo password for the remote user. Similar to the ansible options --become -K. Also turns on console mode since a TTY is expected anyway.

### Log

When called as rrl, reads the JSON file LOG then shows a log of executions for audit trail.

## Audit trail

A log is generated for each script invocation. The JSON file LOG contains an audit trail of all script invocations.

The environment variable LOG is logged as the log field. Without the variable the arguments are used for the field. If no variable or arguments passed then it is logged as UNDEFINED.

This can be useful for audit trails or for changelogs. Before running a script you should set the LOG environment variable.

LOG='Upgrade a thing for bugfix` rr program:upgrade

## Readmes

### Top-level

When invoked without any arguments, rr prints the README in the current working directory.

### namespace/script

Any case insensitive file named readme* in the namespace and script directories can be shown by invoking rr in the following ways:

Prints namespace/readme*

rr namespace
rr namespace/

Prints namespace/script/readme*

rr namespace/script
rr namespace/script/

## Environment variables

Any environment variable prefixed with rr__ are passed to the script.

For example, the rr__USERNAME=test environment variable is passed to the script as USERNAME=test

$ cat tmp/env/script
echo $USERNAME

$ env rr__USERNAME=test rr tmp:env
Thu, 22 Jul 2021 18:13:32 +0800 rr 0.10.0 "Kilowatt Triceps"
Thu, 22 Jul 2021 18:13:32 +0800 Running tmp:env via local...
Thu, 22 Jul 2021 18:13:32 +0800 Running env...
Thu, 22 Jul 2021 18:13:32 +0800 Done. Output:
 local ┌─ stdout
 local │
 local │ test
 local │ 
 local └─
Thu, 22 Jul 2021 18:13:32 +0800 Total run time: 101.648007ms. All OK.

## Idempotence

The default behavior of scripts is to indicate two results: no errors(ok) and failure(failed). To emulate idempotence your script can print the following string to STDERR in it’s own line and exit 0:

__REPAIRED__

When writing your script you have to check if an operation is needed to apply a change or not. Such a precondition can be written like this:

if test -d /some_dir
then
  exit 0
else
  {
    mkdir /some_dir
  } >/dev/null 2>&1 || exit 1
  >&2 printf "__REPAIRED__\\n"
  exit 0
fi

In the JSON log and report mode the result value will say repaired.


No changes, script exited	–>	ok
Script succesfully executed changes	–>	repaired
Repair needed but unable to apply changes	–>	failed

To borrow terminology from *CFEngine*. There are no changes when the **desired state** is already the expected outcome.


Any state	–>	Desired state
Desired state	–>	Desired state

## Hierarchy

TOPLEVEL                            <--- Directory of namespaces or a project
├── .files                          <--- synced to any host
├── .files-avocado                  <--- synced to the avocado host
├── .lib                            <--- files sourced by all scripts
├── VARS                            <--- sourced by all scripts, ideal for frequently changed variables
├── LOG                             <--- Log generated (JSON); can be printed with `rrl`
├── HOSTS                           <--- custom SSH config
└── namespace
    ├── .files                      <--- synced to any host when namespace:* is called
    ├── .files-avocado              <--- synced to the avocado host when namespace:* is called
    ├── .lib                        <--- sourced along with namespace:* scripts
    ├── readme                      <--- Documentation for namespace
    └── script directory
        ├── .files                  <--- synced to any host when namespace:script is called
        ├── .files-avocado          <--- synced to the avocado host when namespace:script is called
        ├── .lib                    <--- sourced along with namespace:script scripts
        ├── readme                  <--- Documentation for script
        ├── script                  <--- the main shell script
        ├── script.pre              <--- run before the main script
        ├── script.post             <--- run after the main script
        └── shell                   <--- shell interpreter for the script

## VARS

The VARS file is sourced by all scripts in all namespaces. It is ideal for variables that frequently change.

## Shell

The shell used to run the script can be set within the file namespace/script/shell. Example:

$ cat namespace/script/shell
/usr/bin/fish

## Notes

Tested on Linux and macOS.

Remote host only requires OpenSSH server and tar(1).

Scripts should reference $@ if it wants to use arguments passed.

## Decision record

Environment variables are not logged. They may contain secrets.

Examples

propolice@gmail.com (Ed) — Sun, 30 Jun 2024 00:00:00 +0000

Table of Contents

## Tutorial

In the Git source tree, the TUTORIAL directory inside README.d contains a hierarchy that persistently enables IP forwarding through sysctl upon the remote SSH host named avocado

First you have to setup SSH public-key authentication for a remote host with hostname avocado.

Once that is done, you can proceed with the example:

cd TUTORIAL
rr avocado sysctl:apply

### Steps that `rr` performs

Copies .files-avocado/ to avocado:/
Generates the script:

#!/bin/sh
unset IFS
set -efu
PATH=/bin:/sbin:/usr/bin:/usr/sbin
LC_ALL=C
sysctl --system

Runs the script on host avocado via SSH.

## Local invocation of scripts

Run locally, the default without a hostname or container PID specified.
Requires tar(1) for .files
The following are equivalent:

rr namespace:script

rr localhost namespace:script

## Run scripts inside a local container

NOTE: Linux only

Run on a local container’s PID via nsenter(1). Requires tar(1) for .files.

rr 1333 namespace:script

## Run scripts remote via SSH

rr remotehost namespace:script

If a file named .ssh/config is in the current directory then that is used as the SSH config instead of the default ~/.ssh/config.

## Install cri-tools to remote Ubuntu host

$ mkdir -p examples/install-critools
$ cat > examples/install-critools/script <<-'EXAMPLE'
. /etc/os-release
echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/ /" | tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
curl -L "https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/Release.key" | apt-key add -
apt-get update
apt-get -y install cri-tools
EXAMPLE
$ cat .ssh/config
Host remotemost
Hostname 203.0.113.1
User root
$ rr remotehost example:install-critools
...
...

Howto

propolice@gmail.com (Ed) — Sat, 29 Jun 2024 00:00:00 +0000

Table of Contents

## Parallel execution

Shell scripts can execute rr natively.

#!/bin/sh

# Run namespace:script in parallel
rr somewhere namespace:script &
rr another namespace:script &
wait
# Next step after above
rr remotehost namespace:anotherscript

## Multiple hosts

For loops can be used to execute scripts on multiple hosts.

$ for x in remotehost1 remotehost2; do rru $x namespace:script; done

taarr

Changelog

### 2.4.0

### 2.3.0

### 2.2.1

### 2.2.0

### 2.1.1

### 2.1.0

### 2.0.1

### 2.0.0

### 1.0.4

### 1.0.3

### 1.0.2

### 1.0.1

### 1.0.0

### 0.19.0

### 0.18.0

### 0.17.0

### 0.16.0

### 0.15.0

### 0.14.0

### 0.13.0

### 0.12.0

### 0.11.0

Reference

## Why

## Multicall modes

### Console

### Silent

### Dump

### Sudo

### Log

## Audit trail

## Readmes

### Top-level

### namespace/script

## Environment variables

## Idempotence

## Hierarchy

## VARS

## Shell

## Notes

## Decision record

Examples

## Tutorial

### Steps that rr performs

## Local invocation of scripts

## Run scripts inside a local container

## Run scripts remote via SSH

## Install cri-tools to remote Ubuntu host

Howto

## Parallel execution

## Multiple hosts

### Steps that `rr` performs