yadm — Initialization with imported key failing ( Issues with yadm 2/2)

[YggdrasiI]

Hello, I've integrated transcrypt into my yadm repository and stumbled over
a few things. No critical errors. I just wanna post some notes about the issues.

Version: 2.3.1-pre
Commit: 016b2e4

cd .local/share/yadm/repo.git
transcrypt --import-gpg="$CREDENTIALS_PATH_GPG"

Shows:

*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.

Repository metadata:

  GIT_WORK_TREE:  /home/username
  GIT_DIR:        /home/username/.local/share/yadm/repo.git
  GIT_ATTRIBUTES: /home/username/.gitattributes

The following configuration will be saved:

  CONTEXT:  default
  CIPHER:   aes-256-cbc
  PASSWORD: XXXX

Does this look correct? [Y/n] 

fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).

Maybe, transcrypt stumbled about the GIT_WORK_TREE ?! It aborts during the decryption of the first file.

 yadm status
 […]

Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	deleted:    ../alt/.i3/lock_screens##class.Private/Info.txt

So it stops at the decryption of the first file of yadm ls-crypt.
After yadm checkout -f the decryption/encryption with transcrypt works, so the importing itself works.

Regards,
Olaf

[jmurty]

It looks like it might be the force_checkout function that is failing, it calls git ls-crypt-${CONTEXT} (CONTEXT is "default" by default) then git checkout --force HEAD -- "$file" for each file listed.

I'm not familiar with yadm. Do you know what is different about how it invokes git, or the environment variables it sets, that could cause these command to fail?

[YggdrasiI]

Similar to issue 189 the explicit setting of GIT_DIR will work around the issue:

cd /tmp/tmp.jK4dyZgEQw/.local/share/yadm/repo.git/
export GIT_DIR=$(pwd)
transcrypt --import-gpg=[SomeKey.asc]

During the debugging I've found a typo in export_gpg. Missing '.' before cipher:

@@ -1072,7 +1072,7 @@ export_gpg() {
        fi
 
        local current_cipher
-       current_cipher=$(git config --get --local "transcrypt${CONTEXT_CONFIG_GROUP}cipher")
+       current_cipher=$(git config --get --local "transcrypt${CONTEXT_CONFIG_GROUP}.cipher")
        local current_password
        current_password=$(load_password "$CONTEXT_CONFIG_GROUP")
        mkdir -p "${CRYPT_DIR}"

Regards Olaf

[YggdrasiI]

I've wrote a little test script for the error, see below. My conclusion: The force_checkout() function assumes that the repo directory and the worktree directory are the same. Using "$GIT_DIR" instead of "$REPO" and fixing relative paths in the for-loop resolving the issue.

diff --git a/transcrypt b/transcrypt
index e75ae7a..db81628 100755
--- a/transcrypt
+++ b/transcrypt
@@ -781,11 +781,13 @@ force_checkout() {
                # but we already made sure the repo was clean during the safety checks
                local encrypted_files
                encrypted_files=$(git "ls-crypt-${CONTEXT}")
-               cd "$REPO" >/dev/null || die 1 'could not change into the "%s" directory' "$REPO"
+               # Fix for 'yadm': If 'git config --local core.worktree' is set, the paths '$REPO' and '$GIT_DIR' can differ
+               # Using GIT_DIR here and in for-loop prefix '${REPO:-.}/'
+               cd "$GIT_DIR" >/dev/null || die 1 'could not change into the "%s" directory' "$GIT_DIR"
                IFS=$'\n'
                for file in $encrypted_files; do
-                       rm -f "$file"
-                       git checkout --force HEAD -- "$file" >/dev/null
+                       rm -f "${REPO:-.}/${file}"
+                       git checkout --force HEAD -- "${REPO:-.}/${file}" >/dev/null
                done
                unset IFS
        fi

Test script for --import-gpg, etc:

#!/bin/bash
# Test script to check 'transcrypt --import-gpg' inside of yadm repo.
#
# It creates an yadm repo in a temporary folder, encrypting a file and then tries to run --upgrade.
# 
# Script assumes that yadm is installed.
# Debian:        sudo apt install yadm
#

set -euo pipefail
#set -x

## Create temp dir and delete it at exit of script.
YADM_TMP_DIR=$(mktemp -d)
test "$?" -ne 0 && exit -1

function introspect {
  echo -e "\n\n\tWrite 'bash' to open interactive shell for introspection." \
    "\n\tWrite [Enter] to clear temp directory '$YADM_TMP_DIR'\n"
  read SHELL_OR_QUIT

  if [ "$SHELL_OR_QUIT" = "bash" ] ; then
    #bash --rcfile <(echo "export PS1='$PS1 |> '")
    bash --rcfile <(echo "export PS1='|> '")
  fi
}

function cleanup {                                                                                                                       
  introspect
  rm -rf "$YADM_TMP_DIR"
}                                                                                                                                        
trap cleanup EXIT                                                                                                                        
trap cleanup SIGINT  


## Download current transcrypt version
cd "$YADM_TMP_DIR"
wget "https://raw.githubusercontent.com/elasticdog/transcrypt/refs/heads/main/transcrypt"
#cp $HOME/transcrypt transcrypt

# Make it available for the other tools
chmod +x transcrypt
PATH=$(realpath "$YADM_TMP_DIR"):$PATH

## Init yadm repo
#  Using temp dir instead of $HOME
WORKTREE_ROOT="$YADM_TMP_DIR"

#   To avoid writing yadm config into $HOME/.config/yadm, relocate home dir…
export HOME="$YADM_TMP_DIR"

# Note that 'git commit' still complains about missing global variables even
# if the local variables are set. Thus, we have to set .gitconfig
echo "[user]
email = test@transcrypt.git
name = Test Test" > "${YADM_TMP_DIR}/.gitconfig"

yadm init -w "$WORKTREE_ROOT"

# Add a file
cd "$WORKTREE_ROOT"
echo "Public content" > "public_file"
yadm add "public_file"
yadm commit -m "Adding unencrypted file"

echo "Yadm is placing bare repo files here: $(yadm introspect repo)"

## Setup transcrypt in repo…
yadm transcrypt -y


## Add info about file to encrypt
echo 'sensitive_file  filter=crypt diff=crypt merge=crypt' > \
  "$(yadm introspect repo)/info/attributes"

echo 'Secret content' > "sensitive_file"
yadm add sensitive_file
yadm commit -m "Adding encrypted file"


## Test --export-gpg and --import-gpg
# Prevent update of users keys (even if $HOME is not redirected)
export GNUPGHOME="$YADM_TMP_DIR/.gnupg"

# Generate Test Key
gpg --batch --passphrase '' --quick-gen-key Transcrypt
#gpg --list-keys

# Export key
cd "$(yadm introspect repo)"
transcrypt --export-gpg=Transcrypt
test -f "crypt/Transcrypt.asc" && echo "Export ok"

# Uninstall transcrypt as preparation for --import-gpg
mv "crypt/Transcrypt.asc" "$YADM_TMP_DIR/."  # otherwise uninstall is failing
#transcrypt --uninstall -y  # Again, GIT_DIR is missing.
yadm transcrypt --uninstall -y
yadm checkout -f  # Because trancrypt leaving repo  in dirty state

# Import key
transcrypt --import-gpg="$YADM_TMP_DIR/Transcrypt.asc" -y

[jmurty]

Thanks for digging into this issue and providing the great test scripts.

I have run some more tests and used your test scripts and I'm coming to the unfortunate conclusion that this probably isn't an issue that can be fixed inside transcrypt. The root issue seems to be that in your yadm worktrees setup, git itself needs to have the $GIT_DIR environment variable set correctly to be able to provide even basic settings required by transcrypt.

In other words, you need to run yadm transcrypt instead of just transcrypt or to set GIT_DIR=$(yadm introspect repo) before running transcrypt.

For example in a working tree directory set up by yadm per your test script, these key config lookup commands required by transcrypt fail with different error messages: git rev-parse --show-toplevel and git config --get --local transcrypt.version

As soon as you set GIT_DIR=$(yadm introspect repo) those same commands work as expected in the working tree directory.

But unless GIT_DIR is set somehow by yadm, as far as I can tell there isn't a way for transcrypt to learn what that GIT_DIR path should be. Any attempt to look up basic Git config settings fails, per the above. So unless I'm missing something, transcrypt (and most likely any git commands or tools) need to be run inside the yadm environment to work correctly.

[YggdrasiI]

[…]
In other words, you need to run yadm transcrypt instead of just transcrypt or to set GIT_DIR=$(yadm introspect repo) before running transcrypt.

Yes, during my tests in the last days this solution works. In my memory, this wasn't enough for an unknown transcrypt command, but I can no longer reproduce this.

[…]
But unless GIT_DIR is set somehow by yadm, as far as I can tell there isn't a way for transcrypt to learn what that GIT_DIR path should be. Any attempt to look up basic Git config settings fails, per the above. So unless I'm missing something, transcrypt (and most likely any git commands or tools) need to be run inside the yadm environment to work correctly.

Nesting transcrypt commands with yadm is probably a sufficient solution, but I want note that's there is no magic learning about the path.

The changes made by yadm just affects GIT_DIR and GIT_WORK_TREE, see yadm enter env | grep -i git . And in transcrypt you fetching the same values: GIT_DIR and REPO.

  # the current git repository's top-level directory
  readonly REPO=$(git rev-parse --show-toplevel 2>/dev/null)                                                                             
                                                                                                                                         
  # the current git repository's .git directory                                                                                          
  readonly RELATIVE_GIT_DIR=$(git rev-parse --git-dir 2>/dev/null || printf '')                                                          
  readonly GIT_DIR=$(realpath "$RELATIVE_GIT_DIR" 2>/dev/null) 

Just the usage of git config --local after cd "$REPO" is problematic because you could not rely on "$GIT_DIR" == "$REPO/.git".
I found just the one occasion of cd "$REPO" (mentioned in the previous post) With this change I can use transcrypt without nesting by yadm.

[jmurty]

You are right that there is nothing magic about $GIT_DIR because Git itself understands and uses this environment variable, but when the git repo/working-copy and metadata directories are in unusual places (i.e .git/ directory is not in the repo) that environment variable is required for git config --local lookups to work.

Prefixing commands with yadm does this job of providing the required GIT_DIR environment variable. For non-standard setups there is no way for transcrypt to figure out what GIT_DIR should be if it's not provided.