Tech Brief: The sls (see-latest-screenshot) Command

How the sls command finds the newest screenshot, loads it into Hermes, and gets it analyzed with vision – the current implementation and how to extend it.

1. What It Does

sls (“see latest screenshot”):

• Finds the newest image in $SCREENSHOTDIR (default ~/Pictures/Screenshots; in this workspace it is set to ~/av/ast/img/screenshots).
• Copies it to a space-free temp path (/tmp/sls_latest.png) – vision tooling breaks on paths with spaces.
• Runs a one-shot Hermes call (hermes -z) with a prompt instructing the agent to use vision_analyze on that path.
• The screenshot_tool skill makes the same flow work in-session: when the user types sls or “see latest screenshot”, the agent finds the latest file, copies it, and calls vision_analyze on it.

2. Implementation

The command is a small bash script in ~/av/bin/sls (the workspace bin dir is on PATH):

#!/usr/bin/env bash
# sls -- see-latest-screenshot
# Finds the newest screenshot in SCREENSHOTDIR and asks Hermes to analyze it.
#
# Usage:
#   sls                  # describe the latest screenshot
#   sls "your question"  # ask a specific question about it

SCREENSHOTDIR="${SCREENSHOTDIR:-$HOME/Pictures/Screenshots}"

LATEST=$(find "$SCREENSHOTDIR" -maxdepth 1 \
  \( -name "*.png" -o -name "*.jpg" -o -name "*.jpeg" \) \
  -print0 2>/dev/null \
  | xargs -0 ls -t 2>/dev/null | head -1)

if [[ -z "$LATEST" ]]; then
  echo "sls: no screenshots found in $SCREENSHOTDIR" >&2
  exit 1
fi

# Copy to a space-free tmp path -- vision_analyze breaks on paths with spaces
TMPFILE="/tmp/sls_latest.png"
cp "$LATEST" "$TMPFILE"

QUESTION="${1:-Describe everything you see in this screenshot in detail.}"

echo "sls: loading $LATEST" >&2

PROMPT="Use vision_analyze to look at the image at this path: ${TMPFILE}
Question: ${QUESTION}"

exec hermes -z "$PROMPT"

2.2 The Hermes skill

The screenshot_tool skill (at ~/.hermes/skills/screenshot_tool/SKILL.md) handles the in-session flow:

• Triggers: user says sls, “see latest screenshot”, “show latest screenshot”, “look at latest screenshot”, or “what’s on my screen”.
• Steps: find the most recent file in $SCREENSHOTDIR (same find pipeline as the script), then call vision_analyze with the path exactly as given – no URL-encoding.

2.3 Registration

• The script is executable in ~/av/bin/ (the workspace PATH convention).
• The skill lives at ~/.hermes/skills/screenshot_tool/ and is picked up when Hermes loads its skills.
• SCREENSHOTDIR is exported in the shell environment.

2.4 Restart / reload

• PATH changes need a new shell.
• Skill changes need a Hermes session restart.

3. How It Works Internally

1. File discovery: find lists .png/.jpg/.jpeg files one level deep; xargs -0 ls -t | head -1 picks the newest by mtime (space-safe via null separators).
2. Normalization: the file is copied to /tmp/sls_latest.png so the path handed to the vision tool has no spaces.
3. Invocation: hermes -z runs a one-shot agent call with a prompt containing the path and the question.
4. Vision tool auto-trigger: the prompt tells the agent to use vision_analyze(image_url=/tmp/sls_latest.png); the skill triggers the same call when the user types sls in-session.

Why find + xargs -0 instead of a plain glob? - Handles filenames with spaces safely (-print0 / -0). - Faster than a Python glob for directories with thousands of screenshots. - ls -t gives the newest by modification time without per-file stat calls in the shell.

4. Testing the Command

Manual test:

1. Ensure you have screenshots in $SCREENSHOTDIR (or the default ~/Pictures/Screenshots).
2. Run sls – expect “sls: loading ” then a vision analysis of the newest image.
3. Run sls "what browser tabs are open?" – expect an answer specific to the image.
4. In-session, type sls – expect the same analysis without leaving the session.

Debugging:

If the command fails silently: - Check the script’s stderr: run sls directly and read the error. - Verify SCREENSHOTDIR is set correctly for a custom location. - Test the find pipeline manually:

find "${SCREENSHOTDIR:-$HOME/Pictures/Screenshots}" -maxdepth 1 \
  \( -name "*.png" -o -name "*.jpg" -o -name "*.jpeg" \) \
  -print0 | xargs -0 ls -t | head -1

• Check /tmp/sls_latest.png exists after a run.

5. Design Considerations

5.1 Error Handling

• Missing/empty SCREENSHOTDIR or no matching images: prints a message to stderr and exits 1 – never crashes the shell.
• The copy to tmp preserves the original; the tmp file is simply overwritten on each run.

5.2 Performance

Finding the newest file in a directory of 10,000+ screenshots: - find + xargs -0 ls -t | head -1: ~100ms - naive shell glob + stat loop: slower and space-fragile

5.3 Security

• No arbitrary command injection: file paths come from find/ls, not user input; the question argument is passed inside the prompt string, not executed.
• The tmp copy avoids path-quoting issues with unusual filenames.

5.4 Extensibility

Future improvements could include: - sls 5 – show the 5 latest screenshots - sls --today – filter by date - sls --dir=/custom/path – override SCREENSHOTDIR - sls --no-vision – just print the path, don’t analyze

6. Best Practices for CLI-Plus-Skill Commands

1. Keep the script small – do work in helpers if logic grows.
2. Use find with -print0 / xargs -0 – space-safe for real-world filenames.
3. Normalize to a space-free path before handing to tooling.
4. Respect env conventions – SCREENSHOTDIR with a sane default.
5. Make the skill match the script – same discovery logic, same trigger vocabulary, so both entry points behave identically.
6. Test on a fresh session – PATH/skill changes take effect on restart.

7. Reverting Changes

Revert via git:

cd ~/av
git checkout -- bin/sls

Manual revert: - Remove ~/av/bin/sls. - Remove ~/.hermes/skills/screenshot_tool/.

Restart Hermes after any change.

8. Alternative Approaches

Without a script: - In-session, the agent can do it directly: find the latest file in SCREENSHOTDIR, copy to /tmp/sls_latest.png, call vision_analyze. - A shell alias that expands to the same one-shot:

alias sls='hermes -z "Use vision_analyze on /tmp/sls_latest.png"'