Loading project...
# libby-to-kindle
**Status:** 🟢 Active
**Phase:** 🎉 FULL PIPELINE + CRON ACTIVE + STATE TRACKING
**Last Activity:** 2026-02-07 (state tracking and cleanup implemented)
---
## Linear Metadata
**Project ID:** `159568cf-6c1b-451b-bc51-2177dc2fb542`
**Team ID:** `96b685fe-2252-47c5-97ee-273d8c484942`
**Last Synced:** `2026-02-06T13:20:08.155Z`
## Overview
Automated pipeline to transfer Libby library loans to Kindle.
**End State:** Cron-based skill that runs overnight, detects new Libby loans, captures EPUBs, decrypts, sends to Kindle, confirms delivery, returns books, and **adds to John's reading list**.
---
## ⚠️ Critical Discovery (2026-02-01)
**Libby has TWO book formats:**
| Format | Indicator | EPUB Download | Our Pipeline |
|--------|-----------|---------------|--------------|
| **Legacy OverDrive** | "Read With..." button | ✅ Yes (ACSM) | ✅ Works |
| **Libby-native** | "Open In Libby" only | ❌ No | ❌ Not supported |
**Format choice locks in!** Once you pick Kindle/Libby/EPUB, it's locked for that loan. Must select EPUB FIRST on new loans before any other choice.
### ⛔ CRITICAL: Correct Flow for EPUB Capture
```
✅ CORRECT:
"Manage Loan" → "Read With..." → "Other Options" → "EPUB" → Get ACSM URL
❌ WRONG (locks format forever):
"Read With Kindle" button on loan card
```
**NEVER click "Read With Kindle"** — it permanently locks the loan to Kindle format and EPUB becomes unavailable.
---
## Current Architecture
```
libby-to-kindle/
├── inbox/ # ACSM files to process
├── processing/ # Temp: DRM'd EPUBs
├── done/ # Clean EPUBs ready for Kindle
├── sent/ # Archive of sent books
├── logs/ # Processing logs
├── config.json # Settings (kindle_email, etc)
└── libby2kindle.py # Main CLI
```
**CLI Commands:**
| Command | Action |
|---------|--------|
| `setup` | One-time initialization |
| `status` | Show queue counts |
| `process book.acsm` | Single book pipeline |
| `process-all` | Process entire inbox |
| `calibre-add book.epub` | Add to Calibre |
| `send book.epub` | Email to Kindle |
| `cleanup` | Remove temp files |
| `config [key] [value]` | View/edit settings |
---
### End Goal
## Workflow
**Phase 1: Capture (Minna)**
1. Open browser (profile=openclaw, Libby authenticated)
2. Navigate to libbyapp.com/shelf/loans
3. For each book:
- Find "Read With..." (on card or via "Manage Loan" dialog)
- If not found → skip (format-locked or Libby-native)
- Click "Other Options..." → "EPUB" → capture URL
4. Download ACSM to inbox/
**Phase 2: Process (CLI)**
```bash
python3 libby2kindle.py process-all
```
ACSM → fulfill → decrypt → cleanup → done/
**Phase 3: Deliver**
- Upload EPUB to Amazon Send to Kindle
- Wait for "In library" status
**Phase 4: Return**
- Return each book in Libby after confirming Kindle delivery
- Frees up loan slots for other library patrons
**Phase 5: Booklist**
- Use **booklist-parser skill** to add processed books
- Skill handles research, personalization, duplicate checking, and append
- Sheet: https://docs.google.com/spreadsheets/d/1-Odbjt2bHC0iJIBA9-yqoeKYnVwl1gGF6wTuaux_dzI
---
## Test Results
| Book | Kindle | Libby Return | Booklist |
|------|--------|--------------|----------|
| I, Robot (Asimov) | ✅ In library | ✅ Returned | ✅ Added |
| The Rest of Our Lives (Markovits) | ✅ In library | ✅ Returned | ✅ Added |
| Rosarita (Desai) | ✅ In library | ✅ Returned | ✅ Already in list |
### Run Log: 2026-02-01 21:18 PST
**Books Processed:**
1. The Rest of Our Lives (Markovits) - sent, returned
2. I, Robot (Asimov) - sent, returned
3. Rosarita (Desai) - sent, returned
**Skipped:**
- Ordinary Magic (audiobook) - not EPUB
- The Ten Year Affair - skip-the-line loan, no EPUB option available
### 🎉 Full Pipeline Proof of Concept (2026-02-01)
Successfully completed the **entire 6-step workflow**:
1. **ACSM Download** - Captured from Libby via browser console
2. **Fulfillment** - acsm-calibre-plugin → encrypted EPUB
3. **Decryption** - DeDRM tools → clean EPUB
4. **Delivery** - Amazon Send to Kindle → Processing → Library
5. **Return** - Books returned in Libby (freeing slots for others)
6. **Booklist** - Both books added to John's reading list spreadsheet
Both books are now in John's Kindle library AND on his curated reading list!
---
### Run Log: 2026-02-07 00:53 PST
**Automated Check (Cron)**
**Books Processed:**
1. Bad Bad Girl (Gish Jen) - ACSM captured, fulfilled, decrypted, added to booklist
**Status:**
- ✅ ACSM capture: Success (via browser console, EPUB format selected)
- ✅ Fulfillment: Success (OverDrive ACSM → encrypted EPUB)
- ✅ DRM removal: Success (48 files decrypted)
- ✅ Booklist: Added (Literary Fiction/Memoir, 352 pages, 2025)
- ⏸️ Kindle delivery: Pending (manual upload needed - mail command incompatible with macOS)
- ⏸️ Libby return: Pending (will return after Kindle delivery confirmed)
**Notes:**
- Book used Legacy OverDrive format (had "Read With..." option)
- Set reading preference to "Kindle" first (for UI consistency), then used "Read With... → EPUB" flow
- Send-to-Kindle email command failed (macOS mail incompatibility)
- Clean EPUB ready in: `done/BadBadGirl_9780593803745_11568273.epub`
**Next Steps:**
- Manual upload to Amazon Send to Kindle: https://www.amazon.com/sendtokindle
- Return book in Libby after confirming Kindle delivery
---
## Recent Improvements (2026-03-01)
### Architecture Change: SSH+Mac ACSM Capture ✅
**Issue:** OverDrive started blocking DRM fulfillment from datacenter IPs. All fulfill requests from the Hetzner VPS returned 403 — even through the VPS browser with correct identity tokens.
**Root Cause:** OverDrive's WAF blocks non-residential IPs for DRM-protected operations (fulfill, chip clone/transfer). Read-only operations (sync, search, availability) still work from any IP.
**Solution — 3-layer fix:**
1. **WireGuard VPN** (VPS ↔ Mac)
- VPS `10.66.66.1` ↔ Mac `10.66.66.2`
- Routes Libby domain traffic through Mac's residential IP
- Config: `/etc/wireguard/wg0.conf`, `vpn/mac-client.conf`
2. **SSH+AppleScript ACSM Capture**
- VPS SSHs to Mac (`johngirard@10.66.66.2`)
- Uses AppleScript to automate Chrome on Mac (navigate to Libby, click EPUB)
- ACSM downloads on Mac, then SCPs back to VPS
- Mac's Chrome has real browser fingerprint + residential IP
3. **Identity Token Injection**
- Token maintained by Mac's Chrome browser (auto-refreshes)
- VPS reads token from Mac's localStorage via SSH+AppleScript
- Injected into VPS browser for loan scanning
**Also fixed:**
- OpenSSL 3.x compatibility: `openssl-legacy.cnf` enables RC2 cipher for Adobe DRM
- oscrypto version regex patched for 3-digit version numbers (3.0.13)
- `libby2kindle.py` auto-loads OpenSSL legacy config
- Kindle upload: uses `upload --ref` (combined arm+click) instead of two-step pattern
**Updated Files:**
- `SKILL.md` — Full rewrite of LIBBY-ACSM-CAPTURE agent, new infrastructure section
- `libby2kindle.py` — Auto-loads `openssl-legacy.cnf`
- `openssl-legacy.cnf` — Persisted OpenSSL 3 legacy provider config
- `vpn/` — WireGuard configs + routing script
- `config/libby-config.json` — Updated chip ID, added VPN/SSH metadata
**Test Results:**
| Step | Status |
|------|--------|
| WireGuard tunnel | ✅ 34ms latency |
| SSH VPS → Mac | ✅ Ed25519 key auth |
| AppleScript Chrome EPUB download | ✅ ACSM captured |
| ACSM → EPUB pipeline | ✅ 67 files decrypted |
| Kindle upload (VPS browser) | ✅ "In library" confirmed |
| Lightbreakers by Aja Gabel | ✅ Full pipeline success |
---
## Recent Improvements (2026-02-20)
### State Cleanup & Orphan Retry Pattern ✅
**Issue:** EPUBs getting stuck in `done/` when kindle-send succeeds but confirm command fails silently. This causes:
- Duplicate ACSM captures on next cron run
- State file out of sync with reality
- Manual cleanup needed
**Root Cause:** The coordinator wasn't verifying that `confirm` and `returned` commands succeeded. These were fire-and-forget `exec()` calls that could fail silently.
**Fixes Applied:**
1. **Kindle Library Verification (Step 0a)**
- Coordinator starts by checking Amazon's actual Kindle library
- Gets ground truth: what books are REALLY uploaded
- Uses filename matching - Amazon filenames match EPUB filenames exactly
- Cross-references Kindle library vs done/ folder to find true orphans
2. **State File Metadata Enhancement (Step 2)**
- After processing ACSM → EPUB, coordinator saves filename → {title, author} mapping in state file
- Captures EPUB filename (newest file in done/) and maps it to real metadata from ACSM capture
- Works for all filenames: with or without ISBN (Rosarita.epub, TheTenYearAffair.epub, atonement_9781400075553.epub)
- Enables safe filename → title lookup for orphan retry
3. **Safe Orphan Retry with Filename Lookup (Step 0b)**
- Uses EPUB filename directly: `atonement_9781400075553` or `Rosarita`
- Looks up real title in state file: `"Atonement"` or `"Rosarita"`
- Passes exact title to libby-return agent (no fuzzy matching, no ISBN dependency)
- No risk of returning wrong book
4. **Error Handling (Step 3)**
- All `exec()` calls for `confirm` and `returned` now check exit codes
- Errors logged and tracked in errors array
- Coordinator won't proceed to return if confirm fails
3. **Current State Cleaned**
- Moved 7 legacy EPUBs (pre-state-tracking) to `archive-pre-state/`
- Confirmed 3 books that were uploaded but untracked: who-knows-you-by-heart, toms-crossing, half-his-age
- Only atonement remains in `done/` for next upload attempt
**Updated Files:**
- `SKILL.md` - Added Step 0 orphan retry pattern
- `SKILL.md` - Added error handling for confirm/returned commands
- `book_state.json` - Cleaned to reflect reality
**Next Run:** Tonight's cron will retry atonement upload, then capture any new books.
---
## Recent Improvements (2026-02-14)
### MIN-458: State Tracking System ✅
**Priority:** P1
Implemented comprehensive state tracking to prevent duplicate ACSM captures when multiple cron runs overlap.
**Features:**
- ISBN-based state file (`book_state.json`) tracks full book lifecycle
- Duplicate detection before processing any ACSM
- Tracks: `processed_at`, `kindle_confirmed`, `libby_returned` timestamps
- Python API for state management (`book_state.py`)
- CLI commands: `pending`, `confirm`, `returned`
**Documentation:** `STATE-TRACKING.md`
### MIN-459: Move EPUBs After Kindle Confirmation ✅
**Priority:** P3
EPUBs now remain in `done/` until Kindle delivery is confirmed, then automatically move to `sent/`.
**Workflow:**
1. Process ACSM → EPUB lands in `done/`
2. Upload to Kindle manually (Amazon Send to Kindle)
3. Run `libby2kindle.py confirm "Book Title"` → moves to `sent/`
4. Run `libby2kindle.py returned "Book Title"` → marks complete
### MIN-460: Standardized EPUB Filenames ✅
**Priority:** P4
All EPUBs now use format: `{slug}_{isbn}.epub`
**Examples:**
- `i-robot_9780316769174.epub`
- `bad-bad-girl_9780593803745.epub`
- `the-ten-year-affair_9780593728307.epub`
**Benefits:**
- Consistent, predictable filenames
- ISBN extractable from filename
- No special characters causing filesystem issues
### MIN-461: Weekly Cleanup Job ✅
**Priority:** P4
Automated cleanup script archives old EPUBs and manages storage.
**Script:** `cleanup_archives.py`
**Features:**
- Archives EPUBs older than 90 days from `sent/` to `archive/`
- Groups by month: `2026-01-books.tar.gz`
- Cleans stale processing files
- Removes old log files (>6 months)
- Storage statistics reporting
**Schedule:** Weekly (Sundays at 3:00 AM)
**Documentation:** `CRON-SETUP.md`
---
## Monitoring