# Recurring Report Ingestion Plan

## Goal
Create a durable, repeatable place for recurring report sources such as:
- Cloudbeds
- QriditHomeWatch
- similar vendor/system report feeds

The system should:
1. download/store raw files when useful
2. parse/extract what matters
3. append durable markdown summaries
4. keep locations predictable
5. clean up disposable raw/download files periodically

## Canonical workspace structure

### Source families
Use one folder per recurring source under:
- `/Users/bobeagle/.openclaw/workspace/reports/sources/`

Examples:
- `/Users/bobeagle/.openclaw/workspace/reports/sources/cloudbeds/`
- `/Users/bobeagle/.openclaw/workspace/reports/sources/qridithomewatch/`
- `/Users/bobeagle/.openclaw/workspace/reports/sources/other/`

### Per-source subfolders
For each source, use:
- `raw/` → original downloaded files
- `parsed/` → JSON/CSV/structured extraction
- `output/` → rendered reports / operator views
- `logs/` → source-specific markdown audit logs if needed

Example:
- `reports/sources/cloudbeds/raw/`
- `reports/sources/cloudbeds/parsed/`
- `reports/sources/cloudbeds/output/`
- `reports/sources/cloudbeds/logs/`

## Durable markdown layer

### Master index
Create / maintain:
- `/Users/bobeagle/.openclaw/workspace/RECURRING_REPORT_SOURCES.md`

### Per-source markdown logs
Examples:
- `/Users/bobeagle/.openclaw/workspace/CLOUDBEDS_READ_LOG.md`
- `/Users/bobeagle/.openclaw/workspace/QRIDITHOMEWATCH_READ_LOG.md`

These markdown files should be treated as the durable narrative history.

## Raw file policy
Raw files are allowed and useful, especially for debugging, re-parsing, and validating source truth.

Rule:
- store first
- summarize into markdown
- keep raw files in known folders
- prune old disposable raw files monthly if they are no longer needed

## Cleanup policy

### Monthly cleanup target
Once per month:
- remove disposable duplicate downloads
- remove outdated temp files
- keep canonical archived raw files only where needed
- keep markdown histories
- keep parsed structured outputs that provide historical continuity

### Important distinction
Do **not** delete durable markdown logs or useful parsed history.

The cleanup target is mostly:
- junk downloads
- temp files
- duplicate raw exports

## Operational rule
For recurring report emails or links:
1. identify the source family
2. save the raw file to the correct source folder if useful
3. extract/parse the content
4. append a markdown summary entry
5. publish a clean summary/report if relevant

## Current immediate source families
- Cloudbeds
- QriditHomeWatch
- other recurring hotel/business system reports as discovered

## Design principle
Markdown is the durable human memory.
Raw files are the short-to-medium-term source archive.
Parsed structured files are the reusable machine layer.