outdated

This document has been the base for developing lmwb for RSS.

Actually the RSS-Feed is no longer used as master input, as it turned out to be incomplete and partly inaccurate.

outdated

RSS Parser Architecture & Specifications

1. Execution Modes & CLI Actions

The script can be invoked with specific actions (positional arguments without leading dashes). If no action is specified, the default behavior is triggered.

• Default (No Action Specified): Executes crawl FIRST, followed immediately by post.
• crawl: Processes the RSS feed, checks for new alerts, parses/splits them, and writes .post files into the queue folder.
• post: Processes the files currently waiting in the queue folder (either publishes them or prints them to screen, depending on options).

2. CLI Options (Flags with Dashes)

Global Options

• -h / --help : Displays standard usage information and exits.
• -ts / --timestamp : Writes timestamps to logfiles (supports two separate logfiles like in lmwb).
• -t / --testlogin : Test Mastodon login using access token from ini-file
• -v / --verbose : Outputs detailed trace/debug messages directly to the screen (STDOUT).

Crawl-Specific Options (Only active/meaningful during 'crawl')

• -alltest : Ignores database/timestamps entirely. Processes all ~250 items in the feed to test parsing and mass-splitting.
• -init : Marks the current feed state as read by updating the DB timestamp, but writes zero posts to the queue.
• -stat : Outputs processing statistics (total alerts, generated posts, split rates) at the end of the crawl.

Post-Specific Options (Only active/meaningful during 'post')

• -sim : Simulation mode. Reads the queue in reverse order to print the posts to the terminal in the exact chronological sequence a user would read them (top-to-bottom).
• -keep : Retains the .post files in the queue folder instead of deleting them after processing (mandatory for -sim, optional for debugging real runs).

3. Queue Mechanics

• The queue consists of exactly ONE text file per final post (not per alert). If an alert is split into multiple parts, each part becomes its own individual file.
• Files are written sequentially by crawl (Part X first, Part 1 last) to ensure proper timeline ordering when pushed to the server.

4. INI Configuration & Key Logic

• Abandoned automated camelCase conversion for INI keys.
• Underscores (_) in INI keys are now strictly used as space placeholders for final output formatting.
• Variable naming convention: Compact, camelCase where necessary for readability, no underscores (avoiding Shift key on German layout).

INI Processing Snippet

# Convert space placeholders globally during INI load
$key =~ s/_/ /g;

5. XML Parsing & Data Extraction (parseRSS)

• Target: BVL Lebensmittelwarnung RSS Feed.
• Detail fields are embedded as plain text inside the XML <description> tag, formatted like Field: Content.
• Parsing strategy uses XML::Twig and iterates over keys defined in $ini->{rss}.

Matching Regex

Uses explicit character class matching letters and spaces inside double brackets, protected via \Q...\E for special characters:

# Inside description line-by-line loop
if ($line =~ /^\s*\Q$xmlfield\E\s*:\s*(.*)$/i) {
    $extracted{$alias} = $1;
}

6. Post Structure & Splitting Logic

Posts are queued in chronological order of posting, but due to Timeline physics (newest on top), Part X is posted FIRST and lands at the bottom. Part 1 is posted LAST and lands at the top. Character tracking uses mastoLen. Framework overhead is calculated dynamically.

Part 1 (Top of Timeline - Posted LAST)

⚠️#Lebensmittelwarnung⚠️ (1/X)
Ware: [Content]

[Payload Part 1]

Quelle: [Link]
[Tags from INI]
Botinfo: [Text from INI]

Subsequent Parts up to Part X (Bottom of Timeline - Posted FIRST)

⚠️#Lebensmittelwarnung⚠️ (X/X)
Ware: [Content]

[Payload Part X]

Quelle: [Link]
[Tags from INI]

Seitenverzeichnis