Feapder

Overview

Implement feapder 1.9.2 code in the style expected by the framework's own docs, templates, and tests. Choose the right spider base class first, then follow the matching startup, request, parse, persistence, and debugging patterns.

Workflow Decision Tree

1. Classify the request before writing code:
2. Use AirSpider for small, local, non-distributed jobs.
3. Use Spider for Redis-backed distributed crawling, resumable queues, and automatic item persistence.
4. Use TaskSpider when seeds come from MySQL or Redis task tables and the framework should manage seed loading.
5. Use BatchSpider for periodic batches with batch records and explicit task-state transitions.
6. Read only the reference file that matches the task:
7. Spider selection, CLI scaffolding, and project layout: references/spider-types-and-scaffolding.md
8. Code patterns for request flow, items, task updates, and render usage: references/code-patterns.md
9. Settings, pipelines, shell debugging, and upstream source anchors: references/settings-debugging-and-sources.md
10. Reuse feapder conventions instead of inventing abstractions:
11. Import feapder directly and subclass the correct base spider.
12. Emit work with yield feapder.Request(...).
13. Keep parsing in parse or explicit callback methods passed through callback=....
14. Prefer __custom_setting__ for spider-local overrides and setting.py for project-wide config.
15. Default to from feapder.utils.log import log and log.info(...) for runtime logs instead of print(...) unless the user explicitly asks to mirror upstream demo output.
16. Default to concise Chinese log messages and Chinese code comments when comments are needed.
17. If the user asks to create new feapder code, prefer the framework's generated structure and naming conventions over bespoke layouts.
18. If the user asks for a simple crawler demo and does not mention persistence, Item, UpdateItem, pipeline wiring, Redis, MySQL, task tables, batch scheduling, distributed workers, or multi-file output, default to the single-file AirSpider style shown in references/vendor/feapder-1.9.2/tests/air-spider/test_air_spider.py: one file, one spider class, minimal startup block, and only the code required to demonstrate the requested behavior.

Working Rules

• Keep output aligned with feapder 1.9.2, not newer community variants.
• Prefer minimal, working spider code over framework-agnostic architecture.
• When the request is underspecified, do not scaffold a full feapder project by default. Write the smallest single-file AirSpider demo only for pure crawling or parsing requests. If the request mentions Item, UpdateItem, 入库, ITEM_PIPELINES, Redis, MySQL, task tables, or batch/task state, choose the corresponding spider and project shape instead.
• When modifying an existing feapder project, preserve its current spider type, main.py dispatch style, item modules, and setting.py shape unless the user asks for a migration.
• When persistence is needed, decide explicitly between:
• Manual DB calls with MysqlDB or RedisDB
• Automatic batch persistence via Item or UpdateItem
• Custom ITEM_PIPELINES
• When render is required, set render=True on the request and then align downloader configuration with the project's setting.py or __custom_setting__.
• When handling BatchSpider or TaskSpider, treat master and worker entrypoints separately. Do not collapse them into a single ambiguous startup path.
• When writing new spider code, default to log.info(...), log.error(...), and similar logger methods for runtime output. Avoid print(...) unless the user explicitly asks for raw console prints or exact upstream reproduction.
• Write log messages in Chinese by default, for example log.info(f"第{request.page}页原始结果 | {response.json}").
• Write comments in Chinese by default and only keep comments that clarify non-obvious logic.

Chinese Trigger Hints

Treat these Chinese requests as strong signals to use this skill:

• 用 feapder 写一个爬虫
• 帮我改这个 AirSpider
• 把这个项目迁移到 Spider 或 BatchSpider
• 写 feapder 的 item / pipeline / setting.py
• 用 feapder shell 调试
• 分析 feapder 1.9.2 示例代码

Implementation Checklist

Before finishing a feapder task, verify the code against this checklist:

• The spider inherits the correct feapder base class.
• If the task is a pure crawling or parsing demo and does not require persistence or task orchestration, the new spider defaults to the single-file AirSpider pattern anchored by references/vendor/feapder-1.9.2/tests/air-spider/test_air_spider.py.
• Startup code passes the required constructor args such as redis_key, task_table, task_keys, or batch metadata when needed.
• Request callbacks, carried parameters, and middleware hooks use feapder's expected method signatures.
• Distributed spiders update task state where required instead of relying on implicit completion.
• Item, pipeline, and setting references match the project's existing module layout.
• Debugging guidance uses feapder-native tools such as feapder shell, to_DebugSpider, or to_DebugBatchSpider when appropriate.
• New example code uses log for runtime logs, with Chinese log text and Chinese comments unless the user requested a different style.

Source Anchors

Use the vendored feapder 1.9.2 snapshot under references/vendor/ as the source of truth. This keeps the skill portable when copied to another machine:

• Core overview: references/vendor/feapder-1.9.2/README.md
• Usage docs: references/vendor/feapder-1.9.2/docs/usage/AirSpider.md, references/vendor/feapder-1.9.2/docs/usage/Spider.md, references/vendor/feapder-1.9.2/docs/usage/TaskSpider.md, references/vendor/feapder-1.9.2/docs/usage/BatchSpider.md
• CLI and project generation: references/vendor/feapder-1.9.2/docs/command/cmdline.md
• Minimal runnable examples: references/vendor/feapder-1.9.2/tests/air-spider/test_air_spider.py, references/vendor/feapder-1.9.2/tests/spider/spiders/test_spider.py, references/vendor/feapder-1.9.2/tests/batch-spider/spiders/test_spider.py, references/vendor/feapder-1.9.2/tests/test-pipeline/pipeline.py