Vardchives

Crawl4AI ​

Setup and CLI usage for crawl4ai. Base steps come from Python Project Setup.

Create venv ​

bash
python3 -m venv .venv
source .venv/bin/activate

Install crawl4ai ​

bash
pip install crawl4ai

Run Setup ​

bash
crawl4ai-setup

Check Setup ​

bash
crawl4ai-doctor

CLI ​

bash
crwl "https://www.autotrader.com/cars-for-sale/all-cars/toyota/4runner/dallas-tx?searchRadius=0&sortBy=mileageDESC&zip=75236" -B browser.yml -C crawler.yml -e extract_css.yml -s css_schema.json -o json

browser.yml ​

yaml
# browser.yml
browser_type: "chromium"
extra_args: ["--disable-extensions"]
headless: false
ignore_https_errors: true
java_script_enabled: true
light_mode: false
text_mode: false
verbose: true
viewport_height: 1080
viewport_width: 1920
user_agent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"

crawler.yml ​

yaml
# crawler.yml
cache_mode: "bypass"
delay_before_return_html: 0.5
magic: true
page_timeout: 30000
process_iframes: false
remove_overlay_elements: true
scan_full_page: true
scroll_delay: 0.3
verbose: true
wait_until: "domcontentloaded"

extract_css.yml ​

yaml
# extract_css.yml
type: "json-css"
params:
  verbose: true

Notes ​

-s css_schema.json is a required companion to the json-css extraction strategy: it defines the selectors to pull out, and has to exist alongside the yml configs.

user_agent sets a fixed User-Agent string. user_agent_mode is a separate field whose only value is "random" — putting a UA string in it silently does nothing.