Web Scrape

Overview#

Web Scrape fetches a public URL from Edgaze's servers and returns a frozen output object. Every field listed below is always present. Missing data is null, "", or [], never a missing key, so downstream Template and LLM Chat nodes behave the same for every buyer run.

Identical frozen config plus an identical live page yields byte-identical output. There is no response cache, no User-Agent rotation, and no implicit Wayback fallback.

Ports#

• Input: URL: Optional upstream URL string, or an object with a url field. It is hidden when URL source is set to ask at run time.
• Output: Page: The frozen scrape result. A blocked, missing, invalid, or empty page fails the node with a clear message instead of emitting empty success data.

Inspector#

• URL source: Set a URL in the builder or ask the user for one when the workflow runs.
• Page URL / Question / Description: The fixed URL, or the run-form copy shown in entry-block mode.
• Primary content: Which field is mirrored into content (markdown default, text, structured, html, links, or custom).
• Max pages: Hard cap. 1 is a single page. Higher values enable crawl / pagination.
• Respect robots.txt: On by default. The setting is frozen into the published graph.
• Blocked-page fallback: none (default), amp, or archive. Archive sets _meta.source to archive with the snapshot timestamp, never silently.
• Allowed / Denied hosts: Same SSRF policy style as HTTP Request.
• Crawl, pagination, selectors, limits, and debug fields: In Advanced. HTML and app-state payloads are off by default because they are large.

Output schema#

Top-level fields (always present):

• url, finalUrl, status, ok
• title, description, author, publishedAt, siteName, lang, canonicalUrl, image
• content, markdown, text, html, structured, appState
• links, images, fields
• _meta, pages

_meta fields (always present):

• strategy, source, extractorUsed, hadStructuredData
• blocked, blockReason, robotsAllowed
• attempts, bytes, truncated, durationMs
• pagesFetched, pagesFailed, archiveSnapshot

pages is always an array. A single-URL scrape has length 1. The first page is also mirrored onto the root for convenient {{markdown}} binding.

Metadata precedence#

When sources disagree, the first non-empty value wins in this fixed order: JSON-LD → OpenGraph → Twitter card → <meta> → <title>.

Block reasons#

Stable strings you can branch on: cloudflare_challenge, captcha_required, login_required, rate_limited, js_required, not_found, forbidden, robots_disallowed, adapter_failed, timeout, too_large, invalid_url, network_error, empty_content.

Size caps#

Markdown and text are truncated at 200 KB (UTF-8, code-point safe) with _meta.truncated = true. HTML (when enabled) caps at 500 KB. Response body hard cap is 5 MB.

Site adapters#

Some hosts use free public endpoints instead of HTML scraping. If an adapter matches the URL and then fails, the node errors with a named reason. It never falls through to generic HTML, which would look like a successful empty scrape.

Contract-stable: Wikipedia REST, Hacker News Algolia, GitHub REST, Stack Exchange, YouTube oEmbed (use YouTube Transcript for captions).

Best-effort (undocumented upstream): X/Twitter single posts via syndication, Reddit .json permalinks.

LinkedIn: OpenGraph only when available; otherwise blockReason = login_required.

Limits#

Web Scrape cannot reach:

• LinkedIn profiles, feeds, or messaging
• X/Twitter timelines, search, or profiles (single /status/<id> posts only)
• Instagram, or any page behind login or an interactive captcha
• JavaScript-only apps that ship no embedded state and no readable HTML

There is no headless browser in v1. Most modern sites still work via JSON-LD, OpenGraph, and embedded app state (__NEXT_DATA__, Nuxt, Apollo, and similar).

Tips#

• Prefer markdown as Primary content for LLM Chat prompts.
• Handle scrape failures at the workflow run level; failed pages do not continue with empty content.
• Keep Max pages small. Crawl length is deterministic (document order up to the cap), but each page costs time and infra.
• Leave Include HTML / Include app state off unless you need them for debugging.

Related#

• HTTP Request
• YouTube Transcript
• Workflow Studio