---
title: Web Scrape
description: Fetch a public web page and extract Markdown, text, structured data, and links with a fixed, always-present output shape.
source: https://www.edgaze.ai/docs/builder/nodes/web-scrape
section: builder
---
# Web Scrape

> Fetch a public web page and extract Markdown, text, structured data, and links with a fixed, always-present output shape.

**Audience: Creators building and publishing workflows.**

## Overview

Web Scrape fetches a public URL from Edgaze's servers and returns a **frozen output object**. Every field listed below is always present. Missing data is `null`, `""`, or `[]`, never a missing key, so downstream Template and LLM Chat nodes behave the same for every buyer run.

Identical frozen config plus an identical live page yields byte-identical output. There is no response cache, no User-Agent rotation, and no implicit Wayback fallback.

## Ports

- **Input: URL:** Optional upstream URL string, or an object with a `url` field. Falls back to the inspector URL.
- **Output: Success:** The frozen scrape result when the page was fetched and extracted successfully.
- **Output: Error:** The same frozen shape when the page is blocked, missing, or empty. Network and policy failures that Temporal should retry still throw (rate limit, timeout, network error).

## Inspector

- **URL:** Target when nothing is connected, or as fallback.
- **Primary content:** Which field is mirrored into `content` (`markdown` default, `text`, `structured`, `html`, `links`, or `custom`).
- **Max pages:** Hard cap. `1` is a single page. Higher values enable crawl / pagination.
- **Respect robots.txt:** On by default. The setting is frozen into the published graph.
- **Blocked-page fallback:** `none` (default), `amp`, or `archive`. Archive sets `_meta.source` to `archive` with the snapshot timestamp, never silently.
- **Allowed / Denied hosts:** Same SSRF policy style as HTTP Request.
- **Crawl & pagination / selectors / debug fields:** In the Extraction panel. HTML and app-state payloads are off by default because they are large.

## Output schema

Top-level fields (always present):

- `url`, `finalUrl`, `status`, `ok`
- `title`, `description`, `author`, `publishedAt`, `siteName`, `lang`, `canonicalUrl`, `image`
- `content`, `markdown`, `text`, `html`, `structured`, `appState`
- `links`, `images`, `fields`
- `_meta`, `pages`

`_meta` fields (always present):

- `strategy`, `source`, `extractorUsed`, `hadStructuredData`
- `blocked`, `blockReason`, `robotsAllowed`
- `attempts`, `bytes`, `truncated`, `durationMs`
- `pagesFetched`, `pagesFailed`, `archiveSnapshot`

`pages` is always an array. A single-URL scrape has length 1. The first page is also mirrored onto the root for convenient `{{markdown}}` binding.

### Metadata precedence

When sources disagree, the first non-empty value wins in this fixed order: JSON-LD → OpenGraph → Twitter card → `<meta>` → `<title>`.

### Block reasons

Stable strings you can branch on: `cloudflare_challenge`, `captcha_required`, `login_required`, `rate_limited`, `js_required`, `not_found`, `forbidden`, `robots_disallowed`, `adapter_failed`, `timeout`, `too_large`, `invalid_url`, `network_error`, `empty_content`.

### Size caps

Markdown and text are truncated at 200 KB (UTF-8, code-point safe) with `_meta.truncated = true`. HTML (when enabled) caps at 500 KB. Response body hard cap is 5 MB.

## Site adapters

Some hosts use free public endpoints instead of HTML scraping. If an adapter matches the URL and then fails, the node **errors with a named reason**. It never falls through to generic HTML, which would look like a successful empty scrape.

Contract-stable: Wikipedia REST, Hacker News Algolia, GitHub REST, Stack Exchange, YouTube oEmbed (use [YouTube Transcript](/docs/builder/nodes/youtube-transcript) for captions).

Best-effort (undocumented upstream): X/Twitter single posts via syndication, Reddit `.json` permalinks.

LinkedIn: OpenGraph only when available; otherwise `blockReason = login_required`.

## Limits

Web Scrape cannot reach:

- LinkedIn profiles, feeds, or messaging
- X/Twitter timelines, search, or profiles (single `/status/<id>` posts only)
- Instagram, or any page behind login or an interactive captcha
- JavaScript-only apps that ship no embedded state and no readable HTML

There is no headless browser in v1. Most modern sites still work via JSON-LD, OpenGraph, and embedded app state (`__NEXT_DATA__`, Nuxt, Apollo, and similar).

## Tips

- Prefer `markdown` as Primary content for LLM Chat prompts.
- Branch the Error port on `_meta.blockReason` when a target site may block datacenter IPs.
- Keep Max pages small. Crawl length is deterministic (document order up to the cap), but each page costs time and infra.
- Leave Include HTML / Include app state off unless you need them for debugging.

## Related

- [HTTP Request](/docs/builder/nodes/http-request)
- [YouTube Transcript](/docs/builder/nodes/youtube-transcript)
- [Workflow Studio](/docs/builder/workflow-studio)