Interface representing the parameters for initializing a SitemapLoader. SitemapLoaderParams

Hierarchy

Implements

Constructors

Properties

allowUrlPatterns: undefined | string[]
caller: AsyncCaller
chunkSize: number

The size to chunk the sitemap URLs into for scraping.

Default

{300}
timeout: number

The timeout in milliseconds for the fetch request. Defaults to 10s.

webPath: string
selector?: SelectorType

The selector to use to extract the text from the document. Defaults to "body".

textDecoder?: TextDecoder

The text decoder to use to decode the response. Defaults to UTF-8.

Methods

  • Extracts the text content from the loaded document using the selector and creates a Document instance with the extracted text and metadata. It returns an array of Document instances.

    Returns Promise<Document[]>

    A Promise that resolves to an array of Document instances.

  • A static method that dynamically imports the Cheerio library and returns the load function. If the import fails, it throws an error.

    Returns Promise<{
        load: ((content, options?, isDocument?) => CheerioAPI);
    }>

    A Promise that resolves to an object containing the load function from the Cheerio library.

  • Fetches web documents from the given array of URLs and loads them using Cheerio. It returns an array of CheerioAPI instances.

    Parameters

    • urls: string[]

      An array of URLs to fetch and load.

    • caller: AsyncCaller
    • timeout: undefined | number
    • Optional textDecoder: TextDecoder
    • Optional options: CheerioOptions

    Returns Promise<CheerioAPI[]>

    A Promise that resolves to an array of CheerioAPI instances.

Generated using TypeDoc