ReworkdAI

What is Reworkd?

Reworkd is an AI-powered web data extraction platform designed for enterprises and development teams. After the user specifies the targets to be extracted and defines the data schema, the agent analyzes the website’s structure, generates Playwright code, runs a browser, verifies the results, and saves the structured data.

The team first gained attention thanks to the open-source AgentGPT; later, it shifted its commercial focus to web scraping that can be maintained on a scalable basis. AgentGPT is still accessible, but it should no longer be presented as Reworkd’s main product.

Core functions

• Automated crawler generation: Code specific to a site is created based on natural language objectives and a Schema.
• Multimodal page understanding: Selecting elements by combining page structure, screenshots, and browser status.
• Self-healing script: Detects issues when the website changes or selectors become invalid, and regenerates the code.
• Dynamic web page handling: Supports clicking, expanding, pagination, infinite scrolling, and waiting for content to load.
• Structured output: Saves data according to field type and hierarchy.
• Scheduled execution: Repeatedly fetch a set of sources at a fixed frequency.
• Deduplication and updating: The primary key is used to determine whether a record is new, duplicate, or has changed.
• File download: Asynchronous access and saving of PDFs and other attachments.
• API and export: Reading results using keys, managing session data, and handling the connection process.
• Execution analysis: View successes, failures, number of outputs, and website changes.

AI code generation process

Reworkd does not have large models guess the content of a webpage directly; instead, it first has an agent understand the page and then generate executable code for scraping. This code runs in a hosted browser, and the extracted results are subjected to Schema validation.

• Load the target website and analyze its interactive elements.
• Search for candidate content based on the field schema.
• Generate code for accessing pages and extracting data.
• Run the code and observe the browser events and network requests.
• Verify that the output conforms to the field types and structure.
• When an error occurs, make the necessary modifications based on the logs and redeploy.

Self-healing gripper

After the layout, class names, or loading logic of a website are modified, traditional selectors may return null values without any warning. Reworkd monitors the results and operation status, detects failures, and attempts to regenerate the relevant code.

• The detection selector fails to find the element, or the output level drops suddenly.
• Locate the corresponding fields based on the new page structure.
• Generate modified scripts specific to the site.
• View the specific changes by examining the code differences.
• Retaining operation records facilitates rollback and manual inspection.
• Perform Schema validation on the repaired output once again.

Schema and data types

A Schema defines which fields should be included in each row of data, their data types, and the nested relationships between them; it also serves as a basis for code generation and result verification. A well-designed Schema allows for more stable results compared to using vague instructions.

• Use explicit field names to convey the actual business meaning.
• Set the correct type for price, date, URL, and boolean status.
• Use objects and arrays to represent product variants or detailed lists.
• Distinguish between required fields and optional fields that may be missing.
• Avoid including multiple incompatible values in the same field.
• Select a stable field as the primary key for deduplication.

Pagination and dynamic content

The official Harambe scraping SDK is used to handle pagination, page navigation, dynamic loading, HTML parsing, and data saving. The scripts it generates allow users to click on the next page, scroll through the pages, or go to the details page to fill in additional fields.

• Click the page number or \"Next\" until no new results are available.
• Handle infinite scrolling and lazy-loading lists.
• Open foldable panels, tabs, and pop-ups.
• Go from the list page to the details page to collect more data.
• Wait for a network request or for the specified element to appear.
• Cookies and local storage are used while logged in.

Scheduled tasks and templates

The scraping group can be run again at regular intervals to enable continuous monitoring of prices, inventory levels, job openings, or regulatory documents. For websites that use the same website construction system, the scraping logic can be reused, which reduces the costs associated with generating code and carrying out debugging tasks.

• Set the frequency of repeated execution according to business needs.
• Add websites with the same structure to the same template.
• View the successes and failures from various sources in one place.
• Update and reuse the script uniformly after changes to the website.
• Avoid generating code repeatedly for each identical website.
• Use weekly reports to track fluctuations in output volume and sources.

Deduplication and change tracking

Reworkd creates materialized result views based on the unique key or composite key selected by the user. New records are marked as created, new values for existing records are marked as updated, and completely identical content is not inserted again.

• SKU, UPC, and stable page ID are suitable as single-field primary keys.
• Brands, models, and colors can be combined to form composite keys.
• Price, inventory level, and update time are not suitable as unique keys.
• The same object should use consistent key rules across different sources.
• Before modifying the primary key, it is necessary to assess whether there will be duplicate records in the history.
• The result obtained through the API is a materialized view with duplicates removed.

Download of files and attachments

When the capture results include file addresses, Reworkd can access and download the attachments asynchronously, while also saving the file type, checksum, source address, and storage information. This capability is suitable for regulatory announcements, tender documents, and product manuals.

• Download PDFs, documents, and other attachments provided on web pages.
• Record the original source and file name.
• Use checksums to identify duplicate or altered files.
• Associate the download task with the corresponding data row.
• Perform a browser download for dynamically generated or click-required files.
• Continue processing the saved files via API.

APIs and developer capabilities

Organizations can create API keys to retrieve the output of scraping tasks and their status from external systems. The public documentation also provides interfaces related to cookies, local storage, review status, and file processing.

• Retrieve the deduplicated results for a specified crawling group.
• Check the latest status of manual review and comments.
• Read or set browser cookies.
• Read or set the browser’s local storage.
• Download data through batch export.
• Clients are generated using open interface specifications.
• Integrate the results into database, analysis, or model training processes.

Browser debugging and observability

The platform offers a unified browser trace that displays Reworkd events, Playwright events, console logs, network activities, and browser protocol events in the same view. It also allows for comparing code differences, showing what was added or removed before and after repairs.

• Page loading timeout and resource blocking occurred.
• Check for script errors in the console.
• Check network requests, status codes, and changes in responses.
• Compare the old and new crawling codes of the proxy.
• Determine whether the drop in output is caused by changes to the page or by operational issues.
• Use the real-time stream to view task status and output volume.

Which users are it suitable for

• E-commerce team: Monitors products, prices, inventory, and reviews.
• Market research team: Compiles information on companies, industries, and competition.
• Recruitment and talent platform: Collects information on available job positions and companies.
• Finance and Procurement teams: Monitor announcements, tenders, and regulatory documents.
• AI team: Develop training, fine-tuning, and RAG datasets.
• Data engineering team: Replaces a large number of fragile, site-specific scripts.
• Startups: Establishing pipelines quickly in the absence of full-time crawler engineers.

Typical use cases

• Synchronize the product prices and availability status of multiple stores on a daily basis.
• Extract names, industry, team, and contact information from the company directory.
• Download the new regulations and tender attachments available on the government website.
• Collect the list of positions and go to the details page to add additional requirements.
• Continuously monitor changes in the real estate, vehicle, or travel lists.
• Prepare millions of rows of public web data for the domain model.
• Monitor website addresses, service descriptions, and updates on operational status.

Product advantages

• It covers the entire process, from requirements and code generation to execution and bug fixing.
• Code-based extraction is easier to verify and reproduce than having the model generate data directly.
• It includes a built-in managed browser, proxy, CAPTCHA processing, and scheduling capabilities.
• It supports pagination, dynamic content, detail pages, and file downloads.
• The deduplication and update mechanism is suitable for long-term data monitoring.
• It provides APIs, open interface specifications, and an open-source Harambe SDK.
• Trace and code Diff help in identifying issues in production.

Usage restrictions

• Automatically generated code may still miss boundary pages or incorrect fields.
• Website anti-scraping measures, login requirements, geographical restrictions, and verification codes increase costs.
• After self-repair, it is necessary to verify whether the semantics still correspond to the original fields.
• Crawling public web pages may also be subject to terms, copyright, and database rights restrictions.
• High concurrency increases the costs associated with browsers, proxies, traffic, and verification codes.
• When the structure of websites differs greatly, the effectiveness of template reuse is limited.
• The official page retains outdated and inactive banners, and the product information is not consistently maintained.
• Sampling and verification against business rules are still required for key data.

Subscription plan

The public pricing page currently lists three tiers: Hobby, Pro, and Enterprise. It also shows an outdated banner indicating that certain older products are no longer available, but it is still possible to access the login page for the new platform, the status page, as well as the documentation and any subsequent updates. Before making a purchase, it is advisable to check with the authorities regarding the current service terms.

Infrastructure usage costs

In addition to the subscription fee, the platform also charges based on the actual amount of infrastructure usage. The standard prices displayed on the public page are as follows; corporate plans can be customized according to the terms of the contract.

The results extracted from the pricing page do not show the label for the first item of infrastructure costs in its entirety; therefore, it should not be added directly to the subscription fee to obtain a fixed monthly cost. Before making a payment, it is necessary to check the account for information regarding the free quota, minimum spending requirements, and the names of various usage amounts.

How to create the first crawler

1. Register Reworkd and create a new crawling group.
2. Enter the target website and a description of the data to be collected.
3. Define field names, types, nesting relationships, and optional fields.
4. Choose a stable and unique deduplication key.
5. Have the agent scan the page and generate the first version of the scraping code.
6. Watch the browser in action and check the pagination, clicking, and detail pages.
7. Check line by line to ensure that the output matches the original text on the page.
8. Use chatting to indicate missing fields or special interactions.
9. Verify multiple pages before saving and enabling scheduling.

How to launch a long-term data pipeline

1. Confirm the allowed methods and frequency of access to the target website.
2. Prepare samples for normal cases, empty results, abnormal layouts, and error pages.
3. Set retry options, timeout values, rate limits, and maximum concurrency.
4. Configure a stable primary key and test the addition and updating of records.
5. Use the API to write the results to a temporary area instead of directly overwriting the production data.
6. Set up alerts for field completeness rate, output volume, and failure rate.
7. Manually review the code differences and sample results after self-repair.
8. Record template, code, and Schema version.
9. Evaluate the costs of agents, computing, verification codes, and storage on a monthly basis.

Data quality recommendations

• Save the original page address along with each line of results.
• Add format and range validation for price and date.
• Monitor sudden increases or decreases in daily output.
• Create test samples for different page types separately.
• Regularly compare the crawl results with page screenshots.
• Re-verify the semantics of the fields after changes to the website.
• Do not equate successful operation with completely accurate data.

Legal and compliance considerations

Companies need to verify for themselves the terms of the target website, the robots rules, copyright issues, database rights, privacy regulations, and access permissions. Data that can be read by technical means does not necessarily allow for free copying, redistribution, or use in commercial training purposes.

• Give priority to using public APIs or obtaining permission from the data owner.
• Do not attempt to obtain unauthorized content by bypassing login processes, paywalls, and access controls.
• Restrict the collection of personal identity and sensitive information.
• Comply with the requirements regarding the frequency of visits to the target website.
• Retain the data source, collection time, and deletion process.
• Check the license scope separately for model training purposes.

GitHub and the open-source status

On its official GitHub repository, Reworkd makes projects such as AgentGPT, Harambe, Tarsier, and Bananalyzer available, but the complete proxy systems, hosting infrastructure, and self-repair services for commercial web scraping platforms are not open-source products.

The differences between AgentGPT and the current Reworkd

Basic information

Recommendation score

4.5 / 5. Reworkd is suitable for companies and data teams that need to maintain a large number of site crawlers; it provides a comprehensive pipeline for code generation, self-repair, deduplication, and monitoring. However, the costs associated with its use, legal aspects, and the consistency of information on official websites need to be carefully evaluated.

Frequently Asked Questions

What does Reworkd do mainly these days?

The key focus at present is to have AI generate and maintain web scraping code, in order to convert dynamic websites and attachments into structured data.

What is the relationship between Reworkd and AgentGPT?

AgentGPT was an early open-source general-purpose agent project developed by the Reworkd team; the current commercial products focus on web data extraction.

Do I need to write the crawler code myself?

It is usually not necessary; after the user defines the goals and schema, the agent generates the code. However, complex sites may still require guidance via chat and manual debugging.

Will the website be automatically fixed after its redesign?

The platform will detect the failure and attempt to regenerate the code, but the semantics of the fields and the accuracy of the data after the correction still need to be verified.

Does Reworkd offer a free plan?

Yes; Hobby costs $0 per month, and the public details specify 10 concurrent browsers, 30 days of data retention, and API access.

How much is the Pro package?

The public price is $99 per month; it includes 50 concurrent browsers and 90 days of data retention, as well as support for scheduled tasks and captcha processing.

Are the infrastructure costs charged separately?

Yes, services such as proxy traffic, browser processing, and anti-robot verification are charged based on usage, and it is necessary to check the free quota before settling the account balance.

Can I download the PDF attachment?

Yes, the platform supports asynchronous file downloads and retains the source, file type, and verification information.

Does Reworkd provide an API?

It provides functions for reading deduplicated output, managing cookies and local storage, and can be integrated through open interface specifications.

Is Reworkd open source?

Commercial platforms are not open source, but official projects such as AgentGPT, Harambe, Tarsier, and Bananalyzer are available publicly.