How to Connect Firecrawl as Data Search
Integration with Firecrawl is implemented as a separate Search block, which can be placed between the Data Source and Data Destination. This allows you to retrieve data fields from the Data Source, query Firecrawl, for example, for the "URL Parsing" action or other specified values, and pass the resulting data to the Data Intake fields. This way, you can automatically retrieve data from Firecrawl through Search and transmit it to the services and systems you use.
The function allows you to search for information on a specific request using the "URL Parsing" action in Firecrawl and refresh the data.
Navigation:
Connecting Google Sheets as a Data Source:
1. What data can you get from Google Sheets?
2. How to connect a Google Sheets account to ApiX-Drive?
3. Selecting the table and sheet from which rows will be exported.
4. An example of data that will be transferred from Google Sheets.
Connecting Firecrawl as a Data Search:
1. What data can be obtained from Firecrawl?
2. How do I connect my Firecrawl account to ApiX-Drive?
3. How do I configure Firecrawl data search in a selected action?
4. An example of the data that will be transmitted from Firecrawl.
Setting up row updates in Google Sheets:
1. What will the Google Sheets integration do?
2. How to connect a Google Sheets account to ApiX-Drive?
3. How do I configure the selected action to transfer data to Google Sheets?
4. Example of data that will be sent to your Google Sheets.
5. Auto-update and communication response interval.
Setting Data SOURCE: Google Sheets
Let's take a look at how the Search function works, using the example of updating a Row in Google Sheets.
First, you need to create a new connection.
Select a system as the Data Source. In this case, you need to select Google Sheets.
Next, you need to specify the action "Get ROWS (new)".
The next step is to select the Google Sheets account from which the data will be uploaded.
If there are no logins connected to the system, click "Connect account".
Select which account you want to connect to ApiX-Drive and grant all permissions to work with this account.
Once the connected account appears in the "active accounts" list, select it for further work.
Attention! If your account is listed as "inactive accounts," please verify your access to this login.
Select the Google Sheets table and sheet where the data you need is located.
At this step, you can add a data filter to load not all rows, but only those for which you need to update the data, for example:
1. Select the column from the table that contains the data you are looking for.
2. Select the "Does not contain" action.
3. Write the value by which filtering will occur.
With this filter, you won't unload rows that already contain this value. Why waste extra effort overwriting this value?
You can now see sample data for one of the rows in your Google Sheets spreadsheet.
If you want to update the test data, click "Load test data from Google Sheets".
If you want to change the settings, click "Edit" and you will go back one step.
Now let's add the additional Data Search block. To do this, click the "+" button and select "Add data SEARCH" from the list.
Select "Firecrawl" as the system in which the search will be performed.
Next, select an action, for example, "URL Parsing".
The next step is to select a Firecrawl account.
If there are no logins connected to the ApiX-Drive system, click "Connect account".
Enter the API Key to connect to your Firecrawl account.
Go to your Firecrawl account, copy your API key, and paste it into the appropriate field in your ApiX-Drive dashboard.
Once the connected account appears in the "active accounts" list, select it for further work.
Then fill in the required fields such as:
- URL Scrap;
- Formats;
and additional fields if necessary.
In the "Prompt" field, you must specify a query for which the system will return the results in the next section, Test, in the Answer line. For example, "Collect all links on the site, including subdomain links."
Now you see test data for one of the queries. You can transfer this data to your receiving table.
If the test data does not appear automatically, click "Search in Firecrawl".
If you're not happy with something, click "Edit," go back a step, and change the search field settings.
This completes the Data Search setup!
Now we can begin setting up Google Sheets as a Data Destination system.
To do this, click "Add data DESTINATION".
Setup Data DESTINATION: Google Sheets
Select the system as Data Destination. In this case, you need to select Google Sheets.
Next, you need to specify the action "Update ROW".
The next step is to select the Google Sheets account to which the URL from Firecrawl will be transferred. If this is the same account, select it.
If you need to connect another login to the system, click "Connect account" and repeat the same steps, described when connecting Google Sheets as a Data Source.
Now you need to select the File (Table) and Sheet in which the data will be updated for the "URL Parsing" action in Firecrawl.
In the "Search Column" field, select the column by which to search for data. This means where to look for data in the table.
Next, in the "What to search for" field, you need to select a variable from the drop-down list or enter the data manually. This is the value the system will search for to update the row you need. In our case, we'll select the "S" column, which contains the URL being parsed. The system will update the data in the desired row only if the URL matches.
You also need to specify the Search Type in case multiple lines with the same URL are found:
"First found row" - data search and updating will occur in the first found row that satisfies the search conditions.
"Last found row" - the search and update of data will occur in the last found row that satisfies the search conditions.
"All found rows" - the search and data update will occur for all found rows that meet the search conditions.
Also specify the parameter as Overwrite data:
- Overwrite old data - new data will be written over the old data.
- Append to old data - new data from the Data Source will be appended to the old data.
Now you need to assign the Firecrawl response variable, which we retrieve from the Data Search block, to the empty column. This column will subsequently be updated with data based on parsing of the specified URL.
After setting up, click "Continue".
Thus, the Data Search block takes the Data Source field in which you have the URL specified, requests data from Firecrawl for this URL, in this case, this is information on our request in the Answer line, and passes this response to the Data Destination field, for example, in column "G".
This completes the Data Destination system setup!
Now you can start choosing the update interval and enabling auto-update.
To do this, click "Enable update".
On the main screen, click on the gear icon to select the required update interval or set up scheduled launch. To start the connection by time, select scheduled start and specify the desired time for the connection update to be triggered, or add several options at once when you need the connection to be triggered.
Attention! In order for the scheduled run to work at the specified time, the interval between the current time and the specified time must be more than 5 minutes. For example, you select the time 12:10 and the current time is 12:08 - in this case, the automatic update of the connection will occur at 12:10 the next day. If you select the time 12:20 and the current time is 12:13 - the auto-update of the connection will work today and then every day at 12:20.
To make the current connection transmit data only after another connection, check the box "Update connection only after start other connection" and specify the connection after which the current connection will be started.
To enable auto-update, switch the slider to the position as shown in the picture.
To force the connection, click on the arrow icon.
This completes the Firecrawl URL Parsing Action Data Search setup! See how easy it is!?
Now don't worry, ApiX-Drive will do everything on its own!