[Feature request] Add API tool to agents and use vector store for tools retrieval

[kairoswealth]

Currently retrieval of data is done either by search, webscraping, or vector stores.
The ability to use API by describing their specification in a tool description would be a great addition to extend the search capabilities of GPT Researcher.

One challenge with such approach though is that some APIs can have many endpoints, making it challenging by having to declare many tools for the agent.
One way to overcome this is to provide API endpoint description, path and specification for each endpoint, and index these in a vector database with the description as a embedding, and the path and specification as metadata.
In that case, the agent would first make a similarity search for its current task over the tools vector db, retrieve the tool to use, and generate the payload to make the api call based on the specification retrieved.
We could either index other methods (websearch), or having a first step as a router that would determine if this is within the scope of a web search or attempt to find a tool to accomplish the task.

In any case, as a first step, getting an API tool would be amazing

[ElishaKay]

Welcome @kairoswealth

Have a look here at the Custom Retrievers section:

https://docs.gptr.dev/docs/gpt-researcher/search-engines/retrievers

If you need something more complex, perhaps editing this file will solve your use case:

https://github.com/assafelovic/gpt-researcher/blob/master/gpt_researcher/retrievers/custom/custom.py

[kairoswealth]

Thanks. That could work, but are retrievers passed as tools to the Agents?
Right now my understanding is that the agents are only passed a retriever, or in the hybrid search case, it is passed 2 retrievers, ie one search engine and one local files retriever.

I really think GPT researcher would benefit a lot if it would give the ability to set a list of tools (standard and custom retrievers) that the agent could choose from.