Crawl More Pages option
complete
J
Jarod Ackerman
We're losing out on deals because Inbox Pro can only crawl so many pages. We've had two where there's an issue with Inbox not being able to crawl their 5000+ pages of products.
If this costs additional, it would be a nice add-on for customers who sell many products on their website.
Log In
Vendasta Team
complete
Thanks for suggestion, we'd recommend not using website scraping for adding live product inventory to AI since live inventory that was scraped would very quickly become out of date.
The better way to do it is to add a custom AI capability that does a live API lookup of the inventory, based on the user's query. This way, the AI can answer questions based on thousands of pages of inventory products or services.
Advanced users can create custom capabilities today, or you can work with Vendasta Services team to help you build this for a client! https://partners.vendasta.com/marketplace/products/MP-K48XKQV7DXLGQZ67ZH4MNZDM2FQCK8LM
R
Ryan Freeman
Vendasta Team that's remarkably disappointing. Not every client has a rapidly evolving product selection. And not every client has an API available for product lookup. Not to mention having the info available for recommendations.
Outside the scope of Jarod's request, it's really not hard for a client to have a website with more than 100 non-product pages. So we're back to the same issue of trying to train the AI on the client's content but hitting limits. Yes, there are hacks to get around this, but they are time consuming and can be fragile.
Instead of rejecting requests from your clients, it could be more productive to hear us telling you these are pain points with a product that we would like to sell more often if you can make it suitable.
Vendasta Team
Thanks for comments Ryan Freeman – we're not outright rejecting anything, just marked this as complete since the underlying request is now possible with our new custom capabilities creator!
Some additional context: we've been testing internally AI employees with larger knowledge base with more than 100 scraped pages, e.g. we have one with 1000 pages scraped, and at the moment the ability for AI to respond with correct and useful information when this much website content is scraped really degrades the more pages that are added. Unless the underlying web content is really immaculately prepared (most business websites are not) the vector search returns poor results.
In our research the best way to get the AI to answer based on really large data sets like product inventories is actually by using APIs to do a live lookup. A query is built by the AI, that includes the right filters, and then the AI returns really great results.
Very open to hearing more about your use case, and collaborating on the best solutions - please feel free to share more context here about what you're trying to accomplish.
R
Ryan Freeman
Vendasta Team thanks for the info. It's good to know when your AI starts to decrease in effectiveness.
In some cases, we have clients with 800+ products, mostly segmented by occasion, and most viable for 3+ years. Some new seasonal items might be added, but overall the product rotation is minimal. In these cases, the AI will be expected to field questions about products and suggest products that meet the user's query.
Other clients may publish a lot of material, either about their products/services or as a subject matter expert. The AI needs to be able to process the pages and posts in order to answer effectively for these clients - otherwise, the answers will range from unhelpful to absolutely wrong.
Am I correct in assuming that the current issue is more about information scope, and less about crawl capacity?