Maison Merling
France
Reviews (4)
Knowledge Base still needs a lot of foundational work, especially to be used reliably with Inbox. Content hallucination. There's no control over AI hallucination, and the tool sometimes invents answers. Test-and-learn scenarios should feed Knowledge Base more broadly. Right now they seem limited to a single conversation, when they should let the AI learn to adapt to the e-commerce store in question, for wider question coverage and training. No control over the data source. We have no way to select or correct the data source used for a FAQ, which is surprising for a tool meant to accurately represent our brand. During testing, a URL was invented, along with other incorrect data. The FAQ creation system is a good idea, but it's missing key configuration options (data source, reliability, hallucination level...) to make the tool trustworthy.
The Shopify Inbox AI agent has real potential and is a genuine step up from the previous Inbox (which was limited to live chat management). But several issues still need fixing before it can be used reliably in production. 1. Response language The AI has replied in English even though the entire conversation was in French. 2. Inconsistency depending on how the question is phrased Two questions with the same intent, phrased differently, don't give the same answer: one omits relevant information/product, the other doesn't. This is a real reliability concern if the result depends so heavily on the customer's exact wording. 3. Hallucinated non-existent link On certain requests, the AI tries to share a link to a page that doesn't exist at all on our site. How can this type of URL hallucination be avoided? 4. Scope of learning Does the training/learning happen only at the conversation level, or does it also benefit future conversations? 5. Disabling the AI mid-conversation After disabling the AI agent while a conversation was in progress, it kept being handled by the AI despite the box being unchecked. Is this a bug? 6. Limited training options For training, with Knowledge Base you can suggest answers, but not suggest new questions in the scenarios. Is this planned? We also don't have control over selecting/correcting which data source is used, which is surprising. Overall, a great tool at first glance, but one that still needs a lot of foundational work. Looking forward to updates. In the meantime, to get reliable answers, I'm forced to use another paid tool instead of this native, free one.
Developer reply ·
Thank you for taking the time to share such a detailed and thoughtful review. We reviewed the issues you raised against the actual conversations on your store and have shared the findings with our engineering team who are actively digging into the underlying causes to fix them. To clarify two points: The training/learning will benefit future conversations and also allow our background agent to suggest improved Agent persona if that will be more helpful for representing your business to buyers accurately. Also, yes we are planning to surface more questions in Knowledge Base based on the conversations buyers have with your agent. We appreciate the thorough testing and candid feedback. This is exactly the kind of input that helps us make the product more reliable.
We really like the experience with this app Custumer service support is welcoming, friendly and fast. I recommend !
Awesome !! Had a problem with the app and they helped me fix evrything. Great support team at Judge.me !
