Hacker News AI· Yigtwx·· 15 days agoAI score18
An AI agent read a Shift_JIS page as 208,953 replacement characters
An AI agent read a Shift_JIS page as 208,953 replacement characters
AI summary
Link understanding decoded every fetched page with a bare UTF-8 TextDecoder, so pages declaring Shift_JIS, ISO-8859-1 or another charset reached processors and the model as replacement characters. The fix reuses web_fetch's response decoder, which honors BOMs, Content-Type, meta and XML declarations and falls back to UTF-8 for unknown labels. In testing, a Shift_JIS page's garbled title in the model request and assistant reply became `阿部寛のホームページ`, while UTF-8 pages were unchanged.
Source: Hacker News AI · github.com