feat(rag): skip the files openRAG declares it cannot index - #4937
paultranvan merged 2 commits into
Conversation
Sorry, something went wrong.
resolveContent decides that name and opens the content in one go, so nothing could ask for the name alone. Splitting it out keeps the two renaming rules in a single place, ready for a caller that needs the name before reading anything.
openRAG answers 415 on an extension its deployment does not accept and stores nothing, so every run sent the file again. One Drive walked on 2026-09-22 made about 147k refused POSTs for 25k distinct files, mostly node_modules trees (87k .js, 36k .map, 6k .json, 3k LICENSE, 2k .xlsx), each with a GET and a full upload for nothing. Read GET /indexer/supported/types once per job and skip those files before any request. The route is optional: on any failure the job warns and sends every file as before. Extension only, no mimetype sent: openRAG picks its parser from the extension and falls back to plain text, so a file accepted on its mimetype would be indexed as noise.
| if err := SetIndexStatus(ctx.inst, f.ID, StatusNotSupported, f.Rev); err != nil { | ||
| return err | ||
| } | ||
| return errUnsupportedFormat |
There was a problem hiding this comment.
when we are moving file from one workspace to another, and in the second one it's nop supported, it will be still indexed in the firs one
Sorry, something went wrong.
There was a problem hiding this comment.
That's quite a good catch. I think this situation is quite unlikely in practice, as all the workspace will have the same supported files.
We could however have this situation: file1.txt is renamed file1.xyz (not supported), and then moved to another workspace: we'll still have file1.txt in first workspace.
We can fix it, but I think it's not worth the effort for now: it's not a common situation, and I prefer we put effort on a future rabbitmq indexer instead, outside of cozy-stack. I'll make an issue about that so we don't forget
Sorry, something went wrong.
Summary