Follow-up work is broken out into TIKA-4771 (pluggable external parsers) and TIKA-4772 (document event streaming) rather than growing this PR.

@nddipiazza @tballison what do you think of this design instead? far less fields - ability to roll your own protobuf model in the future. Best of both worlds. Document structure is very markdown-friendly. I'll make the output of this be able to be the input of the grpc OpenNLP grpc server.

@nddipiazza great feedback..

Review: e2e test coverage for the new typed Document contract

Reviewed the design overall — the shift from mirroring Tika's metadata taxonomy in protobuf to a small, stable document.proto (208 lines) plus per-parser mapper code is a solid answer to the #2916 feedback. A few things worth addressing before merge:

Testing gap: e2e coverage does not exercise the new contract

The only change in tika-e2e-tests/tika-grpc is a 2-line fix in HandlerTypeTest.java:

- String htmlContent = htmlReply.getFieldsMap().get("X-TIKA:content");
+ String htmlContent = htmlReply.getDocument().getMarkdown();

(same for the TEXT handler case). This is a compile-fix to keep the pre-existing assertion working against the new API — it only proves document.markdown is non-empty over a real gRPC round-trip for two handler types.

It does not exercise, end-to-end, through the live server:

• DocumentMetadata typed fields (title/authors/dates/counts) — no e2e assertion anywhere
• extra tagged-tail typing (int/bool/date/string) — no e2e assertion
• blocks structured content tree (headings/tables/lists/code) — untouched by any e2e test
• embedded recursion for container formats (e.g. an Office doc with an embedded image) — no e2e coverage at all
• format_category routing hint — untouched

The other e2e tests (FileSystemFetcherTest, IgniteConfigStoreTest, ExternalTestBase) only ever inspect getFetchKey()/getStatus() on FetchAndParseReply — none of them look at getDocument(), so the bulk-corpus/streaming/ignite e2e paths give zero signal on the new typed contract.

The real proof of correctness for the mapping logic lives entirely in tika-grpc-mapper's unit tests (DocumentBuilderTest, MarkdownBlockTreeBuilderTest, one test class per transformer), which feed fixture-derived Metadata/markdown into DocumentBuilder in-process. That's good for the mapping logic itself, but it bypasses the actual gRPC wire serialization, the live server (TikaGrpcServerImpl), the pipes client, and fetcher plumbing entirely.

Ask: add (or extend HandlerTypeTest) at least one e2e case per format that fetches a real file through the live gRPC server and asserts on document.metadata (a couple of typed fields), document.extra (at least one tagged key), and one document.embedded case (e.g. an Office/PDF file with an embedded image) — not just that markdown is non-empty. Right now this new, larger surface area has no live-server coverage at all.

Other findings from code inspection

1. Likely-dead error path: DocumentBuilder.build()'s primary == null branch (returns FAILED with "No metadata returned from parse") looks unreachable in production — the only caller, TikaGrpcServerImpl, always passes a non-null Metadata (tikaMetadata = new Metadata() as fallback when emitData()/metadata list is empty). Worth double-checking this is intentional, since a real fetch failure with no emit data currently produces an "empty-but-success-shaped" Document rather than hitting this explicit error branch.
2. Duplication: 7 of 8 format transformers (Generic, Pdf, Html, Image, Office, Rtf, Epub) repeat an identical block mapping TITLE/DESCRIPTION/CREATOR/SUBJECT/LANGUAGE/CREATED/MODIFIED. Consider factoring this into a shared helper in TransformSupport called by each transformer, keeping only the format-specific additions per-transformer.
3. Minor: parseTimeMs in TikaGrpcServerImpl measures fetch+parse combined (timer starts before pipesClient.process()), though the field's doc/intent reads as parse-only time.

Nice work on the overall shape of the contract — the main blocker from my read is the e2e coverage gap above.

My bad; the e2e suite really was only asserting "markdown is non-empty". Added three live-server cases to HandlerTypeTest:

• PDF (testPDF.pdf): asserts typed document.metadata (title, page count), a boolean-typed document.extra key (pdf:encrypted), document.blocks, and format_category.
• HTML (sample.html): asserts typed title, FORMAT_WEB, and a string-typed extra key.
• Embedded (test_recursive_embedded.docx): asserts FORMAT_OFFICE, typed created/modified dates, and that document.embedded recurses with per-child metadata.

Two things fell out of writing these:

1. The extra-tail typing was actually broken over the live server - MetadataTagger relies on Property.get(key), but Tika's property registry is populated lazily by class init, so in the server JVM pdf:encrypted came back as a string, not a boolean. The unit tests never caught it because they happen to load the vocabulary classes first. Fixed by force-loading the tika-core metadata vocabularies in a static initializer in MetadataTagger, and the new e2e test now pins the behavior.
2. The e2e configs needed grpc.allowComponentManagement / allowPerRequestConfig opt-ins after TIKA-4764 landed on main, otherwise every test dies with PERMISSION_DENIED - updated the three config JSONs.

Note the CI e2e job doesn't currently execute these (the -pl tika-e2e-tests -am verify invocation only builds the aggregator POM and no plugins are provisioned) - that's a pre-existing infra gap, but the suite passes locally against a live server.

Thanks for the review @nddipiazza, all four points addressed:

• e2e tests now exercise the typed contract over a live server (PDF/HTML/embedded DOCX), which flushed out and fixed a real bug: lazy Property registry init made typed extra keys degrade to strings in the server JVM.
• The primary == null branch is now reachable (server passes null on empty output) and correctly treats EMPTY_OUTPUT as success.
• Common field mapping deduped into TransformSupport.mapCommonFields.
• Renamed parse_time_ms to fetch_parse_time_ms to match what's actually measured.

Also merged latest main (the lombok removal and split-packages changes). Full build plus mapper, grpc, and e2e suites pass locally.

@nddipiazza @tballison -

I figured I'd explain why and what I'm trying to do. I view tika is a primary input plane and think the gRPC server is where I'd want it to integrate.

So a Tika Document shape first, including any revisions you think it needs before it becomes a contract. I would use this as my input.

The OpenNLP gRPC server is deliberately following OpenNLP 3.0 as it develops. As features are merged to OpenNLP main, I add the corresponding capabilities to the server. My goal is for the gRPC server to be fully compliant with the OpenNLP 3.0 API when I release it and have the ability to extend it (you can see the OpenVINO and TEI extensions already and they're working prototypes).

If Tika adopts this Document contract, I will make it a first-class input to that server. The remaining integration details include flattening the block tree, preserving block-to-text offsets, mapping annotations back to source blocks, and deciding which operations should stream. Those can be worked out after the contract is settled. I can also contribute to the fetching, parsing, and performance work here - which I'd love to do.

I want to avoid defining and implementing a second document input format before we settle this boundary. If Tika later chooses a different shape, OpenNLP would have an unused input contract and both projects would carry unnecessary compatibility surface.

So I'm curious: Is this Document direction an acceptable shared boundary for Tika output and downstream consumers such as OpenNLP?

If not, which parts need to change before it can serve that role?

I'm in no rush to get the answer - but I've never had an opportunity explicit about the vision and goal. By agreeing to a model, it can allow both projects to launch with an amazing open source story to tell.

The current OpenNLP gRPC work is here:

https://github.com/apache/opennlp-sandbox/tree/OPENNLP-1833-grpc-expansion

Once we agree on a shape, I can make the OpenNLP server conform to it as the rest of the 3.0 features land.

This is not going to happen right away - we're still going to tease more releases so I'd expect both servers to go through many iterations. What I'm trying to do on our side is to have a generic enough output shape so we can survive a lot of new features for point releases. With Tika - I want to help integrate and transform a lot of parsing surfaces and connectors (as you saw with the markdown parser).

But it all starts with a document shape - then there's no rush to add on more features.

Thoughts?

@nddipiazza up for another look? If you land this shape, I can demo how this would work w/opennlp 3's grpc server + make an enhancement to tika to integrate it directly.

Let me know what you think -

Another suggestion is to make this a separate build under the tika umbrella since you'll need to keep up with CVEs and stuff for creating a docker image. I'd help regardless. Up to you..

Follow-up after digging further into the tika-grpc/tika-grpc-api contract specifically (cc @tballison since this bears directly on your #2916 feedback).

Two scope concerns in the current diff

1. ParseBytesRequest/ParseBytes RPC is new scope, not described anywhere in this PR.

tika.proto adds a whole new RPC (rpc ParseBytes(ParseBytesRequest) returns (FetchAndParseReply)) for parsing inline bytes without registering a fetcher. It isn't mentioned in the PR description, the "what's in / deliberately not in this PR" table, or the client migration table — it has nothing to do with the typed Document contract.

The implementation in TikaGrpcServerImpl is also a real workaround, not a thin RPC:

• On every server start it creates a temp directory (Files.createTempDirectory("tika-grpc-parse-bytes")) and rewrites the loaded Tika config to inject a synthetic file-system-fetcher rooted there (augmentWithParseBytesFetcher), so the forked pipes worker can "fetch" bytes the caller pushed inline.
• Neither shutdown() nor postShutdown() clean up parseBytesDir, nor the temp config file written by augmentWithParseBytesFetcher (Files.createTempFile("tika-grpc-config", ".json")). Every server start leaks a temp dir + temp file that's never removed.
• It changes config-loading behavior for every request (the augmented config becomes the one config path the server ever loads) for the sake of one RPC.

Given the whole point of this PR (and #2916) is keeping tika-grpc small and reviewable, I'd pull ParseBytes into its own PR/JIRA with its own review.

2. FormatCategory re-bakes per-format buckets into the wire contract.

document.proto's FormatCategory enum (PDF/OFFICE/IMAGE/HTML/RTF/EPUB/WARC/GENERIC) plus its detector (FormatCategoryDetector, mime/extension string matching) reintroduces the pattern called out in #2916 point 3 — re-deriving something Tika already told us (content_type already carries the detected MIME type) via a hand-maintained second classification. Every new "coarse category" is a public proto enum change — the same wire churn this PR's premise is meant to avoid. RTF/EPUB/WARC already getting dedicated enum values shows the pull. The docs pitch "adding a format = adding a DocumentTransformer, wire contract doesn't move" isn't quite true here.

Aside from those two, the actual mapper transformers (PdfDocumentTransformer, OfficeDocumentTransformer, EpubDocumentTransformer, WarcDocumentTransformer, etc.) are appropriately thin now — 40-70 lines each, no format-structure-extraction logic living in tika-grpc-mapper — and the module's main-scope deps are clean (tika-core + tika-grpc-api + commonmark + protobuf-java; every parser module is test-scoped only). That part of the #2916 "wrong altitude" concern looks resolved.

Proposal: stage this as a sequence of small PRs

Given @tballison's core objection was scale/maintenance risk of a big bang change, I think this whole effort would merge faster broken into independently reviewable, independently testable PRs — mostly a split of what already exists here, not a rewrite:

1. Minimal typed contract. document.proto trimmed to: Document envelope (id, content_type, parsed_at, status, origin), DocumentMetadata with DC fields only (title/authors/description/keywords/languages/publishers/identifiers/created/modified/rights), the MetadataField/MetadataValue tagged tail, ParseStatus. No blocks/markdown, no FormatCategory, no embedded. Pair with GenericDocumentTransformer only, wired into tika-grpc (FetchAndParseReply.document) with a real e2e test through the live server. This is close to the ~80-line proto Tim said he'd be okay with, and directly answers the top TIKA-4766: Typed parse response grpc #2916 concern on its own.
2. Per-format typed transformers, one PR (or 2-3 grouped): PdfDocumentTransformer, OfficeDocumentTransformer, HtmlDocumentTransformer, ImageDocumentTransformer. Additive only, no proto change — review is just "is this mapping correct."
3. Content tree (blocks/markdown) as its own PR — a genuinely separate feature (commonmark parsing, tables/lists/inline runs), with its own live-server e2e round-trip test.
4. embedded recursion as its own PR — distinct complexity (depth limits, cycles), with an Office/PDF-with-embedded-image e2e test.
5. Niche formats (Epub, Warc, Rtf, Creative Commons) last, only once the mechanism is proven on common formats — each small and additive.

Would drop or defer indefinitely:

• FormatCategory — let clients bucket on content_type instead, unless a client has a concrete need.
• ParseBytes — separate JIRA/PR entirely, unrelated to this contract.
• External pluggable parsers, .md parser — already correctly deferred in the description.

Each stage above is independently testable (unit + e2e) and independently mergeable, which should make review a lot less daunting than reviewing ~3.9k lines at once.

This sounds great and goes exactly in the direction I'm hoping. Multiple tickets = real collaboration and proper pacing

I'll close this and post the first suggested change but keep the overall thread alive here unless you can do something in jira after a couple tickets land.

I like the feedback's instinct because shaping the API is by far the most important part.

Honestly I understand this is a big feature and I appreciate the time you're taking to look into it. seriously.

Closing in favor of #2961