Process data sheets and technical product data with AI

What sounds like a classic job for AI-supported document processing quickly becomes a complex development project for industrial companies: countless product series that are sold in various versions, individual measured values, test standards and certifications - all in different documents and layouts.

Engineers, purchasing teams and dealers spend hours leafing through the PDFs. They are actually “only” interested in which sealing material is heat-resistant up to 200 degrees, or whether one of the 30 supplier offers actually meets the required certifications. Automated checking of data sheets no longer seems quite so trivial - and requires engineering approaches that go far beyond classic text recognition.

What is possible with structured product data

A technical data sheet (TDS), also known as a product or safety data sheet in some industries, contains all the product information that is essential for technical communication between companies: Specifications on material and processing properties as well as test standards support companies in their purchasing decisions and risk management - classic application scenarios for artificial intelligence (AI).

Automatically converting product data into a searchable form makes internal data management and updates easier and significantly reduces the need for personnel checks. Interactive product selectors and AI chatbots can also be set up to make it easier for customers to select products based on structured data. There were technical reasons why this has not been the industry standard for years - until now.

Overcoming the complexity of technical data

First the good news: yes, the problem has been solved. With modern AI pipelines based on multimodal language models, thousands of technical data sheets - regardless of layout, language or document type - can now be converted fully automatically into structured, searchable data records. What used to take weeks of manual data entry or schema creation now takes hours. The extraction accuracy reaches values that allow productive use without manual rechecking.

Why this is more difficult than you might think:

• Heterogeneous document structure - Similar documents from different manufacturers, product lines or decades do not follow a standardized layout. Section names, table structures and field sequences vary greatly.
• Product types do not follow a uniform pattern - Not every product has the same properties. A machine screw has a strength class, a wood screw has a tip type. An LED lamp has lumens and color temperature, a light bulb only watts and base. Rigid data models inevitably lose information.
• Measured values spread over several lines - Key, unit, measurement condition and value are rarely in the same line. Conventional parsers read them as separate entries and produce nonsensical results.
• Values are not simple numbers - Data sheets contain ranges, inequalities, approximations and qualitative information. If you put everything in a text field, you destroy the filterability.
• Synonyms and inconsistent vocabulary - The same property has dozens of names in different documents. Without normalization, data records are created that are not comparable.
• OCR quality as a hard limit - Special characters, units and scientific symbols are not recognized by conventional PDF parsers is often corrupted (e.g. Ω becomes Q). No language model can repair what text recognition destroys.
• Completeness beats accuracy - A missing measured value makes a product invisible in search queries. The pipeline must be exhaustive, not just precise.

Solutions for heterogeneous data sheets

The decisive design decisions in the processing of data sheets do not concern individual algorithms, but the interaction of schema design, extraction strategy and post-processing. Five principles prove to be effective:

• Design the scheme for the target, not the source - A data sheet follows the logic of the engineer, a product selector that of the customer. Mapping the document structure in the data model results in a deep hierarchy full of empty fields. Better: a flat schema in which each field answers a filter question or contributes to a measurement.
• Split extraction into subtasks - One extractor for classification and measured values at the same time is worse for both. Two independent extractors with a focused prompt - one for metadata, one for measured values - deliver better results. The merging takes place downstream.
• First extract, then classify - Forcing values directly into predefined categories during extraction is obvious, but leads to incorrect assignments. It is more reliable to initially take information from the document verbatim and only standardize it in a separate step.
• Make value types explicit -„< 5 MPa“, „200-250 °C“ and „conditionally resistant“ are all valid product specifications, but with completely different meanings. If you save them uniformly as text, you lose filterability. Instead, each measured value should have a type - number, range, limit value, approximation or qualitative specification - so that downstream systems can work with it in a meaningful way.
• Keep conditions as free text - Modeling test conditions as structured objects sounds elegant, but fails due to the variety of combinations. A free text string is more reliable and can be parsed downstream. Structure what you can verify - leave the rest as text.

Automatic quality improvement - without fine-tuning

The extraction quality can be increased iteratively without having to adjust the model weights. The principle: A test set is extracted, an independent review agent checks each extraction field by field against the original PDF and suggests specific schema changes. A second agent aggregates the findings, discards individual errors as noise and makes specific adjustments to field descriptions, rules and vocabularies.

The cycle is repeated until the quality reaches its target plateau. A few iterations are often enough for significant improvements - for problems that a human reviewer would only find after hours of reviewing hundreds of pages. The entire learning process takes place as human-readable text in the prompt: testable, comprehensible and revisable at any time.

The software solution at a glance

The automated processing of technical data is carried out as shown by the Konfuzio software from Helm & Nagel GmbH. The system combines multimodal language models with OCR-supported text recognition, to not only extract product data, but also validate it against existing specifications and standards - for example, whether an extracted test value matches the specified DIN/ISO standard. Where the automatic plausibility check reaches its limits, a graphical user interface enables manual correction by specialist personnel.

Do you have a specific set of documents in mind as you read this? Take now Contact and receive a non-binding initial assessment.

Conclusion

Technical data sheets are just one example. The same patterns recur wherever companies are sitting on hundreds or thousands of non-machine-readable PDFs - Acceptance test certificates, safety data sheets, laboratory reports, certificates of conformity. In all these cases, the business leverage lies not in the extraction itself, but in what becomes possible afterwards: Product selectors that actually work, comparisons across suppliers, risk assessments at the touch of a button and procurement decisions based on complete data instead of gut feeling.

Three insights can be generalized: Think of the schema from the end user's perspective, not from the document - what the customer filters determines the fields. Separate reading, normalization and validation consistently instead of forcing everything in one step. And automate not only extraction, but also quality assurance - the biggest improvements come from feedback loops that independently produce better results.

The following applies: every pipeline has documents that cannot be processed reliably. These cases belong in the accuracy balance, not under the carpet. And completeness counts at least as much as precision - because in a filter-controlled application, a missing field makes the product disappear.