Encoding issue on default backend DoclingParseV2DocumentBackend for PDF

[Seigneurhol]

Bug

When I use the default parser (DoclingParseV2DocumentBackend) for parsing a PDF I have encoding issue : "ao\u00fbt, facturation \u00e0". But it works fine with PyPdfiumDocumentBackend.

Steps to reproduce

Use the default DocumentConverter without specifying a backend.

    pipeline_options.generate_picture_images = True
    pipeline_options.do_ocr = True
    pipeline_options.ocr_options = EasyOcrOptions()
    pipeline_options.do_table_structure = True
    pipeline_options.table_structure_options.do_cell_matching = True
    pipeline_options.ocr_options.lang = ["fr", "en"]
    pipeline_options.accelerator_options = AcceleratorOptions(
        num_threads=4, device=AcceleratorDevice.AUTO
    )
    
    converter = DocumentConverter(
        format_options={
            InputFormat.PDF: PdfFormatOption(
                pipeline_options=pipeline_options,
            )
        }
    )

Then read a PDF and convert it to markdown.

    doc_stream = io.BytesIO(content)
    input_name = filename if filename else "document.pdf"
    
    # Convert document
    result = converter.convert(
        DocumentStream(name=input_name, stream=doc_stream)
    ) 
    result.document.export_to_markdown()

Docling version

Docling version: 2.14.0

Python version

Python 3.12.3

[trinanjan12]

@Seigneurhol I tested this code, this seems working for me with docling 2.14 and python 3.11

from docling.datamodel.pipeline_options import PdfPipelineOptions
from docling.datamodel.pipeline_options import EasyOcrOptions
from docling.datamodel.pipeline_options import AcceleratorDevice, AcceleratorOptions
from docling.document_converter import DocumentConverter, PdfFormatOption
from docling.datamodel.base_models import InputFormat


pipeline_options = PdfPipelineOptions()
pipeline_options.generate_picture_images = True
pipeline_options.do_ocr = True
pipeline_options.ocr_options = EasyOcrOptions()
pipeline_options.do_table_structure = True
# pipeline_options.table_structure_options.do_cell_matching = True
pipeline_options.ocr_options.lang = ["fr", "en"]
pipeline_options.accelerator_options = AcceleratorOptions(
    num_threads=4, device=AcceleratorDevice.AUTO)

converter = DocumentConverter(
    format_options={
        InputFormat.PDF: PdfFormatOption(pipeline_options=pipeline_options)
    })
x = converter.convert(source="./tests/data/2305.03393v1-pg9.pdf")

x.document

[Seigneurhol]

You don't have any problem with accent or special characters ?

[trinanjan12]

[Seigneurhol]

Yes you are right. On some document it works fine. But on other there are some encoding issue that don't happen in PyPdfiumDocumentBackend

[PeterStaar-IBM]

@Seigneurhol Can you provide the PDF that gives you problems? I am trying to fix all font related issues.

[zvictor]

I get encoding issues on a simple command:

docling --from pdf --to md --image-export-mode placeholder \
    https://venda-imoveis.caixa.gov.br/editais/EL00820224CPARE.PDF

That produces artifacts such as Leilªo Pœblico and Alienaçªo FiduciÆria instead of Leilão Público
and Alienação Fiduciária.

[Seigneurhol]

@PeterStaar-IBM I get the same error as @zvictor with word with french accent. I can't provide the same document for legal issues. Do you want another with the same issues or does the @zvictor document is enough ?

[PeterStaar-IBM]

@Seigneurhol The more debug examples the better. Working now on fixing as many font issues as possible ;)

[Seigneurhol]

I can't found another PDF that produce the same encoding errors but I will post when I found one :)

[PeterStaar-IBM]

Yes, please do!