MarkItDown
MarkItDown source walkthrough: trace a UTF-8 text conversion
Inspect the core and PlainTextConverter at a fixed commit to understand acceptance, charset handling, stream restoration and a focused plugin regression test.
What you will learn
- The plain-text converter accepts metadata before decoding content.
- A declared charset changes the decoding path.
- Cursor and empty-output tests target different contracts.
Before you start
- Basic Python and command-line usage
- A non-sensitive document whose contents you can verify
Use the chapter checklist to explain and verify this part of a document ingestion workflow.
Key takeaways
- The plain-text converter accepts metadata before decoding content.
- A declared charset changes the decoding path.
- Cursor and empty-output tests target different contracts.
Use two small source files as the reading route
This walkthrough inspects _markitdown.py and converters/_plain_text_converter.py at commit b6e8bbdce628d564c6af031b5f26cda6e818ea10. Start with the public input method, then the _convert dispatch loop, and finally PlainTextConverter. These observations concern those files; they do not establish the behavior of every PDF, Office or cloud converter.
A small UTF-8 text fixture makes that route understandable. Use bytes containing a heading and an accented word, supply a matching StreamInfo extension and charset, then inspect the returned Markdown. The example below is a regression fixture to run in the pinned environment, not a claim of a benchmark or an execution performed for this article.
Acceptance and decoding are separate operations
PlainTextConverter.accepts returns true when StreamInfo has a charset. Otherwise it checks a list of recognized extensions and MIME prefixes. Its acceptance method examines metadata without consuming the stream, satisfying the invariant enforced by the dispatcher.
PlainTextConverter.convert reads the bytes and decodes them with the supplied charset when present. Without one, it asks charset_normalizer for its best result and turns that result into text. It wraps the text in DocumentConverterResult; it does not add a document outline or infer table structure from arbitrary plain text.
Turn the trace into narrow regression tests
Test the documented stream contract with a custom converter whose acceptance probe temporarily reads bytes and then seeks back. A deliberately broken probe that leaves the cursor advanced should expose the dispatcher assertion. This is more diagnostic than running a directory of unrelated files and comparing only exit codes.
Test invalid declared encodings separately from unknown formats. A decoding exception occurs after a converter accepted the input and belongs to the failed-attempt path. Also test a converter returning empty text: a result object can exist even when your ingestion application should reject the output as unhelpful.
Implementation steps
- 1
Pin the cited commit and inspect the two linked source files.
- 2
Run the BytesIO fixture in that environment.
- 3
Add an invalid-charset fixture and inspect the exception.
- 4
Add a custom acceptance probe test that verifies cursor restoration.
Copy-ready example
from io import BytesIO
from markitdown import MarkItDown, StreamInfo
stream = BytesIO("# Example\n\nCafé\n".encode("utf-8"))
result = MarkItDown(enable_plugins=False).convert_stream(
stream, stream_info=StreamInfo(extension=".txt", charset="utf-8")
)
assert "Café" in result.markdown
print(result.markdown)Frequently asked questions
Is this a source analysis or a measured runtime trace?
It is an analysis of the cited source files, accompanied by fixtures for readers to execute. It does not report execution measurements.
Does plain-text conversion reconstruct semantic structure?
The inspected converter decodes bytes and returns text. Any headings already present in a text fixture come from that fixture.
Sources
- Pinned sourceSource checked 2026-09-07
- Core dispatcherSource checked 2026-09-07
- Plain-text converterSource checked 2026-09-07
- Public exportsSource checked 2026-09-07