Cataloging & MARC
Broken characters (�, é) in Koha after a MARC import
Diamonds with question marks, or é shown as é, mean the file was staged under the wrong character encoding. The characters cannot be repaired in Koha; undo the batch and import again with the right encoding.
By Babar Junaid
Published 2026-11-19 · Tested against Koha 26.05
Characters shown as �, or é shown as é, mean the MARC file was staged under the wrong character encoding. Koha's staging screen assumes UTF-8 and reports success either way: the only sign is a warning in the server log. A MARC-8 or Windows file staged that way has every accented, Urdu or Arabic letter replaced. Revert the import batch, convert the file to UTF-8 (or pick its real encoding when staging) and import it again.
The damaged records cannot be repaired in place
- Keep the original file exactly as you received it.
- If staff have edited the imported records since, note which ones before you revert the batch.
Step 1: Tell the two patterns apart
| What you see | What happened |
|---|---|
� next to an unaccented letter: Garc�ia M�arquez | A MARC-8 file staged as UTF-8. MARC-8 stores the accent as its own byte before the letter, so the letter survives and the accent becomes �. Stage it as MARC 8 |
� in place of the letter: Caf� cr�me, or whole Urdu and Arabic words as � � � | A single-byte file (Latin-1, a Windows code page such as Windows-1256) staged as UTF-8. The letter itself is gone. Convert it (step 4) |
é, ü, ال and similar pairs | A file that was already UTF-8, staged as MARC 8 or ISO 8859-1, so it was converted a second time |
| Only some records broken | The file mixes encodings, usually several exports joined together |
Step 2: Undo the import
- 1Go to Cataloging → Manage staged records.
- 2Open the batch you imported.
- 3Click Undo import into catalog. Records the batch created are deleted; records it overlaid are put back as they were.
Step 3: Find out what encoding the file really is
$ yaz-marcdump records.mrc | head -n 300155nam 2200061 a 4500001 kb40-1100 1 $a Garc�ia M�arquez, Gabriel.Response from a live run on Koha 26.05
The first line printed is the leader. Its 10th character (position 09) is a when the file claims UTF-8 and a space when it claims MARC-8, as here (nam 22: two spaces). Do not trust it on its own; many systems set it wrongly. Look at a record you know: if its accented or Urdu letters print correctly in a UTF-8 terminal, the file is UTF-8. This one is MARC-8, and it shows the MARC-8 pattern: the base letter survives and the accent beside it turns into � (Garc�ia, not Garc�a).
Files exported from older desktop systems in Urdu, Arabic or Persian are often in a Windows code page such as Windows-1256. There every letter of the script prints as �, for example 245 00 $a ����� ����. Koha's staging screen has no option for it, so those files must be converted first (step 4).
If the records came from a service you export from regularly (OCLC, a vendor portal, another library system), check its export setting too and set it to UTF-8, so the next file arrives right and needs no conversion.
Step 4: Convert the file to UTF-8
yaz-marcdump is installed with Koha. Give it the real source encoding with -f, the target with -t, binary MARC out with -o marc, and -l 9=97 to set leader position 09 to a (97 is the code for the letter a):
$ yaz-marcdump -f MARC-8 -t UTF-8 -o marc -l 9=97 records.mrc > records-utf8.mrc$ yaz-marcdump records-utf8.mrc | head -n 300160nam a2200061 a 4500001 kb40-1100 1 $a García Márquez, Gabriel.Response from a live run on Koha 26.05
The conversion prints nothing; check the new file, as the second command does. Position 09 of the leader is now a and the letters read correctly. Converting from MARC-8, yaz-marcdump writes each accented letter as two characters, the letter and a separate accent. It looks the same, and Koha joins them back into one when it stages the file, so nothing more is needed. For a Latin-1 file use -f ISO-8859-1. For a Windows Arabic-script file use -f CP1256:
$ yaz-marcdump -f CP1256 -t UTF-8 -o marc -l 9=97 urdu.mrc > urdu-utf8.mrc$ yaz-marcdump urdu-utf8.mrc00081nam a2200049 a 4500001 kb40-2245 00 $a ديوان غالبResponse from a live run on Koha 26.05
One thing to know for Urdu: Windows-1256 has no Urdu ی (U+06CC), so a file in that code page always stored the Arabic ي (U+064A) in its place, and the converted record keeps it. The text reads correctly, but it is not the letter an Urdu keyboard types. After importing, search for a title that contains it to see whether your catalog finds it. Check a few more records with yaz-marcdump records-utf8.mrc | less before importing.
Step 5: Stage it again with the right setting
Stage records-utf8.mrc in Cataloging → Stage records for import with Character encoding left at UTF-8. If you did not convert the file, choose its real encoding instead: MARC 8 for most older MARC21 exports, ISO 8859-1 for Latin-1. Before you import, open one record from the staged batch preview and check its title.
Why this happens
Koha stores everything as UTF-8 and converts what it imports. To convert, it has to know what the file was, and the file cannot say so reliably. MARC21 has one flag for it, leader position 09, and so many systems have set it wrongly over the years that Koha's own code ignores it. The staging screen therefore asks you, and defaults to UTF-8.
When the answer is wrong, Koha has two bad choices: refuse the record or keep it with the unreadable bytes replaced. It keeps it, replacing each unreadable byte with the Unicode replacement character, so the import "succeeds" and the damage shows up later at the OPAC. It is not only cosmetic: a title or name with � in it no longer matches what readers type, so the record drops out of search, sorts in the wrong place and fails to match its authority record. The reverse mistake, converting a file that was already UTF-8, gives the é pairs: the conversion has run twice.
Troubleshooting
| Problem | What to check |
|---|---|
| Letters right in the staff interface, wrong in exported files | The program opening the export, often Excel, is guessing the encoding. Open it as UTF-8 |
| Right in Koha, wrong in a Z39.50 record you copied | The target server's encoding is set wrongly: see adding a Z39.50 server |
| Urdu or Arabic letters right but in the wrong order | A display direction problem, not encoding: the record's script and the OPAC language settings |
| Records you just imported cannot be found by search | The index, not the characters: rebuild the Zebra index |
The full staging and matching workflow is in importing MARC records into Koha, and the queue that runs the import is explained in background jobs stuck at New.
Tested on live Koha 26.05.05
Passed all 13 checks · Read the run log
- Koha release
- 26.05.05
- System
- Ubuntu 24.04
- Setup
- A fresh Ubuntu 24.04 server reached over SSH as a sudo user, Koha installed from the Koha repository as our installation guide leaves it.
- Source read at
- Koha commit d3fc229
Earlier live checks (1)
- , Koha 26.05.05 (commit d3fc229): all 11 claims held. Read the run log
In the Koha manual (26.05)
Verified against the Koha source (5 files)
- C4/ImportBatch.pm (lines 370-376)
- koha-tmpl/intranet-tmpl/prog/en/modules/tools/stage-marc-import.tt (lines 158-170)
- C4/ImportBatch.pm (lines 1624-1647)
- Koha/ImportBatch.pm (lines 85-110)
- C4/Charset.pm (lines 214-330, 769-784)
The label under each command says where its response came from; a command with no label was not run live. How we test and which standards we follow.
Characters still broken after re-importing? Send us a sample record and we will convert the whole file for you.
Want your staff trained on this? We teach it hands on with a live Koha, online or at your library. Ask about staff training
Related
More on cataloging & marc
Answers to the questions that usually arrive with this one.
Importing MARC records into Koha
Staging first, committing second — and the reason that two-step design is the most useful safety feature Koha has. With the command-line equivalents for files too big for a browser.
"Something went wrong when loading the table" on a Koha record
The record opens, the holdings table does not. It is one bad row rather than a broken Koha — usually items with no home or holding library, which an import can create in seconds.
MARC21 or UNIMARC: the choice you cannot undo
Koha asks for a MARC flavour before you catalogue anything, and changing it later means re-importing the catalogue. Unless you have a specific reason, choose MARC21.
Adding a Z39.50 target to Koha for copy cataloging
Add a remote catalog once and your catalogers stop typing records by hand. The settings, the targets that answer today, how to put your preferred source first, and why a search returns nothing.

