Borislav Hadzhiev
Last updated: May 2, 2022
Photo from Unsplash
The Python "UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in
position: invalid start byte" occurs when we specify an incorrect encoding when
decoding a bytes object. To solve the error, specify the correct encoding, e.g.
cp1252
.
Here is an example of how the error occurs.
my_bytes = 'abc ’ def'.encode('cp1252') # ⛔️ UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 4: invalid start byte my_str = my_bytes.decode('utf-8')
The 0x92
byte can be decoded using the cp1252
encoding.
string
to a bytes
object and decoding is the process of converting a bytes
object to a string
.When decoding a bytes object, we have to use the same encoding that was used to encode the string to a bytes object.
In the example, we can set the encoding to cp1252
.
my_bytes = 'abc ’ def'.encode('cp1252') my_str = my_bytes.decode('cp1252') print(my_str) # 👉️ "abc ’ def"
Windows-1252 or cp1252
is a single-byte character encoding of the Latin
alphabet.
If that doesn't work, try using the latin-1
encoding and see if the data is
legible.
If you got the error when reading from a file using pandas
, try setting the
encoding to cp1252
or latin-1
.
import pandas as pd # 👇️ set encoding to cp1252 df = pd.read_csv('employees.csv', sep='|', encoding='cp1252') print(df)
You can try doing the same if using the native open()
function.
import csv with open('employees.csv', newline='', encoding='cp1252') as csvfile: csv_reader = list(csv.reader(csvfile, delimiter='|')) print(csv_reader)
If the error persists, you could set the errors
keyword argument to ignore
to ignore the characters that cannot be decoded.
Note that ignoring characters that cannot be decoded can lead to data loss.
import csv # 👇️ set errors to ignore with open('employees.csv', newline='', encoding='utf-8', errors='ignore') as csvfile: csv_reader = list(csv.reader(csvfile, delimiter='|')) print(csv_reader)
Opening the file with an incorrect encoding with errors
set to ignore
won't
raise a UnicodeDecodeError
.
Make sure you didn't open a file in rb
(binary) mode if you have to read from
it.
with open('example.txt', 'rb') as f: data = f.read() print(data)
We opened the file in binary mode (using the rb
mode), so the data
variable
contains a bytes object.
You shouldn't specify encoding
when opening a file in binary mode.
You can use this approach if you need to upload the file to a remote server and don't need to decode it.
string
to a bytes
object and decoding is the process of converting a bytes
object to a string
.When decoding a bytes object, we have to use the same encoding that was used to encode the string to a bytes object.
Here is an example that shows how using a different encoding to encode a string to bytes than the one used to decode the bytes object causes the error.
my_text = 'a ’ b' my_binary_data = my_text.encode('cp1252') # ⛔️ UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 2: invalid start byte my_text_again = my_binary_data.decode('utf-8')
We can solve the error by using the cp1252
encoding to decode the bytes
object.
my_text = 'a ’ b' my_binary_data = my_text.encode('cp1252') my_text_again = my_binary_data.decode('cp1252') print(my_text_again) # 👉️ "a ’ b"