7.2. codecs codec

: Lib/codecs.py

Python codec () codec Python codec codec codec codec bytes

codec

codecs.encode(obj, encoding='utf-8', errors='strict')

encoding codec obj

errors () 'strict' ValueError ( UnicodeEncodeError codec ) codec Codec

codecs.decode(obj, encoding='utf-8', errors='strict')

encoding codec obj

errors 'strict' ValueError ( UnicodeDecodeError codec ) codec Codec

codec

codecs.lookup(encoding)

Python codec codec CodecInfo

CodecInfo LookupError CodecInfo

class codecs.CodecInfo(encode, decode, streamreader=None, streamwriter=None, incrementalencoder=None, incrementaldecoder=None, name=None)

codec codec

name

encode
decode

Codec encode() decode() (see Codec )

incrementalencoder
incrementaldecoder

IncrementalEncoder IncrementalDecoder codec ()

streamwriter
streamreader

StreamWriter StreamReader codec

codec codec lookup() :

codecs.getencoder(encoding)

codec

LookupError

codecs.getdecoder(encoding)

codec

LookupError

codecs.getincrementalencoder(encoding)

codec

codec LookupError

codecs.getincrementaldecoder(encoding)

codec

codec LookupError

codecs.getreader(encoding)

codec StreamReader

LookupError

codecs.getwriter(encoding)

codec StreamWriter

LookupError

codec codecs

codecs.register(search_function)

codec 1 CodecInfo None

open() io codecs

codecs.open(filename, mode='r', encoding=None, errors='strict', buffering=1)

mode / StreamReaderWriter 'r'

'\n' mode open() 'b'

encoding codec

errors 'strict' ValueError

buffering open()

codecs.EncodedFile(file, data_encoding, file_encoding=None, errors='strict')

StreamRecoder

data_encoding file_encoding file_encoding data_encoding

file_encoding data_encoding

errors 'strict' ValueError

codecs.iterencode(iterator, encoding, errors='strict', **kwargs)

iterator generator errors ()

str base64_codec

codecs.iterdecode(iterator, encoding, errors='strict', **kwargs)

iterator generator errors ()

bytes rot_13 iterencode()

:

codecs.BOM
codecs.BOM_BE
codecs.BOM_LE
codecs.BOM_UTF8
codecs.BOM_UTF16
codecs.BOM_UTF16_BE
codecs.BOM_UTF16_LE
codecs.BOM_UTF32
codecs.BOM_UTF32_BE
codecs.BOM_UTF32_LE

Unicode (BOM) UTF-16 UTF-32 UTF-8 Unicode BOM_UTF16 BOM_UTF16_BE BOM_UTF16_LE BOM BOM_UTF16 BOM_LE BOM_UTF16_LE BOM_BE BOM_UTF16_BE UTF-8 UTF-32 BOM

7.2.1. Codec

codecs codec codec

Python codec 4 codec codec

7.2.1.1.

errors Python codec :
'strict' UnicodeError () strict_errors()
'ignore' ignore_errors()

'replace' Python codec U+FFFD '?' replace_errors()
'xmlcharrefreplace' XML () xmlcharrefreplace_errors()
'backslashreplace' backslashreplace_errors()
'namereplace' \N{...} () namereplace_errors()
'surrogateescape' U+DC80 U+DCFF 'surrogateescape' ( PEP 383 )

codec :

Codecs
'surrogatepass' utf-8, utf-16, utf-32, utf-16-be, utf-16-le, utf-32-be, utf-32-le codecc

3.1 : 'surrogateescape' 'surrogatepass'

3.4 : 'surrogatepass' utf-16* utf-32*

3.5 : 'namereplace'

3.5 : 'backslashreplace'

codecs.register_error(name, error_handler)

error_handler name name errors error_handler

error_handler UnicodeEncodeError str bytes IndexError

UnicodeDecodeError UnicodeTranslateError

()

codecs.lookup_error(name)

name

LookupError

codecs.strict_errors(exception)

strict UnicodeError

codecs.replace_errors(exception)

'replace' ( )(codec ) '?' '\ufffd' (Unicode )

codecs.ignore_errors(exception)

ignore

codecs.xmlcharrefreplace_errors(exception)

'xmlcharrefreplace' ( ) XML

codecs.backslashreplace_errors(exception)

'backslashreplace' ( )

codecs.namereplace_errors(exception)

'namereplace' ( )\N{...}

3.5 .

7.2.1.2.

Codec :

Codec.encode(input[, errors])

input (, ) ( cp1252 iso-8859-1)

errors 'strict'

Codec codecs StreamWriter

0

Codec.decode(input[, errors])

input (, )

codec input bytes buffer

errors 'strict'

Codec codecs StreamReader

0

7.2.1.3.

IncrementalEncoder IncrementalDecoder encode()/decode()

encode()/decode()

7.2.1.3.1. IncrementalEncoder

IncrementalEncoder Python codec

class codecs.IncrementalEncoder(errors='strict')

IncrementalEncoder

Python codec

IncrementalEncoder errors

errors IncrementalEncoder

encode(object[, final])

object () encode() final ()

reset()

.encode(object, final=True)

getstate()

0 ( marshal/pickle )

setstate(state)

state state getstate()

7.2.1.3.2. IncrementalDecoder

IncrementalDecoder Python codec

class codecs.IncrementalDecoder(errors='strict')

IncrementalDecoder

Python codec

IncrementalDecoder errors

errors IncrementalDecoder

decode(object[, final])

object () decode() final () final ()()

reset()

getstate()

212 ( 0 ) 0 0 ( marshal/pickle )

setstate(state)

(: ?) state state getstate()

7.2.1.4.

StreamWriter StreamReader encodings.utf_8

7.2.1.4.1. StreamWriter

StreamWriter Codec Python codec

class codecs.StreamWriter(stream, errors='strict')

StreamWriter

Python codec

stream codec

StreamWriter errors codec

errors StreamWriter

write(object)

object

writelines(list)

( write() ) codecs

reset()

codec

StreamWriter

7.2.1.4.2. StreamReader

StreamReader Codec Python codec

class codecs.StreamReader(stream, errors='strict')

StreamReader

Python codec

stream codec

StreamReader errors codec

errors StreamReader

errors register_error()

read([size[, chars[, firstline]]])

chars read()

size -1

firstline 1

size

readline([size[, keepends]])

1

size read() size

keepends

readlines([sizehint[, keepends]])

keepends codec

sizehint read() size

reset()

codec

StreamReader

7.2.1.4.3. StreamReaderWriter

StreamReaderWriter

lookup()

class codecs.StreamReaderWriter(stream, Reader, Writer, errors='strict')

StreamReaderWriter stream Reader Writer StreamReader StreamWriter

StreamReaderWriter StreamReader StreamWriter

7.2.1.4.4. StreamRecoder

StreamRecoder

lookup()

class codecs.StreamRecoder(stream, encode, decode, Reader, Writer, errors='strict')

StreamRecoder encode decode (read() write() ) Reader Writer (stream )

Latin-1 UTF-8

stream

encode decode Codec Reader Writer StreamReader StreamWriter

StreamRecoder StreamReader StreamWriter

7.2.2. Unicode

0x00x10FFFF ( PEP 393 ) CPU term: <text encoding>

('latin-1' 'iso-8859-1') 0255 0x00xff codec U+00FF UnicodeEncodeError (): UnicodeEncodeError: 'latin-1' codec can't encode character '\u1234' in position 3: ordinal not in range(256)

(charmap ) Unicode 0x00xff encodings/cp1252.py ( Windows ) 256

Unicode 1114112 256 Unicode 42: 2 UTF-32-BE UTF-32-LE UTF-32-BE UTF-32 : CPU UTF-16 UTF-32 BOM () Unicode U+FEFF UTF-16 UTF-32 (0xFFFE) Unicode UTF-16 UTF-32 U+FFFE U+FEFF ZERO WIDTH NO-BREAK SPACE : Unicode 4.0 ZERO WIDTH NO-BREAK SPACE U+FEFF ( U+2060 (WORD JOINER) ) Unicode U+FEFF : BOM ; ZERO WIDTH NO-BREAK SPACE

Unicode UTF-8 UTF-8 8 UTF-8 UTF-8 ()04 1 0 Unicode (x Unicode ):
U-00000000 U-0000007F 0xxxxxxx
U-00000080 U-000007FF 110xxxxx 10xxxxxx
U-00000800 U-0000FFFF 1110xxxx 10xxxxxx 10xxxxxx
U-00010000 U-0010FFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

Unicode x

UTF-8 8 BOM U+FEFF () ZERO WIDTH NO-BREAK SPACE

charmap UTF-8 UTF-8 Microsoft Notepad UTF-8 (Python 2.5 "utf-8-sig" ) Unicode UTF-8 BOM ( 0xef, 0xbb, 0xbf ) charmap ( iso-8859-1

LATIN SMALL LETTER I WITH DIAERESIS
RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK
INVERTED QUESTION MARK

)utf-8-sig BOM utf-8-sig codec 3 0xef, 0xbb, 0xbf 3UTF-8 BOM

7.2.3.

Python codec C codec ; 'utf-8' 'utf_8' codec

CPython : codec CPython () : utf-8, utf8, latin-1, latin1, iso-8859-1, iso8859-1, mbcs (Windows ), ascii, us-ascii, utf-16, utf16, utf-32, utf32

3.6 : us-ascii

(EURO SIGN ) :

  • ISO 8859
  • Microsoft Windows 8859
  • IBM EBCDIC
  • ASCII IBM PC
Codec
ascii 646, us-ascii
big5 big5-tw, csbig5
big5hkscs big5-hkscs, hkscs
cp037 IBM037, IBM039
cp273 273, IBM273, csIBM273

3.4 .
cp424 EBCDIC-CP-HE, IBM424
cp437 437, IBM437
cp500 EBCDIC-CP-BE, EBCDIC-CP-CH, IBM500
cp720  
cp737  
cp775 IBM775
cp850 850, IBM850
cp852 852, IBM852
cp855 855, IBM855
cp856  
cp857 857, IBM857
cp858 858, IBM858
cp860 860, IBM860
cp861 861, CP-IS, IBM861
cp862 862, IBM862
cp863 863, IBM863
cp864 IBM864
cp865 865, IBM865
cp866 866, IBM866
cp869 869, CP-GR, IBM869
cp874  
cp875  
cp932 932, ms932, mskanji, ms-kanji
cp949 949, ms949, uhc
cp950 950, ms950
cp1006   Urdu
cp1026 ibm1026
cp1125 1125, ibm1125, cp866u, ruscii

3.4 .
cp1140 ibm1140
cp1250 windows-1250
cp1251 windows-1251
cp1252 windows-1252
cp1253 windows-1253
cp1254 windows-1254
cp1255 windows-1255
cp1256 windows-1256
cp1257 windows-1257
cp1258 windows-1258
cp65001  

Windows : Windows UTF-8 (CP_UTF8)

3.3 .
euc_jp eucjp, ujis, u-jis
euc_jis_2004 jisx0213, eucjis2004
euc_jisx0213 eucjisx0213
euc_kr euckr, korean, ksc5601, ks_c-5601, ks_c-5601-1987, ksx1001, ks_x-1001
gb2312 chinese, csiso58gb231280, euc- cn, euccn, eucgb2312-cn, gb2312-1980, gb2312-80, iso- ir-58
gbk 936, cp936, ms936 Unified Chinese
gb18030 gb18030-2000 Unified Chinese
hz hzgb, hz-gb, hz-gb-2312
iso2022_jp csiso2022jp, iso2022jp, iso-2022-jp
iso2022_jp_1 iso2022jp-1, iso-2022-jp-1
iso2022_jp_2 iso2022jp-2, iso-2022-jp-2 , , , ,
iso2022_jp_2004 iso2022jp-2004, iso-2022-jp-2004
iso2022_jp_3 iso2022jp-3, iso-2022-jp-3
iso2022_jp_ext iso2022jp-ext, iso-2022-jp-ext
iso2022_kr csiso2022kr, iso2022kr, iso-2022-kr
latin_1 iso-8859-1, iso8859-1, 8859, cp819, latin, latin1, L1
iso8859_2 iso-8859-2, latin2, L2
iso8859_3 iso-8859-3, latin3, L3
iso8859_4 iso-8859-4, latin4, L4
iso8859_5 iso-8859-5, cyrillic
iso8859_6 iso-8859-6, arabic
iso8859_7 iso-8859-7, greek, greek8
iso8859_8 iso-8859-8, hebrew
iso8859_9 iso-8859-9, latin5, L5
iso8859_10 iso-8859-10, latin6, L6
iso8859_11 iso-8859-11, thai
iso8859_13 iso-8859-13, latin7, L7
iso8859_14 iso-8859-14, latin8, L8
iso8859_15 iso-8859-15, latin9, L9
iso8859_16 iso-8859-16, latin10, L10
johab cp1361, ms1361
koi8_r  
koi8_t  

3.5 .
koi8_u  
kz1048 kz_1048, strk1048_2002, rk1048

3.5 .
mac_cyrillic maccyrillic
mac_greek macgreek
mac_iceland maciceland
mac_latin2 maclatin2, maccentraleurope
mac_roman macroman, macintosh
mac_turkish macturkish
ptcp154 csptcp154, pt154, cp154, cyrillic-asian
shift_jis csshiftjis, shiftjis, sjis, s_jis
shift_jis_2004 shiftjis2004, sjis_2004, sjis2004
shift_jisx0213 shiftjisx0213, sjisx0213, s_jisx0213
utf_32 U32, utf32
utf_32_be UTF-32BE
utf_32_le UTF-32LE
utf_16 U16, utf16
utf_16_be UTF-16BE
utf_16_le UTF-16LE
utf_7 U7, unicode-1-1-utf-7
utf_8 U8, UTF, utf8
utf_8_sig  

3.4 : utf-16* utf-32* (U+D800U+DFFF) utf-32*

7.2.4. Python

codec Python codec Python codec codec codec

7.2.4.1.

codec Unicode str bytes bytes-like object str

Codec
idna   RFC 3490 encodings.idna errors='strict'
mbcs ansi, dbcs Windows : ANSI (CP_ACP)
oem  

Windows : OEM (CP_OEMCP)

3.6 .
palmos   PalmOS 3.5
punycode   RFC 3492 codecs
raw_unicode_escape   \uXXXX \UXXXXXXXX Latin-1 Python pickle
undefined  
unicode_escape   ASCII Python Unicode Latin-1 Python UTF-8
unicode_internal  

codecs

3.3 : PEP 393

7.2.4.2. (Binary Transforms)

codec bytes-like object bytes bytes.decode() str

Codec /
base64_codec [1] base64, base_64

MIME base64 ( '\n' )

3.4 : bytes-like object

base64.encodebytes() / base64.decodebytes()
bz2_codec bz2 bz2 bz2.compress() / bz2.decompress()
hex_codec hex 2 16 binascii.b2a_hex() / binascii.a2b_hex()
quopri_codec quopri, quotedprintable, quoted_printable MIME quoted printable quotetabs=True quopri.encode() / quopri.decode()
uu_codec uu uuencode uu.encode() / uu.decode()
zlib_codec zip, zlib gzip zlib.compress() / zlib.decompress()
[1] 'base64_codec' str ASCII

3.2 : (: 2.x 3.0 )

3.4 : (: 2.x 3.2 )

7.2.4.3. (Text Transforms)

codec str str str.encode() bytes

Codec
rot_13 rot13 (Caesar-cypher)

3.2 : rot_13 (: 2.x 3.0 )

3.4 : rot13 (: 2.x 3.2 )

7.2.5. encodings.idna (IDNA)

RFC 3490 ( IDNA: Internationalized Domain Names in Applications) RFC 3492 (Nameprep: (IDN) stringprep ) punycode stringprep

RFC ASCII (www.Alliancefranaise.nu ) ASCII ASCII (ACE www.xn--alliancefranaise-npb.nu ) ACE DNS HTTP Host ; : Unicode IDNA ACE Unicode

Python supports this conversion in several ways: the idna codec performs conversion between Unicode and ACE, separating an input string into labels based on the separator characters defined in section 3.1 (1) of RFC 3490 and converting each label to ACE as required, and conversely separating an input byte string into labels based on the . separator and converting any ACE labels found into unicode. Furthermore, the socket module transparently converts Unicode host names to ACE, so that applications need not be concerned about converting host names themselves when they pass them to the socket module. On top of that, modules that have host names as function parameters, such as http.client and ftplib, accept Unicode host names (http.client then also transparently sends an IDNA hostname in the Host field if it sends that field at all).

() Unicode : Unicode

encodings.idna nameprep nameprep nameprep

encodings.idna.nameprep(label)

label nameprep AllowUnassigned

encodings.idna.ToASCII(label)

RFC 3490 ASCII UseSTD3ASCIIRules

encodings.idna.ToUnicode(label)

RFC 3490 Unicode

7.2.6. encodings.mbcs Windows ANSI

ANSI (CP_ACP)

: Windows

3.3 :

3.2 : 3.2 errors ; 'replace' 'ignore'

7.2.7. encodings.utf_8_sig BOM UTF-8

UTF-8 codec UTF-8 UTF-8 BOM () UTF-8 BOM