The Byte Order Marker (BOM) is a series of byte values placed on the beginning of an encoded text stream (or file). This data allows the reader to correctly decide which character encoding to use when decoding the stream back into a sequence of characters. The use of byte order markers within files and streams is not specific to XML, but its typical to see them in use when XML data is streamed or stored in files.

The first thing to appreciate is there are a number of ways in which a sequence of characters can be represented in a binary form. The process of turning characters into binary data is termed character encoding.

The following table is a list of the standard BOM's in use for common encodings.

Example

As you can see the first 3 byte of the file are 0xEF 0xBB, 0xBF, looking this up in our table we can see that the encoding is UTF-8.

The rest of the file can therefore be read and decoded using this encoding.

Common Problems

Most platforms and languages have built into support for decoding character streams, so most of the time you are unaware that they are even in use.

using (Stream s = new FileStream("XmlTheSmartWay.txt"))
{
    using (TextReader tr = new StreamReader(s))
    {
        string txt = tr.ReadToEnd();
    }
}

The above sample will read and decode the values correctly, however if support is missing or bypassed then problems can occur. In this example the reader is assuming 1 byte per char, so if the file contains utf-8 data any values above 127 will be incorrectly interpreted.

StringBuilder sb = new StringBuilder();

using (Stream s = new FileStream("XmlTheSmartWay.txt"))
{
    while ((int value = s.ReadByte()) != -1)
    {
        sb.Append((char)value);
    }
}
string txt = sb.ToString();