Recommendations for designing magic numbers of binary file formats

7 comments

• conaclos 40 minutes ago

     SHOULD include a zero byte
  

  I guess it is expected to be at the end of the magic number to act as a null-termibated string?

     MUST include a byte sequence that is invalid UTF-8
  

  I guess it is to differentiate a text file from a specific format?

     MUST include at least one byte with the high bit set
  

  Any reason?
  [-]
  • robinhouston a few seconds ago

    I think the idea of all these is to make the file not be recognised as text (which doesn't allow nulls), ASCII (which doesn't use the high bit), UTF-8 (which doesn't allow invalid UTF-8 sequences).

    Basically so that no valid file in this binary format will be incorrectly misidentified as a text file.

  • wjholden 36 minutes ago

    The author explains their reasoning in the next post: https://hackers.town/@zwol/114155807716413069

• hgomersall 39 minutes ago

  As anyone able to break down why those requirements are desirable?

• weinzierl 2 hours ago

  Why is ELF a good example?

      7F 45 4C 46
  

  - MUST be the very first N bytes in the file -> check

  - MUST be at least four bytes long, eight is better -> check, but only four

  - MUST include at least one byte with the high bit set -> nope

  - MUST include a byte sequence that is invalid UTF-8 -> nope

  - SHOULD include a zero byte -> nope

  So, just 1.5 out of 5. Not good.

  By the way, does anyone know the reason it starts with DEL (7F) specifically?

  [-]
  • 0xFF0123 an hour ago

    It's (7F) ELF

    [-]
    • indigoabstract 13 minutes ago

      Hmm, I would expect that to be 31F, if it stood for "ELF" in correct Hexspeak.