# Entropy

This page is the answer to the task Entropy in the Rosetta Code.

### Description (from Rosetta Code)

Task
Calculate the Shannon entropy H of a given input string. Given the discrete random variable that is a string of "symbols" (total characters) consisting of different characters (n=2 for binary), the Shannon entropy of X in where is the count of character . For this task, use X=" This coding problem calculates the "specific" or "intensive" entropy that finds its parallel in physics with "specific entropy" S The "total", "absolute", or "extensive" information entropy is - bits
This is not the entropy being coded here, but it is the closest to physical entropy and a measure of the information content of a string. But it does not look for any patterns that might be available for compression, so it is a very restricted, basic, and certain measure of "information". Every binary file with an equal number of 1's and 0's will have S=N bits. All hex files with equal symbol frequencies will have bits of entropy. The total entropy in bits of the example above is S= 10*18.4644 = 18.4644 bits. The H function does not look for any patterns in data or check if X was a random variable. For example, X=000000111111 gives the same calculated entropy in all senses as Y=010011100101. For most purposes it is usually more relevant to divide the gzip length by the length of the original data to get an informal measure of how much "order" was in the data. Two other "entropies" are useful: Normalized specific entropy: which varies from 0 to 1 and it has units of "entropy/symbol" or just 1/symbol. For this example, H Normalized total (extensive) entropy: which varies from 0 to N and does not have units. It is simply the "entropy", but it needs to be called "total normalized extensive entropy" so that it is not confused with Shannon's (specific) entropy or physical entropy. For this example, S Shannon himself is the reason his "entropy/symbol" H function is very confusingly called "entropy". That's like calling a function that returns a speed a "meter". See section 1.7 of his classic A Mathematical Theory of Communication and search on "per symbol" and "units" to see he always stated his entropy H has units of "bits/symbol" or "entropy/symbol" or "information/symbol". So it is legitimate to say entropy NH is "information". In keeping with Landauer's limit, the physics entropy generated from erasing N bits is if the bit storage device is perfectly efficient. This can be solved for H |

### Program

**Explanation**

The `Count`

expression return the number of occurrences of every distinct character of a string, as follows:

The `Lg`

expression is the binary logarithm.

### Example

It seems confusing at start, because Fōrmulæ always try to give exact results.

To get a strictly numeric value, use the `N`

expression: