Issue link: http://dc.ee.ubm-us.com/i/71849

In the language of statistics, the lightning has hung a fat tail on our probability distribution. adding up the values and dividing by N. We will deal with such practical details in a moment. Another, more fundamental, rea on calls for a short diversion into theory. If the data had an underlying statis- tical distribution with thin tai ls- something like the classic normal den- sity function shown in Figw·e 1 (red trace)-the ordina•·y mean would work be tter. In this ideal case, the median would be noisier than the mean by a factor near -JW2l "' 1.25. However, distributions in the real world often have fat tai ls. Consider the di stributio n shown in Figure 1 (blue trace). It is 90% the same as before, but 10% of the probabili ty follows a normal distribution 10 Limes wider. What I'm trying to rep- resent is a composite process: some- thing well-behaved is making the na r- row central peak, but something else is going on at a low level and making outlie rs. The mean allows the outlier -rep- resented by those fat tails-to affect its estimate u111·easonably strongly, and is 3.3 times as noisy with the new data. In contrast, the median focuses its atten- tion on the largely undamaged proba- bility peak in the center, and is noisie r by a factor near 1 .1. This time, the median is quieter than the mean by a factor near 2.5. Quite a difference for such a subtle change in the shape of tl1e disu·ibution! So it matte rs what the unde rlying statistical distributio n is. As is usual, I have used th e normal distribution for conve ni e n ce a nd familiarity when doing theory. Some real-world situa tions, like o ur lightning storm, have eve n longer tails that are hard- e r to treat analytically, but show off the median to even better advan- tage. I o nce saw a really good exam- ple at very close range: a pair of Confidence intervals So how did I get those theoretical results? Confidence intervals for the mean are based on the variance, a2, of the composite distribution. To form that. you form the appro- priate weighted average of variances of the two distributions that make it up. As always when averaging squared quantities, the larger a quickly takes control of the result. In sharp contrast, confidence intervals for the median are based on the central den- sity instead\ and that is where we win. For the two normal distributions of our exam- ple, the central density goes like 1 Ia and we take the appropriate weighted average of that quantity. As always when averaging reciprocals, the smaller a takes control! In cal- culating the median, the most common, organized data behavior dominates any rare large excursions just as we wish it to. 1. The median m is approximately normal with , 2 I neglect a factor of N/(N-1) in the noise comparisons. For the normal distribution, f(m) = f(p) = 1/n"'(2:r). Mathematical Statisttcs, Freund, Prentice-Hal! (1962). = 1/(4 (N-1) f2(m)], f(m) the central density. N odd FIGURE 1 Density functions . ·. :::: · _ : ;I I, l! 0.3 ' 0.25 ' li ,, 'I 11 !! 0.2 0.15 I. o.1 I· 0.05 10 20 30 Embedded Systems Programming NOVEMBER 2ooo 101

- November001
- November002
- November003
- November004
- November005
- November006
- November007
- November008
- November009
- November010
- November011
- November012
- November013
- November014
- November015
- November016
- November017
- November018
- November019
- November020
- November021
- November022
- November023
- November024
- November025
- November026
- November027
- November028
- November029
- November030
- November031
- November032
- November033
- November034
- November035
- November036
- November037
- November038
- November039
- November040
- November042
- November043
- November044
- November045
- November046
- November047
- November048
- November049
- November050
- November051
- November052
- November053
- November054
- November055
- November056
- November057
- November058
- November059
- November060
- November061
- November062
- November063
- November064
- November065
- November066
- November067
- November068
- November069
- November070
- November071
- November072
- November073
- November074
- November075
- November076
- November077
- November078
- November079
- November080
- November081
- November082
- November083
- November084
- November085
- November086
- November087
- November088
- November089
- November090
- November091
- November092
- November093
- November094
- November095
- November096
- November097
- November098
- November099
- November100
- November101
- November102
- November103
- November104
- November105
- November106
- November107
- November108
- November109
- November110
- November111
- November112
- November113
- November114
- November115
- November116
- November117
- November118
- November119
- November120
- November121
- November122
- November123
- November124
- November125
- November126
- November127
- November128
- November129
- November130
- November131
- November132
- November133
- November134
- November135
- November136
- November137
- November138
- November139
- November140
- November141
- November142
- November143
- November144
- November145
- November146
- November147
- November148
- November149
- November150
- November151
- November152
- November153
- November154
- November155
- November156
- November157
- November158
- November159
- November160
- November161
- November162
- November163
- November164
- November165
- November166
- November167
- November168
- November169
- November170
- November171
- November172
- November173
- November174
- November175
- November176
- November177
- November178
- November179
- November180
- November181
- November182
- November183
- November184
- November185
- November186
- November187
- November188
- November189
- November190