Length of a Nautical Mile at a Given Latitude

Date: 11/06/2002 at 15:58:11
From: Keith Shears
Subject: Length of a Nautical Mile in a given Latitude

As the actual length of a nautical mile varies with latitude, I am 
trying to work out a formula for accurately predicting the exact 
length of a nautical mile in any given latitude, in metres. I believe 
this may require the use of some form of spheroidal calculation to 
allow for the flattening for the earths ellipsoid shape. I was given 
one formula as follows:

   true length of a nautical mile = 1852.3 - 9.4 Cosine(2 x Latitude)

But I have no reference or proof of this formula or its origins. It 
would appear to use the length of a nautical mile at Latitude 45 
degrees - so I can't prove whether this is accurate or not.

Date: 11/07/2002 at 10:41:01
From: Doctor Rick
Subject: Re: Length of a Nautical Mile in a given Latitude

Hi, Keith.

I have never heard anyone say that the nautical mile is not a fixed 
distance. See the How Many? web site linked from the Dr. Math FAQ:

   A Dictionary of Units of Measurement - Russ Rowlett
   http://www.unc.edu/~rowlett/units/dictN.html 

It says:

"nautical mile (nmi, naut mi or NM) 

"a unit of distance used primarily at sea and in aviation. The 
nautical mile is defined to be the average distance on the Earth's 
surface represented by one minute of latitude. This may seem odd to 
landlubbers, but it makes good sense at sea, where there are no mile 
markers but latitude can be measured. Because the Earth is not a 
perfect sphere, it is not easy to measure the length of the nautical 
mile in terms of the statute mile used on land. For many years the 
British set the nautical mile at 6080 feet (1853.18 meters), exactly 
800 feet longer than a statute mile; this unit was called the 
Admiralty mile. Until 1954 the U.S. nautical mile was equal to 
6080.20 feet (1853.24 meters). In 1929 an international conference in 
Monaco redefined the nautical mile to be exactly 1852 meters or 
6076.115 49 feet, a distance known as the international nautical 
mile. The international nautical mile equals about 1.1508 statute 
miles. There are usually 3 nautical miles in a league. The unit is 
designed to equal 1/60 degree [2], although actual degrees of 
latitude vary from about 59.7 to 60.3 nautical miles."

The nautical mile is the AVERAGE distance of a minute of latitude, 
which is a fixed distance, not the actual distance at each particular 
point (which does vary with latitude due to the non-sphericity of the 
earth). 

Perhaps you are really asking for a formula for the length of a minute 
of latitude at any latitude on an ellipsoidal model of the earth.

The radius of curvature in the meridian (north-south plane) is greater 
at the poles and less at the equator. The length of a minute of 
latitude is proportional to the radius of curvature. This is 
consistent with your formula. Let's check out the details.

The formula for the radius of curvature in the meridian at a given 
latitude is

  M = a(1-e^2)/(1-e^2*sin^2(lat))^3/2

where a is the semi-major axis (radius of the earth at the equator) 
and e is the eccentricity of the earth. In the World Geodetic System 
(WGS),

  a = 6378137 m
  e^2 = 0.00669438002290

Thus

  M = 6335439.32708 / (1 - 0.00669438002290*sin^2(lat))^3/2

One minute of arc is 1/60 degree, and a degree is pi/180 radians, so

  1 min arc = pi/10800 = 0.000290888208665 radian

We multiply this by the radius of curvature M to get the length of a 
minute of arc:

  1842.90459693 / (1 - 0.00669438002290*sin^2(lat))^3/2

This does not look like

  1852.3 - 9.4*cos(2*lat)

However, when I calculate both formulas using a spreadsheet, the 
differences aren't so great:

Lat  Yours         Mine
------------------------------
0    1842.9        1842.904597
10   1843.466889   1843.462752
20   1845.099182   1845.071468
30   1847.6        1847.540708
40   1850.667707   1850.57721
50   1853.932293   1853.817733
60   1857          1856.871458
70   1859.500818   1859.367085
80   1861.133111   1860.999007
90   1861.7        1861.566326

Maybe with a little work we could put our formulas into forms that 
would show the relation more clearly. I'll leave it at this, though, 
unless you want to pursue it further.

- Doctor Rick, The Math Forum
  http://mathforum.org/dr.math/ 

Date: 11/07/2002 at 14:48:18
From: Keith Shears
Subject: Thank you (Length of a Nautical Mile in a given Latitude)

A big thank you for your efforts; at least now I have two formulae, 
and I can see the proof of one of them - yours - and understand its 
origins!

Keith

Date: 11/07/2002 at 15:17:46
From: Doctor Rick
Subject: Re: Thank you (Length of a Nautical Mile in a given Latitude)

Hi, Keith.

I thought I'd take a quick look at how my formula might be 
approximated. Using the fact that

  (1+x)^n = 1 + nx + (n(n-1)/2!)x^2 + ...

we can approximate my formula

  1842.90459693 / (1 - 0.00669438002290*sin^2(lat))^3/2

as

  1842.90459693(1 + (3/2)0.00669438002290*sin^2(lat)) =
  1842.90459693 + 18.5056555767sin^2(lat)

Since sin^2(lat) = 1 - cos^2(lat), this equals

  1842.90459693 + 18.5056555767(1-cos^2(lat)) =
  1861.4102525067 - 18.5056555767cos^2(lat)

Your formula had cos(2*lat). Using

  cos(2x) = 2*cos^2(x) - 1

  cos^2(x) = (1+cos(2x))/2

the approximate formula becomes

  1861.4102525067 - 18.5056555767(1+cos(2*lat))/2 =
  1852.15742471835 - 9.25282778835cos(2*lat)

Compare this with your formula:

  1852.3 - 9.4*cos(2*lat)

We're pretty close. We can get an idea of the error in the 
approximation by looking at the next term in the series:

  (n(n-1)/2!)x^2 = (3/2)(1/2)/2 * (0.00669438002290*sin^2(lat))^2 =
  0.0000168055214591*sin^4(lat)

This error is greatest at the poles, where sin(lat) = 1 and we get an 
error of 0.0000168 * 1842.90459693 = 0.031 meters. The difference 
between our formulas at 90 degrees latitude was 0.14 meter; but then, 
the coefficients weren't the same as in my approximation.

- Doctor Rick, The Math Forum
  http://mathforum.org/dr.math/