Dgdrxrt

Let $t=fracpi5$ (so $t$ is $36^circ$). Since $108=180-72$, we have $3t=pi-2t$ and therefore
$$sin(3t)=sin(pi-2t).$$
But $sin(pi-2t)=sin(2t)=2sin tcos t$.

Also, by the identity you were given, $sin(3t)=3sin t-4sin^3 t$. Thus
$$3sin t-4sin^3 t=2sin tcos t.$$
But $sin tne 0$, so we can cancel a $sin t$ and obtain
$$3-4sin^2 t=2cos t.$$
Substitute $1-cos^2 t$ for $sin^2 t$ and simplify a bit. We get
$$4cos^2 t-2cos t-1=0.$$
Use the Quadratic Formula to solve this quadratic equation for $cos t$, rejecting the negative root. We get
$$cos t=frac2+sqrt208.$$
We can simplify this to $dfrac1+sqrt54.$

To find $cospi/5$, note that

$$sin(3 pi/5) = sin(2 pi/5)$$

and

$$sin(3 pi/5) = 3 sin(pi/5) - 4 sin^3(pi/5) = 2 sin(pi/5) cos(pi/5)$$

Thus

$$2 cos(pi/5) = 3 - 4 sin^2(pi/5) = 4 cos^2(pi/5) - 1$$

Let $y=cos(pi/5)$. Then

$$4 y^2-2 y-1=0 implies y = frac1 + sqrt54$$

because $y>0$. Thus, $cos(pi/5) = (1+sqrt5)/4$.

Since rlgordonma has given the answer suggested by the supplied hint, here is another method of computing $cos(pi/5)$.

In this answer, it is shown that
$$
tan(5theta)=frac5tan(theta)-10tan^3(theta)+tan^5(theta)1-10tan^2(theta)+5tan^4(theta)
$$
which, since $tan(pi/2)=infty$, implies that
$$
5tan^4(pi/10)-10tan^2(pi/10)+1=0
$$
which, by the quadratic formula, yields
$$
tan^2(pi/10)=frac5-2sqrt55
$$
Adding $1$ and taking the reciprocal yields
$$
frac1+cos(pi/5)2=cos^2(pi/10)=frac5+sqrt58
$$
Therefore,
$$
cos(pi/5)=frac1+sqrt54=phi/2
$$
where $phi$ is the Golden Ratio.

Here is a way using roots of unity.

We have $cos 36 = fracomega + omega^-12$ where $omega = textexp left ( frac2 pi10 right )$. We have $-w$ is a primitive $5^th$ root of unity, so it follows $omega^4 - omega^3 + omega^2 - omega + 1 = 0$. Now, $x^4 - x^3 + x^2 - x + 1 = x^2(x^2 - x + 1 - frac1x + frac1x^2) = x^2left ( left (x + frac1x right )^2 - left (x + frac1x right ) -1 right )$. Thus $0 = omega^2 cdot ((2 cos 36)^2 - (2 cos 36) - 1)$. Therefore $4 cos^3 36 - 2 cos 36 - 1 = 0$. Using the quadratic formula we then arrive at $$cos 36 = frac1 + sqrt54$$

http://mathuprising.comlu.com/images/pentagon.png

This is a graphic that I created for my website about 2 days ago. Using $Delta ABC$ invoke the Law of Cosines ($angle BAC = angle BCA = 36^circ$).

BEGIN QUOTE: Ptolemy used geometric reasoning based on Proposition 10 of Book XIII of Euclid's Elements to find the chords of $72^circ$ and $36^circ$. That Proposition states that if an equilateral pentagon is inscribed in a circle, then the area of the square on the side of the pentagon equals the sum of the areas of the squares on the sides of the hexagon and the decagon inscribed in the same circle. END QUOTE

http://en.wikipedia.org/wiki/Ptolemy%27s_table_of_chords#How_Ptolemy_computed_chords

Finding the chord of $72^circ$ amounts to finding the sine of $36^circ$.

Here is Proposition  10 of Book VIII.

Another with roots of unity ...

Let $omega:=expleft(fracpi i5right)$ and $phi$ is the golden ratio defined via relation $$phi^2:=phi+1qquad|quadphi>0taga$$i.e., explicitly : $phi=frac1+sqrt52$ (but only (a) is sufficient to proceed further through this text, without radicals of 5's at start), then following holds true :

Via Euler's formula: $$Reomega=Rebigg expleft(fracpi i5right) bigg = cos left(fracpi5right)$$

Similarly

$$Imomega=sin left(fracpi5right)$$

Moreover, it's clear, by properities of the function cosine (resp. sine) on the interval $(0,fracpi2)$, that $$Reomega>0qquadmathrmresp.qquadImomega>0tag0$$ By Moivre's formula also $$omega^5 = e^ipi=-1$$
I.e.:
$$omega^5+1=0$$

Since $omeganeq -1$ we divide this by factor $omega+1$, hence

$$omega^4-omega^3+omega^2-omega+1=0tag1$$

This could be wieved as a polynomial degree 4 in $omega$, so we may guess, it can be factorised more suggesting :

$$omega^4-omega^3+omega^2-omega+1 = (omega^2-alphaomega+1)(omega^2-betaomega+1)tag2$$
Expanding out we get a system for $alpha$ and $beta$ :

$$alpha+beta=1$$
$$alphabeta=-1$$
Subctracting $phi$ in the equation for golden ratio and dividing through $phi$ we get an equivalent relation $$phi-frac1phi=1tagb$$

So the conditions for $alpha$ and $beta$ are met when :

$$alpha=phi$$ $$beta=-frac1phi$$

Now we have from $(1)$ and $(2)$ two quadratic equations for $omega$ :

$$omega=fracalpha2pmfrac12sqrtalpha^2-4qquad mathrmorqquad omega=fracbeta2pmfrac12sqrtbeta^2-4tag3$$

Dividing $(b)$ by $phi$ and rearanging we get
$$frac1phi=1-frac1phi^2tagc$$

So then $$alpha^2-4=phi^2-4overset(a)=phi-3overset(b)=frac1phi-2overset(c)=-frac1phi^2-1<0 quadmathrmsincequad phi>0$$

... and also ... $$beta^2-4=frac1phi^2-4overset(c)=-frac1phi-3<0 quadmathrmsincequad phi>0$$

Taking square roots make us some imaginary numbers, from (3) then, taking real part

$$Reomega=fracalpha2qquad mathrmorqquad Reomega=fracbeta2$$

However, since $beta=-frac1phi<0$ is by $(0)$ and $(3)$ nessesarily and uniquely :

$$omega=fracalpha2+fraci2sqrt4-alpha^2=fracphi2+fraci2sqrt3-phi$$

Ergo

$$cosleft(fracpi5right)=Reomega=fracphi2=frac1+sqrt54$$

and, as a bonus :

$$sinleft(fracpi5right)=Imomega=frac12sqrt3-phi=fracsqrt5-sqrt54$$