computer science - Closed form of a nonlinear recurrence sequence.

As has already been explained, there is no hope in general of finding explicit solutions to nonlinear recurrences. However, for your example, it is possible to find $lim_{ntoinfty}f_n(X)$ for all real $X$.

The function $g(x)=(x^2+x)/2$ has two fixed points: $x=0$ (atractor) and $x=1$ (repulsor). Its respective stable sets are $(-2,1)$ and ${-2,1}$; $(-infty,-2)cup(1,+infty)$ is the stable set of $+infty$. Thus,

$$lim_{ntoinfty}f_n(X)=left{matrix{0, & Xin(-2,1)cr 1, & Xin{-2,1}cr +infty, & Xin(-infty,-2)cup(1,+infty)}right.$$