Classical approaches to solution of the FG matrix
The force constants can be treated as parameters and used to fit vibrational spectra. However, this problem has an inherent limitation due to the large number of force constants. In the general case the F matrix has dimension (3N - 6)2, however since it is a symmetric matrix the number of independent parameters is (3N-6)(3N-5)/2. This is still a relatively large number of parameters and is larger than the number of fundamental frequencies for all but the smallest of molecules. Isotopic molecules can provide additional data with the assumption that the potential function is not changed by isotopic substitution.
In Molecular Vibrations by Wilson, Decius, and Cross Sections 8-3 and 8-4 are devoted to the estimation of the force constants. The valence force field approach can give the diagonal force constants (bond stretching, valence angle bending, torsions etc.) but not the off-diagonal interaction force constants (e.g. stretch-bend interaction, stretch-stretch interaction etc.). Ultimately, the approach taken by Wilson, Decius and Cross is to hope that the interaction force constants are small.
The use of group theory can greatly simplify the GF matrix. This is illustrated in Molecular Vibrations and in Cotton's "Chemical Applications of Group Theory".