Linear Model
fθ(x)=∑j=1bθjϕj(x)=θTϕ(x)
where ϕ(x) is the basis function vector and θ is the parameter vector. As you can see, there are b basis functions.
Basis functions can be of different form, such as polynomial:
ϕ(x)=(1,x,x2,⋯,xb−1)T
or triangular polynomial:
ϕ(x)=(1,sinx,cosx,sin2x,cos2x,⋯,sinmx,cosmx)T
x may be a vector rather than a scalar. However, linear models have nothing to do with the training set.
Multiplicative Model
Taking the one-dim basis functions as factors in order to get multi-dim basis functions.
fθ(x)=∑j1=1b′⋯∑jd=1b′θj1,⋯jdϕj1(x(1))⋯ϕjd(x(d))
Obvious curse of dimension.
Additive Model
fθ(x)=∑k=1d∑j=1b′θk,jϕj(x(k))
Kernal Model
Kernal functions are binary, relate to the training set. Kernal models are defined as linear combinations of kernal functions.
fθ(x)=∑j=1nθjK(x,xj)
There are many types of kernal functions, however, Gaussian kernal functions enjoy the most popularity.
K(x,c)=exp(−∥x−c∥22h2)
As you can see, kernal models assign kernals with respect to the training sample xi and then learn their height θi. Therefore, they can approximate functions only in the neighborhood of training samples regardless of the dimension of xi.
Hierarchy Model
Hierarchy models belong to nonlinear models.
fθ(x)=∑j=1bαjϕ(x,βj)
Hence, θ=(αT,βT1,⋯,βTb)T. ϕ(x,βj) are basis functions. There are 2 typical basis functions:
S function (or Artificial Neural Network)
ϕ(x,β)=11+exp(−xTω−γ),β=(ωT,γ)T
Gaussian function
ϕ(x,β)=exp(−∥x−c∥22h2),β=(cT,h)T
Note that θ and fθ are not one to one corresponded.