There are two shape of wave guide , rectangular and cylindrical where to use each one ?

http://www.rfwireless-world.com/Terminology/Rectangular-waveguide-vs-Circular-waveguide.html

cylindrical wave guides and Elliptical wave guides are much easier to pend and  install so it is being used commonly for harder to reach places, rectangular wave guides are less losses but harder to twist or install.

Rectangular wave guides are used with Microwave dishes using vertical or Horizontal polarization, where as circular wave guides are used with radars.

.The parameters of rectangular waveguide and circular waveguide has been verified with the simulated results in Matlab,this paper use to analyze the parameters of waveguides like as (Cut off frequency, Cut off wavelength,Guide wavelength,Phase velocity, Group velocity, Impedance, power, Attenuation, VSWR) and compare their results with the help of output figures so it is easy to analyze from the above results that rectangular waveguide is much better than the circular waveguide.These parameters of Rectangular waveguide showing that the guided wavelength is high,less power required,cutoff wavelength is high but attenuation is more in comparision of circular waveguide.So with the help of these parameters it is easy to select the best waveguide for the applications like in communications field and in military purposes.

Circular waveguide is basically seems like the circular structure where it consist tubular circular conductor.As the frequency of transmitted signal changes the inner diameter of circular waveguide also changes.There are so many uses of Circular waveguides like in communication systems used in specific areas of radar and uses as rotating joints of the mechanical point of the antennas rotation.Generally the values of radius is very important for manufacturing of circular waveguides. 

A rectangular waveguide supports TM and TE modes but not TEM modes because it cannot define a unique voltage since there is only one conductor in a rectangular waveguide. The shape of a rectangular waveguide based on their dimensions. A material with permittivity (e )and permeability (m) fills the inside of the conductor.The mode in which the magnetic field is always transverse in the direction of propagation is called the Transverse magnetic (TM) mode,In Tm mode Hz =0 & Ez ≠ 0.The mode in which the electric field is always transverse to the direction of propagation is called the Transverse electric mode.Thus in TE mode EZ =0 & HZ ≠ 0.