The aim of this study was to optimize the nanoencapsulation of fish Omega 3 fatty acids with use of complex coasevation technique which was performed by using response surface methodology (RSM) in the central composite design (CCD). Nanoparticles containing omega-3 fatty acid were produced by Rainbow trout skin gelatin and Arabic gum as walls using complex coacervation technique. The effects of three independent variables as fish gelatin concentration (33-66%), the concentration of omega-3 fatty acids (10-30%) and the homogenizer speed at three levels (10000, 15000 and 20000 rpm) on the dependent variables such as particle size, oil level, encapsulated oil and nanoencapsulation efficiency were evaluated. The results showed that the homogenizer speed significantly had the greatest impact on the particle size (P<0.01). Determination of the nanoparticle surface oil indicated that surface oil was higher in the treatments with higher levels of omega-3 fatty acid than the treatments with lower concentration of omega-3 fatty acid. Nanoencapsulation efficiency of produced powder was recorded at about 41.98-74.69 %.  Therefore, it is recommended to use 33% fish gelatin with 30% omega-3 fatty acids at 20000 rpm for production of omega-3 fatty acids nanocapsules with desired properties.