### Abstract

The present contribution concerns the study of pattern formation in the Marangoni convection in a binary liquid layer with a nondeformable interface. The long-wave instability was found by A. Oron and A.A. Nepomnyashchy (Phys. Rev E, 2004). It was shown that both monotonic and oscillatory modes are possible depending on the Soret number. In particular, the set of equations describing the 2D oscillatory convection was obtained. The bifurcation analysis shows that the traveling waves are the selected kind of flow in 2D case; the standing waves are unstable. The purpose of the present study is further investigation of the oscillatory mode.

The numerical study of the 2D regime is performed by Galerkin method. It is found that after some relaxation process the solution evolves to the monochromatic traveling wave; all subharmonics decay. Moreover, it is shown, that with the increase of the Marangoni number the traveling wave with any wave number becomes stable.

The set of amplitude equations for description 3D thermocapillary flows is obtained. The bifurcation analysis performed within the framework of this set reveals that the plane traveling wave is unstable with respect to the 3D perturbation.

A special case of perturbations is found: a superposition of the traveling square pattern and the plane traveling wave propagating along the diagonal of the square. This mode is described by a dynamic system of the fourth order. Two qualitatively different nonlinear regimes are obtained: one is close to square pattern and another - to the plane traveling wave. The stability of these regimes is studied. It is shown, that the pattern close to the square is stable, while the pattern close to the plane wave is unstable.

The numerical simulations within the framework of the set of amplitude equations are also carried out to study 3D thermocapillary convection.