The Alternative fuer Deutschland (AfD) party took 21% of the vote behind the centre-left SPD's 30.5%.
The German chancellor's CDU was supported by only 19% of those who voted, according to the exit polls.
The vote was seen as a key test before German parliamentary elections in 2017.
Before the vote in Mecklenburg-West Pomerania, all of Germany's other parties ruled out forming a governing coalition with the AfD.
However, the party's strong showing could weaken Mrs Merkel ahead of the national elections next year.
Mecklenburg-West Pomerania, in the former East Germany, is where the chancellor's own constituency is located.
Under her leadership, Germany has been taking in large numbers of refugees and migrants - 1.1 million last year - and anti-immigrant feeling has increased.
The AfD, initially an anti-euro party, has become the party of choice for voters dismayed by Mrs Merkel's policy.
Read the whole article here