The benchmark Nikkei .N225 shed as much as 3.4 percent to 14,121.05 before steadying to end the morning session 2.62 percent down at 14,236.59.
After its stellar gains of 57 percent in 2013, the Nikkei has dropped 12.6 percent since the start of this year to become one of the worst performing markets.
"What is of concern is that there appears to be a substantial overhang of foreign ownership in the Japanese stock market," said Stefan Worrall, director of equity cash sales at Credit Suisse in Tokyo.
In a major bearish signal for chartists, the Nikkei fell decisively below its 200-day moving average, around 14,425, for the first time since November 2012 when the market started rallying on expectations of radical monetary easing.
"Many market players are trying to determine whether the recent declines are just a short-term correction or a change of tide in the market," said Kenichi Hirano, a strategist at Tachibana Securities.
But global investor sentiment was rocked by U.S. data showing manufacturing activity slowed sharply in January on the back of the biggest drop in new orders in 33 years, while construction spending barely rose.
On Wall Street, the S&P 500 suffered its worst drop since June.
On the Nikkei, index heavyweight SoftBank Corp plunged as much as 5.8 percent at one point, adding to its 6.6 percent decline in the previous session, before bargain-hunting by retail investors forced a sharp rebound.
The stock was the most traded stock on the main board and last traded up 2.6 percent on the day, though it was still down more than 20 percent so far this year.
Other names with high foreign ownership also took a beating. Mazda Motor Corp fell 5.4 percent.
Currency-sensitive exporters came under pressure as the yen rose to 2 1/2-month high on safe-haven buying.
Toyota Motor Corp, which will announce its quarterly earnings results after the market closes, dropped 4.5 percent and was the third-most traded stock by turnover on the Topix.
Aside from Toyota, several major companies, including Panasonic Corp, Sharp and Hitachi Ltd are set to announce quarterly results later in the day. Nippon Paint Co Ltd nosedived as much as 14.9 percent on dilution concerns after the paintmaker said it would issue new shares.
The broader Topix index was down 3.1 percent at 1,159.29 in heavy trade, with volume at 85.1 percent of the fully daily average for the past 90 trading days.
The JPX-Nikkei Index 400, an index launched this year comprised of firms with high return on equity and strong corporate governance shed 3 percent to 10,509.44.