A fine Keisuke Honda strike after 16 minutes had put Japan in front at the Pernambuco Arena in Recife, but the Ivorians introduced Didier Drogba from the bench just after the hour mark before going on to turn the game on its head.
Within five minutes of the veteran striker's arrival on the rain-soaked surface, the Elephants scored twice, both times from Serge Aurier crosses.
The equaliser came when the right-back's curling delivery was headed home by Wilfried Bony, before Gervinho connected with another Aurier centre to stun the Japanese contingent inside the stadium.
The result sees the Ivory Coast join Colombia -- who beat Greece 3-0 earlier in the day -- on three points atop Group C, and Sabri Lamouchi's side must face the South Americans next in Brasilia on Thursday.
"We didn't start the match very well and we switched off at the goal but then we started to create chances. For us it was so important to start this World Cup positively," said Lamouchi.
"Having a player like Didier Drogba in your side, whether he is starting or on the bench, changes a match. The players are delighted with the victory, but we are not through yet."
Meanwhile, Japan face Greece in Natal next knowing that they cannot afford another setback.
"The Ivory Coast stepped up the intensity as the game went on. Physically they are so strong, more than we are capable of being," admitted Japan coach Alberto Zaccheroni.
"They are stronger than us in the penalty box so we needed to prevent the crosses from coming in. I thought we could win the game but now our backs are against the wall."
The Elephants are desperate to finally progress beyond the group stage at a World Cup after failing in 2006 and 2010, and they were boosted in Recife by the presence in their starting line-up of Yaya Toure, who shrugged off a hamstring problem in time to captain the side.
Drogba was on the bench at kick-off, with Swansea City's Bony preferred in attack, but the Ivorians still maintained a considerable physical advantage over the Asian champions.
However, what Japan lack in muscle they make up for in technical ability, and Honda displayed his class to put the Blue Samurai ahead.
Following a throw-in from the left flank, the ball arrived at the feet of AC Milan playmaker Honda at the edge of the box, and he took a touch before letting fly with a powerful, rising left-foot strike that gave Boubacar Barry Copa in goal no chance.
Honda also scored Japan's first goal at the 2010 World Cup, in a 1-0 victory against Cameroon in South Africa, before the Asian giants eventually bowed out in the last 16.
This time, his goal gave Japan a platform which right-back Atsuto Uchida almost built on, as he broke into the box and turned inside Serey Die only to see his shot beaten away by the goalkeeper.
Drogba unsettles Japan
As the rain continued to fall, the Ivorians gradually came into the contest with Yaya Toure and Arthur Boka threatening from free-kicks and the latter seeing a fizzing long-range strike parried away by Eiji Kawashima in the Japanese goal.
Lamouchi opted to throw Drogba into the fray in the 62nd minute, withdrawing a defensive midfielder in Serey Die, and his presence unsettled the Japanese defence enough for them to concede two quick goals.
Aurier, of French club Toulouse, was the source of both goals with two identical balls into the box from the right.
For the equaliser, Bony connected with his head to send the ball flying past Kawashima and into the net at the far post.
Two minutes later, Gervinho was left unmarked to head home at the near post, and Japan never really looked like coming back into the contest after that.
Indeed, the Ivorians could have added to their lead, with Drogba twice almost getting on the scoresheet, including from a 30-yard free-kick that Kawashima turned behind.