Samuel Chukwueze hit the opener as he controlled Alex Iwobi's cross, before scoring at the second attempt.
Bongani Zungu equalised with a header initially ruled out for offside before VAR replays showed a free-kick hit a Nigerian player on the way to him.
But Udinese's Troost-Ekong bundled home from close range late on to win it.
His finish was into an empty net after goalkeeper Ronwen Williams totally missed Moses Simon's corner.
Nigeria face Ivory Coast or Algeria in Sunday's second semi-final (20:00 BST).
The Super Eagles were deserved winners and led through Villarreal youngster Chukwueze's first goal for his country. They should have been ahead by more when Williams pushed Peter Etebo's free-kick onto the bar.
Zungu's goal was the first in the Africa Cup of Nations given by the VAR, which has just been introduced for the quarter-finals. It looked offside initially but Percy Tau's free-kick hit Nigeria's Odion Ighalo on the way on to Amiens midfielder Zungu's head.
The game looked set for extra time before Williams flapped at a cross to leave Troost-Ekong with a simple finish.
Senegal, who beat Benin 1-0 earlier on Wednesday, play either Madagascar or Tunisia in the other semi-final.
All three goals were quite scrappy. The first one for Chukwueze took a deflection.
Then South Africa's equaliser thanks to the VAR decision, which was a bit of history. Everyone in the stadium thought it was offside - that was what the linesman flagged for - but the replay proved VAR was useful, correct and worth doing because it proved the ball had come off Ighalo and not Thulani Hlatshwayo, and so it stood.
But South Africa didn't do enough to win this - they were caught between trying to pump it long and trying to pass it around a very solid Nigerian defence.
And it was Troost-Ekong who was at the heart of that defence and scored the winner. Nigeria had class through the side though - Iwobi had another outstanding game, as did Etebo and Chukwueze.