Spain now face title contenders and Pool A toppers Australia in the last-eight stage on Tuesday.
After the two sides were goalless at half time, Faizal Saari put Malaysia ahead in the 35th minute but Marc Miralles (41st) and Xavier Gispert (42nd) scored in quick succession to put Spain 2-1 up.
Shello Silverius then scored a fine field goal to make the scoreline 2-2.
In the penalty shootout, it was 3-3 after the first set of five players took their shots. Firhan Ashari, Faizal Saari and Suhaimi Shahmie Irfan scored for Malaysia, while Marhan Jalil and Shello Silverius missed. Marc Miralles, Bonastre Jordi and Gispert Xavier scored for Spain while Alvaro Iglesias and Marc Reyne failed to score.
As the two teams were tied 3-3, sudden death was applied.
Spain’s Marc Miralles scored while Firhan Ashari missed.
Spain, who had finished third in Pool D behind England and India, dominated the match with a lot of circle penetration but the speedy Malaysians were impressive in the counter-attack.
Spain got as many as eight penalty corners but wasted all of them. Malaysia earned one PC which they could not convert.
Malaysia were able to put the ball in the Spanish goal in the first minute of the fourth quarter but the goal was disallowed with the umpire ruling that the ball had touched the back stick of Faizal Saari.