Pirates starter Jeff Locke looked strong through 1 2/3 innings, but surrendered hits to seven of the last 13 men he faced. Included in that stretch was L.J. Hoes' run-scoring single and a long home run from Lew Ford, his second of the spring.
Baltimore made it 3-0 in the fifth on Danny Valencia's RBI single off Phil Irwin.
After Luis Ayala pitched a hitless fifth, Zach Britton, who like Arrieta is fighting for a rotation spot, allowed a run in two innings with two strikeouts.
Britton lost the shutout with two outs in the seventh on Josh Harrison's RBI double, the only hit he allowed.
Brad Hawpe homered to right off Orioles reliever Daniel Schlereth in the eighth to make it 3-2.
Harrison's second run-scoring hit tied the game in the bottom of ninth, and then Alex Presley, who singled with one out, scored on Paulino's grounder to second. Even with the Orioles' infield playing in, Presley beat Ryan Flaherty's throw home to win it.
Jared Goedert singled off losing pitcher Adam Russell to lead off the inning and was forced out on Clint Robinson's grounder to second. Jordy Mercer ran for Robinson, and took third on Presley's single before scoring on Harrison's single.
Conor Jackson, Valencia, Ford and Hoes each had two hits for Baltimore.
Kris Johnson pitched a scoreless top of the ninth to earn the win for Pittsburgh.
Up next: The Orioles are technically off Tuesday, but will play an intrasquad game in the back of the Ed Smith Stadium complex at 10:30 a.m. ET. Jason Hammel will start the game, while Jair Jurrjens, Steve Johnson, Mike Wright, Eduardo Rodriguez, Troy Patton and Daniel Schlereth are also scheduled to pitch. Nolan Reimold, Wilson Betemit and Taylor Teagarden will be in the lineup -- along with some of the organization's Minor Leaguers -- to get in some extra at-bats. Brian Matusz will start Wednesday's 7 p.m. ET game against the Twins in Fort Myers. The game be heard live on Gameday Audio.