Top Donald Trump spokesman Jason Miller abruptly announced Saturday that he will not take the job of White House communications director – two days after the transition team said he was joining the Trump administration in that role.

Miller, a longtime spokesman for Trump’s campaign and later the transition operation, had been named communications director and assistant to the president as part of the senior White House communications team.

But he said Saturday he wants to spend more time with his family, and is passing up the job.

“After spending this past week with my family, the most amount of time I have been able to spend with them since March 2015, it is clear they need to be my top priority right now and this is not the right time to start a new job as demanding as White House Communications Director,” he said in a statement. “My wife and I are also excited about the arrival of our second daughter in January, and I need to put them in front of my career. I look forward to continuing to support the President-elect from the outside after my work on the Transition concludes.”

He said his planned responsibilities will be “handed off” to Sean Spicer, the incoming White House press secretary who “will also serve as Communications Director moving forward.”

As previously announced, Trump aides Hope Hicks and Dan Scavino also are joining the White House team, as director of strategic communications and director of social media, respectively.

Fox News’ Christopher Snyder and Jenny Buchholz and contributed to this report.