Cowell , girlfriend Lauren Silverman and son Eric were at the £35million mansion when the burglars struck.
A source said: “It was clearly a very traumatic incident. Simon is just relieved that his family was safe and all right. Their safety is his number one priority.”
Police rushed to the luxury house in Holland Park, West London, but the burglars had fled.
It is understood Cowell, 56, and his family were asleep at the time of the break-in and that a security guard called the police.
Scotland Yard said: “No arrests have been made but inquiries are ongoing.”
It is not the first time Simon's home has been targeted. In March 2012, stalker Leanne Zaloumis broke in wielding a brick and hid on a shelf inside his giant wardrobe.