Ministers have agreed to consider whether it should be a criminal offence for powerful people like the Sports Direct founder, Mike Ashley, to refuse to give evidence when called by MPs.

Chris Bryant, the shadow leader of the House of Commons, said the system was not working properly when some of those called could refuse to take part in inquiries by taking advantage of unclear rules governing the select committees.

He cited the cases of businessman Robert Maxwell, media executive Rebekah Brooks, Rupert and James Murdoch, former BHS owner Sir Philip Green, and Vote Leave chief executive Matthew Elliott, who all initially refused to attend and had to be formally summonsed or persuaded to do so.

Speaking in the House of Commons, Bryant said: “Irene Rosenfeld, chief executive of Kraft Foods, point-blank refused to appear to discuss the takeover of Cadbury and got away with it ... Some people think our powers are unclear and witnesses are beginning to call our bluff, so we have to do something.

“In 2013, the joint committee on parliamentary privilege recommended changes to standing orders to make it absolutely clear that parliament can arrest, punish and fine offenders and said that ‘if the problems we have identified ... are not resolved ... today’s parliament should stand ready to legislate’.

“The committee said doing nothing was not an option but that is exactly what the government has done, absolutely nothing.

“So surely it is time for us to make it a criminal offence to fail to appear or refuse to appear, without reasonable excuse, before a committee of this house.”

Chris Grayling, the leader of the Commons, said he was open to discussing the issue, as it was essential to parliament for witnesses like Ashley to appear before select committees when they are summonsed.

“On this point you and I do agree,” he said. “It is essential for the workings of this house that if people are summoned to appear before a select committee, that they do so.

“And I am very happy that in the new session we have cross-party discussions on how we ensure that that happens.”

It comes after a battle between MPs on the Commons business committee and Ashley, who has refused to appear before their inquiry investigating pay and working conditions at Sports Direct.

The billionaire called the parliamentarians “a joke” after the Commons’ business, innovation and skills (BIS) committee took the unusual step of issuing a summons to the Newcastle United owner.

The inquiry by the committee, which has threatened Ashley with being in contempt of parliament if he fails to attend a hearing on 7 June, follows a Guardian investigation last year that found workers at the sportswear group’s Shirebrook warehouse were receiving, in effect, rates of pay below the minimum wage.