Lord Walney is not the most popular member of the upper chamber, at least among Labour MPs and ministers.