Could some of the growth in book length be due to technology. Manuscripts were once delivered to the publisher in ink on paper format. Actual human beings had to convert it into a printable form, probably several times. This makes the act of including something in the final product a result of at least one conscious choice. I think these days works are submitted electronically so each exclusion requires a conscious choice.