The fifth brother was the first to build with the stone