Indirect distillation among LLMs
Eric Schmidt, who was the CEO of Google from 2001 to 2011, and an early investor in Anthropic, gave a talk to Stanford in April 2024. The talk became controversial and was taken down. One of the insightful remarks that he made was about how LLMs have already been trained on all the public data, and he wondered how these LLMs will get more of it to improve. ...