Are there any limits to large neural networks?

OpenAI will release GPT-4 in the next few years. It will have around 100 trillion parameters.