"What kind of confusing plot is this?"
Earlier, I saw the instructor from the School of Electrical Engineering came to him in a menacing manner, and I thought this guy was a problem student.
How did it turn into such a profound and professional discussion?
who I am? where am I? What are they talking about?
Why do I know every word, but I can't understand them all together?
Time passed slowly, and before I knew it, Dean Fu's drafts and deductions had filled six or seven pages of A4 paper.
"If you simply use y=f(x)+b to understand the behavior of a layer, then for any positive integer k, there is a neural network with a depth of k^3 and a width of a constant, which cannot be covered by a k-layer Neural network fitting unless its width is 2^k times the original."
The more Dean Fu calculated, the more he felt that this boy's intuition was good.
"If, as you said, batch normalization is used to avoid the problem of covariate shift, and nonlinear functions provide additional expressive capabilities for the layer, then even in extreme cases, 2^ is needed k^3 times the width. It seems that your idea of focusing on solving the depth problem first, and then considering solving the training difficulty problem is a very smart choice."
Dean Fu pondered for a while, reviewed his calculation process, and asked, "Do you understand?"
Meng Fanqi shook his head very seriously and said, "I don't quite understand."
Dean Fu smiled and went through the entire process in order without being anxious or annoyed.
Even though Meng Fanqi had read these three papers several times in his previous life, he had never been able to fundamentally understand the mathematical relationships among them. The AI community and the mathematics community have completely different perspectives on this issue. Dean Fu brought Meng Fanqi new knowledge and understanding.
A senior mathematics professor sorted out the mathematical principles and relationships behind it for him. Meng Fanqi suddenly felt enlightened, but he was always confused, separated by a layer of gauze.
After looking at it carefully for a long time, it seemed to be somewhat clear, "I seem to understand it now."
"Read it a few more times to consolidate it." Dean Fu stood up after hearing this, patted Meng Fanqi on the back, and said, "My office is in Building 503 of the Institute of Science and Technology. If you have any questions in this regard, you are welcome to come to see me. I discuss communication.”
After saying that, he turned around and left without asking what Meng Fanqi's name was, and only had a purely academic exchange.
Before leaving, he glanced at the two graduate students from the School of Mathematics who were craning their necks and looking around. It probably meant, "Look at me, I'm an undergraduate, and I've almost written an article. Look at me, look at you."
The two-digit graduate student quickly lowered his head and looked away.
Meng Fanqi was left alone to savor the complicated argumentation process. After a while, Meng Fanqi felt that he understood it, but he seemed to have forgotten everything, and entered a mysterious and mysterious state.
This must be how Zhang Wuji felt when he learned Zhang Sanfeng's Tai Chi sword.
--------------------------
The friendly guidance from Dean Fu of the School of Mathematics and Physics has made up for the weakest link in Meng Fanqi's current thesis, which is enough solid mathematical analysis and formula deduction.
In the middle and late era of the AI discipline, no truly convincing theory has been found to explain the powerful power of deep neural networks. Therefore, many articles that focus on performance and practical aspects are becoming more and more like experimental reports rather than papers. This has been criticized by many people.
The number of contributors has increased sharply every year. Around 2017, AI-related fields have become floodplains in the eyes of many researchers. The rolling Yangtze River is full of water, and few heroes can be found in the waves.
However, at this point in time, reviewers at many conferences and journals still pay great attention to the theoretical part. If the argument and reasoning in this part are not clear and smooth enough, no matter how good your results are, even if you are one of the three giants of AI. Lecun will also be ruthlessly rejected.
Meng Fanqi received this honor by chance and was not polite at all. In his early days, there were more than a dozen papers that needed to be completed as soon as possible, and many of them troubled him and made him not know how to write.
Nowadays, with such a good teacher, it is natural to ask for advice frequently.
He doesn't need Dean Fu to spend a lot of time reading specific articles. It's just that through the succession and transition of a few key formulas, the tiny details that he never paid attention to when reading in his previous life have opened up that layer of information for him. The seemingly invisible veil allows us to peek into the true mystery.
A few weeks later, around five o'clock on a Friday afternoon, Meng Fanqi just walked out of the dean's office in the Mathematical Sciences Building.
Dean Fu inside did not get up and go to the cafeteria to eat as usual. Instead, he took out all the papers that Meng Fanqi had asked these questions and looked at them carefully.
Although Dean Fu’s expertise in deep learning and image algorithms is not the right one here.
But if the picture content wants to be displayed on the monitor and stored in the computer, it must still be in the form of a matrix. In fact, to put it bluntly, it is just a bunch of rectangular numbers, just like a face of a Rubik's Cube. There are nine squares on it that can store nine numbers. That is a 3 by 3 square matrix. Each number is a pixel, which is the smallest unit of an image.
When it comes to operations and transformations between matrices, Dean Fu is a master in this field.
Although Meng Fanqi never came here with a complete paper, from these scattered questions, Dean Fu was able to get a glimpse of the leopard.
"The first time we discussed was an extremely deep network optimization problem. The residual and batch normalization he proposed should belong to the category of model structure."
"But he later came to me to discuss first-order and second-order gradient calculations, as well as several variations, which should be the content of the parameter optimizer."
"Further, he asked about some error analysis and transmission, which is the content of numerical analysis." Dean Fu frowned and thought carefully, "This should involve different ways of processing the same number in the computer. Storage, storage methods are different, and the computer resources occupied are naturally different. But some errors are bound to be introduced."
"And today, the things he asked were even more imaginative." Dean Fu recalled the prerequisites and background of Meng Fanqi's question today, and marveled at his bold imagination. "With two deep networks, one is responsible for generation and the other is responsible for judgment."
“The generator continuously creates images that it believes are real, and the discriminator is responsible for distinguishing whether the input is real or generated by the generator.”
"This mutual confrontation method can get rid of the human supervision mode where model training always relies on humans to provide correct answers to these data one by one."
Dean Fu murmured to himself that Academician E is actively promoting matters related to big data. As one of the top disciples, he naturally understands the magnitude of today's data, as well as the difficulty and workload of providing answers and labels for them one by one. How big.
"The world is already dominated by young people." Dean Fu let out a sigh of relief. At this moment, there was a knock on the door.
