Why has everyone set their sights on optical chips?

The booming development of artificial intelligence infrastructure in Silicon Valley has brought about a surprising physical problem: the amount of data that can be transmitted through copper wires is limited, and beyond a certain limit, heat, distance, and power consumption become difficult to bear. This is why investors, chip manufacturers, and cloud giants have suddenly begun to pay close attention to photonics, using light instead of electrical signals to transmit data between artificial intelligence chips and servers. I recently visited the headquarters of Lightmatter in Silicon Valley, where this startup company showcased its latest photonics hardware for artificial intelligence data centers. After the event, I spoke with Lightmatter CEO Nick Harris to discuss why optical technology may become indispensable infrastructure for the era of artificial intelligence. Harris looks annoyingly young and annoyingly smart, with a PhD from MIT. Lightmatter has also achieved annoying success, raising $850 million from major investors including Google, Fidelity, and T. Rowe Price. On Tuesday, Lightmatter joined NVIDIA's NVLink Fusion ecosystem, which should help the startup's technology better collaborate with NVIDIA's dominant AI hardware. The following is my edited conversation with Harris for clarity and conciseness. Q: Why are artificial intelligence companies suddenly interested in photonics? Harris explained that the artificial intelligence industry has reached a stage where improving performance no longer means increasing the speed of individual chips, but effectively connecting a large number of GPUs together. Today's artificial intelligence systems heavily rely on copper wire connections between GPUs. This method works well in small-scale applications. However, as enterprises connect hundreds or thousands of GPUs to build cutting-edge artificial intelligence models, copper wire connections become a bottleneck because electrical signals decay over relatively short distances and generate a lot of heat. Photonics use light to transmit data in optical fibers. This allows for longer data transmission distances, faster speeds, and lower energy consumption. "Assume you have 500 GPUs, and there are copper wires connecting them so they can communicate in the so-called vertical scaling domain. People run model training workloads on these systems." Harris told me that if you use copper cables, you need four independent GPU server racks to achieve 500 GPU servers, but "if you switch to optical fiber, you can directly connect all 500 GPU servers. This way, the time it takes to train AI models will be significantly reduced. Think about cutting-edge models like Claude, the speed has tripled." "Whoever masters this technology first will bring new models to market faster in this frontier competition. They have two choices. One is to launch a new model every month, and the other is to take three months to launch a larger-scale model," he added. "Assuming my power supply is limited. The same power, I can get triple the performance. So, 1 gigawatt of power feels like 3 gigawatts. Or, with the same power, you only need one-third of the time to finish," Harris explained. Q: What are the problems with copper cable connections inside artificial intelligence data centers? Harris said, "Copper wires can only transmit about one meter," because electrical signals rapidly decay when transmitted in copper cables. "The signal will create an electrical signal in the conductor, but as the transmission distance increases, the signal strength weakens. After about one meter, the data is lost." This physical limitation brings another problem: heat dissipation. Since the transmission distance of copper cables is limited, GPU server racks in artificial intelligence data centers are now closely packed together. "They are crammed together," Harris said. "The issue is that I need them all stacked together so that the copper can touch. But the downside is that heat dissipation is very challenging." Photonics changes this because light signals can propagate further and faster without decay, allowing GPU servers and racks to be spaced further apart. "Optics don't care about the distance between objects," Harris said. "They may be a kilometer apart." As a result, data center operators have greater flexibility in designing and cooling AI clusters, potentially saving more on power costs for cooling these systems. Q: What is BiDi (Bi-Directional Communication), and why is it important? One more practical innovation that Lightmatter is researching is reducing the amount of wiring required inside artificial intelligence data centers. Harris explained that some next-generation artificial intelligence clusters require about 300 miles of cables. Lightmatter aims to halve the cable length with a technology called BiDi (short for Bi-Directional Communication). "Typically, whether using copper cables or optical fibers, if I want to establish a connection between two GPUs, I need two cables," Harris explained. "One is for sending, the other is for receiving." Lightmatter's solution is to merge the cables for the two directions into one cable. For super-sized data centers, reducing the length of optical fibers is crucial because optical cables take up space, generate heat, increase maintenance difficulty, and raise costs. Lightmatter points out that reducing the total demand for optical fibers from 300 miles to 150 miles can significantly simplify the construction of large-scale artificial intelligence clusters. Q: Why hasn't photonics been applied earlier? Harris said the main issue was cost. "Photonics is too expensive," he said. Harris explained that this is changing as manufacturing technology improves, and the demand for artificial intelligence infrastructure is growing exponentially. "People designing these systems are looking for a reliable 2x performance increase. Bandwidth doubling, performance doubling, and they strive to achieve this goal regularly. In the past, copper cables could extract some performance improvements, but now that opportunity is gone. Furthermore, there is a stronger driving force, which is the realization that companies that are ahead in adopting and deploying photonics technologyNVIDIA is likely one of themhave a huge performance advantage." "It used to be 'change out of necessity,' now it's 'change for a competitive advantage,'" Harris said. This article is reproduced from the WeChat public account "Observations on the Semiconductor Industry", edited by Xu Wenqiang.