Special Feature

TechInsights’ AI View

Silicon underpinning AI

Although AI tasks can be performed on general purpose CPUs, they are typically not very efficient. Most of the interest in AI chips has been around GPUs, FPGAs, and to a lesser extent ASICs.

GPUs

For those outside the semiconductor industry, the use of GPUs (Graphics Processing Units) might seem a little confusing.

GPUs were initially designed to handle high end graphics by offering plenty of parallel processing power. CPUs typically have 1-8 cores, where a multi-core architecture might allow some parallel processing. In contrast, GPUs have hundreds or thousands of cores capable of running many tasks simultaneously.

This also makes them ideally suited to machine learning and AI. The term GPU no longer necessarily relates to graphics, but any chip with this parallel architecture. As a result, companies such as Nvidia (profiled later in this issue) have made a successful transition into AI hardware.

ASIC’s (Application Specific Integrated Circuit)

An ASIC (Application Specific Integrated Circuit) differs from CPUs and GPUs in that they are not general-purpose computing hardware but heavily customized to a specific use case. As a result, while they offer advantages in both speed and power consumption, they lack flexibility and typically take a long time to come to market for a novel use case such as AI.

We are now starting to see ASICs for AI appear. For example, at its Infra @Scale event in May this year, Meta announced its MTIA (Meta Training and Inference Accelerator) chip, which will run on its own cloud architecture and accelerate AI functions on its LlaMA 2 generative AI service. Other companies requiring AI at scale, such as Tesla and Google, have also invested in custom AI chips.

As well as ASICs for cloud infrastructure, on-device AI is also being enabled by some smartphone ASICs such as Google’s Tensor processors.

FPGAs (Field Programmable Gate Arrays)

Sitting between GPUs and ASICs are FPGAs (Field Programmable Gate Arrays). FPGAs are integrated circuits with a programmable hardware fabric. Unlike GPUs or ASICs, the circuitry inside an FPGA chip is not hard etched and can be reconfigured as needed. They are therefore often used to prototype new hardware and in the development of an ASIC for a final consumer device.

They can offer the speed and power consumption advantages of ASICs – although typically not to quite the same level. They are available off-the-shelf like GPUs so are faster to market – some work is required to define the hardware before it can be used, but there is much less work required than designing an ASIC.

Another advantage of the flexible FPGA architecture is that it can be configured to receive multiple inputs and outputs, processing them with low latency and memory usage. This is important in some device categories, such as in autonomous driving or advanced AR headsets, where data from a range of sensors is analyzed by AI in near-real time to map the environment and receive inputs from the user through gesture control.

AMD and Intel are the key manufacturers of FPGAs.

Where does the intelligence sit?

AI is used for a range of different tasks and there are a vast array of AI tools. One key difference is where that intelligence sits: on the device itself, or in the cloud.

For some tasks, an on-device solution is best. Autonomous driving, for instance, requires very low latency and the vehicle will not always be connected to a network. On-device AI is the only practical solution.

For generative AI, which generates new text or images, a large data set is required for that AI to “learn” from to generate that content. The Large Language Models (LLMs) used by such systems are vast; OpenAI’s GPT-3 LLM has over 175 billion parameters. Cloud infrastructure is the only practical solution for hosting this type of AI.

The choice for where the AI sits is dependent on the type of AI being used, the use case it is addressing, the desired features of the device, and cost.

Cloud AI

Cloud AI is mainly focused on enterprise use cases. ChatGPT and other generative AI tools are an emerging area for a “pure play” consumer cloud AI use case.

On-device AI

AI implemented directly on the device caters to a diverse array of use cases, intricately shaped by the specific capabilities of the device. This encompasses tasks such as picture editing and real-time language translation on smartphones, voice control and facial recognition within smart home devices, autonomous driving capabilities in vehicles, and even gesture control functionalities within augmented and virtual reality headsets.

Device and Cloud Together

Some applications use both; home automation may have some intelligence on the device but also use a cloud solution for certain features. For autonomous driving, although while in use the vehicle relies on its own AI, it will feed what it has “learned” up to the cloud so that systems are improved with real world data.

Founded in 1993, Nvidia’s rose rapidly to become one of the major fabless players in the chip industry. The company’s focus on GPUs and parallel processing, at its core since the company’s inception, is a long term technology bet that has paid dividends.

Although AI tasks can be performed on general purpose CPUs, they are typically not very efficient. Most of the interest in AI chips has been around GPUs, FPGAs, and to a lesser extent ASICs.

Company Financials

Nvidia’s revenues in fiscal 2023 were flat year on year, the same as the $27.0B in fiscal 2022. So why is everyone excited about Nvidia lately? The flat total revenue picture hides the changing nature of Nvidia’s business. 2022 was the year Nvidia transitioned from a graphics company to an AI pioneer.

Nvidia’s revenues from its graphics segment tumbled 25%. The rapid growth in the previous 2 years could not be sustained. With COVID restrictions people were stuck inside looking at screens and gaming hardware sales rose considerably over this period. The macroeconomic picture has also worsened considerably in the past year. Consumer demand for its graphics cards, previously the majority of Nvidia’s business, has shrunk.

Nvidia’s revenues from data center clients has now overtaken the gaming segment to become the company’s largest revenue driver. Up 41% YoY, the major driver was increased demand from its hyperscaler customers for AI chips and services, along with continued growth in the automotive sector, where AI again plays an important role, both in the efficiency of manufacturing and in the autonomous driving capabilities of the end product.

A Brief History of Nvidia

Nvidia claims to have invented the GPU, but the term GPU existed before Nvidia did. Sony used the term to describe the CXD8514Q chip inside the original PlayStation console.

They can claim to have invented its modern architecture, and they certainly popularised the terminology. In 1999, the Nvidia GeForce 256 launched, propelling the term, and Nvidia, into the public consciousness.

Nvidia’s later wins with Microsoft and Sony in the console wars (see Look Back Through Time below) cemented their position. In the PC space, Nvidia competed in its own “graphics card wars” for the last 20 years, primarily with AMD, for the PC gaming market.

Nvidia’s Key Partners

Nvidia’s strategic alliances and partnerships in the tech industry are both diverse and impactful. While TSMC remains a key foundry partner, recent reports suggest discussions with Intel for escalated production capabilities. Among its relationships, the bond with Microsoft stands out, characterized by a comprehensive collaboration encompassing robust software, services, and hardware integration.

In the automotive sector, Nvidia boasts partnerships with 20 of the top 30 electric vehicle manufacturers. The association with Mercedes Benz not only extends to autonomous driving solutions but also encompasses AI-driven enhancements for factory automation. Despite ceding ground to AMD for the latest generation of Xbox and PlayStation consoles, Nvidia’s pivotal role in supplying GPUs for the leading Nintendo Switch console underscores its enduring influence in the gaming market.

Look Back Through Time

One of Nvidia’s key early wins was to supply Microsoft with a GPU for its entry into the “console wars” – the Xbox. That was an undoubted success, seeing Microsoft establish Itself as a major player competing directly against the market leading PlayStation 2 and precipitating Sega’s withdrawal from games hardware. Nvidia’s GPUs impressed – Sony chose Nvidia as a supplier for its own next generation PlayStation 3 console.

Nvidia’s AI Success

Nvidia’s first significant move into AI came in 2006 with the launch of its CUDA general purpose parallel processing platform and the G80 chip. It launched its Tesla range of data center GPGPUs early the following year. While the Tesla brand is now retired to avoid confusion with the auto company, Nvidia’s product line continues with its most recent offering, the N100 Hopper.

Optimized for AI, the N100 can deliver up to 30x faster performance in AI applications compared to Nvidia’s previous generation chip (A100). With an order of magnitude increase, Nvidia has leapfrogged its competitors. Nvidia also launched a CPU, Grace, designed to pair with this new generation GPU and it is offered to market as the “Grace Hopper Superchip”.

Demand has outstripped supply for these offerings, which are driving Nvidia’s Data Center segment revenues up to record highs every quarter. Hyperscalers including AWS, Microsoft, Google and Oracle are all using this new technology. Nvidia’s competitors look on as green as the company’s logo with envy as the company climbs to the top of the AI tree. How long can it’s success last?

Nvidia Closer Observations

TechInsights’ Platform includes over 1,000 reports and insights on Nvidia through the years. Here are some recent key reports to help you gain a greater knowledge of the company and the reasons for its success.

Total Connected Consumer Electronics Devices

We love our gadgets!

TechInsights tracks volumes and wholesale revenues across a broad range of mobile and home consumer electronics, mapping consumer demand ultimately for silicon itself. Anticipating a noteworthy resurgence, the Consumer Electronics market is poised to reclaim growth this year, rebounding from a 2.9% decline in wholesale revenues experienced in 2022 and projected to achieve a 3.1% growth rate in 2023.

$1 Trillion

revenues in 2024

17 Billion

CE devices in use today

Connected CE Devices Market Forecast: Global 2018-2028 (Report)

Artificial Intelligence

• Generative AI starting to compete with humans in creativity
• Driving demand for AI chips, notably GPUs for datacenters
• Nvidia Hopper H100 flagship is capturing attention
  • H800 – lower specification GPU can be exported into China
  • Orders spike as Chinese internet giants worry sales will be ended by trade restrictions

Geopolitics

• Technology trade war between China and Western economies continues
• “Tit for tat” with China restricting Ga/Ge exports
• “Chip Acts” in US, EU beginning to bear fruit, multiple new foundries announced

Smaller Better Faster

• Samsung first to market with 3nm GAA chip
• China demonstrating technical prowess despite US restrictions
  • H800 – lower specification GPU can be exported into China
  • Orders spike as Chinese internet giants worry sales will be ended by trade restrictions

Battery Markets Charging Up

• Samsung rumored to move to stacked battery technology in flagship Galaxy smartphones
• Toyota going into production with solid state batteries for electric vehicles

Macroeconomy

• Fears of a deep Global recession ease, although picture is mixed
• Global Consumer Electronics market is set to return to growth this year and expected to surpass $1 trillion for the first time in 2024

Geopolitics

Geopolitics is orchestrating a transformative shift in the global technological arena, as the collaborative efforts between the EU and USA materialize in the form of “chip acts.” This strategic policy is proving fruitful, as industry giants such as Intel, TSMC, and other players seize the opportunity, announcing new foundries while capitalizing on the incentives provided. This orchestrated move is strategically designed to cultivate supply chain flexibility, a calculated approach that seeks to bolster resilience rather than create over-capacity. Meanwhile, Chinese electronics and infrastructure vendors are encountering a dwindling foothold in Western markets, spurring China’s strategic recalibration. Emphasizing self-reliance, China is progressively integrating local components into its manufacturing landscape, while adopting a “tit for tat” stance through reciprocal trade restrictions. Notably, the recent imposition of export restrictions on Gallium (Ga) and Germanium (Ge) underlines China’s evolving tactics. In this intricate geopolitical ballet, the tech industry is a stage where economic strategies and global power dynamics entwine, reshaping the contours of the technological landscape.

Smaller Better Faster

Samsung has taken the tech industry by surprise, emerging as the pioneering force in introducing a groundbreaking 3nm Gate-All-Around (GAA) chip to the market. This significant achievement was initially unveiled by TechInsights, within the Whatsminer M56S++ crypto-mining ASIC. Meanwhile, China persists in showcasing remarkable technical prowess, undeterred by the stringent US restrictions on technology and tools. In a commendable stride, Semiconductor Manufacturing International Corporation (SMIC) is reported to have commenced mass production of 7nm chips, attesting to China’s impressive advancement. Notably, as part of its ongoing progress, China is embracing a shift towards the innovative “chiplet” architecture. Amidst a dynamic technological landscape, these developments underscore the intricate interplay between innovation, geopolitical dynamics, and industrial strategy.

Battery Markets Charging Up

The realm of battery technology is undergoing a fascinating evolution, driven by remarkable innovations from industry leaders. Samsung’s rumored shift to stacked battery technology for its flagship Galaxy smartphones holds immense promise, with layers designed to enhance power density and extend battery lifespan through efficient heat distribution. In parallel, Toyota has ushered in a new era with a groundbreaking announcement regarding solid-state batteries for electric vehicles. The automaker’s achievement includes a substantial reduction in size, weight, and cost, while the projected attributes of a 1,200km range and a mere 10-minute charging time promise to reshape the landscape of electric mobility. Anticipated to hit the roads in 2027, these advancements underscore a future where batteries play a pivotal role in driving sustainable and efficient transportation systems, setting the stage for a transformative era of mobility.

Macroeconomy

The macroeconomic landscape is displaying a complex array of signals as concerns over a profound global recession begin to recede, although the overall outlook remains nuanced. Economists in the US are cautiously optimistic, predicting a “soft landing” with minimal or no recession in sight. Conversely, the Eurozone portrays a more varied picture, with economies experiencing a cooling effect driven by high inflation and escalating costs of living. China’s economic trajectory has faced challenges, yet a growth rate close to the government’s target of 5% for 2023 remains within reach. Against this backdrop, TechInsights’ projections offer a glimmer of hope, forecasting a resurgence in the Global Consumer Electronics market. This industry is poised to rejuvenate this year and is anticipated to cross the historic threshold of $1 trillion for the first time in 2024, a milestone that holds significant implications for global economic dynamics. As we navigate these intricate economic currents, the Data Observatory provides a comprehensive overview of the pivotal trends shaping this dynamic market.

Is Artificial Intelligence a powerful new tool, or an existential threat to humanity?

The answer to that question depends on whom you ask.

Our own human intelligence, hosted in the 1.4kg of fatty tissue in our cranium rather than a few grams of silicon, has provided us with both useful tools and the means of our own destruction. As I write this, the feature film Oppenheimer is playing at my local movie theater, a powerful drama of a time when our own intelligence overstepped a line that we might wish to draw for AI. Human intelligence is neither a force entirely for good nor evil; should we expect an artificial intelligence to be different?

We have met a tipping point when AI is beginning to compete with human intelligence and indeed even replace humans in some specific tasks. At the London premiere of Oppenheimer, the stars walked out, part of a writers’ and actors’ strike. One of the demands is a framework for compensation when AI is used as a substitute for a real actor’s performance or a human writer’s script. As a species we must wrestle with the question of the work which AI will do for us – and therefore how far it might go to “replace” humans.

The study of AI has moved from a philosophical thought exercise into a pressing need for answers. TechInsights data on AI chips to devices to its use in the consumer and enterprise sectors can provide an unique insight into this space.

Generative AI

Generative AI has been a particularly hot topic in the press.

Tools such as Chat GPT (Open AI & Microsoft), LLAMA 2 (Meta) and Bard (Google) are focused on generating human-like text outputs to prompts and questions. They can even extend to writing computer code. Other tools such as DALL-E (Open AI) and Midjourney (Midjourney Labs) create vivid artworks and virtual “photographs”.

It is these tools which have captured the imagination (and fears) of the broader public. They have changed the nature of discourse around AI as it moves from a set of largely technical use cases into competing with human creativity.

Curated highlights of the top reports published on the TechInsights platform in the past month.

In the current edition, our focus turns to two pivotal trends reshaping the technological landscape. We spotlight the intriguing development of a decelerating decline in smartphone sales, shedding light on the emerging dynamics of winners and losers within the market as it embarks on a journey of resurgence. Meanwhile, the automotive sector takes center stage with a profound shift towards subscriptions prevailing over ownership, heralding a trend that possesses the potential to fundamentally reshape the intricate bond between society and the automobile. Join us as we delve into these transformative narratives, offering insights into the shifting tides that define our digital age.

Q2 ’23: Apple: China’s iPhone Shipments Surpassed US’ For The First Time

Apple’s iPhone shipments declined 9%, but it registered its highest ever Q2 ASP. Tepid demand in North America, Western Europe and Japan is largely attributable to Apple’s mixed performance. Lacklustre performance elsewhere meant China surpassed US being the largest single market for iPhone for the first time ever.

Vehicle Subscriptions Services Must Take 3 Critical Steps for Shared Model Transition

The automotive business model is shifting from ownership to shared services. Driving this trend are the spike in vehicle prices, rise of remote work, and emergence of electrification and “software-defined” cars. This report details the automotive industry’s progress toward this vision, the hurdles to adoption, and opportunities for car makers, service providers, and other market sectors.

Transsion Nudged Up the Fifth Largest Smartphone Vendor in Q2 2023

Global smartphone shipments dropped 8% YoY to 269 million units, marking the eighth consecutive quarter of decline. The rate of decrease has slowed from double digits to -8%, hinting at a potential mild market rebound in the second half of 2023 due to normalized inventory, upcoming holidays, new product launches, and Transsion’s entry into the top five list for the first time.