Below you’ll find the privately published (i.e. appropriate only for viewing by Alliance Member company representatives) videos from the Alliance’s September 2023 Edge AI and Vision Innovation Forum. They are: The presentation “Welcome Remarks” from Pulin Desai of Cadence The presentation “Opening Remarks” from Jeff Bier of the Edge AI…

September 2023 Edge AI and Vision Innovation Forum Presentation Videos

Register or sign in to access this content.

Registration is free and takes less than one minute. Click here to register and get full access to the Edge AI and Vision Alliance's valuable content.

The post September 2023 Edge AI and Vision Innovation Forum Presentation Videos appeared first on Edge AI and Vision Alliance.

This market research report was originally published at the Yole Group’s website. It is reprinted here with the permission of the Yole Group.

PISÉO broadens its field of expertise in optical radiation measurement.

OUTLINE

• The new service offering is aimed at the medical, automotive, XR, Industry 4.0, defense, and aeronautics markets.

• Among the characteristics measured are MTF, geometric distortion, signal-to-noise ratio, field of view, sharpness, contrast, colorimetry, etc.

• PISÉO’s expertise are recognized by the certifications received from relevant approving bodies, including product approval organizations in the automotive market segment.

PISÉO, Yole Group’s partner, is pleased to announce the launch of its optical imaging testing service. This new offering is the result of a journey based on 12 years of in-depth experience in lighting. The company, ISO 17025-accredited by COFRAC, is a major player in photometry and colorimetry. PISÉO is uniquely positioned with a comprehensive service offering, including support in the design and industrialization of optical solutions. Its global expertise in norms and standards and its active involvement in the Committees for Standardization since its creation give PISÉO a significant advantage in providing adequate testing programs and meeting the requirements of a complex normative system in constant evolution.

Set up by experts in optical imaging solutions over a year ago, the laboratory benefits from cutting-edge equipment essential for a reliable and accurate characterization of the entire optical chain for imaging systems in high-value-added areas. Several companies in the medical field (manufacturers of medical endoscopes or dental equipment) and XR – which includes AR, MR , and VR – already rely on PISÉO. The laboratory is involved during the development phase as part of potential product creation, or just before the product launch for the purpose of certification.

In the automotive industry, the company is also mandated by approval bodies in order to demonstrate, through independent and rigorous testing, the compliance of on-board optical imaging systems with international regulatory requirements – CMS, front cameras, DMS, etc. – which supports the features of ADAS. Indeed, the market demand is very strong in this sector. In its Imaging for Automotive report, Yole Intelligence, part of Yole Group, states that there were 2.6 cameras per car produced in 2021. This number should rise to 4.6 cameras per car produced in 2027, with high-end vehicles potentially fitted with at least double that number.

“We offer our clients assistance in understanding the standards, interpreting the results, and certifying their products.”
Joël Thomé
CEO, PISÉO

Acronyms

• MTF : Modulation Transfer Function
• XR : Extended Reality
• AR : Augmented Reality
• MR : Mixed Reality
• VR : Virtual Reality
• CMS : Camera Monitor Systems
• DMS : Driver Monitoring Systems
• ADAS : Advanced Driver Assistance Systems

Yole Group and PISÉO teams invite you to follow the technologies, related devices, applications, and markets on www.yolegroup.com and piseo.fr.

Stay tuned!

The post Optical Imaging: PISÉO On the Starting Blocks appeared first on Edge AI and Vision Alliance.

Jae-sun Seo, Associate Professor at Arizona State University, presents the “Efficient Neuromorphic Computing with Dynamic Vision Sensor, Spiking Neural Network Accelerator and Hardware-aware Algorithms” tutorial at the May 2023 Embedded Vision Summit. Spiking neural networks (SNNs) mimic biological nervous systems. Using event-driven computation and communication, SNNs achieve very low power…

“Efficient Neuromorphic Computing with Dynamic Vision Sensor, Spiking Neural Network Accelerator and Hardware-aware Algorithms,” a Presentation from Arizona State University

Register or sign in to access this content.

Registration is free and takes less than one minute. Click here to register and get full access to the Edge AI and Vision Alliance's valuable content.

The post “Efficient Neuromorphic Computing with Dynamic Vision Sensor, Spiking Neural Network Accelerator and Hardware-aware Algorithms,” a Presentation from Arizona State University appeared first on Edge AI and Vision Alliance.

Laguna Hills, Calif. – September 20, 2023 – BrainChip Holdings Ltd (ASX: BRN, OTCQX: BRCHF, ADR: BCHPY), the world’s first commercial producer of ultra-low power, fully digital, event-based, neuromorphic AI IP, today announced that it has partnered with VVDN Technologies, a premier electronics engineering and manufacturing company, to deliver the industry’s first Edge box based on neuromorphic technology.

VVDN Technologies is an industry-leading device manufacture and solutions provider that has extensive experience in developing and deploying vision-based solutions for various domains, such as automotive, industrial, security surveillance, enterprise, medical and others. VVDN will complement BrainChip’s AI capability with their domain expertise in hardware design, firmware development, cloud integration and manufacturing for the Edge box product.

“We are excited to bring the benefits of neuromorphic computing to the Edge AI market with VVDN as our lead partner,” said Sean Hehir, CEO of BrainChip. “This portable and compact Edge box is a game-changer that enables customers to deploy AI applications cost-effectively with unprecedented speed and efficiency to proliferate the benefits of intelligent compute.”

The Edge box is a compact and powerful device that can run various AI applications at the Edge of the network, such as video analytics, face recognition, object detection and more. The Edge box leverages BrainChip’s Akida processors, which are designed to mimic the structure and function of the human brain. Akida processors offer high performance, low power consumption and scalability for Edge AI solutions.

“Edge boxes are a rapidly growing segment in AI and are currently based on platforms from major players,” said Bram Geenen Co-founder of Wevolver, providers of one of the most subscribed Edge AI analyst reports. “The cost-effectiveness, efficiency and scalability of BrainChip’s Akida neuromorphic processor, coupled with VVDN’s solutions expertise should deliver a boost to the proliferation of customizable and secure AI applications at the Edge.”

“Edge AI is currently dominated by Nvidia and Qualcomm. But these systems are not ideal for the Edge from a power, size, and cost perspective,” said Marc Staimer, President Dragon Slayer Consulting. “Edge AI systems specifically architected to meet the performance, power, cooling, portability, and cost requirements are necessary to drive market growth.”

The Edge box will be available for presale from BrainChip and VVDN later this year.

About BrainChip Holdings Ltd (ASX: BRN, OTCQX: BRCHF, ADR: BCHPY)

BrainChip is the worldwide leader in Edge AI on-chip processing and learning. The company’s first-to-market, fully digital, event-based AI processor, Akida TM , uses neuromorphic principles to mimic the human brain, analyzing only essential sensor inputs at the point of acquisition, processing data with unparalleled efficiency, precision, and economy of energy. Akida uniquely enables Edge learning local to the chip, independent of the cloud, dramatically reducing latency while improving privacy and data security. Akida Neural processor IP, which can be integrated into SoCs on any process technology, has shown substantial benefits on today’s workloads and networks, and offers a platform for developers to create, tune and run their models using standard AI workflows like Tensorflow/Keras. In enabling effective Edge compute to be universally deployable across real world applications such as connected cars, consumer electronics, and industrial IoT, BrainChip is proving that on-chip AI, close to the sensor, is the future, for its customers’ products, as well as the planet. Explore the benefits of Essential AI at www.brainchip.com.

Follow BrainChip on Twitter: https://www.twitter.com/BrainChip_inc
Follow BrainChip on LinkedIn: https://www.linkedin.com/company/7792006

The post BrainChip Engages VVDN to Deliver Industry’s First Commercial Edge Box Based on Neuromorphic Technology appeared first on Edge AI and Vision Alliance.

Laguna Hills, Calif. – September 20, 2023 – BrainChip Holdings Ltd (ASX: BRN, OTCQX: BRCHF, ADR: BCHPY), the world’s first commercial producer of ultra-low power, fully digital, event-based, neuromorphic AI IP, has been invited to present at the conference on Emerging Growth in AI, presented by Maxim Group LLC, Tuesday, September 26 and Wednesday, September 27, 2023, at 8:00 a.m. ET.

The continuous evolution of AI is paving the way for groundbreaking applications in the technology sector. Participants will dive deep into how companies are leveraging AI and computer vision to unlock new opportunities, spanning from semiconductor innovations to fintech breakthroughs. Maxim senior analysts will facilitate engaging dialogues with CEOs and key management of emerging growth companies with a strong focus on AI. BrainChip CEO Sean Hehir will discuss the role BrainChip’s technology plays in shaping the future of in-situ intelligence on tiny devices in a virtual fireside chat September 26 at 3:00pm ET.

BrainChip’s Akida neural processor is designed to provide a complete ultra-low power Edge AI network processor for vision, audio, smart transducers, vital signs and, more broadly, any sensor application. BrainChip’s scalable solutions, which can be used standalone or integrated into systems on chip to execute today’s models and future networks directly in hardware, empowers the market to create much more intelligent, cost-effective devices and services that can be universally deployed across real-world applications in connected cars, healthcare, consumer electronics, industrial IoT, smart-agriculture and more, including use in a space mission and in the most stringent conditions.

“The Emerging Growth in AI Conference is an ideal venue to showcase how advanced IoT infrastructure and AI technologies inspired by the human brain, deliver highly efficient problem solving, with more intuitive and secure human-to-machine interactions,” said Hehir. “As a leader in Edge AI, we see just how ubiquitous the technology will be and will highlight how we are driving AI into a plethora of use cases that weren’t possible before.”

The two-day Emerging Growth in AI event features more than 40 companies with a focus on artificial intelligence participating in a series of virtual panels, live roundtables, and company discussions. This conference will be live on M-Vest. To attend, interested parties must sign up to become an M-Vest member and reserve their seat.

About BrainChip Holdings Ltd (ASX: BRN, OTCQX: BRCHF, ADR: BCHPY)

BrainChip is the worldwide leader in Edge AI on-chip processing and learning. The company’s first-to-market, fully digital, event-based AI processor, Akida TM , uses neuromorphic principles to mimic the human brain, analyzing only essential sensor inputs at the point of acquisition, processing data with unparalleled efficiency, precision, and economy of energy. Akida uniquely enables Edge learning local to the chip, independent of the cloud, dramatically reducing latency while improving privacy and data security. Akida Neural processor IP, which can be integrated into SoCs on any process technology, has shown substantial benefits on today’s workloads and networks, and offers a platform for developers to create, tune and run their models using standard AI workflows like Tensorflow/Keras. In enabling effective Edge compute to be universally deployable across real world applications such as connected cars, consumer electronics, and industrial IoT, BrainChip is proving that on-chip AI, close to the sensor, is the future, for its customers’ products, as well as the planet. Explore the benefits of Essential AI at www.brainchip.com.

Follow BrainChip on Twitter: https://www.twitter.com/BrainChip_inc
Follow BrainChip on LinkedIn: https://www.linkedin.com/company/7792006

The post BrainChip to Participate in the Virtual Tech Conference Series: Emerging Growth in AI, Presented by Maxim Group LLC appeared first on Edge AI and Vision Alliance.

VeriSilicon’s IP solutions optimized for demanding Robotics and AR/VR/MR applications

Shanghai, China, September 21, 2023–VeriSilicon (688521.SH)  today announced that Inuitive, a leading vision-on-chip processor company, has adopted its dual-channel Image Signal Processor (ISP) IP, featuring low latency and low power capabilities, into its mass-produced NU4100 vision AI processor. This integration brings outstanding imaging and vision experience to Robots, Augmented Reality (AR), Virtual Reality (VR), Mixed Reality (MR), Drones, and many other applications.

Inuitive’s NU4100 is a highly-integrated single-chip with high-quality 3D depth sensing, enhanced AI processing and on-chip VSLAM (Visual Simultaneous Localization and Mapping) capabilities. It can process large volumes of data from multiple 4K resolution cameras with low latency and low power consumption. As a powerful vision processor, it introduces an optimized embedded vision architecture that effectively combines a set of computing blocks including computer vision engine and deep learning (CNN) processing.

VeriSilicon’s ISP provides high throughput processing capability, catering to a wide range of applications for the NU4100 vision AI processor. It supports multiple cameras with multi-context management, HDR (High Dynamic Range), advanced noise reduction, and more, greatly enhancing the overall performance of the vision processor.

Dor Zepeniuk, CTO at Inuitive, says, “Our NU4100 high-performance vision AI processor has set a new benchmark in the industry with a unique set of combined 3D vision, AI and VSLAM capabilities. We greatly value VeriSilicon’s advanced technologies and services. With an aim of providing the ultimate single-chip solutions for Robotics and various applications, we have deepened our collaboration with VeriSilicon on roadmap processors, to bring the industry  solutions that addressing fundamental vision challenges.”

Wei-Jin Dai, Executive VP and GM of IP Division at VeriSilicon, says, “Robotics and AR/VR/MR applications represent the forefront of innovation driven by progressive technologies. With our comprehensive Glass to Glass intelligent pixel processing IP portfolio, and IP subsystems which leverage our unique FLEXA technology that delivers short latency from sight to video and AI processing, VeriSilicon will actively innovate in these fields through collaborations with our partners like Inuitive.”

About Inuitive

Inuitive’s disruptive Vision-on-Chip processors introduce all-in-one chips with a wide range of integrated capabilities, outstanding performance, optimal size, and cost efficiency. These game-changing processors support simultaneous depth sensing, positioning, location algorithms (SLAM), and AI-based object detection and recognition while dramatically shortening both system latency and response time, saving power and improving overall performance (high frame rate and camera resolution, and a wide FOV).

Together with its technological ecosystem of partners in the field of machine sensing, software development, and commercial manufacturing, Inuitive integrates its enterprise-ready sensor-and-processor modules into its customers’ robotics, drones, AR, VR, MR, AIoT, and 3D sensing applications, providing human-like visual understanding with optimal capabilities and superior performance. For more information, please visit www.inuitive-tech.com.

About VeriSilicon

VeriSilicon Microelectronics (Shanghai) Co., Ltd. (VeriSilicon, 688521.SH) is committed to providing customers with platform-based, all-around, one-stop custom silicon services and semiconductor IP licensing services leveraging its in-house semiconductor IP. Under the unique “Silicon Platform as a Service” (SiPaaS) business model, depending on the comprehensive IP portfolio, VeriSilicon can create silicon products from definition to testing and packaging in a short time, and provides high-performance and cost-efficient semiconductor alternative products for fabless, IDM, system vendors (OEM/ODM), large internet companies and cloud service providers, etc. VeriSilicon’s business covers consumer electronics, automotive electronics, computer and peripheral, industry, data processing, Internet of Things (IoT) and other applications.

VeriSilicon presents a variety of customized silicon solutions, including high-definition video, high-definition audio and voice, in-vehicle infotainment, video surveillance, IoT connectivity, smart wearable, high-end application processor, video transcoding acceleration and intelligent pixel processing, etc. In addition, VeriSilicon has six categories of in-house processor IPs, namely GPU IP, NPU IP, VPU IP, DSP IP, ISP IP and Display Processor IP, as well as more than 1,500 analog and mixed signal IPs and RF IPs.

Founded in 2001 and headquartered in Shanghai, China, VeriSilicon has 7 design and R&D centers in China and the United States, as well as 11 sales and customer service offices worldwide. VeriSilicon currently has more than 1,400 employees. For more information, please visit: www.verisilicon.com.

The post Inuitive Adopts VeriSilicon’s Advanced ISP IP for its Vision AI Processor appeared first on Edge AI and Vision Alliance.

The ZIP file linked to below contains PDFs of the various presentations from the Alliance’s September 2023 Edge AI and Vision Innovation Forum. Please be aware that these materials are for Alliance Member company internal use only. Note that the presentation from Yoav Banin of Nauto is not yet available.…

September 2023 Edge AI and Vision Innovation Forum Presentation Files

Register or sign in to access this content.

Registration is free and takes less than one minute. Click here to register and get full access to the Edge AI and Vision Alliance's valuable content.

The post September 2023 Edge AI and Vision Innovation Forum Presentation Files appeared first on Edge AI and Vision Alliance.

Sumedh Datar, Senior Machine Learning Engineer at 7-Eleven, presents the “Item Recognition in Retail” tutorial at the May 2023 Embedded Vision Summit. Computer vision has vast potential in the retail space. 7-Eleven is working on fast frictionless checkout applications to better serve customers. These solutions range from faster checkout systems…

“Item Recognition in Retail,” a Presentation from 7-Eleven

Register or sign in to access this content.

Registration is free and takes less than one minute. Click here to register and get full access to the Edge AI and Vision Alliance's valuable content.

The post “Item Recognition in Retail,” a Presentation from 7-Eleven appeared first on Edge AI and Vision Alliance.

News highlights:

• Newly released NeuroMosAIc Processor family extends serviceable edge AI market through 60% better performance efficiency.

• AiM Future joins the Arm AI Partner Program expanding the serviceable market for concurrent multimodal inferencing at the edge.

• AiM Future partners with Design & Reuse to reach a global semiconductor design community.

• The European Commission accepts the Horizon Program proposal including AiM Future, world-class universities, research institutes and industry leaders.

Seoul, Korea, and San Jose, CA, [September 20, 2023] – AiM Future, the pioneer in concurrent multimodal inference accelerators for edge and end-point devices, announced a new family of NeuroMosAIc Processors delivering sixty percent higher performance efficiency over the previous generation. The products leverage numerous architecture and technology advancements enabling edge and end-point devices to perform complex AI tasks at the lowest power and cost.

The new family is comprised of the fully configurable NeuroMosAIc Processor and three pre-configured options, the ultra-low power NMP-350, a performance-efficiency-focused NMP-550, and an edge-computing-focused NMP-750. All are supported by the accompanying NeuroMosAIc Studio, extending applications in computer vision, audio, and multi-sensor fusion. The family leverages architecture version 2.6 enhancements including doubling AI compute density, expanded activation function acceleration, and improved system utilization resulting in a twenty-five percent smaller area and fifty percent lower power consumption when compared to the previous generation products executing the same neural network.

AiM Future continues to forge the path to natural cognition at the edge by enabling the simultaneous processing of multiple real-time sensor data streams within the existing size, weight, and power (SWaP) constraints of consumer electronics, automotive and mobile robotics. The offering is also compatible as a coprocessor to industry standard CPUs including Arm® Cortex®-A and Cortex-M processors.

AiM Future is committed to supporting its customers’ need to stay ahead of the pace of innovation. As part of this mission, it has joined the Arm AI Partner Program, an ecosystem of companies committed to accelerating innovation and helping simplify AI deployment across multiple segments. The Arm AI Partner Program brings partners together to collaborate and provide best-in-class tools, algorithms, and applications to businesses worldwide.

“The pace of change in the field of AI is increasing every day with new technologies and use cases emerging across every industry, and developers need access to a wide range of hardware and software solutions to scale edge AI”, said Paul Williamson, senior vice president and general manager, IoT Line of Business at Arm. “We created the Arm AI Partner Program in 2019 to help address the opportunities AI can provide, and the program has continued to gain momentum, with 112 partners to date. With innovators like AiM Future on board, we look forward to seeing even more AI applications built on Arm.”

By joining the program, Aim Future continues to grow its network of strategic partnerships in support of customer success. The combination of the NeuroMosAIc Processor with Arm’s CPUs and platform solutions eases engineering design and reduces time to production.

“We are thrilled to partner with Arm and bring our technology to Arm’s global ecosystem. As part of LG Electronics, the first silicon using the NeuroMosAIc Processor was produced with an integrated Arm Cortex-A53”, said ChangSoo Kim, Chief Executive Officer of AiM Future. “We are excited to extend that partnership as AiM Future and address the rapidly expanding domain of edge AI, where consumers expect greater flexibility and real-time, multi-sensor analytics within existing cost and size budgets.”

Adding to AiM Future’s global ecosystem development, it announced a partnership with Design & Reuse, the de facto semiconductor intellectual property (IP), subsystems and solutions marketplace. Since 1997, Design & Reuse has been connecting semiconductor designers with enabling technology providers across the globe.

”Today’s SoCs are exponentially more complex than previous generations and addressing advanced artificial intelligence and machine learning applications introduces yet another layer for SoC designers to navigate”, said Gabriele Saucier, CEO of Design and Reuse. “We are grateful to partner with AiM Future, which is breaking down the barriers to adding low power, cost-optimized AI accelerators to common electronic systems across multiple industries.”

By working with Design & Reuse, AiM Future reaches thousands of semiconductor designers and product planners continually searching for advanced technologies to stay ahead of the competition. AI and machine learning acceleration technologies have emerged as one of the fastest-growing applications in the semiconductor industry.

”Leading silicon design teams for multiple decades has proven great products are the result of great talent and great IP”, says CEO ChangSoo Kim, AiM Future. “We are excited by partnering with D&R as another step on the path to enabling a global SoC design community pursuing greatness in the next generation of edge and end-point AI.”

Meeting the sensor-to-cloud AI deployment solutions driven by industry and national interests requires collaboration from enabling technology providers to cloud and network service providers. As a result, government and industry investment is intensifying across the globe to drive cooperation and innovation through international partnerships.

To this end, AiM Future announced the European Commission (EC) has accepted the proposal submitted by a consortium of its partners to the Horizon Program. SINTEF, one of Europe’s largest research institutes, leads a group of world-class universities, emerging technology innovators and industry leaders, including EMC, Ericsson, Intel, and Telenor, with the goal of bringing human-like cognition to the edge computing continuum.

“We are honored to lead a collection of leading innovators such as AiM Future with a common goal of addressing the many challenges standing in front of making seamless and ubiquitous AI a reality for society”, stated Hui Song, Senior Scientist and Acting Research Manager of SINTEF. “We are grateful the EC has recognized the importance of our goals and look forward to working with this talented collection of researchers and engineers.”

The three-year program sets out to demonstrate solutions to the circular challenges of delivering machine cognition at each layer of a sensor-to-cloud AI deployment, where each layer is required to deliver a unique level of autonomy through the data pipeline.

“While our NeuroMosAIc Processor addresses the problems closest to the point of data creation, achieving the goals of the program requires the integration of numerous complex technologies,” says Jaehwa Kwak, CTO of AiM Future, “The program offers an excellent opportunity for AiM Future to collaborate with a collection of international experts at the forefront of AI research and development. We are eager to demonstrate what the future holds for these innovative technologies.”

About AiM Future

Based in Seoul, South Korea, the company was founded in late 2020 with LG Electronics as a strategic investor. Subsequently, it attracted a Series-A round investment from leading venture capital firms including L&S Venture Capital, Hi Investment Partners, Daedeok Venture Partners, KB Investment, and We Ventures. The company is focused on the development of neural network hardware accelerator IP and compiler software. Its flagship NeuroMosAIc architecture achieved commercialization in 2019 and the company has executed several license agreements with partners around the globe. Learn more at https://www.aimfuture.ai.

The post AiM Future Introduces Next-generation NeuroMosAIc Processors, Expands Partnerships appeared first on Edge AI and Vision Alliance.

Jason Lavene, Director of Advanced Development Engineering at Keurig Dr Pepper, talks with Jeff Bier, Founder of the Edge AI and Vision Alliance, for the “Lessons Learned in Developing a High-Volume, Vision-Enabled Coffee Maker” interview at the May 2023 Embedded Vision Summit. Why did Keurig Dr Pepper—a $12B beverage company—spend…

“Lessons Learned in Developing a High-Volume, Vision-Enabled Coffee Maker,” an Interview with Keurig Dr Pepper

Register or sign in to access this content.

Registration is free and takes less than one minute. Click here to register and get full access to the Edge AI and Vision Alliance's valuable content.

The post “Lessons Learned in Developing a High-Volume, Vision-Enabled Coffee Maker,” an Interview with Keurig Dr Pepper appeared first on Edge AI and Vision Alliance.

The PDF file linked to below is the general sponsorship and exhibit prospectus for the May 2024 Embedded Vision Summit: May 2024 Embedded Vision Summit General Sponsorship Prospectus For more information, please contact Amy Ardzrooni at ardzrooni@edge-ai-vision.com and Patrick Ahern at ahern@edge-ai-vision.com.

May 2024 Embedded Vision Summit Sponsorship and Exhibit Opportunities

Register or sign in to access this content.

Registration is free and takes less than one minute. Click here to register and get full access to the Edge AI and Vision Alliance's valuable content.

The post May 2024 Embedded Vision Summit Sponsorship and Exhibit Opportunities appeared first on Edge AI and Vision Alliance.

SAN JOSE, Calif., September 19, 2023 (Newswire.com) – The SHD Group, a leading strategic marketing and business development firm, today announced its plans to create a detailed market research report that will provide an in-depth analysis of the RISC-V market. It will include design-start projections from 40nm to 2nm by application, RISC-V IP, ecosystem directions, and a 5-year forecast. The report will be available in the 4^th quarter of 2023.

The analyst report will be authored by Rich Wawrzyniak, who has over 20 years of semiconductor industry experience, specifically in market research and analysis. His research experience encompasses ASICs, SoCs, SIP, memory, design starts, and emerging technologies, such as AI, RISC-V, and chiplets. Formerly a principal market analyst at Semico, he is currently a principal analyst at The SHD Group. Rich is a graduate of Loyola University in Chicago, Ill.

This new RISC-V Market Report will include:

• Market analysis of the current and future landscape for RISC-V applications.
• RISC-V-based SoC unit volume and revenue projections with 5-year forecasts for RISC-V growth by application segment.
• Related IP market trends, including projected growth, licensing, royalty, and maintenance revenues.
• Software tools and ecosystem for RISC-V, highlighting their role in enabling RISC-V technology design wins.
• Regional insights related to RISC-V adoption and growth in North America, Europe, Japan, China, and the Asia Pacific region.

“We are developing this RISC-V Market Report to provide much-needed insights into this fast-paced and evolving market,” said Derek Meyer, CEO of The SHD Group. “RISC-V has emerged as a disruptive force in the semiconductor industry, and our report will equip stakeholders with the knowledge needed to make informed decisions about the future direction of their companies and products.”

Calista Redmond, CEO of RISC-V International, commented: “RISC-V stands at the forefront of innovation, driving the future of semiconductor design with its open standard instruction set architecture (ISA). We are pleased that The SHD Group is developing this report and believe it will serve as an invaluable guide for our 3,800 members in over 70 countries who define RISC-V open specifications.”

RISC-V, an open and versatile ISA, has gained significant traction in shaping the future of the next generation of semiconductor design. The RISC-V Market Report promises to be an invaluable resource for industry executives, investors, and designers seeking a comprehensive understanding of this evolving landscape.

If you are a RISC-V technology provider and wish to schedule a briefing with Rich Wawrzyniak about your products for this upcoming Market Report, please reserve a time at: https://calendly.com/shd-r-wawrzyniak/risc-v-briefing-with-rich-wawrzyniak

About The SHD Group

The leadership team at The SHD Group has a decades-long track record of successful business development rooted in strategic market analysis. The company specializes in providing its clients with essential insights required to navigate the complexities of emerging markets and to formulate strategies for capturing market share in both new and established technology sectors. Our experience spans diverse industries, including AI, processors, sensors, systems, design services, and silicon, with a significant presence in consumer, automotive, industrial, and computing sectors. At The SHD Group, we are committed to being your reliable partner in achieving lasting success. Learn more at TheSHDGroup.com.

The post New RISC-V Market Report Will Provide 5-year Growth Projections for Semiconductor Devices and Insights on the Enabling Ecosystem appeared first on Edge AI and Vision Alliance.

Dean Kamen, Founder of DEKA Research and Development, talks with Jeff Bier, Founder of the Edge AI and Vision Alliance, for the “Embedded Vision in Robotics, Biotech and Education” interview at the May 2023 Embedded Vision Summit. In his 2018 keynote presentation at the Embedded Vision Summit, legendary inventor and…

“Embedded Vision in Robotics, Biotech and Education,” an Interview with Dean Kamen

Register or sign in to access this content.

Registration is free and takes less than one minute. Click here to register and get full access to the Edge AI and Vision Alliance's valuable content.

The post “Embedded Vision in Robotics, Biotech and Education,” an Interview with Dean Kamen appeared first on Edge AI and Vision Alliance.

Mobile robots have experienced substantial growth in the last decade due to their autonomous mobility, which has been propelled by advancements in robotics technology, autonomous navigation, and artificial intelligence. IDTechEx’s market research report, titled “Mobile Robotics in Logistics, Warehousing, and Delivery 2024-2044“, delves into the technical, regulatory, and market aspects influencing the emerging logistics mobile robot industry.

According to IDTechEx’s research, there has been a significant cumulative increase in funding for various types of mobile robots (e.g., mobile picking, intralogistics, material handling, etc.) from 2015 to 2022. This trend highlights the potential of mobile robots to automate various logistics operations, including material handling, material picking, long-haul distribution, and last-mile delivery.

As of 2023, some applications, such as material transport using automated guided vehicles (AGVs), have already reached a mature stage, attracting billions of dollars in annual revenue. However, other applications, like drone delivery, are still emerging and are not expected to see widespread deployment until the end of this decade due to regulatory constraints and specific technological challenges. Despite varying market readiness levels, each segment is experiencing rapid growth, with instances of technology giants acquiring start-up companies. Some noteworthy recent examples include the acquisition of Robotnik by United Robotics Group, Amazon’s acquisition of iRobot, and the merger of Mobile Industrial Robots (MiR, a Teradyne company) with AutoGuide Mobile Robots.

IDTechEx’s report, “Mobile Robotics in Logistics, Warehousing and Delivery 2024-2044”, examines the key products used in various logistics operations, including AGVs (e.g., tow tractors, forklift AGVs, unit load carts, ‘goods to person’ grid based AGVs, etc.), autonomous mobile robots (AMRs), case-picking robots, mobile manipulators, heavy load autonomous mobile vehicles (AMVs), and last-mile delivery robots (vans, sidewalk robots, and drones). The report highlighted recent technology advancements and commercial transitions, such as how Amazon’s first fully autonomous mobile robot, Proteus, is expected to drive the adoption of AMRs in warehouses.

In terms of the technology, IDTechEx analyzed the teardown components of various mobile robots to offer an in-depth technology analysis of emerging technologies underpinning the growth of mobile robots. These technologies range from sensors such as LiDAR, ultrasonic sensors, and cameras to software such as computer vision, simultaneous localization and mapping (SLAM), and other navigation and sensing technologies. IDTechEx interviewed a number of industry players ranging from leading companies such as Omron, MiR, and ForwardX Robotics to mid-sized companies and start-ups to discuss the unique value propositions of their technologies, how they can help to address the pain points, along with the most updated business information such as regulatory approvals, funding status, and latest flagship products.

One of the transitions IDTechEx spotted is the transition from AGVs to AMRs. AGVs have been widely adopted in many warehouses. Unlike AMRs, AGVs rely on infrastructures such as magnetic tapes, QR codes, and markers to navigate. Although this provides high navigation accuracy, it increases the total cost of setting up the infrastructure and restricts the flexibility of AGVs. With the emerging SLAM technology and more companies demanding flexible operation, IDTechEx has noticed a few AMRs being used in factories and warehouses, for example, Amazon’s Proteus. The IDTechEx report analyzes both AGVs and AMRs and provides an independent analysis of what scenarios these two robot types would be favorable to users.

The report also provides 20-year market forecasts for the future of mobile robotics in the logistics industry, including granular breakdowns by application area and product categories. Through IDTechEx’s research, the yearly market size of mobile robots (L4 trucks exclusive) will reach around US$150 billion, representing significant opportunities for component suppliers, robot OEMs, and end-users.

To find out more about IDTechEx’s technical and commercial analysis of mobile robotics in the logistics industry, please see the IDTechEx report, “Mobile Robotics in Logistics, Warehousing, and Delivery 2024-2044“.

Free-to-Attend Upcoming Webinar: Mobile Robotics in Logistics – Over 20-fold Revenue Increase in the Next Two Decades

Yulin Wang, Technology Analyst at IDTechEx and author of this article, will be presenting a free-to-attend webinar on the topic on Tuesday 10 October 2023 – Mobile Robotics in Logistics – Over 20-fold Revenue Increase in the Next Two Decades.

The webinar’s agenda includes the following topics:

• An overview of mobile robots and a breakdown of different types of mobile robots used in logistics
• Exploring the reasons behind the rapid growth of mobile robots in logistics and the benefits they offer to various industries
• Discussion of the fundamental technologies and components that empower mobile robot systems
• Analyzing the present state of the market, including noteworthy developments such as acquisitions and new product launches by industry leaders
• Delving into the obstacles and regulatory considerations that mobile robots face, along with a timeline for addressing these challenges
• Providing an overview of IDTechEx’s independent forecast for the future of mobile robots in the logistics sector

Click here to find out more and register your place on one of the three sessions on the 10th October.

About IDTechEx

IDTechEx guides your strategic business decisions through its Research, Subscription and Consultancy products, helping you profit from emerging technologies. For more information, contact research@IDTechEx.com or visit www.IDTechEx.com.

The post Mobile Robotics: Increasing Flexibility Enables Increasing Efficiency in Logistics appeared first on Edge AI and Vision Alliance.

Jeff Bier, Founder of the Edge AI and Vision Alliance, welcomes attendees to the May 2023 Embedded Vision Summit on May 24, 2023. Bier provides an overview of the edge AI and vision market opportunities, challenges, solutions and trends. He also introduces the Edge AI and Vision Alliance and the…

May 2023 Embedded Vision Summit Opening Remarks (May 24)

Register or sign in to access this content.

Registration is free and takes less than one minute. Click here to register and get full access to the Edge AI and Vision Alliance's valuable content.

The post May 2023 Embedded Vision Summit Opening Remarks (May 24) appeared first on Edge AI and Vision Alliance.

Jeff Bier, Founder of the Edge AI and Vision Alliance, welcomes attendees to the May 2023 Embedded Vision Summit on May 23, 2023. Bier provides an overview of the edge AI and vision market opportunities, challenges, solutions and trends. He also introduces the Edge AI and Vision Alliance and the…

May 2023 Embedded Vision Summit Opening Remarks (May 23)

Register or sign in to access this content.

Registration is free and takes less than one minute. Click here to register and get full access to the Edge AI and Vision Alliance's valuable content.

The post May 2023 Embedded Vision Summit Opening Remarks (May 23) appeared first on Edge AI and Vision Alliance.

This blog post was originally published at SOYNET’s website. It is reprinted here with the permission of SOYNET.

A Multi-Modal Conversational Model for Image Understanding and Generation

Visual ChatGPT allows users to perform complex visual tasks using text and visual inputs. With the rapid advancements in AI, there is a growing need for models that can handle multiple modalities beyond text and images, such as videos or voices. One of the challenges in building such a system is the amount of data and computational resources required. But worry no more because the newly released Visual ChatGPT has got us covered; it is based on ChatGPT and incorporates a variety of Visual Foundation Models (VFMs) to bridge the gap between ChatGPT and visual information. It uses a Prompt Manager that supports various functions, including explicitly telling ChatGPT the capability of each VFM and handling the histories, priorities, and conflicts of different VFMs.

The Prompt Manager

The Prompt Manager in Visual ChatGPT aims to help the language model accurately understand and handle various visual language tasks by providing clear guidelines and prompts for different scenarios. The Prompt Manager distinguishes among the various Visual Foundation Models (VFMs) available in Visual ChatGPT and helps select the appropriate model for a specific task. It defines the name, usage, inputs/outputs, and optional examples for each VFM to help the model decide which VFM to use for a particular task.

The Prompt Manager also handles user queries by generating a unique filename for newly uploaded images and appending a suffix Prompt to force Visual ChatGPT to use foundation models instead of relying solely on its imagination. Using these prompts encourages the model to provide specific outputs generated by the foundation models rather than generic responses and to use the appropriate VFM in a particular scenario. This helps improve the accuracy and relevance of Visual ChatGPT’s responses to user queries, particularly those related to visual language tasks.

Visual foundation model (VFM)

It is a machine-learning model that processes visual information, such as images or videos. VFMs are usually trained on large amounts of visual data. They can recognize and extract features from visual inputs. Other machine learning models, such as ChatGPT, can then use these features to perform more complex tasks requiring language and visual understanding. For example, a VFM could identify objects in an image. Then ChatGPT could generate a caption describing the objects in natural language. VFMs can be trained using various techniques, including convolutional neural networks (CNNs) and recurrent neural networks (RNNs).

Sources: Visual ChatGpt Paper

Example of Visual ChatGPT in Action

To understand how Visual ChatGPT works, consider the following scenario: a user uploads an image of a yellow flower and enters a complex language instruction “please generate a red flower conditioned on the predicted depth of this image and then make it like a cartoon, step by step.” With the help of Prompt Manager, Visual ChatGPT starts a chain of execution of related Visual Foundation Models.

In this case, it first applies the depth estimation model to detect the depth information, then utilizes the depth-to-image model to generate a figure of a red flower with the depth information, and finally leverages the style transfer VFM based on the Stable Diffusion model to change the style of this image into a cartoon.

During the above pipeline, Prompt Manager serves as a dispatcher for ChatGPT by providing the type of visual formats and recording the information transformation process. Finally, when Visual ChatGPT obtains the hints of “cartoon” from the Prompt Manager, it will end the execution pipeline and show the final result.

Customized System Principles

Visual ChatGPT is designed to assist with various text and visual-related tasks, such as VQA, image generation, and editing. The system relies on a list of VFMs to solve various VL tasks. Visual ChatGPT is designed to avoid ambiguity and be strict about filename usage, ensuring that it retrieves and manipulates the correct image files.

Filename sensitivity is critical in Visual ChatGPT since one conversation round may contain multiple images and their updated versions. To tackle more challenging queries by decomposing them into subproblems, Chain-of-Thought (CoT) is introduced in Visual ChatGPT to help decide, leverage, and dispatch multiple VFMs. Visual ChatGPT must follow strict reasoning formats.

The intermediate reasoning results are parsed using elaborate regex matching algorithms to construct the rational input format for the ChatGPT model to help determine the next execution, e.g., triggering a new VFM or returning the final response. Visual ChatGPT must be reliable as a language model and not fabricate image content or filenames. Therefore, prompts are designed to ensure that the model is loyal to the output of the vision foundation models.

Limitations

Visual ChatGPT’s limitations are its dependence on ChatGPT and VFMs. The accuracy and effectiveness of individual models invoked heavily influence the performance of Visual ChatGPT. Additionally, the heavy prompt engineering required to convert VFMs into language and make them distinguishable can be time-consuming and involves computer vision and natural language processing expertise.

Another limitation of Visual ChatGPT is its limited real-time capabilities. Since it automatically decomposes complex tasks into several subtasks, handling a specific task may involve invoking multiple VFMs, resulting in limited real-time capabilities.

To deal with the unsatisfactory results due to some VFMs failure, a self-correction module is suggested in the paper; this is for checking the consistency between execution results and human intentions and making the corresponding editing. This self-correction behavior can lead to more complex thinking of the model, significantly increasing the inference time.

Despite the limitations, Visual ChatGPT has demonstrated great potential and competence for different tasks despite these limitations. Its integration of visual information into dialogue tasks holds great promise for the future of AI systems.

Sweta Chaturvedi
Marketing Manager, SOYNET

Sources: Original Paper Visual ChatGpt

The post Visual ChatGPT Explained appeared first on Edge AI and Vision Alliance.

This market research report was originally published at the Yole Group’s website. It is reprinted here with the permission of the Yole Group.

Overcoming a global economic downturn, the MEMS market is set to grow to US$20 billion by 2028 as MEMS allow OEMs in the consumer, automotive, and other industries to optimize the cost, size, and performance of their systems.

In today’s article, Pierre Delbos and Pierre-Marie Visse from Yole Intelligence and Khrystyna Kruk from Yole SystemPlus reflect on the last 20 years of the MEMS market, highlighting the evolving drivers, the most significant MEMS innovations, and the future areas set to drive MEMS demand.

This analysis forms part of the report, Status of the MEMS Industry 2023, along with related reverse engineering & costing analyses, including:

• Consumer MEMS Pressure Sensors Comparison 2023
• Bosch IMU in iPhone 14 Pro
• Vesper MEMS Piezoelectric Voice Accelerometers : VA1200 & VA1210…

These products are fully based on Yole Group’s 20 years of expertise in the MEMS market and technologies.

Yole Intelligence and Yole SystemPlus are part of Yole Group.

Increased market readiness of emerging devices will drive the MEMS market of the future

In addition to the market being driven by a growing number of devices and increased MEMS-enabled functionality, we expect to see a rise in market adoption of other devices that will fuel additional demand in the near future.

For example, MEMS microspeakers offer potential for their very small footprint but have typically not performed as well as legacy technology at low frequencies. As microspeaker companies start to improve the performance and achieve design wins – such as within TWS headphones and hearing aids – the volumes could push down the price such that they can compete with traditional speakers.

Furthermore, as emerging systems edge closer to mass adoption, we expect to see greater MEMS adoption.

For example, with the high-volume arrival of AR glasses in 2030, Yole Group expects MEMS micromirrors for LBS microprojection to act as a stop-gap solution (competing with LCoS) before microLEDs become viable.

“While MEMS gas sensors traditionally have performed worse than other sensors, we expect them to find a place within low-cost, low-power-consumption applications with space constraints and commoditization of consumer apps. Applications such as monitoring indoor air quality, where there is renewed interest following the COVID-19 pandemic, could help drive penetration in this market.”
Pierre-Marie Visse
Technology and Market Analyst, Sensing and Actuating, Yole Intelligence (part of the Yole Group)

Another noticeable trend is the transition from 8 to 12-inch MEMS manufacturing. While this involves a significant investment for MEMS players, it allows for better integration with 12-inch CMOS wafers and supports optimal equipment performance, such as for lithography and DRIE steps.

Though this report has existed in various iterations for 20 years now, the MEMS industry is still bringing disruptive technologies and applications to the table. To succeed in this industry, it is necessary to meet the market’s requirements: a growing total accessible market, a technology with an actual added value for specific applications, and enough funding and support to sustain fierce competition against an established industry. But with all these new players, technologies, and megatrends, the future of the MEMS industry looks bright!

Back to the future: How MEMS market drivers have evolved in 20 years

The success of MEMS technologies is indisputable, and MEMS devices continue to be increasingly adopted in almost all markets.

“With the ongoing global demand for sensorization and data-driven applications, the past 20 years during which we have analyzed the MEMS industry have shown continuous innovation and even opened new product perspectives. Over the years, various market drivers, successive crises, and ecosystem changes shaped the US$14 billion MEMS industry of today.”
Pierre Delbos
Technology and Market Analyst, Sensing and Actuating, Yole Intelligence (part of the Yole Group)

In 2003, automotive was a major driver as more advanced safety features started to be incorporated into vehicles. Accelerometers used in airbags, gyroscopes for ESP systems, and the early adoption of pressure sensors for tire pressure monitoring were some of the first automotive applications for MEMS.

MEMS inkjet printheads from HP created significant demand around this time, surpassing automotive in terms of volume and sales: around US$1 billion, compared to US $800 million for automotive. However, one of the most symbolic MEMS devices in the early 2000s was Texas Instruments’ DLP for projection applications that created significant sales, about US$300 million, and Texas Instruments became an early market leader along with HP.

The arrival of the iPhone in 2007 and subsequent widespread adoption of smartphones caused a surge in MEMS demand in the consumer sector, which is the largest market today. Automatic screen rotation created early demand for accelerometers, and the introduction of more advanced smartphone features such as navigation assistance, step counting, and gaming further fueled the demand for inertial sensors. MEMS microphones also started to be used in smartphones – the Motorola Razr was one of the first adopters – and ended up as one of the most shipped MEMS devices in the industry. Microphones were the perfect example of how MEMS’ small size and low power consumption matched smartphone requirements.

A second wave in the consumer electronics industry started in 2016 and saw increased adoption of wearable devices, such as smartwatches and TWS headphones, which also boosted the MEMS industry. In addition to microphones that allow beamforming, inertial sensors were used in TWS headphones for bone conduction sensing, allowing perfect voice pick-up and 3D audio functionalities. This second consumer wave and subsequent surge in demand for inertial MEMS sensors allowed STMicroelectronics and Bosch to take the leading market positions from early leaders such as TI and HP, with revenues surpassing US$1 billion (STMicroelectronics) and nearly US$2 billion (Bosch) today.

Pierre Delbos adds: “As we wait for the consumer market to breathe new life following the recent fall in smartphone demand, the automotive market is now leading growth in the MEMS industry. Automotive is being propelled by the electrification of cars and the introduction of autonomous driving, leading to the complete sensorization of vehicles.”

Indeed, even though more pressure sensors were used in an ICE car than in an electric vehicle, we expect other car domains to undergo massive transformation and reshuffle the needs for MEMS. Autonomous automotive functions are driving the adoption of MEMS inertial sensors, micromirrors, magnetometers, and more. MEMS oscillators are also increasingly being used to support the exchange of the ever-growing amount of data in the automotive sector and within telecommunications, particularly with the arrival of 5G.

Technology outlook: mature, but innovation ongoing across the entire value chain

MEMS technologies are well established in their target markets. But despite the maturity of MEMS components such as pressure sensors, inertial sensors, and microphones, notable innovation is taking place, allowing OEMs to optimize cost, size, and performance, further driving demand.

“Innovations are occurring across the entire MEMS manufacturing process, from the choice of material all the way to the final integration of dies in the package, while noticeable technology trends have developed.”
Khrystyna Kruck
Technology and Cost Analyst, Yole SystemPlus (part of the Yole Group)

From the design and material standpoint, MEMS players are innovating to improve the performance of their devices. For example, Infineon Technologies sealed dual-membrane technology within Goertek’s MEMS microphone in Apple Airpods Pro significantly improves SNR performance.

By moving away from its single-back (2010) and dual-back (2014) designs to a sealed dual-membrane (2019) structure, water and dust are prevented from being trapped between the membrane and the backplate, enabling practically noise-free audio signal capturing (68-75dB(A)). In addition, Vesper’s mono-membrane design inside its MEMS piezoelectric microphone represents a radical technology shift, improving water- and dustproofing for better performance in applications that require high robustness and reliability. This possibly explains why the start-up seduced the 3^rd biggest MEMS player, Qualcomm.

But MEMS leaders are also innovating on the integration side. From 2015 to 2018, Bosch’s pressure sensors in the Apple iPhone moved from LGA packaging to an o-ring waterproof package, allowing Apple to improve durability. During this period, Bosch more than halved the size of the MEMS die, from 0.8mm² to 0.35mm², following an industry pattern to miniaturize. And from 2018 to 2023, iPhone pressure sensors had the same footprint but were packaged differently. Bosch’s 2023 BMPxxx v2 MEMS pressure sensor die in the latest iPhones, for example, is glued with an adhesive on the ASIC die, which is then glued onto the ceramic substrate, with the electrical connections using wire bondings. Previous versions used flip-chip bonding for the ASIC integration inside the package.

Bosch even came out with a new manufacturing technique! Its new laser reseal process significantly reduces pressure variation to maximize the performance of inertial sensors within the iPhone 14 Pro. While the process is three times more expensive than previous processes, it allows MEMS gyroscopes and accelerometers to be integrated onto the same die, thus enabling further miniaturization of the sensor and better control of the vacuum level inside the cavity.

Stay tuned on yolegroup.com!

Yole Group will attend the MEMS & Imaging Sensors Forum, powered by SEMI. Analysts will be pleased to present key results of the annual MEMS report with a dedicated presentation on Sep. 20, “Future of the MEMS Technology” session. Yole Group’s speaker, Pierre-Marie Visse, will explore the latest and future MEMS trends for a smarter world…

Analysts are glad to meet Yole Group’s customers and business partners, establish new contacts, and help drive business forward during the show.

Come and meet them and look through the latest market, technology, reverse engineering and reverse costing analyses. Ask for a meeting at Yole Group’s booth: events@yolegroup.com.

The post The MEMS Industry: Looking Back at the Last 20 Years of Innovation and Growth appeared first on Edge AI and Vision Alliance.

This blog post was originally published at Tenyks’ website. It is reprinted here with the permission of Tenyks.

We introduce the Multiclass Confusion Matrix for Object Detection, a table that can help you perform failure analysis identifying otherwise unnoticeable errors, such as edge cases or non-representative issues in your data.

In this article we introduce a practical error prediction matrix: the Multiclass Confusion Matrix for Object Detection (MCM). This matrix serves as a good first step to systematically conduct failure analysis of your models.

We describe what the matrix is, and how it allows you to get deeper insights into mispredicted examples. This more granular view of errors can reveal patterns that might go undetected when simply aggregating errors into false positives and false negatives.

We build on our previous article, and use the same example of helmet detection, as shown in Figure 1.

Figure 1. Use-case: detecting classic and welding helmets

Failure analysis in object detection

In a nutshell, failure analysis focuses on examining why an object detector fails on certain inputs, analyzing the causes of the model’s errors or mistakes [1], [2].

At a high level, failure analysis may involve:

1. Identifying instances that the model fails on (i.e. incorrect predictions).
2. Determining why the model failed on these particular instances (e.g. failing on edge cases the model wasn’t trained on).
3. Collecting additional data to address the causes (e.g. gathering more diverse data for the underrepresented categories).
4. (Possibly) Improving the model setup (e.g. using a more powerful architecture).

‍Some of the most common reasons for model failures include:

• Lack of representative training data: The training dataset does not have enough examples of certain objects or scenarios, causing the model to not generalize well.
• Class imbalance: The distribution of the classes in the data is skewed, as a result the model doesn’t learn some classes well.
• Quality of images: The images are degraded by noise, poor lighting, occlusion, or other factors, making the objects difficult to detect.
• Similarity of objects: In this case the model is unable to reliably distinguish between objects that are very similar in appearance.

Figure 2. Bounding boxes affected by occlusion and blur can cause models to fail

Metrics in Object Detection

As we described in our previous post, the primary metric to evaluate object detection performance is mean average precision (mAP). However, when using mAP, failures in object detection can only be coarsely described as false positives or false negatives.

Remember, there are 3 types of predictions in object detection:

• True Positive (TP): correct model prediction. An annotation is correctly matched with a prediction.
• False Positive (FP): incorrect model prediction. The model predicted a bounding box but no corresponding annotation existed.
• False Negative (FN): missing prediction. An annotation is not matched to any prediction (i.e. the object is present but was not detected by the model).

Why grouping together TPs, FPs and FNs is limiting

The downside of organizing predictions in the traditional way (see Figure 3) is that you are left with a numeric score that doesn’t provide any insights to identify why some samples were classified as false positives or false negatives.

Without this information, how can you improve your dataset to have a better model?

Figure 3. Classic approach to account for false positive and false negative predictions

What if you are tasked to answer questions such as:

• How many examples of Class A did the model confuse with Class B?
• How many undetected predictions were there (e.g. due to non-representative or rare samples)?
• How many examples don’t have any corresponding annotation (i.e. ghost predictions)?

A solid first step to answering the previous set of questions is to use the Multiclass Confusion Matrix for Object Detection (MCM), which is described below.

Multiclass Confusion Matrix for Object Detection

The MCM arranges predictions into an (N+1) * (N+1) matrix (where N is the number of classes). Every cell (i, j) specifies the number of matched bounding boxes that were predicted as class i (or not predicted, if i=N+1), and had an annotation class j (or no annotation, if j=N+1).

Selecting the best match across all classes, not just within the same class

Given a ground truth label, this matrix finds the best matching prediction, even if the predicted label is from a different class than the true label.

‍For example, in our helmet detection task, this matrix might select a “classic helmet” prediction as the best match for a true “welding helmet” label, if this is a better option. This allows the MCM to capture errors where the model confuses one class for another, rather than just errors within the same class.

Figure 4. The Multiclass Confusion Matrix for Object Detection

This closely resembles multiclass confusion matrices used in classification, with the addition of “Undetected” and “Ghost Prediction” rows and columns.

In particular, the MCM defines 4 types of predictions:

• True Positive: correct model prediction. A prediction is matched with an annotation of the same class (this definition is the same as always).
• Undetected: missing model prediction. An annotation is not matched to any prediction (this definition is the same as the False Negative definition from before).

• Mispredicted: incorrectly-predicted class. A prediction is matched with an annotation of a different class (this would have been graded as a False Positive).

• Ghost Prediction: incorrect prediction. A prediction is not matched with any annotation (this would have also been graded as a False Positive).

How can the MCM improve your failure analysis process?

The MCM offers a more granular view into your model errors, including:

• Observing how errors are distributed across class combinations (instead of producing a single aggregate score, such as mAP, or AP)
• Better understanding the different types of errors (i.e. is it a ghost prediction, misprediction, or undetected case?)

‍This type of analysis can be invaluable in gaining a deeper understanding into why & where your model is failing. For instance:

• A high number of mispredictions between two classes can indicate a poor class definition, in which both classes share a lot of common samples. For instance, if you have classes “Pedestrian” and “Cyclist”, but then you have many samples of a person walking next to a bike.
• A high number of undetected objects could mean a large number of outliers and edge cases (i.e. data representativity issues). For example, cars of an unusual color, or old models of cars that are not frequently-present in the dataset.
• A high number of ghost predictions could indicate the need to set a higher confidence threshold for the model, or could indicate a low representativity of certain cases in your data. For instance, if you have a small number of cyclist samples, your model may start to incorrectly detect parked bikes as cyclists.

This kind of analysis is impossible to conduct on aggregate metrics, such as mAP or mAR.

‍In comparison to aggregating predictions into FP, and FN, Figure 5 shows a MCM table, where the predictions lie on the Y axis, and the ground truth labels are on the X axis:

• The misprediction occurs because for a classic helmet ground truth label (green solid bounding box on the left side), the model predicts a welding helmet (yellow dashed bounding box).
• The ghost prediction takes place because the model predicts a classic helmet (green dashed bounding box in the center) but this object does not exist in our ground truth labels.

‍Key takeaway: The Multiclass Confusion Matrix for Object Detection (MCM) helps you identify undetected objects, ghost predictions, and mispredictions in a more intuitive way than simply using TP, FP, and FN.

Figure 5. The MCM helps you quickly spot undetected, ghost predictions and mispredicted instances

Example: Identifying undetected pedestrians in the Kitti dataset

In this section we provide a practical application example of the Multiclass Confusion Matrix for Object Detection (MCM), using the Tenyks platform.

‍Kitti dataset: We use Kitti, a popular dataset containing data for object detection, tracking, depth perception, and more — all from real-world driving scenarios. This makes it useful for machine learning models in self-driving cars, where not detecting pedestrians is very dangerous, since disastrous consequences can happen when a self-driving car fails to spot a pedestrian in time.

‍Figure 6 shows how the MCM helps you quickly visualize where a trained detector fails at detecting pedestrians: after clicking on the number of undetected objects with the ground truth label of Pedestrian, you can see what images contain these undetected samples.

Beyond that, using the Tenyks platform you can filter out these undetected objects to obtain more insights into why the model failed to predict these examples. As Figure 6 shows, many of the undetected Pedestrians objects are occluded, noisy or are very small. This is a truly effective way of investigating the root cause of why your model is failing, which is the first step to devising a strategy to improving your model’s performance!

Figure 6. The Multiclass Confusion Matrix for Object Detection (MCM) in action!

Summary

In this post we introduced the Multiclass Confusion Matrix for Object Detection (MCM), a granular first step to help you identify model failures in object detection. This matrix categorizes predictions into true positives, undetected, mispredicted and ghost predictions. The MCM allows you to identify samples that may lead to problems in your dataset such as edge cases or representative issues, that otherwise would go unnoticed by traditional approaches.

The road ahead: a hierarchical approach!

Generally, we can breakdown error analysis of an object detection model into three scenarios:

• High-level error analysis: including aggregate metrics such as mAP, mAR, TP, FP, and FN.
• Medium-level error analysis: a more granular approach than the previous one, such instance MCM.
• Low-level error analysis: a very low-level, detailed view of what is wrong and why, conducted by automatic strategies such as Data Quality Checks.

‍The Tenyks platform enables analysis of failure cases at all three different levels. As we have covered in this Series, you can analyze errors at the high and at the medium level. In the upcoming third instalment, we will explore how Tenyks enables Data Quality Checks to identify systemic issues, at the very low-level, that may be impacting model performance.

‍By examining failures at these three hierarchy levels, you can gain a holistic understanding of the weaknesses in your model and data, and take targeted actions to improve results.

Stay tuned!

‍Note: All images for this post are from unsplash.com based on the unsplash license.

‍References

1. Why Object Detectors Fail: Investigating the Influence of the Dataset
2. Imbalance Problems in Object Detection: A Review

‍Authors: Jose Gabriel Islas Montero, Dmitry Kazhdan.

The post Multiclass Confusion Matrix for Object Detection appeared first on Edge AI and Vision Alliance.

This blog post was originally published at NVIDIA’s website. It is reprinted here with the permission of NVIDIA.

The camera module is the most integral part of an AI-based embedded system. With so many camera module choices on the market, the selection process may seem overwhelming. This post breaks down the process to help make the right selection for an embedded application, including the NVIDIA Jetson.

Camera selection considerations

Camera module selection involves consideration of three key aspects: sensor, interface (connector), and optics.

Sensor

The two main types of electronic image sensors are the charge-coupled device (CCD) and the active-pixel sensor (CMOS). For a CCD sensor, pixel values can only be read on a per-row basis. Each row of pixels is shifted, one by one, into a readout register. For a CMOS sensor, each pixel can be read individually and in parallel.

CMOS is less expensive and consumes less energy without sacrificing image quality, in most cases. It can also achieve higher frame rates due to the parallel readout of pixel values. However, there are some specific scenarios in which CCD sensors still prevail—for example, when long exposure is necessary and very low-noise images are required, such as in astronomy.

Electronic shutter

There are two options for the electronic shutter: global or rolling. A global shutter exposes each pixel to incoming light at the same time. A rolling shutter exposes the pixel rows in a certain order (top to bottom, for example) and can cause distortion (Figure 1).

Figure 1. Distortion of rotor blades caused by rolling shutter

The global shutter is not impacted by motion blur and distortion due to object movement. It is much easier to sync multiple cameras with a global shutter because there is a single point in time when exposure starts. However, sensors with a global shutter are much more expensive than those with a rolling shutter.

Color or monochrome

In most cases, a monochrome image sensor is sufficient for typical machine vision tasks like fault detection, presence monitoring, and recording measurements.

With a monochrome sensor, each pixel is usually described by eight bits. With a color sensor, each pixel has eight bits for the red channel, eight bits for the green channel, and eight bits for the blue channel. The color sensor requires processing three times the amount of data, resulting in a higher processing time and, consequently, a slower frame rate.

Dynamic range

Dynamic range is the ratio between the maximum and minimum signal that is acquired by the sensor. At the upper limit, pixels appear white for higher values of intensity (saturation), while pixels appear black at the lower limit and below. An HDR of at least 80db is needed for indoor application and up to 140db is needed for outdoor application.

Resolution

Resolution is a sensor’s ability to reproduce object details. It can be influenced by factors such as the type of lighting used, the sensor pixel size, and the capabilities of the optics. The smaller the object detail, the higher the required resolution.

Pixel resolution translates to how many millimeters each pixel is equal to on the image. The higher the resolution, the sharper your image will be. The camera or sensor’s resolution should enable coverage of a feature’s area of at least two pixels.

CMOS sensors with high resolutions tend to have low frame rates. While a sensor may achieve the resolution you need, it will not capture the quality images you need without achieving enough frames per second. It is important to evaluate the speed of the sensor.

A general rule of thumb to determine the resolution needed for the use case is shown below and in Figure 2.  The multiplier (2) represents the typical desire to have a minimum two pixels on an object in order to successfully detect it.

INSERT GRAPHIC
Figure 2. Sensor resolution required is determined by lens field of view and feature of interest size

For example, suppose you have an image of an injury around the eye of a boxer.

• 
• FOV, mm = 2000mm
• Size of feature of interest (the eye), mm = 4mm

Based on the calculation, 1000 x 1000, a one-megapixel camera should be sufficient to detect the eye using a CV or AI algorithm.

Note that a sensor is made up of multiple rows of pixels. These pixels are also called photosites. The number of photons collected by a pixel is directly proportional to the size of the pixel. Selecting a larger pixel may seem tempting but may not be the optimal choice in all the cases.

Table 1.  Pros and cons of small and large pixel size

Back-illuminated sensors maximize the amount of light being captured and converted by each photodiode. In front-illuminated sensors, metal wiring above the photodiodes blocks off some photons, hence reducing the amount of light captured.

Figure 3. Cross-section of a front-illuminated structure (left) and a back-illuminated structure (right)

Frame rate and shutter speed

The frame rate refers to the number of frames (or images captured) per second (FPS). The frame rate should be determined based on the number of inspections required per second. This correlates with the shutter speed (or exposure time), which is the time that the camera sensor is exposed to capture the image.

Theoretically, the maximum frame rate is equal to the inverse of the exposure time. But achievable FPS is lower because of latency introduced by frame readout, sensor resolution, and the data transfer rate of the interface including cabling.

FPS can be increased by reducing the need for large exposure times by adding additional lighting, binning the pixels.

CMOS sensors can achieve higher FPS, as the process of reading out each pixel can be done more quickly than with the charge transfer in a CCD sensor’s shift register.

Interface

There are multiple ways to connect the camera module to an embedded system. Typically, for evaluation purposes, cameras with USB and Ethernet interfaces are used because custom driver development is not needed.

Other important parameters for interface selection are transmission length, data rate, and operating conditions. Table 2 lists the most popular interfaces. Each option has its pros and cons.

Table 2. Comparison of various camera interfaces

Optics

The basic purpose of an optical lens is to collect the light scattered by an object and recreate an image of the object on a light-sensitive image sensor (CCD or CMOS). The following factors should be considered when selecting an optimized lens-focal length, sensor format, field of view, aperture, chief ray angle, resolving power, and distortion.

Lenses are manufactured with a limited number of standard focal lengths. Common lens focal lengths include 6mm, 8mm, 12.5mm, 25mm, and 50mm.

Once you choose a lens with a focal length closest to the focal length required by your imaging system, you need to adjust the working distance to get the object under inspection in focus. Lenses with short focal lengths (less than 12mm) produce images with a significant amount of distortion.

If your application is sensitive to image distortion, try to increase the working distance and use a lens with a higher focal length. If you cannot change the working distance, you are somewhat limited in choosing an optimized lens.

Table 3. Main types of camera lenses

To attach a lens to a camera requires some type of mounting system. Both mechanical stability (a loose lens will deliver an out-of-focus image) and the distance to the sensor must be defined.

To ensure compatibility between different lenses and cameras, the following standard lens mounts are defined.

Table 4. Common lens mounts used in embedded space

NVIDIA camera module partners

NVIDIA maintains a rich ecosystem of partnerships with highly competent camera module makers all over the world. See Jetson Partner Supported Cameras for details. These partners can help you design imaging systems for your application from concept to production for the NVIDIA Jetson.

Figure 4. NVIDIA Jetson in combination with camera modules can be used across industries for various needs

Summary

This post has explained the most important camera characteristics to consider when selecting a camera for an embedded application. Although the selection process may seem daunting, the first step is to understand your key constraints based on design, performance, environment, and cost.

Once you understand the constraints, then focus on the characteristics most relevant to your use case. For example, if the camera will be deployed away from the compute or in a rugged environment, consider using the GMSL interface. If the camera will be used in low-light conditions, consider a camera module with larger pixel and sensor sizes. If the camera will be used in a motion application, consider using a camera with a global shutter.

To learn more, watch Optimize Your Edge Application: Unveiling the Right Combination of Jetson Processors and Cameras. For detailed specs on AI performance, GPU, CPU, and more for both Xavier and Orin-based Jetson modules, visit Jetson Modules.

Related resources

• GTC session: Real-Time Onboard Processing for Autonomous Indoor Drones using NVIDIA Jetson (Spring 2023)
• GTC session: Jetson Edge AI Developer Days: Getting the Most Out of Your Jetson Orin Using NVIDIA Nsight Developer Tools (Spring 2023)
• SDK: JetPack SDK
• Webinar: Implementing Computer Vision and Image Processing Solutions with VPI
• Webinar: ACCELERATE COMPUTER VISION AND IMAGE PROCESSING USING VPI 1.1
• Webinar: JetPack 5.0 Deep Dive: New Kernel, Bootloader, Trusted OS and Support for Jetson AGX Orin

Vikas Sharma
Product Manager, NVIDIA

The post Selecting the Right Camera for the NVIDIA Jetson and Other Embedded Systems appeared first on Edge AI and Vision Alliance.