This market research report was originally published at Yole Group’s website. It is reprinted here with the permission of Yole Group.

After a blip in the market due to the COVID-19 pandemic, the market is now back to normal and will be driven by traditional applications to a value of $9.2 billion in 2027.

OUTLINE

• The overall thermal imaging market is expected to reach US$8.7 billion in 2027. In parallel, the overall thermal detector market will be close to US$598 million in the same year.

• The general technical trend is a move towards pixel pitch reduction in microbolometers.

• Teledyne FLIR retains its historical leadership in thermal imaging but is slowing down in 2022.

• The Chinese industry holds its position with a 38% market share by volume in 2022 and is animated by notable advances.

• The U.S. market is increasingly supplied by local players (Teledyne FLIR, and Seek Thermal), while China prefers Hikvision.

The Chinese ecosystem is consolidating its position. Therefore, from the Chinese side, the thermal imager manufacturers are clearly consolidating their positions in 2022. Indeed, after ramping their shipments in 2020 for fever detection systems, they have managed to maintain their position in 2022 and ship nearly 40% of thermal imagers globally. Among them, Hikvision and InfiRay are the two leaders. Both companies have significantly improved the performance of their products and adapted their portfolio to be stronger in classical thermal applications, such as surveillance for Hikvision and the consumer market for InfiRay.

In this dynamic context, Yole Intelligence has developed a dedicated report to provide market trends and data on key thermal detectors (thermopiles, pyroelectrics) and imagers (microbolometers), including metrics and dynamics. In its new Thermal Imaging and Sensing 2022 report, the company, part of Yole Group, provides key technical insights into future technology trends and challenges and delivers an in-depth understanding of the thermal imaging value chain, infrastructure, and players. In addition, this study presents an in-depth analysis of thermal detectors and imagers for defense, automotive, industrial, and consumer applications.

“In 2021, the thermal imaging and sensing market returned to traditional applications as the demand to test body temperature for fever detection collapsed. As a result, the thermal detector market is expected to decrease from US$460 million in 2021 to US$439 million in 2022”.
Axel Clouet, Ph.D.
Technology and Market Analyst, Imaging and Display, Yole Intelligence (part of Yole Group)

The thermal imaging market is similarly expected to shrink from US$6.2 billion in 2021 to US$5.9 billion in 2022. The demand for classical applications is still solid, but the global component shortage results in delays for critical components necessary for manufacturing thermal cameras. If the situation returns to normal in 2023, the overall imaging & sensing market is expected to grow from US$6.6 billion in 2021 to US$9.2 billion in 2027, with a CAGR_21-27 of 6%.

Thermal detectors are becoming more digital, with many functionalities added to the sensor. In addition, the pixel pitch of thermopile detectors is still decreasing and may compete directly with thermal imagers, mainly for smart building applications.

“In thermal imagers, there is still a general trend for pixel pitch reduction, with R&D targets of 10µm to 8µm, whereas most commercial products have a 17µm or 12µm pitch.”
Dalil Oussalah, Ph.D.
Technology and Market Analyst, Sensing and Actuating, Yole Intelligence

Small pixel pitches could help reduce the size of chips and so their price while maintaining sufficient resolution. There is also a growing interest in thermal and visible data fusion for commercial applications, like automotive…

Yole Intelligence’s imaging team invites you to follow the thermal imaging and sensing industry and the evolution of the technology on www.yolegroup.com.

Stay tuned!

The post Thermal Imaging: The Chinese Advantage appeared first on Edge AI and Vision Alliance.

HILLSBORO, Ore. – Dec. 5, 2022 – Lattice Semiconductor (NASDAQ: LSCC), the low power programmable leader, today unveiled Lattice Avant, a new FPGA platform purpose-built to bring the company’s power efficient architecture, small size, and performance leadership to mid-range FPGAs. Lattice Avant offers best-in-class power efficiency, advanced connectivity, and optimized compute that enable Lattice to address an expanded set of customer applications across the Communications, Computing, Industrial, and Automotive markets.

“With Lattice Avant, we extend our low power leadership position in the FPGA industry and are poised to continue our rapid pace of innovation, while also doubling the addressable market for our product portfolio,” said Jim Anderson, President and CEO, Lattice Semiconductor. “We created Avant to address our customers’ need for compelling mid-range FPGA solutions, and we’re excited to help them accelerate their designs with new levels of power efficiency and performance.”

“Enormous amounts of data are being generated every day by billions of connected sensors, devices, and systems supported by AI algorithms, creating an accelerated need for intelligence at the Edge. This trend requires developers and OEMs to look for more flexible and adaptable solutions,” said Patrick Moorhead, CEO and Chief Analyst at Moor Insights & Strategy. “The introduction of Lattice Avant addresses this trend with its high-performance data processing capability where the need for innovation, efficiency, and flexibility is rapidly growing.”

Lattice Avant enables system and application developers to address important industry challenges around increases in technology connectedness and intelligence, the accelerating demand for innovation, and the growing need for efficiency and flexibility in system and application design. Combining its power optimized programmable fabric, class-leading aggregate bandwidth, adaptable hardware acceleration, and expanded application support, Lattice Avant is built from the ground up for low power FPGA leadership.

“We are proud to build on our history of power efficient innovation with the introduction of Lattice Avant by delivering breakthrough architectural advancements, features, and capabilities to address our customers’ needs across an expanded set of applications,” said Steve Douglass, Senior Vice President of R&D, Lattice Semiconductor. “Just as we’ve done with our Nexus platform, we have a strong and steady roadmap of future product introductions based on the Avant platform. To make designing with Lattice Avant FPGAs as easy as possible, Avant will be fully supported by our robust software tools and application-specific solution stacks.”

The Lattice Avant platform equips customers with:

• Power Efficiency
  • Up to 2.5X lower power than similar class competitive devices helping system and application engineers achieve power and thermal design efficiencies, improve operating costs, and enhance reliability
• Performance
  • Up to 2X faster throughput at lower power than similar class competitive devices delivers increased bandwidth and reduces links, system cost and size, while enabling datapath applications
• Form Factor
  • Up to 6X smaller package size compared to similar class competitive devices enables size efficient system designs
• Connectivity
  • Combines configurable SERDES up to 25 Gbps, hardened support for PCIe® Gen 4, high performance I/O, and high speed memory interface support including LPDDR4 and DDR5
• Software Support
  • Leverages existing, familiar Lattice software solutions including easy-to-use design tools, reference designs and SDKs, a broad IP portfolio, in addition to Lattice’s portfolio of application-specific solution stacks, to help customers achieve more efficient design development and faster go-to-market.

The Lattice Avant platform is architected for scalability that will enable the rapid development of multiple new device families, starting today with the launch of the Lattice Avant-E FPGA family. Lattice Avant-E FPGAs are designed to solve key customer challenges at the Edge by combining class-leading power efficiency, size and performance with an optimized feature set tailored to the needs of Edge applications like data processing and AI.

The Lattice Avant platform and Lattice Avant-E devices were announced today at a livecast event, a replay of which will soon be available on the company’s website.

For more information on the Lattice Avant, please visit:

• Lattice Avant platform
• Lattice Avant-E FPGAs

About Lattice Semiconductor

Lattice Semiconductor (NASDAQ: LSCC) is the low power programmable leader. We solve customer problems across the network, from the Edge to the Cloud, in the growing Communications, Computing, Industrial, Automotive, and Consumer markets. Our technology, long-standing relationships, and commitment to world-class support let our customers quickly and easily unleash their innovation to create a smart, secure, and connected world.

For more information about Lattice, please visit www.latticesemi.com. You can also follow us via LinkedIn, Twitter, Facebook, YouTube, WeChat, Weibo or Youku.

The post Lattice Extends Low Power Leadership with New Lattice Avant FPGA Platform appeared first on Edge AI and Vision Alliance.

• NT98530 Smart Camera SoC integrates CEVA SensPro2 DSP to support advanced computer vision and edge AI workloads, on-device

• Companies to showcase the SoC at CES 2023 in CEVA’s meeting suite

ROCKVILLE, MD – December 5, 2022 – CEVA, Inc. (NASDAQ: CEVA), the leading licensor of wireless connectivity and smart sensing technologies and co-creation solutions today announced that Novatek Microelelctonics Corp, a leading fabless chip design company, has licensed and deployed the SP500 DSP from CEVA’s SensPro2 Sensor Hub DSP architecture family in its newest generation NT98530 multi-sensor IP Camera SoC targeting surveillance, retail, smart city, transportation and more. Novatek was ranked No.2 in Taiwan and No. 6 in the world by fabless chip design house revenue in 2021, and specializes in smart display and smart image total solutions, ranging from full line of display ICs and SoC for all types of display and image applications. To date, Novatek has shipped more than 1 billion SoCs worldwide.

This latest engagement extends the long and successful partnership between the two companies, which spans almost a decade of licensing multiple generations of CEVA’s computer vision, audio/voice and AI DSPs. The companies will showcase this latest Edge AI camera solution at CES 2023 in CEVA’s meeting suite.

“Today’s multi-sensor security cameras need powerful computer vision and edge AI processing capabilities to maximize performance and minimize the reliance on cloud connectivity for intelligent analysis,” said Novatek AVP Jimmy Su. “CEVA’s SP500 DSP delivers superior real-time performance for computer vision, AI-based analytics and multi-sensing sensor fusion at the edge, ensuring our customers have a powerful, flexible edge AI camera solution which they can customize to their requirements.”

The Novatek NT98530 SoC is a highly-integrated SoC with high image quality, low bitrate, low power consumption, providing the best cost/performance Edge-Computing IP camera solution in the market today for multi-sensor security cameras. It supports up to 640M pixel/sec to achieve over 8MPixel/60FPS video performance while simultaneously performing advanced AI on each frame, with minimal power consumption. The integrated SensPro2, CEVA’s latest DSP generation along with its comprehensive & optimized SDKs delivers up to 6X DSP performance improvement in computer vision, 2X AI inferencing improvement and 20% lower power compared to its predecessor, ensuring that Novatek’s latest SoC has ample performance available to support even the most complex computer vision algorithms and neural networks inferencing.

“We’re pleased to extend our collaboration with Novatek to enable their latest generation surveillance SoC support multiple cameras and the most advanced computer/vision AI use cases,” said Ran Snir, Vice President and General Manager of the Vision Business Unit at CEVA. “Our SensPro2 DSP is primed to support more of the camera analytics workload in real time at the edge, including the ability to perform vision/AI processing for multiple camera sensors in parallel, at minimal power consumption”.

CEVA and Novatek at CES 2023

CEVA and Novatek will showcase the NT98530 Evaluation Board at CES 2023, January 5th to 8th 2023 in Las Vegas, NV, in CEVA’s meeting suite #2937 at the Westgate Hotel. To arrange a viewing or to meet with CEVA’s computer vision/AI experts attending the show, contact events@ceva-dsp.com.

About SensPro2

SensPro2 is the highly-scalable and enhanced 2nd generation high performance sensor hub DSP for multitasking sensing and AI of multiple sensors including camera, Radar, LiDar, Time-of-Flight, microphones and inertial measurement units. The SensPro2 family is designed to handle multiple sensor processing workload for contextually aware devices and can be utilized in modern intelligent systems in automotive (inc. ISO26262 functional safety compliance), robotics, surveillance, AR/VR, voice assistants, wearables, mobile and smart home devices. The SensPro2 maximizes performance-per-watt for multi-sensor processing use cases by utilizing a combination of high performance single and half precision floating point math, point cloud creation and Deep Neural Network processing, along with parallel processing capacity for voice, imaging, and Simultaneous Localization and Mapping (SLAM). The SensPro2 platform approach is paired with advanced software and development tools that include LLVM C/C++ compiler, Eclipse based integrated development environment, CEVA Deep Neural Network (CDNN) graph AI compiler for neural networks including the CDNN-Invite API for inclusion of custom AI engines, TensorFlow Lite Micro support, OpenVX API, and a multitude of software libraries for OpenCL, CEVA-CV imaging functions and CEVA-SLAM SDK. For more information, visit https://www.ceva-dsp.com/product/ceva-senspro/.

About CEVA, Inc.

CEVA is the leading licensor of wireless connectivity and smart sensing technologies and co-creation solutions for a smarter, safer, connected world. We provide Digital Signal Processors, AI engines, wireless platforms, cryptography cores and complementary software for sensor fusion, image enhancement, computer vision, voice input and artificial intelligence. These technologies are offered in combination with our Intrinsix IP integration services, helping our customers address their most complex and time-critical integrated circuit design projects. Leveraging our technologies and chip design skills, many of the world’s leading semiconductors, system companies and OEMs create power-efficient, intelligent, secure and connected devices for a range of end markets, including mobile, consumer, automotive, robotics, industrial, aerospace & defense and IoT.

Our DSP-based solutions include platforms for 5G baseband processing in mobile, IoT and infrastructure, advanced imaging and computer vision for any camera-enabled device, audio/voice/speech and ultra-low-power always-on/sensing applications for multiple IoT markets. For sensor fusion, our Hillcrest Labs sensor processing technologies provide a broad range of sensor fusion software and inertial measurement unit (“IMU”) solutions for markets including hearables, wearables, AR/VR, PC, robotics, remote controls and IoT. For wireless IoT, our platforms for Bluetooth (low energy and dual mode), Wi-Fi 4/5/6 (802.11n/ac/ax), Ultra-wideband (UWB), NB-IoT and GNSS are the most broadly licensed connectivity platforms in the industry.

Visit us at www.ceva-dsp.com and follow us on Twitter, YouTube, Facebook, LinkedIn and Instagram.

The post Novatek Adopts CEVA’s Latest Sensor Hub DSP for New Multi-sensor IP Camera SoC appeared first on Edge AI and Vision Alliance.

This blog post was originally published at Xailient’s website. It is reprinted here with the permission of Xailient.

How can we speed up video processing? Parallel processing is the answer!

If you want to process a number of video files, it might take from minutes to hours, depending on the size of the video, frame count, and frame dimensions.

If you are processing images in batches, you can utilize the power of parallel processing and speed up the task.

In this post, we will look at how to use Python for parallel processing of videos.

We will read video from the disk, perform face detection, and write the video with output of face detection (bounding boxes) back to the disk.

Let’s get started.

Install Dependencies

We will need the following packages:

OpenCV: is a computer vision library that is commonly used. In this post, we will use OpenCV to read and write video files.

To install OpenCV on your device, you can use either pip command or apt-get command.

or

FFmpeg: is a cross-platform solution to record, convert, and stream audio and video. In this post, we will use FFmpeg to join multiple video files.

To install FFmpeg, use the following apt-get command:

Let’s import the required Python libraries.

Details of libraries used:

cv2: OpenCV library to read from and write to video files.

time: to get the current time for calculating code execution time.

subprocess: to start new processes, connect to their input/output/error pipes, and obtain their return codes.

multiprocessing: to parallelize the execution of a function across multiple input values, distributing the input data across processes

xailient: library for face detection

Video Processing Pipeline Using Single Process

We will first define a method to process video using a single process. This is the way we would normally read a video file, process each frame, and write the output frames back to the disk.

Let’s create another function that calls the video processor, takes note of start and end times, and calculates the time taken to execute the pipeline and the frames processed per second.

Video Processing Pipeline Using Parallel Processing

Now, let’s define another function for processing video that utilizes multiprocessing.

The way the above function works is that the video processing job, which is normally done using one process, is now divided equally amongst the total number of processors available on the executing device.

If there are four processors, and the total number of frames in the video to be processed is 1,000, then each processor gets 250 frames to process, and these are executed in parallel. This means that each process will create a separate output file, and so when the video processing is completed, there are four different output videos.

To combine these output videos, we will use FFmpeg.

Create a pipeline to run the multiprocessing of video and calculate time to execute and frames processed per second.

Single Processing Versus Parallel Processing

I ran this experiment on a Lenovo Yoga 920 with the Ubuntu18.04 operating system. The number of logical processors available on this device is eight.

From this experiment, we can observe 2x more frames being processed per second when using all cores to process the video, making parallel processing a far more efficient choice for your video processing needs.

Sabina Pokhrel
AI Specialist and Machine Learning Engineer, Xailient

The post Parallel Processing Using Python for Faster Video Processing appeared first on Edge AI and Vision Alliance.

This market research report was originally published at Yole Group’s website. It is reprinted here with the permission of Yole Group.

Driven by electrification, smart cockpit and ADAS, automotive is the fastest-growing memory market segment, with a 20% CAGR between 2021 and 2027.

OUTLINE

• In terms of revenue, at US$4.3 billion in 2021 and ~20% CAGR 21-27, automotive memory is a relatively small yet fast-growing market.

• The automotive memory industry is driven by digital cockpit and ADAS & AD applications.

• NOR flash, DRAM and NAND technologies dominate the automotive memory market.

• Micron is the leading player in the market, with an estimated market share of 45% in 2021.

In 2021, at US$167 billion, stand-alone memory represented 28% of the total semiconductor market. Comparatively, the automotive memory market, worth US$4.3 billion in 2021 and with 2.6% of the global memory market revenue, represented 10% of automotive semiconductors. These figures highlight the prevalence of non-memory electronic components in current vehicles.

To illustrate the importance of this rapidly growing and robust market, Yole Intelligence has developed a dedicated market and technology report which provides an overview of memory technologies and business opportunities for automotive applications. In its new Memory for Automotive 2022 report, the company, part of Yole Group, details and analyzes technology trends and forecasts by application and describes the memory use cases and associated memories.

This analysis is part of the memory expertise developed by Yole Group over several years. The group offers a wide collection of products, all fully dedicated to understanding the memory markets and technologies. As an example, the memory market monitors – the NAND Market Monitor and DRAM Market Monitor – follow the memory ecosystem and highlight the strategies of the leading memory makers. These products include detailed figures by market segment, by technology, … and support the memory players through their strategic decision processes.

“The automotive market indeed represents a strong business opportunity for memory makers. At Yole Intelligence, we forecast memory to triple in revenue. It will reach 17% of the automotive semiconductor market by 2027, a 20% CAGR21-27, outperforming the global memory market with 8% CAGR21-27 and the automotive semiconductor market with 10% CAGR21-27.”
Thibault Grossi
Senior Technology and Market Analyst, Yole Intelligence

The overall automotive memory market is currently dominated by the leading memory maker, Micron, with an estimated market share of 45% by revenue. Samsung follows distantly behind in 2^nd place with a ~13% market share. Then, Infineon Technologies, Kioxia, SK Hynix, and ISSI are all behind Samsung, with market shares of ≤7%.

“In the global memory market, NAND and DRAM accounted together for ~96% of the 2021 revenue, while NOR flash accounted for only slightly more than 2%.”
Simone Bertolazzi, Ph.D.,
Senior Technology and Market Analyst for Memory, Yole Intelligence

NAND and DRAM also dominated the automotive memory market, with a combined share of 80%: 41% for DRAM and 39% for NAND. NOR flash has a much stronger presence in automotive, with a market share of 15% and ~US$0.7 billion.

The cockpit, with the main infotainment unit, instrument cluster, and connectivity, is currently the primary memory user. Aiming to reproduce a “smartphone-like” user experience, the cockpit memory content follows a similar trend. NOR flash is still being used, but managed NAND and DRAM represent most of the revenue.

ADAS & AD arrive as the second most significant memory user in-vehicle, with 24% of the revenue in 2021. The memory was mainly DRAM and high-density NOR flash, with some SLC NAND for smart sensors. It is the fastest-growing application domain for memory.

Other applications (powertrain, chassis & safety, and body & comfort) are estimated to represent about 5% of the bit demand. These domains, having the most constraints, mainly use robust memory technologies such as EEPROM and NOR flash.

In 2027, cockpit is expected to remain the largest application memory consumer, but ADAS & AD will see its share of the revenue increase to 36%. Technology-wise, DRAM and NAND are forecasted to generate almost 90% of automotive memory revenue.

The Yole Intelligence memory team invites you to follow the memory for automotive industry on www.yolegroup.com. In this regard, do not miss the Semiconductor trends in an automotive webcast on December 14^th! Program and registration here.

Stay tuned!

Acronyms

• CAGR : Compound Annual Growth Rate
• ADAS : Advanced Driver-Assistance System
• AD : Autonomous Driving
• NOR : Flash memory with logical NOR-type structure
• DRAM : Dynamic Random-Access Memory
• NAND : Flash memory with logical NAND-type structure
• SLC : Single-Level Cell
• EEPROM : Electrically Erasable Programmable Read-Only Memory

The post Memory For Automotive: A Fast-growing and Robust Market appeared first on Edge AI and Vision Alliance.

This article was originally published at Au-Zone Technologies’ website. It is reprinted here with the permission of Au-Zone Technologies.

Unlike image classifiers, which simply report on the most important objects within an image, object detectors determine where objects of interest are located, their sizes and class labels within an image. Consequently, object detectors are central to numerous computer vision applications.

In this article, we provide a technical introduction to deep-neural-network-based object detectors. We explain how these algorithms work, and how they have evolved in recent years, utilizing examples of popular object detectors. We will discuss some of the trade-offs to consider when selecting an object detector for an application, and touch on accuracy measurement. We also discuss performance comparison among the models discussed in this article.

Early Approaches

The seminal work in this regard was reported in the form of technical report at UC Berkeley, in 2014 by Ross Girshick et al. entitled “Rich feature hierarchies for accurate object detection and semantic segmentation”. This is popularly known as “R-CNN” (Regions with CNN features). They approached this problem in a methodical way with having 3 distinct algorithmic stages which is shown in Figure 1 below.

Figure 1: R-CNN [“Rich feature hierarchies for accurate object detection and semantic segmentation”]

As seen above, there are three stages:

• Region Proposal: Generate and extract category independent region proposals, using selective search.
• Feature Extractor: Extract feature from each candidate region using a deep CNN
• Classifier. Classify features as one of the known classes using linear SVM classifier model.

Here the main issue was that it produces lots of overlapping bounding boxes and post processing and filtering was needed to produce reliable results. Consequently, it was very slow and require 3 different algorithms to work in tandem. A faster version of this approach was published in 2015 entitled “Fast R-CNN” where authors combined second and third stages into one network with two outputs – one for classification and the other for bounding-box regression.

Shortly afterwards, they published “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks” where region proposal scheme was also implemented using a separate neural network.

Recent Approaches

YOLO

In 2016 a mark departure was proposed by Joseph Redmon et al. entitled “You Only Look Once: Unified, Real-Time Object Detection” or in short, YOLO. It divides the input image into an S × S grid. If the centre of an object falls into a grid cell, that grid cell is responsible for detecting that object. Each grid cell predicts B bounding boxes, confidence for those boxes, and C class probabilities. The Figure 2 below shows the basic idea.

Figure 2: YOLO [You Only Look Once: Unified, Real-Time Object Detection]

Figure 3 below shows the network architecture for YOLO detector. The network has 24 convolutional layers followed by 2 fully connected layers. Alternating 1 × 1 layers reduce the features space from preceding layers. The convolutional layers were pretrained on the ImageNet classification task.

Figure 3:YOLO Network Architecture

The predictions are encoded as an S × S × (B ∗ 5 + C) tensor. YOLO imposes strong spatial constraints on bounding box predictions since each grid cell only predicts two boxes. It can only have one class in a cell. This model struggles with small objects that appear in groups, such as flocks of birds.

YOLO-v3 was proposed in 2018 which utilizes multi-resolution approach with three different grid sizes 8×8, 16×16 and 32×32. Objects were learned based on the resolution and anchors. Each resolution has at least 3 anchors to define objects ratios at each cell.

Recently, in 2021 YOLOX was proposed which is anchor free and utilized decoupled output heads for class labels, bounding box regression and IoU, as shown in Figure 4 below.

Figure 4: YOLOX

SSD (Single Shot Detector)

This approach evaluates a small set of default boxes of different aspect ratios (anchor boxes) at each location in several feature maps with different scales (8 × 8 and 4 × 4 in (b) and (c) in Figure 5 below).

Figure 5: [SSD: Single Shot MultiBox Detector]

The SSD model adds several “feature scaler layers” to the end of a backbone network, which predict the offsets to default boxes of different scales and aspect ratios and their associated confidences. The figure below highlights the differences between YOLO and SSD. For each default box, it predicts both the shape offsets and the confidences for all object categories.

Figure 6: SSD vs YOLO

Inference frame rate (speed with batch size 8 using Titan X and cuDNN v4 with Intel Xeon E5-2667v3@3.20GHz.) is 46 fps for SSD of 300×300 pixel images with mAP = 74.3%. Whereas for YOLO (VGG-16) frame rate is 21 fps of 448×448 pixel images with mAP=66.4%. For SSD, output candidate bounding boxes can be very large (~8732) therefore Non-Maxima Suppression (NMS) post-processing needs to be optimized.

RetinaNet

Figure 7: [Focal Loss for Dense Object Detection]

This technique utilizes Feature Pyramid Network (FPN) backbone on top of feedforward ResNet architecture. They introduced Focal Loss to handle class imbalance in object detection dataset.

RetinaNet with ResNet-101-FPN matches the accuracy of ResNet- 101-FPN Faster R-CNN. The inference time is 172 msec per image on nVidia M40 GPU.

CenterNet

In 2019 CenterNet was proposed which had very different approach as compared to other object detection techniques discussed so far. It is trained to produce gaussian blob (heatmaps) at the centre of the object, thereby eliminating the need for anchor boxes as well as NMS post filtering.

Figure 8:[Objects as Points]

Object are localized by detecting the local maxima from the heatmaps. Single network predicts the keypoints (local maxima of heatmaps), offset, and size. The network predicts a total of C + 4 outputs at each location.

CenterNet is not the best in terms of accuracy but provides excellent accuracy-speed trade-offs. It can easily be extended to 3D object detection, multi-object tracking and pose estimation etc.

Evaluation Framework – Accuracy Measurement

True Positive (TP) — Correct detection made by the model.
False Positive (FP) — Incorrect detection made by the detector.
False Negative (FN) — A Ground-truth missed (not detected) by the object detector.

Intersection over Union (IoU)

The IoU metric in object detection evaluates the degree of overlap between the ground (gt) truth and predicted (pd) bounding boxes. It is calculated as follows:

IoU ranges between 0 and 1, where 0 shows no overlap and 1 means perfect overlap between ground truth and predictions.

IoU is used as a threshold (say, α), and using this threshold one can decide if a detection is correct or not. Consequently:

True Positive (TP) is a detection for which IoU(gt,pd)≥ α
False Positive (FP) is a detection for which IoU(gt,pd)< α
False Negative (FN) is a ground-truth missed with gt for which IoU(gt,pd)< α

If we consider IoU threshold, α = 0.5 , then TP, FP and FNs can be identified as shown in the Figure above. If we raise IoU threshold above 0.86, the first instance will be a FP and if we lower IoU threshold below 0.24, the second instance becomes TP.

Precision is the degree of exactness of the model in identifying only relevant objects. It is the ratio of TPs over all detections made by the model.
Recall measures the ability of the model to detect all ground truths—TPs among all ground truths.

A good model is supposed to have high precision and high recall.

Raising the IoU threshold means that more objects will be missed by the model (more FNs and therefore low recall and high precision). Conversely, a low IoU threshold will mean that the model gets more FPs (hence low precision and high recall).

The precision-recall (PR) curve is a plot of precision and recall at varying values of IoU thresholds. As shown in the figures below, the P-R curve is not monotonic, therefore smoothing is applied.

Figure 9: Precision-Recall Curve

Figure 10: Precision-Recall Curve (smoothed)

Average Precision (AP)

AP@α is Area Under the Precision-Recall Curve(AUC-PR) evaluated at α IoU threshold. AP50 and AP75 mean AP calculated at IoU=0.5 and IoU=0.75, respectively.

Mean Average Precision (mAP)

There are as many AP values as the number of classes. These AP values are averaged to obtain the metric – mean Average Precision (mAP). Pascal VOC benchmark requires computing AP at 11 points – AP@[0, 0.1, …, 1.0]. MS COCO describes 12 evaluation metrics shown below.

Comparisons

In this section we summarize and present comparisons which are published in the publicly available research papers.

The table above provides comparisons between SSD, YOLO and others on PASCAL VOC 2017 dataset. The tests were done on Titan X GPU and cuDNN v4 with Intel Xeon E5-2667v3@3.20GHz.

Here we see that SSD model is faster with higher mAP but produces lots of boxes and therefore decoding and NMS post filtering can be time consuming.

The table below provides comparison for RetinaNet with others on MS COCO dataset. Here RetinaNet with ResNet-101-FPN and a 600×600 pixel image size (on nVidia M40 GPU) runs at 122 msec per image. It shows that RetinaNet can good very good accuracy with ResNext-101-FPN backbone.

The next table below provides comparisons with CenterNet and the previous detectors on COCO dataset. The tests were performed with Intel Core i7-8086K CPU, Titan Xp GPU, Pytorch 0.4.1, CUDA 9.0 and CUDNN 7.1.

Here it can be observed that CenterNet gives best tradeoff between FPS and accuracy.

Choosing An Object Detector

There is no single best detector and choosing one involves looking several factors including application and hardware requirements. In general, highly accurate models are more complex and of larger sizes making them unattractive for embedded or resource constrained devices. On the other hand, smaller models tend to be faster but less accurate. Model size can be reduced, and inference speed can be enhanced using quantization from 32-bit floats to 16-bit floats or 8-bit integers. Obviously, significant boost in speed can be achieved on dedicated hardware such as GPU, NPU etc. at the expense of portability. The table below provides general observations when it comes to deployment.

Challenges On Edge Devices

In earlier sections we established some guidelines about Object Detection algorithms as well as pros and cons. To select the model that fits best for a given problem is a challenging task when working on edge devices. Most of the time, the objective is to maximise the accuracy as much as possible while maintaining a low inference workload and timing. The problem is that these are often conflicting requirements when compute resources are constrained or fixed as shown in the diagram below.

Training on detectors with custom datasets with eIQ

Usually, selected models should be fine-tuned or retrained on custom dataset without having clues about the generalization level. The end goal is the model learns to generalize as much as possible real-world images after seeing training samples. This point constitutes a challenge by itself given the vast number of parameters the model must train. On the other hand, there are few external parameters that play important roles within the training process, commonly known as hyper parameters. Hyper parameters involve learning rate, batch size, epochs, decay, etc. Since the training process oversees tuning millions of parameters we need to be carefully when selecting these hyper parameters because that could be the difference in having successfully training sessions or not. Along with the hyper parameters, we need good initialization values, so our model learns to generalize in custom dataset as faster as possible. If initialization values are not properly set, we could have undesirable situations like overfitting or poor generalization.

Deploying trained detectors to embedded platforms

Before deploying any model on any embedded platforms, it is important to have a well-defined and widely tested pipeline, otherwise compatibility problems, memory errors and issues related to unsupported ops could arise. Additionally, a deep understanding on the model architecture as well as in Machine Learning might be required to modify models and make them able to run on the edge devices.

Unlike classification, object detection algorithms return raw predictions that need to be decoded for specific functions. In most of the cases these functions do not have a side-by-side support on embedded platforms libraries. Quantization support is not always fully achievable in all the models, which make them to run in mixed precisions, some ops run on low bits (uint8, int8) while the rest run on (float16, float32). Even when this characteristic is not a big problem, to have the full model running in low bits (uint8, int8) always improve timings, and reduce memory and energy consumption.

To mitigate the impact of these common problems encountered when performing object detection at the edge, eIQ Portal provides a set of tools to handle these issues for embedded developer. These tools are integrated within easy-to-use GUI for quick and transparent execution – thereby reducing trial and error steps in the process. Complex concepts like data annotation, data augmentation, training a model (classification or detection), model evaluation on targets devices and model deployment are transformed into GUI components so the normal user does not have to deal with all the intricacies of training and deploying AI models on target hardware.

Conclusions

In this blog we introduced early DNN-based object detectors, R-CNN, Fast R-CNN, Faster R-CNN. The recent ones include YOLO, YOLOX, SSD, RetinaNet and CenterNet. We introduced object detection error types, precision-recall curve as well details of computation of mAP. We discussed performance and speed comparisons mentioned in the literature along with our own (@Au-Zone) inference performance on in-house dataset on iMx8mPlus HW. Finally, we provided general guidelines and factors involved in choosing object detectors. We will be following up on this post with Part II, which will provide details on real world performance for many common DNN based detection models on NXP processors including the i.MX8MPlus. i.MXRT1060 and i.MXRT1170.

Azhar Quddus, PhD.
Senior Computer Vision Engineer, Au-Zone Technologies

References

R-CNN
Ross Girshick Jeff Donahue Trevor Darrell Jitendra Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation”, Tech report (v5), UC Berkeley, 2014
https://arxiv.org/pdf/1311.2524.pdf

Fast R-CNN
Ross Girshick, “Fast R-CNN”, Microsoft Research, 2015.
https://arxiv.org/pdf/1504.08083.pdf

Faster R-CNN
Shaoqing Ren, Kaiming He, Ross Girshick and Jian Sun, “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks”, Advances in Neural Information Processing Systems, 2015.
https://arxiv.org/pdf/1506.01497.pdf

YOLO-v1
Joseph Redmon, Santosh Divvalay, Ross Girshick, Ali Farhadiy, “You Only Look Once: Unified, Real-Time Object Detection”, 2015.
https://arxiv.org/pdf/1506.02640.pdf

YOLO-v3
Redmon, Joseph, and Ali Farhadi. “Yolov3: An incremental improvement.“, 2018.
https://arxiv.org/pdf/1804.02767.pdf

YOLO-v4
Bochkovskiy A, Wang CY, Liao HY. Yolov4: Optimal speed and accuracy of object detection, 2020.
https://arxiv.org/pdf/2004.10934.pdf

YOLOX
Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, Jian Sun. YOLOX: Exceeding YOLO Series in 2021.
https://arxiv.org/pdf/2107.08430.pdf

SSD
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, Alexander C. Berg, “SSD: Single Shot MultiBox Detector“, 2015
https://arxiv.org/pdf/1512.02325.pdf

RetinaNet
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, Piotr Dollár, “Focal Loss for Dense Object Detection”, 2017.
https://arxiv.org/pdf/1708.02002.pdf

CenterNet
Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, Qi Tian, “CenterNet: Keypoint Triplets for Object Detection”, 2019.
https://arxiv.org/pdf/1904.08189.pdf

Accuracy
Object Detection Metrics With Worked Example
https://towardsdatascience.com/on-object-detection-metrics-with-worked-example-216f173ed31e#:~:text=Average%20Precision%20(AP)%20and%20mean,COCO%20and%20PASCAL%20VOC%20challenges

The post DNN-Based Object Detectors appeared first on Edge AI and Vision Alliance.

This video was originally published at Network Optix’ website. It is reprinted here with the permission of Network Optix.

Quanergy LiDAR solutions provide actionable insights to organizations across major industries and enable partners and their end-users to deploy innovative solutions to drive their business growth and ultimately, improve the quality of life for people around the world.

How LiDAR 3D Object Detection Works with Nx

Quanergy’s LiDAR 3D Object Detection is integrated with Powered by Nx products via the Nx Server API. Once an event is detected by LiDAR sensors, the event is 3D projected in the LiDAR interface and an API call is sent to the Nx system, triggering a Generic Event. Users can designate certain Actions to be triggered in response to detected events in the Nx desktop client.

Quanergy LiDAR 3D Object Detection Capabilities

Quanergy’s LiDAR platform and sensors analyze spaces in 3D with cm-level accuracy to provide volumetric sensing and insights including object location, direction, speed, and the type of objects detected.

Quanergy’s smart LiDAR solutions were designed to transform business operations with real-time, advanced 3D insights for a variety of industries including physical security, smart cities, smart spaces, industrial automation, and much more. Some applications include PIDS, RTLS, Counting and Flow Management.

Quanergy 3D LiDAR sensors, coupled with QORTEX DTC (Detect, Track, Classify), enables reliable and real-time tracking of:

• People and vehicles for security
• Smart space, and
• Smart city applications

The post Quanergy LiDAR 3D Object Detection Works with Nx appeared first on Edge AI and Vision Alliance.

This market research report was originally published at Yole Group’s website. It is reprinted here with the permission of Yole Group.

Automotive semiconductor growth is inevitable and leads to new technologies.

OUTLINE

• The market for semiconductor chips is expected to reach more than US$80 billion in 2027.

• Electrification and ADAS are the main drivers for this growth.

• Electrification: the most common devices are Si IGBT modules and SiC MOSFET modules, used in the main inverters for e-motor traction.

• The conventional automotive supply chain needs to thoroughly examine its position and transform to retain a competitive portfolio.

• Most players, whether OEM or Tier 1 supplier, do not yet have a well-defined strategy toward semiconductors.

The automotive semiconductor market is showing continuous growth, which is inevitable as the penetration of semiconductor-based applications, such as higher levels of ADAS and electrification, increases.

Despite a relatively flat light vehicle market, the market for semiconductor chips is expected to increase from US$44 billion in 2021 to US$80.7 billion in 2027 at a CAGR of 11.1%. This represents a semiconductor chip value of ~US$550/car, growing to ~US$912 in 2027. It is also an increase in the number of chips implemented in cars, from ~820 chips today to ~1100 chips per car in 2027.

In this dynamic context, Yole Intelligence has developed a dedicated report to provide an in-depth understanding of the changing automotive industry ecosystem and supply chain. In its new Semiconductor Trends in Automotive 2022 report, the company, part of Yole Group, details the current and future internal and external factors affecting the durability of the car industry. It also provides a complete overview of the current technology trends and market forecasts, including wafers, for cars. This study delivers key technical insights and analyses about future technology trends and challenges.

In addition, Yole SystemPlus experts, also part of Yole Group, are daily working on automotive Teardown Tracks. They are Reverse Costing® analyses of electronic systems with technology and cost assessments. Yole SystemPlus’ experts deconstruct these systems with the purpose of scrutinizing the automotive systems of various types right down to the semiconductor level, determining their supply chains in the process… More information.

“The rapid increase in car electrification demands new types of substrates, such as SiC for power electronics. It is expected to represent 1,130 kwafers in 2027. While still low compared to the ~30,500 kwafers of Si expected for 2027, SiC will grow faster than Si and GaAs /Sapphire. ADAS is also an important driver, and MCU s with cutting-edge technology as low as 16 nm / 10 nm will go into ADAS, including radar and other sensor controls. Levels 4 and 5 of autonomy will drive increasing demand for more memory (DRAM) and computing power”.
Pierrick Boulay
Senior Technology & Market Analyst, Photonics and Sensing Division, Yole Intelligence

For electrification, vertical integration is becoming popular among OEMs. It can work out in multiple ways: full integration down to the component level, system integration and subcontracting build-to-print parts, strategic cooperation/direct investments with key component suppliers, etc. The conventional automotive supply chain needs to examine its position thoroughly and transform through JVs, M&As, and new investments and divestments to retain its competitive edge. Although semiconductors are critical to the automotive industry in the ongoing disruptive transition, most players, both OEMs and T1 suppliers, do not have well-defined strategies for semiconductors yet. Specific expertise in semiconductor technologies and their supply chains, both internally and externally, is urgently needed to prepare for the future.

“Supply chain management will change as OEMs will need to negotiate directly with chip manufacturers, learn from the consumer industry, and keep ‘buffer stock’, “They must work closer with chip manufacturers on volume forecasts and long-term orders. Just-in-time manufacturing, pioneered by Toyota in the 1960s, no longer works with chip manufacturers in the current geopolitical climate”.
Eric Mounier, Ph.D.,
Director of Market Research, Yole Intelligence

Yole Intelligence and Yole SystemPlus teams invite you to follow the automotive semiconductor industry, the technology evolution, and the competitive landscape on www.yolegroup.com. In this regard, do not miss the Semiconductor trends in automotive webcast on December 14^th! Program and registration here.

Stay tuned!

Acronyms

• ADAS : Advanced Driver Assistance Systems
• Si : Silicon
• SiC : Silicon Carbide
• GaAs : Gallium Arsenide
• MCU : Microcontroller Unit

The post The Automotive Semiconductor Industry: Facing Disruptive Transitions and a New Deal appeared first on Edge AI and Vision Alliance.

This blog post was originally published at CEVA’s website. It is reprinted here with the permission of CEVA.

AI is becoming an increasingly popular technology, finding uses in more and more applications in sectors such as automotive, vision processing, and telecoms. AI is enabling new capabilities and replacing many conventional algorithms, for example to provide de-noising and image stabilization for smartphone cameras.

But while many AI implementations send data to a cloud data center, this has some major drawbacks: including added latency, privacy risks, and the need for an internet connection.

Instead, designers are looking to create AI systems that run on edge devices, which are often battery-powered – but this raises new challenges to balance the required performance and capabilities against power consumption, particularly as the demand for more computational power continues to grow rapidly.

AI processing challenges

While there are many different edge devices with varying requirements, they basically all aim to maximize performance while reducing power consumption, and minimizing the physical space required. How can design engineers make the right trade-offs to meet these challenges?

With existing AI processors, performance is often limited by bandwidth, and the bottlenecks involved in moving data to and from external memory. This leads to low system utilization, and means the performance/power figure (measured in TOPS/Watt) is restricted.

Another important issue is how to plan ahead for future needs. AI processor chips typically have a long deployment period, and an AI solution must be able to adapt to new requirements in the future, including supporting new neural networks that aren’t yet defined. This means any solution must have sufficient flexibility, and the scalability to grow with increasing performance demands.

AI systems must also be secure, and must meet the highest quality and safety standards, particularly for automotive applications and others where the AI system is potentially involved in life-critical decisions. If a pedestrian steps out in front of an autonomous vehicle, for example, there’s minimal time to respond.

To help overcome these challenges, a comprehensive software toolchain is needed, which also makes implementation simpler for customers and reduces development time.

AI processors step up

Let’s look at how AI solution providers are addressing these challenges, taking vision machine learning  as an example.

Figure 1: NeuPro-M AI processor block diagram, showing memory architecture

Firstly, if we consider the issues of bandwidth-limited performance and memory accesses, these can be addressed with a dynamically-configured two level memory architecture (see Figure 1). This minimizes power consumption due to data transfers to and from external SDRAM. By using local memory resources in a hierarchical manner, this achieves more than 90% utilization, and protects against ‘data starvation’ of the co-processors and accelerators – while also enabling each engine to process independently.

Another way to optimize AI processing is by enabling the processor architecture to support a mixed-precision neural engine. The ability to process data from 2 bits up to 16 bits means less system bandwidth consumption on top of flexibility to run mixed precision networks per use case. Furthermore, bandwidth reduction mechanism like data compression gives the ability to compress the data and weights in real-time, while writing or reading from the external memory. This reduces the memory bandwidth needed, further improving performance and significantly reduces overall power consumption.

Figure 2: segmentation with quad-engine core

This is the approach taken by CEVA’s NeuPro-M AI processor, which is a self-contained heterogeneous processor architecture for AI/ ML inference workloads. Using this solution as an example, figure 2 shows how a machine vision application could be segmented amongst four AI engines, in this case lane detection on the road ahead. The image data is loaded from external memory or from an external interface, and then split into four tiles, which are each handled by a different engine. Alternatively, the engines might each take on a sub-graph or different task, for example object detection as well as lane identification, to optimize the performance for a specific application.

Each engine has its own on-chip L1 memory, to minimize any bottlenecks or delays. This also means the AI processor can run almost fully independently once it is configured and, in most cases, can run ‘head-to-tail’ “fused” operations pipeline that doesn’t require internal memory access at all, with very little external memory access. This makes it more flexible, and helps improve its power efficiency.

The requirements we discussed at the start of this article also included providing a future-proof, flexible solution. A fully programmable vector processing unit (VPU) that can work in parallel to the coprocessors on the same engine L1 data, can meet this need, by ensuring software-based support of new neural network topologies and advances in AI workloads.

Machine vision optimizations

There are many optimizations that can boost performance in specific AI applications. In vision processing, one such optimization is the Winograd Transform. This is an alternative, efficient way of performing convolutions, such as for a Fourier transform, that uses only half the previously-required number of MACs (multiply and accumulate operations).

For 3×3 convolution layers, the Winograd transform can double the performance achieved, while keeping the same precision as the original convolution method.

Another essential optimization is the use of sparsity, which is the ability to ignore zero in the data or weights. By avoiding multiplication by zero, performance is improved, while preserving accuracy. While some processors require structured data to get the benefits of sparsity, better results can be achieved with a processor that provides full support for unstructured sparsity.

Typically, an AI system would need to hand some of these optimization functions or network inherent operations, such as Winograd transforms, sparsity mechanisms or pulling, self-attention operations & scaling, to specialized engines. This means that data needs to be offloaded, and then loaded back after processing – which adds latency and reduces performance. Instead, a better alternative is to connect accelerators directly to engine local shared L1 memory or better-off in most cases, conduct fused operations meaning on-the-fly end to end pipeline processing from one coprocessor to another, without the need to access any memory during execution.

How significant are these optimizations? Figure 3 shows the performance improvements achieved by a single engine NPM11 core, compared to CEVA’s previous generation of AI processor, for a typical ResNet50 implementation. You can see that the basic, native operation achieves a performance boost of nearly five times.

Adding a Winograd transform and then sparsity engine takes this performance increase further, up to 9.3 times the previous generation process. Finally, using mixed precision (8×8 & low-resolution 4×4) weights & activations for some of the network layers gives a further increase with negligible accuracy loss – achieving nearly 15 times faster performance than the previous processor, as well as a 2.9x increase over the native implementation.

Figure 3: NPM11 (single engine core) performance improvements

Conclusion

We’ve seen how a new memory architecture and local ‘load balancing’ control implementation (pipeline processing vs. consecutive processing of the same data), minimizes external accesses and utilizing hardware to the full, can improve performance without requiring more power, and how optimizations such as Winograd transforms and sparsity can give a further boost.

Overall, a modern AI processor can provide a fully programmable hardware/software development environment, with the performance, power efficiency and flexibility needed for demanding edge AI applications. This enables design engineers to benefit from effective AI implementations within their systems, without increasing power consumption beyond the budget of their portable edge devices.

Gil Abraham
Business Development Director and Product Marketing, Artificial Intelligence and Vision Business Unit, CEVA

The post Meeting the Challenges of Edge AI appeared first on Edge AI and Vision Alliance.

This blog post was originally published at Plumerai’s website. It is reprinted here with the permission of Plumerai.

Earlier this year in April we presented our MLPerf Tiny 0.7 benchmark scores, showing that our inference engine runs your AI models faster than any other tool. Today, MLPerf released the Tiny 1.0 scores and Plumerai has done it again: we still have the world’s fastest inference engine for Arm Cortex-M architectures. Faster inferencing means you can run larger and more accurate AI models, go into sleep mode earlier to save power, and run them on smaller and lower cost MCUs. Our inference engine executes the AI model as-is and does no additional quantization, no binarization, no pruning, and no model compression. There is no accuracy loss. It simply runs faster and in a smaller memory footprint.

Here’s the overview of the MLPerf Tiny results for an Arm Cortex-M4 MCU:

Official MLPerf Tiny 1.0 inference results for an Arm Cortex-M4 at 120MHz (STM32L4R5ZIT6U).

Compared to TensorFlow Lite for Microcontrollers with Arm’s CMSIS-NN optimized kernels, we run 1.74x faster:

MLPerf Tiny inference latency for the Arm Cortex-M4 at 120MHz inside the STM32L4R9 MCU.

But not only latency is important. Since MCUs often have very limited memory on board it’s important that the inference engine uses minimal memory while executing the neural network. MLPerf Tiny does not report numbers for memory usage, but here are the memory savings we can achieve on the benchmarks compared to TensorFlow Lite for Microcontrollers. On average we use less than half the memory:

MLPerf Tiny memory usage is lower by a factor of 2.04x.

MLPerf doesn’t report code size, so again we compare against TensorFlow Lite for Microcontrollers. The table below shows that we reduce code size on average by a factor of 2.18x. Using our inference engine means you can use MCUs that have significantly smaller flash size.

MLPerf Tiny code size is lower by a factor of 2.18x.

Want to see how fast your models can run? You can submit them for free on our Plumerai Benchmark service. We email you the results in minutes.

Are you deploying AI on microcontrollers? Let’s talk.

The post MLPerf Tiny 1.0 Confirms: Plumerai’s Inference Engine is Again the World’s Fastest appeared first on Edge AI and Vision Alliance.

Ever wondered how a touch-sensitive screen works? The answer is capacitive sensing, which can detect the presence of your finger through the protective glass sheet and doesn’t require any pressure. However, this perhaps under-appreciated technology is not limited to touch-sensitive displays: it is increasingly being utilized for many other established and emerging applications, ranging from automotive controls to leak sensing and even eye tracking.

How Does Capacitive Sensing Work?

Understanding the operating mechanism of capacitive sensors is key to understanding the applications to which they are well suited. Briefly, the sensors work via capacitive coupling to detect objects that have dielectric properties substantially different from air – fingers, being mainly water, work well. Since electric fields can travel through dielectric materials such as air or glass, there is no requirement for electrical contact. However, depending on sensitivity and readout methodology, the object being sensed often needs to be close to the patterned conductive layer, which is why it’s often difficult to use touchscreens or other capacitive controls while wearing gloves.

Opaque Touch Sensing

Although capacitive sensing is widely associated with touch-sensitive displays, it is also increasingly used on opaque surfaces. These include car steering wheel controls and touch panels – the conductive material used to make the capacitive sensor is sometimes a transparent metal mesh so that the controls can be backlit. Capacitive touch sensing is also an essential feature of in-mold electronics, in which conductive traces and LEDs are first printed/mounted on a flat substrate, then thermoformed, and finally, injection molded.

Moving from conventional mechanical switches to capacitive switches brings multiple benefits: fewer components, a reduction in materials used, an interface that can be wiped clean (since there are no grooves between buttons), and a sleek modern aesthetic. However, some drivers have expressed concerns that capacitive sensors are less tactile and hence more difficult to use while driving – integrating haptic feedback is a possible solution.

Leak Sensing

As anyone who has tried to operate their smartphone in the rain will attest, capacitive sensors are extremely sensitive to water due to its high dielectric constant. While this might be undesirable on a touch-sensitive display, it does enable capacitive sensors to be used for leak detection.

By utilizing low-cost materials such as carbon inks printed onto plastic using conventional methods, it is possible to produce rolls of printed/flexible sensors for just a few dollars. These can then be placed around plumbing, under kitchen appliances, and even along pipes – a small control unit housing a circuit board with a micro-controller and batteries measures the changing capacitance. If a leak occurs, a change in capacitance is detected, and a signal is sent to an app to provide a notification. This capability is proving compelling to insurance companies since limited capital investment would substantially limit the property damage caused by any leak.

Eye-Tracking

Perhaps the most surprising emerging application of capacitive sensors is an early-stage technology that provides eye-tracking for virtual/augmented reality glasses. Based on multiwalled carbon nanotubes (MWCNTs) embedded within paper to produce a conductive material with a very large surface area and hence high sensitivity to changes in the dielectric environment. Utilizing three sensors enables projected capacitance, in which the relative response of each sensor enables the location of the change in dielectric properties of the target object to be determined via triangulation. Compared to competing approaches such as machine vision, capacitive sensing promises a much shorter processing time and hence a more responsive system.

Technological and commercial readiness of capacitive sensing technologies and applications.

Emerging Printed & Flexible Sensor Technologies

IDTechEx’s report “Printed and Flexible Sensors 2022-2032: Technologies, Players, Markets” assesses the technologies and market landscape across 9 distinct printed sensor technologies: piezoresistive sensors, piezoelectric sensors, printed photodetectors, temperature sensors, strain sensors, capacitive touch sensors, gas sensors, biological sensors, wearable electrodes. The report draws on detailed profiles of over 50 companies, the majority based on interviews, to evaluate each of these printed sensor categories in considerable detail, evaluating the different technologies and the challenges to adoption. IDTechEx develop 10-year market forecasts for each technology and application sector, resulting in 33 individual forecast segments delineated by revenue and printed sensor area.

Dr Matthew Dyson
Senior Technology Analyst, IDTechEx

The post Untapped Potential for Capacitive Sensors? appeared first on Edge AI and Vision Alliance.

LETTER FROM THE EDITOR

Dear Colleague, Registration is now open for the 2023 Embedded Vision Summit, coming up May 22-25, 2023 in Santa Clara, California! The Summit is the premier conference and tradeshow for innovators incorporating computer vision and visual or perceptual AI in products. The program is designed to cover the most important technical and business aspects of practical computer vision, deep learning and perceptual AI. Register before December 31 and you can save 35%—the best price you’ll ever be able to get on the Summit! We’re currently creating the presentation program for the Summit, and we invite you to share your expertise and experience with peers at this premier event! To learn more about the topics we’re focusing on this year and to submit your idea for a presentation, check out the Summit Call for Proposals or contact us at present@edge-ai-vision.com. We will be accepting session proposals through December 7, 2022, but space is limited, so submit soon! On Tuesday, February 7, 2023 at 9 am PT, Hailo will deliver the free webinar “Implementing Advanced AI-based Analytics for Video Management Systems” in partnership with the Edge AI and Vision Alliance. The term video management systems (VMS) encompasses all software elements involved in handling multiple video channels at scale in a variety of applications: surveillance, smart cities, retail, etc. Recent developments in VMS solutions, such as the migration to high-resolution smart cameras as well as the introduction of advanced video analytics software, require VMS providers to integrate high-performance computing and artificial intelligence (AI) capabilities into their products. Hailo supplies top-performing AI accelerators to enable accurate, real-time analytics for detection and search on cameras and other edge devices, even at high resolutions and frame rates. The company’s comprehensive software suite supports a wide range of platforms, operating systems, and neural network models, enabling easy integration into all video management systems. The result is higher accuracy and lower latency with lower cost of ownership–all of which translate into enhanced safety and better service in surveillance and smart city applications, along with improved customer service in retail applications. In this webinar, Avi Baum, CTO, and Mark Grobman, Machine Learning CTO, both of Hailo, will explore various VMS system configurations where analytics play a role. The presenters will describe how to implement the enhanced capabilities enabled by Hailo processors and software, along with detailing performance metrics. Baum and Grobman will also address emerging trends in machine learning related to video analytics, which promise to unlock even more capabilities. A question-and-answer session will follow the presentation. For more information and to register, please see the event page. Brian Dipert Editor-In-Chief, Edge AI and Vision Alliance

DESIGNING ROBUST EDGE IMPLEMENTATIONS

Seamless Deployment of Multimedia and Machine Learning Applications at the Edge There has been an explosion of opportunities for edge compute solutions across the internet of things. This growth in opportunities and the diversity of applications is leading to fragmentation in the IoT space both in hardware and software, which creates challenges for developers. In addition, customers and developers are facing challenges in efficient data management and optimized application deployment on embedded edge platforms. In this presentation, Megha Daga, Senior Director of Product Management for AIoT at Qualcomm, introduces the Qualcomm Intelligent Multimedia SDK, which empowers developers to tackle these challenges and deploy edge compute applications in a scalable, flexible and optimized way. The Qualcomm Intelligent Multimedia SDK easily decodes and organizes sensor data and executes applications efficiently on edge platforms. Developing Sustainable Edge AI Products for the Real World For edge AI systems to succeed in the real world, they must be adaptable. They must adapt as customer requirements change, as their environments evolve and as AI techniques advance. Creating adaptable edge AI systems requires a holistic approach that encompasses efficient deep neural network inference and tools and techniques that enable effective testing, deployment and management of the edge AI device over its entire life cycle. This presentation explores this holistic approach and shows how Blaize enables developers to create adaptable, sustainable products for a diverse range of applications. Barrie Mullins, former Senior Director of Product Marketing at Blaize, introduces the Blaize Pathfinder and Blaize Xplorer accelerators, Blaize AI Studio (a code-free development environment), and Blaize Picasso SDK (a traditional software development kit for experienced engineers who want to get under the hood and customize their AI solutions). He also shows how these innovative offerings help quickly take edge AI products from idea to production.

SILICON, SOFTWARE AND SERVICES FOR RAPID DEPLOYMENTS

Converging AI, Media and Computing in a Compact and Open Platform As a system integrator, solution provider or AI developer, you need to run your AI applications efficiently at the edge with sufficient throughput. Does your edge device run either generic computing or deep learning inferencing, but not both? The Intel Video AI Box with Core CPU and integrated Xe LP graphics offers a compact solution to run video AI analytics at the edge with the support to orchestrate AI applications and workloads in cloud-to-edge deployments. In this presentation, you’ll learn about Intel’s new platform, comprising an Intel CPU with integrated graphics and the Edge AI Box for Video Analytics software package, and how it enables developing cutting-edge video solutions faster. Richard Chuang, Principal AI Engineer at Intel, also explores EFLOW enablement on the platform, which allows Windows-based business applications to run rich Linux AI workload containers with Azure cloud connections for scalable deployments. A Platform Approach to Developing Networked Visual AI Systems Connected cameras are becoming ubiquitous. Coupled with CV and ML, they enable a growing range of applications that monitor people, facilities and other objects and environments. But creating robust, scalable applications requires more than connected cameras and algorithms. For example, applications require robust video storage management, including the ability to manage limited bandwidth and recover from hardware failures. In addition, applications often must be able to discover and interact with a variety of camera and stream types on a network. Applications also typically require media servers and clients that can run on mobile, desktop, server and cloud. And they need the ability to integrate with a variety of software stacks and ecosystems. In this presentation, Nathan Wheeler, Chairman and CEO, and Tony Luce, Vice President of Product Marketing, both of Network Optix, introduce the Nx Meta intelligent video platform, which enables developers to quickly create cross-platform networked visual AI solutions that incorporate device and stream discovery and interoperability, robust storage management and extensibility. They also show application examples.

UPCOMING INDUSTRY EVENTS

How to Successfully Deploy Deep Learning Models on Edge Devices – Deci Webinar: December 13, 2022, 9:00 am PT Implementing Advanced AI-based Analytics for Video Management Systems – Hailo Webinar: February 7, 2023, 9:00 am PT Embedded Vision Summit: May 22-25, 2023, Santa Clara, California More Events

FEATURED NEWS

AiM Future Introduces the NeuroMosAIc Family of Edge AI Accelerators and Software Studio Qualcomm Launches the Snapdragon AR2, Designed to Revolutionize AR Glasses CEVA Introduces a Voice User Interface Solution for TI SimpleLink Wi-Fi Wireless MCUs e-con Systems Launches Its First Full HD Global Shutter FPD-LINK III Camera STMicroelectronics Extends Embedded AI Solutions with Advanced Features to Ease Machine-Learning Development More News

EDGE AI AND VISION PRODUCT OF THE YEAR WINNER SHOWCASE

Grabango Checkout-free Technology (Best Enterprise Edge AI End Product) Grabango’s Checkout-free Technology is the 2022 Edge AI and Vision Product of the Year Award winner in the Enterprise Edge AI End Product category. Grabango’s system for existing, top-tier grocery and convenience stores is pure computer vision (CV), based on machine learning with targeted retrainings (AI). With a handful of other startups boasting a basic proof-of-concept here and there, Grabango has leapt ahead to announce 5 mature stores with Giant Eagle, 6 stores with Circle K, and 10 stores with bp, with additional announcements pending from two more convenience chains, and two major grocery chains. All of these stores are fast retrofits, serving the same customers as before installation. Since going live, Grabango remains the only provider delivering high-volume operations, in true retrofit settings, with exceptionally accurate receipts. Please see here for more information on Grabango’s Checkout-free Technology. The Edge AI and Vision Product of the Year Awards celebrate the innovation of the industry’s leading companies that are developing and enabling the next generation of edge AI and computer vision products. Winning a Product of the Year award recognizes a company’s leadership in edge AI and computer vision as evaluated by independent industry experts. The Edge AI and Vision Alliance is now accepting applications for the 2023 Awards competition; for more information and to enter, please see the program page.

The post Edge AI and Vision Insights: November 23, 2022 Edition appeared first on Edge AI and Vision Alliance.

Additional funding enables Israeli startup to accelerate adoption of AI based vision enhancement

Jerusalem, Israel, November 22, 2022 – Visionary.ai, a developer of AI for real time video enhancement, today announced that it has received a multi-million-dollar investment from National Grid Partners (NGP), the venture investment and innovation arm of National Grid plc., one of world’s largest investor-owned energy companies.

Lisa Lambert, Chief Technology and Innovation Officer at National Grid and Founder and President of National Grid Partners said: “Electricity grids by their nature incorporate many geographically distributed assets that need to be monitored for safety, security and signs of damage or malfunction. The transition to renewable energy sources drives this distribution of assets even further. Vision is critical to monitoring these assets, and cameras need to be dependable in all lighting and weather conditions. The technology that Visionary.ai is developing can help with the need for increased security of electricity supply and the battle against climate change.”

One of the principal benefits of Visionary.ai’s technology is that it enables cameras to operate in extremely low lighting, in real time, in colour and while maintaining high picture sharpness. In addition, Visionary.ai has developed a novel software image signal processor, which can adapt image processing to the needs of particular applications. Today’s hardware-based solutions cannot offer this level of flexibility.

Oren Debbi, CEO and Co-founder of Visionary.ai, commented: ”We’re experiencing a surge of interest in our technology from many industries, including from several of the world’s most renowned technology companies. The need for better real-time video clearly resonates across both human and machine vision applications. Our engineering team has made incredible progress over the last 18 months, and our first products are now production-ready. This investment from National Grid Partners will enable us to build on this foundation and expand the range of solutions we can offer our customers.”

With its investment in Visionary.ai, NGP has backed a total of 40 startups dedicated to combating climate change across a range of applications, from cybersecurity to customer-focused solutions.

About Visionary.ai

Visionary.ai empowers cameras to achieve the next generation of image quality. Their de-noiser and software-based ISP (Image Signal Processor) leverage AI on the Edge to achieve market-leading performance for a wide range of imaging tasks. Their technology has applications across industries, including consumer electronics, drones, robotics, mobility, and medical imaging. For more information, visit www.visionary.ai.

About National Grid Partners

National Grid Partners (NGP) is the venture investment and innovation arm of National Grid plc., one of the largest investor-owned energy companies in the world. NGP invests for strategic and financial impact and leads companywide disruptive innovation efforts. The organization provides a multi- functional approach to building startups, including innovation (new business creation), pathfinding and incubation, corporate venture capital, business development and venture acceleration. NGP is headquartered in Silicon Valley and has offices in Boston, London, and New York. Visit ngpartners.com or follow us at www.twitter.com/ngpartners_ and www.linkedin.com/showcase/national-grid-partners.

The post Visionary.ai Receives Investment from National Grid Partners appeared first on Edge AI and Vision Alliance.

Scottsdale, Arizona and Mississauga, Ontario – Nov 21, 2022. Piera Systems Inc. (“Piera”) today announced that it has been named a CES® 2023 Innovation Awards Honoree for its ground-breaking Canāree Air Quality Monitor family of products. This year’s CES Innovation Awards program received a record high number of over 2100 submissions. The announcement was made ahead of CES 2023, the world’s most influential technology event, happening Jan. 5-8 in Las Vegas, NV.

The highly accurate Canāree family of Air Quality Monitors (AQM) are built using Piera’s patented Intelligent Particle Sensor (“IPS”) technology, combining ease-of-use and rapid deployment with unparalleled particle measurement and environmental sensing capabilities. Canāree AQMs are deployed indoors at schools, hospitals, hotels, and offices where they provide continuous, real-time air monitoring and vaping detection.

“The pandemic put the focus on monitoring indoor air quality, and  customers worldwide responded by deploying Canāree AQMs throughout their facilities and institutions,” said  Vin Ratford, CEO of Piera Systems..  “This award recognizes the importance of Canāree AQM’s many ground-breaking innovations, including sensor miniaturization, fine particulate sensing to 0.1 micron, and easy-to-use interfaces with IT systems.”

“Aruba partnered with Piera Systems to integrate Canāree AQMs with Aruba’s Wi-Fi access points so new and existing customers can monitor air quality without installing additional networking infrastructure,” said Tim Vanevenhoven, Director of Partnerships at Aruba, a Hewlett Packard Enterprise company.  “A customer need only plug a Canāree AQM into an access point’s USB port, enter the IP address of the data collector, and they are done.  The simplicity of this design, and the sensitivity of the sensor, set the Canāree AQM apart from its competitors and make it a deserving recipient of the CES award.”

“Vaping and smoking constitutes a serious threat to children in schools across America and we at EMT are working hard to provide schools with innovative solutions to improve health outcomes,” said Karl Seebacher, COO of Extreme Microbial Technologies, Inc. “We are proud to be working with the award-winning Piera team and their highly effective Canāree family of air quality monitors with built-in vape-smoke detection capabilities.”

Indoor air quality has direct and substantial impacts on human health, from loss of productivity to causing serious diseases including lung cancer, Alzheimer’s, and cardiovascular diseases. The smallest particles, less than 1 micron in size, are the most dangerous as they are readily absorbed into the bloodstream. Piera’s Canāree Air Quality Monitors and IPS technology are independently certified to accurately detect these sub-micron particles, a feat not possible with most other sensors which only estimate air quality.

Visit us at CES 2023 Booth # 55429 at Venetian Expo Hall D to learn more. Schedule an appointment at CES here.

About Piera Systems, Inc.

Piera Systems Inc. is on a mission to make air quality measurement accurate, simple, inexpensive, and pervasive, enabling a major improvement in the health of all humans. Piera has developed a family of ‘Intelligent Particle Sensors’ and Canāree Air Quality Monitors utilizing a breakthrough custom processor to detect particulate matter (PM), a major component of air pollution. Unlike existing low-cost PM sensors and the AQMs that rely on them, IPS and Canāree have superior accuracy over a wider range, including harmful ‘very fine particles’, smaller than 1.0 micron, and reports particle size and count in real-time at low power. IPS and Canāree data is analyzed using AI/ML algorithms to identify and classify pollution sources such as vape, cigarette smoke, cooking, etc. IPS sensors and Canāree air quality monitors answer the question, ‘What’s In Your Air?’

Headquartered in Canada, more information on Piera Systems air quality solutions can be found at https://www.pierasystems.com

The post Piera Systems Named CES 2023 Innovation Awards Honoree appeared first on Edge AI and Vision Alliance.

This blog post was originally published at e-con Systems’ website. It is reprinted here with the permission of e-con Systems.

An IR-cut filter is a mechanical shutter that blocks or delivers light in the IR spectrum to enable true color reproduction. Learn how an IR-cut filter works and why your embedded vision application might need it.

In recent years, day/night video surveillance has expanded beyond the traditional practice of monitoring only critical infrastructure. Generally, systems such as CMOS cameras and CCD cameras capture accurate images in daylight. Their sensors can detect near-infrared light that is invisible to the human eye. While this can be an important feature for night-vision recording –the infrared light distorts the colors of recorded images during daytime hours.

Before deeply discussing the IR-cut filter, let’s take a quick look at what “infrared” means.

What is Infrared?

Infrared (IR) is also known as radiant energy. It is electromagnetic radiation that has longer wavelengths than visible light. Normally, human eyes can view light with a wavelength ranging between 320nm – 760nm. The IR ranges between 700nm to 1000nm on the electromagnetic spectrum.

It is far more than what a human eye can view – as only NIR cameras can capture images in this wavelength range.

What is an IR-cut filter?

IR-cut filter is a mechanical shutter design that blocks or delivers the IR – providing a high-quality image with true color reproduction – irrespective of day or night.

Unlike human eyes, camera sensors can detect near IR light outside the range of the visible spectrum. To make the image more similar to what humans can view, most OEM cameras are fitted with an IR-cut filter which allows only the visible light to pass through – thereby reflecting the unwanted IR.

Below are sample images with and without an IR-cut filter:

Figure 1 – comparison of images with and without IR-cut filter

How does an IR-cut filter work?

Generally, an IR-cut filter is placed between the image sensors and the lens as shown below.

Figure 2 – placement of IR-cut filter in a camera

This filter is controlled by a motor or with the help of an electromagnet. When the IR filter is turned on during the daytime, it helps to block IR – letting only the visible light pass through. This process is known as True Day Night (TDN) because it delivers true color images during the day and black and white or night vision images at night. It ensures that the image colors are not disturbed and generates a true representation of colors – as seen naturally by the human eye.

When the IR filter is turned off during the night or in low-light conditions, it helps IR and other forms of light to reach the CCD/CMOS sensor. The image sensor, in turn, absorbs enough light – and the camera turns to black-and-white mode, which is more sensitive to IR light.

An IR-cut filter can be used in devices with bright incandescent light bulbs to prevent unwanted heating.

If you are interested in learning more about NIR imaging, please have a look at the article What is NIR imaging and how do NIR cameras work?

Now, let us discuss in detail the advantages of using an IR-cut filter.

Three reasons to use an IR-cut filter

1. To avoid color distortion – Generally, IR light leads to color distortion during the day. To solve this issue, an IR-cut filter is used to keep the IR light disturbance out of the image sensor during the day. When the light falls below a certain level, the filter automatically swivels out of the way so that the IR light does hit the image sensor. Also, the camera switches to black-and-white mode to use the IR light optimally.
2. To achieve realistic colors – The IR-cut filter in a color camera can achieve realistic colors in white light. In general, the color spectrum viewed by the human eye is quite limited compared to the spectrum viewed by a CCD camera. Especially, the difference in sensitivity is significant in the near IR region of the spectrum. If an IR-cut filter is not used, a huge amount of infrared light will appear – resulting in strange colors.
3. To achieve color correction in lenses – In many cases, it is difficult to design imaging optics covering both the visible spectrum and the near IR spectrum. Therefore, many lenses have different depths of focus for the visible and the IR spectrum. In such cases, the IR-cut filter cuts away a significant amount of the collected light and achieves color correction in lenses. Please have a look at the below image to understand the difference between a standard lens output and a color corrected lens output:

Figure 3 – comparison of standard lens and IR-corrected lens

Embedded vision applications that rely on IR-cut Filter

Some of the embedded vision applications where an IR-cut filter is used are discussed below:

Video Conferencing

In a video conferencing camera, the lenses should ensure high image quality and a large field of view (FOV) while maintaining a very low distortion at the same time. So, an IR-cut filter helps to produce accurate color images with color CCD or CMOS cameras. The filter blocks the transmission of IR light while passing visible light. It can be performed by absorption or reflection optical techniques. The absorptive filters are made with special optical glass to absorb the near IR radiation. In comparison, the reflection type filters are short-pass interference filters that reflect IR light with high efficiency.

Automatic Number Plate Recognition (ANPR)

An ANPR system is expected to accurately recognize a vehicle’s plate number based on an image captured by an ANPR camera. It can be performed through the combination of steps, such as capturing the real image, locating and identifying individual characters on the plate, and recognizing the optical character. During the nighttime, a color camera with an IR-cut filter can capture clear images. Therefore, it helps identify the individual symbol image on the plate, the number of characters on the license plate, and the text luminance level (dark text on a light background or light text on a dark background).

To learn how to do number plate recognition using e-con’s global shutter camera and Raspberry Pi4, please have a look at the article Automatic number plate recognition using e-con Systems’ global shutter camera and Raspberry Pi4.

Smart Surveillance System

Nowadays, most of the CCTV cameras are equipped with IR illumination. The IR-cut filter is placed in front of the cameras with a night vision sensor. The human eye can’t notice the IR illumination light, but the camera sensor can view it perfectly. In monochrome mode, if the picture quality is good, the IR radiation has a major effect on the image in the daytime. Also, smart surveillance cameras with night vision are equipped with an IR cut filter to protect against rays, thus achieving an accurate image.

NIR cameras from e-con Systems

With more than 18 years of experience in the camera technology space, e-con Systems has developed a wide portfolio of NIR cameras. PFB the comprehensive list of NIR cameras from e-con Systems.

• See3CAM_CU135M – 4K Monochrome USB 3.1 Gen 1 Camera
• See3CAM_CU55M – 5MP Monochrome USB NIR Camera
• See3CAM_CU27 – Sony® Starvis IMX462 Ultra Low Light USB 3.1 Gen 1 Camera
• See3CAM_20CUG – 2MP OV2311 Monochrome Global Shutter Camera
• See3CAM_CU30 – 3.4 MP Low Light USB Camera
• Conversa® – Low Light USB Camera with Microphone

Hope this piece of content threw some light into how an IR-cut filter works. If you have any further queries on the topic, please leave a comment. We shall get back to you as soon as possible.

If you are interested in integrating embedded cameras into your products, kindly write to us at camerasolutions@e-consystems.com. You could also visit our Camera Selector to get a complete view of e-con Systems’ camera portfolio.

Prabu Kumar
Chief Technology Officer and Head of Camera Products, e-con Systems

The post What is an IR-cut Filter, and Why Do Embedded Vision Applications Need It? appeared first on Edge AI and Vision Alliance.

London, England – 21st November 2022 – Imagination Technologies (“Imagination”) has announced a new partnership with MulticoreWare Inc. (“MulticoreWare”) as a preferred software development partner to optimise algorithms on Imagination GPUs. MulticoreWare Inc. is a software solutions and services company that specialises in optimising computer vision, sensor data processing, and AI applications on a variety of low-power, embedded, and heterogeneous systems.

Imagination collaborated with MulticoreWare to showcase accelerated stereo block matching algorithm performance on the UNISOC-T710 development platform. MulticoreWare leveraged its expertise in OpenCL by re-implementing the stereo BM algorithm to improve compute resource utilisation and memory optimisation to deliver greater than ~50x performance gain on the Imagination GPUs.

Vish Rajalingam, Vice President & Co-GM – Autonomous Vehicle & Automotive BU, MulticoreWare, says: “Power-efficient GPUs are now essential for all computer vision, artificial intelligence, and sensing applications. We are excited to be partnering with Imagination to enable their customers in implementing algorithm optimisations and software accelerations on Imagination’s PowerVR GPUs, with a planned roadmap to include RISC-V software acceleration in future.”

Gilberto Rodriguez, Director of Product Management, Imagination, says: “Imagination’s GPUs can be used to deploy computer vision tasks as well as machine learning acceleration easily and efficiently on edge devices. MulticoreWare is utilising our IP to its actual potential for general-purpose GPU applications. By working together, we can offer our customers a truly optimised PowerVR deployment experience.”

The StereoBM algorithm was chosen for optimisation based on customer interest. MulticoreWare analysed the CPU performance to identify bottlenecks. The goal was to achieve maximum GPU parallelism, which was enabled by efficient implementation of internal register usage and the configuration of an appropriate global workgroup size that is adaptive to the image resolution.

Imagination’s GM9446 GPU memory layout was used to calculate the adaptive global workgroup size. In addition to the computational optimisation, the algorithm parameters were modified to obtain greater accuracy combined with a very-high performant implementation on the GPUs compared to the CPUs on the same platform. The CPU time in one configuration was 54.25ms, whereas the MCW implementation was 0.78ms, a ~70x gain in performance.

About Imagination Technologies

Imagination is a UK-based company that creates silicon and software IP (intellectual property) designed to give its customers an edge in competitive global technology markets. Its GPU, CPU and AI technologies enable outstanding power, performance, and area (PPA), fast time-to-market and lower total cost of ownership. Products based on Imagination IP are used by billions of people across the globe in their smartphones, cars, homes, and workplaces.

See www.imaginationtech.com.

Follow Imagination on Twitter, YouTube, LinkedIn, Facebook and Blog.

About MultiCoreWare

MulticoreWare delivers software IP Solutions and Engineering Services serving a wide group of customers with Compilers & Toolchains, Libraries for SDK, Video codec and AI analytics solutions using various vision & non-vision (Radar, LiDAR, IMU, GPS, etc.) sensors on various Heterogeneous Computing Platforms. Our solutions are used in Automotive (ADAS/AD), Surveillance, Defense, Medical Imaging, IoT, Retail, Logistics, Industrial, Robotics, Smart City. More information at www.multicorewareinc.com.

The post Imagination Partners with MulticoreWare Inc. and Demonstrates Superior Performance of Its GPUs appeared first on Edge AI and Vision Alliance.

This video was originally published at Intel’s YouTube channel. It is reprinted here with the permission of Intel.

In this episode, Ragesh and Steph talk about OpenVINO Integration with Tensorflow which is used for speeding up inferencing of a TensorFlow based real-time application at the edge. Ragesh pulls and runs a docker image on his Intel® Core i7 CPU to showcase what it can do, uses a notebook that covers an object detection use case and demonstrates acceleration of YOLOv4 model, trained on coco dataset. In the end, Steph mentions 4 short training videos for you on Container Playground from Intel® DevCloud: Edge Workloads.

OpenVINO integration with TensorFlow GitHub Homepage: https://bit.ly/3QHNxzy

Validated models supported by OpenVINO integration with TensorFlow: https://bit.ly/3A5dSlr

Dockerfiles and prebuilt images: https://bit.ly/3neu4co

Sample applications and tutorials on Intel® DevCloud: https://intel.ly/3OgVEBG

Object Detection Jupyter Notebook on GitHub: https://bit.ly/3OznOaG

How OpenVINO integration with TensorFlow really works under the hood?: https://intel.ly/3xO1YJK

Video playlist to learn about Intel® DevCloud container playground: https://bit.ly/39NrK8Z

Performance improvement of SSD & Faster-RCNN object detection models are in progress and will be addressed in the upcoming release. Stay tuned to this page: https://bit.ly/3njwg2j

Please submit your questions, feature requests and bug reports via GitHub issues: https://bit.ly/3ni3teu

About Intel Software

The Intel® Developer Zone encourages and supports software developers that are developing applications for Intel hardware and software products. The Intel Software YouTube channel is a place to learn tips and tricks, get the latest news, watch product demos from both Intel, and our many partners across multiple fields. You’ll find videos covering the topics listed below, and to learn more, you can follow the links provided!

Connect with Intel Software:

• Visit the Intel Software Website: https://intel.ly/2KeP1hD
• Like Intel Software on Facebook: http://bit.ly/2z8MPFF
• Follow Intel Software on Twitter: http://bit.ly/2zahGSn
• Intel Software Github: http://bit.ly/2zaih6z
• Intel Developer Zone on LinkedIn: http://bit.ly/2z979qs
• Intel Developer Zone on Instagram: http://bit.ly/2z9Xsby
• Intel Game Developer on Twitch: http://bit.ly/2BkNshu

The post Magical OpenVINO Integration with TensorFlow appeared first on Edge AI and Vision Alliance.

This blog post was originally published at Qualcomm’s website. It is reprinted here with the permission of Qualcomm.

Qualcomm AI Research establishes firsts in AI across research and proof-of-concept.

At Qualcomm, we have a culture of innovation. We take pride in researching and developing fundamental technologies that will change the world. It’s also in our DNA to prototype these ideas and ensure that they can be deployable and scalable under real-world conditions. We have a saying that if you haven’t tested it, it doesn’t work — so we seed ideas, build prototype systems, and seeing if they are worth the effort to commercialize.

Accelerating the pipeline from AI research to commercialization has been daunting because deploying the technology in the real world faces many challenges beyond a lab setting where theoretical work is developed. Qualcomm AI Research has taken on the task of not only generating novel AI research but also being first to demonstrate proof-of-concepts on commercial devices, paving the path for technology to scale in the real world. In this blog post, I’ll cover both our novel AI research and our full stack AI research that has enabled first-ever proof-of-concept demonstrations on commercial mobile devices.

Innovating across the entire spectrum of AI

At Qualcomm AI Research, we do novel research that covers the entire spectrum of AI R&D to push the boundaries of what is possible with machine learning. Our research directions can be classified into three areas:

• Fundamental research looks the furthest into the future and is generally more foundational. Examples include Group-equivariant CNNs, or G-CNNs, and quantum AI.
• Platform research focuses on achieving the best power, performance, and latency when running AI models on hardware by employing techniques, such as quantization and compression to models, as well as enhancing compiler algorithms.
• Applied research is where we leverage both fundamental and platform advances in AI to achieve state-of-the-art in specific use cases, like video recognition and prediction.

We cover the entire spectrum of AI research and development.

Deploying our innovations through full stack AI research

To bring these innovations to the masses, we need to dive deeper into full-stack AI research. This means researching optimization techniques across the application, the neural network model, the algorithms, the software, and the hardware, as well as working across disciplines within the company (and sometimes partnering with other companies). There are many layers of hardware and software that can be tuned and optimized to squeeze out every bit of performance at the lowest energy consumption. Once we have done full-stack optimization and proven out the technology, we can enable the ecosystem toward rapid commercialization at scale.

Our full stack research is essential to making fundamental research deployable.

Driving AI firsts in research and proof-of-concepts

Our purposeful innovation has led to many AI firsts – these are firsts both in novel research and in proof-of-concepts running on target devices due to our full-stack optimizations.

Our AI firsts in both research and proof-of-concept.

I’ll briefly describe a few examples, but please tune into my webinar and download the presentation for a more detailed explanation across all other examples.

Model quantization

Although AI is bringing tremendous new capabilities as models sizes increase, it often comes at the cost of increased computation, energy consumption, and latency. Quantization reduces the precision of weight parameters and activation node computations for an AI model. If you can maintain model accuracy, then quantization offers compelling benefits, such as improved power efficiency, increased performance, and lower memory usage.

We’ve invented the best techniques for fast deployment of 8-bit quantization while maintaining model accuracy, such as data-free quantization, AdaRound, and transformer quantization. We’re also driving the industry toward integer inference by open sourcing the research through the best power efficiency toolkit in the industry, the AI Model Efficiency Toolkit (AIMET).

We’ve invented the best techniques for fast deployment of 8-bit quantization.

Federated learning

The need for intelligent, personalized experiences powered by AI is ever-growing. Devices are producing more and more data that could help improve our AI experiences. However, how can we preserve privacy while taking advantage of this insightful data? Federated learning addresses this through on-device learning and only aggregating model updates in a central location. To improve federated learning, we’ve invented methods for combining differential privacy and compression. Our Differentially Private Relative Entropy Coding (DP-REC) method preserves privacy and enables high compression for a drastic reduction in communications. We also demonstrated the first federated learning end-to-end research software framework for the mobile ecosystem, showing a voice user verification example.

Neural video compression

The demand for improved data compression continues to grow since the scale of video being created and consumed is massive. AI-based compression has compelling benefits over conventional algorithms and codecs. We invented an instance-adaptive compression method that can customize the codec for the specific video rather than using a generic codec. This research achieved state-of-the art results, including a 24 percent BD-rate savings over the leading neural codec by Google. We also demonstrated the first HD neural video decoder on mobile. Through full stack research, like a novel parallel entropy decoding algorithm, quantization, and hardware optimization, we can run the neural decoders in real time for HD videos.

Super resolution

Super resolution creates a high-resolution image with superior visual quality from a low-resolution image. This capability is very useful across many applications, from gaming and photography to XR and autonomous driving. AI-based super resolution offers improved visual quality over traditional methods. Until now, on-device deployment that meets real-time performance, latency, and power requirements has not been feasible. With our full-stack optimization, which included a hardware-optimized custom model architecture and quantization, we achieved the first 4K super resolution at 100+ frames per second on a mobile device.

2X super resolution running on a mobile device at more than 100 frames per second.

AI for wireless

AI applied to wireless can offer many enhanced or new capabilities, such as improved communications or radio frequency (RF) sensing. Neural networks can be used to augment traditional algorithms in wireless communications, thereby achieving increased performance – we call this neural augmentation. We can combine the inductive bias from wireless domain knowledge with neural networks to address interpretability, out-of-domain generalization, and achieve better sample complexity. We invented neural augmentation to enhance physical layer algorithms. Two examples of neural augmentation are hypernetwork Kalman filtering and generative channel modeling. For RF sensing, we showcased the first weakly supervised method for real-world passive RF sensing at MWC 2021 and NeurIPS 2021. Our new machine-learning based method works on large floor plans and only requires weakly labeled training data and a floor plan.

Innovating for our next AI firsts

Our purposeful innovation is continuing, and we expect many more AI firsts across a diverse range of topics as we work on novel research and full stack optimizations.

Our innovation cycle is continuing, as well as our next potential AI future AI firsts.

We will continue to solve system and feasibility challenges with full-stack optimizations to quickly move our research toward commercialization. If you are excited about solving big problems with cutting-edge AI technology and improving the lives of billions of people, come join our team.

Dr. Jilei Hou
Vice President, Engineering, Qualcomm Technologies

The post Leading Research Delivers AI Firsts appeared first on Edge AI and Vision Alliance.

This blog post was originally published at Hasty’s website. It is reprinted here with the permission of Hasty.

The previous post featuring our results on the PASCAL VOC 2012 cleansing project was successful. We attracted your attention and got you talking about the Data-centric AI development approach on Reddit and the Slack communities. There was some criticism, which was great, but also, there seemed to be a lot of interest in the idea of a cleaner, better PASCAL. Nevertheless, the obtained results were just the tip of the iceberg, as the project still had an expansive room for experimentation and improvements. So, in this blog post, we will unveil what was going on for the past two months and share our plans for the future.

What’s new?

If you want a quick recap of our initial results, you can always skim through the original blog post. Still, we suggest you give your eyes a bit of the rest and take a look at our YouTube video, where we explain the project’s workflow and provide some interesting insights about it.

Over the past weeks, we explored the field a bit more while trying not to rush for a result but to be more systematic and accurate. Overall, we ran two massive experiments:

• We trained custom models on the initial and updated PASCAL without any LR scheduler;
• We trained custom models on the initial and updated PASCAL with the StepLR scheduler.

No scheduler

After posting our initial results on Reddit, we got many valuable comments from the community. It helped us identify that the initial improvements might have happened because of the ReduceLROnPlateau scheduler that, for some reason, worked only for the model trained on cleansed PASCAL. So, we decided to see what happens if we do not use any scheduler.

The experiment’s settings (hyperparameters, dataset partitioning, etc.) were the same as in the first iteration of the PASCAL cleansing project. To remind you as hyperparameters, we used the following options:

• Faster R-CNN architecture with ResNet101 FPN as a backbone;
• R101-FPN COCO weights were used for model initialization;
• Blur, Horizontal Flip, Random Crop, Rotate, and Color Jitter as augmentations;
• AdamW was the solver;
• There was no scheduler;
• Just like in every other OD task, a combination of losses was used (RPN Bounding Box loss, RPN Classification loss, final Bounding Box regression loss, and final Classification loss);
• As a metric, we had COCO mAP.

Additionally, we decided to run this experiment 4 times in a row as we wanted to achieve a more robust result. Across all these experiments the parameters were the same. Our goal was to ensure that the random processes inside neural networks do not massively affect the final result. As an example, please check out the mean Average Precision curves for one of the experiments:

Example: mAP curve (PASCAL dirty)

Example: mAP curve (PASCAL clean)

To evaluate this part of the experiment, we separately averaged mAP curves obtained on dirty and clean PASCAL and built the following graph.

Head-to-head comparison

The average COCO mAP result we achieved on dirty PASCAL was 0.372 mAP on validation, whereas for the model trained on the cleansed PASCAL, this value is 0.4 mAP. This experiment shows that even without a scheduler, the clean model outperformed the dirty one by 8%. So, ReduceLROnPlateau gave the clean model a more significant performance boost in the previous iteration. Still, as you can see, the scheduler was not the only cause of the metric improvement – the improved annotations make the difference.

If you are curious and want to see how the final COCO mAP changed for the models trained on dirty and cleansed PASCAL from experiment to experiment, please check out the table down below:

StepLR scheduler

The second idea was to use a scheduler that will 100% reduce the learning rate simultaneously for both models. We decided to use the StepLR scheduler and reduce the learning rate every 3000 iterations by 0.1. All the other parameters remained the same.

Just as in the No scheduler case, we ran the StepLR experiment 4 times in a row as we wanted to achieve a more robust result. As an example, please check out the mean Average Precision curves for one of the runs:

Example: mAP curve (PASCAL dirty)

Example: mAP curve (PASCAL clean)

Once again, to evaluate this part of the experiment, we separately averaged mAP curves obtained on dirty and clean PASCAL and built the following graph.

Head-to-head comparison

The average COCO mAP result we achieved on dirty PASCAL was 0.387 mAP on validation, whereas for the model trained on the cleansed PASCAL, this value is 0.409 mAP. The clean model outperformed the dirty one by 5.6%. So, having a scheduler gave us an additional logical and expected overall metric boost.

If you are curious and want to see how the final COCO mAP changed for the models trained on dirty and cleansed PASCAL from experiment to experiment, please check out the table down below:

To summarize, although our second iteration might not be as extensive and fascinating as the initial one, we still got the results that prove the original experiment was legit. With this in mind, we again encourage you to give Data-centric AI development a shot, as it might be the exact thing you need to push your solution to the top.

What is the next step?

Getting excellent results and proving that the Data-centric AI development approach works are great achievements, but, unfortunately, they are not enough to make a difference in the grand scheme. Even with an understanding that better data results in better models, if you have a partially labeled dataset like PASCAL, you still have to work hard and invest lots of time and money to improve the data quality.

However, the bigger picture is that cleaning up even a section of a dataset has its benefits. First, it can push your AI solution to new heights and we have demonstrated that it two iterations of our project. Second, by improving the data (especially if you are tuning a well-known free dataset) and sharing it with the Data Science community, you are helping future researchers to achieve better results. From this point of view, the Data-centric AI development approach is not only relevant for commercial organizations. We strongly believe that better data for research teams also means more accurate results and advancement for the entire AI field.

These assumptions defined our next step with the PASCAL cleansing project. We have always tried to make our customers’ life easier and spare their time, money, and nerves. Today is the time to set a new goal and contribute a bit to the future. The PASCAL project was an unforgettable experience as we had never thought such little work could push a solution so much further. And we want to share this experience with you and the AI community.

Today, we are launching our PASCAL crowdsourcing campaign, where you will work side by side with us to thoroughly cleanse PASCAL from all its issues using the full power of the Hasty AI assistants and AI Consensus Scoring.

Why should I help?

If you join us, you will have a chance to work on a fascinating project that might be helpful for future AI research. When the job is done, we will post the updated version of the dataset on Kaggle so that anyone around the globe can use it without experiencing any issues with annotations. Additionally, we will donate to charity depending on the results achieved, making your contribution precious and karma cleaner.

Do you want to get in the gauntlet, push PASCAL to its limits, and help future AI researchers? Please fill in the sign-up form, and we will get back to you in no time.

Vladimir Lyashenko
Content Manager, Hasty

The post Further Pascal VOC 2012 Exploration and Plans for the Future appeared first on Edge AI and Vision Alliance.

This blog post was originally published at Tryolabs’ website. It is reprinted here with the permission of Tryolabs.

Artificial intelligence (AI) and machine learning (ML) significantly impact the retail world, particularly for companies that rely on online sales, where using some kind of AI technology is very common nowadays.

Big players like eBay, Amazon or Alibaba have successfully integrated AI across the entire sales cycle, from storage logistics to post-sale customer service.

However, you do not have to be a big company or sell exclusively online to take advantage from the tremendous power of Machine Learning.

In this guide we will see how online retailers and brick and mortar stores of any size can integrate Machine Learning technology to stay ahead of their competitors, by increasing sales and reducing costs.

From clothes to groceries to household items, the possibilities in the retail space are full of promise. The use cases presented in this guide are a fraction of the feasible Machine Learning projects and serve as examples of what can be done today in the Retail space. That being said, many companies have very unique needs that could be served with data and custom Machine Learning development. We hope you find this guide useful.

Retail intelligence

Pricing strategy

How you price an item and scale pricing operations intelligently, consistently, and efficiently is critical for retail and  e-commerce businesses. AI and data intelligence dramatically improve pricing decisions.

At Tryolabs, we have created AI solutions for retail that come in different flavors depending on the pricing strategy objectives.

Auto-pricing

Sometimes, the price of an item is defined by a person based on how she values a given item. This is usually the case for shopping and resale platforms. This manual process sometimes leads to inconsistent results, as different people value the same item differently. The human bottleneck is also a factor: every piece published on a site needs to be priced by a human, impacting costs and publication speed. It is also difficult for a person to consider the various factors influencing demand and price.

At Tryolabs, we have extensively worked in Machine Learning models that automatically and consistently price every new item by looking at data. The model leverages image recognition technology to understand the product’s look and consider other aspects such as brand, condition, fabric, category, description, historical sales data, and past pricing decisions of similar items. With these data points, it computes an optimal price for each SKU. It is also possible to expand the model to include other features, including competitor price if it’s available.

Some of our largest customers have used such models to automate the vast majority of their pricing operations and scale to more items effortlessly.

Pricing optimization

Pricing is a critical predictor of profitability. Based on econometric science, a Machine Learning algorithm can take key pricing variables into account to define an automatic pricing strategy with real-time, dynamic prices.

The willingness to pay can be estimated from the customer’s behavior, for example, considering the items they look at and purchase, or the time they spend on each web page.

The goal is to understand how clients react to items at different price points and estimate the right price for each product if they want to sell it in a certain period, ultimately maximizing profit.

The model forecasts demand and considers many factors to advise on dynamic price adjustments.  By including supply, seasonality, external events related to your business (e.g. a concert, a match, a festival) and market demand and offer, an automatic pricing system can test and learn from past experiences which is the most profitable or estimated “optimal” price.

Such a system needs historical sales data to be trained, but unlike auto-pricing, the system benefits from variability in the past prices used for each product. This is because it needs “exploration” in order to estimate the price-demand relation (usually referred to as demand elasticity).

Suppose there’s no variability in past prices. In that case, the first iterations will be mainly in “exploration mode,” which consists of introducing some randomness to the prices and evaluating how the sales data for each product reacts.

Significant deviations from previous prices are not needed, the range can be expanded gradually as the system gains information about what’s the right direction to move in each case.

In our experience, there’s an enormous potential for improvement in the pricing strategy compared to other approaches, and the benefits are usually for both sellers and customers, since low prices in some products can end up driving a surprising increase in profitability.

Product matching

Product matching is the process of identifying and linking identical items between two or more catalogs.

Some of the reasons why you might want to do that linking are: price comparison with competitors or different providers of the same products, merging multiple offers from different sellers into a single product page (e.g: Amazon’s buy box), among other less obvious uses like performing assortment gap analysis against your competitors.

To understand the business value of solving this problem, let’s consider the first example mentioned above. In retail, 96% of customers compare prices before making a purchase. Applying product matching allows any business to track the price their competitors are offering for the same products.

As businesses grow, product catalogs tend to become large and messy. If you’re lucky, products sometimes have codes that uniquely identify them and make the matching trivial. But more often than not, these codes are not available or sellers do not expose them. Sometimes, the codes are even incorrect! Another thing that may happen is that the products may indeed be the same, but their presentation may not. For example, single bottle vs. pack of 6.

Because of all this, you may be left with a complex problem: using whatever information of the products is available to do the matching. This can be a product title, a free-text description, some other metadata, and images.

This matching task can be done manually up to a point, but it escalates out of control rather quickly when catalogs grow in size. Even worse if you are matching multiple catalogs (multiple competitors or providers).

In our experience with retailers, we have faced this problem quite often. From this, we have built a solution that leverages AI through natural language processing (NLP) and computer vision techniques to find the most similar pairs by comparing each product’s image or unstructured text fields, like title, attributes and description.

In most cases, it’s not possible to guarantee that the matching pairs are perfect, so we use a human-in-the-loop process in which the task becomes a quick verification of the suggested matches that gets progressively easier with time. This is how it works:

1. The first batch of manually verified matches is leveraged to re-train the system (using them as a list of “verified matching examples”).
2. 1. The AI learns what features it should use to identify actual matches for each type of product.
3. Now we can perform a second iteration for the remaining items and the proposed matches will be more trustworthy.
4. The system is re-trained again.

This process can be repeated as needed depending on the confidence achieved and requirements of the particular problem.

Marketing Campaigns Optimization

Like pricing strategy, Marketing Campaigns, besides being essential for retail business, are complex and require a deep understanding of the market. Machine Learning models augment the decision-making process by leveraging historical data to forecast the ROI and provide optimal parameters for its execution. The model predictions aid decision makers in the process of balancing the costs and profits when designing the campaign.

At Tryolabs, we have created Machine Learning models customized to target specific business metrics when designing Marketing Campaigns such as the net revenue or margin contribution. Primarily we have worked on optimizing product promotion and replenishment campaigns.

As these solutions are employed, they produce more data so that we can evaluate them by measuring business KPIs. This in turn produced more data for improving the models, eventually creating a virtuous learning cycle.

Product Promotion optimization (discounts to offer, similar to pricing optimization but not the same)

Product promotion campaigns can have different flavors, for example, offering discounts to increase the net revenue. However, how can we optimize the discount offers provided so that margin contributions are maximized and therefore, maximize the overall net revenue?

For this task, we can define a function modeling the discount offering problem and optimize it to maximize profit  using Machine Learning. We then combine demand forecasting together with the unit costs of each SKU as well as their price data to train a model that recommends the decision makers what discount to offer according to the business goals.

By employing Machine Learning to optimize Product Promotion Campaigns, Marketing Teams are able to use their time more efficiently and use it where their creativity and experience can be leveraged the most.

Product Replenishment optimization (recommend what to buy for clients in b2b2c)

In retail, especially in the case of B2B2C businesses, the goal of Product Replenishment Campaigns is to increase repeat buys and decrease purchasing cycles by using specialized recommendations. Achieving these goals requires a deep understanding of the market and a thorough analysis to target the campaign successfully.

By utilizing market and business data, we employed a customized Machine Learning Clustering model to perform customer segmentation and enable the business to optimize their recommendations to their buyers with better targeting. This did not just increase their own net revenue, it also increased their buyers’ for the case of B2B2C businesses. Moreover, this kind of customer segmentation can be used for better targeting other Marketing Efforts downstream.

Predicting customer behavior

The goal of a predicting customer behavior system is to estimate how buyers will behave in the future based on data of previous behaviors. These systems allow retailers to segment customers and perform personalized marketing actions that are more effective than general approaches. Moreover, taking actions based on predicted customer needs increases loyalty and retention.

A typical application is to predict purchases. For example, to know which customers are likely to make a purchase in the next seven days. More complex predictions may have to do with important events in people’s lives. For example, to predict marriage or pregnancy, and then send custom offers.

The predictive models basically need consumer behavior data. That is, for example, purchasing history or buying trends, but it could also include social media activity and domain specific knowledge.

Predicting the needs of consumers is a challenging task where Machine Learning algorithms are of great help. How often do your customers make transactions? Do they buy during sales time or before their birthday? How many items do they usually buy? What are they currently buying? What topics are they talking about on social media? All this kind of information is used by the models to predict future behavior.

And last but not least, retailer experience is very important to choose business specific criteria and fine tune the models.

Retail Stocking and Inventory

Optimizing inventory planning and predictive maintenance is a key issue and a very important logistic concern for retailers. Moreover, this could affect the UX of end users of e-commerce platforms if a product they buy is actually out-of-stock and their order cannot be completed.

Predicting inventory

Machine Learning algorithms can exploit purchase data to predict inventory needs in real-time. Based on the day of the week, the season, nearby events, social media data and customer past behavior, these algorithms can provide a daily dashboard of suggested orders to a purchasing manager.

From a different perspective, we have built similar solutions to predict if an item a user is purchasing might be out of stock in scenarios where the e-commerce platform is not managing inventories directly. A predictive model leverages historical data such as past stockout events, demand, price, stock level for each product. It also uses real-time data such as price updates and date-time features to make accurate predictions. In this way, e-commerce platforms could advise users their items might be out of stock in a timely manner.

The inventory planning models need consumer behavior data. That is, for example, purchasing history or buying trends, but it could also include social media activity and domain specific knowledge.

Also, the algorithms used for pricing optimization, which need a sales forecasting model (as a function of the price, in order to calculate what’s the optimal one) can also be leveraged for stock management purposes, and they usually work integrated in order to avoid prices that will create an early out-of-stock event (since that’s always a suboptimal solution).

e-Commerce

Tagging and copywriting automation

Searching for items on a crowded site can be a daunting experience. That is why retail teams spend days writing descriptive copy for items and tagging them according to their primary attributes and preferred taxonomy. Traditionally, this is a manual process that relies on each person’s time and criteria, often resulting in a lack of consistency in data.

With the traditional manual process, it is common to face some of these issues:

• Inconsistent descriptions among similar items (some too long and detailed, some too short)
• Missing attributes
• Lack of human alignment in criteria for specific attributes or values

Tryolabs has developed an AI solution that uses computer vision technology to automate item tagging and descriptions, obtaining consistent and standardized data. The information comes directly from the image. The machine automatically fills product attributes and tags them accordingly to sit in the correct category and hierarchy on the website by simply uploading the photo. This can be tuned for each particular taxonomy unique for each retailer.

Image retouching

Images ensure online shoppers get as close as possible to fully understanding product features. Professional-looking photos can give an edge to an e-commerce site, making customers more likely to buy, and brands happier with the portrayal of their products.

Usually, pictures are taken in a photo studio and then sent out for retouching. The retouching process includes several different steps, depending on each particular site. The most common ones are calibrating colors for accurate depiction (you don’t want customers returning items because the colors don’t match their expectations!), removing backgrounds, aligning/rotating items, smoothing creases, and erasing mannequins. Unfortunately, this process can take several days, delaying the whole item uploading process.

But AI is increasingly used to cut corners and automate most of this time-consuming process.

Tryolabs has worked on automated image retouching using AI that cuts both time and costs to a fraction. The system replicates the traditional “Photoshop pipeline” done by artists but on a much larger scale. The opportunity is huge: one of our largest clients reduced the turnaround time for retouching hundreds of thousands of images from three days to only 30 minutes.

Browsing experience

Browsing online catalogs can be exhausting like a needle in a haystack situation, or an enjoyable process in which you may end up learning a lot more about the products you’re looking for. It depends mostly on the relevancy of the results that the system provides while you interact with it in various ways. The problem is known in computer science as Information Retrieval, and it’s not limited to introducing a search term and hitting Enter. In general, we can think of it as: given a query, provide the most relevant results, usually defined -in this context- as those which the user is most likely to click, so it’s evaluated by measuring the click-through rate.

Let’s see some examples that follow this pattern, in which there’s a query (e.g: a search term or product that the user is currently looking at) and a list of results that might be relevant to the user. The interaction with such elements ends up defining to a great extent the browsing experience for the users.

Visual search

Customers tend to search for visual content prior to making a purchase. However, in some cases they cannot easily find good keywords to describe what they want. The goal of visual search is to make it much easier for consumers to find exactly what it is they’re looking for.

Instead of typing a query such as ‘cordless combo kit with soft case’, which will probably return a lot of general results, customers can upload an image to help narrow the search down to more specific items. With the huge and increasing amount of snapping and sharing images, Machine Learning algorithms can currently achieve amazing results.

This is one of the most trending use cases in online content. Leading companies like Microsoft, Google or eBay have presented in 2017 Bing Visual Search, Google Lens and Image Search.

Regarding the e-commerce market, Pinterest introduced Lens Your Look, a visual search engine that allows you to find outfit ideas inspired by your wardrobe. So if you are looking for new ways to wear your favorite jean or blazer, you can add a photo of it to your search to find outfit ideas that you eventually can buy.

The solution to this problem is actually very related to finding similar products (see next section), because the visual search operation consists of finding the most visually similar products to your query image, hence it’s a sub-problem of the next one. If you have the best deep learning model to perform visual search, you also want to use it to find similar products in general, maybe adding other attributes and text descriptions.

Similar products

As mentioned above, finding similar products makes use of the visual information (if it’s available), but also the text information and relevant attributes.

The background models are also similar to those used in product matching (described above), but the purpose here is not to find exact matches, but those that are visually similar and also have certain attributes that make them similar.

The exact features that need to be prioritized in order to suggest similar products, depend a lot on the use case and the business strategy. For example, some retailers may want to prioritize products from the same brand or manufacturer, while others may prefer the exact opposite, and show diverse brands when there are many similar items for the same product category. Another example is the color in the case of fashion, is it important that the suggestions have similar colors? Those things can be configured as needed, using heuristics defined with some business strategy in mind, or leveraging domain knowledge from the clients.

Having a similar products system deployed is also a robust step before a good recommendation engine. The reason is that in order to retrieve similar products, there’s no need for prior user interaction data. But it’s precisely a way to generate that data, because these suggestions will result in a lot of interaction between users and products, and that information can be then used to train a recommendation system that takes into consideration usage metrics like click-through rate.

Recommendation engines

Many e-commerce and retail companies are leveraging the power of data and boosting sales by implementing recommender systems on their websites.

Companies using recommender systems focus on increasing sales as a result of very personalized offers and an enhanced customer experience. Recommendations typically speed up searches and make it easier for users to access content they’re interested in, and surprise them with offers they would have never searched for.

What is more, companies are able to gain and retain customers by sending out emails with links to new offers that meet the recipients’ interests, or suggestions of films and TV shows that suit their profiles.

The user starts to feel known and understood and is more likely to buy additional products or consume more content. By knowing what a user wants and showing it right away, it is less likely that it leaves the platform. This translates into a higher chance of purchase and a decrease in the threat of losing a customer to a competitor.

Providing that added value to users by including recommendations in systems and products is appealing. Furthermore, it allows companies to position ahead of their competitors and eventually increase their earnings.

Catalog tree optimization

Ecommerce platform catalog trees serve various purposes including facilitating search by buyers, categorizing products for the sellers, and allowing the platform to personalize the experience and provide better recommendations for the buyers among others. Additionally, they enable better SEO.

The main issue platforms face comes from dealing with uncategorized products which end up in overdimensioned Other categories. These end up being the go to option for users categorizing products that don’t fit the existing tree, be it because the category was overlooked or it is a new kind of product. This in turn makes it more difficult for users to find the products they are looking for, it affects the website’s SEO, the buying user experience, and eventually the sales volume.

With a combination of recommender systems and NLP techniques, we built a solution that allows ecommerce platforms to reduce the size of their Other categories by predicting the category of new products for users and recommending operators new categories to move uncategorized products over to. This brought better SEO, better discoverability of items, better buying experience, and therefore, an increase in revenue.

Brick and mortar

Behavioral Tracking via Video Analytics

A good thing about physical stores is that the behavior and interaction of humans with products can generate valuable insights in ways that online retail cannot. Computer Vision algorithms can recognize faces and people’s characteristics such as gender or range of age, generating precious exploitable data.

Human flow counting

Being able to measure human flow in stores allows retailers to know not only the effectiveness of their stores when it comes to converting customers, but also the distribution of people entering the stores during the day. This is useful to optimize the use of operational resources and provide better customer service.

Using computer vision algorithms, we built a video analytics solution that counts human flow in stores – the number of people entering and exiting physical stores at each given time, based on videos from pre-installed security cameras. This allows retailers to leverage their existing CCTV infrastructure to gain further insight into their business with the aid of Machine Learning.

Analyzing navigational routes

Where to put different items is a crucial matter for physical retailers, who always look for additional ways to understand the customer’s path to purchase.

Computer vision algorithms can track customers’ journey in stores to understand how they are interacting with it. These algorithms can detect the walking patterns and the direction of the gaze of the customers. Retailers can use this information to restructure store layouts or to measure the interest in their products. They can also discover locations that get a lot of traffic and visual attention.

Do elderly people shop more on weekdays? Do teenagers tend to cover only part of the store, for example the front part? Is the store more visited in winter? Variables such as age, day of the week or season could be used to generate insights that help to dynamically modify product placements and create efficient promotions.

Theft prevention

Theft Prevention is a common problem in retail with a strong ROI, where Machine Learning technologies can go beyond the typical use of video cameras to detect shoplifters.

Facial recognition algorithms can be trained to spot known shoplifters when they enter the store. Walmart has tested this technology in 2015 as an anti-theft mechanism. In the same way computer vision can detect if someone picks an item, it could detect if someone hides an item in their backpack or jacket. Moreover, the same approach can be used to detect when checkout clerks skip scanning items, either inadvertently or on purpose.

A system based on Machine Learning can alert in real-time security personnel or managers and send them video excerpts so they can judge by themselves before confronting the individual in the store.

Product tracking and gesture recognition

Brick and mortar retailers usually have no information about the items that customers pick up, glance at, and put back on the shelf. They do not have any information either about what customers look at next.

A computer vision algorithm can monitor shoppers’ facial and hand gestures to estimate how interesting an item is. These kinds of applications generate precious data about how many times an item is picked up from the shelf, and then purchased or put back on the shelf.

Conclusions

The retail industry keeps growing and finding new challenges in an ever-evolving consumer paradigm. Serving these necessities with state-of-the-art technology can determine whether you succeed or become outdated.

At Tryolabs we help retailers leverage the power of AI by providing expertise in all the phases of AI initiatives, from ideas to execution: from definition of opportunities in an AI Roadmap, up until implementation in production, iteration and improvements.

We build data-driven solutions to improve company’s KPIs. This means that we partner up with retailers to conceptualize & build machine learning systems that either increase revenue or reduce costs.

If you’re thinking of ways to integrate AI into your strategy, get in touch with us and we’d love to discuss your case.

Javier Couto
Research Scientist, Tryolabs

The post The Guide to AI and Machine Learning in Retail: Applications and Use Cases appeared first on Edge AI and Vision Alliance.