Solution

Focusing on FPGA image solution customization

Position:

HomeSolutionImage solutions

FPGA simulation MIPI camera successfully integrated into NVIDIA AGX developer

Author:admin / Time:2024-02-24

FPGA simulated MIPI camera, successfully connectedNVIDIAAGX Developer

1. preface

Visual cameras are key to machine intelligence/automation, and NVIDIA Jetson supports cameras from multiple partners, including cameras with USB, Ethernet, MIPI, and other interfaces. Jetson provides a complete Jetpack SDK for these cameras, enabling users to quickly connect to these cameras and implement corresponding designs.

The list of cameras supported by Jetson can be found on the official website as follows:

https://developer.nvidia.com/embedded/jetson-partner-supported-cameras?t -1 supported jet son products=Origin&t1 supported jet son products=Xavier

image.png 

Select the CSI/MIPI interface and Orin suite, and the webpage will display a list of Jetson Partner supported cameras, mainly Sony's IMX290 IMX334、IMX179、IMX335、IMX415 , And Aptina's AR0234, AR0144, AR0822, etc. Therefore, for example, if users use NVIDIA's Jetson AGX Orin developer suite, the SDK of the suite natively supports CSI access to the sensors listed above, allowing users to quickly develop their own solutions.

image.png 

If you choose the camera from the above list and use it with the accompanying CSI adapter board, as the Camera Partner has already pre docked with various models of NVIDIA Jetson, the driver in the original factory image is already supported by default, and can be plug and play.

howeverIf we choose a sensor model outside the list, which is not a Camera Partner product, we cannot directly use it,For example, if I want to use Sony's IMX472, IMX252, Sitewe's SC2210, SC235HGS, etc., thenIt is necessary to upgrade the supporting hardware and driver software.For this, Aoweisi provides you with a complete solution, which can be equipped with any camera or choose FPGA integrated ISP.

2. FPGA analog MIPI camera accessJetsonprogramme

 

As shown in the following figure: The scheme adopts Yilingsi Titanium 16nm FPGA Ti60F225, collects image sensor data through MIPI CSI RX, undergoes a series of complex ISP operations inside the FPGA, and then sends the image data to the backend through MIPI CSI TX.

image.png 

Among them, VC-MIPI-AGX is the MIPI access sub card of the NVIDIA Jetson AGX Orin developer kit developed by Aoweisi. As shown in the figure below, it is the MIPI card slot of AGX and the access sub card we designed. We use a high-speed and stable TypeC 3.0 data cable to transmit MIPI signals. The current solution can support 2 MIPI CSI channels, with a maximum bandwidth of 6Gbps per channel. At present, the solution has been successfully integrated into AGX, stable and reliable, and the baking machine has passed the test.

image.png 

 

2.1Introduction to Yilingsi MIPI

The low power consumption, high bandwidth, and small size characteristics of the Yilingsi 16nm titanium series FPGA fully empower terminal products to make them innovative and competitive. The titanium series Ti60 can support multi camera input, multi type screen driver, high-performance RISC-V soft core and other functions with MIPI/LVDS interfaces, suitable for various camera and sensor systems and other scene applications.

image.png 

Therefore, we chose Yilingsi's FPGA as the main controller, fully utilizing its high-speed and low-power characteristics to complete the collection of MIPI CSI, ISP image processing, and transmission of MIPI CSI. The MIPI PHY of Yilingsi Ti60 FPGA is 1.5Gbps single channel, and 4 lanes support 6Gbps. At 85% effective bandwidth, the effective transmission resolution/frame rate is evaluated as follows:

RGB888 transmission

Effective bandwidth

Actual bandwidth

(85%)

result

RGB888 24bit transmission

1920* 1080@60

Effective bandwidth=1920 * 1080 * 8bit * 60 * 3=2.986Gbps

3.513Gbps

1920* 1080@100

Effective bandwidth=1920 * 1080 * 8bit * 100 * 3=5.286Gbps

6.219Gbps

X

1920* 1080@120

Effective bandwidth=1920 * 1080 * 8bit * 120 * 3=5.972Gbps

7.026Gbps

X

Bayer 8-bit transmission

1920* 1080@60

Effective bandwidth=1920 * 1080 * 8bit * 60=0.9953Gbps

1.171Gbps

1920* 1080@100

Effective bandwidth=1920 * 1080 * 8bit * 100=1.659Gbps

1.952Gbps

1920* 1080@120

Effective bandwidth=1920 * 1080 * 8bit * 120=1.991Gbps

2.342Gbps

4096* 2304@60

Effective bandwidth=1920 * 1080 * 8bit * 60=4.530Gbps

5.330Gbps

Adopting RGB888 or Bayer 8-bit transmission has its advantages and disadvantages:

1) RGB888 8-bit transmission, AGX can directly obtain image data, with the advantage of no additional source data processing overhead; The downside is that due to Bayer's 3x bandwidth, the current FPGA scheme can only support up to 1080P60 scheme.

2) Bayer 8-bit transmission, with a bandwidth of 1/3 of RGB888, can support higher resolutions (4K60), and at the same resolution, can support longer transmission distances than RGB888; The shortcoming is that AGX's CPU (Neon) is needed to process the Bayer2RGB algorithm. Currently, testing 1080P120 consumes 43% of the resources of one CPU (a total of 12 CPUs, with single core resource consumption as shown in the following figure).

Based on comprehensive evaluation, due to the current use of only one AGX CPU and the user's algorithm mainly running CUDA on the GPU, the advantage of using Bayer transmission is more obvious.

image.png 

2.2Introduction to Aoweisi ISP

NVIDIA can also perform complete ISP image processing, and even the ISP function and performance implemented by ASIC are more powerful than FPGA. However, in this solution, we have to use FPGA for complete ISP processing, mainly for the following reasons:

1) NVIDIA still requires a lot of tuning work for the selected sensors;

2) NVIDIA's ISP does not support 8-bit RAW input mode;

3) Some application engineers are not familiar with the ISP field;

4) Some ISP algorithms are not included in NVIDIA;

5) The demand for low latency requires a pipeline to complete ISP.

In view of this, we have completed a complete ISP processing process using FPGA. MIPI CSI directly outputs the data after FPGA image processing. After NVIDIA AGX receives the RGB data, users can directly carry out subsequent application algorithms. The ISP image processing pipeline based on Yilingsi FPGA is as follows:

 

image.png 

The self-developed assembly line ISP by Aoweisi and its related features are shown in the table below:

parameter

describe

Parameter switch

Black Leveling

Black level correction

Dead Pixel Correction

Bad Point Correction Module (DPC)

Auto Exposure

Automatic exposure algorithm module

U adjustable switch

U AE brightness target value

U AE sensitivity

Auto White Balance

Automatic white balance algorithm module

Bayer2RGB

Bayer to RGB domain

RGB2YUV

RGB to YUV domain

Gamma Correction

YUV domain Gamma correction

U adjustable parameters

Color Correction

RGB color correction

2DNR

2D noise reduction algorithm module

U adjustable switch

U adjustable strength

Sharping

Edge sharpening module

U adjustable switch

U adjustable strength

Saturation

Saturation adjustment

U adjustable strength

Contrast

Contrast enhancement module

U adjustable strength

Brightness

Brightness adjustment module

U adjustable strength

Scaler&Crop

Zoom/Screenshot Module

U adjustable parameters

YUV2RGB

YUV to RGB module

 

2.3AGX Orin Access Driver

In terms of hardware, the VC-MIPI-AGX sub card designed by Ovis is used to achieve user MIPICSI to AGX Samtec input. In terms of software, it is necessary to develop specific Sensor driver adaptations based on hardware board and module design. If there are special customized control requirements, such as ISP parameter control, it is necessary to modify the Camera framework for deep customization development.

For the protection of system security, Jetson's kernel compilation was turned on when it left the factoryDriver Signature OptionsIf the newly developed driver signature certificate does not match, the driver ko file cannot be loaded. However, most users are unable to obtain the signature certificate from NVIDIA when compiling factory images, which cannot solve the ko signature problem. If you need to use a new sensor module, you will need toRecompile the kernel, generate a full system image, and burnsystemThe workload is quite enormous.

 

Aoweisi solves the above-mentioned software and hardware problems for customers and creatively uses typeC 3.0 cables as data transmission media, with reliable connections and stable communication. Based on the NVIDIA Jetson AGX Orin developer platform, the self-developed FPGA MIPI ISP camera connection environment is connected as follows:

 image.png

3. Effect display

image.png 

302, Building 17, Nanshan Ruiyuan, Xili Longzhu 3rd Road, Nanshan District, Shenzhen City, Guangdong Province

19926595670

cb@szovs.com

Official WeChat

Skype

CopyRight © 2023 Shenzhen Aoweisi(SZOVS) All Right Reserved.
Service

Tel

Hotline:

199 2659 5670

website

二维码

FPGA website

Official account

二维码

Official account

Skype

skype

Skype

Mailbox

Mailbox

cb@szovs.com

Comeback