🎣
KV260アプリケーションの開発方法（4）
1.はじめに
前回はSmartCamアプリケーションをインストールして動作確認しようとしましたが、カメラドライバーが正しく認識せず動かすことができませんでした。今回はその原因について調べて行きたいと思います。
2.原因調査
まずは、KV260でSmartCamのファームウエアをロードした時のdmesgログを見てみます。
% sudo xmutil unloadapp
% sudo xmutil loadapp user-firmware
% dmesg
ログを見ると、mipi_csi2_rxでプローブエラーが発生していることが確認できます。
3.デバイスツリーの修正（1行追加）
XSAファイルから生成したpl.dtsiのMIPIノードにxlnx,csi-pxl-formatプロパティを追加してもう一度試してみます。値は<0x18> : MIPI_CSI_DT_YUV_420_8Bを指定します。
capture_pipeline_mipi_csi2_rx_subsyst_0: mipi_csi2_rx_subsystem@80000000 {
clock-names = "lite_aclk", "dphy_clk_200M", "video_aclk";
clocks = <&misc_clk_0>, <&misc_clk_3>, <&misc_clk_2>;
compatible = "xlnx,mipi-csi2-rx-subsystem-5.1", "xlnx,mipi-csi2-rx-subsystem-5.0";
interrupt-names = "csirxss_csi_irq";
interrupt-parent = <&gic>;
interrupts = <0 104 4>;
reg = <0x0 0x80000000 0x0 0x2000>;
xlnx,csi-pxl-format = <0x18>;
xlnx,axis-tdata-width = <32>;
xlnx,max-lanes = <4>;
xlnx,ppc = <2>;
xlnx,vfb ;
・・・・・
};
リビルドしたpetalinuxで、user-firmwareをロードしてdmesgログを再確認してみると、no sink port foundのエラーに代わっています。
3.デバイスツリーの再修正（ガッツり修正）
pl.dtsiのmipi_csi2の部分を見てみると、videoのfield変更やらremote-endpointの追加を設定しろとコメントに書いてあります。。。そのままでは使えないんですね。
capture_pipeline_mipi_csi2_rx_subsyst_0: mipi_csi2_rx_subsystem@80000000 {
・・・
mipi_csi_portscapture_pipeline_mipi_csi2_rx_subsyst_0: ports {
#address-cells = <1>;
#size-cells = <0>;
mipi_csi_port0capture_pipeline_mipi_csi2_rx_subsyst_0: port@1 {
/* Fill cfa-pattern=rggb for raw data types, other fields video-format and video-width user needs to fill */
・・・
};
mipi_csi_port1capture_pipeline_mipi_csi2_rx_subsyst_0: port@0 {
/* Fill cfa-pattern=rggb for raw data types, other fields video-format,video-width user needs to fill */
/* User need to add something like remote-endpoint=<&out> under the node csiss_in:endpoint */
・・・
};
};
};
そういう訳で、以下のSmartCamのDeviceTreeファイルを参考にしてpl.dtsiを修正します。
修正後のpl.dtsi
/*
* CAUTION: This file is automatically generated by Xilinx.
* Version: XSCT 2021.1
* Today is: Fri Apr 1 00:58:30 2022
*/
/dts-v1/;
/plugin/;
/ {
fragment@0 {
target = <&fpga_full>;
overlay0: __overlay__ {
#address-cells = <2>;
#size-cells = <2>;
firmware-name = "user-firmware.bit.bin";
resets = <&zynqmp_reset 116>, <&zynqmp_reset 117>, <&zynqmp_reset 118>, <&zynqmp_reset 119>;
};
};
fragment@1 {
target = <&amba>;
overlay1: __overlay__ {
afi0: afi0 {
compatible = "xlnx,afi-fpga";
config-afi = < 0 0>, <1 0>, <2 0>, <3 0>, <4 0>, <5 0>, <6 0>, <7 0>, <8 0>, <9 0>, <10 0>, <11 0>, <12 0>, <13 0>, <14 0x0>, <15 0x000>;
};
clocking0: clocking0 {
#clock-cells = <0>;
assigned-clock-rates = <99999001>;
assigned-clocks = <&zynqmp_clk 71>;
clock-output-names = "fabric_clk";
clocks = <&zynqmp_clk 71>;
compatible = "xlnx,fclk";
};
clocking1: clocking1 {
#clock-cells = <0>;
assigned-clock-rates = <99999001>;
assigned-clocks = <&zynqmp_clk 72>;
clock-output-names = "fabric_clk";
clocks = <&zynqmp_clk 72>;
compatible = "xlnx,fclk";
};
};
};
fragment@2 {
target = <&amba>;
overlay2: __overlay__ {
#address-cells = <2>;
#size-cells = <2>;
audio_ss_0_audio_formatter_0: audio_formatter@80040000 {
clock-names = "s_axi_lite_aclk", "m_axis_mm2s_aclk", "aud_mclk", "s_axis_s2mm_aclk";
clocks = <&misc_clk_0>, <&misc_clk_1>, <&misc_clk_1>, <&misc_clk_0>;
compatible = "xlnx,audio-formatter-1.0", "xlnx,audio-formatter-1.0";
interrupt-names = "irq_mm2s", "irq_s2mm";
interrupt-parent = <&gic>;
interrupts = <0 111 4 0 110 4>;
reg = <0x0 0x80040000 0x0 0x10000>;
xlnx,include-mm2s = <0x1>;
xlnx,include-s2mm = <0x1>;
xlnx,max-num-channels-mm2s = <0x2>;
xlnx,max-num-channels-s2mm = <0x2>;
xlnx,mm2s-addr-width = <0x40>;
xlnx,mm2s-async-clock = <0x1>;
xlnx,mm2s-dataformat = <0x3>;
xlnx,packing-mode-mm2s = <0x0>;
xlnx,packing-mode-s2mm = <0x0>;
xlnx,rx = <&audio_ss_0_i2s_receiver_0>;
xlnx,s2mm-addr-width = <0x40>;
xlnx,s2mm-async-clock = <0x1>;
xlnx,s2mm-dataformat = <0x1>;
xlnx,tx = <&audio_ss_0_i2s_transmitter_0>;
};
misc_clk_0: misc_clk_0 {
#clock-cells = <0>;
clock-frequency = <99999000>;
compatible = "fixed-clock";
};
misc_clk_1: misc_clk_1 {
#clock-cells = <0>;
clock-frequency = <18432995>;
compatible = "fixed-clock";
};
audio_ss_0_i2s_receiver_0: i2s_receiver@80060000 {
aud_mclk = <18432995>;
clock-names = "s_axi_ctrl_aclk", "aud_mclk", "m_axis_aud_aclk";
clocks = <&misc_clk_0>, <&misc_clk_1>, <&misc_clk_0>;
compatible = "xlnx,i2s-receiver-1.0", "xlnx,i2s-receiver-1.0";
interrupt-names = "irq";
interrupt-parent = <&gic>;
interrupts = <0 108 4>;
reg = <0x0 0x80060000 0x0 0x10000>;
xlnx,depth = <0x80>;
xlnx,dwidth = <0x18>;
xlnx,num-channels = <0x1>;
xlnx,snd-pcm = <&audio_ss_0_audio_formatter_0>;
};
audio_ss_0_i2s_transmitter_0: i2s_transmitter@80070000 {
aud_mclk = <18432995>;
clock-names = "s_axi_ctrl_aclk", "aud_mclk", "s_axis_aud_aclk";
clocks = <&misc_clk_0>, <&misc_clk_1>, <&misc_clk_1>;
compatible = "xlnx,i2s-transmitter-1.0", "xlnx,i2s-transmitter-1.0";
interrupt-names = "irq";
interrupt-parent = <&gic>;
interrupts = <0 109 4>;
reg = <0x0 0x80070000 0x0 0x10000>;
xlnx,depth = <0x80>;
xlnx,dwidth = <0x18>;
xlnx,num-channels = <0x1>;
xlnx,snd-pcm = <&audio_ss_0_audio_formatter_0>;
};
ap1302_clk: sensor_clk {
#clock-cells = <0x0>;
compatible = "fixed-clock";
clock-frequency = <0x48000000>;
};
ap1302_vdd: fixedregulator@0 {
compatible = "regulator-fixed";
regulator-name = "ap1302_vdd";
regulator-min-microvolt = <2800000>;
regulator-max-microvolt = <2800000>;
enable-active-high;
};
ap1302_vaa: fixedregulator@1 {
compatible = "regulator-fixed";
regulator-name = "ap1302_vaa";
regulator-min-microvolt = <1800000>;
regulator-max-microvolt = <1800000>;
};
ap1302_vddio: fixedregulator@2 {
compatible = "regulator-fixed";
regulator-name = "ap1302_vddio";
regulator-min-microvolt = <1200000>;
regulator-max-microvolt = <1200000>;
};
axi_iic_0: i2c@80030000 {
#address-cells = <1>;
#size-cells = <0>;
clock-names = "s_axi_aclk";
clocks = <&misc_clk_0>;
compatible = "xlnx,axi-iic-2.1", "xlnx,xps-iic-2.00.a";
interrupt-names = "iic2intc_irpt";
interrupt-parent = <&gic>;
interrupts = <0 107 4>;
reg = <0x0 0x80030000 0x0 0x10000>;
i2c_mux: i2c-mux@74 {
compatible = "nxp,pca9546";
#address-cells = <1>;
#size-cells = <0>;
reg = <0x74>;
i2c@0 {
#address-cells = <1>;
#size-cells = <0>;
reg = <0>;
ap1302: isp@3c {
compatible = "onnn,ap1302";
reg = <0x3c>;
#address-cells = <1>;
#size-cells = <0>;
reset-gpios = <&gpio 79 1>;
clocks = <&ap1302_clk>;
sensors {
#address-cells = <1>;
#size-cells = <0>;
onnn,model = "onnn,ar1335";
sensor@0 {
reg = <0>;
vdd-supply = <&ap1302_vdd>;
vaa-supply = <&ap1302_vaa>;
vddio-supply = <&ap1302_vddio>;
};
};
ports {
#address-cells = <1>;
#size-cells = <0>;
port@0 {
reg = <2>;
isp_out: endpoint {
remote-endpoint = <&mipi_csi_incapture_pipeline_mipi_csi2_rx_subsyst_0>;
data-lanes = <1 2 3 4>;
};
};
};
};
};
};
};
axi_vip_0: axi_vip@a0000000 {
/* This is a place holder node for a custom IP, user may need to update the entries */
clock-names = "aclk";
clocks = <&misc_clk_2>;
compatible = "xlnx,axi-vip-1.1";
reg = <0x0 0xa0000000 0x0 0x10000>;
xlnx,axi-addr-width = <0x20>;
xlnx,axi-aruser-width = <0x10>;
xlnx,axi-awuser-width = <0x10>;
xlnx,axi-buser-width = <0x0>;
xlnx,axi-has-aresetn = <0x1>;
xlnx,axi-has-bresp = <0x1>;
xlnx,axi-has-burst = <0x1>;
xlnx,axi-has-cache = <0x1>;
xlnx,axi-has-lock = <0x1>;
xlnx,axi-has-prot = <0x1>;
xlnx,axi-has-qos = <0x1>;
xlnx,axi-has-region = <0x0>;
xlnx,axi-has-rresp = <0x1>;
xlnx,axi-has-wstrb = <0x1>;
xlnx,axi-interface-mode = <0x2>;
xlnx,axi-protocol = <0x0>;
xlnx,axi-rdata-width = <0x20>;
xlnx,axi-rid-width = <0x10>;
xlnx,axi-ruser-width = <0x0>;
xlnx,axi-supports-narrow = <0x1>;
xlnx,axi-wdata-width = <0x20>;
xlnx,axi-wid-width = <0x10>;
xlnx,axi-wuser-width = <0x0>;
};
misc_clk_2: misc_clk_2 {
#clock-cells = <0>;
clock-frequency = <299997000>;
compatible = "fixed-clock";
};
capture_pipeline_mipi_csi2_rx_subsyst_0: mipi_csi2_rx_subsystem@80000000 {
compatible = "xlnx,mipi-csi2-rx-subsystem-5.1", "xlnx,mipi-csi2-rx-subsystem-5.0";
reg = <0x0 0x80000000 0x0 0x10000>;
clock-names = "lite_aclk", "dphy_clk_200M", "video_aclk";
clocks = <&misc_clk_0>, <&misc_clk_3>, <&misc_clk_2>;
interrupt-parent = <&gic>;
interrupts = <0 104 4>;
xlnx,csi-pxl-format = <0x18>;
xlnx,axis-tdata-width = <32>;
xlnx,max-lanes = <4>;
xlnx,en-active-lanes;
xlnx,vc = <4>;
xlnx,ppc = <2>;
xlnx,vfb;
mipi_csi_portscapture_pipeline_mipi_csi2_rx_subsyst_0: ports {
#address-cells = <1>;
#size-cells = <0>;
port@1 {
/* Fill cfa-pattern=rggb for raw data types, other fields video-format and video-width user needs to fill */
reg = <1>;
xlnx,video-format = <0x3>;
xlnx,video-width = <0x8>;
mipi_csirx_outcapture_pipeline_mipi_csi2_rx_subsyst_0: endpoint {
remote-endpoint = <&capture_pipeline_v_frmbuf_wr_0capture_pipeline_mipi_csi2_rx_subsyst_0>;
};
};
port@0 {
/* Fill cfa-pattern=rggb for raw data types, other fields video-format,video-width user needs to fill */
/* User need to add something like remote-endpoint=<&out> under the node csiss_in:endpoint */
reg = <0>;
xlnx,video-format = <0x3>;
xlnx,video-width = <0x8>;
mipi_csi_incapture_pipeline_mipi_csi2_rx_subsyst_0: endpoint {
data-lanes = <1 2 3 4>;
remote-endpoint = <&isp_out>;
};
};
};
};
misc_clk_3: misc_clk_3 {
#clock-cells = <0>;
clock-frequency = <199998000>;
compatible = "fixed-clock";
};
capture_pipeline_v_frmbuf_wr_0: v_frmbuf_wr@b0010000 {
#dma-cells = <1>;
clock-names = "ap_clk";
clocks = <&misc_clk_2>;
compatible = "xlnx,v-frmbuf-wr-2.2", "xlnx,axi-frmbuf-wr-v2.1";
interrupt-names = "interrupt";
interrupt-parent = <&gic>;
interrupts = <0 105 4>;
reg = <0x0 0xb0010000 0x0 0x10000>;
reset-gpios = <&gpio 78 1>;
xlnx,dma-addr-width = <32>;
xlnx,dma-align = <16>;
xlnx,max-height = <2160>;
xlnx,max-width = <3840>;
xlnx,pixels-per-clock = <2>;
xlnx,s-axi-ctrl-addr-width = <0x7>;
xlnx,s-axi-ctrl-data-width = <0x20>;
xlnx,vid-formats = "nv12";
xlnx,video-width = <8>;
};
vcu_vcu_0: vcu@80100000 {
#address-cells = <2>;
#clock-cells = <1>;
#size-cells = <2>;
clock-names = "pll_ref", "aclk", "vcu_core_enc", "vcu_core_dec", "vcu_mcu_enc", "vcu_mcu_dec";
clocks = <&misc_clk_4>, <&misc_clk_0>, <&vcu_vcu_0 1>, <&vcu_vcu_0 2>, <&vcu_vcu_0 3>, <&vcu_vcu_0 4>;
compatible = "xlnx,vcu-1.2", "xlnx,vcu";
interrupt-names = "vcu_host_interrupt";
interrupt-parent = <&gic>;
interrupts = <0 106 4>;
ranges ;
reg = <0x0 0x80140000 0x0 0x1000>, <0x0 0x80141000 0x0 0x1000>;
reg-names = "vcu_slcr", "logicore";
reset-gpios = <&gpio 80 0>;
encoder: al5e@80100000 {
compatible = "al,al5e-1.2", "al,al5e";
interrupt-parent = <&gic>;
interrupts = <0 106 4>;
reg = <0x0 0x80100000 0x0 0x10000>;
};
decoder: al5d@80120000 {
compatible = "al,al5d-1.2", "al,al5d";
interrupt-parent = <&gic>;
interrupts = <0 106 4>;
reg = <0x0 0x80120000 0x0 0x10000>;
};
};
misc_clk_4: misc_clk_4 {
#clock-cells = <0>;
clock-frequency = <49999500>;
compatible = "fixed-clock";
};
vcap_capture_pipeline_mipi_csi2_rx_subsyst_0 {
compatible = "xlnx,video";
dma-names = "port0";
dmas = <&capture_pipeline_v_frmbuf_wr_0 0>;
vcap_portscapture_pipeline_mipi_csi2_rx_subsyst_0: ports {
#address-cells = <1>;
#size-cells = <0>;
vcap_portcapture_pipeline_mipi_csi2_rx_subsyst_0: port@0 {
direction = "input";
reg = <0>;
capture_pipeline_v_frmbuf_wr_0capture_pipeline_mipi_csi2_rx_subsyst_0: endpoint {
remote-endpoint = <&mipi_csirx_outcapture_pipeline_mipi_csi2_rx_subsyst_0>;
};
};
};
};
};
};
};
リビルドしたpetalinuxで、user-firmwareをロードしてdmesgログを再確認してみると、mipi_csi2のエントリーは正しく出来ていそうです。逆にAP1302 ISPのfirmwareロードに失敗しているログが新たに発生しています。
この時点で、/dev/video0は認識されますが、/dev/media0はまだ認識されない状態です。
また、デバイスツリーに登録したモジュールは一通り登録に成功していることが確認できます。
/sys/devices/platform/axiのリスト
% xilinx-k26-starterkit-2021_1:/sys/devices/platform/axi$ ls
80000000.mipi_csi2_rx_subsystem axi:si5332_5 ff0f0000.spi
80030000.i2c axi:vcap_capture_pipeline_mipi_csi2_rx_subsyst_0 ff150000.watchdog
80040000.audio_formatter b0010000.v_frmbuf_wr ff170000.mmc
80060000.i2s_receiver driver_override ff960000.memory-controller
80070000.i2s_transmitter fd070000.memory-controller ff9d0000.usb0
80140000.vcu fd0b0000.perf-monitor ffa00000.perf-monitor
a0000000.axi_vip fd400000.phy ffa10000.perf-monitor
axi:afi0 fd490000.perf-monitor ffa50000.ams
axi:clocking0 fd4b0000.gpu ffa60000.rtc
axi:clocking1 fd4c0000.dma-controller ffa80000.dma
axi:fixedregulator@0 fd4d0000.watchdog ffa90000.dma
axi:fixedregulator@1 fd500000.dma ffaa0000.dma
axi:fixedregulator@2 fd510000.dma ffab0000.dma
axi:ina260-u14 fd520000.dma ffac0000.dma
axi:misc_clk_0 fd530000.dma ffad0000.dma
axi:misc_clk_1 fd540000.dma ffae0000.dma
axi:misc_clk_2 fd550000.dma ffaf0000.dma
axi:misc_clk_3 fd560000.dma modalias
axi:misc_clk_4 fd570000.dma of_node
axi:sensor_clk fd6e0000.cci power
axi:si5332_0 ff010000.serial subsystem
axi:si5332_1 ff030000.i2c supplier:firmware:zynqmp-firmware
axi:si5332_2 ff050000.spi supplier:firmware:zynqmp-firmware:clock-controller
axi:si5332_3 ff0a0000.gpio uevent
axi:si5332_4 ff0e0000.ethernet
4.AP1302のパッケージインストール
AP1302 ISPのfirmwareが無いということなので、以下のコマンドを実行することでfirmwareをインストールしてみます。
$ sudo dnf update
$ sudo dnf clean all
$ do dnf install ap1302-ar1335-single-firmware.cortexa72_cortexa53
インストールした後にuser-firmwareをロードしてdmesgログを再確認してみると、AP1302 ISPのfirmwareロードに成功している事が確認できました。この時点で/dev/media0も正しく認識されている事が確認できました。
5.動作確認
インストールが完了したらアプリケーションを動作させてみます。
最初にファームウエアをロードします。
$ sudo xmutil unloadapp
$ sudo xmutil loadapp user-firmware
受信PC側のGstreamerのスクリプトを実行します。
受信側
gst-launch-1.0.exe udpsrc name=video_src port=554 caps="application/x-rtp,media=(string)video,clock-rate=(int)90000,encoding-name=(string)H264" ! rtph264depay ! queue ! h264parse ! queue ! d3d11h264dec ! videoconvert ! d3d11videosink
KV260側のGstremerのスクリプトを実行してストリーミングを開始します。
送信側
gst-launch-1.0 v4l2src io-mode=4 device=/dev/video0 ! videoconvert ! video/x-raw, format=NV12, framerate=30/1, width=1920, height=1080 ! omxh264enc ! h264parse ! queue ! rtph264pay ! udpsink host=192.168.0.15 port=554 buffer-size=20000000
結果、受信PC側で無事に動画再生されることが確認できました！
6.終わりに
以上４回に渡って、プラットフォームとアプリケーションを作成して、KV260で動作させるところまで確認してきました。ここで確認してきたことの応用で独自アプリケーションも作っていけるのではないでしょうか。
Discussion