😽
第5回 React+TypeScriptなWebアプリで、QRコードをARしてみた。(完成編)
<- [第4回]React+TypeScriptなWebアプリで、QRコードをARしてみた。(検出補助線を引く編)
[第6回]React+TypeScriptなWebアプリで、QRコードをARしてみた。(リリース編) ->
Abstract
React+TypescriptのWebアプリで、ARを実装してみた。第5回。
前回で検出QRコードの中心がとれたのでその位置に3Dモデルを移動させる。
あんまり、検出の精度がよくなくって、トラッキングがカクカクするけど、それは次の課題に。(たぶん、その日はこない。。。)
結論
今回の成果物はココ↓
前提
- React+Typescriptの開発環境は構築済 [環境構築]WindowsにVSCode+React+TypeScriptの開発環境を構築してみた。
- 前回プロジェクトから。ReactTs-QrAr004
- Webカメラは準備しておく。
手順
1.プロジェクト生成 -> VSCodeで開く
このテンプレートコードから始める。ReactTs-QrAr004
git cloneとか、フォルダリネームとか
$ D:
$ cd .\Products\React.js\ # ご自身の適当なフォルダに読み替えてね。
$ rd /q /s D:\Products\React.js\ReactTs-QrAr004
$ git clone https://github.com/aaaa1597/ReactTs-QrAr004.git
$ rd /q /s "ReactTs-QrAr004/.git"
$ ren ReactTs-QrAr004 ReactTs-QrAr005
$ cd ReactTs-QrAr004
準備
コマンドプロンプト
$ npm install
実行してみる
コマンドプロンプト
$ npm start
ここまでで、一旦動作するのが確認できる。
実装
検知カウンタを追加
別にカウンタである必要はないけど、検知イベントを通知する手段が必要で追加。
setContext/setPoints4/setPoints3はデバッグ用だったので削除。
App.tsx
-10: const FBXModel = (props:{setActionName: React.Dispatch<React.SetStateAction<string>>, detectcount: number}) => {
+10: const FBXModel = (props:{setActionName: React.Dispatch<React.SetStateAction<string>>, detectcount: number}) => {
- {/* デバッグ用canvas */}
- const [context, setContext] = useState<CanvasRenderingContext2D>();
- const [points4, setPoints4] = useState<ResultPoint[]>();
- const [points3, setPoints3] = useState<ResultPoint[]>();
アクションNoの初期値を-1に。
アクションNoの初期値を-1にすることで、検知してから動き出す様に修正。
App.tsx
-24: const [ animIdx, setAnimIdx ] = useState<number>(0);
+24: const [ animIdx, setAnimIdx ] = useState<number>(-1);
アクションの動き出しを削除
検知してから動き出す様にするため、初期開始点を削除
App.tsx
- new Promise(() => setTimeout(() => {0}, 1000)).then(()=>animActions[0].play())
QRコード検知処理
props.detectcount値がインクリメントする=検知した。
App.tsx
+37: useEffect(() => {
+38: if(props.detectcount > 0 && animIdx == -1) {
+39: setAnimIdx(0);
+40: animActions[0].play();
+41: }
+42: }, [props.detectcount])
FPS処理の変更
animIdx == -1の時は処理しない。
setAnimIdx()でアクションをシーケンス制御する。一番最後になったら次のQRコードを検知するまで停止。
App.tsx
54: /* FPS処理 */
55: useFrame((state, delta) => {
+56: if(animIdx == -1) return;
57:
58: if(mixer.current)
59: mixer.current.update(delta);
60: const durationtime: number= animActions[animIdx].getClip().duration
61: const currenttime: number = animActions[animIdx].time
62: if(currenttime/durationtime > 0.9/*90%を超えたら次のモーションへ*/) {
- const index: number = (animIdx+1) % (animCrips.length)
- setAnimIdx( index )
+63: if(animIdx+1 == animCrips.length)
+64: setAnimIdx(-1)
+65: else
+66: setAnimIdx( (animIdx+1) % (animCrips.length) )
67: }
68: });
QRコード検出時の中心点設定関数と検知イベント関数の追加
ついでに検知カウンタ(detectcount)も追加
App.tsx
- const ZxingQRCodeReader = (props:{setSize: React.Dispatch<React.SetStateAction<React.CSSProperties>>}) => {
+75: const ZxingQRCodeReader = (props:{setSize: React.Dispatch<React.SetStateAction<strSize>>, setCenter: React.Dispatch<React.SetStateAction<strPoint>>, setDetecte: React.Dispatch<React.SetStateAction<number>>}) => {
+76: const detectcount = useRef<number>(0);
検知QRコードの中心点算出処理
元の処理は削除。
App.tsx
- if(points.length%4 == 0)
- setPoints4(points);
- else if(points.length%3 == 0)
- setPoints3(points);
+95: /* 中心を求める */
+96: const center: ResultPoint = (points.length==4) ? CenterofLine(points[2], points[0]) :
+97: (points.length==3) ? CrossLineLine( points[0], points[1], points[2], points[3]) :
+98: new ResultPoint(0,0);
+99: props.setCenter({x: center.getX(), y: center.getY()});
+100: props.setDetecte(detectcount.current++);
+101: console.log('aaa detected!! count=', detectcount);
canvas描画処理削除
デバッグ用だったので削除
App.tsx
- const canvas = document.getElementById("canvas") as HTMLCanvasElement
- canvas.width = ref.current.videoWidth;
- canvas.width = ref.current.videoWidth;
- const context = canvas.getContext("2d")
- if(!context) return;
- setContext(context);
QRコード検出時の枠線描画
デバッグ用の描画処理削除。あー、スッキリ。
App.tsx
- {/* デバッグ用 3点取れた時の確認 */}
- useEffect(() => {
- if(!context) return;
- if(!points3) return;
- /* 対角線の中点を求める */
- const xpwr = Math.abs(points3[2].getX() - points3[0].getX());
- const xm = Math.min(points3[2].getX() , points3[0].getX()) + xpwr/2;
- const ypwr = Math.abs(points3[2].getY() - points3[0].getY());
- const ym = Math.min(points3[2].getY() , points3[0].getY()) + ypwr/2;
- const m = new ResultPoint( xm, ym);
- context.clearRect(0, 0, 1920, 1080)
- /* 中点を描画 */
- context.beginPath();
- context.arc(m.getX(), m.getY(), 10, 0, 2*Math.PI, false);
- context.fillStyle = 'green';
- context.fill();
- context.stroke()
- /* 矩形 */
- context.beginPath()
- context.moveTo( points3[0].getX(), points3[0].getY())
- context.lineTo( points3[1].getX(), points3[1].getY())
- context.lineTo( points3[2].getX(), points3[2].getY())
- /* 中心線 */
- context.moveTo( points3[0].getX(), points3[0].getY())
- context.lineTo( m.getX(), m.getY())
- context.moveTo( points3[1].getX(), points3[1].getY())
- context.lineTo( m.getX(), m.getY())
- context.moveTo( points3[2].getX(), points3[2].getY())
- context.lineTo( m.getX(), m.getY())
- /* 描画 */
- context.strokeStyle = "red";
- context.lineWidth = 2;
- context.stroke()
- context.font = "48px serif";
- context.fillText("0",points3[0].getX(), points3[0].getY())
- context.fillText("1",points3[1].getX(), points3[1].getY())
- context.fillText("2",points3[2].getX(), points3[2].getY())
- }, [points3]);
-
- {/* デバッグ用 4点取れた時の確認 */}
- useEffect(() => {
- if(!context) return;
- if(!points4) return;
- context.clearRect(0, 0, 1920, 1080)
- /* 中点を求める */
- const m = CrossLineLine( points4[0], points4[1], points4[2], points4[3]);
- /* 中点を描画 */
- context.beginPath();
- context.arc(m.getX(), m.getY(), 10, 0, 2*Math.PI, false);
- context.fillStyle = 'green';
- context.fill();
- context.stroke()
- /* 矩形 */
- context.beginPath()
- context.moveTo( points4[0].getX(), points4[0].getY())
- context.lineTo( points4[1].getX(), points4[1].getY())
- context.lineTo( points4[2].getX(), points4[2].getY())
- context.lineTo( points4[3].getX(), points4[3].getY())
- /* 対角線 */
- context.moveTo( points4[0].getX(), points4[0].getY())
- context.lineTo( points4[2].getX(), points4[2].getY())
- context.moveTo( points4[1].getX(), points4[1].getY())
- context.lineTo( points4[3].getX(), points4[3].getY())
- context.strokeStyle = "red";
- context.lineWidth = 2;
- context.stroke()
- context.font = "48px serif";
- context.fillText("0",points4[0].getX(), points4[0].getY())
- context.fillText("1",points4[1].getX(), points4[1].getY())
- context.fillText("2",points4[2].getX(), points4[2].getY())
- context.fillText("3",points4[3].getX(), points4[3].getY())
- }, [points4]);
- <>
119: <video ref={ref} />
- {/* デバッグ用canvas */}
- <canvas id="canvas" width="1920" height="1080" style={{ position: "absolute", left: "0px", top: "0px", background: "#0088ff44"}}></canvas>
- </>
中心点を求める関数の実装
線分の中心点を求める関数を追加
App.tsx
+123: /**********************/
+124: /* 線分の中心点を求める */
+125: /**********************/
+126: const CenterofLine = (p00: ResultPoint, p01: ResultPoint) => {
+127: const xpwr = Math.abs(p00.getX() - p01.getX());
+128: const xm = Math.min(p00.getX() , p01.getX()) + xpwr/2;
+129: const ypwr = Math.abs(p00.getY() - p01.getY());
+130: const ym = Math.min(p00.getY() , p01.getY()) + ypwr/2;
+131: return new ResultPoint( xm, ym);
+132: }
中心点設定/検知イベントの実装
中心点設定/検知イベントの実装
App.tsx
+150: type strSize = { width: number; height: number; };
+151: type strPoint= { x: number; y: number; };
152:
153: const App = () => {
154: const [actionName, setActionName] = useState<string>('aaabbb');
155: const [size, setSize] = useState<strSize>({width: 300, height: 200});
+156: const [center, setCenter] = useState<strPoint>({x: 0, y: 0});
+157: const [detectcount, setDetecte] = useState<number>(0);
App.tsx
全体
App.tsx
1: import React, { useEffect, Suspense, useRef, useState, useMemo } from 'react';
2: import './App.css';
3: import { Canvas, useLoader, useFrame } from '@react-three/fiber'
4: import * as THREE from 'three'
5: import { OrbitControls, useFBX } from '@react-three/drei'
6: import { FBXLoader } from "three/examples/jsm/loaders/FBXLoader";
7: import { useZxing } from "react-zxing";
8: import ResultPoint from '@zxing/library/esm/core/ResultPoint';
9:
-const FBXModel = (props:{setActionName: React.Dispatch<React.SetStateAction<string>>}) => {
+10: const FBXModel = (props:{setActionName: React.Dispatch<React.SetStateAction<string>>, detectcount: number}) => {
11: /* FBXモデル読込み */
12: const fbx = useLoader(FBXLoader, "assets/Ch09_nonPBR.fbx");
13: /* AnimationClip(s)読込み */
14: const animCrips: THREE.AnimationClip[][] = []
15: animCrips[0] = useFBX('./assets/BreakdanceEnding2.fbx').animations
16: animCrips[1] = useFBX('./assets/BreakdanceUprockVar1.fbx').animations
17: animCrips[2] = useFBX('./assets/HipHopDancing.fbx').animations
18: animCrips[3] = useFBX('./assets/NorthernSoulSpin.fbx').animations
19: animCrips[4] = useFBX('./assets/SwingDancing.fbx').animations
20: animCrips[5] = useFBX('./assets/BreakdanceEnding1.fbx').animations
21: const animNames = ['BreakdanceEnding2', 'BreakdanceUprockVar1', 'HipHopDancing', 'NorthernSoulSpin', 'SwingDancing', 'BreakdanceEnding1']
22: /* 変数定義 */
23: const mixer = useRef<THREE.AnimationMixer>();
- const [ animIdx, setAnimIdx ] = useState<number>(0);
+24: const [ animIdx, setAnimIdx ] = useState<number>(-1);
25: const animActions = useMemo(() => [] as THREE.AnimationAction[], [])
26:
27: /* 初期化 */
28: useEffect(() => {
29: fbx.scale.multiplyScalar(0.02)
30: mixer.current = new THREE.AnimationMixer(fbx)
31: animCrips.forEach((val: THREE.AnimationClip[], idx: number) => {
32: if(!mixer.current) return;
33: animActions[idx] = mixer.current.clipAction(val[0])
34: })
- new Promise(() => setTimeout(() => {0}, 1000)).then(()=>animActions[0].play())
35: }, [])
36:
37: useEffect(() => {
38: if(props.detectcount > 0 && animIdx == -1) {
39: setAnimIdx(0);
40: animActions[0].play();
41: }
42: }, [props.detectcount])
43:
44: /* モーション切替え処理 */
+45: useEffect(() => {
+46: const act: THREE.AnimationAction = animActions[animIdx]
+47: act?.reset().fadeIn(0.3).play()
+48: props.setActionName(animNames[animIdx] + ' : ' + animIdx)
+49: return () => {
+50: act?.fadeOut(0.3)
+51: }
52: }, [animIdx])
53:
54: /* FPS処理 */
55: useFrame((state, delta) => {
+56: if(animIdx == -1) return;
57:
58: if(mixer.current)
59: mixer.current.update(delta);
60: const durationtime: number= animActions[animIdx].getClip().duration
61: const currenttime: number = animActions[animIdx].time
62: if(currenttime/durationtime > 0.9/*90%を超えたら次のモーションへ*/) {
- const index: number = (animIdx+1) % (animCrips.length)
- setAnimIdx( index )
+63: if(animIdx+1 == animCrips.length)
+64: setAnimIdx(-1)
+65: else
+66: setAnimIdx( (animIdx+1) % (animCrips.length) )
67: }
68: });
69:
70: return (
71: <primitive object={fbx} position={[1, -1, 1]} />
72: )
73: }
74:
- const ZxingQRCodeReader = (props:{setSize: React.Dispatch<React.SetStateAction<React.CSSProperties>>}) => {
+75: const ZxingQRCodeReader = (props:{setSize: React.Dispatch<React.SetStateAction<strSize>>, setCenter: React.Dispatch<React.SetStateAction<strPoint>>, setDetecte: React.Dispatch<React.SetStateAction<number>>}) => {
- {/* デバッグ用canvas */}
- const [context, setContext] = useState<CanvasRenderingContext2D>();
- const [points4, setPoints4] = useState<ResultPoint[]>();
- const [points3, setPoints3] = useState<ResultPoint[]>();
+76: const detectcount = useRef<number>(0);
77: const { ref } = useZxing({
78: constraints: {
79: audio: false,
80: video: {
81: facingMode: 'environment',
82: width: { min: 1024, ideal: 1920, max: 1920 },
83: height: { min: 576, ideal: 1080, max: 1080 },
84: },
85: },
86: timeBetweenDecodingAttempts: 100,
87: onDecodeResult(result) {
88: console.log('onDecodeResult::result=', result);
89: if(result.getResultPoints().length <= 0) return;
90:
91: // setResult(result.getText());
92:
93: const points: ResultPoint[] = result.getResultPoints()
- if(points.length%4 == 0)
- setPoints4(points);
- else if(points.length%3 == 0)
- setPoints3(points);
94:
+95: /* 中心を求める */
+96: const center: ResultPoint = (points.length==4) ? CenterofLine(points[2], points[0]) :
+97: (points.length==3) ? CrossLineLine( points[0], points[1], points[2], points[3]) :
+98: new ResultPoint(0,0);
+99: props.setCenter({x: center.getX(), y: center.getY()});
+100: props.setDetecte(detectcount.current++);
+101: console.log('aaa detected!! count=', detectcount);
102:
103: console.log(points.length, " -----[0]: ", points[0]?.getX(), " ,", points[0]?.getY(),)
104: console.log(points.length, " -----[1]: ", points[1]?.getX(), " ,", points[1]?.getY(),)
105: console.log(points.length, " -----[2]: ", points[2]?.getX(), " ,", points[2]?.getY(),)
106: console.log(points.length, " -----[3]: ", points[3]?.getX(), " ,", points[3]?.getY(),)
107: },
108: });
109:
110: /* Videoサイズ変更に合わせてCanvasサイズを変更する */
111: useEffect(() => {
112: if(!ref.current) return;
113: props.setSize({width: ref.current.videoWidth, height: ref.current.videoHeight});
- const canvas = document.getElementById("canvas") as HTMLCanvasElement
- canvas.width = ref.current.videoWidth;
- canvas.width = ref.current.videoWidth;
- const context = canvas.getContext("2d")
- if(!context) return;
- setContext(context);
114: }, [ref.current?.videoWidth, ref.current?.videoHeight]);
115:
116: console.log("ref.current?.videoxxx=(", ref.current?.videoWidth, ",", ref.current?.videoHeight, ")" );
117:
- {/* デバッグ用 3点取れた時の確認 */}
- useEffect(() => {
- if(!context) return;
- if(!points3) return;
- /* 対角線の中点を求める */
- const xpwr = Math.abs(points3[2].getX() - points3[0].getX());
- const xm = Math.min(points3[2].getX() , points3[0].getX()) + xpwr/2;
- const ypwr = Math.abs(points3[2].getY() - points3[0].getY());
- const ym = Math.min(points3[2].getY() , points3[0].getY()) + ypwr/2;
- const m = new ResultPoint( xm, ym);
- context.clearRect(0, 0, 1920, 1080)
- /* 中点を描画 */
- context.beginPath();
- context.arc(m.getX(), m.getY(), 10, 0, 2*Math.PI, false);
- context.fillStyle = 'green';
- context.fill();
- context.stroke()
- /* 矩形 */
- context.beginPath()
- context.moveTo( points3[0].getX(), points3[0].getY())
- context.lineTo( points3[1].getX(), points3[1].getY())
- context.lineTo( points3[2].getX(), points3[2].getY())
- /* 中心線 */
- context.moveTo( points3[0].getX(), points3[0].getY())
- context.lineTo( m.getX(), m.getY())
- context.moveTo( points3[1].getX(), points3[1].getY())
- context.lineTo( m.getX(), m.getY())
- context.moveTo( points3[2].getX(), points3[2].getY())
- context.lineTo( m.getX(), m.getY())
- /* 描画 */
- context.strokeStyle = "red";
- context.lineWidth = 2;
- context.stroke()
- context.font = "48px serif";
- context.fillText("0",points3[0].getX(), points3[0].getY())
- context.fillText("1",points3[1].getX(), points3[1].getY())
- context.fillText("2",points3[2].getX(), points3[2].getY())
- }, [points3]);
-
- {/* デバッグ用 4点取れた時の確認 */}
- useEffect(() => {
- if(!context) return;
- if(!points4) return;
- context.clearRect(0, 0, 1920, 1080)
- /* 中点を求める */
- const m = CrossLineLine( points4[0], points4[1], points4[2], points4[3]);
- /* 中点を描画 */
- context.beginPath();
- context.arc(m.getX(), m.getY(), 10, 0, 2*Math.PI, false);
- context.fillStyle = 'green';
- context.fill();
- context.stroke()
- /* 矩形 */
- context.beginPath()
- context.moveTo( points4[0].getX(), points4[0].getY())
- context.lineTo( points4[1].getX(), points4[1].getY())
- context.lineTo( points4[2].getX(), points4[2].getY())
- context.lineTo( points4[3].getX(), points4[3].getY())
- /* 対角線 */
- context.moveTo( points4[0].getX(), points4[0].getY())
- context.lineTo( points4[2].getX(), points4[2].getY())
- context.moveTo( points4[1].getX(), points4[1].getY())
- context.lineTo( points4[3].getX(), points4[3].getY())
- context.strokeStyle = "red";
- context.lineWidth = 2;
- context.stroke()
- context.font = "48px serif";
- context.fillText("0",points4[0].getX(), points4[0].getY())
- context.fillText("1",points4[1].getX(), points4[1].getY())
- context.fillText("2",points4[2].getX(), points4[2].getY())
- context.fillText("3",points4[3].getX(), points4[3].getY())
- }, [points4]);
118: return (
- <>
119: <video ref={ref} />
- {/* デバッグ用canvas */}
- <canvas id="canvas" width="1920" height="1080" style={{ position: "absolute", left: "0px", top: "0px", background: "#0088ff44"}}></canvas>
- </>
120: );
121: };
122:
+123: /**********************/
+124: /* 線分の中心点を求める */
+125: /**********************/
+126: const CenterofLine = (p00: ResultPoint, p01: ResultPoint) => {
+127: const xpwr = Math.abs(p00.getX() - p01.getX());
+128: const xm = Math.min(p00.getX() , p01.getX()) + xpwr/2;
+129: const ypwr = Math.abs(p00.getY() - p01.getY());
+130: const ym = Math.min(p00.getY() , p01.getY()) + ypwr/2;
+131: return new ResultPoint( xm, ym);
+132: }
133:
134: /**********************/
135: /* 2線分の交点を求める */
136: /* p1 ------ p2 */
137: /* | | */
138: /* | | */
139: /* | | */
140: /* p0 ------ p3 */
141: /**********************/
142: const CrossLineLine = (p00: ResultPoint, p01: ResultPoint, p02: ResultPoint, p03: ResultPoint) => {
143: const s1: number = ((p02.getX()-p00.getX())*(p01.getY()-p00.getY())-(p02.getY()-p00.getY())*(p01.getX()-p00.getX())) / 2.0;
144: const s2: number = ((p02.getX()-p00.getX())*(p00.getY()-p03.getY())-(p02.getY()-p00.getY())*(p00.getX()-p03.getX())) / 2.0;
145: const x: number = p01.getX()+(p03.getX()-p01.getX()) * s1 / (s1+s2);
146: const y: number = p01.getY()+(p03.getY()-p01.getY()) * s1 / (s1+s2);
147: return new ResultPoint( x, y);
148: }
149:
+150: type strSize = { width: number; height: number; };
+151: type strPoint= { x: number; y: number; };
152:
153: const App = () => {
height: "200px"});
154: const [actionName, setActionName] = useState<string>('aaabbb');
- const [size, setSize] = useState<React.CSSProperties>({width: "300px", +155: const [size, setSize] = useState<strSize>({width: 300, height: 200});
+156: const [center, setCenter] = useState<strPoint>({x: 0, y: 0});
+157: const [detectcount, setDetecte] = useState<number>(0);
158:
159: return (
160: <div>
- <ZxingQRCodeReader setSize={setSize}/>
- <Canvas camera={{ position: [3, 1, 3] }} style={{ position: "absolute", left: "0px", top: "0px", width: `${size.width}px`, height: `${size.height}px`,}}>
+161: <ZxingQRCodeReader setSize={setSize} setCenter={setCenter} setDetecte={setDetecte}/>
+162: <Canvas camera={{ position: [3, 1, 3] }} style={{ position: "absolute", left: `${center.x-(size.width/2)}px`, top: `${center.y-(size.height/2)}px`, width: `${size.width}px`, height: `${size.height}px`,}}>
163: <ambientLight intensity={2} />
164: <pointLight position={[40, 40, 40]} />
165: <Suspense fallback={null}>
- <FBXModel setActionName={setActionName}/>
+166: <FBXModel setActionName={setActionName} detectcount={detectcount}/>
167: </Suspense>
168: <OrbitControls />
169: <axesHelper args={[5]} />
170: <gridHelper />
171: </Canvas>
172: <div id="summry" style={{background: "rgba(255, 192, 192, 0.7)"}}>{actionName}</div>
173: </div>
174: );
175: }
176:
177: export default App;
で、実行。
出来た!!
QRコードの上に3Dモデル描画が出来ている。
トラッキングが弱いけど。。。
<- [第4回]React+TypeScriptなWebアプリで、QRコードをARしてみた。(検出補助線を引く編)
[第6回]React+TypeScriptなWebアプリで、QRコードをARしてみた。(リリース編) ->
Discussion