In this example, we will try to get a vision application running on the PLC itself. We are using open-source frameworks as:

Please note, that this is a fairly advanced topic, and as many steps are involved the basics are not explained in detail.

Requirements

We need the following equipment:

image

image

image

image

Assumptions

Settings up Build Environment

Please ensure you have a working Linux workstation + the Phoenix PLCNext SDK installed in /opt/pxc/<current I advise to make a symlink

We need to run the following commands:

vincent@vincent-VirtualBox ~ $ source /opt/pxc/current/environment-setup-cortexa9t2hf-neon-pxc-linux-gnueabi

This configures the crosscompiler provided by the Phoenix SDK.

We created an extra file (/opt/pxc/setupCrossCompiler) which has the content:

export ARMPREFIX=/opt/pxc/install
export PKG_CONFIG_PATH=$ARMPREFIX/lib/pkgconfig:$PKG_CONFIG_PATH

We also need to run source /opt/pxc/setupCrossCompiler, to enable this variable.

Last, we create a folder: /opt/pxc/install wich we use to install our cross-compiled libaries into, compiled for the PLCNext platform

::: {.tip} ::: {.admonition-title} Tip :::

Run the source <filename in every terminal session you are working in, as it is limited to the instance executed in, in this example for both files, so twice! :::

Compiling FFMPEG

We also will compile ffmpeg which contains various codecs, usefull for OpenCV.

proceed with the following steps:

git clone https://github.com/FFmpeg/FFmpeg.git

vincent@vincent-VirtualBox /opt/pxc/ $ cd ffmpeg
git checkout release/2.6

Now we have checked-out this version.

for FFMpeg I've created a file to ease configuration, due to the extensive options. Create the file config.sh in the root of our ffmpeg directory, and make it executable by issuing chmod +x config.sh.

This config.sh file has to containg following content:

export CFLAGS="-mfpu=neon -mfloat-abi=hard -I${ARMPREFIX}/include"
export LDFLAGS="-L${ARMPREFIX}/lib"
export SYSR=/opt/pxc/current/sysroots/cortexa9t2hf-neon-pxc-linux-gnueabi

export FFMPEG_FLAGS="--target-os=linux \
--arch=arm \
--enable-shared \
--enable-pic \
--disable-static \
--disable-symver \
--disable-doc \
--disable-ffplay \
--disable-ffprobe \
--disable-ffserver \
--disable-avfilter \
--disable-encoders \
--disable-muxers \
--disable-filters \
--disable-devices \
--disable-everything \
--disable-encoder=aac \
--disable-libgme \
--disable-libgsm \
--disable-libiec61883 \
--disable-libilbc \
--disable-libmodplug \
--disable-libmp3lame \
--disable-libopencore-amrwb \
--disable-libopencv \
--disable-libopenh264 \
--disable-libopenjpeg \
--disable-libopus \
--disable-libpulse \
--disable-librtmp \
--disable-libshine \
--disable-libsmbclient \
--disable-libsoxr \
--disable-libspeex \
--disable-libssh \
--disable-libtheora \
--disable-libtwolame \
--disable-libv4l2 \
--disable-libvidstab \
--disable-libvo-amrwbenc \
--disable-libvorbis \
--disable-libvpx \
--disable-libwavpack \
--disable-libwebp \
--disable-libx264 \
--disable-libx265 \
--disable-libxavs \
--disable-libxcb \
--disable-libxcb-shm \
--enable-protocols  \
--enable-parsers \
--enable-demuxers \
--enable-decoders \
--enable-muxers \
--enable-bsfs \
--enable-network \
--enable-postproc \
--enable-swscale  \
--enable-ffmpeg \
--enable-avfilter \
--enable-avresample \
--enable-pthreads \
--disable-stripping \
--disable-iconv \
--disable-xlib \
--disable-zlib \
--disable-bzlib \
--disable-lzma \
--disable-demuxer=sbg \
--disable-demuxer=dts \
--disable-parser=dca \
--disable-decoder=dca \
--enable-ffmpeg \
--sysroot=${SYSR} \
--prefix=${ARMPREFIX} \
--cross-prefix=arm-pxc-linux-gnueabi-
"

export EXTRA_CFLAGS="-I ${ARMPREFIX}/include"
export LDFLAGS="-L ${ARMPREFIX}/lib"

./configure $FFMPEG_FLAGS --extra-cflags="$CFLAGS $EXTRA_CFLAGS" --extra-ldflags="$LDFLAGS"

With this file saved, it is time to execute it, and proceed with the build:

chmod  +x config.sh
./config.sh
make -j2
make install

The make -j2 build the ffmpeg, with 2 concurrent processes. The make install command installes our crosscompiled files into ${ARMPREFIX} directory, used by our other crosscompiled applications. These libraries here, will also be copied to the target later on.

<div note=“note”

The -j2 suffix tells make to use to concurent processes, one for every available CPU core (this will speedup compilation on multi-core systems) </div

to test if ffmpeg works on our PLCNext platform and can extract imagery from the webcam, we can copy the ffmpeg binary to th CURRENTPLC to test if ffmpeg has indeed compiled correctly and can access our webcam:

#navigate to the root folder of ffmpeg
scp ffmpeg root@192.168.1.10:/tmp/

Now ssh to the plc ssh root@192.168.1.10 and execute:

cd /tmp
./ffmpeg -i http://root:root@192.168.1.2/mjpg/video.mjpg -r 10 -f image2  test%03d.jpg 

ffmpeg version N-86746-g2f84f40 Copyright (c) 2000-2017 the FFmpeg developers
built with gcc 6.2.0 (GCC)
configuration: --target-os=linux --arch=arm --enable-static --disable-symver --disable-doc --disable-ffplay --disable-ffprobe --disable-ffserver --disable-avfilter --disable-encoders --disable-muxers --disable-filters --disable-devices --disable-everything --enable-protocols --enable-parsers --enable-demuxers --enable-decoders --enable-encoders --enable-muxers --enable-bsfs --enable-network --enable-postproc --enable-swscale --enable-ffmpeg --enable-avfilter --enable-avresample --disable-iconv --disable-xlib --disable-zlib --disable-bzlib --disable-lzma --disable-demuxer=sbg --disable-demuxer=dts --disable-parser=dca --disable-decoder=dca --enable-ffmpeg --enable-version3 --sysroot=/opt/pxc/current/sysroots/cortexa9t2hf-neon-pxc-linux-gnueabi --disable-x86asm --prefix=/opt/pxc/install --cross-prefix=arm-pxc-linux-gnueabi- --extra-cflags='-mfpu=neon -mfloat-abi=hard -I/opt/pxc/install/include ' --extra-ldflags=-L/opt/pxc/install/lib
libavutil      55. 67.100 / 55. 67.100
libavcodec     57.100.104 / 57.100.104
libavformat    57. 75.100 / 57. 75.100
libavdevice    57.  7.100 / 57.  7.100
libavfilter     6. 95.100 /  6. 95.100
libavresample   3.  6.  0 /  3.  6.  0
libswscale      4.  7.101 /  4.  7.101
libswresample   2.  8.100 /  2.  8.100
Input #0, mpjpeg, from 'http://root:root@192.168.1.2/mjpg/video.mjpg':
Duration: N/A, bitrate: N/A
    Stream #0:0: Video: mjpeg, yuvj420p(pc, bt470bg/unknown/unknown), 320x240 [SAR 1:1 DAR 4:3], 25 tbr, 25 tbn, 25 tbc
Stream mapping:
Stream #0:0 -#0:0 (mjpeg (native) -mjpeg (native))
Press [q] to stop, [?] for help
Output #0, image2, to 'test%03d.jpg':
Metadata:
    encoder         : Lavf57.75.100
    Stream #0:0: Video: mjpeg, yuvj420p(pc), 320x240 [SAR 1:1 DAR 4:3], q=2-31, 200 kb/s, 10 fps, 10 tbn, 10 tbc
    Metadata:
    encoder         : Lavc57.100.104 mjpeg
    Side data:
    cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: -1
frame=   30 fps= 13 q=24.8 Lsize=N/A time=00:00:03.00 bitrate=N/A dup=0 drop=41 speed=1.29x    
video:188kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Exiting normally, received signal 2.

after a few seconds we can cancel by pressing ctrl+c and when executing ls we see that we have a lot of new files:

root@axcf2152:/tmp# ls
OpenCv_Cam   test006.jpg  test015.jpg  test024.jpg
ffmpeg       test007.jpg  test016.jpg  test025.jpg
test         test008.jpg  test017.jpg  test026.jpg
test.jpg     test009.jpg  test018.jpg  test027.jpg
test001.jpg  test010.jpg  test019.jpg  test028.jpg
test002.jpg  test011.jpg  test020.jpg  test029.jpg
test003.jpg  test012.jpg  test021.jpg  test030.jpg
test004.jpg  test013.jpg  test022.jpg
test005.jpg  test014.jpg  test023.jpg
root@axcf2152:/tmp#

copying the files over to the pc by executing

scp root@192.168.1.10:/tmp/test*.jpg .

Will copy the images to the local directory, where we might view them.

Compiling OpenCV

Create the following directory: /opt/pxc/opencv-arm/ and create a subdirectory build

Now we clone the OpenCV Git-repository:

cd /opt/pxc/opencv-arm
git clone https://github.com/Itseez/opencv.git
cd opencv       
git checkout 2.4.13.2

We need to create a toolchain.cmake file.

Create the file in /opt/pxc/opencv-arm/

its content:

#!/bin/sh

set(ENV{PKG_CONFIG_PATH} $ENV{ARMPREFIX}/lib/pkgconfig)
set(ENV{LD_LIBRARY_PATH} $ENV{ARMPREFIX}/lib)
set(ENV{C_INCLUDE_PATH} $ENV{ARMPREFIX}/include)
set(ENV{CPLUS_INCLUDE_PATH} $ENV{ARMPREFIX}/include)

set(CMAKE_LIBRARY_PATH ${CMAKE_LIBRARY_PATH} $ENV{ARMPREFIX}/lib)
set(MY_FLAGS "-I$ENV{ARMPREFIX}/include -L$ENV{ARMPREFIX}/lib -lswscale -lavformat -lavutil -lswresample -lavcodec")

set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}  ${MY_FLAGS}")
set(CMAKE_C_FLAGS   "${CMAKE_C_FLAGS}  ${MY_FLAGS}")

set(CMAKE_INSTALL_PREFIX $ENV{ARMPREFIX})

now we create a start.sh file in /opt/pxc/opencv-arm/build/ to configure cmake:

cmake -DENABLE_PRECOMPILED_HEADERS=OFF -DWITH_FFMPEG=ON -DWITH_JPEG=ON -DCMAKE_TOOLCHAIN_FILE=../toolchain.cmake -D CMAKE_BUILD_TYPE=RELEASE ../opencv

When this is completed run:

Building a dice counter

Now we have our libraries ready, we can put them to good use.

First in Eclipse, we create a new C++ program. And we add an aditional Build Configuration. This enables us to build our application for the desktop and the targetplatform as well, so we can test our code (using the GUI) on the pc, and cross-compile using our Phoenix SDK and cross-compiled libraries. I've called it PNext, and set the toolchain in accordance to the way libraries are build.

image

image

The Compiler

The compiler (both gcc and g++ ) needs to know where it can find the OpenCV Header files, so we add this.

image

image

projectsettings -G++ Compiler -*AND*** projectsettings -GCC Compiler -

/opt/pxc/install/include

The Linker

In order to ensure our program is linked against our precomiled libraries, we need to add this list as linker-libraries:

image

image

In the Libraries (-l) field, add:

``` {.sourceCode .shell} opencv_objdetect opencv_stitching opencv_videostab opencv_calib3d opencv_features2d opencv_highgui opencv_ml opencv_video opencv_ocl opencv_photo opencv_imgproc opencv_flann opencv_core libprotobuf libjpeg libwebp libpng libtiff libjasper IlmImf tegra_hal swscale avformat avutil swresample avcodec opencv_nonfree opencv_gpu

Library search path (-L)

/opt/pxc/install/lib
/opt/pxc/install/share/OpenCV/3rdparty/lib

The Actual Program

Our fairly simple program does:

  1. Get image from IPcam
  2. Makes the image blurry for faster computation
  3. Counts the full circles (effectively the dots on the dice).
  4. Prints the # of circles to the console:

``` {.sourceCode .c++} //============================================================================

// Name : OpenCv_Cam.cpp // Author : Vincent Gijsen // Version : 0.1 // Copyright : // Description : //============================================================================ /* * main.c * * Created on: Jul 29, 2017 * Author: Vincent Gijsen */

#include <opencv2/core/core.hpp\#include <opencv2/highgui/highgui.hpp\#include "opencv2/imgproc/imgproc.hpp"

#include "iostream"

using namespace cv; using namespace std;

int main(int argc, char** argv) {

Mat image; Mat img_gray, img_blur, img_small; Scalar Color; size_t i;

string cam = "0 <http://root:root@192.168.1.2/mjpg/video.mjpg"; string img = "1 /tmp/image.png";

if (argc < 3) {
cout << "usage: " << argv[0] << " " << cam << " || " << img << endl; exit(1);

}

string input = argv[2]; string cmd = argv[1];

VideoCapture vcap;

if (cmd == "0") {

vcap.open(input); if (!vcap.isOpened()) { cout << "Error opening videostream: " << cam << endl; return -1;

}

}

for (;;) {
if (cmd == "0") {

//We are in film mode if (!vcap.read(image)) { cout << "no frame" << endl;

waitKey();

}

} else {

cout << "Reading image: " << input << endl; //We are in image mode image = imread(input, CV_LOAD_IMAGE_COLOR);

}

cvtColor(image, img_gray, CV_BGR2GRAY);

vector<Vec3f;

blur(img_gray, img_blur, Size(10, 10)); HoughCircles(img_blur, circles, CV_HOUGH_GRADIENT, 1, img_blur.rows / 4, 150, 25, 0, 0); ;

if (circles.size() ) {

cout << "Found " << circles.size() << " dots" << endl;

// Draw for fun! for (i = 0; i < circles.size(); ++i) { Point pip(cvRound(circles[i][0]), cvRound(circles[i][1])); circle(image, pip, 3, Scalar(0, 255, 0), -1, 8, 0); }

}

if (cmd == "1") {
break;

}

}

}

```

We need to to compile our program.

If all is well, the Console in Eclipse should look like:

Copying the Program and our Libraries

As our program is called OpenCV_Cam, we need to copy this binary.

For testing purposes, I chose to upload everything to /tmp on the controller. Do keep in mind, this /tmp is mapped to RAM, and therefore:

  1. Limited in available space
  2. Wiped on powercycle/loss
  3. Therefore fairly safe to experiment on, as changes are not persistend.

to copy the program, navigate in our project-directory (ecplise) to the build directory (name of the build-configuration) and copy the file:

scp OpenCv_Cam root@192.168.1.10:/tmp

We also need to copy all of our libraris to this directory (or to say it proper, copy it to direcotry that is either indexed by ldconfig, or instructed to be indext).

navigate to our /opt/pxc/install/lib directory.

scp *.4 root@192.168.1.10:/tmp
scp libav* root@192.168.1.10:/tmp
scp libsw* root@192.168.1.10:/tmp

we no have the libraries copied, but executables are not eable to use them.

When we issue the ldconfig -p on the plc, we see this:

root@axcf2152:/tmp# ldconfig -p
65 libs found in cache `/etc/ld.so.cache`
libz.so.1 (libc6,hard-float) =/lib/libz.so.1
libxml2.so.2 (libc6,hard-float) =/usr/lib/libxml2.so.2

...

we need to execute these commands to add them to the LDCACHE:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/tmp
ldconfig

now our new libraries are also available:

root@axcf2152:/tmp# ldconfig -p
91 libs found in cache `/etc/ld.so.cache`
...
libswscale.so.3 (libc6,hard-float) =/tmp/libswscale.so.3
libswresample.so.1 (libc6,hard-float) =/tmp/libswresample.so.1
libopencv_videostab.so.2.4 (libc6,hard-float) =/tmp/libopencv_videostab.so.2.4
libopencv_video.so.2.4 (libc6,hard-float) =/tmp/libopencv_video.so.2.4
libopencv_superres.so.2.4 (libc6,hard-float) =/tmp/libopencv_superres.so.2.4
libopencv_stitching.so.2.4 (libc6,hard-float) =/tmp/libopencv_stitching.so.2.4
libopencv_photo.so.2.4 (libc6,hard-float) =/tmp/libopencv_photo.so.2.4
libopencv_ocl.so.2.4 (libc6,hard-float) =/tmp/libopencv_ocl.so.2.4
libopencv_objdetect.so.2.4 (libc6,hard-float) =/tmp/libopencv_objdetect.so.2.4
libopencv_nonfree.so.2.4 (libc6,hard-float) =/tmp/libopencv_nonfree.so.2.4
libopencv_ml.so.2.4 (libc6,hard-float) =/tmp/libopencv_ml.so.2.4
libopencv_legacy.so.2.4 (libc6,hard-float) =/tmp/libopencv_legacy.so.2.4
libopencv_imgproc.so.2.4 (libc6,hard-float) =/tmp/libopencv_imgproc.so.2.4
libopencv_highgui.so.2.4 (libc6,hard-float) =/tmp/libopencv_highgui.so.2.4
libopencv_gpu.so.2.4 (libc6,hard-float) =/tmp/libopencv_gpu.so.2.4
libopencv_flann.so.2.4 (libc6,hard-float) =/tmp/libopencv_flann.so.2.4
libopencv_features2d.so.2.4 (libc6,hard-float) =/tmp/libopencv_features2d.so.2.4
libopencv_core.so.2.4 (libc6,hard-float) =/tmp/libopencv_core.so.2.4
libopencv_contrib.so.2.4 (libc6,hard-float) =/tmp/libopencv_contrib.so.2.4
libopencv_calib3d.so.2.4 (libc6,hard-float) =/tmp/libopencv_calib3d.so.2.4
libavutil.so.54 (libc6,hard-float) =/tmp/libavutil.so.54
libavresample.so.2 (libc6,hard-float) =/tmp/libavresample.so.2
libavformat.so.56 (libc6,hard-float) =/tmp/libavformat.so.56
libavfilter.so.5 (libc6,hard-float) =/tmp/libavfilter.so.5
libavdevice.so.56 (libc6,hard-float) =/tmp/libavdevice.so.56
libavcodec.so.56 (libc6,hard-float) =/tmp/libavcodec.so.56
...

Running the program

Now we are ready to run the program:

cd /tmp
./OpenCv_Cam 0 http://root:root@192.168.1.2/mjpg/video.mjpg

If we present our 'dice' to the camera:

image

image

If now 'dots are recognized, these will be printed

image

image

Here you see the program running:

image

image