Efficient OpenCV in Unity using direct OpenGL rendering

What this post is about

I had the need for using Unity to visualize OpenCV data efficiently, without having to break the programming of my numerous C++ DLLs. This post will explain how to share a texture handle created in Unity with your plugin DLL, allowing it to directly modify the OpenGL buffer without having to transfer or convert any image data between the C++  plugin code and Unity.

This method should theoretically work with DirectX too, but I leave the technical details to any interested parties.

Feel free to fork the code on GitHub at your leisure.

What this post isn't about

This will not be a copy/paste the code and get it working kind of deal. There is provided code, yes, but it will probably not do everything you want it to do. which is why I will be explaining the ideas and architecture behind the code, to help you better understand and change it to suit your needs.

Using OpenCV in Unity

Setting up Unity to use OpenGL

First of all you need to set up Unity to use OpenGL rendering (it uses DirectX by default). You can do this by accessing the Player Setting from Edit->Settings->Player and go to the Other Settings tab:

Just uncheck Auto Grahics API for each platform and move OpenGLCore to the top.

Creating a placeholder material

Just create an unlit texture material, it will act as a placeholder for OpenGL rendering later on. Give it a name: I called it empty for this post's purposes.

Now just create a cube and affect the empty material to it.

A look at the native plugin code

In case you haven't done it, you can get the code on GitHub. It uses a standard Unity low-level native plugin interface, which I will not explain as it is beyond the scope of this post.

The code for the DLL is in the OpenCVPlugin folder. It contains a Visual Studio 2015 solution which has to be modified to use the correct paths for your OpenCV libraries if you want to use it directly. It also uses SWIG to compile the interface between C++ and C#. More on that below.

The meat of the work is done in OpenCVDllInterface.cpp. First we need to store the handle of the texture somewhere:

void OpenCVDllInterface::setTextureHandle(void* handle)
{
 m_textureHandle = (GLuint)(size_t)handle;
}

And now that we have our texture handle, we just update it every frame with whatever OpenCV data we want:

void OpenCVDllInterface::updateFrameDataOGL(int eventID)
{
 if (m_captureDevice.isOpened())
 {
  cv::Mat originalFrame;
  m_captureDevice >> originalFrame;

  if (!originalFrame.empty())
  {
   cv::Mat frame;

   flip(originalFrame, frame, 0);

   size_t currentFrameSize = frame.total() * 3;

   if (m_frameBufferSize < currentFrameSize)
   {
    m_frameBufferSize = currentFrameSize;
    m_frameBuffer = (uchar*)realloc(m_frameBuffer, m_frameBufferSize);
   }

   Mat continuousRGB(frame.size(), CV_8UC4, m_frameBuffer);
   cvtColor(frame, continuousRGB, CV_BGR2RGB, 3);

   glBindTexture(GL_TEXTURE_2D, m_textureHandle);

   glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
   glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);

   // Set texture clamping method
   glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP);
   glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP);

   //set length of one complete row in data (doesn't need to equal image.cols)
   glPixelStorei(GL_UNPACK_ROW_LENGTH, (int)(frame.step / frame.elemSize()));

   glTexSubImage2D(GL_TEXTURE_2D,
    0,
    0,
    0,
    continuousRGB.size().width,
    continuousRGB.size().height,
    GL_RGB,
    GL_UNSIGNED_BYTE,
    continuousRGB.data);
  }
 }
}

This code just directly outputs the video capture device but it can easily be adapted to display any image manipulations OpenCV can store in a cv::Mat.

And finally, a method to inform Unity of the size of the texture we will need it to create:

void OpenCVDllInterface::getFrameBufferInfo(FrameInfo& bufferInfo)
{
 if (m_captureDevice.isOpened())
 {
  bufferInfo.width = int(m_captureDevice.get(CAP_PROP_FRAME_WIDTH));
  bufferInfo.height = int(m_captureDevice.get(CAP_PROP_FRAME_HEIGHT));

  cv::Mat frame;
  m_captureDevice >> frame;
  if (frame.empty())
  {
   bufferInfo.sizeInBytes = 0;
  }
  else
  {
   bufferInfo.sizeInBytes = unsigned int(frame.total() * 3);
  }
 }
}

This assumes RGB byte sized data (hence the *3 to compute buffer size) and has to be adapted to your own needs.

Scripting in Unity

The Unity project is located in the OpenCVUnityProject folder. Let's take a look at the OpenCVBridge.cs script in the Assets/Scripts folder.

On Start(), we need to ask the DLL, about the texture we need to create, create said texture, and pass it on to the DLL. We also need to setup an event callback to trigger the OpenGL update from within the DLL using a delegate function in a coroutine:

IEnumerator Start() {
        m_dllInterface = new OpenCVDllInterface();
        m_frameInfo = new OpenCVDllInterface.FrameInfo();

        m_eventCallback = new eventCallbackDelegate(m_dllInterface.updateFrameDataOGL);

        m_dllInterface.getFrameBufferInfo(m_frameInfo);


        m_texture = new Texture2D(m_frameInfo.width, m_frameInfo.height, TextureFormat.RGB24, false);
        m_texture.filterMode = FilterMode.Point;
        m_texture.Apply();
        m_material.mainTexture = m_texture;

        m_dllInterface.setTextureHandle(m_texture.GetNativeTexturePtr());

        yield return StartCoroutine("CallPluginAtEndOfFrames");
    }

And the coroutine itself is pretty straightforward. Just call the correct method in the DLL:

private IEnumerator CallPluginAtEndOfFrames()
    {
        if (m_eventCallback != null)
        {
            while (m_dllInterface != null)
            {
                yield return null;

                GL.IssuePluginEvent(Marshal.GetFunctionPointerForDelegate(m_eventCallback), 1);
            }
        }
    } 

Now just attach the script to the cube we created earlier, you know, the one with the empty material. And don' forget to affect the empty material to the script.

Communicating between C++ and C#

Usually when communicating between C++ and C# there's a whole slew of manual dllimport directives and other hassles. But luckily, you can avoid all this if you use SWIG. I will definitely not go into the details of SWIG so feel free to browse their website for more information.

The Visual Studio solution on GitHub is already configured to automatically run SWIG and compile its output. If you wan to know how that is done, you can check out this page on Stackoverflow. It describes the process for Python but is easily modifiable for C#.

The SWIG interface file is pretty straightforward, you can find it in the OpenCVPlugin/Sources folder:

%module OpenCVPlugin

%{
#include "OpenCVDllInterface.h"
%}

//Use System.IntPtr from C# for void* pointers
%typemap( cstype ) void* "System.IntPtr"
%typemap( csin ) void* %{ $csinput %}
%typemap( imtype ) void* "System.IntPtr"

%include "OpenCVDllInterface.h"

The file is pretty straightforward. The only code gymnastics here instruct SWIG to user C# System.IntPtr when it encouters void* in C++.

Putting it all together

Before compiling your DLL make sure you use the same architecture as your Unity editor: a 64-bit Unity Editor will only work with a 64-bit DLL plugin.

When you compile your C++ project, SWIG will create .cs C# script files in the same folder as your DLL. If it's not already there, create the Assets/Plugins folder in your Unity project and copy the DLL and the generated scripts.

Run it and enjoy!

Using a canvas for AR applications

For real AR applications, displaying the OpenCV video feed on a cube is not the best approach. What can be done is assign the empty material to a Unity canvas that is attached to the main camera, with coordinates in camera space, set at a distance a little below the camera's far plane. The canvas aspect ratio should match the OpenCV video aspect ratio.

With that, you should be golden.