Compiling Ceres Solver on Windows with SuiteSparse Support

It took me a while to get all the CMake parameters just right, and a few colleagues needed to do the same. So I decided to write this document and share it with the world. Hope it helps someone, somewhere, sometime.

Environment

My building environment is as follows:

• Windows 7 (yeah our work PCs are obsolete). But it should work on 10.
• Visual Studio 2015, 64 bits compiler
• CMake 3.10.3

It might work in other environments, but I haven't tested it, sorry...

Downloads

Here are the versions of Ceres and it's dependencies I used:

• Ceres Solver 1.1.4
• Eigen 3.3.4
• glog 0.3.5
• gflags 2.2.1
• SuiteSparse Metis for Windows 1.3.1

So download these, unzip them, and let's get to work.

Configuring and building dependencies

Eigen

Nothing to do here really, just unzip it somewhere.

gflags

• Run CMake.
• Set the CMAKE_INSTALL_PREFIX if you want.
• Build and install.

The problem with gflags is that its Debug and Release targets produce the same libraries, with the exact same name, in the exact same folder. So, with the default CMAke settings, you can only have either Debug or Release, but not both.

And since both glog and ceres depend on gflags, you have to choose the same configuration for both of them. However, you might get away with compiling glog without the gflags dependency (it compiles but I haven't tested it any further).

However, you can have CMake generate a Debug library with a suffix, so it will generate gflagsd.lib. To do so you just need to click on the "+Add Entry" button in your CMake GUI and create the following variable:

• CMAKE_DEBUG_POSTFIX, Type String, Value d.

glog

• Run CMake.
• Set the SUITESPARSE_INSTALL_PREFIX if you want.
• It should detect the gflags install folder automatically. If not, specify it manually.

• Build and install.

Don't forget to build the same configuration (Debug or Release) as gflags. Or if you want to have both, just do as above and add the CMAKE_DEBUG_POSTFIX CMake variable.

SuiteSparse

If you're using Visual Studio 2013 or newer (which you should be because this guide assumes you're using VS20015), there is a problem with the declaration of rint().

Int the folder where you unzipped SuiteSparse go to the folder metis/GKlib, open the file gk_arch.h. At line 62 you should see this code block:

#ifdef __MSC__
/* MSC does not have rint() function */
#define rint(x) ((int)((x)+0.5))

Replace it with this:

#ifdef __MSC__
#if (_MSC_VER < 1800)
/* MSC does not have rint() function */
#define rint(x) ((int)((x)+0.5))  
#endif

Now you can:

• Run CMake.
• Set the CMAKE_INSTALL_PREFIX if you want.
• Build and install.

Configuring and building Ceres Solver

This section assumes you will be building a Release version of ceres. For building in Debug, see section Building in Debug below.

Fire up CMake, click Configure and let's get to work.

First, we need to set up a few things for the first configuration run:

1. Set CMAKE_INSTALL_PREFIX if you want.
2. Set EIGEN_INCLUDE_DIR to the folder where you unzipped Eigen earlier.
3. Make sure CUSTOM_BLAS is checked.
4. Make sure LAPACK is checked.
5. Make sure SUITESPARSE is checked.
6. Click Configure.

THE CONFIGURATION WILL FAIL!

Now, before going any further, you need to check the Advanced box in the CMake GUI:

YOU NEED TO RECHECK SUITESPARSE and LAPACK BECAUSE THEY ARE AUTOMATICALLY UNCHECKED EVERY TIME CMAKE FAILS TO FIND THEM!

We now need to manually st ALL dependcies, one by one. I will be using SUITESPARSE_INSTALL_DIR to refer to the folder where you installed SuiteSparse. It should be the same value you gave CMake in SUITESPARSE_INSTALL_PREFIX when you configured and built SuiteSparse:

7. Set AMD_INCLUDE_DIR to SUITESPARSE_INSTALL_DIR/include/suitesparse
8. Set AMD_LIBRARY to SUITESPARSE_INSTALL_DIR/lib64/libamd.lib
9. Set BLAS_blas_LIBRARY to SUITESPARSE_INSTALL_DIR/lib64/lapack_blas_windows/libblas.lib
10. Set CAMD_INCLUDE_DIR to SUITESPARSE_INSTALL_DIR/include/suitesparse
11. Set CAMD_LIBRARY to SUITESPARSE_INSTALL_DIR/lib64/libcamd.lib
12. Set CCOLAMD_INCLUDE_DIR to SUITESPARSE_INCLUDE_DIR/include/suitesparse
13. Set CCOLAMD_LIBRARY to SUITESPARSE_INSTALL_DIR/lib64/libccolamd.lib
14. Set CHOLMOD_INCLUDE_DIR to SUITESPARSE_INSTALL_DIR/include/suitesparse
15. Set CHOLMOD_LIBRARY to SUITESPARSE_INSTALL_DIR/lib64/libcholmod.lib
16. Set COLAMD_INCLUDE_DIR to SUITESPARSE_INSTALL_DIR/include/suitesparse
17. Set COLAMD_LIBRARY to SUITESPARSE_INSTALL_DIR/lib64/libcolamd.lib
18. Set METIS_LIBRARY to SUITESPARSE_INSTALL_DIR/lib64/metis.lib
19. Set SUITESPARSEQR_INCLUDE_DIR to SUITESPARSE_INSTALL_DIR/include/suitesparse
20. Set SUITESPARSEQR_LIBRARY to SUITESPARSE_INSTALL_DIR/lib64/libspqr.lib
21. Set SUITESPARSE_CONFIG_INCLUDE_DIR to SUITESPARSE_INSTALL_DIR/include/suitesparse
22. Set SUITESPARSE_CONFIG_LIBRARY to SUITESPARSE_INSTALL_DIR/lib64/suitesparseconfig.lib
23. Check CUSTOM_BLAS, if it's not already checked.
24. Check LAPACK, if it's not already checked.
25. Check SUITE_SPARSE, if it's not already checked.
26. Finally, set CMAKE_BUILD_TYPE to Release.

Click Configure again. IT WILL FAIL AGAIN!

But now we can setup the final missing variables concerning Lapack.

YOU NEED TO RECHECK SUITESPARSE and LAPACK BECAUSE THEY ARE AUTOMATICALLY UNCHECKED EVERY TIME CMAKE FAILS TO FIND THEM!

27. Set LAPACK_lapack_LIBRARY to SUITESPARSE_INSTALL_DIR/lib64/lapack_blas_windows/liblapack.lib
28. Check LAPACK
29. Check SUITE_SPARSE

Your final CMake options should look like this (click to enlarge and/or download full size image):

You can now build the Release version, install and enjoy!

Building in Debug

• First you need to build gflags and glog in Debug.
• In step 26 above, set CMAKE_BUILD_TYPE to Debug.
• Add the suffix d to all libraries in steps, 8, 11, 13, 15, 17, 18, 20 and 22. (i.e. libamd.lib becomes libamdd.lib). Do NOT do this for BLAS and LAPACK.
• Add the flag /bigobj to CMAKE_CXX_FLAGS_DEBUG.

Configure and build (in Debug of course).

Troubleshooting the Debug Build in your project

First, check out how to correctly use ceres with CMake on the official ceres installation page.

This works perfectly if you have either a Release build or a Debug build, but sometimes, not all the time, for some reason I cannot understand, CMake doesn't generate the proper include paths for glog and gflags for the Debug config of your project.

So you just need to simply manually add them to your CMake file, either hardcoded (which is bad, bad, bad, shame on you!) or using some kind of Environment or CMake variable (because, for another reason I don't understand GLOG_INCLUDE_DIRS and GFLAGS_INCLUDE_DIRS stay blank even though CMake finds the libraries and paths).

Here's what mine looks like:

set( INCLUDE_DIRS "$ENV{THIRD_PARTY_LIBS}/glog/include;${INCLUDE_DIRS}")
set( INCLUDE_DIRS "$ENV{THIRD_PARTY_LIBS}/gflags/include;${INCLUDE_DIRS}")

Troubleshooting

If you encounter the following errors, here's how to solve them:

Error LNK2038

gflags_static.lib(gflags.obj) : error LNK2038: mismatch detected for '_ITERATOR_DEBUG_LEVEL': value '0' doesn't match value '2' in stl_logging_unittest.obj

It's Debug/Release mismatch.It means you're trying to build glog or ceres in Debug when you compiled gflags in Release (or the opposite). You need to compile everything in either Debug or Release.

Error MSB3491

Could not write lines to file "ba_iterschur_suitesparse_clustjacobi_auto_threads_test.dir\Release\ba_iters.8A65E343.tlog\ba_iterschur_suitesparse_clustjacobi_auto_threads_test.lastbuildstate". The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters.

Windows has a limit on the number of characters in a path. Try moving your ceres source files to a shallower folder with a shorter name. In my example, the source files are in D:/Dev_MAW/ceres.

Fatal Error C1128

cwisenullaryop.h(84): fatal error C1128: number of sections exceeded object file format limit: compile with /bigobj

You are building in Debug and forgot to add the /bigobj flag to CMAKE_CXX_FLAGS_DEBUG as explained in the Building in Debug section above.

Fatal Error C1083

fatal error C1083: Cannot open include file: 'glog/glog.h': No such file or directory

Try to manually add the include paths for gflags and glog as explained here.

Efficient OpenCV in Unity using direct OpenGL rendering

What this post is about

I had the need for using Unity to visualize OpenCV data efficiently, without having to break the programming of my numerous C++ DLLs. This post will explain how to share a texture handle created in Unity with your plugin DLL, allowing it to directly modify the OpenGL buffer without having to transfer or convert any image data between the C++  plugin code and Unity.

This method should theoretically work with DirectX too, but I leave the technical details to any interested parties.

Feel free to fork the code on GitHub at your leisure.

What this post isn't about

This will not be a copy/paste the code and get it working kind of deal. There is provided code, yes, but it will probably not do everything you want it to do. which is why I will be explaining the ideas and architecture behind the code, to help you better understand and change it to suit your needs.

Using OpenCV in Unity

Setting up Unity to use OpenGL

First of all you need to set up Unity to use OpenGL rendering (it uses DirectX by default). You can do this by accessing the Player Setting from Edit->Settings->Player and go to the Other Settings tab:

Just uncheck Auto Grahics API for each platform and move OpenGLCore to the top.

Creating a placeholder material

Just create an unlit texture material, it will act as a placeholder for OpenGL rendering later on. Give it a name: I called it empty for this post's purposes.

Now just create a cube and affect the empty material to it.

A look at the native plugin code

In case you haven't done it, you can get the code on GitHub. It uses a standard Unity low-level native plugin interface, which I will not explain as it is beyond the scope of this post.

The code for the DLL is in the OpenCVPlugin folder. It contains a Visual Studio 2015 solution which has to be modified to use the correct paths for your OpenCV libraries if you want to use it directly. It also uses SWIG to compile the interface between C++ and C#. More on that below.

The meat of the work is done in OpenCVDllInterface.cpp. First we need to store the handle of the texture somewhere:

void OpenCVDllInterface::setTextureHandle(void* handle)
{
 m_textureHandle = (GLuint)(size_t)handle;
}

And now that we have our texture handle, we just update it every frame with whatever OpenCV data we want:

void OpenCVDllInterface::updateFrameDataOGL(int eventID)
{
 if (m_captureDevice.isOpened())
 {
  cv::Mat originalFrame;
  m_captureDevice >> originalFrame;

  if (!originalFrame.empty())
  {
   cv::Mat frame;

   flip(originalFrame, frame, 0);

   size_t currentFrameSize = frame.total() * 3;

   if (m_frameBufferSize < currentFrameSize)
   {
    m_frameBufferSize = currentFrameSize;
    m_frameBuffer = (uchar*)realloc(m_frameBuffer, m_frameBufferSize);
   }

   Mat continuousRGB(frame.size(), CV_8UC4, m_frameBuffer);
   cvtColor(frame, continuousRGB, CV_BGR2RGB, 3);

   glBindTexture(GL_TEXTURE_2D, m_textureHandle);

   glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
   glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);

   // Set texture clamping method
   glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP);
   glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP);

   //set length of one complete row in data (doesn't need to equal image.cols)
   glPixelStorei(GL_UNPACK_ROW_LENGTH, (int)(frame.step / frame.elemSize()));

   glTexSubImage2D(GL_TEXTURE_2D,
    0,
    0,
    0,
    continuousRGB.size().width,
    continuousRGB.size().height,
    GL_RGB,
    GL_UNSIGNED_BYTE,
    continuousRGB.data);
  }
 }
}

This code just directly outputs the video capture device but it can easily be adapted to display any image manipulations OpenCV can store in a cv::Mat.

And finally, a method to inform Unity of the size of the texture we will need it to create:

void OpenCVDllInterface::getFrameBufferInfo(FrameInfo& bufferInfo)
{
 if (m_captureDevice.isOpened())
 {
  bufferInfo.width = int(m_captureDevice.get(CAP_PROP_FRAME_WIDTH));
  bufferInfo.height = int(m_captureDevice.get(CAP_PROP_FRAME_HEIGHT));

  cv::Mat frame;
  m_captureDevice >> frame;
  if (frame.empty())
  {
   bufferInfo.sizeInBytes = 0;
  }
  else
  {
   bufferInfo.sizeInBytes = unsigned int(frame.total() * 3);
  }
 }
}

This assumes RGB byte sized data (hence the *3 to compute buffer size) and has to be adapted to your own needs.

Scripting in Unity

The Unity project is located in the OpenCVUnityProject folder. Let's take a look at the OpenCVBridge.cs script in the Assets/Scripts folder.

On Start(), we need to ask the DLL, about the texture we need to create, create said texture, and pass it on to the DLL. We also need to setup an event callback to trigger the OpenGL update from within the DLL using a delegate function in a coroutine:

IEnumerator Start() {
        m_dllInterface = new OpenCVDllInterface();
        m_frameInfo = new OpenCVDllInterface.FrameInfo();

        m_eventCallback = new eventCallbackDelegate(m_dllInterface.updateFrameDataOGL);

        m_dllInterface.getFrameBufferInfo(m_frameInfo);


        m_texture = new Texture2D(m_frameInfo.width, m_frameInfo.height, TextureFormat.RGB24, false);
        m_texture.filterMode = FilterMode.Point;
        m_texture.Apply();
        m_material.mainTexture = m_texture;

        m_dllInterface.setTextureHandle(m_texture.GetNativeTexturePtr());

        yield return StartCoroutine("CallPluginAtEndOfFrames");
    }

And the coroutine itself is pretty straightforward. Just call the correct method in the DLL:

private IEnumerator CallPluginAtEndOfFrames()
    {
        if (m_eventCallback != null)
        {
            while (m_dllInterface != null)
            {
                yield return null;

                GL.IssuePluginEvent(Marshal.GetFunctionPointerForDelegate(m_eventCallback), 1);
            }
        }
    } 

Now just attach the script to the cube we created earlier, you know, the one with the empty material. And don' forget to affect the empty material to the script.

Communicating between C++ and C#

Usually when communicating between C++ and C# there's a whole slew of manual dllimport directives and other hassles. But luckily, you can avoid all this if you use SWIG. I will definitely not go into the details of SWIG so feel free to browse their website for more information.

The Visual Studio solution on GitHub is already configured to automatically run SWIG and compile its output. If you wan to know how that is done, you can check out this page on Stackoverflow. It describes the process for Python but is easily modifiable for C#.

The SWIG interface file is pretty straightforward, you can find it in the OpenCVPlugin/Sources folder:

%module OpenCVPlugin

%{
#include "OpenCVDllInterface.h"
%}

//Use System.IntPtr from C# for void* pointers
%typemap( cstype ) void* "System.IntPtr"
%typemap( csin ) void* %{ $csinput %}
%typemap( imtype ) void* "System.IntPtr"

%include "OpenCVDllInterface.h"

The file is pretty straightforward. The only code gymnastics here instruct SWIG to user C# System.IntPtr when it encouters void* in C++.

Putting it all together

Before compiling your DLL make sure you use the same architecture as your Unity editor: a 64-bit Unity Editor will only work with a 64-bit DLL plugin.

When you compile your C++ project, SWIG will create .cs C# script files in the same folder as your DLL. If it's not already there, create the Assets/Plugins folder in your Unity project and copy the DLL and the generated scripts.

Run it and enjoy!

Using a canvas for AR applications

For real AR applications, displaying the OpenCV video feed on a cube is not the best approach. What can be done is assign the empty material to a Unity canvas that is attached to the main camera, with coordinates in camera space, set at a distance a little below the camera's far plane. The canvas aspect ratio should match the OpenCV video aspect ratio.

With that, you should be golden.