The last tracker commit added normalized tracking. This makes
tracking patches undergoing uniform illumination change easier.
However, the prepass which computes a quick translation-only
estimate of the warp did not take this into account. This commit
fixes that.
This works reasonably well but in some examples the brute
initialization fails. I suspect this is due to the warped template
estimate in the current frame being too different from the
original, so there are multiple peaks in the normalized-SAD
correlation function.
The solution is to use the previous frame for the brute
initialization and the keyframe for refinement, but that requires
architecture changes.
This commit adds the ability to normalize patterns by their
average value while tracking, to make them invariant to global
illumination changes.
To see this in action, check out the "Lobby" scene from Hollywood
VFX. If you track the markers that are shadowed by the actress,
previously they would not track. With the scale adaption on, the
tracker would shrink the area to compensate for the changed
illumination, losing the track. With "Normalize" turned on, the
patch is correctly tracked and scale is maintained.
A remaining problem is that only the Ceres cost function is
updated to handle the normalization. The brute translation search
does not take this into account. Perhaps "Prepass" (see below)
should get disabled if normalization is enabled until I fix the
prepass to normalize as well.
There are a few other changes:
- Cleanups in tracking RNA comments.
- Bail out of the sampling loop early if the mask is zero; this
saves expensive samples of the image derivatives.
- Rename the wordy "Translation initialization" to "Prepass" at
Sebastian's suggestion.
- Fix a bug where the mask was ignored when sampling in the cost
functor.
This replaces the old style tracker configuration panel with the
new planar tracking panel. From a users perspective, this means:
- The old "tracking algorithm" picker is gone. There is only 1
algorithm now. We may revisit this later, but I would much
prefer to have only 1 algorithm. So far no optimization work
has been done so the speed is not there yet.
- There is now a dropdown to select the motion model. Choices:
* Translation
* Translation, rotation
* Translation, scale
* Translation, rotation, scale
* Affine (Not implemented yet)
* Perspective
The most stable is the "translation" parameterization. The
others work but still require some tweaking.
- The old "Hybrid" mode is gone; instead there is a toggle to
enable or disable translation-only tracker initialization. This
is the equivalent of the hyrbid mode before, but rewritten to work
with the new planar tracking modes.
- The pyramid levels setting is gone. At a future date, the planar
tracker will decide to use pyramids or not automatically. The
pyramid setting was ultimately a mistake; with the brute force
initialization it is unnecessary.
Also made BKE_tracking_get_search_imbuf use space conversion utility functions,
so now it's not so annoying that search area calculation is happening differently
in different paces.
Also allow even sizes for search area.
Another small fix is about flipping search area dumping by libmv-capi.
It used to be flipped since in blender Y axis is up-aimed.
This commit removes the use of the legacy RegionTracker API from
Blender, and replaces it with the new TrackRegion API. This also
adds several features to the planar tracker in libmv:
- Do a brute-force initialization of tracking similar to "Hybrid"
mode in the stable release, but using all floats. This is slower
but more accurate. It is still necessary to evaluate if the
performance loss is worth it. In particular, this change is
necessary to support high bit depth imagery.
- Add support for masks over the search window. This is a step
towards supporting user-defined tracker masks. The tracker masks
will make it easy for users to make a mask for e.g. a ball.
- Add Pearson product moment correlation coefficient checking (aka
"Correlation" in the UI. This causes tracking failure if the
tracked patch is not linearly related to the template.
- Add support for warping a few points in addition to the supplied
points. This is useful because the tracking code deliberately
does not expose the underlying warp representation. Instead,
warps are specified in an aparametric way via the correspondences.
- Remove the "num_samples_xy" concept and replace it with
automatic determination of the number of samples. This makes the
API easier for users.
- Fix various bugs in the parameterizations.
There remains a bug with subpixel precision tracking when in
"keyframe" mode; this will get fixed shortly.
- Remove strict flags from files, which are using FFmpeg stuff
We're still using some symbols which are marked as deprecated.
Ideally, we shall switch to new API, but it's a bit larger challenge
because we don't want to break compatibility withotu actual need.
- Replace MAKE_ID with BT_MAKE_ID in bullet library.
This is needed to prevent re-definition of MAKE_ID in bullet library.
Seems it's only used to read blender files, so should be quite safe
change.
This implements the "Efficient Second-order Minimization"
scheme, as supported by the existing translation tracker.
This increases the amount of per-iteration work, but
decreases the number of iterations required to converge and
also increases the size of the basin of attraction for the
optimization.
This adds a new planar tracking implementation to libmv. The
tracker is based on Ceres[1], the new nonlinear minimizer that
myself and Sameer released from Google as open source. Since
the motion model is more involved, the interface is
different than the RegionTracker interface used previously
in Blender.
The start of a C API in libmv-capi.{cpp,h} is also included.
The ESM tracker, also known as the KLT tracker in the UI, is
temporarily changed to use the new Ceres-based planar
tracker in translation-only mode. Currently it is a bit
slower than ESM and also doesn't have all the bells and
whistles implemented. Those will come soon. Longer term,
both trackers will remain since Ceres is unlikely to be as
fast as ESM for pure translation solving, due to its
generality.
The next step is to implement a new tracking UI. The current
UI assumes a translational motion model; the new one must
support arbitrary perspective transforms of the pattern
regions.
[1] http://code.google.com/p/ceres-solver
Currently only put sources of Ceres library into extern/libmv/third_party and
setup CMake and SCons building systems.
Integration details:
- Even CMake build files are not re-used from Ceres's trunk: they're using some
automatic stuff detection like glog, pthreads, protobuf and so and it's not
so clear how to re-use that files without modifications.
And IMO it's easier if build files are getting re-generated automatically to
match Blender-specific setup rather than keeping changes made locally in
Blender in sync when re-bundling Ceres library. Especially in case when it's
alerady needed to support SCons build system.
- Integrated only actual sources, all tests were stripped. Probably it'll be nice
to have them, but they'll need clear integration with current module test stuff
in Blender.
Hopefully integration went smooth.
- Suitesparse was disabled. It'll help a lot having it, but there are some difficulties
making cholmod working fine on windows. Would be added in future
- collections_port.cc was also stripped. It's not used by Ceres's upstream and
it gives compilation error (undefined uint32 -- looks like namespace issue).
- Currently all schur eliminators are included. Not sure if it makes sense,
also not sure if it makes sense having them switchable on and off -- IMO better
to have single configuration which works and does not require special tweaks
after everything was set up.
- Personally i'd say if some of Ceres modules are not used better to drop it
away -- all symbols would be stripped anyway, but it'll be waste of compilation
time which is annoying in cases when one doing, say, binary search of revision
at which some regression was introduced. Especially when it's easy to add modules
which should be used by Blender.
But as long as it stays in Tomato i'm not worrying much about this.
To bundle updated version of Ceres:
- You'll need to use GIT-SVN checkout,
Re-bundling Ceres using SVN is still NOT supported!
- Go to extern/libmv/third_party/ceres folder
- Run ./bundle.sh
This will checkout fresh Ceres snapshot of Windows branch (which is currently
most interesting from integration into Blender POV), apply all patches listed
in patches/series and copy needed files into Blender's working copy. This will
also re-generate CMake/SCons build rules.
If you'll need extra files from Ceres repository which are not present in
Blender, you'll need to copy them manually and then run ./mkfiles.sh from
extern/libmv/third_party/ceres folder which will update list of files used
by Blender.
Thanks all Ceres developers for this library and thanks to Keir Mierle with
help integrating Ceres into Blender!
Expose option into interface to use modal solver which currently
supports only tripod motion.
This solver requires two tracks at least to reconstruct motion.
Using more tracks aren't improving solution in general, just adds
instability into solution and slows down things a lot.
Refirement of camera intrinsics was disabled due to it's not only
refines camera intrinsics but also adjusts camera position which
isn't necessary here
To use this solver just activate "Tripod Motion" checkbox in
solver panel.
Merged from tomato: svn merge ^/branches/soc-2011-tomato -r45622:45624 -r46036:46037
P.S. Quite experimental yet, requires more checking and probably
tweaks to prevent camera jumps when tracks apperars/disappears
from the screen.
Expose option into interface to use modal solver which currently
supports only tripod motion.
This solver requires two tracks at least to reconstruct motion.
Using more tracks aren't improving solution in general, just adds
instability into solution and slows down things a lot.
Refirement of camera intrinsics is supported by this solver.
To use this solver just activate "Tripod Motion" checkbox in
solver panel.
- Deduplicate patetrn sampling used in esm and lmicklt trackers and
move SamplePattern to image/sample.h - Move computation of Pearson
product-moment correlation into own function in new file image/correlation.h
so all trackers can use it to check final correlation.
- Remove SAD tracker. It's almost the same as brute tracker, with only two differences:
1. It does brute search of affine transformation which in some cases helps to track
rotating features 2. It didn't use common tracker api which probably gave some
speed advantage, but lead to a real headache to use it together with other
trackers leading to duplicated code in blender side.
- Switch blenedr to use brute tracker instead of sad tracker which tracking made
source code much more simple to follow.
Should resolve compilation error on some platforms when using linux and
compilation error of libmv on FreeBSB.
It was a regression caused by not applied changes on config_linux.h
and some changes made to utilities.cc were also occasionally missed.
This version of libmv includes new gflags and glog libraries which makes
it possible to compile libmv with clang compiler.
Also remove code from CMakeLists which was disabling libmv if using clang.
Tested on linux with gcc-4.6 and clang-3.0, windows cmake+msvc and scons+mingw.
Could be some issues with other platforms/build system which shall be simple to resolve.
