Light Field Super-Resolution via LFBM5D Sparse Coding
9th February 2018In this paper, we propose a spatial super-resolution method for light fields, which combines the SR-BM3D single image super-resolution filter and the recently introduced LFBM5D light field denoising filter. The proposed algorithm iteratively alternates between an LFBM5D filtering step and a back-projection step. The LFBM5D filter creates disparity compensated 4D patches which are then stacked together with similar 4D patches along a 5th dimension. The 5D patches are then filtered in the 5D transform domain to enforce a sparse coding of the high-resolution light field, which is a powerful prior to solve the ill-posed super-resolution problem. The back-projection step then impose the consistency between the known low-resolution light field and the-high resolution estimate. We further improve this step by using image guided filtering to remove ringing artifacts. Results show that significant improvement can be achieved compared to state-of-the-art methods, for both light fields captured with a lenslet camera or a gantry.
Implementation
The MATLAB/C/C++ source code is available on github.
Related publications
Additional results
Visual results complementing the paper are shown below.
We then show in Tables 1 and 2 the average results presented in the paper. The corresponding detailed results are shown in Table 3 and 4.
Lytro Illum dataset
We use in our experiments a Lytro Illum dataset consisting of light fields taken from the EPFL and the INRIA datasets.
These light fields have been further processed with the V-SENSE enhancement pipeline described here.
Visual results
We show in videos below (click on an image to start a video) side by side comparisons of low-resolution (uspampled with a bicubic filter) and super-resolved light fields for different scaling factors ?. On the top left corner of each video is highlighted the sub-aperture image being displayed. Note that some videos may exhibit encoding artifacts.
?=2 | ?=3 | ?=4 | |
Chess | |||
Lego Bulldozer | |||
Lego Knights | |||
Tarot Cards and Crystal Ball (Large Angle) | |||
Bee 2 | |||
Bikes | |||
Danger de mort | |||
Vespa |
Below, we show results for ?=4 without (left) and with (right) the light field guided filtering described in section 2.2 of the paper to improve back-projection (click on an image to start a video).
Chess | Lego Knights | Bikes | Vespa |
Average PSNR results
Values highlighted in bold and italic correspond to the best and second best performing methods respectively for a given magnification factor.
?=2 | ?=3 | ?=4 | |
Bicubic | 27.78 | 26.08 | 24.62 |
SR-BM3D | 30.21 | 28.45 | 26.58 |
BM+PCA+RR | 29.95 | 28.55 | 27.08 |
GB | 29.80 | 28.65 | 27.45 |
SR-LFBM5D 1st step | 30.17 | 28.62 | 26.87 |
SR-LFBM5D 2nd step | 30.25 | 28.60 | 26.82 |
?=2 | ?=3 | ?=4 | |
Bicubic | 29.00 | 26.81 | 25.06 |
SR-BM3D | 34.10 | 30.90 | 28.00 |
BM+PCA+RR | 32.81 | 30.85 | 28.73 |
GB | 33.01 | 31.42 | 29.44 |
SR-LFBM5D 1st step | 34.15 | 31.81 | 29.10 |
SR-LFBM5D 2nd step | 34.27 | 31.77 | 29.02 |
Detailed PSNR results
Ankylosorus & Dipplodocus 1 | Bee 1 | Bee 2 | Bikes | Chez Edgar | Danger de Mort | Friends 1 | Fruits | Magnets 1 | Posts | Rose | Vespa | |
Bicubic | ||||||||||||
?=2 | 33.5414 | 27.0979 | 27.5808 | 26.2794 | 22.3877 | 24.3675 | 28.713 | 26.0159 | 31.0621 | 30.4165 | 28.5095 | 27.3734 |
?=3 | 31.5025 | 25.4013 | 25.9392 | 24.5069 | 21.1371 | 22.8849 | 26.589 | 24.3747 | 29.0306 | 29.2154 | 26.8529 | 25.5526 |
?=4 | 29.7729 | 23.8938 | 24.4516 | 22.9869 | 20.0919 | 21.7029 | 24.769 | 22.9985 | 27.2962 | 28.0506 | 25.4262 | 24.0135 |
SR-BM3D | ||||||||||||
?=2 | 36.075 | 29.023 | 29.346 | 30.058 | 24.220 | 27.101 | 32.185 | 28.791 | 33.858 | 30.958 | 30.597 | 30.354 |
?=3 | 34.207 | 27.892 | 28.265 | 27.978 | 22.433 | 24.979 | 30.423 | 26.827 | 31.500 | 29.667 | 28.733 | 28.534 |
?=4 | 32.477 | 26.421 | 26.959 | 25.369 | 20.908 | 22.756 | 28.256 | 24.748 | 29.300 | 28.299 | 26.801 | 26.698 |
BM+PCA+RR | ||||||||||||
?=2 | 35.6127 | 28.7435 | 29.0858 | 29.5911 | 24.2046 | 26.8869 | 31.8674 | 28.6242 | 33.4409 | 30.8375 | 30.4213 | 30.1102 |
?=3 | 34.3607 | 27.6675 | 28.225 | 27.8739 | 22.7327 | 25.2586 | 30.4348 | 27.1949 | 31.5474 | 29.704 | 28.9552 | 28.6078 |
?=4 | 32.9217 | 26.4726 | 27.1327 | 26.0074 | 21.4623 | 23.7469 | 28.7222 | 25.6303 | 29.7057 | 28.5436 | 27.481 | 27.1216 |
GB | ||||||||||||
?=2 | 35.7291 | 28.7144 | 29.159 | 29.329 | 23.753 | 26.618 | 31.629 | 28.249 | 33.353 | 30.9148 | 30.292 | 29.918 |
?=3 | 34.606 | 27.774 | 28.395 | 28.073 | 22.644 | 25.356 | 30.531 | 27.159 | 31.672 | 29.781 | 28.976 | 28.781 |
?=4 | 33.441 | 26.757 | 27.588 | 26.643 | 21.536 | 24.126 | 29.114 | 26.108 | 29.991 | 28.627 | 27.793 | 27.619 |
SR-LFBM5D 1st step | ||||||||||||
?=2 | 36.044 | 28.992 | 29.303 | 29.977 | 24.222 | 27.059 | 32.136 | 28.714 | 33.840 | 30.917 | 30.495 | 30.326 |
?=3 | 34.559 | 27.921 | 28.366 | 28.127 | 22.459 | 25.171 | 30.529 | 27.139 | 31.586 | 29.767 | 29.039 | 28.756 |
?=4 | 33.068 | 26.587 | 27.203 | 25.678 | 20.977 | 22.794 | 28.649 | 25.246 | 29.415 | 28.456 | 27.262 | 27.133 |
SR-LFBM5D 2nd step | ||||||||||||
?=2 | 36.123 | 29.037 | 29.350 | 30.098 | 24.282 | 27.144 | 32.242 | 28.816 | 33.899 | 30.966 | 30.602 | 30.425 |
?=3 | 34.532 | 27.908 | 28.348 | 28.115 | 22.439 | 25.133 | 30.513 | 27.106 | 31.573 | 29.768 | 29.007 | 28.731 |
?=4 | 33.032 | 26.561 | 27.170 | 25.614 | 20.909 | 22.650 | 28.595 | 25.161 | 29.379 | 28.447 | 27.201 | 27.074 |
Amethyst | Bracelet | Chess | Eucalyptus Flowers | Jelly beans | Lego Bulldozer | Lego Knights | Lego Truck | Tarot Cards and Crystal Ball (Large Angle) | Tarot Cards and Crystal Ball (Small Angle) | The Stanford Bunny | Treasure Chest | |
Bicubic | ||||||||||||
?=2 | 27.9143 | 26.3944 | 31.0063 | 23.3153 | 41.5464 | 26.9215 | 28.5825 | 27.7637 | 27.0215 | 27.103 | 36.0681 | 24.3827 |
?=3 | 25.846 | 24.1524 | 28.707 | 22.1331 | 38.0155 | 24.7319 | 26.3982 | 25.8623 | 25.2307 | 24.8391 | 32.8963 | 22.9615 |
?=4 | 24.1607 | 22.3826 | 26.8413 | 21.2074 | 35.4326 | 22.9613 | 24.5788 | 24.2874 | 23.5296 | 23.1016 | 30.4554 | 21.745 |
SR-BM3D | ||||||||||||
?=2 | 32.591 | 34.325 | 37.250 | 26.424 | 45.191 | 32.029 | 34.014 | 31.820 | 32.241 | 33.764 | 41.051 | 28.487 |
?=3 | 29.935 | 29.172 | 33.646 | 23.929 | 42.740 | 29.198 | 30.944 | 28.929 | 29.048 | 28.878 | 38.889 | 25.505 |
?=4 | 27.491 | 25.111 | 30.247 | 22.269 | 39.682 | 26.305 | 28.053 | 26.103 | 26.087 | 25.339 | 36.061 | 23.264 |
BM+PCA+RR | ||||||||||||
?=2 | 32.1632 | 32.8753 | 36.1938 | 26.2293 | 43.3373 | 30.5675 | 32.8232 | 31.4071 | 28.4677 | 31.6728 | 39.7559 | 28.1899 |
?=3 | 30.281 | 30.260 | 33.903 | 24.775 | 41.575 | 28.482 | 30.898 | 29.497 | 26.363 | 29.435 | 38.600 | 26.169 |
?=4 | 28.428 | 26.799 | 31.713 | 23.344 | 39.239 | 26.367 | 28.695 | 27.502 | 24.485 | 27.127 | 36.452 | 24.578 |
GB | ||||||||||||
?=2 | 32.164 | 32.164 | 35.715 | 25.529 | 45.301 | 30.928 | 32.768 | 30.968 | 31.457 | 30.939 | 41.017 | 27.192 |
?=3 | 30.245 | 30.34 | 34.136 | 24.595 | 43.182 | 29.293 | 31.475 | 29.549 | 29.298 | 29.620 | 39.330 | 25.976 |
?=4 | 29.127 | 28.16 | 32.636 | 24.012 | 40.835 | 27.828 | 29.995 | 28.007 | 21.534 | 28.148 | 37.816 | 25.203 |
SR-LFBM5D 1st step | ||||||||||||
?=2 | 32.620 | 34.214 | 37.455 | 26.450 | 45.396 | 32.007 | 34.210 | 31.896 | 32.278 | 33.673 | 41.109 | 28.514 |
?=3 | 30.440 | 30.535 | 35.242 | 24.653 | 43.099 | 29.804 | 32.467 | 29.949 | 30.077 | 29.863 | 39.412 | 26.141 |
?=4 | 28.125 | 26.351 | 31.670 | 22.838 | 40.689 | 27.241 | 29.767 | 27.510 | 26.837 | 26.724 | 37.157 | 24.324 |
SR-LFBM5D 2nd step | ||||||||||||
?=2 | 32.775 | 34.508 | 37.567 | 26.584 | 45.261 | 32.125 | 34.312 | 31.982 | 32.445 | 33.902 | 41.172 | 28.659 |
?=3 | 30.400 | 30.477 | 35.206 | 24.629 | 43.032 | 29.769 | 32.427 | 29.923 | 30.046 | 29.811 | 39.385 | 26.112 |
?=4 | 28.046 | 26.209 | 31.618 | 22.754 | 40.626 | 27.152 | 29.707 | 27.463 | 26.738 | 26.614 | 37.073 | 24.236 |