Install Caffe with CMake on vision machines/eldar

cd caffe;
mkdir cmake_build;
cmake .. -DBLAS=Open;

Final CMakeCache.txt should look like (both non-cuda version from vision machines and cuda version from eldar included):
https://gist.github.com/dineshj1/431828a10d230a5f767b73ecdfacf744

Install latest opencv from github repository on vision machines/ eldar

Clone opencv from Itseez's repository
cd opencv; mkdir release; cd release;
cmake .. -DWITH_IPP=OFF -DCMAKE_BUILD_TYPE=RELEASE -DCMAKE_INSTALL_PREFIX=/vision/vision_users/dineshj/local_installs/ -DBUILD_TIFF=ON -DBUILD_PNG=ON -DBUILD_opencv_viz=OFF

Final CMakeCache.txt file should look like:
https://gist.github.com/dineshj1/3058f8a439a82ff7cd1ae07113a5b148

Pycaffe training

https://gist.github.com/dineshj1/839e08576d441944fd6f36ca6896453b

Hyperparameter optimization on a cluster - job submission script

Job submission scripts can be written in such a way as to make hyperparameter optimization easy.

I personally find it useful to loop over some set of configurations of hyperparameters within a Matlab/Python script, and also simultaneously dump those parameters into a spreadsheet. I then copy the cells from that spreadsheet into a Google Sheets spreadsheet, one run on each row. Later, I will record results in that Google Sheets spreadsheet too.

https://gist.github.com/dineshj1/7e38e6a68d6f7d81cc771ed77ce3d656

kitti227_submitjobs.m and kitti227_submitjobs.py are Matlab and Python job submission scripts that do similar things. Python does things a little more elegantly, using the Caffe python interface to automatically generate network prototxts (see layer_stack.py)

job_submission_script_example.m is just another job submission Matlab script, similar to kitti227_submitjobs.m.

Torch-hdf5

Don't use luarocks install hdf5. It might install a different library. What you want is:
https://github.com/deepmind/torch-hdf5/blob/master/doc/usage.md

To install it, clone the repository and type:

luarocks make hdf5-0-0.rockspec

Test installation with:
th -e "require 'hdf5'"

If it works, you're set. If it throws up errors like  "Error: unable to locate HDF5 header file at hdf5.h",

then you need to go to $TORCHPATH/share/lua/5.1/hdf5/config.lua and edit HDF5_INCLUDE_PATH. For e.g.

    HDF5_INCLUDE_PATH = "/vision/vision_users/dineshj/local_installs/include/",

Caffe+dependencies installation with cudnn on vision/eldar machines (both with and without cuda)

https://gist.github.com/dineshj1/6f0371ba5615d7e9d50d870fd53c8d01

For this installation, install_caffe_new.sh was run on vision machines entirely. Not sure if installing dependencies would work just as well on eldar. OpenCV in particular caused problems when trying to install from eldar-1.


PS: The cudnn version installs, and make runtest passes all the important tests, but throws a segmentation fault when printing date:
[----------] Global test environment tear-down
[==========] 2081 tests from 277 test cases ran. (812540 ms total)
[  PASSED  ] 2081 tests.
*** Aborted at 1471619205 (unix time) try "date -d @1471619205" if you are using GNU date ***
PC: @     0x2b8879ed77db (unknown)
*** SIGSEGV (@0x2b886bac1188) received by PID 29519 (TID 0x2b887324c480) from PID 1806438792; stack trace: ***
    @     0x2b8879e8fcb0 (unknown)
    @     0x2b8879ed77db (unknown)
    @     0x2b8879ed8ce8 (unknown)
    @     0x2b8879edc1dc __libc_calloc
    @     0x2b8885d91279 (unknown)
make: *** [runtest] Segmentation fault

Download full directories with wget

See: http://stackoverflow.com/a/273776/2009491

In particular, what you probably need is this:

wget -r --no-parent --reject "index.html*" http://example.com/configs/.vim/

Install osmesa on vision machines (in progress)

wget https://mesa.freedesktop.org/archive/current/mesa-11.0.7.tar.gz
dtrx mesa-11.0.7.tar.gz

cd mesa-11.0.7/
./configure --prefix=/vision/vision_users/dineshj/local_installs/ --disable-dri3
--> error saying libudev-dev or sysfs required


Was trying to do this to view OFF files. Found ways to do it within Matlab instead.

Reduce file size of eps image file in Linux (eg., during arXiv submission)

It's not clear how to do this within Inkscape (even if you are creating your eps in Inkscape)

So, open the file in Gimp. Choose resolution 100 (default).

Then export to eps in Gimp. This should create a much smaller file already.

Tradeoff compactness and quality by adjusting resolution at the time of loading the eps into Gimp.

How to merge folders from command line in Linux

The standard mv has no way to do this. Instead, use rsync.

rsync -av <source_folder> <target_folder> --remove-source-files

See: http://unix.stackexchange.com/questions/127712/merging-folders-with-mv

How to monitor the progress of system commands like mv, cp, rsync in Linux

http://unix.stackexchange.com/a/144284

https://github.com/Xfennec/progress


$ git clone git@github.com:Xfennec/progress.git
$ export PREFIX=<install_location>
$ make
$ make install

Then, to continuously monitor commands like mv, cp, rsync etc. simply run:
$ watch progress -q

Matlab's keyboard command, but for IPython

Add the following to the preamble.

from IPython.core.debugger import Tracer; debug_here = Tracer()

Then in code, you can call "debug_here()" where you'd use "keyboard" in Matlab.

YouCompleteMe setup on remote Linux machine (vision)

.vimrc:

vundle link:
valloric/YouCompleteMe

compilation:
cd .vim/bundle/YouCompleteMe
./install.py --clang-completer

To avoid issue "HTTPConnectionPool(): Read timed out. (read timeout=0.5)", reported at:  https://github.com/Valloric/YouCompleteMe/issues/777

change one line per: https://github.com/Valloric/YouCompleteMe/issues/777#issuecomment-42033173

neovim from source

git clone https://github.com/neovim/neovim
cd neovim
rm -r build/
make CMAKE_EXTRA_FLAGS="-DCMAKE_INSTALL_PREFIX:PATH=<custom-install-location>"
make install

pip install --upgrade --force-reinstall neovim

Monitor disk io usage statistics on Linux

$ dstat

http://unix.stackexchange.com/questions/55212/how-can-i-monitor-disk-io

Color differences between normal terminal and tmux terminal

In ~/.tmux.conf:
      set -g default-terminal "screen-256color"

In .bashrc or .profile:
     export TERM=xterm-256color
   alias tmux ="TERM=xterm-256color tmux"

-------------------------------------------------------------------------

Updated: 5/11/16

Actually, everything works okay with just the change in ~/.tmux.conf...

I removed:
export TERM=xterm-256color from .bashrc
alias tmux="TERM=xterm-256color tmux" from .profile


Discovered this when trying to overcome the following problem created by running tmux in xterm-256color mode. Matlab in commandline mode inside tmux does not recognize home and end keys etc. Displays them as "[~1" etc. rather than moving the cursor.

Switching between Latex compilers when using vim latex-suite

https://gist.github.com/dineshj1/8f0783ae5211c31a8aa6ea04a9664fa3

Use latexmk with pdflatex as the default compiler. Mapped to <leader><leader>

But now, use latexmk with xelatex as a compiler by using shortcut <leader>x.

Viewing images on remote machine with X11 forwarding

gpicview: http://lxde.sourceforge.net/gpicview/

Easy to install:
./configure
make
make install

Then alias "imshow" to point to the gpicview binary.

After this:
$ imshow <imgfilename>
will display the image.

LaTeX in HTML pages with Latexit

http://www.codecogs.com/latex/integration/htmlequations.php

How to expose only one of several GPUs to a process (masking GPUs)

At the command line:
export CUDA_VISIBLE_DEVICES=0

See: http://www.acceleware.com/blog/cudavisibledevices-masking-gpus

LaTeX math mode in Microsoft Powerpoint

Writing equations in Powerpoint doesn't have to be a pain. There's Math Mode.
Alt+= will get you there. Mostly LaTeX compatible.

Note: doesn't work on my Powerpoint for Mac 2016 installation. :/ Alt, N, E, I gets you into insert equation mode there, which also mostly recognizes LaTeX-style math entry.

For more handy tips on mathematical presentations in Powerpoint, see:

• https://www.cs.bu.edu/~reyzin/pptips.html
• http://www.chem.mtu.edu/~tbco/cm416/EquationEditor_main.pdf

Look up machine name from IP address

IP address, say, "128.83.139.20".
Simply type:

$ nslookup 128.83.139.20

Output:
Server: 127.0.1.1
Address: 127.0.1.1#53

20.139.83.128.in-addr.arpa name = eldar-9.cs.utexas.edu.

List all files in order of size starting at a certain root folder

$ du -a ./ | sort -n -r > du_out.txt

Install new NVIDIA graphics card and cuda on Linux Mint

NVIDIA driver

Manual install instructions from https://sites.google.com/site/easylinuxtipsproject/12

• Power off
• Install card in motherboard slot
• Connect monitor cables to card outputs
• Find out what graphics card you have: $ lspci | grep VGA
• Find out which driver you should download by entering card details at: http://www.nvidia.com/Download/index.aspx?lang=en-us
• Download the driver
• Remove all previous nvidia drivers by running: $ sudo apt-get purge nvidia*
• $ chmod +x -v ~/Downloads/NVIDIA-Linux-*.run
• Switch to terminal only mode: $ sudo service mdm stop
• Login in terminal-only mode
• Run: $ sudo sh ~/Downloads/NVIDIA-Linux-*.run
• Accept all options
• After everything is done, $ sudo reboot
• Did it work?
• Everything should display properly in high res
• $ lsmod | grep nvidia
• $ lspci | grep VGA
• Check if nouveau is still being used (if properly installed, no), by: $ lsmod | grep nouveau

If things didn't go to plan, and you need to uninstall, then:

$ sudo sh ~/Downloads/NVIDIA-Linux-*.run --uninstall

$ sudo rm -v /etc/X11/xorg.conf

$ sudo reboot

CUDA

Again, prefer manual install from https://developer.nvidia.com/cuda-toolkit-archive over official repository cuda packages. Use the run file instructions from http://docs.nvidia.com/cuda/cuda-getting-started-guide-for-linux/#axzz45oaAPyTF. Don't accept offer to reinstall Nvidia drivers during cuda installation.

Nicknames for ssh hosts

You can specify ssh host nicknames through your .ssh/config file. See: https://gist.github.com/dineshj1/9f304084a1f4af27ae2ccf79c134b0e6

Install improved_dense_trajectories code on vision machines

https://gist.github.com/dineshj1/873850c07a0ff2a5a899a14008dc8abe

Run install_IDT_code.sh, but this will run into errors at first. Follow the instructions given in install_IDT_code.sh to edit Makefile. That should do the job.

Matlab script to plot caffe outputs

https://gist.github.com/dineshj1/931bbfe054ca1a60d0b3041554f2a15c

Setting primary group on UTCS machines

The code snippet in https://gist.github.com/dineshj1/d6b975a8d3d160e95c69 should be added to the end of the .profile file.

NOTE: This causes issues with getting graphics working on eldar-1.

Download large SUN panoramas parallelly on condor

https://gist.github.com/dineshj1/c3ff8c55b28b8c4ea30d

Screenshot shortcuts on a new Linux Mint OS

Linux Mint 17.3 did not have the keyboard shortcuts for screenshots of selected areas working.
I want the behavior to be:

• Shift-PrintScr = save selected area screenshot (to /home/<username>/Pictures/)
• Ctrl-Shift-PrintScr = copy selected area screenshot to clipboard

The following steps proved sufficient:

• Copy scripts from https://gist.github.com/dineshj1/d2c671a3ad82f89879c2
  
• Navigate to keyboard>shortcuts>custom shortcuts
• Add shortcut commands: 
• bash /home/dineshj/screenshot_area.sh, with keyboard bindings: Shift+Print 
• bash /home/dineshj/screenshot_area_to_clipboard.sh, with keyboard bindings: Ctrl+Shift+Print 

Compiling and submitting matlab scripts to condor on the UTCS cluster

Gist at: https://gist.github.com/dineshj1/935a703f93c12a3efc19

1. Write the script in condor-compatible way: (see https://gist.github.com/dineshj1/935a703f93c12a3efc19#file-sample_condorized_function-m, and read tips and details)
2. Compile code after adding all necessary paths to Matlab path: (see https://gist.github.com/dineshj1/935a703f93c12a3efc19#file-compilecodes-m-example)
3. Create submit script: (see https://gist.github.com/dineshj1/935a703f93c12a3efc19#file-sample_submit_file)
4. Some edits to PATH, LD_LIBRARY_PATH etc. may be necessary (see full .profile at https://gist.github.com/dineshj1/935a703f93c12a3efc19#file-profile-when-working): 
1. export PATH=$PATH:/lusr/share/software/matlab-r2015b/bin/glnxa64/:/lusr/share/software/matlab-r2015b/toolbox/compiler/deploy/;
2. export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/v/filer4b/software/matlab-r2015b/bin/glnxa64/:/v/filer4b/software/matlab-r2015b/runtime/glnxa64/;
3. export MCR_CACHE_ROOT=/scratch/vision/dineshj/mcrCache/ # where executables will extract ctf archives
4. export KMP_DUPLICATE_LIB_OK=true;

Python jobs on condor

Condor submit file could look like:

+Group="GRAD"
+Project="AI_ROBOTICS"
+ProjectDescription=""
Universe = vanilla
Getenv = True
Log    = ../SUN_download_condor.log
Output = ../SUN_download_condor.out
Error  = ../SUN_download_condor.err
Executable = ./download_SUN_large_panos.py
Queue 100

And the python script itself could look like:

#!/usr/bin/env python

import os
import urllib
import sys

def touch(fname, times=None):
  with open(fname, 'a'):
    os.utime(fname, times)

target_root='./dataset/9104x4552/';
if not os.path.exists(target_root):
  os.mkdir(target_root);


Note the header at the top (shown in bold). This is important for condor to know how to execute the script.

Also, obviously, the python script would have to have executable permissions.

Caffe installation on Maverick, from scratch, without CMake

Scripts at: https://gist.github.com/dineshj1/69cc2d2aae149bed519d
(very basic bashrc, dependency installation script and Makefile.config)

1) Run dependency installation script to install all deps to a  common install/ folder (and anaconda separately).
2) Makefile.config and bashrc are configured to point to these folders.
3) Compile!

• make all -j; 
• make pycaffe -j; 
• make matcaffe -j; 
• make runtest -j; 
• make pytest -j;
• make mattest -j; 

Everything works!

Caffe installation on Maverick using CMake

Gist at: https://gist.github.com/dineshj1/8162cd8e72647aed2108
(all the following should be done from an interactive session on one of the visualization machines ./request_vis.sh)

Dependency installation:
mkdir caffe_deps
cd caffe_deps; cp <install_caffe_deps.sh> ./;
./install_caffe_deps.sh (Not completely tested... may need tinkering from time to time)


Main Caffe installation (after installing all dependencies):

1) cd caffe; mkdir build; cd build;
2) ln -s <caffe_dependencies_folder> .caffe_deps
3) cp <caffe_cmake_script.sh from gist>  ./
4) ./caffe_cmake_script.sh
5) make all -j; make pycaffe-j; make runtest -j; make pytest -j;

Cross-check against .bashrc in case of trouble.

Torch installation on Maverick

https://gist.github.com/dineshj1/ee4264f60dea3ac71def

- 1) Run install-torch.sh. This will in turn, create and call install-script_cuda.
- 2) "Install" the visual debugger zbs-torch (https://github.com/soumith/zbs-torch). Instructions as of 1/28/2016:

• git clone https://github.com/soumith/zbs-torch
• cd zbs-torch
• ./zbstudio.sh (to start ... no need to "install" anything)

(If trouble, crosscheck against .bashrc)

Torch installation on eldar

- Gists at: https://gist.github.com/dineshj1/6ec0ccb4267f3698b7ac

- 1) Run install-torch.sh. This will in turn call install-script_cuda.
- 2) "Install" the visual debugger zbs-torch (https://github.com/soumith/zbs-torch). Instructions as of 1/28/2016:

• git clone https://github.com/soumith/zbs-torch
• cd zbs-torch
• ./zbstudio.sh (to start ... no need to "install" anything)

(If trouble, crosscheck against .profile)