Renaming HDF5 Datasets in Matlab/Python

MATLAB:

Suppose you have a HDF5 file that looks like:
>> h5disp('myh5file.h5')                                            

HDF5 myh5file.h5
Group '/'                                                                  
    Dataset 'a'                                              
        Size:  227x227x6x5000                                  
        MaxSize:  227x227x6xInf                              
        Datatype:   H5T_IEEE_F32LE (single)          
        ChunkSize:  227x227x6x1000                        
        Filters:  none                                                    
        FillValue:  0.000000                                        
    Dataset 'b'                                        
        Size:  6x5000                                                    
        MaxSize:  6xInf                                                
        Datatype:   H5T_IEEE_F32LE (single)          
        ChunkSize:  6x1000                                        
        Filters:  none                                                    
        FillValue:  0.000000

If you want to rename the datasets "/a" to "/a_new", do:

  fid=H5F.open('myfile.h5', 'H5F_ACC_RDWR', 'H5P_DEFAULT');
  gid=H5G.open(fid, '/');                        
  H5L.move(fid, 'a', gid, 'a_new', 'H5P_DEFAULT', 'H5P_DEFAULT');              
  H5G.close(gid);
  H5F.close(fid);


PYTHON:
h5py makes it easy to "hard-link" the same dataset to a new name.

import h5py
h5file=h5py.open("myfile.h5", "r+");
h5file["new_dataset_name"]=h5file["orig_dataset_name"];
h5file.close();

If you had wanted to delete orig_dataset_name entirely, you'd simply add (before closing h5file):
del h5file["orig_dataset_name"];

Caffe install on delta (done)

• git clone git@github.com:BVLC/caffe.git
• Dependency installation script at https://gist.github.com/dineshj1/ad960e8361127c647194
• CMake installation works.
• glog installation works
• gflags installation works
• protobuf installation complicated: 
• 3.0 installation failing with pthread library error. Fixed hackily.
• Installer can't file pthread library, sets HAVE_PTHREAD to false.
• In src/Makefile.am, around line 15, edit the if HAVE_PTHREAD, ... , else, .. endif block, to execute the code for HAVE_PTHREAD = true, without checking the condition. i.e. PTHREAD_DEF = -DHAVE_PTHREAD=1
• protobuf 2.6.1 installation appears to work cleanly. 
• leveldb installation works
• snappy installation works
• hdf5 installation works
• opencv installation ?
• first failed with "Unsupported gpu architecture compute_11. fixed by cmake rerun with -DCUDA_GENERATION=Kepler flag (specific to Tesla C2075 GPU -- figure out which gpu with nvidia-smi)
• failed again, due to possible mismatch between CUDA-7.5 and opencv-2.4.9. Fix by downloading and replacing the following file: http://code.opencv.org/projects/opencv/repository/revisions/feb74b125d7923c0bc11054b66863e1e9f753141/entry/modules/gpu/src/nvidia/core/NCVPixelOperations.hpp
• Works (modified CMakeCache.txt that worked successfully: https://gist.github.com/dineshj1/759e846dc7cb7cf45182)
• opencv-3.0.0 installation?. Also possible to disable opencv in caffe installation
• fails again with "undefined reference to 'parallel_pthreads_set_threads_num(int)'
• Boost installation works following steps at: http://www.boost.org/doc/libs/1_59_0/more/getting_started/unix-variants.html#easy-build-and-install (Section 5.1)
• Main Caffe installation
• Using Makefile.config
• module list output looks like: 1) cuda/7.5            2) intel/14.0.0        3) mkl/10.2.5.035      4) java/1.6.0-x86_64   5) vim/7.2             6) cmake/2.8.1         7) atlas/3.10.2
• cp Makefile.config.example Makefile.config. Small changes, to point to anaconda etc. https://gist.github.com/dineshj1/209aecc7b753b8fa92aff82571cf0ede
• Error when $ make runtest,  with system gflags (libgflags.so.0) getting linked to installed glog. Fixed by: /N/u/dineshj/local/$ ln -s libgflags.so.2.2 libgflags.so.0
• bash_profile at: 
• Works right away!
• Using cmake
• module list output looks like: 1) vim/7.2        2) cmake/2.8.1    3) gcc/4.8.2      4) cuda/7.5       5) atlas/3.10.2
• Modifying various items in CMakeCache.txt: ATLAS_lapack_library, boost libraries, leveldb, protobuf, python, snappy, opencv (modified CMakeCache.txt at https://gist.github.com/dineshj1/8424162f5e67476d455d)
• First attempt produces errors:
• boost version already present is too old. Installation fixed this.
• Second attempt nearly there, but produces errors:
• Using system's original installation of opencv causes errors. Install opencv-2.4.9, then try again. 
• make all -j works. 
• make pycaffe -j works.
• make runtest -j works!
• Error with system gflags (libgflags.so.0) getting linked to installed glog. Fixed by: /N/u/dineshj/local/$ ln -s libgflags.so.2.2 libgflags.so.0
• add ~/local/python/ to PYTHONPATH environment variable in .bash_profile.
• make pytest fails saying, google protobuf missing. Fix by going to ~/local_pkgs/protobuf-2.6.1/python, then typing (Instructions in the README.txt file in this directory):
• python setup.py build
• python setup.py google_test
• python setup.py test --cpp_implementation
• python setup.py install --cpp_implementation
• make pytest -j works!

Caffe install on eldar (done)

Using Makefile.config directly

• git clone git@github.com:BVLC/caffe.git
• cp Makefile.config.example Makefile.config
• make all -j 10
• make test -j 10
• make runtest
• Works! (? - failed at some directory creation error in make runtest, possibly freak)

Using Makefile.config with some changes

• git clone git@github.com:BVLC/caffe.git
• cp Makefile.config.example Makefile.config
• edit Makefile.config (many of these lines may be already present, only need to uncomment): 
• CUSTOM_CXX := g++-4.6
• CUDA_DIR :=/opt/cuda-7.0/
• MATLAB_DIR := /lusr/share/software/matlab-r2014a/
• ANACONDA_HOME := $(HOME)/anaconda
• PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
• $(ANACONDA_HOME)/include/python2.7 \
• $(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include \
•  
• PYTHON_LIB := $(ANACONDA_HOME)/lib
• WITH_PYTHON_LAYER := 1
 
• make all -j 10  # Works
• make matcaffe -j 10 # Works, but with warning about gcc version (changed above, why still?)
• make pycaffe -j 10 # Works
• make test - j 10 # Works
• make runtest -j 10 # Works
• make pytest -j 10 # Works

SSH without password from machine A to machine B (done)

Login on A. Then:

cat ~/.ssh/id_rsa.pub | ssh b@B 'cat >> .ssh/authorized_keys'

Reference: http://www.linuxproblem.org/art_9.html

Torch installation on Delta HPC cluster (in progress)

1) Use Oxford's Torch installation script, modified: https://bitbucket.org/dj1989/active_vis/src/5aeec61c09dd2ffeccf92e4944a2056037459c3e/misc/torch-install/install-torch.sh?at=panocon_torch&fileviewer=file-view-default

2) In ~/local/luajit-rocks/luajit-rocks/lua-5.1/CMakeLists.txt, comment out the lines searching for readline. Do the same in ~/local/luajit-rocks/luajit-rocks/luajit-2.0/CMakeLists.txt. (This avoids "undefined reference to PC" errors when linking to readline, but will build without readline support though)

3) install-script-delta script works for installation of luajit, sundown, cwrap, paths, but fails at torch installation, with cmake complaining: "Performing Test OpenMP_FLAG_DETECTED - Failed"
Possibly only an issue on the login machine. To try on actual delta cluster machine.