Welcome to FUC-CHEN¶
Commands Basics¶
Help commands¶
$ man <command> # man page
$ whatis <command> # whatis
$ <command> -h # option -h
Environment variables¶
Create variable in shell
$ <env_var>="var_value"
$ export <env_var>
Define vairable in bash configuration file
export <env_var>="var_value" # add the setting to ~/.bashrc
source ~/.bashrc # load new environment variables to shell
Print variable
$ echo $<env_var>
$ printenv <env_var>
History commands¶
$ history # check recent commands
$ history | grep <string> # find previous command with specific string
Directory change¶
$ pwd # check current directory
$ cd ~ # go to home directory
$ cd - # go to previous directory
$ cd .. # go to parent directory
Download from the internet¶
$ wget <url>
$ curl <url>
Git Commands¶
Delete branch¶
Delete local branch
$ git branch -D <branch_name>
Delete remote branch
$ git push <remote_name> --delete <branch_name>
Tag¶
Add tag to current submit
$ git tag -a <tag_name> -m <comment>
List all tags
$ git tag -l
Delete local tag
$ git tag -d <tag_name>
Delete remote tag
$ git push --delete origin <tag_name>
Remote¶
Set up upstream remote
git remote add upstream https://github.com/repo
Update local repo
git pull upstream <branch_name>
Push to upstream
git push upstream <branch_name>
File Operations¶
Find files contain specifc text¶
$ grep -rin <text> .
Check specifc lines inside a text file¶
$ sed -n <start_line_num>:<end_line_num>p <text_file>
Get number of all files¶
When number of file is small, less than 10, 000
$ ls -ls *.<ext> | wc -l
When more than 10, 000 files
$ find -type f -name '*.<ext>' | wc -l
Storage usage¶
Check disk partition usage
$ df -h
Check the size of a directory
$ du -hs <directory>
Check size of all subdirectories¶
$ du -hs *
Searches directory recursively in subdirectories¶
$ find . -type d -name <directory>
Searches file recursively in subdirectories¶
$ find . -type f -name <file>
Delete file/directory recursively in subdirectories¶
$ find . -name <file/directory> -exec rm -rf {} ;
Docker Management¶
Docker is an open platform for developers and sysadmins to build, ship, and run distributed applications, whether on laptops, data center VMs, or the cloud.
Docker Installation¶
$ sudo apt install docker.io # install docker
$ sudo usermod -aG docker $USER # add user to docker group
Image management¶
Create image from Dockerfile
$ docker build -t <image_name>:[tag_name] .
Create image from a specific Dockerfile
$ docker build -f <dockerfile_name> -t <image_name>:[tag_name] .
Image management
$ docker save <image_name>:[tag_name] <image_name>.tar # save image
$ docker load --input <image_name>.tar # load image
$ docker images # list available images
$ docker rmi <image_name> # remove image by name
$ docker rmi <image_id> # remove image by id
$ docker images purge # remove dangling images
Container management¶
Start container
$ docker run -it --restart always --name <container_name> <image_name>:[tag_name]
Options:
-v <local_dir>:<docker_dir>:ro # map local directory to docker
--runtime=nvidia # expose NVIDIA GPUs
-e NVIDIA_VISIBLE_DEVICES=5 # cuda device setting
--shm-size 16G # set size of shared memory
--rm # remove container file system when exits
Copy files
$ docker cp <container_name>:<src_dir> <local_dst_dir> # copy files from docker to local
$ docker cp <local_src_dir> <container_name>:<dst_dir> # copy files from local to docker
Container management
$ docker ps # list all available containers
$ docker stop <container_name> # stop specific container
$ docker rm <container_name> # remove specific stopped container
Upload image to docker hub¶
Login to docker hub
$ export DOCKER_ID_USER="user_name" # set docker hub username
$ docker login # login in to docker hub
Tag image
$ docker tag <image_name>:<version> $DOCKER_ID_USER/<image_name>:<version>
Push to docker cloud
$ docker push $DOCKER_ID_USER/<image_name>:<version>
Convert image to singularity¶
Create docker image tarball
$ docker save <image_name>:<version> -o <image_name>.tar
Build singularity from image tarball
$ singularity build <image_name>.sif docker-archive://<image_name>.tar
Build singularity from DockerHub image
$ singularity pull <image_name>.sif docker://<user_name>/<image_name>:<version>
User Management¶
Add user with full profile¶
$ adduser <username>
Add user with only name¶
$ useradd <username>
Add user to sudo group¶
$ usermod -aG sudo <username>
List all user names¶
$ compgen -u
Delete user
$ userdel -r <username>
Compression Operations¶
Tar and untar¶
Create tar archive File
$ tar -cvf tar-archive-name.tar <source>
Untar files in current directory
$ tar -xvf tar-archive-name.tar
Untar files to a specific directory
$ tar -xvf tar-archive-name.tar -C <destination>
List content of tar archive file
$ tar -tvf tar-archive-name.tar
Compress and uncompress¶
Compress folder to tar.gz
$ tar -czvf tar-archive-name.tar.gz <source>
Extract a tar.gz compressed archive
$ tar -xzvf tar-archive-name.tar.gz
Compress folder to tar.bz2
$ tar -cjvf tar-archive-name.tar.bz2 <source>
Extract a tar.bz2 compressed archive
$ tar -xjvf tar-archive-name.tar.bz2
Process Management¶
Check process¶
View your system’s resource usage and see the processes that are taking up the most system resources.
$ top
Improved top
$ htop
Kill process¶
By process id
$ kill -9 <pid>
By application name
$pkill <app>
By filtering conditions
ps -ef | grep <command> | awk '{print $<col_num>}' | xargs kill -9
Check Ubuntu version¶
$ lsb_release -a
Seadragon¶
Seadragon is a supercomputing resource of MD Anderson’s High Performance Computing (HPC), which is dedicated for use by the research community.
Login Seadragon¶
$ ssh seadragon
Interactive Running¶
# cpu interactive
$ busb -Is -q interactive -W 1:00 -M 64 -R rusage[mem=64] –n 4 /bin/bash
# gpu interactive
$ busb -Is -q gpu-medium -gpu num=1:gmem=16 -W 3:00 -M 64 -R rusage[mem=64] –n 10 /bin/bash
Load Modules¶
$ module load singularity/3.5.2
$ module load cuda10.1/toolkit/10.1.243
Start Singularity Instance¶
$ singularity run --nv --bind <local_dir>:<container_dir> <singularity_path> <program>
More Useful Commands¶
$ bsub < <lsf_script> # submit job via script
$ bjobs -u all | more # check all running jobs
$ bjobs -u all | grep gpu # check all gpu jobs
$ bjobs -p # show all pending jobs
$ bjobs -l xxxxxxx # check the details of one specific job
$ bkill -l xxxxxxx # kill one specific job
Conda Env¶
List available envs¶
$ conda info --envs
Create env¶
$ conda create --name <env_name> python=3.6
Activate env¶
$ source activate <env_name>
Deactivate env¶
$ source deactivate
Remove env¶
$ conda remove --name <env_name> --all
Virtual Env¶
Installation¶
$ pip install virtualenv
Create virtual environment¶
Create with specific python version
$ virtualenv -p python3 myvenv
Activate environment¶
$ source myvenv/bin/activate
Deactivate the environment¶
$ deactivate
tmux¶
tmux is a terminal multiplexer: it enables a number of terminals to be created, accessed, and controlled from a single screen. tmux may be detached from a screen and continue running in the background, then later reattached.
Installation¶
$ sudo apt-get install tmux
List available sessions¶
$ tmux ls
Start new session¶
$ tmux new -s <session_name>
Attach to session¶
$ tmux a -t <session_name>
Kill session¶
$ tmux kill-session -t <session_name>
sftp¶
SSH File Transfer Protocol, a network protocol used for secure file transfer over secure shell.
Login to server¶
Login to user home directory of the server.
$ sftp <user_name>@<server_ip>
Login to specific directory of the server.
$ sftp <user_name>@<server_ip>:<remote_dir>
File transfer¶
Download file from server.
$ get <server_path> <local_path>
Upload local file to server.
$ get <local_path> <server_path>
Command¶
Command for local host.
$ !<command>
Command for server.
$ <command>
Exit server¶
Exit from server.
$ bye
apache¶
Restart¶
$ service apache2 restart
Check error log¶
$ tail -n 20 /var/log/apache2/error.log