As biomedical researchers, we often run analyses on Linux, whether on a high-performance computing (HPC) cluster or in the cloud. I created this tutorial to make Linux more approachable by connecting its underlying structure to the commands and applications we use in our research.
The tutorial uses an upside-down tree as a visual metaphor: the roots represent the hardware, the trunk the kernel, the branches the system services and tools, and the leaves the applications we use. Together, these layers offer a simplified view of how the system fits together.
Click any part of the tree to explore its role, or follow a task as it moves through the layers. New to the terminal? Begin with the story, which follows a researcher through a day of real work at the command line.
Level: introductory and conceptual, for researchers new to Linux. After this tutorial you should be able to explain how a command travels from the shell through the kernel to the hardware and back; read paths, permissions and common error messages and place them on the tree; and recognize where research tools such as Python environments, SSH, Slurm and containers fit. It is not a hands-on course: Where to go next lists the following steps.
Development note: I used Claude Code to translate my ideas and technical knowledge into the interactive diagram and accompanying stories. Feedback on clarity, accuracy and usability is welcome.
Branches group related parts of the system; they are not required execution paths. Applications, tools, services and libraries can interact across groups, and each reaches the kernel through system calls.
Swipe sideways inside the diagram to see the whole tree. It scrolls to follow each animation step.
The story
A Day Inside the Tree
Meet Safir, a research student in a genetics lab. Follow him through a working day at the terminal, from logging in to running jobs on a cluster. Along the way, see how commands connect applications, system tools, the kernel and hardware. Each chapter includes a Watch it on the tree button that animates the key interactions on the diagram.
About this tutorial: Safir uses an Ubuntu-based lab workstation with Bash and systemd. Commands, file locations and login methods may differ on other Linux distributions, HPC clusters and cloud systems. The tree is a simplified learning model, not a literal map of every interaction. The terminal sessions are illustrative: outputs were not all captured on one tested machine, and the research scripts are not included, so treat them as examples rather than a ready-to-run exercise (see Sources and scope).
08:30 · switching on
The tree wakes up
Before Safir types anything, Linux grows its tree from the roots up. Several startup steps prepare the system before he sees the login screen:
The firmware finds the disk and starts GRUB, the boot loader, whose files live in /boot.
GRUB loads the kernel (/boot/vmlinuz-…) and an initial RAM file system (/boot/initrd.img-…) into memory.
The kernel initializes devices and loads the required drivers. Many devices, including disks and terminals, are represented by special files in /dev.
Programs in the initial RAM file system help locate the main disk and mount it as /. The kernel also provides /proc and /sys, folders that exist only in memory.
Once the real root file system is mounted, control passes to the main startup program, /sbin/init, process number 1. On Safir's workstation that is systemd.
systemd reads its settings in /etc and starts the services: networking, logging into /var/log, SSH, the login screen.
Notice where init lives: /sbin. Historically, /bin held essential everyday commands such as ls and cp, while /sbin held system-administration tools such as fsck and fdisk. On many modern distributions both are symbolic links to locations under /usr. A command's location does not by itself determine who may run it.
Why it matters: kernel updates need enough free space in /boot. If an update reports that it is full, contact the system administrator. On a personally managed Ubuntu system, use the package manager rather than deleting files by hand, and review the list of changes it proposes before you confirm.
08:50 · before the day starts
How commands reach the system
When a program needs an operating-system service, such as reading a file or starting another program, it asks the kernel through the system call interface. The kernel manages access to the resources involved. In the tree picture, the leaves (apps) and branches (shell, tools, libraries) rely on the trunk (the kernel), which manages the roots (CPU, memory, disk, network).
The diagram shows a typical path for a command that accesses a file. It highlights the important layers rather than every implementation detail.
The dashed red line on the diagram, the system call interface, is the border. Above it is kernel space, where the kernel manages the hardware. Below it is user space, where your programs run. Crossing it means asking politely, through one of a few hundred system calls such as open, read, write and execve.
safir@genome01:~$ strace -c ls
# shows a summary table of the system calls ls made; the count varies
safir@genome01:~$ strace ls 2>&1 | head
# shows the individual calls, one per line
Why it matters: most chapters below follow a version of this path with different stops. Once you can picture it, error messages hint at which layer to check.
08:55 · multi-user login
Two people, one computer
Safir and his labmate Omar share the lab workstation genome01. They log in at the same time. Linux was built for this from day one.
When he typed his password, the login service checked who he is. On this workstation it uses an authentication framework called PAM. For a local password-based account, it looks him up in /etc/passwd (name, user ID, home folder, shell) and checks his password against the password hash stored in /etc/shadow, a file restricted to privileged access. After successful authentication, the login service starts bash running with his user and group identities, user ID 1001. The kernel uses those identities when enforcing access permissions, so every file he touches is checked against that number.
Curious, he peeks into Omar's folder:
safir@genome01:~$ ls /home/omar
ls: cannot open directory '/home/omar': Permission denied
safir@genome01:~$ ls -ld /home/omar
drwx------ 12 omar omar 4096 Oct 12 08:57 /home/omar
Read drwx------ in groups of three: d = directory, then rwx = the owner may read, write and enter, then --- = the group gets nothing, then --- = everyone else gets nothing. It was the kernel's security check that said no, not ls.
One account, root (user 0), has full administrative privileges. Safir's workstation is configured to allow members of the sudo group to run authorized commands with elevated privileges, so he can borrow those powers for one command with sudo. Which commands are permitted, and how their use is logged, depends on the system's sudo policy.
Why it matters: "Permission denied" is not a bug. It is the trunk doing its job. ls -l and id tell you who owns what and who you are.
09:10 · creating files and folders
A home for the project
safir@genome01:~$ mkdir -p ~/projects/hla-study/{data,scripts,results}
safir@genome01:~$ cd ~/projects/hla-study
safir@genome01:~/projects/hla-study$ echo "sample,allele" > data/samples.csv
safir@genome01:~/projects/hla-study$ touch scripts/clean.py
safir@genome01:~/projects/hla-study$ tree
.
├── data
│ └── samples.csv
├── results
└── scripts
└── clean.py
Bash first expands ~ into Safir's home-directory path (/home/safir) and {data,scripts,results} into three directory names. The mkdir program then asks the kernel to create those directories, typically through the C library (glibc). The kernel checks access permissions and passes the request to the file system, ext4 on this workstation.
A surprise: the kernel first keeps the change in RAM (the page cache) and writes it to disk a moment later. That is why an unexpected power loss can lose the last few seconds of work, and why sync exists.
Unlike deleting through many desktop file managers, rm normally does not move files to a trash folder. Treat it as permanent deletion.
Names starting with a dot, like .bashrc, are hidden. See them with ls -a.
/tmp is temporary storage and may be cleared automatically or at reboot. Do not use it for results you need to keep.
df -h shows used and available space on mounted file systems (a file system can sit on a disk partition, a logical volume, a network server or memory). du -sh * shows which files and directories are using the most space.
On a shared research system, follow your institution's guidance on project storage, scratch space and backups.
Why it matters: your files live in /home (yours), shared programs in /usr (read-only for you), scratch in /tmp. Knowing this tells you where to look, and where never to save.
09:40 · using bash
Bash, the glue
Safir has added 40,000 records to his sample table. He wants the three most frequently recorded HLA alleles in its second column, without opening a spreadsheet:
tail -n +2 skips the header line. This example assumes a simple two-column file with one allele per row and no quoted fields containing commas.
Bash starts the programs and connects them into a pipeline: each program's standard output becomes the next program's standard input. The programs can run concurrently while the operating system schedules their work. Some stages, such as sorting, must collect all their input before producing results.
Solid arrows are kernel-managed pipes, not intermediate files that Safir must create. The dashed arrow is ordinary output to the terminal, not a pipe. sort may still use its own temporary files when its input exceeds the available sorting buffer.
How does Bash find sort? For an external command, it searches the directories in PATH in order, unless it already has a cached location. Bash also recognizes functions and built-in commands, so not every command comes from a file in a PATH directory:
safir@genome01:~/projects/hla-study$ printf '%s\n' "$PATH"
/home/safir/.local/bin:/usr/local/bin:/usr/bin:/bin
safir@genome01:~/projects/hla-study$ command -v sort
/usr/bin/sort
safir@genome01:~/projects/hla-study$ type cd sort
cd is a shell builtin
sort is /usr/bin/sort
Look deeper: shared libraries
Even sort does not work alone. It borrows shared code from /lib, the System Libraries branch of the tree:
The first line, #!/bin/bash (the "shebang"), tells the kernel which program should read the script. chmod +x switches on its "may run" permission. echo $? prints the previous command's exit status. Zero normally indicates success, while a nonzero value indicates an error or another command-specific condition. For a pipeline, Bash normally reports the last command's status, not the status of every stage. Automated workflows use exit codes to detect problems, as the CI chapter shows.
Look deeper: checking every stage of a pipeline
set -o pipefail makes a pipeline fail if any stage fails. Use it with care when the pipeline ends in head: head may stop reading early, and the program before it can then receive a "broken pipe" signal that counts as a failure.
Why it matters: "command not found" means Bash found no alias, function, built-in or file in a PATH directory with that name. type name shows what Bash would run instead. Personal shortcuts such as alias ll='ls -lh' go in ~/.bashrc.
The first command refreshes the package information; it does not upgrade the installed software.
APT reads the configured software repositories under /etc/apt/, retrieves the required packages, and verifies them using signed repository metadata and package checksums. The package's files are installed in designated locations: the program in /usr/bin, its libraries in /usr/lib, its manual in /usr/share/man, its settings in /etc. The package manager records what was installed under /var/lib/dpkg. Removing a package later does not necessarily delete its configuration files or data you created.
Where software lives
Who puts it there
Usually needs administrator access?
/usr/bin
the package manager (apt, dnf)
yes
/usr/local/bin
you, building from source
yes
/opt/<vendor>
vendor installers (Chrome, MATLAB)
yes
~/.local/bin
your own scripts; command-line tools installed with pipx
no
~/venvs, ~/miniconda3
virtual environments, conda
no
On current Ubuntu releases the system Python is externally managed: pip install outside a virtual environment, even with --user, is refused so it cannot break packages apt installed. Project dependencies belong in a virtual environment, the next chapter.
Why it matters: on a shared cluster, Safir normally cannot install system-wide packages. He first checks the software the institution already provides, then follows local guidance for user-managed environments and installations, such as the last two rows of the table.
10:45 · Python virtual environment
A private toolbox
Safir's analysis was written and validated against pandas 2, so he keeps it below pandas 3 (the current major version) until he has tested the upgrade. Omar's older pipeline breaks with anything newer than pandas 1.5. If either of them upgraded the system-wide copy, the other's work would break. The fix is a virtual environment. On Ubuntu, venv support is a separate package, installed once by an administrator with sudo apt install python3-venv (or python3-full):
A virtual environment is a directory containing its own Python environment and package-installation location (lib/python3.x/site-packages). Its Python executable may be a copy of, or a symbolic link to, the interpreter used to create it. Activating the environment places its bin directory at the front of PATH, sets environment-related variables, and usually changes the shell prompt. No administrator privileges are needed to create it in Safir's home directory. Conda environments work on a similar idea and can also bring their own copy of Python and C libraries.
Activation is convenient but not required. Safir can call the environment's Python directly:
Why it matters: when an import fails or a package version is unexpected, first check which Python interpreter is running. This confirms that the analysis is using the intended environment.
11:30 · SSH to a server
A terminal on another computer
The lab's big server, hla-srv, sits in a locked server room. Safir works on it from his laptop. First, a one-time key setup:
safir@laptop:~$ ssh-keygen -t ed25519
Enter passphrase (empty for no passphrase):
Your identification has been saved in /home/safir/.ssh/id_ed25519
Your public key has been saved in /home/safir/.ssh/id_ed25519.pub
safir@laptop:~$ ssh-copy-id safir@hla-srv.example.edu
He sets a passphrase, so a stolen key file is not enough on its own. On his first connection, Safir verifies the server's host-key fingerprint using information supplied by the lab's system administrator. SSH records the server's identity in ~/.ssh/known_hosts and checks it on later connections.
The ssh client (a networking tool) uses a crypto library to encrypt the connection. The laptop's kernel sends packets through the network card to the server, on port 22 by default. On the server, the same tree in reverse: the network card, the kernel, then sshd, the SSH service, listening on port 22 (on recent Ubuntu, systemd starts it when the first connection arrives). During public-key authentication, the client proves that it can use Safir's private key. The server checks whether the matching public key is authorized for his account, typically through ~/.ssh/authorized_keys. The private key is never sent to the server. Once he is authenticated, sshd starts bash there as safir.
Two trees connected through a network.
authorized_keys (on the server) lists which user keys the server accepts. known_hosts (on the laptop) lists which server identities the client recognizes.
The private key ~/.ssh/id_ed25519 stays on the laptop. Only the .pub half goes to servers.
SSH refuses keys that others can read: chmod 600 ~/.ssh/id_ed25519.
A shortcut in ~/.ssh/config on the laptop saves typing:
Why it matters:Permission denied (publickey) means public-key authentication did not succeed. Check the username, the selected key, server authorization and relevant file permissions. ssh -v shows which authentication steps the client attempted.
Laptop:Server:
12:15 · checking on the machine
Files that are not really files
His script feels slow. Before lunch, Safir checks the machine's health:
safir@genome01:~$ free -h
total used free shared buff/cache available
Mem: 62Gi 42Gi 2.1Gi 1.0Gi 19Gi 20Gi
safir@genome01:~$ grep -E '^(MemTotal|MemFree|MemAvailable):' /proc/meminfo
MemTotal: 65011712 kB
MemFree: 2202008 kB
MemAvailable: 20971520 kB
safir@genome01:~$ ls -l /proc/meminfo
-r--r--r-- 1 root root 0 Oct 12 12:15 /proc/meminfo
Look at the size: 0 bytes, yet it is full of numbers. /proc is a window into the kernel. Nothing in it is stored on disk; the kernel writes the answer at the moment you read it. free, top and ps are just friendly readers of /proc. Each process even gets its own folder, named after its process ID:
safir@genome01:~$ echo $$
48211
safir@genome01:~$ head -1 /proc/48211/status
Name: bash
/dev is the hardware side of the same idea: df -h /home shows that the file system holding his files is on the partition /dev/nvme0n1p2.
Logs may be stored under /var/log; journalctl reads the systemd journal. journalctl -p err --since today lists today's errors.
The system reports about 20 GiB of available memory, so this snapshot does not suggest immediate system-wide memory pressure. Safir next checks CPU activity with top and finds another job heavily using the machine. CPU contention is a possible explanation for the slowdown, but he would need further checks to confirm it.
Why it matters: a bare "Killed" message does not identify the cause. Memory exhaustion is one possibility; a scheduler limit or another termination request may also be involved. Check the relevant logs and, on a cluster, the scheduler's job records.
13:00 · cloud and HPC over SSH
Thousands of trees, one front door
Based on earlier benchmarking, Safir requests 64 CPUs for his scoring workflow; his pandas code does not use a GPU, so he requests none. He logs in to the university cluster and, once, creates a Python environment there. The workstation environment at ~/venvs/hla does not appear on another machine automatically.
The login node is the cluster's front door, shared by hundreds of people. It is for editing, copying and submitting, not for heavy work. sbatch hands the script to Slurm, whose scheduling service (slurmctld) waits until a compute node has the requested resources free. That node is another full Linux tree. Slurm allocates the resources; how strictly they are enforced depends on the cluster's configuration. Because the job sets --cpus-per-task, Slurm provides SLURM_CPUS_PER_TASK with that requested number, so the script passes it to score.py instead of repeating a hard-coded value. It describes this job's request, not the machine's total CPUs. Requesting CPUs does not by itself make a program parallel; the program has to use them.
Modules configure the software environment, often updating PATH and other variables.
Partition, account and memory settings must follow local documentation. Requirements differ; some institutions require an explicit memory request.
Home-directory quotas and storage policies vary by institution. Large data, environments and model caches usually belong in project or scratch storage such as /work or /scratch.
Request GPUs only for GPU-enabled code. When a job does hold GPUs, they appear as device files such as /dev/nvidia0, and nvidia-smi on the compute node shows whether they are actually busy.
Cloud machines (AWS, Azure, Google Cloud) are reached the same way: ssh into a Linux VM.
Why it matters: heavy work on the login node may be stopped by administrators. A full home directory gives "Disk quota exceeded". A GPU waiting on a slow data loader wastes the allocation. These problems may involve system configuration, resource allocation or application behavior.
14:30 · containers
Docker and Apptainer: a tree in a box
A reviewer asks: "Can someone else rerun this analysis using the same software environment?" Safir packs the environment into a container image:
safir@laptop:~/hla-study$ cat Dockerfile
FROM python:3.11-slim
RUN pip install "pandas==2.2.*"
COPY scripts/ /app/
CMD ["python", "/app/score.py"]
safir@laptop:~/hla-study$ docker build -t hla-score:1.0 .
safir@laptop:~/hla-study$ docker run --rm -v "$PWD/data":/data hla-score:1.0
safir@laptop:~$ uname -r
6.8.0-45-generic
safir@laptop:~$ docker run --rm python:3.11-slim uname -r
6.8.0-45-generic
# same kernel inside and outside the container
That last line is the key idea. A container is not a little computer. It is ordinary processes that the host kernel wraps in namespaces (their own view of /, their own process list, their own network) and cgroups (which account for CPU and memory use, and enforce limits when you set them, for example with --cpus or --memory; the command above sets none). The container brings its own branches and leaves (/usr, /lib, Python, pandas) but borrows the host's trunk. A virtual machine brings its own trunk too.
Container
Virtual machine
Safir's laptop runs Docker Engine directly on Linux in its conventional mode: a background service, dockerd, runs as root (a rootless mode also exists). Docker keeps its data under /var/lib/docker; on fresh Docker Engine 29 and later installations, image contents and container snapshots live in /var/lib/containerd instead. A running container sees the image's read-only layers plus a thin writable layer of its own. -v "$PWD/data":/data mounts a real folder from his tree into the container's tree. Anything written to the writable layer, outside a mounted folder, vanishes when the container is removed.
The tag 1.0 is a convenient label, but a tag can later be moved to a different image. The image digest (sha256:…) identifies the exact content; a locally built image gets its registry digest when it is pushed, and docker push prints it. Rebuilding later from python:3.11-slim and pandas==2.2.* could pull in newer versions, so Safir publishes the built image to the lab's registry and records its digest with his results.
safir@laptop:~/hla-study$ docker tag hla-score:1.0 registry.example.edu/hla-lab/hla-score:1.0
safir@laptop:~/hla-study$ docker push registry.example.edu/hla-lab/hla-score:1.0
On the cluster, Docker's root service is not allowed. Apptainer fills the gap: no background service, the image is one .sif file, and he stays himself (uid 1001) inside. He pulls the image by the digest he recorded (replace the placeholder with that value), so he gets exactly the content he tested, then runs the analysis in a scheduled job on a compute node, not on the login node:
This job uses CPUs only, because pandas does not use a GPU. For a GPU-enabled application, he would add #SBATCH --gres=gpu:1 and apptainer exec --nv: --nv makes NVIDIA devices and host driver libraries available inside the container, but neither the request nor the flag makes ordinary code use a GPU. His home directory is usually available inside the container; other paths depend on the cluster's configuration. Some clusters also ask users to pull or build images inside a job.
Why it matters: once you know containers share the kernel, the puzzles disappear: why a Linux container cannot run Windows, why results must go to a mounted folder, and why --nv is needed to see the GPU.
GitHub starts a fresh Ubuntu virtual machine in the cloud, called a runner. A runner service on it reads the YAML file and runs the steps in order. setup-python installs Python 3.11, matching the container (3.11 is older but still receives security fixes until October 2027). Each run step starts a new noninteractive shell, and GitHub records the exit status of each step. If every step succeeds, the run gets a green check; a nonzero exit code fails that step, marks the run with a red X and stops the remaining steps. This workflow covers testing and building. The tests run in the runner’s Python, not inside the image, so they do not show that the image itself works; a further step could run the tests in a container started from it. Deployment, the "CD" half, would need extra steps: push the image to a registry, or connect to a server and restart a service. On that web server, the site often lives in /srv/www: /srv is the folder set aside for data a machine serves to others.
Why it matters: a CI log is just a terminal you didn't type into. When it turns red, you debug it with everything above: PATH, permissions, packages, exit codes.
17:00 · backing up
Plugging in: /dev, /media and /mnt
Before going home, Safir copies his results to a USB drive.
/dev/sdb1 identifies a partition; /media/safir/BACKUP is the directory where its file system is mounted, meaning attached to the tree, so its files look like any other folder. The desktop did that automatically, which is what /media is for: removable things.
/mnt is the do-it-yourself spot. His supervisor attaches the lab's network share by hand:
Why it matters: remember the RAM cache from 09:10? Unmount before pulling the drive, or the last files may never reach it. "Target is busy" means a process is still using the file system, for example through an open file or a working directory. Close those references, then retry.
17:30 · wrapping up
When something breaks, find the layer
An error message provides clues about which layer to investigate first. Some problems cross several layers, so use the tree to organize your checks rather than assume the cause.
You see
Layer on the tree
First thing to try
command not found
Shell (PATH)
type name, printf '%s\n' "$PATH"
Permission denied
Kernel security
ls -l file, id
No such file or directory
File system
pwd, ls; names are case-sensitive
No space left on device
Storage
df -h and df -i, then du -sh * | sort -h
Disk quota exceeded
Storage (quota)
the institution's user or project quota tools
Killed
Process forcibly terminated; investigate the cause
free -h, system logs; on clusters, sacct -j JOBID
Connection refused / timed out
Network or service
ssh -v host, then systemctl status sshd on the server
Permission denied (publickey)
SSH keys
username, key, ls -l ~/.ssh, ssh -v host
ModuleNotFoundError
Module unavailable to the active interpreter
python -c "import sys; print(sys.executable)", activate the intended venv
A service silently stops
Services
journalctl -u name, /var/log
17:45 · closing the laptop
The map of the day
In one day Safir used every layer of the tree: logins checked at the trunk, files written to the roots, tools on the branches, apps as leaves, and other trees reached through the network. Along the way, he encountered many common directories in the Linux filesystem hierarchy. Here is where each one showed up:
/etc comes from "et cetera". "Editable Text Configuration" is a later explanation that happens to fit.
/tmp is not emptied when a program finishes. Most systems clear it at reboot, and some also delete files there after several days.
/bin, /sbin and /lib are now usually shortcuts into /usr, so /bin/ls and /usr/bin/ls are the same file.
Once you can see the tree, a path stops being a string of letters and becomes an address, and an error message stops being a wall and becomes a map.
Where to go next
This tutorial gives the picture; mastery comes from practice. The table maps a full Linux learning path for research computing onto what this page covers, with one official or well-established resource for each next step. For bioinformatics work, Bash, permissions, SSH, environments, Git, containers, a workflow manager and cluster jobs repay the effort first.
Topic
In this tutorial
Next step
Start with
Linux fundamentals
Introduced: the system's layers, paths, directories, everyday commands
The examples assume Ubuntu 24.04 LTS with Bash, systemd and Docker Engine on Linux, as of October 2026. The command outputs are illustrative and were not all captured on one tested machine, and the research scripts (score.py, its tests and data) are not included: the snippets show the shape of the workflow, not a ready-to-run exercise. Cluster settings follow your institution's documentation.