UP | HOME

Advanced Processes
Using the Command-Line
COP-3402

Table of Contents

Pipelines

Create complex tools from simple, existing programs.

UNIX philosophy

  1. Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new "features".
  2. Expect the output of every program to become the input to another, as yet unknown, program. Don't clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don't insist on interactive input.
  3. Design and build software, even operating systems, to be tried early, ideally within weeks. Don't hesitate to throw away the clumsy parts and rebuild them.
  4. Use tools in preference to unskilled help to lighten a programming task, even if you have to detour to build the tools and expect to throw some of them out after you've finished using them.

Doug McIlroy

Example: searching for a file

  • find lists each file in a file tree
  • grep searches for text

Use stdio to connect these program

# list all files
find > findresults

# get only files with hello in its path
grep hello < findresults

Use a pipe and avoid making new files

find | grep hello

Separate bash commands with a pipe | symbol to pass stdout to stdin 

cat /usr/include/stdio.h | grep head

What are pipes?

  • OS creates two "virtual" files
  • Writing to first file outputs to second file
  • Allows processes to pass data between them

Diagram

The operating system handle intermediate files for us. Made possible because all programs have standard in and out.

We'll use pipes to build our own shell command processor later in the semester.

xargs executing a command on many arguments

find | xargs file

xargs takes each line from stdin and turns them into arguments for the given command, in this case the file command.

What does file do?

How can we run file on a lot of things? Use xargs

Building complex tools from simpler ones

Print the first few lines of every header file 

find /usr/include | grep "\.h$" | xargs head -n3
  • head -n3 prints the first 3 lines of a file
  • xargs takes each line from stdin and executes the given command on it
  • xargs head -n3 takes each line from stdin and executes head -n3 on it

Complex text processing

I want to get all configuration options defined by the Linux build system. 

# search for all configuration options that you can choose when building the Linux kernel
find | grep Kconfig | xargs grep "^config"

# deduplicate and sort them as well
find | grep Kconfig | xargs grep -h "^config" | sort | uniq

# trim off the "config " keyword
find | grep Kconfig | xargs grep -h "^config" | sort | uniq \
     | cut -f2 -d' '

# count the results
find | grep Kconfig | xargs grep -h "^config" | sort | uniq \
     | cut -f2 -d' ' | wc -l

Pipe both stdout/stderr

Use |& instead of | 

grep define /root |& grep -i Permission

Job management

Managing multipe processes from the shell.

Multiprocessing means we can have multiple programs running concurrently.

Killing jobs

Ctrl-C kills a running program.

Suspending jobs

Ctrl-Z suspends a running program. 

find /
# type ctrl-z

[1]+  Stopped                 find /

Resuming a job

fg resumes a suspended program to the foreground. 

fg
# find continues running
# type ctrl-z to suspend again

Foreground vs. background

  • Foreground means process is running over the shell
  • Background means process running separately from the shell

Foreground and background are relative to the interactive shell. The kernel doesn't distinguish foreground or background processes, all of which are concurrent.

The technical details are more subtle. See man bash JOB CONTROL

Foreground processes

  • Will block the user from running additional shell commands
  • Can receive signals, e.g., Ctrl-z

Background processes

  • No bash signals will work (no Ctrl-C or Ctrl-Z)
  • Output still to terminal (if not redirected)

Put process into background with bg

find /
# type ctrl-z
bg
# ctrl-c will not terminate it
fg
# now ctrl-c will work

You won't be able to see your command-line, because the find command is emitted text so quickly, that the bash shell, which using the same terminal as output, is quickly moved up the screen.

Have faith and type fg then Ctrl-Z to suspend the program.

Put process into the background immediately &

Suffix the command with & to put it in background immediately. 

cat &
# to bring into foreground, use fg
fg
# suspend again with Ctrl-Z
# bring to back again with bg
bg

Warning: the stdout/stderr will still be the terminal if not redirected.

Viewing running jobs

find / > /dev/null 2>&1 &
cat &
grep hello &
jobs
fg 2 # bring up the second job, cat &

Really killing a process

  • Ctrl-C basically asks the process to terminate (SIGTERM)
  • SIGTERM can be ignored by the program
  • SIGKILL cannot be
find / > /dev/null 2>&1 &
ps
kill -9

See man kill for more details

See man 7 signal for signals and their codes

echo $! will tell the last command's pid, so we can do this: 

find / > /dev/null 2>&1 &
ps
kill -9 $!

Quick-and-dirty development workflow

  • Use job control to keep an editor running
  • Don't exit, just use Ctrl-Z to suspend editor
  • Compile, test, etc
  • Use fg to resume editing
emacs hello.c
# type Ctrl-Z
gcc -o hello hello.c
./hello
fg

There are better workflows, but this is great for simple, quick scripting tasks for instance.

Terminal multiplexing

A terminal multiplexer is like a remote desktop for command-line shells.

We'll use byobu in this class.

byobu

A wrapper for managing terminal multiplexers

byobu is really a wrapper around multiplexers and using tmux by default. GNU screen is an alternative backend for byobu.

Initialization

Setup the multiplexer for bash/emacs' Ctrl-A keybinding. 

byobu-ctrl-a
# Type 2 and hit enter

This is done historically multiplexers use ctrl-a to enter multiplexer commands. We will use F# commands instead.

Entering byobu

In eustis, run 

byobu

You can tell you are in byobu (tmux), because there is a status bar at the bottom of the screen.

If you already have byobu sessions, byobu will prompt you to connect to an existing one or allow you to make a new one. Otherwise, byobu will create a new multiplexer session.

Creating additional windows

Press F2 to create a new terminal "window".

byobu_status.png

Note the new "tab" with number 1 near the bottom-left of the terminal.

If you can't see the whole status bar, try expanding the terminal window.

Additionally, you can turn off status notifications interactively with F9. Remember, you will only be able to use keyboard (arrow keys, <tab>, <enter>) to navigate.

Detaching and re-attaching

F6 detaches from the byobu session

Rerun byobu to reattach 

echo "hello, world!" # to show that we indeed are reattaching
# press ~F6~ to detach
# you will return to the original eustis bash session.
exit # you can even exit and reconnect
ssh eustis..
byobu
# select (1) or your last byobu session

Navigating byobu

  • F3 or Alt-<left> to go to the previous window
  • F4 or Alt-<right> to go to the next window

byobu Cheat Sheet

byobu-ctrl-a
# Type 2 and hit enter
Command Description
F6 Detach from byobu
F2 New byobu terminal
F3 or Alt-<left> Go to left terminal
F4 or Alt-<right> Go to right terminal
exit or Ctrl-D exit terminal (not byobu-specific)

Takeaways

Pipes

Chain multiple programs together, e.g., 

find / | grep bin | wc -l

Manage multiple processes

Suspend and resume processes into the background or foreground to work with multiple programs. 

emacs hello.c
# Ctrl-Z to suspend
gcc -o hello hello.c
./hello
fg # to resume

Manage remote terminal sessions

Use a multiplexer to manage multiple terminal screens and preserve (though eustis is configured to prevent this) remote terminal sessions. 

byobu
emacs hello.c
# F2 to create a new terminal window
gcc -o hello hello.c
./hello
# F3 to switch back to the editor
# F6 to detach and resume working later

Author: Paul Gazzillo

Created: 2024-09-24 Tue 22:30

Validate