How to master environment variables

If you have been using Linux for a while you might have encountered the term “environment variables” a few times. You might even have run the command export FOO=bar occasionally. But what are environment variables really and how can you master them?

In this post I will go through how you can manipulate environment variables both permanently and temporarily. Lastly I will round up with some tips on how to properly use environment variables in Ansible.

Check your environment

So what is your environment? You can inspect it by running env on the command line and search with a simple grep:

$ env
COLORTERM=truecolor
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
DESKTOP_SESSION=gnome
DISPLAY=:1
GDMSESSION=gnome
GDM_LANG=en_US.UTF-8
GJS_DEBUG_OUTPUT=stderr
GJS_DEBUG_TOPICS=JS ERROR;JS LOG

... snip ...

$ env | grep -i path
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
OMF_PATH=/home/ephracis/.local/share/omf
PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin
WINDOWPATH=2

So what are all these variables coming from and how can we change them or add more, both permanently and temporarily?

Know your session

Before we can talk about how the environment is created and populated, we need to understand how sessions work. Different kinds of sessions reads different files for populating their environment.

Login shells

Login shells are created when you SSH to the server, or login at the physical terminal. These are easy to spot since you need to actually log in (hence the name) to the server in order to create the session. You can also identify these sessions by noting the small dash in front of the shell name when you run ps -f:

$ ps -f
UID        PID  PPID  C STIME TTY          TIME CMD
ephracis 23382 23375  0 10:59 pts/0    00:00:00 -fish
ephracis 23957 23382  0 11:06 pts/0    00:00:00 ps -f

Interactive shells

Interactive shells are the ones that reads your input. This means most sessions that you, the human, are working with. For example every tab in your graphical terminal app is an interactive shell. Note that this means that the session created when you login to your server either over SSH or from the physical terminal, is both an interactive and a login shell.

Non-interactive shells

Apart from interactive shells (of which login shells are a subset) we have non-interactive shells. These would be the ones created from various scripts and tools that do not attach anything to stdin and can thus not provide interactive input to the session.

Know your environment files

Now when we know about the different types of sessions that can be created, we can start to talk about how the environment of these sessions are populated with variables. On most systems we use Bash since that’s the default shell on virtually all distributions. But you might have changed this to some other shell like Zsh or Fish, especially on your workstation where you spend most of your time. What kind of shell you use will determine which files are used to populate the environment.

Bash

Bash will look for the following files:

  • /etc/profile
    Run for login shells.
  • ~/.bash_profile
    Run for login shells.
  • /etc/bashrc
    Run for interactive, non-login shells.
  • ~/.bashrc
    Run for interactive, non-login shells.

That part about non-login is important and a reason why many users and distributions configure bash_profile to read bashrc so that it is applied in all sessions, like so:

[[ -r ~/.bashrc ]] && . ~/.bashrc

Zsh

Zsh will look for a bit more files than Bash does:

  • /etc/zshenv
    Run for every zsh shell.
  • ~/.zshenv
    Run for every zsh shell.
  • /etc/zprofile
    Run for login shells.
  • ~/.zprofile
    Run for login shells.
  • /etc/zshrc
    Run for interactive shells.
  • ~/.zshrc
    Run for interactive shells.
  • /etc/zlogin
    Run for login shells.
  • ~/.zlogin
    Run for login shells.

Fish

Fish will read the following files on start up:

  • /etc/fish/config.fish
    Run for every fish shell.
  • /etc/fish/conf.d/*.fish
    Run for every fish shell.
  • ~/.config/fish/config.fish
    Run for every fish shell.
  • ~/.config/fish/conf.d/*.fish
    Run for every fish shell.

As you can see Fish does not distinguish between login shell and interactive shells when it reads its startup files. If you need to run something only on login or interactive shells you can use if status --is-login or if status --is-interactive inside your scripts.

Manipulate the environment

So that’s a bit complicated but hopefully things are more clear now. Next step is to start manipulating the environment. First of all, you can obviously edit those files and wait until the next session is created, or load the newly edited file into your current session using either source /path/to/file or the shorthand . /path/to/file. That would be the way to make permanent changes to your environment. But sometimes you only want to change this temporarily.

To apply variables for a single command you just insert it to the beginning of the command like so:

# for bash or zsh
$ FOO=one BAR=two my_cool_command ...

# for fish
$ env FOO=one BAR=two my_cool_command ...

This will make the variable available for the command, and then go away as soon as the command finishes.

If you want to keep the variable and have it available to all future commands in your session you run the assignment as a stand alone command like so:

# for bash or zsh
$ FOO=one

# for fish
$ set FOO bar

# then use it later in your session
$ echo $FOO
one

As you can see the variable is available for the echo command run later in the session. The variable will not be available to other sessions, and will disappear when the current session ends.

Finally, you can export the variable to make it available to subprocess that are spawned from the session:

# for bash or zsh
[parent] $ export FOO=one

# for fish
[parent] $ set --export FOO one

# then spawn a subsession and access the variable
[parent] $ bash
[child] $ echo $FOO
one

What about Ansible

If you are running Ansible to orchestration your servers you might ask your self what kind of session that is and what files you should change to manipulate the environment Ansible uses on the target servers. While you could go down that road, a much simpler approach is to use the environment keyword in Ansible:

- name: Manipulating environment in Ansible
  hosts: my_hosts

  # play level environment
  environment:
    FOO: one
    BAR: two

  tasks:

    # task level environment
    - name: My task
      environment:
        FOO: uno
      some_module: ...

This can be combined with vars, environment: "{{ my_environment }}", allowing you to use group vars or host vars to adapt the environment for different servers and scenarios.

Conclusion

The environment in Linux is a complex beast but armed with the knowledge above you should be able to tame it and harvest its powers for your own benefit. The environment is populated by different files depending on the kind of session and shell used. You can temporarily set variable from a one shot command, or for the remaining duration of the session. To make subshells inherit a variable use the export keyword/flag.

Lastly, if you are using Ansible you should really look into the environment keyword before you start to experiment with the different profile and rc-files on the target system.

If you’re using netstat you’re doing it wrong – an ss tutorial for oldies

Become a modern master with some serious ss skills

If you are still using netstat you are doing it wrong. Netstat was replaced by ss many moons ago and it’s long overdue to throw out the old and learn how to get the same result but in a whole new way. Because we all love to learn stuff just for the fun of it, right.

But seriously, ss is way better than nestat because it talks to the kernel directly via Netlink and can thus give you much more info than the old netstat ever could. So to help old folks like me transition from netstat to ss I’ll give you a translation table to port you over. But first, in case there are some newcomers whom isn’t encumbered with old baggage I’ll quickly describe a few common tasks you can do in ss.

Check open ports that someone is listening to

One of my most common use cases is to see if my process is up and running and listening to connections, or if there’s is something listening to a port I wanna know who it is. To do this use the flags --listening to get sessions with the LISTEN state, --processes to get the process that is listening, and to clean up we use --numeric since I never remember that sunrpc means port 111:

$ ss --listening --tcp --numeric --processes
State     Recv-Q  Send-Q  Local Address:Port    Peer Address:Port                                                                                    
LISTEN    0       128     0.0.0.0:111           0.0.0.0:*                                                                                       
LISTEN    0       128     127.0.0.1:27060       0.0.0.0:*        users:(("steam",pid=29811,fd=45))                                              
LISTEN    0       10      0.0.0.0:57621         0.0.0.0:*        users:(("spotify",pid=11223,fd=106))                                           
LISTEN    0       32      192.168.122.1:53      0.0.0.0:*                                                                                       
LISTEN    0       128     0.0.0.0:22            0.0.0.0:*                                                                                       
LISTEN    0       5       127.0.0.1:631         0.0.0.0:*                                                                                       
LISTEN    0       128     0.0.0.0:17500         0.0.0.0:*        users:(("dropbox",pid=13706,fd=98))                                            
LISTEN    0       128     0.0.0.0:27036         0.0.0.0:*        users:(("steam",pid=29811,fd=82))                                              
LISTEN    0       128     127.0.0.1:57343       0.0.0.0:*        users:(("steam",pid=29811,fd=39))

Check active connections

Checking just active sessions is easy. Just type ss. If you want to filter and show only TCP connection use the --tcp flag like so:

$ ss --tcp
State        Recv-Q   Send-Q   Local Address:Port     Peer Address:Port     
ESTAB        0        0        192.168.1.102:57044    162.125.18.133:https    
ESTAB        0        0        192.168.1.102:34008    104.16.3.35:https    
CLOSE-WAIT   32       0        192.168.1.102:52008    162.125.70.7:https

The same goes for UDP and the --udp flag.

Get a summary

Instead of listing individual sessions you can also get a nice summary of all sessions by using the --summary flag:

$ ss --summary
Total: 1625
TCP:   77 (estab 40, closed 12, orphaned 0, timewait 6)

Transport Total     IP        IPv6
RAW       0         0         0        
UDP       33        29        4        
TCP       65        59        6        
INET      98        88        10       
FRAG      0         0         0

Translation table going from netstat to ss

Lastly, as promised here is a nice table to help you transition. Believe me, it’s quite easy to remember.

netstat -ass
netstat -auss -u
netstat -ap | grep sshss -p | grep ssh
netstat -lss -l
netstat -lpnss -lpn
netstat -rip route
netstat -gip maddr

Automatic testing of Docker containers

So we are all building and packaging our stuff inside containers. That’s great! Containers lets us focus more on configuration and less on installation. We can scale stuff on Kubernetes. We can run stuff on everything from macOS to CentOS. In short, containers opens up a ton of opportunities for deployment and operations. Great.

If you are using containers you are probably also aware of continuous integration and how important it is to use automatic tests of all your new code. But do you test your containers, or just the code inside it?

At Basalt we don’t make applications. We make systems. We put together a bunch of third party applications and integrate them together. Most of it run inside containers. So for us it is important to test these containers, to make sure they behave correctly, before we push them to the registry.

Use encapsulation

So the approach that we chose was to encapsulate the container we wanted to test inside a second test container. This is easily done by using the FROM directive when building the test container.

Screenshot from 2019-03-30 09-26-04

The test container installs additional test tools and copies the test code into the container. We chose to use InSpec as the test framework for this, but any other framework, or just plain Bash, works just as well.

Testing Nginx with InSpec

So let’s make an example test of a container. In this example I will build a very simple web service using the Nginx container. Then I will use InSpec to verify that the container works properly.

Let’s start by creating all files and folders:

$ mkdir -p mycontainer mycontainer_test/specs
$ touch mycontainer/Dockerfile \
    mycontainer_test/{Dockerfile,specs/web.rb}
$ tree .
.
├── mycontainer
│   └── Dockerfile
├── mycontainer_test
│   ├── Dockerfile
│   └── specs
│       └── web.rb
└── test_container

4 directories, 3 files

Then add the following content to mycontainer/Dockerfile:

FROM nginx
RUN echo "Hello, friend" > \
  /usr/share/nginx/html/index.html

Now we can build the app container:

$ docker build -t mycontainer mycontainer/.
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM nginx
 ---> 2bcb04bdb83f
Step 2/2 : RUN echo Hello > /usr/share/nginx/html/index.html
 ---> Running in 7af1cec318f9
Removing intermediate container 7af1cec318f9
 ---> fe25cbbf80f9
Successfully built fe25cbbf80f9
Successfully tagged mycontainer:latest

$ docker run -d --name hello -p 8080:80 mycontainer
cfd3c3ea70c3512c5a7cc9ac2c9d74244aec4dd1a4bb68645e671cbe551af4ab

$ curl localhost:8080
Hello, friend

$ docker rm -f hello
hello

Great! So the container build and manual testing shows that it works. Next step is to build the test container. Add the following content to mycontainer_test/Dockerfile:

FROM mycontainer
ARG INSPEC_VERSION=3.7.11

# install inspec
RUN apt-get update \
 && apt-get -y install curl procps \
 && curl -o /tmp/inspec.deb \
    https://packages.chef.io/files/stable/inspec/${INSPEC_VERSION}/ubuntu/18.04/inspec_${INSPEC_VERSION}-1_amd64.deb \
 && dpkg -i /tmp/inspec.deb \
 && rm /tmp/inspec.deb

# copy specs
COPY spec /spec

Next we write our tests. Add the following to mycontainer_test/specs/web.rb:

name = 'nginx: master process nginx -g daemon off;'
describe processes(name) do
  it { should exist }
  its('users') { should eq ['root'] }
end

describe processes('nginx: worker process') do
  it { should exist }
  its('users') { should eq ['nginx'] }
end

describe http('http://localhost') do
  its('status') { should eq 200 }
  its('body') { should eq "Hello, friend\n" }
end

Now we can build and run our test container:

$ docker build -t mycontainer:test mycontainer_test/.
Sending build context to Docker daemon  4.608kB

  ... snip ...

Successfully built 6b270e36447a
Successfully tagged mycontainer:test

$ docker run -d --name hello_test mycontainer:test
e3f0e4a06efa416167d5d30458785bf66975d7837d9fc2b04634bb8291bc5679

$ docker exec hello_test inspec exec /specs

Profile: tests from /specs (tests from .specs)
Version: (not specified)
Target:  local://

  Processes nginx: master process nginx -g daemon off;
     ✔  should exist
     ✔  users should eq ["root"]
  Processes nginx: worker process
     ✔  should exist
     ✔  users should eq ["nginx"]
  HTTP GET on http://localhost
     ✔  status should eq 200
     ✔  body should eq "Hello, friend\n"

Test Summary: 6 successful, 0 failures, 0 skipped

$ docker rm -f hello_test
hello_test

And that’s it! We have successfully built an app container and tested it using InSpec. If you want to use another version of InSpec you can specify it as an argument when building the test container:

$ docker build -t mycontainer:test \
    --build-arg INSPEC_VERSION=1.2.3 mycontainer_test/.

Happy testing!