Code as a Liberal Art, Spring 2025

Unit 2, Lesson 1 — Thursday, March 6

Metadata and modular programming with UNIX

Table of contents for the topics today: (Click each to jump down to the corresponding section.)

The UNIX pipe comamnd
Creating a utility to use with pipe
Data and metadata

An example of a lenticular wall installation from a UK graffiti artist named dr.d. Source: moillusions.com

I. The UNIX pipe comamnd

In the reading this week, Tara McPherson discussed various ways that modularity is manifested in the UNIX operating system, focusing on the text-based command line interface and the pipe command which signified with the vertical bar character, |.

Let's experiment today with how that works.

To start, let's cd into a folder with a lot of files. For most of us, we can probably use our Documents, Desktop, or Downloads folder. Open a command line shell in VS Code, type cd (note the space), then find the folder you'd like to examine, drag it into the command line window, and press enter. For me, that produces this command:

$ cd /Users/rory/Desktop

By now we're probably all familiar with the ls command, which stands for "list", and displays all the files contained within the directory (folder) that your shell is presently "in".

UNIX commands often allow the user to modify their behavior by specifying flags, which are optional command line arguments that start with a hyphen: -. The ls command accepts many such flags that customize its behavior in various ways. Type man ls to see a full list of all of them. (man is short for manual.) Type the space bar to page through the manual pages, and type q to exit the manual.

Let's look at a few flags.

First let's try this one. (That's a lower case "L".)

ls -l

This displays a detailed list view that gives you the owner of the file, as well as whether or not its executable, and its last modified date.

Next let's try this. (That's the number one. Very visually confusing!)

ls -1

This flag displays the contents of the directory in a list with one filename per line.

Working with that last version, now we can see how we can string this command together with the pipe command |.

Try typing this:

$ ls -1 | grep jpg

That uses a UNIX utility called grep (for global regular expression print) which allows you to filter text input in many advanced ways. We're using a pretty simple filter here and just filtering only filenames that include jpg. There are many many more options we can use with grep and we could spend days or weeks just working with this one command, but let's put that aside for now and think about what is going on here.

Using the UNIX pipe command |, the operating system is taking the output of the ls command, and sending it directly as the input to the grep command. We the users never "see" this output, but it is passed on, or "piped" from one program to another. As McPherson explained, modularity manifests here in the way that the ls command is implemented completely independently of the grep command, and vice versa — grep "knows" nothing about where its input is coming from. But because these commands share common assumptions about inputs and outputs (namely, that inputs and outputs will be a stream of lines of text), they can exchange data in this way.

Let's see how we can chain additional commands together with more pipe experiments.

Try adding the following additional command to the chain:

$ ls -1 | grep jpg | sort

The UNIX sort command takes whatever input it receives, sorts it, and displays the results as output. By default this sorts by alphabetical order, but with various flags we can change that.

Depending on the files in your folder, you will probably see that uppercase files A-Z precede lowercase files a-z. We can modify that by adding -f (or synonymously, --ignore-case) which sorts the input alphabetically with upper and lower case interleaved, effectively ignoring case:

$ ls -1 | grep jpg | sort -f

We'll see some additional flags for sort later today.

For now, let's turn to thinking how we can create our own utilities that join in on this linked chain of inputs and outputs.

(jump back up to table of contents)

II. Creating a utility to use with pipe

a. Making an executable

Making it a little easier to execute our Python programs.

Type this in the command line:

$ which python3

Make a new folder named

Unit 2, Lesson
	1

for this week. Open a new window in VS Code, drag Unit 2, Lesson 1 into it. Make a new file. Let's call it executable.py. As the first line in your new Python file, add #!, then copy/paste the resulting output of the above command. For me, that's:

#!/Library/Frameworks/Python.framework/Versions/3.8/bin/python3

This command, known as hash bang notation, tells your operating system that if you are try to run this file, the file should be interpreted by the command that you have specified there, which is the full path to python3 on your system. Basically this means that now we can run this Python program just by invoking its name, which will be equivalent to typing python3 and then the name of the file.

To make this work we have to do one more little bit of file management. At the command line, type the following:

$ chmod a+x step1.py

Let's just add one more line to our Python file, for testing. I'll make a simple print() statement.

#!/Library/Frameworks/Python.framework/Versions/3.8/bin/python3

print("Hello")

Now you should be able to type ./executable.py from the command line and run your program.

(jump back up to table of contents)

b. Reading input from a pipe

Creating a program that can read input from a pipe command is easy. We have already seen many different inputs and outputs for our programs. Unit 1, Lesson 1 showed how you can use the input() and print() commands for interactive text input and output with the user in a command line interface, as well as how you can use open() with readline() to read input from a file, and open() with write() to write output to a file. And Unit 1, Lesson 3 showed how we can also use command line arguments as a form of user input.

Working with data from a pipe command as input is just as straightforward as these other techniques. The way it works is by reading from stdin, which is short for standard input. We treat stdin just like a regular file: we open it, and read lines of text from it, which we can then process. The only difference is that the actual file never exists! It comes as a form of output from a previous command.

Let's see how it works. Create a new file. Let's call it pipe.py. Type the following into it. Remember to replace the hash bang line with the full path to Python on your system.

#!<FULL PATH TO PYTHON ON YOUR SYSTEM>

import sys

print("Piped input:")
i = 0
for line in sys.stdin:
    
    print("Line",i,line)

    i = i + 1

This program is simply opening stdin to access piped input, and then displaying that line-by-line with a little counter to show how many lines there are.

Save that file and run it with ls -l | ./pipe.py.

If you accidentally run this without piping in any input, the program will wait for input from the user interactively. Try running the program this way and see what happens. You can type any text, line-by-line, and see it echoed back to you. Type CONTROL-D (Mac) to exit. (I believe the Windows command is CONTROL-Z.)

(jump back up to table of contents)

c. Batch processing images

Now let's make a command line utility that is a little more interesting. Looking back to Unit 1, Lesson 3 Homework part 2, let's make a command that can rotate images, but now working in batch mode from piped input.

Let's make a new file called batch_rotate.py

#!/Library/Frameworks/Python.framework/Versions/3.8/bin/python3

import sys
import os
from PIL import Image

for line in sys.stdin:
    
    # Remove the newline character \n from the end of the line
    filename = line.rstrip()
    
    img = Image.open(filename)

    # Modify the image
    rotated_img = img.rotate( 45 )

This code is similar to the previous example. It takes stdin, which it now presumes to be a list of filenames. The rstrip() command removes the newline character at the end of each line. Lines are indicated in standard input by being terminated with a new line character. In the above example we didn't care about this, which is why you saw the blank lines between each line in the output. But now, this would cause trouble, because filenames do not generally have newline characters within them. So we remove them with this handy Python utility.

Then, we treat each line as the filename of an image, open it with the Python Image Library, and rotate it by 45 degrees.

We haven't saved the images yet, so you won't see any ouput! If we just saved all these modified images in the current directory, we could clutter up our code files with a lot of image files. Let's prompt the user for a command line argument which will correspond to their preferred output directory. Then use that as the place to save the modified images, using os.path to join directory names to filenames:

#!/Library/Frameworks/Python.framework/Versions/3.8/bin/python3

import sys
import os
from PIL import Image

output_directory = sys.argv[1]

for line in sys.stdin:
    
    # Remove the newline character \n from the end of the line
    filename = line.rstrip()
    
    img = Image.open(filename)

    # Modify the image
    rotated_img = img.rotate( 45 )

    # Save the new, modified image
    original_filename = os.path.basename(filename)
    new_filename = "rotated_" + original_filename
    
    rotated_img.save( os.path.join(output_directory,new_filename) )

One other issue here is that the ls command only shows the filename, but depending on how you are using this, you will most likely want the complete path to the file that your code will be opening. For this purpose, you can use the find command. So to run this, make sure you have a folder called output in your current directory, and run it like this:

$ find people -type f | ./batch_rotate.py output

(The -type f flag tells find to only return files, not any folder names.)

If you wanted to run this on the original folder that we were experimenting with about (e.g. Downloads or Desktop) you would likely get an error when your batch_rotate.py program encounters any files that are not images. So we could add grep to the chain of pipe commands like this:

$ find people -type f | grep jpg | ./batch_rotate.py output

If you want to improve usability for your user, consider adding some error checking and helpful output, like this:

#!/Library/Frameworks/Python.framework/Versions/3.8/bin/python3

import sys
import os
from PIL import Image

if len(sys.argv) != 2:
    exit("""
This program requires one command line argument: the name of a directory to save images in
The directory must already exist.
""")

output_directory = sys.argv[1]

for line in sys.stdin:
    
    # Remove the newline character \n from the end of the line
    filename = line.rstrip()
    
    img = Image.open(filename)

    # Modify the image
    rotated_img = img.rotate( 45 )

    # Save the new, modified image    
    original_filename = os.path.basename(filename)
    new_filename = "rotated_" + original_filename
    
    rotated_img.save( os.path.join(output_directory,new_filename) )

And maybe also had a help message, which is common with command line utilities, displayed if the user enters a -h flag:

#!/Library/Frameworks/Python.framework/Versions/3.8/bin/python3

import sys
import os
from PIL import Image

if "-h" in sys.argv:
    exit("""
This program expects a list of image filenames.
They should be full paths, so use the find command.
And they should all be images, so use grep to filter the input.
""")

if len(sys.argv) != 2:
    exit("""
This program requires one command line argument: the name of a directory to save images in
The directory must already exist.
""")

output_directory = sys.argv[1]

for line in sys.stdin:
    
    # Remove the newline character \n from the end of the line
    filename = line.rstrip()
    
    img = Image.open(filename)

    # Modify the image
    rotated_img = img.rotate( 45 )

    # Save the new, modified image    
    original_filename = os.path.basename(filename)
    new_filename = "rotated_" + original_filename
    
    rotated_img.save( os.path.join(output_directory,new_filename) )

Next, let's look at how we can create a utility that can pipe output to another program — not just have data piped into it, as we just saw above.

(jump back up to table of contents)

III. Data and metadata

As an optional reading this week, consider taking a look at an article about metadata by Matthew Mayernik. In it, he explains that the term "metadata" likely originated in the late 1960s "in the context of computer system design to refer to the use of one data element to describe or represent some characteristic of another data element." [1] In other words, it's data about other data.

We might debate the usefullness of this distinction (after all, isn't it all, just, data?) and perhaps there is some ideological baggage that this comes loaded with in terms of computer scientists wanting to create this distinction - as if to say that as a field, it does not want to get its hands dirty mucking about with actual data itself, which might be the messy, poorly structured stuff of culture and humanistic communication. But it is a distinction that seems to have some usefulness, and while Mayernik does not offer any hard and fast definitions, he lists many contenders, such as "data attributes that describe, provide context, indicate the quality, or document other object (or data) characteristics." [2] Let's go with that ... and let's see if we can manage some successful metadata experiments that use these data attributes to provide context or analysis about some data objects.

The type of metadata that I'd like to start exploring is Exif data. This term stands for "exchangeable image file format", and it refers to a number of standardized fields that can be embedded in image files, currently only JPG and TIFF image formats. Exif data can include information about the camera device that was used to take the picture, including numerous photographic settings on that camera; it can also include information about the resolution and color properties of the image; and it can include contextual information like the date, time, and location of where and when the photo was taken.

As you are probably thinking, this can pose some serious privacy concerns.

One high profile case illustrating this is the story of John McAffee, who was on the run, and whose location was then revealed to be Guatamala after doing an interview with Vice media, who left GPS information embedded in Exif data on photographic images they took of him. (thenextweb.com)

Exif data can be very challenging to work with in a reliable way. First of all it is usually only included in photographic images, not other types of graphics or images created with other tools. Also, not all cameras save the same fields. And lastly, owing to the privacy concerns (see sidenote below) many platforms strip out some or all of the Exif data in images before sharing them on their platforms. So while most of the pictures you take with your phone probably have this embedded, trying to find images on the internet with these fields embedded can be frustrating.

Privacy? I would argue that concern for privacy is not necessarily the primary motivating concern for why these platforms strip Exif data from images before publishing them. For one, most of these platforms, such as Facebook, have proven themselves to be wholly unconcerned with the privacy of their users. On the contrary: their entire business model is predicated on extracting as much data about their users as possible. I contend that the performance of privacy concern regarding Exif data is actually a way of holding on to as much data as they can. After all, it is not the case that these companies ignore Exif data. Rather, they strip it out and save it for their own uses. Facebook and Google will show you where a photo was taken, even though they don't let you collect this data from files yourself. Removing it from the photo files that are published, I argue, is a way for them to prevent everyone from deriving value from this data and to maintain exclusive access to it.

For this work, I will work with a collection of 15 images. I saved these from Flickr, and I found them by doing a search for "people", sorted by "most interesting" (a property which Flickr determines by means that I'm not aware of) and filtered to only allow images with a Creative Commons license. Since Flickr is a site geared toward photography enthusiasts, they leave in much Exif metadata because it often includes technical information about the photograph (like equipment, shutter speed, etc). So most of these have Exif data, but not all. I plan to work with these images as examples over the next couple weeks. You can find a zip file containing the folder of images here.

Download the above zip file, unzip it, and move the resulting folder into your Unit 2, Lesson 1 folder for this week. Now, if you open a shell in VS Code, you should be able to type ls and see the people folder in the listing.

Let's start by playing around with Exif data in the Python shell.

>>> from PIL import Image, ExifTags
>>> img = Image.open("people/11767919503_335aa61249_o.jpg")
>>> img.getexif()
<PIL.Image.Exif object at 0x7fba16ed2a60>

These few commands simply import the libraries we need to work with this, opens an image, and gets the Exif data for that image. This Exif object is somethign like a dictionary, which is a data structure, like a list but one that we have not seen yet — I will explain this next week. For now, let's make a new file called exif.py and copy/paste the following code into it:

#!/Library/Frameworks/Python.framework/Versions/3.8/bin/python3

import sys
from os import listdir
from PIL import Image, ExifTags

filename = sys.argv[1]
img = Image.open( filename )

img_exif = img.getexif()
print("Printing Exif data for " + filename)
for key in img_exif.keys():

    print( ExifTags.TAGS[key], "is", img_exif[key])

This requires you to specify an image filename as a command line argument, and then displays all the Exif metadata for that file.

Here is an example:

$ ./exif.py people/11767919503_335aa61249_o.jpg
Printing Exif data for people/11767919503_335aa61249_o.jpg
ExifVersion is b'0230'
ShutterSpeedValue is 7.643856
ApertureValue is 4.970854
DateTimeOriginal is 2012:02:28 13:09:19
DateTimeDigitized is 2012:02:28 13:09:19
ExposureBiasValue is 0.0
MaxApertureValue is 4.0
...

The output continues on from there.

Running that command, now I can see the date / time for all images that contain this field, and my program simply skips over images that don't contain it.

Can you start to think about some things that we might do with this data? In the homework I'll ask you to try to sort the images based on date, or any other Exif field you wish.

In working with this Exif stuff, it is useful to have a non-programmatic tool to quickly access the Exif (meta)data. I can recommend this simple online tool: http://exif.regex.info/exif.cgi. I have been using it and it works well for me. There are many other ways you could access this. On Mac I believe you can use built-in tools such as clicking the file in Finder, and then clicking File > Get Info.

VI. Wrapping up & homework

Hopefully this gives you some building blocks in algorithmic thinking that you can play with. The homework for this week has some opportunities for you to experiment with these ideas. Have fun!