Home | Schedule & Readings | Links & Resources

Radical Software

LCST 2234, Fall 2021 (CRN 9430)
Rory Solomon

Project 1, Tutorial 4: Scheduling and publishing

Today we're going to talk about some techniques for publishing and deploying a software project, which you'll use to package up and submit Project 1.

More specifically, we'll talk about:

1. (1) Controlling the schedule on which your bot will run, using a Unix tool called cron, and
2. (2) More git commands to tag a particular version of your code files, and to push your code changes to your repository on GitHub.

Documenting your project

Let's start with some documentation. At the top of your main.py, add a block comment using three double quotes that includes some identifying information like your name, our class, the date, and the project number. Please also include whether you are using python or python3 in your development:

"""
  First Last
  Radical Software, Fall 2021
  Project 1
  Sept 23, 2021
  python (or python3)
"""

Next, working in Atom, please add some comments to README.md. It has become a common convention (if not standard practice) on GitHub to include a file with this name in the root directory of your project.

This is a place to put a short description of your project that someone browsing on GitHub would see first. Modify the description that I included with your code as you wish. I described a bot inspired by Christopher Strachey's love letter generator — that may or may not be the case for you at this point in your project work, so feel free to change.

This .md file uses a kind of formatting called Markdown: "a lightweight markup language for creating formatted text using a plain-text editor". (Incidentally, this project was co-created by John Gruber and a person named Aaron Swartz, whom we'll be talking about later in the semester.) I don't want to spend time explaining how to use Markdown, and I don't wish to impose any expectations on you to make use of it, but it is quite easy to work with, so if you are curious you can read more about it here:

guides.github.com/features/mastering-markdown
(Scroll down to the "Syntax guide" for a concise "cheat sheet".)

You can add your real name here if you wish but you do not need to as I have record of everyone's GitHub username at this point. Keep in mind that this will be publicly visible on GitHub. And all student projects that were created as forks of my original code are available here:

github.com/Radical-Software-Fall-2021/project-1/network/members

If you don't wish to have this project publicly visible on your GitHub account, that is fine, but I ask that you at least leave it this way until we get through this project, so you get some practice submitting with GitHub, and I can collect your work. After that you can delete this repository if you wish.

Deploying your bot

The over-arching paradigm of how a bot likes this works is that the code you have written will be uploaded onto a server, where it will be run at some periodic interval that you can define.

A server is just a computer, like your laptop but typically housed in a data center or some facility where it can be always on and always connected to a network. Server machines will typically not have any sort of screen and are usually maintained by system administrators who login to them remotely for maintenance and configuration.

Most servers will have many computer programs running on them simultaneously to do various things such as keep the time, serve webpages, perform logging, automated backups, and other tasks. The technical term for a computer program running independently on a computer (server, laptop, or mobile device) is a process. When a process runs autonomously on a server and does tasks, it is commonly referred to as a daemon (usually pronounced DAY-mon, but also as DEE-mon), a technical term used to signify a process that is running in the background, not being invoked by a user directly.

Your laptop has many daemons running on it as well. There are daemons that run continuously to generate the indexes that let you search for files, that create system logs, that handle keyboard and mouse events, and other things. Generally, a process whose name ends in "d" signifies a daemon. If you open your system's Acvity Monitor (Mac) or Task Manager (Windows), you can see all the processes running on your computer — some of which will have been invoked by you (for example when you double-click on Atom) others are daemons run automatically by your operating system.

So in a way, to deploy your bot we are going to run it as a kind of daemon on a server that I will set up. You will submit your code to GitHub, tell me how to schedule it, and I will pull it out of GitHub onto our class server and configure it to be run.

The important thing to keep in mind here is that your bot will not be running continuously, nor will it be able to be somehow "triggered" in response to any events or user actions. It is not the case that everytime someone mentions you, your bot will automatically be invoked. Rather, your bot will be run based on some fixed periodic interval that you will schedule. Each time your bot runs, the process will begin and end in a flash as quickly as it happens when you run your bot on the command line.

It can be tricky to think about your bot in this way. Twitter does not send out alerts everytime some event happens. Instead, it is up to developers like you to schedule your code to run automatically, and then check on what's happened since the last time your code ran.

The cron system

The common way that people schedule processes to run like this is with a Unix tool called cron, the name being a reference to "chronos", the Greek word for time.

If you'd like to read more about the cron system, I can actually recommend the Wikipedia page on the topic, which is quite informative. This hosting company also has a helpful blog post with some explanation that I found useful.

Technically, your code will not run as a daemon, but rather, there is one cron daemon running on our server, which will be configured to invoke your code at a specified interval.

The cron system is comprised of a crontab file, which lists a number of individual cronjobs. This is what a crontab file might look like:

The lines that start with # are comments & documentation. The seven lines towards the bottom are a list of cronjobs.

The first few characters of each of those lines is a scheduling string: some specially formatted characters that define the periodic interval at which this job will run. The rest of each line specifies the command that will be run (in this case, a command called docker run, in your case it will be python app/main.py) and what to do after (in this case, send an email to the developer and CC me).

For your project, I am asking you to craft a scheduling string that defines how often you want your bot to run.

In putting together scheduling strings, I find this tool to be extremely useful (and kind of fun):

crontab.guru

As that site indicates, there are five parts to a scheduling string:

• minute
• hour
• day of month
• month
• day of week

Specifying a star, *, indicates "every". So all stars means: every minute of every hour of every day of every month of the year. That's a bit too much for our case!

Let's think through an example:

0 12 * * 2,4

This is specifying: the zero minute of the twelve twelfth hour of every day of every month, but only the second and fourth days of the week, i.e. Tuesday and Thursday. So this would be the scheduling string for our class.

As I've done here, you can use commas to specify a list of values. As crontab.guru explains, you can also use dashes to indicate a range of values. So 0 8-10 in the first and second position would indicate the first minute of the hour at 8am, 9am, and 10am. You can also use a slash / to specify intervals. So */15 8-10 would mean every 15 minutes starting at 8am through the end of the 10th hour (so until 10:59am).

How about another example:

*/15 17-22 * * 0-4

Every 15 minutes from 5-10pm, Sunday through Thursday. So like a schoolnight bot.

TO DO: Work out a scheduling string that works conceptually for the project that you are developing, and add this to the class bot spreadsheet, along with a plain English description of what you expect this string to do.

I will assemble all of these into a crontab file on our shared server.

Submitting your work (more source code management)

Last week we talked about some git techniques that you can use for better source code management: git add to add files to git and tell it which files you're currently working on, git commit to save all your recent edits to your local repository, and git status to see the current state of your local code files (whether any are changed and need to be committed).

Today I want to talk about how you can take your local repository, after you have added a bunch of code changes to it, tag it as a stable version, and transfer your version of your repository up to the GitHub server.

First let's see what you have going on with your local files:

$ git status
On branch main
Your branch is up-to-date with 'origin/main'.
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   README.md
	modified:   app/main.py

The second line of output will look a little different for you. It will probably say something like Your branch is ahead of 'origin/main' by X commits, depending on how many commits you made last week in class or since.

The list of modified files will also look different for everyone, depending on what you've been working on since the last tutorial. But you should definitely have some modified files. Regardless of how complete you think your work is at the moment, let's do some git practice.

Whenever you have "Changes not staged for commit", and you plan to commit those changes to your local repository or push them to a git server (like GitHub), you need to first add those files. This is like putting together a packet of files to deliver. First you assemble the packet, then you send it off.

So type git add FILENAME for each new or modified file that you wish to commit. After running this command, you probably won't see any output in response.

After that you can type git status again and you should see something like:

$ git status
On branch main
Your branch is up-to-date with 'origin/main'.
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

	modified:   README.md
	modified:   app/main.py

(Again, instead of up-to-date you'll probably see ahead ... by X commits.)

Now commit your changes like you did last week:

$ git commit -m"Add your own informative commit message here"
[main 44dc962] a sample commit of changes
 2 files changed, 3 insertions(+)

Replace the blue text with whatever informative note-to-self you wish to use. The last line will be different depending on how you have modified your files. This is telling you some stats about the changes that you have just committed.

Note the seven characters in brackets. These are the commit ID, known as a SHA-1 hash. This is actually a kind of shortcut nickname for a longer 40-character ID which uniquely identifies this particular commit. The seven character shortcut is usually unique enough. There are more complicated git operations that you might do later that would require you to use this.

Now let's look at all commits that have been made to your repository:

$ git log

This will show you a list of all the commits you have made, preceded by all the commits that I made to this code when I was creating the repository that you forked when you started your project.

Let's look for these commits on GitHub.com. Visit the site, and under "Repositories" on the left, click on your project-1 repository. Now, under the green button that says "Code", click on the word "commits". This should show you a similar view as git log did above.

But your newer commits are missing! Why? Because you have made these commits locally. When you started your project by running git clone, you created a copy of the entire repository that exists locally, only on your computer. GitHub stores a version of the same repository on its server. But if you want your local commits to be visible on GitHub (for example, to share your code with me or with potential collaborators) you have to update the server with your version of the repository. You do that with the git push command. If you try this now, you'll likely get an error message:

$ git push 
Username for 'https://github.com': YOUR-USERNAME
Password for 'https://YOUR-USERNAME@github.com': 
remote: Support for password authentication was removed on August 13, 2021.
Please use a personal access token instead.
remote: Please see https://github.blog/2020-12-15-token-authentication-requirements-for-git-operations/
for more information.
	  fatal: Authentication failed for 'https://github.com/YOUR-USERNAME/project-1.git/'

You should hopefully see a very helpful & informative error message like this. When you try to push, you are transfering code changes that you made locally up into your repository on the GitHub site, so naturally GiutHub wants to authenticate you to make sure that you are in fact a person who has access to modify this repository. You can try entering your GitHub username and password here, but you'll get this error message beseching you instead to use a "personal access token." This is a security measure GitHub has taken.

To do this, click here and follow the instructions under "Creating a token" to create a personal access token. For the "Note" you can enter whatever you'd like. For "Expiration" you can select 90 days. For "Scopes" I recommend checking the "repo" box, which will let you do everything with your repository. Once you have that, copy it to your clipboard, try git push again. Enter your username and when prompted for a password, paste the token. It won't look like you are entering any text, but that is OK. Just try pasting the token and pressing enter.

$ git push 
Username for 'https://github.com': YOUR-USERNAME
Password for 'https://YOUR-USERNAME@github.com': 
Counting objects: 4, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (4/4), 360 bytes | 0 bytes/s, done.
Total 4 (delta 2), reused 0 (delta 0)
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
To https://github.com/YOUR-USERNAME/project-1.git
   44dc962..788e75d  main -> main

(If you're still having problems and you've done any work with git before, you may need to delete github.com keys from your KeyChain. You can see instructions for how to do that here.)

Now if you return to GitHub.com, and click on "commits", you should see a log of repository commits that matches what you see when you run git log. Congratulations, you have just pushed a set of source code changes to a public repository.

Creating tags

The last thing that I'd like you to do to submit this project is somehow label your code as complete when you are ready to do so. We'll do this with a git mechanism called a tag.

Git tags are like user-friendly commit IDs. They are a way to label a particular version of all your files in a way that is easier for a person to remember than a long SHA-1 hash.

Tags are used to create things like the official releases for widely used software projects. For example, after a handful of bugs have been fixed, a development team may decide to release a new version. They would probably create a tag that specifies the version number, like v2.42.8, and then share that in different places as a way of indicating that this particular commit of changes is a stable, usable release of this software.

I would like you to submit your project by creating a tag. Doing so is very simple. You just type: git tag followed by the name of your tag. For now, let's just do a test. The text in blue is a message and can be whatever you'd like:

git tag test

You shouldn't see any output in response. But after you do this, you'll need to push the tag to GitHub if you want me or anyone to see it. Do git push, but use the --tags parameter. You should see something like this:

$ git push --tags
Total 0 (delta 0), reused 0 (delta 0)
To https://github.com/YOUR-USERNAME/project-1.git
 * [new tag]         test -> test

Now if you return to your repository in GitHub, to the left of the green "Code" button, you should see an icon that says "1 tag". Click that, and it will show you the tag you've just created with options for someone to download this code as if it was a release version of your project.

TO DO: When you are ready to submit your work for this project, please create a tag called v1.0 and push it to GitHub. This is what I will look for as the submitted version of project 1.

Concluding

Remember to create a Lab Notebook for each tutorial including this one. For today, include some commentary about your experiences working through all these items.

Reminder: Make sure to create your own Twitter developer at account to create a bot of your own. This process can take several days, so if you have not started yet, you need to start as soon as possible.

Remember that you can find help in this process here: Twitter setup "mini" tutorial. In particular, the verbiage from my email exchange with Twitter may be helpful - especially the last one.

And add your bot name to our class spreadsheet: add a row to this spreadsheet. Seven people have added their names already. Nice work!