Home | Schedule & Readings | Links & Resources

Radical Software

LCST 2234, Fall 2021 (CRN 9430)
Rory Solomon

Project 1, Tutorial 3: Search, mentions, replies, and other APIs

Today we're going to talk about how you can take advantage of some additional features that the Twitter API provides, features which are made available to you by virtue of being implemented by the tweepy library.

Remember, Twitter (as you know) is a website & platform that has a human-friendly user interface (what we might call a GUI), it also offers an interface for use by computer programs (an API). The functionality of that API requires a bunch of code in order to access, but thankfully, many developers have already written that code in various languages (Python, Javascript, Java, others) and provided them open source as freely-available libraries. The Python library that we are using, as you know, is tweepy.

Quiz question: How many different types of interfaces have we encountered this semester? Highlight here for the answer: Three: GUI (graphical user interface), API (application program interface), and CLI (command line interface).

Workspace cleanup & source code management

Before we get into the technical work of today, let's take a moment to rearrange our files a bit to make things a bit easier to work on.

Last week, I asked you to put all your new code into main.py as you worked through tutorial 2.

For today, let's start with a clean, fresh file so that we're all starting in the same place. But I don't want you to delete any of your work from last week! So let's start by renaming main.py to tutorial2.py. You can do this in Finder (select the file, then click on it again or press ENTER) or Windows Explorer. You can also do this in the command line with the mv command like so: (Make sure you are in your project-1 directory.) 

$ pwd
/Users/rory/Documents/Radical Software/Project 1/project-1
$ mv app/main.py app/tutorial2.py

The first line is not necessary. I included it only to show you how you might confirm that you're in the right place. And the output of that command will be different for each of you depending on where you've placed your Radical Software folder on your computer.

Now let's experiment with some source code management for a moment. This semester we are using git and GitHub to manage our source code files. Somewhat analagously to Twitter and tweepy, GitHub and git are two different things. GitHub is a website and platform for socially managing and sharing source code projects. git is an open source command line tool for managing code files. git has existed for years before GitHub came along, and GitHub was created in a way to be compatible with git. The git tool offers you many commands and features. Let's look at some now.

Type git status. You should see something like this: 

$ git status
On branch main
Your branch is up-to-date with 'origin/main'.
Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	deleted:    app/main.py

Untracked files:
  (use "git add <file>..." to include in what will be committed)

	app/tutorial2.py

no changes added to commit (use "git add" and/or "git commit -a")

git thinks that you deleted main.py because you renamed it. That's fine.

But we've created a new file. Let's add that to our source code repository: 

$ git add app/tutorial2.py

If you type git status again, you should now see something like: 

Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

	new file:   app/tutorial2.py

Let's commit this new file, which means it will be added to your git repository. This will help you keep track of different versions of this file, make it easier to share the file with others, to work on it collaboratively, and make it harder to accidentally delete.

To commit the file, type the following: 

git commit app/tutorial2.py -m"Adding new file for tutorial 2"

The text that I marked in blue is a message for you, a note-to-self, and it can be whatever you'd like. It should be short and concise, but also informative because you may look at it later.

Now what about that missing app/main.py file?

You've deleted it (actually renamed it) but because it is being managed by git, you can get it back. At least, you can get back the last version of the file that you committed, which is the version of the file that I originally gave you.

The git command to retrieve a missing file is checkout: 

$ git checkout -- app/main.py

Now you should have three source code files. Type ls app to confirm: 

$ ls app
authorization_tokens.py		main.py		tutorial2.py

For today, let's rename app/main.py again, this time to tutorial3.py. In Finder, Explorer, or: 

$ mv app/main.py app/tutorial3.py

So today we'll edit and run this file. And let's commit it to git at the end of the day.

So for now, I propose that you work through the tutorials in these files, which I hope will help you feel organized and less confused. As you work on your project by tweaking, customizing, and combining these techniques together, I suggest that you do that work in main.py.

Accessing additional API features

Let's talk about accessing some new features of the Twitter API.

  1. Option 5: Basic search. (I'm numbering these options as building on last week.)

    So remember that now we'll work in tutorial3.py, and that should look like this, back with the simple code that I started you off with:

    Today, we're going to work on the API, which means we'll need to use this variable called api. So go to the end of line 8, add some new lines, and we'll work there.

    Create the message variable again with an empty string, modify api.update_status() to use that, and start by adding an informative comment. (I'll use the tear edge to indicate that this is a code snippet: a piece of a larger file.) 

    	  
api = tweepy.API(auth)

message = ""

# Option 5: Basic search


api.update_status(message)

    To get a sense of what search functionality the Twitter API and the tweepy library offer us, have a look at the documentation for tweepy:

    We have already worked through the first few bullet points there. Today, let's jump down to click on "API reference", and search for "API.search". This explains how this command works and the many options available with it.

    Let's do a search! 

    
# Option 5: Basic search	  
search_results = api.search(q="Bezos filter:safe", lang="en")

    Here I'm using two tweepy named parameters: q and lang. q is a string that includes the search I'll pass on to Twitter, and lang is because we're at Eugene Lang College — just kidding it's because I'm indicating I only want English tweets in this case.

    Inside my q string I've specified that I only want "safe" tweets. This is a feature Twitter offers that allows you to filter out offensive content. I'm doing this because we're working together in a group today, but feel free to use this option on your own or in your project or not as you wish. But please keep in mind the Community Guidelines that we spoke about on the first day.

    Now let's access one tweet in the search results. The api.search() command returns a Python list, so we can use the same list manipulation techniques that we used last time, like random.choice() (but remember: you have to import random again at the top of your file, say, after line 2): 

    	  
# Option 5: Basic search	  
search_results = api.search(q="Bezos filter:safe", lang="en")
random_tweet = random.choice(search_results)

    We've created a new variable here called random_tweet. (This is a variable, and you can name it whatever you'd like.) This is an object that contains a tweet. An object is a term from Python (and many other programming languages) that means a variable that has many values (called properties) instead of one simple value. You can access these properties with dot notation, using a period. Let's examine this tweet object, but don't try to post a tweet yet. Comment out the api.update_status(message) line, and print some values: 

    
# Option 5: Basic search	  
search_results = api.search(q="Bezos filter:safe", lang="en")
random_tweet = random.choice(search_results)
print( dir(random_tweet) )

    Save your code and run it. Remember we're working on tutorial3.py now. Also remember to use python3 if that is applicable to you. 

    $ python app/tutorial3.py

    Examine the results. I know it might feel like you're looking at the green text from The Matrix, but what do you see? These are all the properties (a.k.a. attributes) available to you through this one tweet object that we have called random_tweet (but you can call anything you'd like).

    One of these in particular is interesting to me and that is _json. JSON stands for Javascript Object Notation, and it is a way of representing objects in text. Let's use a Python library called PrettyPrint to examine this: 

    
# Option 5: Basic search	  
search_results = api.search(q="Bezos filter:safe", lang="en")
random_tweet = random.choice(search_results)
# print( dir(random_tweet) )
import pprint
pp = pprint.PrettyPrinter(indent=4)
pp.pprint(random_tweet._json)

    Run this again: 

    $ python app/tutorial3.py

    What do you see now? A somewhat more nicely displayed view of the above information. This also expands some of those properties and shows you what additional properties they themselves have! (Like turtles all the way down ...)

    The properties that we'll work with for this option are text and full_text. Let's access that and do some string manipulation on it. Let's also comment out all those lines that display text 

    
# Option 5: Basic search	  
search_results = api.search(q="Bezos filter:safe", lang="en")
random_tweet = random.choice(search_results)
# print( dir(random_tweet) )
# import pprint
# pp = pprint.PrettyPrinter(indent=4)
# pp.pprint(random_tweet._json)

text = random_tweet.text
message = text.replace("Bezos", "Bezos, who pays no taxes,")
print(message)

    Now run your code and see what you get. (By the way, this snarky text replacement is of course a big oversimplification of this billionaire and the US tax code, but if you'd like to read more here is a great article from Propublica.)

    It may not work precisely as you think, because of things like case sensitive string matching. One problem is that tweepy was designed back when tweets were only 140 characters. Twitter has since expanded them to 280. So by default, you will get only the first 140 characters of the tweets in your search results. If you'd like the full version, you can use the tweet_mode="extended" option in your search call, but then you have to use full_text atttribute instead of text. Like this: 

    
# Option 5: Basic search	  
search_results = api.search(q="Bezos filter:safe", lang="en", tweet_mode="extended")
random_tweet = random.choice(search_results)

text = random_tweet.full_text
message = text.replace("Bezos", "Bezos, who pays no taxes,")
print(message)

    We can talk later about case insensitive string replacement and other techniques to clean this up. If you get something that you like, uncomment api.update_status(message), and run your code again to post a tweet!

  2. Option 6: Mentions

    Let's comment out all that. Add a new comment for this option, and add our new code in betwewen # Option 6 and api.update_status(message): 

    	  
# Option 5: Basic search
...
# print(message)

# Option 6: Mentions



api.update_status(message)

    Accessing your mentions (people who have tagged you) is pretty easy. Check the documentation for API.mentions_timeline().

    You use that like this: 

    	    
# Option 6: Mentions
mentions = api.mentions_timeline()

    As with api.search(), this command gives you a Python list, so we can work on it using the same Python list and random techniques. Let's select a random mention from the list: 

    	  
# Option 6: Mentions
mentions = api.mentions_timeline()
mention_tweet = random.choice(mentions)

    Now the variable mention_tweet here has a different name than what I called random_tweet above, but they are both the same kind of object, so they both have the same properties. This time, we'll access the user and id properties. 

    	  
# Option 6: Mentions
mentions = api.mentions_timeline()
mention_tweet = random.choice(mentions)
thanks = " I'm a radical gratitude bot. Thank you for the mention."
message = "@" + mention_tweet.user.screen_name + thanks

    Here I'm using the + operator as string concatenation to create a reply tweet. I'm starting the tweet with "@" and the username of the person who mentioned me to tag them back.

    Now you can either print(message) and comment out api.update_status(message), or comment out print() and uncomment the update line, depending on whether you want to test by displaying the result to yourself, or trying to post a tweet.

    You may know that Twitter also has threaded replies. If you want this message to actually appear in a thread as a reply to your mention, you have to reference the id of the original tweet in your reply. Like this: 

    	  
# Option 6: Mentions
mentions = api.mentions_timeline()
mention_tweet = random.choice(mentions)
thanks = " I'm a radical gratitude bot. Thank you for the mention."
message = "@" + mention_tweet.user.screen_name + thanks

api.update_status(message, in_reply_to_status_id=mention_tweet.id)

  3. Option 7: Working with an additional external library

    For this last option of the day, I want to give you one example of how you might work with an additional external library. Again, remember, tweepy itself is an external library: it provides Python code to help you interact with the Twitter API. There are many (many!) other Python libraries out there that can help you access other platforms (access services over the network), but also libraries to let you access additional functionality and databases locally on your computer.

    Today, for the purposes of demonstration, I'm going to show you how to use a rhyming dictionary API called Pronouncing. You can read more about it here:

    First things first. To work with an external library, you have to install that code onto your system. Just like we did with tweepy. You can do that with pip (or pip3).

    You can install the Pronouncing library by itself, but I prefer a common convention that many people use which is to put all the necessary libraries for a project into the requirements.txt file, and use that with pip.

    To do that, edit requirements.txt in Atom and add the following line: 

    tweepy==3.9.0
pronouncing==0.2.0

    If you use other libraries in your project, add them all here.

    Then run the following command to install that: 

    $ pip install -r requirements.txt

    Check your output. If you get warnings, it's probably OK. If you get errors, we can look into it.

    Once you have that installed, let's go back to tutorial3.py. Comment out our work from option 6, and add a new comment below for option 7: 

    	  
# Option 6: Mentions
# mentions = api.mentions_timeline()
# mention_tweet = random.choice(mentions)
# thanks = " I'm a radical gratitude bot. Thank you for the mention."
# message = "@" + mention_tweet.user.screen_name + thanks

# Option 7: External API



api.update_status(message, in_reply_to_status_id=mention_tweet.id)

    Let's start by replicating some of the code from option 6 that accesses mentions: 

    	  
# Option 7: External API
mentions = api.mentions_timeline()
mention_tweet = random.choice(mentions)

    Next, I'm going to use a really convenient string manipulation technique that Python offers, which is common to most programming languages: split(). This command takes a string, breaks it up by spaces or punctuation, and returns a list of words: 

    	  
# Option 7: External API
mentions = api.mentions_timeline()
mention_tweet = random.choice(mentions)
mention_tweet_words = mention_tweet.text.split()

    Now, mention_tweet_words contains a Python list comprised of all the words in the tweet that mentions me. (And again, this is not a special keyword, but just a variable that I created that you can call anything you'd like.)

    Let's select a random word from that list: 

    	  
# Option 7: External API
mentions = api.mentions_timeline()
mention_tweet = random.choice(mentions)
mention_tweet_words = mention_tweet.text.split()
word = random.choice(mention_tweet_words)

    And now let's use this word with our rhyming dictionary API: 

    	  
# Option 7: External API
mentions = api.mentions_timeline()
mention_tweet = random.choice(mentions)
mention_tweet_words = mention_tweet.text.split()
word = random.choice(mention_tweet_words)
rhyming_word_list = pronouncing.rhymes(word)
rhyme_word = random.choice(rhyming_word_list)

    The first line returns a list of rhyming words (print it out if you're curious!) and the second line selects one random word from that list.

    What should we do with it? Let's go back to some string template techniques from last week: 

    	  
# Option 7: External API
mentions = api.mentions_timeline()
mention_tweet = random.choice(mentions)
mention_tweet_words = mention_tweet.text.split()
word = random.choice(mention_tweet_words)
rhyming_word_list = pronouncing.rhymes(word)
rhyme_word = random.choice(rhyming_word_list)

template = "You say {}, I say {}."
message = template.format(word,rhyme_word)

    Now you can either print(message) to see what your code produces, or, if you like what it's doing, you can call api.update_status(message) and post the tweet!

    And again, if you want this to appear as a threaded reply to the mention, you need to start with "@" and the username who mentioned you, and add in_reply_to_status_id=mention_tweet.id in api.update_status(): 

    	  
# Option 7: External API
mentions = api.mentions_timeline()
mention_tweet = random.choice(mentions)
mention_tweet_words = mention_tweet.text.split()
word = random.choice(mention_tweet_words)
rhyming_word_list = pronouncing.rhymes(word)
rhyme_word = random.choice(rhyming_word_list)

template = "You say {}, I say {}."
message = template.format(word,rhyme_word)

message = "@" + message
api.update_status(message, in_reply_to_status_id=mention_tweet.id)

    The point here is to show you how to access an external library, not how to use a rhyming dictionary, per se. Rhyming is fun and cool, but it is up to you to determine if this is something that you wish to use in your project, or whether you'd like to take this as example of how to work with external libraries.

    You can find Python libraries simply by doing online research, checking projects like the NYC.gov open data platform, or asking me.

For your homework, as last week, make sure you complete each of these options and add your reflections about them. Which make sense? Which can you see as having potential for radical software, and how? Pick one of these options and try to exapand and embellish it.

Lastly, have a look at this Twitter setup "mini" tutorial, which talks about how you can create your own Twitter developer account in order to make your own Twitter bot.