The Twitter streaming API for those without firehose access is still useful and interesting, you just need to get your feet wet.
Because you should love Python, today we’ll focus on how to mine tweets from the Twitter streaming API using Python and Tweepy. Before we begin, you need to create a new application on Twitter and get your API keys and secrets ready to roll.
Got them? Good, let’s continue.
Oh also, I’d like to assume you’re using virtualenv, if not, no worries… but please, you’re going to ruin your life.
So let’s setup our dumb little development environment.
|
1 2 3 4 5 |
~# mkdir twitter_miner && cd twitter_miner ~/twitter_miner# virtualenv venv ~/twitter_miner# echo "Hurray\!" ~/twitter_miner# . venv/bin/activate (venv)~/twitter_miner# pip install tweepy |
Alrighty, so now we have Tweepy setup and we’re just about ready to get down to brass tacks. We’re going to sucking down 1% of the Twitter feed via the sample streaming API… while that may not sound like a lot, it does add up, so let’s use sqlite to handle the task.
Really though, the complication pretty much ends there. Tweepy is an amazing library that makes the next part pretty easy.
Before we get into our Python code below, let’s quickly create a table to hold all of the information we want to keep by entering our Python interpreter.
|
1 2 3 4 5 6 7 8 9 10 |
~# venv/bin/python Python 2.6.8 (unknown, Sep 17 2012, 03:13:50) [GCC 4.6.2 20111027 (Red Hat 4.6.2-1)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import sqlite3 >>> conn = sqlite3.connect('tweets.db') >>> curs = conn.cursor() >>> curs.execute("CREATE TABLE tweets (tid integer, username text, created_at text, content text, reply_to text, coordinates text, source text)") <sqlite3.Cursor object at 0x7fabc9f63e48> >>> |
Okay, cool. Now we have a healthy place to store our tweets. Feel free to tweak that one to your desired data you want to capture, just be sure to modify the code below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 |
#!/usr/bin/env python # -*- coding: utf-8 -*- # # twitter client import tweepy # database interface import sqlite3 conn = sqlite3.connect('tweets.db') curs = conn.cursor() class StreamWatcherHandler(tweepy.StreamListener): """ Handles all incoming tweets as discrete tweet objects. """ def on_status(self, status): """Called when status (tweet) object received. See the following link for more information: https://github.com/tweepy/tweepy/blob/master/tweepy/models.py """ try: tid = status.id_str usr = status.author.screen_name.strip() txt = status.text.strip() in_reply_to = status.in_reply_to_status_id coord = status.coordinates src = status.source.strip() cat = status.created_at # Now that we have our tweet information, let's stow it away in our # sqlite database curs.execute("insert into tweets (tid, username, created_at, \ content, reply_to, coordinates, source) \ values(?, ?, ?, ?, ?, ?, ?)", \ (tid, usr, cat, txt, in_reply_to, coord, src)) conn.commit() except Exception as e: # Most errors we're going to see relate to the handling of UTF-8 messages (sorry) print(e) def on_error(self, status_code): print('An error has occured! Status code = %s' % status_code) return True def main(): # establish stream consumer_key = "CONSUMER_KEY" consumer_secret = "CONSUMER_SECRET" auth1 = tweepy.auth.OAuthHandler(consumer_key, consumer_secret) access_token = "ACCESS_TOKEN" access_token_secret = "ACCESS_TOKEN_SECRET" auth1.set_access_token(access_token, access_token_secret) print "Establishing stream...", stream = tweepy.Stream(auth1, StreamWatcherHandler(), timeout=None) print "Done" # Start pulling our sample streaming API from Twitter to be handled by StreamWatcherHandler stream.sample() if __name__ == '__main__': try: main() except KeyboardInterrupt: print "Disconnecting from database... ", conn.commit() conn.close() print "Done" |
So, let’s pretend we put that code into a file called tweet_gobbler.py and said okay, let’s go!
|
1 |
~# venv/bin/python tweet_gobbler.py |
That’s it! Now whenever we run this script, whilst it runs, it will continue to update the sqlite database tweets.db and after a few days you will have tons and tons of tweets from all around the world. Lucky you!
I set this up on an Amazon EC2 micro instance and let it run for a few days and pulled down about 400MiB of tweets, so it’s not a bad way to build that awesome dataset you’ve been craving.
Now, go save the world.
Oh, also, you can hone in on specific users or tweets if you like using the streaming API filter functionality. Maybe more on that later, stick around.