Twitch Clipper was a scraping bot I made to create 10 minute videos of highlights from twitch streams. I had the idea when I was watching a video of twitch clips put into a compilation and thought that it was such a simple task that I could probably automate it myself.
The task ended up having many interesting challenges, and became a great learning opportunity. Like many modern websites, Twitch generates much of the page dynamically, meaning it is impossible to simply use a GET request to fetch the page you want to visit and parse the data from there. The HTML from a simple GET request does not include all of the main content dynamically loaded. My solution was to use the selenium library to get all of the dynamic html content. This library uses a browser's webdriver to load dynamic content and interact with the website using scripts to target different elements on the page programatically.
I coded this project with a functional programming paradigm in mind since I find this easiest to visualize and maintain. This is a little difficult since the webdriver object had to be passed around, and obviously had its own state. The bulk of the code was parsing the metadata of the different pages and then using this data to determine which videos to download. This decision was made based on the number of views on the clips and the length of the clips. The clips were essentially chosen by popularity, and chosen until there was 10 minutes worth of clips to use.
Since on the twitch clips website the clips are stored as video files which can be downloaded directly, there was no need to use a stream based video downloader. The open source command line video editing tool FFMPEG was used to compile all of the videos together. FFMPEG is quite powerful and so I was also able to add titles to each of the clips. I simply took the title which the user who originally clipped the video used since this would add more context then I would be able to extract, and often also added its own comedy to the clip.
| Message: |
|---|