Version control repositories like CVS, Subversion or Git can be a real gold mine for software developers. They contain every change to the source code including the date (the "when"), the responsible developer (the "who"), as well as little message that describes the intention (the "what") of a change.
In this notebook, we will analyze the evolution of a very famous open-source project – the Linux kernel. The Linux kernel is the heart of some Linux distributions like Debian, Ubuntu or CentOS.
We get some first insights into the work of the development efforts by
Linus Torvalds, the (spoiler alert!) main contributor to the Linux kernel (and also the creator of Git), created a mirror of the Linux repository on GitHub. It contains the complete history of kernel development for the last 13 years.
For our analysis, we will use a Git log file with the following content:
# Printing the content of git_log_excerpt.csv # ... YOUR CODE FOR TASK 1 ... log = open('datasets/git_log_excerpt.csv', 'r') print(log.read())
1502382966#Linus Torvalds 1501368308#Max Gurtovoy 1501625560#James Smart 1501625559#James Smart 1500568442#Martin Wilck 1502273719#Xin Long 1502278684#Nikolay Borisov 1502238384#Girish Moodalbail 1502228709#Florian Fainelli 1502223836#Jon Paul Maloy
The dataset was created by using the command
git log --encoding=latin-1 --pretty="%at#%aN". The
latin-1 encoded text output was saved in a header-less csv file. In this file, each row is a commit entry with the following information:
timestamp: the time of the commit as a UNIX timestamp in seconds since 1970-01-01 00:00:00 (Git log placeholder "
author: the name of the author that performed the commit (Git log placeholder "
The columns are separated by the number sign
#. The complete dataset is in the
datasets/ directory. It is a
gz-compressed csv file named
# Loading in the pandas module # ... YOUR CODE FOR TASK 2 ... import pandas as pd # Reading in the log file git_log = pd.read_csv('datasets/git_log.gz', sep='#', encoding='latin-1', header=None, names=['timestamp', 'author']) # Printing out the first 5 rows # ... YOUR CODE FOR TASK 2 ... print(git_log.head(5))
timestamp author 0 1502826583 Linus Torvalds 1 1501749089 Adrian Hunter 2 1501749088 Adrian Hunter 3 1501882480 Kees Cook 4 1497271395 Rob Clark
The dataset contains the information about every single code contribution (a "commit") to the Linux kernel over the last 13 years. We'll first take a look at the number of authors and their commits to the repository.
# calculating number of commits number_of_commits = git_log.shape # calculating number of authors number_of_authors = len(git_log.dropna().author.unique()) # printing out the results print("%s authors committed %s code changes." % (number_of_authors, number_of_commits))
17385 authors committed 699071 code changes.
There are some very important people that changed the Linux kernel very often. To see if there are any bottlenecks, we take a look at the TOP 10 authors with the most commits.
# Identifying the top 10 authors top_10_authors = git_log.groupby(['author']).agg('count').sort_values(by=['timestamp'], ascending=False)[0:10] # Listing contents of 'top_10_authors' top_10_authors
|David S. Miller||9106|
|H Hartley Sweeten||5938|
|Mauro Carvalho Chehab||5204|
For our analysis, we want to visualize the contributions over time. For this, we use the information in the
timestamp column to create a time series-based column.
# converting the timestamp column # ... YOUR CODE FOR TASK 5 ... git_log.timestamp = pd.to_datetime(git_log['timestamp'], unit='s') # summarizing the converted timestamp column # ... YOUR CODE FOR TASK 5 ... git_log.timestamp.describe()
count 699071 unique 668448 top 2008-09-04 05:30:19 freq 99 first 1970-01-01 00:00:01 last 2037-04-25 08:08:26 Name: timestamp, dtype: object
As we can see from the results above, some contributors had their operating system's time incorrectly set when they committed to the repository. We'll clean up the
timestamp column by dropping the rows with the incorrect timestamps.
# determining the first real commit timestamp from_2005_to_2018 = (git_log.timestamp > '2005-04-16 22:20:00') & (git_log.timestamp <= '2018-02-17 00:00:00') valid_timestamps = git_log[from_2005_to_2018].sort_values(by=['timestamp']).timestamp valid_timestamps = valid_timestamps.reset_index() first_commit_timestamp = valid_timestamps.timestamp # determining the last sensible commit timestamp last_commit_timestamp = valid_timestamps.timestamp[len(valid_timestamps)-1] # filtering out wrong timestamps corrected_log = git_log[(git_log.timestamp >= first_commit_timestamp) & (git_log.timestamp <= last_commit_timestamp)] # summarizing the corrected timestamp column # ... YOUR CODE FOR TASK 6 ... corrected_log.describe()
|top||2008-09-04 05:30:19||Linus Torvalds|
To find out how the development activity has increased over time, we'll group the commits by year and count them up.
# Counting the no. commits per year # commits_per_year = corrected_log.groupby(corrected_log.timestamp.dt.year).agg('count') commits_per_year = corrected_log.groupby(pd.Grouper(key='timestamp', freq='AS')).agg('count') # Listing the first rows # ... YOUR CODE FOR TASK 7 ... print(commits_per_year.head())
author timestamp 2005-01-01 16229 2006-01-01 29255 2007-01-01 33759 2008-01-01 48847 2009-01-01 52572
Finally, we'll make a plot out of these counts to better see how the development effort on Linux has increased over the the last few years.
# Setting up plotting in Jupyter notebooks %matplotlib inline # plot the data # ... YOUR CODE FOR TASK 8 ... commits_per_year.plot(title='Linux commits evolution', kind='bar', legend='off')
<matplotlib.axes._subplots.AxesSubplot at 0x7f7b157f0b38>
Thanks to the solid foundation and caretaking of Linux Torvalds, many other developers are now able to contribute to the Linux kernel as well. There is no decrease of development activity at sight!
# calculating or setting the year with the most commits to Linux year_with_most_commits = commits_per_year.idxmax()