REBECA Practice: Data Scientist

As we are working in the marketing department of a movie production firm, our problem is to maximize the revenue of our film productions. To solve this, your team needs consequently acquire the necessary data. These are the first two steps in the workflow of a data scientist team: defining the problem and acquiring data which allows to solve it.

NOTE: we have streamlined the process for acquisition for you, and you only need to import the data.

ACTION: Import the vega_datasets library, a repository of example data sets. Load the movies data from the library and have a first look at the features. Use the initial Problem “How can we achieve the largest revenue?” and reformulate it to define a more specific problem which you want to solve. Which variable do you choose as a target, which as features? Make an initial guess: Which variables have the greatest effect on your target variable?

TIP FORM THE EXPERT: Acquiring the data can be a long and cumbersome problem and often the availability of data poses constraints in the exact and specific definition of the problem. In this respect, acquiring the data and fine-tuning the problem statement is an iterative process.

SOLUTION 1

After acquiring (here: loading) the data, you need to clean and prepare the data set for further processing. Experience shows that this crucial step in the data analysis workflow can be the most challenging one.

Please, note that not all observations are complete, many of them do not contain information about all features, sometimes it appears a “None” or “NaN”. This missing data can affect your data analysis and later modeling, so you need to find a strategy to deal with it.

You discuss with your team how to proceed: one simple option would be to exclude observations with missing features from further analysis, but you need to check if this strategy will end up deleting many of the observations if most of them are incomplete. Another option would be to exclude features instead of observations, but none or only very few features are contained in all observations, so again you could end up deleting too much information. This means you must compromise: Select few entries with more attributes or more entries with fewer attributes.

NOTE: Creating a model later usually requires numerical data as input. This means other data formats, such as strings, need to be mapped to numeric values. For simplicity in this exercise, we restrict our further analysis to the numeric data type features of the data set.

ACTION: Clean and prepare the data by selecting the columns you want to analyze and drop incomplete entries following the comprise strategy. Limit yourself to numeric columns. Play around with selecting fewer or more (numeric) columns and remove missing observations. Balance the use of as many features as possible while dropping only as few entries as needed. During all of this, keep in mind your specific problem which you are planning to solve.
Once you are happy with your cleaning and preparation of the data set you can move on to the next task.

TIP FROM THE EXPERT: Alternative methods to deal with missing data exist. For example, you could infer the missing information, e.g., set to the mean. We refrain from these more complex ways to deal with missing data here though, but bear in mind that they also exist.

NOTE: Remember! have you documented all this?

SOLUTION 2

Now that you have cleaned and prepared your data set. Before continuing, you can have a first look and understand the data better. Get a first feeling of its statistics, i.e., distributions of features and correlations between distinct features.

ACTION: Calculate the mean and standard deviation of each of the features using the Dataframe.describe() in Pandas.

• Calculate all correlation coefficient using the Dataframe.corr() method in Pandas.
• Finally, create a histogram of the features of interest and scatter plots of the correlation using the Seaborn.pairplot() function.

SOLUTION 3

To make predictions about the target (which is unknown) of an observation based on its features, it is needed to create a model. This model could range from a simple linear fit to a complex deep neural network. The product owner of the project has assigned this task to the new junior Machine Learning Expert in your team.

You received the code snippet, which is used to model the data form your colleague. He claims that a Linear Regression is sufficient to model this data. Before you assemble the mode to the whole pipeline, you need to review the code and the produced result.

ACTION: Before you incorporate the model, review the code and the produced result. Check the code for bugs. If there is any, draft an email explaining to your college the error, and what you are going to do to solve it.

TIP FROM THE EXPERT: not only code is reviewed among members of a team, but also code style. In companies, there are style guidelines that are mandatory to be follow. Your team will also review how the code compiles the company’s style guidelines. This is a regular procedure in companies, where mistakes are part of the norm and team, not individuals, resolve them.

SOLUTION 4

Now that your team has created a model you can make predictions about events that are not contained in the data set or even lie in the future.

ACTION: Let’s assume a putative movie production of your stakeholder with a production budget of 0.5B USD, an IMDB Rating of 6.8 and 30000 IMDB votes.

1. What is the expected Worldwide Gross?
2. How would a 10% increase in each one of the three variables Production Budget, IMDB Rating, IMDB Votes affect the Worldwide Gross?

SOLUTION 5

Now you need you to evaluate and interpret your findings. Out of the python code and numbers you want to draw conclusions, which can be understood by your stakeholders and help them in their decision making.

ACTION: Look at your model and the prediction. Try to draw conclusions and formulate a take home message. Make a statement like

• “If we increase the production budget by 10%, we expect an increase of the revenue by …”,
• “If we can achieve an increase in IMDB rating by 10%, we expect an increase of the revenue by …” or “The return of interest is not dependent on variable X, but strongly depends on variable Y”.

Now you just need to sell it to your stakeholders! The Global and Strategy Department likes visuals!

ACTION: Create an appealing figure which your team will use together with your previously formulated key take home message. Remember, the visualization has the main goal to support your message.

The “Quality Control Department” wants to check the whole process. Have you documented all?

TIP FROM THE EXPERT: The standard tool for documentation and version control of (Python) code is Git. Github is a famous provider and platform for Git-based version control.

END