A Simple Guide to Scraping IMDB Data Using Selenium & Store into SQL

Overview

In this project, I’ll walk you through how to scrape movie data from the IMDb website, store it in an SQL database, and finally, visualize it using Python’s Matplotlib library.

Part 1: Scraping IMDb with Selenium

We’ll start by using Selenium, a powerful tool that allows us to automate web browsing. With Selenium, we can extract data like movie titles, ratings, and release dates directly from IMDb. Instead of manually going through each movie, Selenium will do the job for us!

Part 2: Storing Data in SQL Database

Once we have the data, the next step is to store it in an SQL database. This way, we can easily manage and retrieve the information anytime we need it. SQL databases are perfect for handling large sets of structured data, and I’ll show you how to insert our scraped movie data directly into a database.

Part 3: Visualizing with Matplotlib

After storing the data, we’ll use Matplotlib to create stunning visualizations. Whether it’s showing the top 10 rated movies or analyzing trends over time, matplotlib will help us turn our raw data into meaningful graphs and charts.

Part 4:  Displaying Data into graph Using PowerBI

In the final step, we’ll use PowerBI to bring our analysis to life. After scraping and storing the data, and visualizing some trends with Matplotlib, we’ll now leverage PowerBI's advanced features to create interactive dashboards. By plotting data into various charts such as column, donut, stacked area, and decomposition tree, we can further analyze TV shows and movies, highlighting patterns in gross earnings, rating distributions, and ranking shifts. PowerBI will allow us to dynamically explore these insights with ease.

By the end of this project, you’ll know how to automate data collection, store it efficiently, and create visuals that make your data come to life. Let’s get started!

Technologies used in this project:

Selenium:

• I used Selenium for automating the browser to scrape data from websites like IMDb.
• It helped me navigate web pages, extract data using specific selectors, and interact with various web elements.

WebDriver Manager:

• To manage the browser driver, I used WebDriver Manager, which automatically downloads and configures the correct version of Chrome-Driver.
• This eliminated the need for me to manually handle browser drivers.

pyodbc:

• I used pyodbc to connect my project to an SQL database.
• It allowed me to store and retrieve the scraped movie data in a structured format within the database.

Matplotlib:

• For visualizing the data, I used Matplotlib to create various types of graphs, such as bar charts and histograms.

• This helped me represent the movie data graphically for better understanding and analysis.

Pandas (pd):

• I relied on Pandas to manipulate and analyze the scraped data efficiently.

• With Pandas, I could clean, transform, and prepare the data both for visualization and storage in the SQL database.

Part 1 - Scraping IMDB using Selenium

What is Selenium?

Selenium uses the Web-driver protocol to automate processes on various popular browsers such as Firefox, Chrome, and Safari. This automation can be carried out locally (for purposes such as testing a web page) or remotely (for purposes such as web scraping).

Lets Set Up the Pre-Requisites:

Here’s a step-by-step guide on how to use Selenium with the example being extracting IMDB Data.

Step 1 — Install and Imports

pip install selenium webdriver-manager pyodbc

Once installed, you’re ready for the imports.

import time
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from webdriver_manager.chrome import ChromeDriverManager
import pyodbc

Step 2 — Install and Access WebDriver

A WebDriver is essential for automating browser actions. It opens your browser and interacts with websites. For Chrome users, you'll need the ChromeDriver. You can download it from https://developer.chrome.com/docs/chromedriver/downloads based on your browser version. To find your Chrome version, click the three dots in the top-right corner of the browser, go to Help > About Google Chrome. This will show your version.

×

You’ll need to know where the WebDriver is saved on your computer. By default, it’s often in your Downloads folder. Once you locate it, create a driver variable that points to the file path where the WebDriver is stored.

driver = webdriver.Chrome('/Users/MyUsername/Downloads/chromedriver')

Note: If you haven't installed ChromeDriver, you can do it automatically in your script using

service = Service(ChromeDriverManager().install())

This command automatically downloads and installs the correct version of ChromeDriver for your Chrome browser. It saves time and ensures you always have the right version without worrying about manual updates.

Note: When scraping websites like IMDb, they often block bots from accessing their content. To bypass this, we can make our script appear more like a regular browser by adding a User-Agent header. This header mimics a real user browsing the site. For example, the line:

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36'
}

Step 3 — Access the IMDB Website with Python

This is a simple yet essential step in the web scraping process. To begin scraping, you need your Python code to open the IMDB website (or any site you want to scrape). Using Selenium, the browser will automatically navigate to the desired webpage, allowing you to interact with and extract data from it. With the proper configuration, your script will simulate a regular browser session, which is crucial for accessing dynamic websites like IMDB.

driver.get(Website_Url_you_want_to_scrape)

When run, this code snippet will open the browser to your desired website.

Let's just start our project:

First of all, Navigate to IMDb's Top Movies Page. The first thing you'll need is the URL of the IMDb page you want to scrape. In this case, we will scrape movie data from IMDb's Top 250 Movies page.

To get the correct URL:

• Open your browser.

• Visit the IMDb website (www.imdb.com).

Navigate to the "Top 250 Movies" chart page.

Set Up Web Scraping Configuration

The image below showcases the current IMDb Top 250 Movies list. This list ranks the highest-rated movies of all time based on user ratings. The following Python script will be used to scrape this data programmatically, making it easier to analyze and gather movie details directly from IMDb.

This step defines the function scrape_imdb_top_250(): Movies and sets up a Chrome WebDriver using Selenium and WebDriver Manager. The user-agent header is added to mimic a real browser visit to avoid getting blocked by IMDb. 

def scrape_imdb_top_250():
    url = 'https://www.imdb.com/chart/top/'
    
    # Set up the Chrome WebDriver using WebDriver Manager
    options = webdriver.ChromeOptions()
    options.add_argument(f"user-agent={headers['User-Agent']}")
    service = Service(ChromeDriverManager().install())
    driver = webdriver.Chrome(service=service, options=options)

    driver.get(url)
    time.sleep(3)

The driver.get(url) method instructs the Chrome browser to navigate to the IMDb page. We also add a short delay (time.sleep(3)) to ensure that the entire page content is loaded before we attempt to scrape any data.

Scrape Movie Data

Now that the page is loaded, we'll locate and extract the relevant movie data like Title, IMDb rating, Year, Time, Runtime Rating, Vote Count, Description, Poster URL, Stars and the Director.

we use Selenium to grab all the movie items using their class name. This creates a list of movies to loop through and extract details from.

# Find all movies listed in the Top 250 table
movies = driver.find_elements(By.CLASS_NAME, 'ipc-metadata-list-summary-item__c')
movie_list = []

Extract Movie Details

In this step, we extract the movie title. The script checks if there is a period (e.g., "1. The Shawshank Redemption") in the title and removes the number before it. If no period exists, the full title is used as is.

As you can see the Title is included a numeric value so we will Scrape the name only and divide/Split it into two parts and then we'll get only textual data.

1. Movie Title

  for movie in movies:
        # Extract title
        title_column = movie.find_element(By.CLASS_NAME, 'ipc-title__text')
        full_title = title_column.text.strip()

        # Check if the title contains a period before splitting
        if '.' in full_title:
            title = full_title.split('.', 1)[1].strip()
        else:
            title = full_title  # If no period, use the entire title
  

2. Extract Year of Release

year = movie.find_element(By.XPATH, './/span[contains(@class, "sc-b189961a-8")]').text.strip() if movie.find_elements(By.XPATH, './/span[contains(@class, "sc-b189961a-8")]') else None
year = int(year) if year else None

Here, we extract the year of the movie's release using the XPath. If the year is available, it's converted into an integer. Otherwise, it's set as None.

3. Extract IMDb Rating

imdb_rating = movie.find_element(By.CLASS_NAME, 'ipc-rating-star--rating').text.strip()
imdb_rating = float(imdb_rating) if imdb_rating else None

In this step, the script finds the movie’s IMDb rating using the class name for the rating element. If a rating is present, it is converted to a float; otherwise, it returns None.

4. Extract Runtime and Genre

# Extract Time
MovieTime = movie.find_element(By.XPATH, './/span[contains(@class, "sc-b189961a-8") and (contains(text(), "h") or contains(text(), "m"))]').text.strip() if movie.find_elements(By.XPATH, './/span[contains(@class, "sc-b189961a-8") and (contains(text(), "h") or contains(text(), "m"))]') else 'N/A'

• To Find the Runtime Genre

# Genre (Movies ratings like PG, G, etc.)
runtimeGen = movie.find_element(By.XPATH, './/span[contains(@class, "sc-b189961a-8") and (text()="Approved" or text()="Passed" or text()="PG" or text()="G" or text()="R" or text()="PG-13" or text()="NC-17" or text()="Not Rated")]').text.strip() if movie.find_elements(By.XPATH, './/span[contains(@class, "sc-b189961a-8") and (text()="Approved" or text()="Passed" or text()="PG" or text()="G" or text()="R" or text()="PG-13" or text()="NC-17" or text()="Not Rated")]') else 'N/A'

The script extracts the movie’s Time (in hours and minutes) and its Runtime Rating (such as PG, R, etc.). If any information is missing, it sets the value as 'N/A'.

5. Extract Rating Count

rating_count = movie.find_element(By.CLASS_NAME, 'ipc-rating-star--voteCount').text.strip() if movie.find_elements(By.CLASS_NAME, 'ipc-rating-star--voteCount') else 'N/A'

 In this step, the script scrapes the number of votes (rating count). If either element is missing, it assigns 'N/A'.

6. Extract Description

In the above Screenshot, the pointing arrow it means when you click on that then you will see the Description, Director and Stars Data (The very First image of Top 250 Movies see there is no such data). So First we have to click that Button then it will tell Selenium what to do now. Otherwise it won't scrape that data.

How we will do that add the below code into the first function that we created scraped_imdb_top_250(): 

  # Click the button to trigger the detailed view
    try:
        # Locate the button by ID and click it
        button = driver.find_element(By.ID, 'list-view-option-detailed')
        button.click()
        time.sleep(2)  # Wait for the page to load the detailed view
    except Exception as e:
        print(f"Error clicking the button: {e}")

7. Extract Director and Starring Cast

director = movie.find_element(By.XPATH, './/span/a[@class="ipc-link ipc-link--base dli-director-item"]').text.strip() if movie.find_element(By.XPATH, './/span/a[@class="ipc-link ipc-link--base dli-director-item"]') else 'N/A'

stars = [star.text.strip() for star in movie.find_elements(By.XPATH, './/span/a[@class="ipc-link ipc-link--base dli-cast-item"]')]

This step scrapes the director’s name and the list of starring actors for each movie using the XPath for each element. The stars are returned as a list of names.

8. Extract the Movie Poster URL

image_element = movie.find_element(By.XPATH, './/div/img[@class="ipc-image"]')
image_url = image_element.get_attribute('src')

Finally, the script grabs the URL of the movie poster by extracting the src attribute of the image tag.

9. Append Data to the Movie List

    movie_list.append({
    'Title': title,
    'Year': year,
    'IMDb Rating': imdb_rating,
    'Movie Time': MovieTime,
    'Genre': runtimeGen,
    'Rating Count': rating_count,
    'Description': description,
    'Director': director,
    'Stars': ", ".join(stars),
    'Image URL': image_url
    })
    

Once all the details for a movie are scraped, they are stored in a dictionary and appended to the movie_list. This list will contain all the movie data once the script completes its run.

10. Close the WebDriver

driver.quit()
return movie_list

Part 2 - Storing Data in SQL Database

Setting Up the Database Connection

In this step, the insert_into_database function is defined. Inside the function, a connection is created to a SQL Server using pyodbc. The connection string uses the "ODBC Driver 17 for SQL Server," and specifies your server name (Your_Server_Name\SQLEXPRESS), the database name (Your_DataBase_Name), and Windows Authentication (Trusted_Connection=yes).

# Function to insert data into SQL Server
def insert_into_database(movies):
    # SQL Server connection setup with Windows Authentication
    connection = pyodbc.connect(
        'DRIVER={ODBC Driver 17 for SQL Server};'
        'SERVER= Add_Your_SQL_ServerName;'  # Add your SQL server name
        'DATABASE=Add_Database_Name;'  # SQL your Database name
        'Trusted_Connection=yes;'
    )

When connecting to a SQL Server, there are typically two ways to authenticate:

i) SQL Server Authentication: This method requires you to provide a specific username and password to access the server. It's useful when you want to create custom login credentials for different users.

ii) Windows Authentication: SQL Server uses your existing Windows credentials to authenticate, so you don't need to manually enter a username or password. This method is especially convenient if you don't remember your SQL Server credentials, or if you're working in a local environment where your Windows account already has the necessary permissions.

Creating the Cursor and Checking for Table

The cursor is used to execute SQL commands. This step checks if a table called IMDB_Top_250_Movies exists in your database.

If it doesn't, the table is created with the following fields:

id: Auto-incrementing primary key.
Title: Movie title (text, required).
Year: Release year (integer).
MovieTime: Movie duration (e.g., '2h 14min').
Runtime: Movie rating (e.g., 'PG', 'R').
IMDB_Rating: IMDb rating (e.g., 8.5). 
Rating Count: Number of votes (e.g., '1.2M').
Description: Short movie description.
Director: Name of the director.
Stars: Main stars, stored as a comma-separated string.
ImageURL: URL of the movie poster image.

cursor = connection.cursor()

# Create table if it doesn't exist
cursor.execute(''' 
    IF NOT EXISTS (SELECT * FROM sysobjects WHERE name='IMDB_Top_250_Movies' AND xtype='U')
    BEGIN
        CREATE TABLE IMDB_Top_250_Movies (
            id INT IDENTITY(1,1) PRIMARY KEY,  -- Auto-incrementing primary key
            Title NVARCHAR(255) NOT NULL,      -- Title of the movie
            Year INT,                           -- Year of release
            MovieTime NVARCHAR(10),             -- Time
            Runtime NVARCHAR(10),               -- Runtime (e.g., R, PG, NC-17, etc.)
            IMDB_Rating FLOAT,                  -- IMDb rating
            Rating_Count NVARCHAR(20),          -- Rating count (e.g., "2.9M")
            Description TEXT,                   -- Description of the movie
            Director VARCHAR(255),              -- Director of the movie
            Stars VARCHAR(255),                 -- Comma-separated list of stars
            ImageURL NVARCHAR(255)              -- URL of the movie image
        )
    END
''')

Inserting Data into the Table

The below loop iterates through each movie in the movies list (which is passed to the insert_into_database function). For each movie, it inserts the movie's details (like title, year, runtime, rating, etc.) into the database. Placeholders (?) are used for values, and the corresponding movie data (like movie['Title'], movie['Year'], etc.) is passed in the correct order.

# Insert data into the table
for movie in movies:
    cursor.execute('''
        INSERT INTO IMDB_Top_250_Movies (Title, Year, MovieTime, Runtime, IMDB_Rating, Rating_Count, Description, Director, Stars, ImageURL)
        VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
    ''', 
    movie['Title'], 
    movie['Year'], 
    movie['Time'],
    movie['Runtime'],  
    movie['IMDb Rating'], 
    movie['Rating Count'], 
    movie['Description'], 
    movie['Director'], 
    movie['Stars'], 
    movie['Image URL']
    )

Committing Changes and Closing the Connection

Once all movie data has been inserted into the database, connection.commit() is called to save the changes. Then, the cursor and database connection are closed.

connection.commit()
cursor.close()
connection.close()

Main Function - Running the Scraper and Inserting Data into SQL Server

 In this final step, we define the main entry point of our Python script. Here's how it works:

Scraping the Data: The script calls the scrape_imdb_top_250() function to collect data from IMDb's Top 250 movies list. If successful, it prints the number of movies scraped.

Inserting Data into SQL Server: If movies were successfully scraped, the script proceeds to call the insert_into_database() function, inserting the scraped movie details into the SQL Server database.

Error Handling: If no data is scraped (in case of an error or failure), the script will notify you by printing "No data scraped."

# Main function
if __name__ == '__main__':
    # Scrape the data
    movies = scrape_imdb_top_250()
    
    if movies:
        print(f"Scraped {len(movies)} movies successfully!")
        
        # Insert the scraped data into SQL Server
        insert_into_database(movies)
    else:
        print("No data scraped.")

Conclusion

By following the steps outlined in this blog, you’ve learned how to scrape movie data from IMDb's Top 250 list and store that data in a SQL Server database using Python. This process includes setting up a web scraper with Selenium, handling the extracted data, and inserting it into a database using Windows Authentication for seamless integration.

Whether you're working on a personal project or a professional application, this approach can be adapted to various scenarios, from simple data collection to more complex web scraping projects.

In this Screenshot, There is a pop-up sort by that include if you want to see the movies in any order. To scrape that just replace the URL and add the specific order by URL you'll choose.

Here is an example URL of an Alphabetical Order: 

url = 'https://www.imdb.com/chart/top/?sort=alpha%2Casc' 

If you would like to see the complete code without explanations, click the link below:

Click Me to get the code!

Part 3 - Visualizing with Matplotlib

Once you've scraped and stored your data, you can take it a step further by visualizing IMDb's Top 250 movies using Matplotlib. Visualizations can help make your data more engaging and easier to analyze.

Check out the next part of this series where I demonstrate how to use Matplotlib to visualize the movie data, including rating distributions, release years, and more.

Click here for the Matplotlib visualization Blog! 

Click Me to see the blog! 

Part 4 -  Displaying Data into graph Using PowerBI

In this section, we will explore how to visualize entertainment industry data using PowerBI. By plotting data into various charts such as column, donut, stacked area, and decomposition tree, we can gain deeper insights into TV shows and movies. This step demonstrates how data-driven visuals can uncover trends in genre popularity, gross earnings, IMDb ratings, and runtime comparisons.

1. Clustered Column Chart: Displaying TV Shows ID by Genre and TV Rating

This chart breaks down TV shows by genre and their respective TV ratings, helping us understand which genres are popular among different audience demographics.

TV Mini-Series shows tend to be rated more frequently in the TV-MA and TV-14 categories, indicating that mature audiences are more drawn to this genre.

TV Series is more evenly spread across ratings like TV-G, TV-PG, and TV-Y7, making it more suitable for broader audiences, including younger viewers.

2. Donut Chart: Count of ID, Title, and Rating Count by Total Gross and Weekly Gross

This donut chart visualizes movies by their total and weekly gross earnings, accompanied by the number of ratings each title has received.

Transformers One dominates the weekly gross with $9.2M, while Deadpool & Wolverine leads in total gross with $631M.

The inner section of the chart helps compare total gross distributions, while the outer section focuses on weekly performance, showing the variance in earnings for different movies.

3. Stacked Area Chart: IMDb Rating by Title

The stacked area chart offers a comparison of IMDb ratings for various movie titles.

12 Angry Men holds the highest IMDb rating of 9.0, followed by classics like The Godfather and Schindler's List.

The general trend shows a slight decline in ratings, with many popular titles landing in the 8.6 - 9.0 range, indicating consistent quality among top-rated films.

4. Stacked Column Chart: IMDb Actor Rankings by Movie

The graph showcases a comparison of IMDb actor rankings across various movies using a stacked column format. Each bar represents a specific movie, with the rankings categorized into "High" and "Low" values. The left panel provides additional details such as the ranking category, ranking value, the name of the movie, and the actor associated with the ranking.

The chart gives a clear visual representation of how actors performed in their respective movies based on IMDb ratings, allowing users to easily distinguish between high and low rankings.

Full Source Code on GitHub - Check This Out!

For a more detailed understanding of how the code functions and its structure, check out the complete project on my GitHub repository here. The repository provides comprehensive insights into the code, with clear documentation to help you navigate through the various features and components. Whether you want to understand the logic behind certain functionalities or you're interested in customizing the project for your own needs, you'll find all the resources necessary to get started. Feel free to download, experiment with, and run the project at your convenience.

Your feedback and contributions are always welcome!