PDF Text Processing with Python

In a previous Medium article, I worked through the process of using the Python library Tabula to extract tables from PDF documents and convert them into an Excel sheet. But PDFs don’t only contain tables, they also often contain large amounts of text. Therefore, it is also useful to know how to extract text from PDFs and convert said text into a Word document.

Following the theme of my last post, I’m going to use another PDF focused on Indonesia’s current energy situation with the Indonesia Energy Outlook 2019 Report published by the Secretariat General of the National Energy Council.

If you open the link to the PDF you will find a long report with many pages and figures. For the purpose of this post, I am only going to focus on extracting the text from the Executive Summary on pages xii and xiii.

With the PDF and text identified let’s move on to using python to extract the Executive Summary.

Note: The following code explanation is designed for the Google colab environment.

Our Python Code: Extracting the text

The library we will use to extract the PDF text is called PyPDF2. PyPDF2 can do much more than just extract text and if you are curious about its other capabilities, you can read about them here. This library isn’t pre-installed in the Google colab environment so we will have to install it before importing the PyPDF2 into our code.

!pip install PyPDF2

import PyPDF2

Before we move to the next step make sure you have loaded the PDF document into the file repository on the left of the colab environment. We must save the PDF as an object before we can start using PyPDF2 on it.

pdf_file_obj = open("/content/content-indonesia-energy-outlook-2019-english-version.pdf","rb")

Now we need to convert pdf_file_obj into a PyPDF2 object so that we can use the library to search through the Indonesia Energy Outlook to extract our text of interest.

# Converting the object into a PDF Reader Object

pdf_reader = PyPDF2.PdfFileReader(pdf_file_obj)

# If you want to find out the number of pages in the PDF use this   # command

print(pdf_reader.numPages)

We know from looking at the original PDF that we are interested in pages 12 and 13 where the Executive Summary resides. We can pull out an individual page using the following method.

# How to create a page objec

page_obj = pdf_reader.getPage(12)

If you print the page_obj you will get something quite unreadable to the human eye.

So we need to go a step further to actually get to the text.

# Extracting text from a page object

print(page_obj.extractText())

Much better!

Now let’s pull all the text from pages 12 and 13 and combine them to get the executive summary.

# Getting Executive Summary

page_obj1 = pdf_reader.getPage(12)

page_obj2 = pdf_reader.getPage(13)

executive_summary = page_obj1.extractText() + page_obj2.extractText()

You’ll notice that the text has many instances of “\n” within it when you print it out. We can remove this with a simple one-liner.

clean_text = executive_summary.replace("\n","")

Our Python Code: Making our word document

We need one more library now so that we can create our word document. The library we will be using is called python-docx. First, we install and import it into our environment.

!pip install python-docx

from docx import Document

With the necessary library installed we must first create an empty document object and then build that empty object by doing the following steps.

# Create Document object

document = Document()

# Add a heading to our word document

document.add_heading('Executive Summary', 0)

# Create a paragraph by feeding our document the extracted text

p = document.add_paragraph(clean_text)

All we need to do now is save our document and it will appear in our file repository on the right side of the Google colab environment.

document.save("Executive Summary.docx")

The document won’t look perfect and there will likely be a few minor cleanups to do, but you should have all the text from the executive summary.

I hope you can make use of this!