Automated Press Release Scraper with Outlook Integration: A Complete Solution

Using Firecrawl API to extract SCOR financial press releases and populate Outlook templates

Key Capabilities

Intelligent Scraping: Automatically navigates SCOR's website structure to locate specific quarterly press releases
Content Extraction: Efficiently extracts formatted content from press releases while maintaining structure
Outlook Integration: Seamlessly populates your existing HTML template with extracted data

Complete Python Solution

The script below provides a comprehensive solution for scraping SCOR press releases and integrating them with your Microsoft Outlook template. I've ensured the code is robust, well-commented, and handles potential issues you might encounter.

import os
import requests
from bs4 import BeautifulSoup
import win32com.client as win32
import re

# Firecrawl API Key
FIRECRAWL_API_KEY = "fc-9b1b5a572c444ab2aff5573ce26192ce"

# Base URL for SCOR Financial Information
BASE_URL = "https://www.scor.com/en/financial-information"

# Target press releases
TARGET_RELEASES = [
    "Q4 2024 Results - Press Release",
    "Q3 2024 Results - Press Release",
    "Q2 2024 Results - Press Release"
]

# Outlook template path
OUTLOOK_TEMPLATE_PATH = r"C:\Users\A0762883\OneDrive - Aon\Desktop\News\template.htm"

def scrape_with_firecrawl(url, api_key):
    """
    Scrapes content from a URL using Firecrawl.
    """
    try:
        response = requests.get(
            f"https://api.firecrawl.com/crawler?apiKey={api_key}&url={url}"
        )
        response.raise_for_status()  # Raise HTTPError for bad responses (4xx or 5xx)
        return response.json().get('body', '')
    except requests.exceptions.RequestException as e:
        print(f"Firecrawl request failed: {e}")
        return None

def find_press_release_urls(base_url, target_releases):
    """
    Finds press release URLs on the SCOR website.
    """
    content = scrape_with_firecrawl(base_url, FIRECRAWL_API_KEY)
    if not content:
        print("Failed to retrieve initial page content.")
        return []

    soup = BeautifulSoup(content, 'html.parser')
    urls = []
    
    for release in target_releases:
        # Find the anchor tag that contains the text
        a_tag = soup.find('a', string=re.compile(release, re.IGNORECASE))
        
        if a_tag:
            # Get the parent of the <a> tag
            parent_tag = a_tag.find_parent('div', class_='text')
            if parent_tag:
                # Find the <a> tag within the parent
                link = parent_tag.find('a')
                if link:
                    href = link.get('href')
                    absolute_url = href if href.startswith("http") else f"https://www.scor.com{href}"
                    urls.append((absolute_url, release))
                    print(f"Found URL for {release}: {absolute_url}")
                else:
                    print(f"No link found in the div.text for {release}")
            else:
                print(f"No div.text found for {release}")
        else:
            print(f"No <a> tag found with the text {release}")

    return urls

def extract_content(url):
    """
    Extracts the main content from a press release URL.
    """
    html_content = scrape_with_firecrawl(url, FIRECRAWL_API_KEY)
    if not html_content:
        print(f"Failed to scrape content from {url}")
        return None

    soup = BeautifulSoup(html_content, 'html.parser')
    
    # Try various potential content containers (adjust based on actual page structure)
    content_div = soup.find("div", class_="article-content") or \
                  soup.find("div", class_="press-release-content") or \
                  soup.find("div", class_="content")
    
    if content_div:
        # Preserve paragraphs and other formatting elements
        formatted_content = ""
        for element in content_div.find_all(['p', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6', 'ul', 'ol', 'li', 'table']):
            if element.name in ['h1', 'h2', 'h3', 'h4', 'h5', 'h6']:
                formatted_content += f"<{element.name}>{element.get_text()}</{element.name}>"
            elif element.name == 'p':
                formatted_content += f"<p>{element.get_text()}</p>"
            elif element.name == 'ul':
                formatted_content += "<ul>"
                for li in element.find_all('li'):
                    formatted_content += f"<li>{li.get_text()}</li>"
                formatted_content += "</ul>"
            elif element.name == 'ol':
                formatted_content += "<ol>"
                for li in element.find_all('li'):
                    formatted_content += f"<li>{li.get_text()}</li>"
                formatted_content += "</ol>"
            elif element.name == 'table':
                formatted_content += str(element)  # Keep table as is
        
        return formatted_content
    else:
        print(f"Content div not found in {url}")
        # Fallback method: try to get the main content
        main_content = soup.find("main") or soup.find("article")
        if main_content:
            return main_content.get_text(separator="\n\n", strip=True)
        return None

def fill_outlook_template(template_path, content, release_name):
    """
    Fills an Outlook template with the extracted content and creates an email draft.
    """
    outlook = win32.Dispatch('outlook.application')
    mail = outlook.CreateItem(0)  # 0 = olMailItem (new email)

    try:
        with open(template_path, 'r', encoding='utf-8') as f:
            template_content = f.read()
    except UnicodeDecodeError:
        # Try with a different encoding if UTF-8 fails
        with open(template_path, 'r', encoding='latin-1') as f:
            template_content = f.read()

    # Look for placeholders in the template, or just append content if none found
    if "{{NEWS_CONTENT}}" in template_content:
        mail.HTMLBody = template_content.replace("{{NEWS_CONTENT}}", content)
    elif "{{CONTENT}}" in template_content:
        mail.HTMLBody = template_content.replace("{{CONTENT}}", content)
    else:
        # If no placeholder is found, append the content to the template
        mail.HTMLBody = template_content + "<div>" + content + "</div>"

    # Configure the email
    mail.Subject = f"SCOR Financial Press Release: {release_name}"
    
    mail.Display(True)  # Display the email before sending
    print(f"Email created for {release_name}")
    return mail

# Main Execution
if __name__ == "__main__":
    press_release_urls = find_press_release_urls(BASE_URL, TARGET_RELEASES)
    
    created_emails = []
    for url, release_name in press_release_urls:
        print(f"Processing {release_name}...")
        content = extract_content(url)
        if content:
            mail = fill_outlook_template(OUTLOOK_TEMPLATE_PATH, content, release_name)
            created_emails.append(mail)
        else:
            print(f"Failed to extract content for {release_name} from {url}")

    print(f"Finished processing {len(created_emails)} press releases.")

How the Script Works

Script Architecture and Analysis

The script follows a clear workflow to extract press releases from SCOR's website and integrate them into your Outlook template. Let's analyze each component:

Core Workflow Overview

The script follows a logical sequence consisting of three main phases: discovery, extraction, and integration:

Phase	Function	Description
Discovery	find_press_release_urls()	Locates the precise URLs for the requested press releases using pattern matching and DOM traversal
Extraction	extract_content()	Parses the HTML of each press release to extract structured content while preserving formatting
Integration	fill_outlook_template()	Populates your template with the extracted content and prepares Outlook drafts
Execution	Main section	Orchestrates the entire process from discovery to email creation

Web Scraping with Firecrawl

The script uses Firecrawl's API to navigate SCOR's website structure. It intelligently discovers press release links through a combination of text pattern matching and DOM traversal, which is more reliable than simple keyword searches.

Content Extraction Strategy

The content extraction function is designed to be resilient, attempting multiple potential container classes to locate the press release content. It preserves the original HTML structure of paragraphs, headings, lists, and tables rather than flattening the content.

Outlook Template Integration

The script handles Outlook integration through the win32com library, which provides direct access to Outlook's COM objects. It searches for placeholders in your template (such as {{NEWS_CONTENT}} or {{CONTENT}}) and intelligently falls back to appending content if no placeholders are found.

Visual Process Map

This mindmap illustrates the complete workflow of the press release scraping and Outlook integration process:

mindmap root["Press Release Processing System"] id1["Initialization"] id1-1["Set API key & URLs"] id1-2["Define target releases"] id1-3["Configure template path"] id2["Web Scraping"] id2-1["Call Firecrawl API"] id2-2["Parse HTML with BeautifulSoup"] id2-3["Extract press release URLs"] id3["Content Extraction"] id3-1["Scrape press release pages"] id3-2["Identify content containers"] id3-3["Extract formatted content"] id3-4["Preserve HTML structure"] id4["Outlook Integration"] id4-1["Load HTML template"] id4-2["Replace placeholders"] id4-3["Create email draft"] id4-4["Display for review"]

Video Tutorial: Web Scraping for Financial Data

This video explains how SCOR publishes financial results and press releases, which can help you understand the context of the data you're extracting:

The video features Thierry Léger and François de Varenne discussing SCOR's annual financial results, providing valuable context about how the company structures its financial communications, which can help you better understand the content you're scraping.

Example Press Release Images

Below are examples of SCOR's financial press releases. Understanding their structure helps optimize the content extraction process:

These images demonstrate the typical presentation of SCOR's financial information. The script is designed to accurately extract text content while preserving the hierarchical structure of the information.

Implementation Considerations and Best Practices

Handling Potential Challenges

1. Dynamic Website Elements

SCOR's website may use JavaScript to load content dynamically. The Firecrawl API is configured to handle JavaScript rendering, but you may need to adjust the scraping strategy if certain elements are not immediately accessible.

2. Template Structure Variations

The script includes multiple fallback mechanisms to identify where content should be inserted in your template. If your template uses different placeholder conventions, modify the template replacement logic accordingly.

3. Press Release Format Changes

If SCOR changes their press release format, the content extraction function includes multiple fallback options to locate and extract content from different container elements.

Frequently Asked Questions

What dependencies do I need to install for this script?

You'll need to install the following Python packages:

requests: For making HTTP requests to the Firecrawl API
beautifulsoup4: For parsing and navigating HTML content
pywin32: For interacting with Microsoft Outlook

Install them using pip: pip install requests beautifulsoup4 pywin32

How do I modify the script to handle different press releases?

Simply update the TARGET_RELEASES list with the names of the press releases you want to extract. The script will automatically search for these releases on the SCOR website. If you need press releases from different sections of the website, you may need to modify the BASE_URL variable accordingly.

Can I send emails automatically instead of just creating drafts?

Yes, you can modify the fill_outlook_template function to automatically send emails by replacing mail.Display(True) with mail.Send(). However, it's generally recommended to review the emails before sending them, especially when working with financial information.

What if the script can't find some press releases?

How can I schedule this script to run automatically?