Pandas Archives - Open Source Automation

09Oct 2021 by Andrew Treadway

Faster alternatives to pandas

Pandas, Python

Background If you've done any type of data analysis in Python, chances are you've probably used pandas. Though widely used in the data world, if you've run into space or computational issues with it, you're not alone. This post discusses several faster alternatives to pandas. R's data table in Python If you've used R, you're probably familiar with the data.table package. A port of this library is also available in Python. In this example, we show how you can read in a CSV file faster than using standard pandas. For our purposes, we'll be using an open source dataset from the UCI repository. Using datatable, we can read in the CSV file in ~20 seconds. Reading the same file using pandas takes almost 76 seconds! Next, we can also sort…

14Jan 2020 by Andrew Treadway

How to import Python classes into R

Pandas, Python, R

Background This post is going to talk about how to import Python classes into R, which can be done using a really awesome package in R called reticulate. reticulate allows you to call Python code from R, including sourcing Python scripts, using Python packages, and porting functions and classes. To install reticulate, we can run: Creating a Python class Let's create a simple class in Python. Porting the Python class to R There's a couple ways we can can port our Python class to R. One way is by sourcing a Python script defining the class. Let's suppose our class is defined in a script called "sample_class.py". We can use the reticulate function source_python. Running the command above will not only make the class we defined will available to us…

17Apr 2019 by Andrew Treadway

How to get options data with Python

Python, Web Scraping

In a previous post, we talked about how to get real-time stock prices with Python. This post will go through how to download financial options data with Python. We will be using the yahoo_fin package. The yahoo_fin package comes with a module called options. This module allows you to scrape option chains and get option expiration dates. To get started we'll just import this module from yahoo_fin. How to get options expiration dates Any option contract has an expiration date. To get all of the option expiration dates for a particular stock, we can use the get_expiration_dates method in the options package. This method is equivalent to scraping all of the date selection boxes on the options page for an individual stock (e.g. https://finance.yahoo.com/quote/NFLX/options?p=NFLX). How to get the option chain…

18Dec 2018 by Andrew Treadway

2 packages for extracting dates from a string of text in Python

Pandas, Python

This post will cover two different ways to extract dates from strings with Python. The main purpose here is that the strings we will parse contain additional text - not just the date. Scraping a date out of text can be useful in many different situations. Option 1) dateutil The first option we'll show is using the dateutil package. Here's an example: Above, we use a method in dateutil called parse. The first parameter to this method is the string of the text we want to use to search for a date. The second parameter, fuzzy_with_tokens, is set equal to True - this causes the method to return where the date is found in the string. In other words, in the string above, the date is found after the substring…

31Oct 2018 by Andrew Treadway

Data Analysis with Python Course: How to read, wrangle, and analyze data

Pandas, Python

For anyone in the NYC area, I will be holding an in-person training session December 3rd on doing data analysis with Python. We will be covering the pandas, pyodbc, and matplotlib packages. Please register at Eventbrite here: https://www.eventbrite.com/e/data-analysis-with-python-how-to-read-wrangle-and-analyze-data-tickets-51945542516. Overview Learn how to apply Python to read, wrangle, visualize, and analyze data! This course provides a hands-on session where we'll walk through a prepared curriculum on doing data analysis with Python. All code and practice exercises during the session will be made available after the course is complete. About the course During this hands-on class, you will learn the fundamentals of doing data analysis in Python, the powerful pandas package, and pyodbc for connecting to databases. We will walk through using Python to analyze and answer key questions on sales…

16Sep 2017 by Andrew Treadway

Parsing Dates with Pandas

Pandas, Python

The pandas package is one of the most powerful Python packages available. One useful feature of pandas is its Timestamp method. This provides functionality to convert strings in a variety of formats to dates. The problem we're trying to solve in this article is how to parse dates from strings that may contain additional text / words. We will look at this problem using pandas. In the first step, we'll load the pandas package. Next, let's create a sample string containing a made-up date with other text. For now, assume the dates will not contain spaces (we will re-examine this later). Taking this assumption, we use the split method, available for strings in Python, to create a list of the sub-strings of our variable, string, which are separated by spaces.…

Tag: Pandas