2021 - Open Source Automation

27Nov 2021 by Andrew Treadway

How to stop long-running code in Python

Python

Ever had long-running code that you don't know when it's going to finish running? If you have, then Python's stopit library is for you. In a previous post, we talked about how to create a progress bar to monitor Python code. This post will show you how to automatically stop long-running code with the stopit package. Getting started with stopit To get started with stopit, you can install it via pip: In our first example, we'll use a context manager to stop the code we want to execute after a timeout limit is reached. The result of running the above code will print out "DID NOT FINISH...". Above, we just need to specify the number of seconds we want the timeout limit to be - in this case, 5. Inside…

09Oct 2021 by Andrew Treadway

Faster alternatives to pandas

Pandas, Python

Background If you've done any type of data analysis in Python, chances are you've probably used pandas. Though widely used in the data world, if you've run into space or computational issues with it, you're not alone. This post discusses several faster alternatives to pandas. R's data table in Python If you've used R, you're probably familiar with the data.table package. A port of this library is also available in Python. In this example, we show how you can read in a CSV file faster than using standard pandas. For our purposes, we'll be using an open source dataset from the UCI repository. Using datatable, we can read in the CSV file in ~20 seconds. Reading the same file using pandas takes almost 76 seconds! Next, we can also sort…

02Jul 2021 by Andrew Treadway

Automated EDA with Python

Pandas, Python

In this post, we will investigate the pandas_profiling and sweetviz packages, which can be used to speed up EDA (exploratory data analysis) with Python. In a previous article, we talked about an analagous package in R (see this link). Getting started with pandas_profiling pandas_profiling can be installed using pip, like this: Next, let's read in our dataset. The data we'll be using is a heart attack-related dataset, which can be found here. Now, let's import ProfileReport from pandas_profiling. If you're running this code in Jupyter Notebook, you should see the report generated within your notebook file. The report shows several pieces of analysis. First, it gives a summary glimpse of the data, giving the number of variables, observations, missing values and percentages, data type information, and number of duplicate rows…

28Apr 2021 by Andrew Treadway

How to plot XGBoost trees in R

Machine Learning, R

In this post, we're going to cover how to plot XGBoost trees in R. XGBoost is a very popular machine learning algorithm, which is frequently used in Kaggle competitions and has many practical use cases. Let's start by loading the packages we'll need. Note that plotting XGBoost trees requires the DiagrammeR package to be installed, so even if you have xgboost installed already, you'll need to make sure you have DiagrammeR also. Next, let's read in our dataset. In this post, we'll be using this customer churn dataset. The label we'll be trying to predict is called "Exited" and is a binary variable with 1 meaning the customer churned (canceled account) vs. 0 meaning the customer did not churn (did not cancel account). Preparing the dataset for modeling Now, let's…

14Apr 2021 by Andrew Treadway

Python collections tutorial

Python

In this post, we'll discuss the underrated Python collections package, which is part of the standard library. Collections allows you to utilize several data structures beyond base Python. How to get a count of all the elements in a list One very useful function in collections is the Counter method, which you can use to return a count of all the elements in a list. The Counter object that gets returned is also modifiable. Let's define a variable equal to the result above. Notice how we can add the number 20 to our Counter object without having to initialize it with a 0 value. Counter can also be used if you have a list of tuples. This can be handy if you want to count pairs of numbers or other…

25Mar 2021 by Andrew Treadway

How to create PDF files with Python

Python

In a previous article we talked about several ways to read PDF files with Python. This post will cover two packages used to create PDF files with Python, including pdfkit and ReportLab. Create PDF files with Python and pdfkit pdfkit was the first library I learned for creating PDF files. A nice feature of pdfkit is that you can use it to create PDF files from URLs. To get started, you'll need to install it along with a utility called wkhtmltopdf. Use pip to install pdfkit from PyPI: Once you're set up, you can start using pdfkit. In the example below, we download Wikipedia's main page as a PDF file. To get pdfkit working, you'll need to either add wkhtmltopdf to your PATH, or configure pdfkit to point to where…

03Mar 2021 by Andrew Treadway

Faster data exploration with DataExplorer

R

Data exploration is an important part of the modeling process. It can also take up a fair amount of time. The awesome DataExplorer package in R aims to make this process easier. To get started with DataExplorer, you'll need to install it like below: Let's use DataExplorer to explore a dataset on diabetes. Running the create_report line of code above will generate an HTML report file containing a collection of useful information about the data. This includes: Basic statistics, such as number of rows and columns, number of columns with missing data, count of continuous variables vs. discrete, and the total memory allocation Data type for each field Missing data percentages for each column Univariate distribution for each column QQ plots Correlation analysis PCA That's right - a single line…

16Feb 2021 by Andrew Treadway

How to get stock earnings data with Python

Python

In this post, we'll walk through a few examples for getting stock earnings data with Python. We will be using yahoo_fin, which was recently updated. The latest version now includes functionality to easily pull earnings calendar information for individual stocks or dates. If you need to install yahoo_fin, you can use pip: If you already have it installed and need to upgrade, you can update your version like this: To get started, let's import yahoo_fin: Getting stock earnings calendar data The first method we'll cover is the get_earnings_history function. get_earnings_history returns a list of dictionaries. Each dictionary contains an earnings date along with EPS actual / expected information. Let's test it out with a few sample tickers. Depending on the ticker, the data returned may include future earnings dates. Any…

02Feb 2021 by Andrew Treadway

Technical analysis with Python

Python

In this post, we will introduce how to do technical analysis with Python. Python has several libraries for performing technical analysis of investments. We're going to compare three libraries - ta, pandas_ta, and bta-lib. The ta library for technical analysis One of the nicest features of the ta package is that it allows you to add dozen of technical indicators all at once. To get started, install the ta library using pip: Next, let's import the packages we need. We'll be using yahoo_fin to pull in stock price data. Now, data contains the historical prices for AAPL. Next, let's use ta to add in a collection of technical features. Below, we just need to specify what fields correspond to the open, high, low, close, and volume. This single call automatically…

05Jan 2021 by Andrew Treadway

Python’s rich library – a tutorial

Python, System Administration

The Python rich library is a package for having clearer, styled, and colored output in the terminal. rich works across multiple operating systems - including Windows, Linux, and macOS. In this post, we'll give an introduction to what it can do for you. You can get started with rich by installing it with pip. Once you have it installed, open up the command line and type in python. In order to get the additional functionality from rich, you'll need to do one more step, which you can see below. Running this snippet will allow you to have styled / formatted code interactively. You'll only need to do this once. Here's a couple examples of automatic coloring for a list. Creating "rich" progress bars In a previous post, we talked about…

Year: 2021