Software Development

A 43-post collection

Using Docker for local web development

By Matt Button |  May 9, 2020  | asp.net, docker, sql, docker-compose

If you’re new to Docker, it’s hard figuring out even what to search for, and how the concepts tie together, let alone what to pay attention to for your use case.

To make things more difficult, there has been a lot of change over time, and stuff that was once recommended is now not recommended, for example, a lot of older video tutorials recommend using container links which have since been deprecated in favor of user-defined bridge networks .. in case you’re unfamiliar with this, I’ll cover what this means, and why you’d use it later in the blog post.

Once you learn the basics, using Docker in practice is fairly straightforward, and if you’re using Docker at home or at work, the time you spend learning now will save you a massive amount of time and effort in the future.

If you’ve used Docker a bit, are familiar with some of the concepts, how dabbled here and there, but sometimes struggle with connecting some of the concepts, or how to get things set up end-to-end, then this blog is for you.

In this blog post, I’ll guide you through step by step on:

  • The distinction between docker image and containers
  • Best practices on how to persist data within containers, and how to control where data is persisted
  • How to continue developing on your local machine as usual, while your app is running within a docker container on your local machine
  • How to connect your containerised web app to a containerised Microsoft SQL Server
Continue Reading...

Redux made easy with Redux Toolkit and Typescript

By Matt Button |  Jan 20, 2020  | react, redux, typescript

Using Redux with React can be incredibly challenging for people to get their head around; while the core principles of Redux itself are fairly straightforward and don’t take too long to grasp, a lot of people struggle with using Redux in React apps. Partly because there are so many damn things to integrate:

  1. Redux
  2. React Redux
  3. Redux Thunk
  4. Typescript (getting more common in recent years)
  5. Redux Saga / Redux Observable (in more complex apps)

I spotted this quote which summed it up for me:

Redux was such a mental overhead for me when I started out with it. The amount of boilerplate code that I had to write for the project I worked on was very frustrating.

I struggled with that stuff too, even after I became familiar with the concepts, I often found it difficult to understand the flow of data and events within React Redux apps. This is not uncommon, even the original author of Redux had this to say:

Yup. Can totally relate.

Then I found Redux Toolkit, written by the maintainers of Redux, and recommended by the Redux style guide.

Continue Reading...

How to scrape stock upgrades and downgrades from Yahoo Finance

By Matt Button |  Dec 2, 2019  | yahoo-finance, python, beautiful-soup, web-scraping, pandas

In this blog post I’ll show you how to scrape stock upgrade and downgrade data from Yahoo Finance using Python, Beautiful Soup, json, and a bit of regular expression magic.

This technique can be applied to any stock symbol on Yahoo Finance, but for this blog post we’ll be scraping data for Apple (AAPL).

I’ll be guiding you through the process, describing my thought processes and techiques, and if you follow along, by the end of this blog post, you’ll have extracted stock upgrade and downgrade data in a Pandas dataframe which will look something like this:

From there, you can easily export the data into an Excel file, or if you’re a more advanced Python user, host the code in a Web API which can be accessed by Google Sheets, or your own custom code.

Continue Reading...

Accelerate - Building and Scaling High Performing Technology Organisations

By Matt Button |  Sep 19, 2019  | book-notes

I first read Accelerate by Nicole Forsgren, Jez Humble, Gene Kim back in 2018 when it was a hot topic in the office; only an hour or two would go by between overhearing somebody talk about it, internal blog posts were being written, and we’d receive emails about following the findings.

One of the goals behind the book was to run a series of surveys, gathering 23000 responses across 2000 organizations, then using statistical methods on the results in order to find a way to measure software delivery performance, and to find out what drives it. One of the most impressive things about the book is that it not only presents the research findings, but also goes into detail about why the findings are trustworthy, meaning that they not only tell us what they learned, but how they learned it.

I recently read through the book again and took a bunch of notes which cover the main points.

Continue Reading...

13 years ago we were rallying against nested HTML tables - Notes from Webstock 2019

By Matt Button |  Jun 29, 2019  | webstock, notes

I’m not sure why it’s taken so long, but after 13 years of Webstock existing, this year was the first time that I attended.

While I haven’t attended Webstock until now, I have watched many of the recordings which are usually available a few months after each event was held, and it’s been interesting to see how the event, and the industry in general has changed over the years.

None of this year’s talks were technical, and all were about the human side of the industry - topics such as ethics, personal experiences, psychology, communication, and team health.

One of the original organisers stood up on stage at the end of the event, and said it best:

13 years ago we were rallying against nested HTML tables

Continue Reading...

Using Git on Windows

By Matt Button |  Mar 24, 2019  | git

Using Git on Windows is way easier once you find a combination of tools that you’re comfortable with.

In this blog post you’ll learn about:

  • Using Git via the command line on Windows
  • A great Git GUI client for Windows and Mac
  • The tool that I use for diffs, and resolving any Git merge conflict that may occur (including 3-way merges)
  • Using Git with SSH authentication on Windows via .bashrc and Bash shell
  • My .gitconfig
Continue Reading...

How to scrape Yahoo Finance and extract fundamental stock market data using Python, LXML, and Pandas

By Matt Button |  Apr 3, 2020  | python, pandas, lxml, scraping, finance, featured, yahoo-finance

In this blog post I’ll show you how to scrape Income Statement, Balance Sheet, and Cash Flow data for companies from Yahoo Finance using Python, LXML, and Pandas.

I’ll use data from Mainfreight NZ (MFT.NZ) as an example, but the code will work for any stock symbol on Yahoo Finance.

The screenshot below shows a Pandas DataFrame with MFT.NZ balance sheet data, which you can expect to get by following the steps in this blog post:

After taking you step by step on how to fetch data from the balance sheet, I’ll show you how to generalise the code to also generate a DataFrame containing data from the Income Statement, and Cash Flow statement.

After creating the Pandas DataFrames, I’ll then show you how to scrape data for multiple symbols, and finally, export everything to an Excel file, so you’ll have output that looks something like this:

This post was last updated in April, 2020.

Prior to October, 2019, Yahoo Finance conveniently had all this data in a regular HTML table, which made extracting the data super easy. Since then, they’ve updated the page with a new structure, which was a wee bit tricker to get the data from. Fortunately, it’s still possible. Read on to find out how.

Continue Reading...