How I built a simple stock "analytics" site with web scraping and serverless functions


## A little backstory: the GameStop short squeeze

Everyone who even just loosely follows american news must have heard of the craziness going on on the US stock market in the past few weeks. In short the story is that a large group of Reddit users from the [WallStreetBets](https://www.reddit.com/r/wallstreetbets/) subreddit, which describes itself as "a community for making money and being amused while doing it. Or, realistically, a place to come and upvote memes when your portfolio is down." almost brought Wall Street to its knees. They discovered that the GameStop stock was massively overshorted by a number of large hedge funds and in an effort to stick it to those greedy Wall Street investors they decided to start mass buying the company's shares. This drove up the stock's price so much that some hedge funds lost billions and needed to be bailed out to avoid bankruptcy. There's more to the story than this but let's get to the topic of the article.

![The GameStop rally](/images/posts/post4/gamestop.jpg)

## We Like The Stock!

Okay, I admit calling [WeLikeTheStock.live](https://welikethestock.vercel.app) a stock analytics site is a bit generous for this ridiculous looking meme site. My goal was not to build a [Yahoo Finance](https://finance.yahoo.com/) competitor, but be a small part of this historic movement and build something that can bring a smile to some disgruntled Reddit investors' faces.

![We Like The Stock](/images/posts/post4/post4-thumbnail.jpg)

## Web scraping, serverless functions and serverless database

I've been a heavy user of [React](https://reactjs.org/) and [Next.js](https://nextjs.org/) so they were obvious choices for this project as well. Next.js makes the development of React sites a breeze with many useful features such as static & server side rendering, easy routing etc. If you use their own hosting solution, [Vercel](https://vercel.com/) and hook it up to your site's repository then the whole deployment process is as easy as just pushing your changes to GitHub (or BitBucket or GitLab). Vercel also has built-in support for serverless functions which you can write in the same repo as your frontend code and push it to production along with your site. It doensn't get much easier than that!

For the web scraping part I went with [Scarper API](https://www.scraperapi.com/) which is another wonderfully simple to use framework, allowing developers to easily grab the HTML content from any page.

I used [Lambda Store](https://lambda.store/) to store data, which is a serverless database solution which takes only a few clicks to set up. It was my first time using it but I was suprised at the simplicity of it. It may not be suitable for a large scale application but it was perfect for my needs here.

## Hooking it all up

So we've got everything we need but how does it all come together to form this gorgeous looking website? Well the frontend code is not much to look at, the juicy part is the serverless function. It's one simple endpoint which provides the data to the site through Next.js's [`getServerSideProps`](https://nextjs.org/docs/basic-features/data-fetching#getserversideprops-server-side-rendering) function, which lets you pre-render the page on each request before serving it to the user. This function is an `async` function which can go and fetch data or do any kind of work before returning the result which will be passed in as render props to the React site.

The serverless function itself does the following:

-   It checks the Redis database for recently updated data.
-   If it doesn't find anything or if the data is out of date (older than 20 mins), it goes ahead and scrapes [https://www.highshortinterest.com/](https://www.highshortinterest.com/) which is a site listing the most shorted stocks. Scraper API takes care of all the scraping work, returning just the raw HTML code.
-   With the help of [JSSoup](https://github.com/chishui/JSSoup) it pulls out the data from the HTML and transforms it into a JSON object.
-   Once that's done it saves the new data to the database, and returns it for the frontend to use.

And that's how you build a site using web scraping and serverless functions in no time! While this toolset would not scale well for larger applications, it's best not to overcomplicate things when all we want is a simple website that will break America's financial system.

## [Go check it out now!](https://welikethestock.vercel.app)


Do you like GeoGuessr? I made a multiplayer version of it that you need to try.


## Geo... what?

In case you don't know [GeoGuessr](https://geoguessr.com) is an online game where you're dropped somewhere on Earth and with the help of Google Street View you need to figure out where you are. It's a very popular online game and a personal favorite of mine.

## So why make another one?

A little background story: besides being a GeoGuessr fan, I've also spent countless hours playing .io-type online multiplayer games, like: [skribbl.io](https://skribbl.io) and [slither.io](https://slither.io). The multiplayer element (among other things) makes these games a lot of fun and (unfortunately) highly addictive.

And that's what's missing from GeoGuessr.. Don't get me wrong it's a great game, but I figured it could be even greater if it had some sort of multiplayer functionality. So I built my own version of it.

## Introducing... Hide & Seek World!

In [Hide & Seek World](https://hideandseek.world) you're up against real players instead of just getting computer-generated random locations.

The game consists of a few turns, and in each turn there's one hider who selects a location somewhere in the world, and 1-5 seekers who try to guess the location based on Google Street View images.
As a hider your goal is hide well because the better seekers are able to guess your location the less points you get. As a seeker you get more points if you guess better.

![screenshot1](/images/posts/post3/screenshot1.jpg)

The concept is similar to the popular children's game, hence the name. The exception is that you have a bigger playground here, the whole world. At the end of each round you can see the guesses of all the players and how far everyone was from the actual hiding place.

![screenshot2](/images/posts/post3/screenshot3.jpg)

## Game modes

There's 2 game modes: Street View and Satellite. In Street View mode you hide by selecting a street view location somewhere and in Satellite mode your hiding spot is a zoomed in rectangular area on the surface of Earth. In Street View mode you need to be able to read visual cues about architectural styles, vegetation, people etc. to guess well, while Satellite mode relies more on your geographical knowledge.

You can also create a private room and play against your friends, where you can customize more things like: how much time you have to make a guess or restrict the map to only certain countries.

## How I built it

I built the frontend using Next.js & React, and the backend with Firebase services: Authentication, Realtime Database, FireStore, Functions, Analytics. For the maps & Street View related things I used Bing Maps and Google Maps APIs. Payments are handled with Stripe Checkout. The site is hosted on Netlify.

## [Go check it out now!](https://hideandseek.world)


My first 3 months of web development


## From zero to hero in 3 months

It's been more than 3 months since I went all in on web development, so it's about time I wrote a status update. In March, I wrote my [first blog post](/posts/post1) outlining the plan I was going to follow to get myself up to speed on the latest and greatest web dev technologies and build up a skillset that can land me a job in today's market. For the most part I followed my plan pretty faithfully with only slight deviations where I deemed a change was necessary.

With that being said, the actual "learning" part (that is taking online courses & tutorials) only took me about 3.5 (intense) weeks and the rest was putting that knowledge to good use and building things.
We all know that to be a good developer, learning new things is only a small part of the game. You have to be able to use your skills to build them into your "muscle memory" so it's not just some vague theoretical knowledge but knowledge you can apply in real life.

## So what have I been up to?

Here's all the resources I used, courses I took, skills I acquired and projects I built in this 3 months period:

#### Courses

1. [The Complete 2020 Web Development Bootcamp](https://www.udemy.com/course/the-complete-web-development-bootcamp/): This one was only a refresher since I took this course a long time ago. It's a course that touches a bit on everything, so it was a good way to start and brush up that little unused knowledge I had beforehand.
2. [NodeJS - The Complete Guide (incl. MVC, REST APIs, GraphQL)](https://www.udemy.com/course/nodejs-the-complete-guide/): Another refresher, since I took this one some time ago as well. I only skimmed through it since my primary focus this time is frontend development.
3. [Get Started With CSS (Plus Flexbox, Grid, and Sass)](https://www.udemy.com/course/css-the-complete-guide-incl-flexbox-grid-sass/): Excellent and very in-depth course from one of my favorite instructors, [Maximilian Schwarzmüller](https://www.udemy.com/user/maximilian-schwarzmuller/). More than 22 hours of CSS goodies. I feel like it gave me some pretty strong foundations on the topic.
4. [JavaScript - The Complete Guide 2020 (Beginner + Advanced)](https://www.udemy.com/course/javascript-the-complete-guide-2020-beginner-advanced/): Another mega course (~52 hours) from the same instructor, this time on JavaScript. It was great, comprehensive and had both sufficient breadth and depth. This and the CSS one gave me the foundations that I beleive is the bare minimum every web developer should have before moving on to more advanced frameworks.
5. [React 16: The Complete Course (incl. React Router 4 & Redux)](https://www.udemy.com/course/react-the-complete-guide-incl-redux/): Once I got the foundations down (and made sure I understood them well, more on that later), it was time to move on to my frontend framework of choice, React. At this point I was so happy with Max's other courses that I figured I can't go wrong with his React course either, and I was not disappointed. This course teaches you the ins and outs of React and some related technologies. This was the last course I took, then it was time to build some sites.

#### Other resources

1. [syntax.fm](https://syntax.fm): Web dev podcast. I use it to get informed about trends, popular and upcoming technologies & frameworks. Listening to it gives me a clear perspective of what's out there in the web dev world and what are the strengths and weaknesses of each technology.
2. Medium articles, MDN docs, blog posts etc.

#### Skills I learned

-   HTML
-   Javascript
-   (S)CSS
-   React
-   Next.js
-   Netlify
-   Jamstack
-   Serverless functions
-   Firebase: Authentication, Hosting, Realtime Database, Firestore, Cloud Functions, Analytics
-   Google Cloud Platform
-   Visual Studio Code
-   Google Maps JavaScript API

#### Web dev projects

1. [Prio](https://prio-landing.firebaseapp.com/): My first "serious" project. It is a full-featured todo list app that has both web and iOS components (both made by me), and the two sync together in real time, thanks to Firebase Firestore. It was built in vanilla Javascript without the use of any frontend framework, using Web Components and Firebase services (Authentication, FireStore, Hosting) under the hood. My primary goal with this was to build a project using only the fundamentals (so no frontend framework), to really drill in that HTML, CSS and JS knowledge. I made this before I even started learning React, intentionally so I couldn't get tempted to use anything but the basics. From start to finish it took me about 1.5 weeks to build the landing page and the app itself.

2. [This website](https://istvan-kreisz.com/): My second project, that I started after I finished all the courses outlined above. This time my goal was to build a simple blog, personal site with my newfound React knowledge. No restrictions this time, so I built it using the React, Next.js and Netlify trio, which I instantly fell in love with. Next.js's static site generation and Netlify's automatic deployment are incredible tools that take care of all the heavy-lifting for you, so you can just focus on writing code. I had a lot of fun with this project and I'll continue improving it as I go. The first version took about a week to build, since then I've spent an additional few days on tweaking and improving it.

3. [Hide & Seek World](https://hideandseek.world/): This really is the crown jewel that took me about 2 months to build. It's a turn-based online multiplayer game inspired by two of my personal favorites: [GeoGuessr](https://www.geoguessr.com/) and [Skribble.io](https://skribbl.io/). The premise of the game is simple: hide somewhere on Earth by selecting a street view location, then the other players have to find you based on the street view image. As for the tech stack: the backend component was built in a serverless way, using Google Cloud Functions. I used Firebase Firestore and Firebase Realtime Database for data storage. The frontend uses React, Next.js, Firebase Authentication, Google Maps JavaScript API and Stripe for payment processing. It's hosted on Netlify. It was an interesting and complex project that really put my skills to the test. There will be a follow-up post about it soon, detailing the development process.

#### What's next?

Obviously 3 months is not enough to learn everything, but with a solid engineering background (including some basic web dev knowledge), clear goals and a good amount of hard work I feel like I managed to get a lot out of it. I am now ready to tackle complex web dev tasks and I feel confident in being able to perform in a professional environment. With that being said, I am not stopping here. I am in this for the long haul and I can't wait to see where I'll be in 1, 2, 5 years. Hopefully a rockstar developer with millions in the bank!


My web development learning plan


## How to make a plan?

Okay so I’ll be learning web development, but where do I start? A quick search on Google for something like “learn web development” will give you about a bazillion options to choose from. It can get pretty overwhelming. And no matter how many reddit threads & articles you go through you won’t find “THE BEST” way to go about it.

Now, my methods may not work for everyone. I won’t get into the pros and cons of the different ways of learning new technologies here, because that could have an article on its own.
Without further explanation, let’s see my criteria for setting up my learning path:

-   **It shouldn’t cost much**: College, bootcamps were out of the question.
-   **I am a self learner, thus I only need an internet connection and the rest I’ll take care of**: Another point against college, or bootcamps.
-   **It should be the most efficient way to learn**: For me that usually means a few carefully selected online courses with lots of hands-on practice
-   **It should take no longer than 3-4 months**: This is again specific to my situation. I already have a strong foundation in software development. I am no stranger to things like software development principles, system architecture, using git in a team workflow, continuous integration, building working software from scratch (mostly iOS apps). I am also a pretty fast learner. On the other hand I won’t spend every waking hour learning web dev during this period, since I have other things to pursue as well: working on my next product, writing this blog, building up an online presence, networking and maybe even traveling. Oh and also I’ll still have my full time job as an iOS dev for the next month so my time in the beginning will be limited.
-   **It should be fun**: I could read through the entire JavaScript documentation and try to memorize it but I’d probably break down crying after 10 minutes. Not to mention it’s probably not a very efficient way to learn.

## The tech stack

Keep in mind that I’ll be focusing on frontend for now but I’ll cover some backend technologies as well. There are countless tech stacks to learn on both frontend and backend. I selected some of ones that seem to be the most popular, widely-adopted and intuitive to use as of now. I won’t give a comparison of different stacks but they all have their own use cases. At the end of the day, you just have to pick one that looks like a viable option for your goals and stick with it.

### **My tech stack:**

_Frontend:_

-   **HTML**
-   **CSS**, SASS
-   **JavaScript**
-   TypeScript
-   webpack
-   BootStrap
-   **DOM**
-   **React**
-   **Redux**
-   Next.js

_Backend:_

-   **NodeJS**
-   GraphQL
-   **npm**
-   **REST APIs**
-   Websockets
-   **Express**
-   Authentication
-   Sessions, Cookies
-   **MongoDB, Mongoose**

The ones in bold are the ones I want to get a deeper level understanding of. They are mostly the foundations plus 1-2 popular higher-level frameworks.

## Courses, resources

All the courses in the next sections are from three recognized web developer instructors: **Dr. Angela Yu**, **Maximilian Schwarzmüller** and **Wes Bos**.
I am not affiliated with any of them, I just found them to be awesome teachers who can break down complex concepts in ways that are easily understandable. What’s equally as important to me is that they can make learning fun and not feel like a chore.

### What I’ve done so far

Before I started writing this blog I already dabbled with some small projects and took some courses so I’m not starting from absolute zero. For the sake of completeness, here’s what I’ve done so far.

#### Courses

-   [The Complete 2020 Web Development Bootcamp](https://www.udemy.com/course/the-complete-web-development-bootcamp/): Awesome course that touches a little bit on everything. It goes over some of the foundations like HTML, CSS and JS, covers both backend and frontend, and even gives you an intro into some of the popular frameworks on both platforms. Of course, the sheer amount of technologies covered here makes it impossible to get a deep understanding of any of them but it gives you a nice overview of the whole stack. After finishing this course I no longer felt like people were speaking Chinese when they mentioned something about web dev.
-   [NodeJS - The Complete Guide (incl. MVC, REST APIs, GraphQL)](https://www.udemy.com/course/nodejs-the-complete-guide/): Another great course, only this one is narrower in its focus. It only deals with backend technologies, more specifically with NodeJS. Again, you won’t be ready to take on the world as a backend engineer after this but will for sure learn a lot.

#### Works

-   ~~This website which is essentially just a fork of a Jekyll theme [reverie](https://github.com/amitmerchant1990/reverie) with some modifications~~ (UPDATE: this refers to my previous site, which since then I've taken down)
-   My old website which is no longer live. That one I actually built from scratch but I needed something more extensible so I pulled the plug on it.
-   All the assignments for the courses I took

And that’s about all I’ve done so far. Not a whole lot, but enough to give me some understanding of what I’m getting myself into.

### What’s next

Alright onto the exciting stuff! That is what things I’m gonna learn and in what order. What comes next is simply an ordered list of Udemy courses that I’m gonna take (with only one exception).

Besides all these course, I plan to build some sites from scratch as I go along to push myself to apply the knowledge and not just be stuck in [tutorial purgatory](https://www.freecodecamp.org/news/how-to-escape-tutorial-purgatory-as-a-new-developer-or-at-any-time-in-your-career-e3a4b2384a40/).

#### Courses

1. I’ll start off by reviewing the 2 courses I’ve taken. The first one I took over a year ago and the second one a few months ago so I’ll need a bit of a refresher.
2. **CSS:** [Get Started With CSS (Plus Flexbox, Grid, and Sass)](https://www.udemy.com/course/css-the-complete-guide-incl-flexbox-grid-sass/) Before I move on to the more advanced stuff I want to really get the foundations down so I’ll start off with a pretty in-depth CSS course. As I already have some CSS knowledge, I’ll probably skip sections here and there.
3. **JS:** [JavaScript - The Complete Guide 2020 (Beginner + Advanced)](https://www.udemy.com/course/javascript-the-complete-guide-2020-beginner-advanced/)
   More foundational stuff, this time it’s JavaScript. This is gonna be a tough one, as this course is massive with over 50 hours of video material. Again I’ll probably skip some of the introductory parts.
4. **JS:** [https://javascript30.com](https://javascript30.com/):
   Wes Bos’s JS 30 challenge, that lots of people recommended. When I’m finished with the JS course and this, I should have a pretty good understanding of JavaScript and will be ready to tackle more advanced frameworks.
5. **React:** [The Complete 2020 Web Development Bootcamp](https://www.udemy.com/course/the-complete-web-development-bootcamp/) (React Section): This belongs to the course which I’ve already taken, with the exception of the React section which was not yet available at the time.
6. **(Optional) React:** [React 16: The Complete Course (incl. React Router 4 & Redux)](https://www.udemy.com/course/react-the-complete-guide-incl-redux/): If I am still not sick of studying yet at this point, I’ll finish it off with another loooong course. It’s possible that by this point tutorial fatigue will kick in and I’ll just be itching to jump into building sites so I made this one optional.

#### Other resources

-   [syntax.fm](https://syntax.fm): A great podcast about all web dev related things.
-   [Effective_Engineer.md · GitHub](https://gist.github.com/rondy/af1dee1d28c02e9a225ae55da2674a6f): Very good notes on how to maximize your potential as an engineer. I read this every so often to check myself if I’m on the right track or I need some adjustment.
-   [Learn to become a modern frontend developer](https://roadmap.sh/frontend): A very nicely put together roadmap for learning front end development. This was a major inspiration for my learning plan too.
-   There will probably be lots of other resources, like documentations, articles etc. that I’ll use if I need some more clarification on something.

## Let’s get to it!

So there you have it, my full learning plan. As I mentioned earlier, I still have a full time job for another month and I’ll be working on other projects next to this so my progress in the beginning will be rather slow. Once I’m free of all my obligations I’ll go full speed on it.

Check back here for my weekly updates on my progress & other development related articles.
And follow me on [Twitter](https://twitter.com/IKreisz)!


My personal blog about Web & iOS development, productivity and developing products the indie way.

Welcome to my blog!

How I built a simple stock "analytics" site with web scraping and serverless functions

Do you like GeoGuessr? I made a multiplayer version of it that you need to try.

My first 3 months of web development

My web development learning plan