FCC's Weather App Project Part Two: Bittersweet Callbacks

Upon gathering my data from Weather Underground, I had written a simple script calling each piece of data thinking that is all I would need. I was wrong, however, and the reason is due to JavaScript's advantage as well as its disadvantage as a programming language.

JavaScript executes programs in an asynchronous fashion, which is unusual to a Ruby programmer like me. What's the difference? A synchronous program is executed line by line and whenever a defined function is called, the computer waits until the function has completed its work to move execute the remaining code. Asynchronous languages execute code whether or not a function has completed its work or not. This is a little confusing, but it is one the main reasons JavaScript is so popular: its quick. So why would this cause programming problems?

If I were to simply assign the "$.getJSON" call to a variable and log it to the console, it would return the JSON history, but not be accessible. Due to the asynchronous nature of JavaScript, the function executes after the JSON data is no longer available. However, JavaScript has a unique property that is both unusual, but also extremely useful: passing functions into functions. In order to retain the data from the API call we have to assign the data to a function and app it into other functions that use the data for our purposes. Imagine if you will that you have predefined functions and you’re essentially calling back to them throughout your script. This is called a callback and it is quite confusing but also quick. Here’s an idea of what a callback looks like in my script:

function testIdea(value){

console.log(value);

var apiCall = "https://api.wunderground.com/api/f946ee8192f9f7ca/conditions/q/" + value.location.state + "/" + value.location.city + ".json";

callData(apiCall);

}

var geoLocate = $.getJSON("https://api.wunderground.com/api/f946ee8192f9f7ca/geolookup/q/autoip.json", function(api) {

testIdea(api);

})

This tends to be very confusing, but not too bad once you get the hang of it. However, this outlines one of the disadvantages JavaScript has for large scale apps versus another backend language like Java. Imagine having different sets of callbacks for multiple types of information. The script will get quite confusing. However, now my scripts are set up and all I have to do it make my site pretty and add some minor functionality. The hard work is over and its nothing but sunny skies ahead!

FCC's JavaScript Weather App: Introduction to API's

The Goal: To create an app that takes weather data from a weather information site and display it to a site so that any user can get their local weather data (in Celsius and Fahrenheit) from my app. Here's an example of my end goal.

As part of my goal to become really good at JavaScript and Java, I've started to take on projects on Free Code Camp again. This project required me to find a good API or Application Programming Interface. What is an Application Programming Interface? Interface is just a fancy word for a meeting ground of two organizations or systems. So an Application Programming Interface is a sort of meeting ground for one application to retrieve data from another application. This is important because, generally speaking, there are two ways to obtain data. One way landed me in trouble with a web admin and is incredibly slow and tedious and generally not allowed. That is to scrape the data off a website with a web crawler. The other is to use an API. Use API's. Its a skills employers need and its what developers have set up (with certain conditions such as a payment or providing credit to the data they have obtained) so that you can safely and securely obtain valuable information.

With JavaScript, an API returns an object in JSON form. Here is an example from my project's api call that was printed to the console.

 This is a pretty sweet deal and I must provide credit to the best weather API that is both simple and easy to use: Weather Underground. So how can we make this object useful to our application? The default language of data transmission on the web is JSON (which is why JavaScript is popular). This means that with an API call with jQuery (outlined next post) that I can simply pull data from this object with a command such as (object name).current_observation.UV. Any guess what data this would return? It would return the integer value of 3, where (object name) is the name of a local variable. This might be a little cloudy now, but I plan on outlining how this works exactly in my next post. Get it? Cloudy?! Weather??? Alright, till next time...

Aspirations

One of the key success factors with me landing my first software job, was blogging. This allowed me to do two things:

1. Maintain accountability in my coding journey
2. Write a blog as if I were teaching the code I was learning. (You remember more of what you teach)

As such, I think it is important to write out what I want to learn for the first two reasons as well as show my intended path. When I started my journey, my goal was to: "become fluent in Ruby and conversant in JavaScript." Now that I'm employed, my goals serve to first master my current role (automated tester) and secondly to explore what ultimate career path I want to take. I enjoy testing, but I am also very curious about becoming a developer. Luckily, the next two languages I'm delving into: JavaScript and Java are both useful in testing. However, I will be learning some web development skills and begin demonstrating that here.  I will say that any developer should be an excellent tester first. And anyone wishing to become a tester should also learn to become a great developer for the sake of proficiency in their career (imagine being able to point out code bugs upon finding errors).

My goals (in this order too) are as follows:

1. Master my current field: automated testing
2. Learn Node.js and complete a simple project with it
• I am currently working through Manuel Kiessling's Node Beginner Book
• I hope this will also serve my learn JavaScript goal
3. Master the MVC framework
4. Become proficient in Java (and possibly Java implementations of Ruby like JRuby and Groovy)

My interest, should I become any sort of developer, is to become a backend developer rather than a front end. I find this far more fascinating than the frontend and Node and Java are excellent skills to have for this reason. 

However, I will say the most important skills to learn to succeed in this field are the soft skills. Learn to speak in front of a group, learn to communicate and ask for help, and learn to be aggressive: give talks and network hard. This is a topic for next blog. See you then!

My Plan to Automate Myself Out of a Job

Ever see a cop on the side of the road directing traffic at a construction site? Or see a cop just standing around providing security at a football game? That must be a pretty cushy job right? That guy or gal probably went to roll call that day, crossing their fingers, hoping they would get to work the football game that shift. That would be cool, but who's out doing the routine patrols? Are we, the taxpayers, paying the cops overtime for this?

A lot of people don't know this, but things like directing traffic at construction sites, providing security at events and graduation parties are all duties that fall outside an officer's regular duty hours and the cost of the officer is paid for by the individuals and organizations requesting them. For instance, a construction company needing traffic control actually pays the officer and rents a police cruiser (if one is requested). This is called special duty and I'm temporarily administrating all of this while the regular coordinator is on maternity leave. With her absence, I've begun to appreciate her mental organization skills and memory. This job involves taking requests from multiple groups of people, checking inventories and staffing, getting command approval to work the job, posting the job, recording those who bid on the job, reviewing their personnel file to see if they've done anything bad to warrant not being allowed to work special duty, and then assigning the job and notifying the person who made the request. That's not all, you've also got to generate invoices, process and verify checks from the vendors, and distribute checks for the officers who work.

That might all sound difficult and I often get praised for simply taking on the job, but theoretically it's not. Its all a matter of being detailed-oriented and doing the same thing over and over and over without making a mistake. Wait a second...

Doing the same thing over and over and over and over and over again = routine work
routine work = an opportunity to enhance a human being's function or eliminate them completely through automation!

I've been teaching myself automated testing tools and developing the skill set necessary to land a job in that arena, but I understand Ruby well enough to know that everything I've done at this job can be replaced with a few scripts. In fact, they are scripts I've used before. However, remember I'm really not automating myself out of a job, I'm automating the well-organized young lady who just had a baby... Don't hate me just yet because there are two important facts we must remember:

1. We work for a law enforcement agency, which is a government agency and so we'll always have a job unless we really screw up or leave. 
2. There are certain human aspects to this that you simply cannot program. I will likely make her job 100,000 times easier rather than put her out of work. In fact, my goal is the purest of the two: make her job easier!

While, I believe I have all the tools to make this work there's one big problem that I'm not sure how to solve yet: making it look pretty. When she comes back to work, I want to make her program pretty and I'll need something that I believe can handle the task: rails. I'm going to work through the tutorial and learn rails as best I can with the end goal of automating us out of work. And really by automating us out of work, I really mean doing away with a large part of the workload.

This excites me because I don't have to wait to solve real problems, I'm working on solving a real problems  now that will make the lives of people I know easier now and create a cost savings down the line. This is what programming is all about. Let the fun begin!

Uploading files and testing popups with Capybara

So, I took on the task of testing my friend's file upload site. You can find that site here. It's a tool that allows you to upload a file and you will get a popup alerting you the file size in bytes.

When you go to a website and click something that allows you to upload a file, typically you'll get a GUI interface that interacts with your system to retrieve a file for you. However, requires a different set of tools because this means you now have to interact with your system rather than the browser. Luckily, Capybara has a wonderful method: attach_file. This method is simply amazing because it takes two arguments: the button style identifier you want to press that opens the system to attach a file and the file's path. Amazing. The method while in action would look like this:
attach_file("the-file" , 'file path')

So I ran this through cucumber and the test is well-designed, except I didn't know how switch Capybara from testing the browser to a popup window. Its really simple! Give your driver variable the method the "switch_to_alert" and Capybara will look at popups (officially know as alerts).

This shows how powerful a tool Capybara is. I feel like I'm practically speaking English to the computer and getting the computer to do amazing stuff for me.

Capybara. Can it be that simple?

So, a lot of the testing groups and employers keep telling me "you need to learn Capybara and Rspec to go with your Cucumber!" After fiddling with Capybara, I can tell you why. It makes a lot of test automation about ten times easier and the software is very intelligent.

Before I used Selenium-Webdriver to navigate to different pages, click on buttons, and identified things with HMTL and CSS tags. However, what if I could simply identify something by the text a link is displaying? What if I could just write visit(page) into ruby and it visits the page for me? Things would be much simpler and there would be far less chance of error. Also, I could write my scripts much faster! In this post, I want to concentrate on rewriting the same methods in my TrekToday automated test in Capybara.

In our Given Statement, we told Selenium Webdriver to navigate to a page using the "navigate.to" method. How do we do that in Capybara? Simple: visit(page), where page is the place you want to go to. Now, does that mean Capybara replaces SWD? Not exactly. It's actually using SWD methods to build something a little more intelligent. How do I know? Its in the gem's documentation:

Here you can see that Capybara's visit method is defined using SWD methods. Which is really cool, but also makes this kinda like mathematics. Calculus is good for a lot of things, but you have to understand algebra before you truly understand what's going on. In order to find the code I just found with any gem, go to the Rubydoc page for that gem, find the method, and hit "view source." This really opened up my world! Back to the task at hand:

So now we can visit the page and we have to resize it. Remember Capybara is still using SWD. This is a little more extraneous with Capybara, but you simply assign "Capybara.current_session.driver.browser.manage.window" to the variable window then use the .resize_to(x,y) method on the window variable.

Where this become really easy is the when and then statements. Before we told the driver to find a CSS ID which locates a button and then asked the computer to click it for us. This then took us to the CSI Files site. However with Capybara all we have to do is tell the computer to click the button via the actual text on the button. One thing to remember in this case is to still inspect the element and check out the actual text. Sometimes some fonts are all uppercase no matter what case is input. So the link text for us is "CSI Files" and we can use the "click_link('CSI Files')" method to click the button. Now that we're on the CSI Files site, I want to use the CSI Files logo to see if we have correctly navigated to the site. All I have to do with Capybara is say page.has_content?('CSI FILES'). This really makes our code simpler to read. Here's the final code, so it's a little easier to understand what I did.

Learning Cucumber: Step Definitions Part Two (The Coding)

So referring to my last post, I showed you the output of running a feature file through Cucumber. It's pretty neat! You actually get a shell of code in which we can begin to work. This makes our workflow really efficient. Now, for this we need to stick with tools we know. Later on, I'll be blogging about Capybara, but for now let's work with Selenium-Webdriver, which we used in a previous blog post.

If you haven't already, create a directory called "step_definitions" and create a ruby file. Copy and paste the code from the Cucumber output into your ruby file. It should look like this:



Now, to understand what's going on, you have to fiddle with regular expressions a little bit. I highly recommend http://regexr.com/. Ruby uses regular expressions to communicate with the .feature file and Cucumber writes them out for us whenever we write out Gherkin and run Cucumber. Using Selenium WebDriver, let's visit the webpage, click the CSI Files Button and see if it takes us to the site! I've done something extremely similar to this in a former post. We require SWD (Selenium-Webdriver), assign "Selenium::WebDriver.for :firefox" to the variable 'driver' and use the 'navigate.to' method to go to the site. I'm going to help those of you who use this from a silly error. The WebDriver will actually open most pages in the mobile resolution making our test a little more difficult, so we need to use SWD's maximize window method. We will write out our code and paste it where it says "pending." Our code should look like this:


Now we can run Cucumber and see what happens! Our webdriver script took us to the page and maximized the window. This sets up our "given" part of Cucumber. Essentially we're saying "we're on the webpage, now we can do stuff." Our terminal should give us this output:







Now we can see the other statuses of Gherkin features. We have the Ruby code written out and nothing shows as undefined. We have properly written executable code that takes us to the TrekToday Homepage as our "given." This shows as "passed" because it did what it was supposed to do, no problems. The next step shows as "pending" because of the pending statement in our code. This means we have defined our test, but do not have code for it yet. The last step shows as "skipped" due to the way Cucumber has to work: We check the given, then when, and then then statement and in that order only. We didn't pass on our when step, so we can't even look at the then step yet. Without repeating what I did in my Selenium-Webdriver tutorial, I'll just post the code here that gets us to the site and show you the results of a fully passed test.

When I run cucumber on this, the webdriver opens TrekToday, maxes out the page size, looks for the CSI Files button using a CSS ID, clicks the button and scans the page for a specific CSS ID that identifies we're the CSI Files site. Now we know this code does this correctly and so we should see that all the tests passed.









And that's what we get. We can see the great importance of this tool. It brings everyone together and makes testing much more efficient. Much, much more efficient. I can now write an automated test in a matter of minutes. More importantly, I can show someone who doesn't know code my instruction set in plain english. They know exactly what I'm doing and that means better collaboration which also means better software.

Learning Cucumber: Step Definitions Part 1 (The Setup)

So Cucumber's Gherkin language is awesome. We can very simply write out a feature with plain english. And one rule I've learned with cucumber the simpler the better. What's really cool about Cucumber is that you can run tests based off the features you have made in plain English! How do you do that? Well, you write code that backs up the plain english instructions Gherkin instructions.

When you run Cucumber, you run it "above" the features file by typing the word "cucumber." The features file will contain, when we're all done, a plain english Gherkin file denoted with the extension .feature. Then next to that you will have a folder labeled "step_definitions." Step definitions are the codes that give Gherkin features meaning to a computer. These will have a ".rb" extension are written in ruby. This all makes sense when you start to run each step in the terminal, so let's do just that. Create a folder and name it whatever you want. I created a folder called "blogger_demo." Inside that folder type "mkdir features" into your folder. Now simply type "cucumber."
 
This is exactly what we wanted. The features file has nothing in it, so Cucumber is reading that there are no scenarios and no steps to even run this command on. So, let's choose a site (I'm using TrekToday.com) and see what features it may have that we can use to run a small test. Trek Today is awesome for Trekkers like me and has a handy navigation bar that I could use to check out all kinds of cool stuff that isn't just Star Trek, but also CSI Files and other CBS television treasures. 

What I want to do is see if Trek Today's CSI Files button works. Now as a Cucumber newbie, I decided it is best to do a manual test to see what the button actually does. It takes us to another site, CSI Files, which is similar in format to Trek Today. So let's create a file called "navbar_buttons.feature" and write out our Gherkin. I should be able to press the CSI Files button on the navbar and be taken to the CSI Files site. In Gherkin that is said even more simply:



Now that this is written out in Gherkin the computer does have something it can read. However that doesn't mean that the computer is going to carry out instructions that you have. To understand this a little better type "cucumber" into your terminal.

So our computer has read our Gherkin file and determined that there are three steps: a given, when and then step. However, the computer can't do anything with that and therefore the tests are undefined. A lot of our work is cut out for us. If you copy from the Given all the way down to the last "end" statement and paste it into a ".rb" file, you have a lot of the coding work done for you. All we have to do is add the executable code into the file. However that is best explained in Part 2.

Bringing People Together with Cucumber

So as a newbie trying to enter into the QA world, I've consistently heard "you've gotta know Cucumber!" What in the world is that? Well, firstly it is a skill that is in very high demand and for a very good reason. Cucumber is a testing framework driven by plain english and it serves to aid testing, documentation, and development. Simply put, it streamlines a company's workflow meaning: faster releases, more releases, better testing, and better development.

To me, of course, this is all theoretical. However, I have had to do technical work before (accounting and operations metrics) and serve managers, who to put it lightly, didn't understand anything I was doing. What happens is unrealistic deadlines and friction between the management expert and the expert of another field. I've even had a job where we lost tens of thousands of dollars in productivity waste. Now I can imagine what the problems could be between a programmer and a person whose expertise is something else. Cucumber reduces this problem as well using a wonderful little language known as Gherkin (which in English is a baby Cucumber).

How does it do that? Well this is better explained by examples. I'll give credit to this example to the Cucumber Book, the resource I am currently working through. Let's imagine that testsite.com is an online retailer. My friend goes to testsite and she wants to buy some camping equipment. She goes and enters her credit card information, but instead of entering 16 digits, she enters 15 by accident. Now, we could just spit something back out that says "transaction error," but that isn't user friendly and people would get frustrated with our site and we would go bankrupt. We want to give the user something that will give an error message specifying what they did wrong, so they can fix it easily.

This something is commonly referred to a feature and luckily enough, it is called a feature in Gherkin. Features are the basic units of Cucumber. Features are items that address multiple scenarios described by the given, when, then syntax. Most importantly, the first step of this is written in plain english! There are also some other keywords that may help us like "but," "and," as well as a small list of other. So breaking it down:

• We want a feature that tell us what went wrong when we enter incorrect card details
• We want them to address different scenarios

For the sake of brevity, I'll do the one aforementioned scenario. My friend enters 15 digits and should get an error message. This is simply a scenario, but the feature will address multiple problems. The Feature will address a card that is expired, stolen, all kinds of other things. The scenario addresses a single case where something would go wrong. So how do we write this in Gherkin? It's rather intuitive:

Here's a blank example:

Feature:

Scenario

Given I am someone (you can add some place at some time)
When I do something
Then I expect this result

Ok so this is rather blank, how does that translate to our problem? Like so:

Feature: Feedback when entering invalid card information

  Background:

    Given I have chosen some items to buy

    And I have entered my card details

  Scenario: Credit card number too short

    When I enter a card number that's only 15 digits long

    And all the other details are correct

    And I submit the form

    Then the form should be redisplayed

    And I should see a message advising me of the correct number of digits

Here Feature and Background apply to multiple scenarios. This one scenario addresses my friend's problem. However, what is really cool about this is that everything typed above goes into a .feature folder. This plain english language can be backed up by code, which in our case is a Ruby script. This means that cucumber is both a helpful tool to those who code and serves as a meeting table between the coder and the "layman."

     

Using Selenium-Webdriver to have the surfing done for you

So now that we have functions for scraping data from sites, let's imagine we had to do it for a whole bunch of site. In the case of webpagetest.org, simply putting this in any array won't work. Whenever you go to webpagetest.org, you're entering a URL that, when submitted, changes the path of the webpage. So you start on webpagetest.org, but entering in your site changes the URL from "http://www.webpagetest.org" to "http://www.webpagetest.org/result." This means in order to do a simple automation of our test, we need to have something do the web surfing for us.  Before that, however, I have to make a confession: if you use this test to do an large amount of testing, as I did, you're not being a good web citizen. Ensure that you only do this for a small batch of URL's (like 100 at the MOST). Ask for an API. Be wise, unlike myself.

Anyways, there is a tool to do our aforementioned task. Its called Selenium-Webdriver and it made me giggle the first time I used it! There are some awesome and surprisingly intuitive methods with this gem. First to start off this program, you'll have to require the gem. Now let's say we wanted to create a real simple demo program that opened up the browser, navigated to google, and search google for a something (I'm picking Star Trek Beyond Trailers). That's relatively simple and we can draw on some skills we learned from Nokogiri.

The only way a computer is going to know what a google search bar is, is through code. For us, that means HTML and CSS id's. To find these, we right click and click inspect elements. Luckily most element inspectors make finding elements in code easy. For the google search bar the HTML ID is "gs_lc0." Now for the code:



Firstly, we call on Selenium-Webdriver and this takes an argument for which browser you'll be using. So far, I've only tried this on Firefox and Chrome. Using Firefox is extremely easy, so I'd recommend that. Next you're going to want to go to google. The ".navigate.to" method does just that, it navigates to a page. Now, we want to do something to the page, namely enter a value into the google search bar and submit it to search. As mentioned earlier, the google search bar id is "gs_lc0." The find_element method finds the element with your two given arguments. Using ID is the most specific as each ID is unique to the item, "gs_lc0" will only be used for the search bar.

So now your computer has identified the search bar, what do we do? The same things we do as human beings, we just don't really think about it! We click on the search bar, type some stuff, and click or press enter. We just have to tell Selenium_Webdriver to do that now. For this, we use ".click" which prompts the search bar with a click. Then we type something out with the "send_keys" method which takes a string value argument. Finally we wrap it up with ."submit," which is our enter key, making Google search for us Star Trek Beyond Trailers. Which surely to be a great movie!

Data Scraping with Nokogiri

Imagine that while working for testsite, you are asked to create an automated test that will audit one of our page's performance metrics. After a bunch of research, you find that your best bet for gathering website performance metrics is from a handy website called: webpagetest.org. Check it out, you can enter any URL and get all kinds of data about said site's performance without a whole lot of time spent.

In order to complete this project, we need two things:

1. Something that takes the performance data printed out on the site and turns into something you can manipulate. This process is called "data scraping" and it is what this blog post is all about!
2. Something, which I will cover later, that navigates to the site for us, inputs the data for us, and runs the test for us.

Luckily Ruby has an awesome gem called Nokogiri, which turns a website into a Ruby object. Let me show you a manual way of testing a webpage. I'm a Star Trek fan, so I'm going to test "http://www.trektoday.com." I enter my URL and then I'll get a nice chart with all kinds of useful data like so:

Let's say I wanted to get that first value "14.630s," in the "Load Time" column and "First View" row. How in the world would you do that? You have to delve into the front-end coding languages of the website: HTML and CSS. HTML shows us the plain text of the site and CSS shows us the style. HTML and CSS often times have specific identifiers that let us know what each text is. Since we're scraping a very specific data we have to use a very specific identifier for Nokogiri: 'id.' In HTML and CSS 'id' is specific to one thing and one thing only. It can't be used anywhere else. So how do we get "14.630s" id? 

First highlight "14.630s" and right click. In Google Chrome, you'll see an option "inspect." This opens up the HTML and CSS we'll use to scrape our data. One nice thing is that the exact element for "14.630s" should already be highlighted for us! Here's what it will look like: 

We now have an important CSS 'id' which equals "fvLoadTime" and this gives us the "14.630s" we're looking for. Now we can write our code. We need to require 'open-uri' as well as 'nokogiri' since we will be working with URL's.

Now we can define the page we'll be using to scrape data from, which will be the results page (not simply webpagetest.org). We'll call on Nokogiri and its HTML library and assign it to a variable to make it easier to use for future use. Nokogiri has the method ".at_css" which accesses our specific HTML id. The above code yields these results:

The first result simply spits out the HTML code we pulled our data from. The second result will (though it did not in the case) pull out any input text between the <td></td> tags (meaning we will \n's as well as the text we want). By adding .strip to data.at_css('fvLoadTime').text, we actually are able to get just the data we want. Our function has returned to us a string which will be extremely useful to us. Instead of copying and pasting all of our data to some document, we now have Ruby strings results we can work with. Now to automate the actions in the browsers! Stay tuned!

What's up with those dollar signs?!

So you may have noticed that I put dollar signs next to my variables in my code. Like here:





















The reason behind these dollars signs is simple and might save you some forehead wrinkles! Lets look at my gmail function. My goal was to make my program send an email to whatever input a user would enter. However I couldn't simply use a variable called "email" because of one computer science principle: global and local variables. Within functions, such as our email function, variables are local to the function or to say only exist within the function. This also means, I can't call upon another variable to use in the function. Now this is easily fixable! By adding a "$" to our variable name, the variable become global. This means that these variables permeate the script and can be used anywhere! So by adding $ to "email," I can ask a question that receives input outside the function and have the input used within the function! However, be aware variables "email" and "$email" are not equal! If these two are defined within a program, they will have two separate values.

Making a web crawler with Anemone

In case you're wondering how to pronounce "Anemone," I've got this useful YouTube video.

So, let's put ourselves in a real-life possible business situation: we work for a company that manages an online retail website. This website works for buyers and sellers and sells just about everything. Now our boss wants us to make sure that all our products and departments will show up without error, specifically no 400 or 500 level errors. Well, we already have a 404 finder, but we don't have the each page name. You could check this simply, but also inefficiently by having a team go through every single page, copy and paste into the .txt file our 404 finder uses, and then run the program. However, with a little understanding of how our webpage (and all webpages work) we can do this very efficiently!

Our site is called "testsite." Our homepage where you can see all the departments and customer options is "http://www.testsite.com." Let's say you wanted to look at something in the outdoors section, you click on a button or link and you'll be taken to "http://www.testsite.com/outdoors" and if you look at the hiking boots in the section you'll see "http:www.testsite.com/outdoors/hikingboots#123." Essentially, each link or button is another folder or branch on the tree in the website. Being that we sell a lot of stuff at testsite.com, there's going to be a ton of links! However, there's an age old tool that we can use to collect all the links to be used for testing: a web crawler.

Web crawlers, simply put, will index entire webpages based on certain user inputs. One important input is depth, which like it sounds, tests a page's "branches." Now, it is important to say here and now, that I am not yet at the level to create a webcrawler from scratch, but Anemone is a wonderful prepackaged little gem that makes this extremely easy.


















Before you start crawling, install the gem with terminal command "gem install anemone" and open a new script in your text editor. To start, you have to require the anemone gem with "require 'anemone.'" Now, if you check anemone's documentation it has this web crawler script prewritten for us. However what I want to concentrate on is :obey_robots_txt and threads part of Anemone.

1. If you go to any website and type in "/robots.txt" at the end of it, you'll receive two possible responses: a 200 (successful) response which indicates which, if any, webcrawlers are allowed to crawl the site, or a 404 (broken link) response. The 404 response means that the developers haven't said that crawling is disallowed. You only want to crawl websites that yield a 404 when you add robots.txt. This is formally called Robots Exclusion Protocol and you want to be polite in your early coding days. Luckily, Anemone has an intrinsic setting ":obey_robots_txt" that when set to "false" doesn't obey the REP, but is easily changed when the setting is "true." Simple enough!
   2. Multithreading is a programming tool that allows for processes to be completed simultaneously. Here is a really simple demonstration of the actual code you have to write in Ruby to do this. Essentially what multithreading does is give a single set of code to multiple processors at once. I will demonstrate the effectiveness of this timewise in a later post. Again, we are super lucky that Anemone is intrinsically multithreaded and you can use the ":threads => *num" to perform multithreading. However, you have to wise about your multithreading use: you can't have too many without actually slowing down your program. Anemone's documentation recommended 4 threads, so that's what I chose.

Getting a script to send an email (gmail)

OK, so I had a lot of trouble out of this and it took a couple days and a couple humble moments of asking a few people for help to get it right. How could I run a script, say the 404 finder, and have the results of a test sent directly to your user or whoever automatically? Luckily Ruby has multiple Gems to do this, but Gmail is a little different. Now the first thing we're going to do is login into your Google Account and allow less secure apps (https://support.google.com/accounts/answer/6010255?hl=en). Now we can code and that's the fun part!

The gem I am using for this project is the gmail gem which can be installed by "gem install gmail." I found a little piece of code in the documentation that allowed me to do exactly what I wanted aside from one little thing, but let me explain what code I have thus far.

So this should be pretty intuitive. The function takes certain inputs. Sender email with the password so the program can log into gmail. Then it will take the recipient email, a subject, and a text body. Of course, I had a problem I want my program to be fluid, I want to be able to ask the user "Who do you want to send it to?" and then use user input as the email. You can't just make a variable like "email" and equal it to gets.chomp and use it within a function. Variables within a function are local, so you have to make the variables we use global. You can do this very easily by adding a "$" to whatever you want to do. So instead of recipientEmail as our variable, we can use $recipientEmail. This means it is usable within the entire program. 

404 Finder. Possible User Stories

So we have a completed 404 Finder, but in my view it is only a skeleton project. What if the user accidentally enters a bunch of white space? What if the user doesn't input or incorrectly inputs the "http://" part? I'm going to answer each of these questions and more in this post.

User Story: Excessive Whitespace
Let's say our user enters too much whitespace by accident (or to test our script). What do we do? Honestly, I'm not sure how to handle whitespace in the middle of a URL. For instance, the user inputs "http://www.g o o g l e.com." I'm not sure how to handle that other than send it back to the user and tell them that it was malformed. However if the user inputs "     http://www.google.com    " I know how to handle that and its quite easy. Just run .strip for each item in your good URL array. This method removes trailing and leading whitespace. But this brings a bigger question: how do we know which URL's are good and which are poor input.

User Story: Run poor URL's without breaking the script!
So let's say we enter in a bad URL: "htp://www.goole.com." Our script is designed to get response codes! You can't get the response code of something that doesn't exist. We need to tell the computer to sort these out. Originally my instinct was to use .slice and sort through the array's first characters, for instance .slice(0,7). However, I discovered the wonderful regular expression and the website: regexpr.com. So I made a regular expression that found matches for "http://" and then sorted them. Now, whenever you have a match that is more than one character, you will be returned an integer 0. Whenever you have only one match, you will be returned an integer telling you which character in the string it matches. For instance if I were matching 'o' in the string 'dog', I would get back integer 1. However, if there are no matches, then I will get back a nil. This helps me sort my URL's. I create two arrays. One for malformed URL's and then one for the one's I'll be running. Here's a good demonstration of a regular expression sorting the URL's. Keep in mind, the regular expression is defined globally.











This checks to see if the regular expression has a match by testing for being returned a nil response. If it does, then it is malformed. If not, then the regular expression returned an integer and it matched. In this case it must return 0 or nil since I'm testing for more than one character. Now, you might be wondering why I'm typing "https?:\/\/" into my regular expression. Regular expressions are situated between two slash marks. One slash begins the regular expression and one ends it. This means, like using quotes within quotes in a string, that I must break slashes. So I use "\/" to mean "/". And the reason for the 'https?'? A user could enter in "https://www.amazon.com" and I should still be able to test for it. It's not wrong or malformed. So entering a 's?' along with the 'http' means that I can test for 'http' and 'https' with the 's' character being optional.


I hope this helps folks in some way and that I'm explaining it well! I work full time while learning to code, so if you see any errors or have any suggestions, let me know!

Making a 404 Finder. Part 3

So one thing to note about working in the law enforcement field is that we're late to everything and technology is one of the big ones. I use computer programs that look like they come out of the 1980's. One of my favorite coworkers is a 67 year old man. He gets so flustered around computers that he'll leave a job with cushy days off and daytime hours (rare in law enforcement) if it involves computers. I imagine that the client I'm making my 404 finder for is him. I want to make things as streamlined and easy as possible. As far as our project, what do we have so far? Well we have a script that reads a file, assigns the string data in the file to a variable in our program, and then splits each URL into its own data point on the assumption that our use will input items with a space between each item. 

Now we have to run through this array and find out the response code and then sort each URL into its respective group. For this we need to create some empty arrays which will be our groups.

Sweet! Now all we have to do is run our comparison and then use the .push method to push the items into their own arrays. Now, since the first language I learned was JavaScript, my first instinct was use a for loop. However, a .each is more appropriate iteration in Ruby. Not only is this more efficient, but it is also a lot easier than formatting a JavaScript for loop.








So this iterator will sort through your .txt file and tell what kind of link each link is! Now, you could consider yourself done, but that is entirely dependent on your user stories. I've got a few more ideas to make my code a little more friendly to people like my coworker.

Making a 404 Finder. Part 2

So in my last post, I walked you through the beginning steps of creating a 404 finder. What we discovered was that Ruby has a nice little class (Net::HTTP) and a method (.get_response) that when used returned to us, a string value indicating the URL's response code. I also laid out what we can now do with this response code. I chose (possibly due to a lack of experience) to use an if...else statement and I'm going to sort it into 6 classes:

1. Informational (100 code)
2. Successful (200 code) 
3.  Redirectional (300 code)
4. Client Error (400 code)
5. Server Error (500 code)
6. 404 Errors (404 code)

I can sort this for instance by saying if the response code is greater than or equal to 100 or less than 200 then it is informational. Or if it is greater than or equal to 400, not equal to 404, and less than 500 then it is a client error and will be sorted into that bunch. Now be careful, if you choose to sort more than that, you might run into trouble: the get_response() method returns a string. Remember that in Ruby, if a = '33' and b = '230,' Ruby will return a as larger when compared to b due to being a string. If you use a to_i method on both variables, you will get a correct comparison. 

Before we do that, we must go back to our original user story: take a .txt file filled with URL's and then run them through our program to find out which links are working and which ones are broken. So we need to a group of some string data's, which happen to be URL's, and then sort through them using our code and place them into their own groups. So, we will be working with array's. We will be putting our URL's into an unsorted arrays, run our code through the array, then place them into their own arrays.

The next part here is on you. Go ahead and create a .txt file and put some URL's of each type in there. Broken, successful, informational etc. Once you've done that, we need to figure out how to put that information into an array we can read. To do this we need to use the File class. The easiest way to do this is assign a variable to File.open('yourtxtfile.txt', 'r'). This tells the computer to: open the .txt file and then read (hence the 'r') it. However, that variable we just assigned is still a file type and we can't run any methods on it besides read and write. To check this you can always put a variable and .class next to it. The computer will tell what kind of data it is!

So we have a variable that is reading the data inside the file. If we assign that variable to another variable and use a .read method on it we'll have a string value to work with. Here's what your code should look like (I called my URL file ural.txt):



We're still not done. We now have one big long string with a bunch of URL's in it. How do we turn that into an array. One thing I remember from JavaScript that is also in Ruby is the .split() method. This takes an argument between the parenthesis that takes a string and splits it with whatever character you provide. Here's how it works:

So in my nonsensical string, the .split method with the argument 'g,' removed the 'g's and returned 5 strings in one array. This is quite useful. I'm counting on my user inputting the URL's into the file with one space in between each URL. So now, I can run .split(' ') on the string variable contents and I should return an array with each URL occupying its own space.

Now we can program our computer to check each URL's response code and we won't have input them manually! But that's a topic for a different day. 

Making a 404 finder. Part 1

Objective: I have been tasked to create a script that will determine whether URL is a broken link or not. This script will take a text (.txt) file with a list of URL's and check its status.

So, how do we do this? First we need to understand the internet and what a HTTP code really is. What is a 404 error? When you get on the internet you use a browser to interact with web pages. Your browser sends a request to a server which holds the contents of your intended web page. That server sends a response back. This response is key to our URL finder. Here are the HTTP responses in a nutshell (keyword: nutshell):
100 series code: Informational URL's
200 series code: Successful URL's
300 series code: Redirectional URL (Think typing Wikipedia.org instead of Wikipedia.com)
400 series code: Client Side error
500 series code: Server Side error

So, let's go back to one of our original questions: what is a 404 error? A 404 error is a client side error, or more specifically a response code sent to indicate that the link is not functional. The question is what can we type into our text editor to see get said response data and use it to determine whether something is a broken link or not?

Well you have to do a little research (which is what coding often times is). When you do that, you discover that Ruby has a wonderful class to manipulate HTTP. The class Net:HTTP has a method that can get us the response code of a specific URL. Firstly, you'll have to require the class by the command require 'net/http'.

Now we can start using some of the methods! Net::HTTP can send a request to get a response code through the code line: 'Net::HTTP.get_response(URI(#link))'. Try running the code yourself and you should get some type of code that falls between 100 and 500. Your output is a string. So for 'https://www.wikipedia.com,' you'll receive a string with the value of '200.' To make things we can assign this to a variable and convert it to an integer using .to_i and then sort it. We can use a case statement or a comparison operator to have our computer sort it for us.

Now that we have things sorted (pun) out and we know how to turn a website's response code into something we can manipulate, our work is pretty much laid out for us. What are the user stories? We can discuss this more in detail in our next part.

Terminal: Top to Bottom

I remember the first time I did a pair programming event and had to use a terminal for the first time... It seemed like a thousand different steps with no real logic. Whatever instructions I was given sounds like: "OK Matt, type 'bash dash m and a plus sign and the words cp dash r and the letter y.'" What is that?

However, it has become so intuitive to me now. Whenever I create a program, it is second nature to me to "subble" it (subl programname.rb or .js or .html whatever it is). In fact, my friend and I were working on a small side project this weekend in HTML and CSS. He's a very skilled programmer who learned HTML, CSS, and JavaScript through Free Code Camp. I highly admire FCC, but they teach you nothing about how to use the terminal. I was able to teach him a lot of what I've learned.

While it's cool to manipulate files and seem like a genius doing it, I must admit there is much for me to learn. I am nearing the end the Zed Shaw's Learn Ruby the Hard Way and I made it to exercise #50. The goal was to create a simple "Hello, World!" web application. I made the application and ran it with the ruby command in the terminal. My intuition, from doing mathematics, is to break it down. So I change directory (cd) all the way down to the specific file. While the program ran without a flaw, Zed was one step ahead of me (and probably a lot of other aspiring developers) and pointed out the mistake.

When you run something in the terminal, you need to run everything from the top down. My program worked because it simply did not have any other dependencies. Whenever it does have dependencies, you need to specify to run the program within a directory rather than changing the directory the computer is looking at. Whenever you use the cd command into a directory, that directory is like the computer's whole world at that moment. To illustrate, I will show you a tinker project I came up with. I tinker in order to understand truly what's going on under the hood of anything, including a computer! Keep in mind, I'm writing these commands for the Mac.

mkdir Tinkering
cd Tinkering
mkdir TinkerOne TinkerTwo
cd TinkerOne
subl tinker_requirement.rb

inside tinker_requirement.rb:
class Tinker
def Tinker.test
puts "See you need to run from within the parent directory!"
end
end
(Command + S to save)
in the terminal again:
cd
cd Tinkering
subl tinker_test.rb

inside tinker_test.rb:
require './TinkerOne/tinker_requirement'

Tinker.test
(Save)

Now, if you know ruby, you know you have one program that contains a class with the method called ".test" that will print the string above. Whenever I run the program tinker_test.rb, it calls the class and method from tinker_requirement.rb to print that string into my terminal. Now try this
cd Tinkering/TinkerTwo
ruby tinker_two

You'll get a "cannot load such file" error meaning the computer doesn't know what file you're referring to in the require statement. Therefore the class and the class method cannot be executed. However back out:
cd
cd Tinkering
ruby TinkerTwo/tinker_two.rb

What do you get? You get the string defined in the method of the class Tinker.

"See you need to run from within the parent directory!"

Hope this helps! I still have a lot to learn, but teaching helps one learn faster!
Matt

Why I'm Doing This

Truth be told, I never knew exactly what I wanted to do with my life until I discovered coding. I only knew that I wanted to do something that was valuable to others and I wanted to make a lot of money doing it. My family continually pushed me to enter into science and engineering, but I was scared because of the math involved. I did what all kids who are unsure of what path to take in college do, I majored in business and graduated in 2013 from Ohio University.

Since graduating, I've held a couple job titles while in search of something that would satisfy my two aforementioned criteria. I've worked for a freight company, braving the elements and working with hands. I moved up in that job, but the long hours and working conditions pushed me to find my path in life. I fooled around in nursing and didn't like it. I eventually retook some math and science classes and found that I really enjoy them and I'm good at it. I came to the decision that I would become some sort of engineer. I wasn't sure which kind, all I knew was that engineers help create products that people use on a daily basis to make life easier. I started taking engineering survey classes and found that I really enjoy working with software. Making a computer say "Hello, World!" is a pretty cool experience. Imagine what else you could do!

I found a new job working for my local Sheriff's Office in the jail and got so caught up in that job that I forgot about programming for a few months. While this job does afford me a lot of time to learn programming, the money I make isn't substantial enough to pay for my education and I have enough student loans, I don't want or need anymore of those. So, I am learning to code on my own. The great thing about this industry is that you can unlock the doors and get jobs without a degree! I've taken a few paths on this journey. I started with a textbook on Python and found myself ineffective. I ventured into Free Code Camp and learning JavaScript and even made a couple websites. This includes a Star Trek Random Quote Machine, a tribute page, and a personal portfolio. While, I learned a lot the community is a little lacking and no man is an island in the coding world. Luckily I found a coach and mentor, by the name of Josh Kemp. He and I are venturing through the world of Ruby with a goal of landing an entry-level position in four months that will eventually lead to me becoming a software developer by January 2017.

Of course, learning to code requires a significant amount of discipline that has led me to spend at least 3 hours a day coding. I've skipped going on dates, hanging out with friends, going to concerts, etc. to learn to code. People often inquire "why are you doing this?" I felt that I would share my reasons why:

1. Money. Boo. This isn't a good reason to do something and I think millennials get this. People of my generation are said to be intrinsically motivated (i.e. motivated primarily by factors other than money.) So I list all of my motivators that aren't money!

2. I love coding problems. It may be frustrating for others, but I enjoy the battle of exploiting a problem and finding a solution. When I was kid, I would play StarCraft religiously and I found that my best moments (the ones where I still watch the replays) were the long, drawn-out battles where I had to attack my opponent from every angle. You would test and test and test to exploit a weakness in the enemy defense and eventually overcome after hours of testing. In a weird sort of way, coding creates the same feeling. You get a problem, say reversing a string, and while you don't know everything you know where to look. You delve into the documentation, hitting stackoverflow, MDN, and a thousand other websites, and you try running the program. You fail and fail and fail until you get it right. There's no greater feeling than the moment when you finally get it right. That sense of accomplishment is unbeatable in my opinion. If you're like me, you'll laugh at yourself because you should've known to use camelCase.

3. Of course, doing those problems means that I can eventually create a working product. Whatever that is, I know that it is going to help someone. It could help businesspeople or the general public.

4. Being creative instead of a drone. Every job I've had, I've been a drone. This isn't always bad and it isn't bad for everyone, but I believe I was meant to do something else. After exploring some of the software companies around the area at events like PairColumbus or the Columbus Web Group, I found that these companies foster a work environment of creativity. The company that inspired me to learn to code, covermymeds, has pool tables, open floor plans, an on-staff chef, and a little brewery. Hearing people talk about the company, the teams they work with, and the products they create inspires a little healthy envy in me. I am motivated because I want to experience the same thing in my work.

5. "Your code was merged..." I love the coding community. It is full of people from all walks of life who aim to create. The best part is most people in this community realize that coding is hard! How many times have you done something that was difficult and failed only to hear "you're stupid!" or "come on, you should get this by now!" While the coding community has its harsher elements, most of the people I've met will meet you where you are and they're eager to help. Their vocabulary is often "instead of this, try this." They often times teach you socratically, which is the least insulting form of pedagogy.

Of course, there's more to the story with #5, which is the most important reason to me. A month after I typed my first "Hello,World!", I learned HTML and CSS. I was simply proficient, but when I went to PairColumbus at the beginning of February, my friend David wanted me to help him with the creation of a song lyrics app. He wanted song lyrics to appear on an element that looked like notebook paper. At first, my reaction was "uhhhh...." I didn't know where to start, so I started researching. Eventually I found the right code and modified it to make it look the way David wanted.

I felt that I hadn't contributed much, but I also know I am a beginner! However, I felt rather wonderful when I got an email from GitHub saying David had merged my code into his own. While I hadn't found the cure for cancer, I knew that David would be successful creating his app and once released, I would see a piece of my hard work along with everyone using the app.

So why am I doing this? I want to get paid doing something I love, like coding and creating! I know that if I continue on this path, my hard work will useful to someone else and help others and I'll make a fine living doing it. So, I finally found what I want to do with my life. I want to touch the-world.rb.