(Hopefully) the last of the bugs

The main final bugs I kept finding were the buttons reappearing at random times when the sidebar was reshown. On Friday night when recording a video demo just in case, I realized it is because I never changed the criteria for what button should be shown when in the show sidebar function. I had them set it the other functions but decided to apparently just show them all no matter what in the show sidebar function.

There is one other bug that I know of that after you zoom in and then hide the sidebar the key flies off the left side of the screen. I think it has something to do with the margins or padding, but it only happens after zooming in. I have no clue how to fix this as I think it is something with the Google controls padding.

Presentation

The majority of my week was spent creating and perfecting my presentation. I presented it a few different times to different people and in very different stages of development and got a lot of helpful feedback.

Overall, I think my presentation went really well, there were a few mistakes and a question that I wish I would have answered better, but no presentation will ever be perfect.

You can view a copy of my presentation here:

One of my favorite moments of the week came on Friday night when a few of us were practicing for presentations. We were listening to music and laughing and just having a good time. Sometimes as a CS major and especially over the course of this project, my life was incredibly stressful, but the friendships, the learning, and the fun times make it all worth it.

I am so thankful for everyone who has supported me over the past four years and especially those who were able to be there for my presentation Saturday including my parents (who I didn’t get a picture with), my roommates (Nicole, Katie, and Lauren), Krissy, Nick, Annicka, and Dr. McVey-Pankratz.

It’s sad to see this chapter closing so quickly, but I’m ready for the next and to see what life has in store for me!

Fixing the Circles

Although I am presenting the original version that uses the state outlines, I wanted to fix up the circle version a bit to present it as an extension for the person who would have it next. After a lot of trial and error, I was able to create the circles in an array and that allowed me to call them one by one to update them.

I am going to spend a few hours this weekend trying to create a few new points using weighted averages. To be honest, I’m not super hopeful that it will work, but I’m going to try.

Info Box Errors

The main errors this week focused on my info div and the errors that I had with it.

I now have it so when you click the info button when the div is already open, it closes it. There is also a hide button on the bottom, but each option does the same thing.

When the animation is done and the user wants to check the info box again, I noticed the bottom was covered up by the key. Because of this, when the info div is open I hid the key and when the info div is closed I have the key reappear.

The last “bug” I fixed was disabling the play button when the info div was open. I think I could have left it the way it was, but I was worried about the user accidentally hitting play before reading and picking all of the options.

In the text of the info box div, I mention what the current start and end years are set to, but I grabbed that data when the box was open. If the user changed it while the box was open, it didn’t update. I was able to add a listener to the start input box to update the text. I did the same for the end text and now when the start or end year are changed, the text updates.

If the year is invalid, the text changes to “INVALID”. I wanted to warn the user when they clicked out of the box, but because it was a listener, an alert would continually appear and you couldn’t get out of it to change the value. You cannot run the program with invalid data, so I don’t think this is the worst solution.

A Silly Mistake

One of my friends, Leah, was able to find an error I hadn’t thought of and it led to me figuring out that the way I was doing my math was incorrect. She let me know that if you do the same start and end year the program is confused and tries running the default. This was because I was making a simple math mistake and my count was one off. In order to show the data for each year between 1900 and 2000, there are 101 years because you have to include both 1900 and 2000. So after figuring that out, each year should be showing correctly.

In the case of starting and ending on the same year, the count I created was zero, but it needed to be one. Once I added one to the final count, it now only displays the one year.

I also realized that the program was showing the text a year off. So if the program started in 1900, the text showed that the data was instead for 1901. This was because I was updating the current year before showing the text.

The next week is going to be focused on creating and practicing my presentation, fixing any other small errors that come up, cleaning up my code, and preparing for defenses. I also want to record a video for my demo just in case something would happen to compsci04.

I made a rough version of this earlier this week by finding the center latitude and longitude of each state and drawing a circle with the corresponding color.

You can see the rough version here: https://compsci04.snc.edu/cs460/2022/nooycr/circles/

There are a few bugs with this version, the main being I realized for every new year circles are being drawn again on top of the old ones. This means the old color is drawn on top of the new one and after about 20 years the program glitches out really bad.

I think I figured out a solution by creating unique names for each circle when they are created and updating them based on that new name in a loop. However, I didn’t have the time yet to sit down to fix it, so I am hoping to do that tomorrow.

The next step in this process is to try to create data for more circles. I’m not really sure how to accomplish this instead of just doing it manually which is obviously not the idea and way too much work.

I like the concept of having circles, but if I had to present today I would present the state map. With only two weeks left before the presentations, I don’t know if I can make this new concept work in time.

Some other errors with the old version (and the new version) still exist, so I want to fix those this week. One is clicking the info button if the info div is displayed does not make the div disappear which a few test users have remarked that they hoped it would. Another is adding a zoom-out lock and pan coordinate locks so they can’t look outside the US and Canada.

Overall, I think I want to focus these last two weeks on making my site as perfect as possible using the state coloring and creating my presentation. I have the new example as an idea to show what it would have looked like on my site, but I think it is what I would like for the next student who has a similar project to focus on.

New Features

After walkthroughs, there were a few new features I wanted to add. I also asked the class for more feedback mainly on some user interface issues I was having and fixed some things based on that conversation as well.

Adding Temperature Change Each Year

The main feature I added was a temperature change each year option on the dropdown. This takes the current year’s data and subtracts the previous year’s data to show the difference from one year to the next. I also updated the temperature change over time option to use the start year the user entered as a baseline. Instead of being allowed to start at 1743, I instead restricted the two temperature options to start at 1850, when every state (except Hawaii) was guaranteed to have data.

Rewind 5 Years

Another feature I added was when the animation is paused, the user can rewind five years. The button only appears when the animation is paused and then disappears again when resumed or stopped. This was a suggestion from walkthroughs that I really appreciated and agreed with and was excited when I got it working.

Adding an info button and option

Lastly, I added an info button and created a new div that explained the options much better. This also takes up the majority of the screen and allows the user to read it much better than the alert box I had previously. You can see what this looks like in the image above.

The Little Things

Throughout the rest of the week, I fixed a wide variety of little errors I encountered, some of which led to another list of errors.

These errors I was able to fix include:

1. The coloring of buttons when inactive
2. The color did not change when rewind was pressed
3. Values of the input boxes do not show start and end years when animation is running
4. Allowing the animation to run with bad year data
5. Not having the Fahrenheit equivalents to the keys to make it easier to read
6. The spacing of the sidebar so five buttons and five input boxes can fit
7. Not allowing the “Stop” button to clear the data when the data animation has been completed
8. Correcting the order of subtraction for the change over time data (I had it backward)
9. The key trying to show without the play button being pressed (it never loaded an image)
10. The current year text was too far from the animation (I decided to center it on the top of the screen)

And probably the biggest error I had to deal with this week that I think I might have finally figured out is the spacing of the key on the bottom of the screen. I was trying to use padding and forced margins to make the key in the correct spot. However, I was using the same control location as the buttons on the lower left, so when the user zoomed in, the buttons jumped up. I hid the Keyboard Shortcuts and Terms of Use from the bottom right corner and was able to anchor the key from the bottom right instead of the bottom left. I have to fix the margin on the larger key when the sidebar is hidden, but the smaller main one should be fixed. I want to do some more testing on different size screens to confirm, but it seemed to be working on mine.

Next Steps

The major requirement I have not met yet is to implement data smoothing. However, the more I research this, the more I do not know if this is possible in Google Maps. The first step I want to do is draw a circle of color for each state and have it change based on the data.

Even if I cannot figure this out, I think my project is extremely close if not fully ready for presentations in three weeks! I am hoping to take a bit of my Easter break to start creating my presentation.

With the help of Dr. McVey, I was also able to fix my data passing issue, which is huge!

Thursday was also walkthrough presentations and if you want to hear some of the suggestions I received you can check out my post here.

Moving the Current Year

Surprisingly, the hardest cosmetic update of the page this week was trying to move around the Current Year text box. I kept having issues with CSS not updating and multiple rules being applied to the same text box which pushed it down under part of the input box. I finally got it working and now it lives next to the map controls on the top left of the page above the input box.

Hiding the Sidebar

A new feature I added this week was a button that hides the sidebar to give the user more real estate to view the map. Talking to a few people, the sidebar does take up a lot of page space, but I also think it needs that much for all the options it houses. So, adding the option to hide it I thought was a win-win for people who want to see more of the map or those who want to see all the options.

Creating Keys

I was also able to create the needed keys to add to the bottom of the screen for the user to understand what the colors mean. The temperatures right now are in Celcius, so I want to try to add Fahrenheit equivalents under. They might not line up perfectly, but I think it will make it easier to understand the coloring.

Data Sending Issue

The reason the data was not coming back from the retrieve.php file is that I was never catching the request. So after adding a req.onreadystatechange function in the play button, the data is now able to be split into a data array to display.

A Few Current Errors

To account for user error, I made certain buttons inactive depending on where in the lifecycle you are. However, it is hard to tell whether they are inactive currently. I added a CSS property to turn the text white because I was planning to change the opacity of the button, but after thinking about it, I think I am just going to hide the buttons you can’t use. I am also trying to make the options uneditable when the animation is running, which is working fine for the drop-down boxes, but I cannot get it to work for the input text boxes. They appear as a different color and the text is gray instead of black and just hard to read. If I can’t figure it out this weekend, I am hoping to enlist the help of Annicka to look it over.

Because of the input box errors, you can’t currently restart the animation after stopping or it ending it which is a huge problem. I am hoping that once I figure out the inactive issues, it goes back to running multiple times.

Another major error right now that is occurring is that you can enter bad data for the year (ex. 120 for the start year) and it yells at you but tries to run anyway. I return out of the function, but it seems to still want to loop. I am thinking if I call the stop function before returning, that might do the trick.

There is also one really weird bug that I am not sure how to solve. If you hide the sidebar before starting the animation and zoom in on the map, when you try to show the map again, the play and pause buttons disappear. I might try to lock the zoom until the animation starts, but I am not sure yet.

What’s the Plan?

With only four weeks until presentations, I am working on prioritizing what needs to be done. The main focus is fixing the current major bugs that exist, which I’m hoping to figure out this weekend. I also want to add a few features including an option to compare each year to the previous, a learn more button that explains the options, and a back 5 years button when the data is paused. Next weekend, I want to focus on trying to figure out the data smoothing algorithm. In a dream world, I would be able to get this working right around Easter and then the rest of the time would be allotted to having as many people as I can testing it on different devices and fixing bugs that come up.

I would also like to fix the mobile version, but that is lower on my priority list because the more I think about it, this is really meant to be a desktop application.

New Additions

There are a few good suggestions that were made that I’m excited to try to add.

The first addition is an option where the coloring of each state is based on the year directly previous. Personally, I would never use this feature because I’m not sure how much info it would tell you, but I think it would be a good option.

Next, I want to change the option for “change over time” to use a baseline of whatever the starting year is. Right now, it uses a baseline of 1850 automatically, but I am hoping to use the user’s input.

I also want to add a “Rewind 5 Years” button up next to the current year on the top left. This would only appear when the map is paused.

There was also a suggestion to try and make the options faster, but I am not sure how much faster I want to make it. I am going to continue to play with it to see how fast the data can come back without breaking it.

I am excited to add these new features over the next couple weeks before presentations which are a month away!

The Progress

The first win of the week was being able to see that the data was being sorted through and encoded correctly after the help of Dr. McVey. Knowing that the data was being read through correctly was very promising! Secondly, I was able to get the states to fill with their individual colors this week. It took a lot of debugging and finding some rookie mistakes, but I was able to make it work in the end.

For the longest time, Canada did not want to change colors and kept coming up as undefined. I realized this is because when I combined the two GeoJSON files, the STATE_NAME property that was coloring the US states didn’t exist in Canada, so I had to make sure to add those into the property section. It still didn’t seem to be working, but after sending it to someone else to check it out and it working for them, I realized I had not closed out of my browser in too long, and the cache needed to be cleared.

After getting the color to work for the year 1950, I added a few more years of data and was able to get a slight animation working for the six years that I have. I am incredibly hopeful that once my data issues are solved (see below), I will be able to have a working animation in no time. Right now I am planning to go and fetch the new data array each time, which I know is not efficient, but it is a start. Eventually, I am hoping to pull around 5-10 years at a time and send the next while the animation is running.

Overall, a lot of the week was learning how to better debug and use the console, and thanks to Dr. McVey and Nick, I have a much better handle on using JavaScript and PHP.

The Issues

Although the data is encoding correctly, the data array is not correctly getting sent back to the JavaScript file, which means none of the colorings can be correctly applied. Because of this issue, the data right now is hardcoded into the JavaScript files for the years 1950-1955. Most of the examples I was looking at had both the PHP and JavaScript code in one file, so I might have to go that route and combine two of the files. Once this issue is fixed, I have hope that the animation at least for the Summer Temperature Data would work for any entered years.

After debugging for a bit too long, I was realizing that the index value of the color array was always returning 45 or 47. This was because I was using the Winter data array instead of the Summer data which meant almost every temperature was at the top end of the range. To make it work, I hardcoded in using the Summer data, but have to work on getting the data from both drop-down boxes and setting the data array correctly.

What’s Next?

Besides working to fix the few issues I have been encountering this week, I also want to work on creating the key for each data set so the user can understand what the colors mean. The current year right now is on the right side above the zoom, but I don’t love that location, so I want to try moving it up to the top left next to where the base map options are.

The other major project of this weekend and next week is to create a presentation and prepare for walkthroughs that are occurring next Thursday, the 31st.

The main thing I worked with this week was Jared’s code and learning what each section did. However, my input files looked quite different than his, so I had to change the way the file is read in. Right now I am reading in one line at a time from a CSV and checking the first value for the year. If the year matches, I have the line put into an array. The main issue I think I’m having is that I can’t figure out how to send the array back to the JavaScript file once it is filled. This means when the JavaScript file tries to color each state, it has no data to read in from.

Right now, I am using Jared’s code as a base, but I have heavily modified it by removing a lot of features that he decided were a good idea for his project. I also changed things like coloring and font choices. The two main features I changed (besides the data reading) were the state outlines and options in the sidebar.

To add the state outlines, I had to find or create a GeoJSON file. After searching for a long time online, I wasn’t really able to find one that had the states of both the US and Canada. Because of this, I tried to load two separate files. When I had just the US loaded, it worked totally fine, but when I tried to add in Canada, random state outlines disappeared. It took me the longest time to figure out what was happening, but I figured out that the US and Canada states both had the same IDs, so when the Canada file loaded, it wrote over some of the US state outlines. So, to fix the problem, I combined the two files and had to update both the ObjectId and Id of the Canada states for them all to appear.

I also created a color code for both the change over time and average temperature options that I think will be able to show a good variety of changes even if the temperature doesn’t change much.

Overall, after fixing up the code and making my adjustments, my project currently looks like this:

You can view it on compsci04 as well, but as I continue to make updates the site will update as well.

I haven’t fully settled on the base map option, but Google Maps has a few other options I am going to play around with.

What’s Next?

The next step I want to work on is to have data for one year show up with the correct coloring in the states and then eventually create the animation aspect. So over the early parts of next week, I want to make sure that I can at least get the coloring part done. For sure, I want to get a basic animation working before walk-throughs happen in a little less than two weeks.

After I can get the animation working, the next step I want to work on is the data smoothing between the center of each state. Also, if the feature exists, I would be interested in adding a time slider instead of just the start and end year options for the animation.

Looking forward, some features I would like to add if there is enough time would be adding a button to hide the sidebar if the user wanted, adding default values so you don’t have to enter data every time, and creating a mobile version of the site for users that is a portrait layout instead of the landscape layout for desktop.

So overall, I didn’t have the incredibly productive week that I was hoping for, but I was slightly expecting that. I think I am fairly close to getting the coloring to appear, but adding the animation I think is going to be a bigger project.

Original Data

The data set I am using was found on a website called Kaggle. It is titled, “Climate Change: Earth Surface Temperature Data” by Berkeley Earth. It has four main Global Land Temperature Excel files: by city, country, major city, and by state. The one I decided to use for my project is Global Land Temperatures By State. This had the most widespread data for both the United States and Canada where I want to focus my initial efforts.

The inital data set looks something like this and spans close to 650,000 rows of data. I am only looking at the US and Canada data, but it also includes data from places including Australia, Russia, Brazil, China, and India.

Formatting

I did not love the format of the original data, as I thought trying to read in data similar to this line by line would take forever for the program to compute. So instead, I created a grid of data with each location being a separate row and each year being a separate column.

Eventually, I want my project to focus on four different sets of data. I will have an option to view the map showing the average temperature, and a set focused on showing change over time for a particular region. Then, for each of those options, I will have a spring temperature option using July data and a winter temperature option using January data.

So overall there were four data sets to create:

1. Average Summer Temp
2. Summer Change Over Time
3. Average Winter Temp
4. Winter Change Over Time

To create both of these spreadsheets, I had to use some magic to fix up the data. I knew the end goal was using a VLookup to pull the data, but I had to be able to find something that could be searched. To do this, I had to concatenate the year the data was from, the state, and if the data was from January (Winter) or July (Summer). This new string allowed me to pull the data all into one new sheet of data.

Creating this string was easier said than done. With the data I used, Excel wouldn’t recognize any of the dates before 1900 in a date format, meaning I couldn’t easily pull the year off. Instead I just pulled the first four digits off of the date from years before 1900. After the hodgepodge of formulas, I finally had the correct data to apply the VLookups on the main data sheets to create.

Replacing Unfound Data

After I finished all the VLookups and got all of the data in the correct orientation, I saved each sheet as a CSV file. I wanted to be able to replace the “N/A” that the VLookups created if data couldn’t be found. I installed a program called “Modern CSV” to be able to do the replace. I don’t love the way Excel works with CSVs, so I was thankful to find a different program. The program allowed me to do a find and replace for the whole cell which was helpful because when searching for 0 to replace, all of the years also wanted to be replaced. After all the replacements, the CSV looked like this:

Converting to JSON

The last step that I did this week was making each CSV into a JSON file for easier reading on the program. However, there are over 150 years of data, so reading in the entire file might make for way too long of a load time. Instead, I want to try to be able to only read in a few years of data at a time and continue to load while also showing the start of the animation. Because of that, I am thinking I orientated my JSON files wrong. Currently, the file is based on location, but I think I may try to flip it so the file is based on year.

The next step is to try to read in some data and create at least a starting screen of color, but eventually somewhat of an animation. I am planning to take a deep dive into Jared’s code to see if I can understand how his works. If time allows I also want to start creating my options screen and read in data from the user.

After following this Simple Map tutorial, I was able to get a map to load using JavaScript and HTML. I changed the center of the map and the zoom to focus on North America because that is where the data I am starting with is located.

You can view the web map here: https://compsci04.snc.edu/cs460/2022/nooycr/simple-map/

I met with Dr. McVey Wednesday afternoon and have some solid steps to work forward with. The first one is to format the data sets that I have found to make it easier to read in as a JSON file and remove the rest of the extra data I won’t be using. Next, we looked briefly through a past student’s code that had a similar project a few years ago. So looking through that, I feel like I have a great jumping-off point to start working through code over the next few weeks.

So overall, it was not a super productive week because of being stuck figuring out how to get an API key. But, this weekend I am going to work on formatting the data using VLookups and then looking at possibilities to change the base map because I don’t love the current one.